Detecting cognitive impairment in cerebrovascular disease using gait, dual tasks, and machine learning

Abstract

Background

Cognitive impairment is common after a stroke, but it can often go undetected. In this study, we investigated whether using gait and dual tasks could help detect cognitive impairment after stroke.

Methods

We analyzed gait and neuropsychological data from 47 participants who were part of the Ontario Neurodegenerative Disease Research Initiative. Based on neuropsychological criteria, participants were categorized as impaired (n = 29) or cognitively normal (n = 18). Nested cross-validation was used for model training, hyperparameter tuning, and evaluation. Grid search with cross-validation was used to optimize the hyperparameters of a set of feature selectors and classifiers. Different gait tests were assessed separately.

Results

The best classification performance was achieved using a comprehensive set of gait metrics, measured by the electronic walkway, that included dual-task costs while performing subtractions by ones. Using a Support Vector Machine (SVM), we could achieve a sensitivity of 96.6%, and a specificity of 61.1%. An optimized threshold of 27 in the Montreal Cognitive Assessment (MoCA) revealed lower classification performance than the gait metrics, although differences in classification results were not significant. Combining the classifications provided by MoCA with those provided by gait metrics in a majority voting approach resulted in a higher specificity of 72.2%, and a high sensitivity of 93.1%.

Conclusions

Our results suggest that gait analysis can be a useful tool for detecting cognitive impairment in patients with cerebrovascular disease, serving as a suitable alternative or complement to MoCA in the screening for cognitive impairment.

Background

Cognitive impairment (CI) due to cerebrovascular disease (CVD), also known as vascular cognitive impairment (VCI) is common after stroke. It affects mostly attention and executive functions [1, 2], ranging from mild to major CI or dementia [1, 3, 4]. Vascular dementia, which is the second most common cause of dementia after Alzheimer’s disease [1, 5, 6], can significantly impact patients’ quality of life and independence [5, 7], being associated with increased mortality, disability, and institutionalization [8].

Despite significant progress in stroke care in recent decades, CI after stroke remains a common and underdiagnosed problem [7, 9, 10]. Studies estimate that 30–50% of stroke survivors experience CI [7], but recent systematic reviews suggest that the prevalence may be even higher, with more than half of stroke survivors experiencing some degree of CI [11]. In the first year after a stroke, the prevalence of mild CI ranges from 17.5% to 54.9%, with a pooled prevalence of 38% [12]. As the population ages and the number of stroke survivors increases, effective management of post-stroke CI will become increasingly important for both patients and caregivers [9].

The first step in the management of CI is diagnosis. The gold standard for diagnosis is a comprehensive neuropsychological examination [9] that is not feasible at a population level. In practice, a two-step approach is advised, starting with a primary screening test such as the Montreal Cognitive Assessment (MoCA) [4, 8]. While MoCA is commonly used for this purpose, its low specificity, and moderate to good sensitivity for the diagnosis of post-stroke CI coupled with a lack of standardization still limit its use [6, 9].

Researchers have been exploring alternative methods to improve the screening of CI. Gait analysis, for instance, was shown to distinguish normal aging from mild CI in non-stroke patients [13]. Further research showed that gait performance is affected by cognitive load in people with CI due to cognitive-motor, or, dual-task, interference [14, 15]. A recent meta-analysis showed that dual-task gait characteristics allow better differentiation between groups, with an increased sensitivity for the detection of mild CI [13].

Because attention is often impaired, patients with VCI may struggle with multitasking [9]. Studies suggest that performance on dual tasks can be compromised in patients with VCI [16], and suggest that gait assessment under dual-task conditions could be a useful tool for early detection of VCI [17]. Dual-task assessment is attractive because it is less influenced by educational level, is practical, fast, and easy to administer in clinical practice [18], and may be a good alternative to screening tests like MoCA.

Digital biomarker technologies, often combined with machine learning-based predictive models, are increasingly explored for early detection of CI in community settings [19]. Support Vector Machines (SVMs) are often used for this purpose by analyzing gait characteristics [19]. For example, Boettcher et al. [20] used dual-task gait data collected from a computerized walkway to differentiate subjects with mild CI from healthy individuals, achieving an accuracy of 77.2%. Aoki et al. [18] used the Microsoft Kinect sensor to capture dual-task gait, achieving an AUC of 74.7% in differentiating subjects with different levels of cognitive performance, as measured by the Mini-Mental State Examination (MMSE). Ghoraani et al. [21] employed an SVM in a one-vs-one manner with a majority vote to differentiate healthy subjects, older adults with mild CI, and older adults with Alzheimer’s disease. An accuracy of 86.0% was achieved in the differentiation of cognitively impaired and healthy subjects using gait features extracted from an electronic walkway. Shahzad et al. [22] used an inertial sensor-based gait analysis and a machine learning framework to distinguish mild CI due to Alzheimer’s disease from healthy subjects, finding that dual-task walking provided a better distinction than single-task, with a classification accuracy of 70.0%. Finally, Jung et al. [23] used sequential gait characteristics extracted from inertial sensors coupled with a long short-term memory network to categorize community-dwelling older adults into three groups based on their scores of the MMSE; they achieved improved F1 scores of 97.4% using temporal gait features extracted from usual and fast pace walking trials. Taken together, these studies suggest that machine learning and gait assessments have the potential to be objective tools for cognitive screening that do not heavily rely on cognitive testing.

Although some research has investigated the use of gait characteristics to detect CI, it is unclear whether machine learning and gait assessments can effectively distinguish CI from cognitively normal in post-stroke patients. In this study, we aim to objectively evaluate the clinical utility of gait in classifying CI in post-stroke patients using a machine learning-based approach. We used as input data from different gait tests, including dual tasks and fast walking conditions. We also tested multiple machine learning classifiers and two different gait assessment technologies that provided varying levels of detail about gait metrics. Finally, we objectively compared the classification results with those obtained using MoCA and proposed a majority voting approach to combine MoCA with gait. We hypothesized that: i) machine learning models based on gait metrics differentiate VCI from healthy individuals with a performance comparable to MoCA; ii) the dual-tasks provide better classification results than the single-tasks; iii) a more comprehensive analysis of gait allows better differentiation of the groups than simpler gait metrics; and iv) the combination of MoCA with gait metrics leads to better performances than using each modality alone.

Materials and methods

Dataset

The data used in this study were obtained from the Ontario Neurodegenerative Disease Research Initiative (ONDRI) [24, 25]. ONDRI is a multi-site prospective cohort study investigating Alzheimer’s dementia and mild CI (AD and MCI), amyotrophic lateral sclerosis (ALS), frontotemporal lobar degeneration (FTD), Parkinson’s disease (PD), and cerebrovascular disease (CVD). The ONDRI protocol includes assessments for genomics, neuroimaging, ocular function, gait and balance, and neuropsychological testing. In the present study, we have only used the baseline data from the cerebrovascular disease (CVD) cohort, including demographic, clinical, neuropsychological, and gait data.

Participants in the CVD group experienced a mild to moderate ischemic (or silent) stroke event verified on neuroimaging 3 or more months before enrollment. They also met the following inclusion criteria: (a) age between 55 and 85 years old, (b) proficient in speaking and understanding English, (c) 8 or more years of formal education, (d) mild-moderate stroke severity defined by scores 0–3 in the modified Rankin scale, (e) a MoCA score of at least 18. Exclusion criteria included vascular cause of symptoms, large cortical strokes, severe CI, aphasia, history of dementia prior to the stroke, inability to write, and/or severe functional disability limiting ability to perform assessments [24, 26, 27].

The current study received approval from the Ethics Committee of the Faculty of Medicine of the University of Porto (51/CEFMUP/2022). Access to the ONDRI Foundational Study Data was granted after approval by Brain-CODE’s Data Access Committee [28].

Assessments

Demographic and clinical data

The dataset included demographic data, such as age, education (reported in years), height, weight, and leg length. Additionally, it included a rate of depression, as assessed by the Quick Inventory of Depressive Symptomatology (QIDS) [29].

Study partners rated the participant’s ability to function independently across eight instrumental activities of daily living (iADLs; i.e., telephone use, shopping, food preparation, housekeeping, laundering, use of transportation, medication management, and financial management) and six basic ADLs (i.e., feeding, dressing, grooming, ambulating, bathing, and toileting). The percent of independence on relevant items was provided for iADLs and ADLs.

MoCA

MoCA is a paper-and-pencil screening tool for CI [30]. It evaluates multiple cognitive domains, including, executive function, memory, language, visuospatial ability, orientation, attention, concentration, and working memory. The score of MoCA is corrected for low education (≤12 years) by adding an extra point, having a maximum score of 30 points [30]. With a high sensitivity of 83–97%, MoCA is currently the preferred tool for the screening of CI at primary care level [31]. The cutoff of <26 is commonly used to detect CI [8, 27, 30].

Neuropsychological assessment

The neuropsychological assessment consisted of a standardized battery administered to all participants in the ONDRI study. The ONDRI study followed standard quality assurance and quality control procedures, as described in [32], ensuring the rigor and accuracy of the data.

To characterize areas of CI in individual participants, the tests were categorized into five cognitive domains, as proposed by [33] after a consensus agreement among the ONDRI Clinical Neuropsychologists. In addition to raw scores from the neuropsychological tests, the ONDRI dataset provided standard scores (z-scores, t-scores, percents, or scaled scores) based on published normative data (education- and/or age-adjusted). Normative values were not available in two of the considered tests (BDAE: Semantic probe and BVMT-R: Copy trial) and they were, thus, excluded from the analysis. The resulting cognitive domains and associated test measures are shown in Table 1.

Table 1 Neuropsychological assessment

We categorized participants as impaired or as cognitively normal using the comprehensive criteria proposed by [34]. According to this criteria, individuals were classified as cognitively normal if, at most, performance on one measure within one or two cognitive domains fell more than 1 SD below age/education-appropriate norms [34].

Before applying the comprehensive criteria, we filled in missing values using k-Nearest Neighbors (KNN). Missing neuropsychology data were imputed using the mean value from the 5 nearest neighbors weighted by the inverse of their Euclidean distance. Since KNN is based on distances, before imputation, we standardized the data, i.e. removed their mean and scaled to unit variance, to generate unbiased estimations of missing values [35]. After imputation, the values were transformed to the original scale.

Gait

Quantitative gait parameters were assessed using wearable inertial sensors (Gulf Coast Inc.; Shimmer Inc.) worn bilaterally on the ankles and at the hip. In two study sites, gait performance was additionally assessed using electronic walkway systems (GAITRite® or PKMas®) [24, 36]. To facilitate a comparison between inertial sensors and electronic walkway systems, we have only considered the participants who were assessed by the two systems.

Gait was assessed under three conditions, using the following order: (1) preferred walking speed, or single-task walking (SS), (2) dual-task walking (DT), and (3) fast walking (F). The secondary tasks were administered in the following order: (1) subtracting serial ones from 100 (DS1), (2) naming animals (DAn), and (3) subtracting serial sevens from 100 (DS7). The cognitive tasks were performed out loud, without any prioritization instruction. All participants walked along a 6-m path, starting and finishing 1 m away to ensure that only steady-state walking was captured [24, 36].

The preferred walking speed condition was repeated three times, and average values were calculated. Gait characteristics such as speed, cadence, and total number of steps were reported for the participants assessed with the accelerometer. The participants assessed using the electronic walkways had additional gait characteristics, including, gait speed, stride velocity, step and stride time, step and stride length, double support time, swing time, step and stride width, cadence, and corresponding variabilities, calculated using the coefficient of variation, CoV, obtained from [(standard deviation/mean) ×100] [36].

Since some of the metrics extracted from the electronic walkway provide similar information (e.g., stride time and cadence, step and stride metrics), we excluded step metrics and cadence from the analysis. The metrics included were, thus, stride length, double support time, stride time, stride velocity, stride width, swing time, and their respective variabilities.

For all these metrics we calculated dual-task costs (DTC, in %) using the formula [(single-task metric—dual-task metric)/single-task metric] ×100, which quantifies the magnitude of the effect of cognitive load on gait performance [36]. We also calculated the capacity index comparing preferred speed with fast walking trials using the formula [(fast metric—single-task metric)/single-task metric] ×100.

Machine learning pipeline

We developed a machine learning pipeline for classifying participants into normal or cognitively impaired using the metrics from gait tests. We assessed each gait test separately (i.e., SS, DAn, DS1, DS7, and F) to compare each test’s ability to differentiate both groups. Additionally, we tested the combinations of dual-task tests (and fast-walking test) with SS, i.e., DAn + SS, DS1 + SS, DS7 + SS, and F + SS. To this purpose, we analyzed and reported the best-performing experiment from the following three possibilities: (a) using dual-task costs (or capacity indexes) alone, (b) combining dual-task costs (or capacity indexes) and SS metrics, and (c) combining dual-task costs (or capacity indexes) and dual-task (or fast-walking) test metrics. The five gait tests and the four combinations resulted in a total of nine reported experiments.

The experiments were performed using metrics provided by the accelerometer and repeated using the metrics provided by the electronic walkway. Since our goal was to compare the limited set of features provided in this study by the accelerometer with the more comprehensive set provided by the electronic walkway (described in the previous section), we have only included participants assessed by the two systems.

One participant in the cognitively normal group had no accelerometer data on DAn, DS7 and F tests and was, as such, excluded from experiments including these data. The same happened with one participant in the cognitively impaired group that had no accelerometer data on the DAn test. All the remaining variables were present, yielding 29 participants in the impaired group (or 28 when data were missing) and 18 in the cognitively normal group (or 17 when data were missing).

To avoid overly optimistic estimations and account for the limited number of participants in this study, we used a nested cross-validation (CV) approach [37], which consists of an outer and inner CV loop, as shown in Fig. 1. Although leave-one-subject-out (LOSO) cross-validation is a common approach for small datasets, we opted for a k-fold split with k = 20 in nested CV to reduce computational load while maintaining sufficient training data and robust model evaluation. The dataset was, thus, first divided into 20 equally-sized outer folds, yielding 42–45 participants’ data for hyperparameter tuning and model training (training sets) and 2–3 participants’ data for evaluation (test sets). In the inner loop, the outer training data were further divided into 5 folds, forming the inner training and validation folds. The inner loops were used for hyperparameter tuning in a grid search. When the best set of hyperparameters was found, the whole (outer) training set was used to retrain the model. The model was then used to classify the test set in the outer loop.

Fig. 1

Machine learning pipeline with nested cross-validation

As illustrated in Fig. 1, the machine learning pipeline consists of: (1) preprocessing, (2) feature selection, (3) sampling, and (4) classification. The preprocessing step consisted of a z-score standardization that scaled the features to zero mean and unit standard deviation. For the feature selection, we tested different methods: we optimized the number of features to select (1, 2, 3, 5, or 8) using the highest-scored features (KBest) according to their mutual information (or mutual dependence) with the target [38], or using recursive feature elimination (RFE) based on the feature ranking provided by a Support Vector Machine (SVM) with a linear kernel; we also tested feature selection based on feature importance as provided by Random Forest (RF) [39] with 50 estimators, selecting all those features with importance above the mean. We also tested the pipeline without any feature selection. In the sampling step, we used the Synthetic Minority Oversampling Technique (SMOTE) algorithm [40]. Since the incidence of cognitively normal participants in the dataset was only 38.3%, SMOTE was used to create new synthetic examples of the minority class. According to [40] SMOTE provides better results when applied after feature selection. We also evaluated the performance of the classifiers without applying any resampling strategy, as all the classifiers tested in the subsequent stage were able to adjust for dataset imbalance by using class weighting. In the classification stage, we optimized the following hyperparameters in three different classifiers: the regularization strength, C (0.001, 0.01, 0.1, 1.0, 10, or 100) and kernel (linear or rbf) of the Support Vector Machine (SVM); the number of estimators (10, 50, or 100) of the Random Forest (RF); and the regularization strength, C (0.001, 0.01, 0.1, 1.0, 10, or 100) of the Logistic Regression (LR) classifier.

The total number of true positives (TP), true negatives (TN), false positives (FP), and false negatives (FN) were obtained for all test partitions in the outer loop and used to calculate the global metrics of performance, where the cognitively impaired group was considered the positive class. To evaluate and compare the performance of the machine learning models, we calculated balanced accuracy, sensitivity, specificity, F1 score, and the area under the receiver operating characteristic (ROC) curve (AUC). Since we were dealing with imbalanced data, balanced accuracy was used as the scoring criteria to select the best set of hyperparameters in the grid search. Balanced accuracy was also used to compare multiple machine-learning experiments.

Experiments were performed using Python 3 and the package scikit-learn v1.1.3 [41].

Classification using MoCA scores

We compared the performance of the machine learning experiments with MoCA. Two different experiments were conducted: (1) we used the classical threshold of 26 to identify the groups and (2) we used a ROC analysis to determine the optimal MoCA cut-point that maximized sensitivity and specificity in the current dataset. The optimal cut-point was considered the point of the ROC curve closest to the upper left corner (0,1), defined as that yielding the minimum value of (1–sensitivity)² + (1–specificity)² [42]. A third experiment combined the classifications obtained with MoCA with those obtained using gait metrics in two of the best-performing conditions. The classifications obtained by the three different approaches were combined using a majority voting scheme [43].

Statistical analysis

Demographic and medical characteristics were summarized using either means and standard deviations, or frequencies and percentages, as appropriate, to characterize both groups. The comparisons between groups were performed using the Independent Samples T-test or the Mann-Whitney U test for continuous variables, and Person’s chi-square test (${\chi ^2}$) for categorical variables. The Mann-Whitney U test was used when data were not normally distributed or when outliers were present outside ±1.5 of the interquartile range. To test for normality, we used the Shapiro-Wilk test.

To compare the results of the machine learning experiments, we used Paired Samples T-Tests or Wilcoxon signed-rank tests, that compared the average performance of the validation sets in all inner cross-validation (CV) runs. Wilcoxon signed-rank tests were used when paired data differences were not normally distributed or when outliers were present. Bonferroni-adjusted p-values were reported for the multiple comparisons [44].

McNemar’s Test was used to compare the diagnosis provided by MoCA, and the classifications provided by machine learning experiments, as recommended in [45].

The statistical analysis was performed using Python 3 and the statistical tools from SciPy v1.9.3, NumPy v1.23.5, and statsmodels v0.13.5 [46]. A p-value of less than 0.05 indicated statistical significance.

Results

Descriptive statistics

Of the 161 participants with CVD available in the ONDRI dataset, only 47 had gait features measured by both the accelerometer and the electronic walkway. Of the 47 participants, 29 (61.7%) met the criteria for CI, whereas 18 (38.3%) were classified as cognitively normal. The descriptive statistics of the two groups are provided in Table 2.

Table 2 Descriptive statistics (n = 47)

Significant differences between groups were obtained on MoCA scores and iADLs percent, indicating lower global cognitive performance in the impaired group and less independence on iADLs. The difference in ADLs was not significant, denoting that both groups were still equally capable of performing these tasks. Additionally, there were no significant differences in depression scores, age, and education between the two groups.

The group with CI had a slower gait speed, as indicated by significant differences in multiple gait tests. The electronic walkway revealed significant differences in gait speed in more tests than the accelerometer. No significant differences between groups were obtained for gait speed while walking fast, as measured either by the accelerometer or the electronic walkway. Although the impaired group had significantly shorter height, leg length presented no significant differences between groups, which should not contribute to differences in gait performance [47].

Classification using accelerometer metrics

The best-performing classifiers, feature selectors, and sampling methods are shown in Table 3 for each tested gait condition using metrics from the accelerometer. The average number (and standard deviation) of features selected (within each outer training fold) are also shown in this table. Performance metrics are reported for the test set.

Table 3 Performance on test sets using gait metrics from the accelerometer (in %)

Balanced accuracy spanned from 60.4% (using F walking features) to 70.6% (using DS1 + SS walking features). The features extracted from the SS condition provided better performance than DAn, DS1, DS7, and F conditions, with 72.4% of sensitivity, 66.7% of specificity, and 66.3% of AUC. The performances of dual-task conditions (and F) improved when dual-task costs (or capacity indexes) were included (e.g., the performance of DAn + SS was higher than DAn alone). The best performing combination overall was with DS1 + SS, with a sensitivity of 69.0%, a specificity of 72.2%, and an AUC of 75.5% (Table 3).

The performance (balanced accuracy) on validation and test sets using gait metrics from the accelerometer is visually compared in Fig. 2a. Statistical results are provided in this figure comparing DS1 + SS with the other conditions.

From the analysis of Fig. 2a we verify that classification performance (measured on validation folds) of the best performing condition, DS1 + SS, although superior to the SS condition, does not differ significantly. All the remaining conditions had a lower average performance, differing significantly from the DS1 + SS condition.

We note that in the SS condition only 3.0 ± 0.0 features were included, whereas in the DS1 + SS condition an average of 4.6 ± 1.0 were included (Table 3). In the case of the accelerometer, as only three metrics were measured (i.e., number of steps, cadence, and speed), the maximum number of features selected was either three or six in the case of the experiments including dual-task costs. The features that were most commonly used in the DS1 + SS condition are shown in Fig. 2c. In this condition, the most relevant features were the cost in velocity, single-task velocity, single-task cadence, the cost in total steps, and the cost in cadence. The total number of steps in the single-task condition was the least relevant feature. Figure 2b shows the confusion matrix of the DS1 + SS condition.

Fig. 2

Performance using features extracted from the accelerometer. (a) Performance on validation and test sets, where vertical lines indicate standard deviation of the 20 inner CVs. The statistical analysis compares the best-performing experiment (DS1 + SS) with the results achieved in the other conditions, using Paired Samples T-tests or Wilcoxon signed-rank tests with Bonferroni correction, where *p < 0.05, **p < 0.01, and ***p < 0.001 indicate statistical significance. (b) Confusion matrix of the DS1 + SS condition. (c) Feature selection frequency for the DS1 + SS condition

Classification using walkway metrics

The results achieved using the metrics extracted from the electronic walkway are shown in Table 4. As with the accelerometer, the best performing condition was the combination DS1 + SS, with 78.8% of balanced accuracy, 96.6% of sensitivity, 61.1% of specificity, and 68.6% of AUC. When using features from the gait test alone, DS7 (with 77.5% of balanced accuracy) revealed the highest performance in comparison with SS, DAn, DS1, and F conditions. Balanced accuracies spanned from 60.5% (with F) to 78.8% (with DS1 + SS), being test scores higher than its equivalents with the accelerometer.

Table 4 Performance on test sets using gait metrics from the electronic walkway (in %)

Although the electronic walkway provided an additional number of gait metrics than the accelerometer, the number of features used by machine learning models was not always higher. The SS condition, for instance, used only 2.0 ± 2.2 features, whereas in the case of the accelerometer an average number of 3.0 ± 0.0 features were used in this condition. However, the features included were different. In the SS condition, for instance, double support time was selected for all the 20 outer training folds and stride width variability was selected 5 times (results not shown). Double support time was one of the most relevant features of the electronic walkway in several conditions where feature selection has been applied, including, SS, DAn, DS1, F, DS7 + SS, and F + SS. In the case of the DS1 + SS condition, the best balanced accuracy of 78.8% was achieved with 1.7 ± 2.1 features, being stride length cost one of the most relevant features, selected in all folds of the outer training loop (Fig. 3c). Only 1 participant in the impaired group was misclassified using DS1 + SS features (Fig. 3b), leading to a very high sensitivity of 96.6%.

Fig. 3

Performance using features extracted from the walkway. (a) Performance on validation and test sets, where vertical lines indicate standard deviation of the 20 inner CVs. The statistical analysis compares the best-performing experiment (DS1 + SS) with the results achieved in the other conditions, using Paired Samples T-tests or Wilcoxon signed-rank tests with Bonferroni correction, where *p < 0.05, **p < 0.01, and ***p < 0.001 indicate statistical significance. (b) Confusion matrix of the DS1 + SS condition. (c) Feature selection frequency for the DS1 + SS condition

Compared to the DS1 + SS condition, only DS1, F, DAn + SS, and F + SS conditions revealed significantly lower performances (Fig. 3a). The performance of the DS1 + SS was superior to the SS in validation and test folds, although the differences in validation performances were not statistically significant.

A Paired Samples T-test revealed that the performance of the machine learning models in the DS1 + SS condition using walkway features was significantly higher than the performance of the DS1 + SS condition using the features extracted from the accelerometer (T = −4.446, p < 0.001).

Fig. 4

Classification performance using MoCA and a majority voting approach. (a) ROC analysis used to determine the optimal MoCA cut-point and (b) Confusion matrix resulting from the majority voting approach, which combined the SS condition, the DS1 + SS condition (walkway) and MoCA<27. AUC: Area under the curve; ROC: receiver operating characteristic

Performance using MoCA

Table 5 and Fig. 4 summarize the results of the classification when a single cut-point was employed to MoCA. Using the traditional cut-point of 26, the MoCA’s sensitivity was 62.1% and the specificity was 77.8% in the present sample (Table 5). The ROC analysis (ROC curve shown in Fig. 4a) indicated that in the present sample the cutoff of 27 would optimize sensitivity and specificity, which was according to the results obtained in [27]. The cutoff of 27 yielded a sensitivity of 75.9%, a specificity of 77.8%, and an AUC of 66.0%.

Table 5 Performance using MoCA and a combination with gait

Since the DS1 + SS condition measured by the walkway yielded the best classification performance when using gait metrics (Table 4), and because the DS1 + SS condition requires a SS test in order to measure the dual-task costs, we combined the classification results using a majority voting approach that considered the classifications provided by the SS condition, the DS1 + SS condition and MoCA<27. The majority voting resulted in an improved balanced accuracy of 82.7%, a sensitivity of 93.1%, and a specificity of 72.2%. The resulting confusion matrix is shown in Fig. 4b.

An exact McNemar’s Test showed that there were no statistically significant differences in the disagreements between the classifications provided by MoCA<27 and the classifications provided by the machine learning models using the best performing condition of DS1 + SS (walkway), with p = 0.549. The observed differences in classification results were also not statistically significant when the traditional threshold (<26) was employed, with p = 0.092. Although the majority voting scheme revealed the highest balanced accuracy overall, the observed differences in classification results were not significantly different from DS1 + SS (p = 1.0), SS (p = 0.289), or MoCA<27 (p = 0.219), but differed significantly from MoCA<26 (p < 0.05).

Discussion

This study was the first to investigate the use of machine learning and gait characteristics for detecting CI in post-stroke patients using a gold-standard neuropsychological battery as a reference. We used a nested cross-validation approach to evaluate the performance of multiple classifiers and feature selection methods on different walking conditions, including single-task (SS), dual-task (DAn, DS1, and DS7), and fast walking (F). Moreover, we tested the inclusion of dual-task costs and capacity indexes resulting from the combination of SS with DAn, DS1, DS7, or F. In line with our initial hypothesis, gait characteristics were able to distinguish between post-stroke individuals with and without CI. Moreover, a comprehensive set of gait metrics and dual-task costs allowed better differentiation between the two groups, with classification performances that were comparable to or even superior to those achieved with MoCA. Combining gait metrics with MoCA improved overall classification performance.

Performance using gait metrics

The machine learning experiment leading to the best classification overall was achieved using DS1 + SS gait metrics extracted from the electronic walkway. Although disagreements in classification results were not significantly different from MoCA (using cutoffs of <26 or <27), the balanced accuracy achieved with gait metrics was higher (78.8% versus 69.9% or 76.8%, respectively), which is a positive indication that gait analysis can be a useful tool for the screening of CI in post-stroke patients. Compared to MoCA, gait tests are simpler and allow more objective and faster evaluations, which justifies their relevance in the context of cognitive screening [18].

The DS1 + SS condition was consistently identified as the best-performing condition, either using the accelerometer or the walkway. However, in both cases, the differences between the DS1 + SS condition and the SS condition were not statistically significant. This suggests that even a simple single-task gait analysis could provide acceptable classification results, without the need for participants to complete additional tests. Despite this, the sensitivity obtained in the DS1 + SS condition using the electronic walkway was much higher at 96.6%. This suggests that the DS1 + SS condition should be a preferred choice for cognitive screening, where the main goal is to detect all potential cases for further assessment [8].

The lower classification performances obtained with the accelerometer may be in part due to the lower level of detail it provided compared to the electronic walkway. While the accelerometer could only provide general gait parameters, like, velocity, cadence, and total steps, the electronic walkway provided additional spatio-temporal metrics that characterized the strides (e.g., length, speed, width) and gait phases (e.g. swing and double support time). We should note that the limitations of the accelerometer in this regard are not due to the device itself, but rather to the algorithms used to process its data [48,49,50]. It’s also worth noting that in the analysis of the descriptive statistics, the electronic walkway detected significant differences in gait speeds between the two groups in more tests than the accelerometer (Table 2). This may suggest that there are differences in the performance of the two technologies. In fact, different algorithms can be used to process data from inertial sensors, and the inclusion of a gyroscope can often lead to improved results [48,49,50]. Therefore, the choice of algorithm and sensors used may have an impact on the results obtained. Thus, a possible lack of instrument accuracy (and/or precision) could partially justify the obtained results. Yet, including more detailed evaluations of gait seems to be beneficial, which is according to previous results on the detection of CI. According to previous studies, gait speed may be associated with several adverse health outcomes [51, 52], and therefore, it is not specific for the detection of CI [53, 54]. Since no other pathologies were included in our study, these results could not be confirmed.

Previous studies showed that different cognitive tasks can impact walking in different ways. In general, more cognitively demanding tasks tend to lead to greater interference effects, although the results are not always consistent [13]. The serial subtraction by sevens is particularly cognitively demanding and has been shown to be the most sensitive in distinguishing patients with CI in non-stroke populations [13]. In this study, the classification performance achieved with the DS7 and DS7 + SS conditions was significantly lower than the DS1 + SS condition when using the accelerometer (Fig. 2a), but did not differ from the DS1 + SS condition when using the walkway (Fig. 3a). However, the DS1 + SS condition led to higher performances in both cases, indicating better discrimination ability. It’s worth noting that post-stroke patients often have other disabilities, particularly physical ones [9], and the mechanisms underlying motor-cognitive interferences may be different in this population compared to patients with other neurocognitive disorders such as Alzheimer’s disease. Therefore, further research is needed to understand the effects of different cognitive tasks on post-stroke patients.

Previous research has also shown that combining multiple gait metrics can provide better discrimination ability than using a single gait metric [47, 55]. However, some gait metrics may be highly related and their simultaneous inclusion may not provide additional discrimination ability [56]. In this study, we used feature selection methods to identify the most relevant features based on different criteria. In the best-performing condition (DS1 + SS using the walkway), features were selected based on feature importance as provided by the Random Forest. In this experiment, only dual-task costs were selected for classification. According to previous studies, dual-task costs represent only the effect of adding the cognitive task and are not influenced by individual baseline characteristics (e.g., leg length) that could otherwise be considered covariates [14, 57]. For this reason, dual-task costs constitute an interesting evaluation metric for the detection of cognitive disorders.

The performances achieved in this study are at the level of the performances reported in the literature for the detection of CI in non-stroke patients. For instance, using dual-task gait data from an electronic walkway, Boettcher et al. [20] reported sensitivities of 82.0% and specificities of 67.7% on the detection of mild CI. Also with an electronic walkway, Ghoraani et al. [21] differentiated cognitively impaired groups from the healthy subjects with an accuracy of 86.0% and an F1 score of 88.0%. Using a detailed evaluation of gait provided by an inertial sensor-based gait analysis solution, Shahzad et al. [22] could achieve classification accuracies of 70.0% and sensitivities of 83.3% for the detection of mild CI, which proves the capacity of inertial sensor-based solutions.

Performance using MoCA

Similarly to the study by Zaidi et al. [27], the threshold of 27 in MoCA was the one achieving the best combination of sensitivity and specificity (Table 5). Although this study used the same dataset, they included all 161 participants and employed different neuropsychological criteria to characterize CI. In [27] the threshold of 27 resulted in a maximized sensitivity of 79% and specificity of 67% in detecting CI post-stroke. As in our study, increasing the cutoff from 26 to 27 allowed an increase in sensitivity.

To increase specificity, some studies recommend lowering the threshold of MoCA [8, 10]. A pooled analysis showed that MoCA scores below 26 are able to detect post-stroke CI with a sensitivity of 95% and a specificity of 45%, but lowering the threshold to 22 results in a lower sensitivity of 84% and a better specificity of 78% [8]. The differences in sensitivity and specificity reported by the different studies are mostly justified by the different neuropsychological criteria used to characterize CI [12].

Although disagreements in classification results obtained with MoCA<27 and with the best-performing gait condition did not differ significantly when compared to the majority voting approach, the combination of these two modalities resulted in an overall improvement in classification performance (Table 5). Compared to MoCA<26, the disagreements in classifications were significantly different, with the majority voting scheme achieving better sensitivity (93.1% versus 62.1%) for detecting CI. The sensitivity also improved compared to MoCA<27 (93.1% versus 75.9%), making the majority voting approach more suitable for screening purposes [8]. Although its sensitivity was lower than that of the DS1 + SS condition (93.1% versus 96.6%), it still remained very high, with a large improvement in specificity (72.2% versus 61.1%), which should favor its choice. The majority voting approach has received much attention in the literature for its simplicity and good performance on real data and has been applied in various fields [43]. To the best of our knowledge, this is the first study to propose the combination of MoCA with gait metrics for classifying cognitive status in post-stroke patients.

Study limitations

Several factors constitute the limitations of our study. First, the sample size was relatively small, which may affect the robustness of the models and the generalizability of our findings. Second, the criteria for classifying participants as impaired or cognitively normal based on neuropsychological assessments are not universally accepted and may vary depending on the cognitive domains evaluated, the tests used, their grouping criteria, the cutoffs demarcating impairment (most commonly 1 SD, 1.5 SD, or 2 SD), and the number of tests required to declare a deficit in a certain domain [12, 34]. In our study, we employed the comprehensive criteria recommended by Jak et al. [34] and the cognitive domains proposed by Dilliott et al. [33], which resulted in 61.7% of the patients meeting the criteria for CI. This frequency is above the pooled prevalence rates reported in the literature for the first year after stroke [12], but it is important to note that different criteria could lead to different results. Third, the participants in this study may have higher functioning levels than those typically seen in the general clinic, as those with severe functional disabilities were excluded from the study. Moreover, only individuals with MoCA>18 were included, which reduced the range of cognitive disorders associated with stroke that could be tested in this study. However, it is worth noting that our approach was able to detect CI in this group, which included more subtle symptoms that are usually harder to detect. Fourth, we did not evaluate the impact of the different study sites on the results. Although efforts were made to standardize assessments and ensure data quality across sites [24, 32, 36], differences could still be observed due to discrepancies in acquisition. This effect should be explored in future research. Finally, it is important to note that we did not evaluate the impact of the walking task on cognitive performance, although Plummer et al. [58] argue that the correct interpretation of the dual-task interference requires an objective evaluation of both tasks. Motor-cognitive interference may be observed in just one of the tasks, or in both simultaneously, and, for this reason, the performance on both tasks should be taken into account [58]. Additionally, it is worth noting that the order of the gait tests was maintained for all participants, which could potentially affect the results.

Future work

Further research is needed to determine the feasibility and effectiveness of gait assessments and machine learning models in detecting CI in post-stroke patients within a clinical setting. To achieve this, it would be necessary to include a larger and more diverse group of participants, including those with lower functioning levels and a wider range of cognitive abilities. Additionally, we should explore the detection of CI in the context of chronic CVD without a previous stroke [9].

Utilizing additional gait metrics as measured by inertial sensors could also be a valuable area of investigation. Inertial sensors have gained popularity in gait assessment due to their flexibility, low cost, and performance [59], and have the potential to extract gait metrics that have not yet been explored in this context [48, 49]. As suggested by Jung et al. [23], using sequential gait parameters extracted from inertial sensors could improve classification results and should, thus, be explored in the future. Furthermore, it is important to include an analysis of the performance on the cognitive task to provide a complete assessment of motor-cognitive interference in these patients [58].

Conclusion

Cognitive impairment (CI) after stroke is a common but often undiagnosed condition. To better manage cognitive disorders resulting from cardiovascular disease, it is necessary to develop new techniques to improve the screening of CI. In this study, we examined the use of gait, dual tasks, and machine learning to detect CI in post-stroke patients. Our results showed that a machine learning approach combined with dual-task gait metrics can effectively differentiate impaired and cognitively normal groups, with a performance that is not inferior to commonly used screening tests like MoCA. We also found that when using gait to assess cognitive status, it is important to include gait features that evaluate individual strides and gait phases, as opposed to just overall gait metrics like speed, cadence, or number of steps, as these result in poorer classification results. Using a majority voting approach that combines MoCA and dual-task gait metrics resulted in improved classification results, with better combinations of sensitivity and specificity for the detection CI. This study demonstrates the potential of machine learning and gait assessments as objective tools for cognitive screening, offering a good alternative or complement to MoCA in identifying CI in post-stroke patients. Further research is required to evaluate the practical and effective performance of this approach in a clinical setting.