Machine learning–based predictive model for post-stroke dementia

Abstract

Background

Post-stroke dementia (PSD), a common complication, diminishes rehabilitation efficacy and affects disease prognosis in stroke patients. Many factors may be related to PSD, including demographic, comorbidities, and examination characteristics. However, most existing methods are qualitative evaluations of independent factors, which ignore the interaction amongst various factors. Therefore, the purpose of this study is to explore the applicability of machine learning (ML) methods for predicting PSD.

Methods

9 acceptable features were screened out by the Spearman correlation analysis and Boruta algorithm. We developed and evaluated 8 ML models: logistic regression, elastic net, k-nearest neighbors, decision tree, extreme gradient boosting, support vector machine, random forest, and multilayer perceptron.

Results

A total of 539 stroke patients were included in this study. Among the 8 models used to predict PSD, extreme gradient boosting and random forest showed the highest area under the curve (AUC) of the receiver operating characteristic curve (ROC), with values of 0.7287 and 0.7285, respectively. The most important features for predicting PSD included age, high sensitivity C-reactive protein, stroke side and location, and the occurrence of cerebral hemorrhage.

Conclusion

Our findings suggest that ML models, especially extreme gradient boosting, can best predict the risk of PSD.

Introduction

Stroke, also known as cerebrovascular accident (CVA), stands as the second leading cause of death worldwide. It is characterized by a high incidence rate, elevated mortality, and increased disability rates [1]. According to the Global Burden of Disease study in 2019, there were 6.55 million fatalities attributed to stroke, with 12.2 million new cases reported. Notably, China ranks among the countries with the highest proportion of stroke risk factors globally [2]. Advances in medical treatments have notably reduced the mortality associated with stroke. It is noteworthy that post-stroke disability complications, post-stroke cognitive impairment (PSCI) persist widely. According to severity, PSCI is divided into post-stroke cognitive impairment no dementia (PSCIND) and post-stroke dementia (PSD) [3].

According to the Diagnostic and Statistical Manual of Mental Disorders, Fifth Edition (DSM-5), cognitive impairment can be classified into two categories based on severity: mild cognitive impairment (MCI) and dementia. Additionally, research has found that MCI is reversible and may progress back to a normal cognitive state [4]. About 20–30% of PSCI patients will deteriorate and develop into PSD [5, 6]. Thus, early diagnosis and treatment of PSCI and PSD contribute to improving the prognosis of stroke patients, alleviating societal and economic burdens. Numerous studies have reported many independent risk factors for PSCI and PSD related to age, hypertension, high cholesterol, diabetes, smoking, atrial fibrillation [7,8,9]. Stroke history and vascular risk factors (such as high blood pressure, obesity, and smoking) may accelerate the development of PSCI and PSD [10]. Thus, clinicians face challenges in integrating multiple factors for the early diagnosis of PSCI and PSD. The CHANGE and SIGNAL2 scales have demonstrated accuracy in predicting PSCI, with area under the curve (AUC) of receiver operating characteristic curve (ROC) ranging from 0.740 to 0.829 [11, 12]. Due to the complexity of the pathophysiology and multiple contributing factors associated with PSCI, conventional scoring systems with limited variables may not optimally predict PSCI. Researchers have applied cognitive impairment risk prediction models to PSCI, but the predictive performance has not been ideal [13].

Machine learning (ML) algorithms are adaptable to various types and sizes of data and have garnered significant attention in the development of patient-centered prediction/prognosis models. These models can help optimize treatment plans and facilitate the monitoring and management of health conditions. Recently, ML has demonstrated immense potential in enhancing the speed and accuracy of stroke imaging assessments [14, 15]. Consequently, some researchers are pivoting their focus towards ML, aiming to develop a more accurate predictive model for PSCI [16,17,18]. In this context, we conducted a retrospective study focusing on a more severe form of cognitive decline after stroke, PSD, with the goal of developing and testing the applicability of ML models in predicting PSD.

Materials and methods

Data source

For this retrospective cohort study, we selected 545 stroke inpatients from the rehabilitation department and geriatrics department of Shaoxing People’s Hospital, spanning from January 2019 to August 2021.

Inclusion Criteria: (1) admitted for stroke between January 2019 and August 2021; (2) aged 18 years and above; (3) signed an informed consent form.

Exclusion Criteria: (1) Pre-existing cognitive impairment before the current stroke; (2) Presence of other diseases severely affecting cognitive function, such as anxiety, depression, or brain tumors.

Clinical variables

A total of 46 variables were collected, including demographic data, vascular risk factors, and examination findings. Demographic data included variables such as gender, age, occupation, and education level, and Body mass index (BMI). The vascular risk factors recorded were smoking status, alcohol consumption, hypertension, hyperlipidemia, and any history of stroke. Clinical examination findings compiled for analysis included the type of stroke, fibrinogen levels, high-sensitivity C-reactive protein (CRP), blood uric acid levels, blood homocysteine levels (Hcy), and specifics regarding the stroke’s side and location.

Model construction and verification

The initial sample (n = 545) underwent multicollinearity assessment to ensure the stability of subsequent analyses. Based on Spearman correlation analysis, we identified and removed individual significant and highly correlated features to mitigate the effects of multicollinearity. Additionally, features with over 20% missing values (BMI) were discarded, and 6 samples with extreme deviations and missing data were excluded. 539 patients and 45 variables were included in the study. Supplementary Table 1 shows the detailed information of all variables. The continuous variables in the remaining sample were standardized through Z-score normalization. Finally, the data of 539 patients were randomly divided into a training set and a test set in a 3:1 ratio.

In this study, we preselected features for the training set using the Boruta algorithm, a feature selection method based on the random forest algorithm [19]. The Boruta algorithm is a highly popular and robust feature selection method that helps in retaining only the most statistically significant variables [20], thereby making a practical contribution to our model. It identifies crucial features by creating shadow features (randomized copies of features) and evaluating the importance of original features using the random forest algorithm. Subsequently, we input these preselected features into 8 different ML models: logistic regression (LR), elastic net (EN), k-nearest neighbors (KNN), decision tree (DT), extreme gradient boosting (XGB), support vector machine (SVM), random forest (RF), and multilayer perceptron (MLP). For each model, we selected a set of hyperparameters that maximized the AUC of ROC on the training set using a Bayesian optimizer, ensuring optimal performance and effective prediction and comparison on the test set. Detailed hyperparameters are available in the supplementary material. All models were subjected to five-fold cross-validation to ensure robustness and reliability. We plotted the ROC of each model on the test set to evaluate their predictive performance and decision curves analysis (DCA) and clinical impact curve (CIC) to assess their clinical utility.

Comparing models, we selected the best model based on AUC values and created a Shapley Additive exPlanations (SHAP) explainer to calculate SHAP values, which indicate the contribution of each feature to the prediction outcome. We plotted a SHAP summary chart to illustrate the impact of model features.

Statistical analysis

R software (version 4.3.0, https://www.R-project.org/) was used to support grouping and data statistical analysis. Continuous variables were presented as median with interquartile range (Mean, IQR), while categorical data were represented by numerical values and corresponding percentages (n, %). We used t-test or the Wilcoxon rank-sum non-parametric test for continuous variables, and chi-square test for categorical variables to compare the demographic and clinical characteristics between the PSD group and the non-PSD group.

Results

Characteristics of patients

Among the 539 stroke patients included in this analysis, 194 did not develop PSD, while 345 did. The baseline characteristics of the PSD and no PSD groups are shown in Table 1. There were significant differences in age and High-sensitive CRP between the two groups (P < 0.001).

Table 1 Clinical characteristics of patients (n = 539)

Model building and verification

Through Spearman correlation analysis, we removed 6 individual significant and highly correlated features. Feature selection was then conducted using the Boruta algorithm, identifying 9 acceptable variables: brain stem, age, temporal lobe, right lesion, cerebral hemorrhage, high-sensitive CRP, subarachnoid space, outer capsule, and Island leaves (Fig. 1). Using these 9 variables, we developed 8 ML models to predict the risk of PSD in stroke patients. Fig. 2 displays the ROC of all models, with their predictive discrimination represented by the AUC. The XGB model had the highest AUC (0.7287), followed by RF (0.7285), KNN (0.7113), MLP (0.7082), EN (0.7033), LR (0.7022), DT (0.6502), and SVM (0.6098). The clinical applicability of all models was further assessed using DCA and CIC. DCA was utilized to evaluate the clinical benefit of the predictive models. The threshold range for the XGB model was approximately 0-0.87, slightly narrower than those for the EN, LR, and KNN models. Additionally, the DCA overlapped partially or entirely with other models across most threshold ranges, indicating no significant difference in net benefits. However, in the threshold range of approximately 0.76–0.8, the net benefit of the XGB model was significantly higher than that of other models (Fig. 3). Supplementary Fig. 1 shows the CIC for all models, assessing the efficiency of the models. When the threshold was greater than 0.7, the high-risk PSD group identified by the XGB prediction model closely matched the actual PSD occurrences, confirming the model’s high clinical efficiency. Additionally, we calculated the accuracy, sensitivity, and specificity for all models (Table 2). The XGB model achieved the highest accuracy with a value of 0.72939. Although the sensitivity and specificity of the XGB model were not the highest, its overall performance is better when these metrics are considered comprehensively.

Fig. 1

Feature selection based on Boruta algorithm. Green represents acceptable variables

Fig. 2

The receiver operating characteristic curves of the eight machine learning models. LR, logistic regression; XGB, extreme gradient boosting; DT, decision tree; SVM, support vector machine; KNN, k-nearest neighbors; RF, random forest; MLP, multilayer perceptron; EN, elastic net

Fig. 3

The DCA curves of the eight machine learning models. LR, logistic regression; XGB, extreme gradient boosting; DT, decision tree; SVM, support vector machine; KNN, k-nearest neighbors; RF, random forest; MLP, multilayer perceptron; EN, elastic net

Table 2 Performances of various prediction models predicting PSD using a testing data set

Feature importance

To visually present the selected variables, we analyzed the best predictive model, XGB, using the SHAP package to show the positive and negative impact of each feature on a given sample. Fig. 4A displays the absolute values of the average SHAP values for different features. Age had the most significant impact on model output, followed by high-sensitive CRP, right lesion, temporal lobe, cerebral hemorrhage, brain stem, undergraduate, bilateral lesions, and subarachnoid space. Fig. 4B provides a more detailed view of the impact of each feature on individual predictions.

Fig. 4

The Shapley Additive exPlanations values of the best prediction model, XGB. (A) Average impact of features on model predictions. (B) Detailed impact analysis of each feature

Discussion

In this retrospective cohort study, we selected 9 acceptable features using the Boruta algorithm and developed 8 ML models to predict the risk of PSD in stroke patients. Among the 8 models, the XGB model showed the highest AUC and good clinical applicability. Furthermore, the most impactful features for predicting PSD, in descending order of importance, were age, high-sensitive CRP, right lesion, temporal lobe, and cerebral hemorrhage.

Increasingly, research acknowledges that the risk factors and progression of post-stroke cognitive impairment and dementia (PSCID) are determined by a multitude of factors, including age, comorbidities, type of stroke (ischemic and hemorrhagic), education level, and the location and size of the stroke [3, 21]. Yan et al.‘s investigation, which developed 8 model for predicting the occurrence of MCI after stroke, revealed that the LR model achieved the highest AUC of 0.8595. It also exhibited high accuracy, sensitivity, and specificity, at 0.770, 0.778, and 0.765 respectively [18]. However, owing to the small sample size (n = 199), the model may not be sufficiently effective. Therefore, more patients were included in this study to provide better confidence and stability. Given the complex and varied factors influencing PSCID, prediction in clinical settings is challenging, with lower accuracy compared to post-stroke functional outcomes. ML methods are often more suitable when dealing with complex influencing factors. Stroke can be categorized into hemorrhagic strokes (HS) and ischemic strokes (IS), with IS constituting a significantly higher proportion of all strokes, approximately 87% according to data from Johns Hopkins Medical Center [22]. Thus, research focus and resources are more inclined towards IS. Our study encompassed patients with both types of strokes, offering a more comprehensive perspective for understanding and predicting PSD. The studies of Lee et al. (n = 951) and Ji et al. (n = 397) developed a variety of ML models to predict the risk of PSCI in patients with acute IS (AIS). They found that the XGB and Gaussian Naive Bayes (GNB) models had the best discrimination, with AUC of 0.7919 and 0.925, respectively. Additionally, these models also surpassed the LR model in other metrics, including accuracy and F1 score [16, 17]. In our study, XGB also showed better overall performance for PSD prediction. Although PSCI and PSD may clinically overlap, they differ in definition, severity, treatment strategies, and management. Dementia represents a more severe clinical outcome, significantly impacting patients’ daily lives and independence. Therefore, by concentrating on PSD as an outcome, effectively predicting its occurrence can facilitate the implementation of prevention and intervention measures at an earlier stage.

Despite the SIGNAL2 risk score and CHANGE risk score models exhibiting good discriminative ability in predicting PSCI, both models use Mini-Mental Status Examination (MMSE) ≤ 25 or Montreal Cognitive Assessment (MOCA) ≤ 22 as cut-off raw scores and incorporate age and education level as variables with high weight. However, since MMSE and MOCA scores are highly dependent on age and education level [11, 12]. Therefore, regardless of the clinical characteristics of the patient, it may have little effect on the predictive outcome. In our study, the ML models we constructed can continually learn and adapt to new data, improving their accuracy over time, unlike the static SIGNAL2 and CHANGE models. By applying the Boruta algorithm, we identified the most important features in the dataset for the predictive models. This approach, unconstrained by data type, allows for comprehensive feature selection, eliminating irrelevant features to reduce the risk of overfitting, thereby enhancing the model’s accuracy and interpretability. We incorporated variables in the green zone into the 8 models, with the XGB model achieving the highest AUC value. Lastly, we employed SHAP to quantify the contribution of each feature to the predictive outcome. In the XGB model, the most important features were age, high-sensitive CRP, right lesion, temporal lobe, and cerebral hemorrhage. These key features show both consistency and differences with previous research on PSCID risk factors [16,17,18, 23].

Most studies agree on the close association between age and cognitive decline. For instance, the REGARDS study found that each additional year of baseline age increased the likelihood of cognitive impairment by 17% during the follow-up period [23], aligning with non-stroke population studies that identified older age as a significant risk factor for cognitive impairment [24]. However, Yan et al.‘s findings suggested no correlation between age and the occurrence of MCI after stroke [18]. A systematic review and meta-analysis assessed the potential of various blood-derived proteins as biomarkers for PSCI, recommending Hcy, CRP, total cholesterol, and low-density lipoprotein as potential biomarkers for PSCI [25]. The high-sensitivity CRP test, capable of measuring low concentrations of CRP in blood, is useful for assessing low-grade inflammation and cardiovascular risk. A significant association between high-sensitive CRP concentrations and long-term cognitive decline was observed in a large study involving 5257 participants [26], consistent with our findings. Our research provides additional evidence supporting high-sensitive CRP as a potential biomarker for PSD, enhancing its potential in PSD prediction and monitoring. No association between Hcy and PSD was found in our study, although high Hcy levels have been confirmed as a risk factor for cerebrovascular events and cognitive decline [25]. The reasons for these differences may include sample selection bias, differences in data collection, or analytical methods.

At present, ML prediction models related to PSCID primarily focus on PSCI. To the best of our knowledge, this study is the first to construct and compare the performance of eight different risk prediction models for PSD. Furthermore, this research integrates ML techniques with demographic and imaging features to predict PSD. However, there are several limitations to our study that cannot be ignored. Firstly, the dataset used in this study was sourced from patients in the geriatrics department and those transferred to the rehabilitation department, who may exhibit significant differences in distributions of various demographic characteristics, such as age and gender, compared to the general stroke patient population. These disparities could potentially limit the generalizability of the model across different demographic groups. Secondly, this is a single-center retrospective cohort study, and the data quality and diversity might be affected, necessitating external validation and optimization. Thirdly, not all patients underwent all examinations, leading to missing features in some cases. Although we excluded patients with more than 20% missing values and employed multiple imputation for features with less than 10% missing values to mitigate this concern, the possibility of residual effects remains. Finally, our model has overlooked social, psychological, and behavioral elements like social support and lifestyle, which are crucial for understanding cognitive outcomes after a stroke. Including these factors in future research could enhance the model’s ability.

Conclusion

We proved that the ML model, especially the XGB model, can accurately predict PSD and is expected to be an effective assistant tool for the diagnosis and treatment of PSD. Among the variables included, age and high-sensitive CRP are the two most significant factors influencing the XGB model’s output. However, the efficacy of this model in external cohorts and its potential to mitigate the occurrence of PSD remains to be determined.