Predicting in-hospital mortality in patients with heart failure combined with atrial fibrillation using stacking ensemble model: an analysis of the medical information mart for intensive care IV (MIMIC-IV)

Abstract

Background

Heart failure (HF) and atrial fibrillation (AF) usually coexist and are associated with a poorer prognosis. This study aimed to develop a model to predict in-hospital mortality in patients with HF combined with AF.

Methods

Patients with HF and AF were obtained from the Medical Information Mart for Intensive Care IV (MIMIC-IV) database from 2008 to 2019. Feature selection was based on the Mann-Whitney U test and the least absolute shrinkage and selection operator (LASSO) regression model. Random Forest, eXtreme Gradient Boosting (XGBoost), Light Gradient Boosting Machine (LGBM), K-Nearest Neighbor (KNN) models, and their stacked model (the stacking ensemble model) were established. The area under of the curve (AUC) with 95% confidence interval (CI), sensitivity, specificity, as well as accuracy were applied to assess the performance of the predictive models.

Results

A total of 5,998 patients with HF combined with AF were included, of which 4,198 patients were assigned to the training set and 1,800 to the testing set (7:3). Among these 4,198 patients, 624 (14.86%) died in-hospital and 3,574 (85.14%) survived. Twenty-two features were used to construct the predictive model. Among these four single models, the AUC was 0.747 (95%CI: 0.717–0.777) for the Random Forest model, 0.755 (95%CI: 0.725–0.785) for the XGBoost model, 0.754 (95%CI: 0.724–0.784) for the LGBM model, and 0.746 (95%CI: 0.716–0.776) for the KNN model in the testing set. The stacking ensemble model had the highest AUC compared to the four single models, with AUCs of 0.837 (95%CI: 0.821–0.852) and 0.768 (95%CI: 0.740–0.796) for the training set and testing set, respectively.

Conclusion

The stacking ensemble model showed a good predictive effect in predicting in-hospital mortality in patients with HF combined with AF and may provide clinicians with a reference tool for early identification of mortality risk.

Background

Heart failure (HF) is a cardiovascular disease caused by abnormalities in the structure or function of the heart that can lead to increased intracardiac pressure or decreased cardiac output [1]. Atrial fibrillation (AF) is the most common arrhythmia and frequently coexists in patients with HF [2]. AF occurs in approximately one-third of patients with HF and these patients have higher morbidity and mortality than either HF or AF alone [3]. Therefore, early identification of the risk of mortality in patients with HF combined with AF is important for disease management and burden reduction.

Many tools for risk or prognosis prediction in patients with HF have been reported, including biomarkers, risk scores, and their combined metrics, but most of them have limited predictive validity [4,5,6,7]. A meta-analysis demonstrated that the predictive ability of the existing model was mediocre (c-index < 0.71) and was not applicable to the general population (e.g., only to those who were able to calculate a risk score) [8]. Adler et al. indicated that machine learning algorithms can be used to capture features associated with mortality in patients with HF to construct models that can improve the predictive effectiveness of existing models [9]. Different machine learning algorithms have been applied in the diagnosis and prognosis prediction of both HF and AF, but model effectiveness varies depending on the modeling approach [10,11,12]. Since each machine learning algorithm may excel or have drawbacks in different situations, models integrating multiple machine learning methods are applied. Stacking is a powerful integration technique that utilizes the predictions of multiple base learners as features to train new meta-learners, often exhibiting better performance than any single model [13]. Recently, Chiu et al. constructed a stacking ensemble model for predicting mortality in HF patients based on six base classifiers, and their model demonstrated good prediction results [14]. However, the effectiveness of the stacking ensemble models for predicting in-hospital mortality in patients with HF combined with AF is unclear. Thus, the purpose of this study was to construct a stacking ensemble model for predicting the risk of mortality in patients with HF combined with AF for use in assisting the clinical management of patients.

Methods

Study design and data source

This study utilized a retrospective cohort study design to develop models for predicting mortality in patients with HF combined with AF. Data were obtained from the Medical Information Mart for Intensive Care IV (MIMIC-IV) database between 2008 and 2019 period. MIMIC-IV is a large, single-center database containing real hospitalization data for patients admitted to the ICU at Beth Israel Deaconess Medical Center between 2008 and 2019 (https://mimic.mit.edu/docs/iv/). MIMIC-IV contains comprehensive information for each patient, including demographics, vital signs, laboratory measurements, medications, clinical measurements, and medical history [15]. Patients diagnosed with HF combined with AF at first admission to the ICU were included. Patients younger than 18 years of age, admitted to the ICU for less than 24 h, or missing survival data were excluded. HF and AF were identified according to the International Classification of Diseases, ninth/ten revision (ICD-9, ICD-10) codes. HF includes acute HF (ICD-9: 42821, 42823, 42831, 42833, 42841, 42843; ICD-10: I5021, I5023, I5031, I5033, I5041, I5043, I50811, I50813) and chronic HF (ICD-9: 42822, 42832, 42842; ICD-10: I5022, I5032, I5042, I50812). The ICD codes for AF are 42,731 for ICD-9 and I480-, I481-, I482-, I4891- for ICD-10. MIMIC-IV was approved by the Institutional Review Boards of Beth Israel Deaconess Medical Center and the Massachusetts Institute of Technology. Informed consent was not required because all protected health in the database was de-identified and did not influence clinical care [16]. All methods were performed in accordance with the relevant guidelines and regulations.

Outcome and data collection

The outcome was the in-hospital mortality in patients with HF combined with AF. Follow-up was conducted from admission to hospital until discharge or death. For patients with multiple admissions records, only data from the patient’s first admission were used. Data collection included age, gender (female, male), race (Black, White, others, unknown), ICU type [cardiac care unit (CCU), medical ICU (MICU), surgical ICU (SICU), others], HF type (acute, chronic, unspecified), weight, heart rate, systolic blood pressure (SBP), diastolic blood pressure (DBP), respiratory rate, temperature, saturation of peripheral oxygen (SPO₂), Charlson comorbidity index, Simplified Acute Physiology Score (SAPS) II, ventilation (no, yes), vasopressor (no, yes), antiarrhythmic (no, yes), antiplatelet (no, yes), anticoagulation (no, yes), Beta 1 receptor agonist (no, yes), coronary artery bypass grafting (CABG) (no, yes), catheter ablation (no, yes), white blood cell (WBC), platelet, hemoglobin, red blood cell distribution width (RDW), creatinine, blood urea nitrogen (BUN), glucose, sodium, potassium, chloride, bicarbonate, estimated glomerular filtration rate (eGFR), anion gap, and in-hospital follow-up time.

Predictive model construction and evaluation

All data were randomly divided into the training set and the testing set in a ratio of 7:3. The data from the training set was used for the construction of the model (feature selection, model training), and the data from the testing set was used for the internal validation of the model. Due to the imbalance in the incidence of outcomes in the dataset, the synthetic minority oversampling technique (SMOTE) was used to address the data imbalance during model training (training set). The SMOTE method balances the data set by oversampling to increase the number of minority class samples [17].

Feature selection was first performed using the Mann-Whitney U test, which was utilized to compare differences in characteristics between survivors and non-survivors. Among the initial 33 features, 8 features were excluded with a P > 0.05. Then, the remaining 25 features were screened using the least absolute shrinkage and selection operator (LASSO) regression model. LASSO is a compression estimation that compresses the regression coefficients of some features by constructing a penalty function. To ensure the stability and efficacy of the features, the feature set with a higher value is selected from the 10-fold cross-validation results, that is, the features whose coefficient is not 0 are retained. Finally, 22 of these 25 features were retained and included in the predictive model.

Four single models including the Random Forest model, eXtreme Gradient Boosting (XGBoost) model, Light Gradient Boosting Machine (LGBM), and K-Nearest Neighbor (KNN) model were constructed. In addition, a stacking ensemble model consisting of these four single models was established. Random Forest is an extension of Bagging integrated learning that uses decision trees as the basic classifiers. Random Forest generates many classifiers and combines their results by majority voting. XGBoost is an efficient gradient boosting decision tree algorithm, which integrates multiple weak learners into a strong learner by certain methods, i.e., the results of all the classifiers are accumulated to get the result. LGBM is a machine learning algorithm based on gradient boosting decision trees, which progressively improves the performance of a model by iteratively training multiple decision trees. KNN is one of the most basic and simple algorithms in the machine learning algorithm model, which can be used for both classification and regression by measuring the distance between different feature values. The stacking ensemble model of this study was performed using the categorical boosting technique. Stacking is an integration method that connects several different types of classification models through meta-classifiers, by combining several weak learners to obtain a model with stronger generalization ability. In the stacking ensemble model of this study, Random Forest, XGBoost, LGBM, and KNN were used as the base classifiers in the first stage, and the outputs obtained from each single model in the first stage were fed into the meta-classifiers in the second stage. Then, the meta-classifiers were fitted to the output meta-features of each classification model by the categorical boosting integration technique.

The performances of the four single models and the stacking ensemble model were assessed by the area under of the receiver operating characteristic curve (AUC) with 95% confidence interval (CI), sensitivity, specificity, as well as accuracy. The value of AUC is greater than 0.75 indicating that the model has good predictive ability. The modeling process of this study was shown in Fig. 1. The optimal parameters for the different models were presented in Supplementary Table 1.

Fig. 1

The modeling process of this study. MIMIC-IV, the Medical Information Mart for Intensive Care IV database; LASSO, the least absolute shrinkage and selection operator; XGBoost, eXtreme Gradient Boosting; LGBM, Light Gradient Boosting Machine; KNN, K-Nearest Neighbor; CV, cross-validation; AUC, the area under of the curve

Statistical analysis

Continuous data were presented as mean ± standard deviation (SD) or median and quartiles [M (Q1, Q3)], and categorical data were presented as numbers and percentages [n (%)]. Differences in continuous data were compared using the t-test or Wilcoxon signed-rank test, and differences in categorical data were compared by the Chi-square test or Fisher’s exact probability test.

For missing values, variables with ≥ 20% missing data (e.g., lactate) were excluded, and variables with < 20% missing data (e.g., respiratory rate) were interpolated using the random forest imputation. Sensitivity analysis was conducted before and after missing data processing. Descriptive statistical analyses were completed using R version 4.3.1 software (Institute for Statistics and Mathematics, Vienna, Austria). The construction and visualization of the model were performed using Python version 3.9.12 software (Python Software Foundation, Delaware, USA). A P-value < 0.05 was considered statistically significant.

Results

Characteristics of patients

Between 2008 and 2019, MIMIC-IV documented a total of 7,097 patients diagnosed with HF combined with AF. After excluding 1,099 patients who were admitted to ICU for less than 24 h, 5,998 patients were included in the analysis (Supplement Fig. 1). Of these 5,998 patients, 4,198 were assigned to the training set and 1,800 to the testing set. Table 1 presents the baseline characteristics of patients in the training set. Among these 4,198 patients, 624 (14.86%) died in-hospital and 3,574 (85.14%) survived. The mean age was 74.35 ± 11.48 years, 2,397 (57.1%) were males, and 3,112 (74.13%) were White. There were 1,040 (24.77%) patients from CCU, 1,428 (34.02%) patients from MICU, and 693 (16.51%) patients from SICU. For the type of HF, 2,205 (52.53%) patients had acute HF, 1,257 (29.94%) patients had chronic HF, and 736 (17.53%) patients had unspecified HF. The mean SAPS II score was 41.23 ± 12.69. The median length of in-hospital follow-up time and ICU stay was 7.89 (5.17, 12.91) days and 2.98 (1.83, 5.28) days, respectively.

Table 1 Baseline characteristics of patients with heart failure (HF) combined with atrial fibrillation (AF) in the training set

The results of LASSO regression on feature screening were shown in Fig. 2. The 10-fold cross-validation was used for LASSO regression, a λ-value (λ = 0.00187) was determined when the mean squared error (MSE) value was the smallest (Fig. 2A) and 22 features were selected based on the λ-value (Fig. 2B). A total of 22 features were used to construct the predictive model, including age, weight, heart rate, SBP, respiratory rate, SPO₂, Charlson comorbidity index, SAPS II, RDW, BUN, glucose, eGFR, anion gap, race (White), ICU type (MICU, SICU, others), HF type (chronic), ventilation (yes), vasopressor (yes), anticoagulation (yes), and beta-1 receptor agonist (yes). The correlation heat map for these 22 features was presented in Supplement Fig. 2.

Fig. 2

Feature selection using the least absolute shrinkage and selection operator (LASSO) regression. (A) changes in mean squared error (MSE) during LASSO regression screening; (B) Changes in the coefficient profiles during LASSO regression screening

Model performance for predicting in-hospital mortality in patients with HF combined with AF

Table 2 shows the performance of the Random Forest model, XGBoost model, LGBM model, KNN model, and stacking ensemble model in predicting in-hospital mortality in patients with HF combined with AF. The single models of Random Forest, XGBoost, LGBM, and KNN performed better in predicting the in-hospital mortality, with model AUCs of 0.818 (95%CI: 0.801–0.835), 0.827 (95%CI: 0.811–0.843), 0.811 (95%CI: 0.794–0.829), and 0.824 (95%CI: 0.808–0.840) in the training set, respectively. In the testing set, the AUC was 0.747 (95%CI: 0.717–0.777) for the Random Forest model, 0.755 (95%CI: 0.725–0.785) for the XGBoost model, 0.754 (95%CI: 0.724–0.784) for the LGBM model, and 0.746 (95%CI: 0.716–0.776) for the KNN model. Moreover, the stacking ensemble model had the highest AUC compared to the four single models, with AUCs of 0.837 (95%CI: 0.821–0.852) and 0.768 (95%CI: 0.740–0.796) for the training set and testing set, respectively. The receiver operating characteristic (ROC) curves of these models were shown in Fig. 3.

Table 2 Model performance in predicting in-hospital mortality in patients with HF combined with AF

Fig. 3

The receiver operating characteristic (ROC) curves of the models to predict in-hospital mortality in patients with heart failure (HF) combined with atrial fibrillation (AF). XGBoost, eXtreme Gradient Boosting; LGBM, Light Gradient Boosting Machine; KNN, K-Nearest Neighbor; AUC, the area under of the curve; CI, confidence interval

Comparisons of AUC between the stacking ensemble model and the four single models were presented in Table 3. In predicting in-hospital mortality in patients with HF combined with AF, the AUC of the stacking ensemble model was superior to that of the four single models on both the training set and the testing set (P < 0.05).

Table 3 Delong test for comparison of AUC of different models

In addition, the predictive performance of these models was analyzed in populations with different HF types (chronic, acute) (Supplement Table 2). The stacking ensemble model still showed good ability to predict in-hospital mortality in both patients with chronic HF combined with AF [Training set: (AUC = 0.907, 95%CI: 0.886–0.928); Testing set: (AUC = 0.800, 95%CI: 0.746–0.853)] and patients with acute HF combined with AF [Training set: (AUC = 0.828, 95%CI: 0.806–0.851); Testing set: (AUC = 0.743, 95%CI: 0.699–0.786)].

Discussion

Patients with HF combined with AF tend to have a poorer prognosis. This study constructed a model to predict in-hospital mortality in patients with HF combined with AF using four single models and the stacking ensemble model, respectively. Among the four single models, the LGBM model and the XGBoost model had good predictive ability for in-hospital mortality, with model AUCs of 0.754 and 0.755 in the testing set, respectively. The AUC of the stacking ensemble model was superior to that of the four single models, with AUCs of 0.837 and 0.768 for the training set and testing set, respectively.

Previous studies have reported models for predicting mortality in patients with HF [12, 18]. Li et al. used machine learning methods to build a model for predicting mortality in patients with HF, in which the XGBoost model had the best prediction, with an AUC of 0.824 on the training set [12]. Chen et al. demonstrated that the XGBoost model used in the prediction of in-hospital mortality in patients with HF outperformed conventional risk prediction methods, with an AUC of 0.771 for the model in the external validation set [18]. Segar et al. showed that machine learning models predicted HF mortality better than the traditional Get With The Guidelines-Heart Failure (GWTG-HF) model (C-statistic: 0.82 vs. 0.69) [19]. Since each machine learning model may have strengths and weaknesses in different situations, the stacking ensemble model achieves better model performance by integrating multiple machine learning models. In addition, HF and AF frequently coexist and are associated with a worse prognosis. However, models for predicting the risk of mortality in patients with HF combined with AF have not been reported. Our study used a machine learning approach to compare the performance of different models in predicting mortality in patients with HF combined with AF. All four single models (Random Forest, XGBoost, LGBM, KNN) showed good predictive ability of in-hospital mortality, and the AUC of the models in the training set exceeded 0.81. However, the predictive ability of these models on the testing set (AUC > 0.746) is slightly weaker than on the training set. This may be related to the fact that the data distributions of the training and testing sets are inconsistent (the SMOTE method was used in the training set to deal with the data imbalance problem) leading to difficulties in generalizing the models to the testing set. The stacking ensemble model consisting of these four single models showed better predictions than any of the single models. The AUC of the stacking ensemble model in the training set and testing set were 0.837 and 0.768, respectively.

In this study, the sensitivity and specificity of the model represent the model’s recognition of patients at risk of mortality and patients not at risk of mortality, respectively, while the accuracy represents the overall recognition performance of the model for patients at risk of mortality and patients not at risk of mortality. Our stacking ensemble model had a sensitivity of 0.812, a specificity of 0.705, and an accuracy of 0.721. Although the specificity of the model was not high, the model had a high sensitivity of 0.812. For models with mortality as an outcome, models with high sensitivity for the prediction of patient mortality may be more clinically valuable. However, the performance of the stacking ensemble model in the testing set was similarly weaker than in the training set. Moreover, chronic HF and acute HF were combined for analysis in this study. To test whether this was reasonable, we examined the performance of the model in the acute HF and chronic HF populations separately. The results demonstrated that the stacking ensemble model showed a good ability to predict in-hospital mortality in both patients with chronic HF combined with AF (AUC = 0.907) and patients with acute HF combined with AF (AUC = 0.828). This suggests that it is feasible to combine chronic HF and acute HF for predictive models with mortality as the outcome. The stacking ensemble model may provide a reference for a real mortality risk assessment tool for clinical practice. HF and AF interact, with common risk factors (e.g., age, hypertension, obesity) and comorbidities (e.g., valvular and ischemic and cardiac disease), neurohormonal and electrophysiologic changes, as well as alterations to cardiac myocytes combining to create an environment in which the heart is susceptible to HF and AF [20, 21]. Increased ventricular rate and arrhythmias caused by AF can shorten the left ventricular filling time, resulting in decreased cardiac output, which increases left atrial pressure. The increased cardiac filling pressures in HF can lead to atrial stretching, cardiac fibrosis, dysregulation of intracellular calcium regulation, and autonomic and neuroendocrine dysfunction, all of which may cause AF [22]. Weak atrial contraction impairs ventricular filling and worsens diastolic function. Ventricular remodeling with ventricular dilatation is a response to chronically elevated blood pressure, and this remodeling can lead to worsening of AF and HF [20, 23, 24].

In our predictive model, 22 characteristics (e.g., age, RDW, BUN, blood glucose, eGFR, anion gap) were used to construct the model. The relationship between these characteristics and HF or AF has also been reported. Age was an independent predictor of all-cause mortality in patients with HF, and age was found to significantly influence the effect of body mass index on patient mortality [25]. RDW reflects the variability of circulating red blood cell size, and a high RDW was associated with morbidity and mortality in patients with HF [26, 27]. The association between RDW and HF may be related to nutritional deficiencies, renal insufficiency, hepatic congestion, and inflammatory stress [26, 27]. BUN is a marker of kidney function that measures protein metabolism in the blood. Previous studies have shown that BUN is a key predictor of mortality in patients with HF [8, 9, 28]. Diabetes is a common risk factor for HF, and elevated blood glucose levels are an independent predictor of 30-day mortality in patients with HF [29, 30]. eGFR is an assessment of renal function, and impaired renal function is a prognostic indicator of acute and chronic HF [31]. Serum anion gap is used in the differential diagnosis of acid-base imbalance and metabolic acidosis, and high serum anion gap levels were linked to an increased risk of mortality in patients with HF [32]. However, the order of importance of these 22 features for the stacking ensemble model cannot be known. Since the results of the stacking ensemble model are based on the outputs of four single models, there are differences in the importance of these 22 features for each single model.

Our study constructed a stacking ensemble model for predicting in-hospital mortality in patients with HF combined with AF. The stacking ensemble model combines the strengths of multiple machine learning models and shows better predictive performance than a single model. As no predictive model for in-hospital mortality in HF combined with AF has been reported, our stacking ensemble model may provide a reference for a true mortality risk assessment tool in clinical practice.

Limitations

Some limitations of this study should be noted. First, this study mainly included ICU patients, and the model’s prediction of mortality risk in the general population needs to be further tested. Second, some biomarkers such as Troponin-T and N-terminal pro-B-type natriuretic peptides were not considered due to too many missing values (more than 60%), which may affect the prediction effect of the model. Third, the model lacks external validation, which is necessary before the model can be applied in clinical practice. Fourth, this study was unable to analyze HF into three phenotypes, HF with preserved ejection fraction (HFpEF), HF with reduced ejection fraction (HFrEF), and HF with midrange ejection fraction (HFmEF), and thus patients with HFrEF, HFpEF, and HFmrEF phenotypes could not be analyzed separately. Fifth, this study could not determine whether some patients had isolated or combined pulmonary hypertension associated with right ventricular dysfunction, making it impossible to perform a classification analysis based on isolated or combined pulmonary hypertension. Sixth, the cause of the patient’s heart failure is unknown.

Conclusions

This study constructed models for predicting in-hospital mortality in patients with HF combined with AF. The stacking ensemble model consisting of Random Forest, XGBoost, LGBM, and KNN has better AUC than any of single model. The stacking ensemble model may provide a reference for a true mortality risk assessment tool in clinical practice among patients with HF combined with AF. Moreover, external validation is necessary before the model can be applied in clinical practice.