Automated machine learning for early prediction of systemic inflammatory response syndrome in acute pancreatitis

Abstract

Background

Systemic inflammatory response syndrome (SIRS) is a frequent and serious complication of acute pancreatitis (AP), often associated with increased mortality. This study aims to leverage automated machine learning (AutoML) algorithms to create a model for the early and precise prediction of SIRS in AP.

Methods

This study retrospectively analyzed patients diagnosed with AP across multiple centers from January 2017 to December 2021. Data from the First Affiliated Hospital of Soochow University and Changshu Hospital were used for training and internal validation, while testing was conducted with data from the Second Affiliated Hospital. Predictive models were constructed and validated using the least absolute shrinkage and selection operator (LASSO) and AutoML. A nomogram was developed based on multivariable logistic regression (LR) analysis, and the performance of the models was assessed through receiver operating characteristic (ROC) curves, calibration curves, and decision curve analysis (DCA). Additionally, the AutoML model’s effectiveness and interpretability were assessed through DCA, feature importance, SHapley Additive exPlanation (SHAP) plots, and locally interpretable model-agnostic explanations (LIME).

Results

A total of 1,224 patients were included, with 812 in the training cohort, 200 in validation, and 212 in testing. SIRS occurred in 33.7% of the training cohort, 34.0% in validation, and 22.2% in testing. AutoML models outperformed traditional LR, with the deep learning (DL) model achieving an area under the ROC curve of 0.843 in the training set, and 0.848 and 0.867 in validation and testing, respectively.

Conclusion

The AutoML model using the DL algorithm is clinically significant for the early prediction of SIRS in AP.

Introduction

Acute pancreatitis (AP) is a frequent cause of severe abdominal pain, occurring due to the activation of pancreatic enzymes that cause self-digestion of the pancreas and initiate a chain of inflammatory responses, potentially leading to various complications [1]. While the symptoms of acute pancreatitis (AP) typically resolve on their own in most patients, approximately 20% may develop severe complications, including prolonged organ failure and the onset of systemic inflammatory response syndrome (SIRS) [2]. If SIRS persists, it heightens the risk of persistent organ dysfunction, potentially progressing AP to severe acute pancreatitis (SAP) [3]. SIRS is a key factor in the progression of SAP and is closely linked to its severity, which can lead to multiple organ failure and even death [1, 4, 5]. Therefore, early detection and treatment of SIRS are essential.

In the early stages of AP, the condition starts as a localized sterile inflammation of the pancreas and quickly progresses to SIRS, often accompanied by compensatory anti-inflammatory response syndrome (CARS) [6]. The mechanism of SIRS is complex and is associated with the anti-inflammatory response mediated by Th2/Treg differentiation [7,8,9]. Studies have shown that gut microbiota translocation, the release of extracellular ATP, and activation of endogenous coagulation pathways are all closely related to inflammatory factors [7, 10, 11]. Additionally, studies have also shown that elevated serum levels of Pentraxin 3 (PTX3) are closely associated with SIRS and ultimately result in fatal outcomes in critically ill patients [12]. Staubli SM et al. indicates that PTX3 performs worse than C-reactive protein (CRP) and Acute Physiology and Chronic Health Evaluation II (APACHE II) in predicting SIRS. All three biomarkers exhibited weak predictive discrimination, with area under the receiver operating characteristic curve (AUROC) values of 0.54, 0.69, and 0.69, respectively. When CRP and PTX3 are used in combination, the AUC value increases to 0.7 [13].

Most of these studies relied on traditional logistic regression (LR) models, with some not fully addressing the problem of multicollinearity among variables. Recently, the growing use of machine learning (ML) in healthcare has attracted attention, with both supervised and unsupervised learning offering powerful algorithms for handling large amounts of clinical data. Unlike traditional LR models, ML provides clear advantages in predicting patient outcomes, complications, and prognosis [14, 15]. Traditional ML algorithms typically include Support Vector Machines (SVM), Artificial Neural Networks (ANN), Bayesian learning, and Random Forests. A newer approach, AutoML, automates the selection of algorithms and hyperparameters to build customized models tailored to specific data. In contrast to traditional ML, AutoML employs techniques like intelligent early stopping, cross-validation, regularization, and hyperparameter optimization, allowing for the development of more accurate models in less time.

In this study, we will utilize the H2O AutoML platform to develop and validate a range of ML models for the early and accurate prediction of SIRS in patients with AP. The performance of these models will be compared with that of the traditional LR method.

Materials & methods

Patients

This study retrospectively gathered clinical data from hospitalized AP patients at Changshu Hospital, the First Affiliated Hospital of Soochow University, and the Second Affiliated Hospital of Soochow University, covering the period from January 2017 to December 2021. All three hospitals are tertiary teaching institutions located in Suzhou, Jiangsu, China. The data from Changshu Hospital and the First Affiliated Hospital were split into a training set and an internal validation set in an 8:2 ratio, while the data from the Second Affiliated Hospital served as an independent test set.

The diagnostic criteria for AP were based on the revised 2012 Atlanta classification [16]. To diagnose acute pancreatitis (AP), a patient must satisfy at least two of the following three criteria: (1) typical abdominal pain; (2) serum amylase levels exceeding three times the normal upper limit; and (3) imaging findings consistent with AP [16]. Adults aged 18 years and older who meet the above criteria will be included in this study. According to the criteria for SIRS, a diagnosis of AP can be made in patients who exhibit two or more of the following conditions: (1) a body temperature greater than 38 °C or less than 36 °C; (2) a heart rate exceeding 90 beats per minute; (3) a respiratory rate greater than 20 breaths per minute or arterial carbon dioxide tension (PaCO2) less than 32 mm Hg; and (4) a white blood cell count greater than 12,000 cells/mm³, less than 4,000 cells/mm³, or more than 10% immature (band) forms [17]. During hospitalization, patients who meet the diagnostic criteria for SIRS will be classified into the SIRS group, while those who do not meet the criteria will be classified into the non-SIRS group. Patients with chronic liver disease, chronic kidney disease, hematological disorders, recurrent, chronic, traumatic, or idiopathic pancreatitis, pancreatic cancer, a history of pancreatic surgery, those who have undergone chemoradiotherapy, and pregnant patients will be excluded from the study. All included patients will receive treatment following the established guidelines for managing AP. The ethics committee of Changshu Hospital Affiliated to Soochow University has approved this study.

Data collection

Demographic information, clinical data, and details of comorbidities were extracted from electronic medical records. Relevant laboratory tests, including complete blood counts, coagulation tests, and serum biochemical markers, were collected within 24 h of admission. A total of 34 variables were analyzed, with more details provided in Supplementary Table S1. Missing data were assumed to be random and were imputed using a random forest algorithm through the “mice” package in R software [18]. The study flowchart is shown in Fig. 1.

Fig. 1

The flowchart of this study

Statistical analysis

In this study, we used the Shapiro-Wilk test to assess whether continuous variables followed a normal distribution. Continuous variables were expressed as mean ± standard deviation (SD) if they followed a normal distribution, and as median (interquartile range) if not. Categorical variables were presented as frequencies. For group comparisons, categorical variables were analyzed using Pearson’s Chi-square test or Fisher’s exact test, while continuous variables were compared using the nonparametric Mann-Whitney U test. A two-sided p-value of less than 0.05 was considered statistically significant. All statistical analyses were performed using R software (version 4.2.1) with the following packages: H2O (version 3.36.0.2), tableone (version 0.12.0), tidyverse (version 1.3.0), tidyquant (version 1.0.2), and lime (version 0.5.1).

Logistic regression algorithms and automated machine learning algorithms

To address multicollinearity among variables, the LASSO regression model was applied using the “λmin + 1se” criterion for univariate analysis. Independent risk factors were identified through binary LR with a backward stepwise method, and a nomogram was constructed. The model’s predictive ability was evaluated using ROC curves, calibration curves, and decision curve analysis (DCA).

For the ML component, this study utilized the H2O package (www.h2o.ai) for AutoML, which automatically selects suitable algorithms and combines them into various ensemble models. These included the default Random Forest (DRF), a random grid of Gradient Boosting Machines (GBM), Extremely Randomized Forest (XRF), a random grid of Deep Learning (DLs), and a fixed grid of Generalized Linear Models (GLMs). Hyperparameters were optimized through a grid search with 5-fold cross-validation on the training set, and the area under the curve (AUC) was used to evaluate the performance of different hyperparameter combinations. AutoML visualizations included feature importance, SHapley Additive exPlanation (SHAP) plots, and Local Interpretable Model-agnostic Explanations (LIME). Feature importance highlights each feature’s contribution to the ML model’s predictions. SHAP provides insights into how specific features influence individual predictions, offering clear, interpretable explanations. LIME interprets predictions by fitting a locally interpretable model around a specific instance.

Results

Baseline characteristics

The clinical characteristics and baseline data of 1,224 patients are detailed in Table 1. Based on the diagnostic criteria, 389 patients (31.8%) were identified as having developed SIRS. For model validation, patients from Changshu Hospital and the First Affiliated Hospital of Soochow University were randomly allocated into a training set (n = 812, 80%) and a validation set (n = 200, 20%). Furthermore, patients from the Second Affiliated Hospital of Soochow University served as an independent test set (n = 212). Within the training, internal validation, and test sets, 274 patients (33.7%), 68 patients (34.0%), and 47 patients (22.2%) respectively developed SIRS.

Table 1 Baseline characteristics of patients in training, validation and test groups

Development of prediction model

Univariate analysis was performed using the LASSO regression model with the “λmin + 1se (0.059)” criterion and 5-fold cross-validation to address multicollinearity among variables (Supplementary Figure S1). For multivariate analysis, backward stepwise LR identified three independent risk factors out of 34 variables, which were used to create a nomogram (Fig. 2). And, the results of the multivariate LR can be found in Supplementary Table 2. Calibration curves for the training, validation, and test datasets (Fig. 3) showed mean absolute errors of 0.049, 0.023, and 0.044, respectively, highlighting the LASSO model’s strong predictive accuracy compared to actual outcomes. DCA for the test set indicated that the LASSO model provided additional benefits when predicting SIRS probabilities between 10% and 90%, with a net benefit ranging from 1 to 17%. For instance, if a clinician estimated a 40% probability of SIRS in a patient, early intervention could offer a 6% additional benefit, equating to detecting 6 cases of SIRS without unnecessary treatments out of 100 patients. This approach outperforms the “treat none” strategy (represented by the horizontal line in Supplementary Figure S2), which assumes no true positives or false positives. Net benefit analysis confirms that the LASSO model improves patient outcomes regardless of individual preferences. The test set’s ROC curve (Supplementary Figure S3) achieved an AUC of 0.856, as detailed in Table 2.

Fig. 2

Nomogram of the LASSO model for the early prediction of SIRS

Table 2 Comparison of LR and automl models for early prediction of SIRS in the test cohort

Fig. 3

Calibration curve of the LASSO model in the training, validation, and test set

The study constructed 53 models using four ML algorithms: Deep Learning (DL), Gradient Boosting Machine (GBM), Generalized Linear Model (GLM), and Distributed Random Forest (DRF). Stacked ensemble models were omitted due to challenges in interpretation. Among these, the DL model demonstrated the best performance, achieving the highest AUC of 0.867 in the test cohort and effectively managing imbalanced data. Figure 4 highlights that in the GBM model, neutrophil count (N), white blood cell count (WBC), and C-reactive protein (CRP) were the most critical features, followed by lactate dehydrogenase (LDH), serum calcium (Ca²⁺), alkaline phosphatase (ALP), lymphocyte ratio (Lr), activated partial thromboplastin time (APTT), gamma-glutamyl transferase (GGT), and international normalized ratio (INR). N and CRP were common significant variables in both the GBM and LASSO models. The SHAP plot in Fig. 5 ranks the top ten important features in the GBM model: WBC, CRP, N, LDH, Ca²⁺, Lr, ALP, GGT, prothrombin time (PT), and total bilirubin (TB). Variables closer to a value of 1 were more strongly associated with a higher likelihood of SIRS progression. For example, the red portion of CRP, concentrated on the right of axis = 0, suggests that elevated CRP levels in patients with AP are linked to an increased risk of SIRS. The LIME plot of the GBM model demonstrates the contributions of key variables to SIRS progression, with Supplementary Figure S4 showing that both cases 1 and 2 were predicted to have a high risk of SIRS. In both cases, Ca²⁺ was the most influential feature, while N contributed the least to the risk of SIRS progression.

Fig. 4

Variable importance of the GBM model in the training set

Fig. 5

SHAP of the GBM model in the training set

DCA for the test set showed that AutoML models predicting SIRS probabilities between 10% and 100% could offer additional benefits ranging from 1 to 20%. For instance, if a clinician assessed a patient’s SIRS likelihood at 20%, early intervention could improve outcomes by at least 15%. This approach significantly outperforms the “treat none” strategy (represented by the horizontal line in Supplementary Figure S5), which results in no true positives or false positives.

Comparisons models developed by LR and automl

In the test set, the AUC values of the five models were as follows: the DL model had the highest AUC at 0.867, followed by the LASSO model at 0.856, the GLM at 0.853, the GBM at 0.833, and the DRF at 0.830. Among these models, the DL model achieved an AUC and accuracy both exceeding 0.80, demonstrating better predictive performance compared to the other models. Details are listed in Table 2.

Discussion

AP can be classified into three stages based on its pathological progression: the acute response phase, systemic infection phase, and residual infection phase. During the acute response phase, the self-digestion of pancreatic enzymes triggers the activation of inflammatory factors, accompanied by the activation of inflammatory cells, the release of chemokines, adhesion molecules, reactive oxygen species, platelet-activating factor (PAF), and endothelin. This causes local inflammation to escalate into SIRS [3, 5]. If SIRS persists, endotoxins and phospholipase A2, through systemic circulation and the mesenteric lymphatic pathway, sustain and amplify the inflammatory cascade, potentially leading to more severe SIRS and even multiple organ dysfunction syndrome (MODS) [3]. Research by Tan et al. and Sharma et al. further found that AP patients with SIRS are more prone to developing infected pancreatic necrosis (IPN), and the duration of SIRS is comparable to APACHE II and CT Severity Index (CTSI) scores in predicting IPN, MODS, and mortality [19, 20]. Studies have shown that early onset of SIRS in AP is usually associated with a higher mortality rate [21]. Therefore, early and accurate assessment of AP with SIRS is crucial for improving prognosis.

In this study, we developed and validated a range of models for the early detection of SIRS using both AutoML and LR methods. Compared to traditional univariate and multivariate analyses, AutoML significantly reduced the time required while improving accuracy, which greatly enhanced operational efficiency. Additionally, the ensemble model integrated multiple machine learning algorithms and used several classifiers to combine prediction outcomes, resulting in improved overall performance [22]. We applied four AutoML algorithms (GBM, DRF, GLM, and DL) for the early prediction of SIRS, with all models outperforming standard algorithmic techniques. The DL model achieved the highest AUC in the test set, a critical indicator for evaluating model effectiveness. Since our goal was to quickly identify AP patients at risk of developing SIRS, sensitivity and accuracy were also crucial assessment metrics. Both the DRF and GLM models demonstrated sensitivities above 0.950, while the DL model achieved an accuracy of 0.807. Therefore, in this study, the DL model was found to be the most successful.

WBC and N are important indicators of inflammatory response and are widely used in the diagnosis and prognostic assessment of inflammatory diseases [23, 24]. When the body experiences severe infection or an inflammatory response, both WBC and N levels typically rise significantly, reflecting the immune system’s reaction to pathogens or injury [24, 25]. Therefore, it is not surprising that both are considered predictor variables for SIRS. CRP is a common acute phase reactant closely related to inflammatory responses. It is synthesized and secreted by the liver and is essential for maintaining normal immune function. Under normal circumstances, CRP levels remain within a standard range, but they rise rapidly when the body is subjected to an inflammatory cell invasion. CRP has excellent stability and detectability, making its level changes highly sensitive indicators of the degree of inflammatory response and cellular or tissue damage [13]. Relevant studies indicate that changes in CRP levels can occur significantly earlier than the onset of clinical symptoms [24]. Some research also suggests that the CRP level at admission serves as a predictive factor for the severity of acute pancreatitis [26].

Kolodecik TR et al. found that measuring LDH activity within 12 h of onset can serve as a biomarker for the early prediction of acute pancreatitis (AP) prognosis, with a sensitivity of 63.6% and specificity of 89.6% [27]. Additionally, LDH is also a key variable in the RANSON score [28]. In an in vitro study of lymphocytes from burned rats, it was found that the transformation activity of lymphocytes significantly increased, accompanied by elevated levels of interleukins (IL-1, IL-2, IL-6), which was consistent with changes in cell proliferation [29]. Furthermore, the study indicated that in addition to macrophages and neutrophils, lymphocytes also expressed Toll-like receptor 4 (TLR4) and nuclear factor kappa B (NF-κB), both of which provide the basis for initiating and promoting the inflammatory response [30]. TLR4, one of the earliest discovered TLR subtypes, is a type I transmembrane protein composed of extracellular, I transmembrane, and intracellular regions. It is the only confirmed receptor and signaling molecule within the TLR family that recognizes lipopolysaccharide (LPS), a key endotoxin component, thus playing an initiating role in the inflammatory response [31]. When LPS binds to TLR4 on lymphocytes, it activates the main signaling pathway, the NF-κB pathway [32]. This pathway triggers and promotes the transcription of various genes related to the inflammatory response, encoding a range of cytokines and inflammatory mediators, including TNF-α, intercellular adhesion molecule-1, IL-1, IL-2, IL-6, and prostaglandins, thereby driving the inflammatory process. If these cytokines and inflammatory mediators are excessively expressed and released, they can lead to a cascade of inflammatory mediators, rapidly escalating local inflammation into SIRS.

When the host’s immune system is unable to combat pathogenic factors, it may lead to hepatocyte damage, obstructed bile secretion, and disrupted bilirubin metabolism, resulting in the appearance of jaundice. Research indicates that the dysfunction of stem cells during the development of sepsis may be associated with the release of inflammatory cytokines, which is often closely related to the progression of SIRS. SIRS is a systemic inflammatory response of the body to infection or other pathogenic factors, characterized by the excessive release of inflammatory cytokines. GGT, ALP, and TB are widely distributed in the biliary system and are primarily excreted through the bile ducts, making them important serum markers for assessing biliary obstruction [33]. In our study, patients with higher levels of these three markers had an increased risk of developing SIRS, which may be related to prolonged biliary obstruction and poor bacterial growth in the bile. Animal studies have also shown that the concentration of the GGT marker is closely associated with the risk of death from SIRS [34].

There is a close interaction between the inflammatory response and the coagulation mechanism. The inflammatory response can activate the coagulation system, while the coagulation mechanism also plays a regulatory role in the inflammatory response. During the development of sepsis, pathogens produce endotoxins and exotoxins, which stimulate the release of a large number of inflammatory cytokines and mediators, activating the coagulation system and leading to a hyper-coagulable state in the blood. At the same time, this response also inhibits the fibrinolytic and anticoagulant systems to varying degrees. The formation of microthrombi in the microvasculature can result in microcirculatory disturbances [35]. Therefore, it is not surprising that PT is used as a predictive indicator for SIRS. In the systemic inflammatory response of sepsis, inflammatory factors such as interleukin-1β and interleukin-6 can bind to calcium-sensing receptors (CaSR) present in the parathyroid gland and renal tubules, leading to a decrease in parathyroid hormone (PTH), 1,25-dihydroxyvitamin D, and blood calcium levels [36]. Vitamin D is closely related to calcium regulation, and Watkins has noted a high incidence of vitamin D deficiency in patients with sepsis. The SIRS can impair the expression of vitamin D receptors in immune cells and reduce the levels of vitamin D binding protein. Supplementing with vitamin D may have anti-inflammatory effects and could help improve sepsis and the coagulopathy associated with disseminated intravascular coagulation (DIC) [37].

This study has the advantage of utilizing AutoML to construct a series of models that are more accurate and sensitive in the early prediction of SIRS compared to traditional algorithms. The predictive factors in these models are based on routine detection indicators for AP, and our study is multicenter, facilitating the models’ application and promotion. However, there are some limitations to this study. First, novel biomarkers highlighted in recent research were not included, as they have not yet been widely implemented in clinical practice. Second, since this is a retrospective study, further prospective research is required to validate our findings. Finally, the sample size of this study is relatively small, and larger sample-size studies are needed to confirm our conclusions.

Conclusions

We developed and validated a series of models using the AutoML platform for the early prediction of SIRS in patients with AP. These models outperformed scoring systems constructed with traditional algorithms and may provide direction for the application of AutoML in future medical research. Additionally, the performance of the DL model is better than that of the other models.