Accuracy of artificial intelligence algorithms in predicting acute respiratory distress syndrome: a systematic review and meta-analysis

Abstract

Background

Acute respiratory distress syndrome (ARDS) is a serious threat to human life. Hence, early and accurate diagnosis and treatment are crucial for patient survival. This meta-analysis evaluates the accuracy of artificial intelligence in the early diagnosis of ARDS and provides guidance for future research and applications.

Methods

A search on PubMed, Embase, Cochrane, Web of Science, CNKI, Wanfang, Chinese Biomedical Literature (CBM), and VIP databases was systematically conducted, from their establishment to November 2023, to obtain eligible studies for the analysis and evaluation of the predictive effect of AI on ARDS. The retrieved literature was screened according to inclusion and exclusion criteria, the quality of the included studies was assessed using QUADAS-2, and the included studies were statistically analyzed.

Results

Among the 2, 996 studies, 33 were included in this meta-analysis, which showed that the pooled sensitivity of AI in predicting ARDS was 0.81 (0.76–0.85), the pooled specificity was 0.88 (0.84–0.91), and the area under the receiver operating characteristic curve (AUC) was 0.91 (0.88–0.93). The analyzed studies included 28 models, with a pooled sensitivity of 0.79 (0.76–0.82), a pooled specificity of 0.85 (0.83–0.88), and an AUC of 0.89 (0.86–0.91). In the subgroup analysis, the pooled AUC of the AI models ANN, CNN, LR, RF, SVM, and XGB were 0.86 (0.83–0.89), 0.91 (0.88–0.93), 0.86 (0.83–0.89), and 0.89 (0.86–0.91), 0.90 (0.87–0.92), 0.93 (0.90–0.95), respectively. In an additional subgroup analysis, we evaluated the predictive performance of the AI models trained using different predictors. This meta-analysis was registered in PROSPERO (CRD42023491546).

Conclusion

AI has good sensitivity and specificity for predicting ARDS, indicating a high clinical application value. Algorithmic models such as CNN, SVM, and XGB have improved prediction performance. The subgroup analysis revealed that the model trained using images combined with other predictors had the best predictive performance.

Background

Acute respiratory distress syndrome (ARDS) is a severe condition, affecting more than 3 million patients worldwide every year. Further, the mortality rate of severe ARDS can reach up to 46.1%, indicating a severe threat to patients’ lives [1]. The etiology of ARDS is complex and its disease progression is rapid, thus accurate early prediction, diagnosis, and individualized treatment plans are crucial for improving patient survival rates. However, the early identification of potentially high-risk ARDS requires the timely integration and analysis of basic patient information, disease characteristics, laboratory assays, imaging data, and respiratory mechanical characteristics for predictive analytics [2]. Developing methods or tools to accurately predict ARDS early is crucial for effectively treating patients with ARDS.

With the rapid development of artificial intelligence (AI) technology, its application in medicine has attracted widespread attention. AI technologies, such as machine learning and deep learning, show great potential for analyzing medical images, bioinformatics, and clinical decision support. Their ability to process data and recognize patterns makes them powerful tools for processing large-scale clinical data. These technologies have been applied to the early diagnosis of diseases, patient management, and prognosis assessment in clinical medicine [3, 4].

Patients with ARDS often exhibit complex and highly heterogeneous conditions, accompanied by an abundance of clinical, biomarker, and imaging data. This establishes a solid informational foundation for AI research in the field of ARDS. Studies have demonstrated that AI technology can effectively be utilized for early prediction of ARDS, by conducting in-depth analysis of clinical data, imaging materials, and monitoring information. This aids doctors in achieving more rapid and precise diagnoses and predictions, thereby facilitating timely interventions and improving patient outcomes [5]. However, no meta-analyses of the efficacy of AI in the prediction and diagnosis of ARDS have been conducted. A search on research databases shows that studies on AI in ARDS have been proliferating, especially since 2021. Hence, this study conducted a meta-analysis of AI prediction for ARDS to evaluate the accuracy of AI in predicting ARDS, providing useful guidance for future research and applications.

Methods

This meta-analysis was conducted according to the PRISMA guidelines [6]. This meta-analysis was registered in PROSPERO (CRD42023491546).

Literature search strategy and screening

We searched PubMed, Embase, Cochrane, Web of Science, CNKI, Wanfang, Chinese Biomedical Literature (CBM), and VIP databases from their earliest available records up to November 2023.The search terms were “acute respiratory distress syndrome,” “artificial intelligence,” “computer intelligence,” “machine learning,” “computer reasoning,” “deep learning,” and “random forest.” (The specific search strategy is described in Supplementary Material 3)

Inclusion and exclusion criteria

Articles were screened against the inclusion criteria by two independent researchers, first by title or abstract. If this step did not provide clear results, the entire text was examined to determine whether the article satisfied the inclusion criteria. Disagreements between researchers were resolved by consulting a third researcher.

The studies included in this paper were retrospective, and satisfied the following inclusion criteria: (1) literature in different languages; (2) studies conducted in patients with ARDS or those likely to develop ARDS; (3) studies that provide direct or indirect data to calculate the true positive (TP), false positive (FP), true negative (TN), and false negative (FN) values of the study to construct a complete four-fold table; (4) studies that present clear descriptions of the AI models and predictors used; (5) studies that clearly illustrate the source of the dataset used.

The exclusion criteria were as follows: (1) studies with incomplete data in the literature and inability to obtain TP, FP, TN, and FN directly or indirectly; (2) reviews, conference reports, letters, and experiments with animals; and (3) Literature duplicating experimental data.

Data extraction and literature quality assessment

Information was extracted and crosschecked independently by two researchers. Disagreements were resolved through discussion or consultation with a third researcher. We extracted the results from the validation or test sets of the study, and when there were no clear grouping in the text, we analyzed them using the total sample size. Data extracted included authors, year of publication, study population, study area, study type, sample size, prevalence, cross-validation method, TP, FP, TN, FN, sensitivity, and specificity. Quality evaluation of the included studies was performed using QUADAS-2 [7].

Statistical analysis

Statistical analysis was performed for the included studies. Sensitivity, specificity, positive likelihood ratio (PLR), negative likelihood ratio (NLR), and diagnostic odds ratio (DOR) were summarized using a bivariate mixed-effects model (MIDAS), and a summary receiver operating characteristic (SROC) curve was plotted, with the area under the curve (AUC) value calculated. All results were expressed with 95% confidence intervals. Calculate the Spearman correlation coefficient to detect heterogeneity caused by threshold effects. The magnitude of heterogeneity was assessed using the I² statistic. Heterogeneity was considered high if I² was > 50%, and sources of heterogeneity were explored through meta-regression. The stability of the results of diagnostic studies is tested through sensitivity analysis. The Deeks test was employed to assess the publication bias of the included studies, which were deemed to have publication bias when P < 0.05.

Results

Literature search results and characteristics of the included studies

A total of 2, 996 studies were searched through databases. After removing duplicates, the titles and abstracts of 1919 studies were reviewed, and the full text of 126 studies was screened. Ultimately, 33 studies were included in the pooled analysis [2, 8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39]. The specific literature screening process is shown in Fig. 1. The characteristics of the included studies are summarized in Table 1. The selected studies were conducted in eight countries and regions, with 21 single-center and 12 multicenter studies. All 33 studies were retrospective.

Fig. 1

Literature screening flow

Table 1 Characteristics of the included studies

Literature quality evaluation

According to the QUADAS-2 tool, the overall risk of bias in patient selection was “high” for 1 study and “unclear” for 6 studies. The risk of bias related to the index test and the flow and timing was “low” for all studies. The risk of bias for the reference standard test was “unclear” for 6 studies. In terms of overall applicability, the risk of patient selection bias was “unclear” for 5 studies, and the risk of bias for the reference standard test was “unclear” for 6 studies. (Supplementary Material 2 Figures S1, S2)

Results of the meta-analysis

Best models

The performance of the best AI model in predicting ARDS was evaluated in the 33 studies. The pooled sensitivity was 0.81 (0.76–0.85), the pooled specificity was 0.88 (0.84–0.91), the pooled PLR was 6.66 (4.97–8.93), the pooled NLR was 0.22 (0.17–0.27), the pooled diagnostic odds ratio (DOR) was 31 (20–48), and the overall pooled AUC was 0.91 (0.88–0.93). (Figures 2, 3, 4 and 5) After excluding seven studies with patient selection bias risk and six studies with reference standard test bias risk, it was found that their impact on the final results was very limited, with AUC values of 0.91 (0.88–0.93) and 0.91 (0.89–0.94), respectively. (Supplementary Material 1 Table S2) Five studies included external test data, and 12 studies were multicenter studies. Their pooled AUC values were 0.91 (0.88–0.93) and 0.92 (0.89–0.94), respectively, further validating the reliability of our research findings. (Supplementary Material 1 Table S3)

Fig. 2

Forest plots of the pooled sensitivity and specificity for best models

Fig. 3

Forest plots of the pooled PLR and NLR for best models

Fig. 4

Forest plots of the pooled DOR for best models

Fig. 5

SROC of best models for predicting ARDS

All models

A total of 28 models predicted ARDS. The pooled sensitivity was 0.79 (0.76–0.82), the pooled specificity was 0.85 (0.83–0.88), the pooled PLR was 5.37 (4.55–6.35), the pooled NLR was 0.25 (0.21–0.28), the DOR was 22 (17–28), and the overall pooled AUC was 0.89 (0.86–0.91). (Supplementary Material 2 Figures S3-S6)

Different types of AI models

In this meta-analysis, we analyzed the performance of XGB, RF, LR, CNN, SVM, and ANN models in predicting ARDS. Their pooled AUCs were 0.93 (0.90–0.95), 0.89 (0.86–0.91), 0.86 (0.83–0.89), 0.91 (0.88–0.93), 0.90 (0.87–0.92), and 0.86 (0.83–0.89), respectively. The XGB was the best predictive model, with the pooled sensitivity of 0.84 (0.75–0.89), the pooled specificity of 0.89 (0.83–0.93), the pooled PLR of 7.66 (4.92–11.92), the pooled NLR of 0.19 (0.12–0.28), the pooled DOR of 41 (20–85) (Table 2; Figs. 6, 7, 8 and 9). (Forest plots and SROC curves for the other analyzed models are shown in Supplementary Material 2 Figures S7-S26.)

Fig. 6

Forest plots of the pooled sensitivity and specificity for XGB models

Fig. 7

Forest plots of the pooled PLR and NLR for XGB models

Fig. 8

Forest plots of the pooled DOR for XGB models

Fig. 9

SROC of XGB models for predicting ARDS

Table 2 Different types of AI models

Models with different predictors

The effectiveness of different predictors as variables to train models for predicting ARDS was analyzed by training the models with images, images combined with other predictors, mechanical ventilation parameters, laboratory assays, and other predictors. Their pooled AUC were 0.90 (0.88–0.93), 0.92 (0.89–0.94), 0.87 (0.83–0.89), 0.91 (0.88–0.93), and 0.78 (0.74–0.81) respectively. In particular, the model trained with images combined with other predictors exhibited the best prediction with the pooled sensitivity of 0.85 (0.80–0.89), the pooled specificity of 0.86 (0.82–0.89), the pooled PLR of 6.10 (4.98–7.46), the pooled NLR of 0.18 (0.14–0.22), and the pooled DOR of 35 (29–41) (Table 3; Figs. 10, 11, 12 and 13). (Forest plots and SROC curves for training models for different predictors are shown in Supplementary Material 2 Figures S27-S42) (Details of the predictors are shown in Supplementary Material 1 Table S1)

Fig. 10

The pooled sensitivity and specificity for models trained with images combined with other predictors

Fig. 11

The pooled PLR and NLR for models trained with images combined with other predictors

Fig. 12

The pooled DOR for models trained with images combined with other predictors

Fig. 13

SROC of models trained with images combined with other predictors for predicting ARDS

Table 3 Models with different predictors

Model trained with specific factors

We conducted a thorough analysis of models trained using specific predictive variables. Specifically, CNNs were primarily trained using Images data, achieving an AUC of 0.91 (0.88–0.93); LR focused on Mechanical ventilation parameters, yielding an AUC of 0.80 (0.76–0.83); and Laboratory assays were utilized to train LR, ANN, and SVM, with their respective AUCs being 0.88 (0.85–0.90), 0.92 (0.89–0.94), and 0.91 (0.89–0.94). (Supplementary Material 1 Table S4)

Publication bias detection

The publication bias test using Deeks showed P = 0.95 (P > 0.05). The Deeks funnel plot (Fig. 14) revealed that the angle between the regression line and the DOR axis was close to 90, and no significant asymmetry was present, suggesting a low likelihood of publication bias.

Fig. 14

Funnel plot of studies included in the meta-analysis

Heterogeneity analysis

Threshold effect analysis

The threshold effect analysis showed a Spearman correlation coefficient of 0.090 and a P-value of 0.619. There was no threshold effect between the studies in this inclusion.

Meta-regression analysis

In the combined analysis of the optimal models from 33 studies, significant heterogeneity was observed in both sensitivity and specificity (with I² values of 92.86% and 99.00%, respectively). To explore the underlying causes of this heterogeneity, we conducted a meta-regression analysis, examining factors such as study region, definition of ARDS, type of research center (multi-center vs. single-center), number of patients, incidence of ARDS, type of AI model, and type of predictor variable. As illustrated in fig. 15 all factors except for the type of research center were considered as sources of heterogeneity (with P-values < 0.05).

Fig. 15

Meta-regression of heterogeneous sources

Sensitivity analysis

We conducted a comprehensive sensitivity analysis on the included studies, with particular attention to the impact of excluding each of the studies by Suganya D/2023, Wanyue Zhang/2023, Anoop Mayampurath/2020, and Yu Wang/2023. After recalculating the I² values for the remaining studies, we found that heterogeneity still persisted (I² value > 50%). Using MIDAS to integrate the effect sizes, the combined AUC value did not exhibit significant fluctuation compared to the total combined AUC value of 0.91 (0.88–0.93). (Fig. 16; Table 4) This suggests that these four studies have a limited impact on the accuracy of AI in predicting ARDS, and the research findings are stable and reliable. Additionally, we performed sensitivity analyses on various subgroups and observed that after excluding the study by Wanyue Zhang/XGB 2023, the combined AUC value for the XGB group decreased to 0.82 (with a total combined AUC value of 0.93). Similarly, after removing the study by Suganya D/Mask R-CNN 2023, the combined AUC values for the CNN group and the image group decreased to 0.80 (with a total combined AUC value of 0.91) and 0.82 (with a total combined AUC value of 0.90), respectively. (Supplementary Material 1 Table S5, Supplementary Material 2 Figs. 43, 44, 45, 46, 47, 48, 49, 50 and 51).

Fig. 16

The analysis of sensitivity

Table 4 The results of excluding each of the four studies one by one

Discussion

AI, with several powerful algorithms, has significantly progressed the fields of image recognition, analysis of big data, natural language processing, and decision-making assistance, substantially developing various medical fields. In recent years, AI has been gradually applied to predict, recognize, and diagnose multiple diseases. Bacci et al. systematically evaluated the use of AI models in AKI prediction. The authors reported that the AUC results of the included studies could reach up to 0.70 [40]. Moreover, Silva et al. also indicated that AI performed well to detect prostate cancer with an optimal sensitivity of 1.0 (0.93–1.0) and a specificity of 0.78 (0.64–0.89) [41]. Several studies have conducted systematic evaluations on the application of AI in ARDS, yet these studies have solely focused on analyzing AI models that interpret imaging data [42,43,44,45]. In contrast, the research by Muhammed Rashid et al. provides a comprehensive examination of the multifaceted applications of AI in ARDS, highlighting its promising role in diagnosis, risk stratification, severity prediction, management, mortality prediction, and decision-making analysis [46]. Similarly, T.K. Tran et al. have analyzed the utilization of machine learning in ARDS research, encompassing a broad range of research areas spanning seven distinct categories, notably including diagnosis [47]. However, none of these studies conducted a meta-analysis to delve deeper into the research questions. This study addresses this gap by assessing the accuracy of AI in predicting ARDS, as well as the actual effectiveness and advantages of AI in managing ARDS.

In this meta-analysis, the best models exhibited a pooled sensitivity of 0.81 (0.76–0.85), a pooled specificity of 0.88 (0.84–0.91), and a pooled AUC of 0.91 (0.88–0.93). This suggests that AI can recognize sick and non-sick people with credible results. The results of subgroup analysis conducted on various artificial intelligence models indicate that XGB demonstrates optimal performance, with an AUC value of 0.93 (0.90–0.95), which aligns with the general trends observed in machine learning. As an ensemble tree model, XGB excels in handling data with complex interactions and high dimensionality. Compared to other machine learning models, XGB not only leads in prediction accuracy but also exhibits superior training efficiency and scalability [48]. A study suggests that AI models with high AUC values should incorporate additional predictive indicators [48]. However, Izadi et al. suggested that a large number of predictors might not improve the predictive performance of a model. In particular, selecting appropriate predictors may be crucial, which is more practical in clinical practice [26]. In this meta-analysis, the model trained with images combined with other predictors had an AUC of 0.92 (0.89–0.94), showing the best prediction performance. The AUCs of the models trained for images and laboratory assays were 0.90 (0.88–0.93) and 0.91 (0.88–0.93), respectively. This suggests that imaging and laboratory assays may be more suitable for training ARDS prediction models. The two subgroups analyzed in this study provide useful references for selecting appropriate models and predictors in future research.

We further observed that CNNs, a deep learning model specialized for processing image and video data, yielded promising prediction outcomes. Models trained based on laboratory examinations encompassed LR, ANN, and SVM. Among them, ANNs demonstrated the superior predictive performance, attributed to their high accuracy, parallel distributed processing capabilities, and robustness against noise. In our dataset, LR models were more frequently utilized in conjunction with mechanical ventilation parameters for training; however, their predictive performance was not satisfactory, with an AUC of 0.80 (0.76–0.83). This may be attributed to the fact that LR models typically only consider pairwise interactions while neglecting other variables and are susceptible to disturbances from nonlinear relationships among predictor variables [35, 49]. In conclusion, selecting an appropriate model according to the specific task and data characteristics is crucial for achieving satisfactory prediction results and performance.

The implementation of artificial intelligence algorithms for real-time prediction aids in timely stratified care for high-risk patients with ARDS, optimizes resource allocation in ICUs, and enhances treatment efficiency. Our comprehensive analysis, integrating numerous related studies, reveals that the application of artificial intelligence in predicting ARDS is accurate and reliable. However, significant challenges persist in data sharing and regulation during the implementation of AI algorithms. To address these issues, the establishment of unified data standards and sharing mechanisms, along with the enhancement of data regulation, is imperative.

Protecting patient privacy is a fundamental principle that must be adhered to in the clinical application of AI. The studies included in this meta-analysis have adopted rigorous data encryption and anonymization measures to ensure the security of patient data, and future research should also focus on this to ensure the reasonable, safe, and sustainable application of AI technology.

This study acknowledges its limitations as well. Firstly, all 33 studies included were retrospective, leading to a high degree of heterogeneity among them. Meta-regression analysis identified various sources of heterogeneity, including study region, definition of ARDS, type of research center, patient number, incidence of ARDS, type of AI model used, and type of predictor variables utilized. High heterogeneity was also observed within subgroups. Different models vary in algorithmic principles, data processing methods, and predictive capabilities, which may serve as sources of heterogeneity. Furthermore, different studies employed various predictors for model training, reflecting different aspects of patients’ pathophysiological processes with different sensitivities and specificities, thereby influencing the study results. Sensitivity analysis indicated a lack of robustness in the study outcomes for the XGB group. Within this group, the study by Wanyue Zhang/XGB 2023 reported good prediction results, potentially due to the selection of more appropriate predictors. Upon excluding this study, the combined effect size of the remaining studies significantly decreased. The study by Suganya D/Mask R-CNN 2023 had a large sample size and excellent predictive performance; after excluding this study, the combined effect sizes of both the CNN group and the imaging group declined. Future studies should include more high-quality research for further analysis. Secondly, during the literature retrieval process, we only searched English and Chinese databases, and all articles eventually included in the meta-analysis were published in either English or Chinese. This outcome may inadvertently exclude important studies in other languages, introducing language bias and affecting the comprehensiveness and accuracy of the results. While we did conduct a comprehensive search across multiple authoritative databases and adhered to a systematic literature review method, even the most thorough search may fail to capture all relevant studies, particularly those that are unpublished or difficult to access, which may result in selection bias in the literature. Lastly, the included studies lacked data on model calibration metrics (such as the Brier score), which hindered our accurate assessment of the model’s reliability. The interpretability of a model is crucial for understanding its practicality. In the studies included in this paper, although various variables such as imaging, vital signs, and laboratory indicators were used to train the model, the lack of data that can quantify variable contributions, such as SHAP values, prevented us from conducting an in-depth integrated analysis of these variables

Conclusion

AI models show good sensitivity and specificity for predicting ARDS, promising future clinical applications. Among them, CNN, SVM, and XGB models exhibited the best prediction performance. The subgroup analyses revealed that models trained with images combined with other predictors showed the best predictive performance. A future work could focus on selecting the model and predictors according to the specific task and data characteristics.