Sharing reliable information worldwide: healthcare strategies based on artificial intelligence need external validation. Position paper

Abstract

Training machine learning models using data from severe COVID-19 patients admitted to a central hospital, where entire wards are specifically dedicated to COVID-19, may yield predictions that differ significantly from those generated using data collected from patients admitted to a high-volume specialized hospital for orthopedic surgery, where COVID-19 is only a secondary diagnosis. This disparity arises despite the two hospitals being geographically close (within20 kilometers). While machine learning can facilitate rapid public health responses, rigorous external validation and continuous monitoring are essential to ensure reliability and safety.

The Artificial Intelligence (AI) Act was released by the European Commission on December 9, 2023 while Machine Learning (ML) methodologies gain increasing interest worldwide [1–2]. ML technologies offer to the global community of patients, clinicians and policy makers the unprecedented opportunity to collect, elaborate and share a huge quantity of data in short time. Predictive models need to be validated in different settings and contexts to be reliable. In case of sudden global health crises, such models can support infection diagnosis and screening, predict individual outcomes and adopt effective preventative measures on time, like isolating individuals in the early contagion outbreak, developing differential pathways in care access, predicting recovery curves based on patient stratification, organizing intermediate care facilities and minimize hospitalization; these measures, in turn, can save healthcare systems from collapse [3].

However, we are far from reproducing good AI systems’ performance in different settings. Lack of generalizability makes the so-called reproducibility crisis a clinical (cf. poor predictions, automation bias) and moral issue (cf. iatrogenic harm), as the substantially different meaning of validation between medical literature and ML [4,5,6] is often neglected. In the medical context, validation generally implies confirming the consistent fulfillment of specific requirements for intended use. In contrast, ML validation tends to focus on fine-tuning model parameters or comparing model performances, also known among ML practitioners and researchers as “internal validation”. Internal validation is based on the same data used to develop and test a model or on data coming from the same facility. Another thing is external validation, which evaluates the model’s performance under varying conditions and datasets from different settings. An externally validated model is a more robust model, considered robustness as the capability of a system (a) to maintain its level of performance under any circumstances (ISO/IEC FDIS 22989:2022), (b) to have comparable performance on inputs dissimilar to those on which it has been trained (ISO/IEC TR 24029-1). Medical validation and machine-learning validation differ significantly in both scope and execution. While machine learning models are often validated based on historical datasets and performance metrics like accuracy, external medical validation is essential to assess how these models perform in real-world and diverse clinical environments. Providing reliable and timely information across different healthcare settings is critical to address the reproducibility crisis of AI applications in healthcare.

The literature reports many cases in which robust external validation (performed across especially diverse clinical settings and patient populations) would have revealed the limitations of the models before full deployment, allowing for recalibration or model redesign. The most known cases are IBM Watson for Oncology, Deep mind Retinal Screening for Diabetic Retinopathy and Epic’s Sepsis Prediction Model [7, 8]. IBM Watson for Oncology was developed to provide treatment recommendations for cancer patients. The system was initially trained on data from Memorial Sloan Kettering Cancer Center (MSKCC); however, when applied in different hospitals (particularly in Asia), it became evident that its recommendations were not always appropriate for local clinical settings [9,10,11,12]. The system had been trained and validated on a narrow dataset, which lacked diversity in terms of population, disease variants and treatment approaches. Deep Mind AI developed a deep learning model for detecting diabetic retinopathy from retinal images. Initial results published from testing in highly controlled clinical settings were promising [13, 14]. However, in real-world settings in rural Thailand, the model faced significant challenges. Factors like image quality, variability in equipment, and differences in technician expertise led to poor model performance [15,16,17]. Epic Systems is a widely used electronic health records company which developed a sepsis prediction model to identify patients at risk. However, research showed that its performance in real-world hospital settings was significantly poorer than expected, with high rates of false positives [18]. The model had not been thoroughly externally validated in diverse hospital settings and clinical workflows.

In all of these cases, validation accomplished across hospitals with different patient populations and clinical workflows, would have mitigated the risk of overfitting on specific hospital data and would have allowed hospitals to adapt the model for their specific treatment, diagnostic and screening protocols, ensuring that the predictions were more aligned with local clinical realities. Similarly, collecting data from severe COVID-19 patients admitted in a central hospital with entire wards specifically dedicated to COVID-19 may be different from collecting data from patients admitted to a high-volume specialistic hospital to undergo orthopedic surgery, who are diagnosed COVID-19 as a simultaneous condition, although these hospitals are geographically equivalent. Major reference centres for the treatment of common musculoskeletal conditions (e.g. joint arthroplasty in elderly patients affected by osteoarthritis or femoral head fractures), therefore, may appear highly representative of the local population in vulnerable patients and areas to COVID-19, efficiently training predictive models in short time; however, they may result clinically misleading when used on patients with different age and/or medical characteristics just 20 km far [19]. A concrete and straightforward way to ascertain external validation, therefore, is to test the system (a) on data coming from different healthcare settings (e.g., laboratories, radiological departments, hospitals), possibly from different regions and countries (cross-sectional validation); (b) on data purposely collected for the system validation at different times (ideally prospective, and ideally years apart) (longitudinal validation). If the results are replicated, then the system can be deemed externally valid and robust.

Moreover, all models– no matter how weak or robust they are– can be invalidated by concept drift (e.g., phenotype evolution) or label shift (e.g., changing the name we give to medical phenotypes, e.g. calling disease what is previously or elsewhere called symptom). To ensure that reliable clinical information is provided from different contexts, on different patients and over time, monitoring actions like techno-vigilance are needed, just like they are in case of drugs and medical devices, in order to identify, evaluate, understand and prevent the underperformance and unwanted effects of each predictive model; even more so when a certain model tested in a certain part of the world must be available to other patients in other contexts affected by the same public health emergency, as it may happen in a next pandemic.

To be more specific on the concept of techno-vigilance, we draw a parallel with well-established practices and procedures of pharmacovigilance and adapt them to the specific challenges posed by AI. Indeed, while pharmacovigilance is designed to monitor, assess and prevent adverse effects or other problems related to pharmaceuticals, similarly, techno-vigilance is proposed to monitor the safety, performance and ethical use of AI systems in healthcare. A comprehensive techno-vigilance framework for AI in healthcare begins with pre-deployment risk assessment, where clinical and technical risks are identified, and risk profiles are developed to define the AI model’s reliability. Post-deployment, continuous monitoring is also essential, with mechanisms in place to track performance in real-world settings, log failures, and detect performance drifts. Oversight bodies, or techno-vigilance committees comprising multidisciplinary experts, should regularly evaluate the AI system’s safety, performance and compliance with regulations. Continuous revalidation and mandatory update cycles ensure that AI models remain aligned with evolving clinical guidelines and population data, with each update undergoing rigorous external validation.

Training healthcare providers on AI model limitations, and implementing feedback mechanisms, helps identify potential issues early. Regulatory compliance is maintained through the alignment of techno-vigilance processes with standards, ensuring accountability via regulatory audits. Additionally, ethical monitoring systems are necessary to safeguard against biases, incorporating ethical oversight to ensure AI models are used responsibly and equitably. The above outlined framework parallels pharmacovigilance in drug safety, by also emphasizing continuous monitoring, regulatory alignment and ethical integrity so as to ensure that AI systems are safely and effectively integrated into healthcare. By adhering to these techno-vigilance practices, we believe healthcare systems can monitor AI models in real time, ensure their safety and efficacy across diverse populations, achieve and maintain necessarily-high ethical standards.

Surprising few articles performed external validation analyses in the last two years, although the number is increasing, considering that 192 out of 194 ML articles reporting external validation were published in the last 5 years. Assuming the Covid-19 pandemic as the last global health crisis, launching on PubMed the query ((“machine learning“[Title/Abstract]) OR (“deep learning“[Title/Abstract])) AND ((“external validation“[Title/Abstract]) OR (“externally validated“[Title/Abstract])) AND ((COVID[Title/Abstract]) OR (COVID-19[Title/Abstract]) OR (SARS-COV-2[Title/Abstract])) gave 124 results, while removing the external validation terms the result is 6,689. Most ML models reported in the literature about risk stratification, radiological discrimination and cardiological tasks [20,21,22] perform poorly on external data, or significantly worse than on internal data, which means they could be practically useless if not even harmful to real-world patients and practitioners [23].

Conclusions

Medical validation and machine-learning validation differ significantly in both scope and execution. While machine learning models often undergo validation based on historical datasets and performance metrics like accuracy, external medical validation is essential to assess how these models perform in real-world, diverse clinical environments. Providing reliable and timely information across different healthcare settings is critical to address the reproducibility crisis of AI applications in healthcare.

For policymakers, we advocate the establishment of clear regulatory frameworks that mandate external validation as a fundamental component of AI certification. Regulatory bodies should require AI developers to conduct multi-site external validation studies before any system can be widely deployed. Additionally, policies should encourage the creation of publicly available datasets that represent diverse populations, diseases, and healthcare environments. Such datasets would help ensure that AI models do not reinforce health disparities or perform inadequately in underrepresented communities. A systematic regulatory audit should be implemented to monitor the continuous performance of AI models over time, ensuring that models remain compliant with safety and efficacy standards as healthcare environments evolve.

For healthcare providers, the adoption of AI models must be coupled with an understanding of their limitations. Providers should actively participate in external validation processes by contributing real-world clinical data and providing feedback on model performance. Furthermore, hospitals and healthcare organizations must develop internal protocols for the ongoing monitoring of AI tools, identifying any performance deviations that could lead to adverse outcomes. Healthcare providers play a pivotal role in identifying early signals of AI model failure and should have the authority to report issues directly to regulatory bodies and AI developers, much like adverse event reporting systems in pharmacovigilance.

For AI researchers, external validation should be integrated into the model development lifecycle from the outset. Researchers must prioritize the diversity of training and validation datasets, ensuring that their models generalize across different patient populations, clinical settings, and geographies. Collaborative efforts with healthcare institutions should be encouraged to facilitate the collection of real-world data for validation purposes. AI researchers should also develop adaptive learning mechanisms within AI models, allowing them to adjust based on new validation data from different settings. Additionally, establishing a standard for transparent reporting of validation outcomes, including instances where models underperform in external settings, is essential for advancing AI trustworthiness in healthcare.

In contrast to previous global health crises, we now have the unprecedented opportunity to leverage AI models as public health measures that can stratify patient populations, activate appropriate care pathways, and organize healthcare networks in response to emergencies. The consequences of inadequate or non-existent external validation are profound, as poorly implemented models can harm patient outcomes and erode trust in AI technologies. Therefore, it is of paramount importance that certification-oriented validation studies take external validation seriously and integrate it as a critical step to ensure the safe, equitable, and effective deployment of AI systems in healthcare.