Leveraging large language models to mimic domain expert labeling in unstructured text-based electronic healthcare records in non-english languages

Abstract

Background

The integration of big data and artificial intelligence (AI) in healthcare, particularly through the analysis of electronic health records (EHR), presents significant opportunities for improving diagnostic accuracy and patient outcomes. However, the challenge of processing and accurately labeling vast amounts of unstructured data remains a critical bottleneck, necessitating efficient and reliable solutions. This study investigates the ability of domain specific, fine-tuned large language models (LLMs) to classify unstructured EHR texts with typographical errors through named entity recognition tasks, aiming to improve the efficiency and reliability of supervised learning AI models in healthcare.

Methods

Turkish clinical notes from pediatric emergency room admissions at Hacettepe University İhsan Doğramacı Children’s Hospital from 2018 to 2023 were analyzed. The data were preprocessed with open source Python libraries and categorized using a pretrained GPT-3 model, “text-davinci-003,” before and after fine-tuning with domain-specific data on respiratory tract infections (RTI). The model’s predictions were compared against ground truth labels established by pediatric specialists.

Results

Out of 24,229 patient records classified as poorly labeled, 18,879 were identified without typographical errors and confirmed for RTI through filtering methods. The fine-tuned model achieved a 99.88% accuracy, significantly outperforming the pretrained model’s 78.54% accuracy in identifying RTI cases among the remaining records. The fine-tuned model demonstrated superior performance metrics across all evaluated aspects compared to the pretrained model.

Conclusions

Fine-tuned LLMs can categorize unstructured EHR data with high accuracy, closely approximating the performance of domain experts. This approach significantly reduces the time and costs associated with manual data labeling, demonstrating the potential to streamline the processing of large-scale healthcare data for AI applications.

Introduction

The healthcare industry is undergoing a transformative era, fueled by the rapid advancement of technology and the ever-increasing volume of data. The advent of Big Data (BD), characterized by its vastness, velocity, and variety, has unlocked unprecedented opportunities for healthcare providers and researchers [1,2,3]. In recent years, artificial intelligence (AI) applications have demonstrated significant improvements in safety, quality, and diagnostic accuracy across various clinical settings. Leveraging BD from Electronic Health Records (EHR), these AI-based techniques offer the potential to revolutionize medicine by enhancing outcomes and providing numerous benefits [2, 4,5,6,7,8,9,10,11]. However, these large-scale data are often unstructured, requiring extensive processing and labeling, which poses the most significant bottleneck [12]. In a precise field like medicine, errors in the labeling and preprocessing process can lead to poor outcomes in terms of the reliability of AI models and the impact of model results [11, 13,14,15,16,17]. Therefore, domain experts are often employed for labeling tasks in the present day, a process that is both time-consuming and costly [6, 18].

Moreover, when considering non-English datasets such as those in Turkish, the challenges intensify. The Turkish language presents distinct challenges for automated text analysis due to its agglutinative nature, morphological richness, and extensive inflection [19]. These linguistic characteristics can significantly complicate the extraction and classification of information from medical records, where precision is crucial [20, 21]. Addressing these challenges, our study leverages a fine-tuned large language model to effectively interpret and categorize unstructured EHR in Turkish. This adaptation not only enhances the accuracy of data processing but also contributes valuable insights into the application of AI in underrepresented linguistic contexts within the healthcare domain.

Challenges in generating datasets and data extractions from clinical notes for artificial intelligence models from BD sources containing unstructured EHR texts are primarily attributed to the complex process required for structuring and standardizing these texts for effective supervised learning AI models utilization. Unstructured EHRs are characterized by a wide array of data formats, including free-text clinical notes, laboratory findings, and imaging narratives. Each of these formats exhibits unique terminological and syntactical features, ambiguous jargon, and non-standard phrasal structures [17, 22,23,24,25,26]. To mitigate such complexity, the encoding of patients’ diseases in EHRs using universally accepted disease classification coding systems like the International Classification of Disease (ICD) facilitates the clustering of patients, providing convenience. However, these codes can sometimes be misencoded due to intentional or unintentional information transfer by the patient or clinician [27], and these inaccuracies can significantly impact the performance of supervised learning AI models, including machine learning (ML), deep learning (DL), time series analysis, and Natural Language Processing (NLP) [26, 28, 29].

Recently, NLP methods for EHR-based computational phenotyping have seen extensive development, in information technology, the knowledge graph can transform complex unstructured data into structured form [30, 31]. Serving as a pivotal task in the construction of knowledge graphs, Named Entity Recognition (NER) enables the automatic extraction of predefined entities from extensive volumes of intricate texts, thereby facilitating the structuring of information. Through NER methods, the extraction of information from large-scale unstructured text-based datasets is substantially simplified [32,33,34,35,36,37]. However, human-induced typo errors, such as homophone, typographical, grammatical, and spacing errors, can still be present in manually entered data, with reported error rates ranging from 5 to 17% [38, 39]. These errors significantly impact the performance of NER methods [26, 40, 41]. In 2017, Google’s introduction of transformer architecture marked a significant breakthrough in artificial intelligence, paving the way for the creation of advanced large language models (LLM). Trained on vast amounts of internet data using self-supervised learning techniques, these LLMs showcased an unprecedented ability to comprehend and produce text closely resembling human writing [6, 42]. Furthermore, in NER tasks, transformer-based language models have demonstrated the highest performance [26]. Unlike other LLMs, OpenAI’s GPT model is used more frequently than others due to its availability through the ChatGPT interface and an API [43]. It has been demonstrated that ChatGPT can accurately predict diagnoses for patients based on clinical notes, achieving results comparable to those of human practitioners in the domain of clinical information extraction from such notes [6, 44,45,46,47].

In this research, the ability to precisely classify target labels containing typographical errors through NER tasks was explored, aiming to alleviate the detrimental effects of missing data on the efficacy of supervised learning AI models. This investigation was conducted utilizing domain specific, fine-tuned LLMs, highlighting their potential to enhance model accuracy and reliability.

Method

Data structure

In the present study, the primary data source comprised clinical notes from Pediatric Emergency Department (PED) admissions, which were extracted from the EHR system of Hacettepe University İhsan Doğramacı Children’s Hospital. The structure of the data is centered around the initial assessments conducted by pediatric residents at the PER triage point. These assessments include a variety of patient information, such as presenting complaints, evaluations based on the pediatric assessment triangle [45], body temperature, heart rate, respiratory rate, and SpO2% levels. This information forms the basis of the triage process, wherein patients are categorized for further examination. Notably, during this initial triage phase, patients are not assigned specific diagnostic codes due to the preliminary nature of the assessments. Instead, patient complaints are categorized into specific, institution-prepared complaint categories such as abdominal pain, headache, and fever, in a structured format. When the patient’s presenting complaint does not align with these predefined structured categories, the physician recording the triage selects the “Others ()” category, and this input is taken as unstructured Turkish text. Consequently, this results in poorly labeled data, which poses challenges for researchers in subsequent retrospective studies.

Data collection

Data collection for this study was conducted by encompassing all patient visits to the PED at Hacettepe University İhsan Doğramacı Children’s Hospital during the period from 2018 to 2023. Records were obtained from the hospital’s Electronic Health Record (EHR) system, through which a systematic approach was employed to compile relevant clinical notes and assessment data.

Data preprocessing

For preprocessing tasks, open-source Python libraries such as Pandas, NLTK, and Re (regex) were utilized. Initially, poorly labeled unstructured texts categorized as “Others (Complaints)” from structured categorical diagnostic descriptions were selected, ensuring that the anonymized dataset contained only the complaints without any personal patient information. These categories were then normalized by removing the “Others ()” part to leave only ‘Complaints,’ and subsequently, all characters were converted to lowercase as ‘complaints’ using a regular expression task designed with the NLTK and Re libraries. After this normalization, the filtered data were iteratively classified syntactically using common NLP methods to determine if they contained various combinations of well-known respiratory tract infections (RTI) findings, such as fever, cough, and shortness of breath. Using a simple NER task, findings were extracted from the lowercase poorly labeled texts and subsequently categorized with rule-based methods using a series of dictionaries. Subsequently, words that could not be processed in the iteration were identified as either typographical errors or combinations of extremely rare findings. A dataset containing these poorly labeled data, which included typographical errors and those that could not be classified with simple NLP methods, was prepared for further queries using GPT.

Prompt engineering and fine-tuning of a GPT-3 model and prediction

In this study, we utilized the “text-davinci-003” model, a GPT-3 language model accessible through OpenAI’s API, established as of May 2023. This low-code approach allowed us to provide preprocessed, poorly labeled data directly to the model without extensive coding requirements. Using a predefined prompt, “Based on the symptoms and findings presented, does this align with the characteristics of an RTI? If the evidence strongly suggests an RTI, please respond with ‘True’. If the findings do not support an RTI diagnosis, respond with ‘False’,” we iteratively collected model responses. These were recorded in a Boolean list to capture the model’s diagnostic alignment with the RTI characteristics. Following this initial application, the “text-davinci-003” model was fine-tuned using a specific corpus describing RTI symptoms in Turkish, enhancing its diagnostic accuracy. The fine-tuned model was then reapplied to the dataset with the same prompt to evaluate improvements in prediction accuracy.

Ground truth establishment

For the ground truth labels, four pediatric specialists were asked to determine whether the presenting complaint data, which were distributed equally and randomly among them, indicated findings of an RTI.

Model evaluation and data analysis

In the evaluation of the model outcomes, assessments were conducted using classification metrics from the Scikit Learn library, including accuracy, ROC-AUC, precision, recall, F1 score, and MCC metrics. For this project, Python version 3.9 and OpenAI’s Python library version 0.26.5 were used.

Results

Between 2018 and 2023, 321,672 patients presented to the Pediatric Emergency Room (PER). In this study, 31.9% (n = 102,732) of the patients were determined to have RTI complaints through standard filtering methods. Subsequently, 7.53% (n = 24,229) of the patients were recorded in the EHR system as “Others ()”, with 77.91% (n = 18,879) of these patients accurately identified with RTI findings through filtering methods, showing no typographical errors. Moreover, standard filtering methods revelaed that 20.2% (n = 3,828) of these patients had RTI. The presenting complaint targets of the remaining 22% (n = 5,350) were assessed as poorly labeled. These 5,350 patients received ground truth labels from four pediatric specialists within two business days. From these labels, 16.9% (n = 909) were identified as RTI cases. In Table 1, the most frequent occurrences of presenting complaints containing RTI findings across the data clusters are displayed. Following the correction of errors within the unstructured poorly labeled data and typographical error-containing data cluster, the most frequently presenting complaints were, in order, control revisits, falling, diarrhea, patients sent for hospitalization from the outpatient clinic, patients with suspected COVID among upper respiratory tract infections, epistaxis, constipation, patients receiving injections, nasal discharge, and cough.

Table 1 Distribution of presenting complaints by data clusters

The labeling process, which was conducted by four pediatric specialists, each of whom dedicated two business days, was completed within a total of eight business days, resulting in a labeling rate of 27 labels per hour. The pretrained LLM completed the same task using a zero-shot approach in approximately six hours, with a labeling rate of 891 labels per hour. The fine-tuning process of the pretrained model, utilizing a document containing 4,724 tokens pertaining to RTI findings in Turkish, lasted approximately three hours. Similarly, employing a zero-shot approach, the fine-tuning process completed the entire labeling task in approximately six hours, akin to the performance of the pretrained model. The performances of both models were evaluated against the established ground truth labels. The pretrained model identified 714 (78.54%) patients with RTIs, and the fine-tuned model identified 908 (99.88%) patients with RTIs. The data processing stages are also demonstrated in Fig. 1, and the detailed performance metrics are available in Table 2.

Fig. 1

Data Processing and Model Performance for RTI Identification: This figure shows the filtering of URTI symptoms from the dataset after processing all cases. It focuses on the analysis of 5,350 poorly labeled cases, comparing the ROC-AUC performance of the pretrained and fine-tuned GPT-3 models. The fine-tuned model demonstrates significant improvement in identifying RTI cases

Table 2 Comparison of performance between pretrained models and fine-tuned models

Discussion

Due to typographical errors, the categorization of unstructured text-based EHR clinical notes that cannot be classified through standard filtering and NER tasks is a costly and time-consuming process when dealing with large-scale data. As the data scale increases, it becomes imperative to automate the processes of data manipulation that require domain knowledge for more efficient supervised learning AI models. In the context of the pediatric emergency room visits where this study was conducted, nearly one-third of the patients had RTI, representing the largest patient cohort. Therefore, it is valuable to demonstrate that RTIs as presenting complaints can be recognized by LLMs. In this study, a solution to this bottleneck is presented, demonstrating that LLMs fine-tuned on a specific subject can be categorized with an accuracy approaching that of domain experts, in contrast to the general-use LLM models.

Our findings are particularly significant given the complexity of the Turkish language, which has been underrepresented in NLP research, especially in medical applications. The successful application of LLMs to Turkish EHR texts not only demonstrates the model’s robustness but also its adaptability to diverse linguistic contexts. This capability is crucial for extending AI applications to non-English datasets, which are often less studied but equally in need of advanced analytical tools.

Evaluating the accuracy of LLM in medical data classification

In this study, ground truth labels were determined by pediatric specialists, and the primary focus was not a direct comparison between humans and LLMs, but rather an investigation into how closely LLMs could approximate domain expert human encoders. Accordingly, the performance of a general-use GPT-3 submodel, “text-davinci-003,” resulted in 78% accuracy, while its version fine-tuned specifically for RTI findings demonstrated a significantly higher accuracy of 99.88%, surpassing that of the general model and closely matching the performance of domain experts. This efficiency and the low-code integration of the API not only expedited the research process but also minimized the potential for errors typically associated with manual coding, thus enhancing the reliability and speed of medical data analysis. The ease of API utilization democratizes the use of sophisticated AI in clinical research, expanding the potential for broader adoption across various medical disciplines.

This finding aligns with existing literature, where comparisons between humans and LLMs, including a meta-analysis by Takita et al., revealed that the pooled accuracy of all models was 57%. Specifically, for the GPT-3 model utilized in this study, the average accuracy was reported to be 60% (range 51–69%). Additionally, model performance across specialties showed the highest efficacy in pediatric studies (93%) [48]. The above-average performance of the general-use model in our study could be attributed to this relatively high efficacy of LLMs in pediatric contexts.

In a related vein, Rosoł et al.‘s study comparing humans and LLMs in medical exam questions found that the GPT-4 model, even without fine-tuning, outperformed the GPT-3 model [49]. Furthermore, the MedPaLM2 model, which is fine-tuned for the medical domain, demonstrated a high accuracy of 86.5% in Singhal et al.‘s study, matching the performance of the GPT-4 model used in the study by Nori et al., which also showed an accuracy of 86.1% in USMLE exam questions [50, 51]. These findings highlight the substantial performance improvements brought about by fine-tuning, as reaffirmed in our study and supported by meta-analyses by Takita et al., where the pooled accuracy of the PaLM2 model was 43%. This underscores the significant enhancement effect of fine-tuning on model performance [48, 50, 52,53,54,55,56].

Moreover, another promising method for obtaining domain-specific responses through LLMs is the retrieval augmented generation (RAG) method [57], which enables a pretrained LLM to generate task-specific answers by sourcing information from specific external resources. This approach may offer an alternative solution for NER tasks [58,59,60,61]. Naik and colleagues, for instance, developed a language model that performs binary classification of clinical outcomes from EHR clinical notes using RAG methods, which have been shown to enhance answer generation performance [62, 63]. Balaguer et al.‘s study comparing LLMs utilized with RAG and fine-tuning found that while the fine-tuned model produced correct answers 47% of the time, the use of RAG alone increased this to 72%, and to 74% when both were used in conjunction. The utilization of RAG methods in unstructured text-based EHR data holds significant potential for NER tasks, as demonstrated in various studies, and could provide a cost-effective alternative to solely fine-tuning models [64].

Time efficiency and cost comparison

Compared with humans, LLMs are capable of labeling both more rapidly and in a continuous, uninterrupted manner. Wang et al. demonstrated that labeling with GPT-3 is not only faster but also less expensive. Their comparison involved the GPT-3 model and human labellers on the Google Cloud Platform, where billing is based on the number of tokens. According to their findings, utilizing GPT resulted in a cost reduction of 50–96%, translating into an approximate cost of $453 for this study [65]. The work of the human encoders in this research was voluntary, with no compensation requested, and the study itself was not focused on cost analysis. However, the comparison is considered striking. Approximately $13 was spent on the labeling process in this study, including the use of a fine-tuning model that can be subsequently utilized with GPT-3. Consequently, achieving an accuracy of 98%, this method, which operates 33 times faster and can be 34 times less expensive, allows expert clinicians to allocate their time more effectively to other tasks.

Agarwal et al. work highlights the potential of using weak supervision to deploy smaller, task-specific models, thereby emphasizing the importance of models that are cost-effective and capable of generating more issue-specific responses [17]. Recently, various open-source, fine-tunable tiny language models have become available. Tasks such as those in our study can be trained on these models, significantly reducing costs through local usage while also addressing ethical concerns by enhancing local security.

Ethical consideration and data security

There are major concerns about the impact of LLMs on patient data. The large datasets used in the training of LLMs may contain sensitive patient information, which is thought to increase the risk of data breaches or unauthorized access [66]. In this study, a language model accessed via API provided by OpenAI was used, and OpenAI’s data usage policy guarantees that data used through APIs cannot be accessed or used by anyone, including model developers. This security is ensured through special protocols such as SAML SSO, SOC2, AES-256, and TLS. Some countries have their own data policies, and for this study, Turkey’s personal data protection law was considered. No personal information of the patient was present in any text sent to the model via API, thus in these processes, data security is more dependent on developer compliance. Additionally, as mentioned by Agrawal et al. and Jimenez et al., security can be enhanced by using smaller, task-specific language models that can run on local systems, avoiding the use of APIs [67].

Conclusion

In conclusion, this study demonstrates that fine-tuned LLMs can effectively categorize unstructured EHR data with high accuracy, mirroring the performance of domain experts. By utilizing a fine-tuned GPT-3 model, the classification of pediatric emergency room data on respiratory tract infections achieved a remarkable accuracy of 99.88%. Notably, this performance was achieved even with data in a non-English language, highlighting the model’s versatility and effectiveness. This approach significantly enhances the efficiency and cost-effectiveness of data labeling, reducing reliance on manual processes. Moreover, the successful adaptation to diverse linguistic contexts suggests a scalable model for global health systems, potentially addressing language barriers in medical data analytics. The findings underscore the potential of LLMs to streamline large-scale healthcare data processing, paving the way for more efficient and reliable AI applications in clinical settings.