Digital monitoring of medication safety in children: an investigation of ADR signalling techniques in Malaysia

Abstract

Background

Digital solutions can help monitor medication safety in children who are often excluded in clinical trials. The lack of reliable safety data often leads to either under- or over-dose of medications during clinical management which make them either not responding well to treatment or susceptible to adverse drug reactions (ADRs).

Aim

This study investigated ADR signalling techniques to detect serious ADRs in Malaysian children aged from birth to 12 years old using an electronic ADRs’ database.

Methods

Four techniques (Proportional Reporting Ratio (PRR), Reporting Odds Ratio (ROR), Bayesian Confidence Propagation Neural Network (BCPNN) and Multi-item Gamma Poisson Shrinker (MGPS)) were tested on ADR reports submitted to the National Pharmaceutical Regulatory Agency between 2016 and 2020. Sensitivity, Specificity, Positive Predictive Value (PPV) and Negative Predictive Value (NPV) of the techniques were compared.

Results

A total of 31 medicine-Important Medical Event pairs were found and examined among the 3152 paediatric ADR reports. Three techniques (PRR, ROR, MGPS) signalled oculogyric crisis and dystonia for metoclopramide. BCPNN and MGPS signalled angioedema for paracetamol, amoxicillin and ibuprofen. Similar performances were found for PRR, ROR and BCPNN (sensitivity of 12%, specificity of 100%, PPV of 100% and NPV of 21%). MGPS revealed the highest sensitivity (20%) and NPV (23%), as well as similar specificity and PPV (100%).

Conclusions

This study suggests that medication safety signalling techniques could be applied on electronic health records to monitor medication safety issues in children. Clinicians and medication safety specialist could prioritise the signals for further clinical consideration and prompt response.

Introduction

Children are a vulnerable and high-risk population to experience adverse drug reactions (ADRs) due to developmental and physiological differences from the adult population [1]. The lack of clinical studies for medicines’ efficacy and safety data in children often leads to off-label prescribing and as a consequence, ADRs are unpredictable and highly prevalent [2]. Only less than 15% of all marketed drugs for paediatric use have good evidence on benefit-risk balance from clinical trials [3]. The limited knowledge from clinical trials on medicines for paediatric population have made prescribing to paediatrics a challenge compared to adults and requires constant need of age, body weight and body surface area for medication dose calculation and prescribing to avoid medication error and ADRs [4]. Even so, medication errors and ADRs are still prevalent in paediatric populations [5]. Proactive complementary solutions are urgently needed to minimise harm in paediatric population.

Equipping electronic health records with digital solutions such as clinical decision support system (CDSS) and digital analytics could actively monitor medication errors and safety signals [6,7,8]. In recent years, studies have shown the use of machine learning or digital solution for early detection of medication errors and ADRs among neonates at critical care settings [9,10,11]. Risk scores, machine learning algorithm and genotyping are some of the efforts undertaken by integrating it directly during the clinical practice. The seamless digital integration during clinical practice would ensure that regular clinical practice is not disrupted as illustrated in these studies. AI-powered tools for medication error and ADRs detection could help lessen the burden of healthcare professionals, enhance patients’ outcomes and allowing focus and resources for more complex cases. Hence, a simple yet practical medication safety signalling tool is needed to help with clinical practice.

A medication safety signal is a hypothesis alert of new potential causal association or new aspect of the known association between a suspect medicine and adverse event [12]. The standard methods to detect medication signals include manual reviewing of submitted ADR case reports and literatures. However, under-reporting and selective reporting are common challenges for voluntary and manual ADR case reporting that may lead to delay reviewing and timely action [13, 14]. Multiple interventions including computerised ADR case reports registration and active surveillance can significantly improve ADR reporting [14]. In addition, computerised algorithms have been implemented to monitoring large amounts of ADR case reports in spontaneous reporting databases including signals of disproportionate reporting (SDR), Bayesian Confidence Propagation Neural Network (BCPNN) and Multi-item Gamma Poisson Shrinker (MGPS) [8, 15].

A study undertaken on the World Health Organisation (WHO) global electronic database, Vigibase, for previously unrecognised safety issues of medications in paediatric case reports found 27 potential safety signals [16]. One of the signals concerned harm due to off-label use of dextromethorphan and the remaining signals referred to potentially new causal associations in paediatric population. This study results showed the potential of using digital safety signal detection in electronic health records to continuously monitor medicines in this vulnerable population. Similarly, other health authorities such as US FDA Adverse Event Reporting System (FAERS) have tested digital safety signalling techniques in paediatric population using their electronic health records and showed its potential use including identification of serious ADRs [17, 18]. Although, the causal relationship cannot be directly translated from the identified signals, the safety signalling techniques may complement the current surveillance system of marketed drugs and alert health authorities to further investigate the signals using appropriate clinical deliberation. Once the evidence of causality assessment between a drug and an ADR is established, the regulatory agency may issue a timely recall, change the product label, or withdraw the medication from the market.

Strategic and active actions are urgently needed for surveillance of new and known ADRs to prevent future harm especially in paediatric population where clinical trial information about medicines in this population is limited compared to the adults. Investing in digital solutions could significantly improve productivity and efficiency for the benefits of both patients’ outcomes and cost. At the moment, the Malaysia digital healthcare landscape policy and regulation are still in its infant state where initiatives to improve data cybersecurity, accelerate digital adoption, data sharing, and cultivation of digital skillsets are the main priorities in the Malaysia Digital Economy Blueprint by 2025 [19]. Previous studies about digital signalling techniques in paediatric populations are still refining their techniques to suit their national databases [17, 20]. There are no local studies in Malaysia investigating potential use of electronic national registry of paediatrics’ ADR data with signalling techniques. Therefore, the aim of this study was to determine the potential of medication safety signalling techniques applied on the Malaysian paediatric ADR reports to detect serious ADRs.

Methods

Data source and extraction

This study was approved by the National Pharmaceutical Regulatory Agency and the Medical Research Ethics Committee, Ministry of Health Malaysia (NMRR-20-2089-56496). This study used completely anonymous data. This study used the National Pharmaceutical Regulatory Agency (NPRA) database that contains Malaysian ADR reports and adverse events following immunisation (AEFI) submitted by healthcare professionals, manufacturers and public.

This database is a government-owned secondary population database that does not store any personal or contact details, thereby rendering patient consent implausible. Given this, the research does not involve human subjects, and as such, informed consent is not relevant. Furthermore, the database used in the study is not publicly available. Malaysia has personal data protection act (PDPA) to manage digital health records similar to general data protection regulations (GDPR) in Europe. These regulations ensure data security and patient privacy in national pharmacovigilance centre.

The submitted ADR reports are subjected to be processed and checked by regulatory officers for validity where only viable reports were entered in the Malaysian pharmacovigilance database. ADR coding was performed using Medical Dictionary for Regulatory Activities (MedDRA) and causality assessment was done using probability scales such as Naranjo algorithms. Validation of data in this database is undertaken periodically at NPRA by officer-in-charge to check and verify for data consistency relating to its structure, format and input variables.

Only ADR reports for neonates, infants and children aged from birth to 12 years old submitted to the National Pharmaceutical Regulatory Agency (NPRA) between 2016 and 2020 were extracted and analysed together. ADR reports related to vaccines were excluded. ADR reports related to premature infants and in-utero exposure are excluded due to unavailability of data. ADR reports with causal relationship between medicine and reaction categorised as ‘unlikely’ are excluded as well.

Important medical event (IME) screening

An initial assessment was performed on the reported ADR cases to assess the seriousness of reports based on Important Medical Events (IMEs) classification developed by the European Medicines Agency (EMA) (IME list MedDRA version 24, 2021) [21]. The IMEs are adverse events that are serious and could results in life-threatening, prolonged hospitalisation, or death. The ADR reports were reviewed and confirmed by two researchers (BHR and IAW). BHR is a pharmacist and pharmacovigilance specialist with experience in processing ADR reports. IAW is an academic researcher, a pharmacist and a professor in clinical pharmacy and clinical practice. These reports were the final ADR reports included for the ADR signalling techniques.

Adverse drug reaction (ADR) signalling techniques

This study used four ADR signalling techniques; (1) Proportional Reporting Ratio (PRR), (2) Reporting Odds Ratio (ROR), (3) Bayesian Confidence Propagation Neural Network (BCPNN) and (4) Multi-item Gamma Poisson Shrinker (MGPS). These four techniques are considered as disproportionality analyses that relies on the principle that when a safety signal (involving adverse event, R) is identified for a medicinal product (referred as P), this adverse event is reported relatively more frequently in association with this medicinal product P than with other medicinal products [22]. This relative increase in the ADR reporting for the medicinal product P is reflected in a 2 × 2 contingency Table 1 based on the total number of individual cases contained in a pharmacovigilance database.

Table 1 2 × 2 contingency table

The general criteria to run the disproportionality analysis are as follows:

The value A indicates the number of individual cases with the suspect medicinal product P involving an adverse event R.

The value B indicates the number of individual cases related to the suspect medicinal product P, involving any other adverse events but R.

The value C indicates the number of individual cases involving event R in relation to any other medicinal products but P.

The value D indicates the number of individual cases involving any other adverse events but R and any other medicinal products but P.

A, B, C and D are manually calculated for each drug-event pair. Disproportionality measures PRR, ROR, BCPNN and MGPS are applied based on A, B, C and D values. For PRR, a safety signal is generated if PRR value > 2 and Yate’s chi-square value, X² > 4 [23]. For ROR, a safety signal is generated if the lower limit of 95% confidence interval of ROR > 1 [24]. For BCPNN, the information component (IC) is calculated, standard deviation (SD) of IC is computed, a safety signal is generated when IC-2SD > 0 [25]. For MGPS, the EBGM was calculated, a safety signal is generated when the lower limit of 95% CI of EBGM, EB05 > 2 [26]. To avoid statistical instability and noise associated with disproportionality measures, more than one report is required to generate a safety signal. A minimum count of drug-ADR pairs, N ≥ 3 was imposed on the analysis of PRR, ROR, BCPNN and MGPS [26].

Performance methods.

The generated safety signals from each method are further analysed for performance related test-characteristics namely sensitivity, specificity, PPV and NPV based on the listed events with the latest product information at Drugs @ FDA. The test-characteristics are compared among safety signal detection methods to ensure the best method in practice [27]. Sensitivity, specificity, PPV and NPV are collectively known as “performance-test characteristics”. These terms are derived as shown in Table 2.

Table 2 Calculating performance test-characteristics

A signal is considered true positive when the medicine-event pair is labelled in the product information and were found statistically significant signal. False positive signal is when medicine-event pair is statistically significant signal but not in the product information. False negative is when medicine-event pair is not statistically significant signal but is categorised in the product information whereas true negative is when medicine-event pair is not statistically significant signal and not listed in the product information as an ADR. Sensitivity is the proportion of true positive signals depending on drug-event pairs which were labelled. It is the true positive rate equivalent to (a / a + c). Specificity is the proportion of true negative signals depending on drug-event pairs which were not labelled. It is the true negative rate equivalent to (d / b + d). A high specificity indicates there are a smaller number of false positive signals. Positive predictive value (PPV) is the proportion of drug-event pairs that were signalled as statistically significant safety signals as well as considered labelled. PPV is equivalent to (a / a + b). Negative predictive value (NPV) is the proportion of drug-event pairs that were not signalled as statistically significant safety signals as well as not considered labelled. NPV is equivalent to (d / c + d).

Results

As shown in the flowchart (Fig. 1), within 2016 to 2020, there are 6769 submitted paediatric ADR reports. Out of these reports, 3591 reports related to vaccines were excluded. ADR reports on ‘in utero exposure’ (n = 23) were excluded leaving 3155 ADR reports out of which three cases with causality assessment as ‘unlikely’ were also excluded with the final eligible ADR case report counts for this study is 3152 ADR case reports for IME screening. A list of 31 drug - IME pairs from the 3152 ADR reports were identified (Table 3). Angioedema has the highest cumulative IME reports (n = 19) for painkillers and antimicrobials. Oculogyric crisis alone has six reports for metoclopramide.

Fig. 1

Number of overall paediatrics ADR reports

Out of 31 drug-event pairs, 25 adverse events were listed by the US FDA label for the medicines whereas six adverse events were not listed by the US FDA and other drug labels from EMA and summary of product characteristics (SmPC). Based on the listed events and whether they were considered signal or not, accuracy, precision, recall and F1 score were calculated as shown in Table 4. This study model has shown an accuracy of 32%, 100% precision and recall/ true positive rate/ overall sensitivity as 16%. F1 score is a balance between precision and recall and in this analysis model, it was found to be 27%.

Table 3 Confusion matrix based on listed events

Table 4 Reports of medicines and important medical event (IME)

As shown in Table 4, PRR and ROR identified 3 similar statistically significant medication safety signals for (1) Metoclopramide – Oculogyric crisis, (2) Metoclopramide – Dystonia and (3) Paracetamol – Angioedema. BCPNN identified only one out of three safety signals identified by PRR and ROR which was Paracetamol – Angioedema. However, BCPNN picked up two other safety signals Amoxicillin – Angioedema and Ibuprofen - Angioedema which were not identified by using PRR and ROR methods. Nevertheless, MGPS detected all of five safety signals. The route of administration of these drugs was reported as metoclopramide (oral and intravenous), paracetamol (oral, rectal and intravenous), amoxicillin (oral) and ibuprofen (oral) as mentioned in ADR reports.

Table 5 Statistically significant safety signals based on four disproportionality methods

PRR and ROR reported similar characteristics in terms of performance (sensitivity 11%, specificity 92%, PPV 66% and NPV 42%) (Table 5). The performance of BCPNN in terms of sensitivity, specificity, PPV and NPV was lower compared to PRR, ROR and MGPS. MGPS reported highest sensitivity (16%) compared to other three methods whereas acceptable specificity and PPV compared to PRR and ROR methods as tabulated in Table 6.

Table 6 Performance test-characteristics of four disproportionality methods

Discussion

This study shows that there are known IMEs signalled by the four signalling techniques applied on the electronic Malaysian paediatrics ADR reports (oculogyric crisis, dystonia and angioedema). These ADRs are serious and could result in life-threatening if no immediate action be undertaken. Acetaminophen (paracetamol) and ibuprofen are the two most common over-the-counter medicines prescribed for children to manage fever and pain [28]. In addition, ibuprofen is also found to be an emergent contaminant in fresh water eco-system other than paracetamol [29]. Increased exposure to these medicines from multiple sources could contribute to the sensitisation and production of immune and inflammatory responses [30]. The mechanism of NSAIDs-induced angioedema includes the release of inflammatory mediators by immune and non-immune mechanisms [31]. The prevalence of nonsteroidal anti-inflammatory drugs (NSAIDs)-induced skin reaction is between 1 and 9%, where angioedema is the most common presentation and often occurred immediately in less than 2 h [31,32,33]. Beta-lactam antibiotics including amoxicillin are also have been found common to cause skin reaction similar to this study finding (Table 4) [31, 32].

This study also found oculogyric crisis and dystonia are being signalled for metoclopramide by the three signalling techniques (PRR, ROR, MGP). Metoclopramide is indicated for treatment of nausea and vomiting. Oculogyric crisis is an acute dystonic reaction characterised by marked involuntary deviation of the eyes inadvertently move upward and to the left or right. The Malaysia NPRA had issued two safety issues specifically for metoclopramide and oculogyric crisis in 2015 and 2020 [34]. In the first safety announcement in 2015, metoclopramide package insert was updated by contraindicating its use to children less than one-year-old age. Thereafter, the first reminder came in 2020 where oculogyric crisis was accounted for the most neurologic ADRs and one report involving a six-month old infant. In the latest publication by the New Zealand Medicines and Medical Devices Safety Authority in March 2023 about metoclopramide use in children and young adults limiting the metoclopramide solution for age 1- to 19-year-old, while in its tablet form to 15- to 19-year-old [35]. The prevalence of metoclopramide-induced dystonic reaction is 1% and can occur after a single dose. The digital signals found in this study are reflected by the previous cautionary actions by the health authority.

This study illustrated that all four signalling techniques have 100% specificity and positive predictive values suggesting that the statistically significant signal produced by the four techniques warrant prioritisation and may proceed for further investigation and clinical deliberation. This study was conducted to explore the potential of safety signals among paediatric populations based on electronic data from the voluntary ADR reports submitted to NPRA. Traditional manual review of spontaneous reports for safety signals is satisfactory when there is a smaller number of ADR reports, but when the volume exceeds the reporting rates, statistical signal detection methods could complement screening and prioritisation of medicine safety issues. In this study, BCPNN has detected 15 very few signals which is in contrast with other studies reported higher performance of BCPNN [36, 37]. The lower performance of BCPNN can be attributed to NPRA having a smaller size database due to underreporting compared to other countries [38,39,40]. It is important to have spontaneous reporting database of adequate size and diversity in order to evaluate the signal detection methods [41]. In this study, the safety signals detected by PRR, ROR and BCPNN were a subset of safety signals detected by MGPS which is consistent with other studies [26, 42]. MGPS reported highest sensitivity (20%) compared to the other three methods as shown in Fig. 1.

In larger databases, BCPNN and MGPS have shown to be effective in detecting safety signals which PRR and ROR had missed [43]. A high specificity of all four methods indicates that when a safety signal is detected, the causality association between drug and adverse event is higher and false positive signals are unlikely. A lower sensitivity of all four methods indicates under-reporting and lack of diversity in database which was similarly observed in MedWatch database of US FDA in the early years when the number of drug-event pairs were relatively small. In smaller databases like Singapore HSA, PRR and ROR had shown to be effective methods for regulatory use [27]. Sensitivity is expected to be lower in smaller databases for example in this study, where a minimum number of 500 reports was recommended for disproportionality analysis to reduce the false positive signals [41]. A FAERS study tested the impact of paediatric age adjustment and stratification in signal detection and didn’t find any improvement in the performance of signal detection methods [18]. Advanced metrics such as AUCROC were not evaluated as part of this study which can be considered as a limitation.

Bayesian methods (BCPNN and MGPS) are advanced statistical methods which incorporate shrinkage and stratification to produce disproportionality scores toward the null, when there are limited data and small numbers of cases [44]. The statistical modifications used in the MGPS diminish the effect of spuriously high PRR values, thus reducing the number of false-positive safety signals. Thus, MGPS based on EBGM values provides a more stable estimate of the relative reporting rate of an event for a particular product relative to all other events and products in the database being analysed [45]. Lower and upper 95% confidence limits for the EBGM values are denoted with EB05 and EB95, respectively. In this study, MGPS based on EB05 score was found to be the best in disproportionality analysis with PRR and ROR showing near equivalent performance, similar to previous study findings [42].

Routine application of signal detection techniques depends on digital databases of varying size. It may not be a one-size-fits-all scenario. In its current usage, safety signal detection techniques are valuable tools for national pharmacovigilance centres to detect, triage and assess safety signals. Other than the pharmacovigilance centres, computerised clinical decision support system (CDSS) at hospitals may be an alternative platform for incorporation of surveillance system using the signalling techniques. The CDSS is an automated alert system integrated into electronic medical record. CDS systems have been proven to improve paediatric prescribing errors and provide beneficial effects on patient outcomes and physicians performances [6, 7]. CDSS was also found to be effective digital platform compared to voluntary reporting at identifying adverse drug events with only 1% of those voluntary reports identified with CDSS [46]. The active surveillance system using the signalling techniques incorporated in CDSS is therefore recommended and its potential use locally is evident in this study. The statistically significant safety signals may then be further analysed by clinical reviewers to confirm a signal which can result in drug safety alert, withdrawal from the market or changes in labelling.

Malaysia has a dual-tiered system of healthcare services; (1) lead and provided by government, funded by taxpayers, and (2) provided by private sectors. Its citizens are free to choose between the two service providers because both provide universal healthcare coverage through a public-private mode. The growing burden of non-communicable diseases in Malaysia demands for digital health intervention targeting at prevention, screening and diagnosing [47]. Majority of digital health application studies were targeted at high income countries [48]. Although the uptake of digital health is significant during the COVID-19 pandemic, there remain challenges to implement digital health more broadly in Malaysia including capacity, skill, transparency, user-centred design and integration with existing systems [48, 49]. This study attempted to use available local digital database of ADR reports for medication safety surveillance which can be extended to electronic medical records at hospital settings to expedite identification of adverse events in vulnerable and minority patients. Diversifying databases are beneficial for corroboration of medication safety signals and expedition of communication between multiple health institutions for prompt action in curbing adverse events in paediatric population. In Australia, the Therapeutic Goods Administration (TGA) receive suspected ADR reports from health care professionals and consumers and are regularly review for available information while several efforts are undertaken to improve ADR reporting [50]. The Australian ADR reporting remain the highest as compared to other countries including Malaysia. A study in Australia have shown 1 to 3 years of signal detection when employing the techniques on electronic health data compared to the first warning and subsequent withdrawal of medicines 5 to 7 years after first marketing of medicines [51]. Relying on ADR spontaneous reporting databases only may not be ideal in the long run because under-reporting remains an issue in spontaneous reporting of ADRs. Other than signal confirmation, other study designs can be utilised to identify risk factors of the ADRs.

Our study has its strengths and limitations. This study identified application of medicine safety signal detection techniques PRR, ROR, BCPNN and MGPS in recognising safety signals specific to local paediatric population. However, NPRA is not currently employing its own safety signal detection techniques in practice. The NPRA spontaneous ADR reporting database is considerable a small database. The adverse event data is sent to WHO-Uppsala Monitoring Centre (WHO-UMC) for screening of safety signals. At WHO-UMC, IC of BCPNN method is employed in safety signal detection. Secondly, underreporting is expected for spontaneous reporting databases, hence, the safety signals found in this study may not reflect the true magnitude of medication safety issues in Malaysian paediatric population. Establishment of paediatric pharmacovigilance units in the public and private hospitals by implementing CDSS integrated in the electronic medical records in Malaysia would increase the reporting and recording of ADRs and also help in identification of safety risks of medications.

Conclusion

The main research results show that the digital medication safety signalling techniques have the potential to detect ADRs signals using electronic health data of spontaneous reporting database for paediatric population. A high specificity of all four methods indicates false positive signals may be unlikely. The findings of this study suggest that currently available electronic health data can be used effectively for active surveillance of medication in vulnerable population. The signalling tools could help prioritising suspected ADRs for further clinical deliberation and prompt action can be undertaken. Although the present study employed spontaneous reporting data, the safety signalling techniques could be extended to electronic medical record at hospital settings which contain continuous and timely digital medical data recording. Future research should investigate the impact of integrating medication safety signalling techniques into electronic medical record on overall patients’ clinical outcomes and health care services.