Leverage machine learning to identify key measures in hospital operations management: a retrospective study to explore feasibility and performance of four common algorithms

Abstract

Background

Measures in operations management are pivotal for monitoring and assessing various aspects of hospital performance. Existing literature highlights the importance of regularly updating key management measures to reflect changing trends and organizational goals. Advancements in machine learning (ML) have presented promising opportunities for enhancing the process of updating operations management measures. However, their specific application and performance remain relatively unexplored. We aimed to investigate the feasibility and effectiveness of using common ML techniques to identify and update key measures in hospital operations management.

Methods

Historical data on 43 measures on financial balance and quality of care under 4 categories were retrieved from the BI system of a regional health system in Central China. The dataset included 17 surgical and 15 non-surgical departments over 48 months. Four common ML techniques, linear models (LM), random forest (RF), partial least squares (PLS), and neural networks (NN), were used to identify the most important measures. Ordinary least square was employed to investigate the impact of the top 10 measures. A ground truth validation compared the ML-identified key measures against the humanly decided strategic measures from annual meeting minutes.

Results

For financial balancing, inpatient treatment revenue was an important measure in 3/4 years, followed by equipment depreciation costs. The measures identified using the same technique differed between years, though RF and PLS yielded relatively consistent results. For quality of care, none of the ML-identified measures repeated over the years. Those consistently important over four years differed almost entirely among four techniques. On ground truth validation, the 2016–2019 ML-identified measures were among the humanly identified measures, with the exception of equipment depreciation from the 2019 dataset. All the ML-identified measures for quality of care failed to coincide with the humanly decided measures.

Conclusions

Using ML to identify key hospital operational measures is viable but performance of ML techniques vary considerably. RF performs best among the four techniques in identifying key measures in financial balance. None of the ML techniques seem effective for identifying quality of care measures. ML is suggested as a decision support tool to remind and inspire decision-makers in certain aspects of hospital operations management.

Contributions to the literature
• We employed a novel method to investigate the feasibility and efficacy of four common machine learning techniques for identifying and updating key hospital operations management measures.
• Further investigation should be conducted using more comprehensive data, especially to quantify such domains as quality of care.

Background

In recent years, the healthcare industry has witnessed an increased emphasis on data-driven decision-making and performance evaluation to optimize hospital operations and enhance patient outcomes [1,2,3]. Measures in operations management play a pivotal role in this process, especially those measures serving as essential metrics to monitor and assess various aspects of hospital performance. However, the dynamic and complex nature of the healthcare landscape necessitates a proactive approach to ensure their relevance and alignment with evolving strategic priorities.

Existing literature on hospital performance evaluation highlights the importance of regularly updating key management measures to reflect changing healthcare trends and organizational goals. Several studies have underscored the need for a data-driven approach to measure selection and refinement, with a focus on real-time performance monitoring and continuous improvement initiatives [4,5,6].

Moreover, advancements in machine learning (ML) have presented promising opportunities for enhancing the process of updating hospital operations management measures. ML algorithms have demonstrated potential in efficiently analyzing vast amounts of data, identifying patterns, and generating insights that can drive evidence-based decision-making [7, 8]. Though several studies have explored the use of ML algorithms in various healthcare domains, their specific application and performance in updating hospital operations management measures remain relatively unexplored [9,10,11].

In our current practice, key measures for hospital operations management are determined manually during annual work meetings. The decisions are supported using data from the health group’s Business Intelligence (BI) system, which provides a somewhat structured basis for strategic planning. However, this manual method to identify and update measures is labor-intensive and occurs repeatedly each year. Moreover, dependency on manual selection limits the scope of data exploration and might overlook emerging trends or subtle patterns potentially captured through more sophisticated analytical methods. This prompted our exploration of ML as a tool to enhance efficiency and efficacy. Consequently, we initiated the current study to evaluate the feasibility and performance of some common ML algorithms to identify techniques to identify and update key measures in hospital operations management, with a focus on financial balance and healthcare quality.

The current work included evaluating the performance of linear models (LM), random forest (RF), partial least squares (PLS), and neural networks (NN) in identifying key operations measures from historical operation data. Each technique was selected for its relevance and common usage within the healthcare [12]. LM are fundamental in modeling relationships between variables and are extensively utilized for tasks such as prediction and process optimization [13]. RF, an ensemble method, is favored for its robustness and ability to manage large datasets without overfitting, making it suitable for both classification and regression tasks [14]. PLS may deal with multicollinearity, especially when traditional regression models are inadequate due to a high ratio of independent variables to observations [15]. NN, inspired by biological neural networks, are capable of capturing complex, non-linear data relationships, which is advantageous for classification, regression, and pattern recognition tasks [16, 17].

Additionally, we incorporated a ground truth validation process by comparing the ML-identified measures with the actual measures extracted from the annual hospital operations management meetings. Our findings may contribute to advancing knowledge in healthcare management and data-driven decision-making in hospital operations.

Materials and methods

Study setting and design

This study was conducted in a major regional healthcare system in Hubei Province, Central China, which hosts over 2,200 beds and offers a comprehensive range of clinical services at multiple facilities. The facilities are connected through a centralized business intelligence (BI) network, which consolidates all operational data. Annually, key operational management measures are determined during the operations management meetings attended by the leadership and senior administrators. The decision-making is supported by data from the BI system and occasional specialized data presentations. By the time of study, the BI system was limited to basic data computations and lacked advanced computing capabilities, such as ML.

This exploratory study aimed to assess the performance of four prevalent ML techniques in identifying and updating key measures in hospital operations management by analyzing historical operational data. The study did not involve patients or the public.

Measure selection

A total of 43 measures related to financial balance and quality of care were selected, guided by the strategic goals of the healthcare system and the performance evaluation criteria established by the national health authority. The measures were grouped into four categories, including 21 revenue measures, 8 service measures, 9 cost measures, and 5 quality measures. The revenue, service, and cost measures were used for analysis from the financial balance perspective, ranging from financial transactions such as outpatient registration and medical procedures to operational efficiency metrics including patient bed utilization and cost management for surgical operations and facility maintenance. With fewer measures, the quality category was used for analysis from the quality of care perspective, including nursing quality scores, both outpatient and inpatient antibiotic use density (AUD), and commendation & complaint cases. (Table 1)

Table 1 Hospital operations management measures selected for data mining: financial balance and quality of care (N = 43)

Data collection and preprocessing

Data on the selected measures were extracted from the BI system, including data from 17 surgical departments and 15 non-surgical departments over a period of 48 months (January 2015 through December 2019).

Several rigorous steps were taken to ensure the quality, consistency, and compliance with privacy standards of the raw dataset. Data cleansing involved addressing missing entries (N = 172) through mean imputation for continuous variables and mode imputation for categorical variables. Outliers (N = 54), identified via the interquartile range (IQR) method, were managed through winsorization by replacing extreme values with the nearest values within the IQR. Validation rules were applied to check the accuracy of the records by assessing range validity and logical consistency, where 61 errors were identified and addressed.

The data were then transformed, where numerical features were normalized using the Min-Max scaling technique to align the values within a [0, 1] range and categorical variables were converted into a binary format through one-hot encoding to ensure compatibility with the ML algorithms. The dataset, which was drawn from various departments, were integrated using unique identifiers such as patient and equipment numbers to maintain coherence. The preprocessed data were also divided into annual subsets to facilitate trend analysis over time.

Data analyses

The four selected ML techniques (i.e., LM, RF, PLS, and NN) were used to train predictive models, focusing on two critical components of hospital operations, i.e. financial balance and quality of care. The model training utilized data inputs under each measure item, including 126,295 data entries in total. Each model’s performance was assessed through ten-fold cross-validation to estimate its generalizability to unseen data. The best performing model was then used to derive normalized importance scores (range 0-100) for various operational measures, with higher scores indicating greater importance. The top 10 important measures were identified for further analysis.

Importance scores were calculated for yearly subsets and for the entire four-year set to identify consistently significant measures and those emerging as significant over time. The measures with the greatest overall influence were determined by aggregating the importance scores from all techniques. The results were then ranked to determine the top 10 measures. The ordinary least squares (OLS) method was employed to assess the impact of the measures on financial balance and quality of care, complemented by ridge regression to address multicollinearity issues and enhance the reliability of the findings.

Ground truth validation

To validate the ML-identified key operational measures, the minutes from annual management meetings (2017–2020) were reviewed. The strategic key measures identified in the documents were compared with those identified by the ML models. This comparison, to extent, could reflect the alignment of ML algorithms with stakeholder expectations and provide an additional layer of performance evaluation.

Results

ML-identified financial balance measures

Figure 1 presents the top 10 important measures for financial balance, categorized by ML technique and year (annually and for the total period 2016–2019). The X-axis shows the importance scores, while the Y-axis lists the top measures.

Fig. 1

Top 10 important operational management measures by types of machine learning techniques and years: Financial balance

According to Fig. 1, certain measures consistently emerged as significant across different years and techniques, despite variations in their ranking. Inpatient treatment revenue was notably important in three out of the four years (2017, 2018, 2019), followed by equipment depreciation costs in two years (2016, 2019). (Table 2) In contrast, the findings demonstrated some consistency in the financial balance measures identified using the same ML techniques, particularly RF and PLS, which showed more stable results year-over-year. Inpatient treatment revenue was consistently identified by all four ML techniques, followed by labor costs and inpatient examination revenue by two techniques (RF, PLS). (Table 3) As a result, the measure of inpatient treatment revenue was deemed the most important.

Table 2 Important measures identified with all four machine learning techniques by year: financial balance

Table 3 Measures identified as important over four consecutive years with the same machine learning technique: financial balance

ML-identified quality of care measures

Similarly, Fig. 2 presents the top 10 quality of care measures, with importance scoring on the X-axis and the measures on the Y-axis.

Fig. 2

Top 10 important operational management measures by types of machine learning techniques and years: Quality of care

There were significant variations in the measures identified as important for quality of care, with no common measures consistently recognized across the years by all techniques. The LM predominantly identified inpatient measures as significant, while the RF identified outpatient measures. The PLS and NN were more balanced, which incorporated both inpatient and outpatient measures. Notably, inpatient nursing revenue was frequently recognized across all techniques, suggesting its recurring importance in assessing quality care.

Overall importance

After normalized into a 0-100 scale, the technique-specific importance scores were aggregated to identify the top 10 measures with the highest overall importance for financial balance (Table 4). Consistent with the analyses above, both annual and technique-specific, inpatient treatment revenue also emerged as the measure of highest overall importance.

Table 4 Top 10 measures with the highest overall importance for financial balance

The OLS analysis revealed that the measure of admitted patients had the highest influence coefficient, with all top 10 measures being significant except for labor costs (Table 5). The OLS analysis further indicated that admitted patients and outpatient pharmaceutical revenue exerted a negative influence, whereas the remaining measures had a positive impact. Ridge regression analysis corroborated the findings but with slight variations in the ranking of measures. The polarity of measure influences remained consistent. (Table 6)

Table 5 Ordinary least square (OLS) analysis of top 10 operation management measures with the highest overall importance: financial balance (in descending order of estimates)

Table 6 Ridge analysis of top 10 measures with the highest overall importance: financial balance (in descending order of estimates)

For quality of care, equipment depreciation ranked highest in importance across all four techniques after standardization and weighting, followed by measures associated with inpatient and outpatient revenues (Table 7). OLS analysis indicated that admitted patients had the most significant impact on quality of care, with all measures showing significance except for Level III & IV surgeries (Table 8). The rankings were slightly altered by the ridge regression results, where inpatient laboratory revenue was the most influential, followed by admitted patients. The polarity of influences remained consistent, with the exception of Level III & IV surgeries (Table 9).

Overall, the top 10 measures for quality of care demonstrated greater variability and were less consistent across years and ML techniques compared to financial balance measures. Remarkably, none of the quality care measures ranked among the top 10 in terms of their influence.

Table 7 Top 10 measures with the highest overall importance for quality of care

Table 8 Ordinary least square (OLS) analysis of top 10 operation management measures with the highest overall importance: quality of care (in descending order of estimates)

Table 9 Ridge analysis of top 10 measures with the highest overall importance: quality of care (in descending order of estimates)

Ground truth validation

The minutes from the 2017–2020 annual operations management meetings were examined to compare the significant measures identified by the four ML techniques annually against those determined through manual decision-making. Measures related to financial balance showed strong alignment with strategically identified measures in the meetings. The measures identified from the 2016–2019 datasets corresponded with the strategic measures designated for subsequent years, except for equipment depreciation in 2019, which was not selected for 2020.

In contrast, the ML-identified measures for quality of care did not align with those from the meeting minutes. The manually determined measures predominantly included qualitative metrics such as commendation & complaint cases.

Discussion

ML has exhibited substantial promise across various healthcare domains, notably in disease prediction, hospital outcome analysis, and medical imaging [12, 17]. While traditional methods such as the Analytical Hierarchy Process, Delphi method, and text mining have been employed to identify key hospital performance measures, the use of ML introduces a novel perspective [18, 19]. The novel ML technologies can facilitate deeper insights from large datasets for improved decision-making, patient outcomes, and research efficiency while streamlining daily workflows [20,21,22]. In this study, we assessed four commonly used ML techniques to assist with the identification and updating of key operational measures in hospital management. Our findings were mixed.

Significant variability in ML performance

According to our findings, ML techniques are effective at identifying key measures related to financial balance in hospital operations. The top 10 key financial measures identified using ML demonstrate good consistency and align well with decisions made by humans. However, the effectiveness of the ML techniques varied. RF is particularly effective, identifying the most key measures (n = 4), whereas LM and NN each only identified one key measure. This variability stresses the importance of selecting the appropriate ML technique for specific aspects of operations management.

Although ML techniques are generally designed to be versatile, they are not universally optimal across all tasks or datasets, particularly when dealing with unique data characteristics or complex operational data in large volumes [23]. Therefore, customizing algorithms to the specific tasks and datasets could significantly enhance the performance of ML models, where unique characteristics such as specific patterns, trends, or anomalies may be taken into account. This may allow for more accurate and reliable results in tasks such as identification of key measures in hospital operations management.

Factors influencing ML effectiveness

Despite the favorable performance in identifying financial balance measures, the ML techniques failed to identify key measures in quality of care. This discrepancy can be attributed to several factors. Financial data is typically structured, quantitative, and consistent, making it easier for ML models to process and identify patterns [24]. In contrast, quality of care data often involves subjective or qualitative metrics, which pose greater challenges for ML models [25, 26]. Furthermore, financial measures such as costs, revenues, and resource allocations often exhibit consistent patterns that ML models may easily capture. Quality of care, however, is influenced by a broader range of factors, many of which are difficult to quantify or predict. Moreover, financial data is generally more readily available compared to quality of care data, which can be harder to collect and standardize [27].

It is noteworthy that none of the routinely monitored quality of care measures ranked among the top 10 important measures, according to our results. Instead, all ML-identified measures were related to other types such as equipment depreciation, and these identified measures failed to align with those selected during the annual strategic meetings. While many key quality of care measures are mandated by health authorities, some ML techniques are better suited for analyzing structured, quantitative data typical of financial transactions than the complex, qualitative data.

ML as potential decision support

According to our findings, the use of ML can enhance decision-making processes in hospital operations. Techniques such as RF demonstrate potential in identifying key operational measures, which may help decision-makers by highlighting strategic areas otherwise overlooked due to human limitations, which is consistent with previous applications reported in higher education [28]. The integration of ML into a hospital’s operational framework offers several practical advantages. Firstly, ML can rapidly analyze extensive datasets, which allows administrators to gain deeper and more frequent insights into operational dynamics and make informed decisions promptly [7, 29]. Secondly, the adaptive feature of ML supports ongoing learning and refinement of parameters. This ensures that the decision support capability continues to evolve in response to changing needs and data environments.

Furthermore, by ensuring that critical operational measures are continuously monitored and optimized, the application of ML may lead to improved overall quality of care and eventually translate into better patient outcomes and higher service quality. Similar benefits have been documented in previous research. For example, Na and colleagues reported that ML models could enhance decision-making and reduce average hospital stays by predicting patient outcomes in a large hospital network [30]. Additionally, Bishara et al. have shown that the application of ML in acute care settings can improve hospital operational management and patient outcomes using supervised, unsupervised, and reinforcement learning algorithms [31].

Limitations

This study has several limitations, including the reliance on retrospective data from a single healthcare system, which may limit the generalizability of our findings. The performance of ML models could vary with different algorithms or more comprehensive datasets, and our data were limited to those available from the hospital’s BI system. Conducting multi-center studies could help validate our findings across various settings and refine the algorithms’ generalizability and sensitivity to diverse operational environments. Additionally, the qualitative quality-of-care data were not structured for optimal ML use, which likely impacted the effectiveness of our models. For future work, it is suggestible that such technologies as text mining should be incorporated to transform unstructured quality-of-care data into structured inputs for ML models, which might enhance their predictive capabilities.

Conclusions

RF demonstrates superior performance in identify financial balance measures over the other ML techniques. However, none of the examined techniques effectively capture key quality of care measures, potentially attributable to a gap in handling qualitative data. ML may be employed as a decision support tool for enhancing data-driven management by offering decision-makers overlooked options. Future research should may consider to refine ML algorithms to better integrate with Chinese hospital management systems by customizing them to specific operational contexts.