Skip to main content
Download PDF

Human-centered design of a health recommender system for orthopaedic shoulder treatment

BMC Medical Informatics and Decision Making volume 25, Article number: 17 (2025) Cite this article

Abstract

Background

Rich data on diverse patients and their treatments and outcomes within Electronic Health Record (EHR) systems can be used to generate real world evidence. A health recommender system (HRS) framework can be applied to a decision support system application to generate data summaries for similar patients during the clinical encounter to assist physicians and patients in making evidence-based shared treatment decisions.

Objective

A human-centered design (HCD) process was used to develop a HRS for treatment decision support in orthopaedic medicine, the Informatics Consult for Individualized Treatment (I-C-IT). We also evaluate the usability and utility of the system from the physician’s perspective, focusing on elements of utility and shared decision-making in orthopaedic medicine.

Methods

The HCD process for I-C-IT included 6 steps across three phases of analysis, design, and evaluation. A team of health informatics and comparative effectiveness researchers directly engaged with orthopaedic surgeon subject matter experts in a collaborative I-C-IT prototype design process. Ten orthopaedic surgeons participated in a mixed methods evaluation of the I-C-IT prototype that was produced.

Results

The HCD process resulted in a prototype system, I-C-IT, with 14 data visualization elements and a set of design principles crucial for HRS for decision support. The overall standard system usability scale (SUS) score for the I-C-IT Webapp prototype was 88.75 indicating high usability. In addition, utility questions addressing shared decision-making found that 90% of orthopaedic surgeon respondents either strongly agreed or agreed that I-C-IT would help them make data informed decisions with their patients.

Conclusion

The HCD process produced an HRS prototype that is capable of supporting orthopaedic surgeons and patients in their information needs during clinical encounters. Future research should focus on refining I-C-IT by incorporating patient feedback in future iterative cycles of system design and evaluation.

Peer Review reports

Introduction

Evidence-based medicine has advanced significantly since its popularization in the 1990s [1, 2], though many practice-specific gaps remain for the care of individual patients. It is well recognized that treatment effect heterogeneity erodes the value of average treatment effect estimates across groups of diverse patients [3,4,5,6], and a gap remains to generate evidence that applies to individual patients [1, 2, 7]. Rich data on diverse patients and their treatments and outcomes stored in Electronic Health Record (EHR) systems can be used to generate real world evidence suited for individual patients [8,9,10,11,12,13]. Using historical patient data stored in EHR systems to assist with clinical decision-making was first conceptualized by researchers at Stanford and was referred to as an informatics consult service (ICS) [12,13,14,15]. ICSs are designed to summarize historical data to help physicians make data-informed choices for current patients. However, the initial ICSs were focused on questions of prognosis not treatment choice [15], and they were not fully automated, requiring a team of researchers and data scientists to field requests and return responses over a period of days. New computing advancements can be applied to fill the analytical gaps of early ICS systems and deliver real-time evidence to guide individual treatment decisions at the point of care.

Comparative effectiveness research (CER) has been used to generate evidence about treatment alternatives to inform treatment decision-making. Traditional CER approaches require researchers to generate average treatment effect estimates for a population subgroup or a “reference classes” based on a combination of measured baseline factors specified prior to estimation [16,17,18]. The need to define reference classes is a significant barrier to finding appropriate treatment effect estimates for individual patients [18]. Even with a small number of measured baseline factors, an individual patient could potentially be placed in an indefinite number of reference classes [16,17,18], and it is often unclear which reference class is best aligned to each patient [12, 16, 17, 19,20,21]. Here we start from the premise that physicians possess greater accumulated clinical wisdom on the patient factors that impact treatment effectiveness than researchers [15, 22, 23]. As such, physicians may be best positioned to define the appropriate reference class for a given patient in a specific setting and need a tool to generate CER-based evidence to summarize outcomes for similar patients under treatment alternatives.

Conceptually, Recommender Systems (RSs) provide an effective system foundation for delivering tailored evidence to guide individual decision making. RS are intelligent applications [24] that provide personalized information fit for the end user [25]. A health recommender system (HRS) focuses on health-related information needs. The core HRS functionality of selectively presenting or “recommending” health data can be adapted to the need of presenting clinically appropriate reference class data summaries which closely match each patient being treated [26]. Presenting personalized treatment evidence can enable engagement in shared decision-making, a priority goal in orthopaedic medicine [27]. Shared decision-making includes physicians and patients being informed about treatment options, evidence of the treatment effects on outcomes the patient can expect, and patient preferences across outcomes [28], so that a patient may play an engaged role in their care. HRSs may offer a unique tool to facilitate shared decision-making in orthopaedic encounters, as HRSs can retrieve and present historical reference class patient data summaries that are matched to the individual patient. These reference class summaries would include the frequency of treatment alternative use, and related outcomes for patients that match the profile of the patient being treated. This type of personalized data has never been available in the clinical encounter and can enhance patient understanding surrounding treatment options and outcomes the patient can expect following treatment. Together, this information will better help patients choose the treatment that offers outcomes best aligned with their preferences. We use the HRS conceptual foundation to develop a clinical decision support system for orthopaedic medicine, the Informatics Consult for Individualized Treatment (I-C-IT, pronounced “I see it”).

The management of proximal humerus fracture (PHF) provides a useful clinical scenario for developing I-C-IT. There is significant variation in the management of PHF [29,30,31,32,33], which is partially due to a lack of evidence to inform the choice between surgical options or conservative care [34, 35]. Meta-analyses across clinical trials seems to validate these concerns, suggesting the comparative effect of surgery versus conservative care is likely heterogenous across patients [32]. Surgical treatment is more costly, increases the risk of surgical complications and mortality [36], and has not been shown in clinical studies to produce superior pain reduction or functional outcomes for many patients [30, 31]. Yet, surgery continues to be used at high rates in many regions of the country [37, 38]. It is unclear whether current surgery rates for many subsets of patients with PHF represent over or underutilization, with the key question not being whether surgery or conservative management is “the” best treatment, but rather which treatment is best suited for each patient [39,40,41,42,43,44,45]. Evidence on the comparative effects of alternative treatments that are appropriate for each patient [46] is needed to guide individual treatment choice.

In this study, we adopt a human-centered design (HCD) [47, 48] approach to identify design requirements and develop the HRS prototype, the I-C-IT, for PHF treatment decision support. HCD prioritizes users’ needs and requirements during the system development process and is concerned with ways in which components of interactive systems can enhance human-system interaction. This work builds upon the original ICS concept [15, 49] to design and develop a HRS to perform dynamic searches and receive immediate comparative treatment and outcome data summaries to aid in shared treatment decision-making by physicians and patients at the point of care for PHF. In the final stage of the HCD process, we engaged ten orthopaedic surgeons in a mixed methods evaluation of the I-C-IT prototype that was produced.

Methods

The HCD process used to develop I-C-IT included 6 steps across three phases of design, analysis, and evaluation (Fig. 1). In addition to traditional steps involving process analysis (Step 3), and feedback through usability assessments (Step 6), we used a collaborative design approach where physician end-users were engaged in the design and evaluation process. Table 1 outlines the HCD phase, steps, and activities performed. A team of informaticians and comparative effectiveness researchers collaborated with input from orthopaedic surgeon subject matter experts. This study was reviewed and approved by the Institutional Review Board at Prisma Health (Protocol #1990879-1).

Fig. 1
figure 1

Human-centered design process for health recommender system instance – I-C-IT

Full size image
Table 1 Human-centered design process for I-C-IT
Full size table

Step 1. Literature search of patient factors for reference class queries

A priori the research team envisioned a system in which physicians apply accumulated wisdom to define the most appropriate reference class for a given patient. To achieve that envisioned design feature, a list of patient factors thought to modify treatment effectiveness was needed. A team of two researchers conducted a search of the literature to identify a list of patient factors associated with treatment effectiveness and treatment choice for PHF. After review, a final group of 10 articles were identified and reviewed from PubMed and Web of Sciences databases. Forty-five individual patient factors were identified and grouped into five domains: social independence, bone quality, patient mobility and activity level, surgical risk factors, fracture characteristics. These factors provided a comprehensive list of patient factors that could be used for reference class queries.

Step 2. Clinical chart review and data modeling

A team of three comparative effectiveness and medical student researchers (S.L., P.M., J.M.) conducted a manual chart review in the EHR system to extract each of the individual patient factors identified from the literature review. Nine hundred and fifty-three patient charts were reviewed. Structured and unstructured data fields were scanned to identify factors of interest. A regression analysis was used to identify the patient factors most relevant to treatment choice and relevant for reference class selection. Step 1 and 2 provided a refined list of patient factors which could be used for reference class identification.

Step 3. Ethnographic study to understand the orthopaedic treatment process

We conducted an ethnographic study [50, 51] (completed by author A.S.) to understand the process of treatment decision-making for orthopaedic surgeons and identify the decision points during an orthopaedic clinical visit, the design factors for HRS prototype, and elements for reference class for similar patient cohort selection. We shadowed two orthopaedic surgeons with different experience levels (< 15 years and 25 + years) in a large and busy clinical practice for a total of 11 h encompassing 42 patient visits. The ethnographic study undertaken in this step helped identify the decision points, information elements involved in the treatment decisions, data evidence requirements and other design requirements to guide the design of desired functionality for the proposed I-C-IT system. The team intended to design a tool which would align with the current information seeking and decision-making process undertaken by physicians. This step elucidated key design features for the system, to maximize usability and utility for physicians. Based on observing the data needs, treatment decision-making process, and patient and physician discussions across all 42 patient visits, we theorized the design elements to include data visualizations across three main areas (1) initial treatment utilization patterns and trends, (2) patient-reported outcome comparisons across treatment alternatives, and (3) 1-year healthcare utilization comparisons across treatment alternatives.

Step 4. Collaborative design process to create wireframes

Next, we iteratively designed wireframes (i.e., visual mockups) for the I-C-IT prototype using the Adobe XD modeling tool. Two informaticians and three comparative effectiveness researchers conducted agile design sessions consisting of fifteen-day intervals inclusive of wireframe designing, gaining user feedback, prioritizing feedback across researchers, and creating revised designs. The collaborative design process spanned 6 months and produced roughly 15 iterations of I-C-IT. The resulting wireframes provided a vision for the desired system. This collaborative design process resulted in a system design that allows providers to define appropriate reference classes for a given patient using patient factor filters to query historical data. For that reference class, summarized data visualizations are presented for initial treatment utilization patterns and trends, patient-reported outcome and 1-year healthcare utilization comparisons across treatment groups.

At the end of the collaborative design process, 30-minute semi-structured interviews (conducted by S.F. and A.S.) were conducted with four orthopaedic surgeons specializing in shoulder conditions. Physicians feedback on design elements was used to inform design enhancements and help refine overall design features in the I-C-IT prototype wireframes. Examples of enhancements that came from interviews included grouping patient demographic, fracture characteristics, and comorbidities together. Additionally, physicians desired an expanded PHF factor list to include the specific fracture location and amount of displacement, beyond just displacement and the number of fracture parts. On the third tab for healthcare utilization outcomes, one physician suggested adding a data element to report the percentage of initial conservatively managed patients that failed and converted to surgery. Smaller revisions included having treatment comparators in distinct colors to easily contrast treatment groups.

Step 5. Informed development of I-C-IT Webapp Prototype

In this step, we developed the first high fidelity webapp prototype of I-C-IT. This prototype was built as a webapp driven by simulated data elements for 2,000 unique patients with PHF. It was designed to support two key future functions. The first function was integration into an EHR by incorporating a Java Script Notation (JSON) framework for patient search criteria and related outcome data. Second, a semantic web framework was applied using data elements in I-C-IT to provide a platform for AI-based algorithms to be implemented to support a background inference engine, while being integrated with EHR systems. Figure 2 shows images of the I-C-IT Webapp Prototype.

Fig. 2a
figure 2

I-C-IT Webapp prototype screen 1

Full size image
Fig. 2b
figure 3

I-C-IT Webapp prototype screen 2

Full size image
Fig. 2c
figure 4

I-C-IT Webapp prototype screen 3

Full size image

Step 6. Evaluation of HRS I-C-IT prototype design

This phase of the HCD process focused on enhancing the I-C-IT prototype design by gathering feedback from 10 orthopaedic surgeons in a mixed methods study. We used a purposive, convenience sampling approach to recruit shoulder specialist orthopaedic surgeons from the regional practice network associated with our orthopaedic research group, the Center for Effectiveness Research in Orthopaedics (CERortho). A total of 15 physician participants were emailed and invited to participate in an interview, of which 10 accepted and were interviewed. Those physicians declining to participate stated time constraints n=2, lack of interest in research participation n=2, or lack of expertise in PHF n=1 for the reasons they declined to participate. Physician participants and researchers had previously collaborated on orthopaedic research and were in the same research network. Both researchers conducting the interviews (S.F. and A.S.) reported their interest in the research topic to physician participants. Physician participants were not incentivized for their time participating.

For the purposes of this study we developed and conducted 60-minute qualitative semi-structured interviews with each orthopaedic surgeon where we prompted them surrounding I-C-IT facilitators and barriers of use, and how I-C-IT might enhance their current practice in terms of engaging in shared decision-making (see full interview guide in the Appendix). We used prompts for the semi-structured interview that included a combination of prompts for I-C-IT usability and utility and modified questions from previous work from our research team. Authors S.F. and A.S., both female, academic researchers holding PhDs in health services research and health informatics, conducted physician interviews via Zoom. We saved Zoom audio recordings for each interview and transcribed them for analysis purposes. We did not collect any other field notes during the interview for analysis purposes, and transcripts were not returned to participants for their review. Both S.F. and A.S. had previously conducted physician interviews with orthopaedic surgeons and were experienced in leading physician interviews. Only researchers and the physician participant were in attendance during the interviews and no repeat interviews were conducted for this step.

We conducted a thematic analysis of interview data using the NVivo-12 software and applied a peer debriefing qualitative research method [52]. Two interviewers independently reviewed the identified themes (B.S. and A.S.) and reconciled the analysis to produce final evaluation results. Final results were derived from the data.

At the end of the interview, we asked participants to complete a standard system usability scale (SUS) questionnaire [53] as well as a customized utility survey to evaluate several associated utility constructs including relevance of reference class filters, ease of navigation and understanding, interpretability of data representations, utility of the system in making evidence informed treatment decisions, and I-C-IT helpfulness in discussion with patients for shared decision-making. Table 2 provides information on participants that completed the interview and utility survey in Step 6.Results.

Table 2 Demographics of physician participants
Full size table

Patient demographics and clinical factors for reference class selection

The goal of Step 1 and 2 of the HCD process was to generate a comprehensive list of patient factors that are relevant to treatment effectiveness and choice. The research team identified a total of 45 patient factors that were eventually minimized to 14 based on collaborative design feedback in Step 4. Patient factors selected for inclusion included patient age, gender, injury of the dominant shoulder, fracture displacement, amount of displacement, location(s) of the fracture, number of topographical parts involved, presence of concurrent fractures, patient frailty measured by a history of falls and a subjective frailty assessment, diabetes, dementia, osteoarthritis and whether the patient lives independently.

Information elements for I-C-IT

Decisions about what treatment and outcome information to present in the results tabs of I-C-IT were first envisioned during Step 3 and iteratively designed and evaluated in Step 4. Approximately 15 rounds of design and feedback were undertaken, in which discussion centered on information presentation and visualization modification. The process was concluded when all feedback had been addressed and no new suggestions were made by the team. The final version of I-C-IT (see Fig. 2a, 2b, 2c) contains 3 results tabs and a total of 14 data visualizations for the reference class. Table 3 provides the identified 14 informational elements for use in I-C-IT.

Table 3 Information elements for I-C-IT
Full size table

Human-centered design of I-C-IT

The aim of I-C-IT was to provide a clinical decision support tool that could stratify historical patient cohorts and summarize personalized treatment evidence to aid in clinical decision support. The design of the I-C-IT interface contains reference class search criteria on the left-hand side, while the middle pane contains various tabs presenting summarized data on treatment utilization, patient-reported outcome measure comparisons, and healthcare utilization for surgical and non-surgical treatment groups within the matched reference class criteria. The prototype was presented to orthopaedic surgeons in this study to assess whether the system captures and presents the design elements needed to support PHF treatment decision-making. Participants affirmed that the main key evidence and decision factors were represented. Future iterations will continue to refine desired system elements and functionality. Also revealed during the HCD process was a set of inductive HCD principles for HRS design to maximize usability and utility in a shared decision-making clinical context and setting. Table 4 represents the design features as they link to a human factor and design principle for HRS design.

Table 4 Human-centered design principles for HRS
Full size table

Mixed method evaluation of usability and utility of I-C-IT webapp prototype

Usability was first addressed with individual orthopaedic surgeons in Step 4 and formally evaluated in Step 6. In the formal evaluation, the overall SUS score for the I-C-IT Webapp prototype was 88.75 indicating high usability [54]. The utility survey results indicated overall positive feedback on the design of I-C-IT (Fig. 3). The response scores were on a 5-point Likert scale (5 = Strongly Agree to 1 = Strongly Disagree). Most orthopaedic surgeons responded between agree to strongly agree for all questions, indicating mostly strong agreement to questions on ease of navigation, ease of understanding, interpretability, utility of system in making data evidence informed treatment decisions, relevant cohort matching filters, helpfulness in discussion with patients for shared decision-making in orthopaedic treatments, appropriateness of data visualizations for PHFs, and willingness to use a similar system integrated in an EHR in their day to day treatment decisions for other orthopaedic conditions. Elements of shared decision-making were specifically addressed in Question 7, in which 90% of respondents either strongly agreed or agreed that I-C-IT would help them make data informed decisions with their patients (Fig. 3).

Fig. 3
figure 5

Results of utility survey

Full size image

From the thematic analysis of the qualitative interview data in Step 6 of the HCD design process, a theme emerged surrounding how I-C-IT could enhance PHF treatment decision-making at point of care. Physicians confirmed the availability of tailored data elements and summaries within I-C-IT would facilitate richer conversations with patients about their preferences for outcomes of care and treatment choice. For instance, when asking one physician participant how he approached thinking about treatment decisions, he responded “I think it’s comorbidities, mechanism, and then patient goals.” This result signaled that beyond the clinical presentation of the patient (comorbidities and fracture characteristics), patient goals are highly relevant for treatment decision-making, and a summative presentation of outcomes would be helpful to share with patients as they weigh treatment choices.

Moreover, the amount of shoulder functionality desired by a patient at the end of treatment was described as one major factor in choosing a treatment option. One physician described an example of a functional goal for an older woman, “The majority of the people who break their shoulder are older, and a lot of them are women, and the biggest problem is having difficulties with putting their bra on or combing their hair.” Therefore, the presentation of shoulder range of motion scores in I-C-IT was highly valuable to have available for patient conversations when weighing treatment decisions. Additionally, many physicians consider pain reduction as a primary goal for most PHF patients, thus the availability of pain score outcomes by treatment group was highly desirable to physicians as it will be key for shared decision-making discussions. One physician participant noted how and when the I-C-IT tool could be used to educate and engage in shared decision making with their PHF patients on treatment options, “When I talk to patients, I’d pull this up in a user-friendly way, I can spend 15 min with the patient coaching them in the office.” All physician participants responded positively to the data elements that I-C-IT contained and the ways in which I-C-IT could be used to enhance clinical encounters with their patients.

Discussion

We describe the HCD approach used to develop I-C-IT, a prototype HRS designed to provide real-time CER-based evidence for the treatment of PHF. The long-term goal for I-C-IT is to improve physicians’ and patients’ ability to make evidence-based data-informed treatment decisions and select the treatment that is best aligned with treatment evidence and patient preferences. I-C-IT offers personalized evidence for the unique patient that is sitting in front of the physician, which provides more tailored information than systematic reviews or meta-analyses on groups of patients that may deviate from the current patient. Visualized data summaries on how similar patients have been treated brings a unique perspective to the clinical encounter that is not currently available and historical data summarized by I-C-IT may reveal previously concealed data relationships and patterns to help overcome common cognitive pitfalls. Further, this patient-specific data will enhance shared decision-making within orthopaedic encounters.

Also, I-C-IT delivers up-to-date evidence on rapidly evolving practice trends which is an advancement over the sluggish translation of peer-reviewed literature. In the future, each patient that is treated in clinical practice will be looped back into the I-C-IT data pool for future analysis through recommender system data pipelines, which creates an ongoing learning cycle for personalized visualizations. Furthermore, as new treatments are adopted and gain momentum in practice, then the ICIT system interface could be modified to show the utilization and outcome data associated with new treatment options. Thus, future patient treatment choice can be improved from the treatment of patients today. Through ongoing development, HRSs like I-C-IT may be useful for supplementing shared decision-making between physician and patients in making on-demand informatics consults during the clinical encounter at the point of care. Physicians and patients can together review the personalized results presented in I-C-IT and discuss treatment options and expected outcomes. Having data visualizations available during the clinical encounter, patients may be better able to understand expected treatment outcomes or may better be able to articulate questions surrounding treatment alternatives and expected outcomes.

I-C-IT is similar in many ways to other decision support applications, though our system differs in a few meaningful ways. First, unlike other systems that depend solely upon computer-based algorithms to phenotype patients [55] or identify patient reference classes, we leverage a physician-driven reference class identification and data recommendation process to summarize information. Machine learning or artificial intelligence-based solutions use statistical methods to group patients by closeness or fit, but these methods are void of physician acumen and intuition [9, 56, 57]. Computer-based approaches lack transparency in how patient reference classes are defined and can lead to challenges in the ability to explain and interpret results ex post because physicians cannot clearly identify the defining factors used to group patients [58].

Furthermore, there is hesitancy among patients and physicians to use computer-based recommender systems that lack the art of the human element of medicine [22, 23]. Physicians possess accumulated clinical wisdom on the patient characteristics that are relevant when making treatment decisions [15, 59, 60]. Because of this unique knowledge, physicians may be better positioned to phenotype or define reference classes of patients. Furthermore, I-C-IT has the capability to track physician reference class search patterns over time and learn from physician behavior to create ‘user profiles’ that exist in most content-filtering recommender systems. This provides the best translation of physician expert knowledge into computerized patterns of patient cohort searches and their specific outcomes. The expertise that a physician “knows what to search for” can be used to train computerized models to automate searches around the index patient in future integrations with EHR systems.

Limitations and future work

Our work is not without limitations. First, this study focuses on the development of a HRS for one orthopaedic condition. More studies are needed to expand the HCD methodology to the development of HRSs for other leading orthopaedic conditions such as atraumatic rotator cuff tears, spine injuries, or knee injuries. We envision that one day I-C-IT, or a future iteration on our HRS, will be embedded within the EHR so that it is readily available and accessible during the clinical encounter. Thus, since our target point of care environment for system use is the clinical encounter, we believe physicians will be the primary end-users. However, the data visualization summaries produced by I-C-IT would also be shared with patients during the clinical encounter, and thus a system evaluation from patients’ perspective is also critical to evaluate the impact of I-C-IT on the overall shared decision-making process at point of care. The next step in our research will focus on (i) evaluating the patient perspective of the design and data visualizations contained within I-C-IT and (ii) developing interpretability from the perspectives of the patient user. Furthermore, the varying health literacy of patients must be considered when designing data visualizations and finalizing the I-C-IT design. Additionally, the surgeon sample that provided initial usability and utility scores for I-C-IT was selected from a sample of physicians within an orthopaedic research network. It’s possible their scores were influenced by their proclivity to engage in orthopaedic research. Finally, for the utility of the system to be thoroughly tested for its real-world application, I-C-IT must be integrated with a dynamic data source that periodically integrates new data evidence, such as the Orthopaedic Data Repository (OPDR) built at the Center for Effectiveness Research in Orthopaedics (CERortho), that contains multiple years of EHR data. The current I-C-IT prototype operates using a limited data set for the purpose of studying physician interactions with the interface. While this work provided significant foundational knowledge for a human centered orthopaedic HRS interface design, follow-on large-scale studies are needed to understand the possible clinical implications of the I-C-IT design and to assess impact of using HRSs such as I-C-IT for treatment decision support in clinical settings.

Conclusion

Through this study we described a HCD process for development of a prototype HRS refined for orthopaedic medicine. Historical EHR data holds important answers about how current and future patients might be treated. Informatics consult systems that enable dynamic learning from historical treatment decisions and outcomes hold great potential for improving the quality of care for all orthopaedic patients. However, studies such as the one presented in this paper are needed to build the foundational logic, decision factors, and human-computer interactions required for an HRS to have a positive clinical impact. The foundational design framework of I-C-IT offers a novel paradigm to accelerate learning from historical practice and associated outcomes for orthopaedic conditions.

Data availability

The datasets generated and/or analyzed during the current study are not publicly available due to the ongoing development of the clinical decision support tool but are available from the corresponding author on reasonable request.

Abbreviations

EHR:

Electronic Health Record

CER:

Comparative effectiveness research

RS:

Recommender system

HRS:

Health recommender system

I-C-IT:

Informatics Consult for Individualized Treatment

HCD:

Human centered design

SUS:

System usability scale

PHF:

Proximal Humerus Fracture

PROM:

Patient-Reported Outcome Measure

SANE:

Single Assessment Numeric Evaluation

References

  1. Sheridan DJ, Julian DG. Achievements and limitations of evidence-based medicine. J Am Coll Cardiol. 2016;68(2):204–13.

    Article  PubMed  Google Scholar 

  2. Sackett DL, Rosenberg WMC, Gray JAM, Haynes RB, Richardson WS. Evidence based medicine: what it is and what it isn’t. BMJ. 1996;312(7023):71–2.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  3. Brooks JM, Chrischilles EA. Heterogeneity and the interpretation of treatment effect estimates from risk adjustment and instrumental variable methods. Med Care [Internet]. 2007 Oct [cited 2020 Mar 22];45(10 Supl 2):S123-30. Available from: http://www.ncbi.nlm.nih.gov/pubmed/17909370

  4. Brooks JM, Chapman CG, Schroeder MC. Understanding Treatment Effect Estimates When Treatment Effects Are Heterogeneous for More Than One Outcome. Appl Health Econ Health Policy [Internet]. 2018 Jun 1 [cited 2020 Mar 22];16(3):381–93. Available from: http://www.ncbi.nlm.nih.gov/pubmed/29589296

  5. Angrist JD. Treatment Effect Heterogeneity in Theory and Practice. The Economic Journal [Internet]. 2004 Mar 1 [cited 2023 Nov 27];114(494):C52–83. Available from: https://doiorg.publicaciones.saludcastillayleon.es/10.1111/j.0013-0133.2003.00195.x

  6. Floyd SB, Thigpen C, Kissenberth M, Brooks JM. Association of Surgical Treatment With Adverse Events and Mortality Among Medicare Beneficiaries With Proximal Humerus Fracture. JAMA Netw Open [Internet]. 2020 Jan 3 [cited 2023 Nov 27];3(1):e1918663–e1918663. Available from: https://jamanetwork.com/journals/jamanetworkopen/fullarticle/2758471

  7. Evidence-Based Medicine and the Changing Nature of Healthcare. 2007 IOM Annual Meeting Summary. Washington (DC): National Academies Press ; 2008.

  8. Ross MK, Wei W, Ohno-Machado L. Big Data and the Electronic Health Record. Yearb Med Inf. 2018;23(01):97–104.

    Google Scholar 

  9. Tang PC, Miller S, Stavropoulos H, Kartoun U, Zambrano J, Ng K. Precision population analytics: population management at the point-of-care. J Am Med Inform Assoc. 2021;28(3):588–95.

    Article  PubMed  Google Scholar 

  10. Gallego B, Dunn AG, Coiera E. Role of electronic health records in comparative effectiveness research. J Comp Eff Res. 2013;2(6):529–32.

    Article  PubMed  Google Scholar 

  11. Frankovich J, Longhurst CA, Sutherland SM. Evidence-based medicine in the EMR era. NEJM. 2011.

  12. Gallego B, Walter SR, Day RO, Dunn AG, Sivaraman V, Shah N, et al. Bringing cohort studies to the bedside: framework for a green button’’ to support clinical decision-making. J Comp Eff Res. 2015;4(3):191–7.

    Article  PubMed  Google Scholar 

  13. Longhurst CA, Harrington RA, Shah NH. A green button for using aggregate patient data at the point of care. Health Aff (Millwood). 2014;33(7):1229–35.

    Article  PubMed  Google Scholar 

  14. Gombar S, Callahan A, Califf R, Harrington R, Shah N. It is time to learn from patients like mine. Digit Med. 2019;2:16.

    Article  Google Scholar 

  15. Schuler A, Callahan A, Jung K, Shah NH. Performing an Informatics Consult: methods and challenges. J Am Coll Radiol. 2018;15(3):563–8.

    Article  PubMed  PubMed Central  Google Scholar 

  16. Dahabreh IJ, Hayward R, Kent DM. Using group data to treat individuals: understanding heterogeneous treatment effects in the age of precision medicine and patient-centred evidence. Int J Epidemiol. 2016;45(6):2184–93.

    PubMed  PubMed Central  Google Scholar 

  17. Kent DM, Van Klaveren D, Paulus JK, D’Agostino R, Goodman S, Hayward R, et al. The predictive approaches to treatment effect heterogeneity (PATH) statement: explanation and elaboration. Ann Intern Med. 2020;172(1):W1–25.

    Article  PubMed  Google Scholar 

  18. Kent DM, Steyerberg E, Van Klaveren D. Personalized evidence based medicine: predictive approaches to heterogeneous treatment effects. BMJ. 2018;363.

  19. Kravitz RL, Duan N, Braslow J, Evidence-Based, Medicine. Heterogeneity of Treatment effects, and the trouble with averages. Milbank Q. 2004;82(4):661–87.

    Article  PubMed  PubMed Central  Google Scholar 

  20. Wendling T, Jung K, Callahan A, Schuler A, Shah NH, Gallego B. Comparing methods for estimation of heterogeneous treatment effects using observational data from health care databases. Stat Med. 2018;37(23):3309–24.

    Article  CAS  PubMed  Google Scholar 

  21. Brooks JM, Chapman CG, Schroeder MC. Understanding treatment effect estimates when Treatment effects are heterogeneous for more Than one outcome. Appl Health Econ Health Policy. 2018;16(3):381–93.

    Article  PubMed  PubMed Central  Google Scholar 

  22. Richardson JP, Smith C, Curtis S, Watson S, Zhu X, Barry B, et al. Patient apprehensions about the use of artificial intelligence in healthcare. NPJ Digit Med. 2021;4(1):140.

    Article  PubMed  PubMed Central  Google Scholar 

  23. Soellner M, Koenigstorfer J. Compliance with medical recommendations depending on the use of artificial intelligence as a diagnostic method. BMC Med Inf Decis Mak. 2021;21(1):236.

    Article  Google Scholar 

  24. Resnick P, Varian HR. Recommender systems. Commun ACM. 1997;40(3):56–8.

    Article  Google Scholar 

  25. Wiesner M, Pfeifer D. Health Recommender systems: concepts, requirements, Technical Basics and challenges. Int J Environ Res Public Health. 2014;11:11.

    Article  Google Scholar 

  26. Wiesner M, Pfeifer D. Health Recommender Systems: Concepts, Requirements, Technical Basics and Challenges. Int J Environ Res Public Health [Internet]. 2014 [cited 2022 Apr 17];11:11. Available from: www.mdpi.com/journal/ijerph

  27. Frosch DL, Kaplan RM. Shared decision making in clinical medicine: past research and future directions. Am J Prev Med. 1999;17(4):285–94.

    Article  CAS  PubMed  Google Scholar 

  28. Singh A, Schooley B, Floyd SB, Pill SG, Brooks JM. Patient preferences as human factors for health data recommender systems and shared decision making in orthopaedic practice. Front Digit Health [Internet]. 2023 Jun 20 [cited 2023 Jun 24];5:1137066. Available from: https://www.frontiersin.org/articles/https://doiorg.publicaciones.saludcastillayleon.es/10.3389/fdgth.2023.1137066/full

  29. Handoll H, Brealey S, Rangan A, Keding A, Corbacho B, Jefferson L, et al. The ProFHER (PROximal fracture of the Humerus: evaluation by Randomisation) trial - a pragmatic multicentre randomised controlled trial evaluating the clinical effectiveness and cost-effectiveness of surgical compared with non-surgical treatment for proxi. Health Technol Assess. 2015;19(24):1–280. 2015/03/31.

    Article  PubMed  PubMed Central  Google Scholar 

  30. Handoll HH, Keding A, Corbacho B, Brealey SD, Hewitt C, Rangan A. Five-year follow-up results of the PROFHER trial comparing operative and non-operative treatment of adults with a displaced fracture of the proximal humerus. Bone Joint J. 2017;99:383–92.

    Article  PubMed  Google Scholar 

  31. Rangan A, Handoll H, Brealey S, Jefferson L, Keding A, Martin BC, et al. Surgical vs Nonsurgical treatment of adults with displaced fractures of the proximal humerus: the PROFHER Randomized Clinical Trial. JAMA. 2015;313(10):1037–47.

    Article  CAS  PubMed  Google Scholar 

  32. PN SSK, G D, A T, GC M. P. R. Trials based on specific fracture configuration and surgical procedures likely to be more relevant for decision making in the management of fractures of the proximal humerus. https://doiorg.publicaciones.saludcastillayleon.es/101302/2046-37585102000638. 2016;5(10):470–80.

  33. LaMartina J, Christmas KN, Simon P, Streit JJ, Allert JW, Clark J, et al. Difficulty in decision making in the treatment of displaced proximal humerus fractures: the effect of uncertainty on surgical outcomes. J Shoulder Elb Surg. 2018;27(3):470–7.

    Article  Google Scholar 

  34. Handoll HH, Ollivere BJ, Rollins KE. Interventions for treating proximal humeral fractures in adults. Cochrane Database Syst Rev. 2012/12/14. 2012;12:Cd000434.

  35. Handoll HH, Brorson S. Interventions for treating proximal humeral fractures in adults. Cochrane Database Syst Rev. 2015;11.

  36. Floyd S, Thigpen C, Kissenberth M, Brooks J. Association of Surgical Treatment with adverse events and mortality among Medicare beneficiaries with proximal Humerus fracture. JAMA Open. 2020;3(1):1–14.

    Google Scholar 

  37. Floyd S, Campbell J, Chapman C, Thigpen C, Kissenberth M, Brooks J. Geographic variation in the treatment of proximal humerus fracture: an update on surgery rates and treatment consensus. J Orthop Surg Res. 2019;14.

  38. Bell LB, Spratt K, Koval K, Weinstein J, Goodman D. others. Trends and Variation in Incidence, Surgical Treatment, and Repeat Surgery of Proximal Humeral Fractures in the Elderly. Journal of Bone and Joint Surgery-American Volume. 2011;93A(2):121–31.

  39. Wennberg JE. Which rate is right? N Engl J Med. 1986;315(13):810–5.

    PubMed  Google Scholar 

  40. McCulloch P, Nagendran M, Campbell WB, Price A, Jani A, Birkmeyer JD, et al. Strategies to reduce variation in the use of surgery. The Lancet. Volume 382. Lancet Publishing Group; 2013. pp. 1130–9.

  41. Bernstein J. Not the last word: Bhandari’s Paradox. Clin Orthop Relat Res 2018/03/15. 2018;476(4):674–7.

    Article  Google Scholar 

  42. Bernstein J. Not the last word: choosing wisely. Clin Orthop Relat Res. 2015;473(10):3091–7.

    Article  PubMed  PubMed Central  Google Scholar 

  43. Lurie JD, Bell JE, Weinstein J. What rate of utilization is appropriate in musculoskeletal care? Clin Orthop Relat Res. 2009;467(10):2506–11.

    Article  PubMed  PubMed Central  Google Scholar 

  44. Cram P, Landon BE, Matelski J, Ling V, Perruccio AV, Paterson JM et al. Utilization and outcomes for spine surgery in the United States and Canada. Spine (Phila Pa 1976). 2019/07/02. 2019;44(19):1371–80.

  45. Austin DC, Torchia MT, Lurie JD, Jevsevar DS, Bell JE. Identifying regional characteristics influencing variation in the utilization of rotator cuff repair in the United States. J Shoulder Elb Surg. 2019;28(8):1568–77. 2019/04/09.

    Article  Google Scholar 

  46. Gombar Callahan A, Califf R, Harrington R, Shah NS. It is time to learn from patients like mine. Digit Med. 2019;2:16.

    Article  Google Scholar 

  47. ISO 9241 – 210. 2019 - Ergonomics of human-system interaction — Part 210: Human-centred design for interactive systems [Internet]. [cited 2023 May 30]. Available from: https://www.iso.org/standard/77520.html

  48. Carayon P, Hose BZ, Wooldridge A, Brazelton TB, Dean SM, Eithun BL et al. Human-centered design of team health IT for pediatric trauma care transitions. Int J Med Inf. 2022;162.

  49. Lai AG, Chang WH, Parisinos CA, Katsoulis M, Blackburn RM, Shah AD, et al. An informatics consult approach for generating clinical evidence for treatment decisions. BMC Med Inf Decis Mak. 2021;21(1):1–14.

    Google Scholar 

  50. Ventres W, Kooienga S, Vuckovic N, Marlin R, Nygren P, Stewart V, Physicians. Patients, and the Electronic Health Record: An Ethnographic Analysis. The Annals of Family Medicine [Internet]. 2006 Mar 1 [cited 2024 Apr 21];4(2):124–31. Available from: https://www.annfammed.org/content/4/2/124

  51. Grando A, Manataki A, Furniss SK, Duncan B, Solomon A, Kaufman D et al. Multi-Method Study of Electronic Health Records Workflows. AMIA Annual Symposium Proceedings [Internet]. 2018 [cited 2024 Apr 21];2018:498. Available from: /pmc/articles/PMC6371265/

  52. Spall S, Austin SF. Peer Debriefing in Qualitative Research: Emerging Operational Models. http://dx.doi.org/101177/107780049800400208. 1998;4(2):280–92.

  53. Bangor A, Kortum PT, Miller JT. An Empirical Evaluation of the System Usability Scale. International Journal of Human–Computer Interaction [Internet]. 2008 Jul 30 [cited 2023 Feb 16];24(6):574–94. Available from: https://www.tandfonline.com/doi/abs/10.1080/10447310802205776

  54. Bangor A, Kortum P, Miller J. Determining what Individual SUS scores Mean: adding an adjective rating scale. J Usability Stud. 2009;4:114–23.

    Google Scholar 

  55. Newton KM, Peissig PL, Kho AN, Bielinski SJ, Berg RL, Choudhary V, et al. Validation of electronic medical record-based phenotyping algorithms: results and lessons learned from the eMERGE network. J Am Med Inf Assoc. 2013;20(e1):147–54.

    Article  Google Scholar 

  56. Ng K, Kartoun U, Stavropoulos H, Zambrano JA, Tang PC. Personalized treatment options for chronic diseases using precision cohort analytics. Scientific Reports 2021 11:1. 2021;11(1):1–13.

  57. Lee J, Maslove DM, Dubin JA. Personalized mortality prediction driven by electronic medical data and a patient similarity metric. PLoS One. 2015/05/16. 2015;10(5):e0127428.

  58. Hsa F, Tan NC, Tan WY, Oei RW, Lee ML, Hsu W. Patient similarity analytics for explainable clinical risk prediction. BMC Med Inf Decis Mak. 2021;21:1.

    Google Scholar 

  59. Pearson H. Science and intuition: do both have a place in clinical decision making? Br J Nurs. 2013/03/02. 2013;22(4):212–5.

  60. Jayakumar P, Moore MLG, Bozic KJ. Value-based Healthcare: can Artificial Intelligence provide value in orthopaedic surgery? Clin Orthop Relat Res. 2019;477(8):1777–80.

    Article  PubMed  PubMed Central  Google Scholar 

Download references

Acknowledgements

Not applicable.

Funding

Open access funding provided by the Carolinas Consortium.

This work was partially supported by P20 GM121342 and CERortho internal funding. The funding sources had no role in the design, collection, analysis, or interpretation of the data, or the decision to submit the manuscript for publication. The authors declare no conflict of interest in this study.

Author information

Authors and Affiliations

  1. Department of Health Services Administration, School of Health Professions, University of Alabama at Birmingham, Birmingham, AL, USA

    Akanksha Singh

  2. Department of Public Health Sciences, Clemson University, Clemson, SC, USA

    Sydney Lindros & Sarah B. Floyd

  3. Department of Electrical and Computer Engineering, Ira A. Fulton College of Engineering, Brigham Young University, Provo, UT, USA

    Benjamin Schooley

  4. University of South Carolina School of Medicine Greenville, Greenville, SC, USA

    John Mobley

  5. Department of Health Services Policy and Management, University of South Carolina, Arnold School of Public Health, Columbia, SC, USA

    John M. Brooks

  6. Department of Exercise Science, University of South Carolina, Arnold School of Public Health, Columbia, SC, USA

    Patrick Mobley

  7. Center for Effectiveness Research in Orthopaedics, Greenville, SC, USA

    Akanksha Singh, Benjamin Schooley, John Mobley, Patrick Mobley, John M. Brooks & Sarah B. Floyd

Authors
  1. Akanksha Singh
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Benjamin Schooley
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. John Mobley
    View author publications

    You can also search for this author inPubMed Google Scholar

  4. Patrick Mobley
    View author publications

    You can also search for this author inPubMed Google Scholar

  5. Sydney Lindros
    View author publications

    You can also search for this author inPubMed Google Scholar

  6. John M. Brooks
    View author publications

    You can also search for this author inPubMed Google Scholar

  7. Sarah B. Floyd
    View author publications

    You can also search for this author inPubMed Google Scholar

Contributions

AS, BS, SL, JB and SF all made substantial contributions to the conception of the work, and the analysis and interpretation of the data. JM, PM and SL all made substantial contributions to the data collection and analysis. All authors read, substantially modified, and approved the final manuscript prior to submission.

Corresponding author

Correspondence to Sarah B. Floyd.

Ethics declarations

Ethics approval and consent to participate

This final study protocol was reviewed and approved by the Institutional Review Board at Prisma Health. The IRB waived the requirement of signed informed consent, but physician participants were provided an information sheet about the study and were allowed to decline enrollment in the I-C-IT Evaluation. Their verbal agreement to continue with the interview served as consent to participate. All methods were carried out in accordance with relevant guidelines and regulations.

Consent for publication

Not applicable.

Competing interests

The authors declare no competing interests.

Conflict of interest

None to disclose.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary Material 1

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Singh, A., Schooley, B., Mobley, J. et al. Human-centered design of a health recommender system for orthopaedic shoulder treatment. BMC Med Inform Decis Mak 25, 17 (2025). https://doiorg.publicaciones.saludcastillayleon.es/10.1186/s12911-025-02850-x

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doiorg.publicaciones.saludcastillayleon.es/10.1186/s12911-025-02850-x

Keywords

BMC Medical Informatics and Decision Making

ISSN: 1472-6947

Contact us