Combining theory and usability testing to inform optimization and implementation of an online primary care depression management tool

Abstract

Background

The ‘Ottawa Depression Algorithm’ is an evidence-based online tool developed to support primary care professionals care for adults with depression. Uptake of such tools require provider behaviour change. Identifying issues which may impact use of an innovation in routine practice (i.e. barriers to and enablers of behaviour change) informs the selection of implementation strategies that can be deployed with the tool to support use. However, established theory-informed barriers/enablers assessment methods may be less well suited to identifying issues with tool usability. User testing methods can help to determine whether the tool itself is optimally designed. We aimed to integrate these two methodological approaches to i) identify issues impacting the usability of algorithm; and ii) identify barriers to and enablers of algorithm use in routine practice.

Methods

We conducted semi-structured interviews with primary care professionals in Ottawa, Canada. To evaluate usability, participants used a written patient scenario to work through the algorithm while verbalizing their thoughts (‘Think Aloud’). Participants were then asked about factors influencing algorithm use in routine practice (informed by the Theoretical Domains Framework). We used the codebook approach to thematic analysis to assign statements to pre-specified codes and develop themes pertaining to usability and routine use.

Results

We interviewed 20 professionals from seven practices. Usability issues were summarised within five themes: Optimizing content and flow to align with issues faced in practice, Enhancing the most useful algorithm components, Interactivity of the algorithm and embedded tools, Clarity of presence, purpose, or function of components, and Navigational challenges and functionality of links. Barriers to and enablers of routine use were summarised within five themes: Getting to know the algorithm, Alignment with roles and pathways of influence, Integration with current ways of working, Contexts for use, and Anticipated benefits and concerns about patient communication.

Conclusions

Whilst the Ottawa Depression Algorithm was viewed as a useful tool, specific usability issues and barriers to use were identified. Supplementing a theory-based barriers/enablers assessment with usability testing provided enhanced insights to inform optimization and implementation of this clinical tool. We have provided a methods guide for others who may wish to apply this approach.

Background

Depression is the second leading cause of global disability [1]. Primary care settings are often the first point of contact for those experiencing symptoms of depression [2,3,4]. However, challenges to care provision include time constraints, lack of experience or expertise in supporting patients with different levels of depressive severity, and lack of mental health care resources, which can lead to suboptimal care [5,6,7]. The “Ottawa Depression Algorithm” [8], an online tool grounded in clinical guidelines and evidence-based practices [9], was developed to support primary care professionals in providing care for adult patients presenting with symptoms of depression. The algorithm presents an interactive clinical pathway to help with assessment, diagnosis, and treatment according to severity, and contains links to resources and appropriate treatment avenues.

Uptake of the algorithm in routine practice will require primary care professionals to change their typical practice behaviour, in order to integrate algorithm use into their existing workflows and care processes. Implementation science focuses on understanding the best ways to move evidence into practice [10], a key component of which is understanding factors that impede change (barriers) or support change (enablers). Using evidence-supported behaviour change theories to guide barriers and enablers assessments allows us to draw on what is already known about factors which influence behaviour [11]. Barriers and enablers assessments are often informed by the Theoretical Domains Framework (TDF) [12, 13]. Developed from a synthesis of theories, the TDF comprises 14 factors (‘domains’) that influence healthcare professional behaviour. The TDF focuses on both internal (e.g. knowledge, skills, motivation, self-efficacy) and external (e.g. organizational, physical and social) influences and can be used to identify key barriers and enablers which can then be addressed by specific strategies to move evidence into practice via behaviour change [14, 15].

Whilst theory-informed barriers and enablers assessments can identify issues which may impact uptake of an innovation into routine practice, they may be less well-suited to identifying specific issues which impact the usability of a new clinical tool. User-centred design provides a framework for developing products (including new clinical tools) which starts with understanding the end-users of the product and the needs that the product is intended to fulfil [16, 17]. It includes a set of methods for developing products whereby end-users are involved to influence aspects of the design to optimize their interaction with the product and ultimately improve product effectiveness [16, 18]. The set of methods comprise different ways to involve users in the design process [18]. Usability testing is one such method, which “involves hands-on evaluation of the extent to which a product or innovation can be used by specified users to achieve specified goals” [19]. This can help to identify usability issues and as such, usability testing is another approach gaining recognition for the value it can add within the field of implementation science.

Combining these two methodological approaches may be useful in situations where supporting healthcare professional behaviour change to move evidence into practice includes the integration of a new clinical tool. In this study, we present an illustrative example of combining usability testing with a theory-informed barriers and enablers assessment. Our aims were to i) identify issues impacting the usability of the Ottawa Depression Algorithm by primary care professionals; and ii) identify barriers to and enablers of the use the Ottawa Depression Algorithm in routine primary care to support the diagnosis and treatment of adult patients presenting with symptoms of depression.

Methods

Study design

We conducted a qualitative study comprising semi-structured, one-to-one, in person interviews.

Setting

This study took place in primary care practices in the Champlain Local Health Integration Network (now called Home and Community Care Support Services Champlain) in the Province of Ontario, Canada. In Ontario, primary care is publicly funded: permanent residents are insured for medically necessary hospital and physician services through the Ontario Health Insurance Plan, and primary care visits are free at the point of care [20].

Online tool development

The algorithm was designed to be used in primary care to treat and support adult patients with depression at the point of care. It was developed by psychiatrists based at The Ottawa Hospital and the University of Ottawa. Development began by mapping the typical pathway by which patients are diagnosed and treated in primary care through consultation with family physicians. Next, a psychiatry resident integrated recommendations from evidence-based clinical guidelines [1, 9]. Further input was provided by a colleague who developed an online tool to support access to mental health help (www.ementalhealth.ca). Resources were embedded into the pathway to support providers (e.g. diagnostic tools, a process for selecting medication, information about where to refer patients for psychotherapy and other community supports). Primary care colleagues were consulted throughout the development process. The algorithm was initially designed as a PDF document and then transformed into an online tool with the aim of improving accessibility and reach.

Participants and recruitment

Eligible participants were family physicians, residents in Family Medicine specialty training, nurses, nurse practitioners, and administrators working in primary care settings. We purposively recruited participants to ensure a mix of individuals with differing familiarity with the tool and differing levels of experience in primary care. DG (a psychiatrist and developer of the algorithm) emailed existing contacts at local practices with a request to distribute information about the study. NMc emailed individuals who indicated interest to invite them to participate and arrange a time for the interview. Up to two reminders emails were sent. Participants were informed that completion of the interview was taken as implied informed consent to participate. Informed verbal consent was also obtained at the beginning of each interview. As a token of appreciation, participants were offered entry into a prize draw to win one of two $200 gift cards.

We aimed to recruit participants until we achieved data saturation for the theory-based barriers and enablers assessment. We applied the ‘10 + 3 rule’ whereby at least 10 interviews were conducted and analyzed, followed by additional sets of three interviews: when the additional three interviews did not raise any new shared beliefs, we would take this as evidence of saturation [21].

Data collection

An interview guide was developed to facilitate interview processes (Additional File 1). The interview comprised two parts. Part one focused on usability testing. Participants were asked to work through the algorithm while “thinking aloud”. The Think-Aloud method involves participants verbalizing their thoughts while completing a task (in this case, using the algorithm to provide care for a patient with symptoms of depression described in a written scenario) [22, 23]. Initial patient scenarios were drafted by BM and then refined by NMc and checked for clinical realism by DG and CK. The final scenario used (Additional File 2) was presented to participants in two parts, representing an initial consultation and a follow-up visit. Part two of the interview focused on exploring barriers to and enablers of using the algorithm in routine practice. Interview questions were informed by the TDF and associated guidance for its application [12, 13, 24]. The TDF was selected because it is theory-based, focuses on factors which are modifiable and can therefore be addressed with interventions, and because it was developed specifically to support understanding of healthcare provider behaviour. The interview process was piloted with primary care providers and subsequently optimized. Interviews were conducted by NMc in-person and audio-recorded. As the Ottawa Depression Algorithm is available as an online resource, access to a computer and an internet connection were required to access the algorithm during the interview.

Data analysis

Digital audio files were transcribed verbatim by a third party. Transcripts were de-identified and assigned a unique study number. NMc reviewed the accuracy of the transcriptions before proceeding with the analyses. This step also facilitated familiarisation with the data. Transcripts were imported into NVivo (QSR International) qualitative analysis software and analyzed using the codebook approach to thematic analysis [25, 26]. One researcher (NMc) coded the transcripts using a codebook (Additional File 3) which listed codes representing key usability categories [27,28,29,30,31] and the TDF domains [12, 13]. The usability categories and their descriptions were drawn from two sources: descriptions of established methodological approaches for usability testing of medical informatics innovations such as clinical decision support tools (categories such as workflow, content, usefulness, understandability, visibility, and navigation) [27,28,29], and evidence-based guidelines developed to support the design of information-oriented websites (categories such as layout or organization, links, search, graphics, and hardware or software) [30, 31]. These sources were chosen since the algorithm is a clinical decision support tool delivered in a website format. Meaningful units of text within transcripts were assigned to one or more of the usability categories/TDF domains. Coding was discussed with JP and DG as this initial analysis progressed, and refined accordingly. A second researcher (IP) then reviewed and coded the transcripts containing the first researcher’s coding. The two researchers met regularly to discuss areas of disagreement, reach consensus, and update the codebook where appropriate. One researcher (NMc) then reviewed the text coded within each usability category and TDF domain and developed belief statements to represent responses with a similar underlying belief that suggested an influence on the target behaviour (use of the algorithm) [24]. One researcher (NMc) then developed themes across usability categories and TDF domains by reviewing the belief statements and considering how they may be combined to form over-arching patterns of shared meaning which are coherent around a central concept [26]. Themes were refined through discussion with another researcher (JP).

Results

Participants

We interviewed 20 participants from seven practices. Participant characteristics are summarised in Table 1. Half were family physicians, and half were nurse practitioners, residents, or administrators. No new shared beliefs concerning barriers to or enablers of algorithm use were identified in interviews 13, 14, and 15, meeting our definition of data saturation: however, we conducted five more interviews to increase the variation in participant roles. All participants practiced in interdisciplinary team-based models of care. Twelve were not aware of the algorithm, six were aware but hadn’t used it, and two had some experience using it.

Table 1 Participant characteristics (n = 20)

Usability testing

Participants viewed the algorithm and its content positively, thought it was user-friendly, and were enthusiastic about its potential to support them in providing depression care. For example, participants said “I like the quality of the resources” (p19); “I love that when you open things up, it gives you things” (p06); “It’s very comprehensive and user-friendly, you just click button and all the information will show up” (p07); “To actually be able to go through this [medication side effects content] myself or with the patient would be really helpful… otherwise it’s more of an abstract conversation so this is really nice for a lot of my patients. It’s really good actually.” (p15).

However, specific issues impacting usability were identified and coded within the following usability categories: Workflow, Content, Usefulness, Understandability, Completeness, Layout or organisation, Visibility, Navigation, and Links. Key issues were grouped into five themes, described below. Table 2 presents sample quotes for each theme. Usability categories included in the codebook but not represented in these themes are listed in Additional File 4 with a rationale for exclusion. Figure 1 provides example algorithm content to help situate the usability findings.

Table 2 Relevant categories, key usability issues, and sample quotes for five over-arching themes regarding algorithm usability

Fig. 1

Ottawa Depression Algorithm – example content. Panel A The interactive clinical pathway at the core of the algorithm and presented on the homepage. Clicking the boxes with solid borders takes users to further information on the stated topic, as presented in Panels B and C. This version was viewed by study participants and has since been updated. Panel B The patient resources section (formerly labelled the patient education section). This is the current version which was not viewed by participants but is similar in content and structure to the version they viewed. Panel C The medications section. This is the current version which was not viewed by participants but is similar in content and structure to the version they viewed

Optimizing content and flow to align with issues faced in practice

Most participants commented that aspects of the content or flow of operations aligned with the approaches they use in practice. However, some noted differences, or highlighted areas where content could more closely align with issues faced in practice. For example, additional content to support treatment of mild depression (including advice on medications to prevent worsening of symptoms) was suggested. Whilst some were enthusiastic about the ‘email this page’ function, which can be used to send content directly to patients, usability was limited due to misalignment with existing patient communication infrastructures. One key issue was raised with the overall algorithm flow diagram itself: participant queried the direct link between the presence of a complex presentation and recommendation for a psychiatry consultation, with some suggesting softening of the language to more closely align with the nuanced clinical judgment that would be made in practice. Some also noted that there are differences in the ‘level’ or nature of complexity of the listed factors, noting that whilst this is the first section where anxiety appears, anxiety is often considered earlier in the diagnostic process and so should appear earlier in the algorithm.

Enhancing the most useful algorithm components

Specific algorithm components were commonly referred to as being good features, helpful, or useful, namely: the medications section, the patient education section, and the ‘email this page’ function. Participants emphasised the usefulness of the criteria for choosing an antidepressant, and the table listing dosage guidance and side effects information. Having this information in one place was viewed as a great asset of the algorithm, and helpful for comparing side effects profiles to tailor medication choices to patient preferences. Some participants recommended further columns be added to provide information on the average time to impact, or medication cost/coverage through the provincial drug funding system. Ensuring that the embedded links to additional tools outside the algorithm took users directly to those tools, rather than to descriptions of those tools, was also suggested. In a few instances, suggestions for additional tools, information, or functions were made.

Interactivity of the algorithm and embedded tools

Some embedded tools (such as screening/diagnostic questionnaires) were in a static PDF format (needing to be printed, filled out, then scanned back and saved to the patient’s chart), and others were interactive (could be filled out on the computer then saved directly into the chart). Whilst some participants expressed a desire for tools to be interactive, others were happy with the static format. A few participants expected the algorithm to be more interactive or responsive, i.e. taking users through the relevant steps based on their previous responses or inputs, noting that because it is labelled as an algorithm, it should “choose my path for me” (p05). Others noted that the algorithm as designed can be used flexibly: users can dip in and out of it during a consultation as they see fit. One participant noted that it may be difficult to interact with the algorithm during a consultation since it involves switching between the algorithm and the patient’s chart.

Clarity of presence, purpose, or function of components

Many participants noticed that some algorithm boxes are duplicates which contain the same information (i.e. the patient education, medications, and psychotherapy boxes). Whilst this was not viewed as problematic, it took time to realise this. A bigger issue was the lack of clarity regarding which boxes were ‘clickable’ (where clicking would take users to another page containing more detail), and which were not. There were numerous instances of participants attempting to click an ‘un-clickable’ box and waiting for a new page to load. Specific clarification issues were identified within the medications section. Some participants queried the order the medications were listed in, as the rationale for the existing order was not immediately apparent. Some of these participants also expressed a desire for functionality to change the order, which would support the prioritisation of medications depending on priorities regarding side effects. There were also some issues with understandability of abbreviations and shorthand notation. Some participants did not notice the links to more information on switching or augmenting medications, or the print/email functions, as evidenced by their verbalisations requesting these features when visible on-screen. Finally, some queried the purpose and functioning of the ‘email this page’ function, querying what it sends, what/who the sender is, whether/how to send specific collections of items, and the relative appropriateness of the function on a range of different page types.

Navigational challenges and functionality of links

Some participants found it easy to navigate through the algorithm. However, many participants inadvertently closed the algorithm webpage after clicking on and then closing an embedded tool, or a link to an outside resource, not realising that it had opened on the same web browser tab. Some noted it would be better if the tool or resource had opened in a new tab. Others queried how to return to the algorithm after clicking a resource, or how to return to previous pages after moving to different pages. Two pertinent comments were made about embedded links, with one participant noting that clicked links changing colour would help them keep track of what they had done (e.g. patient resources they wanted to check back to), and another identifying a link to a password-protected document.

Barriers to and enablers of algorithm use in day-to-day practice

We identified important barriers/enablers within ten TDF domains: Knowledge, Social/professional role and identity, Social influences, Intention, Goals, Beliefs about consequences, Memory, attention, and decision processes, Behavioural regulation, Environmental context and resources and Nature of the behaviour. Key belief statements coded to these domains were grouped into five themes, described below. Table 3 presents sample quotes for each theme. Domains not represented in these themes are listed in Additional File 4 with a rationale for exclusion.

Table 3 Relevant domains, key belief statements, and sample quotes for five over-arching themes regarding barriers to and enablers of algorithm use in routine practice

Getting to know the algorithm

Most participants were unfamiliar with the algorithm and at least some of the embedded tools. Many acknowledged that they would need to get to know the algorithm before using it directly with patients or consulting it to help support depression care, and find time in their already hectic work or home contexts to learn about algorithm content, establish which components might be most helpful or applicable, and develop the procedural knowledge required to use those components. It was also noted that this algorithm is more ‘involved’ than some others that they use, and that it would help if this was made clear to users up-front. Some participants raised concerns about how up-to-date the algorithm is with respect to the best available evidence regarding depression care, and emphasised that they would need to know more about how the algorithm would be kept up-to-date and by whom.

Alignment with roles and pathways of influence

Most participants agreed that the algorithm could be used by range of primary care professionals. Some noted that because nurses are only involved in specific aspects of depression care, guidance regarding the algorithm components most applicable to their role could encourage uptake in nurses. Despite the algorithm being designed to assist all practitioners regardless of experience level, a few of the more experienced physicians explicitly noted that they did not perceive a need for the algorithm, and that it would be more helpful to less experienced practitioners. Some participants noted that uptake amongst their colleagues or their team more generally could encourage them to use the algorithm. Two nurse practitioners highlighted that the patients they saw were rostered to the physicians they worked with, and the physicians would ultimately influence the extent to which nurse practitioners would use the algorithm. However, one nurse practitioner noted that this would not apply to nurse practitioner-led clinics. Two participants noted that uptake with residents could be a powerful influence on more senior colleagues through the demonstration of how useful or helpful the algorithm is. Finally, a few participants commented that they knew and respected the psychiatrist who led algorithm development, which would encourage them to use it.

Integration with current ways of working

Some participants noted that they did not see any conflicts between the algorithm and other recommendations or evidence-based standards they worked to. However, some noted concerns regarding use of the algorithm and their imperative to maintain patient confidentiality (particularly in relation to the email function and leaving the algorithm open/visible on screen). Many agreed that the algorithm adds to existing guidelines/standards, and could replace other resources currently used. Some noted that they had access to all the resources they needed to use the algorithm (e.g. a computer, an internet connection, printing facilities); however, practice internet connections can be slow, which may hinder use. Despite the potential ease of integration into practice, participants thought that they might forget to use the algorithm during a consultation where it might have helped them. Simple solutions such as saving the algorithm as a favourite in their preferred browser or as an icon on their desktop were described as potentially helpful strategies for integration into practice routines. Most participants felt that integration into their electronic medical record (EMR) would be the key strategy for helping them remember to use it and for enabling them to build it into their workflows. Participants mentioned different possible levels of integration (e.g. an electronic reminder which would link out to the algorithm, or full integration whereby any screening tools or questionnaires completed using the algorithm would be automatically saved into patient charts).

Contexts for use

Most participants intended or wanted to use the algorithm, with some noting that they could see using the algorithm becoming a priority. However, most noted that time constraints were a barrier to use during a consultation. Motivation varied depending on specific algorithm components and factors such as consultation/patient type and perceived need for help. Participants were motivated to use the patient resources and medications sections. Some participants did not feel the need to use the algorithm is situations which could be described as less challenging, i.e. with patients they know well, with stable patients, or with relatively straightforward cases. Participants would be more likely to use the algorithm in more complex situations. Some described this more generally in terms of getting stuck/not knowing what to do next, whilst others gave specific examples (e.g. an initially selected medication has been unsuccessful and guidance on switching or augmenting medications is needed). For some participants, algorithm use would depend on the emotional state of the patients: they would not want to use it when the patient was very emotional and there was the potential to lose rapport or harm the communicative aspects of care. There were differences of opinion regarding using the algorithm in front of patients. As an alternative, some participants described using the algorithm before or after as a source of ideas.

Anticipated benefits and concerns about patient communication

Participants noted numerous potential benefits of using the algorithm. Many noted that use would increase their access to a broader range of resources for supporting patients, whilst also centralising these resources. Whilst some cautioned that the algorithm could have negative impacts if applied without the co-application of clinical judgment, most agreed that incorporating the algorithm into their practice will help them help their patients and ultimately improve care. Many described the potential for standardisation or streamlining of care, noting the potential for the algorithm to improve consistency in the care that patients receive when they see different professionals within a practice, and also when referred out (e.g. to psychiatry services). Relatedly, some noted that once teams were familiar with the algorithm, using it could result in time savings/reduced workloads, achieved through the ease of access to centralised resources, and streamlining processes within care teams (e.g. nurses completing screening assessments and physicians focusing on treatment approaches). Participants also discussed the potential for negative consequences regarding patient communication. Two noted that visible algorithm use may reduce patient confidence in their clinical expertise. Some were concerned about increased screen time and reduced attention on the patient, with some specifically highlighting a tension between algorithm use and their therapeutic role in mental health-focused consultations where active listening and providing support are crucial. However, two participants felt that using the algorithm in a consultation would not negatively impact relationships, whilst three noted that discussing parts of the algorithm with patients could help to increase patient involvement and satisfaction.

Discussion

This study investigated factors influencing the usability and routine uptake of an online algorithm supporting primary care professionals in caring for people with depression. Participants were enthusiastic about using the algorithm, found it easy to use, and viewed specific components as particularly helpful. Participants thought it could be used by those in different roles, could see it replacing other tools due to its centralisation of resources covering the care pathway, and noted the potential to influence standardisation of care. However, there are opportunities to improve alignment with workflows, enhance usefulness, optimize interactivity, enhance clarity, and mitigate challenges with navigation and links. Participants also emphasised their need to get to know algorithm content before incorporating it into their care, identified those in specific roles whose uptake would influence others, acknowledged that they might forget to use the algorithm when it could have been helpful, noted concerns about increased screen time, and felt that integration into their EMR would support routine use. By combining usability testing methods with a behavioural science framework, this study has provided insights to inform both modifications that could be made to algorithm content and functionality, and broader strategies to support implementation of the algorithm in routine practice.

Implications for optimizing algorithm structure, function, and content

Some changes have already been made to the algorithm to address the usability issues identified. To more closely mirror the diagnostic process, discussion of anxiety has been removed from a later step in the algorithm and integrated earlier, and the GAD-7 for anxiety screening has been added [32]. In response to concerns about the direct link between a complex presentation (described as frequent) and recommendation for a psychiatry consultation (described as not always appropriate or required), algorithm language has been modified to prompt consideration of a referral. The appearance of all boxes has been amended to more clearly visually indicate ‘clickability’, with clickable boxes now looking like buttons. Where feasible, embedded links now take users directly to the relevant service website, rather than to a document describing the service. The email function and links to information on medication switching and augmentation are now more noticeable.

Other potential avenues for optimization require additional resources. These include changes to the medications table, viewed as one of the most useful components of the algorithm. Usefulness could be enhanced by providing information on the average time to impact for each medication, adding a column reporting cost/health plan coverage, and allowing users to change the order of medication presentation depending on patient/clinical priorities for minimising specific side effects. Such changes would help to address important information needs and subsequently support shared decision making [33]. A 2½ minute video has been added to guide use of the medications section as it is currently presented.

In some instances, the potential pros and cons of making changes need to be considered. Some participants wanted more guidance for mild depression, and there was specific interest in symptom-focused medication. Guidelines do not recommend medication as a first-line treatment for mild depression [9], and other protocols developed to support depression care have been viewed positively for encouraging the treatment of mild depression without medication [34]. Therefore, such changes would need to be carefully thought through so as not to risk encouraging over-medication and focus specifically on symptom management. Other potential changes need to be considered from a feasibility perspective. Providing different versions of all embedded tools (i.e. static pdf and interactive formats) requires resources to gather such tools and depends on their availability and requirements for use, since these tools were developed by others. Developing a version of the algorithm which is fully interactive or responds to user inputs would involve significant work to change functionality. In addition, such functionality is less aligned with intended use. The algorithm was designed to be a general resource that can be used flexibly, as opposed to a step-by-step decision support tool, with many participants appreciating this aspect of the design.

Implications for an implementation strategy

The findings provide an evidence base to inform the selection of strategies for encouraging algorithm uptake. The identified lack of knowledge about the algorithm suggests that strategies to raise awareness combined with focused education and training could be a good place to start. Awareness-raising can involve presentations at conferences/meetings, and emailing information about the algorithm to relevant society member lists [34]. Education and training could be operationalised in multiple ways depending on resources available and intended reach. Locally-focused activities could involve interactive workshops embedded within team meetings, whilst instructional videos could be developed and disseminated to reach a broader audience. It may be important to embed instructions on how to use specific components of the algorithm, offer demonstrations of use, and provide opportunities for practice or rehearsal [35]. Various formats could be considered, such as written descriptions, videos of mock interactions with patients, live observations, written scenarios to support individual practice, or role play activities. Since we identified context-specific intention to use the algorithm, demonstrations and opportunities for practice could incorporate various examples of situations it can be used in (e.g. using the algorithm with a patient who is stable; when a patient is very upset, reviewing the algorithm after the patient has left). Whilst this may support algorithm uptake, the range of other factors influencing use indicates that additional strategies would also be necessary, such as focusing on social influence processes, clarifying how those in different roles can use the algorithm in accordance with their scope of practice, and/or embedding reminders at the point of intended use.

Combining usability testing and behavioural theory-informed barriers/enablers assessment

We combined two methodological approaches to provide insights on two distinct but interrelated concepts: factors impacting tool usability, and factors impacting tool use in routine practice. This has resulted in a more comprehensive investigation than would have been achieved if we had only included one component or the other. Assessing barriers to and enablers of implementation is a key component of many implementation process models [15, 36, 37] and is often done using behavioural theory-informed frameworks which focus on identifying factors influencing the target behaviour in routine practice. In instances such as this where a new tool is being implemented, our findings indicate that supplementing this traditional approach with an element of usability testing can provide additional insights. Whilst the findings related to usability allow us to propose changes to the tool itself, the findings from behaviour change theory-informed interviews allow us to propose strategies to encourage uptake of the tool, both of which should ultimately support integration of the tool in day-to-day practice.

Whilst we did not conduct a systematic assessment of conceptual overlap in our coding and theme generation between the two data sources, our coding indicates the usability categories and TDF domains for which some overlap can occur. These are: Workflow and Environmental context and resources (lack of time to use the algorithm); Workflow and Nature of the behaviour (discussion of use in specific situations); Content and Knowledge (familiarity with the resources included in the algorithm); Usefulness and Beliefs about consequences (comments on the increased access to resources); and Hardware & software and Environmental context and resources (slow internet connection hampering use). Since these issues relate to use in everyday practice with likely solutions being strategies to support uptake rather than changes to the algorithm, we felt they were best represented in the TDF section. Other usability issues (Understandability, Completeness, Layout & organization, Visibility, Navigation, and Links) did not overlap with content coded to the TDF domains, which further support the added value of usability testing.

Other researchers have looked at barriers to implementation and usefulness and usability issues in the same study as part of tool development [38, 39]. However, these studies have some key differences with ours. Anderson and colleagues [38] recruited one group of clinicians to firstly watch a video demonstration of the tool and then provide feedback on implementation barriers and enablers, and a different group of clinicians to think aloud while using the tool in a simulated environment. Coleman and colleagues [39] randomised clinicians to either a think-aloud interview to identify specific usability issues or a focus group to identify general usability issues, for which they provide implementation as an example. Whilst these previous approaches provide useful examples, a unique contribution of our work is that we have built our approach on pre-existing frameworks which outline a range of usability issues [27,28,29,30,31] and factors influencing behaviour [12, 13], which may help to increase comprehensiveness. Our work may therefore provide a useful example for those conducting behaviour change theory-based barriers assessments on how they can integrate different methodological approaches to potentially broaden the insights they obtain. In Fig. 2, we provide a methods guide for others who may wish to supplement barriers assessments with usability testing.

Fig. 2

Combining theory-based barriers assessments and usability testing to inform optimization and implementation of clinical tools: methods guide

Strengths and limitations

Through an innovative combination of methodological approaches, this project has identified usability issues with, and barriers to, uptake of an evidence-informed clinical tool. This work has already guided efforts to improve the tool and can now be used to inform an implementation strategy designed to enhance uptake. Our approach may be informative for those looking to implement similar tools in healthcare contexts. All participants practiced in interdisciplinary team-based environments and so our findings may not be generalizable to settings involving different models of care. Whilst we used written scenarios during think-aloud testing, it is possible that realism may be enhanced and/or different usability issues may be identified using different methods such as involving patient actors [38] or using different forms of usability testing altogether such as near-live clinical simulations [29]. However, participant responses during the think-aloud tasks did seem to represent a description of their approach had the patient been real, and participants did not appear to have any difficulties with identifying usability issues during this portion of the interview. Whilst usability testing forms an important part of User-Centred Design approaches, which are increasingly being advocated for in implementation science [19], we did not have the capacity to conduct a User-Centred Design study, wherein multiple rounds of testing is required to use these approaches effectively [19]. Whilst others have integrated behaviour change theory into multi-round User-Centred Design approaches [40], our approach is built on a multi-theory framework which identifies a broad range of behavioural influences and so may increase the comprehensiveness of barriers identified whilst also providing an example of a method for enhancing the informativeness of behaviour change theory-based barriers investigations.

Conclusions

This study identified a range of usability issues and barriers to use of an online algorithm designed to support primary care professionals in caring for people with depression. The findings have informed some initial changes to the algorithm designed to enhance usability, and will also be reflected on to inform subsequent updates. The identified barriers to use indicate that implementation strategies focusing on awareness-raising about algorithm existence, education and training, social influence processes, and changing the physical environment may be best placed to enhance uptake. This study serves as an example of methods for combining two methodological approaches and integrating existing frameworks to broaden the insights obtained from behaviour change theory-based barriers assessments which involve implementation of a new tool.