A heuristic evaluation of a pharmacy surveillance information system

Abstract

Introduction

The pharmacy surveillance information system (PSIS) is intended to manage the dispensing practice of under-controlled drugs and substances. We designed and developed a PSIS for the first time in a developing country. This study aimed to evaluate the usability of this system using a heuristic evaluation method before the pilot implementation in outpatient pharmacies.

Materials and methods

The study was conducted in 2022 during the development of a pharmacy surveillance information system. Five evaluators examined the system using Nielson’s heuristic evaluation method. The detected usability problems were categorized into 10 Nielson’s usability principles, and their severity was calculated.

Results

In total, 91 unique usability problems were identified. The most detected usability problems were minor (60%). The “consistency and standard” (31%), “aesthetic and minimalist design” (28%), and “match between system and the real world” (12%) were the most frequent problems. Also, the “flexibility and efficiency of use” (mean = 2.9), “error prevention” (2.85), and “user control and freedom” (2.8) were the most severe problems.

Conclusion

The study has identified the most common and severe usability issues of an information system. It is important for the system developers to address these issues as it can significantly improve users’ trust and satisfaction. Therefore, all the identified usability problems were resolved before the system was implemented.

Introduction

Several factors contribute to an increased risk of dispensing error rate (DER) in pharmacies [1]. The use of automated information systems is recommended to mitigate errors and enhance dispensing efficiency [1, 2]. The usability of these systems is crucial due to the existence of multiple pharmacy information systems in the pharmacies and the heavy workload of pharmacists. Usability refers to the ability of a product to be effectively and efficiently used by specified users to achieve specified goals in a particular context [3]. A lack of usability in a pharmacy information system can lead to an increase in DER and a decrease in the quality of pharmacy services. This can potentially compromise patient safety and the overall effectiveness of healthcare delivery [4].

Neglecting to evaluate the usability of information systems (ISs) prior to implementation can result in users’ confusion, exhaustion, errors, and reduced satisfaction. This dissatisfaction can lead to the abandonment of the system and impose significant costs on system re-design [5]. Therefore, the usability evaluation of a pharmacy information system prior to the implementation phase is necessary to ensure successful adoption and optimal use of the system.

There are various methods to evaluate the usability of ISs, including expert-based and user-based methods. Expert-based methods involve usability specialists examining the system based on established guidelines, while user-based methods involve actual end-users interacting with the system to identify issues. These methods are used in the different steps of the system development life cycle (SDLC) [6, 7]. The expert-based methods, such as heuristic evaluations (HEs), are less expensive than user-based ones, can be applied to assess system prototypes, and are, therefore, more easily used in the early steps of SDLC [6,7,8]. This usability evaluation technique is related to the inspection methods conducted by experts [6, 7]. Three to five evaluators can identify up to 75% of usability problems in the HE method [7, 9, 10].

The HE can discover important usability problems in the user interface (UI) design efficiently and cost-effectively [11]. Thus, to identify the usability problems and success in system implementation, using HE can be particularly helpful in situations where access to a wide range of end-users is limited. In the HE, evaluators assess the UI based on a list of predetermined usability standard principles and then recognize the violations as usability problems [11, 12]. This method is a quick way to identify usability problems at a low cost [13,14,15]. Jaspers and Khajouei et al. [6, 16] suggested that individuals with experience in health information systems are well-suited to perform HEs in this context, as their familiarity with such systems can enhance the evaluation process.

Prior to this study, there was no information system in Iran to monitor the dispensing practices of under-controlled drugs and substances from outpatient pharmacies [17]. This gap in the healthcare system potentially increases the risk of medication errors and makes it difficult to ensure regulatory compliance. To address this critical need, we designed and developed a pharmacy surveillance information system (PSIS) to manage the dispensing practice of under-controlled drugs and substances based on user-centered design methodology. The implementation of this system is expected to improve the safety and efficiency of pharmacy practices in Iran, particularly in the handling of controlled substances.

Given the critical nature of this system and its potential impact on healthcare delivery, it is crucial to evaluate its usability before pilot implementation. This study aims to conduct a heuristic evaluation of the PSIS, identifying potential usability issues that could hinder its effectiveness or lead to user errors. By addressing these issues early in the development process, we can ensure a more successful implementation, leading to improved pharmacy practices and, ultimately, better patient outcomes.

Materials and methods

Setting and the evaluated information system

The study was conducted in 2022 in Iran. The Food and Drug Administration (FDA) of Kashan University of Medical Sciences (KaUMS) developed a PSIS (SUPMED) to monitor the dispensing practice of under-controlled drugs and substances to patients. During this project, we planned to implement the SUPMED in the outpatient pharmacies affiliated with KaUMS for the first time. The SUPMED is the first PSIS that will be implemented in Iran, with unique functionalities determined in our previous study [17].

Before the implementation of SUPMED in 50 pharmacies in Kashan city, we evaluated its usability to identify UI problems and resolve them. This system enables the users to do many actions, including registering new users (e.g., FDA manager, and pharmacy), defining under-controlled drugs and substances, registering legal prescribers (e.g., physicians), pharmacies, medication companies, generating reports, sending messages, and allocating under-controlled drugs and substances to patients (Fig. 1).

Fig. 1

The screen of SUPMED related to administrator users (low-fidelity prototype)

Evaluation methods and evaluators

We used HE to evaluate the usability of the system. Evaluators were selected based on their expertise in health information systems and experience with HE methods. Five evaluators were invited to evaluate the usability of the SUPMED. The evaluators included Ph.D. candidates of health information management (HIM) with a background in health information technology (HIT) (n = 2), M.Sc. of HIT (n = 1), Ph.D. of medical informatics (MI) with a background in MI (n = 1), and M.Sc. of MI (n = 1). The evaluators had varying levels of experience doing HEs and working with different health ISs (Table 1). All of the evaluators agreed to participate in this study.

Table 1 Evaluators’ information

Data collection

The UI of the IS was evaluated according to the methodology suggested by Nielsen [9,10,11]. The evaluators independently explored the UI without any scenarios and action sequences. Data were collected by a problem list in a Microsoft Word file including a description, location, and screenshot for each problem. The evaluators proposed their recommendations below for each identified usability problem.

Data analysis

The data collection forms were compared in a joint panel with the researcher (as a coordinator) and evaluators. All of the detected problems were added to a main list while the repeated or duplicate problems were removed. The detected problems were discussed by the evaluators and any disagreement resolved by consensus. In the case of no agreement, we considered an identified issue as a usability problem if it was confirmed by at least three evaluators. Then, each evaluator scored each usability problem between zero (no problem) to four (catastrophic problem) to indicate the severity of each problem based on the level of difficulty for end-users by a standardized severity rating template (Supplementary 1). In this regard, the problem severity was determined based on a combination of three factors: the frequency of the problem, the potential impact of the problem on the users, and the persistence of the problem any time the users face a similar condition [16]. Also, the evaluators recommended a practical solution for every problem. Finally, each evaluator categorized the problems based on Nielsen principles. Three out of five evaluators needed to confirm a usability problem located in one of the 10 Nielson principals. In case of disagreement, we discussed each principle to reach a consensus. The mean severity of usability problems was categorized into five categories: no problem (0-0.5), cosmetic problem (0.6–1.5), minor problem (1.6–2.5), major problem (2.6–3.5), and catastrophe problem (3.6-4) [18]. Data were analyzed by SPSS.22 using descriptive statistics.

Results

In total, 91 unique usability problems were identified by HE. The most detected usability problems were minor (60%) and the mean severity of total problems was 2.1 ± 0.42 (Table 2). As shown in Fig. 2, the five top usability problems were related to “consistency and standard” (n = 28, 31%), “aesthetic and minimalist design” (n = 26, 28%), “match between system and the real world” (n = 11, 12%), “error prevention” (n = 8, 9%), and “visibility of system status” (n = 7, 8%). Some examples of heuristic violations are provided in Table 3.

Table 2 The severity of the identified usability problems in HE

Fig. 2

Percentage of usability problems based on Nielsen principles

Table 3 Some examples of usability problems based on Nielsen principles

Figure 3 shows the frequency and mean severity of the problems based on the Nielson principles. The top severe problems were related to “flexibility and efficiency of use” (mean = 2.9), “error prevention” (2.85), “user control and freedom” (2.8), “visibility of system status” (2.67), and “help users recognize, diagnose and recover from error” (2.45). A screen related to some of the identified usability problems is shown in Fig. 4. (And “Figure S1”, “Figure S2”, “Figure S3”, and “Figure S4” in the Supplementary 2).

Fig. 3

Frequency and mean severity of the problems based on the Nielson principles

Fig. 4

A screen with the identified usability problems related to error messages

P: Ambiguous error message R: Display error message in English language in Persian UI S: Lack of user guidance in error message

Discussion

Principal findings

The study showed that most of the usability problems were minor and there was no catastrophic problems identified. The Most usability problems were related to “consistency and standards”, “aesthetic and minimalist design”, and “match between the system and the real world”. The least of the usability problems were related to “help and documentation”, “recognition rather than recall”, and “flexibility and efficiency of use”. However, when examining severity, the most critical problems were identified in “flexibility and efficiency of use”, “error prevention”, and “user control and freedom”.

Comparisons with existing literature

Based on the study, “consistency and standards”, “aesthetic and minimalist design”, and “match between the system and the real world” were the most frequent usability problems identified in the PSIS. Our findings align with previous research evaluating various types of healthcare information systems. The predominance of problems related to “consistency and standards” has been consistently reported in studies evaluating hospital information systems [19,20,21,22] and computerized order system [23] suggesting this is a common challenge across different healthcare information domains. Similarly, our findings regarding “aesthetic and minimalist design” concerns mirror those found in evaluations of emergency and Radiology Information Systems [22, 24], while problems related to “match between the system and the real world” have been documented in study of nursing information system [21]. While most previous studies conducted on well-established commercial healthcare information systems after they had already been implemented, our study evaluated a newly developed PSIS. Despite differences in the systems assessed, the frequent usability problems identified were similar, underscoring that these problems consistently pose challenges in the UIs of a system.

The “consistency and standard” principle has been widely recognized as fundamental to UI design [21]. Consistency refers to the degree to which a UI employs a predictable, coherent pattern in its appearance, behavior, functionality, and terminology. Researches have shown that consistent interfaces help reduce cognitive workload and enhance user trust [25, 26]. Our findings reinforce the importance of maintaining predictable patterns in appearance, behavior, functionality, and terminology across the system.

The “aesthetic and minimalist” principle relates to a style of design focused on eliminating unnecessary elements, simplifying layouts, and promoting visual harmony [27]. This principle seeks to balance form and function, conveying essential information and capabilities to users without distraction or clutter. The literature supports our finding that successful interface design must balance functional requirements with aesthetic considerations [28]. By attending to aesthetics and avoiding irrelevant information or rarely needed features, systems can be made more usable and satisfying through an attractive, streamlined visual experience.

The “match between system and real world” principle, which emerged as a significant concern in our study, states that “the system should speak the users’ language, with words, phrases, and concepts familiar to them, rather than system-oriented terms. It should also follow real-world conventions, making information appear in a natural and logical order” [29]. This principle is well-documented in the literature as crucial for reducing cognitive load [12] and promoting user trust [30]. Our findings reinforce the importance of using familiar terminology and following real-world conventions in system design.

In line with previous studies [20, 22], we found that while “flexibility and efficiency of use” and “user control and freedom” problems were less frequent, they were among the most severe when they occurred. In an optimal system, users should not feel controlled or restricted. Although some problematic interactions may become habitual through repeated use [22], the system developers should ensure users feel empowered and unencumbered when using the system. This minimizes cognitive load and errors while lowering the mental and physical effort needed to complete tasks [20].

Our identification of “error prevention” principle as a critical usability concern is consistent with existing literature [21, 31]. A usable system should aim to prevent error-prone situations and provide clear messaging to help users resolve errors [21, 31]. The solutions implemented in our study to improve error prevention included adding clear labels, removing ambiguous increase/decrease arrows, preventing duplicate entries, and indicating mandatory fields, demonstrate practical applications of error prevention principles documented in previous research. These modifications exemplify ways to enhance this key aspect of usability through design choices that proactively avoid mistakes and guide users to correct them when they occur. Attention to error prevention principles can significantly improve the usability and satisfaction of a system.

Strengths and limitations

A key strength of this study is its integration within the SDLC, allowing for immediate resolution of identified issues before pilot implementation. All of the identified usability problems (ranging from cosmetic to catastrophic problems) were resolved before the pilot implementation. The developing, evaluating, and implementing process of the PSIS is shown at a diagram in the Supplementary 3. However, we acknowledge that the effectiveness of HE depends heavily on evaluator expertise. While our evaluators had strong backgrounds in health information systems and evaluation methods, the absence of pharmacy students or professionals in the evaluation team represents a limitation. Although our health informatics experts were able to identify general usability issues, domain-specific insights from pharmacy practitioners might have revealed additional workflow-related concerns or domain-specific usability challenges. In addition, our evaluators exhibited a gender imbalance. The HE focuses on objective usability criteria, so we don’t expect this gender imbalance to significantly affect our results. However, a more balanced representation could provide a broader user interface perspective. Future research would benefit from incorporating front-line user participation and considering dynamic workflow interactions. We recommend additional studies employing both expert-based and user-based methods to generate more generalizable findings.

Implication of the research

In most developing countries, low attention is paid to non-functional requirements, especially usability attributes when designing a health information system. Usability evaluation in the design phase of SDLC may resolve many crucial problems and help the successful implementation of a system. The HE is one of the good methods for obtaining a usable information system. This method points to key aspects for improvement and reduction of usability problems and their severity.

Conclusion

This study identified common and critical usability problems in a healthcare information system. While some of these problems may not disturb users or significantly impact real-world use, they were deemed addressable by evaluators that could undermine user interaction with the system. Thus, attention to system usability and resolving identified problems is important overall. The study demonstrates the effectiveness of heuristic evaluation in identifying usability issues, while also highlighting the potential value of employing multiple evaluation methods for a more comprehensive assessment. Future research should focus on comparing different usability evaluation techniques and their effectiveness in identifying various types of usability issues. The findings emphasize the importance of incorporating usability evaluation early in system development to ensure successful implementation and user adoption.