Design Approaches for Developing Quality Checklists in Healthcare Organizations: A Scoping Review ================================================================================================= * Elizabeth Kwong * Amy Cole * Dorothy Sippo * Fei Yu * Karthik Adapa * Christopher M. Shea * Carlton Moore * Shiva Das * Lukasz Mazur ## Abstract Quality checklists have demonstrated benefits in healthcare and other high-reliability organizations, but there remains a gap in the understanding of design approaches and levels of stakeholder engagement in the development of these quality checklists. This scoping review aims to synthesize the current knowledge base regarding the use of various design approaches for developing quality checklists in healthcare. Secondary objectives are to explore theoretical frameworks, design principles, stakeholder involvement and engagement, and characteristics of the design methods used for developing quality checklists. The review followed the Preferred Reporting Items for Systematic Reviews 2020 checklist. Seven databases (PubMed, APA PsycInfo, CINAHL, Embase, Scopus, ACM Digital Library, and IEEE Xplore) were searched for studies using a comprehensive search strategy developed in collaboration with a health sciences librarian. Search terms included “checklist” and “user-centered design” and their related terms. The IAP2 Spectrum of Participation Framework was used to categorize studies by level of stakeholder engagement during data extraction. Twenty-nine studies met the inclusion criteria for this review. Twenty-three distinct design methods were identified that were predominantly non-collaborative in nature (e.g., interviews, surveys, and other methods that involved only one researcher and one participant at a given time). Analysis of the levels of stakeholder engagement revealed a gap in studies that empowered their stakeholders in the quality checklist design process. Highly effective, clear, and standardized methodology are needed for the design of quality checklists. Future work needs to explore how stakeholders can be empowered in the design process, and how different levels of stakeholder engagement might impact implementation outcomes. Keywords * design methods * quality checklist * safety checklist * healthcare * scoping review ## Introduction ### Quality Checklists in Healthcare Researchers have used “quality checklist” or “safety checklist” interchangeably to refer to the safety and quality management checklists that outline current evidence-based practices, serve as mnemonic devices, focus on evaluation or performance measurement, assist in maintaining or improving the safety of an organization, or take the form of a cognitive aid or goal sheet (1). A safety and quality management checklist (hereafter referred to as “quality checklist”) is an algorithmic listing of actions to verify if an action has taken place and is used to manage and control the quality of deliverables and ensure no step is forgotten (2,3). Quality checklists have been used informally across various industries, with their widespread adoption following a notable aircraft incident in 1935. In this incident, two expert pilots were killed, and engineers on board were injured when the gust-lock was not released prior to takeoff, rendering the elevators inoperable (4,5). After the incident, Boeing developed a series of checklists for the pilots to ensure critical tasks like this were completed. Since then, checklists have become essential for aviation regulation and safety, used by pilots consistently and mandated by the Federal Aviation Administration (FAA) and other regulators internationally. This practice was subsequently adopted more broadly by the military (4,5). In healthcare, the use of checklists was influenced by the Michigan Health and Hospital Association (MHA) Keystone Center for Patient Safety and Quality Keystone ICU project conducted between 2003 and 2005. During this project, a checklist was used to ensure adherence to evidence-based, infection-control practices, successfully reducing the risk of central line-associated bloodstream infections in intensive care unit patients (6). The development and implementation of the World Health Organization (WHO) surgical safety checklist in 2007-2008 further promoted the use of checklists in healthcare. This 19-item checklist sought to reduce medical errors and adverse events during surgery while improving the consistency of care (5,7). Subsequent healthcare research has explored the development and application of checklists in various areas, including inpatient care (8), obstetrics (9), hospital discharge (10,11), chemotherapy treatment (12), COVID-19 prevention and control (13,14), and other care areas and procedures. Checklists in healthcare have demonstrated several benefits: they serve as memory aids, ameliorating the effects of fatigue, stress, and distraction, standardize task performance, ensure adherence to best practices, and promote team communication (5,15). Notably, checklists in healthcare have also been crucial for safety management, the improvement of care processes, and the reduction of mortality and morbidity (16–18). However, despite these benefits, barriers to effective design, adoption, and implementation quality checklist design persist. These barriers include slow development and adoption, inconsistent use, and a lack of effective, standardized methodology for quality checklist development, despite research and evidence highlighting the benefits of checklists (1,19). ### Design of Quality Checklists Although the concept of “design” has existed before the Industrial Revolution, the emergence and study of design methods originated in the 1950s and 1960s with the application of novel and “scientific” methods to problem-solving after World War II and the recognition of increased complexity in industrial products (20,21). Since then, design methods have evolved with the publication of books and articles on design methods across various industries, as well as the introduction of artificial intelligence and automation. Influential design researchers including Horst Rittel, Nigel Cross, Herbert Simon, Don Norman, and more along with design organizations such as IDEO have been credited with transforming the field and introducing concepts such as design thinking, solutions-focused problem-solving, user-centered design, and human-centered design (22). These design methods and concepts have been adopted and adapted in engineering, architecture, aviation, technology, product design, and healthcare for a variety of tools, systems, and purposes, including quality checklists. Several studies have reported considerations, recommendations, and frameworks for the design of quality checklists, which individual organizations can adapt to their respective procedures and processes (1,23–26). These studies recognize that the content and format of quality checklists depend on their specific context, so they often provide only broad guidelines or basic methods. Nevertheless, some established methods for designing checklists include literature reviews, focus groups, Delphi consensus, task analyses, heuristic evaluation, interviews, surveys, and personal experience (19,23). Guidance for the design of quality checklists often involves a review of existing literature, consideration of the user skills, and the experience, context, systems, and environment in which the checklist is intended to be used (1,19). These guidelines advocate for the involvement of potential users and stakeholders in the design process or design team, but do not provide detailed instruction on how to engage stakeholders effectively. Stakeholder engagement, or the process of incorporating the stakeholder in the design and development process, is being increasingly used and promoted in health research (27,28), but is incompletely described in the design and development of quality checklists in healthcare. While gathering input and insights from stakeholders has demonstrated impact, it also presents challenges, leading to literature-based recommendations on how to best engage stakeholders (25,27–30). Existing studies that have utilized varying levels of stakeholder engagement to design quality checklists acknowledge the importance of stakeholder involvement. However, more research is needed to elucidate the gaps and challenges with user engagement specifically in quality checklists design and enable researchers to develop, evaluate, and implement quality checklists more effectively in their own organizations. While the utilization of quality checklists has been documented in various settings, there is a shortage of published guidance, rigorous design methodology, and understanding of stakeholder engagement for quality checklists, particularly in healthcare (1). Despite growing evidence and various methods for quality checklist design, a lack of a highly effective, standardized methodology for the design of quality checklists in healthcare and medicine has contributed to inconsistent use, adoption barriers, and gaps in implementation (1,5). Due to the nature and purpose of quality checklists, these issues, if not addressed, may negatively impact stakeholder satisfaction, usability, quality of care and safety. Therefore, this scoping review aims to synthesize design approaches in the development of quality checklists in healthcare organizations to better understand gaps in design methodology and stakeholder engagement that can be addressed in future studies and ultimately improve quality of care. ### Aim and Research Questions In this review, we scope and synthesize the current state of knowledge regarding the design approaches and associated levels of stakeholder engagement for developing quality checklists in healthcare organizations. More specifically, our review addresses the following questions: 1. What are the characteristics of the design methods adopted to develop quality checklists in healthcare organizations, and how were the design methods defined and measured? 2. What theoretical frameworks and design principles were used to develop quality checklists in healthcare organizations? 3. Who is involved in the development of quality checklists in healthcare organizations, and what is the level of stakeholder engagement in the process? 4. What knowledge gaps exist in the literature in the utilization of design approaches to develop quality checklists in healthcare organizations? In this study, a quality checklist is defined as a list of items to be performed to complete a task to verify if an action has taken place (or not), and gives information with regard to quality assurance activities and helps control the quality of deliverables (2,3). This scoping review contributes to the growing literature examining how stakeholder engagement affects the design and development of quality checklists, adds to the existing knowledge base of design approaches and frameworks, and highlights gaps and challenges for future research. ## Methods ### Data Sources and Search Strategy A comprehensive search strategy was developed in collaboration with a health sciences librarian and executed in seven databases were included in the scoping review and searched on May 3, 2023, including PubMed, APA PsycInfo, CINAHL, Embase, Scopus, ACM Digital Library, and IEEE Xplore. The search strategy consists of relevant search terms identified based on our research aim and research questions. For example, our search terms included “checklist,” its term variations, and related terms (e.g., “cognitive aid”, “care pathway”, “clinical pathway”, “care map”, critical path”, “clinical path”) in combination with “user-centered design” and related terms (“community-based participatory research”, “human-centered design”, “co-create”, “co-created”, “co-creation”, “co-design”, “user centered design”, “participatory design”, “design thinking”, “rapid prototyping”, “rapid design”, “participatory research”, “community-based research”, “community based research”, “action research”, “design sprint”, “agile design”, “agile method”, “agile methodology”, “agile project management”, “scrum framework”, and “agile” + “scrum”). Following the search, retrieved citations were exported to SciWheel (reference management tool; [sciwheel.com](https://sciwheel.com)) and then uploaded into Covidence (systematic review management tool; [covidence.org](https://covidence.org)) for manual screening. Both SciWheel and Covidence removed duplicated citations from the aggregated search results. ### Eligibility Criteria Eligibility criteria was developed according to the PICOT Framework (31). To be eligible for inclusion, studies had to be an empirical study involving or describing the design, development, or evaluation of a quality checklist in a healthcare organization (e.g., hospital, clinic, cancer center). Studies needed to involve a safety and quality management checklist of any format (e.g., paper or digital) as the intervention but could involve any comparator or outcome. Studies could be from any publishing year or a healthcare organization in any state, country, or location. We excluded studies that did not provide information on the design, development, or evaluation of the quality checklist intervention, as well as those that only used the checklist as part of another study or to assess another intervention. Additionally, studies published in a language other than English were excluded. Table 1 provides details on the inclusion and exclusion criteria that were used to screen search results. View this table: [Table 1.](http://medrxiv.org/content/early/2024/09/28/2024.09.27.24314468/T1) Table 1. Inclusion and exclusion criteria. ### Study Selection Two investigators (EK and AC) independently screened titles and abstracts against the inclusion and exclusion criteria. Discrepancies were resolved by a third investigator (LM). Two investigators (EK and AC) then independently screened full-text articles for inclusion, with discrepancies resolved by a third investigator (LM). ### Data Extraction and Synthesis Data extraction was performed by one investigator (EK) by using the data extraction template provided in Covidence. After customizing the template based on our research questions, it was pilot tested with three included studies and revised to ensure all relevant data were captured. For each study, the following information was extracted: DOI, title, lead author and their contact details, year, country in which the study was conducted, healthcare organization type, study aim, study design (i.e., the framework of research methods to conduct the study), start date, end date, study funding sources, possible conflicts of interest for study authors, checklist type, theoretical frameworks or design principles used, level(s) of stakeholder engagement used, characteristics of design methods used, implementation methods or frameworks used (if applicable), population description, inclusion criteria, exclusion criteria, method of participant recruitment, total number of participants, baseline population characteristics, intervention characteristics, outcomes (e.g., mean, standard deviation, p-value), study strengths, and study limitations. Extracted data were synthesized to address each research question. Specifically, the IAP2 Spectrum of Participation framework and Vaughn’s definitions for levels of participation based on the IAP2 Spectrum of Participation framework (32,33) was used to guide data synthesis for Research Question 3 (Who is involved in the development of quality checklists in healthcare organizations, and what is the level of stakeholder engagement in the process?), classify the level of stakeholder engagement in each study, and qualify what level of participation was used by each study. Stakeholder engagement was defined as one of five levels of engagement according to this framework: * Inform: Provide stakeholders with balanced and objective information * Consult: Obtain feedback from stakeholders on analysis, issues, decisions, etc. * Involve: Work with stakeholders to make sure concern and aspirations are considered and understood * Collaborate: Partner with stakeholders in each aspect of the decision-making * Empower: Place final decision making in the hands of the stakeholders ### Quality Assessment While the scoping review was not focused on reported outcomes and may not have required a quality assessment, we utilized the Agency for Healthcare Research and Quality (AHRQ) Evidence-Based Practice Center (EPC) approach to interpret levels of evidence for each study (34). Using this approach, one investigator (EK) performed a quality assessment of each study within the Covidence system and a second investigator (AC) validated the quality assessment. Studies were assessed using the five EPC domains: study limitations, consistency, directness, precision, and reporting bias. ## Results ### Study Selection The database search yielded 2551 records, of which 518 were duplicates. After excluding the duplicate articles, a total of 2033 records were identified for screening. After initial screening of article titles and abstracts, 1940 records were excluded, leaving 93 articles for full-text retrieval and review. After full-text screening, 32 articles were retained and synthesized. These 32 articles reported on 29 studies (6 articles reported on 3 studies, i.e., 2 articles were published about 1 study with different cuts of information or data presented, and were combined into 1 record, for 3 different studies). Inter-rater reliability statistics were calculated for the title and abstract review stage and the full text review stage, generating a Cohen’s Kappa of 0.632 and 0.745, respectively. The article selection and review process is detailed in the PRISMA flow diagram (35) in Fig 1.  [Fig 1.](http://medrxiv.org/content/early/2024/09/28/2024.09.27.24314468/F1) Fig 1. PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) flow diagram. Modified from Page *et al.*, 2021 (35). ### Study Quality Assessment Included studies had a moderate strength of evidence overall, as they were observational in nature and not randomized controlled trials. The majority of studies (86%) were scored as “direct” (study has evidence that links interventions directly to a health outcome); remaining studies did not have measurable outcome data, intermediate data, or were only available for proxy respondents. One study (3%) (36) provided details of its power analysis, including effect size calculations and optimal information size, demonstrating a “consistent” level (same direction or similar magnitude of effect) and “precise” level (degree of certainty to reach clinically useful conclusion) for consistency and precision domains, respectively. Seventeen of the 29 studies (59%) had “low” levels of study limitations, indicating low protection against bias. These limitations predominantly resulted from small sample size and studies being conducted at one site or within one group. Reporting bias was rated as “suspected” for 66% of studies and “undetected” for 34% of studies. The “suspected” reporting bias resulted from incomplete reporting of the full study, with only a portion of the outcomes disclosed, and limitations to tested tasks and outcomes. ### Characteristics of Included Studies Among the 29 studies identified, 34% were conducted in the United States; studies are representative of 11 different countries or regions. Studies span approximately 23 years from 2001 to 2023. Studies had between 4 and 138 participants, with a mean of 29 participants (Note: one study (37) did not provide a population number). Recruitment of participants included voluntary recruitment, convenience sampling, snowball approaches, word of mouth, social media and regular media (e.g., posters and recruitment flyers), email, and phone. Several studies cited the design, development, or evaluation of the checklist as part of a quality improvement project in the organization or system. Additionally, the majority of the study designs used were pre-and post-studies (59%). While not a primary or secondary research question, one exploratory section of extracted evidence included documentation on whether or not the study included implementation of the checklist, and if so, what implementation methods or frameworks were included. The majority (55%) of the studies did not involve implementation while 31% of the studies discussed the implementation of the checklist, but did not describe an implementation method or framework. Additionally, 50% of the studies assessed a digital/electronic quality checklist, 12% evaluated a paper-based quality checklist, and 12% evaluated both a digital- and paper-based checklist. For additional information, see Tables 2 and 3. View this table: [Table 2.](http://medrxiv.org/content/early/2024/09/28/2024.09.27.24314468/T2) Table 2. Overview of included studies. Summary characteristics for the 34 included studies are provided in alphabetical order by first author’s last name. View this table: [Table 3.](http://medrxiv.org/content/early/2024/09/28/2024.09.27.24314468/T3) Table 3. Characteristics of included studies . Characteristics include: country of publication, study design, year published, total number of participants, and implementation methods/frameworks used (where applicable). ### Design Methods Twenty-three distinct design methods were identified across the 29 studies, with many studies often employing more than one design method. Of these 23 design methods, the most common ones were interviews (41%), literature reviews (34%), simulated sessions/scenarios (31%), Delphi consensus (24%), and co-design / participatory design / consensus generating workshops (24%), which were a mix of collaborative (involve the cooperation of multiple parties working together) and non-collaborative approaches (involve one researcher and one stakeholder at a given time) (Table 4). View this table: [Table 4.](http://medrxiv.org/content/early/2024/09/28/2024.09.27.24314468/T4) Table 4. Overview of design methods used. Design methods were used in various ways for the design and development of checklists within the studies, including generating checklist content, providing feedback on checklist design and format, collecting data from testing, and improving the design through iterations. Studies also varied in their quality of information provided on design methods. For example, the use of a literature review ranged from an informal literature review (e.g., uncomprehensive literature search) to a full systematic review, with the level of detail in the reported information often minimal. For a list of design methods in each study, refer to Table 2. ### Theoretical Frameworks and Design Principles Used The included studies specified the theoretical design framework, the design principles used, both, or neither. Sixteen different theoretical frameworks or design principles were identified across the 29 studies. Co-design approaches were the most commonly used design principles, with 8 studies (28%) reporting a “co-design approach” which included five sub-categories (Table 5). Seven studies (24%) used a Delphi or modified Delphi consensus technique, and six studies (21%) used what they called a “participatory design” approach. Description of the participatory design approach varied, with some studies using consensus generating workshops, expert steering groups, simulation sessions, usability testing, or a combination of these methods (Table 5). View this table: [Table 5.](http://medrxiv.org/content/early/2024/09/28/2024.09.27.24314468/T5) Table 5. Theoretical framework and/or design principle(s) used. ### Level of Stakeholder Engagement Studies varied in the level of stakeholder engagement according to the IAP2 Spectrum of Participation framework (Fig 2). The greatest percentage of studies were categorized as “Consult,” with 11 studies (38%) obtaining feedback from stakeholders on analysis, issues, decisions, etc. None of the 29 studies had an “Empower” level of stakeholder engagement, where final decision-making was placed in the hands of the stakeholders. The majority of studies did not explicitly state what level of participation or engagement the stakeholders had in the study.  [Fig 2.](http://medrxiv.org/content/early/2024/09/28/2024.09.27.24314468/F2) Fig 2. Level of stakeholder engagement. The twenty-nine studies were categorized by level of stakeholder engagement as Inform (n=3), Consult (n=11), Involve (n=7), Collaborate (n=8), and Empower (n=0). The 9 most common design methods used among the studies were also stratified by level of stakeholder engagement using the IAP2 Spectrum of Participation framework (Fig. 3). Results demonstrate studies across the inform, consult, involve, and collaborate levels of stakeholder engagement utilized a variety of design methods, including interviews, literature reviews, simulated scenarios, and surveys. Level of stakeholder engagement was also explored in the context of funding sources available (or not available) for included studies (Fig 4). Of the 29 studies, 76% (n=22) were either nationally, organizationally, or locally/internally funded by a funding source, with 24% (n=7) listing no funding sources. Of the funded studies, the majority (n=13) were nationally funded. Of the 8 studies designated as “collaborate” levels of engagement, n=5 (63%) were nationally funded. Of the 6 studies designated as “inform” levels of engagement, n=4 (67%) did not have funding listed.  [Fig 3.](http://medrxiv.org/content/early/2024/09/28/2024.09.27.24314468/F3) Fig 3. Design methods by level of stakeholder engagement. Most commonly used design methods in relation to level of stakeholder engagement (inform, consult, involve, collaborate, and empower).  [Fig 4.](http://medrxiv.org/content/early/2024/09/28/2024.09.27.24314468/F4) Fig 4. Type of funding in each study by level of stakeholder engagement. Type of funding (whether the study was nationally funded, funded by a research organization, locally/internally funded, or no description of funding at all) in relation to the level of stakeholder engagement (inform, consult, collaborate, and empower). Stakeholder involvement, or the respective stakeholder roles/groups (e.g., nurses, physicians, patients) that participated in the study, also varied among the studies. Approximately 24% (7/29) had only one primary stakeholder group involved in the study; 76% incorporated two or more different stakeholder roles. Levels of stakeholder engagement, synthesized using the IAP2 Spectrum of Participation framework, were also stratified by one versus multiple stakeholder groups (Fig 5).  [Fig 5.](http://medrxiv.org/content/early/2024/09/28/2024.09.27.24314468/F5) Fig 5. Stakeholder involvement by level of stakeholder engagement. Stakeholder involvement (one vs multiple stakeholder groups involved) in relation to level of stakeholder engagement (inform, consult, involve, collaborate, and empower). ## Discussion This scoping review synthesized evidence from 29 studies by disclosing the current state of design approaches for developing quality checklists in healthcare organizations. We summarized major design approaches and their characteristics, adopted theoretical frameworks and design principles, and levels of stakeholder engagement. ### Design Methods Our analysis suggests that the majority of design methods used to design quality checklists were not collaborative in nature (e.g., involve the cooperation of multiple parties working together), with most methods only involving one researcher and one stakeholder (e.g., interviews, surveys/questionnaires). Studies that used collaborative methods such as Delphi consensus, consensus workshops, and focus groups noted some logistical challenges, such as drop-out observed across consensus meeting phases, virtual-only Delphi meetings hindering productive face-to-face discussion, funding constraints, lack of speaking up in mixed groups, and limitations on participant type and locations (38,44,51,52,65). This may suggest that studies favored more feasible design methods over more collaborative yet logistically-challenging ones. As anticipated, a wide variety of different design methods were identified among the 29 studies, and several of the studies used multiple design methods in the development of the quality checklists. A third of the studies (n=10/29) started with a form of literature review before developing their quality checklist. This trend likely stems from the diverse existing literature on checklists that can be used to inform the design and development of new quality checklists. However, several studies noted that the quality checklists discussed were part of a quality improvement project in an organization, which may arise from a local need, local evidence base, or existing system process that needs improvement, thereby not prompting a desire for a full literature review to inform the project. Twelve of the twenty-nine studies (41%) also included a form of interviews in order to gather insights from users and stakeholders prior to checklist development and gather feedback for checklist iterations. Interviews were predominantly described as semi-structured interviews and were done with one to two research team members. All studies used interviews in conjunction with another design method (e.g., observation sessions and interviews, interviews and focus groups). This was anticipated given the growing use of mixed methods to develop a comprehensive understanding of a topic (triangulation) particularly in qualitative research (82,83). Interview methods varied by channel (video, telephone, in person), length (between 15 minutes to 1 hour), and some described using a prepared interview guide. Notably, several studies described multiple rounds of interviews for different purposes – for example, as part of contextual inquiries for data collection and for value specification, between different checklist prototypes with different focuses, or different interview purposes depending on the type of stakeholder (43,48,49,54). Across the studies using interviews, a variety of primary and occasionally secondary stakeholders were included, in line with checklist development and implementation among high reliability organizations (16). Our findings suggest that while interviews, particularly semi-structured interviews, continue to be the most widely used data collection strategy, organizations adapt interviews as a design method to fit the purposes and needs in the development of quality checklists. Simulated sessions or scenarios were also frequently used in the design, development, and evaluation of the quality checklist (n=9/29, 31%). These studies predominantly used more than one simulated case and varied by task, environment, situation, and fidelity; for example, one study ran usability testing with multiple, simple scenarios (50), while another used near-live simulation sessions to evaluate features of the checklist (56). Given the healthcare settings of these studies, simulated sessions provide an effective way of performing testing and iterative evaluation of a quality checklist without risking harm to stakeholders such as patients in an emergency care setting (57). Some studies acknowledged that a lack of objective observations in actual settings impacted the validity and limitations of the study (47,54,64). As the focus of this scoping review was on the design and evaluation and not the implementation of quality checklists, many included studies remained in the simulated scenario phase and did not address the transition from simulation to actual (e.g., clinic) settings, although they discussed the intent to apply the checklists in real-world settings in the future. Other design methods used (Table 4) include 11 different methods that were used in only 1 of the 29 studies. Many of these methods include modeling methods that support conceptual designs or allow the research team to visualize users and their needs, such as personas, storyboarding, and process mapping. These methods were also used in conjunction with other, more commonly used design methods. This may be because the majority of studies had a clear understanding or direct participation of checklist users to qualify data and provide feedback that make these creative modeling methods noncompulsory for research teams. These results indicate opportunities in the design methods of quality checklists that need further exploration. For example, methods such as storyboarding and touchpoint videos have been used with interviews and focus groups to expand the contextual knowledge gathered in the design of electronic patient reported outcomes (84,85). Further research is needed to examine the feasibility and potential added benefit of modeling and creative design methods in conjunction with more established design methods such interviews, focus groups, and simulated sessions/scenarios. ### Theoretical Frameworks and Design Principles Used Studies varied in the way design frameworks and principles were reported. For example, 28% (8/29) of studies described a “co-design approach” which was further described as experience-based, agile/collaborative, in-the-wild, or based in organizational semiotics. The application of co-design methods differed across studies, involving different data gathering techniques (e.g., semi-structured interviews, focus groups, co-design sessions/workshops, use cases, literature review, simulated sessions/scenarios, Delphi consensus group, touchpoint video, and observations). This diversity was somewhat expected given the range of healthcare organization types and specific functions for each quality checklist. Similarly, studies using participatory design (n=6/29) included a variety of design methods, including consensus workshops and meetings, simulated scenarios, mockups, interviews, and questionnaires. Vaughn *et al.* describe participatory research methods and the variety in methods and tools that are participatory by design (33). Rather than emphasizing specific design methods, participatory design focuses more on how the users and stakeholders are engaged in the research process (33). Despite often nuanced differences in practice, we postulate that co-design (28%), participatory design (21%), and user-centered design (14%) approaches were among the more prevalent design approaches due to study inclusion criteria specifically looking at the design, development, and evaluation of quality checklists (Table 1). Although not the primary focus of our review, there was a small but meaningful amount of overlap between theoretical frameworks and design principles used and implementation methods or frameworks adopted. While 55% of studies did not involve implementation of the checklist and 31% of studies involved implementation but did not describe an implementation framework (Table 3), 4 studies integrated implementation frameworks with the design and development principles (46,48,49,55,58). These studies used frameworks such as Action Research / the Action Research Cycle, Theoretical Domains Framework (TDF), a co-design methodology for an integrated pathway management approach, and the Normalization Process Theory (NPT). Eleven of the sixteen distinct theoretical frameworks or design principles were identified in only 1 of the 29 studies (Table 5). These theoretical frameworks and design principles appear to be specific to the type of quality checklist, environment, or checklist purpose within the study. For example, one study used the American Psychiatric Association’s (APA) App Evaluation Model to support decision-making for a digital health checklist because of its application to selecting apps (40,86). Another study used the Systems Engineering Initiative for Patient Safety (SEIPS) work system model which focuses on system design and its impact on processes, outcomes, and their relationships for patient safety, to identify safety hazards for a preliminary safety checklist (45). Further research is needed to identify and evaluate the optimal use of theoretical frameworks for the development and implementation of quality checklists. Overall, the scoping review revealed considerable variation in design quality and definition of design principles and frameworks, with no one definitive framework emerging as preferable or exceptional. Some studies used participatory design, co-design, and/or user-centered design interchangeably or in combination when describing the study methods or design strategies (45,46,49,60,62). Other studies stated a particular approach was used but provided limited information on why the approach was used, definitions for the principle or framework mentioned, and how these principles and frameworks impacted the methods or work accomplished (37,51). Notably, the majority of studies using a participatory design, co-design, or user-centered design approach did *not* include nor clearly specify a validated theoretical framework (47,50,51,56,57,59,60,63,66,67). ### Level of Stakeholder Engagement Level of stakeholder engagement did not appear to be associated with the design approaches or methods used (Fig. 3). For example, the 8 studies that used co-design approaches had different levels of stakeholder participation and engagement, including consult (obtain feedback on analysis, issues, decisions, etc. from stakeholders), involve (work with stakeholders to ensure concerns and aspirations are considered and understood), and collaborate (partner with stakeholders in each aspect of the decision-making) levels. However, of the design methods used in the studies categorized as “consult” levels, the majority were non-collaborative methods in nature (e.g., individualistic methods that only required one stakeholder at a given time, i.e., surveys, interviews). In comparison, methods used in “collaborate” levels of engagement studies often involved design methods that brought stakeholders altogether (i.e., co-design or participatory design sessions, Delphi consensus groups, and focus groups). This suggests a shift towards partnership with multiple stakeholders at once with increased levels of engagement. However, none of the studies were categorized as having “empower” levels of engagement (placing the final decision-making in the hands of stakeholders). This may be because studies developing a quality checklist may not have or require the level of resources needed to empower their stakeholders, or the organizations may not have the capacity to turn over full or final decision-making to stakeholders (87). Nevertheless, this absence suggests a gap in the literature, indicating empowerment of stakeholders to make final decisions in the design of quality checklists is a potential area warranting further exploration. The level of stakeholder engagement was also synthesized in the context of funding source availability. We found a potential association of increased levels of stakeholder engagement (i.e., “collaborate” levels of stakeholder engagement) and the presence of national funding, compared to less engagement by stakeholder (i.e., “inform” levels of stakeholder engagement) with no disclosed funding (Fig 4). Specifically, 63% of studies that used “collaborate” levels of engagement (partnered with stakeholders in each aspect of the decision-making) having some type of national funding source, while 67% of studies using “inform” levels of engagement (provide stakeholders with objective information) did not cite funding of any type. This suggests studies with funding likely have more resources for stakeholder engagement and therefore are more capable of engaging stakeholders. Literature has demonstrated the importance of research funding towards meaningful stakeholder engagement and increased financial efforts and support to engage stakeholders in the design of interventions (88–90). Further research is warranted to understand how funding sources may impact design methods and stakeholder engagement in healthcare organizations. The type of stakeholder involved in the study did not appear to be associated with stakeholder engagement (Fig 5). Studies ranged from one population type (24% of the studies; e.g., just physicians in a particular department) to a variety of stakeholder groups (76% of the studies; e.g., nurses, pharmacists, speech language therapist, physicians, respiratory therapists, dieticians, social workers, survivors, and family members) involved in one study. Studies with only one stakeholder group were spread across inform, consult, involve, and collaborate levels, as were those involving multiple stakeholder groups . One study that identified physicians as the primary user group of the quality checklist and nurses as a secondary user group conducted usability evaluations with only physicians but acknowledged that requirements of nurses (that did not participate in usability evaluations) may differ from those that were identified using only physicians in the evaluation process (64). Other studies acknowledged that further validation of the quality checklist was needed utilizing other stakeholders or a wider range of users (39,45,63,65). Some studies also used design methods for different purposes depending on the stakeholder involved. For example, Chudleigh *et al.* held semi-structured interviews with healthcare professionals to explore intervention acceptability, feasibility, and usability and perceptions of influential factors, while they conducted semi-structured interviews with parents of newborns to ascertain experiences and perceptions of the co-designed interventions (48,49). The quantity or variety of stakeholder roles participating in the study did not appear to impact the level of engagement, suggesting the potential feasibility for an increased level of engagement while maintaining a large variety of stakeholder roles involved in the study. ### Strengths and Limitations #### Strengths The literature review was performed by searching 7 different databases: PubMed, APA PsycInfo, CINAHL, Embase, Scopus, ACM Digital Library, and IEEE Xplore. These databases were selected by recommendation from an expert committee as they span a diverse range of healthcare, biomedical, life sciences, behavioral and social sciences, psychology, physical sciences, health sciences, nursing and allied health, computing, technology, and engineering fields for identification of manuscripts related to healthcare organizations. Title and abstract review and full text review were performed with two reviewers (EK and AC), with a third reviewer (LM) resolving any conflicts at each step. Inter-rater reliability statistics were calculated for the title and abstract review stage and the full text review stage, generating a Cohen’s Kappa of 0.632 and 0.745, respectively. Kappa value interpretations by Landis and Koch (1977) indicate the strength of agreement for both stages was “substantial” (91). This suggests there was a substantial degree of agreement among the independent reviewers, lending validity to literature review results. #### Limitations Limitations to the literature review include studies with small participant sizes and observational study designs. Thirteen of the 29 studies (45%) had sample sizes of less than twenty participants, which likely influenced the design methods and approaches that were reported. Participant size may have been influenced by the number of stakeholder groups or funding sources. Studies may have also been able to achieve data saturation through the use of various design principles, with research suggesting a range of 9-17 participants were adequate in qualitative research and usability studies suggesting approximately 15-16 participants to test the usability of a design (92,93). There was moderate strength of evidence identified by the quality assessment as interpreted using the AHRQ EPIC approach (34). Due to the nature of the literature review, which aimed to answer the primary question of, “what is the current state of utilizing various design approaches for developing quality checklists in healthcare organizations,” studies were observational studies and not conventional randomized controlled trials. As a result, this was expected to impact the study limitations, directness, consistency, precision, and reporting bias domains when assessing for quality. ## Conclusion This literature review demonstrated the variety of design methods, theoretical frameworks, and design principles used in the design and development of quality checklists across healthcare organizations. The studies used predominantly non-collaborative design methods (e.g., interviews, surveys/questionnaires), suggesting a preference for more feasible design methods over more collaborative methods that present logistical challenges. The review also revealed design terminology discrepancies and instances where terms were used interchangeably. The analysis of stakeholder engagement indicated a gap in studies that empowered their stakeholders in the quality checklist design process. Future research needs to clarify theoretical frameworks or design principles and justify the selection of specific design methods, and examine how they impact the outcomes. Further research is also needed into optimal ways to empower stakeholders in the design process, and how different levels of stakeholder engagement might impact design and implementation outcomes. These propositions address the need for a highly effective, standardized methodology for the design of quality checklists that may improve the use, adoption impediments, and implementation barriers that exist in healthcare. ## Author Contributions Elizabeth Kwong and Lukasz Mazur designed and conceptualized the study. Elizabeth Kwong, Amy Cole, and Lukasz Mazur performed data curation, collection, and extraction of the data. Elizabeth Kwong conducted the formal analysis of the data. Elizabeth Kwong, Amy Cole, Dorothy Sippo, Fei Yu, Karthik Adapa, Christopher M. Shea, Carlton Moore, Shiva Das, and Lukasz Mazur interpreted the data. Elizabeth Kwong prepared the original draft of the manuscript. All authors reviewed, edited, and approved the final manuscript. ## Registration The scoping review protocol was not registered. ## Competing Interests The authors have declared that no competing interests exist. ## Data Availability Statement Data underlying the findings are fully available without restriction from the time of publication. All relevant data are within the manuscript. ## Acknowledgements and Support This work would not have been made possible without the financial support of The National Library of Medicine (NLM) Biomedical Informatics and Data Science T-15 Fellowship (T15-LM012500). We would like to give special thanks to Jamie Lynn Conklin and Jennifer Bissram for their assistance in the scoping review and their valuable time and contributions. * Received September 27, 2024. * Revision received September 27, 2024. * Accepted September 28, 2024. * © 2024, Posted by Cold Spring Harbor Laboratory This pre-print is available under a Creative Commons License (Attribution 4.0 International), CC BY 4.0, as described at [http://creativecommons.org/licenses/by/4.0/](http://creativecommons.org/licenses/by/4.0/) ## References 1. 1.Hales B, Terblanche M, Fowler R, Sibbald W. Development of medical checklists for improved quality of patient care. Int J Qual Health Care. 2008 Feb;20(1):22–30. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1093/intqhc/mzm062&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=18073269&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2024%2F09%2F28%2F2024.09.27.24314468.atom) [Web of Science](http://medrxiv.org/lookup/external-ref?access_num=000252305500005&link_type=ISI) 2. 2.Checklists | PSNet [Internet]. [cited 2023 Aug 21]. Available from: [https://psnet.ahrq.gov/primer/checklists](https://psnet.ahrq.gov/primer/checklists) 3. 3.Quality Checklist VS Quality Checksheet - PM Vidya [Internet]. [cited 2023 Aug 21]. Available from: [https://pmvidya.com/blog/quality-checklists-vs-quality-checksheets/](https://pmvidya.com/blog/quality-checklists-vs-quality-checksheets/) 4. 4.Higgins WY, Boorman DJ. An Analysis of the Effectiveness of Checklists when combined with Other Processes, Methods and Tools to Reduce Risk in High Hazard Activities [Internet]. 2016 [cited 2023 Aug 18]. Available from: [http://www.boeing.com/resources/boeingdotcom/specialty/innovation-quarterly/December-2016/BTJ\_Checklist\_full.pdf](http://www.boeing.com/resources/boeingdotcom/specialty/innovation-quarterly/December-2016/BTJ_Checklist_full.pdf) 5. 5.Chaparro A, Keebler JR, Lazzara EH, Diamond A. Checklists: A review of their origins, benefits, and current uses as a cognitive aid in medicine. Ergonomics in Design: The Quarterly of Human Factors Applications. 2019 Apr;27(2):21–6. 6. 6.Pronovost P, Needham D, Berenholtz S, Sinopoli D, Chu H, Cosgrove S, et al. An intervention to decrease catheter-related bloodstream infections in the ICU. N Engl J Med. 2006 Dec 28;355(26):2725–32. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1056/NEJMoa061115&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=17192537&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2024%2F09%2F28%2F2024.09.27.24314468.atom) [Web of Science](http://medrxiv.org/lookup/external-ref?access_num=000243110500004&link_type=ISI) 7. 7.Haynes AB, Weiser TG, Berry WR, Lipsitz SR, Breizat A-HS, Dellinger EP, et al. A surgical safety checklist to reduce morbidity and mortality in a global population. N Engl J Med. 2009 Jan 29;360(5):491–9. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1056/NEJMsa0810119&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=19144931&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2024%2F09%2F28%2F2024.09.27.24314468.atom) [Web of Science](http://medrxiv.org/lookup/external-ref?access_num=000262812400008&link_type=ISI) 8. 8.Wolff AM, Taylor SA, McCabe JF. Using checklists and reminders in clinical pathways to improve hospital inpatient care. Med J Aust. 2004 Oct 18;181(8):428–31. [PubMed](http://medrxiv.org/lookup/external-ref?access_num=15487958&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2024%2F09%2F28%2F2024.09.27.24314468.atom) [Web of Science](http://medrxiv.org/lookup/external-ref?access_num=000225383600008&link_type=ISI) 9. 9.Molina RL, Benski A-C, Bobanski L, Tuller DE, Semrau KEA. Adaptation and implementation of the WHO Safe Childbirth Checklist around the world. Implement Sci Commun. 2021 Jul 8;2(1):76. 10. 10.Halasyamani L, Kripalani S, Coleman E, Schnipper J, van Walraven C, Nagamine J, et al. Transition of care for hospitalized elderly patients--development of a discharge checklist for hospitalists. J Hosp Med. 2006 Nov;1(6):354–60. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1002/jhm.129&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=17219528&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2024%2F09%2F28%2F2024.09.27.24314468.atom) 11. 11.Soong C, Daub S, Lee J, Majewski C, Musing E, Nord P, et al. Development of a checklist of safe discharge practices for hospital patients. J Hosp Med. 2013 Aug;8(8):444–9. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1002/jhm.2032&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=23554352&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2024%2F09%2F28%2F2024.09.27.24314468.atom) 12. 12.Crandell BC, Bates JS, Grgic T. Start using a checklist, PRONTO: Recommendation for a standard review process for chemotherapy orders. J Oncol Pharm Pract. 2018 Dec;24(8):609–16. 13. 13.Grelat M, Pommier B, Portet S, Amelot A, Barrey C, Leroy H-A, et al. Patients with Coronavirus 2019 (COVID-19) and Surgery: Guidelines and Checklist Proposal. World Neurosurg. 2020 Jul;139:e769–73. 14. 14.Cheng L, Zhu H, Pan H, Huang S, Zhuang Y, Zhu C, et al. Development and application of quality checklist for the prevention and control of COVID-19 in fever clinic and isolation ward of the general hospital. Zhejiang Da Xue Xue Bao Yi Xue Ban. 2021 Feb 25;50(1):74–80. 15. 15.Lingard L, Regehr G, Orser B, Reznick R, Baker GR, Doran D, et al. Evaluation of a preoperative checklist and team briefing among surgeons, nurses, and anesthesiologists to reduce failures in communication. Arch Surg. 2008 Jan;143(1):12–7; discussion 18. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1001/archsurg.2007.21&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=18209148&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2024%2F09%2F28%2F2024.09.27.24314468.atom) [Web of Science](http://medrxiv.org/lookup/external-ref?access_num=000252438500003&link_type=ISI) 16. 16.Thomassen Ø, Espeland A, Søfteland E, Lossius HM, Heltne JK, Brattebø G. Implementation of checklists in health care; learning from high-reliability organisations. Scand J Trauma Resusc Emerg Med. 2011 Oct 3;19:53. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1186/1757-7241-19-53&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=21967747&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2024%2F09%2F28%2F2024.09.27.24314468.atom) 17. 17.Thomassen Ø, Brattebø G, Søfteland E, Lossius HM, Heltne JK. The effect of a simple checklist on frequent pre-induction deficiencies. Acta Anaesthesiol Scand. 2010 Nov;54(10):1179–84. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1111/j.1399-6576.2010.02302.x&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=21069898&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2024%2F09%2F28%2F2024.09.27.24314468.atom) [Web of Science](http://medrxiv.org/lookup/external-ref?access_num=000282687400003&link_type=ISI) 18. 18.de Vries EN, Prins HA, Crolla RMPH, den Outer AJ, van Andel G, van Helden SH, et al. Effect of a comprehensive surgical safety system on patient outcomes. N Engl J Med. 2010 Nov 11;363(20):1928–37. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1056/NEJMsa0911535&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=21067384&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2024%2F09%2F28%2F2024.09.27.24314468.atom) [Web of Science](http://medrxiv.org/lookup/external-ref?access_num=000284047600008&link_type=ISI) 19. 19.Winters BD, Gurses AP, Lehmann H, Sexton JB, Rampersad CJ, Pronovost PJ. Clinical review: checklists - translating evidence into practice. Crit Care. 2009 Dec 31;13(6):210. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1186/cc7792&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=20064195&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2024%2F09%2F28%2F2024.09.27.24314468.atom) 20. 20.Cross N. A history of design methodology. In: Vries MJ, Cross N, Grant DP, editors. Design Methodology and Relationships with Science. Dordrecht: Springer Netherlands; 1993. p. 15–27. 21. 21.van der Linden J, Pedroso de Lacerda A, Pedro Ornaghi de Aguiar J. The evolution of design methods. ResearchGate; 2011. 22. 22. Friis Dam R, Siang TY. The History of Design Thinking | IxDF [Internet]. [cited 2023 Sep 12]. Available from: [https://www.interaction-design.org/literature/article/design-thinking-get-a-quick-overview-of-the-history](https://www.interaction-design.org/literature/article/design-thinking-get-a-quick-overview-of-the-history) 23. 23.Burian BK, Clebone A, Dismukes K, Ruskin KJ. More than a tick box: medical checklist development, design, and use. Anesth Analg. 2018 Jan;126(1):223–32. [PubMed](http://medrxiv.org/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2024%2F09%2F28%2F2024.09.27.24314468.atom) 24. 24.What Makes a Good Checklist | PSNet [Internet]. [cited 2023 Sep 11]. Available from: [https://psnet.ahrq.gov/perspective/what-makes-good-checklist](https://psnet.ahrq.gov/perspective/what-makes-good-checklist) 25. 25.Verdaasdonk EGG, Stassen LPS, Widhiasmara PP, Dankelman J. Requirements for the design and implementation of checklists for surgical processes. Surg Endosc. 2009 Apr;23(4):715–26. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1007/s00464-008-0044-4&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=18636292&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2024%2F09%2F28%2F2024.09.27.24314468.atom) [Web of Science](http://medrxiv.org/lookup/external-ref?access_num=000264266800007&link_type=ISI) 26. 26.Fletcher KA, Bedwell WL. Cognitive Aids. Proceedings of the International Symposium on Human Factors and Ergonomics in Health Care. 2014 Jun;3(1):148–52. 27. 27.Petkovic J, Magwood O, Lytvyn L, Khabsa J, Concannon TW, Welch V, et al. Key issues for stakeholder engagement in the development of health and healthcare guidelines. Res Involv Engagem. 2023 Apr 28;9(1):27. 28. 28.Boaz A, Hanney S, Borst R, O’Shea A, Kok M. How to engage stakeholders in research: design principles to support improvement. Health Res Policy Syst. 2018 Jul 11;16(1):60. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1186/s12961-018-0337-6&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2024%2F09%2F28%2F2024.09.27.24314468.atom) 29. 29.Hollmann S, Regierer B, Bechis J, Tobin L, D’Elia D. Ten simple rules on how to develop a stakeholder engagement plan. PLoS Comput Biol. 2022 Oct 13;18(10):e1010520. 30. 30.Concannon TW, Fuster M, Saunders T, Patel K, Wong JB, Leslie LK, et al. A systematic review of stakeholder engagement in comparative effectiveness and patient-centered outcomes research. J Gen Intern Med. 2014 Dec;29(12):1692–701. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1007/s11606-014-2878-x&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=24893581&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2024%2F09%2F28%2F2024.09.27.24314468.atom) 31. 31.Riva JJ, Malik KMP, Burnie SJ, Endicott AR, Busse JW. What is your research question? An introduction to the PICOT format for clinicians. J Can Chiropr Assoc. 2012 Sep;56(3):167–71. [PubMed](http://medrxiv.org/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2024%2F09%2F28%2F2024.09.27.24314468.atom) 32. 32.Bammer G. Key issues in co-creation with stakeholders when research problems are complex. Evid Policy. 2019 Aug 1;15(3):423–35. 33. 33.Vaughn LM, Jacquez F. Participatory research methods – choice points in the research process. J Particip Res Methods. 2020 Jul 21; 34. 34.Berkman ND, Lohr KN, Ansari MT, Balk EM, Kane R, McDonagh M, et al. Grading the strength of a body of evidence when assessing health care interventions: an EPC update. J Clin Epidemiol. 2015 Nov;68(11):1312–24. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1016/j.jclinepi.2014.11.023&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=25721570&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2024%2F09%2F28%2F2024.09.27.24314468.atom) 35. 35.Page M, McKenzie J, Bossuyt P, Boutron I, Hoffmann T, Mulrow CD, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ. 2021;372:n71. 36. 36.Carayon P, Hoonakker P, Hundt AS, Salwei M, Wiegmann D, Brown RL, et al. Application of human factors to improve usability of clinical decision support for diagnostic decision-making: a scenario-based simulation study. BMJ Qual Saf. 2020 Apr;29(4):329–40. [Abstract/FREE Full Text](http://medrxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MzoicWhjIjtzOjU6InJlc2lkIjtzOjg6IjI5LzQvMzI5IjtzOjQ6ImF0b20iO3M6NTA6Ii9tZWRyeGl2L2Vhcmx5LzIwMjQvMDkvMjgvMjAyNC4wOS4yNy4yNDMxNDQ2OC5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 37. 37.Higby C, Pye K. Improving discharge from the paediatric oncology unit. Paediatr Nurs. 2009 May;21(4):30–2. [PubMed](http://medrxiv.org/lookup/external-ref?access_num=19505062&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2024%2F09%2F28%2F2024.09.27.24314468.atom) 38. 38.Ashiru-Oredope D, Garraghan F, Olaoye O, Krockow EM, Matuluko A, Nambatya W, et al. Development and Implementation of an Antimicrobial Stewardship Checklist in Sub-Saharan Africa: A Co-Creation Consensus Approach. Healthcare (Basel). 2022 Sep 6;10(9). 39. 39.Nebeker C, Bartlett Ellis RJ, Torous J. Development of a decision-making checklist tool to support technology selection in digital health research. Transl Behav Med. 2020 Oct 8;10(4):1004–15. 40. 40. Bartlett Ellis R, Wright J, Miller LS, Jake-Schoffman D, Hekler EB, Goldstein CM, et al. Lessons Learned: Beta-Testing the Digital Health Checklist for Researchers Prompts a Call to Action by Behavioral Scientists. J Med Internet Res. 2021 Dec 22;23(12):e25414. 41. 41.Bearsley-Smith C, Sellick K, Chesters J, Francis K, Gippsland Adolescent Depression Research Group. Treatment content in child and adolescent mental health services: development of the treatment recording sheet. Adm Policy Ment Health. 2008 Sep;35(5):423–35. [PubMed](http://medrxiv.org/lookup/external-ref?access_num=18679789&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2024%2F09%2F28%2F2024.09.27.24314468.atom) 42. 42.Benton SE, Hueckel RM, Taicher B, Muckler VC. Usability assessment of an electronic handoff tool to facilitate and improve postoperative communication between anesthesia and intensive care unit staff. Comput Inform Nurs. 2020 Oct;38(10):500–7. 43. 43.Bentvelsen RG, van der Vaart R, Veldkamp KE, Chavannes NH. Systematic development of an mHealth app to prevent healthcare-associated infections by involving patients: ‘Participatient.’ Clinical eHealth. 2021;4:37–44. 44. 44.Bergerød IJ, Braut GS, Fagerdal B, Gilje B, Wiig S. Developing a Next-of-Kin Involvement Guide in Cancer Care-Results From a Consensus Process. Cancer Nurs. 2021 Dec 1;44(6):E447–57. 45. 45.Bowie P, Ferguson J, MacLeod M, Kennedy S, de Wet C, McNab D, et al. Participatory design of a preliminary safety checklist for general practice. Br J Gen Pract. 2015 May;65(634):e330–43. [Abstract/FREE Full Text](http://medrxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NDoiYmpncCI7czo1OiJyZXNpZCI7czoxMToiNjUvNjM0L2UzMzAiO3M6NDoiYXRvbSI7czo1MDoiL21lZHJ4aXYvZWFybHkvMjAyNC8wOS8yOC8yMDI0LjA5LjI3LjI0MzE0NDY4LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 46. 46.Campbell-Yeo M, Dol J, Richardson B, McCulloch H, Hundert A, Foye S, et al. A co-design of clinical virtual care pathways to engage and support families requiring neonatal intensive care in response to the COVID-19 pandemic (COVES study). J Neonatal Nurs. 2021 Dec;27(6):463–70. [PubMed](http://medrxiv.org/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2024%2F09%2F28%2F2024.09.27.24314468.atom) 47. 47.Ceulemans M, Brughmans M, Poortmans L-L, Spreuwers E, Willekens J, Roose N, et al. Development and pilot testing of a dispensing protocol on emergency contraceptive pills for community pharmacists in belgium. Pharmacy (Basel). 2022 Jun 1;10(3). 48. 48.Chudleigh J, Holder P, Moody L, Simpson A, Southern K, Morris S, et al. Process evaluation of co-designed interventions to improve communication of positive newborn bloodspot screening results. BMJ Open. 2021 Aug 27;11(8):e050773. [Abstract/FREE Full Text](http://medrxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NzoiYm1qb3BlbiI7czo1OiJyZXNpZCI7czoxMjoiMTEvOC9lMDUwNzczIjtzOjQ6ImF0b20iO3M6NTA6Ii9tZWRyeGl2L2Vhcmx5LzIwMjQvMDkvMjgvMjAyNC4wOS4yNy4yNDMxNDQ2OC5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 49. 49.Chudleigh J, Shakespeare L, Holder P, Chinnery H, Hack G, Gill T, et al. Co-designing Improved Communication of Newborn Bloodspot Screening Results to Parents: Mixed Methods Study. J Particip Med. 2022 Jul 27;14(1):e33485. 50. 50.Flohr L, Beaudry S, Johnson KT, West N, Burns CM, Ansermino JM, et al. Clinician-Driven Design of VitalPAD-An Intelligent Monitoring and Communication Device to Improve Patient Safety in the Intensive Care Unit. IEEE J Transl Eng Health Med. 2018 Mar 5;6:3000114. 51. 51.Kerwin T, Hittle B, Stredney D, De Boeck P, Wiet G. Multi-Institutional Development of a Mastoidectomy Performance Evaluation Instrument. J Surg Educ. 2017 May 20;74(6):1081–7. 52. 52.Kirkham EJ, Crompton CJ, Iveson MH, Beange I, McIntosh AM, Fletcher-Watson S. Co-development of a Best Practice Checklist for Mental Health Data Science: A Delphi Study. Front Psychiatry. 2021 Jun 10;12:643914. 53. 53.Kulp L, Sarcevic A. Design in the wild: lessons from researcher participation in design of emerging technology. Ext Abstr Hum Factors Computing Syst. 2017 May;2017:1802–8. 54. 54.Kuo P-Y, Saran R, Argentina M, Heung M, Bragg-Gresham JL, Chatoth D, et al. Development of a checklist for the prevention of intradialytic hypotension in hemodialysis care: design considerations based on activity theory. Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. New York, NY, USA: ACM; 2019. p. 1–14. 55. 55.Li W, Liu K, Yang H, Yu C. Integrated clinical pathway management for medical quality improvement – based on a semiotically inspired systems architecture. Eur J Inf Syst. 2014 Jul;23(4):400–17. 56. 56.Mastrianni A, Sarcevic A, Hu A, Almengor L, Tempel P, Gao S, et al. Transitioning Cognitive Aids into Decision Support Platforms: Requirements and Design Guidelines. ACM Trans Comput-Hum Interact. 2023 Jan 30; 57. 57.McIntosh CA, Donnelly D, Marr R. Using simulation to iteratively test and re-design a cognitive aid for use in the management of severe local anaesthetic toxicity. BMJ STEL. 2017 Aug 20;4(1):bmjstel-2017-000221. 58. 58.Moody G, Choong YY, Greenwood J. An action research approach to the development of a clinical pathway for women requiring Caesarean sections. Contemp Nurse. 2001 Dec;11(2– 3):195–205. [PubMed](http://medrxiv.org/lookup/external-ref?access_num=11924616&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2024%2F09%2F28%2F2024.09.27.24314468.atom) 59. 59.Østergaard KL, Simonsen J, Hertzum M. The Handover from Intensive Care Unit to General Ward: Baseline Performance and Participatory Design of an Electronic Follow-Up Plan. Stud Health Technol Inform. 2019 Aug 21;264:1303–7. 60. 60.Østergaard K. A New Model for Intensive Care Unit Follow-up. CIN: Computers, Informatics, Nursing. 2023 Apr;41(4):195–204. 61. 61.Rebic N, Munro S, Norman WV, Soon JA. Pharmacist checklist and resource guide for mifepristone medical abortion: User-centred development and testing. Can Pharm J (Ott). 2021 May 28;154(3):166–74. 62. 62.Rose L, Istanboulian L, Amaral ACK-B, Burry L, Cox CE, Cuthbertson BH, et al. Co-designed and consensus based development of a quality improvement checklist of patient and family-centered actionable processes of care for adults with persistent critical illness. J Crit Care. 2022 Dec;72:154153. 63. 63.Sarcevic A, Rosen BJ, Kulp LJ, Marsic I, Burd RS. Design challenges in converting a paper checklist to digital format for dynamic medical settings. Int Conf Pervasive Comput Technol Healthc. 2016 May;2016:33–40. 64. 64.Schild S, Sedlmayr B, Schumacher A-K, Sedlmayr M, Prokosch H-U, St Pierre M, et al. A Digital Cognitive Aid for Anesthesia to Support Intraoperative Crisis Management: Results of the User-Centered Design Process. JMIR Mhealth Uhealth. 2019 Apr 29;7(4):e13226. 65. 65.Tarola CL, Hirji S, Yule SJ, Gabany JM, Zenati A, Dias RD, et al. Cognitive Support to Promote Shared Mental Models during Safety-Critical Situations in Cardiac Surgery (Late Breaking Report). IEEE Conf Cogn Comput Asp Situat Manag. 2018 Jun;2018:165–7. 66. 66.Tyler N, Wright N, Gregoriou K, Waring J. Improving mental health care transitions through information capture during admission to inpatient mental health services: a quality improvement study. BMC Health Serv Res. 2021 Oct 21;21(1):1132. 67. 67.Wu L, Cirimele J, Leach K, Card S, Chu L, Harrison TK, et al. Supporting crisis response with dynamic procedure aids. Proceedings of the 2014 conference on Designing interactive systems - DIS ’14. New York, New York, USA: ACM Press; 2014. p. 315–24. 68. 68.Lateef F. Simulation-based learning: Just like the real thing. J Emerg Trauma Shock. 2010 Oct;3(4):348–52. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.4103/0974-2700.70743&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=21063557&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2024%2F09%2F28%2F2024.09.27.24314468.atom) 69. 69.Nasa P, Jain R, Juneja D. Delphi methodology in healthcare research: How to decide its appropriateness. World J Methodol. 2021 Jul 20;11(4):116–29. 70. 70.What Is Codesign? — updated 2024 | IxDF [Internet]. Interaction Design Foundation. 2023 [cited 2024 Aug 27]. Available from: [https://www.interaction-design.org/literature/topics/codesign](https://www.interaction-design.org/literature/topics/codesign) 71. 71.A guide for Focus Group, Steering Group and Advisory Panel members. Cambridge Biomedical Research Centre [Internet]. [cited 2024 Aug 27]; Available from: [https://cambridgebrc.nihr.ac.uk/wp-content/uploads/2017/03/PPI-panel-focus-groups.pdf](https://cambridgebrc.nihr.ac.uk/wp-content/uploads/2017/03/PPI-panel-focus-groups.pdf) 72. 72.What is Usability Testing? — updated 2024 | IxDF [Internet]. Interaction Design Foundation. 2016 [cited 2024 Aug 27]. Available from: [https://www.interaction-design.org/literature/topics/usability-testing](https://www.interaction-design.org/literature/topics/usability-testing) 73. 73.Beyer H. Contextual Design: Defining Customer-Centered Systems (Interactive Technologies). 1st ed. Morgan Kaufmann; 1997. 74. 74.Dam RF, Siang TY. Personas – A Simple Introduction | IxDF [Internet]. Interaction Design Foundation. 2024 [cited 2024 Aug 26]. Available from: [https://www.interaction-design.org/literature/article/personas-why-and-how-you-should-use-them](https://www.interaction-design.org/literature/article/personas-why-and-how-you-should-use-them) 75. 75.Sanders EB-N. From user-centered to participatory design approaches. CRC Press. 2002 Apr 25;18–25. 76. 76.Williamson K, Johanson G. Research Methods: Information, Systems, and Contexts. 2nd ed. Cambridge, MA, United States: Chandos Publishing; 2017. 77. 77.Atkins L, Francis J, Islam R, O’Connor D, Patey A, Ivers N, et al. A guide to using the Theoretical Domains Framework of behaviour change to investigate implementation problems. Implement Sci. 2017 Jun 21;12(1):77. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1186/s13012-017-0605-9&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=28637486&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2024%2F09%2F28%2F2024.09.27.24314468.atom) 78. 78.Hajdukiewicz JR, Vicente * KJ. A theoretical note on the relationship between work domain analysis and task analysis. Theoretical Issues in Ergonomics Science. 2004 Nov;5(6):527–38. 79. 79.Carayon P, Schoofs Hundt A, Karsh BT, Gurses AP, Alvarado CJ, Smith M, et al. Work system design for patient safety: the SEIPS model. Qual Saf Health Care. 2006 Dec;15 Suppl 1:i50–8. [Abstract/FREE Full Text](http://medrxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MzoicWhjIjtzOjU6InJlc2lkIjtzOjE0OiIxNS9zdXBwbF8xL2k1MCI7czo0OiJhdG9tIjtzOjUwOiIvbWVkcnhpdi9lYXJseS8yMDI0LzA5LzI4LzIwMjQuMDkuMjcuMjQzMTQ0NjguYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 80. 80.Murray E, Treweek S, Pope C, MacFarlane A, Ballini L, Dowrick C, et al. Normalisation process theory: a framework for developing, evaluating and implementing complex interventions. BMC Med. 2010 Oct 20;8:63. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1186/1741-7015-8-63&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=20961442&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2024%2F09%2F28%2F2024.09.27.24314468.atom) 81. 81.Cross N. Designerly ways of knowing. Design Studies. 1982 Oct;3(4):221–7. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1016/0142-694X(82)90040-0&link_type=DOI) 82. 82.Lambert SD, Loiselle CG. Combining individual interviews and focus groups to enhance data richness. J Adv Nurs. 2008 Apr;62(2):228–37. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1111/j.1365-2648.2007.04559.x&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=18394035&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2024%2F09%2F28%2F2024.09.27.24314468.atom) [Web of Science](http://medrxiv.org/lookup/external-ref?access_num=000254668300010&link_type=ISI) 83. 83.Carter N, Bryant-Lukosius D, DiCenso A, Blythe J, Neville .Alan J. The use of triangulation in qualitative research. Oncol Nurs Forum. 2014 Sep;41(5):545–7. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1188/14.ONF.545-547&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=25158659&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2024%2F09%2F28%2F2024.09.27.24314468.atom) 84. 84.Dunlop E, Ferguson A, Mueller T, Baillie K, Laskey J, Clarke J, et al. Involving patients and clinicians in the design of wireframes for cancer medicines electronic patient reported outcome measures in clinical care: mixed methods study. JMIR Formativ Res. 2023 Dec 21;7:e48296. 85. 85.Rodríguez-Fuertes A, Reinares-Lara P, Garcia-Henche B. Incorporation of the emotional indicators of the patient journey into healthcare organization management. Health Expect. 2023 Feb;26(1):297–306. 86. 86.Nebeker C, Gholami M, Kareem D, Kim E. Applying a digital health checklist and readability tools to improve informed consent for digital health research. Front Digit Health. 2021 Jul 15;3:690901. 87. 87.Spectrum of Public Participation – Organizing Engagement [Internet]. [cited 2024 Jan 29]. Available from: [https://organizingengagement.org/models/spectrum-of-public-participation/](https://organizingengagement.org/models/spectrum-of-public-participation/) 88. 88.Concannon TW, Meissner P, Grunbaum JA, McElwee N, Guise J-M, Santa J, et al. A new taxonomy for stakeholder engagement in patient-centered outcomes research. J Gen Intern Med. 2012 Aug;27(8):985–91. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1007/s11606-012-2037-1&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=22528615&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2024%2F09%2F28%2F2024.09.27.24314468.atom) 89. 89.Lee DJ, Avulova S, Conwill R, Barocas DA. Patient engagement in the design and execution of urologic oncology research. Urol Oncol. 2017 Sep;35(9):552–8. [PubMed](http://medrxiv.org/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2024%2F09%2F28%2F2024.09.27.24314468.atom) 90. 90.Hacker KE, Smith AB. Engaging stakeholders and patient partners. Surg Oncol Clin N Am. 2018 Oct;27(4):665–73. [PubMed](http://medrxiv.org/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2024%2F09%2F28%2F2024.09.27.24314468.atom) 91. 91.Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1977 Mar;33(1):159–74. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.2307/2529310&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=843571&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2024%2F09%2F28%2F2024.09.27.24314468.atom) [Web of Science](http://medrxiv.org/lookup/external-ref?access_num=A1977CY39700012&link_type=ISI) 92. 92.Nielsen J, Landauer TK. A mathematical model of the finding of usability problems. Proceedings of the SIGCHI conference on Human factors in computing systems - CHI ’93. New York, New York, USA: ACM Press; 1993. p. 206–13. 93. 93.Hennink M, Kaiser BN. Sample sizes for saturation in qualitative research: a systematic review of empirical tests. Soc Sci Med. 2022 Jan;292:114523. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1016/j.socscimed.2021.114523&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2024%2F09%2F28%2F2024.09.27.24314468.atom)