Understanding how the design and implementation of Online Consultations influence primary care outcomes: Systematic review of evidence with recommendations for designers, providers, and researchers ===================================================================================================================================================================================================== * Sarah Darley * Tessa Coulson * Niels Peek * Susan Moschogianis * Sabine N van der Veer * David C Wong * Benjamin C Brown ## Abstract **Background** Online consultations (OCs) allow patients to contact their care provider online, and have been promoted as a way to address increasing workload and decreasing workforce capacity in primary care. Globally, OCs have been rolled out rapidly due to policy initiatives and the COVID-19 pandemic, though there is a lack of evidence regarding how their design and implementation influence care outcomes. **Objective** Informed by existing theories, synthesise quantitative and qualitative research on: 1) outcomes of OCs in primary care; 2) how these are influenced by OC system design and implementation. **Methods** We searched Ovid Medline, Embase, Web of Science, Scopus, NTIS, HMIC, and ZETOC from 2010 to November 2021. We included quantitative and qualitative studies of real-world OC use in primary care, written in English, and published 2010 onwards. Quantitative data were transformed into qualitative themes. For objective 1 we used thematic synthesis informed by the Institute of Medicine’s domains of healthcare quality. For objective 2 we used Framework Analysis informed by the NASSS framework and Realistic Evaluation. Critical appraisal was conducted using the Mixed Methods Appraisal Tool and strength of evidence judged using GRADE-CERQual. **Results** We synthesised 62 studies (quantitative n=32, qualitative n=12, mixed methods n=18) in nine countries covering 30 unique OC systems, 13 of which used Artificial Intelligence (AI). Twenty-six were published in 2020 onwards, and 11 were post-COVID-19. There was no quantitative evidence for negative impacts of OCs on patient safety, and qualitative studies suggested perceptions of OC safety varied. Some participants believed OCs improved safety, particularly when patients could describe their queries using unstructured free-text. Staff workload decreased when sufficient resources were allocated to implement OCs, and patients used them for simple problems or could describe their queries using free-text. Staff workload increased when OCs were not integrated with other software or organisational workflows, and patients used them for complex queries. OC systems that required patients to describe their queries using multiple choice questionnaires (MCQs) increased workload for both them and staff. Health costs were reduced when patients used OCs for simple queries, and increased when used for complex ones. Patients using OCs were more likely to be female, younger, native speakers, with higher socioeconomic status than those not using OCs. However, OCs increased primary care access for patients with mental health conditions, verbal communication difficulties, and barriers to attending in-person appointments. Access also increased by providing a timely response to patients’ queries. Patient satisfaction increased when using OCs due to better primary care access, though could decrease when using MCQ formats. **Conclusions** This is the first theoretically-informed synthesis of research on OCs in primary care, and includes studies conducted during COVID-19. It contributes new knowledge that in addition to producing positive outcomes such as increased access and patient satisfaction, they can also have negative outcomes such as increased workload and costs. These negative outcomes can be mitigated by appropriate OC system design (e.g. free-text format), incorporating advanced technologies (e.g. AI), and integration into technical and organisational workflows (e.g. timely responses). **Study protocol** PROSPERO (CRD42020191802). Keywords * General practice * systematic review * remote consultation * online consultation * triage * primary healthcare ## Introduction Online consultation (OC) systems allow patients to contact their healthcare provider over the internet to ask health-related questions and report symptoms [1]. Their query may then be resolved either by a written response, telephone call, video consultation, or in-person visit. Communication between provider and patient through OC systems may therefore be synchronous (real-time e.g. via video) or asynchronous (questions and responses sent at different times e.g. via written message). Many terms are used to describe this type of technology including ‘e-consultation’, ‘e-visit’, and ‘online triage’ (Appendix 1) – in this review we refer to them all as ‘OCs’. We distinguish OCs from ‘symptom checkers’ [2] and other self-service systems that typically do not directly facilitate communication with a human healthcare provider, and from patient portals [3], which may include generic email or secure messaging functionality. OCs are considered by policymakers in many countries as a way to address the increasing workload and decreasing workforce capacity in primary care [4-9], whilst still meeting patient expectations and improving access [10]. However, they have the potential to exacerbate health inequities [11, 12], and increase inappropriate antibiotic prescriptions [13]. Furthermore, there are widely recognised challenges in initiating and sustaining adoption of new technologies in primary care [14]. Although symptom checkers (e.g. [2, 15]) and patient portals (e.g. [3, 16, 17]) have been well studied, only a small number of evidence syntheses directly relevant to OCs have been published: one systematic review of 57 articles on delivering ‘e-consultation’ in primary care largely focused on generic stand-alone applications such as email and video (n=39/57, 68%) [18]; a scoping review of ‘online triage tools’ included 13 papers, four of which (31%) were non-empirical (e.g. opinion pieces) [19]; and a review of 17 studies of ‘intelligent online triage tools’ focused only on those that use ‘artificial intelligence’ (AI) [20]. Since these syntheses were conducted, OCs have gained wider traction in clinical practice around the world: they have been indispensable in helping manage patients remotely to minimise the spread of COVID-19 [21, 22], and English primary care providers have been mandated to offer OCs for all patients since April 2020 [23]. Moreover, OC system product design has progressed significantly to become more specialised and technologically advanced [24], with several more empirical research studies published on their use (e.g. [25-47]). Given this rapid scale-up and increase in diversity and complexity of OCs, further insight is needed into their outcomes on healthcare. Previous reviews have not reported design or implementation details of the OCs they studied (e.g. [18-20]), despite their importance in understanding the causal mechanisms of how they impact outcomes [48]. In this paper, we systematically review and synthesise the empirical quantitative and qualitative literature in a theoretically-informed way to address this knowledge gap. ## Objectives Informed by existing theories, synthesise quantitative and qualitative research on: 1. Outcomes of OCs in primary care. 2. How these are influenced by OC system design and implementation. ## Methods ### Study design We consider OCs as complex interventions and therefore synthesised both quantitative and qualitative evidence to understand their impacts in specific contexts [49]. We did not perform a meta-analysis due to the heterogeneous and non-randomised nature of included studies [50]. We followed the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement [51]. ### Inclusion criteria Papers that met the following criteria were included: empirical studies employing quantitative and/or qualitative methods to examine the real-world use of OCs in primary care in any country, written in English, and published in 2010 or later. We excluded news articles, opinion pieces, literature reviews, non-English language articles, and literature published before 2010. We defined OCs as digital interventions that allow patients to contact their primary care provider by inputting ‘queries’ into healthcare-specific online forms [1]. We included symptom checkers and similar self-service systems (e.g. [29]) if at least one of their outcomes directly facilitated contact with a primary care health professional. We included patient portals if they had secure messaging functionality that used healthcare-specific forms (e.g. [29]). We excluded stand-alone generic communication technologies, such as email or video conferencing software. ### Search strategy We searched databases Ovid Medline, Embase, Web of Science, and Scopus during July 2020 (Appendix 2). Our search strategy was developed from our scoping searches of the literature, and drew on search strategies used in related literature reviews [18, 19]. We searched NTIS (National Technical Information Service), HMIC (Health Management Information Consortium), and ZETOC to find relevant grey literature, conference proceedings, and theses. We found further literature through citation mapping and reference lists of relevant papers, searching during August and September 2020. SD and TC independently screened titles and abstracts, then full papers for eligibility, resolving differences through discussion at each stage. All literature searches were re-run by SD in November 2021. ### Data extraction and quality appraisal We extracted data from included papers as verbatim text, capturing study characteristics (e.g. research design, study setting) and key findings relevant to our research objectives based on the NASSS framework [52] (Appendix 3). We used the NASSS in order to capture ‘a rich, contextualised narrative of technology-supported change efforts and the numerous interacting influences that help explain its successes, failures, and unexpected events’ [53]. Methodological quality of studies was assessed using the Mixed Methods Appraisal Tool (MMAT), which is designed for qualitative, quantitative, and mixed methods studies [54]. We scored each paper using recommended quintile percentages as cut-offs, and considered any paper scoring at least 60% as ‘good’ quality [55]. SD and TC extracted data for ten papers independently, which confirmed high inter-rater agreement. Following this, SD extracted data from the remaining papers, which were checked by TC. ### Data synthesis Data were imported into Nvivo (v12) [56] for synthesis. In order to integrate both quantitative and qualitative data, during data synthesis quantitative data were transformed into qualitative themes (‘qualitising’) [57]. For objective 1, we considered ‘outcomes’ as consequences of using OCs that could relate to patients, primary care staff, or the wider system [48]. We used thematic synthesis [58], which involved SD and TC coding the text from the data extraction forms independently line-by-line, developing higher level themes through regular discussion [58]. Outcomes were synthesised inductively, with emerging themes mapped to the six Institute of Medicine (IOM) domains of healthcare quality [59]: Safe (avoiding harm to patients from care that is intended to help), Effective (providing care based on scientific knowledge to produce better clinical outcomes), Patient-centred (care that is respectful and responsive), Timely (reducing waits and delays for those who receive and give care), Efficient (avoiding waste), and Equitable (care that does not vary in quality because of personal characteristics) [59]. Our emergent findings suggested OCs had both positive and negative outcomes, and therefore theme descriptions were edited to be neutral (e.g. Safe→Safety, Efficient→Efficiency). Findings for the Effective domain were rated as low confidence and are therefore not presented in our main results. For objective 2, we considered OC ‘design’ as material properties of an OC such as features and functionality [52], and ‘implementation’ as the way an OC had been introduced and used in a particular context [48]. As a design feature, we considered AI as the ability of machines to ‘mimic human intelligence as characterised by behaviours such as cognitive ability, memory, learning, and decision making’ [60]. We synthesised extracted data using Framework Analysis [61], which involved SD and TC reading and re-reading each data extraction form, then coding them line-by-line independently: both deductively using domains from the NASSS framework [52] for high-level themes, and inductively by identifying additional subthemes. Through discussion, SD and TC summarised the findings into five high-level themes: Condition (health condition, illness OC is used for), Technology (material properties of OC, required knowledge for use), Adopters (staff, patients, carers expected to use OC), Organisation (extent of work needed for implementation of OC, capacity, and readiness), and Wider System (policy context) [52]. Two NASSS domains, Value Proposition (value of OC to the developer, patients, and healthcare system) and Embedding and Adaptation over Time (learning and adaptation to changing contexts), had limited applicability to our findings and were not included in the final synthesis. Informed by Realistic Evaluation [48], we considered our themes as contextual factors, and identified patterns of explanations for how each led to the outcomes from objective 1 (i.e. ‘causal mechanisms’). We used visual mapping to identify commonalities and discordances in causal mechanisms – firstly within individual papers, then secondly across papers [61]. Where there were discordances, we explored potential explanations where possible (e.g. related to study setting). Strength and quality of our findings for both objective 1 and 2 was assessed using the GRADE-CERQual method [62]. This accounts for: methodological limitations of contributing papers (according to MMAT assessments); relevance to the review question; coherence of the finding; and adequacy of its supporting data [62]. Confidence in each finding was designated as either high, moderate, low, or very low. At each stage of analysis, findings were discussed and agreed with the wider study team. Author BCB reviewed all coded verbatim excerpts from papers included in the final synthesis. ## Results ### Descriptive summary We synthesised 62 papers (Figure 1), including 51 journal papers (e.g. [26]), seven evaluation reports (e.g.[63]), three conference papers (e.g. [64]), and one Master’s degree thesis [65]. Studies were quantitative (n=32, 52%), qualitative (n=12, 19%), and mixed methods (n=18, 29%), and analysed data from patients (16 qualitative, 17 quantitative studies), staff (22 qualitative studies, 9 quantitative studies), and clinical systems (32 quantitative studies). All were set in one of nine high-income countries, with the majority from the US (n=20) and UK (n=20; Appendix 4). Twenty-six (42%) studies were published in 2020 or later, and 11 studies (18%) were conducted after the start of the COVID-19 pandemic. Examples of excluded studies are those that focus on stand-alone video consultations (e.g. [66]), involved communication between physicians and not patients (e.g. [67]), and those not based in primary care (e.g. [68]). Fifty two studies reported levels of OC adoption by patients and staff, out of which 32 (62%; 52% of all studies) were described as ‘low’ by study authors (e.g. [69]). OCs were adopted at a high rate in 33 (63%; 53% of all) studies (e.g. [70]), including high rates of adoption by certain patient groups even when overall OC adoption in the study was low (e.g. [71]). Included papers described 30 OC systems summarised in Table 1 and detailed in Appendix 5. In 16 papers, the OC system was described sufficiently to meet our inclusion criteria but not enough detail to determine specific design features. Of the 30 OCs described, the majority (n=23, 77%) offered asynchronous two-way written communication between patients and staff i.e. text-based messages sent at different times (e.g. [72]), with a few (n=4, 13%) also offering synchronous real-time communication by video (e.g. [25]). No OCs provided synchronous text-based communication. Four (13%) did not provide functionality for staff to reply to patients via the system (i.e. one-way communication only e.g. [71]). Eleven (37%) required patients to describe their queries solely via multiple choice questionnaires (MCQs; e.g. [73]) compared to four (13%) that solely required patients to describe their queries using unstructured free-text (e.g. [31]). Twelve (40%) had a hybrid approach of primarily using MCQs with the option for patients to enter additional free-text (e.g.[74]). No free-text OCs offered optional MCQs. Eight (27%) OC systems integrated with the electronic health record (EHR; e.g.[33]) and one (3%) allowed patients to schedule telephone or in-person appointments with health care professionals themselves (e.g. [29]). View this table: [Table 1:](http://medrxiv.org/content/early/2022/02/21/2022.02.21.22271185/T1) Table 1: OC Features Thirteen MCQ-based OC systems exhibited three types of AI: 1) adapting the questions they asked patients as they submitted their query in response to their answers so far (n=10, 33%; e.g. [75]); 2) prioritising patient queries based on clinical urgency (n=4, 13%; e.g. [29]); and 3) signposting patients to an appropriate care provider based on their query, such as self-care, primary care, or emergency department (n=3, 10%; e.g. [43]). These were mostly powered by pre-programmed logic and “algorithms” (n=10, 33%; e.g. [29]), with the exact AI methodology unclear in the remainder (n=3, 10%; e.g. [76]). The methodological quality of most papers (n=41, 66%) was ‘good’ (i.e. 60% or above according to MMAT [77]; Appendix 6). Common limitations included lack of detail on whether the OC was administered as intended (e.g. [78]), and small sample sizes (e.g. [28]). ### Synthesis To maintain readability we present only moderate and high confidence findings, and provide only one example reference per finding. Tables 2 and 3 provide all references, and specify whether findings are qualitative or quantitative. Appendices 7 and 8 detail low confidence findings. Appendices 9 and 10 provide exemplar data. View this table: [Table 2.](http://medrxiv.org/content/early/2022/02/21/2022.02.21.22271185/T2) Table 2. Outcomes of Online Consultations in primary care View this table: [Table 3.](http://medrxiv.org/content/early/2022/02/21/2022.02.21.22271185/T3) Table 3. How outcomes of Online Consultations in primary care are influenced by system design and implementation #### Objective 1: Outcomes of Online Consultations in primary care ##### Safety In 17 studies, staff and patients expressed general concerns about the impact of OCs on patient safety, particularly around the potential loss of information from patients versus in-person or telephone consultations, and how it could lead to misdiagnosis (e.g. [30]). However, quantitative evidence from 11 studies did not support these concerns in terms of Emergency Department attendance rates (e.g. [78]), hospitalisations (e.g. [79]), deaths (e.g. [72]), and other measures (e.g. [34]). Furthermore, clinicians and patients in 14 studies believed OCs improved patient safety, for example by producing a detailed shared written record of consultations (e.g. [80]) and helping to reduce the spread of communicable diseases such as COVID-19 (e.g. [41]). ##### Timeliness In 29 studies, OCs were perceived to increase access to primary care services. It was easier and more convenient to make initial contact because patients could submit an OC query at any time without waiting on the phone or attending in-person (e.g. [71]). Once a query was submitted, patients also communicated with health professionals sooner because OCs tended to circumvent the traditional appointment-booking process (e.g. [32]). ##### Efficiency In total, 33 papers suggested workload decreased for both staff and patients when using OCs. Patient queries were written rather than spoken, incoming phone calls to receptionists reduced (e.g. [81]), and patient histories did not need manual documentation (e.g. [80]). Written queries were usually more detailed than when communicated verbally, and were received by health care staff asynchronously, thus providing opportunities for more objective examination and more effective triage. Consequently, patient queries could more often be directed to other services or dealt with by other staff members, rather than always physicians (e.g. [28]). Combined with their remote nature, OCs also gave staff more autonomy over how their work was organised, thus providing efficiency gains such as working from home and control over how to contact a patient rather than defaulting to an in-person consultation (e.g. [65]). When telephone or in-person consultations were necessary, they were more focused and therefore quicker because the staff member could read the patient query prior to contact (e.g. [82]). OCs reduced workload for patients by avoiding the need to telephone their primary care provider to make an appointment, which often entailed long queues (e.g. [83]), and avoiding in-person consultations when possible, which typically involved travel, waiting rooms, and organising time off work and childcare (e.g. [76]). In contrast, 29 studies suggested OCs increased workload for staff and patients. Staff described conducting OCs on top of their usual tasks (e.g. [65]) and dealing with them outside normal working hours (e.g. [84]). They believed that because OCs increased access to primary care, patients sought help more readily than they would have previously (e.g. [82]), thus creating ‘supply-induced demand’ [85]. Processing OCs also created new administrative work, such as filing them to EHRs and deciding whether or not they required input from a clinician (e.g. [69]). Workload could also increase for patients if they perceived that entering their query into the OC system was more difficult than explaining them verbally (e.g. [86]). OCs decreased costs for providers in 20 studies largely by reducing in-person visits, which have associated expenditures related to staffing and utilities (e.g. [87]). Patients reported that due to their convenience, having access to OCs stopped them from visiting other costly unscheduled care providers (e.g. [78]). OCs decreased costs for patients in four studies by avoiding in-person visits, which may entail expenses related to travel, unpaid work leave, and childcare (e.g. [32]). In contrast, OCs increased costs for providers in nine studies due to associated technology costs (e.g. [41]), time required for clinicians to triage patient queries (e.g. [88]), and insufficient reduction of in-person visits or telephone consultations (e.g. [70]). ##### Equitable Forty studies suggested that OCs decreased equitable access to care services because their use varied according to patient characteristics (e.g. [41]). Conversely, 19 studies suggested that OCs increased equitable access because they helped particular groups of patients communicate with their primary care providers who had previously struggled (e.g. [71]). These characteristics are discussed in more detail in the Adopters section below. ##### Patient-centredness Although 13 studies uncovered some patient dissatisfaction with OCs (e.g. [74]), 31 studies found most patients were at least as satisfied with OCs, or were more satisfied with them than traditional in-person appointments (e.g. [27]). Patients liked OCs for the reasons described above: they improved access (Timeliness), reduced their workload and costs (Efficiency), and helped particular groups of patients communicate with their care providers (Equitable). #### Objective 2: How outcomes of Online Consultations in primary care are influenced by system design and implementation ##### Condition Eleven studies suggested that OCs decreased staff workload when used for simple queries that were straightforward to resolve because they were more amenable to completion without needing to contact the patient directly via telephone or in-person (e.g. [38]). Simple queries included those related to administrative tasks, new and recurrent minor acute illnesses, prescriptions, tests, requests for advice, follow-up, and some chronic condition reviews (e.g. [31]). These queries also decreased health costs because they saved clinicians time, for example when administrative staff were able to relay messages and there was no direct contact between doctor and patient (e.g. [89]). Seven studies suggested that OCs increased staff workload and costs when used for complex queries, such as those with multiple ill-defined symptoms (e.g. [82]). These queries generally required verbal dialogue with, and physical examination of the patient, and were usually converted to telephone and/or in-person consultations to assess the patient further (e.g. [89]). Staff felt this duplicated the number of contacts with the patient for the same query. ##### Technology Thirteen studies showed that when patients had to use MCQs to input their OC query it increased both patient and staff workload. Filling out long lists of questions shifted work from the clinician to the patient (e.g. [86]), and staff found them burdensome to read (e.g. [69]). MCQs limited the amount of detail patients could enter so staff could not always fully understand their request. This increased workload because they often had to contact the patient to get further information (e.g. [89]). MCQs also asked questions about seemingly ‘irrelevant’ symptoms, which staff were responsible to assess and follow up, diverting attention away from the patient’s primary concern (e.g. [45]). Due to their restrictive nature, patients regularly adapted their responses to MCQs to get the outcome they wanted, even when it was not the most appropriate use of resources. For example, reporting their symptoms differently to get an in-person consultation when self-care may have been more suitable (‘gaming’; e.g. [82]). Nine studies suggested that MCQs could also decrease patient satisfaction. Reasons included the amount of work required to complete them (e.g. [71]), their inflexibility in getting the answers patients wanted from their primary care provider (e.g. [44]), and that they could be confusing to navigate (e.g. [91]). In contrast, seven papers suggested that when patients could primarily report their queries using unstructured free-text, it decreased staff workload and increased patient safety. This was because patients were more able to fully describe their query in sufficient detail using their own words, and clinicians did not have to request further information as often (e.g. [94]). In six studies, asynchronous two-way written communication within the OC decreased workload for both staff and patients. The ability to reply to patients in writing meant queries could be answered and follow-up questions asked at times convenient to both staff and patients, avoiding lengthy telephone and in-person consultations when appropriate (e.g. [30]). It was also easier to communicate complex information, for example by sending educational materials or using pre-set message templates (e.g. [94]). Thirteen papers highlighted that a lack of integration between the OC system and other core software used by providers increased staff workload. Non-integration meant staff had to go through multiple steps to perform a task, such as when filing an OC to a patient’s EHR (e.g. [87]). ##### Adopters Patients using OCs were more likely to be female (n=26, e.g. [79]), younger (n=26, e.g. [75]), a native speaker of the official language in the country they live (n=7, e.g. [91]), and have higher socioeconomic status (n=11, e.g. [32]) than those not using OCs, thus decreasing equity. In contrast, both staff and patients felt that OCs increased access for particular groups of patients who struggled with traditional consultation methods, thus increasing equity and their satisfaction with care. This included patients with: mental health conditions who became anxious when speaking to health professionals on the telephone or in-person (n=8, e.g. [86]); verbal communication difficulties such as hearing loss, who found it easier to communicate in writing (n=7, e.g. [74]); and barriers to attending in-person appointments due to physical disabilities, geography, work commitments, or care responsibilities (n=8, e.g. [89]). Eight papers suggested that when staff and patients viewed traditional in-person methods as the gold standard, it could lead to resistance in adopting OCs (e.g. [84]). ##### Organisation Five papers found that when OCs were minimally advertised to patients, it led to low rates of adoption (e.g. [90]). Seven papers also showed that responding to a patient’s initial OC query quickly led to high patient satisfaction because it provided an advantage over traditional methods of primary care contact (e.g. [39]); by definition this also increased primary care access. Thirteen papers found that staff workload was increased when providers did not integrate OCs into their normal daily workflows. For example, not scheduling time for clinicians to deal with OCs meant they were done in addition their normal tasks (e.g. [80]), and not diverting all incoming patient demand via the OC meant different communication routes were often used for the same issue thereby duplicating work (e.g. [38]). Eight papers suggested that provider workload decreased if sufficient resources were allocated to implementing OCs. This included their initial set-up, for example training to enable staff to more effectively handle OCs (e.g. [76]), and their ongoing processing, for example dedicated facilities such as quiet rooms to help staff respond to OCs without distraction (e.g. [30]). Five papers showed that a lack of continuity between patients and their known doctor impacted negatively on patient satisfaction. This occurred when any doctor could reply to an OC query and patients were not able to specify a doctor to whom to address their query (e.g. [47]). ##### Wider system Six papers showed that government policies mandating OC usage increased their adoption. Example policies aimed to increase digital modes of contact with primary care in general (e.g. [70]), and to minimise in-person contact during the COVID-19 pandemic (e.g. [41]). Five papers demonstrated that a lack of long-term external financial support for OCs limited their sustainability, as healthcare organisations could often not afford to pay their ongoing costs (e.g. [89]). ## Discussion ### Summary of evidence We synthesised qualitative and quantitative evidence from 62 studies in nine countries covering 30 OC systems described in detail, with wide-ranging functionalities including AI. Twenty-six studies were published 2020 onwards and 11 were post COVID-19. Overall adoption of OCs by patients was generally low. Although concerns were raised about patient safety when using OCs, there was no quantitative evidence to support them. Some clinicians and patients believed OCs improved safety, particularly when patients could describe their queries using unstructured free-text. Staff workload decreased when sufficient resources were allocated to implement OCs, and asynchronous two-way written messages to patients could be sent. Workload also decreased when patients used OCs for simple problems or could fully describe their queries using free-text. Staff workload increased when OCs were not integrated with other software or organisational workflows, and when patients used them for complex queries that ultimately required telephone or in-person consultations. OC systems that required patients to describe their queries using multiple choice questionnaires (MCQs) increased workload for both them and staff due to their inflexibility and often irrelevant questions. Health costs were reduced when patients used OCs for simple queries, and increased when used for complex ones. Patients using OCs were more likely to be female, younger, native speakers, with higher socioeconomic status than those not using OCs, thus decreasing equity. However, OCs increased primary care access for patients who previously struggled with traditional consultation methods, such as those with mental health conditions, verbal communication difficulties, and barriers to attending in-person appointments, thus increasing equity. Access was further increased by providing a timely response to patients’ queries. Patient satisfaction increased when using OCs due to better primary care access, though could decrease when using MCQ formats. Patient satisfaction also decreased when OCs did not support continuity of care. OCs were less likely to be adopted if they were not promoted to patients, if prospective users viewed traditional in-person methods as the gold standard, and if there was a lack of long-term external financial support. Government policies mandating OC usage increased adoption. ### Comparison of findings with other reviews Consistent with previous reviews relevant to OCs, we found a limited demographic of patients using OCs leading to potential inequitable care [18, 19]. We also found that studies often did not sufficiently explore patients’ perspectives of OCs in-depth [19]; only nine studies (15%) used interview-based methods and had an average sample size of 23. This hampered efforts to understand how such inequities arose. Contrary to previous reviews we found that OC outcomes are more complex and nuanced than previously reported [18-20]. For example, we identified mixed findings on their impact on workload, patient satisfaction, and equitable care. This contrasts with previous reviews where OCs only increased [20] or had no impact [18] on workload, decreased patient safety [18, 20], and increased inequity [18-20]. These new findings for OCs may be partly explained because 47 (76%) of our included studies had not been covered by these prior reviews. Although there was some overlap of papers (7/57 [18], 7/13 [19], 4/17 [20]), the majority did not meet our inclusion criteria as they were either non-empirical (4/57 [18], 4/13 [19], 4/17 [20]), pre-2010 (26/57 [18], 2/17 [20]), not based in real-world primary care (16/57 [18], 1/13 [19], 6/17 [20]), or did not meet our functional definition of an OC (39/57 [18], 2/13 [19], 6/17 [20]; e.g. symptom checkers with no link to a health professional [104]). By focusing on design and implementation we have identified new ways in which OCs influence primary care outcomes. For example, we found that by increasing access OCs can increase staff workload by creating ‘supply-induced demand’ [85] (e.g. [82]), and that they can decrease workload by enabling more focused consultations (e.g. [82]). Furthermore, because previous reviews often did not analyse the design or implementation of OCs [18-20], we have identified influential factors that have not previously been described. For example, whilst some reviews identified increased workload when clinicians received insufficient patient information via an OC system (e.g. [19]) we found this was particularly associated with MCQ-based OCs (e.g. [89]). We also identified that allowing patients to describe their queries using unstructured free-text had the opposite effect (e.g. [94]), whilst also having a positive impact on patient safety (e.g. [30]). ### Strengths and limitations As evidenced by the range of examples in Appendix 1, we adopted a fundamental functional definition of OCs rather than relying on the name given to them by authors of included studies. When combined with our comprehensive searches across multiple databases and inclusion of grey literature, we have identified more empirical studies relevant to OCs than any previous evidence synthesis on the topic [18-20]. Combined with our focus on causal mechanisms, this has helped us develop a new and theoretically informed understanding of OCs that has not previously been reported. As in all systematic reviews, our synthesis is reliant on what study authors reported. OC features were not always described in sufficient detail to understand how they impacted outcomes (e.g. [40]). There was also a lack of patient perspective in the studies, particularly from OC non-users (e.g. [37]). We made our literature search strategy as inclusive as possible regarding different terms used for OCs (Appendix 1), though due to their wide-ranging nature it is possible that some papers were missed. We updated our searches in November 2021 in order to capture more recently published studies, although due to time constraints only one author (SD) screened these newer papers. This enabled us to capture studies conducted in the context of COVID-19 (n=11, 18% of all included studies). ### Implications for practice and research Our findings show that OC outcomes are complex, and can be influenced by subtle ways in which they are designed and implemented. To maximise their benefit for patients and staff, we therefore provide recommendations for: OC developers on how systems could be designed; healthcare organisations for how they can be implemented and used; and researchers on questions and areas for further investigation. They are discussed below under the high-level themes from objective 2, and summarised in Table 4. View this table: [Table 4:](http://medrxiv.org/content/early/2022/02/21/2022.02.21.22271185/T4) Table 4: Implications for Online Consultation research and practice #### Condition We found that OCs worked more efficiently when used for simple rather than complex patient queries. It is unclear whether OCs are unsuitable for these conditions, or whether workflows and procedures can be better organised, and OC systems better designed to deal with them. As it currently stands, we recommend that complex conditions should be routed through traditional consultation methods (e.g. in-person, telephone), and further research conducted into how these types of conditions could be better handled via OCs to ensure they benefit all patients. One way in which OC designers could help is by highlighting to healthcare providers when patient queries may be unsuitable for resolution via an OC, so they can be dealt with appropriately. Using the IOM care quality domains to synthesise OC outcomes highlighted an evidence gap for ‘Effectiveness’, which received only low confidence ratings (Appendix 7). This domain refers to care based on scientific knowledge to determine whether an intervention produces better outcomes than alternatives [59]. Although we found evidence of OC impacts on quantitative measures of patient satisfaction (e.g. [27]) and patient safety (e.g. [72]), it is unknown if OCs impact clinical outcomes, such as delivery of evidence-based treatment, or rates of morbidity, and should be investigated further. #### Technology We found OCs that allowed patients to fully describe their queries using unstructured free-text rather than via MCQs decreased workload and increased patient safety (e.g. [94]), whereas MCQs could increase both staff and patient workload (e.g. [45]), and decrease patient satisfaction (e.g. [44]). OC developers should therefore provide this functionality, whilst at the same time guiding and supporting patients to provide sufficient detail for their primary care provider to quickly and safely respond to their query. Similarly, we found that OCs with asynchronous two-way written messages decreased workload (e.g. [30]), whereas those that did not integrate with existing core clinical software systems increased workload (e.g. [87]). Where possible, OC developers should therefore also incorporate these designs. Technology design also has a role in mitigating some of the undesirable outcomes we identified from using OCs, including increasing workload and costs. Increased workload is particularly important because it can lead to a mismatch between patient demand and healthcare resource, which can in turn threaten patient safety if providers are unable to deal with OCs in an appropriate timeframe. One way this could happen is through increased demand: if there are too many OCs submitted by patients and not enough staff to deal with them (e.g. [30]). Whether this additional demand is supply-induced [85] or previously unmet (and now unmasked) need was unclear from the studies we included (e.g. [76]) and requires further research. Nevertheless, OC systems could help by: supporting patients to self-care or signposting them to other services when appropriate; matching capacity to demand by limiting the number of OC queries primary care providers can receive from patients; supporting workflow, for example by determining whether OCs require clinical input or not to relieve workload on administrators (e.g. [69]); assist in triaging patient queries to reduce associated costs of solely relying on clinicians to triage (e.g. [88]); and highlighting when patients may require an in-person appointment to facilitate direct booking to avoid work duplication (e.g. [89]). According to our definition [60], many of these functions may require a form of AI to be most effective, which should be explored by OC designers (Figure 2). Thirteen MCQ-based OC systems in our review used AI (Table 1, e.g. [29]), though mainly for adapting the questions they asked patients during query submission, rather than the functions described above. Furthermore, AI was usually not the focus of the studies, and we consequently found only low confidence evidence regarding its use in OCs (Appendix 8). Therefore, how AI could be used by OC systems in clinical practice requires further research. Included papers did not always adequately describe the OC systems they studied, limiting our ability to determine how specific features influenced observed outcomes. Future research should describe OC systems in detail so that evaluation findings can be usefully compared, for example by using the template for intervention description and replication (TIDieR) checklist [105]. #### Adopters OCs offer advantages such as convenience and access to timely care to some patient groups, but for the technology to be equitable it must benefit all patients. Similar to related reviews [18, 19], we found patient characteristics (age, gender, native language, and socioeconomic status) to be important factors in OC adoption. Qualitative evidence contributed to these findings (e.g. [89]), however as described above, there was insufficient exploration of participant (especially patient) experiences to confidently explain why inequitable care arose, or how it could be overcome. Study authors and healthcare staff speculated reasons such as older patients may be less adept at using modern technology (e.g. [83]), however we were unable to formulate evidence-based hypotheses. Future research should therefore explore perspectives from patients using (and not using) OCs from a wide range of backgrounds using in-depth qualitative techniques such as interview-based methods. Patients from a variety of backgrounds should also be involved in how OC systems are designed, and help plan how they are implemented in practice. In contrast, we found that OCs increased access for patients who traditionally struggled to access primary care, such as those with mental health conditions and verbal communication difficulties. These should be viewed as benefits and promoted as such to health care providers and patients. Further research should explore other patient groups that could specifically benefit from OCs. Staff and patients resisted adopting OCs when they viewed traditional in-person consultation methods as the gold standard. Although this was understandable for complex queries (e.g. [82]), it was unclear whether other factors also influenced this view. Future research should address this evidence gap, particularly since COVID-19 has made remote consultations more commonplace [22]. In the meantime, this perception could be challenged by explaining the benefits of OCs found in our review to prospective users [106]. #### Organisation In order for patients and staff to experience the benefits of OCs, they must be widely promoted to patients as a route for them to contact their primary care provider. This can happen through various channels, such as mail out campaigns (e.g. via SMS) or by verbally mentioning OCs when in contact with patients (e.g. when receptionists speak to patients on the telephone). To minimise workload associated with OCs, organisations should allocate sufficient resources to both setting up and processing them. This includes providing adequate training on how to use OCs, and ensuring there are enough staff and facilities (e.g. computers and rooms) to deal with them. They should integrate OCs into their daily workflows by diverting as much incoming patient demand as possible through the system (e.g. by advising patients who telephone to use the OC system instead) and scheduling time for staff to deal with them. Having one communication channel avoids duplication and increases the proportion of patient contacts that benefit from the advantages of OCs. Traditional ‘appointments’ or slots in clinicians’ schedules could be reserved for OCs to give them adequate time to respond and ensure they do not become additional tasks to complete on top of their normal work. This has the added benefit of reducing costs by replacing other more expensive forms of consultation, such as in-person appointments. OC system developers can help by providing staff and patients the functionality to ‘book’ into these schedules automatically. We found little exploration of this approach in our included studies, and it therefore requires further research as to how it could be implemented most effectively. Our findings show that providers can increase access and patient satisfaction by responding quickly to OCs, though definitions of what this involved were unclear. We recommend providing an initial response to patients’ OC queries as soon as possible on the same day – either by asynchronous written message or telephone call. This does not mean the entire query needs to be resolved at this point, just that initial contact has been made and the query acknowledged. The OC can then be booked into a clinician’s schedule (if necessary) to fully deal with it as described above, whilst considering competing clinical priorities and available capacity. We included studies from nine countries, all of which were developed western countries. Due to their remote nature, OCs may have a role in developing countries where there are isolated communities and fewer health care staff per head of population. However, further research is required to understand how their technological and financial barriers could be overcome. #### Wider system Governmental policies to promote OCs are effective at increasing adoption, though centralised funding is needed to sustain their use. It is unclear what the long-term experience of such policies are from the papers we included, particularly in response to those relating to the COVID-19 pandemic. ## Conclusions This is the first theoretically-informed synthesis of empirical research on OCs in primary care, and uniquely includes studies conducted during COVID-19. It contributes new knowledge that OCs are safe and can have positive outcomes, such as increased access to primary care and decreased patient costs. However, they are also complex and often produce conflicting outcomes related to provider costs, workload, patient satisfaction, and equitable care. Some of these are unintended, and conflict with the promotion of OCs by policymakers as a way to address already increasing workload and decreasing workforce capacity in primary care [4-9]. Unlike previous evidence syntheses on the topic, we have shown that negative OC outcomes can be mitigated by appropriate OC system design (e.g. free-text vs MCQ formats, asynchronous two-way communication), incorporating advanced technologies (e.g. AI), and integration into technical (e.g. software) and organisational (e.g. timely responses) workflows. Since the advent of COVID-19, OCs have become indispensable, though further engineering and implementation research is required to realise their full benefits. ## Registration and protocol The study protocol was registered with PROSPERO (CRD42020191802) [107]. The original title was amended to be less general and more specific to the objectives of the review. Objective 2 was amended to include all identified outcomes, rather than limited to the ones originally specified. ## Supporting information Figure 1 [[supplements/271185_file03.pptx]](pending:yes) Figure 2 [[supplements/271185_file04.pptx]](pending:yes) Appendix 1 [[supplements/271185_file05.docx]](pending:yes) Appendix 2 [[supplements/271185_file06.docx]](pending:yes) Appendix 3 [[supplements/271185_file07.docx]](pending:yes) Appendix 4 [[supplements/271185_file08.docx]](pending:yes) Appendix 5 [[supplements/271185_file09.docx]](pending:yes) Appendix 6 [[supplements/271185_file10.docx]](pending:yes) Appendix 7 [[supplements/271185_file11.docx]](pending:yes) Appendix 8 [[supplements/271185_file12.docx]](pending:yes) Appendix 9 [[supplements/271185_file13.docx]](pending:yes) Appendix 10 [[supplements/271185_file14.docx]](pending:yes) ## Data Availability All papers reviewed in our systematic review are publicly available. ## Conflicts of Interest BCB is clinical lead for a commercially available OC system ([www.patchs.ai](http://www.patchs.ai)). ## Acknowledgements This research was funded by Innovate UK [105178] and a Wellcome Trust Clinical Research Career Development Fellowship for BCB [209593/Z/17/Z]. NP’s time was partially funded by the National Institute for Health Research Greater Manchester Patient Safety Translational Research Centre (NIHR Greater Manchester PSTRC). The views expressed are those of the authors and not necessarily those of the NHS, the NIHR, or the Department of Health and Social Care. NIHR had no role in study design, data collection, data analysis, data interpretation, or writing of the report. The corresponding author had full access to all of the data and the final responsibility to submit for publication. SD, TC, and BCB refined the research question; developed the search strings; conducted screening, critical appraisal, data extraction and data analysis; and wrote the first draft of the manuscript. NP contributed to the conception and design of the review. All authors contributed to the final analysis, and approved the final submitted version of the manuscript. ## Abbreviations AI : Artificial Intelligence EHR : Electronic Health Record GRADE-CERQual : Grading of Recommendations Assessment, Development, and Evaluation - Confidence in the Evidence from Reviews of Qualitative research IOM : Institute of Medicine MMAT : Mixed Methods Appraisal Tool NASSS : Non-adoption, abandonment, scale-up, spread, sustainability (framework) MCQ : Multiple choice questionnaire NHS : National Health Service OCs : Online Triage and Consultation systems PRISMA : Preferred Reporting Items for Systematic Reviews and Meta-Analyses PROSPERO : The International Prospective Register of Systematic Reviews UK : United Kingdom US : United States * Received February 21, 2022. * Revision received February 21, 2022. * Accepted February 21, 2022. * © 2022, Posted by Cold Spring Harbor Laboratory This pre-print is available under a Creative Commons License (Attribution 4.0 International), CC BY 4.0, as described at [http://creativecommons.org/licenses/by/4.0/](http://creativecommons.org/licenses/by/4.0/) ## References 1. 1.Bakhai M, Atherton H. How to conduct written online consultations with patients in primary care. BMJ. 2021;372:n264. doi: 10.1136/bmj.n264. [FREE Full Text](http://medrxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiRlVMTCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiYm1qIjtzOjU6InJlc2lkIjtzOjE3OiIzNzIvZmViMjRfMTAvbjI2NCI7czo0OiJhdG9tIjtzOjUwOiIvbWVkcnhpdi9lYXJseS8yMDIyLzAyLzIxLzIwMjIuMDIuMjEuMjIyNzExODUuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 2. 2.Chambers D, Cantrell A, Johnson M, Preston L, Baxter SK, Booth A, et al. Digital and online symptom checkers and health assessment/triage services for urgent health problems: systematic review. BMJ Open. 2019;9(8):e027743. doi: 10.1136/bmjopen-2018-027743. [Abstract/FREE Full Text](http://medrxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NzoiYm1qb3BlbiI7czo1OiJyZXNpZCI7czoxMToiOS84L2UwMjc3NDMiO3M6NDoiYXRvbSI7czo1MDoiL21lZHJ4aXYvZWFybHkvMjAyMi8wMi8yMS8yMDIyLjAyLjIxLjIyMjcxMTg1LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 3. 3.Antonio MG, Petrovskaya O, Lau F. The State of Evidence in Patient Portals: Umbrella Review. J Med Internet Res. 2020 2020/11/11;22(11):e23851. doi: 10.2196/23851. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.2196/23851&link_type=DOI) 4. 4.NHS. The NHS Long Term Plan. 2019 [updated 7 January 2019; cited 2021 1 March]; Available from: [https://www.longtermplan.nhs.uk/publication/nhs-long-term-plan/](https://www.longtermplan.nhs.uk/publication/nhs-long-term-plan/). 5. 5.The Royal New Zealand College Of General Practitioners. Telehealth and technology-based health services in primary care. 2017 [updated November 2017; cited 2021 1 March]; Available from: [https://www.rnzcgp.org.nz/gpdocs/New-website/Advocacy/Position-Statements/Telehealth-and-technology-based-health-services-in-primary-care-updatedpdf](https://www.rnzcgp.org.nz/gpdocs/New-website/Advocacy/Position-Statements/Telehealth-and-technology-based-health-services-in-primary-care-updatedpdf). 6. 6.Gill M. A National Telehealth Strategy For Australia – For Discussion. Canberra: Australian National Consultative Committee on Electronic Health; 2011 [cited 2021 1 March]; Available from: [https://www.who.int/goe/policies/countries/aus\_\_support\_tele.pdf](https://www.who.int/goe/policies/countries/aus__support_tele.pdf). 7. 7.Daniel H, Snyder Sulmasy L. Policy Recommendations to Guide the Use of Telemedicine in Primary Care Settings: An American College of Physicians Position Paper. Ann Intern Med. 2015;163(10):787–9. doi: 10.7326/m15-0498%m26344925. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.7326/M15-0498&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=26344925&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2022%2F02%2F21%2F2022.02.21.22271185.atom) 8. 8.Hobbs FDR, Bankhead C, Mukhtar T, Stevens S, Perera-Salazar R, Holt T, et al. Clinical workload in UK primary care: a retrospective analysis of 100 million consultations in England, 2007– 14. The Lancet. 2016 2016/06/04/;387(10035):2323–30. doi: [https://doi.org/10.1016/S0140-6736(16)00620-6](https://doi.org/10.1016/S0140-6736(16)00620-6). [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1016/S0140-6736(16)00620-6&link_type=DOI) 9. 9.Marchand C, Peckham S. Addressing the crisis of GP recruitment and retention: a systematic review. Br J Gen Pract. 2017;67(657):e227. doi: 10.3399/bjgp17X689929. [Abstract/FREE Full Text](http://medrxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NDoiYmpncCI7czo1OiJyZXNpZCI7czoxMToiNjcvNjU3L2UyMjciO3M6NDoiYXRvbSI7czo1MDoiL21lZHJ4aXYvZWFybHkvMjAyMi8wMi8yMS8yMDIyLjAyLjIxLjIyMjcxMTg1LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 10. 10.Beaney P, Odulaja A, Hadley A, Prince C, Chambers R. GP Online: turning expectations into reality with the new NHS app. Br J Gen Pract. 2019;69(681):172–3. doi: [https://doi.org/10.3399/bjgp19X701333](https://doi.org/10.3399/bjgp19X701333). [FREE Full Text](http://medrxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiRlVMTCI7czoxMToiam91cm5hbENvZGUiO3M6NDoiYmpncCI7czo1OiJyZXNpZCI7czoxMDoiNjkvNjgxLzE3MiI7czo0OiJhdG9tIjtzOjUwOiIvbWVkcnhpdi9lYXJseS8yMDIyLzAyLzIxLzIwMjIuMDIuMjEuMjIyNzExODUuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 11. 11.Parker RF, Figures EL, Paddison CA, Matheson JI, Blane DN, Ford JA. Inequalities in general practice remote consultations: a systematic review. BJGP Open. 2021:BJGPO.2021.0040. doi: 10.3399/bjgpo.2021.0040. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.3399/bjgpo.2021.0040&link_type=DOI) 12. 12.Joy M, McGagh D, Jones N, Liyanage H, Sherlock J, Parimalanathan V, et al. Reorganisation of primary care for older adults during COVID-19: a cross-sectional database study in the UK. Br J Gen Pract. 2020;70((697)):e540–e7. doi: 10.3399/bjgp20X710933. [Abstract/FREE Full Text](http://medrxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NDoiYmpncCI7czo1OiJyZXNpZCI7czoxMToiNzAvNjk3L2U1NDAiO3M6NDoiYXRvbSI7czo1MDoiL21lZHJ4aXYvZWFybHkvMjAyMi8wMi8yMS8yMDIyLjAyLjIxLjIyMjcxMTg1LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 13. 13.Han SM, Greenfield G, Majeed A, Hayhoe B. Impact of Remote Consultations on Antibiotic Prescribing in Primary Health Care: Systematic Review. J Med Internet Res. 2020 2020/11/9;22(11):e23482. doi 10.2196/23482. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.2196/23482&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2022%2F02%2F21%2F2022.02.21.22271185.atom) 14. 14.Cresswell K, Sheikh A. Organizational issues in the implementation and adoption of health information technology innovations: An interpretative review. Int J Med Inform. 2013 2013/05/01/;82(5):e73–e86. doi: [https://doi.org/10.1016/j.ijmedinf.2012.10.007](https://doi.org/10.1016/j.ijmedinf.2012.10.007). [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1016/j.ijmedinf.2012.10.007&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=23146626&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2022%2F02%2F21%2F2022.02.21.22271185.atom) 15. 15.Abd-Alrazaq A, Rababeh A, Alajlani M, Bewick B, Househ M. Effectiveness and Safety of Using Chatbots to Improve Mental Health: Systematic Review and Meta-Analysis. J Med Internet Res. 2020;22(7). doi: 10.2196/16021. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.2196/16021&link_type=DOI) 16. 16.Kruse CS, Bolton K, Freriks G. The Effect of Patient Portals on Quality Outcomes and Its Implications to Meaningful Use: A Systematic Review. J Med Internet Res. 2015 2015/02/10;17(2):e44. doi: 10.2196/jmir.3171. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.2196/jmir.3171&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=25669240&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2022%2F02%2F21%2F2022.02.21.22271185.atom) 17. 17.Irizarry T, DeVito Dabbs A, Curran CR. Patient Portals and Patient Engagement: A State of the Science Review. J Med Internet Res. 2015 2015/06/23;17(6):e148. doi: 10.2196/jmir.4255. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.2196/jmir.4255&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=26104044&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2022%2F02%2F21%2F2022.02.21.22271185.atom) 18. 18.Mold F, Hendy J, Lai Y-L, de Lusignan S. Electronic Consultation in Primary Care Between Providers and Patients: Systematic Review. JMIR Med Inform. 2019 2019/12/3;7(4):e13042. doi: 10.2196/13042. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.2196/13042&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2022%2F02%2F21%2F2022.02.21.22271185.atom) 19. 19.Baines R, Tredinnick-Rowe J, Jones R, Chatterjee A. Barriers and Enablers in Implementing Electronic Consultations in Primary Care: Scoping Review. J Med Internet Res. 2020;22(11):e19375. PMID: 33035177. doi: 10.2196/19375. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.2196/19375&link_type=DOI) 20. 20.Gottliebsen K, Petersson G. Limited evidence of benefits of patient operated intelligent primary care triage tools: findings of a literature review. BMJ Health Care Inform. 2020;27(1):e100114. doi: 10.1136/bmjhci-2019-100114. [Abstract/FREE Full Text](http://medrxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NjoiYm1qaGNpIjtzOjU6InJlc2lkIjtzOjEyOiIyNy8xL2UxMDAxMTQiO3M6NDoiYXRvbSI7czo1MDoiL21lZHJ4aXYvZWFybHkvMjAyMi8wMi8yMS8yMDIyLjAyLjIxLjIyMjcxMTg1LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 21. 21.Fisk M, Livingstone A, Pit SW. Telehealth in the Context of COVID-19: Changing Perspectives in Australia, the United Kingdom, and the United States. J Med Internet Res. 2020 2020/6/9;22(6):e19264. doi: 10.2196/19264. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.2196/19264&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2022%2F02%2F21%2F2022.02.21.22271185.atom) 22. 22.Neves AL, Li E, Gupta PP, Fontana G, Darzi A. Virtual primary care in high-income countries during the COVID-19 pandemic: Policy responses and lessons for the future. Eur J Gen Pract. 2021 2021/01/01;27(1):241–7. doi: 10.1080/13814788.2021.1965120. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1080/13814788.2021.1965120&link_type=DOI) 23. 23.Bakhai M, Croney L, Waller O, Henshall N, Felstead C. Using Online Consultations In Primary Care: Implementation Toolkit. NHS; 2020 [updated 14 January 2020]; Available from: [https://www.england.nhs.uk/publication/using-online-consultations-in-primary-care-implementation-toolkit/](https://www.england.nhs.uk/publication/using-online-consultations-in-primary-care-implementation-toolkit/). 24. 24.Marshall M, Shah R, Stokes-Lampard H. Online consulting in general practice: making the move from disruptive innovation to mainstream service. BMJ. 2018;360:k1195. PMID: 29581174. doi: 10.1136/bmj.k1195. [FREE Full Text](http://medrxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiRlVMTCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiYm1qIjtzOjU6InJlc2lkIjtzOjE3OiIzNjAvbWFyMjZfNy9rMTE5NSI7czo0OiJhdG9tIjtzOjUwOiIvbWVkcnhpdi9lYXJseS8yMDIyLzAyLzIxLzIwMjIuMDIuMjEuMjIyNzExODUuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 25. 25.Stamenova V, Agarwal P, Kelley L, Fujioka J, Nguyen M, Phung M, et al. Uptake and patient and provider communication modality preferences of virtual visits in primary care: a retrospective cohort study in Canada. BMJ Open. 2020;10(7):e037064. doi: 10.1136/bmjopen-2020-037064. [Abstract/FREE Full Text](http://medrxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NzoiYm1qb3BlbiI7czo1OiJyZXNpZCI7czoxMjoiMTAvNy9lMDM3MDY0IjtzOjQ6ImF0b20iO3M6NTA6Ii9tZWRyeGl2L2Vhcmx5LzIwMjIvMDIvMjEvMjAyMi4wMi4yMS4yMjI3MTE4NS5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 26. 26.Fernández OS, Seguí FL, Vidal-Alaball J, Bonet Simo JM, Vian OH, Cabo PR, et al. Primary Care Doctor Characteristics That Determine the Use of Teleconsultations in the Catalan Public Health System: Retrospective Descriptive Cross-Sectional Study. JMIR Med Inform. 2020 2020/2/1;8(1):e16484. doi: 10.2196/16484. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.2196/16484&link_type=DOI) 27. 27.Johansson A, Larsson M, Ivarsson B. Patients’ Experiences With a Digital Primary Health Care Concept Using Written Dialogues: A Pilot Study. J Prim Care Community Health. 2020 01 Jan;11:2150132720910564. doi: 10.1177/2150132720910564. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1177/2150132720910564&link_type=DOI) 28. 28.Johansson A, Larsson M, Ivarsson B. General Practitioners’ Experiences of Digital Written Patient Dialogues: A Pilot Study Using a Mixed Method. J Prim Care Community Health. 2020 Mar;11. PMID: WOS:000523514400001. doi: 10.1177/2150132720909656. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1177/2150132720909656&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=WOS:00052351&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2022%2F02%2F21%2F2022.02.21.22271185.atom) 29. 29.Judson TJ, Odisho AY, Neinstein AB, Chao J, Williams A, Miller C, et al. Rapid design and implementation of an integrated patient self-triage and self-scheduling tool for COVID-19. J Am Med Inform Assoc. 2020;27(6):860-6. PMID: 32267928. doi: 10.1093/jamia/ocaa051. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1093/jamia/ocaa051&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2022%2F02%2F21%2F2022.02.21.22271185.atom) 30. 30.Entezarjou A, Bolmsjö BB, Calling S, Midlöv P, Milos Nymberg V. Experiences of digital communication with automated patient interviews and asynchronous chat in Swedish primary care: a qualitative study. BMJ Open. 2020;10(7):e036585. doi: 10.1136/bmjopen-2019-036585. [Abstract/FREE Full Text](http://medrxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NzoiYm1qb3BlbiI7czo1OiJyZXNpZCI7czoxMjoiMTAvNy9lMDM2NTg1IjtzOjQ6ImF0b20iO3M6NTA6Ii9tZWRyeGl2L2Vhcmx5LzIwMjIvMDIvMjEvMjAyMi4wMi4yMS4yMjI3MTE4NS5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 31. 31.López Seguí F, Walsh S, Solans O, Adroher Mas C, Ferraro G, García-Altés A, et al. Teleconsultation Between Patients and Healthcare Professionals in the Catalan Primary Care Service: Message Annotation Analysis in a Retrospective Cross-Sectional Study. J Med Internet Res. 2020 Sep 17;22(9)(e19149). doi: 10.2196/19149. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.2196/19149&link_type=DOI) 32. 32.Kelley LT, Phung M, Stamenova V, Fujioka J, Agarwal P, Onabajo N, et al. Exploring how virtual primary care visits affect patient burden of treatment. Int J Med Inform. 2020 2020/09/01/;141:104228. doi: [https://doi.org/10.1016/j.ijmedinf.2020.104228](https://doi.org/10.1016/j.ijmedinf.2020.104228). 33. 33.López Seguí F, Vidal-Alaball J, Sagarra Castro M, García-Altés A, García Cuyàs F. General Practitioners’ Perceptions of Whether Teleconsultations Reduce the Number of Face-to-face Visits in the Catalan Public Primary Care System: Retrospective Cross-Sectional Study. J Med Internet Res. 2020 2020/3/16;22(3):e14478. doi: 10.2196/14478. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.2196/14478&link_type=DOI) 34. 34.Murray MA, Penza KS, Myers JF, Furst JW, Pecina JL. Comparison of eVisit Management of Urinary Symptoms and Urinary Tract Infections with Standard Care. Telemed J E Health. 2020 2020/05/01;26(5):639–44. doi: 10.1089/tmj.2019.0044. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1089/tmj.2019.0044&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=31313978&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2022%2F02%2F21%2F2022.02.21.22271185.atom) 35. 35.Penza KS, Murray MA, Myers JF, Furst JW, Pecina JL. Management of Acute Sinusitis via e-Visit. Telemed J E Health. 2021. doi: 10.1089/tmj.2020.0047. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1089/tmj.2020.0047&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=32522103&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2022%2F02%2F21%2F2022.02.21.22271185.atom) 36. 36.Entezarjou A, Bonamy A-KE, Benjaminsson S, Herman P, Midlöv P. Human-Versus Machine Learning–Based Triage Using Digitalized Patient Histories in Primary Care: Comparative Study. JMIR Med Inform. 2020;8(9):e18930. PMID: 32880578. doi: 10.2196/18930. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.2196/18930&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=32880578&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2022%2F02%2F21%2F2022.02.21.22271185.atom) 37. 37.Murphy M, Scott LJ, Salisbury C, Turner A, Scott A, Denholm R, et al. Implementation of remote consulting in UK primary care following the COVID-19 pandemic: a mixed-methods longitudinal study. Br J Gen Pract. 2021;71(704):e166–e77. doi: 10.3399/bjgp.2020.0948. [Abstract/FREE Full Text](http://medrxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NDoiYmpncCI7czo1OiJyZXNpZCI7czoxMToiNzEvNzA0L2UxNjYiO3M6NDoiYXRvbSI7czo1MDoiL21lZHJ4aXYvZWFybHkvMjAyMi8wMi8yMS8yMDIyLjAyLjIxLjIyMjcxMTg1LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 38. 38.Eldh AC, Sverker A, Bendtsen P, Nilsson E. Health Care Professionals’ Experience of a Digital Tool for Patient Exchange, Anamnesis, and Triage in Primary Care: Qualitative Study. JMIR Hum Factors. 2020;7(4):e21698. PMID: 33315014. doi: 10.2196/21698. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.2196/21698&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=33315014&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2022%2F02%2F21%2F2022.02.21.22271185.atom) 39. 39.Leung K, Qureshi S. Managing high frequency users of an electronic consultation system in primary care: a quality improvement project. BMJ Open Qual. 2021;10(2). PMID: WOS:000663456800001. doi: 10.1136/bmjoq-2020-001310. [Abstract/FREE Full Text](http://medrxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NjoiYm1qcWlyIjtzOjU6InJlc2lkIjtzOjEyOiIxMC8yL2UwMDEzMTAiO3M6NDoiYXRvbSI7czo1MDoiL21lZHJ4aXYvZWFybHkvMjAyMi8wMi8yMS8yMDIyLjAyLjIxLjIyMjcxMTg1LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 40. 40.Tarn DM, Hintz C, Mendez-Hernandez E, Sawlani SP, Bholat MA. Using virtual visits to care for primary care patients with COVID-19 symptoms. J Am Board Fam Med. 2021 February;34:S147–S51. doi: [https://doi.org/10.3122/jabfm.2021.S1.200241](https://doi.org/10.3122/jabfm.2021.S1.200241). [Abstract/FREE Full Text](http://medrxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NToiamFiZnAiO3M6NToicmVzaWQiO3M6MTg6IjM0L1N1cHBsZW1lbnQvUzE0NyI7czo0OiJhdG9tIjtzOjUwOiIvbWVkcnhpdi9lYXJseS8yMDIyLzAyLzIxLzIwMjIuMDIuMjEuMjIyNzExODUuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 41. 41.Wilson G, Currie O, Bidwell S, Saeed B, Dowell A, Halim AA, et al. Empty waiting rooms: the New Zealand general practice experience with telehealth during the COVID-19 pandemic. N Z Med J. 2021 09 Jul;134(1537):89–101. 42. 42.Landgren S, Cajander Å. Non-use of Digital Health Consultations Among Swedish Elderly Living in the Countryside. Front Public Health. 2021 2021-September-10;9(1323). doi: 10.3389/fpubh.2021.588583. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.3389/fpubh.2021.588583&link_type=DOI) 43. 43.Nijhof D, Ingram A, Ochieng R, Roberts E-J, Poulton B, Ochieng B. Examining GP online consultation in a primary care setting in East Midlands, UK. BMC Health Serv Res. 2021 2021/09/30;21(1):1030. doi: 10.1186/s12913-021-07039-2. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1186/s12913-021-07039-2&link_type=DOI) 44. 44.Nilsson E, Sverker A, Bendtsen P, Eldh AC. A Human, Organization, and Technology Perspective on Patients’ Experiences of a Chat-Based and Automated Medical History–Taking Service in Primary Health Care: Interview Study Among Primary Care Patients. J Med Internet Res. 2021 2021/10/18;23(10):e29868. doi: 10.2196/29868. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.2196/29868&link_type=DOI) 45. 45.Cajander Å, Larusdottir M, Hedström G. The effects of automation of a patient-centric service in primary care on the work engagement and exhaustion of nurses. Qual User Exp. 2020 19 Sept;5(1):9. doi: 10.1007/s41233-020-00038-x. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1007/s41233-020-00038-x&link_type=DOI) 46. 46.Zanaboni P, Fagerlund AJ. Patients’ use and experiences with e-consultation and other digital health services with their general practitioner in Norway: Results from an online survey. BMJ Open. 2020 17 Jun;10 (6) (e034773). doi: 10.1136/bmjopen-2019-034773. [Abstract/FREE Full Text](http://medrxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NzoiYm1qb3BlbiI7czo1OiJyZXNpZCI7czoxMjoiMTAvNi9lMDM0NzczIjtzOjQ6ImF0b20iO3M6NTA6Ii9tZWRyeGl2L2Vhcmx5LzIwMjIvMDIvMjEvMjAyMi4wMi4yMS4yMjI3MTE4NS5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 47. 47.Turner A, Morris R, Rakhra D, Stevenson F, McDonagh L, Hamilton F, et al. Unintended consequences of online consultations: a qualitative study in UK primary care. Br J Gen Pract. 2022;72(715):e128. doi: 10.3399/BJGP.2021.0426. [Abstract/FREE Full Text](http://medrxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NDoiYmpncCI7czo1OiJyZXNpZCI7czoxMToiNzIvNzE1L2UxMjgiO3M6NDoiYXRvbSI7czo1MDoiL21lZHJ4aXYvZWFybHkvMjAyMi8wMi8yMS8yMDIyLjAyLjIxLjIyMjcxMTg1LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 48. 48.Pawson R, Tilley N. Realistic evaluation. London: SAGE; 1997. ISBN: 0761950087. 49. 49.Noyes J, Booth A, Moore G, Flemming K, Tunçalp Ö, Shakibazadeh E. Synthesising quantitative and qualitative evidence to inform guidelines on complex interventions: clarifying the purposes, designs and outlining some methods. BMJ Glob Health. 2019;4(Suppl 1):e000893. doi: 10.1136/bmjgh-2018-000893. [Abstract/FREE Full Text](http://medrxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NToiYm1qZ2giO3M6NToicmVzaWQiO3M6MTc6IjQvU3VwcGxfMS9lMDAwODkzIjtzOjQ6ImF0b20iO3M6NTA6Ii9tZWRyeGl2L2Vhcmx5LzIwMjIvMDIvMjEvMjAyMi4wMi4yMS4yMjI3MTE4NS5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 50. 50.Centre for Reviews and Dissemination. Systematic Reviews: CRD’s guidance for undertaking reviews in health care. York: University of York; 2009 [updated January 2009; cited 2021 1 March]; Available from: [https://www.york.ac.uk/crd/SysRev/!SSL!/WebHelp/SysRev3.htm](https://www.york.ac.uk/crd/SysRev/!SSL!/WebHelp/SysRev3.htm). 51. 51.Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ. 2021;372:n71. doi: 10.1136/bmj.n71. [FREE Full Text](http://medrxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiRlVMTCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiYm1qIjtzOjU6InJlc2lkIjtzOjE1OiIzNzIvbWFyMjlfMi9uNzEiO3M6NDoiYXRvbSI7czo1MDoiL21lZHJ4aXYvZWFybHkvMjAyMi8wMi8yMS8yMDIyLjAyLjIxLjIyMjcxMTg1LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 52. 52.Greenhalgh T, Wherton J, Papoutsi C, Lynch J, Hughes G, A’Court C, et al. Beyond Adoption: A New Framework for Theorizing and Evaluating Nonadoption, Abandonment, and Challenges to the Scale-Up, Spread, and Sustainability of Health and Care Technologies. J Med Internet Res. 2017 Nov 1;19(11):e367. PMID: 29092808. doi: 10.2196/jmir.8775. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=doi:10.2196/jmir.8775&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=29092808&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2022%2F02%2F21%2F2022.02.21.22271185.atom) 53. 53.Abimbola S, Patel B, Peiris D, Patel A, Harris M, Usherwood T, et al. The NASSS framework for ex post theorisation of technology-supported change in healthcare: worked example of the TORPEDO programme. BMC Med. 2019 2019/12/30;17(1):233. doi: 10.1186/s12916-019-1463-x. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1186/s12916-019-1463-x&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2022%2F02%2F21%2F2022.02.21.22271185.atom) 54. 54.Hong QN, Fàbregues S, Bartlett G, Boardman F, Cargo M, Dagenais P, et al. The Mixed Methods Appraisal Tool (MMAT) version 2018 for information professionals and researchers. Education for Information. 2018;34:285–91. doi: 10.3233/EFI-180221. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.3233/EFI-180221&link_type=DOI) 55. 55.Nha HQ. Reporting the results of the MMAT (version 2018). 2020 [updated 9 Dec 2020; cited 2021 1 March]; Available from: [http://mixedmethodsappraisaltoolpublic.pbworks.com/w/file/fetch/140056890/Reporting%20the%20results%20of%20the%20MMAT.pdf](http://mixedmethodsappraisaltoolpublic.pbworks.com/w/file/fetch/140056890/Reporting%20the%20results%20of%20the%20MMAT.pdf). 56. 56.QSR International Pty Ltd. NVivo qualitative data analysis software. 2018. 57. 57.Stern C, Lizarondo L, Carrier J, Godfrey C, Rieger K, Salmond S, et al. Methodological guidance for the conduct of mixed methods systematic reviews. JBI Evidence Synthesis. 2020;18(10). 58. 58.Thomas J, Harden A. Methods for the thematic synthesis of qualitative research in systematic reviews. BMC Med Res Methodol. 2008 2008/07/10;8(1):45. doi: 10.1186/1471-2288-8-45. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1186/1471-2288-8-45&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=18616818&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2022%2F02%2F21%2F2022.02.21.22271185.atom) 59. 59.Institute of Medicine (US) Committee on Quality of Health Care in America. Crossing the Quality Chasm: A New Health System for the 21st Century. Washington (DC): National Academies Press (US); 2001. 60. 60.1. Ranganathan S, 2. Gribskov M, 3. Nakai K, 4. Schönbach C, editors Chen L, Wong G. Transcriptome Informatics. In: Ranganathan S, Gribskov M, Nakai K, Schönbach C, editors. Encyclopedia of Bioinformatics and Computational Biology. Oxford: Academic Press; 2019. p. 324–40. 61. 61.1. Bryman A, 2. Rg B, editors Ritchie J, Spencer L. Qualitative data analysis for applied policy research. In: Bryman A, Rg B, editors. Analyzing qualitative data. London 1994. p. 173–94. 62. 62.Lewin S, Glenton C, Munthe-Kaas H, Carlsen B, Colvin CJ, Gülmezoglu M, et al. Using Qualitative Evidence in Decision Making for Health and Social Interventions: An Approach to Assess Confidence in Findings from Qualitative Evidence Syntheses (GRADE-CERQual). PLoS Med. 2015;12(10). doi: [https://doi.org/10.1371/journal.pmed.1001895](https://doi.org/10.1371/journal.pmed.1001895). 63. 63.NHS England, NHS Improvement. Online consultations research: Summary research findings. NHS England and NHS Improvement; 2019 [updated February; cited 2020 1 Sept]; Available from: [https://www.england.nhs.uk/wp-content/uploads/2019/09/online-consultations-reserach-summary-of-findings.pdf](https://www.england.nhs.uk/wp-content/uploads/2019/09/online-consultations-reserach-summary-of-findings.pdf). 64. 64.Nijland N, van Gemert-Pijnen J, Kelders SM, Brandenburg BJ, Seydel ER. Evaluation of the use of an “ask-the-expert” e-consultation service for support on health-related requests. Second International Conference on eHealth, Telemedicine, and Social Medicine; 10-16 Feb. 2010: IEEE; 2010.p. 72–6. 65. 65.Peber E, Wästfelt E. Impact of digi-physical healthcare. [Master Thesis] Sweden: Lund University; 2020 [cited 2020 1 Sept]; Available from: [https://lup.lub.lu.se/student-papers/search/publication/9015211](https://lup.lub.lu.se/student-papers/search/publication/9015211). 66. 66.Leng S, MacDougall M, McKinstry B. The acceptability to patients of video-consulting in general practice: semi-structured interviews in three diverse general practices. J Innov Health Inform. 2016;23(2):141. doi: 10.14236/jhi.v23i2.141. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.14236/jhi.v23i2.141&link_type=DOI) 67. 67.Ruas SSM, Assunção AÁ. Teleconsultations by Primary Care Physicians of Belo Horizonte: Challenges in the Diffusion of Innovation. Telemed J E Health. 2013 2013/05/01;19(5):409–14. doi: 10.1089/tmj.2012.0095. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1089/tmj.2012.0095&link_type=DOI) 68. 68.Jury SC, Kornberg AJ. Integrating telehealth in to ‘business as usual’: Is it really possible? J Telemed Telecare. 2016 2016/12/01;22(8):499–503. doi: 10.1177/1357633X16675802. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1177/1357633X16675802&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=27799455&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2022%2F02%2F21%2F2022.02.21.22271185.atom) 69. 69.Casey M, Shaw S, Swinglehurst D. Experiences with online consultation systems in primary care: Case study of one early adopter site. Br J Gen Pract. 2017 November;67(664):e736–e43. doi: 10.3399/bjgp17X693137. [Abstract/FREE Full Text](http://medrxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NDoiYmpncCI7czo1OiJyZXNpZCI7czoxMToiNjcvNjY0L2U3MzYiO3M6NDoiYXRvbSI7czo1MDoiL21lZHJ4aXYvZWFybHkvMjAyMi8wMi8yMS8yMDIyLjAyLjIxLjIyMjcxMTg1LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 70. 70.Andersen KN, Nielsen JA, Kim S. Use, cost, and digital divide in online public health care: lessons from Denmark. Transforming Government-People Process and Policy. 2019 May;13(2):197-211. PMID: WOS:000484192600004. doi: 10.1108/tg-06-2018-0041. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1108/tg-06-2018-0041&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=WOS:00048419&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2022%2F02%2F21%2F2022.02.21.22271185.atom) 71. 71.Cowie J, Calveley E, Bowers G, Bowers J. Evaluation of a digital consultation and self-care advice tool in primary care: A multi-methods study. Int J Environ Res Public Health. 2018 02 May;15 (5) (896). doi: 10.3390/ijerph15050896. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.3390/ijerph15050896&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2022%2F02%2F21%2F2022.02.21.22271185.atom) 72. 72.North F, Crane SJ, Stroebel RJ, Cha SS, Edell ES, Tulledge-Scheitel SM. Patient-generated secure messages and eVisits on a patient portal: are patients at risk? J Am Med Inform Assoc. 2013;20(6):1143–9. doi: 10.1136/amiajnl-2012-001208. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1136/amiajnl-2012-001208&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=23703826&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2022%2F02%2F21%2F2022.02.21.22271185.atom) 73. 73.Player M, O’Bryan E, Sederstrom E, Pinckney J, Diaz V. Electronic Visits For Common Acute Conditions: Evaluation Of A Recently Established Program. Health Aff. 2018 Dec 2018;37(12):2024–30. PMID: 2151129411. doi: [http://dx.doi.org/10.1377/hlthaff.2018.05122](http://dx.doi.org/10.1377/hlthaff.2018.05122). [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1377/hlthaff.2018.05122&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2022%2F02%2F21%2F2022.02.21.22271185.atom) 74. 74.Eccles A, Hopper M, Turk A, Atherton H. Patient use of an online triage platform: A mixed-methods retrospective exploration in UK primary care. Br J Gen Pract. 2019;69(682):E336–E44. doi: 10.3399/bjgp19X702197. [Abstract/FREE Full Text](http://medrxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NDoiYmpncCI7czo1OiJyZXNpZCI7czoxMToiNjkvNjgyL2UzMzYiO3M6NDoiYXRvbSI7czo1MDoiL21lZHJ4aXYvZWFybHkvMjAyMi8wMi8yMS8yMDIyLjAyLjIxLjIyMjcxMTg1LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 75. 75.Jung CM, Padman R. Virtualized healthcare delivery: Understanding users and their usage patterns of online medical consultations. Int J Med Inform. 2014 Dec;83(12):901-14. PMID: WOS:000346052600004. doi: 10.1016/j.ijmedinf.2014.08.004. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1016/j.ijmedinf.2014.08.004&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=WOS:00034605&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2022%2F02%2F21%2F2022.02.21.22271185.atom) 76. 76.Ipsos MORI, York Health Economics Consortium. Evaluation of Babylon GP at Hand: Final evaluation report. London 2019 [cited 2020 1 Sept]; Available from: [https://www.hammersmithfulhamccg.nhs.uk/media/156123/Evaluation-of-Babylon-GP-at-Hand-Final-Report.pdf](https://www.hammersmithfulhamccg.nhs.uk/media/156123/Evaluation-of-Babylon-GP-at-Hand-Final-Report.pdf). 77. 77.Mixed Methods Appraisal Tool. Reporting the results of the MMAT (version 2018). Mixed Methods Appraisal Tool; 2020 [cited 2021 5 February]; Available from: [http://mixedmethodsappraisaltoolpublic.pbworks.com/w/file/fetch/140056890/Reporting%20the%20results%20of%20the%20MMAT.pdf](http://mixedmethodsappraisaltoolpublic.pbworks.com/w/file/fetch/140056890/Reporting%20the%20results%20of%20the%20MMAT.pdf). 78. 78.McGrail KM, Ahuja MA, Leaver CA. Virtual Visits and Patient-Centered Care: Results of a Patient Survey and Observational Study. J Med Internet Res. 2017 2017/05/26;19(5):e177. doi: 10.2196/jmir.7374. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.2196/jmir.7374&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2022%2F02%2F21%2F2022.02.21.22271185.atom) 79. 79.Penza KS, Murray MA, Pecina JL, Myers JF, Furst JW. Electronic Visits for Minor Acute Illnesses: Analysis of Patient Demographics, Prescription Rates, and Follow-Up Care Within an Asynchronous Text-Based Online Visit. Telemed J E Health. 2018 2018/03/01;24(3):210–5. doi: 10.1089/tmj.2017.0091. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1089/tmj.2017.0091&link_type=DOI) 80. 80.Bishop TF, Press MJ, Mendelsohn JL, Casalino LP. Electronic communication improves access, but barriers to its widespread adoption remain. Health Aff. 2013;32(8):1361–7. doi: 10.1377/hlthaff.2012.1151. [Abstract/FREE Full Text](http://medrxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6OToiaGVhbHRoYWZmIjtzOjU6InJlc2lkIjtzOjk6IjMyLzgvMTM2MSI7czo0OiJhdG9tIjtzOjUwOiIvbWVkcnhpdi9lYXJseS8yMDIyLzAyLzIxLzIwMjIuMDIuMjEuMjIyNzExODUuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 81. 81.Fagerlund AJ, Holm IM, Zanaboni P. General practitioners’ perceptions towards the use of digital health services for citizens in primary care: A qualitative interview study. BMJ Open. 2019 01 May;9 (5)(e028251). doi: 10.1136/bmjopen-2018-028251. [Abstract/FREE Full Text](http://medrxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NzoiYm1qb3BlbiI7czo1OiJyZXNpZCI7czoxMToiOS81L2UwMjgyNTEiO3M6NDoiYXRvbSI7czo1MDoiL21lZHJ4aXYvZWFybHkvMjAyMi8wMi8yMS8yMDIyLjAyLjIxLjIyMjcxMTg1LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 82. 82.Banks J, Farr M, Salisbury C, Bernard E, Northstone K, Edwards H, et al. Use of an electronic consultation system primary care: a qualitative interview study. Br J Gen Pract. 2018 Jan;68(666):E1–E8. PMID: WOS:000425962100001. doi: 10.3399/bjgp17X693509. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.3399/bjgp17X693509&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=WOS:00042596&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2022%2F02%2F21%2F2022.02.21.22271185.atom) 83. 83.Atherton H, Brant H, Ziebland S, Bikker A, Campbell J, Gibson A, et al. The potential of alternatives to face-to-face consultation in general practice, and the impact on different patient groups: a mixed-methods case study. Health Serv Deliv Res 2018;6(20). PMID: 29889485. doi: 10.3310/hsdr06200. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.3310/hsdr06200&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=29889485&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2022%2F02%2F21%2F2022.02.21.22271185.atom) 84. 84.Atherton H, Brant H, Ziebland S, Bikker A, Campbell J, Gibson A, et al. Alternatives to the face-to-face consultation in general practice: Focused ethnographic case study. Br J Gen Pract. 2018 April;68(669):e293–e300. doi: 10.3399/bjgp18X694853. [Abstract/FREE Full Text](http://medrxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NDoiYmpncCI7czo1OiJyZXNpZCI7czoxMToiNjgvNjY5L2UyOTMiO3M6NDoiYXRvbSI7czo1MDoiL21lZHJ4aXYvZWFybHkvMjAyMi8wMi8yMS8yMDIyLjAyLjIxLjIyMjcxMTg1LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 85. 85.Rosen R. Meeting need or fueling demand? Improved access to primary care and supply-induced demand. London: Nuffield Trust; 2014; Available from: [https://www.nuffieldtrust.org.uk/files/2017-01/meeting-need-or-fuelling-demand-web-final.pdf](https://www.nuffieldtrust.org.uk/files/2017-01/meeting-need-or-fuelling-demand-web-final.pdf). 86. 86.Carter M, Fletcher E, Sansom A, Warren FC, Campbell JL. Feasibility, acceptability and effectiveness of an online alternative to face-to-face consultation in general practice: a mixed-methods study of webGP in six Devon practices. BMJ Open. 2018 May;8(2). PMID: WOS:000433129800107. doi: 10.1136/bmjopen-2017-018688. [Abstract/FREE Full Text](http://medrxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NzoiYm1qb3BlbiI7czo1OiJyZXNpZCI7czoxMToiOC8yL2UwMTg2ODgiO3M6NDoiYXRvbSI7czo1MDoiL21lZHJ4aXYvZWFybHkvMjAyMi8wMi8yMS8yMDIyLjAyLjIxLjIyMjcxMTg1LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 87. 87.Matheson C. Implementation of WebGP and Econsultations in Wessex GP Practices: Interim Update Report. Southampton, UK: Centre of Implementation Science; 2016 [updated June; cited 2020 1 Sept]; Available from: [https://eprints.soton.ac.uk/397189/1/\_\_soton.ac.uk\_ude\_personalfiles\_users\_cbm1a13\_mydocuments\_CBM%2520WebGP%2520update%252021062016IR.pdf](https://eprints.soton.ac.uk/397189/1/\_\_soton.ac.uk\_ude\_personalfiles\_users_cbm1a13_mydocuments_CBM%20WebGP%20update%2021062016IR.pdf). 88. 88.Edwards HB, Marques E, Hollingworth W, Horwood J, Farr M, Bernard E, et al. Use of a primary care online consultation system, by whom, when and why: Evaluation of a pilot observational study in 36 general practices in South West England. BMJ Open. 2017;7(11). doi: 10.1136/bmjopen-2017-016901. [Abstract/FREE Full Text](http://medrxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NzoiYm1qb3BlbiI7czo1OiJyZXNpZCI7czoxMjoiNy8xMS9lMDE2OTAxIjtzOjQ6ImF0b20iO3M6NTA6Ii9tZWRyeGl2L2Vhcmx5LzIwMjIvMDIvMjEvMjAyMi4wMi4yMS4yMjI3MTE4NS5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 89. 89.Farr M, Banks J, Edwards HB, Northstone K, Bernard E, Salisbury C, et al. Implementing online consultations in primary care: a mixed-method evaluation extending normalisation process theory through service co-production. BMJ Open. 2018 Mar;8(3). PMID: WOS:000433881200172. doi: 10.1136/bmjopen-2017-019966. [Abstract/FREE Full Text](http://medrxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NzoiYm1qb3BlbiI7czo1OiJyZXNpZCI7czoxMToiOC8zL2UwMTk5NjYiO3M6NDoiYXRvbSI7czo1MDoiL21lZHJ4aXYvZWFybHkvMjAyMi8wMi8yMS8yMDIyLjAyLjIxLjIyMjcxMTg1LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 90. 90.Health Innovation Manchester. Greater Manchester digital primary care insight full report: Key findings from engagement June-July 2020. [Report] 2020 [cited 2021 1 Oct]; Available from: [https://healthinnovationmanchester.com/wp-content/uploads/2020/12/PCIE-DPC-Full-report-FINAL-20.10.20-1.pdf](https://healthinnovationmanchester.com/wp-content/uploads/2020/12/PCIE-DPC-Full-report-FINAL-20.10.20-1.pdf). 91. 91.Health Innovation Manchester. GM Digital First Primary Care: Patient and public insights: Workshop results. Report. Manchester: Health Innovation Manchester, 2021 Sept 2021. Report No.: 1 Contract No.: 1 Sept. 92. 92.1. Safran C, 2. Reti S, 3. Marin HF, editors Padman R, Shevchik G, Paone S, Dolezal C, Cervenak J. eVisit: A Pilot Study of a New Kind of Healthcare Delivery. In: Safran C, Reti S, Marin HF, editors. Medinfo 2010, Pts I and Ii 2010. p. 262–6. 93. 93.Albert SM, Shevchik GJ, Paone S, Martich GD. Internet-based medical visit and diagnosis for common medical problems: experience of first user cohort. Telemed J E Health. 2011 May;17(4):304-8. PMID: 21457013. doi: 10.1089/tmj.2010.0156. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1089/tmj.2010.0156&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=21457013&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2022%2F02%2F21%2F2022.02.21.22271185.atom) [Web of Science](http://medrxiv.org/lookup/external-ref?access_num=000292197800014&link_type=ISI) 94. 94.Adamson SC, Bachman JW. Pilot Study of Providing Online Care in a Primary Care Setting. Mayo Clin Proc. 2010;85(8):704–10. doi: 10.4065/mcp.2010.0145. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.4065/mcp.2010.0145&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=20516427&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2022%2F02%2F21%2F2022.02.21.22271185.atom) [Web of Science](http://medrxiv.org/lookup/external-ref?access_num=000280835300002&link_type=ISI) 95. 95.Lawless M, Wright E, Davidson J. A collaborative approach to improving patient access in general practice: Impact of three different pilot schemes in 12 general practices in Greenwich. London J Prim Care. 2016;8(4):56–65. doi: 10.1080/17571472.2016.1173946. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1080/17571472.2016.1173946&link_type=DOI) 96. 96.Bavafa H, Hitt LM, Terwiesch C. The Impact of E-Visits on Visit Frequencies and Patient Health: Evidence from Primary Care. Manage Sci. 2018;64(12):5461-80. PMID: 33033417. doi: 10.1287/mnsc.2017.2900. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1287/mnsc.2017.2900&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=33033417&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2022%2F02%2F21%2F2022.02.21.22271185.atom) 97. 97.Mehrotra A, Paone S, Martich GD, Albert SM, Shevchik GJ. A comparison of care at e-visits and physician office visits for sinusitis and urinary tract infection. JAMA Intern Med. 2013;173(1):72-4. PMID: 23403816. doi: 10.1001/2013.jamainternmed.305. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1001/2013.jamainternmed.305&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=23403816&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2022%2F02%2F21%2F2022.02.21.22271185.atom) [Web of Science](http://medrxiv.org/lookup/external-ref?access_num=000317238800017&link_type=ISI) 98. 98.Rohrer JE, Angstman KB, Adamson SC, Bernard ME, Bachman JW, Morgan ME. Impact of online primary care visits on standard costs: a pilot study. Popul Health Manag. 2010 Apr;13(2):59-63. PMID: 20415617. doi: 10.1089/pop.2009.0018. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1089/pop.2009.0018&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=20415617&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2022%2F02%2F21%2F2022.02.21.22271185.atom) [Web of Science](http://medrxiv.org/lookup/external-ref?access_num=000276990200002&link_type=ISI) 99. 99.North F, Crane SJ, Chaudhry R, Ebbert JO, Ytterberg K, Tulledge-Scheitel SM, et al. Impact of Patient Portal Secure Messages and Electronic Visits on Adult Primary Care Office Visits. Telemed J E Health. 2014 2014/03/01;20(3):192–8. doi: 10.1089/tmj.2013.0097. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1089/tmj.2013.0097&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=24350803&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2022%2F02%2F21%2F2022.02.21.22271185.atom) 100.100.Jung C, Padman R, Shevchik G, Paone S. Who are portal users vs. early e-Visit adopters? A preliminary analysis. AMIA Annu Symp Proc. 2011:1070–9. PMID: 22195168. [PubMed](http://medrxiv.org/lookup/external-ref?access_num=22195168&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2022%2F02%2F21%2F2022.02.21.22271185.atom) 101.101.1. Sarkar I.N, 2. Georgiou A, 3. Mazzoncini de Azevedo MarquesP, editors Bertelsen P, Petersen L. Danish Citizens and General Practitioners’ Use of ICT for their Mutual Communication. In: Sarkar I.N, Georgiou A, Mazzoncini de Azevedo MarquesP, editors. MEDINFO 2015: eHealth-enabled Health; August; São Paulo, Brazil: IOS Press; 2015. p. 376–9. 102.102.Mehrotra A, Paone S, Martich GD, Albert SM, Shevchik GJ. Characteristics of patients who seek care via eVisits instead of office visits. Telemed J E Health. 2013;19(7):515-9. PMID: 23682589. doi: 10.1089/tmj.2012.0221. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1089/tmj.2012.0221&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=23682589&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2022%2F02%2F21%2F2022.02.21.22271185.atom) 103.103.Ekman B, Thulesius H, Wilkens J, Lindgren A, Cronberg O, Arvidsson E. Utilization of digital primary care in Sweden: Descriptive analysis of claims data on demographics, socioeconomics, and diagnoses. Int J Med Inform. 2019;127:134–40. doi: 10.1016/j.ijmedinf.2019.04.016. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1016/j.ijmedinf.2019.04.016&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2022%2F02%2F21%2F2022.02.21.22271185.atom) 104.104.Fraser H, Coiera E, Wong D. Safety of patient-facing digital symptom checkers. The Lancet. 2018;392(10161):2263–4. doi: 10.1016/S0140-6736(18)32819-8. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1016/S0140-6736(18)32819-8&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=30413281&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2022%2F02%2F21%2F2022.02.21.22271185.atom) 105.105.Hoffmann TC, Glasziou PP, Boutron I, Milne R, Perera R, Moher D, et al. Better reporting of interventions: template for intervention description and replication (TIDieR) checklist and guide. BMJ. 2014;348:g1687. doi: 10.1136/bmj.g1687. [Abstract/FREE Full Text](http://medrxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiYm1qIjtzOjU6InJlc2lkIjtzOjE3OiIzNDgvbWFyMDdfMy9nMTY4NyI7czo0OiJhdG9tIjtzOjUwOiIvbWVkcnhpdi9lYXJseS8yMDIyLzAyLzIxLzIwMjIuMDIuMjEuMjIyNzExODUuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 106.106.Rogers EM. Diffusion of innovations. Illinois: Free Press; 1962. ISBN: 0029266505. 107.107.Darley S, Coulson T, Peek N, Vanderveer S, Wong D, Brown BC. Online patient triage and consultation in primary care: a systematic review (CRD42020191802) PROSPERO; 2020.