Design and implementation of a system for automated monitoring of adherence to evidenced-based clinical guideline recommendations ================================================================================================================================= * Gregor Lichtner * Claudia Spies * Carlo Jurth * Thomas Bienert * Anika Müller * Oliver Kumpf * Vanessa Piechotta * Nicole Skoetz * Monika Nothacker * Martin Boeker * Joerg J Meerpohl * Falk von Dincklage ## Abstract **Background** Clinical practice guidelines are systematically developed statements intended to optimize patient care. However, a gap-less implementation of guideline recommendations requires health care personnel not only to be aware of the recommendations and to support their content, but also to recognize every situation in which they are applicable. To not miss situations in which guideline recommendations should be applied, computerized clinical decision support could be given through a system that allows an automated monitoring of adherence to clinical guideline recommendation in individual patients. **Objectives** (1) To derive the requirements for a system that allows to monitor the adherence to evidence-based clinical guideline recommendations in individual patients, and based on these requirements, (2) to implement a software prototype that integrates clinical guideline recommendations with individual patient data and (3) to demonstrate the prototype’s utility on a COVID-19 intensive care treatment recommendation. **Methods** We performed a work process analysis with experienced intensive care clinicians to develop a conceptual model of how to support guideline adherence monitoring in clinical routine and identified which steps in the model could be supported electronically. We then identified the core requirements of a software system for supporting recommendation adherence monitoring in a consensus-based requirements analysis within loosely structured focus group work of key stakeholders (clinicians, guideline developers, health data engineers, software developers). Based on these requirements, we implemented a prototype and demonstrated its functionality by integrating clinical data with a treatment recommendation. **Results** Based on our conceptual flow chart model of recommendation adherence monitoring in clinical routine, we identified four main requirements of a software system for automated support of recommendation adherence monitoring of in-hospital patients: (i) Ability to interpret guideline recommendations’ semantics and logics, (ii) integration of clinical routine data from various underlying data structures, (iii) automatic adoption of new or updated guideline recommendations, and (iv) user interfaces optimized for distinct groups of users. Using a prototype implementation that fulfills these requirements, we demonstrate how such a system could be applied to monitor guideline recommendation adherence over time in clinical patients. **Conclusions** The four main requirements identified through our model-based analysis represent the most important aspects that need to be considered when developing a clinical decision support system for monitoring the adherence to evidence-based clinical guideline recommendations in individual patients. As each of the requirements corresponds to a different expertise (guideline development, health data engineering, software development, patient treatment), a modularized software architecture separated by area of required expertise seems favorable. Our prototype successfully demonstrates how such a modular architecture can be implemented to allow real-time monitoring of guideline recommendation adherence. This prototype, which we released as open source to invigorate collaboration, could serve as a basis for further development to integrate guideline recommendations with clinical information systems. ## Introduction Clinical practice guideline recommendations are intended to optimize patient care by assisting decision making of health care professionals within specific clinical circumstances [1–3]. Thus, clinical practice guideline recommendations are among the most important potential clinical decision support tools [4,5]. Considering and implementing such recommendations during patient management is expected to be associated with improved patient outcome, especially in the case of evidence-based recommendations that were developed based on systematic reviews and appraisal of the available evidence [6–8]. However, a gap-less implementation of clinical practice guideline recommendations in daily routine work requires health care professionals not only to be aware of the existence of the respective guideline recommendations, to understand and support their content, but also to correctly recognize all situations in which specific recommendations should be applied [9]. Meeting the latter requirement becomes particularly demanding in the interdisciplinary treatment of patients with complex conditions that affect multiple organ systems, as it is often the case in critical care medicine [10–12]. To ensure that all health care professionals have an active knowledge about all guideline recommendations that apply in such situations and that they correctly recognize every situation in which these recommendations should be applied can prove difficult. Thus, treatment in critical care medicine is at a comparably high risk of deviating from guideline recommendations [10]. Besides the multitude of simultaneously applicable guideline recommendations in critical care, another aspect that can strongly affect guideline recommendation adherence is a high frequency of changes in recommendations [11]. The coronavirus disease 2019 (COVID-19) pandemic presented an exemplary situation in which dissemination and implementation of guidelines via conventional processes struggled to hold pace with the rapid development of recommendations and the speed at which recommendations were updated and changed over time [13,14]. To counter such difficulties and assist implementation of clinical guideline recommendations by means of computerized clinical decision support, various machine-readable guideline recommendation formalisms have been developed [15–22]. However, these formalisms focus on representing the finalized guideline recommendations and do not consider the systematic development process from which evidence-based guideline recommendations are derived. We have recently developed a *Fast Healthcare Interoperability Resources* (FHIR)-based formalism for the computer-interpretable representation of the whole guideline recommendation development process from developing systematic reviews of primary studies, rating the certainty of the available body of evidence, and finally applying evidence-to-decision frameworks to derive the final recommendation, called *Clinical Practice Guidelines on Evidence-Based Medicine on FHIR* (CPG-on-EBMonFHIR; [23]). Based on this representation, the aim of this study was (1) to collect and analyze the requirements for providing clinical decision support via automated monitoring of individual evidence-based guideline recommendation adherence, and (2) to design & implement a prototype that fulfills the requirements and (3) to test the prototype’s applicability on real patient data. ## Methods ### Overview To derive the requirements for a software system to monitor the adherence to clinical guideline recommendations, we first performed a work process analysis of the clinical processes that are to be supported by the system. After identifying and structuring these processes, we determined which subprocesses can be supported electronically and identified the requirements for how these subprocesses can be supported by a software system. Based on these requirements we designed a modular system architecture, which we implemented as an open-source prototype. To demonstrate the utility of the prototype implementation, we applied it to monitor the adherence to a COVID-19 treatment guideline recommendation based on clinical data from a large European university hospital. ### Work process analysis, identification of electronic support potential and requirements analysis To derive the requirements for the system, we first conducted a needs analysis for the users of the system, the clinical staff. This needs analysis was conducted as a work process analysis, in which five experienced clinicians contributed to flow chart modeling of how the adherence to guideline recommendations would be monitored in clinical practice. Modeling was performed based on an iterative feedback process, in which for each iteration the model was adapted until no further changes were required. Based on the work process model, we identified which parts of this process could be supported electronically and developed a flow chart model of how clinicians would interact with an electronic system to achieve the goal of monitoring patient-specific guideline recommendation applicability and adherence. Identification of support potential and process modeling was again performed in an iterative feedback process among the same group of clinicians together with a health software architect. To identify the requirements of the software system for monitoring individual recommendation applicability and adherence, we performed a comprehensive needs analysis for the system by involving key stakeholders: 1. Clinical staff, as they are the primary users of the system. 2. Clinical practice guideline developers, as they create and maintain the guideline recommendations that are used by the system. 3. Health data engineers familiar with hospital IT infrastructure, as the system is required to process data from electronic health records. 4. Software developers, as they are required to build, test, and maintain the system. Stakeholders were recruited by convenience from participants at senior level of their respective field within the COVID-19 evidence ecosystem (CEOsys) project of the federal network of university medical centers (NUM) in Germany [24]. They were individually approached and requested to participate. No compensation was offered for participation. We required at least two participants per stakeholder group. ### Software design and prototype implementation The software prototype was implemented in an agile, rapid application development approach. The architecture followed a microservice pattern to allow efficient separation of concerns and scalable and exchangeable deployments of the system within the heterogeneous clinical IT infrastructures. Each container exposes a RESTful application programming interface (API) specified according to the OpenAPI 2.0 standard [25]. Backend modules were implemented in Python 3.8 and the frontend modules using RShiny [26]. ### Demonstration of prototype utility To demonstrate the utility of the prototype, we connected it to anonymized clinical data of a large university hospital (Charité - Universitätsmedizin Berlin, Germany) and integrated a recent strong evidence-based guideline recommendation for the treatment of patients with severe or critical COVID-19 [27]. The use of the anonymized clinical data for research was approved by the local ethics committee (Ethikausschuss 4 am Campus Benjamin Franklin, Charité—Universitätsmedizin Berlin, Chairperson Prof. R. Stahlmann, Application Number EA4/008/19, approval date: 06 Feb 2019, amendment date: 14 May 2020). ## Results ### Work process analysis The needs analysis with the clinical staff resulted in a flow chart describing the work process of how the adherence to guideline recommendations should be monitored (Figure 2 a). The core insight of the work process analysis was that to evaluate and monitor guideline recommendation adherence, clinicians always work at the ward level and examine each patient individually to see whether a guideline recommendation applies and whether it is fulfilled. ![Figure 1:](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2023/03/21/2022.07.18.22277750/F1.medium.gif) [Figure 1:](http://medrxiv.org/content/early/2023/03/21/2022.07.18.22277750/F1) Figure 1: Flowchart of the study and the derived artefacts at each step. ![Figure 2:](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2023/03/21/2022.07.18.22277750/F2.medium.gif) [Figure 2:](http://medrxiv.org/content/early/2023/03/21/2022.07.18.22277750/F2) Figure 2: Conceptual process models of recommendation adherence monitoring in current clinical practice and using an electronic support system. ### Identification of electronic support potential Based on the work process model, the clinical staff together with a health software architect identified which steps could be supported or covered by an electronic system. We identified nearly all steps as susceptible to being taken over by a software system (Figure 2 a). The same group then determined how this process should be supported electronically and developed a corresponding model of the digitized work process (Figure 2 b). The most important core insight of this analysis was the necessity to display the raw patient data that underlies the system’s decision on recommendation applicability/adherence, since clinicians using a software-based decision support system to monitor recommendation adherence want to be able to examine the raw patient data that underlies the system’s decision on recommendation applicability/adherence, to make sure that this data and decision is correct, as electronic health record data may contain errors. ### Requirements analysis For a computer system that should support the defined work process (Figure 2 a), we identified four core requirements in a series of focus work group feedback rounds. #### Requirement #1: The system needs to be able to decide whether a guideline recommendation is applicable and whether a guideline recommendation is implemented for a specific patient The task of checking whether a guideline’s recommendation is applicable and whether it is implemented for an individual patient or not requires the system to be able to process both the semantical (i.e., what is the meaning of the words used in the recommendation) and logical (i.e., which of the words used in the recommendation define who the recommendation applies to and which words define what is to be done or not to be done) content of the recommendation. Therefore, the system needs to be provided with guideline recommendations in a format that is semantically correct, complete, and unambiguous. We here focus on evidence-based guideline recommendations, which makes the decoding of the logical content of the recommendations particular easy: In the development process of evidence-based recommendations, it is standard practice to decompose the clinical question in consideration according to the PICO framework (population/patients, intervention, comparison, outcomes,) [28]. Therefore, in these recommendations, the patients to which the guideline recommendation is applicable to (P in PICO) and the intervention (I in PICO) that is recommended are distinctly defined at its best beginning with the systematic reviews supporting evidence-based guideline recommendations. Regarding the decoding and interpretation of the semantical content of guideline recommendations, we have developed a FHIR-based format for the representation of clinical practice guideline recommendations to provide an interoperable, standards-based guideline recommendation exchange format that fulfills the above requirements [23]. Moreover, a variety of formalisms for representing guideline recommendations in a computer-interpretable way exist [29–31] and any of these could be used, provided that they are able to represent guideline recommendations semantically correct, complete and unambiguous. #### Requirement #2: The system needs to be able to integrate clinical data from different data formats and data structures Despite a multitude of initiatives for standardization, patient data is often only available in proprietary and non-standardized data formats and data structures that differ between countries, hospitals or even wards in the same hospital. Therefore, to make the system applicable across various existing IT infrastructural settings, the second core requirement is that it needs to accept data in a standardized, interoperable format, into which all proprietary data formats can be converted. Among data formats that fulfill these requirements are the OMOP common data model (CDM) [32] or FHIR-based formats (e.g. the US Core Profiles [33] or the German Corona Consensus dataset [34]). #### Requirement #3: The system needs to automatically adopt changes in clinical guideline recommendations Clinical guideline recommendations are subject to change as medical knowledge advances. Considering the vast number of new findings being published in the medical literature every day and the subsequent frequency of guideline recommendation updates, any efforts to manually implement updated guideline recommendations in a software system can be expected to delay updates and pose a source of errors [11]. One aspect that especially complicates error-free manual implementation of guideline recommendations in a software system is that such a task requires the expertise of at least two different and highly specialized fields: the expertise of the medical subspecialty providing the guideline recommendation and the software development expertise necessary for implementation into a system. Having both at one’s disposal for every single new or updated guideline recommendation is difficult and costly. Therefore, the system needs to adopt changes in guideline recommendations without requiring changes in the system’s software code itself. Instead, once changes in guideline recommendation are released by the responsible medical societies or other appropriate sources, these changes should be automatically adopted by the system without the requirement of any manual changes in the system’s software. #### Requirement #4: The system needs to provide user interfaces optimized for distinct user groups Different users of the clinical decision support system require different user interfaces, depending on the specific work processes that are to be supported by the system. For example, medical or nursing staff working on individual patients require a system that is highly integrated with their standard patient data visualization used during the treatment process (e.g., the critical care information system used on the ward) to allow to integrate the decision support seamlessly into the individual patient care. In contrast, cross section staff like quality officers or supervising staff require more comprehensive overviews over multiple patients simultaneously, with less integration with other patient data, as their work processes that are to be supported by the system are more disconnected from individual patient care. Therefore, the fourth core requirement is that the user interface of the system must be customizable to meet the specific requirements that allow integration into the work processes of the respective groups of users that are to be supported. ### Prototype implementation Considering the advantages of a modular system where each module corresponds to a specific specialty, we decided for a software architecture with four main modules that correspond to the previously described four main requirements that each require the involvement of only one of the three stakeholder groups besides the software developers (Figure 3; Table 1). View this table: [Table 1:](http://medrxiv.org/content/early/2023/03/21/2022.07.18.22277750/T1) Table 1: Overview of the prototype’s modules and their matching requirement. ![Figure 3:](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2023/03/21/2022.07.18.22277750/F3.medium.gif) [Figure 3:](http://medrxiv.org/content/early/2023/03/21/2022.07.18.22277750/F3) Figure 3: Architecture of the guideline recommendation evaluator for the automated integration of clinical guideline recommendations with real-time clinical data. Numbers indicate the requirement that is associated with the module. The software was implemented as a containerized microservice architecture for effortless on-premises deployment within individual hospitals’ IT infrastructure, preventing any patient data from leaving the hospitals’ networks. We have designed the backend modules to be run within the same local network with the only outside access possible to the user interface backend. Thus, in our prototype implementation the user interface backend provides a user-level authentication to ensure only authorized users can access the output of the system. Depending on the actual deployment scenario, each of the microservices can be augmented by an authentication scheme. For our exemplary prototype we have designed and implemented a user interface aimed to assist supervising medical staff in their task to review whether specific guideline recommendations are applicable and adhered to in individual patients that are currently treated in wards for which they are responsible (**Figure 4**). The user interface designed for this specific task allows the user to select the guideline recommendation to check and then, along with an overview over the patients currently treated in the ward, gives a condensed evaluation to which of the patients this specific recommendation is applicable and in which of the patients it is adhered to and in which it is not. Furthermore, the user interface allows to view the patient data on which the guideline recommendation evaluation was performed on to allow the clinician to review the evaluator’s results. The user interface is implemented as a dashboard website using RShiny [26], but is easily exchangeable by any other user interface framework or implementation due to the REST API interface of the user interface backend through which it receives data. ![Figure 4:](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2023/03/21/2022.07.18.22277750/F4.medium.gif) [Figure 4:](http://medrxiv.org/content/early/2023/03/21/2022.07.18.22277750/F4) Figure 4: Prototype implementation of the user interface. The user can select a guideline recommendation of interest (top right) and view the patient-individual applicability and adherence of the recommendation on all current patients on a selected ward (left), where recommendation adherence is marked by check marks on green background. To allow the user to understand and review the results of the guideline recommendation evaluation, the user can select individual patients to show the original patient data required to assess the recommendation’s applicability and adherence (right). ### Demonstration of prototype utility To demonstrate the utility of our prototype implementation, we have specified a recent strong evidence-based recommendation for the administration of dexamethasone to critically ill COVID-19 patients from the guideline for inpatient treatment of COVID-19 patients as a machine-readable guideline recommendation [35–37]. We connected the prototype to the critical care information systems and clinical information systems of Charité – Universitätsmedizin Berlin by means of a site-specific implementation of the clinical data interface. An exemplary time-dependent analysis of a guideline recommendation’s applicability and adherence is shown in Figure 5. ![Figure 5:](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2023/03/21/2022.07.18.22277750/F5.medium.gif) [Figure 5:](http://medrxiv.org/content/early/2023/03/21/2022.07.18.22277750/F5) Figure 5: Individual applicability and adherence of a guideline recommendation to treat severe and critical COVID-19 patients with steroids. Top: Shown are the number of patients to which the guideline recommendation is applicable (purple) and which have been treated according to the guideline recommendation (orange) between March 2020 and May 2021. Bottom: Shown are the number of patients that are treated according to the guideline recommendation as a fraction of patients to which the guideline recommendation is applicable. Vertical gray lines indicate date of the first press release of the detected Dexamethasone efficacy in the RECOVERY trial (dashed line [38]) and the initial publication of the trial results (solid line; [39]). ## Discussion In this paper, we demonstrated the system architecture and prototype implementation of a clinical decision support system that automatically integrates clinical guideline recommendations with real-time clinical data to assist health care professionals by visualizing whether guideline recommendations apply to individual patients and whether the guidelines recommendations have been followed in individual patients or not. We described our stepwise approach for the development of the system, including the core requirements which shaped our software architectural design, as well as our prototype implementation and demonstrated the prototype’s utility using a COVID-19 treatment guideline recommendation on clinical data. To evaluate our architecture and prototype, we implemented a recent clinical guideline recommendation on treatment of patients with severe or critical COVID-19 disease and integrated the recommendation with data from a large university hospital to analyze the guideline recommendation adherence over time (Figure 5). The integration of the machine-readable guideline recommendation with clinical data could accurately detect the first and second wave of COVID-19 intensive care treatments [40] and the successful implementation of the guideline recommendation for the second wave, as seen by a >70% of relevant patients treated according to the recommendation. The non-100% guideline recommendation implementation in our specific data set is primarily due to the fact that a large part of COVID-19 patients treated in this university medicine center were transferred from other hospitals and in parts have been treated with steroids in accordance with the guideline recommendation before arriving on the intensive care unit of our hospital [41]. Such a situation in which the recommended treatment has already taken place but was recorded in a different patient data management system could be solved by increased interoperable data exchange between different health care providers. An automated integration of guideline recommendations with clinical data as done by our developed systems holds several advantages: The system can provide a certain kind of decision support during individual patient treatment by pointing to applicable guideline recommendations, which the treating health care professionals might either not be aware of or which are known but whose applicability might go unnoticed. Additionally, the monitoring of guideline recommendation adherence across groups of patients provides possibilities for their use as quality and performance indicators [42,43] that can easily be monitored in real-time using a system as we propose it here. Another similar aspect could be the application of the system to monitor process implementation of new guidelines and to provide clinical insights in applicability of guideline recommendations that are useful for guideline updates. Independent from official clinical guidelines, the system can also easily be used to monitor hospital-specific treatment standards by formulating them as machine-readable guideline recommendations and providing them via the guideline interface to the adherence evaluator. We have designed our system to require machine-readable guideline recommendations, however, guideline recommendations are currently available only in narrative, human-readable format. Thus, individual guideline recommendations need to be converted in a machine-readable format first, adding an extra amount of work. However, the specification of guideline recommendations in a machine-readable format enforces a precise and accurate formulation of guideline recommendation preventing ambiguities, as these could not (easily) be resolved or understood by a software system. Additionally, in contrast to converting human-readable recommendations into machine-readable recommendations, the generation of precise human-readable formulations of a guideline recommendation that is formulated in a machine-readable standard is a comparably simple task. Therefore, specifying guideline recommendations from the start in a machine-readable format holds multiple advantages and we therefore consider it a desirable change in the current practice of high quality, evidence-based guideline recommendation development. An alternative approach to using a machine-readable guideline recommendation specification could be the application of recent advances of natural language processing (NLP) methods to allow the computer to “understand” and process human-readable guideline recommendations [44,45]. However, any errors unknowingly introduced by such an approach (e.g., due to imperfect “understanding” of the guideline recommendation by the NLP algorithms) could have severe consequences on patient health and outcomes. Therefore, we believe that the explicit statement of guideline recommendations in a standardized machine-readable format is currently the more suitable choice. In our prototype implementation we have used an EBMonFHIR-based representation of clinical guideline recommendations that covers the whole development process of evidence-based guidelines, from the underlying systematic review of available evidence over the rating of the individual evidence to the final recommendation in a computer-interpretable way [23]. The computer-interpretable links to the development process are particularly useful in meeting the requirement of the system to adapt to updated guideline recommendations (requirement #3), as updates to recommendations based on new evidence or re-appraisal of existing evidence can be automatically passed to the system without the need of newly and manually converting human-readable guideline recommendations into a representation formalism. While requirement #3 could in principle be met by any guideline recommendation formalisms that is semantically correct, complete, and unambiguous, the CPG-on-EBMonFHIR representation offers an advantage especially with guideline recommendations that are updated regularly, such as recommendations from living guidelines. Additionally, the CPG-on-EBMonFHIR representation could allow the users of the software system to evaluate the certainty of the evidence and the evidence-to-decision process underlying individual recommendations. We did not include this functionality in the prototype implementation, which, however, may become part of a later extension stage of the system implementation. In our approach, the user interface is separated from the rest of the software to allow user- and context-specific implementation of user interfaces. We have provided a prototype as a dashboard website to demonstrate its feasibility. However, in clinical practice during individual treatment it might be desirable to integrate the suggestions of guideline recommendations into the clinical information system (CIS) that is implemented on the ward. Due to the separation of user interface and backend in our system, these integrations could be readily implemented depending on the CIS, e.g. as SMART-on-FHIR clinical decision support (CDS) hooks [46,47], if the CIS provides a SMART-on-FHIR interface. We have designed our software system to retrieve guideline recommendations from a centralized repository, which could be hosted by medical societies or national standardization organizations. This allows these societies or organizations to develop guideline recommendations independently of our system, thereby focusing on their expertise. Once new or updated guideline recommendations are published by the medical societies or organizations on their servers, they can be automatically retrieved by our system and integrated with clinical data. However, a checkpoint in this process should be established, where health care professionals of the individual hospitals first review new or updated guideline recommendations retrieved from the central repository before releasing them for implementation in their hospital. This helps to mitigate risks that arise if the central guideline recommendation server is compromised by malicious attackers and also ensures that the guideline recommendations implemented in the hospitals are in accordance with the hospitals’ policies. ## Conclusion In this work, we have designed a system that integrates guideline recommendations with real-time clinical data to evaluate individual guideline recommendation adherence and developed a functional prototype. The proposed system holds advantages both for the individual treatment of patients, as clinical guidelines condense the current state of medical knowledge into treatment recommendations, and for quality management and monitoring of patient treatment standards in hospitals. We have specified a modular software architecture where each module corresponds to a particular area of expertise, allowing experts from different fields (guideline developers, software engineers, medical data engineers and health care professionals) to work independently and focus on their area of expertise. We have released the source code of our system under an open source license (see code availability) and invite for cooperation and collaborative further development of the system [48]. ## Supporting information Supplementary Appendix [[supplements/277750_file02.pdf]](pending:yes) ## Data Availability All code produced is available online at https://github.com/CEOsys/grevaluator. [https://github.com/CEOsys/grevaluator](https://github.com/CEOsys/grevaluator) ## Funding The COVID-19 Evidence Ecosystem (CEOsys) project is funded under a scheme issued by the Network of University Medicine (Nationales Forschungsnetzwerk der Universitätsmedizin (NUM)) by the Federal Ministry of Education and Research of Germany (Bundesministerium für Bildung und Forschung (BMBF)) grant number 01KX2021. The CODEX+ project is funded under a scheme issued by the Network of University Medicine (Nationales Forschungsnetzwerk der Universitätsmedizin (NUM)) by the Federal Ministry of Education and Research of Germany (Bundesministerium für Bildung und Forschung (BMBF)) grant number 01KX2121. ## Code availability The complete code of the clinical guideline recommendation system is published under an open-source license (AGPL v3.0) at [https://github.com/CEOsys/grevaluator](https://github.com/CEOsys/grevaluator). API-level documentation can be found at [https://ceosys.readthedocs.io/](https://ceosys.readthedocs.io/). ## Footnotes * Corrected author affiliations * Received July 18, 2022. * Revision received March 21, 2023. * Accepted March 21, 2023. * © 2023, Posted by Cold Spring Harbor Laboratory This pre-print is available under a Creative Commons License (Attribution-NonCommercial-NoDerivs 4.0 International), CC BY-NC-ND 4.0, as described at [http://creativecommons.org/licenses/by-nc-nd/4.0/](http://creativecommons.org/licenses/by-nc-nd/4.0/) ## References 1. 1.1. Field MJ, 2. Lohr KN Institute of Medicine (US) Committee to Advise the Public Health Service on Clinical Practice Guidelines. Clinical Practice Guidelines: Directions for a New Program [Internet]. Field MJ, Lohr KN, editors. Washington (DC): National Academies Press (US); 1990 [cited 2021 Sep 9]. PMID:25144032ISBN:978-0-309-04346-5 [PubMed](http://medrxiv.org/lookup/external-ref?access_num=25144032&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2023%2F03%2F21%2F2022.07.18.22277750.atom) 2. 2.Institute of Medicine (US) Committee on Quality of Health Care in America. Crossing the Quality Chasm: A New Health System for the 21st Century [Internet]. Washington (DC): National Academies Press (US); 2001 [cited 2021 Sep 9]. PMID:25057539ISBN:978-0-309-07280-9 [PubMed](http://medrxiv.org/lookup/external-ref?access_num=25057539&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2023%2F03%2F21%2F2022.07.18.22277750.atom) 3. 3.1. Graham R, 2. Mancher M, 3. Miller Wolman D, 4. Greenfield S, 5. Steinberg E, editors Institute of Medicine (US) Committee on Standards for Developing Trustworthy Clinical Practice Guidelines. Clinical Practice Guidelines We Can Trust [Internet]. Graham R, Mancher M, Miller Wolman D, Greenfield S, Steinberg E, editors. Washington (DC): National Academies Press (US); 2011 [cited 2021 Sep 14]. PMID:24983061ISBN:978-0-309-16422-1 [PubMed](http://medrxiv.org/lookup/external-ref?access_num=24983061&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2023%2F03%2F21%2F2022.07.18.22277750.atom) 4. 4.Djulbegovic B, Guyatt G. Evidence vs Consensus in Clinical Practice Guidelines. JAMA 2019 Aug 27;322(8):725. [doi: 10.1001/jama.2019.9751] [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1001/jama.2019.9751&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=31322650&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2023%2F03%2F21%2F2022.07.18.22277750.atom) 5. 5.Djulbegovic B, Guyatt GH. Progress in evidence-based medicine: a quarter century on. The Lancet 2017 Jul 22;390(10092):415–423. [doi: 10.1016/S0140-6736(16)31592-6] [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1016/S0140-6736(16)31592-6&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2023%2F03%2F21%2F2022.07.18.22277750.atom) 6. 6.Trogrlic Z, van der Jagt M, Lingsma H, Gommers D, Ponssen HH, Schoonderbeek JFJ, Schreiner F, Verbrugge SJ, Duran S, Bakker J, Ista E. Improved Guideline Adherence and Reduced Brain Dysfunction After a Multicenter Multifaceted Implementation of ICU Delirium Guidelines in 3,930 Patients. Crit Care Med 2019 Mar;47(3):419–427. [doi: 10.1097/CCM.0000000000003596] [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1097/CCM.0000000000003596&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2023%2F03%2F21%2F2022.07.18.22277750.atom) 7. 7.Kavanagh BP, Nurok M. Standardized Intensive Care. Protocol Misalignment and Impact Misattribution. Am J Respir Crit Care Med American Thoracic Society - AJRCCM; 2016 Jan 1;193(1):17–22. [doi: 10.1164/rccm.201502-0314CP] [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1164/rccm.201502-0314CP&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=26393290&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2023%2F03%2F21%2F2022.07.18.22277750.atom) 8. 8.van Steenkiste J, Larson S, Ista E, van der Jagt M, Stevens RD. Impact of structured care systems on mortality in intensive care units. Intensive Care Med 2021 Mar 27;1–3. PMID:33774712 [PubMed](http://medrxiv.org/lookup/external-ref?access_num=33774712&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2023%2F03%2F21%2F2022.07.18.22277750.atom) 9. 9.Cabana MD, Rand CS, Powe NR, Wu AW, Wilson MH, Abboud P-AC, Rubin HR. Why Don’t Physicians Follow Clinical Practice Guidelines?A Framework for Improvement. JAMA 1999 Oct 20;282(15):1458–1465. [doi: 10.1001/jama.282.15.1458] [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1001/jama.282.15.1458&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=10535437&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2023%2F03%2F21%2F2022.07.18.22277750.atom) [Web of Science](http://medrxiv.org/lookup/external-ref?access_num=000083111000032&link_type=ISI) 10. 10.Trogrlic Z, van der Jagt M, Bakker J, Balas MC, Ely EW, van der Voort PHJ, Ista E. A systematic review of implementation strategies for assessment, prevention, and management of ICU delirium and their effect on clinical outcomes. Crit Care Lond Engl 2015 Apr 9;19:157. PMID:25888230 [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1186/s13054-015-0886-9&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=25888230&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2023%2F03%2F21%2F2022.07.18.22277750.atom) 11. 11.Fischer F, Lange K, Klose K, Greiner W, Kraemer A. Barriers and Strategies in Guideline Implementation—A Scoping Review. Healthcare 2016 Jun 29;4(3):36. PMID:27417624 [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.3390/healthcare4030036&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=27417624&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2023%2F03%2F21%2F2022.07.18.22277750.atom) 12. 12.Zhang Z, Hong Y, Liu N. Scientific evidence underlying the recommendations of critical care clinical practice guidelines: a lack of high level evidence. Intensive Care Med 2018 Jul;44(7):1189–1191. [doi: 10.1007/s00134-018-5142-8] [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1007/s00134-018-5142-8&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=29564478&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2023%2F03%2F21%2F2022.07.18.22277750.atom) 13. 13.Stamm TA, Andrews MR, Mosor E, Ritschl V, Li LC, Ma JK, Campo-Arias A, Baker S, Burton NW, Eghbali M, Fernandez N, Ferreira RJO, Gäbler G, Makri S, Mintz S, Moe RH, Morasso E, Murphy SL, Ntuli S, Omara M, Simancas-Pallares MA, Horonieff J, Gartlehner G. The methodological quality is insufficient in clinical practice guidelines in the context of COVID-19: systematic review. J Clin Epidemiol 2021 Jul 1;135:125–135. [doi: 10.1016/j.jclinepi.2021.03.005] [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1016/j.jclinepi.2021.03.005&link_type=DOI) 14. 14.Ash JS, Sittig DF, Campbell EM, Guappone KP, Dykstra RH. Some Unintended Consequences of Clinical Decision Support Systems. AMIA Annu Symp Proc 2007;2007:26–30. PMID:18693791 [PubMed](http://medrxiv.org/lookup/external-ref?access_num=18693791&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2023%2F03%2F21%2F2022.07.18.22277750.atom) 15. 15.Tu SW, Campbell JR, Glasgow J, Nyman MA, McClure R, McClay J, Parker C, Hrabak KM, Berg D, Weida T, Mansfield JG, Musen MA, Abarbanel RM. The SAGE Guideline Model: Achievements and Overview. J Am Med Inform Assoc JAMIA 2007;14(5):589–598. PMID:17600098 [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1197/jamia.M2399&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=17600098&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2023%2F03%2F21%2F2022.07.18.22277750.atom) 16. 16.Tu SW, Musen MA. Modeling Data and Knowledge in the EON Guideline Architecture. MEDINFO 2001 IOS Press; 2001;280–284. [doi: 10.3233/978-1-60750-928-8-280] [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.3233/978-1-60750-928-8-280&link_type=DOI) 17. 17.Boxwala AA, Peleg M, Tu S, Ogunyemi O, Zeng QT, Wang D, Patel VL, Greenes RA, Shortliffe EH. GLIF3: a representation format for sharable computer-interpretable clinical practice guidelines. J Biomed Inform 2004 Jun 1;37(3):147–161. [doi: 10.1016/j.jbi.2004.04.002] [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1016/j.jbi.2004.04.002&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=15196480&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2023%2F03%2F21%2F2022.07.18.22277750.atom) [Web of Science](http://medrxiv.org/lookup/external-ref?access_num=000222373900002&link_type=ISI) 18. 18.Samwald M, Fehre K, de Bruin J, Adlassnig K-P. The Arden Syntax standard for clinical decision support: Experiences and directions. J Biomed Inform 2012 Aug 1;45(4):711–718. [doi: 10.1016/j.jbi.2012.02.001] [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1016/j.jbi.2012.02.001&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=22342733&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2023%2F03%2F21%2F2022.07.18.22277750.atom) 19. 19.Seyfang A, Miksch S, Marcos M. Combining diagnosis and treatment using asbru. Int J Med Inf 2002 Dec 18;68(1):49–57. [doi: 10.1016/S1386-5056(02)00064-3] [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1016/S1386-5056(02)00064-3&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=12467790&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2023%2F03%2F21%2F2022.07.18.22277750.atom) 20. 20.Ciccarese P, Caffi E, Quaglini S, Stefanelli M. Architectures and tools for innovative Health Information Systems: The Guide Project. Int J Med Inf 2005 Aug 1;74(7):553–562. [doi: 10.1016/j.ijmedinf.2005.02.001] [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1016/j.ijmedinf.2005.02.001&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=16043084&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2023%2F03%2F21%2F2022.07.18.22277750.atom) [Web of Science](http://medrxiv.org/lookup/external-ref?access_num=000231184500007&link_type=ISI) 21. 21.Silva A, Oliveira T, Gonçalves F, Novais P. Enhancing decision making by providing a unified system for computer-interpretable guideline management. Expert Syst 2021;38(5):e12412. [doi: 10.1111/exsy.12412] [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1111/exsy.12412&link_type=DOI) 22. 22.Sutton RT, Pincock D, Baumgart DC, Sadowski DC, Fedorak RN, Kroeker KI. An overview of clinical decision support systems: benefits, risks, and strategies for success. Npj Digit Med 2020 Feb 6;3(1):1–10. [doi: 10.1038/s41746-020-0221-y] [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1038/s41746-020-0221-y&link_type=DOI) 23. 23.Lichtner G, Jurth C, Alper BS, Spies C, Boeker M, Meerpohl JJ, Dincklage F von. Representation of evidence-based clinical practice guideline recommendations on FHIR [Internet]. medRxiv; 2022 [cited 2022 May 24]. p. 2022.05.16.22275120. [doi: 10.1101/2022.05.16.22275120] [Abstract/FREE Full Text](http://medrxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NzoibWVkcnhpdiI7czo1OiJyZXNpZCI7czoyMToiMjAyMi4wNS4xNi4yMjI3NTEyMHYxIjtzOjQ6ImF0b20iO3M6NTA6Ii9tZWRyeGl2L2Vhcmx5LzIwMjMvMDMvMjEvMjAyMi4wNy4xOC4yMjI3Nzc1MC5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 24. 24.What is CEOsys? [Internet]. CEOsys. [cited 2022 Apr 8]. Available from: [https://covid-evidenz.de/what-is-ceosys/](https://covid-evidenz.de/what-is-ceosys/) 25. 25.Fielding RT. Architectural styles and the design of network-based software architectures [phd]. University of California, Irvine; 2000. 26. 26.Chang W, Cheng J, Allaire JJ, Sievert C, Schloerke B, Xie Y, Allen J, McPherson J, Dipert A, Borges B. shiny: Web Application Framework for R [Internet]. 2021 [cited 2021 Sep 29]. Available from: [https://CRAN.R-project.org/package=shiny](https://CRAN.R-project.org/package=shiny) 27. 27.Kluge S, Janssens U, Welte T, Weber-Carstens S, Schälte G, Spinner CD, Malin JJ, Gastmeier P, Langer F, Wepler M, Westhoff M, Pfeifer M, Rabe KF, Hoffmann F, Böttiger BW, Weinmann-Menke J, Kersten A, Berlit P, Krawczyk M, Nehls W, Haase R, Nothacker M, Marx G, Karagiannidis C. S3-Leitlinie Empfehlungen zur stationären Therapie von Patienten mit COVID-19 [Internet]. AWMF (Arbeitsgemeinschaft der Wissenschaftlichen Medizinischen Fachgesellschaften e.V.); 2021 May. Report No.: 113/001, Evidence-based Recommendation #5. Available from: [https://www.awmf.org/leitlinien/detail/ll/113-001LG.html](https://www.awmf.org/leitlinien/detail/ll/113-001LG.html) 28. 28.Schardt C, Adams MB, Owens T, Keitz S, Fontelo P. Utilization of the PICO framework to improve searching PubMed for clinical questions. BMC Med Inform Decis Mak 2007 Jun 15;7(1):16. [doi: 10.1186/1472-6947-7-16] [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1186/1472-6947-7-16&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=17573961&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2023%2F03%2F21%2F2022.07.18.22277750.atom) 29. 29.Peleg M. Computer-interpretable clinical guidelines: A methodological review. J Biomed Inform 2013 Aug 1;46(4):744–763. [doi: 10.1016/j.jbi.2013.06.009] [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1016/j.jbi.2013.06.009&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=23806274&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2023%2F03%2F21%2F2022.07.18.22277750.atom) 30. 30.Isern D, Moreno A. Computer-based execution of clinical guidelines: A review. Int J Med Inf 2008 Dec 1;77(12):787–808. [doi: 10.1016/j.ijmedinf.2008.05.010] [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1016/j.ijmedinf.2008.05.010&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=18639485&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2023%2F03%2F21%2F2022.07.18.22277750.atom) [Web of Science](http://medrxiv.org/lookup/external-ref?access_num=000261447100001&link_type=ISI) 31. 31.Riaño D, Peleg M, ten Teije A. Ten years of knowledge representation for health care (2009– 2018): Topics, trends, and challenges. Artif Intell Med 2019 Sep;100:101713. [doi: 10.1016/j.artmed.2019.101713] [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1016/j.artmed.2019.101713&link_type=DOI) 32. 32.OMOP Common Data Model [Internet]. [cited 2022 Mar 24]. Available from: [https://ohdsi.github.io/CommonDataModel/](https://ohdsi.github.io/CommonDataModel/) 33. 33.FHIR US Core [Internet]. [cited 2022 Mar 24]. Available from: [https://www.hl7.org/fhir/us/core/](https://www.hl7.org/fhir/us/core/) 34. 34.Sass J, Bartschke A, Lehne M, Essenwanger A, Rinaldi E, Rudolph S, Heitmann KU, Vehreschild JJ, von Kalle C, Thun S. The German Corona Consensus Dataset (GECCO): a standardized dataset for COVID-19 research in university medicine and beyond. BMC Med Inform Decis Mak 2020 Dec 21;20(1):341. [doi: 10.1186/s12911-020-01374-w] [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1186/s12911-020-01374-w&link_type=DOI) 35. 35.Malin JJ, Spinner CD, Janssens U, Welte T, Weber-Carstens S, Schälte G, Gastmeier P, Langer F, Wepler M, Westhoff M, Pfeifer M, Rabe KF, Hoffmann F, Böttiger BW, Weinmann-Menke J, Kersten A, Berlit P, Krawczyk M, Nehls W, Fichtner F, Laudi S, Stegemann M, Skoetz N, Nothacker M, Marx G, Karagiannidis C, Kluge S. Key summary of German national treatment guidance for hospitalized COVID-19 patients : Key pharmacologic recommendations from a national German living guideline using an Evidence to Decision Framework (last updated 17.05.2021). Infection 2021 Jul 6; PMID:34228347 [PubMed](http://medrxiv.org/lookup/external-ref?access_num=34228347&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2023%2F03%2F21%2F2022.07.18.22277750.atom) 36. 36.AWMF. S3 Leitlinie Empfehlungen zur stationären Therapie von Patienten mit COVID-19 (Registernummer 113-001). Assoc Sci Med Soc Ger AWMF [Internet] 2021 May 17 [cited 2021 Sep 10]; Available from: [https://www.awmf.org/leitlinien/detail/ll/113-001.html](https://www.awmf.org/leitlinien/detail/ll/113-001.html) 37. 37.Kluge S, Janssens U, Spinner CD, Pfeifer M, Marx G, Karagiannidis C, Guideline group. Clinical Practice Guideline: Recommendations on Inpatient Treatment of Patients with COVID-19. Dtsch Arzteblatt Int 2021 Jan 11;118(Forthcoming):arztebl.m2021.0110. PMID:33531113 [PubMed](http://medrxiv.org/lookup/external-ref?access_num=33531113&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2023%2F03%2F21%2F2022.07.18.22277750.atom) 38. 38.Dexamethasone reduces death in hospitalised patients with severe respiratory complications of COVID-19 | University of Oxford [Internet]. [cited 2021 Sep 13]. Available from: [https://www.ox.ac.uk/news/2020-06-16-dexamethasone-reduces-death-hospitalised-patients-severe-respiratory-complications](https://www.ox.ac.uk/news/2020-06-16-dexamethasone-reduces-death-hospitalised-patients-severe-respiratory-complications) 39. 39.Dexamethasone in Hospitalized Patients with Covid-19. N Engl J Med Massachusetts Medical Society; 2021 Feb 25;384(8):693–704. [doi: 10.1056/NEJMoa2021436] [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1056/NEJMoa2021436&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=32678530&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2023%2F03%2F21%2F2022.07.18.22277750.atom) 40. 40.Karagiannidis C, Windisch W, McAuley DF, Welte T, Busse R. Major differences in ICU admissions during the first and second COVID-19 wave in Germany. Lancet Respir Med Elsevier; 2021 May 1;9(5):e47.–e48. PMID:33684356 [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1016/S2213-2600(21)00101-6&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=33684356&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2023%2F03%2F21%2F2022.07.18.22277750.atom) 41. 41.Wiesner B, Bachmann M, Blum T-G, Forchheim S, Geiseler J, Kassin A, Kretzschmar E, Weber-Carstens S, Westhoff M, Witzenrath M, Grohé C. Aufgaben der Weaning-Zentren im Pandemiefall COVID-19. Pneumol Stuttg Ger 2020 Jun;74(6):358–365. PMID:32294763 [PubMed](http://medrxiv.org/lookup/external-ref?access_num=32294763&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2023%2F03%2F21%2F2022.07.18.22277750.atom) 42. 42.Kumpf O, Nothacker M, Braun J, Muhl E. The future development of intensive care quality indicators – a methods paper. GMS Ger Med Sci German Medical Science GMS Publishing House; 2020 Oct 30;18:Doc09. [doi: 10.3205/000285] [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.3205/000285&link_type=DOI) 43. 43.Artis KA, Dweik RA, Patel B, Weiss CH, Wilson KC, Gagliardi AR, Huckson S, Nothacker M, Adhikari NKJ, Kajdacsy-Balla Amaral AC, Barbash IJ, Carlos WG, Costa DK, Metersky ML, Mularski RA, Sjoding MW, Thomson CC, Hyzy RC. Performance Measure Development, Use, and Measurement of Effectiveness Using the Guideline on Mechanical Ventilation in Acute Respiratory Distress Syndrome. An Official American Thoracic Society Workshop Report. Ann Am Thorac Soc 2019 Dec;16(12):1463–1472. PMID:31774323 [PubMed](http://medrxiv.org/lookup/external-ref?access_num=31774323&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2023%2F03%2F21%2F2022.07.18.22277750.atom) 44. 44.Becker M, Böckmann B. Personalized Guideline-Based Treatment Recommendations Using Natural Language Processing Techniques. Stud Health Technol Inform 2017;235:271–275. PMID:28423796 [PubMed](http://medrxiv.org/lookup/external-ref?access_num=28423796&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2023%2F03%2F21%2F2022.07.18.22277750.atom) 45. 45.Fazlic LB, Hallawa A, Schmeink A, Peine A, Martin L, Dartmann G. A Novel NLP-FUZZY System Prototype for Information Extraction from Medical Guidelines. 2019 42nd Int Conv Inf Commun Technol Electron Microelectron MIPRO 2019. p. 1025–1030. [doi: 10.23919/MIPRO.2019.8756929] [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.23919/MIPRO.2019.8756929&link_type=DOI) 46. 46.Mandel JC, Kreda DA, Mandl KD, Kohane IS, Ramoni RB. SMART on FHIR: a standards-based, interoperable apps platform for electronic health records. J Am Med Inform Assoc JAMIA 2016 Sep;23(5):899–908. PMID:26911829 [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1093/jamia/ocv189&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=26911829&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2023%2F03%2F21%2F2022.07.18.22277750.atom) 47. 47.CDS Hooks [Internet]. [cited 2021 Sep 9]. Available from: [https://cds-hooks.hl7.org/](https://cds-hooks.hl7.org/) 48. 48.Paton C, Karopka T. The Role of Free/Libre and Open Source Software in Learning Health Systems. Yearb Med Inform Georg Thieme Verlag KG; 2017 Aug;26(1):53–58. [doi: 10.15265/IY-2017-006] [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.15265/IY-2017-006&link_type=DOI)