Design, EHR Integration and Evaluation of Clinical Decision Support Workflows Driven by a Mortality Prediction Model to Promote Goal Concordant Care^*

Clinical decision support design

Goals of care conversations should happen early and often following an initial cancer diagnosis. We worked backwards towards this objective by prioritizing development of CDS targeting admitted patients predicted to be near end of life to maximize the impact of suggested interventions and build buy-in before expanding our framework to cover earlier timepoints and broader patient populations.

The CDS tools are summarized in Table 1. They include interruptive alerts as well as passive patient flags, icons within inpatient and clinic schedule columns and dynamic order presentation rules. Interruptive alerts were used judiciously and implemented only with strong support from clinical leaders and review by appropriate EHR governance, to introduce intentional pauses in clinical workflows at critical timepoints where patients nearing end of life were missing actionable documentation to guide goal-oriented care.

Advance directive completion alert

The first model-driven alert was launched in February of 2021 to increase AD completion and facilitate assessment for GOC discussions on moderate to high-risk patients. This AD completion alert (Figure 2.L) is presented to CSWs opening the chart of an admitted patient with a model risk score above a decision threshold. We set the threshold with the intent of achieving 50% precision (or positive predictive value), associated to medium mortality risk. While ACP and AD completion can benefit cancer patients at low risk of mortality, a higher threshold was chosen to balance opportunity with clinical capacity. Clinical leaders determined AD completion for patients meeting this threshold would be best facilitated by CSWs, making CSWs the logical recipients for the alert. CSWs are presented with multiple acknowledgement reasons to either indicate follow-up actions for AD completion or justify why an AD cannot be completed. The acknowledgement reason selected drives the interval for which the alert is prevented from appearing for the same individual on the same admission, ranging from three hours to 30 days.

• Download figure
• Open in new tab

Figure 2.

Advance directive completion alert (L) and icons presented in clinic schedules and inpatient lists (R)

Goals of care alerts

Next, in collaboration with hematology and oncology, we implemented alerts to increase SM and CSW engagement in GOC conversations for seriously ill admitted patients. We set the decision threshold on the model predictions with the intent of achieving 60% precision. If the model “flags” an admitted patient without recent GOC documentation or SM consult, the alerts are presented to hematologists, medical oncologists, intensive care physicians, hospitalists, and advance practice providers (APPs) on the care team when they place orders. Based on pre-implementation feedback from hematologists, patients within one month of a bone marrow transplant or CAR-T infusion were excluded due to anticipated variability in lab results which providers were concerned would result in overestimation of mortality risk. Our evaluation of the model during the silent pilot confirmed a decrease in predictive performance for such categories of patients. Within the GOC alert, a 90-day mortality risk score and the top six clinical factors contributing to the score are presented and the provider is asked to indicate whether they believe 90-day mortality is very likely, likely, unclear, unlikely or very unlikely. The alert suggests a consult to CSW for a GOC conversation with an option to add a SM consult. If no orders are placed, the provider is required to indicate a reason using one of five pre-defined acknowledgement options. Selection of one of these options suppresses presentation of the alert for a set timeframe between five hours and 14 days. The alerts were set up to run silently without presentation to clinicians for the first two weeks to verify the volumes of notifications. For the first six months, prognostic information and order suggestions were presented to clinicians in two alerts presented simultaneously upon opening the chart. These were later combined into a single alert that displays upon signing orders for efficiency and better alignment with clinical workflows.

Goals of care flag

In October of 2021, we extended use of the ML model to the ambulatory clinic setting by using it to trigger a silent (not presented to clinicians) alert that automatically applies a patient-level GOC flag for Hematology and Medical Oncology patients with advanced cancer predicted to be at high risk for mortality. Advanced cancer definitions were established by Hematology and Medical Oncology departments using ICD-10 diagnoses, lines of chemotherapy and performance status. Again, patients near bone marrow transplant and CAR-T infusion were excluded. In addition to the GOC flag, a new GOC column was added to the provider schedule and inpatient lists. This column automatically populates with one of four possible icons that serve as a quick visual indicator of GOC conversation and AD completion status for each advanced cancer patient with a GOC flag (Figure 2.R). While outside the scope of this analysis, supporting EHR build was also implemented to enable scheduling of GOC appointments, a patient-facing questionnaire and standardized documentation for GOC conversations.

Supportive medicine consult automation for ICU transfers

Application of the ML model was further expanded in the hospital setting in January 2022 to dynamically present different consult defaults to providers transferring Hematology and Medical Oncology patients to the ICU. Prior to this time, SM and CSWs were consulted on most ICU transfers which was not always beneficial. To address this, the consult sections of ICU admission order sets were updated to present pre-selected SM and CSW consult orders if the patient in context is predicted to be at high risk for mortality. For patients not at high risk, SM and CSW consult orders are presented but not selected by default. Several weeks later, to cover cases where an ICU admission order set is not used, an update was made to present an interruptive alert to automatically add SM and CSW consults each time a standalone ICU transfer order is placed ordered for high-risk patients. Our evaluation of the silent pilots motivated the consult automation application because it showed that the model could achieve a higher recall on near death patients than the previous approach while maintaining the same volume of consults.

Clinical implementation approach

We designed CDS and corresponding workflows in partnership with departmentally endorsed clinical champions, with input from clinical, operational, and technology-focused committees. We estimated volumes of alerts and predictive performance tradeoffs based on the silent deployments run throughout 2020. This information was shared with stakeholders to provide assurance that the proposed CDS tools would benefit patients, families, and the care team while respecting staff capacity and minimizing alert fatigue. We approached the initial development of each CDS tool by working with clinical champions to understand current state workflows and identify points at which CDS could be most seamlessly integrated to achieve intended outcomes. CDS solutions were then iteratively prototyped in Epic, presented to the appropriate stakeholder groups and adjusted per their feedback. Prototypes and formal requirements documentation were then handed off to the IT team for build. Clinicians were proactively engaged for user acceptance testing (UAT) and a final round of design and build adjustments were made to address their feedback. In parallel, CDS tools were socialized at department and division meetings and communicated to impacted staff. Post-implementation changes followed a similar path.

Evaluation

We used Epic reporting tools to regularly monitor alert volumes and interactions following implementation. We met individually with clinicians experiencing the highest volumes of alerts to obtain their feedback and better understand unexpected alert interactions. Several group discussions were also conducted to provide opportunities for clinicians to raise issues and suggest enhancements. Information gleaned from regular monitoring and user interactions was used to further refine and enhance CDS tools post-implementation. In some cases, clinicians provided specific improvement suggestions which we reviewed with clinical champions, prototyped for consideration, and implemented if deemed appropriate. When feedback provided was more general or problem-based, we formulated solutions and presented these for consideration and decision. The AD completion alert was also formally evaluated using an anonymous survey sent to CSWs to assess usability and perceived impact the alert.

Performance of the advance directive completion alert

We considered the February 16^th – October 14^th, 2021 period, choosing the stop date for the evaluation to account for delays in communication of death. The time frame chosen for the overall performance evaluation of the model in Stage I of the Evaluation Showcase was two weeks shorter.¹² In the Stage I submission, we evaluated performance over all the predictions made by the model during the time frame¹². In this work, we focused only on predictions for hospital patients without an AD (the target population for the AD completion alert) and considered only one prediction per patient encounter: either the first prediction above the threshold, or the last prediction before discharge.

Figure 3.L shows the weekly counts of AD alerts along with patients and providers involved. Figure 3.R shows the fraction for different subsets of acknowledgment reasons. We observed an increasing number of charts navigated by the CSWs since the go-live of the alert. As the plot shows, the same alert could fire multiple times until AD completion, depending on the chosen acknowledgment reasons. Figure 4.L shows the different counts of acknowledgment reasons. Figure 4.R shows the distribution of the volume of notifications across different CSWs. For two CSWs in supervising roles who navigate a larger volume of patient charts than non-supervisory CSWs, the AD alert fired over 90 times, i.e. on average 12 times a month.

• Download figure
• Open in new tab

Figure 3.

Weekly counts of firings, patients, unique alerts and providers for the AD completion alert (L). Weekly fractions of aggregated alert acknowledgment reasons for the same tool (R).

• Download figure
• Open in new tab

Figure 4.

Counts of acknowledgment reasons chosen by CSWs for the AD completion alert (L). Distribution of the number of notifications over the number of providers over the Feb 15 – Oct 14, 2021 period.

The Receiver Operating Characteristic (ROC) and Precision-Recall curve (PRC) for the target subpopulations are displayed in Figure 5.L,R. The model achieved 0.83 (95% CI: 0.81 – 0.85) area under the ROC (AUROC), 0.4 (95% CI: 0.36 – 0.44) AUPRC and 0.08 Brier score with 9.8% 90-day mortality prevalence (n=6,324). AUROC measures the ability to discriminate between a positive (patients who die within 90 days of a prediction) and negative (patients who are still alive 90 days after a prediction) observation: a score of 0.5 indicates random guessing. AUPRC is a measure of performance with values between 0 and 1. The value of AUPRC associated to random guessing would be equal to the 90-day mortality rate in our population.

• Download figure
• Open in new tab

Figure 5.

Receiver operating characteristic (L) and precision-recall (R) curves for the 90-day mortality prediction model over the target population for AD completion alert, with recall and precision for model and alert (R).

With the decision threshold we selected, the model achieved 48.5% recall and 38% precision. A prediction above the threshold is necessary but not sufficient condition to trigger the alert. The AD alert necessitates a CSW to navigate a patient’s chart as an additional condition for firing. CSWs do not have the capacity to navigate every admitted patient’s chart before discharge. This resulted in 34% recall and 44% precision for the AD alert, different from the model. Model and alert performance are summarized in Table 3, along with 95% confidence intervals (CIs). There is an overlap between CIs of precisions for model and alert. However, the evaluation of the difference between precisions of model and alert gave a 95% CI of (0.4-12.3%), indicating a statistically significant increase in precision (p < .05) for the alert. The difference in recalls was even higher: (95% CI: −17.5 - −11.9%).

Survey on advance directive completion alert

User feedback for the AD completion alert was obtained via an anonymized survey of CSWs who received the alert at least once. The survey consisted of six statements with mixed positive and negative framings and Likert scale response options, two workflow questions with pre-defined response options, and an opportunity to provide general feedback in narrative form. Of the 29 CSWs surveyed, 23 provided responses (79.3% response rate). 95.7% of respondents agreed or strongly agreed with the statement “It is simple to interact with and understand the alert.” None disagreed or strongly disagreed. When provided the statement “This alert helps me prioritize the patients I approach for advance directive completion”, 65.2% agreed or strongly agreed. Only 17.3% agreed or strongly agreed with the statement “I receive too many alerts for advance directive completion across all patients I see.” When asked “Would it be helpful to regularly receive a list of high-risk admitted patients to avoid potentially missing those whose charts you do not open?”, 43.5% responded “Yes, but I do not have bandwidth to consistently act on this information.” The remaining responses were split between “Yes” and “No, I always open the chart for my assigned patients so this would not provide me with any new information.”

Advance directive completion alert

CSWs prioritize seeing hospitalized patients at highest risk, including patients identified from clinical huddles and referrals. We observed that the CSWs navigated the charts of about 61% of the instances identified by the model (Table 3). The most obvious consequence of this is a reduction in recall for the alert to 34.1% from the 48.8% achieved by the model. The reduction in median lead days between model and alert is due to the gap between the time of flagging and the navigation time. We expected a precision of around 50% for the threshold we chose based on performance of the pilot version of the model in 2020 but actual precision of the model on the target population was 38%. The operating precision associated with the alerts was higher: 44%. During the 2020 prospective pilot we observed a prevalence of 15% which is higher than 9.8% prevalence observed thus far. Note that AUROCs for model post go-live and 2020 silent pilot were remarkably close with only a 1% difference. We believe that higher 2020 prevalence was mostly associated to inpatient population shifts due to the COVID-19 pandemic and that the subsequent drop in prevalence was one the factors that contributed to overestimating the precision of the model. The lift in precision of the alert seems associated with the fact that CSWs are required to prioritize the most seriously ill patients. While the alerts had higher precision than the model, the recall, not surprisingly, was lower due to fact not all the charts of the flagged patients were navigated.

Several months after implementation, CSWs reported a challenge with the alert design. For newly assigned patients, the alert is presented before their chart can be reviewed to assess. As a workaround, CSWs were selecting “I am not the assigned CSW”, resulting in the high utilization of this option as pictured in Figure 4.L. This response prevents the alert from reappearing for the receiving CSW for the next seven days. To address this, a new “Will review chart” option was added that allows CSW to temporarily proceed past the alert, suppressing reappearance for just three hours.

The AD completion alert survey results indicate that overall, CSWs feel the alert is easy to use and serves the intended purpose of assisting in prioritization of patients for AD completion without exceeding their capacity. Feedback from almost half of respondents indicating that eliminating the chart navigation requirement to instead provide notification for all high-risk patients would exceed their capacity reinforces our capacity-conscious design approach. A similar survey to assess clinician perspectives for the GOCs alerts is currently underway. In addition to the survey results, adoption of the model as the primary pathway for identification of patients for ACP interventions suggests broad user acceptance. CSW, Hematology and Medical Oncology departments recognize the prognostic model as a valid way to identify patients for ACP interventions and have ingrained it in their care delivery processes.

Key lessons learned

Operationalizing use of this ML model at our organization was an institutional learning experience. Continued collaboration between the project team, IT department and clinicians surfaced improvement opportunities and enabled swift adaptations in response. Here we describe several experiences of this process in action and the insights gained as a result.

Shortly after implementation, we discovered a discrepancy between the volume of SM consults accepted from GOC alerts and the total volume of signed consult orders. Further investigation led to a realization that orders accepted from alerts often go unsigned. Learning this alongside our clinical partners allowed us to avoid a scenario where the alert was deactivated due to perceived failure to achieve expected process outcomes. Instead, there was immediate operational support to quickly pivot from presenting the alerts upon opening the chart to presenting them within the ordering workflow. This experience heightened our awareness of the importance of understanding EHR utilization patterns prior to deploying new CDS tools and has prompted broader evaluation of provider interactions with all alerts containing order suggestions.

Operational readiness was both an enabler and barrier to successful CDS implementation. The project team spent substantial time meeting with departmental leaders to explain the model, share performance findings and CDS prototypes and review detailed projections around anticipated volumes of interruptive alerts and SM ICU consults. While time-consuming, we believe this work was instrumental in building support for eventual implementation of these tools by providing a forum for collaborative, data-driven cost/benefit discussions. Similarly, engaging CDS users in UAT and post-implementation feedback meetings and directly incorporating their suggestions as enhancements surfaced use cases that likely would not have been identified by the project team, while also reinforcing appropriate use of CDS tools and conveying a commitment for continued improvement. Conversely, inpatient providers frequently expressed discomfort around initiating GOC conversations in the acute care setting, due to patient acuity and widely held beliefs that primary oncologists and hematologists who typically have a longstanding relationship with the patient are better positioned to facilitate these discussions. While this feedback helped to accelerate implementation of the GOC flag and supporting ambulatory workflow, engagement of clinical leaders to set clear expectations around shared ownership of GOC conversations in advance of implementation may have helped to reinforce the role of inpatient providers in initiating these important conversations in the acute care setting.

When operating in dynamic healthcare environments, maintaining a holistic view of the entire ecosystem of ACP and end-of-life workflows and evolving these to maintain a streamlined clinician experience is crucial. Initially, pre-existing CDS to suggest SM engagement for patients manually identified for GOC discussions using departmental criteria was not deactivated in parallel with the introduction of the GOC alerts and GOC patient flag. This resulted in confusion and duplication of work by the model and APPs. This has since been addressed and has enabled a shift in clinician focus from the administrative task of identifying patients for GOC conversations to improved capture of information important for overall patient care, such as ECOG performance status. Mindfulness of the end-to-end clinician experience with constant re-evaluation of CDS is required to craft a unified user experience.

Limitations

Recent implementation, frequent design changes, informal feedback collection methods and delays in recording of external deaths limit the volume of data available for analysis. This is further compounded by the fact that CDS was deployed broadly across entire clinician and patient populations rather than designing randomized controlled trials (RCTs), limiting the ability to draw direct cause-and-effect conclusions around patient outcomes. Recent work demonstrates the feasibility of using rapid RCTs to evaluate CDS embedded in EHRs and opportunity exists to evaluate future CDS using these methods.¹³

Future work

The 90-day mortality risk prediction model is now an integral component of efforts to provide goal-concordant care at our organization. We are building on this work by developing additional CDS interventions, including alerts to suggest hospice education or admission based on specific risk thresholds and documented patient preferences. We are also developing workflows that will leverage the model to support recruitment for studies involving post-mortem tissue collection. The need to improve patients’ end of life experiences continues to be the driving force behind this work. As such, our future roadmap also includes constant evaluation of the impact of model-driven CDS in relation to patient outcomes. This will be the focus of our Stage III submission.