Red flags for remote cognitive assessment: An expert consensus study using the Delphi method on behalf of the Canadian Consortium on Neurodegeneration in Aging

Workgroup creation

The CCNA telemedicine workgroup came together under the auspices of the CCNA and was tasked with advancing guidance for remote cognitive assessment and diagnosis in dementia care. In forming the group, steps were taken to ensure multidisciplinary participation by inviting experts throughout the CCNA network. The workgroup within the CCNA was composed of an expert multidisciplinary team of behavioral neurologists, neuropsychiatrists, neuropsychologists, social workers, geriatricians, and family physicians.

Study design

In our study, the Delphi method was used to find consensus amongst the telemedicine workgroup within the CCNA. This method is particularly useful in the medical field, where evidence is often conflicting or insufficient [28,29]. The main advantages of the method are that it avoids eminence bias, can be performed remotely and asynchronously at low cost, and allows individuals to consider the group opinion when formulating a response. In a three-round process, our participants first proposed data, then evaluated and re-evaluated data (Figure 1).

• Download figure
• Open in new tab

Figure 1 –

Overview of the Delphi process implemented in the current study.

The study took place between January 19, 2023, and September 20, 2023. Data collection for each round was performed using web-based surveys (Microsoft Forms). Though the surveys were distributed to a closed group, responses were anonymous, ensuring each response had equal weight.

Establishing definitions, process, and consensus criteria

An initial virtual meeting with the workgroup served to establish the necessary prerequisites for the Delphi process [28]: This included the definition of ‘red flag’, the quality indicators for scoring candidate flags, and the consensus threshold. The goal was to have a definition of ‘red flag’ that was simple, comprehensive and encompassed the important components of a clinical encounter. The agreed upon definition of a ‘red flag’ was “Features of the patient, clinician, caregiver, context/situation (or other) indicating that a remote cognitive diagnostic assessment likely should be avoided.”

Three quality indicators (QIs) were used to score each flag quantitatively on a 5-point Likert scale [31]. The goal in choosing the indicators was to capture the important attributes of a red flag in as few indicators as possible with clear language. The first QI focused on efficacy, “This flag identified patients who are not well suited for remote cognitive diagnosis.” The second QI focused on reproducibility, “This flag is reproducible at different times and by clinicians with different levels of experience.” The third QI examined feasibility, “Searching for this flag would not significantly prolong the diagnostic process.” Each potential response on the Likert scale was rated as follows: strongly disagree=1, disagree=2, neutral=3, agree=4, strongly agree=5.

Consensus criteria were set a priori to reduce the risk of bias. We considered consensus to be met when the average score for all QIs taken together for a specific potential red flag was 4.0 or greater.

The Delphi process

The first round of the Delphi generated all the potential flags to be voted on in later rounds. It was important to capture a set of flags that covered the spectrum of opinions held by the group, as later steps generally serve to filter out flags that did not meet consensus. We designed the prompts to be as inclusive as possible and stimulate brainstorming. We prompted respondents about important components of a clinical encounter including features of the patient, caregiver, patient, or context (e.g., “What are red flags about the patient’s [caregiver] that indicate a remote assessment should be avoided?”). Participants responded anonymously by generating as many potential red flags as possible. Basic demographic information about the expert participants, including their specialty and years of clinical experience was also collected in this round.

After the first round, all the responses were pooled and screened for validity (whether the potential factor met the agreed up upon definition of “red flag”) and duplicate responses. With CCNA telemedicine workgroup feedback and approval, invalid responses were removed, and duplicate responses were combined under the response with the clearest language. The workgroup concluded that all remaining potential red flags in the set were valid.

In the second round, respondents were shown each potential red flag and asked to score it based upon the three QIs using the above Likert scale. At the end of the round, respondents were given the opportunity to recommend new red flags that they felt were missing at this point, and one additional flag was generated overall. After the second round, the group average scores for each red flag were calculated, including the average for each QI and overall.

In the third round, respondents were shown each red flag again along with the group average scores from the previous round, for each QI and overall, for all but the one flag added in the second round. They were asked to re-score each flag using the same QI Likert scale described above while considering the previous round’s averages.

After the third round, the group average scores for each flag were re-calculated. The consensus threshold was applied (average score for all QIs≥4.0). Flags with scores above the consensus threshold were included in the final set of agreed upon red flags.

Infographic

In collaboration with a knowledge translation specialist from the CCNA Operations Center, the final set of red flags was translated into a user-friendly infographic as a clinical decision-making tool. This tool was designed to aid physicians in determining if a patient is an appropriate candidate for cognitive diagnostic assessment via telemedicine. If one of the final red flags is identified before a clinical encounter, then the patient is likely not well suited for cognitive assessment via telemedicine, and switching to in-person assessment is recommended.

Statistical analysis

In the first round of the Delphi process, there were 12 respondents, possessing an average of 14.6±11.1 years (mean ± standard deviation) of clinical experience within neurology, psychiatry, psychology, social work, geriatrics, and health administration. This group generated a total of 151 flags. Only 3 were considered invalid based on the definition criterion. Of the 148 remaining red flags, 82 were duplicated flags that coalesced into the 66 unique candidate red flags.

In the second round, 8 respondents scored each candidate red flag on the three QIs (Supplemental Table 1). One additional flag was added in this round, which was a modification of a prior candidate red flag, “The patient does not have access to a caregiver during the remote assessment.” The new candidate flag from the second round was “The patient does not have access to a caregiver during the remote assessment AND is more than mildly impaired.”

In the third round, 9 respondents re-scored each flag while considering the previous round’s scores. The mean score and standard deviation for each potential red flag are shown in Table 1. Applying the consensus criterion after this third and final round yielded 14 red flags that scored over threshold, shown in bold at the top of Table 1. An infographic of the synthesized red flags was developed in collaboration with a knowledge translation specialist within the CCNA Operations Center to provide decision-making guidance about the suitability of a patient for remote cognitive assessment (Figure 2). If any of the 14 red flags in the decision-making tool are identified, a decision to shift to an in-person clinical assessment is warranted.

• Download figure
• Open in new tab

• Download figure
• Open in new tab

Figure 2 –

The infographic and clinical decision-making tool.

Infographic

The goal of this study was to create a clinical decision-making tool to determine a patient’s readiness for remote cognitive assessment (Figure 2). If a health professional identifies any of the 14 red flags, a shift to an in-person assessment is preferred. The tool was developed in English and translated to French.

The Delphi method

The Delphi method is versatile and has been used in various domains of research including finance, economics, process planning, and healthcare [28,29]. Most examples of the method retain the core elements of anonymity, iteration, feedback, and consensus, with validity of the results depending on the implementation. [28,29].

In this study, the overall design and consensus criteria were defined prior to beginning the Delphi iterative rounds. Like most studies in healthcare [29], our study used a predetermined, rather than variable, number of rounds. Our aim was to achieve a balance between allowing participants to respond to the group and reducing response fatigue. Setting a predetermined consensus threshold is recommended as it increases transparency within the group, since participants are aware how close a flag is to achieving consensus [29]. This approach also reduces the subjectivity of study operators.

The iterations in this study were carried out using online software, which allowed us to gather de-identified data and uphold the principle of anonymity. This is important to minimize the risk of individuals having disproportionate influence on the group, known as eminence bias. The online forms also allowed data collection in the context of a national workgroup where individuals were dispersed across the country, and working asynchronously over a wide range of time. Our aim in having asynchronous rounds was to minimize the burden on respondents and maximize response rate.

The red flags

The majority of the red flags that reached consensus acknowledge the importance of patient and caregiver factors that could make a remote assessment inappropriate (Flags 2, 3, 4, 6, 8, 10, 11, 12, 14, Table 1). This includes flags focused on comfort and familiarity with technology and access to electronic devices and internet connectivity (Flags 3, 11, 14, Table 1). The importance of logistical and technical considerations has been identified as a limiting factor for people living in rural or remote areas, as those without access to high-speed internet had reduced access to telemedicine during the COVID-19 pandemic [32]. In the context of remote cognitive assessment, a cross-sectional study cited limited access and technological understanding as barriers to participation [33]. Most studies evaluating remote cognitive assessment have used remote medical hubs with an existing technology infrastructure, and this is reflected in new models of telemedicine for remote cognitive assessment [26,34]. There are also many programs to increase access to high-speed internet, such as Internet for All in the United States, and the Universal Broadband Fund in Canada [35,36]. This highlights the potentially modifiable nature of certain red flags, whereby changes in community access and healthcare policy could potentially reduce the barriers to remote cognitive assessment.

The use of accessible satellite clinics could potentially address red flags identified in the current study, including access to a private, safe location, and a location that is quiet and free of distractions (Flags 4, 6, Table 1). Privacy during a medical encounter is critical for patient autonomy, where a patient is free to decide who is present in a clinical encounter. Additionally, the presence of distractions impedes cognitive performance [37], thereby reducing testing validity. Patient preference is significant; if a patient prefers an in-person assessment, this should be respected (Flag 10, Table 1).

A systematic review of remote cognitive assessment identified the presence of a caregiver as a critical facilitating factor [39]. Multiple flags identified by the workgroup underscore the critical role played by caregivers (Flags 11, 12, Table 1). One study comparing in-person and remote cognitive assessment in a memory clinic found that 68% of patients required caregiver assistance to participate in a videoconference clinical encounter [40]. Another study showed that patients with reduced cognitive function required greater caregiver involvement to complete remotely administered computerized and standard cognitive testing [41].

Current legal guidelines allow for remote cognitive assessment, but practicing outside of the jurisdiction of a clinician’s license has implications for patient privacy, quality of care, clinician liability, and remuneration [38]. Thus, legal considerations are essential to determine the appropriateness of a remote diagnostic encounter (Flag 2, Table 1). Furthermore, the group identified that a court-ordered assessment is sufficient reason to switch to an in-person assessment (Flag 8, Table 1). An in-person neurobehavioral assessment ensures that the clinician is less reliant on factors and people in the patient’s environment that may erroneously influence the diagnostic impression.

One flag that came close to, but did not reach, consensus was the presence of sensory impairment (Flag 17, Table 1). Efficacy and inter-rater reliability scored highly; however, potential challenges in collecting this information efficiently prevented this flag from achieving consensus. Previous studies have shown that participants with vision or hearing impairment have lower performance on cognitive assessments, even when adjusting for the real or simulated impairment [42–45]. Sensory impairments are increasingly common with age [1], and the presence of sensory impairment was often an exclusion criterion for studies focused on remote cognitive assessment [46]. Standardized remote methods of screening for visual impairment have been established, yet the accuracy and reliability of hearing impairment remote screening methods are not [47,48]. This suggests that valid strategies for rapid, remote screening of hearing and vision represent a unmet clinical need that would help to optimize patient suitability for remote cognitive assessment.

The group identified several clinician-related flags (Flags 1, 5, 7, 9, 13, Table 1). A fundamental prerequisite for remote cognitive assessment is the clinician’s access to secure, confidential software and appropriate technology (Flags 1, 7, Table 1) [49]. However, this can be costly and often requires administrative support that poses a potential barrier [50]. The group also highlighted the importance of clinician expertise to the validity of the remote clinical encounter (Flags 5, 9, 13, Table 1). Competency in performing remote cognitive assessment includes both knowledge of the benefits and limitations of telemedicine and expertise in neurocognitive disorders in an in-person context [13,20]. A systematic review on barriers to dementia care among primary care physicians identified concerns about knowledge of dementia, lack of training in dementia care, and discomfort with conducting cognitive assessments [51]. While training clinicians in neurobehavioral assessment is anchored by in-person learning, remote assessment is also a skill requiring exposure and practice. Early examples have shown that including telemedicine training in residency programs is desired [52], and that optimizing the residency curriculum to include training in remote care benefited residents’ competency and patient access to care [53,54].

Limitations and future directions

While this research provides guidance on virtual cognitive assessment readiness, it is important to be aware of some of the limitations of this work. First, our findings are based on expert consensus and this guidance and clinical decision-making tool have not yet been validated in real-world clinical settings. Future work could integrate feedback from clinicians that would allow for estimation of the sensitivity and specificity of each flag, along with quantitative measures of reproducibility and feasibility. Second, as with any clinical tool, information from these guidelines must be interpreted within the clinical context. This tool does not replace clinical judgment, and the clinician must make the final decision whether telemedicine is appropriate (Flag 9, Table 1).

To enhance the validity of this novel clinical decision-making tool, future studies could aim to incorporate a wider range of stakeholder perspectives. The perspectives of patients, caregivers, and other persons with lived experience on what factors warrant an in-person assessment are fundamental to providing patient-centered care.

Some of the identified red flags are potentially modifiable and represent opportunities for improving healthcare guidelines and advocacy efforts. Utilizing remote cognitive assessment for referral and diagnosis appropriately has the potential to increase access via population screening programs, as proposed by a recent study in Sweden [18]. Easily accessible internet and technological support could reduce caregiver and patient burden. Rural access to high-speed internet and health resource centers offering internet-enabled, quiet, and secure spaces are proven methods for achieving this goal [34,46,55]. The cost-benefit analysis in creating such centers should be considered against the healthcare system advantages that telemedicine affords [56]. These efforts have important public health implications with an aging population that is living longer, and work has already begun by governments to reduce the digital divide [35,36].