Development and exploratory analysis of a multi-dimensional metric of adherence for digital health interventions

1. Introduction/Background

The use of technology in healthcare (known as digital health) has proliferated over recent years due to their potential in enhancing efficiency, responsiveness, personalisation, and ability to embed behaviour change techniques to improve outcomes. Digital health encapsulates the intersection between healthcare and emerging or established technologies¹ and include mobile health, wearable devices and telehealth. A prominent example of an emerging technology is gamified telerehabilitation, also known as exergames². Exergames are active video games that use body movements in physical exercise as virtual controls within the gameplay. Exergames provide opportunities for home-based rehabilitation training and longitudinal participant monitoring^3,4 that would be costly to achieve with traditional physiotherapy.

One common concept used to define the success or failure of digital health approaches is “adherence”. There are multiple ways in which adherence can be defined, all of which encapsulate one of more aspects of engagement a user has with this new approach⁵. Demonstration of adherence is a necessary causal precursor to any clinical outcomes⁶. In the context of exergames, adherence is of particular interest as a measure of increased or decreased use through the instantaneous feedback that exergames are expected to provide. In addition, the gamification elements of exergames (such as points and level ups) may provide additional incentives⁷ to complete a prescribed programme (to ensure sufficient dose, challenge and progression) that would not be present in traditional physiotherapy.

Despite the promising performance of exergames in different participant groups (e.g., older adults^8,9, and children and adolescents^10,11), definitions of adherence vary, and relatively few studies compare adherence in exergames to other forms ofexercise². Exergames fall at the intersection of healthcare, exercise, and technology, all of which typically have different motives when defining adherence. The World Health Organisation (WHO) definition of adherence focuses on provider-defined change (“the extent to which a person’s behaviour – taking medication, following a diet, and/or executing lifestyle changes, corresponds with agreed recommendations from a health care provider”)¹². Technological approaches may focus instead on usage, and the distinction between usage and other aspects of engagement in digital Health can be inconsistent¹³. Different terms for adherence in separate fields (such as concordance or compliance) only serves to further obfuscate matters.

Skjæret et al.’s systematic general review of exergames² found that, of the studies they investigated (n=54), over 10% did not report the number of participants that completed the study. Almost half did not report on the completed sessions during the intervention, and for the rest the adherence methods varied between reporting raw number of sessions completed, the percentage of sessions completed, and the degree of accuracy with the prescribed intervention. Many of these descriptions of adherence may prove useful, but the inconsistency in their meaning and measurement makes it hard to draw any systematic conclusions about how to define and assess this important concept.

From a user perspective, monitoring adherence is also important as it can enable a digital tool to provide useful feedback in terms of how often and how well someone follows an exercise regime, or not, and give tailored advice that could improve health outcomes long-term. From a service perspective, adherence data is also valuable to help plan and manage the demand for digital health services locally and nationally and how these can be adequately resourced and delivered. From a research perspective, assessing adherence can be beneficial and viewed at either a personal level or a population level. The personal level (e.g., comparing completed vs. assigned sessions of exercise at any given time) may have practical purposes of real time adjustment of workload or predicting dropout. At the macro-level, analysing adherence data could be useful for improving system/technology design and drawing conclusions about the overall effectiveness of research studies on telerehabilitation.

Regardless of the individual or organisation monitoring adherence, the term and its measures should be applied consistently. Useful dimensions of adherence relevant to digital health should be separated from dimensions of adherence related to other types of interventions which are more universally calculable. Furthermore, conclusions in one study drawn based on adherence can only be compared with another study if the same core underlying definition of adherence is clearly defined and agreed upon.

Here, existing scientific literature is used to define a set of individual and common dimensions of adherence relevant to digital physiotherapy interventions (telerehabilitation). Then a set of metrics to quantify each dimension is proposed to provide a holistic set of metrics for adherence. This set of metrics is applied on a digital physiotherapy dataset and exemplar conclusions are drawn about what these different adherence metrics tell us about this dataset, and to provide a basis for future adherence reporting.

2. Methodology

3. Results

Figure 1 shows how adherence as initial adoption varies depending on the base threshold used (ɵ_s). The figure shows that one participant only completed 3 sessions, and a second participant completed only 4 sessions instead of the full 36 sessions. All other participants completed at least 8 sessions. However, the lower graph shows that a third person only completed at least 3 “session days” – meaning they were highly active during the days they did try the system but did not stick with the programme beyond those days. Different digital physiotherapy interventions may find differing thresholds and measures to be useful here.

• Download figure
• Open in new tab

Figure 1:

A figure showing the number of participants who are still active in the clinical trial, given a certain number of (top): sessions or (bottom) unique sessions days. The study started with 56 participants. Participants were asked to complete 3 sessions a week. The dotted line represents ɵ_s, in equation 1 the threshold used to define initial adoption vs initial non-adoption. As this metric is focussed on initial adoption, only the effect of the threshold ≤21 days is shown here.

Figure 2 shows the consistency measure applied to the trial dataset to gauge this dimension of adherence (Eq. 2). Here, only a full week of completed sessions (recommended as ≥3 sessions for that week) is considered in the analysis. The number of ≥3 session weeks is then tallied for each participant. This graph is used to define how many participants consistently adhered to their recommended plan. All participants beyond the red line completed all 36 sessions at the recommended level of exercise. 45/56 participants (80%) failed to complete ≥3 sessions a week for 12 weeks. Some participants made additional recordings on the digital physiotherapy intervention once their initial 12 weeks were completed. The top subfigure is useful for identifying common stopping points within the programme (e.g., a considerable number stopped after achieving only five full weeks, just short of halfway goal of the programme). The bottom subfigure is useful for identifying the number of participants that achieved a certain consistency metric (e.g. 8 participants failed to complete ≥3 sessions a week for at least 2 weeks).

• Download figure
• Open in new tab

Figure 2:

The number of participants that achieved ≥ 3 sessions a week for N weeks, as defined by Eq. 2. The thin red vertical line at 11.5 weeks indicates that all the participants beyond that line completed the recommended exercise plan. Some participant data is recorded beyond the end of the study. The top graph shows the distribution of total weeks completed; the bottom graph indicates the cumulative number of participants who failed to achieve a consistency threshold for specific number of weeks.

Figure 3 demonstrates how duration of exercise can be used as an adherence metric (Eq. 3). Similar to the consistency measure, an activity threshold is used to define an appropriate level of engagement over a timeframe. Here, only weeks in which a participant achieved either 20 minutes per week or 30 minutes per week (for the left and right subplots respectively) are considered, as this was the recommended dose of the digital physiotherapy intervention. 34/56 participants (61%) failed to complete at least 20 minutes per week for 12 weeks, as opposed to 43/56 (77%) of participants who failed to complete 30 minutes per week over the same period. The figure can be used to help determine how representative these results are (e.g. 3 participants reached the threshold for 11/12 weeks for both thresholds but would not be represented in a final summative report).

• Download figure
• Open in new tab

Figure 3:

The number of participants that achieved ≥20 (left graphs) or ≥30 (right graphs) minutes a week for N weeks (Eq. 3). The thin red vertical line at 11.5 weeks indicates the recommended study length. Some participant data is recorded beyond the end of the recommended exercise plan. The top graphs show the distribution of total weeks completed; the bottom graphs indicate the cumulative number of participants who did not meet the duration threshold for a given number of weeks.

The adherence as dropout metric is displayed in figure 4. The top graph mimics the Kaplan-Meier graph used in clinical trials, displaying the use of the telerehabilitation system by the length of time that any participant was still involved in the study. It does not indicate the consistency or level of exercise during the programme. It serves as a more naturally interpretable visualisation of dropout, with a high initial compliance followed by a steeper decline after 75 days. It is worth noting that 84 days would have been the natural stopping point for any individual who completed the full 12 weeks, although the nature of this dataset means that records of many individuals who completed further sessions were available for analysis. Although 84 days was the initial cut-off, it is still worth noting that 80% of individuals made it past the 75-day mark. The bottom graph of figure 4 measures the percentage of sessions completed, out of a theoretical maximum of 36. Any sessions completed beyond the initial 36 are ignored. The bottom graph more clearly indicates the progress through a programme for each participant but does not encode any time information. The AUC of the bottom graph is 0.723, where an AUC of 1 would represent zero dropouts throughout the duration of the study.

• Download figure
• Open in new tab

Figure 4:

Adapted Kaplan-Meier plots, showing the rate of dropout (called “survival” in traditional Kaplan-Meier) by participants across the recommended digital exercise programme (Eq. 4). Top: the number of days between the first and the last session; Bottom: The % of sessions completed out of the recommended 36.

Figure 5 shows the relationship between the separate dimensions of adherence. Initial adoption serves as the simplest metric, with the two exclusions also having low consistency and duration of exercise. The 30-minute duration threshold shows lower visible correlation with the consistency and dropout metrics than the 20-minute duration threshold. While correlation for the consistency, 20-minute duration, and dropout metrics is high it is not perfect, indicating that these dimensions are capturing the same underlying data, but capturing slightly different perspectives of when a given participant may stop adhering to the study protocol.

• Download figure
• Open in new tab

Figure 5:

The relationship between adherence metrics. Each participant is assigned a normalised score for each metric. The diagonals show the score distribution for each metric, and the off-diagonals show the pairwise relationship.

While group population metrics are vital to understanding the overall adherence to a digital physiotherapy intervention, it is also important to visualise exercise curves for each participant so individual level adherence can be gauged. Figure 6 shows four participants who all did less than 36 sessions, but who all had quite different progression curves. Participant 1 completed 24 sessions in 9 weeks (2.7 sessions/week average), with a clear gap around week 7. Participant 2 only missed one session in week 9, and otherwise completed 35 sessions in 12 weeks. Additionally, Participant 2 completed at least 35 minutes of exercise each week while also increasing their total weekly duration throughout the study. Participant 3 had a long gap around weeks 9 and 10, but otherwise had a similar average completion to Participant 2. Participant 4 had a long duration of exercise, with two separate one-week gaps early in the study. These data could be provided to each participant so they understand their daily and weekly exercise progress and personalised recommendations based on these individual data could be developed and provided via electronic notifications so participants can improve their fitness regime.

• Download figure
• Open in new tab

Figure 6:

The consistency and exercise duration metrics of a subset of participants in the study. The red line indicates the 12-week line, by which an “ideal” participant would have completed 36 sessions.

4. Discussion

The secondary analysis of the trial dataset has enabled a more in-depth exploration and development of key adherence metrics for a digital physiotherapy intervention.

The choice of ɵ_s = 4 sessions per week serves as a useful cut-off point for initial adoption (i.e., accepting participants who completed over an equivalent week of recommended sessions) in relation to this intervention, but other baseline usage and timeframes may be appropriate for other types of digital health interventions. Two participants were dissuaded from continuing from this point onwards, yielding a 3% rate of initial dropout. All other participants continued until nearly the 3-week mark or further. These participants have not been excluded from any subsequent measures to allow for cross-dimensional analysis.

Some reasons for dropout are listed for this dataset by Stanmore et al.¹⁴, with some motivations being potentially linked to the intervention (e.g. disinterest with technology), while some are separate (e.g. becoming medically unfit or family issues).

The consistency and duration of exercise are certainly linked (Fig. 5), although they are still worth considering separately. The habit-forming nature of regular exercise is a vital indicator of long-term success, especially in the case of physiotherapy^23,24. Adherence as duration is a marker of the quantity of exercise and might be much better to pair with any physical outcomes (e.g., increased range of movement). It is worth noting that this study did not assign a fixed duration of exercise to the participants, which is likely why the ≥30-minute threshold of Fig.5 produced such spread results when compared with consistency. When considering the full plan only 20% of participants successfully completed ≥3 sessions for 12 weeks (median: 5.5 weeks), only 23% completed ≥30 minutes of exercise a week (median: 5.5 weeks), and 39% completed ≥20 minutes of exercise a week (median: 9 weeks).

“Adherence as duration” is particularly critical if analysing outcomes such as falls for preventative programmes⁴, which suggest 180 minutes/week of progressive exercise for the greatest effects²⁵. Conversely, an older adult is advised to complete 150 minutes/week of exercise according to physical activity guidelines²⁶, and so it should be noted whether an exercise plan is given above and beyond this baseline amount. “Adherence as consistency” can be useful in considering target implementation programmes, tailoring a programme to best fit a participant’s personal routines.

The dropout metric is a much more useful gauge of a whole study, and it would be interesting to observe in future studies how the curves change for different digital health programmes. There were two separate plots used for dropout for this study (Fig. 4). In many cases, the sessions will be of a non-fixed duration, and so the time-based measure of dropout will have to be used. Where the digital exercise programme is more structured, it is likely that the percentage-completion metric will prove more useful to researchers. The time-based dropout measure showed that over 80% of participants had at least 5 days between their first and last recorded session. The AUC of the percentage-completion graph was 0.723, which could be interpreted as 72.3% of the assigned sessions were completed (discounting additional sessions completed beyond the 36). It is also interesting to note that, despite those who completed the 36 sessions all having an a_dropout score of 1, there were a range of consistency and duration values amongst that subgroup (Fig. 5).

The four dimensions – initial adoption, consistency, duration, and dropout - all provide distinct information on the adherence to a digital health intervention, while also identifying specific areas where improvements may be required to match the recommendations of the programme. A low initial adoption/usage rate could indicate an overly complicated software interface or hardware component, poor Internet access, and/or low levels of digital literacy, among other issues that prevent people using digital health tools²⁷. A low consistency could be improved with a reminder system. A low duration of exercise might be addressed with reducing the intensity. If the dropout is extremely high at a given point, it might be worth observing whether the digital exercise programme introduces a difficult exercise at that point.

While it is possible to reduce each of these adherence dimensions to a single summative statement, it is important to note the nuance lost in doing so. Figure 6 shows four participants that each failed to complete the full 36 sessions. Participant 2 missed a single session, and to all intents would likely have very similar outcomes to a participant who completed the full programme. Participant 3 also essentially completed all sessions, but did so over a slightly longer time period, which may be a confounding factor for some programmes depending on the degree of time extension. Both participants completed a much greater percentage of exercise than Participants 1 and 4, and so it may be useful to consider adherence as a more continuous metric that would encapsulate these variations.

Evaluating the efficacy of the trial by comparing the level of improvement between those who completed the assigned set of exercise and those who did not was considered beyond the scope of this paper. It is noted that the duration and consistency metrics could prove an interesting variable to evaluate such outcomes, and that such an analysis would remove the issue of reducing adherence to a single metric. The use of population adherence metrics and individual adherence metrics provide a chance to evaluate overall study effectiveness, as well as identifying individual behaviours within a study. Additionally, examining how these metrics vary across demographics (e.g., age, gender, ethnicity, sexuality, level of education) could provide more precise information about characteristic adherence in separate populations. Finally, a metric designed to capture the “intensity” metric of adherence could also prove useful in efficacy calculations, provided it follows the same general principals established here (e.g., be able to be applied fairly between digital health and non-digital health datasets).

Here, separate adherence metrics have been developed to provide distinct insights into how participants use a given digital health intervention. While “sessions” were distinctly defined here via recommended guidelines, they could also be more broadly defined as “periods of continuous exercise” or “days containing exercise” for other interventions. It is important to note that there are areas where it is harder to reliably obtain information in non-virtual settings. Follow-up questionnaires are often used to gauge levels of exercise, which are more subject to drop out bias and require additional involvement from participants when compared to the automatic data collection of a digital tool. Hence, duration of exercise will also be far less precise in such circumstances.

5. Conclusion

In this paper, four complementary measures of adherence, initial adoption, consistency, duration of exercise, and dropout, to digital health interventions were developed and applied to an exergames dataset from a clinical trial. This dataset had low initial dropout (3%), but few participants completing the full course of the study (20% completed 3 sessions per week for 12 weeks, 39% completed 20 minutes per week for 12 weeks). Only 72.3% of the recommended exercise sessions were completed. Each of these metrics provides distinct information regarding the overall adherence to a digital physiotherapy intervention. The specific types of adherences evaluated are clearly explained and reported, allowing for a clear understanding of the fidelity of different aspects of the programme. The clear reporting of these metrics could prove useful in improving the relevant technology platform and user experience moving forwards.