Using machine learning to design a short test from a full-length test of functional health literacy in adults - the development of a short form of the Danish TOFHLA

1. Introduction

Healthcare systems all over the world are developing in a way that compels patients to become more active in the management of their own health and disease – a development that changes the role of modern patients and the skills needed to navigate the healthcare system. Demographic changes resulting in more elderly people have led to increases in the burden of chronic diseases and put pressure on increasingly scarce healthcare resources (1). One strategy for overcoming this burden is to reduce the utilization of healthcare resources in the secondary sector by reducing the length of stay and placing more health care services in the primary sector, thus allowing more rehabilitation actions, where the goal is to have patients take control of their own life situation and health. This development focuses on both the ability to be an active part in shared decision making with healthcare professionals during the self-management of chronic diseases and general adherence to treatment, thus requiring that patients increase their understanding and application of health information (2) (3) (4). The World Health Organization (WHO), however, describes the presence of a paradox: increasing demands on the individual patient without the information and support necessary for making health-promoting choices (5). In the wake of this development in healthcare systems, the concept of health literacy (HL) is receiving increased attention. HL is defined by WHO as ‘the cognitive and social skills which determine the motivation and ability of individuals to gain access to, understand and use information in ways which promote and maintain good health’ (6).

As HL is a very complex concept, there is no universally accepted definition. A review by Sorensen et al. (7) found 17 different definitions of HL and 12 conceptual models. Even if there is no universally accepted definition, there are elements common to most definitions: obtaining, understanding, and applying health-related information. Don Nutbeam has described these three elements as functional (accessing health-related information), interactive (the ability to understand health-related information) and critical HL (the ability to actively use health-related information) (8). The myriad of definitions and conceptual models causes a lack of a universally agreed screening instrument to assess HL.

A recent review (9) aimed to identify the most optimal screening instrument for assessing HL in a clinical setting. The review identifies the S-TOFHLA (assessing basic numeracy and literacy skills related to healthcare) as the most widely used in the literature (used in nearly half of the studies), the REALM (medical word pronunciation test) as the second most used and NVS (the ability to identify and interpret basic text and mathematical calculations) as the third most used (9). It should be noted that the above-mentioned screening instruments are criticised for not capturing the complexity of HL (10,11) and as a result screening instruments that seek to capture the higher levels of HL has been developed (12,13). However, these instruments have a subjective approach focusing on self-experienced and self-rated abilities to perform tasks relevant to the management of health information. Thus, these more subjective screening instruments reflect self-evaluated skills in relation to the HL demands of specific hypothetical health-related situations (12,13). Further, the reliability of the subjective screening instruments can be questioned, as participants are prone to overrate their abilities, as a low level of HL is associated with shame and embarrassment (14). Despite the ongoing discussion about the nature of HL and how it should be assessed, the S-TOFHLA, REALM and NVS remain the most widely used in existing literature (9). The rapid technological development has added a new dimension to HL: the ability to consult electronic sources for information about health and use this information in relation to treatment and disease – referred to as e-health literacy (15). E-health literacy, and the assessment of this, is outside the scope of this paper.

A full-length Danish version of the full-length American TOFHLA (D-TOFHLA; D denoting Danish) was developed according to acknowledged guidelines with assessment of face validity, content validity, internal consistency etc. and has proven accurate in assessing functional health literacy (FHL) (16–18). Like the full-length American TOFHLA, the D-TOFHLA consists of two parts with a total of 67 items; the first part comprises 17 items assessing numeracy skills (e.g., prescription bottles, appointment cards that is administered by an interviewer) and the second part comprises 50 items assessing reading comprehension skills. As the D-TOFHLA (and the American) require involvement of an interviewer (e.g., healthcare personnel), it is, unfortunately, not suited for clinical routine use and it has thus primarily been used for research purposes (17,19).

This paper aimed to use machine learning to develop a short version of the D-TOFHLA (DS-TOFHLA; DS denoting Danish Short) that is easily applicable in the Danish clinical context and that does not require the involvement of an interviewer (healthcare personnel). This paper seeks to describe a generic model-based approach applied in the development of the short version of the D-TOFHLA that can also be used in the development of short versions of the TOFHLA in other languages or, in general, to develop short versions of full-length screening instruments.

2. Materials and Methods

The development of the DS-TOFHLA was based on the D-TOFHLA (17); the D-TOFHLA was created based on the original full-length American TOFHLA using the technique described by Beaton et al. (20). Similar to the original full-length American TOFHLA, the total score of the D-TOFHLA is divided into three levels: inadequate (0-59), marginal (60-74), and adequate (75-100); inadequate and marginal scores are regarded as ‘low FHL’ (19).

Motivated by a need to reduce the administration time and create an easier-to-use screening instrument, a short version of the original full-length American TOFHLA was designed: the S-TOFHLA (21). Due to the significant variations between the original American TOFHLA and the Danish version, it is not possible to translate and adapt the S-TOFHLA into a Danish version. The development of the S-TOFHLA was based on more subjective decisions and less on objective algorithm-based decisions, including linear regression models (LRM). The development of the DS-TOFHLA was based solely on an algorithm-based selection of variables and linear multiple regression (LMR). The classical method to obtain an unbiased evaluation, when building and testing models, is to have separate training and testing datasets, which can be accomplished by splitting a given dataset into a training set and a test set. In smaller datasets K-fold cross-validation is often used, which makes it possible to use almost the whole dataset for both training and testing while still avoiding bias. The present study used a special form of K-fold cross-validation, the leave-one-out cross-validation, where K is set to the number of samples (K = N), which, on the expense of more computation time during development, ensures the best possible use of the dataset (i.e., 100% of the dataset is used as training data and 100% as test data) (22).

The quality goal in the development of the DS-TOFHLA was expressed by two conditions. First, Pearson’s correlation coefficient between the DS-TOFHLA score (i.e., the predicted Danish TOFHLA total score) and the D-TOFHLA score should be at least 0.9 (r ≥ 0.9 being indicative of a very strong correlation). Second, if possible, the model should not contain numeracy items to eliminate the involvement of an interviewer.

3. Results

The basic demographic characteristics of the 158 participants can be found in Table 2. The mean age was 69.6 years (SD: 9.53). The basic characteristics were relatively balanced, except for educational level; 20% of the participants completed high school or higher education, and 80% completed elementary school or skilled work.

The exhaustive search showed that the number of items in the D-TOFHLA could be reduced to 20 reading comprehension items and that there was no need for numeracy items. The sets of reading comprehension items were item 2-3, item 13-14, item 18-21, item 23-25, item 37-41, and item 42-45, each set corresponding to a sentence in the Danish TOFHLA leading to the following regression model:

Two examples from the six sentences in the DS-TOFHLA are listed in Appendix A. The maximum time for administration could be reduced from 22 minutes (10 minutes for numeracy items and 12 minutes for comprehension items) to 5 minutes (12*20/50 minutes).

Figure 1 illustrates the development of the best possible model performance as a function of the number of reading comprehension items in the model. The figure shows that the best model with only one item had a correlation coefficient of 0.6, and the best model with 20 items had a correlation coefficient of 0.9. The scatter plot in Figure 2 illustrates the relation between the DS-TOFHLA and the D-TOFHLA for the best model with 20 items. The correlation coefficient was 0.90 (SD 0.01), and the correlation was highly significant (p<0.001).

• Download figure
• Open in new tab

Figure 1. The development of the best possible model performance as a function of the number of reading comprehension items in the model.

• Download figure
• Open in new tab

Figure 2. The relation between the DS-TOFHLA and the Danish TOFHLA for the best model with 20 items

The internal consistency measured by Cronbach’s alpha was 0.885. This indicated that the reliability of the DS-TOFHLA was acceptable (>0.7 as set by Houser (25)). Item to scale correlations were assessed for all 20 items using Pearson’s point-biserial correlation coefficient; 12 items showed a high correlation and 8 items showed a medium correlation. The analysis of the Pearson’s point-biserial correlation coefficient showed significant positive correlations between all 20 items and the scale (p < 0.01).

Table 3 shows a confusion matrix illustrating the ability of the model to correctly predict each participant’s HL level. In the confusion matrix, for 126 out of 158 participants, the prediction was correct; for 32 out of the 158 participants, the prediction was off by one level; and no prediction was off by more than one level.

The accuracy of the prediction of the inadequate level (lowest level) (i.e., inadequate vs. marginal or adequate) was 92%. The accuracy of the prediction of the marginal level (middle level) (i.e., marginal vs. adequate or inadequate) was 80%. The accuracy of the prediction of the adequate level (highest level) (i.e., adequate vs. marginal or inadequate) was 88%, which can also be expressed as the accuracy of the prediction of ‘low FHL’ as defined by Parker et al. (19).

To enable a comparison with the performance of the S-TOFHLA, the relation between the D-36-4-TOFHLA (the Danish mirror version of S-TOFHLA) score and the D-TOFHLA score, for the 158 COPD patients recruited from the TeleCare North cohort, is illustrated in Figure 3. Pearson’s correlation coefficient between the two scores was 0.90. Likewise, the relation between the D-36-0-TOFHLA (the Danish mirror version of the Prose S-TOFHLA) score and the D-TOFHLA score, for the 158 COPD patients recruited from the TeleCare North cohort, is illustrated in Figure 3B. Pearson’s correlation coefficient between the two scores was 0.85. Table 4 gives an overview of the various model versions of TOFHLA.

• Download figure
• Open in new tab

Figure 3. The relation between the Danish mirror version of the S-TOFHLA (D-36-4-TOFHLA) and the Danish TOFHLA.

• Download figure
• Open in new tab

Figure 4. The relation between the Danish mirror version of the Prose S-TOFHLA (D-36-0-TOFHLA) and the Danish TOFHLA.

4. Discussion

The aim of this study was to investigate the use of machine learning to develop a short test of FHL in adults (the DS-TOFHLA) that can be used in the development of short versions of the TOFHLA in various languages, including the original version of the American TOFHLA in English. In addition to investigating the machine learning approach, this study also addressed the problem that, to the authors’ knowledge, there are no efficient, suitable, and objective screening instruments for assessing HL in a clinical setting in Denmark and other non-English speaking countries.

A review has shown that most studies using the S-TOFHLA chose to omit the numeracy items to simplify the test and make it usable in a clinical setting (9). In our study, statistical analyses showed that inclusion of numeracy items was not necessary to meet the chosen quality goal of the study. By including only 20 reading comprehension items, it was possible to create a short version of the D-TOFHLA where the use does not require a trained interviewer. Therefore, in contrast to only using this instrument for research purposes, the DS-TOFHLA is also applicable as a screening instrument for the clinical setting. In addition, the maximum time for administration was reduced from 22 minutes to 5 minutes.

For comparison, an assessment of the performance of the S-TOFHLA was performed using Danish mirror versions of the short American versions. Both the S-TOFHLA and the Prose S-TOFHLA (without numeracy items) was assessed, the latter being the most widely used (9). Pearson’s correlation coefficient, when comparing scores from both the Danish mirror version of S-TOFHLA and the DS-TOFHLA with scores from D-TOFHLA, was 0.90. This indicates that DS-TOFHLA, even though it, as opposed to S-TOFHLA, does not include numeracy items and therefore is easily applicable in a clinical setting, has the same level of performance as S-TOFHLA. Furthermore, the maximum time for administration of DS TOFHLA is only 5 minutes compared to 12 minutes for S-TOFHLA. Pearson’s correlation coefficient when comparing scores from the Danish mirror version of the Prose S-TOFHLA with scores from D-TOFHLA was 0.85. This indicates that the Prose S-TOFHLA, which has omitted numeracy items and therefore as opposed to the S-TOFHLA is more applicable in a clinical setting, is inferior in performance to DS-TOFHLA. Furthermore, it should be noted that the maximum time for administration of the Prose S-TOFHLA is significantly longer than for DS-TOFHLA (9 minutes and 5 minutes respectively).

The DS-TOFHLA was developed solely using an algorithm-based selection of variables and LRMs. A major strength of this method was that the design principles were founded on objective algorithm-based decisions, and the LRM used for development selected the items from the D-TOFHLA that led to the most accurate predictions of the level of FHL. The reading comprehension items in both the D-TOFHLA and the American TOFHLA are ordered by increasing difficulty in readability level and thus, it is reasonable to assume that the more difficult items are most accurate in predicting the functional level of HL. In this regard, it should be noted that only 4 of 20 items selected for the DS-TOFHLA by the algorithm herein were from the first and easiest part of the reading comprehension items. The 36 items assessing reading comprehension in the S-TOFHLA is primarily from the first and middle part (lowest difficulty) and none from the latter and most difficult part. In line with this, the regression model presents quite different weights to the selected items for the DS-TOFHLA, e.g., C42 = 0 and C43 = 6. In comparison, the development of the American S-TOFHLA seems based on a subjective decision to include the first 36 items, without explicitly considering if these items contribute to the most accurate prediction of the FHL or considering the ascending difficulty in readability.

The accuracy of the prediction of the three levels was ranged from 80%-92%; the middle level had the lowest accuracy, which can be explained by the fact that this level is defined by a relatively narrow range of 60-74 points. The prediction of the lowest and highest levels was a one-sided classification, whereas the prediction of the middle layer was two-sided. It should be noted that the prediction of ‘low FHL’ (i.e., inadequate or marginal vs. adequate scores), as defined by Parker et al (19), had an accuracy of 88%.

The DS-TOFHLA was based on a predictive LRM and should not be regarded as a de novo questionnaire, and it, therefore, should not go through the same rigorous evaluation. Instead, the focus should be on developing a short questionnaire with the best possible predictions of the validated full-length questionnaire. Likewise, it makes sense to develop the prediction model in the DS-TOFHLA based on the same data that was used to develop and validate the full-length Danish TOFHLA (16,18). However, further work might be carried out to test the model on other datasets and other types of patients. Alternatives to LRMs might be considered. However, even though other classification methods such as neural networks or various clustering methods might have yielded higher correlation coefficients with 20 reading comprehension items, the results from using such models would be harder to explain both to experts in the field and to clinicians using the HL score. The study used an exhaustive search strategy. Many studies have applied forward or backward selection strategies. Even though such strategies reduce the computational burden significantly, they also lead to the risk of not finding the best model and were therefore not used in the study.

5. Conclusion

This study demonstrated how a generic model-based approach could be applied in the development of a short version of the TOFHLA, thereby reducing the 67 items in the full-length version to 20 items. Furthermore, this study showed that the inclusion of numeracy items was not necessary to meet the chosen quality goal of a Pearson’s correlation coefficient ≥0.9, resulting in a short version of TOFHLA where the use does not require a trained interviewer. The work was based on Danish data and a validated Danish full-length version of TOFHLA. The generic model-based approach used herein may also be used in the development of short versions of the TOFHLA in other languages and in the development of short versions of any scaling questionnaire.

Conflict of interest

The authors declare there are no conflicts of interest.

Funding

This research did not receive any specific grants.