Effects of an online plain language tool on health information quality: A randomised controlled trial
=====================================================================================================

* Julie Ayre
* Carissa Bonner
* Danielle M Muscat
* Erin Cvejic
* Olivia Mac
* Dana Mouwad
* Heather L Shepherd
* Parisa Aslani
* Adam G Dunn
* Kirsten J McCaffery

## Abstract

**Objective** Complex and ineffective health communication is a critical and persistent source of inequity in our health systems. This occurs despite repeated policy directives to provide patients and community with health information that is easy to understand and that applies health literacy principles. This study sought to evaluate the effectiveness of the Sydney Health Literacy Lab (SHeLL) Health Literacy Editor, an easy-to-use online plain language tool that supports health information providers to apply health literacy guidelines to written health information.

**Design** Randomised controlled trial with analysts blind to intervention group.

**Setting** Online study, Australia

**Participants** 188 health information providers with no previous experience using the Health Literacy Editor (mean age 41.0 (SD=11.6); 154 female (85%)).

**Intervention** Participants were provided access to the Health Literacy Editor and a 30-minute online training program prior to editing three pre-specified health texts. The Health Literacy Editor gives objective, real-time, and fine-grained feedback on words and sentences. Control participants were asked to revise the texts using their own standard health information development processes.

**Main outcome measure** Pre-registered primary outcome was text grade reading score (using validated instrument, the Simple Measure of Gobbledygook). Secondary outcomes were use of complex language (% of the text) and passive voice (number of instances), subjective expert ratings (Patient Education Materials Assessment Tool), and acceptability ratings (System Usability Scale; Technology Acceptance Model).

**Results** Texts revised in the intervention group had significantly improved grade reading scores relative to control (Mean Difference (MD)=2.48, 95% CI=1.84 to 3.12, p<0.001, d=0.99), lower text complexity (MD=6.86, 95% CI=4.99 to 8.74, p<0.001, d=0.95) and less passive voice (MD=0.95, 95% CI=0.4 to 1.5, p<0.001, d=0.53) in intention-to-treat analyses. Experts rated texts in the intervention group more favourably for word choice and style than those in the control group (MD=0.44, 95% CI=0.25 to 0.63, p<0.001, d=0.63), with no loss of meaning or content. Participants rated the Health Literacy Editor an acceptable product (71.0/100, SD=13.7) that was useful (3.8/5, SD=0.7) and easy to use (4.0/5, SD=0.6).

**Conclusions and relevance** The Health Literacy Editor helped users simplify health information and apply health literacy guidelines to written text. It has high potential to improve development of health information for people who have low health literacy. As an online tool the Health Literacy Editor is also easy to access and implement at scale.

**Trial registration** ACTRN12623000386639

**Section 1: What is already known on this topic?**

*   Most health information is hard for people to understand, particularly those who are older, with less education, or who speak English as a second language.

*   Systematic reviews show that texts that follow health literacy guidelines (e.g. use simpler words, shorter sentences and active voice) are easier for people to understand and recall.

*   There are few automated tools that guide development of easy-to-understand written health information and none that have been rigorously evaluated in a randomised controlled trial.

**Section 2: What this study adds**

*   Participants who used the Health Literacy Editor were able to more effectively simplify health information compared to participants in the control group.

*   On average participants in the intervention group produced texts suitable for a person with almost 2.5 fewer years of school education compared to those in the control group. Similar patterns were observed for complex language and passive voice.

*   The Health Literacy Editor is an effective tool to support development of written health information that adheres to plain language principles. It can be used in clinical and non-clinical settings and implemented at scale.

National and international policies recognise that health literacy, a person’s capacity to access, understand, and act on health information, is a critical source of inequity in our health systems (1–3). Low health literacy contributes to higher mortality, morbidity, rates of hospitalisation, emergency department visits, and medication errors, independently of other social determinants such as age, education, and socioeconomic disadvantage (4). A key feature of these policies is the directive to provide health information that all people can easily understand, including people with low health literacy (5). There has been a failure to systematically integrate such directives into routine public health and clinical practice, despite some of these policies existing for over a decade (6–11).

To support the provision of easy-to-understand health information, there are several freely available and comprehensive guidelines that provide advice about how to apply health literacy and plain language principles to health information (12–16). These guidelines recommend evidence-based strategies to improve knowledge and recall of health information, such as putting essential information first, using simple language, and minimising medical jargon (17, 18). However, accompanying systems, training, and tools are needed to drive meaningful change in health literacy practices within an organisation (19, 20).

Online tools are well placed to help improve the application of health literacy guidelines because of their capacity to provide specific, immediate and actionable feedback on written text (21–23). These tools typically identify difficult words, sentences, and grammatical structures, and sometimes integrate technologies such as natural language processing and artificial intelligence (24–30). However, few have been specifically designed for health contexts (28–30), only two have been formally evaluated, and both evaluations were limited by small sample size and pre-post study design (29, 30). Though these tools hold promise, it is unclear how effectively health information providers incorporate the tool’s feedback when revising and simplifying health information.

To address this gap in the research, the current study aimed to evaluate whether using the ‘Sydney Health Literacy Lab (SHeLL) Health Literacy Editor’ (referred to herein as ‘Health Literacy Editor’) can support health information providers to effectively apply health literacy guidelines to written health information. The Health Literacy Editor is a new online tool that provides objective and immediate assessment of written health information across a range of factors, including feedback on school grade reading levels, complex language, passive voice, text structure, lexical density and diversity, and person-centred language (31). The Health Literacy Editor guides users in real-time, providing simpler or more familiar alternatives for medical and other words, and demonstrates to the user how small changes can incrementally increase use of plain language in written health information.

## Methods

### Study design

The study used a two-arm parallel-group randomised-controlled trial study design with participants randomly assigned to intervention or control groups (Figure 1). This trial was prospectively registered with the Australian New Zealand Clinical Trials Registry (ACTRN12623000386639) and approved by the University of Sydney Human Research Ethics Committee (2023/276).

![Figure 1.](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2024/04/06/2024.04.04.24305365/F1.medium.gif)

[Figure 1.](http://medrxiv.org/content/early/2024/04/06/2024.04.04.24305365/F1)

Figure 1. Participant flow and study design.

### Participants

Eligible participants were adults in Australia whose work involved developing health information. Students in medicine, allied health and health sciences were also eligible to take part. Participants also had to positively affirm their commitment to the trial (“Do you commit to providing thoughtful answers in this study?”). Participants were not eligible to take part if they had previous experience using the Health Literacy Editor. Participants were recruited online through health networks, newsletters, and social media.

### Intervention groups

#### Health Literacy Editor group

The Health Literacy Editor is a browser-based software application that gives objective real-time feedback on the complexity of health information. The Health Literacy Editor comprises six assessments: readability as measured by the Simple Measure of Gobbledygook (SMOG) formula, complex language, passive voice, text structure, lexical density and diversity, and person-centred language. These are each presented as global scores, with additional, more specific feedback flagged in the text itself through highlighting individual words and sentences. Full details of the development are published elsewhere (31). User testing with health staff has helped improve the quality of training, instructions, and feedback that the tool provides (32). Participants randomised to this group attended a 30-minute online training session in which they learned how to use the tool. Three training resources were also embedded within the tool: a help page that contained instructions, video tutorials and worked examples; a self-check tool comprising 5 questions to check understanding of key concepts; and a 2-page PDF introducing key concepts. After completing training, participants were instructed to use the Health Literacy Editor to help them revise the three health texts.

#### Control group

Participants were asked to use their usual processes to revise three health texts. No further training was provided.

### Procedure

Participants provided consent online, along with demographics, details about professional or student role, and their experience developing health information. They were then randomised 1:1 to intervention group or control using the Qualtrics survey platform’s Mersenne Twister algorithm. Participants were emailed a link to a second survey which asked them to revise three health texts, each approximately 200 words and written at a Grade 14 reading level, on the topics of dementia, cancer and sciatica (Appendix A). Three topics relating to different commonly occurring health conditions were selected to mitigate bias attributable to topic area expertise. Selection of the texts ensured that there were enough long words and sentences, complex language, and passive voice that participants could demonstrate their ability to simplify the text according to health literacy guidelines.

Instructions asked all participants to revise each text to make it easy for most people to understand, to aim for Grade 8 to 10 reading level, and to retain any key messages within the text. The reading level range was selected to reduce participant burden and ensure the revision task was feasible. To aid revision, a brief description of the purpose for each text was also provided. After revising the texts, participants completed items about self-reported estimates of time taken to revise the text. Participants in the intervention group reported on the features they used, and the tool’s usability and acceptability. Participants received a $50 gift card to thank them for their time.

## Outcomes

### Primary outcome

The primary outcome was grade reading score as measured by SMOG (further detail provided in Box 1)(33).

#### Box 1. Assessment tools incorporated into evaluation outcomes

View this table:
[Table1](http://medrxiv.org/content/early/2024/04/06/2024.04.04.24305365/T1)

### Secondary outcomes

#### Objective text complexity

Using simple everyday language is a key health literacy recommendation to improve the quality of health information (12). This was assessed using the Health Literacy Editor’s ‘complex language score’ (Box 1). The complex language score reports the proportion of words (%) that are uncommon in English, acronyms, or words with a suggested alternative in the tool’s thesaurus (36).

#### Objective passive voice

Using active voice is a key recommendation to improve how easy health information is to understand (12). Passive voice was assessed using the Health Literacy Editor (Box 1).

#### Subjective expert ratings

Two co-investigators with expertise in health literacy (JA and OM) assessed each revised text, masked to intervention group. Scores represent average ratings on a 5-point Likert scale (Strongly disagree [1] to strongly agree [5]). Three ratings for each text were produced. The first two, ‘Content’ and ‘Word choice and style,’ correspond to PEMAT topics (12) (Box 1). The content topic relates to a clear purpose and absence of distracting information. Word choice and style refers to use of common everyday language, minimal and defined medical terms, and use of the active voice where possible. A third rating ‘retained meaning’ reflects whether key messages were retained in the revised texts. This was added to ensure texts were not simplified by removing content.

#### Time to complete text revisions

Participants in both groups were asked to estimate the number of minutes to complete all three revision tasks.

#### Intervention acceptability

Participants in the intervention group provided acceptability ratings via two validated instruments. The System Usability Scale (38, 39) produces a score from 0 (low) to 100 (high). A score of 70 is considered acceptable and a score of 90 or more is considered superior (39). The Technology Acceptance Model (40) comprises two subscales: perceived usefulness and perceived ease of use, with scores ranging from 1 (low) to 5 (high). Scores are predictive of current and future use of a product (40).

#### Intervention engagement

Participants were asked which Health Literacy Editor features they engaged with when revising the text. Participants who reported using at least two of three key features described in training (Readability, Complex language and Passive voice) were assessed as having adequate engagement.

### Analysis

#### Sample size

A sample size estimate of 120 (60 participants per group) was calculated to have 90% power at α = 0.05 to detect a moderate effect size (Cohen’s f = 0.30) in the main outcome (grade reading score). An additional buffer allowed for up to 33% drop-out before the text revision task was completed for a total 180 participants. Sample size was adjusted during recruitment to account for a larger than expected non-completion rate (N=211).

#### Statistical analysis

Univariable regression models analysed differences for resources developed using the Health Literacy Editor (averaged across the three texts) and those developed by participants in the control condition. Intention-to-treat (ITT) and per-protocol (PP) analyses are presented. For ITT, the scores of the original texts were retained as the scores for revised texts for participants who did not complete the revision task. That is, we assumed no changes were made to the text. PP analysis included only participants who submitted revised versions of the texts (for participants in the intervention group this included attendance at training). One participant in the control group who used the Health Literacy Editor was excluded from the PP analysis. Descriptive statistics summarised information about usability, acceptability, and engagement (intervention group only). For all subjective ratings, assessors were unaware of intervention group.

Analyses were performed using IBM SPSS Statistics 26. The threshold for significance of the primary outcome was p<0.05 and all hypothesis tests were 2-sided. The same significance threshold was used for analyses of secondary outcomes and can be interpreted as exploratory.

### Patient and public involvement

A community member was involved in discussions about study design. Methods and outcome measures related to community ratings of the revised texts incorporated their feedback. This subsequent component of the project is not reported in this manuscript and will be reported separately. Several health services staff and university research staff helped pilot and improve the training materials and ensured that the text revision task was feasible without placing undue burden on participants.

## Results

### Sample characteristics

Participants were recruited between May and November 2023, with follow-up completed by February 2024. The number and flow of participants at each stage is shown in Figure 1. Sample characteristics are summarised in Table 1. Participants were on average 41.0 years (SD=11.6), 85.1% identified as female (n=154), 13.3% (n=24) male, 1.7% (n=3) as non-binary or other gender, and most reported working in health services (62.4%, n=113) and government organisations (64.6%, n=117). Characteristics appeared comparable for the PP sample (Table A1).

View this table:
[Table 1.](http://medrxiv.org/content/early/2024/04/06/2024.04.04.24305365/T2)

Table 1. Participant characteristics by intervention group, ITT analysis sample (N=181) (n (%) unless otherwise stated)

Most participants completed follow-up (81.2%, n=147), though this rate was lower for participants in the intervention group compared to control (73.3%, n=63; 89.5%, n=84, respectively).

### Evaluation of revised texts

#### Primary outcome

Compared to text revised by participants in the control group, the texts revised by those in the intervention group had significantly improved grade reading level (Mean Difference (MD)=2.48, 95% CI=1.8413.12, p<0.001, d=0.99; Table 2 and Table A2). Magnitude of effects were larger in PP analysis, with participants in the intervention group reducing the grade reading level by almost 4 grades relative to those in the control group (MD=3.79, 95% CI = 3.2914.28, p<0.001, d=1.58).

View this table:
[Table 2.](http://medrxiv.org/content/early/2024/04/06/2024.04.04.24305365/T3)

Table 2. Participant scores for revised materials, by intervention group (N=181) (M (SD))

#### Secondary outcomes

The same pattern was observed for secondary outcomes, with texts revised by those in the intervention group showing lower text complexity (MD=6.86, 95% CI=4.9918.74, p<0.001, d=0.95) and less passive voice (MD=0.95, 95% CI=0.4411.47, p<0.001, d=0.53) (Table 2, Table A2). Expert ratings for word choice and style (common everyday language, minimal and defined medical terms, and active voice) were higher in the intervention compared to control group (MD=0.44, 95% CI = 0.2510.63, p<0.001, d=0.63, Table 2). Magnitude of effects for secondary outcomes were larger in PP analysis, including across both objective and subjective expert ratings. Ratings for content (clear purpose and absence of distracting content) and retaining meaning were high and did not differ significantly across the two groups.

#### Acceptability and engagement

On average, participants rated the Health Literacy Editor an acceptable product that was useful and easy to use (Table 3). Participants using the Health Literacy Editor reported spending an average 65.40 minutes revising the three texts (SD=33.02), compared to an estimated 30.13 minutes for the control group (SD=18.28). Almost all participants (n=59, 93.7%) reported using all three of the Health Literacy Editor’s key assessments: readability, complex language and passive voice.

View this table:
[Table 3.](http://medrxiv.org/content/early/2024/04/06/2024.04.04.24305365/T4)

Table 3. Acceptability and engagement with the Health Literacy Editor, n=63

## Discussion

Health information that was revised using the Health Literacy Editor more closely aligned with health literacy and plain language guidelines compared to texts revised according to participants’ standard processes. These texts had a lower grade reading score and used less complex language and passive voice, showing greater potential to meet the health literacy needs of the population, including people who are older, who have had less opportunity for education, and who speak English as a second language. Subjective ratings from health literacy experts provided further evidence that these revised texts were clear and retained the original meaning. Though participants who used the Health Literacy Editor took longer to revise the texts, we believe this time investment is reasonable given the magnitude of effects, the likelihood that participants may become faster with repeated use of the tool, and its strong capability to support scalable and easily accessible health literacy training.

These findings highlight that innovative new tools can meaningfully contribute to bridging the well-documented gap between health literacy policy and practice (1, 2, 6–9). To date, several promising tools have been developed (24–30), with very limited evaluation of their effectiveness (29, 30). This is the first study to use a randomised-controlled trial study design, to show that health literacy software providing objective, real-time and fine-grained feedback on words and sentences is effective in supporting health information providers to develop plain language written materials. Coupled with sound user acceptability ratings, further work is now needed to explore how tools such as Health Literacy Editor can be implemented at scale within an organisation and to evaluate its impact on patient outcomes.

Online tools are well placed to support consistent and scaled uptake of health literacy guidelines by workforce who may have received little training in the area. This is important given that developing health information can be an intermittent activity for clinicians and health staff, particularly when roles are transient or project based. Online tools and training have the advantage of being easily accessed online without geographic or time constraints.

Developing health texts will always need human oversight and expertise. We envisage that health information providers would use the Health Literacy Editor in combination with other tools and strategies whilst maintaining existing quality and safety processes for clinical oversight. These caveats are likely to continue to apply, even with advances in artificial intelligence that may allow these tools to quickly and coherently simplify health information (41, 42).

In addition to the randomised-controlled trial design, several other aspects further strengthen the study findings. For example, though readability is an appropriate primary outcome measure, it has been criticised as a narrow indicator of plain language (43). This study addressed this issue by including a variety of objective and subjective assessments of plain language (complex language and passive voice) and was therefore able to show consistent patterns across a wide range of outcomes. Study findings were also strengthened by asking participants to submit three revised texts on different health topics. This reduced the likelihood that content-area expertise would influence results.

There are also some limitations to this study. Several participants completed intervention training but did not submit revised texts. It is possible that some were overwhelmed by the training or did not see value in the tool if they were already confident in their skills. Qualitative and co-design research may further improve the training and help set appropriate expectations for using the tool. It is also unclear whether results generalise to health information developers in non-government sectors, given the low number of participants from industry, consumer advocacy groups, and tertiary institutions. Secondly, it is unclear whether improved uptake of plain language will translate to improved perceptions of the health information by consumers and patients. Further work is underway to explore whether consumers prefer and can more easily understand texts developed using the Health Literacy Editor. This work may also help understand the relative importance of each objective assessment.

The findings from this study demonstrate that the Health Literacy Editor can support users to apply health literacy and plain language strategies to written text, while retaining key content and meaning. New technologies may make an important practical contribution to achieving the goals set out by health literacy policy for clear health communication, improved health equity, and better health outcomes. These tools have potential to improve health outcomes for people with lower health literacy.

## Supporting information

Appendix A [[supplements/305365_file02.pdf]](pending:yes)

Appendix B [[supplements/305365_file03.docx]](pending:yes)

## Data Availability

Deidentified data will be made available on reasonable request.

## Transparency statement

Julie Ayre affirms the manuscript is an honest, accurate and transparent account of the study being reported. No important aspects of the study have been omitted. Any discrepancies from the study as originally planned have been explained.

## Competing interests

The SHeLL Health Literacy Editor is a tool owned by the University of Sydney. It has been licensed to Health Literacy Solutions PTY Ltd to allow for commercialisation and enable wider public use. DMM, KJM, CB and JA are co-directors of Health Literacy Solutions PTY Ltd. DMM, KJM, CB and JA take no personal income from Health Literacy Solutions PTY Ltd or the SHeLL Health Literacy Editor. The University of Sydney retains IP and is a shareholder.

## Funding source

JA is supported by a National Health and Medical Research Council fellowship (APP 2017278). The funder was not involved in any aspect of the study. Researchers are independent from funders and all authors had full access to all of the data (including statistical reports and tables) in the study and can take responsibility for the integrity of the data and the accuracy of the data analysis.

## Data sharing statement

Deidentified data will be made available on reasonable request.

## Acknowledgements

We would like to acknowledge the support and contributions of Nancy Zaki, Allison Sigmund, Susan Dyer, Dianna Smith-McCue, and Debra Letica in this study. Thank you also to Olivia Mac, Melody Taba, Katie McFadden, and Jenna Smith for their support in piloting the text revision tasks.

*   Received April 4, 2024.
*   Revision received April 4, 2024.
*   Accepted April 6, 2024.


*   © 2024, Posted by Cold Spring Harbor Laboratory

The copyright holder for this pre-print is the author. All rights reserved. The material may not be redistributed, re-used or adapted without the author's permission.

## References

1.  1.World Health Organization. Health literacy development for the prevention and control of noncommunicable diseases: volume 3: recommended actions. 2022.
    
    

2.  2.U.S. Department of Health and Human Services (Office of Disease Prevention and Health Promotion). Healthy People 2030 2020 [Available from: [https://health.gov/healthypeople](https://health.gov/healthypeople).
    
    

3.  3.Department of Health. National Preventive Health Strategy 2021-2030. Canberra, Australia 2021.
    
    

4.  4.Berkman ND, Sheridan SL, Donahue KE, Halpern DJ, Crotty K. Low health literacy and health outcomes: an updated systematic review. Ann Intern Med. 2011;155(2):97–107. doi:10.7326/0003-4819-155-2-201107190-00005.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.7326/0003-4819-155-2-201107190-00005&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=21768583&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2024%2F04%2F06%2F2024.04.04.24305365.atom) 
    
    [Web of Science](http://medrxiv.org/lookup/external-ref?access_num=000292822000016&link_type=ISI) 

5.  5.Trezona A, Rowlands G, Nutbeam D. Progress in Implementing National Policies and Strategies for Health Literacy-What Have We Learned so Far? Int J Environ Res Public Health. 2018;15(7). doi:10.3390/ijerph15071554.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.3390/ijerph15071554&link_type=DOI) 

6.  6.Cheng C, Dunn M. Health literacy and the Internet: a study on the readability of Australian online health information. Aust N Z J Public Health. 2015;39(4):309–14. doi:10.1111/1753-6405.12341.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1111/1753-6405.12341&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=25716142&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2024%2F04%2F06%2F2024.04.04.24305365.atom) 

7.  7.Daraz L, Morrow AS, Ponce OJ, Beuschel B, Farah MH, Katabi A, et al. Can Patients Trust Online Health Information? A Meta-narrative Systematic Review Addressing the Quality of Health Information on the Internet. Journal of General Internal Medicine. 2019;34(9):1884–91. doi:10.1007/s11606-019-05109-0.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1007/s11606-019-05109-0&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2024%2F04%2F06%2F2024.04.04.24305365.atom) 

8.  8.Boutemen L, Miller AN. Readability of publicly available mental health information: A systematic review. Patient Education and Counseling. 2023;111:107682. doi:10.1016/j.pec.2023.107682.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1016/j.pec.2023.107682&link_type=DOI) 

9.  9.Rooney MK, Santiago G, Perni S, Horowitz DP, McCall AR, Einstein AJ, et al. Readability of Patient Education Materials From High-Impact Medical Journals: A 20-Year Analysis. Journal of Patient Experience. 2021;8:2374373521998847. doi:10.1177/2374373521998847.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1177/2374373521998847&link_type=DOI) 

10. 10.Mishra V, Dexter JP. Comparison of Readability of Official Public Health Information About COVID-19 on Websites of International Agencies and the Governments of 15 Countries. JAMA Netw Open. 2020;3(8):e2018033. doi:10.1001/jamanetworkopen.2020.18033.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1001/jamanetworkopen.2020.18033&link_type=DOI) 

11. 11.Mac OA, Muscat DM, Ayre J, Patel P, McCaffery KJ. The readability of official public health information on COVID-19. Med J Aust. 2021;215(8):373–5. doi:10.5694/mja2.51282.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.5694/mja2.51282&link_type=DOI) 

12. 12.Shoemaker SJ, Wolf MS, Brach C. Development of the Patient Education Materials Assessment Tool (PEMAT): a new measure of understandability and actionability for print and audiovisual patient information. Patient Educ Couns. 2014;96(3):395–403. doi:10.1016/j.pec.2014.05.027.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1016/j.pec.2014.05.027&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=24973195&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2024%2F04%2F06%2F2024.04.04.24305365.atom) 

13. 13.Brega A, Barnard J, Mabachi N, Weiss B, DeWalt D, Brach C, et al. AHRQ Health Literacy Universal Precautions Toolkit, 2nd Edition Rockville, MD: Agency for Healthcare Research and Quality; 2015 [updated February 2015. Available from: [http://www.ahrq.gov/professionals/quality-patient-safety/quality-resources/tools/literacy-toolkit/healthlittoolkit2.html](http://www.ahrq.gov/professionals/quality-patient-safety/quality-resources/tools/literacy-toolkit/healthlittoolkit2.html).
    
    

14. 14.Office of Disease Prevention and Health Promotion. Health literacy online: A guide to simplifying the user experience. 2015.
    
    

15. 15.Plain Language Action and Information Network. Federal plain language guidelines, March, 2011 2011 [updated 01 May 2011. Available from: [https://www.plainlanguage.gov/media/FederalPLGuidelines.pdf](https://www.plainlanguage.gov/media/FederalPLGuidelines.pdf).
    
    

16. 16.Baur C, Prue C. The CDC Clear Communication Index Is a New Evidence-Based Tool to Prepare and Review Health Information. Health Promotion Practice. 2014;15(5):629–37. doi:10.1177/1524839914538969.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1177/1524839914538969&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=24951489&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2024%2F04%2F06%2F2024.04.04.24305365.atom) 

17. 17.Sheridan SL, Halpern DJ, Viera AJ, Berkman ND, Donahue KE, Crotty K. Interventions for individuals with low health literacy: a systematic review. J Health Commun. 2011;16 Suppl 3(sup3):30–54. doi:10.1080/10810730.2011.604391.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1080/10810730.2011.604391&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=21951242&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2024%2F04%2F06%2F2024.04.04.24305365.atom) 
    
    [Web of Science](http://medrxiv.org/lookup/external-ref?access_num=000299952500007&link_type=ISI) 

18. 18.Berkman ND, Sheridan SL, Donahue KE, Halpern DJ, Viera A, Crotty K, et al. Health literacy interventions and outcomes: an updated systematic review. Evid Rep Technol Assess (Full Rep). 2011/03/01 ed2011. p. 1–941.
    
    

19. 19.Sørensen K, Levin-Zamir D, Duong TV, Okan O, Brasil VV, Nutbeam D. Building health literacy system capacity: a framework for health literate systems. Health promotion international. 2021;36(Supplement_1):i13–i23. doi:10.1093/heapro/daab153.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1093/heapro/daab153&link_type=DOI) 

20. 20.Farmanova E, Bonneville L, Bouchard L. Organizational Health Literacy: Review of Theories, Frameworks, Guides, and Implementation Issues. Inquiry. 2018;55:46958018757848. doi:10.1177/0046958018757848.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1177/0046958018757848&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2024%2F04%2F06%2F2024.04.04.24305365.atom) 

21. 21.Hysong SJ, Best RG, Pugh JA. Audit and feedback and clinical practice guideline adherence: making feedback actionable. Implement Sci. 2006;1(1):9. doi:10.1186/1748-5908-1-9.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1186/1748-5908-1-9&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=16722539&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2024%2F04%2F06%2F2024.04.04.24305365.atom) 

22. 22.Hattie J, Timperley H. The power of feedback. Review of Educational Research. 2007;77(1):81–112. doi:10.3102/003465430298487.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.3102/003465430298487&link_type=DOI) 
    
    [Web of Science](http://medrxiv.org/lookup/external-ref?access_num=000244731800003&link_type=ISI) 

23. 23.Locke EA, Latham GP. Building a practically useful theory of goal setting and task motivation. A 35-year odyssey. Am Psychol. 2002;57(9):705–17. doi:10.1037//0003-066x.57.9.705.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1037//0003-066x.57.9.705&link_type=DOI) 

24. 24..38 Long LLC. Hemingway Editor 2021 [Available from: [https://hemingwayapp.com/](https://hemingwayapp.com/).
    
    

25. 25.Grammarly Inc. Grammarly 2021 [Available from: [https://app.grammarly.com/](https://app.grammarly.com/).
    
    

26. 26.StyleWriter. StyleWriter Version 4 2022 [Available from: [https://www.stylewriter-usa.com/](https://www.stylewriter-usa.com/).
    
    

27. 27.VisibleThread. The Language Analysis Platform That Means Business 2022 [Available from: [https://www.visiblethread.com/](https://www.visiblethread.com/).
    
    

28. 28.Health Literacy Innovations TM. The Health Literacy Advisor 2023 [Available from: [https://www.healthliteracyinnovations.com/products/hla](https://www.healthliteracyinnovations.com/products/hla).
    
    

29. 29.Leroy G, Kauchak D, Haeger D, Spegman D. Evaluation of an online text simplification editor using manual and automated metrics for perceived and actual text difficulty. JAMIA Open. 2022;5(2):ooac044. doi:10.1093/jamiaopen/ooac044.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1093/jamiaopen/ooac044&link_type=DOI) 

30. 30.Hadden KB. Health literacy training for health professions students. Patient Education and Counseling. 2015;98(7):918–20. doi:10.1016/j.pec.2015.03.016.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1016/j.pec.2015.03.016&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=25850755&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2024%2F04%2F06%2F2024.04.04.24305365.atom) 

31. 31.Ayre J, Bonner C, Muscat DM, Dunn AG, Harrison E, Dalmazzo J, et al. Multiple Automated Health Literacy Assessments of Written Health Information: Development of the SHeLL (Sydney Health Literacy Lab) Health Literacy Editor v1. JMIR Form Res. 2023;7:e40645. doi:10.2196/40645.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.2196/40645&link_type=DOI) 

32. 32.Ayre J, Muscat DM, Mac O, Bonner C, Dunn AG, Dalmazzo J, et al. Helping patient educators meet health literacy needs: End-user testing and iterative development of an innovative health literacy editing tool. PEC Innov. 2023;2:100162. doi:10.1016/j.pecinn.2023.100162.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1016/j.pecinn.2023.100162&link_type=DOI) 

33. 33.McLaughlin GH. SMOG Grading-a New Readability Formula. Journal of Reading. 1969;12(8):639–46.
    
    [Web of Science](http://medrxiv.org/lookup/external-ref?access_num=A1969ZM49800004&link_type=ISI) 

34. 34.Clinical Excellence Commission. NSW Health Literacy Framework. 2019-2024 Sydney: Clinical Excellence Commission; 2019 [Available from: [https://www.cec.health.nsw.gov.au/\_\_data/assets/pdf\_file/0008/487169/NSW-Health-Literacy-Framework-2019-2024.pdf](https://www.cec.health.nsw.gov.au/__data/assets/pdf_file/0008/487169/NSW-Health-Literacy-Framework-2019-2024.pdf).
    
    

35. 35.Wang LW, Miller MJ, Schmitt MR, Wen FK. Assessing readability formula differences with written health information materials: application, results, and recommendations. Res Social Adm Pharm. 2013;9(5):503–16. doi:10.1016/j.sapharm.2012.05.009.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1016/j.sapharm.2012.05.009&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=22835706&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2024%2F04%2F06%2F2024.04.04.24305365.atom) 

36. 36.Mac O, Ayre J, Bell K, McCaffery K, Muscat DM. Comparison of Readability Scores for Written Health Information Across Formulas Using Automated vs Manual Measures. JAMA Network Open. 2022;5(12):e2246051-e. doi:10.1001/jamanetworkopen.2022.46051.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1001/jamanetworkopen.2022.46051&link_type=DOI) 

37. 37.Browne C. The new general service list: Celebrating 60 years of vocabulary learning. The Language Teacher. 2013;37(4):13–6.
    
    

38. 38.Brooke J. Sus: a “quick and dirty’usability. Usability evaluation in industry. 1996;189(3).
    
    

39. 39.Bangor A, Kortum PT, Miller JT. An empirical evaluation of the System Usability Scale. International Journal of Human-Computer Interaction. 2008;24(6):574–94. doi:10.1080/10447310802205776.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1080/10447310802205776&link_type=DOI) 
    
    [Web of Science](http://medrxiv.org/lookup/external-ref?access_num=000258821600003&link_type=ISI) 

40. 40.Davis FD. Perceived Usefulness, Perceived Ease of Use, and User Acceptance of Information Technology. Mis Quarterly. 1989;13(3):319–40. Doi10.2307/249008.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.2307/249008&link_type=DOI) 

41. 41.Ayre J, Mac O, McCaffery K, McKay BR, Liu M, Shi Y, et al. New Frontiers in Health Literacy: Using ChatGPT to Simplify Health Information for People in the Community. Journal of General Internal Medicine. 2023. doi:10.1007/s11606-023-08469-w.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1007/s11606-023-08469-w&link_type=DOI) 

42. 42.Dunn AG, Shih I, Ayre J, Spallek H. What generative AI means for trust in health communications. J Commun Healthc. 2023;16(4):385–8. doi:10.1080/17538068.2023.2277489.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1080/17538068.2023.2277489&link_type=DOI) 

43. 43.Meade CD, Smith CF. Readability formulas: Cautions and criteria. Patient Education and Counseling. 1991;17(2):153–8. doi:10.1016/0738-3991(91)90017-y.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1016/0738-3991(91)90017-Y&link_type=DOI) 
    
    [Web of Science](http://medrxiv.org/lookup/external-ref?access_num=A1991FF64400005&link_type=ISI)