Assessing the quality, readability and reliability of online information on COVID-19: an infoveillance observational study ========================================================================================================================== * Paulo Cardoso Lins-Filho * Thuanny Silva de Macêdo * Andressa Kelly Alves Ferreira * Maria Cecília Freire de Melo * Millena Mirella Silva de Araújo * Jaciel Leandro de Melo Freitas * Thaise Urbano Caldas * Arnaldo de França Caldas, Jr ## Abstract **Objective** This study aimed to assess the quality, reliability and readability of internet-based information on COVID-19 available on Brazil’ most used search engines. **Methods** A total of 68 websites were selected through Google, Bing, and Yahoo. The websites content quality and reliability were evaluated using the DISCERN questionnaire, the Journal of American Medical Association (JAMA) benchmark criteria, and the presence of the Health on Net (HON) certification. Readability was assessed by the Flesch Reading Ease adapted to Brazilian Portuguese (FRE-BP). **Results** The web contents were considered moderate to low quality according to DISCERN and JAMA mean scores. Most of the sample presented very difficult reading levels and only 7.4% displayed HON certification. Websites of Governmental and health-related authorship nature showed lower JAMA mean scores and quality and readability measures did not correlate to the webpages content type. **Conclusion** COVID-19 related contents available online were considered of low to moderate quality and not accessible. Keywords * COVID-19 * Internet * Consumer Health Information * Health Education * Information Dissemination ## Introduction Health care is rapidly transitioning from a paternalistic approach to a person-centered model. This process aims to improve health outcomes by building a shared decision-making process between healthcare professionals and patients, characterized by the greater involvement of people in resolutions and actions concerning their own health (Lee et al. 2018; Petersen et al. 2019). However, the effectiveness of this new model can be hindered by a considerable number of barriers, such as low education, inadequate access to knowledge and social and economic deprivation (Lee et al. 2017). The internet offers a large amount of information, although the quality of health and sanitary information offered is highly variable, ranging from scientific and evidence-based data to home remedies or information of very questionable origin that can be dangerous to health (Eysenbach et al. 2002). The biggest barrier on the internet is not the difficulty of finding health care information, but identifying those that are valid and reliable (Berland et al. 2001; Lopez-Jornet and Camacho-Alonso 2009; Lopez-Jornet and Camacho-Alonso 2010; Passos et al. 2020). During public health emergencies, people must be aware about the health risks they face, and what measures can be taken to protect their health and lives. Reliable information provided early, often, and in accessible language standards, enables individuals to make choices and act to protect themselves, their families and communities from health hazards (WHO 2017). Fake news and misinformation concerning health on the internet represents a threat to global health (Carrieri et al. 2019). The World Health Organization (WHO) warned that the COVID-19 outbreak had been accompanied by a massive abundance of information, some of which was accurate and some of which was not, which made it difficult for people to find reliable sources and trustworthy information when they needed it (Kouzy et al. 2020; Pulido et al. 2020). The consequences of disinformation overload are the spread of uncertainty, fear, anxiety and racism. Thus, monitoring the quality of information available to the population is of great importance to control the spread of the disease itself and to mitigate its socioeconomic impacts (Hua and Shaw 2020). Therefore, the WHO is dedicating tremendous efforts aimed at providing evidence-based information and advice to the population through its social media channels and a new information platform called WHO Information Network for Epidemics (Zarocostas 2020). Online-available information has been increasingly employed as a surrogate tool for estimating epidemiology (Cervellin et al. 2017). Web-based sources are been used in the analysis, detection, and forecasting of diseases and epidemics, and in predicting human behavior toward several health topics. In this context, infoveillance studies have become an integral part of health sciences that focuses on scanning the Internet for user-contributed health-related content, aiming to improve public health, measuring and predicting the quality of health information on the Web (Eysenbach et al. 2009). Several studies have already assessed the quality of the information available on the internet related to different health conditions (Cuan-Baltazar et al. 2020; Jo et al. 2018; Lee et al. 2017; Passos et al. 2020; Priyanka et al. 2018), however, there is no evidence about the quality of COVID-19 contents available in Brazil. This study aimed to assess the reliability, readability and quality of COVID-19 related information retrieved from Brazilian websites. ## Materials and Methods The search strategy was designed with regard to the relevance of terms employed by the internet users. A query was performed on Google Trends to confirm the link of Brazilian Portuguese words to COVID-19 issues. Google Trends enables researchers to study the trends and patterns of Google search queries (Effenberger et al. 2020). The search term “coronavírus” held the most popularity among internet users in Brazil, in April 2020. Sites on the internet were identified using the three most accessed search engines by internet users in Brazil: Google [www.google.com](http://www.google.com)), Bing ([www.bing.com](http://www.bing.com)) and Yahoo [www.yahoo.com](http://www.yahoo.com)), respectively, 97.59%, 1.2% and 1.04% of accesses in April 2020 (Statcounter 2020). In April 2020, the searches were history of each browser. The first 100 consecutive sites in each search were visited and classified. The search was not restricted in terms of file format or domain. The search was limited to the Portuguese language. Duplicate sites were excluded, as were non-operative sites or sites with denied direct access through password requirements, book review sites, or sites offering journal abstracts, and those sites that did not offer information on COVID-19. Websites that could be modified by the general population were also not considered in this investigation. The quality of website information was assessed by four evaluators, who were previously trained in the analysis tools used. Concerning the scientific accuracy and reliability of websites information, WHO official reports and technical guidelines were used as standards. The websites that were divergently qualified by the examiners were reassessed to the achievement of a consensus score. As this was a study of published information and involved no participants, no ethics approval was required. In order to avoid any changes that may be made to the eligible websites during the period of analysis, the sites were assessed in the same day by the evaluators. The sites were classified in terms of affiliation as commercial, news portal, non-profit organization, university or health center and government. The type of content was classified as corresponding to medical facts, human experiences of interest, questions and answers and socioeconomic related content. The quality of information of the selected websites was assessed using criteria of the Journal of the American Medical Association (JAMA) benchmarks (Silberg et al. 1997). These are a display of authorship of medical content, display of attribution or references, display of currency (date of update), and disclosure of ownership, sponsorship, advertising policies or conflicts of interest. This tool lets the reader easily decide if the site has the basic components like transparency and reliability. For each fulfilled criterion, 1 point was given, with a total score ranging from 0 to 4. The DISCERN instrument (Discern) is a valid and reliable tool to evaluate health information. It is the first standardized quality index and was created by the Division of Public Health and Primary Health Care at Oxford University, London. The instrument comprises 16 questions, each representing a different quality criterion. The DISCERN questions are organized into three sections as follows: Questions 1–8 performed using computers connected to the internet, previously set up by clearing the cookies and search address the reliability of the publication and help users to decide whether it can be trusted as a source of information relating to treatment choice. Questions 9–15 address specific details of the information relating to treatment alternatives. In this context, questions 9–11 refer to the active treatments described in the publication (possibly including self-care), while the options without treatment are addressed separately in question 12. In turn, question 16 corresponds to the global quality assessment at the end of the instrument. Each question is scored on a scale of 1–5 (where 1, the publication is poor; and 5, the publication is of good quality). In the present study, only the first section of the questionnaire was used for reliability assessment. The readability (RE) of the websites was assessed by the Flesch Reading Ease adapted to Brazilian Portuguese (FRE) (Lee et al. 2017). This method classifies the readability of a text on a scale from 0 (very difficult) to 100 (very easy) based on a calculation that considers the number of syllables per word and words per sentence. The adapted formula is given by the following equation: RE = 248.835 − (1.015 × ASL) − (84.6 × ASW). Where: ASL = mean number of words per sentence; ASW = mean number of syllables per word. Those metrics were calculated using the online tool Readable.io (Readable.io, Bolney, England) (Readable.io) through the information of the respective Uniform Resource Locator (URL) of each website. All analyses were performed based on the overall written content downloaded from these links. The reading difficulty of a text is presented according to the following scores: very easy (75–100), easy (50–75), difficult (25–50), and very difficult (0–25). The existence of the Health on the Net (HON) Foundation seal was also recorded. HON is a code of conduct for medical and healthcare sites, defining a series of norms allowing users to know the source and the purpose of the medical information presented. The HON contemplates compliance with the following eight basic criteria: 1. authorship; 2. complementarity; 3. privacy; 4. attribution, references and currency; 5. justifiability; 6. author transparency; 7. sponsor transparency (financial disclosure); and 8. honesty in advertising policy. The website may display the HON code seal if they agree to comply with the standards listed, and they are subjected to random audits for compliance(HON). Data were submitted to statistical analysis, all tests were applied considering an error of 5% and the confidence interval of 95%, and the analyzes were carried out using SPSS software version 23.0 (SPSS Inc. Chicago, IL, USA). Descriptive analysis was performed to characterize the Web pages selected for the study. Although the hypothesis of normal distribution of data was not confirmed by the Kolmogorov-Smirnov test, the statistical analysis was performed by the application of nonparametric tests. The correlations between distinct measures were demonstrated by the Spearman rank correlation coefficients. Distinct websites according to the type of content were compared by Kruskal-Wallis test. Mann-Whitney U test was employed to assess the differences between the natures of websites, for this test, the affiliation criteria used in data collection were dichotomized into governmental and health-related authorship nature (grouping university or health center and government affiliation) and nongovernmental nor health-related authorship nature (grouping news portals, commercial sites and non-profit organization affiliation)(Lee et al. 2017). ## Results Over 2.5 billion results were retrieved from the search engines. Of the 300 webpages assessed, 68 fulfilled the inclusion and exclusion criteria. The search retrieval flow diagram is presented in Figure 1. ![Fig. 1](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2020/05/30/2020.05.30.20117614/F1.medium.gif) [Fig. 1](http://medrxiv.org/content/early/2020/05/30/2020.05.30.20117614/F1) Fig. 1 Search retrieval flow diagram According to affiliation most websites were from news portals (51.5%), followed by government (29.4%), commercial sites (13.2%), university or health center (4.4%) and non-profit organization (1.5%). Considering the type of content, the majority of the sites displayed medical facts (88.2%) followed by socioeconomic related content (5.9%), questions and answers (4.4%) and human experiences of interest (1.5%). None of the evaluated websites met all four criteria of JAMA benchmarks, 35.3% had a single criterion, 30.9% did not include any criteria, 23.5% had 2 criteria and 10.3% presented 3 criteria. Only 7.4% of the sites had HON certification. The DISCERN instrument identified that 75% of the websites had moderate reliability, 17.6% showed high reliability and 7.4% low reliability. For the Flesch Index, over half of the sample were classified as very difficult (57.4%), while 41.2% were considered difficult. Only 1 website (1.5%) was classified as easy and none as very easy. Sample means of JAMA, DISCERN and FREBP scores are shown in table 1. View this table: [Table 1](http://medrxiv.org/content/early/2020/05/30/2020.05.30.20117614/T1) Table 1 Descriptive statistics of the scores of DISCERN, the Journal of American Medical Association benchmark, and Flesch Reading Ease The correlation between distinct measures assessed through the instruments was analyzed. Spearman rank correlation coefficients showed a positive significant correlation between JAMA and Discern scores (p<0.001), and between HON certification presence and JAMA (p = 0.006) and Discern < scores, as shown in table 2 View this table: [Table 2](http://medrxiv.org/content/early/2020/05/30/2020.05.30.20117614/T2) Table 2 Correlations between distinct quality measures No significant differences were observed among the mean scores of DICERN, JAMA and FRE-BP according with the type of websites content (table 3). As the human experience of interest type of content represents a single occurrence in the sample it was disregarded in this step of the statistical analyzes. View this table: [Table 3](http://medrxiv.org/content/early/2020/05/30/2020.05.30.20117614/T3) Table 3 Descriptive statistics of different websites type of content for DISCERN, the Journal of American Medical Association benchmark, and Flesch Reading Ease adapted to Brazilian Portuguese The mean JAMA scores were different according to the dichotomized affiliation categorization. Higher mean was observed in nongovernmental nor health-related authorship nature. For DISCERN and FRE-BP mean scores no significant statistical difference was observed, as shown in table 4. View this table: [Table 4](http://medrxiv.org/content/early/2020/05/30/2020.05.30.20117614/T4) Table 4 Descriptive statistics for both affiliation website groups for DISCERN, the Journal of American Medical Association benchmark, and Flesch Reading Ease adapted to Brazilian Portuguese ## Discussion The internet has great potential for spreading health information (Berland et al. 2001). Whatever the communication channel used, it is known that its content is able to influence the decisions of an individual about his health, including changes in lifestyle (Afshin et al. 2016). Web-based information influences how patients comply with advices, clinical diagnoses, and treatment regimens recommended by health professionals (Lu et al. 2018; Lu and Zhang 2019).Treatment adherence and compliance relies on trust and good professional-patient communication (Wahl et al. 2005). In this context, internet-based new technologies are gaining growing global attention and becoming increasingly available for predicting, preventing and monitoring emerging infectious diseases, such as COVID-19 (Effenberger et al. 2020; Yang et al. 2020). However, misinformation spread through the internet can hinder the communication of health entities and professionals with the general population (Lu and Zhang 2019) and so reduce adherence to the confrontations proposed to contain the pandemic, as social distancing (Farooq et al. 2020). Research on the role of internet content, social media messages and dominant discourses that are communicated to the public is an emerging topic of public health interest in scientific work that requires further investigation (Pulido et al. 2020). Despite the harm that misinformation may pose, especially during a pandemic in which the population’s reaction to health measures imposed by governments is of crucial importance to combat the spread of the disease (US Medicine Institute 2002), scientific evidence on the topic is scarce. Besides WHO efforts to monitor and improve the quality of information available online on this subject (Hua and Shaw 2020), the few available evidence warns of the low quality and reliability of data (Abd-Alrazaq et al. 2020; Febres-Cordero et al. 2018; Kouzy et al. 2020). These works are, however, limited to social media content analysis and lacking standard parameters to data evaluation. In the present study, for all parameters used in the data analysis, the quality of information ranged from low to moderate. The HON seal was displayed in only 7.5% of the sample. Other studies carried on Brazilian websites revealed even a small number of sites with this certification (Lee et al. 2017; Passos et al. 2020). The absence of this seal on sites of important institutions indicates that concern for certifying the quality of information in the internet is still scarce. Another possible cause for low seal adhesion is the fact that the annual review required for seal maintenance is not free (Passos et al. 2020). Besides the quality of information, the amount of it provided for an individual is also a concern. Previous research suggests that the vast amount of available information can be confusing, potentially resulting in over-concern and information overload (Farooq et al. 2020). Information overload is being associated to mental health problems during COVID-19 outbreak. These findings inspire the need for greater government involvement to prevent information overload while facing a public health emergency (Gao et al. 2020). However, government represents only 29.4% of the sources of information, while news portals represent 54.5% of the website’s affiliation. In addition, governmental and health-related sources of information showed lower JAMA mean scores when compared to nongovernmental nor health-related sources. This demonstrates the need for better articulation between health entities and government to stand out as the main provider of reliable content, directing users to good quality sources and avoiding information overload. According to the readability scores, the websites were considered difficult and very difficult for most of population. In addition to this finding it is relevant to consider the low level of health literacy, which is the degree to which people have the capacity to understand health information, reported for Brazilian Portuguese speakers (Batista et al. 2018). This may result in a communication gap for laypersons. In addition to ensure the quality and reliability of information, it is important that these quality contents are presented in a comprehensible and accessible manner (Miguens-Vila et al. 2018). Studies reported a negative correlation of readability scores with JAMA (Sobota and Ozakinci 2015) and DISCERN (Lee et al. 2017) scores. These findings can be considered as exacerbating factors of the impact of the low quality of information on internet users, as it demonstrates that more accessible content is of even worse quality. In the present study such correlation was not found possibly due the growing concern about the quality of health-related information available online (Farooq et al. 2020), however, no positive correlation was found, which demonstrates the need for further efforts on improving the accessibility of high-quality health related information available online. The internet and search engines are dynamic processes that constantly change. The sites evaluated in this investigation may not necessarily reflect the information available to patients at another time. This was a limitation for this investigation. However, the search engines used for the consultation represent 99.8% of the access of Brazilian internet users (Statcounter 2020). In addition, to cover a reasonable amount of data, the first 100 consecutive websites of each search engine were accessed. Regarding the present sample of Brazilian websites, COVID-19 contents were considered of low to moderate quality and low readability based on the parameters adopted. This pattern just reasonably correlated with the nature of websites’ authorship. These findings indicate the need for further efforts on improving the quality of health-related content on internet. Health authorities might apply this evidence to measure the effect of the transmission of information on the population and define better risk communication strategies. ## Data Availability The database is available. ## Compliance with Ethical Standards Conflict of interest: The authors declare that they have no conflict of interest. ## Conflict of interest The authors declare no conflict of interest related to the present study. ## Acknowledgments The authors would like to thank the Coordenação de Aperfeiçoamento de Pessoal de Nível Superior(CAPES) for scholarship granting. ## Footnotes * paulocardoso09{at}hotmail.com, thuannymacedo16{at}gmail.com, andressa_kelly{at}msn.com, mceciliafreire{at}hotmail.com, millenamirella{at}hotmail.com, jacielsanguiar{at}hotmail.com * Received May 30, 2020. * Revision received May 30, 2020. * Accepted May 30, 2020. * © 2020, Posted by Cold Spring Harbor Laboratory This pre-print is available under a Creative Commons License (Attribution-NonCommercial-NoDerivs 4.0 International), CC BY-NC-ND 4.0, as described at [http://creativecommons.org/licenses/by-nc-nd/4.0/](http://creativecommons.org/licenses/by-nc-nd/4.0/) ## References 1. 1.Abd-Alrazaq A, Alhuwail D, Househ M et al. (2020) Top Concerns of Tweeters During the COVID-19 Pandemic: Infoveillance Study. J Med Internet Res. 22:e19016. [https://doi:10.3390/diagnostics10040224](https://doi.org/10.3390/diagnostics10040224) 2. 2.Afshin A, Babalola D, McLean M et al. (2016) Information Technology and Lifestyle: A Systematic Evaluation of Internet and Mobile Interventions for Improving Diet, Physical Activity, Obesity, Tobacco, and Alcohol Use. J Am Heart Assoc. 5:e1–18. [http://doi:10.1161/JAHA.115.003058](http://doi.org/10.1161/JAHA.115.003058) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2020%2F05%2F30%2F2020.05.30.20117614.atom) 3. 3.Batista MJ, Lawrence HP, Sousa MLR (2018) Oral health literacy and oral health outcomes in an adult population in Brazil. BMC Public Health. 18:60. [https://doi:10.1186/s12889-017-4443-0](https://doi.org/10.1186/s12889-017-4443-0) 4. 4.Berland GK, Elliott MN, Morales LS et al. (2001) Health information on the Internet: accessibility, quality, and readability in English and Spanish. Jama 285:2612–2621 [https://doi.org/10.1001/jama.285.20.2612](https://doi.org/10.1001/jama.285.20.2612) [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1001/jama.285.20.2612&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=11368735&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2020%2F05%2F30%2F2020.05.30.20117614.atom) [Web of Science](http://medrxiv.org/lookup/external-ref?access_num=000168761900028&link_type=ISI) 5. 5.Carrieri V, Madio L, Principe F (2019) Vaccine hesitancy and (fake) news: Quasi experimental evidence from Italy. Health Econ 28:1377–1382. [https://doi.org/10.1002/hec.3937](https://doi.org/10.1002/hec.3937) 6. 6.Cervellin G, Comelli I, Lippi G (2017) Is Google Trends a Reliable Tool for Digital Epidemiology? Insights From Different Clinical Settings. Journal of epidemiology and global health 7:185–189. [https://doi.org/10.1016/j.jegh.2017.06.001](https://doi.org/10.1016/j.jegh.2017.06.001) 7. 7.Cuan-Baltazar JY, Muñoz-Perez MJ, Robledo-Vega C et al. (2020) Misinformation of COVID-19 on the Internet: Infodemiology Study. JMIR Public Health Surveill 6:e18444. [https://doi.org/10.2196/18444](https://doi.org/10.2196/18444) [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.2196/18444&link_type=DOI) 8. 8.Discern DISCERN TOOL. Division of Public Health and Primary Health Care at Oxford University. [http://www.discern.org.uk/](http://www.discern.org.uk/). Accessed 20 April 2020 9. 9.Effenberger M, Kronbichler A, Shin JI et al. (2020) Association of the COVID-19 pandemic with Internet Search Volumes: A Google Trends (TM) Analysis. Int J Infect Dis 95:192–197. [https://doi.org/10.1016/j.ijid.2020.04.033](https://doi.org/10.1016/j.ijid.2020.04.033) [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1016/j.ijid.2020.04.033&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2020%2F05%2F30%2F2020.05.30.20117614.atom) 10. 10.Eysenbach G, Powell J, Kuss O et al. (2002) Empirical studies assessing the quality of health information for consumers on the world wide web: a systematic review. Jama 287:2691–2700. [https://doi.org/10.1016/j.jegh.2017.06.001](https://doi.org/10.1016/j.jegh.2017.06.001) [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1001/jama.287.20.2691&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=12020305&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2020%2F05%2F30%2F2020.05.30.20117614.atom) [Web of Science](http://medrxiv.org/lookup/external-ref?access_num=000175732400029&link_type=ISI) 11. 11.Eysenbach G (2009) Infodemiology and Infoveillance: Framework for an Emerging Set of Public Health Informatics Methods to Analyze Search, Communication and Publication Behavior on the Internet. Journal of medical Internet research 11:e11. [https://doi.org/10.2196/jmir.1157](https://doi.org/10.2196/jmir.1157) [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.2196/jmir.1157&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=19329408&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2020%2F05%2F30%2F2020.05.30.20117614.atom) 12. 12.Farooq A, Laato S, Islam AKMN (2020) Impact of Online Information on Self-Isolation Intention During the COVID-19 Pandemic: Cross-Sectional Study. Journal of medical Internet research 22: e19128. [https://doi.org/10.2196/19128](https://doi.org/10.2196/19128) 13. 13.Febres-Cordero B, Brouwer KC, Rocha-Jimenez T et al. (2018) Influence of peer support on HIV/STI prevention and safety amongst international migrant sex workers: A qualitative study at the Mexico-Guatemala border. PLoS One. 13: 1–20. [https://doi:10.1371/journal.pone.0190787](https://doi.org/10.1371/journal.pone.0190787) [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1371/journal.pone.0191962&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=29370308&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2020%2F05%2F30%2F2020.05.30.20117614.atom) 14. 14.Gao J, Zheng P, Jia Y et al. (2020) Mental health problems and social media exposure during COVID-19 outbreak. PLoS One. 15:e0231924. [https://doi.org/10.1371/journal.pone.0231924](https://doi.org/10.1371/journal.pone.0231924) [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1371/journal.pone.0231924&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2020%2F05%2F30%2F2020.05.30.20117614.atom) 15. 15.HON T Health on the Net (HON). HON Foundation. [https://www.hon.ch/en/](https://www.hon.ch/en/). Accessed 20 April 2020. 16. 16.Hua J, Shaw R (2020) Corona Virus (COVID-19) “Infodemic” and Emerging Issues through a Data Lens: The Case of China. Int J Environ Res Public Health 17:2309. [https://doi.org/10.3390/ijerph17072309](https://doi.org/10.3390/ijerph17072309) 17. 17.Institute of Medicine (US) Committee on Assuring the Health of the Public in the 21st Century. The Future of the Public’s Health in the 21st Century. Washington (DC): National Academies Press (US); Available from: [https://www.ncbi.nlm.nih.gov/books/NBK221239/](https://www.ncbi.nlm.nih.gov/books/NBK221239/) doi: 10.17226/10548 18. 18.Jo JH, Kim EJ, Kim JR et al. (2018) Quality and readability of internet-based information on halitosis. Oral Surg Oral Med Oral Pathol Oral Radiol 125:215–222. [https://doi.org/10.1016/j.oooo.2017.12.001](https://doi.org/10.1016/j.oooo.2017.12.001) 19. 19.Kouzy R, Abi Jaoude J, Kraitem A et al. (2020) Coronavirus Goes Viral: Quantifying the COVID-19 Misinformation Epidemic on Twitter. Cureus 12:e7255. [https://doi.org/10.7759/cureus.7255](https://doi.org/10.7759/cureus.7255) 20. 20.Lee K, Robins S, Bragazzi N et al. (2017) Evaluating the Dental Caries-Related Information on Brazilian Websites: Qualitative Study. J Med Internet Res 19:e415. [https://doi.org/10.2196/jmir.7681](https://doi.org/10.2196/jmir.7681) 21. 21.Lee H, Chalmers NI, Brow A et al. (2018) Person-centered care model in dentistry. BMC Oral Health 18:198. [https://doi.org/10.1186/s12903-018-0661-9](https://doi.org/10.1186/s12903-018-0661-9) 22. 22.López-Jornet P, Camacho-Alonso F (2009) The quality of internet sites providing information relating to oral cancer. Oral Oncol 45:e95–98. [https://doi.org/10.1016/j.oraloncology.2009.03.017](https://doi.org/10.1016/j.oraloncology.2009.03.017) [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1016/j.oraloncology.2009.03.017&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=19457707&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2020%2F05%2F30%2F2020.05.30.20117614.atom) [Web of Science](http://medrxiv.org/lookup/external-ref?access_num=000269326700005&link_type=ISI) 23. 23.Lopez-Jornet P, Camacho-Alonso F (2010) The quality of internet information relating to oral leukoplakia. Med Oral Patol Oral Cir Bucal 15:e727–731. [http://doi:10.4317/medoral.15.e727](http://doi.org/10.4317/medoral.15.e727) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=20383096&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2020%2F05%2F30%2F2020.05.30.20117614.atom) 24. 24.Lu X, Zhang R, Wu W et al. (2018) Relationship Between Internet Health Information and Patient Compliance Based on Trust: Empirical Study. J Med Internet Res. 20:e253. [https://doi:10.2196/jmir.9364](https://doi.org/10.2196/jmir.9364) 25. 25.Lu X, Zhang R (2019) Impact of Physician-Patient Communication in Online Health Communities on Patient Compliance: Cross-Sectional Questionnaire Study. J Med Internet Res. 21:e12891. [https://10.2196/12891](https://10.2196/12891) 26. 26.Miguéns-Vila R, Ledesma-Ludi Y, Rodríguez-Lozano F et al. (2018) Disparities between English and Spanish in readability of online endodontic information for laypeople. J Am Dent Assoc. 149:960–966. [https://doi:10.1016/j.adaj.2018.07.003](https://doi.org/10.1016/j.adaj.2018.07.003) 27. 27.Passos KK, Leonel AC, Bonan PR et al. (2020) Quality of information about oral cancer in Brazilian Portuguese available on Google, Youtube, and Instagram. Med Oral Patol Oral Cir Bucal 25:e346–e352. [http://doi:10.4317/medoral.23374](http://doi.org/10.4317/medoral.23374) 28. 28.Petersen CL, Weeks WB, Norin O et al. (2019) Development and Implementation of a Person-Centered, Technology-Enhanced Care Model For Managing Chronic Conditions: Cohort Study. JMIR Mhealth Uhealth 7:e11082. [http://doi:10.2196/11082](http://doi.org/10.2196/11082) 29. 29.Priyanka P, Hadi YB, Reynolds GJ (2018) Analysis of the Patient Information Quality and Readability on Esophagogastroduodenoscopy (EGD) on the Internet. Can J Gastroenterol Hepatol 2018:2849390. [http://doi:10.1155/2018/2849390](http://doi.org/10.1155/2018/2849390) 30. 30.Pulido CM, Ruiz-Eugenio L, Redondo-Sama G et al. (2020) A New Application of Social Impact in Social Media for Overcoming Fake News in Health. Int J Environ Res Public Health 17:2430. [http://doi:10.3390/ijerph17072430](http://doi.org/10.3390/ijerph17072430) 31. 31.Readable.io Readibility Score. [https://readable.io/](https://readable.io/). Accessed 20 April 2020 32. 32.Silberg WM, Lundberg GD, Musacchio RA (1997) Assessing, controlling, and assuring the quality of medical information on the Internet: Caveant lector et viewor--Let the reader and viewer beware. Jama 277:1244–1245 [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1001/jama.1997.03540390074039&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=9103351&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2020%2F05%2F30%2F2020.05.30.20117614.atom) [Web of Science](http://medrxiv.org/lookup/external-ref?access_num=A1997WT40300038&link_type=ISI) 33. 33.Sobota A, Ozakinci G (2015) The quality and readability of online consumer information about gynecologic cancer. Int J Gynecol Cancer. 25:537–541. [https://doi:10.1097/IGC.0000000000000362](https://doi.org/10.1097/IGC.0000000000000362) [Abstract/FREE Full Text](http://medrxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NDoiaWpnYyI7czo1OiJyZXNpZCI7czo4OiIyNS8zLzUzNyI7czo0OiJhdG9tIjtzOjUwOiIvbWVkcnhpdi9lYXJseS8yMDIwLzA1LzMwLzIwMjAuMDUuMzAuMjAxMTc2MTQuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 34. 34.Statcounter (2020) Search Engine Market Share Brazil. GlobalStats. [https://gs.statcounter.com/search-engine-market-share/all/brazil](https://gs.statcounter.com/search-engine-market-share/all/brazil). Accessed 20 April 2020 35. 35.Wahl C, Gregoire JP, Teo K et al. (2005) Concordance, compliance and adherence in healthcare: closing gaps and improving outcomes. Healthc Q. 8:65–70. [https://doi:10.12927/hcq..16941](https://doi.org/10.12927/hcq..16941) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=15715337&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2020%2F05%2F30%2F2020.05.30.20117614.atom) 36. 36.World Health Organization (2017) Communicating risk in public health emergencies: a WHO guideline for emergency risk communication (ERC) policy and practice Switzerland ISBN 978-92-4-155020-8 37. 37.Yang T, Gentile M, Shen CF et al. (2020) Combining Point-of-Care Diagnostics and Internet of Medical Things (IoMT) to Combat the COVID-19 Pandemic. Diagnostics (Basel). 10:e224. [https://doi:10.3390/diagnostics10040224](https://doi.org/10.3390/diagnostics10040224) 38. 38.Zarocostas J (2020) How to fight an infodemic. Lancet 395:676. [http://doi:10.1016/s0140-6736(20)30461-x](http://doi.org/10.1016/s0140-6736(20)30461-x) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2020%2F05%2F30%2F2020.05.30.20117614.atom)