RT Journal Article
SR Electronic
T1 Online health search via multi-dimensional information quality assessment based on deep language models
JF medRxiv
FD Cold Spring Harbor Laboratory Press
SP 2023.04.11.22281038
DO 10.1101/2023.04.11.22281038
A1 Zhang, Boya
A1 Naderi, Nona
A1 Mishra, Rahul
A1 Teodoro, Douglas
YR 2024
UL http://medrxiv.org/content/early/2024/01/11/2023.04.11.22281038.abstract
AB Background Widespread misinformation in Web resources can lead to serious implications for individuals seeking health advice. Despite that, information retrieval models are often focused only on the query-document relevance dimension to rank results.Objective We investigate a multi-dimensional information quality retrieval model based on deep learning to enhance the effectiveness of online healthcare information search results.Methods In this study, we simulated online health information search scenarios with a topic set of 32 different health-related inquiries and a corpus containing one billion Web documents from the April 2019 snapshot of Common Crawl. Using state-of-the-art pre-trained language models, we assessed the quality of the retrieved documents according to their usefulness, supportiveness, and credibility dimensions for a given search query on 6,030 human-annotated query-document pairs. We evaluated this approach using transfer learning and more specific domain adaptation techniques.Results In the transfer learning setting, the usefulness model provided the largest distinction between help- and harm-compatible documents with a difference of +5.6%, leading to a majority of helpful documents in the top-10 retrieved. The supportiveness model achieved the best harm compatibility (+2.4%), while the combination of usefulness, supportiveness, and credibility models achieved the largest distinction between help- and harm-compatibility on helpful topics (+16.9%). In the domain adaptation setting, the linear combination of different models showed robust performance with help-harm compatibility above +4.4% for all dimensions and going as high as +6.8%.Conclusions These results suggest that integrating automatic ranking models created for specific information quality dimensions can increase the effectiveness of health-related information retrieval. Thus, our approach could be used to enhance searches made by individuals seeking online health information.Competing Interest StatementThe authors have declared no competing interest.Funding StatementThis study was funded by Innosuisse project funding numbers 55441.1 IP-ICT and 101.466 IP-ICT.Author DeclarationsI confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.YesI confirm that all necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived, and that any patient/participant/sample identifiers included were not known to anyone (e.g., hospital staff, patients or participants themselves) outside the research group so cannot be used to identify individuals.YesI understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).YesI have followed all appropriate research reporting guidelines and uploaded the relevant EQUATOR Network research reporting checklist(s) and other pertinent material as supplementary files, if applicable.YesThe data sets generated during and/or analyzed during this study are available in the TREC Health Misinformation Track repository [75] and GitLab repository [67].BERTBidirectional Encoder Representations from TransformersBM25Okapi Best Match 25CSScascading style sheetsCLEFConference and Labs of the Evaluation ForumC4Colossal, Cleaned Crawl CorpusNISTNational Institute of Standards and TechnologyNLPnatural language processingRBORank Biased OverlapSMOGSimple Measure of GobbledygookTRECText REtrieval Conference