Abstract
Purpose The explosion of molecular biomarker and treatment information in the precision medicine era drastically exacerbated difficulty in identifying patient-relevant knowledge for clinical researchers and practitioners. Curated knowledgebases, such as the JAX Clinical Knowledgebase (CKB) are tools to organize and display knowledge in a readily accessible format; however, curators face the same challenges in comprehensively identifying clinically relevant information for curation. Natural language processing (NLP) has emerged as a promising direction for accelerating manual curation, but prior applications were often conceived as stand-alone efforts to automate curation, and the scope is often limited to simple entity and relation extraction. In this paper, we study the alternative paradigm of assisted curation and identify key desiderata to scale up knowledge curation with human-computer symbiosis.
Methods We chose precision oncology for a case study and introduced self-supervised machine reading, which can automatically generate noisy training examples from unlabeled text. We developed a curation user interface (UI) for precision oncology and through iterative “curathons” (curation hackathons), conducted retrospective and prospective user studies for head-to-head comparison between manual and machine-assisted curation.
Results Contrary to the prevailing assumption, we showed that high recall is more important for end-to-end assisted curation. In extensive user studies, we showed that assisted curation can double the curation speed and increase the number of findings by an order of magnitude for previously scarcely curated drugs.
Conclusion We demonstrated that an iterative and thoughtful collaboration between professional curators and NLP researchers can facilitate rapid advances in assisted curation for precision medicine. Human-machine reading symbiosis can potentially be applicable to clinical care and research scenarios where curation is a major bottleneck.
Competing Interest Statement
The authors have declared no competing interest.
Funding Statement
No external funding was received.
Author Declarations
I confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.
Yes
The details of the IRB/oversight body that provided approval or exemption for the research described are given below:
This study does not require IRB approval.
All necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived.
Yes
I understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).
Yes
I have followed all appropriate research reporting guidelines and uploaded the relevant EQUATOR Network research reporting checklist(s) and other pertinent material as supplementary files, if applicable.
Yes
Footnotes
Disclaimer: None
Data Availability
Full-text publication data used for machine reading was from the PubMed Central Open Access Dataset (PMC) https://www.ncbi.nlm.nih.gov/pmc/. The full dataset for the results can be made available upon request.
