AImedReport: A Prototype Tool to Facilitate Research Reporting and Translation of AI Technologies in Healthcare

Introduction

The core of AI research in healthcare is carried out by AI data scientists, AI engineers, and clinicians; however, successfully evaluating and translating AI technologies into healthcare requires cross-collaboration beyond this group. Throughout ideation, development, and validation, successful translation requires engaging with many domains, including AI ethicists, quality management professionals, systems engineers, and more.^1-5 We found through a scoping review that the prioritization of proactive evaluation of AI technologies, multidisciplinary collaboration, and adherence to investigation and validation protocols, transparency and traceability requirements, and guiding standards and frameworks are expected to help address present barriers to translation.⁶ However, as identified by Lu et al⁷ through a systematic review assessing clinical prediction model adherence to reporting guidelines that no consensus exists regarding model details that are essential to report, with some reporting items being commonly requested across reporting guidelines yet other reporting items being unique to specific reporting guidelines. Unless there is clear, consistent, and unified best practices and communication and collaboration across domains, there will be gaps in development, accountability, and implementation.^6-10 Documentation is a crucial part of reporting and translation, but its coordinated maintenance throughout the AI lifecycle remains a challenge.^6,9-11

We have established a proof-of-concept team-based documentation strategy for AI translation to simplify compliance with evaluation and research reporting standards through the development of AimedReport, a reporting guideline documentation repository. AimedReport organizes available reporting guidelines for different phases of the AI lifecycle, consolidating reporting items from different guidelines, assigning specific roles to team members, and guiding relevant information to capture when knowledge is generated (Appendix A).

• Download figure
• Open in new tab

Figure 1. Prepare phase of AImedReport.

Discussion

The interactions among AI technologies, their users, and the implementation environments actively define the overall potential effectiveness of AI interventions within healthcare, especially because these tools are complex interventions designed as clinical decision support systems, not autonomous agents.^8,19,20 A tailored, step-by-step approach may support the transition of AI technologies from being evaluated by statistical performance to clinical validity. To address this translational gap, AImedReport was developed to assist teams in several key areas, including a) outlining phases of the AI lifecycle and clinical evaluation, b) developing a comprehensive documentation deliverable and historical archive, and c) addressing translation, implementation, and accountability gaps. This is achieved by consolidating the existing landscape of research reporting guidelines into a repository. This repository acts as a centralized documentation hub and provides a standardized list of considerations and accountability assignments as the solution advances across the development lifecycle. AImedReport, accessible here, (Appendix A), is presently a prototype tool housed in a spreadsheet, but is planned to be made available as a Web resource and a software platform. This will likely further enhance the tool’s usability, reproducibility, and convenience by providing the ability to automate the documentation process, enhance task completion and generate deliverables in accordance with relevant reporting measures, and allow for communication and updates to model documents to be centrally available across teams.

Introducing such a platform allows for transparent communication of evaluation and reporting measures, but also embraces anticipated changes and modifications that come with development and maintenance. Each “Reporting Item” can be assigned to a team or team member to define who is responsible, accountable, consulted, and informed, who can then use the Reporting Item description as a reference to satisfy their role.²¹ For example, project managers, user experience researchers, and machine learning operations (MLOps) can contribute model overview, goals, and future state from their respective perspectives and reference one another’s vision. Similarly, data scientists, AI ethicists, informatics teams, and clinical practice committees may use documented demographic data of patient populations to assess items such as bias, differential model performance, appropriate clinical location, and potential clinical workflow location. During deployment and maintenance, the primary user and updater of the documentation will be an MLOps team, to ensure requirements set by previous groups are met, monitoring input and output metrics for drift, volume, and appropriate use. This can also facilitate interoperability between organizations, as the tool provides a standardized format, and documentation can be transferred across organizations and research governing bodies for consumption, auditing, and monitoring. AlmedReport also serves as a source of information describing completed evaluation and research reporting measures and can therefore fulfill reporting requirements to support clinical trial documentation and other publications. Additional descriptions of roles and responsibilities included within AImedReport can be found in Appendix B.

This paper focused on describing the theorization and development of AImedReport as a proof of concept to aid in evaluating, consolidating, and understanding available documentation resources to support AI reporting and facilitate communication across a multidisciplinary team. AlmedReport primarily concentrated on research reporting guidelines to address the immediate gaps identified within documentation practices. We note recent progress as the field rapidly advances towards enhancing implementation strategies within multidisciplinary teams. For example, in a study conducted by van der Vegt et al.¹⁸ titled “Implementation Frameworks for End-to-End Clinical AI: Derivation of the SALIENT Framework,” an extensive mapping exercise was conducted to synchronize various guidelines with an AI implementation framework. We suggest that AImedReport could further contribute to such implementation endeavors as a valuable resource. Planned future work will continue to converge with and align to various frameworks, like the SALIENT framework,¹⁸ ABCDS, ²² and organizations, including the Office of the National Coordinator,²³ Food & Drug Administration,²⁴ Coalition for Health AI,²⁵ National Academy of Medicine,⁵ Health AI Partnership,²⁶ National Institute of Standards and Technology,²⁷ World Health Organization,²⁸ and others.

We believe that AImedReport can be used in its current formative state for researchers and healthcare organizations to adhere to evaluation and research reporting standards, as well as to bridge some of the reporting and documentation requirements for products necessitating design controls under good manufacturing practices of the quality system regulation, such as those that may be Software as a Medical Device (SaMD).^28-30 Future iterations of AlmedReport will better align translational science and regulatory science so that documentation can be used directly by teams pursuing regulated pathways, aligning with the information needed by regulatory review groups, accreditation commissions, and regulatory bodies (e.g., Food & Drug Administration).

Appendix