1. ABSTRACT
Background and rationale Knee osteoarthritis (OA) is a common disease characterized by reduced function, stiffness, and pain. This clinical diagnosis is commonly supported with radiography of the weight-bearing knee. Radiographic features, such as the Kellgren-Lawrence (KL) grading system, are used as eligibility criteria for clinical studies while others, such as the OARSI grades and minimal joint space width, are used as endpoints for structural OA progression. A higher preoperative KL-grade has been correlated with better pain- and functional outcomes after knee arthroplasty surgery. Consequently, the KL-grade is a common requirement for approving knee arthroplasty among health insurance providers and it is commonly used by orthopedic surgeons as part of determining knee arthroplasty candidacy.
Historically, a radiologist was required to draw on and grade radiographs of the knee to extract the features. With increasing computational power and the increased use of deep convolutional neural networks, off-the-shelf artificial intelligence (AI) tools have become available for automatic extraction of these features. They have received regulatory approval for commercialization, but it is apparent that more diligent external validation is required. Finally, as AI tools begin to mature, new versions are released. It is important to assess how these developments change the current performance of the tool.
Objectives The aim of this analysis is to evaluate the performance of a commercially available AI tool and of readers with different experience levels in orthopedic surgery and radiology at clinically relevant Kellgren-Lawrence grading system thresholds. Additionally, the performance of the AI tool for OARSI grades and patellar osteophytes will be evaluated across two versions of the AI tool.
Methods This study is a secondary analysis of the data from the AutoRayValid-RBknee study, a retrospective observer performance study. It consists of non-fixed-flexion radiographs acquired from the production picture archiving and communications system (PACS) from three European centers. The primary outcome will be the difference in area under the receiver operating curve (AUC) between the readers and the AI tool at the prior authorization clinical criteria threshold (KL ≥ 3). Key secondary outcomes will be radiographic knee osteoarthritis (KL ≥ 2), osteoarthritis clinical trial inclusion (2 ≤ KL ≤ 3), and weight-loss trial inclusion (1 ≤ KL ≤ 3). The AUC of the readers will be computed using the SROC approach as proposed by Oakden-Rayner et al. Further, the performance of the AI tool for grading ordinal OARSI grades will be evaluated using the ordinal ROC as proposed by Obuchowski et al. and the AUC is used for estimating binary OARSI-grade and patellar osteophyte classification performance.
Population Patients with knee pain referred for radiography on suspicion of knee osteoarthritis.
Readers Each center will recruit four readers from across radiology and orthopedic surgery, one in-training and one board-certified for each specialty.
AI tool RBknee-2.2.0 (CE version, KL-grading, OARSI grading, patellar osteophytes) and RBknee-2.1.0 (CE version, KL-grading, OARSI grading, patellar osteophytes) will be used to perform the change impact analysis of advancing product development.
Reference test The reference standard will be determined by the majority vote of three readers, one from each participating hospital who are a board-certified musculoskeletal radiology consultant with expertise in clinical and research evaluation of KOA including extensive experience using the KL-grade.
Sample size Not applicable as this is a secondary analysis.
Framework This is a diagnostic test accuracy study assessing the performance of a commercially available AI tool for radiographic evaluation of knee osteoarthritis according to established grading systems. Additionally, change impact analysis will be performed where multiple versions of the AI tool are available.
Confidence intervals and P values All 95% confidence intervals and P values will use an alpha of 5%.
Multiplicity No explicit multiplicity correction will be performed. Instead, a hierarchical approach will be taken based on tabular order of the tested hypotheses in Table 3.
Statistical software R version 4.2.2 (or newer).
Competing Interest Statement
One author, Mikael Boesen, is a medical advisor for and shareholder of Radiobotics ApS.
Funding Statement
This project has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement No 954221 for the EIC SME Instrument project AutoRay. The work only reflects the authors' view and the European Commission is not responsible for any use that may be made from the information it contains.
Author Declarations
I confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.
Yes
The details of the IRB/oversight body that provided approval or exemption for the research described are given below:
Danish Patient Safety Authority of Denmark waived ethical approval for this work. The IRB of Charite Universitatsmedizin - Berlin waived ethical approval for this work. The IRB of Erasmus Medical Center waived the ethical approval for this work.
I confirm that all necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived, and that any patient/participant/sample identifiers included were not known to anyone (e.g., hospital staff, patients or participants themselves) outside the research group so cannot be used to identify individuals.
Yes
I understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).
Yes
I have followed all appropriate research reporting guidelines, such as any relevant EQUATOR Network research reporting checklist(s) and other pertinent material, if applicable.
Yes
Footnotes
Funding: This project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 954221 for the EIC SME Instrument project AutoRay. The work only reflects the authors’ view and the European Commission is not responsible for any use that may be made from the information it contains.
Revision, 2024 May 27. Expanded knowledge on the clinical relevance of the prior authorization clinical criteria at the Kellgren-Lawrence threshold of KL ≥ 3 has led to a thorough revision of this SAP. As originally planned, we will still evaluate the current and previous versions of the AI tool for the full set of semi-quantitative grades that it can analyze. Additions focus on the above-mentioned clinical threshold. To better estimate the real-world value of the AI tool, we have added a comparison of the AI tool and the readers in the original AutoRayValid-RBknee study. This comparison will be based on the SROC approach proposed by Oakden-Rayner et al. where each reader is treated as a diagnostic test study in a meta-analysis. The KL ≥ 3 threshold will be the primary outcome of the study, with key secondary outcomes being other clinically relevant thresholds: KL ≥ 2 (radiographic knee osteoarthritis threshold),2 ≤ KL ≤ 3 (inclusion criteria in osteoarthritis trials), and 1 ≤ KL ≤ 3 (inclusion criteria in osteoarthritis-related weight-loss trials). We removed the analysis of the joint space width measurements. These will be delegated to future research where fixed-flexion images will be included as well.
Data Availability
All data produced in the present study are available upon reasonable request to the corresponding author.
