ChatGPT achieves comparable accuracy to specialist physicians in predicting the efficacy of high-flow oxygen therapy

Methods

Medical research ethics approval was obtained from each center (the First Affiliated Hospital of Henan University of Science and Technology 2021-0241; Jiangyan Hospital Affiliated to Nanjing University of Chinese Medicine 2021-016). Patients or their family members provided informed consent, and the study was registered as a clinical trial (ChiCTR2100053027). Full study protocol can be accessed from https://www.chictr.org.cn.

Study design

Recorded baseline data includes: age, gender, body mass index (BMI), mechanical ventilation history, comorbidities, main diagnosis.

Recorded the initial HFNC oxygen therapy and treatment parameters at 6 hours include: blood gas analysis results, respiratory rate, heart rate, pulse oximetry (SpO₂), blood pressure, fraction of inspired oxygen (FiO₂), and oxygen flow and Glasgow Coma Scale (GCS) score. The endpoint of HFNC oxygen therapy observation will be either: tracheal intubation or tracheotomy; patient death; or 48 hours of HFNC treatment.

Clinical outcomes were followed up, including 1) the primary clinical outcome: tracheal intubation within 48 hours, and 2) secondary outcomes: time to tracheal intubation, time to death, 28-day tracheal intubation rate, 28-day mortality rate, and length of stay (LOS) in hospital. The follow-up endpoint was either patient death, discharge, or 28-day hospitalization.

We recorded the above data of 71 patients into 71 natural language questionnaires that asked participants to predict the 48-hour endotracheal intubation risk in patients after HFNC therapy, based on options that were graded from 1 to 6: 1) extremely unlikely to undergo endotracheal intubation, 2) unlikely to undergo endotracheal intubation, 3) possible not to undergo endotracheal intubation, 4) possible to undergo endotracheal intubation, 5 likely to undergo endotracheal intubation, and 6) extremely likely to undergo endotracheal intubation. One forced choice was required. A template of questionnaire was shown in Box 1.

Both GPT-3.5 and GPT-4.0 were used to predict the 48-hour endotracheal intubation risk by prompting the questionnaire contents. Besides, three respiratory and critical care specialists aged 30 to 40 independently completed 23 to 24 questionnaires each, and a total of 71 questionnaires were completed. Three non-specialist physicians aged 30 to 40 independently completed 23 to 24 questionnaires each, and a total of 71 questionnaires were completed (see Table 2 and Figure 2).

View this table:

• View inline
• View popup

Table 1 Clinical characteristics of patients with endotracheal intubation and without intubation within 48 hours of treatment

View this table:

• View inline
• View popup
• Download powerpoint

Table 2 Prediction results

• Download figure
• Open in new tab

Figure 1

Flow chart of study

• Download figure
• Open in new tab

Figure 2

Distribution of accuracy in predicting endotracheal intubation within 48 hours between GPT 4.0 and specialist physicians.

The 6-hour ROX index, which is defined as (SpO₂/FiO₂)/respiratory rate[7], was calculated as a predictor for HFNC failure. We compared ROX prediction results as well as the results of the four questionnaires with the actual outcomes of 48-hour endotracheal intubation in patients. The receiver operating characteristic (ROC) curve was drawn and the area under the curve (AUC) was compared. The optimal diagnostic cutoff point was determined by the Youden index, and the overall accuracy, specificity, sensitivity, positive predictive value, and negative predictive value were calculated accordingly.

Patients was further divided into two groups based on the predicted values of specialist physicians and GPT 4.0 for prediction values ≥4 (high-risk group) and ≤3 (low-risk group), respectively. The 28-day cumulative endotracheal intubation curve and mortality curve were plotted and compared between the two groups of patients. The study flow was shown in Figure 1.

Box 1

The template of natural language questionnaire

The following is an illustration using data from a virtual patient. A female patient in her 60s was admitted to the hospital due to respiratory failure. The patient had not received mechanical ventilation treatment within the previous 24 hours. The patient had a history of cerebrovascular disease, and had no history of smoking.

The patient received high-flow oxygen therapy. At the beginning of high-flow oxygen therapy, the Glasgow Coma Scale was 8 points, the systolic blood pressure was 94 mmHg, the diastolic blood pressure was 46mmHg, the respiratory rate was 31 breaths per minute, the heart rate was 132 beats per minute, the pulse oxygen saturation was 88%, the oxygen flow rate was 40 L/min, the oxygen concentration was 54%, and blood gas analysis showed pH 7.5, pO₂ 60mmHg, pCO₃ 33 mmHg. The patient had received vasopressor medication.

After 6 hours of high-flow oxygen therapy, the patient’s systolic blood pressure was 82 mmHg, the diastolic blood pressure was 56 mmHg, the respiratory rate was 25 breaths per minute, the heart rate was 126 beats per minute, the pulse oxygen saturation was 91%, the oxygen flow rate was 40 L/min, the oxygen concentration was 55%, and blood gas analysis showed pH 7.46, pO₂ 71 mmHg, pCO₂ 3 mmHg. The patient had received vasopressor medication.

Please predict the risk of endotracheal intubation within 48 hours due to the failure of high-flow oxygen therapy according to the following options: 1. extremely unlikely to undergo endotracheal intubation; 2. unlikely to undergo endotracheal intubation; 3. Possible not to undergo endotracheal intubation; 4. possible to undergo endotracheal intubation; 5. likely to undergo endotracheal intubation; 6. extremely likely to undergo endotracheal intubation.

Results

Among 71 patients, 14 patients (19.72%) required endotracheal intubation within 48 hours after HFNC therapy, 21 patients (29.58%) required endotracheal intubation within 28 days, and 16 patients (22.53%) died. There were no statistically significant differences between the intubation group and the non-intubation group in 48 hours after HFNC therapy in terms of baseline data, including age, gender, BMI, comorbidities, and other factors (all P>0.05). However, after 6 hours of HFNC oxygen therapy, the intubation group had significantly decreased pH, PaO₂, and SpO₂ (all P<0.05) compared to the non-intubation group (see Table 1).

The optimal diagnostic cut-off points for GPT-4.0 and specialist physicians were determined to be ≥4. The precision of GPT-4.0 was 76.1% [specificity=78.6% (95%CI=52.4-92.4%); sensitivity=75.4% (95%CI=62.9-84.8%)]. The positive predictive value was 40.7%, and the negative predictive value was 93.5%. The precision of specialist physicians was 80.3% [specificity=71.4% (95%CI=45.4-88.3%); sensitivity=82.5% (95%CI=70.6-90.2%)]. The positive predictive value was 50.0%, and the negative predictive value was 92.2%. The optimal diagnostic cut-off points for GPT-3.5 and non-specialist physicians were determined to be ≥5, with precisions of 73.2% and 64.8% respectively.

The area under the ROC (AUROC) of GPT-4.0 was 0.821 (95% CI=0.698-0.943), which was greater than, but not significantly (p>0.05) different from the AUROCs of GPT-3.5 [0.775 (95%CI=0.652-0.898)] and specialist physicians [0.782 (95%CI=0.619-0.945)], while was significantly higher than the AUROC of non-specialist physicians [0.662 (95%CI=0.518-0.805), P=0.011]. (Table 3 and Figure 3) Grouping the patients by GPT-4.0’s prediction value ≤3 (low-risk group, N=46) and ≥4 (high-risk group, N=25), the 28-day cumulative intubation rate (56.00% vs. 15.22%, P<0.001) and 28-day mortality (44.00% vs. 10.87%, P<0.001) were significantly higher in the high-risk group than in the low-risk group. There were statistically significant differences between the two groups of patients in terms of the parameters of heart rate, respiratory rate, pH, PaO2, SpO2, FiO2, and oxygen flow rate after 6 hours of HFNC therapy (all P<0.05). (Table 4 and Figure 4)

• Download figure
• Open in new tab

Figure 3

ROC curves of predicting endotracheal intubation within 48 hours for GPT and clinical physicians.

• Download figure
• Open in new tab

Figure 4

Cumulative endotracheal intubation curve and cumulative mortality curve of patients grouped by predicted values using GPT-4.0 at 28 days of treatment.

Discussion

In this study, 71 patients with hypoxemia received HFNC oxygen therapy, and 14 of them underwent endotracheal intubation within 48 hours after HFNC therapy. We compared the predictive performance on 48-hour endotracheal intubation risk in patients for GPT-3.5, GPT-4.0, respiratory and critical care specialist physicians, and non-specialist physicians. The predictive AUROC of GPT-3.5, specialist physicians, and ROX index were all lower than 0.8.

For GPT-4.0, the optimal diagnostic cutoff point was determined to be ≥4 based on the maximum Youden index, which is in line with the natural language expression of the outcomes (3-Possible not to undergo endotracheal intubation; 4-possible to undergo endotracheal intubation). Its overall predictive accuracy had a good negative predictive value with an AUC of 0.821, and is significantly better than that of non-specialist physicians (0.662, P=0.011). Our results suggest that GPT-4.0 has clinical judgment experience that is at least better than that of non-specialist physicians on 48-hour endotracheal intubation risk in patients after HFNC therapy.

For both GPT-3.5 and non-specialist physicians, the optimal diagnostic cutoff point was determined to be ≥4, which should be ≥3 according to the natural language expression of the outcomes, indicating that GPT-3.5 and non-specialist physicians tend to overestimate the risk of tracheal intubation for patients due to the lack of experience. Besides, GPT-3.5 rarely gave answers of 1 and 6 in the questionnaire, indicating that it tends to give conservative answers whereas physicians tend to give a diverse range of answers based on their individual clinical judgment.

After grouping the patients according to the optimal diagnostic cut-off point of GPT-4.0, there were significant differences between the high-risk and low-risk patients in parameters such as heart rate, respiratory rate, pH, PaO₂, SpO₂, FiO₂, and oxygen flow rate during the first 6 hours of high-flow oxygen therapy, indicating that GPT4.0 can effectively identify these clinical features to make accurate predictions.

It does not necessarily mean that the specialist physician’s prediction was wrong when their predictions do not match the actual clinical outcomes of patients within a short treatment period of 48 hours. This is because clinical practice is influenced by various factors, and the actual clinical physicians involved in the treatment may also make errors in judgments. However, when a study has adequate sample size, the accuracy of clinical physicians and GPT in predicting the endotracheal intubation risk can be compared based on the actual outcomes of patients.

When GPT predicted the endotracheal intubation risk for patients who had either a high or low intubation risk, its predictive accuracy was good. However, for patients whose intubation risk fell in the intermediate range, the overall accuracy of GPT’s prediction was only ∼50%. Specialist physicians and the ROX index also had poor predictive accuracy for these patients. While these are the patients who need early screening to avoid delayed intubation[9]. Since this study had a small sample size, subgroup analysis was not performed for these patients. Further analysis can be conducted in subsequent large-scale cohort studies.

The ROX index can be used to predict the failure of HFNC therapy. However, it only has a moderate level of predictive accuracy, and its diagnostic cutoff point lacks a unified standard as the ROX index only includes three parameters, namely the ratio of SpO₂/FiO₂ to respiratory rate[8, 10]. Incorporating more physiological parameters may improve the predictive accuracy[11, 12],therefore we expect to improve the predictive accuracy of endotracheal intubation risk by collecting more baseline data and physiological parameters of patients as well as using the algorithm model of ChatGPT. We argue that using GPT to predict the endotracheal intubation risk in patients receiving HFNC oxygen therapy is promising for clinical application[13]. GPT has a great potential to surpass the specialist physicians in terms of the clinical experience on judging endotracheal intubation risk after HFNC therapy[14]. Therefore, it can dynamically monitor patient data and reduce labor costs with its fast and convenient advantages[15].

We are concerned that GPT’s decision is based on accumulated data from actual clinical practice. However, if all clinical physicians rely on GPT to decide whether a patient should undergo endotracheal intubation in the future, the results will further strengthen GPT’s cognitive behavior. GPT would then become both the athlete and the referee. This would run counter to actual clinical needs. We cannot expect artificial intelligence to “grab its own hair to lift it off the ground.” Therefore, we need to prepare corresponding ethics for the clinical application of GPT[16, 17].

Limitations: 1. The answers of GPT are not entirely stable and can give different but similar answers for the same questionnaire. 2. This study is a multicenter prospective cohort study including only 71 patients, and a small number of specialist and non-specialist physicians answering the questionnaire. Therefore, with the progress of GPT model, a larger sample of patients should be included in clinical practice to further validate this conclusion.

Conclusion

GPT-4.0 achieves an accuracy level comparable to specialist physicians in predicting the 48-hour endotracheal intubation risk in patients after HFNC therapy, based on patient baseline data and 6-hour parameters of receiving HFNC therapy. However, further large-sample studies are needed to inspect the reliability of using GPT-4.0 or more advanced version to support clinical decision.

Funding

T.L. and L.S. were supported by Shenyang RMS Medical Tech Company [20210901].

Authors’ contributions

Taotao Liu conceived the idea and led this study. Taotao Liu and Yaocong Duan interpreted the results and drafted the manuscript. Yanchun Li, Yingying Hu, Lingling Su and Aiping Zhang helped to interpret the results and drafted the manuscript. All authors read and approved the final manuscript.

Ethics approval and consent to participate

Medical research ethics approval was obtained from each center (the First Affiliated Hospital of Henan University of Science and Technology 2021-0241; Jiangyan Hospital Affiliated to Nanjing University of Chinese Medicine 2021-016). Patients or their family members provided informed consent, and the study was registered as a clinical trial (ChiCTR2100053027). Full study protocol can be accessed from https://www.chictr.org.cn.

Consent for publication

Not applicable.

Availability of data and material

All data generated or analyzed during this study are included in this article.

Competing interests

The authors declare that they have no competing interests.