The accuracy of Japanese administrative data in identifying acute exacerbation of idiopathic pulmonary fibrosis

Settings and patients

This retrospective, multicenter, validation study was conducted to verify the diagnostic accuracy of the disease name for AE-IPF. We performed study in eight Japanese tertiary hospitals: Saiseikai Kumamoto Hospital, Hyogo Prefectural Amagasaki General Medical Center, Iizuka Hospital, Kobe City Medical Center General Hospital, Kameda Medical Center, Hoshigaoka Medical Center, Okinawa Chubu Hospital, and the Japanese Red Cross Medical Center.

We included patients aged ≥ 40 years with a disease name that could meet the diagnostic criteria for AE-IPF (Table 1), who were admitted to the aforementioned eight institutes from January 2016 to February 2019. These criteria were developed in advance by two respiratory specialists. We excluded patients with disease names for secondary interstitial pneumonia, including collagen vascular disease-associated interstitial lung disease and chronic and hypersensitivity pneumonitis, as well as those for malignant neoplasms (Table 2). We did not exclude patients with disease names for malignant neoplasms with the suffixes “post-operation” or “post-radiotherapy”. We ignored all “suspected” disease names.

Regarding disease names, we extracted data from the diagnosis procedure combination (DPC), which was launched at Japanese acute-care hospitals across the country in 2003.[13, 14] The following data were extracted from the DPC: some clinical information (e.g. age, gender, etc.), admission route (e.g. scheduled or emergency admission), diagnoses, surgeries and procedures, medications, outcomes, etc. The patients were recorded with diagnostic names using the International Statistical Classification of Diseases and Related Health Problems, Tenth Revision (ICD-10) codes. The diagnoses included the following six categories: main diagnosis, diagnosis that triggered hospitalization, diagnosis using the most medical resources, diagnosis using the second most medical resources, comorbidities present at admission, and conditions occurring after admission. One diagnostic name is coded for each main diagnosis, diagnosis that triggered hospitalization, diagnosis using the most medical resources, and diagnosis using the second most medical resources. Meanwhile, up to 10 diagnostic names can be coded for comorbidities present at admission and conditions occurring after admission. Furthermore, in the Japanese DPC data, modifiers can be added to the disease name in ICD-10 (e.g., regions such as left and right, recurrence, acute exacerbation, suspected, etc.).

This study was approved by the Ethics Committee of leading institute of this study (Kyoto University Graduate School and Faculty of Medicine) (Approved Number: R2071) and other committed institutes. The requirement for written informed consent from the participants was waived because of the retrospective design and use of anonymized data.

Reference standard

We confirmed the AE-IPF diagnosis using the following two steps:

1. At least two respiratory physicians at each institute reviewed the medical records and chest computed tomography (CT) findings on admission. Subsequently, they determined whether the patients met the following diagnostic criteria: an official ATS/ERS/JRS/ALAT clinical practice guideline for IPF diagnosis [15] and the international working group diagnostic criteria for AE-IPF.[16] An IPF diagnosis was determined if UIP or probable UIP pattern was observed on HRCT findings, even without surgical lung biopsy (SLB). If SLB was performed, IPF was considered based on specific combinations of HRCT and histopathological patterns following the ATS/ERS/JRS/ALAT clinical practice guidelines.[15] Discrepancies were resolved through discussion or consultation with a third respiratory physician, as appropriate.

2. Two expert chest radiologists interpreted chest CT findings of cases considered to be AE-IPF in 1) and determined whether the patients met the aforementioned two diagnostic criteria. If both radiologists considered that a case was not AE-IPF, it was determined as not AE-IPF. On the other hand, if a case was considered as not as AE-IPF by only one radiologist, the case was determined as AE-IPF.

Multidisciplinary discussion (MDD), which is held for diagnostic decision-making regarding patients with interstitial lung disease who are clinically suspected to have IPF, has been described in the official ATS/ERS/JRS/ALAT clinical practice guideline.[15] However, MDD is a conditional recommendation that is not widely held in Japanese hospitals. Therefore, we decided not to consider MDD as a diagnostic requirement.

Developing criteria

First, we assessed the diagnostic accuracy of each disease name in the inclusion criteria, with disease names showing high diagnostic accuracy being adopted without modification. Next, we explored combinations of disease names with high diagnostic accuracy for disease names with a high number of true positives. We sought to create two criteria: 1) the criteria with the highest PPV (specificity-maximizing criteria) and 2) criteria with slightly less PPV but more true positives (balanced criteria). Based on previous studies, [11, 17, 18] we excluded disease names of secondary interstitial lung diseases (ILDs) and included those of acute respiratory failure (ARF) (Table 3). Further, we excluded disease names with pneumonia, influenza, and aspiration pneumonia and addition of that of IPF.

External validation of the criteria of a previous study

To compare the PPV with the aforementioned newly developed criteria, we externally validated the criteria of a previous study [11] (pre-existing criteria) using DPC data extracted from the aforementioned eight hospitals. The inclusion criteria were as follows: 1) age ≥ 50 years; 2) having at least one claim to a specific diagnostic code for IPF (either ICD10 code J84.9, J84.10, J84.111, J84.112, or ICD-9 correspond); 3) having acute respiratory symptoms or ARF (J84.114, J84.9, J96.00-02, J96.20-22, J96.91-92, or ICD-9 correspond) and subsequent hospital admission; and 4) lacking secondary ILDs.[11] In accordance with the Japanese DPC data used in this study, we modified this criteria as follows: 1) age ≥ 50 years; 2) having at least one following disease name (either ICD10 code J84.9 [interstitial pneumonia (IP)] or J84.1 [IPF or idiopathic interstitial pneumonia (IIP) or pulmonary fibrosis] in three diagnosis categories (main diagnosis, diagnosis that triggered hospitalization, diagnosis using the most medical resources); 3) having at least one following disease names (ICD10 code J84.1 [acute interstitial pneumonia] or J96.0 [ARF]) in any diagnosis category, and emergency admission; and 4) lacking secondary ILDs.

Statistical analysis

Interobserver agreement for the AE-IPF diagnosis between both respiratory specialists was evaluated using kappa statistics. For cases diagnosed as AE-IPF by two respiratory physicians, the interobserver agreement for AE-IPF diagnosis between both radiologists was assessed using the AC₁-index since most cases were considered to be true positive cases.[19] We calculated the PPV as true positives / (true positives + false positives) for every disease name and their combinations. Further, we calculated the 95% confidence intervals (CIs) for binomial distribution using the exact method. We could not calculate the sensitivity, specificity, and negative predictive values since we did not perform a chart review of patients without target ICD-10 codes. Validation studies for only obtaining a PPV have reported that a sample size of approximately 100 is sufficient.[20] We sought to identify patients from various disease names and search for algorithms that would improve PPV. Therefore, regarding the specificity-maximizing criteria, we identified the final criteria with the largest PPV out of approximately 100 cases. Statistical analyses were performed using STATA/SE version.16.0 (Stata Corp., College Station, TX, USA).

Specificity-maximizing criteria

We explored algorithms that would result in higher PPV for the five disease names (AE-IIP, acute exacerbation of interstitial pneumonia, AE-IP, IPF + emergency admission, IIP + emergency admission, and IP + emergency admission) and had more true positive patients. We attempted to exclude disease names related to secondary ILDs and infection, and add to disease names for ARF and IPF. Consequently, we observed an increased PPV for several combinations of disease names.

Finally, we combined the groups of disease names with a higher PPV and yielded a criteria (AE-IPF OR (AE-IIP - secondary ILDs - infection) OR (AE-IIP + IPF) OR (AE-IP - secondary ILDs - infection + ARF) with a PPV of 0.72 (95% CI 0.62 to 0.81) (Table 4).

Balanced criteria

Using the five disease names, we searched for a criteria that would include more true-positive patients even with a slightly inferior PPV. Finally, we created a criteria (AE-IPF OR AE-IIP OR [AE-IP - secondary ILDs + ARF] OR [IPF + emergency admission + [ARF OR AE-IP] OR [IIP + emergency admission + (ARF OR AE-IP)] with a PPV of 0.61 (95% CI 0.53 to 0.68) (Table 4).

Pre-existing criteria

The validation results for the pre-existing criteria (IPF OR IIP OR pulmonary fibrosis OR IP) + (AIP OR ARF) + emergency admissions - secondary ILDs) showed a PPV of 0.40 (95% CI 0.31 to 0.49) (Table 4).

Clinical implications

We believe our findings regarding the criteria for identifying patients with AE-IPF can contribute to database research in Japan. Meanwhile, using the specificity-maximizing criteria, only about 1/3 of the cases were true positives. To detect more patients with AE-IPF, it may be better to use the balanced criteria even though it has a lower PPV. Additionally, the specificity-maximizing and balanced criteria are both insufficient for selecting patients with AE-IPF patients without omission. It should be recognized that some patients with disease names, including acute respiratory distress syndrome or drug-induced interstitial pneumonia, may meet the criteria for AE-IPF. Furthermore, the institutions involved in this study were tertiary hospitals that treat numerous patients with AE-IPF. Although DPC data are collected from numerous acute care hospitals, there were not all proficient in treating patients with AE-IPF; thus, our findings may not be generalizable as representative data for Japan.

Research implication

We created criteria for defining AE-IPF with a fair PPV, however, it was not high. Although the present study and a previous validation study were conducted only using disease names and hospitalization information, an algorithm with a higher PPV can be employed by adding other information, including blood test and prescription data. Additionally, using PPV alone cannot allow for a database study that covers the entire AE-IPF population. Therefore, there is a need for studies calculating the sensitivity and specificity using methods, including random sampling or “all possible cases” sampling.[24]

Strength and Limitations

Our study has several strengths. First, this is the first validation study on AE-IPF using Japanese DPC data. Second, this was a multi-center study with a sufficient sample size. Third, AE-IPF diagnosis was more reliable because of the interpretation by two chest radiologists in addition to at least two pulmonologists.

This study had several limitations. First, since this study was conducted at tertiary emergency hospitals with respiratory specialists, it remains unclear whether it can be adapted to the entire DPC data in Japan. Therefore, our findings should be externally validated in other facilities in Japan. Second, this was a validation study of the Japanese DPC data, which is unique to Japan and does not apply to other countries. Finally, we cannot eliminate the possibility of misclassification of the IPF diagnosis (e.g., hypersensitivity pneumonitis and collagen vascular disease) or AE. However, we believe that this was reduced by the interpretation of the chest CT findings by two expert chest radiologists.