Selective prediction for extracting unstructured clinical data

Approach

First, manual abstraction of a fixed number of data points was performed by human abstractors (Figure 1a). This decreases the costs typically associated with having highly trained clinicians perform chart abstraction. Second, natural language processing (NLP) selective prediction models were developed using the data collected by manual abstraction. In contrast to typical ML models that generate a prediction for any input, selective prediction models may abstain from generating a prediction for certain data points (Figure 1b)¹⁸. Third, we leveraged selective prediction modeling to generate predictions for data points that the model can predict with high accuracy and used manual abstraction for data points that the model abstains from predicting. Lastly, we applied a series of quality checks to ensure the fidelity of the final variables.

• Download figure
• Open in new tab

Fig. 1a

Diagram depicting the modeling workflow used to extract unstructured variables. EHR data is abstracted, then used to train and test a model that either predicts an outcome or abstains from a prediction. Charts for which a prediction is not generated are abstracted manually. b Depiction of selective prediction via utility-based thresholding. Two thresholds (Threshold 1 and Threshold 2) are applied to a continuous predicted probability to yield three possible classifications: negative (predicted probability < Threshold 1), positive (predicted probability > Threshold 2), and undetermined (Threshold 1 ≤ predicted probability ≤ Threshold 2). Thresholds are selected based on real world utilities so as to minimize the total cost, defined as (# of FP) × (cost of FP) + (# of FN) × (cost of FN) + (# of undetermined) × (cost of undetermined), where # = number, FP = false positive, and FN = false negative.

Glioblastoma cohort

Patients with mention of the word “glioblastoma” in a pathology report were included. Filtering for pathology reports with the word “glioblastoma” is an effective way of identifying patients who are likely to have primary GBM, since any patient diagnosed with primary GBM would have a pathology report indicating that diagnosis (high sensitivity). Patients younger than 18 at the date of the pathology report that contained the word “glioblastoma” were excluded due to differences in the clinical management of pediatric GBM. The remaining 1195 patients were considered for subsequent analyses. Of these, 629 patients (659 charts) were randomly selected for manual abstraction.

Colorectal cancer cohort

The colorectal cancer cohort was created by selecting patients between January 2012 and December 2021 that had ICD-9 and ICD-10 codes for rectal cancer (154.0 and 154.1 for IDC-9 and C19 and C20 for ICD-10), and Current Procedural Terminology (CPT) codes for treatment of rectal cancer (45110, 45111, 45112, 45114, 45116, 45126, 45395, 44146, 44208, 48.4, 48.5, 48.6, 1007599, 1007604). Patients younger than 18 at the date of the pathology report were excluded due to differences in clinical management. Of the remaining 726 patients, 298 patients (601 charts) were randomly selected for manual abstraction and considered for subsequent analysis.

Confirmation of primary glioblastoma diagnosis

Primary glioblastoma diagnosis was determined by reviewing the relevant pathology reports of patients with suspected GBM. In general, patients diagnosed via surgical resection, biopsy, or specimen review with GBM (WHO grade IV), with or without oligodendroglial or gliosarcoma features, were considered to have primary glioblastoma. Patients with transformed or recurrent GBM diagnoses were not considered to have primary glioblastoma.

Confirmation of rectal adenocarcinoma diagnosis

Rectal adenocarcinoma diagnosis was determined by reviewing the relevant pathology reports of patients with suspected rectal adenocarcinoma. In general, patients diagnosed via surgical resection of rectal adenocarcinoma were considered to have a rectal adenocarcinoma diagnosis.

Confirmation of laparoscopic surgical resection of rectal adenocarcinoma

Confirmation of laparoscopic surgical resection (binary) of rectal adenocarcinoma was made after a patient was confirmed to have been diagnosed with rectal adenocarcinoma. There are two laparoscopic surgical approaches for resection of rectal adenocarcinoma of interest. The first approach is low anterior resection (LAR), which involves removing part of the rectum. The second approach is abdominoperineal resection (APR), which involves removing the anus, rectum, and sigmoid colon. A third, less common approach, is called pelvic exenteration (PE), which involves removing reproductive organs, the bladder or rectum or both, and lymph nodes in the pelvis. Here, we built models to identify patients who underwent APR or LAR. A model to identify patients who underwent PE was not built due to insufficient data.

Feature engineering and feature selection

For each variable, modeling was performed at the level of a patient chart, and each patient may have multiple charts. A randomly selected 60% of charts were used for model training, and the remaining 40% were used for model testing. Features were derived entirely from the free text of pathology reports. Using the text of these documents, TF-IDF (term-frequency-inverse-document-frequency) scores were calculated for unigrams, bigrams, trigrams, and 4-grams. Briefly, TF-IDF is a method used to quantify the relative frequency of an n-gram (an n-word phrase) in a corpus of documents. For example, the TF-IDF score for the unigram “cancer” in a patient’s pathology report is calculated by dividing the frequency of the word “cancer” in that patient’s pathology report (term frequency) by the frequency of the word “cancer” across all pathology reports across all patients (inverse document frequency). L1-regularized logistic regression (Lasso) was used for feature selection and prediction. Hyperparameters (regularization for Lasso and sparsity for document term matrix) were tuned using 10-fold cross-validation. The metric for cross-validation was misclassification cost (described below). Selected features were examined for sensibility and clinical meaningfulness (Supplementary Table 1).

Binary classification

For each binary variable, the output of the lasso regression model was a probability between 0 and 1 indicating the probability that the given chart contains the variable of interest. Typically, a probability threshold is applied to convert a probability prediction model into a classifier. In selecting an optimal probability threshold, relative costs for false negatives and false positives must be specified. Here, “cost” refers not to monetary cost but rather real-world utilities. In the example of GBM diagnosis, the cost of a false positive corresponds to the cost of incorrectly labeling a chart as having GBM, and the cost of a false negative corresponds to the cost of incorrectly labeling a chart as not having GBM. The former results in incorrectly including a patient in the final dataset, whereas the latter results in incorrectly excluding a patient from the final dataset.

We implemented selective prediction by selecting two probability thresholds instead of one. Any chart with a predicted probability of falling between the two thresholds is considered “undetermined”. The idea behind this approach is that a predicted probability very close to 0 or 1 indicates higher model confidence. By labeling charts whose predicted probability is far from 0 or 1 as “undetermined”, the model performance on the charts not labeled “undetermined” is likely to be better than the model performance on the overall sample.

The double threshold requires specifying a relative cost to labeling a chart as “undermined”, as well as costs for false positives and false negatives. A grid search was used to identify the two thresholds which minimize the total cost, defined as follows: where, # is number, FP is false positive, and FN is false negative. With this double threshold approach, we can expect abstraction efficiency gains equal to 100% minus the “undetermined rate”, or the percentage of charts labeled as “undetermined”. Although the model may not be able to classify 100% of charts, we expect more accurate performance on the subset of charts the model is able to classify, thus leading to gains in abstraction efficiency with minimal loss of accuracy.

We compare the double threshold approach to the typical single threshold approach, where predicted probabilities above or below the threshold are labeled “positive” or “negative” respectively.

To quantify the relative costs of FPs, FNs, and undetermined as viewed by researchers (the end users of the model), we created a survey that uses a regret-based approach¹⁹. This validated method asks respondents to consider their preferences and willingness to accept tradeoffs in hypothetical scenarios about patient care. Our survey presents a scenario with 100 patient charts to be classified by an imperfect prediction model and asks respondents to quantify how many patient charts they would be willing to review manually to avoid a false positive and false negative. The responses can be used to generate cost ratios for FP:undetermined and FN:undetermined, respectively. A total of seven clinician and abstractor responses were received. Responses were scanned for consistency and follow up conversations were used to understand motivations for all responses. After removing one outlier response, the remaining responses had similar values and reasoning. A simple average of the remaining six preferences gave the cost ratio for FP:FN:undetermined of 13.5:8:1. This means that the model returning a false positive is 13.5 times more costly than abstaining. Similarly, a false negative result is 8 times more costly than the model abstaining.

Model evaluation

Misclassification cost, sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) were calculated, treating the abstracted labels as ground truth. In addition, we introduce three metrics for the evaluation of selective prediction models: undetermined rate, positive undetermined rate, and negative undetermined rate. Positive undetermined rate is defined as the number of positive observations labeled as undetermined (undetermined positives or UP) over the total number of positive observations within the training dataset (TP + FN + UP). Similarly, the negative undetermined rate is defined as the number of negative observations (UN) labeled as undetermined over the total number of negative observations within the training dataset (TN + FP + UN). Positive and negative undetermined rates give insight into the composition of the undetermined rate. The undetermined rate is defined as the number of predictions a selective prediction model abstains from making (UN + UP), over the total number of possible predictions (UN + UP + FN + FP + TN + TP). A selective prediction model with a lower undetermined rate yields greater efficiency gains than a model with a larger undetermined rate and similar levels of accuracy. For example, if there is an equal proportion of cases and controls in the training dataset, but the positive undetermined rate is much larger than the negative undetermined rate, this suggests the selective prediction model is struggling to predict labels for cases accurately.

After training, the final model was applied to all unlabeled data, yielding three types of predictions: “positive”, “negative” or “undetermined”. For charts labeled positive or negative, the model’s prediction was considered final. Charts labeled “undetermined” were those for which the model abstained from making a definitive prediction. These charts were given to abstractors for manual review.

Failure analysis

Failure analysis involved a manual review of observations where the model prediction resulted in false positives or false negatives during cross-validation. The goal of failure analysis was to determine the reasons for the model’s incorrect prediction. In some cases, the model’s prediction was correct and erroneously labeled as a false positive or false negative due to abstractor error. Potential reasons for abstractor error include mixing up dates of records, misreading or misinterpreting similar words or synonyms, and failing to notice words and phrases. In these cases, the underlying data was amended, the model was retrained, and model accuracy metrics were recalculated.

Comparison to structured data proxies

We assessed whether the output of selective prediction models for GBM diagnosis was more accurate than EHR-derived structured proxy variables. This analysis was not performed for the colorectal cancer cohorts because structured variables were considered in the inclusion criteria for those cohorts. Structured proxy variables for diagnosis of GBM may include the presence of an ICD or CPT code or documented treatment with antineoplastics typically used to treat GBM. The following ICD codes relevant to GBM were included: C71.0 -C71.9 and C72.9 in the ICD-10 scheme; C191.9 in the ICD-9 scheme^20–22. CPT codes related to “glioblastoma” (81287, 81345), “malignant neoplasm of the brain” (70554, 70555, 78600, 78601, 78605, 78606, 78608, 78609, 78610), and surgical procedures related to GBM (61510) were included²³. The proxy for treatment included any documented order or administration of temozolomide or bevacizumab. The labels generated by the model were compared with these structured data elements using the performance metrics described above to quantify the increased accuracy obtained by using an NLP-based model.

Cost survey

Based on the survey conducted (n = 6), a cost ratio for FP:FN:undetermined was assigned as 13.5:8:1 (Table 1).

Colorectal cancer models

Overall, our double threshold models performed better than the single threshold models, with 16%, 35%, and 41% decreases in total cost compared to single threshold models for resection of rectal adenocarcinoma (RRA), abdominoperineal resection (APR), and low anterior resection (LAR), respectively (Table 2). Among the three colorectal cancer (CRC) models, the LAR double threshold classifier had the highest undetermined rate (10%), where 90% of charts were automatically labeled by the LAR classifier. This led to improvements over single threshold modeling in sensitivity (0.71 to 0.91), specificity (0.99 to 1.00), PPV (0.94 to 0.98), and NPV (0.94 to 0.99). Our RRA double threshold classifier had the second highest undetermined rate (9%), where 91% of charts were automatically labeled by the RRA classifier. Similarly, we saw improvements over single threshold modeling in sensitivity (0.87 to 0.93), specificity (0.92 to 0.93), PPV (0.90 to 0.92), and NPV (0.90 to 0.94). Lastly, our APR double threshold classifier had the lowest undetermined rate (5%), where 95% of charts were automatically labeled by our APR classifier. This led to improvements over single threshold modeling in sensitivity (0.81 to 0.91), specificity (0.99 to 0.99), PPV (0.88 to 0.94), and NPV (0.98 to 0.99).

Glioblastoma model

The double threshold model performed better than the single threshold models, with a 51% decrease in total cost compared to the single threshold models (Table 2). The GBM double threshold classifier had an undetermined rate of 43%, meaning that 57% of charts were automatically labeled by the model. This led to improvements over single threshold modeling in sensitivity (0.94 to 0.96), specificity (0.79 to 0.96), PPV (0.89 to 0.98), and NPV (0.88 to 0.91).

For GBM, the double threshold model outperformed all structured proxy variables, yielding costs anywhere from 84.8% to 87.1% lower than those of structured proxy variables (Table 3).

Compared to ICD codes, the double threshold model had higher sensitivity (0.96 to 0.62), specificity (0.96 to 0.46), PPV (0.98 to 0.70) and NPV (0.91 to 0.38). Compared to CPT codes, the double threshold model had higher sensitivity (0.96 to 0.36), specificity (0.96 to 0.59), PPV (0.98 to 0.64) and NPV (0.91 to 0.32). Compared to the proxy variable of GBM-related treatment, the double threshold model had higher sensitivity (0.96 to 0.21), specificity (0.96 to 0.84), PPV (0.98 to 0.71) and NPV (0.91 to 0.34).