Measuring Quality-of-Care in Treatment of Children with Attention-Deficit/Hyperactivity Disorder: A Novel Application of Natural Language Processing

Setting and Population

Packard Children’s Health Alliance (PCHA) is a community-based pediatric healthcare network in the San Francisco Bay Area, affiliated with Stanford Children’s Health and Lucile Packard Children’s Hospital. PCHA has 24 pediatrics primary care offices, grouped into 10 practices. We confirmed that visit diagnosis codes are clinician-entered in all practices.

Data Sources

We conducted a retrospective review of EHRs for a cohort of all pediatric patients seen by PCHA PCPs from October 1, 2015 (date coinciding with adoption of ICD-10 codes) to December 31, 2019. We extracted de-identified structured and free-text data from all office encounters. We reviewed patient records of children aged 4-6 years who had at least 2 visits during the examined period, and who had at least one visit in the study period with a disorder (ADHD) or symptom-level diagnosis code (e.g., hyperactivity, distractibility), as done in our previous study (n=423).²⁵ We excluded patients with a diagnosis of autism in the study period. Figure 1 shows the study workflow.

• Download figure
• Open in new tab

Figure 1.

Study workflow

Manual chart review and annotation

The primary outcome of interest was rate of PCP recommendation of Parent Training in Behavioral Management (PTBM) as part of treatment recommendations in the first visit with an ADHD diagnosis. PTBM recommendations included referral to PTBM and/or counseling the family regarding behavioral management (e.g., providing a handout). To create a “gold standard” for mention of PTBM by PCPs, we developed annotation guidelines to be used by two pediatricians (YB, ST) who performed independent chart review and annotation of the first visit with an ADHD diagnosis for each child in the study cohort.²⁶ Inter-annotator agreement was assessed for mention of PTBM. In step 1, document-level annotations of PTBM mentions were compared between the two annotators and disagreements were used to refine the annotation guidelines. In step 2, annotation guidelines were further refined using the same method. In step 3, the annotators completed the annotation of all notes, and all disagreements (13%) were reviewed and discussed. In cases an agreement could not be reached, a 3^rd pediatrician (L.H.) was used to reach a decision.

Data preparation

We annotated the 423 first ADHD visit notes (i.e., the first visit with a visit diagnosis of ADHD), which we considered to have the highest likelihood of PTBM recommendations. We identified an additional 1020 other notes from all subsequent (follow-up) ADHD visits and all well-care visits between the ages of 4-6 years, which we thought may have included recommendations for PTBM. The annotated set of first ADHD visit notes for the cohort was randomly subset into train (n=296) and test (n=127) sets with a 70-30 split. The train set was used for model development and hyperparameter tuning while the test set was set aside to evaluate model performance on first visits. Temporal validation was conducted by deploying the model on all other patient notes from ADHD or well-care visits (n=1020), which were unannotated.

Notes pre-processing was divided into data cleaning and extraction. All notes were processed through a basic cleaning pipeline that included stripping punctuation, extra whitespace, and digits. A pipeline for extracting sections of interest from the notes was created. The notes were organized in the SOAP (Subjective/Objective/Assessment/Plan) structure. To focus on treatment recommendations by clinicians, the chart review was limited to the Assessment and Plan sections where clinicians are expected to document their impressions, clinical reasoning, and treatment recommendations.

NLP model development

We developed a binary classification pipeline based on various language models to classify notes as containing recommendations for PTBM or not. Six transformer models were used for this task: Bidirectional Encoder Representations from Transformers (BERT) uncased, RoBERTa, XLNet, ClinicalBERT, and BioClinicalBERT. BERT uncased is a variation of the original BERT model that is only trained on lowercase text from general purpose corpora. RoBERTa (Robustly Optimized BERT Approach) was trained on diverse general-purpose corpora. XLNet (eXtreme Language understanding NETwork) and is based on BERT and RoBERTa. It differs from BERT models in that it learns dependencies between words in different contexts by predicting all permutations of the words in a sequence. BioClinicalBERT was trained on MIMIC III notes but was also initialized from BioBERT, which was trained on biomedical corpora.

For each transformer model, a classification layer was incorporated that generated softmax values for each set of scores. Hyperparameter tuning was done on the training set, and performance was evaluated on the test set using various metrics including F1-score, accuracy, precision, recall, and area under the receiver operating characteristic curve (AUROC). Because of low prevalence of positive cases in our dataset, we also evaluated the best performing model with the area under the precision-recall curve (AUPRC), which is a metric that is less sensitive to imbalanced classes. Model thresholds were selected on the precision-recall curve to maximize recall and minimize the false negative rate on the train set. As this work aims to assess clinician adherence to guidelines, we intended to avoid wrongly penalizing clinicians by minimizing the false negative rate. The top-performing model on the test set (n=127) was chosen for temporal validation. The code is available via Github at https://github.com/ybannett/NLP_ADHD.

Temporal validation and error analysis

The best model was deployed on all other ADHD or well-care visit notes (n=1020). All notes classified as positive (n=50) and a random subset of 50 notes (5%) classified as negative were manually reviewed. The misclassified notes were analyzed to understand model errors and potential reasoning behind misclassifications.

Cohort characteristics

This study included 423 patients aged 4-6 years with an ADHD diagnosis. Based on manual chart review of all first visits with an ADHD diagnosis, PTBM was recommended for 30.5% of patients (n=129) at their first visit. Cohort characteristics by PTBM recommendation (yes/no) are displayed in Table 1. Of 423 patients, the average patient age at first ADHD visit was 5.4 years with 74% being male. The cohort primarily consisted of non-Hispanic white patients (n=138, 32.6%) and patients with unknown race/ethnicity (n=131, 31.2%). Most patients were privately insured (n=268, 63.4%); about a third were publicly insured (n=124, 29.3%). Within race/ethnicity subgroups, 33.3% (46/138) of non-Hispanic white patients received behavioral treatment recommendations compared to only 16.7% (4/24) of non-Hispanic black patients. When examining the cohort by insurance type, 32.5% (87/268) of patients with private insurance received behavioral treatment recommendations compared to only 28% (35/124) of patients with public insurance.

Classification results on test set

Five models were compared to classify clinical notes as including a PTBM recommendation or not. Models were trained using the training set (n=296), and performance was evaluated on the test set (n=127) (Table 2). The highest performing model was BioClinicalBERT with an F1-score of 0.78 followed by XLNet with an F1-score of 0.71.

BioClinicalBERT was selected as the final model for temporal validation because it had reasonable performance across all metrics (Table 2). XLNet and BioClinicalBERT achieved the same AUROC (0.87). RoBERTa achieved similar recall to XLNet but had low precision. BioClinicalBERT achieved an area under the precision-recall curve (AUPRC) of 0.73; A prediction probability threshold of 0.001842 was selected after maximizing recall and minimizing the false negative rate in training (Figure 2).

• Download figure
• Open in new tab

Figure 2.

BioClinicalBERT precision-recall curve for threshold selection using the training set

Temporal validation results

Temporal validation was conducted with the best performing model, BioClinicalBERT on all other patient notes following the first visit with an ADHD diagnosis (n=1020). After setting the threshold, 970 samples were classified as negative (not including a PTBM recommendation), and 50 samples were classified as positive. All 50 positively predicted notes and a random sample of 50 negatively predicted were examined in an error analysis.

BioClinicalBERT demonstrated improved performance in temporal validation (Table 3) with a recall of 0.92, representing low rates of false negative classifications (8%) and revealing that pediatricians recommended behavioral treatment in only 5% of non-first-ADHD visits.

Error analysis

An error analysis was performed on predictions from the best performing model, BioClinicalBERT. The error analysis from temporal validation was conducted to understand when the model was misclassifying notes. The few false negative classifications missed mentions of in-office counseling on behavior management and mentions of names of specific therapists. A common false positive classification was recommendations for other types of behavioral treatment for co-occurring conditions documented in well-care visits (e.g., cognitive behavioral therapy for anxiety).

Most misclassified notes were cases with borderline prediction probabilities that were erroneously labeled as counseling on behavioral management. Table 4 illustrates misclassification examples where note 1 is a false negative, and note 2 is a false positive. In note 1, a recommendation for behavior modification was provided, but it was describing a type of behavior modification without referral. In note 2, the false positive may have been caused by mention of a referral for developmental behavioral pediatrics (rather than a PTBM referral).

Limitations

Although the study was conducted in a large network of primary care practices, which included a diverse population (including 14% Hispanic, similar to the US census), the sample size was limited, and the dataset was imbalanced with relatively few positive cases. To mitigate the imbalance, we leveraged BERT and evaluated the final model performance with metrics that are less sensitive to imbalanced classes such as precision and recall. In terms of generalizability, though the study was conducted in a single community-based pediatric network, the variable geographic locations of the practices (n=24) with varying available community services support the generalizability of the study. To further analyze generalizability, we plan to deploy the model on EHR data from other healthcare networks.