Accuracy of ICD-10 codes for suicidal ideation and action in pediatric emergency department encounters
======================================================================================================

* Rena Xu
* Louisa Bode
* Alon Geva
* Kenneth D. Mandl
* Andrew J. McMurry

## ABSTRACT

**Objectives** According to the ideation-to-action framework of suicidality, suicidal ideation and suicidal action arise via distinct trajectories. Studying suicidality under this framework requires accurate identification of both ideation and action. We sought to assess the accuracy of ICD-10 codes for suicidal ideation and action in emergency department (ED) encounters.

**Methods** Accuracy of ICD-10 coding for suicidality was assessed through chart review of clinical notes for 205 ED encounters among patients 6-18 years old at a large academic pediatric hospital between June 1, 2016, and June 1, 2022. Physician notes were reviewed for documentation of past or present suicidal ideation, suicidal action, or both. The study cohort consisted of 103 randomly selected “cases,” or encounters assigned at least one ICD-10 code for suicidality, and 102 propensity-matched “non-cases” lacking ICD-10 codes. Accuracy of ICD-10 codes was assessed using sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV).

**Results** Against a gold standard chart review, the PPV for ICD-10 suicidality codes was 86.9%, and the NPV was 76.2%. Nearly half of encounters involving suicidality were not captured by ICD-10 coding (sensitivity=53.4%). Sensitivity was higher for ideation-present (82.4%) than for action-present (33.7%) or action-past (20.4%).

**Conclusions** Many cases of suicidality may be missed by relying on only ICD-10 codes. Accuracy of ICD-10 codes is high for suicidal ideation but low for action. To scale the ideation-to-action model for use in large populations, better data sources are needed to identify cases of suicidal action.

Keywords
*   suicide
*   suicidal ideation
*   attempted suicide
*   self-harm
*   self-injury
*   ICD codes

## INTRODUCTION

Suicidality among youth, encompassing suicidal ideation, intentional self-harm, and suicide attempt,1 increased at an alarming rate during the COVID19 pandemic2 and remains an urgent public health concern. Accurate identification of suicidality is essential for research, allocation of healthcare resources, and development of public health programs and clinical interventions. Using International Classification of Diseases, 10th Revision (ICD-10) codes, which are designed primarily to support billing and reimbursement, can underestimate suicidality.3–5 A systematic review of 34 studies reported 2-19% sensitivity and 83-100% positive predictive value (PPV) for ICD-10 codes for suicide attempt and 43-88% PPV for ICD-10 codes for suicidal ideation among adolescents.1

Prevailing theories of suicide, including the three-step theory (3ST),6 interpersonal theory (IPTS), and integrated motivational-volitional model (IMV), are underpinned by an “ideation-to-action”7 framework. Under this framework, suicidal ideation and suicidal action are associated with distinct risk factors, and progression from ideation to action, which occurs in only a minority of individuals with ideation, depends on capability for suicide.7–10 The ideation-to-action model has been used in studies to identify risk factors for suicidal ideation versus action, as well as in interventions aimed at suicide prevention,11 but further research is needed to compare populations experiencing only ideation with those progressing to suicidal action to understand trajectories of progression. Expanded adoption of the ideation-to-action model hinges on accurate, consistent identification of each suicidality subtype. Health care claims are a frequently used data source for suicidality research, but prior studies of ICD-10 coding accuracy have not distinguished ideation and action sufficiently to allow study of the ideation-to-action framework. Understanding ICD-10 coding accuracy for suicidal ideation versus action is important to determine whether suicidality progression can be reliably identified in electronic health record (EHR) and payor claims data and whether such data can be used for care, research, and surveillance under the ideation-to-action model.

We sought to assess the accuracy of ICD-10 codes for capturing both suicidal ideation and action in emergency department (ED) encounters.

## MATERIALS AND METHODS

This retrospective study included all pediatric patients 6-18 years old presenting to the emergency department (ED) of a large, academic, tertiary care children’s hospital between June 1, 2016, and June 1, 2022. If patients required admission to the hospital from the ED, the admission was considered part of the same encounter.

Encounter data were extracted from the EHR, including patient demographics, diagnoses, and clinical notes. Patient demographics included sex, age at visit, and patient-reported race. The ED provider notes and, if present, psychiatry consultation notes and inpatient discharge summaries were also included.

ICD-10 codes for suicidality were identified from prior studies.2,12–20 “Cases” were defined as encounters including one or more of these suicidality ICD-10 codes (**Supplemental file 1**). “Non-cases” were defined as encounters in the same period without ICD-10 codes for suicidality. Non-cases were selected using propensity score matching (PSM), a technique commonly used to control for confounding. The PSM model was fit using patient demographics, ICD-10 diagnoses, and number of documented encounters as independent variables and presence of a suicidality ICD-10 code as the dependent variable (**Supplemental file 2**). The PSM model outputs both the effect sizes for each variable and the likelihood that the patient encounter has a diagnosis of suicidality (propensity score).

Consistent with the established “ideation-to-action” model, suicidality subtypes were defined in this study as suicidal ideation or suicidal action, with the latter including both intentional self-harm (non-suicidal self-injury) and suicide attempt. Suicidality progression was dichotomized as past or present to distinguish current suicidality from a history of suicidality disclosed in the present encounter. This resulted in four suicidality subtypes: *ideation-past, ideation-present, action-past, action-present*. Each encounter could be assigned between zero and four suicidality subtypes.

Guidelines for chart review were developed to classify each subtype of suicidality (**Figure 1**, and additional details in **Supplemental file 3**). Two reviewers (RX, AM) collectively reviewed a total of 205 encounters, consisting of 103 cases and 102 propensity-matched non-cases. Reviewers were blinded as to whether encounters were cases or non-cases. Following an initial independent review of 50 encounters, inter-reviewer agreement was assessed using Cohen’s kappa (κ)21–23. Instances of disagreement were adjudicated by a third member of the study team (AG) and discussed by reviewers to resolve any disagreements. Reviewers then reviewed 155 additional unique encounters (64 by RX, 91 by AM).

![Figure 1.](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2024/07/24/2024.07.23.24310777/F1.medium.gif)

[Figure 1.](http://medrxiv.org/content/early/2024/07/24/2024.07.23.24310777/F1)

Figure 1. Summary of chart review guidelines for suicidality subtypes

Agreement between classification by reviewers and by ICD-10 coded diagnoses was assessed across all 205 reviewed encounters. Sensitivity, specificity, PPV, and NPV were calculated for *ideation-present, action-past, and action-present (but not ideation-past* since ICD-10 does not include a corresponding code).

## RESULTS

The overall ED population comprised 59,866 patients aged 6-18 years old with 90,980 ED encounters during the study period (**Table 1**). Just over half of ED encounters were for children aged 6-11 years old (51.9%). Slightly over half of encounters (51.7%) were for male patients. ICD-10 codes for suicidality were present in 2,637 ED encounters among 2,230 patients, for an ICD-10-based suicidality prevalence of 2.9%. Most suicidality encounters were for adolescents aged 12-18 years (88.2%) and females (69.2%). The distribution of assigned ICD-10 codes among suicidality encounters was suicidal ideation 86.2%, self-harm 39.1%, and suicide attempt 6.7%.

View this table:
[Table 1.](http://medrxiv.org/content/early/2024/07/24/2024.07.23.24310777/T1)

Table 1. Characteristics of overall ED population and study cohort

As compared to overall ED encounters with ICD-10 code for suicidality, cases randomly sampled for chart review (N=103) as well as non-cases identified via propensity-score matching (N=102) had similar demographic proportions (p>0.05 for both age and gender on chi-square analysis; **Supplemental file 4**).

Overall agreement between chart reviewers was almost perfect (κ=0.862), with agreement by suicidality subtype ranging from almost perfect for ideation-present (κ=0.959) and action-past (κ=0.830) to substantial for action-present (κ=0.747). The PPV of ICD-10 suicidality codes was high (PPV=86.9%) (**Table 2**). However, ICD-10 codes failed to capture roughly half of suicidality cases identified by reviewers (sensitivity=53.4%). Among propensity-matched non-cases, nearly a quarter of encounters represented false negatives (NPV=76.2%) — i.e., chart review indicated suicidality, but ICD-10 codes were lacking.

View this table:
[Table 2.](http://medrxiv.org/content/early/2024/07/24/2024.07.23.24310777/T2)

Table 2. Accuracy of ICD-10 codes for suicidality by subtype

Differences in ICD-10 code accuracy were observed by suicidality subtype. The PPV of ICD-10 codes was higher for *ideation-present* (91%) as compared to either *action-present* (87.8%) or *action-past* (65.8%). ICD-10 codes captured most encounters with present suicidal ideation (sensitivity=82.4%) but only a third of all encounters documenting present action (sensitivity=33.7%) and even fewer encounters documenting past action (sensitivity=20.4%).

Detailed chart review guidelines and results are included in **Supplemental files 3 and 4**, respectively.

## DISCUSSION

Growing rates of suicidality 21–24 among youth in the U.S. heighten the urgency to understand drivers of suicidal ideation and progression from ideation to attempt. Consistent with prior studies,1 ICD-10 coding are reliable for identifying suicidal ideation among youth.

However, we find they do not accurately identify action. While the predictive value of ICD-10 codes for suicidality was high in this study, nearly half of all encounters that involved suicidality lacked corresponding ICD-10 codes.

Most studies to date have focused on PPV1,25,26 and have not assessed the rate of false negatives --i.e., encounters that involved suicidality but were not assigned an ICD code for suicidality. Because propensity score matching was used to identify non-cases with the greatest probability of involving suicidality, targeted review of charts could be performed to approximate NPV and gain a more comprehensive view of coding accuracy. The results suggest the true prevalence27 of suicidality in the evaluation cohort was higher than what was calculated using ICD-10 codes.

Accurate data are needed to use the ideation-to-action conceptual model in care, research, and surveillance; to allocate resources appropriately; and to implement interventions. To our knowledge, this study is the first to adopt an ideation-to-action framework for chart review and assessment of coding accuracy. Limitations of the study include its retrospective design, restriction to a single site, and sole focus on ED visits. Furthermore, this study did not assess possible varying accuracy of ICD-10 codes across sociodemographic groups. Additionally, the PSM model did not account for frequency of inpatient admission from the ED or total length of stay. It is possible that patients who spent more time in the hospital, whether due to an admission or prolonged observation in the ED, were exposed to more opportunities for accurate ICD-10 coding. The study may also underestimate the real-world NPV of ICD-10 codes, since the PSM model selected for non-cases that were most likely to include suicidality missed by ICD-10 codes.

Future research may further elucidate ICD-10 coding accuracy for suicidality. For example, accuracy may vary over time for individuals with multiple ED encounters for suicidality. Bentley et al.28 recently used single-system EHR data to assess the accuracy of ICD codes for patients with multiple suicide attempts and reported a PPV of 90% for distinct encounter codes entered at least five days apart. There may be a difference in ICD-10 code sensitivity for capturing suicidal action in patients who have had multiple prior encounters for suicidality as compared to individuals presenting with suicidal action for the first time. Future studies could help to assess the reliability of ICD-10 coding for mapping progression from suicidal ideation to action and more broadly from subacute mental illness to acute suicidality. Future work should also include automated methods for processing clinical notes. Advances in natural language processing will allow for extraction of computable phenotypes of suicidality. This could facilitate replication of the present study across multiple sites of care29–32 and provide a more accurate source of data on suicidality as compared to ICD-10 codes.

## Supporting information

Supplemental File 1 [[supplements/310777_file02.xlsx]](pending:yes)

Supplemental File 2 [[supplements/310777_file03.docx]](pending:yes)

Supplemental File 3 [[supplements/310777_file04.docx]](pending:yes)

Supplemental File 4 [[supplements/310777_file05.docx]](pending:yes)

## Data Availability

All data produced in the present work are contained in the manuscript

## INSTITUTIONAL REVIEW BOARD STATEMENT

The Institutional Review Board of Boston Children’s Hospital waived ethical approval for this work (IRB-P00043211).

## CONFLICTS OF INTEREST AND SOURCE OF FUNDING

The authors declare no conflict of interest. This study was supported by the Centers for Disease Control and Prevention (CDC) of the U.S. Department of Health and Human Services (HHS) as part of a financial assistance award. The contents are those of the authors and do not represent the official views of, or an endorsement by, CDC, HHS, or the U.S. government. Support was provided by the National Center for Advancing Translational Sciences, National Institutes of Health Cooperative Agreement (U01TR002623).

## ACKNOWLEDGMENTS

We thank our colleagues at the Massachusetts Department of Public Health for providing public health insights into the challenges of suicidality progression in children. The authors especially thank Sita Smith, Rosa Ergas, Alyssa Anwar, Matthew Tumpney, Dylan Keating, Lauren Larochelle, Victoria Nielsen, Lindsey Sagasta, Megha Parikh, Jennifer Ledet, and Anna Hammelrath, as well as valued colleagues at the CDC foundation, Elizabeth Sprouse and Lyndsey Kirchner. We also thank Tim Miller and Karen Olson for their advice on the study methodology.

*   Received July 23, 2024.
*   Revision received July 23, 2024.
*   Accepted July 24, 2024.


*   © 2024, Posted by Cold Spring Harbor Laboratory

The copyright holder for this pre-print is the author. All rights reserved. The material may not be redistributed, re-used or adapted without the author's permission.

## REFERENCES

1.  1.Swain RS, Taylor LG, Braver ER, Liu W, Pinheiro SP, Mosholder AD. A systematic review of validated suicide outcome classification in observational studies. Int J Epidemiol 2019;48(5):1636–49.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1093/ije/dyz038&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2024%2F07%2F24%2F2024.07.23.24310777.atom) 

2.  2.Zipursky AR, Olson KL, Bode L, et al. Emergency department visits and boarding for pediatric patients with suicidality before and during the COVID-19 pandemic. PLoS One 2023;18(11):e0286035.
    
    

3.  3.Data and Surveillance Task Force of the National Action Alliance for Suicide Prevention. Improving national data systems for surveillance of suicide-related events. Am J Prev Med 2014;47(3 Suppl 2):S122–9.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1016/j.amepre.2014.05.026&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=25145729&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2024%2F07%2F24%2F2024.07.23.24310777.atom) 

4.  4.Sveticic J, Stapelberg NC, Turner K. Suicidal and self-harm presentations to Emergency Departments: The challenges of identification through diagnostic codes and presenting complaints. Health Inf Manag 2020;49(1):38–46.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1177/1833358319857188&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=31272232&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2024%2F07%2F24%2F2024.07.23.24310777.atom) 

5.  5.Walkup JT, Townsend L, Crystal S, Olfson M. A systematic review of validated methods for identifying suicide or suicidal ideation using administrative or claims data. Pharmacoepidemiol Drug Saf 2012;21 Suppl 1:174–82.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1002/pds.2335&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=22262604&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2024%2F07%2F24%2F2024.07.23.24310777.atom) 

6.  6.Klonsky ED, Pachkowski MC, Shahnaz A, May AM. The three-step theory of suicide: Description, evidence, and some useful points of clarification. Prev Med 2021;152(Pt 1):106549.
    
    

7.  7.Klonsky ED, Saffer BY, Bryan CJ. Ideation-to-action theories of suicide: a conceptual and empirical update. Curr Opin Psychol 2018;22:38–43.
    
    

8.  8.Okado I, Floyd FJ, Goebert D, Sugimoto-Matsuda J, Hayashi K. Applying ideation-to-action theories to predict suicidal behavior among adolescents. J Affect Disord 2021;295:1292–300.
    
    

9.  9.Li Y, Kwok SYCL. A Longitudinal Network Analysis of the Interactions of Risk and Protective Factors for Suicidal Potential in Early Adolescents. J Youth Adolesc 2023;52(2):306–18.
    
    

10. 10.Klonsky ED, Qiu T, Saffer BY. Recent advances in differentiating suicide attempters from suicide ideators. Curr Opin Psychiatry 2017;30(1):15–20.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1097/YCO.0000000000000294&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2024%2F07%2F24%2F2024.07.23.24310777.atom) 

11. 11.Anestis MD, Law KC, Jin H, Houtsma C, Khazem LR, Assavedo BL. Treating the Capability for Suicide: A Vital and Understudied Frontier in Suicide Prevention. Suicide Life Threat Behav 2017;47(5):523–37.
    
    

12. 12.Barak-Corren Y, Castro VM, Javitt S, et al. Predicting Suicidal Behavior From Longitudinal Electronic Health Records. Am J Psychiatry 2017;174(2):154–62.
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2024%2F07%2F24%2F2024.07.23.24310777.atom) 

13. 13.Hedegaard H, Schoenbaum M, Claassen C, Crosby A, Holland K, Proescholdbell S. Issues in Developing a Surveillance Case Definition for Nonfatal Suicide Attempt and Intentional Self-harm Using International Classification of Diseases, Tenth Revision, Clinical Modification (ICD-10-CM) Coded Data. Natl Health Stat Report 2018;(108):1–19.
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2024%2F07%2F24%2F2024.07.23.24310777.atom) 

14. 14.Ridout KK, Alavi M, Ridout SJ, et al. Emergency Department Encounters Among Youth With Suicidal Thoughts or Behaviors During the COVID-19 Pandemic. JAMA Psychiatry 2021;78(12):1319–28.
    
    

15. 15.Sacco DL, Probst MA, Schultebraucks K, Greene MC, Chang BP. Evaluation of emergency department visits for mental health complaints during the COVID-19 pandemic. J Am Coll Emerg Physicians Open 2022;3(3):e12728.
    
    

16. 16.Tadros A, Sharon M, Crum M, Johnson R, Quedado K, Fang W. Coexistence of Substance Abuse among Emergency Department Patients Presenting with Suicidal Ideation. Biomed Res Int 2020;2020:7460701.
    
    

17. 17.Zima BT, Gay JC, Rodean J, et al. Classification System for International Classification of Diseases, Ninth Revision, Clinical Modification and Tenth Revision Pediatric Mental Health Disorders. JAMA Pediatr 2020;174(6):620–2.
    
    

18. 18.Zhdanava M, Voelker J, Pilon D, et al. Excess healthcare resource utilization and healthcare costs among privately and publicly insured patients with major depressive disorder and acute suicidal ideation or behavior in the United States. J Affect Disord 2022;311:303–10.
    
    

19. 19.Wilimitis D, Turer RW, Ripperger M, et al. Integration of Face-to-Face Screening With Real-time Machine Learning to Predict Risk of Suicide Among Adults. JAMA Netw Open 2022;5(5):e2212095.
    
    

20. 20.Walsh CG, Johnson KB, Ripperger M, et al. Prospective Validation of an Electronic Health Record-Based, Real-Time Suicide Risk Model. JAMA Netw Open 2021;4(3):e211428.
    
    

21. 21.Charpignon M-L, Ontiveros J, Sundaresan S, et al. Evaluation of Suicides Among US Adolescents During the COVID-19 Pandemic. JAMA Pediatr [Internet] 2022; Available from: doi:10.1001/jamapediatrics.2022.0515
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1001/jamapediatrics.2022.0515&link_type=DOI) 

22. 22.Curtin SC, Garnett MF. Suicide and Homicide Death Rates Among Youth and Young Adults Aged 10-24: United States, 2001-2021. NCHS Data Brief 2023;(471):1–8.
    
    

23. 23.Ajluni V, Amarasinghe D. Youth suicide crisis: identifying at-risk individuals and prevention strategies. Child Adolesc Psychiatry Ment Health 2024;18(1):58.
    
    

24. 24.Massachusetts Youth Health Survey (MYHS) [Internet]. Mass.gov. [cited 2024 Jun 17]; Available from: [https://www.mass.gov/lists/massachusetts-youth-health-survey-myhs](https://www.mass.gov/lists/massachusetts-youth-health-survey-myhs)
    
    

25. 25.Hansen A, Slavova D, Cooper G, Zummer J, Costich J. An emergency department medical record review for adolescent intentional self-harm injuries. Inj Epidemiol 2021;8(1):3.
    
    

26. 26.Gabella BA, Hume B, Li L, Mabida M, Costich J. Multi-site medical record review for validation of intentional self-harm coding in emergency departments. Inj Epidemiol 2022;9(1):16.
    
    

27. 27.Habibzadeh F, Habibzadeh P, Yadollahie M. The apparent prevalence, the true prevalence. Biochem Med 2022;32(2):020101.
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=35799992&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2024%2F07%2F24%2F2024.07.23.24310777.atom) 

28. 28.Bentley KH, Madsen EM, Song E, et al. Determining Distinct Suicide Attempts From Recurrent Electronic Health Record Codes: Classification Study. JMIR Form Res 2024;8:e46364.
    
    

29. 29.Mandl KD, Kohane IS. Federalist principles for healthcare data networks. Nat Biotechnol 2015;33(4):360–3.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1038/nbt.3180&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=25850061&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2024%2F07%2F24%2F2024.07.23.24310777.atom) 

30. 30.Mandl KD, Gottlieb D, Mandel JC, et al. Push Button Population Health: The SMART/HL7 FHIR Bulk Data Access Application Programming Interface. NPJ Digit Med 2020;3(1):151.
    
    

31. 31.McMurry AJ, Murphy SN, MacFadden D, et al. SHRINE: enabling nationally scalable multi-site disease studies. PLoS One 2013;8(3):e55811.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1371/journal.pone.0055811&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=23533569&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2024%2F07%2F24%2F2024.07.23.24310777.atom) 

32. 32.McMurry AJ, Gottlieb DI, Miller TA, et al. Cumulus: a federated electronic health record-based learning system powered by Fast Healthcare Interoperability Resources and artificial intelligence. J Am Med Inform Assoc [Internet] 2024; Available from: doi:10.1093/jamia/ocae130
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1093/jamia/ocae130&link_type=DOI)