Analyzing Pain Patterns in the Emergency Department: Leveraging Clinical Text Deep Learning Models for Real-World Insights ========================================================================================================================== * James A Hughes * Yutong Wu * Lee Jones * Clint Douglas * Nathan Brown * Sarah Hazelwood * Anna-Lisa Lyrstedt * Rajeev Jarugula * Kevin Chu * Anthony Nguyen ## ABSTRACT **Objective** To estimate the prevalence of patients presenting in pain to an inner-city emergency department (ED), describing this population, their treatment, and the effect of the COVID-19 pandemic. **Materials and Methods** We applied a clinical text deep learning model to the free text nursing assessments to identify the prevalence of pain on arrival to the ED. Using interrupted time series analysis, we examined the prevalence over three years. We describe this population pre- and post-pandemic in terms of their demographics, arrival patterns and treatment. **Results** 55.16% (95%CI 54.95% - 55.36%) of all patients presenting to this ED had pain on arrival. There were significant differences in demographics, arrival and departure patterns between those patients with and without pain. The COVID-19 pandemic initially precipitated a decrease followed by a sharp, sustained rise in the prevalence of pain on arrival, altering the population arriving in pain and their treatment. **Discussion** The application of a clinical text deep learning model has successfully identified the prevalence of pain on arrival. The description of this population and their treatment forms the basis of intervention to improve care for patients presenting with pain. The combination of the clinical text deep learning model and interrupted time series analysis has identified the effects of the COVID-19 pandemic on pain care in the ED. **Conclusion** A clinical text deep learning model has led to identifying the prevalence of pain on arrival and was able to identify the effect a major pandemic had on pain care in this ED. Keywords * Artificial Intelligence * Deep Learning Models * Electronic Health Records * Symptoms * Prevalence * COVID-19 ## BACKGROUND Historically, the quality of pain care in the emergency department (ED) has been unsatisfactory (1, 2). Although improvements have been made to pain treatments over the last 30 years, many fundamental aspects of pain care, such as the documentation of pain intensity on arrival and pain reassessment, need to be addressed (3). Inadequate documentation about pain and pain intensity can have immediate consequences for the quality of clinical care, and ongoing consequences for research and quality improvement because it makes it difficult to identify patients who have presented in pain. Instead, researchers typically rely on indirect measures, such as diagnosis or the administration of analgesic medication, to identify their target cohort when auditing medical records (4). However, such indirect measures will likely introduce bias into any resulting samples. The problem of how to accurately identify patients in pain has been brought further into the light with the widespread adoption of electronic health records (EHR) and the subsequent availability of “big data” to pain researchers. However, big data-level studies about ED presentations and pain care provided cannot be realized without changes to documentation practices, or the development of tools that provide easily reproducible and scalable ways to accurately identify patients arriving to the ED in pain. Patients arriving to the ED with pain can be identified in audits of medical records by skilled clinicians reviewing information in the initial clinical assessment, even when pain severity is not documented (4, 6). However, the labor-intensive manual abstraction that is required limits the amount of data that can be reviewed. To address this issue we conducted two pilot studies in which human clinical abstractors were replaced by an “artificial intelligence” (AI) model using Natural Language Processing (NLP) and either Machine or Deep Learning. These conventional Machine Learning and Deep Learning models identified patients with pain from the triage (free text) nursing assessment with accuracies between 88% and 91% (7, 8). However, these studies have significant limitations regarding the adequacy of volume and quality of the training data available. The development of better training datasets (4) has made the development of better models possible. In testing, the model with the best performance has been a fine-tuned domain specific transformer-based deep learning model (9). This model was trained on a subset of data from the same hospital over the same period of time as used in this study. The model will be applied to a dataset of all presentations to a large inner city publicly funded ED over a three-year period with an aim to identify the prevalence of pain on arrival, and to describe this population. The COVID-19 pandemic occurred during the middle of data collection and provides a natural point to compare the effect (if any) of the pandemic on pain care in the ED. ### Objectives 1. Estimate the prevalence of pain on arrival over three years in a large inner-city ED. 2. Describe the population presenting in pain in terms of their demographics, treatment and outcomes compared to patients who did not present in pain. 3. Assess the impact of the COVID-19 pandemic on the prevalence of pain on arrival to the ED, the population presenting in pain and its treatment. ## MATERIALS AND METHODS ### Study Design This is a cross-sectional study using data from two electronic health records that have undergone linkage. This study will be reported in line with the REporting of studies Conducted using Observational Routinely-collected health Data (RECORD) Statement (10). ### Study Setting The Emergency and Trauma Centre (ETC) of the Royal Brisbane and Women’s Hospital (RBWH) is a principal referral and trauma centre for the state of Queensland in Australia. This ED sees approximately 85 000 primary presentations per year catering for adults requiring all specialties. The ED and hospital are publicly funded within a universal health care system. ### Data Sources Two data sources were utilized for this study. The Emergency Department Information System (EDIS™) contains the patient presentation, demographic, assessment, diagnosis and deposition information. This includes the free text of the nursing triage assessment relied on for the classification task. The electronic medication dispensing system (PYXIS™) provided information on the pharmacological treatment of pain in the ED. The information contained within these systems was linked based on the hospital’s unique identifiers, admission and discharge dates and times. Records in either system where unique identifiers were missing were removed before linkage. ### Study Population The study population included all patients who presented to the ETC of RBWH from 1st March 2018 until 28th February 2021. Patients who were under 18 years at the time of presentation (as this study focuses on adult patients in an adult facility), arrived from another hospital (and may have had significant treatment before arrival), or were missing unique identifiers (not allowing for data linkage) were excluded from the population. Figure 1 identifies the sample population and exclusions. ![Figure 1:](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2023/09/25/2023.09.24.23296019/F1.medium.gif) [Figure 1:](http://medrxiv.org/content/early/2023/09/25/2023.09.24.23296019/F1) Figure 1: Study Population Flowchart ### Variables Collected Previous applications of Symptom Management Theory to the study of pain in the ED guided the variables collected for this work (6, 11). Patients’ demographics, including socioeconomic status based on postcode, the workload of the ED on arrival, pharmacological treatment provided and health service outcomes such as time to first analgesic medication, were collected. A previous description of each variable collected has been published (4). These variables are summarized in Table 1. In addition to these variables, narrative triage nursing assessment was collected for the purposes of binary classification (pain/no-pain) by the clinical text deep learning model. View this table: [Table 1:](http://medrxiv.org/content/early/2023/09/25/2023.09.24.23296019/T1) Table 1: Description of the population and differences between those presenting with and without pain. ### Classification of Pain/No Pain A fine-tuned domain specific BlueBERT transformer-based deep learning model trained on 7000 manually coded presentations was used for this study (9). BlueBERT is a domain-specific pre-trained model that utilizes large-scale biomedical literature (PubMed abstracts) and clinical notes, and was fine-tuned to allow the large BlueBERT pretrained model to be trained specifically on the gold standard dataset (12). The gold standard dataset used for training, development and testing was created by manual abstraction of 10 000 presentations to the study setting by three experienced triage clinicians. These clinicians showed excellent agreement in their classification of pain/no pain based on the triage nursing assessment (κ = 0.807 (95% CI 0.795, 0.818), z = 80.703, p <0.001) (4). Previous work based on this dataset evaluated several classification models based on Deep Learning architectures, ranging from simpler Deep Learning architectures such as attention-based Recurrent Neural Network models to more complex general and domain-specific pre-trained transformer-based architectures. Through a learning curve analysis, a fine-tuned domain specific BlueBERT model trained on 7000 presentations was identified as having high accuracy (F1=0.9316) when tested on an evaluation dataset of 2000 presentations, with no further statistically significant increases in performances seen with larger training datasets (9, 13). This model was then deployed on the population in this study to undertake the classification task. ### Data Analysis Data analysis was conducted in IBM SPSS Statistics version 28 (IBM Corp) and R version 4.2.2 (14) #### Missing Data With the exception of records missing unique identifiers or triage assessment no other records were removed because of missing data. All variables with missing data were reported on but remain in the descriptive and bivariate analysis. Cases with missing data underwent case-wise deletion in multivariable models. Description of prevalence and the population Prevalence (and its 95% confidence interval) is reported as a proportion of the population that is identified as arriving to the ED with pain. The population is described in terms of its demographics, arrival patterns, and departure patterns. Continuous variables are assessed for normality visually using P-P plots and described using means, standard deviations and 95% confidence intervals (or their non-parametric equivalents), categorical variables with counts and proportions. Differences in the populations presenting with and without pain were assessed using t-test (or Mann Whitney U test) and the chi-squared test of independence with statistical significance set at p<0.05. Variables where there is a significant difference between the pain and no pain group are entered into a multivariable logistic regression, where a variable remains significant (p<0.05) in the regression it is presented as an adjusted odds ratio. Differences in time to first analgesia between groups is assessed using the log-rank test as part of Kaplin- Meier analysis. ### Impact of COVID-19 The impact of the COVID-19 pandemic on the prevalence of pain on arrival to the ED is assessed using weekly timepoints in an Interrupted Time Series (ITS) Analysis via the its.analysis package (version 1.6.0) in R (15). Visual representations of the ITS analysis and diagnostics are prepared using the timetk package (16). The World Health Organization declared COVID-19 a pandemic on the 11th March 2020 (17) and this date is used as the changeover point for the ITS analysis. The pre and post COVID-19 population presenting in pain are described using the same variables and the same statistical analysis as the population presenting in pain. ## RESULTS ### Missing Data The age of the patient was not recorded in 448 (0.19%) records, sex in 38 (0.02%), postcode (to calculate the Index of Relative Socioeconomic Advantage and Disadvantage (IRSAD) score) in 14 958 (6.34%), employment status in 17 (0.01%), Indigenous (ATSI) status in 60 (0.03%). Significant amounts of missing IRSAD data have been reported in similar populations and has been shown to not be random with significant association to homelessness and international visitors/students (6). Therefore, no further analysis or imputation will occur. ### Prevalence of pain on arrival, description of the population and treatment received The prevalence of pain on arrival to the RBWH ETC over a three-year period was 55.16% (95% CI 54.95% - 55.36%). Table 1 describes the population and compares the population presenting with and without pain on arrival. The population presenting in pain were different to those not presenting in pain in terms of demographics, arrival patterns, urgency, treatment and deposition (Table 1). Overall, the person presenting in pain was less likely to be male, or have a higher socioeconomic status. The Indigenous Population were more likely to present in pain. Those from residential aged care facilities were less likely to present in pain. Patients arriving in pain stayed longer in the ED but were more likely to go home and not represent. Pharmacological therapy remains the main treatment for pain in the ED. Of the patients identified as arriving in pain, 45.9% were identified as having an analgesic medication whilst in the ED with a median time to analgesia of 66 (IQR 38 – 117) minutes. Opioids remain the most common form of pain relief in the ED with 65.5% of all patients receiving pharmacological analgesia, receiving an opioid. There were 146 693 doses of analgesic medication delivered to the 59 667 patients in pain who received pharmacological treatment, representing 2.45 (range 0-22) doses per patient. The top analgesic medications dispensed are summarized in Supplementary Table 1. Patients presented with pain as a symptom of numerous conditions. The top 10 ICD-10 diagnosis for patients presenting in pain are described in Supplementary Table 2. ### Impact of the COVID-19 pandemic on the prevalence of pain on arrival The following figure is an interrupted time series of the weekly prevalence of pain on arrival from March 2018 to February 2021 with separation of the pre and post COVID-19 pandemic periods. The mean pre-pandemic incidence was 546 (SD 14.7) / 1000 presentations and the post- pandemic incidence was 562.9 (SD 18.9) / 1000 presentations. This equates to a mean difference of 16.9 patients presenting in pain per 1000 presentations (F(1,154)=18.710, p<0.001). Figure 2 demonstrates a level and slope change associated with the declaration of the pandemic. An initial decrease of approximately 25 / 1000 presentations was seen, which recovered to pre pandemic levels within nine weeks. This growth in incidence continues for 31 weeks prior to leveling off higher than the pre pandemic levels. Table 2 explores the differences in the populations presenting with pain pre and post pandemic. ![Figure 2:](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2023/09/25/2023.09.24.23296019/F2.medium.gif) [Figure 2:](http://medrxiv.org/content/early/2023/09/25/2023.09.24.23296019/F2) Figure 2: Interrupted time series plot of the rate of pain on presentation to the RBWH ETC pre and post the COVID-19 pandemic. View this table: [Table 2:](http://medrxiv.org/content/early/2023/09/25/2023.09.24.23296019/T2) Table 2: Differences in Demographics, Arrival and Departure metrics in patients presenting in pain pre and post pandemic. The population presenting in pain post pandemic differed to the pre pandemic population in a number of key, interrelated factors. Patients post pandemic appear to be more unwell, as they were less likely to make their own way to the ED (aOR 0.928, 95%CI 0.905-0.953, p<0.001) and were significantly less likely to be discharged home (aOR 0.798, 95%CI 0.766-0.831, p<0.001) or to the short stay unit (aOR 0.828, 95%CI 0.791-0.867, p<0.001). Patients presenting in pain post pandemic were more likely to be unemployed (aOR 1.336, 95%CI 1.247-1.400, p<0.001) and born in Australia (aOR1.043, 95%CI 1.016-1.071, p=0.002). Despite a small decrease in the ED LOS (5 minutes), post pandemic patients were more likely to represent for further care a(OR 1.216, 95%CI 1.139-1.298, p<0.001). In the pre pandemic period 52.17% of patients received analgesia and only 33.24% of patients received analgesia in the post pandemic period (difference 18.92% (95%CI 18.37% – 19.48%, p<0.001) (see Figure 3B). A similar reduction in the proportion of patients receiving opioids was observed (33.60% vs 22.87%, difference 10.73% (95%CI 10.23% - 11.24%), p<0.001) (see Figure 3C). The median time to analgesia (TTA) pre pandemic was 66 min (IQR 39 – 114) minutes and post 67 (IQR 37 – 127) minutes (p<0.001) which represents no clinically significant difference. However, when we consider the impact of the significant reduction in the proportion of patients receiving analgesia and the slightly longer TTA in Kaplan-Meier analysis, the differences become stark. There was a significant difference in the cumulative survival between the two groups (log-rank test ᵡ2 4056.22(1df), p<0.001) (Figure 4). ![Figure 3:](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2023/09/25/2023.09.24.23296019/F3.medium.gif) [Figure 3:](http://medrxiv.org/content/early/2023/09/25/2023.09.24.23296019/F3) Figure 3: Proportions of patients A. Presenting in pain, B Receiving pharmacologic analgesia and C Receiving opiate analgesia per month pre and post the Covid-19 pandemic. ![Figure 4:](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2023/09/25/2023.09.24.23296019/F4.medium.gif) [Figure 4:](http://medrxiv.org/content/early/2023/09/25/2023.09.24.23296019/F4) Figure 4: Survival Curve of the differences in time to analgesia pre- and post-pandemic with associated 95% confidence interval. Post pandemic there was an increase in the use of oxycodone (7%), fentanyl (6%) and ketamine (1%) as can be seen in Supplementary Table 1. There was little change to the pre and post- pandemic ICD-10 diagnosis (Supplementary Table 2), with the addition of “Suspected Viral Illness” in the post pandemic period. ## DISCUSSION A clinical text deep learning model (9) was successfully deployed on three years of data from one large inner-city adult ED. The outcome of this deployment is that the prevalence of pain on arrival was identified as 55.16% of all presentations to the ED, with a significant difference between pre- and post-pandemic prevalence. Pain remains the most common symptom on presentation to the ED, with over 55% of all patients experiencing pain on arrival in this department (4). Fundamentally the patient arriving in pain to the ED is different in terms of age, sex, socioeconomic status, ethnicity, residence, and urgency to those not in pain. However, the COVID-19 pandemic changed both the prevalence of pain on arrival and the population presenting in pain, demonstrating a shift in the reasons the community accessed this ED during the pandemic. This study represents the application of a clinical text deep learning model in answering a clinical question related to pain in the ED. This methodology has been in development for four years (4, 7–9) and stems from other reports of the application of natural language processing to symptom identification in electronic health records (18–20). Emergency Medicine, with a large volume of undifferentiated patients, is an ideal setting for the deployment of artificial intelligence approaches to ED management, prediction of medical conditions and symptoms, patient acuity, deposition, and pre-hospital management (21, 22). The algorithm used in this work achieved an average accuracy of 93.2% (9) when compared to a manually coded random sample from the same dataset as used in this work (4). The prevalence of pain on arrival reported by this methodology (55.16%) compared well to the reported prevalence in a manually coded sample from the same time period and setting (55.22%) (4). Previous reports of prevalence using retrospective methods have identified figures between 49.3% (23) and 62.3% (24) in a variety of EDs from multiple countries. The benefit of this methodology over those reported in these studies is the reproducibility and lack of reliance on documentation of pain, diagnosis of pain, pain intensity measurement or limitations of identified wordsearch. Previous work done in the same ED as this study has identified that pain is poorly documented, with only 33.1% of all patients presenting in pain having a pain intensity score documented (3) therefore reliance on this metric to identify a population may miss up to 67% of the population. The characteristics of the population who present with pain to the ED in this study are fundamentally different to those who do not have pain on arrival. The population presenting in pain are younger, have a greater proportion of females, are more likely to be employed. Patients in pain are more likely to walk-in without pre-hospital care, presenting in the afternoon or night. When seen and treated by a provider are more likely to be discharged home or receive ongoing treatment in the Short Stay Unit. Previous studies have identified that the population presenting in pain are younger than those without pain and more likely to be female (24–26). The COVID-19 pandemic changed the prevalence of pain on arrival to the ED, with an initial decrease and then an increase in the prevalence of pain on arrival. This corresponded with a change in the population presenting in pain and the treatment provided to these patients. Although the pandemic was declared in March 2020 (17) Queensland had it first case of COVID-19 in January 2020 and declared a public health emergency on the 29th January 2020 (27). Initial decreases in the prevalence of pain on arrival correspond to lockdowns ordered in Queensland commencing on the 2nd April 2020 and extended on the 9th April (28). Both domestic and international borders were also either tightly controlled or closed at this time, not fully reopening until January 2022. Some of the changes in the population presenting in pain can be attributed to these changes such as patients more likely to be born in Australia due to the border closures and a higher proportion being unemployed due to labor issues associated with lockdowns. Other characteristics such as increased acuity (demonstrated by a higher proportion using ambulance services, and patients more likely to be admitted) may be representative of a hesitancy of the public to attend EDs during a pandemic, with a significant reduction in all ED attendances experienced in this ED (29) during this time. Previous work by Del Mar et al. (2023) has outlined the functional and operational changes made to this ED during the pandemic (29). With an increasing focus on managing patients in respiratory isolation, expediting admission or discharge, there was a reduction in the number of patients presenting with pain treated with analgesics or opiates. Other metrics such as an increasing representation rate for patients presenting in pain post-pandemic would indicate a level of inadequacy of care. However, without associated patient-reported outcome measures for this period of time we are unable to elaborate further on this phenomenon. ### Limitations In general, deep learning models in healthcare suffer from a number of problems and challenges. In general, these are data insufficiency, model interpretability, privacy and ethical issues and heterogeneity (30). This study goes somewhat to addressing all these concerns. Data insufficiently was both measured and overcome in the development of this algorithm. The incremental multiphase framework used for model optimization having been previously published (9) from a manually coded dataset of 10 000 taken from the same population as this study (4). Interpretability remains difficult with deep learning models, however in the development of this approach we have previously identified a mechanism for interpretability within the task (8). Privacy and ethical issues are addressed within the robust nature of research governance within the jurisdiction this study was conducted. Like other applications of Artificial Intelligence, incorporation of the algorithm described into EHR may have potential ethical issues in the future, which may be justified by a low-risk, high benefit argument (31) in this eventuality. ## Conclusion The use of a clinical text deep learning model has been effective at identifying the prevalence of pain on arrival to the ED from the narrative assessment completed on arrival to the ED. The ability to identify pain at a population level has allowed a robust description of the population, its treatment and the impact of a pandemic on these outcomes. Changes in care associated with the pandemic were closely tied to changes in society and the response of the healthcare system to this major event. ## Significance This work has outlined a methodology for the identification of the patient in pain on presentation to the ED that is not reliant on pain intensity scoring, analgesic administration, or diagnosis. This removes many of the limitations of previous methods. This method allows researchers to take full advantage of the mass of EHR data available to them and assess interventions and associations at a population level rather than from representative samples. ## Data Availability Due to local regulations this data is not available ## Competing Interests Nil ## Funding Source This study was funded by a grant from the Emergency Medicine Foundation (Australia). Grant number EMLE-166R34-2020-Hughes ## Supplementary Material View this table: [Supplementary Table 1:](http://medrxiv.org/content/early/2023/09/25/2023.09.24.23296019/T3) Supplementary Table 1: Analgesia medication administered to patients presenting in pain. View this table: [Supplementary Table 2:](http://medrxiv.org/content/early/2023/09/25/2023.09.24.23296019/T4) Supplementary Table 2: ICD-10 Diagnosis of patients presenting in pain. ## Acknowledgements Nil * Received September 24, 2023. * Revision received September 24, 2023. * Accepted September 25, 2023. * © 2023, Posted by Cold Spring Harbor Laboratory This pre-print is available under a Creative Commons License (Attribution-NonCommercial-NoDerivs 4.0 International), CC BY-NC-ND 4.0, as described at [http://creativecommons.org/licenses/by-nc-nd/4.0/](http://creativecommons.org/licenses/by-nc-nd/4.0/) ## References 1. 1.Doherty S, Knott J, Bennetts S, Jazayeri M, Huckson S. National project seeking to improve pain management in the emergency department setting: Findings from the NHMRC- NICS national pain management initiative. Emergency Medicine Australasia. 2013;25(2):120–6. 2. 2.Wilson JE, Pendleton JM. Oligoanalgesia in the emergency department. The American journal of emergency medicine. 1989;7(6):620–3. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1016/0735-6757(89)90286-6&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=2803357&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2023%2F09%2F25%2F2023.09.24.23296019.atom) [Web of Science](http://medrxiv.org/lookup/external-ref?access_num=A1989AX60800013&link_type=ISI) 3. 3.Hughes JA, Chiu J, Brown NJ, Hills A, Allwood B, Chu K. The documentation of pain intensity and its influences on care in the emergency department. International Emergency Nursing. 2021;57:101015. 4. 4.Hughes JA, Douglas C, Jones L, Brown NJ, Nguyen A, Jarugula R, et al. Identifying patients presenting in pain to the adult emergency department: A binary classification task and description of prevalence. medRxiv. 2022:2022.05. 29.22275652. 5. 5.Rajkomar A, Oren E, Chen K, Dai AM, Hajaj N, Hardt M, et al. Scalable and accurate deep learning with electronic health records. NPJ digital medicine. 2018;1(1):18. 6. 6.Hughes JA, Alexander KE, Spencer L, Yates P. Factors associated with time to first analgesic medication in the emergency department. Journal of Clinical Nursing. 2021;30:16. 7. 7.Vu T, Nguyen A, Brown N, Hughes J, editors. Identifying patients with pain in emergency departments using conventional machine learning and deep learning. Proceedings of the The 17th Annual Workshop of the Australasian Language Technology Association; 2019. 8. 8.Hughes J, Brown N, Vu T, Nguyen A. Towards Automatic and Interpretable Assignments of Patients Presenting with Pain to the Emergency Department. Healthier Lives, Digitally Enabled: IOS Press; 2021. p. 20–5. 9. 9.Wu Y, Hughes JA, Lyrstedt AL HS, Brown NJ, Jones L, Douglas C, et al. Developing Robust Clinical Text Deep Learning Models - a “Painless” Approach. Proceedings of The 19th World Congress on Medical and Health Informatics (Medinfo 2023). 2023. 10. 10.Benchimol EI, Smeeth L, Guttmann A, Harron K, Moher D, Petersen I, et al. The REporting of studies Conducted using Observational Routinely-collected health Data (RECORD) statement. PLoS medicine. 2015;12(10):e1001885. 11. 11.Hughes JA, Alexander KE, Spencer L, Yates P. Factors associated with the experience of patients presenting in pain to the emergency department. Journal of Clinical Nursing. 2022. 12. 12.Peng Y, Yan S, Lu Z. Transfer learning in biomedical natural language processing: an evaluation of BERT and ELMo on ten benchmarking datasets. arXiv preprint arXiv:190605474. 2019. 13. 13.Wu Y. Developing Robust Clinical Text Deep Learning Models - a “Painless” Approach. 2023. 14. 14.Computing. RFfS. R: A language and environment for statistical computing. 4.2.2 ed. Vienna, Austria.2022. 15. 15.English P. its. analysis: Running interrupted time series analysis. R package version 1.6. 0. 2021. 16. 16.Dancho M, Vaughan D. Timetk: A tool kit for working with time series in R. R package version. 2021;2(1). 17. 17.Ciotti M, Ciccozzi M, Terrinoni A, Jiang W-C, Wang C-B, Bernardini S. The COVID- 19 pandemic. Critical reviews in clinical laboratory sciences. 2020;57(6):365–88. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1080/10408363.2020.1783198&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2023%2F09%2F25%2F2023.09.24.23296019.atom) 18. 18.Koleck TA, Dreisbach C, Bourne PE, Bakken S. Natural language processing of symptoms documented in free-text narratives of electronic health records: a systematic review. Journal of the American Medical Informatics Association: JAMIA. 2019;26(4):364–79. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1093/jamia/ocy173&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2023%2F09%2F25%2F2023.09.24.23296019.atom) 19. 19.Koleck TA, Tatonetti NP, Bakken S, Mitha S, Henderson MM, George M, et al. Identifying Symptom Information in Clinical Notes Using Natural Language Processing. Nursing research. 2021;70(3):173–83. 20. 20.Naseri H, Kafi K, Skamene S, Tolba M, Faye MD, Ramia P, et al. Development of a generalizable natural language processing pipeline to extract physician-reported pain from clinical reports: Generated using publicly-available datasets and tested on institutional clinical reports for cancer patients with bone metastases. Journal of Biomedical Informatics. 2021;120:103864. 21. 21.Tang KJW, Ang CKE, Constantinides T, Rajinikanth V, Acharya UR, Cheong KH. Artificial intelligence and machine learning in emergency medicine. Biocybernetics and Biomedical Engineering. 2021;41(1):156–72. 22. 22.Kirubarajan A, Taher A, Khan S, Masood S. Artificial intelligence in emergency medicine: a scoping review. Journal of the American College of Emergency Physicians Open. 2020;1(6):1691–702. 23. 23.Thornton HS, Reynolds J, Coats TJ. A week of pain in the emergency department. British journal of pain. 2018;12(1):58–63. 24. 24.Guéant S, Taleb A, Borel-Kühner J, Cauterman M, Raphael M, Nathan G, et al. Quality of pain management in the emergency department: results of a multicentre prospective study. European Journal of Anaesthesiology| EJA. 2011;28(2):97–105. 25. 25.Cordell WH, Keene KK, Giles BK, Jones JB, Jones JH, Brizendine EJ. The high prevalence of pain in emergency medical care. The American journal of emergency medicine. 2002;20(3):165–9. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1053/ajem.2002.32643&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=11992334&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2023%2F09%2F25%2F2023.09.24.23296019.atom) [Web of Science](http://medrxiv.org/lookup/external-ref?access_num=000175469600006&link_type=ISI) 26. 26.Chang H-Y, Daubresse M, Kruszewski SP, Alexander GC. Prevalence and treatment of pain in EDs in the United States, 2000 to 2010. The American Journal of Emergency Medicine. 2014;32(5):421-31. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1016/j.ajem.2014.01.015&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=24560834&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2023%2F09%2F25%2F2023.09.24.23296019.atom) 27. 27.Government Q. Queensland first to make emergency declarations in January. In: Directory TQCaM, editor. Brisbane, Australia 2020. 28. 28.Government Q. Movement and Gathering Direction (No. 12). In: Health Q, editor. Brisbane 2020. 29. 29.Del Mar P, Kim MJ, Brown NJ, Park JM, Chu K, Burke J. Impact of COVID-19 pandemic on emergency department patient volume and flow: Two countries, two hospitals. Emergency Medicine Australasia. 2023;35(1):97–104. 30. 30.Yang S, Zhu F, Ling X, Liu Q, Zhao P. Intelligent health care: Applications of deep learning in computational medicine. Frontiers in Genetics. 2021:444. 31. 31.Müller S. Is there a civic duty to support medical AI development by sharing electronic health records? BMC Medical Ethics. 2022;23(1):1–12.