A comparison of self-triage tools to nurse driven triage in the emergency department ==================================================================================== * Sachin V Trivedi * Rachit Batta * Nicolas Henao – Romero * Prosanta Mondal * Tracy Wilson * James Stempien ## ABSTRACT **INTRODUCTION** Canadian patients presenting to the emergency department (ED) typically undergo a triage process where they are assessed by a specially trained nurse and assigned a Canadian Triage and Acuity Scale (CTAS) score, indicating their level of acuity and urgency of assessment. We sought to assess the ability of patients to self-triage themselves through use of one of two of our proprietary self-triage tools, and how this would compare with the standard nurse-driven triage process. **METHODS** We enrolled a convenience sample of ambulatory ED patients aged 17 years or older who presented with chief complaints of chest pain, abdominal pain, breathing problems, or musculoskeletal pain. Participants completed one, or both, of an algorithm generated self-triage (AGST) survey, or visual acuity scale (VAS) based self-triage tool which subsequently generated a CTAS score. Our primary outcome was to assess the accuracy of these tools to the CTAS score generated through the nurse-driven triage process. **RESULTS** A total of 223 patients were included in our analysis. Of these, 32 (14.3%) presented with chest pain, 25 (11.2%) with shortness of breath, 75 (33.6%) with abdominal pain, and 91 (40.8%) with musculoskeletal pain. Of the total number of patients, 142 (47.2%) completed the AGST tool, 159 (52.8%) completed the VAS tool and 78 (25.9%) completed both tools. When compared to the nurse-driven triage standard, both the AGST and VAS tools had poor levels of agreement for each of the four presenting complaints. **CONCLUSIONS** Self-triage through use of an AGST or VAS tool is inaccurate and does not appear to be a viable option to enhance the current triage process. Further study is required to show if self-triage can be used in the ED to optimize the triage process. ## INTRODUCTION When patients present to the emergency department (ED), triage serves as their gateway to receiving care. Triage involves the patient being assessed by a specially trained registered nurse, and the process results in a score which assigns a level of acuity and urgency to a given patient’s presenting complaint. The Canadian Triage and Acuity Scale (CTAS) is the reference standard for this output in Canada. The current landscape of emergency medicine is mired by prolonged wait times, which are known to have negative patient outcomes. [1 – 2] Incorporating the patient into the triage process may serve to enhance its efficiency. This notion of self-triage has been studied previously with success in specific populations. [3 – 5] However, this approach may be fraught with challenges owing to individual health literacy. Previous literature would suggest that laypersons may be overcautious in deciding their need to seek medical care, or miss identifying potential emergencies. [6] Indeed, laypersons have also been previously demonstrated to not follow prescribed advice when directed to either present, or not present, to an ED to seek care. [6 – 8] We previously sought to compare a computer-assisted self-triage tool to the traditional method of triage and found low rates of agreement and high rates of over triage, a result consistent with other literature. [4, 6, 9] In this study, we aimed to build on the existing knowledge base and further our previous work by assessing the ability for patients to self-triage. We based this on four specific presenting complaints: chest pain, shortness of breath, abdominal pain and musculoskeletal pain. We tested this through two different methods – an algorithm-generated self-triage (AGST) score, and a visual analogue scale (VAS) based score. In an attempt to see if this is a viable option for the evolution of triage, we aimed to compare the accuracy of these tools against the gold standard of nurse driven triage with respect to their resulting CTAS score. ## METHODS ### Setting This was a prospective, observational pilot study conducted within the tertiary care EDs at Royal University Hospital and St. Paul’s Hospital in Saskatoon, Saskatchewan, Canada. We enrolled a convenience sample of ambulatory patients who presented either independently or with a caregiver that were aged 17 and older. We selected for those who presented with chest pain, abdominal pain, shortness of breath, and musculoskeletal chief complaints. Those excluded were the patients who presented by ambulance and who had different presenting complaints as listed above. We further excluded those patients who identified discrepant presenting complaints between our self-triage tools and the nurse driven triage, as well as those who left the ED prior to completing the formal triage process. Eligible participants were identified by a research assistant who was present in the ED every afternoon, or early evening, for five hours at a time from Monday through Friday beginning on November 4, 2019. Enrollment of participants was stopped due to the beginning of the Covid-19 pandemic in March 11, 2020. Once identified, the research assistant explained the purpose of our project. Participants subsequently had the option to opt in or out of the project. This project was funded by the Royal University Hospital Foundation. The University of Saskatchewan’s Research Ethics Board granted approval for this project (BEH #16-239). Operational approval for this study was obtained from the Saskatchewan Health Authority. ### Intervention and Outcome We designed two tools which were used to generate self-triage CTAS scores. An emergency physician and two emergency medicine residents drafted the initial tools, which were then vetted by a heterogenous group of individuals in an iterative process. This group included a departmental research facilitator, senior ED nurses, and representatives of the Saskatchewan Health Authority’s Patient and Family Advisory Council. Ultimately, the two tools took the form of an AGST survey, and a VAS based tool. Both tools provided a CTAS score output, as described below. For the purposes of this project, we combined the higher acuity CTAS 1 and CTAS 2 patients in the same grouping. Similarly, we grouped CTAS 4 and 5 patients together. The AGST survey (seen in Table 1) consisted of five to six symptoms based yes or no questions depending on presenting complaint. If patients answered “yes” to one or two questions, patients were assigned as being either a CTAS 4 or 5 presentation. Similarly, if patients answered “yes” to three or four categories, they were assigned as being a CTAS 3 presentation. Finally, if all questions were responded to as “yes,” patients were assigned as being a CTAS 2. View this table: [Table 1.](http://medrxiv.org/content/early/2024/01/07/2024.01.04.24300856/T1) Table 1. AGST Survey Questions The VAS based tool consisted of a sliding scale from 0 to 10. It asked four questions which were consistent across all presenting complaints. These were: 1) how much pain are you in right now; 2) how worried are you about your condition right now; 3) how urgently for you think you need to see the doctor; and 4) how much is your condition interfering with your daily activities. The scores for all of these were subsequently averaged. If the average VAS score was between 1 and 3.3, the patient was listed as a CTAS 4 or 5 presentation. If the average score was between 3.4 and 6.6, the patient was assigned as a CTAS 3. Finally, if the score was between 6.7 and 10, the patient was noted as a CTAS 2. Both tools were built on the Ethica app software in order to maintain the necessary level of security and privacy required by our local research ethics board. They were loaded on four Samsung Galaxy 6 smartphones. Depending on individual comfort level, participants were given the option to handle the smartphones themselves in filling out the app, or having the research assistant input their answers for them. Three different versions of the survey were administered at random: 1) one version was a visual analog/ Likert scale; 2) the second version a series of dichotomous closed-ended questions wherein the answers were simply yes or no; and 3) the third version a combination of both scale and questions. Participants enrolling in our study indicated their informed consent through completion of a form which was presented to them on the smartphone they were using prior to using any of our self-triage tools. Following this, they provided demographic details as well as their name and birthdate. At the conclusion of the data collection period, we used the latter information to retrospectively review their chart to acquire the CTAS scores which were generated through nurse-driven triage. This retrospective review was completed on March 2, 2021. Our primary outcome of this study was to assess the level of concordance between both sets of self-triage scores and the reference standard of the triage score assigned by the usual process of nurse driven triage. ### Sample Size Calculation and Statistical Analysis As a pilot correlational study, we intended to have a minimum of 16 participants per each individual variable, defined here as the unique presenting complaints. [10] We sought to meet this minimum for each triage tool. Once all data was collected, it was analyzed quantitatively, using Kappa statistics calculated through the SPSS program. Kappa statistics were used to determine level of correlation between the self-triage tools and the nurse driven triage standard. ## RESULTS A total of 279 participants were enrolled in the study and completed the self-triage questionnaire. Of this group, 39 participants were excluded as their self-reported chief complaint did not match their actual chief complaint. An additional 17 participants were excluded as they had left prior to the formal triage process, and so no actual CTAS score could be determined. Ultimately, 223 participants were included for analysis. The emergency department of Royal University Hospital provided the majority of the data, as the emergency department of Saint Paul’s Hospital was often empty during the months in which it was attended or devoid of patients whose conditions were appropriate for the study. With respect to the different forms of our self-triage tools, a total of 159 participants completed the VAS based tool, and 142 participants completed the AGST survey. Of these participants, 78 completed both self-triage tools. Of the four study categories, chest concerns comprised 14.3% (n = 32) of the responses, breathing concerns comprised 11.2% (n = 25) of the responses, abdominal concerns comprised 33.4% (n = 75) of the responses and musculoskeletal concerns comprised 40.8% (n = 91) of the responses. Table 2 outlines the descriptive statistics of our study. Table 3 displays the concordance of the AGST tool with the nurse-driven triage result, and Table 4 displays the same but with the VAS tool. Table 5 presents the data of the subgroup which completed both the AGST and VAS tools. View this table: [Table 2.](http://medrxiv.org/content/early/2024/01/07/2024.01.04.24300856/T2) Table 2. Demographics View this table: [Table 3.](http://medrxiv.org/content/early/2024/01/07/2024.01.04.24300856/T3) Table 3. AGST tool vs. Nurse-driven Triage View this table: [Table 4.](http://medrxiv.org/content/early/2024/01/07/2024.01.04.24300856/T4) Table 4. VAS triage tool vs. Nurse-driven triage View this table: [Table 5.](http://medrxiv.org/content/early/2024/01/07/2024.01.04.24300856/T5) Table 5. Subgroup analysis of participants who completed both AGST and VAS tools vs. Nurse-driven triage ## DISCUSSION In this study, we endeavored to see if ambulatory patients who presented to the ED could use our proprietary self-triage tools to similar acuity levels in comparison to the reference standard of nurse-driven triage. Our intent with this study was to see if the triage process could evolve in a patient-centered format. Our results demonstrate that, in their current form, both the AGST survey and VAS tools are not appropriate for this purpose. In the case of participants who individually completed either tool, agreement was non-existent, minimal, or weak at best. Though the existing literature has demonstrated a high degree of over-triage, in our study patients both over- and under-triaged themselves. [4, 6, 9] The AGST survey tool appeared to slightly outperform the VAS scale, including in the group of participants who completed both self-triage tools. Though it is important to understand the subjective experience of the patient with respect to treating their symptoms and managing their expectations in a given ED visit, it is likely that the subjectivity of the responses from the VAS score influenced the VAS tool’s relative underperformance. We had hoped that by this degree of patient involvement in triage, the triage component in ED-experience for patients could be optimized. Self-assessments have previously demonstrated some success with triage when they are honed down to specific complaints. [3,5,11] One particular ED based study looked at patients presenting with chest pain used a similar “yes” or “no” based approach like our AGST tool and had high levels of agreement with low risk to patient safety. [11] The success of these studies raises the question of how focused, complaint – specific tools may be utilized to optimize triage overall. Kiosk based triage, though largely focused on early registration of patients, has been studied in terms of optimizing efficiency in triage. [12-13] In considering this, it follows that combining kiosk based triage with the development of individual, complaint-focused tools may represent an area for how triage can evolve. Innovation in triage is not limited to this, however, as other technologies have potential for use. Previously, online-symptom checkers have been studied with respect to their overall diagnostic and triage accuracy, but these have similarly been faced with challenges surrounding accuracy, and rates of over or under triage. [14 – 16] Artificial intelligence may also have uses in triage, with machine learning models having been demonstrated as having some success with specific presenting complaints. [17 – 19] The possibilities of how triage can evolve can take a variety of different forms, and these merit further investigation. Although our study had a negative result, we are confident that it provides additional context to studying the future of ED triage. Our study design was strengthened by the fact that we tested two different self-triage tools and included a group where patients completed both tools, comparing their accuracy against each other with a controlled userbase. Further, we had involved patient and family advisors in the development process so as to capture their insights. We did, however, have some limitations. Bias was likely introduced through our recruitment as it was a convenience sample during daytime and weekday hours. Given how the majority of our patients came from the Royal University Hospital, bias was also inherently introduced as we did not capture a significant population from our inner-city hospital. As well, although we had stated that participation in our study would not result in any expedited assessments in the ED, patients may have still intentionally over-triaged themselves in order to avoid waiting for a prolonged period. Finally, though nurse-driven triage was our reference standard, this is an imperfect system with established variability. [20 – 21] Ultimately, our study showed that self-triage through use of an AGST or VAS tool does not appear to be a viable option for use in the ED. Further studies, including those using focused questionnaires around specific presenting complaints, or those leveraging existing technology, are required to optimize the triage process for ED patients. ## Data Availability As we accessed patient data to acquire their nurse-driven triage scores, we cannot release the full data set publically given how this contains identifying information. Should a reviewer want to access the data, we could provide anonymized data upon request. ## ACKNOWLEDGEMENTS The authors would like to thank Thomas Graham, Marissa Evans, Jenna Mee and Nathaniel Osgood for their contributions to this study. * Received January 4, 2024. * Revision received January 4, 2024. * Accepted January 7, 2024. * © 2024, Posted by Cold Spring Harbor Laboratory This pre-print is available under a Creative Commons License (Attribution 4.0 International), CC BY 4.0, as described at [http://creativecommons.org/licenses/by/4.0/](http://creativecommons.org/licenses/by/4.0/) ## REFERENCES 1. 1.Guttmann A, Schull MJ, Vermeulen MJ, Stukel TA. Association between waiting times and short term mortality and hospital admission after departure from emergency department: Population based cohort study from Ontario, Canada. BMJ. 2011 Jun 4;342: d2983 [Abstract/FREE Full Text](http://medrxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiYm1qIjtzOjU6InJlc2lkIjtzOjE3OiIzNDIvanVuMDFfMS9kMjk4MyI7czo0OiJhdG9tIjtzOjUwOiIvbWVkcnhpdi9lYXJseS8yMDI0LzAxLzA3LzIwMjQuMDEuMDQuMjQzMDA4NTYuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 2. 2.Nyce A, Gandhi S, Freeze B, Bosire J, Ricca T, Kupersmith E, et al. Association of Emergency Department Waiting Times With Patient Experience in Admitted and Discharged Patients. J Patient Exp. 2021;8: 1–7. 3. 3.Eijk ESV, Wefers Bettink-Remeijer M, Timman R, Heres MHB, Busschbach JJV. Criterion validity of a computer-assisted instrument of self-triage (ca-ISET) compared to the validity of regular triage in an ophthalmic emergency department. Int J Med Inform. 2014 Sep 9;85(1):61– 7. 4. 4.Dickson SJ, Dewar C, Richardson A, Hunter A, Searle S, Hodgson LE. Agreement and validity of electronic patient self-triage (eTriage) with nurse triage in two UK emergency departments: a retrospective study. European Journal of Emergency Medicine. 2022 Feb 1;29(1):49–55. 5. 5.Huchon C, Dumont A, Chantry A, Falissard B, Fauconnier A. Triage using a self-assessment questionnaire to detect potentially life-threatening emergencies in gynecology. World Journal of Emergency Surgery. 2014 Aug; 9: 46. 6. 6.Kopka M, Feufel MA, Balzer F, Schmieding ML. The Triage Capability of Laypersons: Retrospective Exploratory Analysis. JMIR Form Res. 2022 Oct 1;6(10): e38977. 7. 7.North F, Jensen TB, Stroebel RJ, Nelson EM, Johnson BJ, Thompson MC, et al. Self-Triage Use, Subsequent Healthcare Utilization, and Diagnoses: A Retrospective Study of Process and Clinical Outcomes Following Self-Triage and Self-Scheduling for Ear or Hearing Symptoms. Health Serv Res Manag Epidemiol. 2023 Jan 1;10: 1–11. 8. 8.Hong AS, Hughes A, Courtney DM, Fullington H, Lee SJC, Sweetenham JW, et al. Characteristics of Self-Triaged Emergency Department Visits by Adults With Cancer. American Journal of Managed Care. 2023 Sep 1;29(9):e267–73. 9. 9.Trivedi S, Littmann J, Stempien J, Kapur P, Bryce R, Betz M. A Comparison Between Computer-Assisted Self-Triage by Patients and Triage Performed by Nurses in the Emergency Department. Cureus. 2021 Mar 19; e14002 10. 10.Tabachnick BG and Fidell L. Using multivariate statistics. 1983. New York: Harper and Row Publishers 11. 11.Ng M, Liu Z, Tan JSL, Ponampalam R. Comparison of self-reported EDACS versus physician-reported EDACS for the triage of chest pain patients in the emergency department. JACEP Open. 2021 Apr 1;2: e12393 12. 12.Joseph MJ, Summerscales M, Yogesan S, Bell A, Genevieve M, Kanagasingam Y. The use of kiosks to improve triage efficiency in the emergency department. Vol. 6, npj Digital Medicine. Nature Research; 2023: 6: 19. 13. 13.Coyle N, Kennedy A, Schull MJ, Kiss A, Hefferon D, Sinclair P, et al. The use of a self-check-in kiosk for early patient identification and queuing in the emergency department. Canadian Journal of Emergency Medicine. 2019 Nov 1;21(6):789–92. 14. 14.Chan F, Lai S, Pieterman M, Richardson L, Singh A, Peters J, et al. Performance of a new symptom checker in patient triage: Canadian cohort study. PLoS One. 2021 Dec 1;16 (12): e0260696. 15. 15.Riboli-Sasco E, El-Osta A, Alaa A, Webber I, Karki M, El Asmar ML, et al. Triage and Diagnostic Accuracy of Online Symptom Checkers: Systematic Review. Vol. 25, Journal of Medical Internet Research. JMIR Publications Inc.; 2023; 25: e43803. 16. 16.Cotte F, Mueller T, Gilbert S, Blümke B, Multmeier J, Hirsch MC, et al. Safety of Triage Self-assessment Using a Symptom Assessment App for Walk-in Patients in the Emergency Care Setting: Observational Prospective Cross-sectional Study. JMIR. 2022 Mar 1;10(3): e32340. 17. 17.Sung SF, Hung LC, Hu YH. Developing a stroke alert trigger for clinical decision support at emergency triage using machine learning. Int J Med Inform. 2021 Aug 1;152: 104505. 18. 18.Patel SJ, Chamberlain DB, Chamberlain JM. A Machine Learning Approach to Predicting Need for Hospitalization for Pediatric Asthma Exacerbation at the Time of Emergency Department Triage. Academic Emergency Medicine. 2018 Dec 1;25(12):1463–70. [PubMed](http://medrxiv.org/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2024%2F01%2F07%2F2024.01.04.24300856.atom) 19. 19.Ellertsson S, Hlynsson HD, Loftsson H, Sigur∂Sson EL. Triaging patients with artificial intelligence for respiratory symptoms in primary care to improve patient outcomes: A retrospective diagnostic accuracy study. Ann Fam Med. 2023 May 1;21(3):240–8. [Abstract/FREE Full Text](http://medrxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6ODoiYW5uYWxzZm0iO3M6NToicmVzaWQiO3M6ODoiMjEvMy8yNDAiO3M6NDoiYXRvbSI7czo1MDoiL21lZHJ4aXYvZWFybHkvMjAyNC8wMS8wNy8yMDI0LjAxLjA0LjI0MzAwODU2LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 20. 20.Mistry B, Stewart De Ramirez S, Kelen G, Schmitz PSK, Balhara KS, Levin S, et al. Accuracy and Reliability of Emergency Department Triage Using the Emergency Severity Index: An International Multicenter Assessment. Ann Emerg Med. 2018 May 1;71(5):581–7. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1016/j.annemergmed.2017.09.036&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2024%2F01%2F07%2F2024.01.04.24300856.atom) 21. 21.Farrohknia N, Castrén M, Ehrenberg A, Lind L, Oredsson S, Jonsson H, et al. Emergency Department Triage Scales and Their Components: A Systematic Review of the Scientific Evidence. Vol. 19, Scandinavian Journal of Trauma, Resuscitation and Emergency Medicine. 2011; 19: 42.