Predicting Olfactory Loss In Chronic Rhinosinusitis Using Machine Learning
==========================================================================

* Vijay R. Ramakrishnan
* Jaron Arbet
* Jess C. Mace
* Zachary M. Soler
* Timothy L. Smith

## ABSTRACT

**Objective** Compare machine learning (ML) based predictive analytics methods compared to traditional logistic regression in classification of olfactory dysfunction in chronic rhinosinusitis (CRS-OD), and identify predictors within a large multi-institutional cohort of refractory CRS patients.

**Methods** Adult CRS patients enrolled in a prospective, multi-institutional, observational cohort study were assessed for baseline CRS-OD using a smell identification test (SIT) or brief SIT (bSIT). Four different ML methods were compared to traditional logistic regression for classification of CRS normosmics versus CRS-OD.

**Results** Data were collected for 611 study participants who met inclusion criteria between April 2011 and July 2015. 34% of enrolled patients demonstrated olfactory loss on objective testing. Differences between CRS normosmics and those with smell loss included objective disease measures (CT and endoscopy scores), age, sex, prior surgeries, socioeconomic status, steroid use, polyp presence, asthma, and aspirin sensitivity. Most ML methods outperformed traditional logistic regression in terms of predictive ability. Top predictors include known factors reported in the literature, as well as several socioeconomic factors.

**Conclusion** Olfactory dysfunction is a variable phenomenon within a large multicenter cohort of refractory CRS patients. ML methods outperform traditional logistic regression in classification of normosmia versus smell loss in CRS, and are able to include numerous risk factors into prediction models. These results carry implications for basic science and clinical research in hyposmia secondary to sinonasal disease, the most common cause of persistent olfactory loss in the general population.

## INTRODUCTION

Olfactory loss is common, affecting up to 25% of the adult population, with chronic rhinosinusitis (CRS) being a major cause of persistent olfactory loss [1]. Smell loss is considered a hallmark of CRS, particularly in those with nasal polyps, and is one of the cardinal diagnostic symptoms for CRS [2-4]. Curiously, this symptom does not affect all CRS patients. Just as CRS is a heterogeneous disease [5], previous literature has estimated a 20-80% prevalence of olfactory loss in CRS cohorts, based on the sensitivity of olfactory assessment used and the proportion of enrolled subjects with nasal polyps (CRSwNP). A recent study using well-defined CRS diagnostic criteria and objective testing using Sniffin’ Sticks assessment showed a 44% prevalence of smell loss in CRSsNP subjects and 58% in CRSwNP [6]. Risk factors associated with CRS-OD identified in this study included disease severity score on computed tomography (CT) the presence of comorbid conditions such as diabetes, allergy, and asthma. Adding a healthy control group for comparison, Schlosser et al. noted CRS-OD to affect CRSwNP patients more than CRSsNP, and identified age as an additional risk factor [7]. Given that most existing studies consist of single-institution cohorts, with subjects undergoing variable treatment regimens, and different assessments of smell function, a systematic review and meta-analysis was performed by Kohli et al [8] to clarify the presence of OD in CRS and identify consistent risk factors. The authors included 47 studies in the analysis, and identified age, CT score, and presence of nasal polyps as the three most consistent risk factors for CRS-OD in the published literature. Similarly, olfactory response to CRS treatment is highly variable, with resolution of smell disturbance after standard therapies occurring in only approximately 40% of refractory CRS patients [9].

Olfactory dysfunction in CRS is potentially complex and the interplay among possible mechanisms is poorly understood. The simple model of conductive loss due to impaired odorant transport to the olfactory cleft in CRS may explain a portion of the disease for some patients. Additional contributions of local and systemic inflammation likely contribute to a varying degree in some individuals, with underlying factors such as age and presence of comorbid diabetes influencing the capacity for olfactory sensory neuron regeneration. Given the complex and multifactorial nature of CRS-OD, compounded by inherent challenges in human cohort studies, it is no surprise that putative predictors and their relative importance have been inconsistently identified across studies. Beyond the risk of sampling bias found in most single institution studies, existing publications attempt to identify predictor variables for CRS-OD generally through univariate regression models. But, these analyses cannot account for all possible predictor variables or interactions among variables, and therefore, may offer limited conclusions. Additionally, attempting multivariable regressions that were not developed for predictive performance or assessed as such, cannot be relied upon [10].

Traditional statistical approaches can model how system variables relate to one another and generate corresponding metrics of statistical significance. However, novel data analytics approaches including machine learning (ML) methods have shown improved classification accuracy and can be utilized to predict future outcomes, rather than attempting to interpret individual components within a system. ML approaches are particularly useful when there is a role for including complex interactions that may otherwise be ignored or dismissed as noise when using traditional statistics [11], such as in the case of CRS-OD.

Understanding the heterogeneity of CRS-OD and grasping multi-dimensional risk factors for classification prediction is foundational for future clinical care and research. The objective of this study was to test different ML models used for predictive analytics against traditional regression modeling for classification accuracy of OD in CRS patients. A secondary goal was to explore top predictors in high-performing models to identify common predictor variables of high importance that may help explain the heterogeneity of OD found among CRS patients.

## MATERIALS and METHODS

### Study Population

Adult patients (≥18 years old) were recruited into a prospective, multi-center, observational cohort study developed to evaluate treatment outcomes of endoscopic sinus surgery (ESS), for which multiple reports have been previously published [12-15]. All study patients were diagnosed with medically refractory CRS as defined by both the American Academy of Otolaryngology-Head and Neck Surgery 2007 guidelines and the European Position Paper on Rhinosinusitis and Nasal Polyps 2012 [3,4]. The Institutional Review Board (IRB) at each performance site governed all study protocols, informed consent documentation, and data safety monitoring. Performance sites consisted of tertiary care sinus surgery centers located within academic hospital systems in North America including: Oregon Health & Science University (OHSU; Portland, OR; eIRB#7198), Stanford University (Palo Alto, CA; IRB#4947), the Medical University of South Carolina (Charleston, SC; IRB#12409), and the University of Calgary (Calgary, Alberta, Canada; IRB#E-24208), with central study coordination at OHSU.

At original study enrollment, participants were screened for demographics, social and medical history including factors listed in **Table 1**. The diagnoses of asthma, allergy, and aspirin sensitivity were made by presence of classic symptoms and self-report of prior testing. Subjects with comorbid cystic fibrosis, ciliary dyskinesia, or autoimmune disease were not excluded from final analysis. Participants were followed postoperatively through the standard of care for up to 18-months and asked to complete postoperative, study evaluations at regular 6-month intervals, either during physician-directed appointments or follow-up study mailings utilizing the postal service and self-addressed return envelopes.

View this table:
[Table 1.](http://medrxiv.org/content/early/2020/10/14/2020.10.12.20210500/T1)

Table 1. Study population.
Collected demographics and clinical metadata used as inputs for classification models are shown here for the overall cohort, and those with and without olfactory dysfunction. Data sorted from smallest to largest q-value (multiple testing adjusted p-value).

### Objective Measures of Disease Severity

Diagnostic measures of disease severity were collected as standard of care, and were also used for investigational purposes. These well-established objective measures have been recommended for clinical study [16], and include Lund-Kennedy nasal endoscopy scoring [17] and high-resolution computed tomography (CT) radiographic imaging graded using the Lund-Mackay scoring system [18]. Olfactory function was using the brief Smell Identification Test (bSIT) in the years 2011-2013, and the Smell Identification Tests (SIT) in the years 2013-2015 (Sensonics Inc., Haddon Heights, NJ). Higher SIT total scores reflect a better sense of smell, and categorical ratings of “smell loss” (anosmia + hyposmia) and “normosmia” were assigned based on established scales [19-21].

### Patient Reported Outcome Measures

Other patient reported outcome measures (PROMs) were documented but were not used in the predictive analytics given the potential overlap between particular symptom ratings and objective smell loss. Study PROMs included the 22-item SinoNasal Outcome Test (SNOT-22; ©2006, Washington University, St. Louis, MO) [22,23], the Rhinosinusitis Disability Index (RSDI) [24], and the questionnaire on olfactory dysfunction (QOD-NS) [25].

### Data Management and Statistical Analyses

Protected health information was previously removed and study data was safeguarded using unique study identification number assignment for each participant. Study data was securely transferred from a HIPAA compliant, relational database (Access, Microsoft Corp, Redmond, WA.) to the University of Colorado Department of Otolaryngology-Head and Neck Surgery password protected research server, according to specifications of a Data Use Agreement between OHSU and the University of Colorado.

Study data were evaluated descriptively while continuous and ordinal variables were evaluated for assumptions of normality. Comparing variables between normosmics and those with smell loss, Fisher’s exact test was used for categorical variables and Welch’s two sample t-test was used for continuous variables. Statistical comparisons were reported using a type-I error probability at the 0.050 level of significance. All statistical and data analyses were completed using the R software program [26].

### Machine Learning Approaches to Classification

Several machine learning predictive analytics models were applied, in order to compare classification accuracy to tradition *logistic regression*, which was performed here with backwards step-wise AIC variable selection. *Random forests* was chosen as it allows for non-linear and interaction effects; conditional inference trees were used with permutation-based variable importance scores for unbiased variable selection [27]. Similar to traditional logistic regression, least absolute shrinkage and selection operator (*LASSO*), assumes only linear and additive affects (without interaction) where regression coefficients are shrunk toward zero to perform variable selection. In this model, unimportant variables are given coefficients equal to zero, effectively removing them from the model. Along these lines, multivariate adaptive regression splines (*MARS*) was also applied in a similar fashion to traditional logistic regression, but it uses stepwise methods for variable selection and allows for nonlinear and interaction effects. Lastly, the *Support Vector Machine* approach was applied with a radial basis kernel.

To evaluate classification accuracy, repeated 10-fold cross validation (5 repeats) was used to tune and evaluate each model area. To deal with the fact that the two classes are unbalanced (smell loss vs normosmia), down sampling was used within the cross-validation procedure, as implemented in the caret R package. Up sampling was also applied, but produced equal classification accuracy in terms of AUC. The method of “surrogate splits” [28] was used to handle missing values in Random Forest. The missForest R package [29] was used to impute missing values for all other models.

To identify the important predictor variables included in each classification algorithm, permutation-based variable importance scores were used to rank variables from most to least important in Random Forests, while the absolute value of the standardized regression coefficient was used for LASSO variable rankinds. The IML R package [30] was used to calculate variable importance scores for SVM using a permutation method. All variable importance scores were scaled from 0 to 100, where the most important variable for the model has a score of 100. Of note, for a given categorical variable, RF and LASSO calculate separate variable importance scores for each category, whereas the approach used for SVM only calculates an overall score.

## RESULTS

### Final Cohort Characteristics

A total of 611 study participants met all inclusion criteria and were prospectively enrolled between April 2011 and July 2015. Demographics and comorbid disease characteristics, examined by olfactory classification, are described in **Table 1**. The overall mean age for the final cohort was 51.3 years (38.3, 61.7) years with slight female preponderance (53% female). Approximately one-third of the cohort exhibited nasal polyps (35%).

Study participants were stratified by SIT category. Statistically significant differences were observed between the normosmic and smell loss cohorts in terms of objective disease measures (CT and endoscopy scores), age, sex, previous surgery, socioeconomic status, steroid use, asthma/COPD, and aspirin sensitivity. Local conditions including nasal polyps, allergic fungal rhinosinusitis, and deviated septum were also significant factors.

### Smell loss vs Normosmia Classification Accuracy

A traditional logistic regression model for classification of baseline olfactory dysfunction demonstrated on these criteria demonstrated acceptable accuracy in the context of CRS-OD (AUC 0.707). Three of the four machine learning models outperformed the traditional statistical approach (**Table 2**). The SVM approach was the most accurate, with a relatively good classification accuracy (AUC 0.754) and much higher sensitivity than other the methods. LASSO and Random Forest also outperformed logistic regression, whereas MARS performed worse than all of the approaches.

View this table:
[Table 2.](http://medrxiv.org/content/early/2020/10/14/2020.10.12.20210500/T2)

Table 2. Comparing classification accuracy.
Three ML approaches outperform traditional logistic regression. *AUC = area under receiver operating characteric curve; SVM-radial = support vector machine with a radial basis kernel; Log Reg-Step = stepwise logistic regression*.

### Machine Learning Predictions: Top variables

ML algorithms are adept are recognizing patterns within complex data, but the lack of *a priori* framework results in the inability to understand *why* the algorithm made its decisions. As we see in the variable importance graphs (**Figs 1, 2, Suppl Fig 1**), the ML models each arrived at somewhat different relative importances of a given predictor variable. SVM, the top predictor model, included the most number of variables in its algorithm with nearly all inputs having significant importance (**Fig 1**). Relative importance of predictor variables in Random Forests and LASSO are shown in **Fig 2** (**and Suppl Fig 1**) and in these models, fewer variables were important and the most weight was placed on the presence of nasal polyps and the nasal endoscopy disease severity score.

![Supplementary Figure 1.](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2020/10/14/2020.10.12.20210500/F3.medium.gif)

[Supplementary Figure 1.](http://medrxiv.org/content/early/2020/10/14/2020.10.12.20210500/F3)

Supplementary Figure 1. Complete list of predictors for Random Forest and LASSO models.
Notably, the LASSO model incorporates significantly more features of interest than the Random Forest algorithm (∼25 features with >10% importance in the prediction algorithm).

![Figure 1.](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2020/10/14/2020.10.12.20210500/F1.medium.gif)

[Figure 1.](http://medrxiv.org/content/early/2020/10/14/2020.10.12.20210500/F1)

Figure 1. Variable importance display for most accurate classification model (Support Vector Machine with a radial basis kernel).
Note the inclusion of many predictor variables – 32 are included with > 10% variable importance, suggesting significant interaction between predictors.

![Figure 2.](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2020/10/14/2020.10.12.20210500/F2.medium.gif)

[Figure 2.](http://medrxiv.org/content/early/2020/10/14/2020.10.12.20210500/F2)

Figure 2. Top 10 Predictors and Variable importance for Random Forest and LASSO models.
*AERD = aspirin exacerbated respiratory disease; CF = cystic fibrosis*.

Top predictors across the approaches contain many factors associated with olfactory function in previous studies, including steroid use, prior surgery, nasal polyps, aspirin exacerbated respiratory disease, age, sex, and smoking history. Inflammatory disease severity, as measured by CT and nasal endoscopy scores, are not surprisingly very important predictors of CRS-OD.

Interestingly, socioeconomic factors such as insurance status, household income, race, were among top predictors across the three models displayed. Comorbid medical conditions including asthma, diabetes, septal deviation, alcohol use, and sleep apnea, are also of moderate importance. Notably, these factors can be influenced by unrelated medical care, suggesting that other approaches to improve general health and well-being may be considered as part of the holistic approach to CRS-OD management. Although the presence of nasal polyps, asthma, and AERD, are significantly associated with CRS-OD here and in prior literature, the presence of allergic rhinitis by history or by skin testing appeared to carry relatively minimal weight in the CRS-OD classifications.

## DISCUSSION

CRS is a heterogeneous process in terms of presentation, subtypes, natural history and responses to treatment [2]. In terms of CRS-OD, current data from a multi-institutional cohort studying olfactory loss in CRS with a sensitive olfactory test (Sniffin’ Sticks) demonstrates that nearly 70% of the CRS population has significant olfactory loss, with 20% exhibiting complete anosmia [7]. The pathophysiology of CRS-OD is ascribed to both sensorineural and conductive effects of local tissue inflammation, on top of the myriad factors that can affect olfaction in the general population [1,31]. In previous study, factors associated with worse baseline olfactory function in CRS patients included nasal polyposis, asthma, age, smoking, and eosinophilia [32]. Other works have shown CRS-OD response treatment in ∼40% of patients, with risk factors such as olfactory dysfunction severity, nasal polyps, female gender, high socio-economic status and non-smoking associating with better quality of life results. In multivariate regression, only nasal polyps and degree of baseline olfactory dysfunction maintained statistical significance, highlighting the importance of sample size and analytical methods to glean useful information from a prospectively collected human dataset [10].

Here, we utilized ML-based data science approaches to interrogate a large multicenter CRS cohort to test whether novel predictive analytics approaches could outperform the gold-standard traditional statistical method of logistic regression. ML is defined as the use of algorithms to break down data, learn from it, and then make a determination or prediction about some aspect. In doing so, ML provides a novel approach to uncover previously unrecognized patterns among CRS-OD patients and thereby offers numerous advantages over regression analyses, which have been traditionally employed in studies of CRS disease severity and outcomes [33]. Whereas traditional statistical methods focus on modeling how system variables relate to one another and what statistical inference (e.g. significance in p-values) can be made, the goal of machine learning (ML) is not interpretation of individual components but prediction of future outcomesTop predictorsIn this study, different supervised ML algorithms were used map features of interest to the outcome or “label” of olfactory loss. We observed improved prediction accuracy with most ML approaches, measured by AUC and sensitivity/specificity. This advantage may be attributed to the ability of many ML models to handle high-dimensional data in which more features exist than observations, whereas traditional statistical modeling with a large number of potential features requires some form of dimension reduction or variable selection. As a result, we believe that machine learning and artificial intelligence data analytics are well-suited to prime research and eventually for precision medicine in olfactory disorders, given the idiosyncrasies, nuances, and numerous possible predictor variables.

There are some limitations of the current study that should be addressed, the first is the potential for sampling bias. Two major biases in healthcare ML are gender and race [34]. Details of this cohort have been extensively published and include even distributions by age and sex. Racial minorities and low socioeconomic status groups are underrepresented as a function of the geography and practice referral patterns, the general lack of accessible and affordable healthcare and racial disparities in the U.S. healthcare system [35]. A related concern is that models created from our cohort data may “overfit” if new or unseen clinical data are applied. Certainly, these approaches should be further validated but nonetheless demonstrate the important utility of novel data science approaches to uncover unrecognized patterns and predictors from “noisy” human data in a noisy disease process. There are likely potentially unrecognized but important data inputs (i.e., molecular endotyping data) that are not yet clinically available. Serum markers, such as peripheral eosinophilia and serum IgE, have been proposed for CRS in the past but have generally been disappointing and do not appear to correlate with tissue levels [36,37] Some our clinical data input fields may associate phenotypes with likely molecular endotypes, but such overlap is not universal [5]. It is an exciting direction for the near future to include molecular based biomarkers to the input data, and test additional accuracy gained.

## CONCLUSION

Here we demonstrate the importance of novel predictive analytics approaches in study of clinical olfactory disorders. Although founded in data science principles, and capable of recognizing patterns within noisy data, it is challenging to understand *why* the models make predictions with comparable accuracy despite use of some different predictor variable. Nonetheless, consistently observed predictors include some weighted interest in SES and other potentially modifiable conditions such as asthma/AERD. Such approaches may have value for both clinical counseling and guidance for future basic science and translational research.

## Data Availability

No datasets were generated during the current study. Due to the nature of the primary research study, participants did not agree for their data to be shared publicly, so these supporting data are not available.

## ACKNOWLEDGMENTS

The authors would like to thank Drs. Miranda Kroehl, PhD, John Rice, PhD, and Krithika Suresh, PhD, for biostatistical expertise in machine learning and prediction models, as well as data management considerations. We would also like to thank Drs. Peter Hwang, MD, and Luke Rudmik, MD, for their contributions in original study enrollment and ongoing collaboration.

## Footnotes

*   Financial Disclosures: V.R.R., J.C.M., Z.M.S., and T.L.S. are supported by grants for this investigation from the National Institute on Deafness and Other Communication Disorders (NIDCD), one of the National Institutes of Health, Bethesda, MD., USA [R01 DC005805 (T.L.S.) and K23 DC014747 (V.R.R.)]. Public clinical trial registration ([www.clinicaltrials.gov](http://www.clinicaltrials.gov)) ID# [NCT01332136](http://medrxiv.org/lookup/external-ref?link_type=CLINTRIALGOV&access_num=NCT01332136&atom=%2Fmedrxiv%2Fearly%2F2020%2F10%2F14%2F2020.10.12.20210500.atom). JA is supported by NIH/NCATS Colorado CTSA Grant Number UL1 TR002535. Contents are the authors’ sole responsibility and do not necessarily represent official NIH views.

*   Conflicts of Interest: None related to this study

*   Received October 12, 2020.
*   Revision received October 12, 2020.
*   Accepted October 14, 2020.


*   © 2020, Posted by Cold Spring Harbor Laboratory

This pre-print is available under a Creative Commons License (Attribution-NonCommercial-NoDerivs 4.0 International), CC BY-NC-ND 4.0, as described at [http://creativecommons.org/licenses/by-nc-nd/4.0/](http://creativecommons.org/licenses/by-nc-nd/4.0/)

## REFERENCES

1.  1.Hummel T, Whitcroft KL, Andrews P et al. Position Paper on Olfactory Dysfunction. Rhinology 2016 31;56(1):1–30.
    
    

2.  2.Orlandi RR, Kingdom TT, Hwang PH, et al. International Consensus Statement on Allergy and Rhinology: Rhinosinusitis. Int Forum Allergy Rhinol. 2016;6 Suppl 1:S22-209. Section X.B.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1002/alr.21695&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=26889651&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2020%2F10%2F14%2F2020.10.12.20210500.atom) 

3.  3.Fokkens WJ, Lund VJ, Mullol J, et al. European position paper on rhinosinusitis and nasal polyps 2012. Rhinol. Suppl 2012;23: 1–298.
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=22764607&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2020%2F10%2F14%2F2020.10.12.20210500.atom) 

4.  4.Rosenfeld RM, Piccirillo JF, Chandrasekhar SS, et al. Clinical practice guideline (update): adult sinusitis. Otolaryngol Head Neck Surg. 2015;152(2 Suppl):S1–39.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1177/0194599814561600&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=25644617&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2020%2F10%2F14%2F2020.10.12.20210500.atom) 

5.  5.Akdis CA, Bachert C, Cingi C, et al. Endotypes and phenotypes of chronic rhinosinusitis: a PRACTALL document of the European Academy of Allergy and Clinical Immunology and the American Academy of Allergy, Asthma & Immunology. J Allergy Clin Immunol 2013;131(6):1479–90.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1016/j.jaci.2013.02.036&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=23587334&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2020%2F10%2F14%2F2020.10.12.20210500.atom) 
    
    [Web of Science](http://medrxiv.org/lookup/external-ref?access_num=000320532800003&link_type=ISI) 

6.  6.Soler ZM, Kohli P, Storck KA, Schlosser RJ. Olfactory Impairment in Chronic Rhinosinusitis Using Threshold, Discrimination, and Identification Scores. Chem Senses 2016;41(9):713–719.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1093/chemse/bjw080&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=27469973&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2020%2F10%2F14%2F2020.10.12.20210500.atom) 

7.  7.Schlosser RJ, Smith TL, Mace JC, et al. Factors driving olfactory loss in patients with chronic rhinosinusitis: a case control study. Int Forum Allergy Rhinol 2020;10(1):7–14.
    
    

8.  8.Kohli P, Naik AN, Harruff EE, et al. The prevalence of olfactory dysfunction in chronic rhinosinusitis. Laryngoscope 2017; 127(2):309–320.
    
    

9.  9.DeConde AS, Mace JC, Alt JA, Schlosser RJ, Smith TL, Soler ZM. Comparative effectiveness of medical and surgical therapy on olfaction in chronic rhinosinusitis: a prospective, multi-institutional study. Int Forum Allergy Rhinol. 2014;4(9):725–33.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1002/alr.21350&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=25044658&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2020%2F10%2F14%2F2020.10.12.20210500.atom) 

10. 10.Katotomichelakis M, Simopolous E, Tripsianis G, et al. Predictors of quality of life outcomes in chronic rhinosinusitis after sinus surgery. Eur Arch Otorhinolaryngol 2014; 271(4):733–41.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1007/s00405-013-2626-6&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=23842603&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2020%2F10%2F14%2F2020.10.12.20210500.atom) 

11. 11.Bzdok D, Altman N, Krzywinski M. Statistics versus machine learning. Nat Methods 2018;15(4):233–234.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1038/nmeth.4642&link_type=DOI) 

12. 12.Smith TL, Kern R, Palmer JN, et al. Medical therapy vs surgery for chronic rhinosinusitis: a prospective, multi-institutional study with 1-year follow-up. Int Forum Allergy Rhinol 2013;3(1):4–9.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1002/alr.21065&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=22736422&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2020%2F10%2F14%2F2020.10.12.20210500.atom) 

13. 13.DeConde AS, Mace JC, Alt JA, et al. Longitudinal improvement and stability of the SNOT-22 survey in the evaluation of surgical management for chronic rhinosinusitis. Int Forum Allergy Rhinol. 2015; 5(3):233–239.
    
    

14. 14.Soler ZM, Smith TL, Alt JA, et al. Olfactory-specific quality of life outcomes after endoscopic sinus surgery. Int Forum Allergy Rhinol. 2016; 6(4): 407–413.
    
    

15. 15.Ramakrishnan VR, Mace JC, Soler ZM, Smith TL. Examination of high-antibiotic users in a multi-institutional cohort of chronic rhinosinusitis patients. Int Forum Allergy Rhinol. 2017;7(4):343–51.
    
    

16. 16.Hopkins C, Hettige R, Soni-Jaiswal A, et al. Chronic rhinosinusitis outcome measures (CHROME), developing a core outcome set for trials of interventions in chronic rhinosinusitis. Rhinology 2018;56(1):22–32.
    
    

17. 17.Lund VJ and Kennedy DW. Quantification for staging sinusitis. The Staging and Therapy Group. Ann Otol Rhinol Laryngol Suppl. 1995;167:17–21.
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=7574265&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2020%2F10%2F14%2F2020.10.12.20210500.atom) 

18. 18.Lund VJ and Mackay IS. Staging in rhinosinusitis. Rhinology. 1992;31(4):183–4.
    
    

19. 19.Doty RL. The Smell Identification Test Administration Manual. 3. Haddon Heights, NJ: Sensonics, Inc; 1995.
    
    

20. 20.Doty RL. The Brief Smell Identification Test Administration Manual. Haddon Heights, NJ: Sensonics, Inc; 2001.
    
    

21. 21.El Rassi E, Mace JC, Steele TO, et al. Sensitivity analysis and diagnostic accuracy of the Brief Smell Identification Test in patients with rhinosinusitis. Int Forum Allergy Rhinol. 2016;6(3):287–92.
    
    

22. 22.Piccirillo JF, Merritt MG, Jr, Richards ML. Psychometric and clinimetric validity of the 20-Item Sino-Nasal Outcome Test (SNOT-20) Otolaryngol Head Neck Surg. 2002;126(1):41–47.
    
    

23. 23.Hopkins C, Gillett S, Slack R, et al. Psychometric validity of the 22-item Sinonasal Outcome Test. Clin Otolaryngol. 2009;34(5):447–454.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1111/j.1749-4486.2009.01995.x&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=19793277&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2020%2F10%2F14%2F2020.10.12.20210500.atom) 
    
    [Web of Science](http://medrxiv.org/lookup/external-ref?access_num=000270176700004&link_type=ISI) 

24. 24.Senior BA, Glaze C, Benninger MS. Use of the rhinosinusitis disability index (RSDI) in rhinologic disease. Am J Rhinol. 2001; 15(1):15–20
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.2500/105065801781329428&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=11258649&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2020%2F10%2F14%2F2020.10.12.20210500.atom) 

25. 25.Simopoulos E, Katotomichelakis M, Gouveris H, Tripsianis G, Livaditis M, Danielides V. Olfaction-associated quality of life in chronic rhinosinusitis: adaptation and validation of an olfaction-specific questionnaire. Laryngoscope 2012;122:1450–1454.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1002/lary.23349&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=22566102&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2020%2F10%2F14%2F2020.10.12.20210500.atom) 
    
    [Web of Science](http://medrxiv.org/lookup/external-ref?access_num=000305577400006&link_type=ISI) 

26. 26.R Core Team. 2019. R: A Language and Environment for Statistical Computing. Vienna, Austria: R Foundation for Statistical Computing.
    
    

27. 27.Strobl C, Boulesteix AL, Zeileis A, Hothorn T. Bias in random forest variable importance measures: Illustrations, sources and a solution. BMC bioinformatics 2007;8(1):25.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1186/1471-2105-8-25&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=17254353&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2020%2F10%2F14%2F2020.10.12.20210500.atom) 

28. 28.Strobl C, Malley J, Tutz G. An introduction to recursive partitioning: Rationale, application, and characteristics of classification and regression trees, bagging, and random forests. Psychological methods 2009; 14:323.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1037/a0016973&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=19968396&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2020%2F10%2F14%2F2020.10.12.20210500.atom) 
    
    [Web of Science](http://medrxiv.org/lookup/external-ref?access_num=000272570600002&link_type=ISI) 

29. 29.Stekhoven DJ and Bühlmann, P. MissForest—non-parametric missing value imputation for mixed-type data. Bioinformatics 2012;28(1):112–118.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1093/bioinformatics/btr597&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=22039212&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2020%2F10%2F14%2F2020.10.12.20210500.atom) 
    
    [Web of Science](http://medrxiv.org/lookup/external-ref?access_num=000298383800017&link_type=ISI) 

30. 30.Molnar C, Casalicchio G, Bischl B. iml: An R package for interpretable machine learning. J Open Source Software 2018;3(26), 786.
    
    

31. 31.Rombaux P, Huart C, Levie P, Cingi C, Hummel T. Olfaction in Chronic Rhinosinusitis. Curr Allergy Asthma Rep 2016;16(5):41.
    
    

32. 32.Litvack JR, Fong K, Mace J, et al. Predictors of olfactory dysfunction in patients with chronic rhinosinusitis. Laryngoscope 2008;118:2225–30.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1097/MLG.0b013e318184e216&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=19029858&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2020%2F10%2F14%2F2020.10.12.20210500.atom) 

33. 33.Michie D, Spiegelhalter D, Taylor C. Machine Learning, Neural, and Statistical Classification. New York, NY: Ellis Horwood; 1994.
    
    

34. 34.Rajkomar A, Dean J, Kohane I. Machine learning in medicine. N Engl J Med 2019;380(14):1347–1358.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1056/NEJMra1814259&link_type=DOI) 

35. 35.2018 National Healthcare Quality and Disparities Report. Content last reviewed April 2020. Agency for Healthcare Research and Quality, Rockville, MD. [https://www.ahrq.gov/research/findings/nhqrdr/nhqdr18/index.html](https://www.ahrq.gov/research/findings/nhqrdr/nhqdr18/index.html)
    
    

36. 36.Gitomer SA, Fountain CR, Kingdom TT, et al. Clinical Examination of Tissue Eosinophilia in Patients with Chronic Rhinosinusitis and Nasal Polyposis. Otolaryngol Head Neck Surg 2016;155(1):173–8.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1177/0194599816637856&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=26980909&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2020%2F10%2F14%2F2020.10.12.20210500.atom) 

37. 37.Settipane GA. Nasal polyps and immunoglobulin E (IgE). Allergy Asthma Proc 1996;17(5):269–73.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.2500/108854196778662237&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=8922146&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2020%2F10%2F14%2F2020.10.12.20210500.atom) 
    
    [Web of Science](http://medrxiv.org/lookup/external-ref?access_num=A1996VU27100006&link_type=ISI)