EPICOVID19: Psychometric assessment and validation of a short diagnostic scale for a rapid Covid-19 screening based on reported symptoms

Introduction

SARS Corona Virus 2 has led to a global pandemic (on July 28 reports over 16 million cases and 650805 deaths across more than 200 countries, WHO, Johns Hopkins Center for Health Security) ^1,2. Besides this immediate human toll there are readily acknowledged and potentially long-lasting effects on global economies, politics, health and privacy policies at many levels that will extend beyond the development of vaccines and treatments. The rapid spread of the disease, COVID-19, and its seemingly high degree of variability in its presentation among individuals has led to a level of clinical and scientific focus not previously seen and encompassing both traditionally reviewed and pre-print publications and resources. Collaborative groups are being formed at the local, regional, national and international levels to address both patient data collection/aggregation and analysis in ways that may change the way research is carried out in the future³. To enable these efforts to be both effective and productive is the need for the data to be evaluated as to its suitability for inclusion in these activities while still recognizing that what we understand about COVID-19 is much less than what we do not understand⁴.

Because of the far-reaching scope of the pandemic, we are already confronting:

1. Need to implement individual testing at a level far above current capacities to optimize individual treatment, assess disease spread, anticipate potential strains on healthcare resources and personnel⁵.

2. Need for improvements in available testing, both nasopharyngeal swabs (NPS) and antibody detection, i.e. accuracy, specificity and sensitivity to enable reliable evaluation and interpretation of data for use in clinical care and policy decisions⁶.

3. Need to harmonize clinical observations and definitions to support development of guidelines, prognostic and diagnostic indicators and to develop a comprehensive understanding of the disease and critical factors that differentiate patient susceptibility, presentation of the disease and response to treatment^7,8.

The use of online surveys can greatly enhance access to broader populations in a cost-effective manner and optimize both screening for individuals who may need immediate care as well as provide an approach to 3) above. A cross-sectional national survey, EPICOVID19, was launched on April 13, 2020 and received more than 200000 responses⁹. The survey, which represents the Phase I of the study, was promoted using social media (Facebook, Twitter, Instagram, Whatsapp), press releases, internet pages, local radio and TV stations, and institutional websites that called upon volunteers to contact the study website. The inclusion criteria were: age of >18 years; access to a mobile phone, computer, or tablet with internet connectivity; and on-line consent to participate in the study.

The aim of our study is to assess the capability of the self-reported symptoms collected through the EPICOVID19 questionnaire to discriminate the COVID-19 among symptomatic subjects, in order to identify possible COVID-19 cases to undergo instrumental measurements and clinical examinations (Phase II of the study). The final objectives are to propose a method for the development of a total score for the self-reported symptoms in the EPICOVID19 questionnaire and to validate the scoring method with reference to the molecular and serological clinical diagnosis.

Methods

Study design and participants

This current study is the Phase II of the EPICOVID19 national survey⁹. The Figure 1 shows the overview of EPICOVID19 two-phase study. The Phase I questionnaire investigated six areas through 38 questions (1: Socio-demographic characteristics; 2: Clinical evaluation; 3: Personal characteristics and health status; 4: Housing conditions; 5: Lifestyle; 6: Behaviours after the lockdown).

• Download figure
• Open in new tab

Figure 1. Overview of EPICOVID19 two-phase study

The Phase II questionnaire was mailed to all subjects who underwent testing for COVID-19 by NPS, and who volunteered to be involved in the follow up study in their Phase I response. Phase II focused on results reported for NPS and/or serological IgG/IgM tests and self-reported symptoms with the aim to better identify both symptomatic and asymptomatic cases of SARS-CoV-2 infections¹⁰.

Phase II was implemented using an open source statistical survey framework PHP based, LimeSurvey (vers. 3.17), distributed under the GNU General Public License. In Phase II, responses to 11 questions were required that covered administration of NPS and/or serological tests and on the time elapsed between observed/reported symptoms and clinical examination (NPS and/or antibodies IgG and IgM) (appendix p 1-8). Of the 6864 subjects who underwent testing for COVID-19 by NPS in the phase I, 4094 subjects were invited by e-mail to complete the Phase II questionnaires online. Of these, 38 email invitations were not delivered, e.g. wrong address, mailbox full, host or domain name not found, etc.; 101 individuals did not participate by refusing to provide consent; and 1252 individuals who received the email did not proceed to complete the questionnaire. A total of 2703 subjects (66% response rate) completed the phase II survey. We compared the characteristics of respondents (2703) and non-respondents (4161). Respondents and non-respondents to the Phase II survey appeared similar with respect to gender, age and the perception of their own health as well as their self-reported comorbidities; the details comparing these two groups of subjects are included in appendix p 9. The resulting data of 2703 subjects who completed the Phase II were linked to the results of the self-reported symptoms of Phase I EPICOVID19 questionnaire, which included questions on the presence of 11 symptoms.

Results

The characteristics of the 2703 subjects, as supplied by those who completed the phase II survey, their NPS, IgG and IgM results are shown in table 1. The sample was predominantly women (68.1%) with the average age being 49 ± 15.0 and 52 ± 14.1 years for women and male individuals, respectively.

In the total sample of 2703 individuals, 694 (25.7%) were NPS(+) and of these 84 (12.1% of the 694 NPS(+)’s) were asymptomatic.

Testing statistics: for the subgroup of individuals who performed serological testing, of the 472 who did IgG(+) and 421 who did IgM(+), 22.9% and 11.6% tested positive, respectively. Among IgG(+) 1 of 108 subjects was asymptomatic (0.9%) while 5/49 subjects among IgM(+) were asymptomatic (10.2%). For subjects with NPS(+), the average number of days between initial symptoms and the day of the swab execution was 9.3 ± 9.4 (median 7 days, IQR 3-7). For subjects IgG(+) the average number of days between initial symptoms and the day of the serological test execution was 36.1 ± 15.1 (median 36.5 days, IQR 28-47). For subjects IgM(+) the average number of days from initial symptoms to the day of the serological test execution was 26.1 ± 17.9 (median 28.0 days, IQR 4-40). The frequency of the eleven symptoms reported by the three groups (NPS, IgG, IgM tested) was similar among men and women. In NPS(+) group, women reported higher percentages than male for the Sore throat and/or cold and/or Tachycardia (Palpitations) symptoms only. In the IgG(+), males reported higher frequencies for headaches only, while in the IgM(+) group the females showed the lower frequency of symptoms related to Conjunctivitis.

The frequency of symptoms among NPS(+) subjects (table 1) ranged from a low rate of observation, e.g. Tachycardia (palpitations) (S9: 17.3%) and Conjunctivitis (S11: 16.0%), to a high observation rate, e.g. Fever (S4: 60.7%) and OT-D (S6: 52.6%). For all symptoms, apart from Headache, the frequencies were significantly higher in NPS(+) subjects than in NPS(-) ones (p<0.001). For the subgroup of individuals who also performed serological tests, the symptoms with higher percentages among the subjects tested positive were Fever (IgG(+) 60.2%; IgM(+) 57.1%) and Pain in muscles, bones, joints IgG(+) 67.6%; IgM(+) 55.1%). In the IgG serological group, no difference was observed in the percentages of Sore throat and/or cold (S3) symptoms, while for Respiratory difficulty (S7), Chest pain (S8) and Gastrointestinal symptoms (S10), the percentages in the IgM group were the same.

The EFA performed by Principal-Component Factors and Horn’s PA methods statistics pointed out one factor. Eigenvalues, descriptive indices, and goodness-of-fit indices of cumulative percentages of explained data variability obtained through EFA are displayed in table 2. Principal-Component Factors highlight only one factor with 89.9% of proportion of explained variability, while Horn’s PA method identifies two factors with eigenvalues greater than 1.0, with 49.8% and 10.3% of proportion of explained variability, respectively.

Based on a priori determined cut-off value the factor-loading greater than 0.35 was maintained. The factors loading rule of one factor solution extracted by Principal-Component Factors is available in appendix p 13. Then, the dimensionality indices of the one-factor solution, as the high variance explained by the factor (89.9%), confirms the presence of one latent variable underlying COVID-19 symptoms items. Therefore, we define the one-factor solution as “EPICOVID19 diagnostic scale” (EPICOVID19 DS). On the basis of the CFA, results confirmed the latent construct as uni-dimensional and how the variables contributed to EPICOVID19 DS. The figure 2 shows the values of the standardized factor loadings for the one factor model. The magnitude of each factor loading higher than 0.4 indicates the importance of the corresponding item to EPICOVID19 DS. For example, Pain in muscles, bones, joints were the most important variable with a value equal to 0.814. The other variables with optimal specific validity index were Respiratory difficulty (sense of breathlessness at rest (0.688), Loss of taste and/or smell (0.724) and Gastrointestinal complaints with item-factor correlations equal 0.737. The lowest values were observed for the Sore throat and/or cold and Conjunctivitis items, 0.537 and 0.557, respectively. The goodness of fit (SMSR, RMSEA) of the EPICOVID19 DS was acceptable because two indexes were lower than 0.10 (SMSR 0.072; RMSEA 0.052; CFI 0.977; TLI 0.971). Finally, we compute CFA indexes to measure internal validity of the model (appendix p 14).

• Download figure
• Open in new tab

Figure 2.

Standardized factor loadings and goodness of fit indexes for one factor model. The goodness of fit indexes (SMSR, RMSEA) of the “EPICOVID19 DS”

Given the successful uni-dimensionality testing of EPICOVID19 DS, optimal scaling via Homogeneity Analysis was undertaken. The optimal score proposed was extracted from the HOMALS procedure (single-factor measurement) and for each subject, the computed optimal score was obtained summing the category quantifications of the screening questionnaire item responses. Cronbach (0.88) and Greenacre (78%) indices confirm the uni-dimensionality found through EFA and CFA. The HOMALS optimal category quantifications of the EPICOVID19 symptoms variable are summarized in table 3. Columns show the binary options (YES-NO), while rows show the different symptoms. The HOMALS category quantifications were scaled so that the score obtained from the sum of responses, varies from 0 (if a subject answered NO to all the symptoms) to 10 (if a subject answered YES to all the symptoms). These values are shown in the last column of table 3. An example of the resulting score calculation is reported as follows. If the subject response pattern with respect to symptoms is: YES, NO, YES, NO, NO, YES, YES, NO, NO, NO, YES, the re-coded response pattern is: 0.80, 0, 0.64, 0, 0, 0.97, 1.03, 0, 0, 0, 0.88 and subject optimal score is: 0.8 + 0+ 0.64+0+ 0+ 0.97+ 1.03+ 0+ 0+ 0+ 0.88 = 4.2.

There was no significant difference in the EPICOVID19 DS mean score between men (2.34±2.2) and women (2.49±2.4), while a low negative correlation between the score and the age of the participants was found (Spearman rho: −0.126; <0.001). The screening properties of the EPICOVID-19 DS compared to COVID-19 positive molecular and serological diagnosis are shown in table 4. The best value of Youden Index was observed for EPICOVID19 DS with respect to COVID-19 diagnosed by NPS(+). A good trade-off between sensitivity and specificity was observed (Se 76.56; Sp 68.24 AUC 77.5; 95% CI: 75.6-79.4). The cut-off score obtained was 2.56. Sensitivity and specificity improved when EPICOVID-19 DS was compared to COVID-19 detected by Antibodies IgG(+) (Se 80.37; Sp 80.17 AUC 86.0; 95% CI: 82.3-89.5) and the cut-off obtained was the same as for NPS(+) (2.59). The positive and negative predictive value for the IgG serological test was better than that for the nasopharyngeal swab (PPV IgG 54.43%: NPV IgG 93.27%). Finally, we observed a poor performance of the IgM results and are not presented in the table.

Discussion

Our focus was on developing a tool composed of simple questions related to the COVID-19 symptomatology for the identification of the subjects who are more likely to have SARS-CoV-2 infection in the general population. We validated the EPICOVID19 DS in a sample of voluntary subjects with serological and molecular clinical diagnosis. The optimal score, computed in 2703 adults aged 18 to 84 years, discriminates among symptomatic individuals. Before constructing the score, we performed both exploratory and confirmatory factor analysis to determine the number of factors/dimensions underlying the questionnaire. The results of these analyses supported a one factor model and the uni-dimensionality of the EPICOVID19 questionnaire. The magnitude of all factor loading was satisfactory, showing the highest factor loading value for the Respiratory difficulty, Chest pain, Tachycardia (palpitations) and Loss of taste and/or smell, Gastrointestinal complaints items appeared to be the most essential features of the EPICOVID19 DS. The high value for Chest pain is also explained by the fact that several patients reported it, possibly because of a tracheal pain caused by pneumonia¹¹. Several clinical studies on hospitalised patients have shown that, at the onset of COVID-19, patients frequently show typical symptoms of viral pneumonia¹². Symptoms that are less common, but still reported by a substantial number of patients are nasal congestion, sore throat, Gastrointestinal complaints and olfactory and taste disorders (OT-D)^13–15. Subjects often reported Gastrointestinal complaints, not as isolated symptomatology of SARS-CoV-2 infection but as concurrent symptoms¹⁶. The lowest factor loading value was observed for the Sore throat and/or cold and Conjunctivitis. These lower values may be related to the fact that Conjunctivitis and cold were not the most frequent symptoms of COVID-19¹⁷. In line with other recent works^18,19, the features encountered showed various aspects of the COVID-19 diagnosis definition. Indeed, Cough, loss of Taste and/or Smell and Respiratory difficulty were among the most reported symptoms in previous researches and corresponded to the items that gained the most importance in our score ^11,15,20,21. However, the clinical presentation of COVID-19 disease is varied, and discrepancy may exist between symptoms and disease. A recent meta-analysis of symptoms including 50000 COVID-19 patients, found that fever and cough were the most common symptoms²² (89.1% and 72.2%) and a separate study of hospitalized subjects has suggested that respiratory distress has been reported in the most critical cases of COVID-19²³. With the aim of supporting medical decision making, predicted models were developed for detecting people in the general population at risk of being admitted to hospital and for diagnosis of COVID-19 in patients with symptoms, but the results presented in a recent systematic review describe a poor research performance and a high risk of bias²⁴. Using Homogeneity Analysis, we propose a scoring methodology for developing an improved scale. Therefore, we achieved a numerical weight value (optimal quantification) which represents the importance of the binary response categories (YES/NO) for each question of EPICOVID19 DS. As a result, the various binary items of the eleven questions of EPICOVID19 DS contributed with different weights to the overall score. This produced an improved scale, 0 to 10, reflecting the importance for each symptom. Thus, Respiratory problems and Chest pain were the most important symptoms, with score 1.03 and 1.07 respectively. The other symptoms showing an important contribution to the total score were Gastrointestinal complaint (0.95), Loss of Taste and/or Smell (0.97), Tachycardia (palpitations) (1.05). Subsequently, we computed sensitivity (Se) and specificity (Sp) of EPICOVID19 DS compared to COVID-19 positive serological and molecular diagnosis subjects. In NPS(+) subjects the cut-off score was 2.56 with sensitivity equal to 76.56% and specificity equal to 68.24%. In IgG(+) subjects the cut-off score was 2.59 and we obtained a substantial improvement of sensitivity, specificity, PPV and NPV with respect to NPS(+) subjects (Se 80.37; Sp 80.17; PPV 54.43%; NPV 93.27%). The COVID-19 sensitivity and specificity of serological and molecular diagnostic tests is not fully resolved but some studies suggest the sensitivity could be as low as 80%^25,26. This raises concerns for a high false negative rate, which could result in an increase in the infection spread in the community. There is no absolute answer on the sensitivity and specificity of COVID-19 diagnostic tests because, to determine their accuracy, they must be compared with a “gold standard” test that currently is not identified. Considering estimates of Se and Sp, Positive Predictive Value (PPV) and Negative Predictive Value (NPV) can be gained on the basis of the disease prevalence and the rate of illness in the population but, as known, concerning COVID-19 there is significant uncertainty about this prevalence²⁷. Statistically it is assumed that PPV varies widely, in a range between 30-50% in areas with low prevalence, as stated in a recent US research about COVID-19²⁸. Early recognition screening and rapid diagnosis are essential to prevent transmission and provide supportive care in a timely manner. Nevertheless, screening is distinguished from further, more detailed diagnostic test assessment. This is of particular relevance as resources for full testing remains as a limited resource and optimizing its use is critical. EPICOVID19 DS could be a preliminary assessment that attempts to detect subjects with symptoms potentially associated with COVID-19 among a wide population. EPICOVID19 DS does not enable a clinical interview to determine the complete symptomatic profile and needs but identifies those who may warrant further assessment. An advantage would be to use this screening in primary care settings, so that GPs can avoid people with suspected COVID-19 in primary care’s offices when possible²⁹, or as a first screening tool and then manage the patient remotely by telephone or video consultations³⁰. The EPICOVID19 DS could be applied to the general population. Once the scoring is assigned to each symptom, the EPICOVID19 DS could allow to set different cut-offs according to the subjects involved and to the gold standards used (NPS, serological tests, clinical evaluation by clinicians, etc.). It should be noted that since it is plausible to expect lower prevalence values in the general population than the 22.77% of the present series, the probability of NPV would increase beyond the current 93.27% and consequently the probability of progressing to COVID-19 for subjects who tested negative (1 - NPV) would be less than the current 6.7%. Although the identified symptoms are not specific for COVID-19, they have been found as valid references in a population setting because they are frequently reported by patients affected by COVID-19. Likely, in a non-pandemic scenario, these symptoms could be assessed with different weights because of their aspecificity, configuring the EPICOVID19 DS as a valid diagnostic support mainly in a pandemic situation. Moreover, the health authority is still unable to monitor through classic tests the spread of SARS-CoV-2 infection, and allowing the circulation of unsuspecting positive subjects could represent a risk for the spread of the infection. The validation of an instrument that can easily identify a suspected case, through a score attributed to each symptom related to COVID-19, can be of great importance in facilitating the containment of the epidemic. The proposed score seems worthy of validation in broader populations in order to confirm its clinimetric properties. In the event of a confirmatory answer, it might qualify as a useful means of selecting people amenable to serological and/or molecular diagnostic tests for COVID-19. The availability and offering of diagnostic tests for the SARS-CoV-2 coronavirus has proven to be one of the keys to the containment of the COVID-19 pandemic. The early identification of positive subjects at molecular and serological tests among people with specific symptoms or considered at risk is crucial to limit the spread of the infection. The tool we validated responds to the need to readily identify a suspect case, through a score attributed to each symptom related to COVID-19. Although the validation was satisfactory, the proposed score seems worthy of being further testing in larger populations in order to confirm its clinimetric properties, useful for selecting people susceptible to serological and / or molecular diagnostic tests for COVID-19.

Although this tool could be a public health prevention measure instrument, directing subjects to a self-assessment without trigger panic, alarmism and concern among the screened population, some limitations need to be outlined. First of all, the participation in the study was voluntary and not representative of the general population. This presents potential selection biases that must be taken into consideration. Data were collected in a convenient young-adult and highly educated population sample characterized by low multimorbidity, as resulting from the phase I study⁹ and expectable in the case of an on-line questionnaire promoted by e-mail invitations. Further, in the context of a pandemic, our survey might have interested people who had no opportunity to report symptoms to clinicians. Moreover, the effect linked to recall bias cannot be excluded among the participants who tested positive at COVID-19 and/or presenting symptoms related to the SARS-COV2 infection. Given these limitations, the adoption of EPICOVID19 DS should be considered with caution and the procedures outlined for its development could be applied iteratively as new data is collected to continue refinement of this potentially valuable clinical decision support tool.

Contributors

SM, LB, FA and FP were responsible for the study concept and design. LB, LF and SP were responsible for the literature search. FA, LF and SP were responsible for acquisition of data. LB, FA, FB, FP and SM were responsible for analysis and interpretation of data. LB, LF, SM, SP and FB were responsible for drafting of the manuscript. MG, AG, RAI, CP, MA and GP were responsible for critical revision of the manuscript for important intellectual content. LB, SM were responsible for statistical analysis. GP, SM, CT, MN, NJ, AS, CP, MA critically revised the manuscript for important intellectual content. All authors participated in data interpretation, read, and approved the final version. The corresponding author, SM, attests that all listed authors meet authorship criteria and that no others meeting the criteria have been omitted.

Ethical statement and consent form

EPICOVID19 phase II study was approved by the Ethical Committee of the Istituto Nazionale per le Malattie Infettive I.R.C.C.S. ‘L. Spallanzani’ as an amendment of the EPICOVID19 epidemiological study (approval No. 93 of the trial register). Data transfer was safeguarded by means of password protection and encrypting/decrypting policy; all data were handled and stored in accordance with the EU GDPR 2016/679 (http://gdpr-info.eu/). Informed consent was accessible on the home page of the platform and participants were asked to review before starting the compilation, thus explaining the purpose of the study and which data were to be collected and how data were stored.

Email address is the personal data provided on a voluntary basis in Phase I. In our study it was only used: 1) to send email invitation to the Phase II survey; 2) to link the information related to the results (swabs and/or antibodies IgG, IgM) to the information on symptoms collected during the phase I survey. In the participation mail, the person was able to participate by clicking on the provided link to the survey; to not participate ignoring the invitation; to communicate to a specific address (valid-epicovid19@ifc.cnr.it) the request of deletion of the email address from the database.

Declaration of interests

We declare no competing interests.

Evidence before this study

As of July 28 over 16 million cases of COVID-19 have been reported to WHO.

SARS-CoV-2 strain responsible for the epidemic in Italy entered the country at the end of January. Italy has been the one of the world’s worst affected countries both in terms of infections and deaths. Rapid screening tests and technological support are needed for outbreak control and surveillance. In Italy due to organizational difficulties not yet overcome, the test for determining the presence of viral RNA by RT-PCR on nasopharyngeal swab (NPS) was performed almost exclusively to people with moderate-to-severe respiratory symptoms. Despite the numerous requests made by the population for a dedicated emergency number, the restriction to the access to tests remained throughout the entire most active epidemic phase. It is therefore likely that thousands of people with mild to severe COVID-19 remained at home during the lockdown without being able to access a diagnostic test. In such a situation, the identification of symptoms that, considered together, allow to formulate a diagnosis of probable COVID-19, acquires particular relevance. Large-scale programs using testing and screening are currently under evaluation by different governments, but clear understanding of the optimal population and role for these tests in the care pathway, are currently lacking.

Added value of this study

Our study proposes a short diagnostic scale (EPICOVID19 DS) to early recognize conditions attributable to the infection and for the timely identification of cases with possible unfavorable evolution. The use of the scale could be important in the epidemiological context for a rapid identification of suspected cases (with SARS-CoV-2 infection-like symptoms and signs) to be immediately subjected to quarantine measures. Our study provides a critical method for evaluating and interpreting self-reported data as part of COVID-19 patient management by development of a multi-component score.

Implications of all the available evidence

Early recognition screening and rapid diagnosis are essential to prevent transmission and provide supportive care in a timely manner and our score supports the potential for identifying individuals who need to seek immediate clinical evaluation. In a pandemic scenario, EPICOVID19 DS could be a preliminary assessment that attempts to detect subjects with symptoms potentially associated with COVID-19 among a wide population.

All authors have read and approved the final version of the manuscript.

The paper has not been published before and is not under consideration for publication in whole or in part elsewhere. If accepted by Journal of Medical Internet Research (JMIR; IF 5.03), it will not be published elsewhere in the same form, either in English or in any other language, without the consent of the Editors.

Hoping to hearing from you soon,

Sincerely yours,

Sabrina Molinaro

Institute of Clinical Physiology, Italian National Research Council

Via Moruzzi 1, 56124, Pisa - Italy

Email: sabrina.molinaro{at}ifc.cnr.it