A comparison of prevalence estimates of smoking, alternative nicotine and alcohol use in Great Britain collected via telephone versus face-to-face: Smoking and Alcohol Toolkit surveys

Introduction

The Smoking and Alcohol Toolkit Study (STS/ATS) is a long-running monthly representative household survey. It started in England in November 2006 and expanded to cover all of Great Britain (i.e., England, Wales and Scotland) from October 2020 (1-3). Up to February 2020, the survey was conducted face-to-face, but due to the COVID-19 pandemic, it changed to telephone surveys from April 2020. This mode change could have impacted estimates generated through the survey, which is a particular issue when comparing trends over time. The current study aimed to compare key indicators of nicotine and alcohol use collected via telephone versus face-to-face interviews.

A previous study comparing both data collection modes conducted in March 2022 as part of the STS/ATS showed that more young people participated in the face-to-face compared to the telephone survey, but otherwise sociodemographic characteristics were similar (4). After data were weighted, differences were found in estimates for never smoking and having quit more than a year ago, but when those two were combined, the estimates were similar (4). For alcohol use, there were differences in those reporting not drinking, with a higher estimate found in the face-to-face than the telephone survey (4). Overall, most estimates were comparable between the two data collection modes (4).

We conducted another parallel wave of face-to-face and telephone data collection in STS/ATS in the first quarter of 2024. The current study aimed to update the previous analysis, combining the March 2022 data with the new data collected in the first quarter of 2024. In addition to the variables assessed in the previous study, the present study also included estimates of the use of various nicotine products in the general population, as the use of novel nicotine products has increased among people who do not smoke in recent years (5).

Methods

Results

The face-to-face sample comprised of 4,381 participants and the telephone sample of 4,982 participants. Table 1 shows missing values by variables for both survey modes. These are values that were missing rather than answer responses coded as “don’t know” or “refused”. However, for the variable “AUDIT-C ≥5”, missing values are a combination of participants answering “don’t know” or “refused” to at least one of the AUDIT items, resulting in missing values for the composite score. There were no missing values for the remaining variables due to the way they were derived.

Sociodemographic characteristics

In the unweighted data, face-to-face survey participants were slightly younger than telephone survey participants (Figure 1). There was also a slight difference according to social grade, with fewer people in social grades AB and C1 and fewer in C2 and D in the face-to-face compared to the telephone survey. After weighting, both samples matched the population targets for England by design (supplementary Table S1).

• Download figure
• Open in new tab

Figure 1:

Unweighted sociodemographic estimates from face-to-face (F2F) and telephone (Tel) data collection (N_F2F=4381 and N_Tel=4982). Error bars represent 95% confidence intervals.

Nicotine use

After applying the sampling weights, estimates for most nicotine use variables were comparable across survey modes (Figure 2 and supplementary Table S2). Among the differences were that the face-to-face sample reported lower proportions of having stopped smoking over a year ago and higher proportions of never smoking relative to the telephone sample. Less people responded “don’t know” in the face-to-face than the telephone interviews regarding their tobacco use.

• Download figure
• Open in new tab

Figure 2:

Weighted estimates for smoking and use of other nicotine products from face-to-face (F2F) and telephone (Tel) data collection (N_F2F=4381 and N_Tel=4982; people smoking in past year: N_F2F=702 and N_Tel=794; e-cigarette users: N_F2F=259 and N_Tel=417). Error bars represent 95% confidence intervals. Abbreviations: cig, cigarette; NRT, nicotine replacement therapy; HTP, heated tobacco product; pre-filled cartridges, rechargeable device with pre-filled cartridges; tank to refill, rechargeable device with tank to refill.

A lower proportion of the face-to-face sample than the telephone sample reported using e-cigarettes. During the telephone interviews, some e-cigarette users reported not knowing what device they usually used, while no one provided this response during the face-to-face interviews.

Alcohol use

Similar to nicotine use, weighted estimates of most alcohol use variables were comparable across survey modes (Figure 3 and supplementary Table S3). A higher proportion of those surveyed via face-to-face compared to telephone reported never drinking at all (AUDIT 1) and never having 6 or more standard drinks on one occasion (AUDIT 3). This resulted in a lower prevalence of increasing and higher risk drinking (AUDIT-C ≥5) among the face-to-face sample (face-to-face: 25.3%, 95% CI: 23.9-26.3 vs. telephone: 30.4%, 95% CI: 28.9-32.0). Compared to the face-to-face sample, a higher proportion of the telephone sample provided “don’t know” or “refused” responses.

• Download figure
• Open in new tab

Figure 3:

Estimates for alcohol use from face-to-face (F2F) and telephone (Tel) data collection (N_F2F=4381 and N_Tel=4982). Error bars represent 95% confidence intervals.

Discussion

Overall, most estimates were comparable across the two survey modes. The face-to-face sample included slightly younger and less socioeconomically advantaged people than the telephone sample, which was balanced out through weighting. After weighting, there were only differences in the estimates of the prevalence of never smoking, having stopped smoking over a year ago, overall prevalence of e-cigarette use, never drinking alcohol, and never having 6 or more standard drinks on one occasion. For some variables, there were more “don’t know” or “refused” responses among the telephone compared to the face-to-face sample, including the questions about tobacco use, e-cigarette device type, and the number of standard drinks on a typical day.

These findings are consistent with other research comparing data quality between both survey modes. Generally, differences in mode can affect sample representativeness as well as observed response patterns (10). The first point is less of a concern when data weighting is used as it is routinely applied in the STS/ATS. However, it is possible that people who are not included in our surveys because they are either not reached or refused to participate are systematically different from those who participate (10, 11). The higher “don’t know” response rate in the telephone survey may be due to people putting less effort into answering the questions compared with people in face-to-face surveys (‘satisficing’) (10). Other research has also shown that there is a general tendency towards more “don’t know” responses during telephone compared to face-to-face interviews (12, 13). In face-to-face interviews, the interviewers can use non-verbal communication, potentially increasing engagement and encouraging participants to provide an answer (rather than saying “don’t know”) (10, 12).

We generally observe large month-to-month variations between the rates of never smoking and having quit more than a year ago, even when data are collected using the same survey mode. The reason for the variation could be that people who quit a long time ago and may never have smoked heavily classify themselves as never-smokers on some days and ex-smokers on others. While the prevalence of never smoking was higher in the face-to-face compared to the telephone survey (66.0% vs. 58.5%), the opposite was the case for having stopped over a year ago (17.5% vs. 22.4%). In our analyses, we often combine the two categories to “not smoked in the past year”, in which case these differences in responding would not have a material impact. The differences in alcohol consumption estimates may also be partly due to random variability in responses between never and rare alcohol consumption and partly due to social desirability bias, as the same tendencies were apparent in the 2022 (4) and 2024 datasets.

The present study also found a difference (between 2 to 6 percentage points) in e-cigarette use prevalence in the total population between face-to-face (6.4%) and telephone (10.4%) interviews, which may be driven by differential responses from younger age groups. Other studies also reported some variation in this outcome between survey settings. In the 2021 US National Youth Tobacco Survey, students could complete the survey online during class either in school or at home (14). The proportion of students reporting past-30-day e-cigarette use was about twice as high among those taking the survey at school (15.0%) as among those taking it at home (8.2%) (14). This effect persisted after adjusting for various covariates. The authors attributed at least part of it to response bias — depending on whether the students were among their peers or parents/guardians (14). In a 2019 systematic review, Levy et al. (15) found that online surveys tended to measure higher e-cigarette use rates than telephone or in-person surveys. Since our study showed that the difference in estimates was larger among younger and middle-aged adults, we hypothesise that they may have felt more reluctant to admit to e-cigarettes use in the face-to-face household interviews. The same issue may have led to some underreporting of alcohol consumption in the face-to-face samples. In future, we aim to record whether other people are present in the household while the survey is being completed. Assuming that our results indicate a lower risk of response bias during telephone interviews, our new study methodology may yield more accurate responses.

It is worth noting some limitations of the study. First, the sample sizes may have not been sufficient to detect important differences of some variables. Second, this is the second time we have compared a relatively large number of indicators; therefore, it is possible that some of the highlighted differences may reflect random variation. Third, data were not collected at exactly the same time. We tried to adjust for this by including a sensitivity analysis with telephone data collected between January and March 2024, which is the same timeframe in which the face-to-face survey was collected. The results of this analysis were comparable to that using only February 2024 data.

In summary, face-to-face and telephone surveys generally yielded similar estimates of nicotine and alcohol use. However, there may be some underreporting of vaping and drinking in a face-to-face survey conducted in the home compared with telephone.

Data sharing

Data are available upon reasonable request.