Predicting the future risk of lung cancer: development and validation of QCancer2 (10-year risk) lung model and evaluating the model performance of nine prediction models

Study design, study population, and data source

This is a population-based cohort study. We used EHRs from the QResearch® database (version 45) as the data source. The QResearch® database is one of the largest healthcare databases in England, and a trusted research environment accredited by Health Data Research UK. The inclusion and exclusion criteria were similar to those in the previous QCancer (10-year risk) algorithms¹⁵, which allowed us to compare the model performance between the original and the updated versions of models. We included adult patients aged 25-84 years registered with general practices contributing to the QResearch database between 01 January 2005 and 31 March 2020 and excluded patients with a diagnosis of lung cancer before cohort entry. The broad age range covers the majority of the adult primary care population. It also provides great flexibility to select patients in different age groups to evaluate the model performance (Stage 2) or to perform subgroup analysis. The included patients needed to have been registered in the general practices for at least 12 months, and the general practices needed to have contributed to the QResearch database for a minimum of 12 months before the cohort entry date. This was to ensure complete data before cohort entry.

Primary study outcome

The primary study outcome was an incident diagnosis of lung cancer recorded on one or more of the four linked data sources – primary and secondary (hospital episode statistics, HES) care databases, cancer registry (from Public Health England, PHE) and death registry (from the Office for National Statistics, ONS). We used the earliest date on any of the four records as the date of lung cancer diagnosis. We used Read/SNOMED-CT codes to identify events from the GP record, and ICD-10 codes to identify events from HES, cancer and death registries.

Updating the QCancer (10-year risk) lung model

The predictors for the original models include age, sex, ethnicity, Townsend scores (a proxy for socioeconomic status), smoking status and intensity, alcohol status, BMI, COPD, asthma, asbestos, prior cancer, and family history of lung cancer. Two additional predictors (pneumonia¹¹ and venous thromboembolism¹⁷) were included in the updated model. We used well-established methodologies for developing and validating the risk prediction algorithms^15,18. Three-quarters of general practices were randomly selected for the derivation dataset and the remaining quarter for the validation dataset. Multiple imputation with chained equations was used to replace missing values for ethnicity, body mass index (BMI), alcohol and smoking status, with five imputations. We assumed that the absence of clinical information in the database indicated that the patients did not have specific health conditions, personal or family histories. Therefore, we did not perform multiple imputation for the clinical predictors. We used the imputed values in our main analyses and Rubin’s rules to combine the results across the imputed datasets¹⁹.

Fractional polynomials²⁰ were used to model non-linear relationships between the continuous variables (age, BMI, Townsend scores) and the outcome. Cox proportional hazards models were used to estimate the coefficients for each risk factor for men and women separately, using robust variance estimates to allow for patients clustering within general practices. We initially fitted a full model with all variables for men and women, and retained the variables with a hazard ratio (HR) of <0.91 or >1.10 (for non-continuous variables) for clinical significance and statistical significance at the 0.01 level.

How the QCancer2 (10-year risk) lung model works

We used the coefficients for each predictor from the final Cox regression as weights and combined them with the baseline survivor function evaluated for up to 10 years to derive the risk equations¹⁵. The baseline survivor function was estimated based on the zero values of the centred continuous variables and all binary predictors. The risk equations allow us to derive absolute risk estimates for each year of follow-up, but we had a specific focus on 5-, 6-, and 10-year risk estimates, as we intended to compare the QCancer2 (10-year risk) lung model against other models for lung cancer screening in Stage 2 of this study.

Other prediction models for lung cancer screening to be compared against the QCancer2 (10-year risk) lung model

Toumazis et al conducted a systematic review of risk prediction models for lung cancer screening²¹. There were also empirical studies comparing some mainstream prediction models published in recent years^22,23. We referred to these studies and included models with predictive horizons ≥5 years, as the purpose of screening is to early detect cancer from asymptomatic populations. The sojourn time for lung cancer progressing from preclinical stage (detectable by screening tests) to clinical stages is 3-6 years, longer in women than in men²⁴. Detecting lung cancer at the earliest possible stage is thought to be most effective in reducing mortality, provided over-diagnosis is avoided. Therefore, prediction models for a longer period are more suitable for screening than those designed for a shorter period (e.g. 1 or 2 years) for diagnostic purposes.

We included seven models to be compared against our models, including LLP_v2, LLP_v3, LCRAT, PLCO_M2012, PLCO_M2014, Pittsburgh, and Bach. The LLP_v2 and PLCO_M2012 are used in the TLHC programme to select eligible participants for lung health check. LLP was updated and the newest version (LLP_v3) was published in 2021¹⁴. The authors recalibrated the intercepts for each age group by sex, while the predictors and their coefficients remained unchanged between LLP_v2 and LLP_v3¹⁴. PLCO_M2014 was updated based on PLCO_M2012 and could be used to calculate the risk for never smokers²⁵, while PLCO_M2012 was developed for ever-smokers only. The 5-year Lung Cancer Risk Assessment Tool (LCRAT)²⁶ was selected because an empirical study concluded that this model performed the best in classifying future lung cancer cases using data from three UK cohorts²³. The 6-year Pittsburgh Predictor²⁷ was included for its simplicity – only 4 predictors in the model (age and three smoking variables), with reasonably predictive accuracy. The PLCO, LCRAT, and Pittsburgh models used either all or part of the PLCO/NLST data in their models. Hence, some study samples overlapped in these models. The Bach model²⁸ can predict 10-year risk, which is the same predictive horizon as our models. Most models were developed based on the US population, except for LLP using regional data from Northwest England. A total of nine prediction models were included in this study. The general information and predictors for each model are summarised in Table 1.

Handling EHR data for the variables in other prediction models

Due to different study designs (RCT, case-control study, or survey) and different study samples, variables in other prediction models are not necessarily available in EHR. O’Dowd et al. externally validated the LLP_v2 and PLCO_M2012 models using CPRD in the English population¹³. We handled most variables in a similar way as reported in their paper, except for ethnicity, the categories of smoking intensity, and the age of starting smoking. Information on patients’ ethnicity is available in the QResearch database. Therefore, we need not assume all patients were white as O’Dowd did. There are five smoking categories in the QResearch database, non-smoker, ex-smoker, light smoker (1-9 cigarettes/day), moderate smoker (10-19 cigarettes/day), and heavy smoker (≥20 cigarettes/day). We used the median number of cigarettes smoked per day for each category, i.e. 5 for light smokers, 15 for moderate smokers, and 30 for heavy smokers for the PLCO and other models, while O’Dowd et al used the lower bound of the number for moderate and heavy smoking, which may underestimate the cumulative lifetime smoking exposure. We assumed people started smoking at 15 years, as people aged ≥50 years at cohort entry (2005) were born in the 1930s-1950s. People at that time started to smoke at a much younger age and the smoking prevalence was much higher than nowadays²⁹. The current legal age of smoking at 18 was set in 2007. We wrote Stata codes for the QCancer (10-year risk), LLP_v3, and PLCO_M2014 models, while the risk scores for the LLP_v2, LCRAT, PLCO_M2012, Pittsburgh, and Bach models were calculated using the “lcmodels” R package.

Evaluation of model performance

We calculated the absolute predicted risks of individual patients by each algorithm in the validation datasets. We calculated the following measures for discrimination for each algorithm, including the (explained variation, where higher values indicate a greater proportion of variation in time to event is explained by the model)³⁰, the D statistic (where higher values indicate better discrimination)³¹, and Harrell’s C statistic³² (similar to the area under the receiver operating characteristic curve but taking account of the censored nature of cohort data) at the prediction horizon of each algorithm (5, 6 or 10 years). These statistics were combined across the imputed datasets using Rubin’s rules. To assess calibration, we used the “pmcalplot” package in Stata³³ to compare the mean predicted risks (x-axis) against the observed risk (obtained using the Kaplan-Meier estimates, y-axis) in twentieths by sex for each algorithm.

The three discrimination measures were calculated and calibration plots were made for the updated QCancer2 (10-year risk) lung model for the whole validation cohort (men and women aged 25-84 years). In addition, we evaluated the discrimination, calibration, and net benefits of the prediction models in two subgroups – people aged 40-84 years (including non-smokers) as subgroup 1, and ever-smokers aged 55-74 years (the criteria for the TLHC programme)¹⁰ as subgroup 2. There were two main reasons to set these two subgroups. Firstly, the included algorithms were developed with different inclusion and exclusion criteria (such as age range and smoking exposure), these two subgroups broadly cover the inclusion and exclusion criteria for the included algorithms so that they could be compared fairly. Secondly, we aim to provide research evidence regarding which algorithm will be the best to select eligible people for the TLHC programme if rolling out across England (through research evidence from subgroup 2). Decision curve analysis³⁴ was used to evaluate the net benefit of the prediction models (clinical usefulness) for models in three predictive horizons by sex. We calculated the net benefits across a range of threshold probabilities. In general, the model with the highest net benefit at any given risk threshold is considered to have the most clinical value^35,36.

All the analyses were conducted in the QResearch server using Stata 17.0 and R 4.1.0. We used the TRIPOD (Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis) statement to guide the conduct and reporting of this study^37,38. A TRIPOD checklist for this article is in supplementary table S1.

Ethical approval

This project was approved by the QResearch Scientific Committee on 8 March 2021. QResearch is a research ethics approved database, confirmed by the East Midlands – Derby Research Ethics Committee (Research ethics reference: 18/EM/0400, project reference: OX37 DART).

Patient and public involvement

Two lay representatives from the Roy Castle Lung Cancer Foundation were involved in the project set-up phase. They reviewed our lay summary and provided feedback as part of ethical approval for this project. The lay members were not involved in the research process, but we will engage patients and stakeholders in creating accessible materials in lay language for the public to understand prediction models and risk scores of lung cancer and the benefits and harms of the TLHC or lung cancer screening when we disseminate our study findings.

Population characteristics

Table 2 shows the baseline demographic and clinical characteristics of the study population and the incident lung cancer cases in the derivation and the validation cohorts for the updated QCancer2 (10-year risk) lung model. There were 12,991,042 patients aged 25-84 years in the derivation cohort and 73,380 people developed incident lung cancer during the follow-up period. There were 4,137,199 patients in the validation cohort and 22,838 developed incident lung cancer. The ratio of men and women was almost 1:1 in the derivation and validation cohorts, but men had a higher proportion (56%) than women (44%) in the incident lung cancer cases. The mean age for incident lung cancer cases (66 years) was 21 years older than that of the primary care population (45 years) in this study. The Townsend quintile was equally distributed in general in the derivation and validation cohorts, but had a smaller proportion in the two bottom Townsend quintiles (more deprived) in incident lung cancer cases. The percentages of ethnicity, BMI, alcohol, and smoking status recorded in the EHRs in the whole cohort were around 72%, 83%, 82%, and 93%, respectively. The proportions of comorbidities, family history of lung cancer, and any prior cancer at baseline in incident lung cancer cases were much higher than those in the primary care population.

Comparison of the predictors between the original and the updated QCancer (10-year risk) lung models

The hazard ratios (HRs) for the predictors in the original and updated QCancer (10-year risk) lung models are in Table 3. The coefficients for the predictors in the updated models by sex did not change much, compared with those in the original models. Two new predictors in the updated model, pneumonia and venous thromboembolism, were both significant and had higher HRs in women than in men. Other significant predictors included ethnicity, smoking status and intensity, family history of lung cancer, COPD, asthma, prior oral, blood, and renal cancers. Compared with white, other ethnicities had a smaller HR (HR<1), except that Bangladeshi and Chinese women had an HR>1. The HR increased with the current smoking status and smoking intensity (cigarette per day) and was generally in a dose-response relationship in both sexes. The HR for moderate smokers (10-19 cigarettes/day) was smaller than that in light smokers (1-9 cigarettes/day) in men. Two smoking-related conditions, COPD and prior oral cancer, had an HR>2 in both sexes. Asbestos exposure, prior colorectal and gastric-oesophageal cancers, and drinking status, were all significant in men, but not significant in women. Such differences showed the importance of developing models separately by sex, as men and women may have different significant predictors. Even for the same predictors, the coefficients and the association between predictors and the outcome could be different between the sexes.

Supplementary Figure S1 shows the non-linear relationship (fractional polynomials) between the continuous variables (age, BMI, Townsend score) and the outcome (incident lung cancer cases) by sex. Supplementary Figure S2 shows the non-linear relationship between the interaction terms (age and smoking status) and the outcome by sex.

Evaluating the model performance for prediction models for lung cancer screening

The prediction models being compared and evaluated are summarised in supplementary table S2. The descriptive statistics of the predicted risks for each model in the two subgroup populations in three predictive horizons by sex are in supplementary tables S3 and S4.

Calibration

Calibration plots are presented in Figures 1–3. Figure 1 is the updated QCancer2 (10-year risk) lung model in three predictive horizons by sex. Figures 2 and 3 are calibration plots for the two patient subgroups. The QCancer2 (10-year risk) lung model was generally well-calibrated across different risk levels in both sexes, in three predictive horizons, in the whole validation cohort and the two subgroups. For the other prediction models, in the 5-year horizon, a common problem for the LLP_v2, LLP_v3, and LCRAT models in patient subgroup 2 (ever-smokers aged 55-74 years) was overestimation (the predicted risks were much greater than the observed risks) in both sexes, especially at higher risk bands. Calibration of these three models was better in patient subgroup 1, but still overestimated the risk at high-risk levels. Compared with LLP_v2, LLP_v3 was better calibrated in both patient subgroups. In the 6-year horizon, the PLCO_M2012 severely under-estimated at low-risk thresholds and over-estimated at high-risk thresholds, particularly in patient subgroup 2. PLCO_M2014 under-estimated across all risk bands in both sexes and both patient subgroups, while the Pittsburgh predictor over-estimated the risk. The Bach model was poorly calibrated in both sexes, particularly in patient subgroup 2.

• Download figure
• Open in new tab

Figure 1

Calibration plots for the QCancer2 (10-year risk lung model) in the three predictive horizons by sex

• Download figure
• Open in new tab

• Download figure
• Open in new tab

• Download figure
• Open in new tab

Figure 2

Calibration plots for smokers and non-smokers aged 40-84 years old (patient subgroup 1) in the three predictive horizons and by sex

• Download figure
• Open in new tab

• Download figure
• Open in new tab

• Download figure
• Open in new tab

Figure 3

Calibration plots for ever smokers aged 55–74 years (patient subgroup 2, the targeted lung health check criteria) in the three predictive horizons and by sex

Decision curve analysis

Figures 4–5 show the net benefit curves for the prediction models in the two subgroups by sex in three predictive horizons. The QCancer2 (10-year risk) lung model had the highest net benefit, compared with other prediction models and strategies considering either no patients or all patients for intervention across a range of risk thresholds.

• Download figure
• Open in new tab

Figure 4

Decision curve analysis for patient subgroup 1 (smokers and non-smokers aged 40-84 years old, including never smokers)

• Download figure
• Open in new tab

Figure 5

Decision curve analysis for patient subgroup 2 (ever-smokers aged 55-74 years old, the TLHC criteria)

Principal findings

In this study, we developed and validated the QCancer2 (10-year risk) lung model with more recent data and a larger sample size to quantify the future risk of men and women being diagnosed with lung cancer in the next 10 years. The predictors include sociodemographic characteristics, lifestyle, comorbidities, family history and personal history of other cancers. The updated model performed slightly better than the original version. We compared the QCancer2 (10-year risk) lung model against other prediction models and found that the QCancer2 (10-year risk) lung model had the best performance in discrimination, calibration, and net benefits in two subgroups across three predictive horizons.

Comparison with existing literature

The QCancer (10-year risk) models are prognostic models, which can estimate an individual patient’s future risk of cancer while the patient has not yet presented any symptoms. Prognostic models are more suitable for early detection of cancer through screening in asymptomatic populations, while diagnostic models (known as QCancer) aim to predict the risks of existing cancer in patients already present with symptoms that may indicate cancer.

Compared with the original model using a total of 6.6 million patients aged 25-84 years and 13,570 incident lung cancer cases from 753 practices from 1 January 1998 to 30 September 2013 in the deviation and validation cohorts, the sample size of the whole population and the incidence lung cancer cases in the updated model increased substantially, due to the expansion of the QResearch database in recent years. We have conducted a more thorough assessment of model performance in this study. ten Haaf et al. mentioned little attention has been given to sex-specific risk-stratification in current practice³⁹. We developed and validated our models by sex and found that the predictors differed between men and women. Even for the same predictors, their coefficients may be different. The histological subtypes and preclinical duration of lung cancer may vary between sexes^24,39. It may need to consider sex-specific risk thresholds and screening intervals for lung cancer screening.

When choosing risk prediction models for population-based screening programmes, we need to consider whether the models can be applied to the targeted population (external validity and generalisability), as models may be developed using a specific study sample, with limitations in study design, data, or internal validity of the study findings. Some models may be developed using highly selective study samples or populations, whose baseline risk was higher than the general population. The LLP model had limitations in study design and study sample (case-control study using regional data)¹¹. The PLCO_M2012 model was developed using data from the US population¹². When externally validating these two models using CPRD, O’Dowd found that both models underestimated the risk in patients at low predicted risk (observed risk>predicted risk) but overestimated the risk in patients at higher risk (observed risk<predicted risk, cut-offs at around 1% for both models)¹³. The C statistics (equivalent to AUC in O’Dowd’s report) in this study were even lower. Possible explanations included study sample selection (two patient subgroups in this study vs ever-smokers aged 50-80 years), availability of information, population and geographical coverage in different EHR databases (QResearch vs CPRD), different ways of handling the variables unavailable from the EHR database (described in the methods section), sample size and study period (this study had a larger sample size and a longer study period). Despite these differences, both studies reached the same conclusion that the LLP_v2 and PLCO_M2012 models did not have satisfactory discrimination and were not well-calibrated when externally validated using different English primary care databases. They may not be directly applicable to the English primary care population.

Robbins et al. evaluated the model performance of lung cancer risk models (LLP_v2, LLP_v3, LCRAT, LCDRAT, PLCO_m2012, and Bach) in current and former smokers aged 40-80 years using three UK cohorts (UK Biobank, EPIC-UK, and Generations Study) to define the eligibility for lung cancer screening²³. We did not include LCDRAT in our study, as this model predicts lung cancer death. The AUC values for the models in the two patient subgroups (patients aged 40-74 years and 55-74 years, 0.73-0.82) were much higher in the three cohorts than those in our study. For calibration, all models overestimated risk in all cohorts. The ratio of expected vs observed (E:O) was between 1.20 (LLP_v3) and 2.25 (LLP_v2). We found both overestimation and underestimation appeared in our study samples. Robbins et al. emphasised the importance of validating prediction tools in specific countries. We agreed with this point.

Strengths and limitations of this study

Implications for practice and health policy

As a universal service with a very high population coverage in the UK, primary care has great potential and opportunity for early detection and diagnosis of cancer, and GPs could be strong advocates for the TLHC programme and lung cancer screening. We have demonstrated the model performance in the QCancer2 (10-year risk) lung model. More accurate risk estimation has an impact on risk stratification at the population level and risk communication with the patients. The implementation of the QCancer2 (10-year risk) lung model in primary care computer systems could facilitate patient-GP discussion about the risks and benefits of preventive interventions for individual patients. This could improve patients’ awareness of their health status and risk level, which in turn might increase their willingness to participate in the TLHC programme or lead to behavioural changes such as considering smoking cessation.

The updated QCancer2 (10-year risk) lung model allows batch-mode processes that use existing information in EHRs at each practice to facilitate the selection of eligible patients at high risk for the TLHC programme or lung cancer screening. Compared with using questionnaires to collect information from individual patients and calculating risk scores to check their eligibility, the batch-mode process is more efficient. It can stratify the primary care population based on risk level for better target in prevention or screening programmes. It can not only substantially reduce human resources and costs, but also save time and streamline the administrative process for better patient experience and increased patient satisfaction.

We comprehensively compared nine prediction models using the same validation dataset in this study. We evaluated model performance in two patient subgroups, which broadly covered the inclusion and exclusion criteria of the study sample in each model and the eligibility criteria for the TLHC programme. The LLP_v2 and PLCO_M2012 may not be the best models to select eligible primary care patients for the TLHC programme, due to their unsatisfactory model performance in patient subgroup 2. Through such a thorough assessment in this study, we provide research evidence to assist policymakers in considering the age range and smoking exposure for lung cancer screening and increase equality in health services access. In the end, more patients can be diagnosed at earlier stages and through screening-detected route, which can lead to better survival outcomes and reduced lung cancer mortality in the UK. Considering the sizable incidence and mortality of lung cancer at the population level, the improvement in lung cancer could substantially contribute to the UK government’s ambition that 75% of people with cancer will be diagnosed at early stages by 2028⁴¹. Meanwhile, patients may benefit from primary or secondary prevention of other cardiorespiratory diseases through participating in the TLHC programme. We do not recommend a risk threshold for our models for lung cancer screening, as this needs to consider the balance between benefits and harms, cost-efficiency, health resources, accessibility and health equality, and the potential impact of the screening program at the population level. Our colleagues in another strand of our work package will conduct a separate study to evaluate the cost-effectiveness and determine the risk thresholds for lung cancer screening in England.