ThinkCancer! The multi-method development of a complex behaviour change intervention to improve the early diagnosis of cancer in primary care
=============================================================================================================================================

* A Surgey
* S Disbeschl
* R Lewis
* J Hiscock
* S Nafees
* R J Law
* J L Roberts
* A Hendry
* Z Hoare
* N Goulden
* M A Stanciu
* A Carson-Stevens
* S Smits
* K Brain
* R T Edwards
* N Williams
* R D Neal
* C Wilkinson

## ABSTRACT

**Background** Relatively poor UK cancer outcomes are blamed upon late diagnosis. Despite most cancer patients presenting to their GP with symptoms, diagnostic delay remains a common theme, with many clinical and non-clinical factors responsible. Early diagnosis is key to improving outcomes and survival. This paper reports the multi-method process to design a complex intervention to improve the timely diagnosis of symptomatic cancer.

**Methods** A review of reviews, survey, discrete choice experiment, qualitative interviews and focus groups, all informed a realist evidence synthesis. This in turn informed the design of a complex intervention, guided by the Behaviour Change Wheel framework using a multi-step process.

**Results** Key themes from the realist evidence synthesis included effective safety netting at practitioner and practice system level, increased vigilance and lowering referral thresholds. Qualitative findings explored the tensions, barriers and facilitators affecting suspected cancer referral. The Think Cancer! intervention is an educational and quality improvement workshop directed at the whole primary care team. Bespoke cancer safety netting plans and appointment of cancer champions are key components.

**Conclusions** Think Cancer! is a novel primary care early cancer diagnosis intervention, requiring evaluation through a cluster randomised control trial.

## Background

The role and importance of primary care (PC) in the diagnosis of cancer is well established.1 Wales, like other UK countries, has relatively poor cancer outcomes2,3 with late diagnosis recognised as a major contributor resulting in low one-year survival rates.4,5 Early diagnosis has been shown to be key in improving cancer outcomes6 and cancer survival.7 It is therefore unsurprising that national and international cancer policies are heavily focussed on diagnosis. Referral rates and adherence to guidelines are lower in Wales,8,9 with General Practitioners (GPs) in the UK overall less likely to take action on potential cancer symptoms.2 These findings are unlikely to be unique to Wales with expected similar findings in other countries with comparable health care systems and areas of deprivation.

Most (85%) cancer patients initially present to their GP with symptoms,10,11 yet diagnostic delay remains a common and emotive theme in cancer research.12 Access to secondary care diagnostics and treatment within the UK health services, as with many other countries with similar models of care, places the GP as the first point of contact in a ‘gatekeeping’ role. Therefore, irrespective of scientific advances such as new diagnostic tests, rapid diagnostic centres for vague symptoms and evolving cancer therapies, unless the GP thinks about the possibility that their patient may have an underlying cancer, delays in investigation and referral will continue.

The Wales Interventions and Cancer Knowledge about Early Diagnosis (WICKED) Programme (Figure 1) aims to develop and test a behavioural intervention targeted at GPs and primary care teams to improve early diagnosis of cancer using the Medical Research Council’s framework for developing and evaluating complex interventions.13 The aims of this phase of the project are to synthesise current evidence and to develop a complex intervention that will improve the quality of PC practitioners’ knowledge, attitudes and referral behaviour (work packages 1-3). This paper describes the findings of those work packages and their contribution to intervention development, resulting in the production of a complex behaviour change intervention that will be tested in a feasibility trial (work package 4).

![Figure 1.](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2020/11/23/2020.11.20.20235614/F1.medium.gif)

[Figure 1.](http://medrxiv.org/content/early/2020/11/23/2020.11.20.20235614/F1)

Figure 1. 
Diagram of the WICKED Programme

## Method

The background, rationale, design and methods of the WICKED programme, leading to the development of the ThinkCancer! intervention, have been reported elsewhere. Therefore, 14 this section summarises the methods, focusing on their interconnectivity with the intervention development process.

### Work package 1a: Review of reviews

A systematic review of reviews was undertaken to:

*   identify the reasons for (or causes of) longer primary care intervals (PCIs) in the diagnosis of cancer;

*   identify all interventions designed to reduce the PCI;

*   identify the causes of longer PCIs, which could be modified by a behaviour change intervention;

*   identify interventions designed to modify the behaviour of PC health professionals;

*   identify non-behaviour change intervention components that could contribute to the ThinkCancer! intervention (e.g., structural or organisational modifications that could facilitate the behaviour change elements); and,

*   identify which types of interventions (or intervention components) are applicable to UK and similar health systems.

The review of reviews followed methodological guidelines for rigorous conduct and reporting.16,17 The search strategy and rationale are reported in the protocol paper,14 and are included in detail (supplementary file 1). The searches were conducted in October 2016 and extended to reviews published from 1995 onwards. These searches have not been updated as data was used to inform the starting point of the later intervention development steps, rather than as a stand-alone research output. Two reviewers independently screened titles and abstracts and then the full-text of relevant papers for inclusion. Disagreements were resolved by discussion. Data were extracted in tabular format. We did not formally appraise or exclude studies on the basis of quality; where relevant data contributed to the realist review we considered their ‘trustworthiness’ (i.e., the plausibility and coherence of the methods used to generate the data, and its contribution to the development of the synthesis) in that context.18

### Work package 1b: Realist review

The realist review aimed to identify and test programme theories underpinning identified interventions, and determine what works/or not, for whom and in what contexts. This process informed, and was informed by, components of work package 2, and aided the design and development of the ThinkCancer! intervention. Specific objectives were to:

*   identify evidence about interventions promoting prompt diagnosis/referral of suspected cancer in PC settings;

*   identify the mechanisms through which these interventions directly or indirectly effect changes in the behaviour of individuals, groups and organisations in PC that lead to earlier referral and diagnosis;

*   investigate the contextual characteristics that mediate the potential impact of these mechanisms on early referral and diagnosis in PC; and,

*   develop a practical programme theory from the evidence synthesis that can be applied to the development of the ThinkCancer! Intervention.

The realist review followed on from the systematic review of reviews, in agreement with Rycroft-Malone, et al., 19 adapted from Pawson, et al. 20 and RAMESES methodological guidance and reporting standards.21,22 The first stage of the process involved data collection from multiple sources organised into a matrix of evidence (supplementary file 2), starting with a list of possible causes of longer PCIs in cancer referral, and examples of interventions to promote prompt diagnosis/referral drawn from scoping searches and background reading. Evidence was mapped against categories of barriers and facilitators to timely referral and diagnosis, further developed through a process of discussion and consensus between the research team operational group (see Figure 2).

![Figure 2.](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2020/11/23/2020.11.20.20235614/F2.medium.gif)

[Figure 2.](http://medrxiv.org/content/early/2020/11/23/2020.11.20.20235614/F2)

Figure 2. 
Areas of programme theory used in the realist review.

The six areas of focus were then developed as theory areas for the realist review in the form of conjectured Context-Mechanism-Outcomes (CMOs) by two experienced researchers with a background in primary care cancer research, and further refined through several stakeholder group meetings of clinicians and patient participant involvement (PPI) representatives. The final stage of the realist review involved testing and refining the programme theory by searching for and compiling further evidence. A realist narrative was written up for each programme theory area with recommendations.

### Work Package 2a: Quantitative Survey and Discrete Choice Experiment (DCE)

The survey measured the GPs’ cancer knowledge, investigation and referral attitudes, along with their behaviours, learning needs and preferences, and mechanisms supportive of change. The findings were used to identify the influences of Capability, Opportunity and Motivation on the target behaviour (COM-B). COM-B describes how changing behaviour at the individual and/or system level is a result of changing one or more components of psychological and physical Capability, social and physical Opportunity, and automatic and reflective Motivation which informed subsequent intervention development.14

All GPs in Wales (approximately 2000) were sent an email invitation to complete the survey between 22nd August 2017 and 18th December 2017. A DCE was part of the online survey. The respondents were required to state preferences regarding hypothetical scenarios,23 in order to explore GPs’ preferences for different attributes relating to the earlier diagnosis of cancer in PC, and to explore how GPs are willing to trade between attributes. The attributes and levels were identified from a scoping search, results of which included a review of reviews, experts’ opinions and consensus. Seven attributes used were prompts, audit, type of education and delivery, mode of delivery of education and training, access to diagnostic testing, safety netting and time spent on activities to achieve timely cancer diagnosis.

### Work Package 2b – Qualitative component

The qualitative component explored underlying GP and practice team beliefs and behaviours in identifying, investigating and referring cancer signs and symptoms (including perceived constraints, barriers or concerns), through telephone interviews and focus groups. The purposive sampling procedure and semi-structured topic guide are described elsewhere.14 The focus groups also contributed to understanding the practicalities of how the intervention should be focused.

Twenty GPs were interviewed and four focus groups with PC practice teams were conducted. Both data sets were transcribed verbatim, organised and analysed using the five stages of the Framework process: familiarisation, thematic framework identification, indexing, charting, mapping and interpretation.24

### Work package 3 – Behaviour Change Wheel (BCW) driven intervention development

All data from work packages 1 and 2 were collated to inform the BCW process and guide the development of an intervention designed to change a target behaviour (see Figure 3). 14,25 The starting point in designing a behaviour change intervention is to clarify the target behaviour, or the ‘behavioural diagnosis’.25 For ThinkCancer! this was refined according to the main aims of the programme and discussed within the research team and wider stakeholder group. Our proposed target behaviour was identified as *GPs thinking of and acting on clinical presentations* that could be cancer*.

![Figure 3:](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2020/11/23/2020.11.20.20235614/F3.medium.gif)

[Figure 3:](http://medrxiv.org/content/early/2020/11/23/2020.11.20.20235614/F3)

Figure 3: 
The Behaviour Change Wheel (Reproduced with permission from authors)

**We defined clinical presentations as symptoms, clinical signs and related context (e*.*g*., *past medical history, risk factors, previous consultations) identified during a primary care consultation*.

Figure 3 depicts the different layers of the BCW with the COM-B model at its centre. (see supplementary file 3 for the detailed methodology of the application of the BCW).

Using a matrix of intervention functions and applying the APEASE criteria (Affordability, Practicability, Effectiveness/cost-effectiveness, Acceptability, Side effects/safety and Equity) 25,26 we selected the following functions that best suited our target behaviour: education, training, enablement, environmental restructuring, modelling, persuasion and incentivisation. Intervention functions were then matched to relevant Behaviour Change Techniques (BCTs), for example, in relation to education and training, the relevant intervention functions included: *instruction, information about health consequences, feedback on outcomes, prompts/cues, self-monitoring, and verbal persuasion about capability*. A multidisciplinary intervention working group was convened to develop the intervention, including members of the research team, a wider stakeholder group including GPs to ensure real world applicability and an expert patient.

### Public and Patient Involvement (PPI)

There has been continued and active Public and Patient Involvement (PPI) within the research programme from its initial design and continued through each work package, with regular contribution to research group meetings and advice and comment gained on project documentation.

## Results and Intervention Development

### Work package 1a: Review of reviews

After discarding duplicate references, 334 studies were identified, of which 37 were deemed to be potentially relevant and full papers were retrieved. Seventeen of these were included in the review. 7,27-42 A summary of findings is presented in supplementary file 4.

Thirteen of the included studies were systematic reviews 27,29,31-36,38-42 and three were non-systematic narrative literature reviews.7,30,37 The final study reported a consensus process with key informants that was based upon an earlier literature review.28 Ten studies focused on specific cancers: haematological, breast and colon;27 bladder 29,39; colorectal 30,33,41,42; upper gastro-intestinal 31,32; and colorectal and lung. 36 The remaining seven considered any cancer. Broadly speaking, three studies focused on the impact on cancer referral of features of healthcare systems, 28,33,37 two focussed on clinical roles and practices, 29,31 two on signs, symptoms and the process of diagnosis, 39,40 and six on identifying or describing factors relating to diagnostic intervals.7,27,30,32,35,36

Only three reviews aimed to include studies that evaluated interventions to address delays or promote early diagnosis of cancer.32,34,38 Macdonald, et al.32 evaluated factors associated with longer patient and PC practitioner intervals for upper gastrointestinal cancer and aimed to describe an intervention designed to reduce those intervals, but found no intervention studies that met their inclusion criteria.32 Mansell, et al.34 aimed to identify interventions that reduce PC delay. Although the review covered any cancer, only skin, breast, colorectal studies met the inclusion criteria; 22 interventions were identified, but no evidence that any intervention directly reduced PC delay in the diagnosis of cancer. Limited evidence suggested that complex interventions such as audit and feedback, and specific skills training had the potential to reduce delays. Schichtel, et al.38 examined the evidence for the effectiveness of educational interventions for primary healthcare professionals. Twenty studies were included but the authors concluded that, although some educational interventions delivered at a clinician as well as at a practice level may promote the early diagnosis of cancer in PC, there is limited evidence for their long-term sustainability and effectiveness.

### Work package 1b: Realist review

The following initial conjectured Context-Mechanism-Outcome configurations (CMOs) were developed and refined:

#### CMO 1

Where explicit systems are put in place to plug potential gaps in the patient pathway (i.e. safety netting) **(context)**, and members of the healthcare team identify and perceive themselves to have safety-netting responsibilities that are important and relevant to their role (e.g. to ensure test results are acted upon, referrals sent, follow-up appointments made and attended as advised etc.) **(mechanism)**, then they will act upon them, and patients will be more likely to progress to referral without undue delay **(outcome)**.

#### CMO2

Where systems are in place and patients are engaged in safety-netting processes (e.g. by being advised to come back if symptoms are not resolved or get worse, asked to ring if they don’t hear about a test result, hospital appointment etc.) **(context)**, and, by becoming more involved, they perceive the importance of these actions and are empowered to take more responsibility **(mechanism)**, then they will adopt a more proactive role in their care and will be less likely to miss appointments etc. **(outcome)**.

#### CMO 3

Where all practice staff who have contact with patients have been trained to recognise unexpected signs and uncharacteristic behaviours (e.g. altered appearance such as weight-loss, unusual consulting pattern) **(context)**, and are therefore alert to, and notice these signs, and recognise the possibility of cancer **(mechanism)**, then staff will be more likely to question patients and report suspicions, and GPs may identify more patients who they *should* worry about, pay closer attention to these patients, and perhaps elicit important information (signs, symptoms, previous or family history etc. that the patient may not recognise as relevant and would not otherwise volunteer) **(outcome)**.

#### CMO 4

Where usual practice includes discussion of patient-safety (e.g. ‘safety huddles’, significant event audits), and there is a culture of frank, informal, non-judgemental interaction **(context)**, and by sharing experiences and supporting each other, GPs benefit in terms of improved knowledge and confidence in their actions **(mechanism)** then they will be more likely to recognise when investigations and referrals are warranted, and less likely to delay referral because of uncertainty **(outcome)**.

#### CMO 5

Where a practice audit of cancer referrals is conducted and feedback is given to GPs on whether recommended investigations and examinations were conducted **(context)**, they will be aware of and reflect on their own personal performance, compared with local or national performance, and recognise how it can be improved and **(mechanism)** then they will be more likely to perform these tasks consistently, in line with good clinical practice and NICE guidance **(outcome)**.

#### CMO 6

In the face of uncertainty about individual cancer referral decisions (e.g. vague symptoms, low-risk but not no-risk), if a philosophy of cancer-risk-averseness, and a practice culture of case-finding rather than gate-keeping are engendered **(context)**, GPs will be more likely to lower their individual risk threshold for urgent referrals and Think Cancer! **(mechanism)**, so that more cases of suspected cancer will be identified and referred **(outcome)**.

The final stage of the realist evidence synthesis led to completion of narrative summaries (Supplementary File 2), summarised into the following key headlines:

*   Wider use of high quality and effective safety-netting in every aspect of the patient/health care professional interaction;

*   Dual safety-netting responsibility between patient and practitioner by encouraging aware, engaged and empowered patients to co-navigate the patient journey;

*   Increased vigilance and taking action when suspicions are raised (including all practice staff);

*   Improving the practice level patient safety agenda;

*   The right test, for the right patient, at the right time (guidelines, awareness, false negatives); and,

*   Lowering the threshold to referral trigger (challenging gate keeper mentality, acting on gut feelings).

### Work package 2a: Quantitative Survey

In total, 1993 GPs were invited and 269 responded (13.5%). Respondents were representative of GPs in Wales in terms of personal and practice factors. There was strong agreement amongst respondents that they had changed their behaviour with regard to safety netting of symptoms in the past few years, and changing thresholds for investigation and referral, but less so for the use of decision support tools. The most prominent drivers of change were case discussions with colleagues and significant event audits. There was strong agreement that timelier diagnosis leads to better survival for the six cancers included in the survey (breast, colorectal, endometrial, myeloma, pancreas and renal), and that GPs had a large amount of influence on timely diagnosis of these cancers. Respondents were less confident about patient management when they did not have qualifying symptoms for urgent referral. GPs conveyed that they can, and have, changed their cancer-related diagnostic activity, and believe that they can influence timely diagnosis. They also identified a number of ways in which future interventions may be targeted to maximise behaviour change.

View this table:
[Table 1.](http://medrxiv.org/content/early/2020/11/23/2020.11.20.20235614/T1)

Table 1. 
Exploring Capability, Opportunity and Motivation for behaviour change COM-B

### Work package 2a: Discrete Choice Experiment (DCE)

Of the 269 survey respondents, 151 fully completed the DCE. Seven attributes and preferences were explored, with five showing statistical significance. Open access to diagnostic tests was ranked as most important, followed by practice level cancer audit. The other statistically significant preferences were computerised prompts, blended online and face to face education sessions and a preference towards patient responsibility in safety netting. The focus of educational continuing professional development and time spent doing it were not ranked as statistically significant attributes. (Supplementary File 5 shows the attributes and attribute levels offered along the logit regression model and detailed DCE data analysis).

### Work package 2b: Qualitative component

The qualitative work undertaken as part of the WICKED programme revealed important insights and rich data relating to current challenges of PC teams dealing with patients who might have cancer. A separate scientific paper is proposed to illustrate the methods and findings in detail. To highlight some common themes with regard to intervention development, participants explained how they go about identifying, investigating and referring patients with potential signs and symptoms of cancer. They described how they address the need to diagnose cancer earlier, and what influences their behaviour, including NICE guidance, clinical judgement, patient factors and ‘gut feeling’.

*“Well, some of them* [the guidelines] *you use every day, so it just becomes*…*something you’re thinking about constantly*…*just part of the everyday decision making*.*”* (Male, urban, medium deprivation, < 10yrs experience)

However, they were also influenced by their knowledge of the pressures within secondary care, and a concern not to add to an already strained system. This created a tension between GPs’ internal, individual decision-making and their knowledge of the context-dependent pressures.

*“A big part of me wants to refer everyone. Yeah, 3% risk, you’re more likely to catch that earlier diagnosis and get a curable disease. But at the same time, you don’t want to completely swamp the system so that no one’s getting it. It is very resource-limited. It does affect my practice significantly*.*”*

(Female, urban, high deprivation,15-25yrs experience)

Exploring perspectives from practice teams during the focus groups, it appears that some of this tension can potentially be offset by positive practice culture and helpful practice-based systems. For example, open door communication systems between staff members and a feeling that the whole practice team is united in a common goal.

*“I think on the phone you don’t see people do you, but at the desk you do. So you notice their weight, or the yellowness or, you know if you think there is something not right, I mention it to one of the doctors*.*”*

(Receptionist, urban area, non-training practice)

#### The novel intervention - ThinkCancer!

Data from all earlier work packages were included to build the content and proposed modes of delivery for a host of potential interventions. The recurring themes of safety netting, use of guidance and lowering referral thresholds were strongly represented with regard to content of the intervention, among other key topics. The most common intervention functions identified through the BCW process were education, training, enablement and environmental restructuring.

The resultant ThinkCancer! Intervention (see Figure 4) is a whole practice-based educational and quality improvement workshop, consisting of themed sessions for both clinical and non-clinical staff, the co-production of a bespoke practice-specific Cancer Safety Netting Plan (CSNP) and the appointment of a Cancer Safety Netting Champion (CSNC). The workshop aims to raise awareness and increase knowledge around current cancer diagnosis guidance and be delivered over half a day during GP protected educational time in the form of face-to-face or remotely delivered web based educational sessions. Sessions will consist of a series of interactive activities exploring existing processes within practices and developing plans for change to implement each component. Members of the research team will deliver the intervention; a member qualified as a GP Educator will oversee the workshop, supported by up to two research team staff members.

![Figure 4.](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2020/11/23/2020.11.20.20235614/F4.medium.gif)

[Figure 4.](http://medrxiv.org/content/early/2020/11/23/2020.11.20.20235614/F4)

Figure 4. 
Summary outline of the ThinkCancer! intervention

### Session 1

This session of the workshop will deliver a face-to-face or remote video linked educational session for all clinical staff at the practice. The overall aim is to raise knowledge and awareness around current guidance, hot topics in early diagnosis and safety netting, including the introduction of a new safety netting tool – the Symptom Safety Netting Action Plan (SSNAP). The educational component of the ThinkCancer! Intervention has been developed in accordance with well-established educational principles and theories such as social cognitive theory and andragogical assumptions.43

The curriculum principle, constructive alignment, was considered in mapping the intended learning outcomes (ILOs) against the learning activities and experiences, along with the assessment and evaluation plan.44 Opportunities for discussion and reflection within the workshops will allow learners to connect any new learning with prior experience,45 and making clear the relevance of the learning programme to daily practice should increase both the engagement and effectiveness of the learning experience. The session will specifically focus around key Intended Learning Outcomes (ILOs) as described below. The ILOs were developed from stakeholder review of data collected from work packages 1 and 2, specifically the programme theory developed from the realist review. This was later further developed and refined in collaboration with RedWhale GP Update™ who are an established organisation delivering PC update courses, including a specific not for profit cancer and palliative care training programme. Hot topics in early diagnosis of cancer and common pitfalls were included in the ILOs from their materials. This collaboration allowed educational and content oversight and peer review. The ILOs for the clinician session comprise the following:

1.  Understand the key points about the latest NICE guidelines;

2.  Consider the recurrent themes in delayed diagnosis;

3.  Be aware of guidance around thrombocytosis and cancer;

4.  Be alert to common pitfalls in cancer detection and the dangers of false negative investigations;

5.  Understand the importance of and techniques considered essential for high quality safety netting, with introduction of the safety-netting prescription;

6.  Use the ThinkCancer! toolkit/manual as an information and signposting resource.

Symptom Safety Netting Action Plan (SSNAP) Tool

The SSNAP tool is a co-designed novel safety netting intervention in the form of a physical leaflet that can be given to a patient that formalises the advice given within a consultation detailing when and how to seek further medical review. Developed by a research team within the Yorkshire and Humber Patient Safety Translational Research Centre, the tool was developed from a mixed-method programme of research culminating in a series of iterative co-design workshops and focus groups including PC teams and patients. The use of the tool within the ThinkCancer! intervention practices and incorporation into their own bespoke safety netting plan is entirely at the discretion of the individual practice.

### Session 2

The non-clinical session will focus on patient-facing staff and their role in the early diagnosis of cancer. The aim is to increase awareness of red flag symptoms (using the Cancer Research UK Be Clear on Cancer Campaign)46 and highlight non-specific but concerning symptoms that patients may complain of or present, that could indicate early presentation of cancer.47 The secondary aim of this workshop is to improve confidence in sharing concerns within the practice team by placing the patient at the centre of all healthcare interactions and empowering staff. The format will be interactive and in two parts. Part 1 will include a gallery of posters on the wall with a walk around and discussion focussed on the Be Clear on Cancer campaigns. Symptom cards will be given that will include red flag symptoms for possible cancer and also other common symptoms reception staff are faced with when interacting with patients. Participants will then be asked to separate the symptom cards into two groups; red flag versus non red flag for cancer symptoms. The format for the matching exercise was conceived by a member of the research team with experience in adult and medical education. The second part will be a case discussion exploring common themes in early presentation of cancer, such as patient behaviours, for example changes in consultation patterns.

### Session 3 – Cancer Safety Netting Plan (CSNP) and Cancer Safety Netting Champion (CSNC)

The aim of this component of the workshop is to facilitate the design and implementation of a practice-specific Cancer Safety Netting Plan (CSNP). This component of the workshop will be attended by both clinical and administrative staff. Information gathered prior to the workshop from a baseline practice questionnaire regarding current safety netting procedures will be discussed and reflected upon in an exercise involving the use of the CRUK safety netting flow diagram (Supplementary File 6). The practice team will consider the practice’s existing safety netting arrangements (if any), and the need for modification, augmentation, or the development of an entirely new plan. Overall leadership and responsibility for implementing the plan will rest with the practice-appointed Cancer Safety Netting Champion (CSNC), who will ideally be identified before the workshop but may be identified during the workshop. The use of a champion to affect positive change in projects and organisations is now well established.48 The CSNC will take ‘ownership’ of the CSNP and be responsible for implementing it within and throughout the practice as well as the encouragement of a supportive environment for staff. The production, implementation, monitoring and evaluation of the plan will follow recognised Quality Improvement (QI) principles.49,50 However, the minimum output of the intervention will be the reflective discussion generated and testing whether the development of a safety netting plan is feasible.

### Logic model

In order to assess and evaluate ThinkCancer!, drawing on the UK MRC framework for evaluation of complex interventions,13 a logic model of the proposed intervention has been created (Figure 5). This will allow the research team to gain a greater understanding of each working part of the intervention by mapping out each individual component of the intervention by its input and activity. Detailing the anticipated outputs and outcomes allow the research team to consider the most effective ways of measuring and recording the effects of the intervention, therefore suggesting primary and secondary outcome measures as featured in this logic model. The current primary outcome measures for ThinkCancer! are Two Week Wait referral rate and Primary Care Interval (PCI), where the assumption is made that raising awareness and providing education will raise referral rates, and improving safety netting and GP practice systems will shorten the PCI, thereby reducing diagnostic delay in the more difficult to diagnose cancers. These measures are expected to best assess effectiveness of the intervention during a definitive phase 3 randomised controlled trial. The secondary outcome measures relate to feasibility criteria as this is a novel complex intervention. Refinement of the logic model from data collected during a proposed feasibility trial will allow iterative development of the intervention.

![Figure 5.](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2020/11/23/2020.11.20.20235614/F5.medium.gif)

[Figure 5.](http://medrxiv.org/content/early/2020/11/23/2020.11.20.20235614/F5)

Figure 5. 
The ThinkCancer! Intervention Logic Model

## Discussion

The WICKED programme involved a rigorous and structured process of intervention development, built on a sound foundation of primary research, by a core team of multi-professional researchers supported by methodological specialists. The process was guided at each stage by expert patient involvement and stakeholders, contributing to the validity and future implementation of the intervention. This research programme was originally set up to improve cancer outcomes in Wales, and has developed an intervention that is relevant across the whole of the United Kingdom and for any other country where health care is structured around primary care as the first point of contact. The landscape of cancer diagnosis is topical and fast changing with research and policy change occuring at a rapid rate. Single pathway referral processes may aid referrers and patients along their diagnostic journies when cancer is suspected, and the introduction of Rapid Diagnostic Centres for pateints prssenting with low risk but not no risk symtpoms are likely to reduced diagnostic intervals. However, if the possibility of a cancer diagnosis is not considered, if the clinician does not Think Cancer!, the patient will not be referred and diagnostic delays continue.

This potential limitation to the validity of the data collected in the early review work has been mitigated by the research team regularly reviewing new published papers and policy against the original findings. The strength of this multidimensional approach applied in the WICKED programme lies in two important factors. Firstly, the deep insight gained within the qualitative work in revealing the attitudes, feelings and behaviours of the clinical staff in relation to real world challenges of cancer diagosis and referral, giving sharp focus to what is most important within the literature to exploit in an intervention to promote timely cancer diagnosis. Secondly, in the use of the BCW to guide the intervention development, all data from each work package was recycled to inform the process, eliminating any “research waste”. This environmentally friendly research method has added to the strength, depth and scientific rigour of the format and delivery of the proposed intervetion. It is difficult to objectively measure the quality of intervention development and reporting but there are limited guidelines, such as The Template for Intervention Description and Replication (TIDiER) checklist.51 ThinkCancer! satisfies all domains relating to initial design prior to testing as described in this check list. A recently published eDelphi consensus aiming to provide more indepth guidance on the reporting of intervention development studies in health research (GUIDED),52 achieved consensus on fourteen reporting items. The intervention development described in this paper satisfies the majority of those reporting items. ThinkCancer! will be tested and iteratively developed within a feasibility trial delivered in Wales in 2020, with an economic and mixed method process evaluation.

## Conclusion

ThinkCancer! is a novel, theory-informed educational and quality improvement practice-level intervention, aiming to improve the early diagnosis of cancer in primary care. The content of the intervention and the way it will be delivered has been the output of a comprehensive multi-method programme of primary research, developed in accordance with the MRC framework.13 This complex intervention will be tested in a randomised controlled feasibility trial, with embedded process and economic evaluation, to iteratively adapt and refine the intervention, assess the usual feasibility criteria and test proposed outcome measures. The findings will go on to inform a definitive large-scale phase 3 randomised control trial.

## Supporting information

Supplementary File 1 [[supplements/235614_file02.docx]](pending:yes)

Supplementary File 2 [[supplements/235614_file03.docx]](pending:yes)

Supplementary File 3 [[supplements/235614_file04.docx]](pending:yes)

Supplementary File 4 [[supplements/235614_file05.docx]](pending:yes)

Supplementary File 5 [[supplements/235614_file06.docx]](pending:yes)

Supplementary File 6 [[supplements/235614_file07.docx]](pending:yes)

## Data Availability

Supplementary files referred to in this manuscript are attached.

## Additional Information

## Authors’ contribution

The authors contributions are as follows: conceiving and designing the study: CW,RDN,NW,RTE, KB, SS, ACS, MAS, NG, ZH, RJL, SN, JH, RL; data gathering and interpretation: CW,RDN, NW, RTE, KB, SS, ACS, MAS, NG, ZH, AH, JLR, RJL, SN, JH, RL, SD, AS; drafting or revising the manuscript CW, RDN, NW, RTE, KB, SS, ACS, MAS, NG, ZH, AH, JLR, RJL, SN, JH, RL, SD, AS.

## Ethics approval and consent to participate

Ethics approval granted by Bangor University (School of Healthcare Science) and NHS R&D permissions granted in al Welsh Health Boards (Integrated Research Application System reference number: 213034).

## Competing Interestss

The authors declare no conflict of interests.

## Funding Information

The WICKED research programme is entirely funded by Cancer Research Wales. RDN is an associate director of the CanTest Collaborative funded by Cancer Research UK (C8640/A23385).

## Acknowledgements

The authors would like to thank our patient representative Jan Rose and all clinical and research colleagues who assisted in the design, delivery and data interpretation of the WICKED research programme. We acknowledge the contributions of Maggie Hendry (programme design and review work), Dr Lynne Williams (realist review), Dr Jane Heyhoe (Yorkshire and Humber Patient Safety Translational Research Centre - intervention development and process evaluation) and Dr Caroline Greene (RedWhale GPUpdate – intervention development and medical education expertise).

*   Received November 20, 2020.
*   Revision received November 20, 2020.
*   Accepted November 23, 2020.


*   © 2020, Posted by Cold Spring Harbor Laboratory

The copyright holder for this pre-print is the author. All rights reserved. The material may not be redistributed, re-used or adapted without the author's permission.

## References

1.  1.Emery JD, Shaw K, Williams B, Mazza D, Fallon-Ferguson J, Varlow M, et al. The role of primary care in early detection and follow-up of cancer. Nat Rev Clin Oncol 2014;11(1):38–48.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1038/nrclinonc.2013.212&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=24247164&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2020%2F11%2F23%2F2020.11.20.20235614.atom) 

2.  2.Rose PW, Rubin G, Perera-Salazar R, Almberg SS, Barisic A, Dawes M, et al. Explaining variation in cancer survival between 11 jurisdictions in the International Cancer Benchmarking Partnership: a primary care vignette survey. BMJ Open 2015;5(5):e007212.
    
    [Abstract/FREE Full Text](http://medrxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NzoiYm1qb3BlbiI7czo1OiJyZXNpZCI7czoxMToiNS81L2UwMDcyMTIiO3M6NDoiYXRvbSI7czo1MDoiL21lZHJ4aXYvZWFybHkvMjAyMC8xMS8yMy8yMDIwLjExLjIwLjIwMjM1NjE0LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

3.  3.CRUK. Cancer Research UK position paper: The diagnostic workforce in Wales. London, 2018.
    
    

4.  4.Richards MA. The size of the prize for earlier diagnosis of cancer in England. Br J Cancer 2009;101 Suppl 2:S125–9.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1038/sj.bjc.6605402&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=19956156&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2020%2F11%2F23%2F2020.11.20.20235614.atom) 
    
    [Web of Science](http://medrxiv.org/lookup/external-ref?access_num=000273523000021&link_type=ISI) 

5.  5.Butler J, Foot C, Bomb M, Hiom S, Coleman M, Bryant H, et al. The International Cancer Benchmarking Partnership: an international collaboration to inform cancer policy in Australia, Canada, Denmark, Norway, Sweden and the United Kingdom. Health Policy 2013;112(1-2):148–55.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1016/j.healthpol.2013.03.021&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=23693117&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2020%2F11%2F23%2F2020.11.20.20235614.atom) 
    
    [Web of Science](http://medrxiv.org/lookup/external-ref?access_num=000326364800018&link_type=ISI) 

6.  6.Rubin G, Berendsen A, Crawford SM, Dommett R, Earle C, Emery J, et al. The expanding role of primary care in cancer control. Lancet Oncol 2015;16(12):1231–72.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1016/S1470-2045(15)00205-3&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=26431866&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2020%2F11%2F23%2F2020.11.20.20235614.atom) 

7.  7.Hamilton W, Walter FM, Rubin G, Neal RD. Improving early diagnosis of symptomatic cancer. Nat Rev Clin Oncol 2016;13(12):740–49.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1038/nrclinonc.2016.109&link_type=DOI) 

8.  8.Nicholson BD, Mant D, Neal RD, Hart N, Hamilton W, Shinkins B, et al. International variation in adherence to referral guidelines for suspected cancer: a secondary analysis of survey data. Br J Gen Pract 2016;66(643):e106–13.
    
    [Abstract/FREE Full Text](http://medrxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NDoiYmpncCI7czo1OiJyZXNpZCI7czoxMToiNjYvNjQzL2UxMDYiO3M6NDoiYXRvbSI7czo1MDoiL21lZHJ4aXYvZWFybHkvMjAyMC8xMS8yMy8yMDIwLjExLjIwLjIwMjM1NjE0LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

9.  9.Wales Cancer Network. Cancer Delivery Plan for Wales 2016-2020. The highest standard of care for everyone with cancer. : Welsh Government, 2016.
    
    

10. 10.Hansen RP, Vedsted P, Sokolowski I, Sondergaard J, Olesen F. Time intervals from first symptom to treatment of cancer: a cohort study of 2,212 newly diagnosed cancer patients. BMC Health Serv Res 2011;11:284.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1186/1472-6963-11-284&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=22027084&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2020%2F11%2F23%2F2020.11.20.20235614.atom) 

11. 11.Allgar VL, Neal RD. Delays in the diagnosis of six cancers: analysis of data from the National Survey of NHS Patients: Cancer. Br J Cancer 2005;92(11):1959–70.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1038/sj.bjc.6602587&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=15870714&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2020%2F11%2F23%2F2020.11.20.20235614.atom) 
    
    [Web of Science](http://medrxiv.org/lookup/external-ref?access_num=000229465400002&link_type=ISI) 

12. 12.Neal RD, Din NU, Hamilton W, Ukoumunne OC, Carter B, Stapley S, et al. Comparison of cancer diagnostic intervals before and after implementation of NICE guidelines: analysis of data from the UK General Practice Research Database. Br J Cancer 2014;110(3):584–92.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1038/bjc.2013.791&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=24366304&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2020%2F11%2F23%2F2020.11.20.20235614.atom) 
    
    [Web of Science](http://medrxiv.org/lookup/external-ref?access_num=000331226400007&link_type=ISI) 

13. 13.. [https://mrc.ukri.org/documents/pdf/complex-interventions-guidance/](https://mrc.ukri.org/documents/pdf/complex-interventions-guidance/)
    
    

14. 14.Stanciu MA, Law RJ, Nafees S, Hendry M, Yeo ST, Hiscock J, et al. Development of an intervention to expedite cancer diagnosis through primary care: a protocol. BJGP Open 2018;2(3). DOI: [https://doi.org/10.3399/bjgpopen18x101595](https://doi.org/10.3399/bjgpopen18x101595).
    
    

15. 15.Weller D, Vedsted P, Rubin G, Walter FM, Emery J, Scott S, et al. The Aarhus statement: improving design and reporting of studies on early cancer diagnosis. Br J Cancer 2012;106(7):1262–7.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1038/bjc.2012.68&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=22415239&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2020%2F11%2F23%2F2020.11.20.20235614.atom) 
    
    [Web of Science](http://medrxiv.org/lookup/external-ref?access_num=000302130700003&link_type=ISI) 

16. 16.CRD. Systematic Reviews: CRD’s Guidance for Undertaking Reviews in Healthcare University of York: CRD, 2009.
    
    

17. 17.Smith V, Devane D, Begley CM, Clarke M. Methodology in conducting a systematic review of systematic reviews of healthcare interventions. BMC Med Res Methodol 2011;11(1):15.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1186/1471-2288-11-15&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=21291558&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2020%2F11%2F23%2F2020.11.20.20235614.atom) 

18. 18.Pawson R. Digging for Nuggets: How ‘Bad’ Research Can Yield ‘Good’ Evidence. International Journal of Social Research Methodology 2006;9(2):127–42.
    
    

19. 19.Rycroft-Malone J, McCormack B, Hutchinson AM, DeCorby K, Bucknall TK, Kent B, et al. Realist synthesis: illustrating the method for implementation research. Implement Sci 2012;7:33.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1186/1748-5908-7-33&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=22515663&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2020%2F11%2F23%2F2020.11.20.20235614.atom) 

20. 20.Pawson R, Greenhalgh T, Harvey G, Walshe K. Realist Synthesis: An Introduction. RMP Methods Paper 2/2004. RMP Methods Paper 2/2004. Manchester, UK: ESRC Methods Programme, Manchester University, 2004.
    
    

21. 21.Wong G, Greenhalgh T, Westhorp G, Buckingham J, Pawson R. RAMESES publication standards: realist syntheses. BMC Med 2013;11:21.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1186/1741-7015-11-21&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=23360677&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2020%2F11%2F23%2F2020.11.20.20235614.atom) 

22. 22.Wong G, Greenhalgh T, Westhorp G, Pawson R. Development of methodological guidance, publication standards and training materials for realist and meta-narrative reviews: the RAMESES (Realist And Meta-narrative Evidence Syntheses – Evolving Standards) project. 2014.
    
    

23. 23.Ryan M, Gerard K. Using discrete choice experiments to value health care programmes: current practice and future research reflections. Appl Health Econ Health Policy 2003;2(1):55–64.
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=14619274&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2020%2F11%2F23%2F2020.11.20.20235614.atom) 

24. 24.Ritchie J, Spencer L. Qualitative data analysis for applied policy research. In: Analyzing qualitative data 2002 Sep 9 (pp. 187-208). Routledge.
    
    

25. 25.Michie S, Atkins L, West R. The Behaviour Change Wheel. A guide to designing interventions. Great Britain: Silverback Publishing, 2014.
    
    

26. 26.Cane J, O’Connor D, Michie S. Validation of the theoretical domains framework for use in behaviour change and implementation research. Implement Sci 2012;7:37.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1186/1748-5908-7-37&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=22530986&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2020%2F11%2F23%2F2020.11.20.20235614.atom) 

27. 27.Abel GA, Friese CR, Magazu LS, Richardson LC, Fernandez ME, De Zengotita JJ, et al. Delays in referral and diagnosis for chronic hematologic malignancies: a literature review. Leuk Lymphoma 2008;49(7):1352–9.
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=18604724&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2020%2F11%2F23%2F2020.11.20.20235614.atom) 

28. 28.Brown S, Castelli M, Hunter DJ, Erskine J, Vedsted P, Foot C, et al. How might healthcare systems influence speed of cancer diagnosis: a narrative review. Soc Sci Med 2014;116:56–63.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1016/j.socscimed.2014.06.030&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=24980792&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2020%2F11%2F23%2F2020.11.20.20235614.atom) 

29. 29.Burge F, Kockelbergh R. Closing the Gender Gap: Can We Improve Bladder Cancer Survival in Women? - A Systematic Review of Diagnosis, Treatment and Outcomes. Urol Int 2016;97(4):373–79.
    
    

30. 30.Carter S, Winslet M. Delay in the presentation of colorectal carcinoma: a review of causation. Int J Colorectal Dis 1998;13(1):27–31.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1007/s003840050127&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=9548097&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2020%2F11%2F23%2F2020.11.20.20235614.atom) 
    
    [Web of Science](http://medrxiv.org/lookup/external-ref?access_num=000072509500006&link_type=ISI) 

31. 31.Gikas A, Triantafillidis JK. The role of primary care physicians in early diagnosis and treatment of chronic gastrointestinal diseases. Int J Gen Med 2014;7:159–73.
    
    

32. 32.Macdonald S, Macleod U, Campbell NC, Weller D, Mitchell E. Systematic review of factors influencing patient and practitioner delay in diagnosis of upper gastrointestinal cancer. Br J Cancer 2006;94(9):1272–80.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1038/sj.bjc.6603089&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=16622459&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2020%2F11%2F23%2F2020.11.20.20235614.atom) 
    
    [Web of Science](http://medrxiv.org/lookup/external-ref?access_num=000237238700011&link_type=ISI) 

33. 33.Mangion D, Brennan A. Relationship between the ‘two-week rule’ and colorectal cancer diagnosis. Br J Nurs 2014;23(12):660–7.
    
    

34. 34.Mansell G, Shapley M, Jordan JL, Jordan K. Interventions to reduce primary care delay in cancer referral: a systematic review. Br J Gen Pract 2011;61(593):e821–35.
    
    [Abstract/FREE Full Text](http://medrxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NDoiYmpncCI7czo1OiJyZXNpZCI7czoxMToiNjEvNTkzL2U4MjEiO3M6NDoiYXRvbSI7czo1MDoiL21lZHJ4aXYvZWFybHkvMjAyMC8xMS8yMy8yMDIwLjExLjIwLjIwMjM1NjE0LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

35. 35.Mitchell E, Macdonald S, Campbell NC, Weller D, Macleod U. Influences on pre-hospital delay in the diagnosis of colorectal cancer: a systematic review. Br J Cancer 2008;98(1):60–70.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1038/sj.bjc.6604096&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=18059401&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2020%2F11%2F23%2F2020.11.20.20235614.atom) 
    
    [Web of Science](http://medrxiv.org/lookup/external-ref?access_num=000252247300012&link_type=ISI) 

36. 36.Mitchell ED, Pickwell-Smith B, Macleod U. Risk factors for emergency presentation with lung and colorectal cancers: a systematic review. BMJ Open 2015;5(4):e006965.
    
    [Abstract/FREE Full Text](http://medrxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NzoiYm1qb3BlbiI7czo1OiJyZXNpZCI7czoxMToiNS80L2UwMDY5NjUiO3M6NDoiYXRvbSI7czo1MDoiL21lZHJ4aXYvZWFybHkvMjAyMC8xMS8yMy8yMDIwLjExLjIwLjIwMjM1NjE0LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

37. 37.Ott JJ, Ullrich A, Miller AB. The importance of early symptom recognition in the context of early detection and cancer survival. Eur J Cancer 2009;45(16):2743–8.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1016/j.ejca.2009.08.009&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=19765977&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2020%2F11%2F23%2F2020.11.20.20235614.atom) 
    
    [Web of Science](http://medrxiv.org/lookup/external-ref?access_num=000272284200008&link_type=ISI) 

38. 38.Schichtel M, Rose PW, Sellers C. Educational interventions for primary healthcare professionals to promote the early diagnosis of cancer: a systematic review. Educ Prim Care 2013;24(4):274–90.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1080/14739879.2013.11494186&link_type=DOI) 

39. 39.Schmidt-Hansen M, Berendse S, Hamilton W. The association between symptoms and bladder or renal tract cancer in primary care: a systematic review. Br J Gen Pract 2015;65(640):e769–75.
    
    [Abstract/FREE Full Text](http://medrxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NDoiYmpncCI7czo1OiJyZXNpZCI7czoxMToiNjUvNjQwL2U3NjkiO3M6NDoiYXRvbSI7czo1MDoiL21lZHJ4aXYvZWFybHkvMjAyMC8xMS8yMy8yMDIwLjExLjIwLjIwMjM1NjE0LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

40. 40.Shapley M, Mansell G, Jordan JL, Jordan KP. Positive predictive values of >/=5% in primary care for cancer: systematic review. Br J Gen Pract 2010;60(578):e366–77.
    
    [Abstract/FREE Full Text](http://medrxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NDoiYmpncCI7czo1OiJyZXNpZCI7czoxMToiNjAvNTc4L2UzNjYiO3M6NDoiYXRvbSI7czo1MDoiL21lZHJ4aXYvZWFybHkvMjAyMC8xMS8yMy8yMDIwLjExLjIwLjIwMjM1NjE0LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

41. 41.Thompson MR, Heath I, Swarbrick ET, Wood LF, Ellis BG. Earlier diagnosis and treatment of symptomatic bowel cancer: can it be achieved and how much will it improve survival? Colorectal Dis 2011;13(1):6–16.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1111/j.1463-1318.2009.01986.x&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=19575744&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2020%2F11%2F23%2F2020.11.20.20235614.atom) 
    
    [Web of Science](http://medrxiv.org/lookup/external-ref?access_num=000285765400002&link_type=ISI) 

42. 42.Williams TG, Cubiella J, Griffin SJ, Walter FM, Usher-Smith JA. Risk prediction models for colorectal cancer in people with symptoms: a systematic review. BMC Gastroenterol 2016;16(1):63.
    
    

43. 43.Mann KV. Theoretical perspectives in medical education: past experience and future possibilities. Med Educ 2011;45(1):60–8.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1111/j.1365-2923.2010.03757.x&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=21155869&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2020%2F11%2F23%2F2020.11.20.20235614.atom) 

44. 44.Biggs J. Teaching for Quality Learning at University. 2 ed. Buckingham: The Society for Research into Higher Education and Open University Press, 2003.
    
    

45. 45.Dornan T, Mann K, Scherpbier A, Spencer J. Medical Education: Theory and Practice. Integrates practice, theory and research.: Churchill Livingstone, Elsevier, 2011.
    
    

46. 46.. [https://www.nhs.uk/be-clear-on-cancer#jjh8yMZAr0C6BsP0.97](https://www.nhs.uk/be-clear-on-cancer#jjh8yMZAr0C6BsP0.97)
    
    

47. 47.Pearson C, Poirier V, Fitzgerald K, et al. Crosssectional study using primary care and cancer registration data to investigate patients with cancer presenting with nonspecific symptoms. BMJ Open 2020;10:e033008. doi:10.1136/bmjopen-2019-033008
    
    [Abstract/FREE Full Text](http://medrxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NzoiYm1qb3BlbiI7czo1OiJyZXNpZCI7czoxMjoiMTAvMS9lMDMzMDA4IjtzOjQ6ImF0b20iO3M6NTA6Ii9tZWRyeGl2L2Vhcmx5LzIwMjAvMTEvMjMvMjAyMC4xMS4yMC4yMDIzNTYxNC5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

48. 48.Shaw EK, Howard J, West DR, Crabtree BF, Nease DE, Jr.., Tutt B, et al. The role of the champion in primary care change efforts: from the State Networks of Colorado Ambulatory Practices and Partners (SNOCAP). J Am Board Fam Med 2012;25(5):676–85.
    
    [Abstract/FREE Full Text](http://medrxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NToiamFiZnAiO3M6NToicmVzaWQiO3M6ODoiMjUvNS82NzYiO3M6NDoiYXRvbSI7czo1MDoiL21lZHJ4aXYvZWFybHkvMjAyMC8xMS8yMy8yMDIwLjExLjIwLjIwMjM1NjE0LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

49. 49.Langley GJ, Moen RD, Nolan KM, Nolan TW, Norman CL, Provost LP. The improvement guide: a practical approach to enhancing organizational performance. John Wiley & Sons; 2009.
    
    

50. 50.Parry GJ, Carson-Stevens A, Luff DF, McPherson ME, Goldmann DA. Recommendations for evaluation of health care improvement initiatives. Academic Pediatrics. 2013 Nov 1;13(6):S23–30.
    
    

51. 51.Hoffmann TC, Glasziou PP, Boutron I, et al. Better reporting of interventions: template for intervention description and replication (TIDieR) checklist and guide. BMJ 2014;348:g1687.
    
    [Abstract/FREE Full Text](http://medrxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiYm1qIjtzOjU6InJlc2lkIjtzOjE3OiIzNDgvbWFyMDdfMy9nMTY4NyI7czo0OiJhdG9tIjtzOjUwOiIvbWVkcnhpdi9lYXJseS8yMDIwLzExLzIzLzIwMjAuMTEuMjAuMjAyMzU2MTQuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

52. 52.Duncan E, O’Cathain A, Rousseau N, et al. Guidance for reporting intervention development studies in health research (GUIDED): an evidence-based consensus study. BMJ Open 2020;10:e033516.
    
    [Abstract/FREE Full Text](http://medrxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NzoiYm1qb3BlbiI7czo1OiJyZXNpZCI7czoxMjoiMTAvNC9lMDMzNTE2IjtzOjQ6ImF0b20iO3M6NTA6Ii9tZWRyeGl2L2Vhcmx5LzIwMjAvMTEvMjMvMjAyMC4xMS4yMC4yMDIzNTYxNC5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=)