Dynamic methods for ongoing assessment of site-level risk in risk-based monitoring of clinical trials: a scoping review

Introduction

Monitoring, a major component of assuring the quality of clinical trials, has traditionally relied on frequent on-site monitoring visits,¹ particularly to facilitate sometimes extensive source data verification (SDV).² However, it is increasingly recognised that this model may be inefficient and unnecessary in many cases,^3,4 with trialists questioning the value of 100% SDV.^5–7 In recent years, regulators^8–10 and trialists^1,11 have proposed a risk-based approach to monitoring, whereby monitoring methods, including the frequency and nature of on-site visits, vary across trials depending on the risks specific to each one.

Risk-based monitoring methods can be applied at different stages of a trial. Pre-trial risk assessments can help define the overarching strategies appropriate to the trial’s risks. In some models,^12,13 this is predominantly a one-off assessment during trial setup. However, it is also possible to modify the monitoring strategy, or build in flexibility and responsiveness, based on changing or emerging risks during the course of the trial.¹⁴

Risk-based monitoring is often associated with fewer on-site visits than ‘traditional’ monitoring.¹⁵ Although effective central monitoring methods alone could, in some respects, provide adequate trial monitoring in place of visits, on-site visits offer particular benefits over central monitoring, for example in any activities that require access to site-held source data (such as patients’ medical notes). On-site visits may also be necessary, for example, to investigate potential fraudulent activity. In a risk-based monitoring framework, visits to sites may not be routine, but can be based on assessed risk; we therefore need methods to assess site-level risk on an ongoing basis. We can interpret these methods as assessing the risk of not going to site now. If the risk seems too high, a visit – or some other corrective action – is triggered. Methods of this kind have been referred to using various terms, including ‘triggered monitoring’¹⁴ or, as in ICH GCP guidance, ‘targeted monitoring’,¹⁶ and may employ data-driven approaches from methods known collectively as ‘central statistical monitoring’,¹⁷ or more subjective assessments.^14,18,19

A recent systematic review has established the breadth of tools available to assess overall trial risk (and to use this assessment to define the monitoring strategy) in the setup stage,²⁰ but so far there has been no such exercise for methods to assess ongoing site-level risk once a trial has started. We conducted a scoping review²¹ to identify and characterise available methods.

Our aims were 1) for trialists, to establish if any published methods were supported by adequate evidence to support implementation in routine practice and 2) for researchers in this area, to consolidate the existing evidence and point towards future developments in this growing field.

Methods

We conducted a scoping review to identify methods for using centrally held clinical trial data to assess site-level risk of deviations from Good Clinical Practice (GCP) or the trial protocol, or research misconduct, and thereby to target sites for further monitoring activity. We chose scoping review methodology as we anticipated finding a variety of methods and study types, and we wanted to characterise the extent, range and nature of research activity.²² There is no published protocol for this scoping review.

Results

Figure 1 gives a PRISMA flow diagram²⁸ showing the different stages of the review. From the various data sources, we ultimately included 30 reports in our final dataset. 21 of these are peer-reviewed publications. The results are characterised in Table 1 and listed in full in Table 2. Figure 2 shows reports by year of publication.

• Download figure
• Open in new tab

Figure 1. PRISMA flow diagram

^a Reasons: no relevant methods presented (n=28); no novel methods presented (e.g. review article; n=28); method to measure variation between trial sites but no ‘flagging’ of sites of concern (n=25); abstract only and not enough detail to confirm relevance (n=10); duplicate or abstract where full paper also available (n=8); grey literature not considered to present reproducible methods (n=5); not about ‘monitoring’ according to ICH Good Clinical Practice definition (n=5); trial-level assessment only, not site-level (n=4); focus on consistency of outcome assessment only (n=4); method from observational study only, not clinical trial (n=1).

• Download figure
• Open in new tab

Figure 2. Publications by year and type

When specific trials were mentioned, they involved various health conditions and were in various geographical settings. Information on trial intervention was not often available, but where it was, methods had most often been used in phase III trials of investigational medicinal products (IMP). The IMP risk category,²⁹ when known, was either ‘licensed and used within its licensed indication’, or ‘licensed and used outside its licensed indication’ (i.e. we found no reports involving trials of unlicensed IMPs).

We classified 20/30 of our results as central statistical monitoring methods, of which 7 focussed on detection of investigator fraud or research misconduct. We classified 9, including one of the 20 that used central statistical monitoring, as ‘triggered monitoring’, i.e. review of each trial site against pre-set thresholds in key performance metrics, usually without any statistical testing. A final 2 we could not fit into either of these categories; these involved using measured site metrics to directly compare sites against one another.^30,31

21/30 reports included some assessment of the effectiveness of the methods; these are summarised in Table 3. The most common experimental designs were to explore the methods’ characteristics using real trial data with no known integrity issues (n=9), and simulating data integrity problems at sites within real trial datasets then using the method to try to identify the problem sites (n=6).

Of the 21 reports, 9 had no information about sites’ ‘true’ status, i.e. whether the problems identified through central monitoring constitute meaningful problems (either recorded through on-site monitoring or audit activity, or known because statuses were created through simulation). One report³² only contained case studies, i.e. partial and selective reporting. Seven^14,33–38 had partial information, e.g. some of sites’ true statuses were reported, but not all. Two explored classification ability through extensive simulation,^39,40 and 2 had detailed information from a limited set of scenarios on the number of true and false positives and negatives.^19,41 The best reported or deducible classification ability for the 11 papers with at least some information on sites’ ‘true’ status is shown in Table 4; in many cases this is limited by considerable amounts of missing or unclear data.

Some papers report on actual or theoretical cost savings from reduced on-site monitoring,^33,42,43 and others comment on the risk of incurring costs if their proposed central monitoring method identifies sites that do not in fact have meaningful problems (i.e. false positives).^19,37 However, no papers give detailed costings for the development, implementation and maintenance of the central monitoring methods themselves.

Discussion

We conducted a scoping review to identify published methods for assessing the risk of not taking corrective action at trial sites at a given time. Although our search looked for reports from any time after 1995, over half of our results are from after 2013, highlighting the recent growth of risk-based monitoring concepts. Host trials for these methods were most often phase III trials of licensed investigational medicinal products, i.e. somewhat lower risk compared to earlier phase trials of newer treatments.²⁹ Around a third of our results were not full, peer-reviewed reports, reflecting a wider problem with evidence supporting trial conduct methods, i.e. that researchers can feasibly produce posters and abstracts for conferences, but may not have time or incentive to produce comprehensive reports.⁴⁴

Identified methods were mainly in two broad categories (with some overlap). Most were about central statistical monitoring, which uses statistical testing of all or a subset of trial data items to compare sites and identify atypical trial centres. A minority described triggered monitoring techniques, whereby sites are assessed against pre-specified site metric threshold rules (usually binary), with sites meeting the greatest number of ‘triggers’ being considered the most concerning. Several authors note that central statistical monitoring needs sufficient overall and per-site sample sizes for adequate statistical power.^17,19,37 Triggered monitoring, however, can be used at any stage of a trial’s recruitment (especially with trigger rules based on single instances of a given protocol violation, for example). We therefore suggest use of the techniques is not mutually exclusive.

Although the reports on central statistical monitoring described a range of methods, there were some commonalities. Previous papers have reviewed these methods in more detail.^{17,41,45–47}

Nearly half of the central statistical monitoring reports focused on identification of data fabrication. The possibility of fraud is a serious concern to trialists and a threat to wider trust in science.⁴⁸ It was possibly an important factor in establishing 100% on-site verification of trial data as a common monitoring approach.^49,50 This may help explain the prevalence of reports about fraud detection, as some may see the priority in risk-based monitoring to be establishing its fraud detection ability compared with 100% source data verification. However, although the incidence of data fraud is difficult to quantify, cases of extensive data fabrication appear rare enough to have individual notoriety.⁵¹ Further, methods to detect fraud are necessarily rather selective, and therefore may not alone be suitable for trialists looking to detect more common, lower-level data integrity issues.

We collected data on how the proposed methods we identified had been evaluated. A number of reports only presented proposed, untested methods, or only selected case studies to demonstrate the methods’ performance. Of those that presented more detailed evaluation, a common limitation was that the ‘true’ status both of identified problem sites and sites apparently not of concern was often not available, or only partially available. It was therefore difficult to know if the ‘concern’ status of sites in central monitoring results represented meaningful problems or not. In addition, a number of studies use simulation to create ‘true’ sites of concern; these raise the additional question of whether these simulations reflect real-life issues, though the involvement of clinicians (i.e. those who would provide real-life trial data) in the simulation process of some reports^19,36 is reassuring.

Clearly, a balance must be struck between minimising cost and adequately mitigating key risks. Many false positives could be expensive (e.g. additional on-site monitoring visits for no gain), but many false negatives could be catastrophic (e.g. missing serious risks to trial participants’ safety). Some authors do acknowledge this, and even suggest that a high false positive rate could be acceptable in a method used to broadly target additional scrutiny at particular sites.³⁸

It is important to recognise the limitations of the available ‘gold-standards’ in the classification of sites. When methods are tested using simulated or real-but-adjusted data, it may be difficult to know how well these accurately recreate real-life situations. When central monitoring methods are tested in real, ongoing trials, on-site monitoring may be an imperfect reference test, in that it may not be able to identify all problems. By contrast, it is clear that central monitoring, with its enhanced inter- and intra-site review, can identify issues that a single team at one site for a limited time might not.⁴⁷

It could be argued that at least some of the methods we have identified do not need extensive evaluation because they prove their own worth. For instance, they help identify outliers that in some cases are self-evidently meaningful problems to resolve. We acknowledge that some central monitoring activities identify ‘known’ problems (e.g. identifying weekend visit dates, which in most cases are unlikely to be correct) and are valuable for data cleaning purposes. However, we were specifically interested in the more nuanced use of these methods to identify sites of ‘concern’, at which monitoring activity may be targeted, and consequently sites ‘not of concern’, monitoring of which may be reduced or omitted. This element does need adequate proof before wider adoption and, if it is shown to be effective, could have significant benefits. In light of the limitations we have described here, we do not believe any methods have yet fully demonstrated that they should be adopted more widely as a means to limit on-site monitoring of sites deemed, based on review of centrally-held data, not to be of concern.

Aside from some comments on the potential cost of investigating false positive central monitoring results,^19,52 the reports we identified contained limited information on the cost of developing and implementing their methods. As well as uncertainty about how to develop relevant methods, uncertainty or concern about costs involved is a substantial barrier to adoption of risk-based monitoring.⁵³

Through our various searches, we identified 24 commercial companies which, we were at least reasonably certain, had a method relevant to our search (data not shown). Only a small proportion of these had published details of their methods in peer-reviewed journals, although just over half had some detail in grey literature sources. This highlights another difficulty in disseminating new trial conduct methods, i.e. that they may be commercially sensitive and, unlike evidence about treatments being tested in trials, there is no compulsion to publish new evidence. It is somewhat contradictory that while risk-based monitoring has come about partly to reduce costs of trials, some risk-based monitoring methods are available only for a fee.

Further work is needed to fully demonstrate the effectiveness of these dynamic site risk assessment methods which, alongside pre-trial risk assessments, form the core of risk-based monitoring. We therefore recommend the following:

1. Coordinate research efforts. From the scoping review and contact with report authors, it was clear that various small research projects relevant to this topic were ongoing, but mostly in isolation. Researchers in this area should take stock of existing research, and set clear priorities to ensure research time is well-spent.

2. Standardise monitoring studies. Core outcome sets⁵⁴ or other mechanisms to standardise studies about monitoring would improve study quality and may facilitate cross-study evidence synthesis.

3. Share evidence. Time, commercial sensitivity and perceived reputational risk could all be barriers to publishing evidence about monitoring practices. However, additional, publicly-available evidence to support best monitoring practice will allow trialists in all settings to adopt new methods with confidence.

4. Publish full papers. Conference abstracts and posters are a useful way to disseminate basic information about new ideas, but rarely have enough detail to allow replication or robustly demonstrate effectiveness. As this emerging field cannot be built on abstracts alone, we encourage researchers to publish full, peer-reviewed papers about their monitoring methods.

5. Look to combine complementary methods. Although work has been done on a number of distinct risk-based monitoring methods, an optimal monitoring plan might involve a combination of these, including both central statistical monitoring and triggered monitoring. A collaborative approach to combining existing methods could help develop and test such an idea.

We acknowledge several limitations. Our database searches identified relevant material from disparate locations, including abstracts in conferences in unrelated research fields. It is possible that other abstract collections include relevant material, but it was not feasible to find and hand-search all of these. Although the internet searches made little contribution to the final list of included reports, they may have been limited by known reproducibility problems.⁵⁵

Scoping review methodology advises that relevant experts in a field are surveyed to help identify other relevant work.⁵⁶ We have not formally done this. We have, however, contacted most authors of included reports for clarifications, and this has not highlighted any additional relevant reports.

Although we pre-specified our aims and our eligibility criteria, we added some detail to our exclusion criteria during the course of our review to help us decide about certain reports. It is possible that other researchers repeating the same review might result in a slightly different list, but we believe this might only affect the ‘method-only’ papers, which do not form the key part of our conclusions. The comprehensive nature of our search strategy, including review of reference and citation lists, gives us confidence that our report is a sound overview of the state of the evidence in this research area.

We have not performed a formal quality assessment of reports we found, however, this is considered by some to be unnecessary in scoping review methodology.²² There is also no validated way to review the quality of risk-based monitoring studies, although we used the QUADAS-2 tool, designed to assess the quality of diagnostic studies, to inform our data collection template.

Finally, we acknowledge that some time has now passed since we first conducted our search for relevant evidence. Conscious of this, we repeated the main database search in 2018 (albeit with only one author conducting title and abstract screening) and added three relevant reports. We are not aware of any research published since then that might change our overall conclusions. If evidence is now available that addresses the limitations we have highlighted in the existing literature, we would certainly consider this a positive development.

Our scoping review highlighted some promising evidence for risk-based monitoring in ongoing trials. However, currently published methods may not yet have demonstrated their efficacy or cost-effectiveness well enough for trialists to implement them with confidence as a means to target or omit on-site visits. A more coordinated, collaborative and transparent approach to developing and sharing evidence in this field, including industry and academic partners, could help it grow beyond its current nascent state, and could contribute to risk- based monitoring more quickly entering routine practice.

Declaration of Conflicting Interests

SS and WC are part of the TEMPER study team, one of the studies included in the final results of this paper. CH and VYE declare no conflict of interest.

Funding

This work was supported by the MRC London Hub for Trial Methodology Research (MC_UU_12023/24). The idea for this work arose from the TEMPER study, which was funded by a grant from Cancer Research UK (C1495/A13305).