A systematic review of the methods used to ensure rigour, transparency and validity in rapid evaluations, rapid appraisals and rapid assessments

1. Search strategy

The search strategy for the review included both keyword and subject heading searches across four scientific literature databases: MEDLINE, Embase, Healthcare Management Information Consortium (HMIC), Cumulative Index of Nursing and Allied Health Literature (CINAHL) Plus, and the open search engine Google Scholar. Search words were related to ‘rapid evaluation’, ‘rapid appraisal’ or ‘rapid assessment’. The searches were conducted between 11–16th August 2022. A detailed outline of the search strategy can be found in Appendix 1.

2. Eligibility criteria

Criteria for including studies consisted of (1) study methods being rapid evaluations, rapid appraisals, rapid assessments, or any variation of these that were listed in the introduction of this paper; (2) the study having been completed within 6 months (in keeping with rapid approaches) (Vindrola-Padros, 2021); and (3) the article including sufficient information on the methods used to ensure rapidity, transparency, validity and rigour. There were no restrictions on publication date, language or study design in terms of being qualitative, quantitative or mixed methods. Despite conducting the majority of the searches on health-related databases, we did not specify our inclusion criteria to health-related fields.

3. Selection process

The search results were imported into EndNote for de-duplication (Gotschall, 2021), and then to Rayyan for further de-duplication and screening (Ouzzani et al., 2016). The title and abstract screening based on the eligibility criteria was split between three researchers due to the large number of search results. The three researchers then cross-checked 10% of each other’s exclusions and discussed any disagreements of exclusion decisions until a consensus on the decision was reached. The inclusion criteria were then refined to ensure all reviewers had the same understanding before the remaining screening was conducted. Following this, one researcher progressed the included articles to full text screening against the eligibility criteria, using Microsoft Excel to report the screening decision. Any articles identified in non-English languages were translated using Google Translate. The principal investigator then cross-checked 10% of the excluded articles.

4. Data extraction and quality assessment

A comprehensive list of data extraction categories was developed based on the research question and pre-specified outcomes of interest. These categories were further refined to ensure information was captured appropriately following screening. Data extraction, quality assessments and full text screening was carried out in parallel by one researcher, recording information on Microsoft Excel. The final data extraction categories included: study details (location of study; study design; study duration; general methods used throughout) and the methods used across the different stages of the rapid studies (as shown in Appendix 2) such as during study design; data collection; data analysis; result interpretation; and dissemination. We also extracted any method limitations outlined in papers.

The Mixed Methods Appraisal Tool (MMAT) (Hong et al., 2018) was used to assess the quality of the included articles because of the heterogenous range of study designs. No articles were excluded based on their MMAT scores.

5. Data synthesis

Textual narrative synthesis and a summary of content was conducted to summarise the study characteristics and key findings across the literature (Lucas et al., 2007). We then used the rigorous and accelerated data reduction technique (RADaR) to reduce the data within our findings (Watkins, 2017).

1. Study selection

The search returned 33,144 results, of which 20,922 articles remained following duplicate removal. Of these, 577 articles were identified as potentially relevant and screened at full text. Overall, 169 articles were included in the review. Figure 1 shows the flow of studies through the review process based on PRISMA guidelines (Page et al., 2021), and the full list of included references can be found in Appendix 3.

• Download figure
• Open in new tab

Figure 1.

PRISMA 2020 Flow Diagram of the Selection Process

The reasons for excluding articles at full text screening can be found in Figure 1. This included articles not sharing enough detail on the methods used to ensure transparency and rigour; the study not fitting under a variation of rapid evaluation or appraisal; the study not being conducted to a rapid timeline (within 6 months); no detail on the study timeline; the study not being based on empirical evidence; the study using review methodology; or the study was a duplicate of a previously screened item.

2. Study characteristics

A detailed summary of the characteristics and themes of all the studies included in the review can be found in Appendix 2.

Most of the included studies were conducted in the USA (n=35), followed by the UK (n=12), South Africa (n=9), Uganda (n=8) and Kenya (n=7), more details on study location can be found in Appendix 2. Rapid assessments were the most common form of rapid studies, followed by rapid appraisals and rapid evaluations, more detail can be found in Table 1 below. A large majority of the rapid approaches were conducted within qualitative study designs.

3. Quality assessment

The MMAT scores of each study can be found in Appendix 2. The studies ranged from low quality with a score of 0/5 (n=3) and 1/5 (n=32) to medium quality with a score of 2/5 (n=38) and 3/5 (n=25), to high quality with a score of 4/5 (n=61) and 5/5 (n=10).

4. Characteristics of rapid studies that ensure rigour and/or transparency

Appendix 2 reports the methods used by each included study, which are also summarised below.

4.1 Frameworks, models and reporting guidelines

There were 29 studies that discussed using frameworks, models, theories or existing methodologies to guide their study design. The most common frameworks and models that were cited were the Consolidated Framework for Implementation Research (n=3) and the Health Belief Model (n=3). The most common methodologies that were used to guide the studies were the International Rapid Assessment Response and Evaluation (I-RARE)/RARE methodology (n=4) and the Rapid Assessment Procedure-Informed Clinical Ethnography (RAPICE) methodology (n=3).

There were only six studies that followed reporting guidelines. The Consolidated criteria for reporting qualitative research (COREQ) guidelines were the most commonly cited (n=4).

4.2 Study design

In terms of designing and setting up the study a number of approaches were reported that enabled transparency, rigour and rapidity of studies. There were 41 publications that specifically mentioned following a study protocol or proposal, and six articles that discussed working with a steering group to guide the study design.

Eight articles highlighted they did not need to subject their study to Institutional Review Committees as the projects were classified as non-research and instead classified as service evaluations, or they worked with online anonymised surveys. One study received informal approvals and comments from the technical team of an Institutional Review Committee. Two studies discussed sharing research plans with the local organisations and areas concerned with the evaluations.

The majority of the included articles (n=91) obtained informed consent from participants ahead of data collection in the form of verbal, written and assumed consent (based on voluntary participation in online surveys). There were two studies that shared that consent was not obtained due to examination of services or observational designs, however it was made clear to participants observations were taking place.

A large majority of the publications discussed using sampling methodologies to either increase rapidity or to improve relevance of characteristics of the sample. The main form of sampling discussed in 84 of the studies was the use of non-probability sampling in the form of convenience sampling, purposive sampling or variations of snowball sampling. Fourteen of the studies discussed how sample size was determined based on when data saturation was reached. Three articles discussed methods used to minimise biases in sampling by searching for alternative or disconfirming cases, deviating from arranged locations to try to reach unsensitized groups, and using active, sequential recruitment.

In terms of recruitment, 22 of the articles discussed having support from local key informants, local researchers/evaluators, local networks and organisations to build trust with potential study participants and to advertise the study. Sixteen of the articles discussed recruiting participants via email or through social networking via WhatsApp or social media.

Some of the articles discussed using incentivisation to improve recruitment. Twenty-three of the articles discussed using monetary incentives either in the form of honorarium or in the form of vouchers. Six of the articles discussed using food and drinks as incentives, and four of the articles discussed health incentivisation through referring participants onto support services and providing condoms. There were five articles that specifically mentioned that their study participants were not incentivised or compensated for their participation.

4.3 Data collection

Methods of data collection that improved transparency, validity and rapidity of the studies included piloting methods, using numerous methods for data collection, collecting data in local languages, working with large experienced multidisciplinary teams of researchers/evaluators, conducting team meetings, using participatory methods and recording, transcribing and developing field notes.

There were 26 publications that discussed piloting and reviewing data collection tools to ensure cultural and face validity. The majority of the publications (n=82) also discussed using numerous methods of data collection in the form of interviews, focus groups, surveys and observations for subsequent triangulation.

A large number (n=66) of the studies conducted research or evaluations in local languages working with translators, local researchers/evaluators and with researchers with lived experience. There were nine studies that discussed how this helped to facilitate trust and strong validity of the cultural and traditional knowledge.

The majority of the studies (n=87) discussed the composition and experience of the evaluation or research team. They discussed whether they worked in large multidisciplinary teams with individuals with diverse experiences and previous experiences in research or evaluation. They also shared whether researchers/evaluators had undergone training or were under the supervision of a senior team member. Twenty-two of the studies discussed conducting team meetings during data collection to ensure good data quality, validity and representativeness of samples and to modify any processes or materials.

There were 14 studies that discussed using participatory methods for data collection such as participating in existing community meetings and using mind maps, matrix ranking and other techniques to elicit information for action, and to generate group discussions allowing for room to correct any misinterpretations.

Twenty-three of the studies discussed using technology for data collection, by conducting interviews and focus groups over video conferencing software, over the phone and over social media platforms like WhatsApp. They also discussed conducting surveys through online platforms and using data from online Twitter forums. The majority of the studies (n=84) also discussed recording and transcribing interviews and reviewing any field notes either in place of transcripts or to enable early analysis of data ahead of availability of transcripts.

4.4 Data analysis

Many of the qualitative studies (n=65) discussed using forms of qualitative content analysis including thematic analysis or the Framework approach both using stepwise approaches to data familiarisation, coding, generating themes and reviewing themes, with the latter method having a focus on using matrices or tables to aid analysis. There were 12 papers that discussed using the ‘coding consensus, co-occurrence, and comparison’ approach or the Grounded Theory approach to openly code transcripts and use the constant comparative method to identify connections between the data. There were 11 studies that discussed specifically using rapid analysis techniques such as rapid assessment procedure sheets. One study discussed using the RADaR technique to reduce data and two studies discussed using mind-mapping techniques to help to visualise the data.

Iterative data collection and data analysis was mentioned across 32 studies. They reported that analysing data whilst data collection was ongoing allowed teams to identify when data saturation had been met, to re-shape data collection tools based on emerging findings and to develop initial codebooks for subsequent in-depth analysis. The method of data triangulation was also commonly used across 58 publications. The methods included triangulating data from different subgroups of participants that data were collected from, data from different data collection methods such as surveys and interviews, and data from existing secondary data sources such as historical records and existing statistical data.

There were 42 studies that shared their team had meetings to compare any emerging findings, discuss any discrepancies in analysis and to reach a consensus on the analysis. Two of the studies discussed using field notes of the team members analysis as an audit trail. Twenty-two of the studies discussed using quality assurance methods during analysis to assess consistency in analysis across team members and to assess the quality of the analysed data.

4.5 Result interpretation

Some of the studies went on to discuss how teams interpreted study results, with four of the studies sharing that they conducted team meetings to discuss findings and reach consensus on how best to interpret them. Six of the studies shared that their team underwent a form of reflexive thinking to reflect on their practice throughout the study and how their personal characteristics may have affected their interpretations of the results. There were three studies that shared that they had a peer-review team that reviewed the research/evaluation teams’ interpretations and the draft reports to identify any potential biases.

Many of the studies (n=27) discussed using member checking to validate the findings from the study with the study participants themselves or with key informants from the study. Seven studies also shared that advisory groups separate to the evaluation/research team supported with interpreting interim and initial findings.

4.6 Dissemination

Using iterative dissemination of findings whilst studies were ongoing was discussed in three studies to rapidly share emerging findings with key stakeholders. The studies also shared that this process allowed feedback loops to be created between stakeholders implementing findings and the evaluation/research team, enabling the evaluation of studies or programs to take place as they are being implemented.

5.1 Limitations encountered during study design

In terms of study design, the majority of publications identified having small sample sizes as a limitation, which led to limited generalisability and representativeness of findings. Also mentioned was the introduction of selection bias of participants and limited generalisability of findings because of non-probability sampling. Some of the publications also shared they found it difficult to set up advisory groups and working groups if connections with the groups were not already in place.

5.2 Limitations encountered during data collection

Some of the publications discussed not having enough time to train staff in the methods they had planned to use, or to pre-test the data collection instruments, which affected consistency in data collection between team members. Using mind maps as a form of data collection was identified as a limitation as it did not allow enough time for participants to reflect on the data they had shared, and it could be heavily impacted by evaluator/researcher bias. Some of the studies were unable to audio record interviews due to privacy concerns which affected the accuracy of findings. Using online data collection made it difficult for some studies to determine the non-response rate, sampling frame, and could have led to selection bias of participants that could access and use online technology. Conducting interviews over the phone or through teleconferencing software was thought to affect how open respondents were to questions. Some studies identified that having team members that did not speak the local language affected interpretations of findings and that conducting research or evaluations in a foreign language to the study participants, affected how detailed responses were. The studies also identified that some contextual validity was lost following translation. It was frequently reported that a form of response bias may have occurred based on participants feeling uncomfortable around the type of researcher/evaluator for example based on their gender, age or whether a local researcher/evaluator who they may know was collecting their data. Similarly, response bias was flagged as a potential limitation depending on the location of data collection, for instance in a workplace.

5.3 Limitations encountered during data analysis

Some of the studies mentioned the limitation of being unable to triangulate findings with other sources to corroborate self-reported responses from participants that may have been prone to recall or response bias. Conducting interim analysis was also identified as a limitation as it made it difficult to attribute findings to a particular collection instrument or participant. The issue of over-representing or under-representing themes during analysis was also reported, as were the difficulties in auditing analysis when many decisions around coding had taken place during group discussions rather than during physical coding.

5.4 Limitations encountered during the result interpretation and within team dynamics

Aspects of the team were discussed within the studies as limitations, including team member characteristics and experiences affecting interpretations of findings. Additionally, the difficulty and safety risks of the team when accessing study sites were identified as a limitation, as was the impact of stressful rapid timelines on team member wellbeing. Finally, it was reported in some of the studies that they had under-estimated their resource requirements so did not have enough team members to plan or conduct the study.