Methods used to select results to include in meta-analyses of nutrition research: a meta-research study ======================================================================================================= * Raju Kanukula * Joanne E McKenzie * Lisa Bero * Zhaoli Dai * Sally McDonald * Cynthia M Kroeger * Elizabeth Korevaar * Matthew J Page ## Abstract **Objectives** To investigate the extent of multiplicity of results in study reports of nutrition research, and the methods specified in systematic reviews to select results for inclusion in meta-analyses. **Methods** MEDLINE and Epistemonikos were searched (January 2018 – June 2019) to identify systematic reviews with meta-analysis of the association between food/diet and health-related outcomes. A random sample of these reviews was selected, and for the first presented (‘index’) meta-analysis, rules used to select effect estimates to include in this meta-analysis were extracted from the reviews and their protocols. All effect estimates from the primary studies that were eligible for inclusion in the index meta-analyses were extracted. **Results** Forty-two systematic reviews were included, 14 of which had a protocol. In 29% of review protocols and 69% of reviews, at least one decision rule to select effect estimates when multiple were available was specified. In 69% (204/325) of studies included in the index meta-analyses, there was at least one type of multiplicity of results. **Conclusions** Authors of systematic reviews of nutrition research should anticipate encountering multiplicity of results in the included primary studies. Specification of methods to handle multiplicity when designing reviews is therefore recommended. #### What is new? ##### Key Findings **Key Findings** * Authors of systematic reviews of nutrition research should anticipate encountering multiplicity of results in the included primary studies. * Decision rules to select results for inclusion in meta-analyses of nutrition research were infrequently pre-specified. ##### What this adds to what was known? **What this adds to what was known?** * Previous studies have found that multiplicity of results of continuous outcomes in studies included in systematic reviews was common, and methods used to select results to include in meta-analyses were infrequently pre-specified in systematic review protocols. However, none of the studies examined meta-analyses in nutiriton research, inclusion of randomized or non-randomized studies, or where the outcome was non-continuous (e.g. binary, count or time-to-event); circumstances for which different forms of multiplicity might arise. Our study addressed this gap. ##### What is the implication and what should change now? **What is the implication and what should change now?** * Pre-specification of decision rules to handle multiplicity when designing reviews is recommended. In the systematic review, we recommend reporting any modifications to the specified rules, or any additions that were introduced to cover multiplicity scenarios that had not been anticipated when designing the review. Key words * Meta-analysis * Systematic review * Nutrition * Multiplicity * Eligibility criteria * Decision rule ## 1. Background The Global Burden of Disease study 2019 reported that diet has a significant impact on health outcomes. Diet quality was found to be the fifth leading risk factor for disability adjusted life years [1]. Large and long-term prospective observational studies and short-term clinical trials have found associations between a particular dietary factor and non-communicable diseases [2-4]. Systematic reviews (SRs) based on such studies are being used to inform recommendations in dietary guidelines [5-7]. However, flaws in the design, conduct and reporting of SRs may yield misleading results, and in turn, misinform guideline recommendations [8]. One challenge SR authors often face is a multiplicity of results in the primary studies [9]. For example, a study report may present multiple effect estimates for the association between red meat consumption and gout, where these estimates may arise from the fitting of multiple statistical models with different outcome definitions of gout, different exposure levels, or where adjustment is made for different sets of confounders. While inclusion of multiple effect estimates from a particular study in a meta-analysis is possible (using methods that adjust for statistical dependency) [10], more commonly only one of the available effect estimates is selected for inclusion. There are various methods that can be used to select a single effect estimate [9, 10]. However, when this selection is based on the statistical significance, magnitude or direction of effect, this may introduce bias into the meta-analysis effect estimate [11]. We refer to this selection process as ‘selective inclusion of results’. To help mitigate the potential for selective inclusion of results, it has been recommended that methods for dealing with multiplicity should be pre-specificied [11]. This includes pre-specification of “eligibility criteria” for each meta-analysis, indicating which results are eligible for inclusion in the meta-analysis (e.g. intervention groups, measurement instruments, time points), and “decision rules”, which specify the methods that will be used to select a single result from a study when multiple are eligible for inclusion in the same meta-analysis (see Box 1 for examples of eligibility criteria and decision rules). Box 1. **Examples of eligibility criteria and decision rules to select results** *Example of eligibility criteria to select results:* Systematic reviewers state that only study effect estimates that were adjusted for sex and age would be included in a meta-analysis of the association between fruit consumption and coronary heart disease. *Example of a decision rule to select results when multiple are available:* Systematic reviewers state that if multiple effect estimates quantifying the association between different levels of fruit consumption and coronary heat disease were available, as would arise if intake was categorised into quartiles in a study report, only the contrast between the highest (e.g. quartile 4) and lowest (e.g. quartile 1) intake would be included in the meta-analysis. Previous research has examined the extent of multiplicity of results, and the methods used to select results for inclusion in the meta-analysis [11-13]. These studies have focused on a range of conditions and examined multiplicity in meta-analyses of randomised trials with continuous outcomes. All studies found that multiplicity of results was common, and Page et al.[9] and Tendal et al. [12] found that specification of methods to select results for inclusion were rarely reported. However, none of the studies examined meta-analyses in nutiriton research, inclusion of randomized or non-randomized studies, or where the outcome was non-continuous (e.g. binary, count or time-to-event); circumstances for which different forms of multiplicity might arise. Therefore, we aimed to address this gap and investigate the i) extent of multiplicity of results in sudy reports of nutrition research, and ii) the methods specified in systematic reviews to select results for inclusion in meta-analyses of all outcome types, including randomized or non-randomized study designs. ## 2. Methods The study protocol has been published [14]. Here, we provide an overview of the methods, with modifications to the protocol reported in Supplementary Table 1. ### 2.1. Eligibility criteria, search and selection of SRs SRs that satisfied the definition of an SR, as outlined in the 2019 edition of the Cochrane Handbook for Systematic Reviews of Interventions [15], and that had explicitly stated methods of study identification (e.g. a search strategy) and of study selection (e.g. eligibility criteria and selection process), and included a meta-analysis of study results, were eligible for inclusion in this study. We included such SRs with meta-analysis that: * included studies that enrolled, regardless of their age and background, (a) people who were generally healthy, (b) a mixture of generally healthy people and people with diet-related risk factors (e.g. overweight, high blood pressure) or a particular health condition (e.g. type II diabetes or cardiovascular disease), or (c) people with non-specified health status; * included randomized trials or non-randomized studies that assessed the effects of at least one type of food (e.g. eggs, fish) or at least one food-defined dietary pattern (e.g. Mediterranean diet) on any continuous (e.g. systolic blood pressure) or non-continuous (e.g. gout incidence) health-related outcome; * were published between 1 January 2018 and 30 June 2019; * were written in English; * provided citations for all included studies in the SR, and; * presented the summary statistics or effect estimate and its precision (e.g. standard error or 95% confidence interval) for each included study, and the meta-analytic summary effect estimate and its precision in the text or forest plot, for at least one meta-analysis of a continuous or non-continuous outcome. We excluded * SRs that did not include any meta-analysis of a non-continuous or continuous outcome; * meta-analyses or pooled analyses of studies conducted outside the context of a SR; * SRs that only focused on nutrient-specific associations with outcomes (e.g. examining the effects of single nutrients such as folic acid, salt); * SRs that included studies enrolling only participants with a health condition, who were obese or who were frail or elderly people at risk of malnutrition, and; * SRs that were co-authored by any of our research team members. We searched for eligible SRs indexed in the PubMed and Epistemonikos [16] databases from 1 January 2018 to 30 June 2019 (search strategies reported in supplement 2). The search results were exported into Microsoft Excel, all duplicate records were removed, and the remaining records were randomly sorted. In the piloting phase, four investigators (MJP, CMK, ZD and SM) independently assessed 50 abstracts against the inclusion criteria (rating each as ‘Eligible’, ‘Ineligible’, or ‘Unsure’), discussed any discrepancies, and made any necessary changes to the screening form. Following piloting, two investigators (MJP and one of CMK, ZD or SM) independently screened titles and abstracts of 450 records. Two investigators (MJP and one of CMK, ZD or SM) then independently assessed the full text of records that were rated as ‘Eligible’ or ‘Unsure’ against the eligibility criteria. This screening process was repeated (in batches of 500 records) until we reached the target sample of 50 SRs, including 25 meta-analyses of continuous and 25 meta-analyses of non-continuous outcomes. If the total number of eligible SRs exceeded this target at the end of a batch, we planned to randomly sample 25 SRs of each type. Any discrepancies in screening decisions at each stage were resolved via discussion between investigators, or by consultation with another investigator (JM) where necessary. For each included SR, we retrieved the published protocol for the SR or registration record (e.g. PROSPERO record), if available, as cited or reported in the SR. From each SR meeting the inclusion criteria, one investigator (MJP) selected one meta-analysis for inclusion. The selected meta-analysis was the first meta-analytic result mentioned in the review (with no restrictions on its placement in the manuscipt); we refer to the selected meta-analysis as the “index meta-analysis”. We initially selected an index meta-analysis regardless of the outcome domain (e.g. weight, bladder cancer), effect measure (e.g. odds ratio, standardised mean difference), meta-analytic model and number and type of included studies (i.e. randomized or non-randomized study). However, following selection of 50 index meta-analyses, we decided to restrict our inclusion to only those meta-analyses including fewer than 20 studies, for reasons of feasibility. For each included index meta-analysis, we retrieved the reports of all included studies (see the protocol for further details [14]). ### 2.2 Data collection and management A data collection form was developed in REDCap (see Supplementary Table S2) [17]. Seven investigators (RK, ZD, SM, CMK, EK, LB and MJP) piloted the form on two randomly selected meta-analyses and their included studies. Discrepancies were discussed, and we modified the form accordingly. Following piloting, two investigators (RK and one of ZD, SM, CMK, LK, LB and MJP) independently collected data from a random sample of half of the index meta-analyses and their included studies, while one investigator (RK) collected data on the remaining index meta-analyses and their included studies. Any discrepancies were resolved through discussions between two investigators or adjudication by a third investigator (JM) if necessary. An overview of the data items, and the sources these were obtained from (i.e. systematic review protocol, systematic review or study report), is presented in Table 1; further details are available in the protocol [14]. In the case of data extracted from the reports of studies included in the index meta-analysis, we extracted all outcome data that were eligible for inclusion in the index meta-analysis. This was determined by the eligibility criteria and decision rules stated in the SR protocol if available, and if not available, the SR, and in combination with the comparison and outcome of the meta-analysis. For example, if the systematic reviewers pre-defined in the SR protocol that the eligible intervention and comparator for the meta-analysis of weight gain was “highest versus lowest intake of dairy products”, and pre-defined a decision rule stipulating that they would consider only data at 12 weeks follow-up when data were available at multiple time points, we only extracted data for that comparison and time point, regardless of whether study reports had data for other time points and other comparisons for the same outcome. In the absence of an SR protocol, we assumed that no eligibility criteria and decision rules were pre-specified (‘worst-case scenario’ assumption) and extracted all study outcome data based on how the outcome was specified in the SR. For example, if the systematic reviewers did not state in a protocol which results should be selected when multiple were available, yet defined the meta-analysis as “effect of dairy intake on weight at 6 months”, we extracted all data on weight at 6 months, regardless of the level of intake of dairy, whether results were unadjusted or adjusted, or what analysis sample was used. View this table: [Table 1:](http://medrxiv.org/content/early/2021/08/07/2021.08.05.21261680/T1) Table 1: Data sources and data items (see protocol for further details) [14] ### 2.3 Data analysis We used descriptive statistics to summarise the characteristics of SR protocols, SRs, index meta-analyses and included studies. For categorical variables, we present frequencies and percentages. For continuous variables, we report medians (with interquartile ranges [IQR]). We computed the frequencies and percentages for different types of eligibility criteria and decision rules used to select results, differences in eligibility criteria or decision rules between the SR protocol and the SR, and studies with different types of multiplicity of results. We also calculated risk differences (with 95% confidence intervals [CIs]) to examine whether the percentages of different types of eligibility criteria and decision rules and studies with different types of multiplicity of results, differed between meta-analyses of continuous outcomes and of non-continuous outcomes. Risk differences were calculated using the method of Mee with the Miettinen and Nurminen modification. Analyses were undertaken using the statistical packages Stata (College Station, Tx), except for the calculaton of risk differences, which was undertaken using the library PropCIs [18] in R (Vienna, Austria) [19]. ## 3. Results ### 3.1 Results of search and screening Our search yielded a total of 7,167 references from the PubMed and Epistemonikos databases (Figure 1). After removing duplicates (n=908), we screened the titles and abstracts of 3,013 randomly sorted references, of which 2,777 were excluded, leaving 236 for full-text screening. Of these, 99 SRs met the inclusion criteria, including 25 SRs with a meta-analysis of a continuous outcome and 74 with meta-analysis of a non-continuous outcome. Initially, all SRs with a continuous outcomes were included, and 25 of the SRs with a non-continuous outcome were randomly selected. From these 50 SRs, eight were excluded (6 continuous, 2 non-continuous), because the index meta-analysis had 20 or more studies, leaving 42 included SRs [20-61]. ![Figure 1.](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2021/08/07/2021.08.05.21261680/F1.medium.gif) [Figure 1.](http://medrxiv.org/content/early/2021/08/07/2021.08.05.21261680/F1) Figure 1. PRISMA 2020 flow diagram of identification, screening and inclusion of systematic reviews. *Of the 6259 unique titles and abstracts, we only needed to screen 3,013 randomly sorted titles and abstracts to reach our target sample size. **We initially drew a random sample of 50 systematic reviews, but post-hoc excluded eight systematic reviews with 20 or more included studies in the index meta-analysis to reduce workload. ### 3.2 Characteristics of the included systematic reviews and their index meta-analyses Of the 42 SRs, 14 had accessible protocols (2 were published protocols and 12 were PROSPERO records) (Table 2). In most index meta-analyses the population was unclear (45%, 19/42), 33% (14/42) included only healthy participants and the remainder (22%, 9/42) included a mix of healthy people and people with a health condition. The majority of index meta-analyses included only non-randomized studiess (62%, 26/42), 31% (13/42) included only randomized trials and the remaining 7% included studies of both designs (Table 3). The primacy of the outcome was not identified in most reviews (81%, 34/42). In nearly all meta-analyses, a random-effects model was fitted (90%, 38/42). The 42 index meta-analyses included a total of 325 studies, with a median of seven studies (IQR 5-11; range 2-17) per meta-analysis. View this table: [Table 2:](http://medrxiv.org/content/early/2021/08/07/2021.08.05.21261680/T2) Table 2: Characteristics of the systematic reviews (N=42) View this table: [Table 3:](http://medrxiv.org/content/early/2021/08/07/2021.08.05.21261680/T3) Table 3: Characteristics of index meta-analyses (N=42) ### 3.3 Eligibility criteria and decision rules reported in SR protocols Of the SR protocols (n=14), all included at least one eligibility criterion, and four (29%) reported at least one decision rule to select results (Table 4). Almost all protocols specified eligibility criteria based on interventions/exposures (93%, 13/14) (e.g.specifying which foods or dietary patterns were eligible), but few other types of eligibility criteria were specified (e.g. time points [29%, 4/14], information sources [13%, 3/14]). The most commonly pre-specified decision rule was based on interventions/exposures (reported in 3 of the 4 SR protocols with at least one decision rule to select results). See Supplementary Table S3 for the content of the decision rules. View this table: [Table 4.](http://medrxiv.org/content/early/2021/08/07/2021.08.05.21261680/T4) Table 4. Number of systematic review protocols or registration entries reporting eligibility criteria and decision rules to select results (N=14) ### 3.4 Eligibility criteria and decision rules reported in SRs Of the SRs (n = 42), all included at least one eligibility criterion and 69% reported at least one decision rule to select results (Table 5). Similar to the SR protocols, the most commonly reported eligibility criteria (95%, 40/42) and decision rule (40%, 17/42) in the SRs was based on interventions/exposures. Eligibility criteria and decision rules for the type of analysis were more freqently specified in SRs as compared with their protocols. The most commonly reported decision rules for analyses were rules to select from multiple unadjusted and covariate-adjusted analyses (24%) (Table 5). There were some discrepancies observed between SR protocols and their published SRs, with the most common type being the addition of a new decision rule to deal with multiple unadjusted and covariate-adjusted analyses in the included studies (Supplementary Table S4). View this table: [Table 5.](http://medrxiv.org/content/early/2021/08/07/2021.08.05.21261680/T5) Table 5. Number of systematic reviews reporting eligibility criteria and decision rules to select results The percentage of reviews specifying different types of eligibility criteria and decision rules generally did not differ by outcome type. However, a larger percentage of SRs with an index meta-analysis of a continuous outcome specified eligibility criteria for any type of analysis compared with SRs with a non-continuous outcome (37% vs 17%; risk difference [RD] 20%, 95% CI -7% to 45%). Conversley, a smaller percentage of SRs with an index meta-analysis of a continuous outcome specified a rule for selecting an adjusted/unadjusted result compared with SRs of a non-continuous outcome (11% vs 35%; RD -24%, 95% CI -47% to 2%). ### 3.5 Multiplicity of results in included studies Of the 325 studies, 296 studies were included in reviews (n=38) without any prespecified decision rules to select results to include in meta-analyses (Table 6). These reviews, and the studies within, are therefore used to estimate the extent of multiplicity of results that can be expected when no decision rules to select results are predefined. The median (IQR) number of available effect estimates per study was 2 (1 to 3), and the largest number of effect estimates in a study was 41 [62]. The most common types of multiplicity arose from multiple unadjusted and one or more covariate-adjusted analyses (which occurred in 40% of the included studies), followed by multiple intervention/control groups (24%). The least common types of multiplicity arose from multiple instruments (0%). The studies with continuous outcomes had less multiplicity than the studies with non-continuous outcomes (54% vs 80%; RD -26%, 95% CI -40%, -12%). View this table: [Table 6:](http://medrxiv.org/content/early/2021/08/07/2021.08.05.21261680/T6) Table 6: **Number of studies with different types of multiplicity of results in systematic reviews without prespecified methods to select results (N=38)** ## 4. Discussion Our findings show that decision rules to select results were infrequently pre-specified in the protocols of a randomly selected sample of SRs with meta-analyses of nutrition research. Multiplicity of results in the primary studies included in the index meta-analyses was very common, with 69% having at least one type of multiplicity. The frequency and types of multiplicity in the included studies varied, arising from multiple intervention groups, time points, analyses and subgroups. ### Comparison with previous research The findings of our study are in line with previously published studies, which have observed incomplete pre-specification of SR methods such as eligibility criteria, methods for collecting, handling and analysing data, and pre-specification of eligibility criteria and decision rules in PROSPERO records [63], Cochrane protocols [9, 64] and published SRs [9, 65]. Two previous studies, on which the methodology of the present study is based, examined methods used to select results for inclusion in meta-analyses [12] [9]. Page et al. [9] examined 44 SRs,, nearly half of which were Cochrane reviews, and they included only randomised trials. In Page et al. [9] and our study, all protocols reported at least one eligibility criteria, but at least one decision rule was pre-specified more in the SR protocols included in Page et al. [9] than those in our study (81% vs 29%). The frequency of discrepancies between the SR protocol and SR in the eligibility criteria or decision rules to select results was higher in our study. Tendal et al. [12] examined eighteen Cochrane reviews, all of which had protocols. Eight of the protocols mentioned eligible time points or periods, but only one provided decision rules to handle multiplicity of time points. Interestingly, all of the 18 protocols reported eligibility criteria for the control group, but none reported decision rules to handle multiple control groups in included studies. Three studies (Tendal et al.[12], Mayo-Wilson et al. [13] and Page et al.[9]) assessed the multiplicity of results among the included studies of SRs. Compared to our study, Page et al. [9] and Tendal et al. [12] had fewer studies with multiple estimates that were available for inclusion in a particular meta-analysis. Multiplicity arising from multiple intervention/control groups was slightly less frequent in our study compared to Tendal et al. [12] (24% vs 29%) but was more frequent in ours when compared with Page et al. [9] (24% vs 17%). Our study findings showed less multiplicity in terms of time points (5%) and measurement instruments for outcomes (0%) compared to previous studies [9, 12]. However, we observed greater multiplicity (40%) due to unadjusted and one or more covariate-adjusted analyses. This difference was likely driven by inclusion of non-randomized studies in the present study, which often require adjustment for covariates to reduce risk of bias due to confounding [66]. Similarly, Mayo-Wilson et al. [13] assessed multiplicity in the clinical trials of publicly accessible reports and non-public reports related to gabapentin for neuropathic pain (n=21) and quetiapine for bipolar depression (n=7). In 15/21 (71%) gabapentin trials and 7/7 (100%) quetiapine trials, there was multiplicity of results. ### Strengths and limitations The major strength of our study is that we have pre-specified methods to identify, select and collect data from eligible SRs and studies, and provided the modifications/deviations to our study protocol (supplementary Table S1). We also used extensive search strategies to identify SRs in nutrition research. In addition, all the study authors, who had different levels of expertise, undertook training and pilot-testing of data collection forms. Moreover, given we randomly selected the SRs, our findings are generalisable to SRs meeting this study’s eligibility criteria. A limitation of our study is that we only retrieved reports of studies included in the index meta-analyses that the systematic reviewers cited. It is possible that other papers relating to the studies exist, which contain additional results that are compatible with the index meta-analysis (e.g. results of cohort studies or randomized trials at later time points may have been presented in other papers which were not cited by the systematic reviewers). For this reason, we may have underestimated the true extent of multiplicity of results within studies. We were unable to translate and interpret the data from four included non-English language studies however, their absence is unlikely to have modified the observed extent of multiplicity. Finally, we only searched PubMed and Epistemonikos to find all SRs and included SRs written in English, so generalisation of our study findings to non-indexed SRs and non-English language SRs is potentially limited. ### Implications of this research for practice Our study, in common with previous research ([9, 12, 13]), demonstrates that multiplicity of effect estimates in primary studies is very common, and is therefore an issue that systematic reviwers should prepare for when designing their review. Doing so will have multiple benefits. It will reduce post-hoc decision making, and in doing-so, provide greater assurance to a reader that the results have not been ‘cherry-picked’ for inclusion in the meta-analyses. Furthermore, specification of eligibility criteria and decision rules is likely to lessen the data extraction effort, requiring less information to be extracted from each primary study. Our results suggest that in nutrition research, specification of eligibility criteria and decision rules to select results from among multiple unadjusted or covariate-adjusted analyses and overlapping samples of participants and different subgroups are most important to pre-specify in SRs that include non-randomised studies. On the other hand, methods used to select results arising from both final values and change from baseline values, and multiple time points are most important to pre-specify in SRs that include randomized trials with continuous outcomes. Furthermore, in the SR, we recommend reporting any modifications to the specified rules, or any additions that were introduced to cover multiplicity scenarios that had not been anticipated when designing the review. The Cochrane Handbook for Systematic Reviews of Interventions version 6.1 [67] provides updated guidance to systematic reviewers about how to group interventions with multiple components or co-interventions or how to select from multiple comparisons and handle multiplicity of outcomes when conducting meta-analyses. In addition, the recently updated “Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) statement” [68] includes a new item (10a), which recommends authors “List and define all outcomes for which data were sought. Specify whether all results that were compatible with each outcome domain in each study were sought (e.g. for all measures, time points, analyses), and if not, the methods used to decide which results to collect”. Implementation of recommendations from these sources will allow readers to understand the result selection process. ## 5. Conclusion Our study found that in systematic reviews examining the effects of foods and diet, that multiplicity of results in the included primary studies was very common. Yet, pre-specification of decision rules to select from multiple results was not common. Systematic reviewers are encouraged to consider methods for dealing with multiplicity when designing their reviews. Doing so will limit the potential for selective inclusion of results, thus providing greater assurance to readers as to the trustworthiness of the review. ## Supporting information Supplementary [[supplements/261680_file02.pdf]](pending:yes) ## Data Availability Data are available upon request from the corresponding author. ## DECLARATIONS ### Funding This project was funded by an Australian National Health and Medical Research Council (NHMRC) project grant (APP1139997). RK is supported by a Monash Graduate Scholarship and a Monash International Tuition Scholarship. MJP is supported by an Australian Research Council Discovery Early Career Researcher Award (DE200101618). JEM is supported by an Australian NHMRC Career Development Fellowship (1143429). SM is supported by the Country Women’s Association (NSW) and Edna Winifred Blackman Postgraduate Research Scholarship. The funders had no role in the study design, data collection and analysis, or preparation of the manuscript. ### Conflicts of interest All authors have completed the ICMJE uniform disclosure form at [www.icmje.org/coi\_disclosure.pdf](http://www.icmje.org/coi_disclosure.pdf) and have no conflicts of interest to declare. ### Contributors All authors declare to meet the ICMJE conditions for authorship. MJP and JEM conceived the project. MJP, JEM, LB, ZD, SM, and CMK contributed to the design of the project. MJP, ZD, SM, and CMK screened articles for inclusion. RK, MJP, LB, ZD, SM, CMK, and EK collected data. RK analysed the data. RK wrote the first draft of the manuscript, which was revised in conjunction with MJP and JEM. JEM drafted sections of the manuscript. All authors were involved in revising the article critically for important intellectual content. All authors approved the final version of the article. MJP is the guarantor of this work. * Received August 5, 2021. * Revision received August 5, 2021. * Accepted August 7, 2021. * © 2021, Posted by Cold Spring Harbor Laboratory This pre-print is available under a Creative Commons License (Attribution 4.0 International), CC BY 4.0, as described at [http://creativecommons.org/licenses/by/4.0/](http://creativecommons.org/licenses/by/4.0/) ## References 1. [1].Global burden of 87 risk factors in 204 countries and territories, 1990-2019: a systematic analysis for the Global Burden of Disease Study 2019. Lancet. 2020;396:1223–49. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1016/S0140-6736(20)30752-2&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2021%2F08%2F07%2F2021.08.05.21261680.atom) 2. [2].Estruch R, Ros E, Salas-Salvadó J, Covas MI, Corella D, Arós F, et al. Primary Prevention of Cardiovascular Disease with a Mediterranean Diet Supplemented with Extra-Virgin Olive Oil or Nuts. N Engl J Med. 2018;378:e34. [PubMed](http://medrxiv.org/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2021%2F08%2F07%2F2021.08.05.21261680.atom) 3. [3].Taylor MK, Mahnken JD, Sullivan DK. NHANES 2011-2014 Reveals Cognition of US Older Adults may Benefit from Better Adaptation to the Mediterranean Diet. Nutrients. 2020;12. 4. [4].Dong X, Li S, Chen J, Li Y, Wu Y, Zhang D. Association of dietary ω-3 and ω-6 fatty acids intake with cognitive performance in older adults: National Health and nutrition examination Survey (NHANES) 2011-2014. Nutr J. 2020;19:25. 5. [5].Kelley GA, Kelley KS. Systematic reviews and meta-analysis in nutrition research. Br J Nutr. 2019;122:1279–94. 6. [6].Blake P, Durão S, Naude CE, Bero L. An analysis of methods used to synthesize evidence and grade recommendations in food-based dietary guidelines. Nutr Rev. 2018;76:290–300. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1093/nutrit/nux074&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=29425371&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2021%2F08%2F07%2F2021.08.05.21261680.atom) 7. [7].Dai Z, Kroeger CM, Lawrence M, Scrinis G, Bero L. Comparison of methodological quality between the 2007 and 2019 Canadian dietary guidelines. Public Health Nutr. 2020;23:2879–85. 8. [8].Schunemann HJ, Wiercioch W, Etxeandia I, Falavigna M, Santesso N, Mustafa R, et al. Guidelines 2.0: systematic development of a comprehensive checklist for a successful guideline enterprise. CMAJ. 2014;186:E123–42. [Abstract/FREE Full Text](http://medrxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NDoiY21haiI7czo1OiJyZXNpZCI7czoxMDoiMTg2LzMvRTEyMyI7czo0OiJhdG9tIjtzOjUwOiIvbWVkcnhpdi9lYXJseS8yMDIxLzA4LzA3LzIwMjEuMDguMDUuMjEyNjE2ODAuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 9. [9].Page MJ, McKenzie JE, Chau M, Green SE, Forbes A. Methods to select results to include in meta-analyses deserve more consideration in systematic reviews. J Clin Epidemiol. 2015;68:1282–91. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1016/j.jclinepi.2015.02.009&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=25841706&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2021%2F08%2F07%2F2021.08.05.21261680.atom) 10. [10].Lopez-Lopez JA, Page MJ, Lipsey MW, Higgins JPT. Dealing with effect size multiplicity in systematic reviews and meta-analyses. Res Synth Methods. 2018;9:336–51. 11. [11].Page MJ, Forbes A, Chau M, Green SE, McKenzie JE. Investigation of bias in meta-analyses due to selective inclusion of trial effect estimates: empirical study. BMJ open. 2016;6. 12. [12].Tendal B, Nuesch E, Higgins JP, Juni P, Gotzsche PC. Multiplicity of data in trial reports and the reliability of meta-analyses: empirical study. BMJ. 2011;343:d4829. [Abstract/FREE Full Text](http://medrxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiYm1qIjtzOjU6InJlc2lkIjtzOjE3OiIzNDMvYXVnMzBfMS9kNDgyOSI7czo0OiJhdG9tIjtzOjUwOiIvbWVkcnhpdi9lYXJseS8yMDIxLzA4LzA3LzIwMjEuMDguMDUuMjEyNjE2ODAuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 13. [13].Mayo-Wilson E, Fusco N, Li T, Hong H, Canner JK, Dickersin K. Multiple outcomes and analyses in clinical trials create challenges for interpretation and research synthesis. J Clin Epidemiol. 2017;86:39–50. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1016/j.jclinepi.2017.05.007&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2021%2F08%2F07%2F2021.08.05.21261680.atom) 14. [14].Page MJ, Bero L, Kroeger CM, Dai Z, McDonald S, Forbes A, et al. Investigation of Risk Of Bias due to Unreported and SelecTively included results in meta-analyses of nutrition research: the ROBUST study protocol. F1000Res. 2019;8:1760. 15. [15].Higgins JP, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, et al. Cochrane handbook for systematic reviews of interventions: John Wiley & Sons; 2019. 16. [16].Rada G, Perez D, Araya-Quintanilla F, Avila C, Bravo-Soto G, Bravo-Jeria R, et al. Epistemonikos: a comprehensive database of systematic reviews for health decision-making. BMC Med Res Methodol. 2020;20:286. [PubMed](http://medrxiv.org/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2021%2F08%2F07%2F2021.08.05.21261680.atom) 17. [17].Harris PA, Taylor R, Minor BL, Elliott V, Fernandez M, O’Neal L, et al. The REDCap consortium: Building an international community of software platform partners. J Biomed Inform. 2019;95:103208. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1016/j.jbi.2019.103208&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=31078660&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2021%2F08%2F07%2F2021.08.05.21261680.atom) 18. [18].Scherer R, Scherer MR. Package ‘PropCIs’. 2018. 19. [19].Team RC. R: A language and environment for statistical computing. 2013. 20. [20].Ayoub-Charette S, Liu Q, Khan TA, Au-Yeung F, Mejia SB, de Souza RJ, et al. Important food sources of fructose-containing sugars and incident gout: a systematic review and meta-analysis of prospective cohort studies. BMJ open. 2019;9:e024171. [Abstract/FREE Full Text](http://medrxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NzoiYm1qb3BlbiI7czo1OiJyZXNpZCI7czoxMToiOS81L2UwMjQxNzEiO3M6NDoiYXRvbSI7czo1MDoiL21lZHJ4aXYvZWFybHkvMjAyMS8wOC8wNy8yMDIxLjA4LjA1LjIxMjYxNjgwLmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 21. [21].Choo VL, Viguiliouk E, Mejia SB, Cozma AI, Khan TA, Ha V, et al. Food sources of fructose-containing sugars and glycaemic control: systematic review and meta-analysis of controlled intervention studies. bmj. 2018;363. 22. [22].Cheng S, Zheng Q, Ding G, Li G. Mediterranean dietary pattern and the risk of prostate cancer: A meta-analysis. Medicine. 2019;98. 23. [23].Bermejo LM, López-Plaza B, Santurino C, Cavero-Redondo I, Gómez-Candela C. Milk and dairy product consumption and bladder cancer risk: a systematic review and meta-analysis of observational studies. Adv Nutr. 2019;10:S224–S38. 24. [24].de Magalhães Cunha C, Costa PR, de Oliveira LP, Valterlinda AdO, Pitangueira JC, Oliveira AM. Dietary patterns and cardiometabolic risk factors among adolescents: systematic review and meta-analysis. Br J Nutr. 2018;119:859–79. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1017/S0007114518000533&link_type=DOI) 25. [25].Eaton JC, Rothpletz-Puglia P, Dreker MR, Iannotti L, Lutter C, Kaganda J, et al. Effectiveness of provision of animal-source foods for supporting optimal growth and development in children 6 to 59 months of age. Cochrane Database Syst Rev. 2019. 26. [26].George ES, Marshall S, Mayr HL, Trakman GL, Tatucu-Babet OA, Lassemillante A-CM, et al. The effect of high-polyphenol extra virgin olive oil on cardiovascular risk factors: A systematic review and meta-analysis. Crit Rev Food Sci Nutr. 2019;59:2772–95. 27. [27].Ghaedi E, Mohammadi M, Mohammadi H, Ramezani-Jolfaie N, Malekzadeh J, Hosseinzadeh M, et al. Effects of a Paleolithic diet on cardiovascular disease risk factors: a systematic review and meta-analysis of randomized controlled trials. Adv Nutr. 2019;10:634–46. 28. [28].Haider LM, Schwingshackl L, Hoffmann G, Ekmekcioglu C. The effect of vegetarian diets on iron status in adults: A systematic review and meta-analysis. Crit Rev Food Sci Nutr. 2018;58:1359–74. 29. [29].Hou R, Wei J, Hu Y, Zhang X, Sun X, Chandrasekar EK, et al. Healthy dietary patterns and risk and survival of breast cancer: a meta-analysis of cohort studies. Cancer Causes & Control. 2019;30:835–46. 30. [30].Huang Y, Zheng S, Wang T, Yang X, Luo Q, Li H. Effect of oral nut supplementation on endothelium-dependent vasodilation–a meta-analysis. Vasa. 2018. 31. [31].Iguacel I, Miguel-Berges ML, Gómez-Bruton A, Moreno LA, Julián C. Veganism, vegetarianism, bone mineral density, and fracture risk: a systematic review and meta-analysis. Nutr Rev. 2019;77:1–18. 32. [32].Kang K, Sotunde OF, Weiler HA. Effects of milk and milk-product consumption on growth among children and adolescents aged 6–18 years: a meta-analysis of randomized controlled trials. Adv Nutr. 2019;10:250–61. 33. [33].Kibret KT, Chojenta C, Gresham E, Tegegne TK, Loxton D. Maternal dietary patterns and risk of adverse pregnancy (hypertensive disorders of pregnancy and gestational diabetes mellitus) and birth (preterm birth and low birth weight) outcomes: a systematic review and meta-analysis. Public Health Nutr. 2019;22:506–20. 34. [34].Kodama S, Horikawa C, Fujihara K, Ishii D, Hatta M, Takeda Y, et al. Relationship between intake of fruit separately from vegetables and triglycerides-A meta-analysis. Clin Nutr ESPEN. 2018;27:53–8. 35. [35].Kojima G, Avgerinou C, Iliffe S, Walters K. Adherence to Mediterranean diet reduces incident frailty risk: Systematic review and meta-analysis. J Am Geriatr Soc. 2018;66:783–8. 36. [36].Larsson SC, Drca N, Jensen-Urstad M, Wolk A. Chocolate consumption and risk of atrial fibrillation: two cohort studies and a meta-analysis. Am Heart J. 2018;195:86–90. 37. [37].Li R, Yu K, Li C. Dietary factors and risk of gout and hyperuricemia: a meta-analysis and systematic review. Asia Pac J Clin Nutr. 2018;27:1344. 38. [38].Li W, Ruan W, Peng Y, Wang D. Soy and the risk of type 2 diabetes mellitus: A systematic review and meta-analysis of observational studies. Diabetes Res Clin Pract. 2018;137:190–9. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1016/j.diabres.2018.01.010&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=29407270&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2021%2F08%2F07%2F2021.08.05.21261680.atom) 39. [39].Li L, Lietz G, Seal C. Buckwheat and CVD risk markers: A systematic review and meta-analysis. Nutrients. 2018;10:619. 40. [40].Lopez PD, Cativo EH, Atlas SA, Rosendorff C. The effect of vegan diets on blood pressure in adults: a meta-analysis of randomized controlled trials. Am J Med. 2019;132:875-83. e7. 41. [41].Maki KC, Palacios OM, Koecher K, Sawicki CM, Livingston KA, Bell M, et al. The relationship between whole grain intake and body weight: results of meta-analyses of observational studies and randomized controlled trials. Nutrients. 2019;11:1245. 42. [42].Malmir H, Saneei P, Larijani B, Esmaillzadeh A. Adherence to Mediterranean diet in relation to bone mineral density and risk of fracture: a systematic review and meta-analysis of observational studies. Eur J Nutr. 2018;57:2147–60. 43. [43].Matía-Martín P, Torrego-Ellacuría M, Larrad-Sainz A, Fernández-Pérez C, Cuesta-Triana F, Rubio-Herrera MÁ. Effects of milk and dairy products on the prevention of osteoporosis and osteoporotic fractures in Europeans and non-Hispanic Whites from North America: a systematic review and updated meta-analysis. Adv Nutr. 2019;10:S120–S43. 44. [44].Mena-Sánchez G, Becerra-Tomás N, Babio N, Salas-Salvadó J. Dairy product consumption in the prevention of metabolic syndrome: a systematic review and meta-analysis of prospective cohort studies. Adv Nutr. 2019;10:S144–S53. 45. [45].Milajerdi A, Namazi N, Larijani B, Azadbakht L. The association of dietary quality indices and cancer mortality: a systematic review and meta-analysis of cohort studies. Nutr Cancer. 2018;70:1091–105. 46. [46].Mishali M, Prizant-Passal S, Avrech T, Shoenfeld Y. Association between dairy intake and the risk of contracting type 2 diabetes and cardiovascular diseases: a systematic review and meta-analysis with subgroup analysis of men versus women. Nutr Rev. 2019;77:417–29. 47. [47].Mohseni R, Abbasi S, Mohseni F, Rahimi F, Alizadeh S. Association between dietary inflammatory index and the risk of prostate cancer: a meta-analysis. Nutr Cancer. 2019;71:359–66. 48. [48].Musa-Veloso K, Poon T, Harkness LS, O’Shea M, Chu Y. The effects of whole-grain compared with refined wheat, rice, and rye on the postprandial blood glucose response: a systematic review and meta-analysis of randomized controlled trials. Am J Clin Nutr. 2018;108:759–74. 49. [49].Namazi N, Larijani B, Azadbakht L. Dietary inflammatory index and its association with the risk of cardiovascular diseases, metabolic syndrome, and mortality: a systematic review and meta-analysis. Horm Metab Re. 2018;50:345–58. 50. [50].Picasso MC, Lo-Tayraco JA, Ramos-Villanueva JM, Pasupuleti V, Hernandez AV. Effect of vegetarian diets on the presentation of metabolic syndrome or its components: A systematic review and meta-analysis. Clin Nutr. 2019;38:1117–32. 51. [51].Qin Z-Z, Xu J-Y, Chen G-C, Ma Y-X, Qin L-Q. Effects of fatty and lean fish intake on stroke risk: A meta-analysis of prospective cohort studies. Lipids Health Dis. 2018;17:1–7. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1186/s12944-017-0646-8&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=29298716&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2021%2F08%2F07%2F2021.08.05.21261680.atom) 52. [52].Rees K, Takeda A, Martin N, Ellis L, Wijesekara D, Vepa A, et al. Mediterranean-style diet for the primary and secondary prevention of cardiovascular disease. Cochrane Database Syst Rev. 2019. 53. [53].Ren Y, Liu Y, Sun X-Z, Wang B-Y, Zhao Y, Liu D-C, et al. Chocolate consumption and risk of cardiovascular diseases: a meta-analysis of prospective studies. Heart. 2019;105:49–55. [Abstract/FREE Full Text](http://medrxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6ODoiaGVhcnRqbmwiO3M6NToicmVzaWQiO3M6ODoiMTA1LzEvNDkiO3M6NDoiYXRvbSI7czo1MDoiL21lZHJ4aXYvZWFybHkvMjAyMS8wOC8wNy8yMDIxLjA4LjA1LjIxMjYxNjgwLmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 54. [54].Shab-Bidar S, Golzarand M, Hajimohammadi M, Mansouri S. A posteriori dietary patterns and metabolic syndrome in adults: a systematic review and meta-analysis of observational studies. Public Health Nutr. 2018;21:1681–92. 55. [55].Shafiei F, Salari-Moghaddam A, Larijani B, Esmaillzadeh A. Adherence to the Mediterranean diet and risk of depression: A systematic review and updated meta-analysis of observational studies. Nutr Rev. 2019;77:230–9. 56. [56].Teoh SL, Lai NM, Vanichkulpitak P, Vuksan V, Ho H, Chaiyakunapruk N. Clinical evidence on dietary supplementation with chia seed (Salvia hispanica L.): a systematic review and meta-analysis. Nutr Rev. 2018;76:219–42. 57. [57].Voon PT, Lee ST, Ng TKW, Ng YT, Yong XS, Lee VKM, et al. Intake of palm olein and lipid status in healthy adults: a meta-analysis. Adv Nutr. 2019;10:647–59. 58. [58].Wang L, Liu C, Zhou C, Zhuang J, Tang S, Yu J, et al. Meta-analysis of the association between the dietary inflammatory index (DII) and breast cancer risk. Eur J Clin Nutr. 2019;73:509–17. 59. [59].Xiao Y, Ke Y, Wu S, Huang S, Li S, Lv Z, et al. Association between whole grain intake and breast cancer risk: a systematic review and meta-analysis of observational studies. Nutr j. 2018;17:1–12. 60. [60].Xu Y, Yang J, Du L, Li K, Zhou Y. Association of whole grain, refined grain, and cereal consumption with gastric cancer risk: A meta-analysis of observational studies. Food Sci Nutr. 2019;7:256–65. 61. [61].Zhang Z, Chen G-C, Qin Z-Z, Tong X, Li D-P, Qin L-Q. Poultry and fish consumption in relation to total cancer mortality: a meta-analysis of prospective studies. Nutr Cancer. 2018;70:204–12. 62. [62].Adjibade M, Assmann KE, Andreeva VA, Lemogne C, Hercberg S, Galan P, et al. Prospective association between adherence to the Mediterranean diet and risk of depressive symptoms in the French SU.VI.MAX cohort. Eur J Nutr. 2018;57:1225–35. 63. [63].Page MJ, Shamseer L, Tricco AC. Registration of systematic reviews in PROSPERO: 30,000 records and counting. Systematic reviews. 2018;7:1–9. 64. [64].Liu Z, Saldanha IJ, Margolis D, Dumville JC, Cullum NA. Outcomes in Cochrane systematic reviews related to wound care: an investigation into prespecification. Wound Repair Regen. 2017;25:292–308. 65. [65].Saldanha IJ, Dickersin K, Wang X, Li T. Outcomes in Cochrane systematic reviews addressing four common eye conditions: an evaluation of completeness and comparability. PloS one. 2014;9:e109400. 66. [66].Trutschel D, Palm R, Holle B, Simon M. Methodological approaches in analysing observational data: a practical example on how to address clustering and selection bias. Int J Nurs Stud. 2017;76:36–44. 67. [67].McKenzie JE, Brennan SE, Ryan RE, Thomson HJ, Johnston RV, Thomas J. Defining the criteria for including studies and how they will be grouped for the synthesis 2019. 68. [68].Page MJ, Moher D, Bossuyt P, Boutron I, Hoffmann T, Mulrow C, et al. PRISMA 2020 explanation and elaboration: updated guidance and exemplars for reporting systematic reviews. 2020.