Prospective meta‐analyses and Cochranes role in embracing next‐generation methodologies

Anna Lene SeidlerKylie E HunterSaskia CheyneJesse A BerlinDavina GhersiLisa M AskieView authors' declarations of interest

https://island123.medium.com/%E8%A7%92%E9%A0%AD-%E6%B5%AA%E6%B5%81%E9%80%A3-%E7%B7%9A%E4%B8%8A%E7%9C%8B-%E5%AE%8C%E6%95%B4%E7%89%88-2021-tw4k-d33e88f297f8
https://bruningdavenport.instructure.com/eportfolios/12625/Home/__The_last_stray1080P2021
https://people.instructure.com/eportfolios/22919/Home/_20214K1080PThe_last_stray_
https://cggs.instructure.com/eportfolios/6637/Home/__The_last_stray_2021HD__
https://republicschools.instructure.com/eportfolios/8516/Home/20211080P4K

Cochrane systematic reviews and meta‐analyses are regarded as the ‘gold standard’ for high‐quality information and are widely used to inform healthcare policy and practice. The nature of how conventional systematic reviews are conceived and conducted after at least some of the included studies are completed means that reviewers can inadvertently introduce bias when faced with heterogeneous studies that cannot be easily synthesized. Prospective meta‐analysis (PMA) is now gaining traction as a means of reducing research waste and producing meaningful and less biased evidence syntheses.[1] PMA has been lauded as a ‘next‐ generation’ method,[2] and Ioannidis has argued that “all primary original research may be designed, executed, and interpreted as prospective meta‐analysis”.[3]

In a PMA, eligible studies or cohorts are identified for inclusion, and hypotheses and analysis strategies are specified before the results are known.[1] The approach relies on prospective trial registration and early contact with investigators in the field. In some instances, PMAs may begin with initiating a co‐ordinated program of research for a high‐priority research question that encourages and incentivizes researchers to plan studies that can be included in the PMA. Investigators of identified planned or ongoing studies are then invited to form a collaboration, harmonize their studies (to the extent possible) and combine their results upon completion, usually after all the individual studies' results have been published.

Though a paradigm shift is likely to require systemic change, the benefits of PMA for researchers and decision‐makers are manifold, and recent advances with collaborative technologies and prospective study registration have opened new doors for PMA to become more commonplace.[1, 4] The Cochrane Prospective Meta‐Analysis Methods Group has published in‐depth guidance on PMA.[1, 5] This guidance includes detailed advice for each step of conducting a PMA, such as writing a PMA protocol, searching for unpublished studies, forming and managing a collaboration, and harmonizing key study features.

PMA can reduce several of the limitations and potential sources of bias associated with traditional retrospective Cochrane Reviews. First, the risk of publication bias and selective outcome reporting can be reduced or removed by agreeing to a standard set of core outcomes, time points, and measures with investigators in advance.[6, 7] Doing so has the added benefits of aligning outcome definitions to facilitate meta‐analysis, maximizing the ability to analyze rare but important outcomes (such as adverse events) that individual studies would not have had the power to detect, and reducing research waste.[8] In a recent PMA on childhood obesity prevention, the decision to collaborate increased the number of core outcomes collected by all trials (and thus the number of outcomes that could be combined in a meta‐analysis) from 18% to 91%.[9] Second, by specifying eligibility criteria and outcomes before results are known, PMA can prevent bias introduced by prior knowledge of study context and findings.[5] Third, PMA can facilitate access to individual participant data (IPD) and allow more complete interrogation of primary datasets.[10] Collaboration with investigators in a PMA allows data to be tracked through the application of clear instructions and variable descriptions. An important associated advantage of IPD meta‐analysis is the ability to conduct subgroup analyses that are less prone to ecological bias,[11] which is further improved with PMA as these analyses can be planned de novo to ensure all included studies collect the appropriate subgroup variables. PMA has become particularly relevant in light of the current global pandemic of COVID‐19, in which unprecedented numbers of small ongoing trials are emerging, with insufficient power to detect key clinical outcomes such as mortality.[12] Co‐ordinating these efforts prospectively, worldwide, to align on key elements of study design, populations, core outcomes, etc, is a cost‐effective, timely, and more reliable way of achieving larger sample sizes and thus more powerful and impactful evidence.[13] It ensures trials can be combined upon completion, or even in pre‐specified interim analyses.[12, 13] For instance, a recently published PMA combining seven trials with a total of 1703 participants showed that corticosteroid therapy can reduce 28‐day mortality of patients critically ill with COVID‐19 compared with usual care or placebo.[14]

PMA is one of several next‐generation systematic review methodologies that are evolving and increasing in number. Many of these methodologies can be used in conjunction with one another and so it is important that they are not regarded in isolation. In addition to the common use of IPD in PMA mentioned above, the ability to harmonize collected variables in a PMA means it also has a place in network meta‐analysis and systematic reviews of prognosis studies to understand effect modifiers and predictors and assess the validity of indirect comparisons.[15, 16] Sequential meta‐analysis methods may also be used in conjunction with PMA methods to determine when an optimal information size has been reached and no further data need to be collected,[5] or with adaptive trial methodologies to incorporate new interventions, prioritize promising comparisons, and discourage research waste.[17] Finally, PMA can be combined with living evidence frameworks, systematically updating the PMA as new evidence becomes available.[18] There is also the possibility of integrating previous evidence into a PMA, by conducting a nested PMA.[1]

When Cochrane was founded in 1993, systematic reviews mostly combined evidence from standard trial designs that compared an intervention and a control group. Now, the landscape of systematic review methods is more diverse and complex. Most of the leaders and developers of these new methods have longstanding and strong links with Cochrane but find it difficult to apply new methods in accordance with Cochrane's production tools, standards, and policies. In the context of PMA, Cochrane's usual requirement to have an updated search within 12 months (intended to provide up‐to‐date evidence) is challenged. The search for planned and ongoing studies in a PMA is usually conducted years before the eligible studies are completed and, when collecting IPD, many time‐consuming steps happen after the search (collaborators need to be invited, and data need to be prepared, shared, processed, merged, and analyzed). Cochrane's review production tools are not optimized for inputting non‐standard data or displaying adjusted analyses, and template headings can be restrictive, redundant, or insufficient.

The challenge for Cochrane in supporting next‐generation methods is to allow flexibility in its production tools, standards, and policies to facilitate the use and publication of diverse methods and review types while upholding standards of consistency and quality assurance. To achieve the balance, Cochrane and the methods community need to work together to facilitate the appraisal and uptake of novel methods by identifying:

criteria to help authors and editors assess where PMA could be considered (e.g., infectious disease outbreaks, novel therapeutics)

areas of Cochrane's Methodological Expectations of Cochrane Intervention Reviews (MECIR) that may need revision or extension to accommodate new methods;

guidance needed for authors and editors to ensure the approach can be applied accurately and scrutinized; and

developments needed for review production software to present analyses and results in a digestible format (e.g. flexibility with headings, options to import results from other software).

As Cochrane reaches the end of its Strategy to 2020 and begins to define a new strategic framework, engagement with the methods community will be essential to consider next‐generation approaches and build innovation and agility into Cochrane's future.[19] If Cochrane can integrate novel methods and flexibility into its processes while maintaining high levels of quality assurance and rigor, the organization can retain its reputation at the forefront of systematic review methods and healthcare decision making.

Database:
Cochrane Database of Systematic Reviews
Published:
30 October 2020
Copyright:
Copyright © 2020 The Cochrane Collaboration. Published by John Wiley & Sons, Ltd.

Declarations of interest
The authors have completed the ICMJE form for disclosure of potential conflicts of interest. The authors are convenors (LMA, JAB, DG) or editors (ALS, KEH, SC) for the Cochrane Prospective Meta‐Analysis Methods Group. ALS is a Research Fellow for the Australian New Zealand Clinical Trials Registry (ANZCTR), and LMA is manager at ANZCTR. JAB is employed by and owns shares in Johnson & Johnson and contributed to the current work as part of normal duties.

Editorial note: This Editorial was accepted for publication in September 2020. It would not comply with Cochrane's Conflict of Interest Policy for Cochrane Library content (which applies to content initiated or accepted from 14 October 2020) due to one of the authors (JAB) being an employee of a pharmaceutical company.

Provenance and peer review
This editorial was commissioned based on a proposal by the Cochrane Prospective Meta‐Analysis Methods Group, and it was not externally peer‐reviewed.

References
Seidler AL, Hunter KE, Cheyne S, Ghersi D, Berlin JA, Askie L. A guide to prospective meta‐analysis. BMJ 2019;367:l5342. https://doi.org/10.1136/bmj.l5342
Ioannidis J. Next‐generation systematic reviews: prospective meta‐analysis, individual‐level data, networks and umbrella reviews. British Journal of Sports Medicine2017;51:1456–8. https://doi.org/10.1136/bjsports-2017-097621
Ioannidis JP. Meta‐research: The art of getting it wrong. Research Synthesis Methods2010;1(3–4):169–84. https://doi.org/10.1002/jrsm.19
Tan AC, Jiang I, Askie L, Hunter K, Simes RJ, Seidler AL. Prevalence of trial registration varies by study characteristics and risk of bias. Journal of Clinical Epidemiology2019;113:64–74. https://doi.org/10.1016/j.jclinepi.2019.05.009
Thomas J, Askie LM, Berlin JA, Elliott JH, Ghersi D, Simmonds M et al. Chapter 22: Prospective approaches to accumulating evidence. In: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA (editors). Cochrane Handbook for Systematic Reviews of Interventions version 6.0 (updated July 2019). Cochrane, 2019. Available from training.cochrane.org/handbook.
Simes RJ. Publication bias: the case for an international registry of clinical trial