Trouble With Systematic Reviews and Meta-Analyses
Health Letter, August 2017
By Azza AbuDagga, M.H.A., Ph.D.
Results of biomedical and health care research are disseminated on an enormous scale: Last year alone, nearly 1.3 million new publications were indexed in MEDLINE, a major database of journal articles in these fields.[1]
Systematic reviews and meta-analyses are single studies that synthesize the results of multiple other studies concerning a particular question, such as the effectiveness or safety of a drug or medical device. If conducted properly, systematic reviews and meta-analyses are considered the highest level of evidence on a given subject[2] and are extremely useful for making health care decisions based on evidence from multiple studies instead of evidence from the most recent or the largest individual studies. [3]
However, systematic reviews and meta-analyses may be of limited value or even be misleading if they are not conducted properly or are biased.
A recent study[4] by John Ioannidis, a well-known Stanford University researcher, examined the growth of systematic reviews and meta-analyses and painted a troubling picture of the state of these publications. According to his study, researchers are producing, in large quantities, systematic reviews and meta-analyses that are redundant, misleading or serving vested interests.[5] This study was published in the September 2016 issue of the Milbank Quarterly.[6]
Mass production
Although methods for conducting systematic reviews and meta-analyses have existed for many decades, these studies were not common in biomedical and health care research until the late 1980s and 1990s, mainly due to the lack of popular software to produce them in large quantities at that time.
Researchers from the Cochrane Collaboration, a respected international non-profit organization that specializes in systematic reviews and meta-analyses, anticipated in 2003 that about 10,000 systematic reviews would be needed to cover all clinical trials in the field of health care research.[7] However, Ioannidis found that nearly 267,000 systematic reviews and 59,000 meta-analyses were indexed in MEDLINE from Jan. 1, 1986, to Dec. 4, 2015.[8] In fact, the growth rate of these publications has outpaced the growth rate of studies overall: Annual publications between 1991 and 2014 increased by more than 2,700 percent for systematic reviews and by over 2,600 percent for meta-analyses, compared with a 153 percent annual increase for all MEDLINE-indexed items.
Redundancy
Ioannidis illustrates that the surge in systematic reviews and meta-analyses is primarily due to redundancy, as there are several systematic reviews and meta-analyses for the majority of topics. For example, a survey published in the BMJ[9] that included 73 randomly selected meta-analyses that had been published in 2010 found that for two-thirds of these studies, there was at least one, and sometimes as many as 13, additional meta-analyses published on the same topic by early 2013.
Ioannidis also pointed out that about half of the systematic reviews published in recent years pertain to reviews of clinical trials: There were more than 15,000 systematic reviews related to clinical trials in 2014, representing 53 percent of all systematic reviews published that year. Similarly, the proportion of clinical trial meta-analyses is high compared with the total number of meta-analyses. In contrast, there were approximately 23,100 clinical trial articles in 2014, a number that had remained stable from 2010 to 2014.
It could be argued that there may be some benefit in having several independent authors look at the same data to see whether they reach the same results and conclusions or examine different outcomes than those included in original reviews. However, the aforementioned BMJ survey shows that almost a quarter of subsequent meta-analyses were conducted by some of the same authors of the original meta-analyses, and 65 percent of the subsequent meta-analyses did not include more outcomes than those included in the original meta-analyses.[10]
Moreover, overlapping meta-analyses often can be confusing because they may not include the same primary studies that meet the criteria selected by researchers for inclusion in the original meta-analysis. Although this practice can explain why overlapping reviews reach different results, readers may find it difficult to reconcile different findings.
Ioannidis notes that China is currently the most prolific producer of English-language meta-analyses, especially in the area of genetics. He characterized most of these publications as misleading due to their use of outdated information.
Fragmented evidence
Another major part of the problem with systematic reviews and meta-analyses is that these publications often try to piece together fragments of information from multiple primary studies that are inherently different[11] without highlighting differences in these studies.
For example, a 2016 in-depth study of nearly 700 systematic reviews published in PLOS Medicine found that more than half of these reviews did not consider the risk of publication bias — the fact that many results with negative outcomes are never published — when drawing their conclusions.[12] At least one third of the reviews did not report essential details about their methodologies, such as whether they used a certain study protocol, how they searched the literature, how the information was extracted from the primary studies or how the risk of bias in the results was assessed. Moreover, at least one third of the reviews used statistical methods that are not recommended by leading medical research organizations, such as the Cochrane Collaboration and the Institute of Medicine (IOM).
Therefore, in an April 2017 opinion article in the Journal of the American Medical Association – Psychiatry, Ioannidis recommended that review studies “should systematically probe, detect, dissect, and highlight major errors and biases” in primary studies instead of “taki[ng] a shortcut” of including flawed studies in their reviews and ignoring the differences between the studies.[13]
Vested interests
Many systematic reviews and meta-analyses are conducted by investigators or contracting companies with financial ties to the pharmaceutical or medical device industries. This is a troubling trend because research has shown that industry-sponsored reviews are less transparent about their methodologies[14] and often reach conclusions that are more favorable to the industry[15] than reviews conducted by investigators without conflicts of interest.
For example, meta-analyses that included at least one author employed by a manufacturer of an antidepressant were associated with a 22-fold decrease in the odds of reporting negative statements about the drug compared with meta-analyses conducted entirely by non-industry researchers.[16] Thus, reviews authored by researchers with conflicts of interest can become viable tools for the industry to market products.
Conclusions
Due to the aforementioned limitations involving unnecessary, misleading and conflicting systematic reviews and meta-analyses, Ioannidis concludes that these flawed studies are not promoting evidence-based medicine and health care. In fact, he estimates that only 3 percent of all meta-analyses represent good and truly informative studies.
Therefore, he calls for a major overhaul in the production of biomedical research and its credible synthesis, including planning and conducting systematic reviews and meta-analyses prospectively and without conflicts of interest through teamwork between researchers of primary studies and those of future systematic reviews and meta-analyses. He emphasizes the importance of realigning the funding and authorship of systematic reviews and meta-analyses in ways that serve to reduce vested interests and related biases.
Buy-in from a large number of stakeholders, including funders, scientists, medical journals and consumers, is needed for these changes to occur. In the meantime, readers of systematic reviews and meta-analyses need to take the findings of these publications with a grain of salt. Readers should look for possible conflicts of interest and should check whether other studies had different findings or reached different conclusions. Whenever available, systematic reviews and meta-analyses produced by organizations without conflicts of interest, such as the Agency for Healthcare Research and Quality’s Evidence-based Practice Center Program, the Cochrane Collaboration and the IOM, should be sought first.
References
[1] Medline (PubMed) trend. http://dan.corlan.net/medline-trend.html. Accessed July 21, 2017.
[2] Ioannidis J. Next-generation systematic reviews: Prospective meta-analysis, individual-level data, networks and umbrella reviews. Br J Sport Med. 2017;Feb 21. doi:10.1136/bjsports-2017-097621.
[3] Page MJ, Shamseer L, Altman DG, et al. Epidemiology and reporting characteristics of systematic reviews of biomedical research: A cross-sectional study. PLOS Med. 2016;13(5):e1002028. doi:10.1371/journal.pmed.1002028.
[4] Ioannidis JPA. The mass production of redundant, misleading, and conflicted systematic reviews and meta-analyses: Mass production of systematic reviews and meta-analyses. Milbank Q. 2016;94(3):485-514.
[5] Page MJ, Moher D. Mass production of systematic reviews and meta-analyses: An exercise in mega-silliness? Milbank Q. 2016;94(3):515-519.
[6] Ioannidis JPA. The mass production of redundant, misleading, and conflicted systematic reviews and meta-analyses: Mass production of systematic reviews and meta-analyses. Milbank Q. 2016;94(3):485-514.
[7] Mallett S, Clarke M. How many Cochrane reviews are needed to cover existing evidence on the effects of health care interventions? ACP J Club. 2003;139(1):A11-A12.
[8] Ioannidis JPA. The mass production of redundant, misleading, and conflicted systematic reviews and meta-analyses: Mass production of systematic reviews and meta-analyses. Milbank Q. 2016;94(3):485-514.
[9] Siontis KC, Hernandez-Boussard T, Ioannidis JP. Overlapping meta-analyses on the same topic: Survey of published studies. BMJ. 2013;347:f4501.
[10] Siontis KC, Hernandez-Boussard T, Ioannidis JP. Overlapping meta-analyses on the same topic: Survey of published studies. BMJ. 2013;347:f4501.
[11] Ioannidis JPA. Meta-analyses can be credible and useful: A new standard. JAMA Psychiatry. 2017;74(4):311-312.
[12] Page MJ, Shamseer L, Altman DG, et al. Epidemiology and reporting characteristics of systematic reviews of biomedical research: A cross-sectional study. PLOS Med. 2016;13(5):e1002028. doi:10.1371/journal.pmed.1002028.
[13] Ioannidis JPA. Meta-analyses can be credible and useful: A new standard. JAMA Psychiatry. 2017;74(4):311-312.
[14] Jorgensen A, Maric K, Tendal B, et al. Industry-supported meta-analyses compared with meta-analyses with non-profit or no support: differences in methodological quality and conclusions. BMC Med Res Methodol. 2008;8(September 9):60.
[15] Jørgensen AW, Hilden J, Gøtzsche PC. Cochrane reviews compared with industry supported meta-analyses and other meta-analyses of the same drugs: systematic review. BMJ. 2006;333(7572):782.
[16] Ebrahim S, Bance S, Athale A, et al. Meta-analyses with industry involvement are massively published and report no caveats for antidepressants. J Clin Epidemiol. 2016;70(Feb):155-163.