Skip to main content

Web Content Display Web Content Display

Skip banner

Web Content Display Web Content Display

INCET logo

Web Content Display Web Content Display

BIOUNCERTAINTY - ERC Starting Grant no. 805498

ERC logo

Web Content Display Web Content Display

Web Content Display Web Content Display

Is meta-analysis of RCTs assessing the efficacy of interventions a reliable source of evidence for therapeutic decisions? - a new article by Mariusz Maziarz

Is meta-analysis of RCTs assessing the efficacy of interventions a reliable source of evidence for therapeutic decisions? - a new article by Mariusz Maziarz

Literature-based meta-analysis is a standard technique applied to pool results of individual studies used in medicine and social sciences. It has been criticized for being too malleable to constrain results, averaging incomparable values, lacking a measure of evidence's strength, and problems with a systematic bias of individual studies. We argue against using literature-based meta-analysis of RCTs to assess treatment efficacy and show that therapeutic decisions based on meta-analytic average are not optimal given the full scope of existing evidence.

Full abstract

Literature-based meta-analysis is a standard technique applied to pool results of individual studies used in medicine and social sciences. It has been criticized for being too malleable to constrain results, averaging incomparable values, lacking a measure of evidence's strength, and problems with a systematic bias of individual studies. We argue against using literature-based meta-analysis of RCTs to assess treatment efficacy and show that therapeutic decisions based on meta-analytic average are not optimal given the full scope of existing evidence. The argument proceeds with discussing examples and analyzing the properties of some standard meta-analytic techniques. First, we demonstrate that meta-analysis can lead to reporting statistically significant results despite the treatment's limited efficacy. Second, we show that meta-analytic confidence intervals are too narrow compared to the variability of treatment outcomes reported by individual studies. Third, we argue that literature-based meta-analysis is not a reliable measurement instrument. Finally, we show that meta-analysis averages out the differences among studies and leads to a loss of information. Despite these problems, literature-based meta-analysis is useful for the assessment of harms. We support two alternative approaches to evidence amalgamation: meta-analysis of individual patient data (IPD) and qualitative review employing mechanistic evidence.

Link to the article

Maziarz, M. (2022). Is meta-analysis of RCTs assessing the efficacy of interventions a reliable source of evidence for therapeutic decisions?, Studies in History and Philosophy of Science, 91: 159-167. https://doi.org/10.1016/j.shpsa.2021.11.007.