We have the pleasure to invite you for a research seminar in the ‘BIOUNCERTAINTY’ research project. The subject of the seminar is 'Is meta-analysis a reliable source of evidence for decisions in the clinic?'. The seminar will take place on Monday, 26th of October, at 5:30pm on MS Teams (link below).

Abstract: Literature-based meta-analysis is a standard technique applied to pooling results of individual studies used in medicine and social sciences. It has been criticized for [1] being too malleable to constrain results, [2] averaging incomparable values, [3] lacking a measure of strength of evidence, and [4] problems with systematic bias of individual studies. Despite these epistemic concerns, the current practice of decision-making and preparing clinical guidance relies to a high degree on evidence from meta-analysis. We argue against the use of literature-based meta-analysis of RCTs for assessment of treatment efficacy by showing that therapeutic decisions based on meta-analytic average of individual studies are not optimal given the full scope of existing evidence. The argument proceeds with discussing examples and analyzing properties of some common meta-analytic techniques. First, we demonstrate that the use of meta-analysis can lead to reporting statistically significant results despite limited efficacy of the treatment. This, in turn, leads to a huge number needed to treat (NNT) and an unfavorable tradeoff between benefits and harms of the treatment. Second, we show that meta-analytic confidence intervals (CI) are too narrow comparing to the variability of treatment outcomes reported by individual studies. Third, based on a review of studies comparing results of literature-based meta-analysis and IPD meta-analysis that report non-systematic differences in treatment effect estimates, we conclude that literature-based meta-analysis lacks reliability. Finally, we analyze amalgamating conflicting RCTs (i.e., studies reporting both positive and negative treatment effects). In this case, meta-analysis averages out the differences among studies and leads to a loss of information  if no mediating variable can be found. Despite these problems, literature-based meta-analysis is useful for the assessment of harms. We support two alternative approaches to concluding systematic literature review: meta-analysis of individual patient data (IPD) and qualitative review employing mechanistic evidence.