How to Evaluate a Meta-Analysis

This content is educational. It describes how to read and evaluate meta-analyses. It is not medical advice.

Why this matters

Meta-analyses sit at the top of the evidence hierarchy for many clinical questions. They pool data from multiple individual studies to produce more precise estimates than any single trial. When a meta-analysis concludes that an intervention works, the conclusion is generally stronger than what any single trial supports.

The catch: meta-analyses aren't all equally rigorous. A poorly conducted meta-analysis can produce a confident-looking conclusion that doesn't survive scrutiny. The published literature contains many high-quality meta-analyses and many lower-quality ones, and the difference isn't obvious from the abstract alone.

This page describes how to read a meta-analysis well enough to distinguish the two.

What a meta-analysis actually is

A meta-analysis takes the quantitative results of multiple individual studies and combines them using statistical methods to produce a pooled estimate of an effect. Properly done, it accomplishes several things:

• Increases statistical power by combining sample sizes
• Reduces the influence of any single anomalous study
• Allows estimation of how consistent the effect is across studies
• Surfaces patterns that emerge across populations or settings

A typical meta-analysis question: "Does intervention X improve outcome Y, and by how much?"

The output is usually:

• A pooled effect estimate (with confidence interval)
• A heterogeneity assessment (how much the studies disagree)
• A forest plot visualising each study's contribution
• Subgroup analyses where appropriate
• An assessment of risk of bias and publication bias

The big questions to ask of any meta-analysis

1. What's the question?

Look for explicit population, intervention, comparator, and outcome (the PICO framework we covered in How to read a clinical trial).

Vague questions produce vague meta-analyses. A meta-analysis of "vitamin D for health" is much less rigorous than "vitamin D supplementation at 800-2000 IU/day for fracture risk in postmenopausal women with baseline 25(OH)D <30 ng/mL."

Specific questions allow specific inclusion criteria, specific outcome measurement, and specific interpretation. Vague questions allow heterogeneous study inclusion that can muddy the analysis.

2. What's the inclusion strategy?

The meta-analysis should describe:

• How studies were identified (database search strategy)
• What types of studies were included (RCTs only, observational, both)
• Specific inclusion criteria
• How disagreements between reviewers were resolved
• A PRISMA flow diagram showing how many studies were screened, excluded, and included

Pre-registered systematic reviews (PROSPERO registration) are higher quality than ad hoc meta-analyses. Pre-registration commits the authors to specific methods before they see the results.

3. What's the heterogeneity?

Heterogeneity is how much the included studies disagree with each other. Statistical measures (I², Q, τ²) quantify it.

I² is the most commonly reported:

• 0-25% — low heterogeneity; studies are reasonably consistent
• 25-50% — moderate heterogeneity; some disagreement
• 50-75% — substantial heterogeneity; studies don't fully agree
• 75%+ — high heterogeneity; pooling may not be appropriate

High heterogeneity doesn't mean the meta-analysis is wrong — but it means the pooled estimate hides genuine variation between studies. A pooled effect of 20% improvement that includes studies showing 5% and 60% improvements is telling you less than it appears to.

The authors should investigate sources of heterogeneity through subgroup analysis, meta-regression, or sensitivity analysis. A meta-analysis that ignores high heterogeneity should be read with caution.

4. What's the risk of bias?

Each included study has its own risk of bias. The meta-analysis should assess this systematically using validated tools (Cochrane RoB 2 for RCTs, ROBINS-I for observational studies).

Common biases the assessment should address:

• Random sequence generation
• Allocation concealment
• Blinding of participants and personnel
• Blinding of outcome assessment
• Incomplete outcome data
• Selective reporting

Meta-analyses that include high-risk-of-bias studies should typically report sensitivity analyses excluding those studies.

5. Is publication bias a concern?

If positive studies are more likely to be published than null studies, the meta-analysis will overestimate the true effect. Detection methods:

• Funnel plot — visual symmetry test of effect size vs. study precision; asymmetry suggests publication bias
• Egger's test — statistical test for funnel plot asymmetry
• Trim and fill — sensitivity analysis that estimates the effect if missing studies were included

Meta-analyses with substantial publication bias warning signs should be discounted. Some fields (particularly nutrition and dietary supplement research) have severe publication bias problems.

6. What's the certainty of evidence?

The GRADE framework (which we covered in Why "evidence-based" gets misused) classifies evidence as high, moderate, low, or very low quality based on:

• Study design
• Risk of bias
• Inconsistency
• Indirectness
• Imprecision
• Publication bias
• Magnitude of effect
• Dose-response gradient
• Plausible confounding

A good meta-analysis explicitly applies GRADE to summarise overall certainty. "High-quality evidence" carries more weight than "low-quality evidence" even when both produce statistically significant pooled effects.

How to read a forest plot

The forest plot is the standard visualisation in meta-analyses. Each line represents one study's effect estimate with its confidence interval. A diamond at the bottom represents the pooled effect.

What to look at:

Position of the diamond. Where does the pooled effect sit relative to the line of no effect (typically 1.0 for ratios, 0 for differences)?

Width of the diamond. Narrow = precise pooled estimate. Wide = imprecise. The width matters for clinical interpretation.

Position of individual study points. Are they clustered around the pooled estimate? Scattered widely? Mostly on one side of the line of no effect?

Width of individual study confidence intervals. Larger studies show narrower intervals. Small underpowered studies have wide intervals that can look misleadingly impressive on first glance.

Are any studies dominant? If one large study dominates the analysis, the "meta-analysis" may largely reflect that one study. Some Cochrane reviews include sensitivity analyses excluding dominant studies.

Are confidence intervals crossing the line of no effect? Studies whose confidence intervals cross the line of no effect aren't individually significant. A pooled effect that's significant despite most individual studies being non-significant should be examined carefully.

Common quality issues to watch for

Several patterns suggest a meta-analysis should be read with caution:

Vague or post-hoc question. The question seems to have been shaped after the data was collected.

Inclusion criteria are too broad. Studies of very different interventions, populations, or outcomes are pooled together as if they were equivalent.

High heterogeneity without investigation. I² > 50% without serious discussion of why.

Inclusion of high-risk-of-bias studies without sensitivity analysis. Low-quality studies are pooled with high-quality ones.

Publication bias warning signs ignored. Funnel plot asymmetry, missing trials registered but unpublished.

Conflicts of interest. Industry-funded meta-analyses of industry products produce more favourable conclusions [Lundh et al. 2017 Cochrane].

Cherry-picked subgroup analyses. A null primary result followed by emphasis on positive findings in subgroups.

Updated and superseded. The meta-analysis is older than alternatives that have included newer trials.

What good looks like

The strongest meta-analyses share several features:

• Pre-registered protocol (PROSPERO entry available)
• Comprehensive search of multiple databases
• Two reviewers extracting data independently
• Risk of bias assessed for every included study using validated tools
• Heterogeneity examined and explained
• Publication bias assessed
• GRADE summary of certainty
• Pre-specified subgroup analyses
• Sensitivity analyses excluding high-risk-of-bias studies
• Transparent reporting of conflicts of interest
• Updated regularly as new evidence emerges (this is what Cochrane does well)

Cochrane reviews aren't the only high-quality meta-analyses, but they consistently exemplify most of these features. The Cochrane brand is a useful starting heuristic for evidence quality.

How to integrate a meta-analysis into your own thinking

For a non-specialist reader encountering a meta-analysis claim:

1. Find the actual publication. Don't rely on news coverage. The abstract usually contains the key findings.

2. Check the recency. Meta-analyses from 5+ years ago may have been superseded.

3. Check whether it's Cochrane. Cochrane reviews are generally high-quality starting points.

4. Read the heterogeneity discussion. If the authors don't engage with heterogeneity, the conclusion may be less robust than the headline suggests.

5. Check GRADE certainty if reported. "Low-quality evidence supports..." is meaningfully different from "high-quality evidence supports..."

6. Look at the effect size, not just the p-value. "Statistically significant" with a small effect size is different from a clinically meaningful effect.

7. Check for replications. Independent meta-analyses on the same question are reassuring when they converge; concerning when they don't.

What Proco's editorial position is

Meta-analyses are powerful evidence-synthesis tools when done well. They are also frequently overstated in consumer coverage — the headline conclusion often elides methodological nuance that matters.

When Proco's content cites a meta-analysis, we try to distinguish high-quality from low-quality sources, acknowledge heterogeneity where it's meaningful, and report effect sizes alongside significance. When the meta-analytic evidence is mixed across multiple analyses, we say so.

For readers evaluating health claims: a single meta-analysis is more reliable than a single trial. Multiple high-quality meta-analyses converging on the same conclusion is much more reliable than either. A meta-analysis without GRADE assessment of certainty deserves more scrutiny.

Related Proco pages

• How to read a clinical trial
• Why "evidence-based" gets misused
• Health misinformation: the scale of the problem

Sources

Proco provides educational, research-based information. This page describes meta-analysis methodology. It is not medical advice. Decisions about your own health belong with a qualified healthcare professional.

Schema (for implementation)

{
  "@context": "https://schema.org",
  "@type": "Article",
  "headline": "How to Evaluate a Meta-Analysis",
  "description": "Meta-analyses sit at the top of the evidence hierarchy but aren't all equally rigorous. This page describes how to read a meta-analysis — heterogeneity, GRADE, forest plots, publication bias.",
  "datePublished": "2026-06-01",
  "dateModified": "2026-05-31",
  "author": {"@type": "Organization", "name": "Proco"},
  "publisher": {"@type": "Organization", "name": "Proco", "url": "https://procohq.com"},
  "about": {"@type": "Thing", "name": "Meta-analysis methodology"}
}