Appraising studies

How to appraise a systematic review and meta-analysis

Check how a review found, selected and assessed studies before interpreting its pooled result.

A systematic review uses explicit methods to bring together research addressing a question. A meta-analysis combines study results statistically. A systematic review need not contain a meta-analysis, and a precise pooled estimate can still be based on weak evidence. The scope and methods need attention before the conclusion is accepted.

Read the question and eligibility rules

Before you judge the conclusion, find out who the review is addressing, what interventions it is evaluating, what comparison is being made, what outcomes the reviewers are measuring, and for how long. The follow-up period matters as well. If a reviewer limits themselves to only include very short term studies of follow-up, then they are limited in what they can say about what happens over longer periods of time.

Eligibility criteria need to be set prior to assembling the results. While changes do not always invalidate a review, unexplained changes after the results of a review become known can certainly impact its answers. Transparency and consistency are important during all parts of the process. Readers should know what has changed and why.

Also note, a study and a manuscript are often used interchangeably. However, there can be many different manuscripts describing the same participants. Using each of these manuscripts as independent evidence sources would give those participants too much power.

Follow the search and selection process

The search date provides a clearer picture of the evidence included in a review than the publication date alone. The databases searched, the search terms used, and any restrictions placed upon language or publication status provide clarity into what likely could have been found and what possibly could have been missed.

Each author's assessment of whether each manuscript meets the eligibility criteria should be documented. Documentation of the eligibility criteria assessments allow readers to see the history of how studies were selected. Authors should document the reasons for excluding manuscripts that appear to be eligible. A flow diagram can demonstrate the pathway through which studies were selected; however, a flow diagram cannot verify that the search was thorough.

Documentation of screening and data extraction processes is also necessary. Documentation of screening and data extraction helps readers examine the processes and identify possible errors. It does not ensure error-free processes or reliable subsequent analyses. Independent validation of both screening and data extraction can further support reliable use of source materials, although independent validation cannot establish reliable source material or repair flawed eligibility criteria. The PRISMA statement provides guidance on transparent documentation of screening and data extraction processes. PRISMA is not designed as a methodologic quality rating scale.

Review studies prior to combining them

In order to select studies for a review, investigators must develop comprehensive descriptions of each study. Investigators must report sample demographics, study design, data collection procedures, analytical techniques, and study limitations. Reporting study strengths and limitations will enable readers to assess the implications of potential study variations on the final results.

Study descriptions should indicate differences in participant demographics, types of interventions studied, and definitions of outcomes. An identical patient population does not ensure that a combined estimate will convey meaningful information.

Appropriate risk of bias assessments should guide the review’s synthesis. Sample size does not establish freedom from bias. A large study with significant bias can disproportionately influence the estimated effect size. The review should detail how such concerns were addressed.

The AMSTAR 2 instrument provides investigators with a structured framework for assessing reviews of interventions. Investigators should specifically examine potentially critical issues separately. Summarizing checklist responses can obscure a major deficiency hidden amongst otherwise acceptable checklist responses.

Understand what pooling means

Pooling generally involves forming a weighted average of compatible effect estimates. It is not a simple average of percentage values taken from abstracts; such averages could represent misleading summaries.

As an example, two studies investigate comparable groups using similar measures of the same symptom scores. The mean differences (intervention vs control) are -2 and -4 points respectively with standard errors of 1 and 2 points. In a simple inverse variance-weighted fixed effect calculation, the first study (mean difference -2, standard error 1) has greater precision and receives more weight. The weights are 1 and 0.25. The pooled mean is (-2 × 1 + -4 × 0.25) / (1 + 0.25) = -2.4 points. This demonstrates the mechanics involved in pooling; it does not provide any information about whether the examples should be pooled or whether the assumed conditions are valid or whether the resultant difference in symptoms has practical relevance.

Determine residual variation and uncertainty

Individual studies can be visualized within a forest plot as separate findings. The magnitude and direction of these findings require examination in conjunction with differences in methodology and population examined. Assessments of statistical heterogeneity are part of that examination; they are not substitutes for it.

Random effects models generate a point estimate for the mean effect consistent with their underlying assumptions. Random effects models do not eliminate differences between studies. Confidence intervals generated by random effects models reflect uncertainty around estimates of the mean; they do not describe the range of effect sizes across settings. Prediction intervals are related to estimating variability in effect sizes across settings, and prediction intervals can be developed when applicable.

Sensitivity analyses are conducted to determine if conclusions drawn from pooled estimates vary substantially depending upon specific choices or studies. Substantial changes after removal of a poor-quality study warrant explanation. Similarly, sensitivity analyses examining potential omissions of relevant evidence deserve attention. A useful summary provides estimates, estimates' uncertainties, and limitations -- including limitations associated with applying the review findings to your desired context.

Methodology references

  • The PRISMA 2020 statement: an updated guideline for reporting systematic reviews
  • AMSTAR 2: a critical appraisal tool for systematic reviews
  • Cochrane Handbook, Chapter 10: Analysing data and undertaking meta-analyses