Evidence and decisions

Grading the certainty of research evidence

Understand GRADE, outcome-specific certainty and the reasons confidence in a research finding may be reduced.

When a review gives evidence a GRADE rating, keep the explanation beside the result. The label is shorthand for judgements about a body of evidence, tied to a particular comparison and outcome. It cannot be carried intact to every conclusion in the review.

GRADE uses high, moderate, low and very low certainty. Those words tell readers how confidently to interpret the findings, provided the reasons behind the rating are available. A long list of studies or an impressive sample size cannot supply those reasons on its own. Studies may agree because they share a flaw; a precise estimate may answer a question different from the one that matters.

Start with an outcome and a comparison

The fictional rehabilitation review is a useful way to think about this. Its authors compare two programmes and report walking ability, pain, treatment burden and hospital admission. Each outcome needs its own assessment. Good evidence about walking does not settle whether either programme reduces admissions.

There is also a time dimension. Evidence about pain after six weeks does not establish what happens after a year. The population and setting belong in the description too. Where a review includes substantially different groups, a combined rating may hide differences that deserve separate consideration.

This is why a statement such as the evidence is strong remains incomplete until the reader knows which comparison, outcome and follow-up it describes.

Five reasons for reduced confidence

The GRADE assessment considers several ways in which the evidence might mislead or leave a decision unresolved:

  • Risk of bias concerns distortion introduced through a study's design, conduct or analysis.
  • Inconsistency concerns differences between results that have not been adequately explained.
  • Indirectness arises when the evidence differs from the intended question in its participants, intervention, comparison or outcome.
  • Imprecision matters when the uncertainty is wide enough to permit effects that would lead to different decisions.
  • Publication bias concerns the possibility that unavailable or selectively reported results give readers a distorted view of the research.

These are judgements to explain, rather than points to add up. The same underlying problem can appear in more than one domain, so it should not automatically be counted twice. Readers need to understand how serious a concern is and why it affects the conclusion.

In the traditional GRADE approach, randomised trials commonly begin at high certainty. Important limitations can lower that rating. Non-randomised evidence needs careful handling, including the risk-of-bias method used and whether any upgrading considerations apply. The study design starts the assessment; it does not complete it.

Read the estimate beside the rating

Suppose an entirely fictional review estimates that a programme reduces an event from 100 in 1,000 people to 80 in 1,000 over one year. Its uncertainty interval is compatible with anything from 40 to 130 events in 1,000. The central estimate suggests 20 fewer events, but the interval also allows a substantial reduction or an increase.

That range contains very different possibilities. A reader weighing the result needs to know how much benefit or harm would matter for the particular outcome. Whether the interval includes the no-effect value is only part of that discussion.

A different problem arises when the estimate comes entirely from specialist centres but its intended use is in a community service with fewer staff. Greater numerical precision would not resolve the question of whether the result transfers. The evidence may be answering the original setting's question more securely than the new setting's.

Certainty does not choose an option

A recommendation brings in considerations beyond confidence in an effect estimate. Benefits have to be considered alongside harms, burdens, resources and what people value. Even convincing evidence of a small benefit can leave a difficult choice. Conversely, low certainty does not establish that the true effect is zero.

A summary-of-findings table becomes more useful when its pieces are read together: the outcome, absolute effect, follow-up, rating and reasons. An empty cell may mean that the relevant evidence was not available. It should not be taken as reassurance about an unmeasured outcome.

A fair account of a review leaves the reader with both the finding and its limits. It explains what the evidence suggests, how confidently it suggests it and which uncertainty could change the interpretation.

Methodology references

  • Cochrane Handbook, Chapter 14: Summary of findings and certainty of evidence
  • GRADE Working Group: criteria for applying or using GRADE
  • Guyatt et al. What is quality of evidence and why is it important to clinicians? BMJ 2008