Evidence and decisions

How to appraise clinical practice guidelines

Assess how a guideline was developed, how its recommendations follow from evidence and whether its scope fits the question.

A guideline can look authoritative while leaving important questions unanswered. Who selected the evidence? Which outcomes mattered? Did the authors explain why one option was preferred? Appraising a guideline means tracing the route from a clinical question to a recommendation, rather than accepting its presentation as evidence of quality.

While the date of publication provides some valuable information, it does not necessarily follow that a more recent document will also be of higher quality. Look for scope, date of last search, development processes used, planned updates and supporting evidence for each recommendation you plan to evaluate.

Establish what the guideline covers

Identify the population, setting and decisions included. A document written for specialist assessment may not cover the same circumstances as one intended for first-contact care. Read exclusions carefully. Check whether recommendations concern an entire population or a defined subgroup.

A guideline must state its objectives clearly enough for readers to assess whether the selected evidence addresses them. General statements about improving care are insufficient. Readers need to understand what constitutes improvement and whose outcomes were considered.

Examine the development process

AGREE II is an established instrument for appraising guideline development and reporting. It considers six domains: scope and purpose, stakeholder involvement, rigour of development, clarity of presentation, applicability and editorial independence.

Use those domains to organise a careful reading. Ask whether the development group included the relevant professional perspectives and meaningful patient or public input. Look for systematic methods to find and assess evidence, explicit links between findings and recommendations, and a process for external review and updating.

Disclosures should be considered alongside how competing interests are managed. A list of declared interests cannot demonstrate that they had no influence. The presence of a recognisable sponsor does not establish transparent methodology.

AGREE II evaluates the quality of guideline development and its reporting. It cannot determine whether every clinical claim is accurate. Poorly documented methods can be difficult to evaluate, and attractive writing does not guarantee that a recommendation rests on reliable evidence. Document the rationale for each appraisal judgement rather than summarising the exercise as an unexplained total score.

Follow one recommendation back to its evidence

Having selected a specific recommendation, locate the evidence review that supports it. Identify the important benefits and harms, the certainty of those findings and the assumptions used to interpret them. Determine whether the recommendation extends to populations or settings outside those studied.

For illustration, imagine a fictional intervention that lowers an event risk from 20 in 1,000 to 15 in 1,000 over two years. That is five fewer events per 1,000 people, or a 25% relative reduction. The relative figure alone does not show the size of the expected benefit. A recommendation should also account for burdens, adverse outcomes, uncertainty and relevant alternatives.

A recommendation may reasonably involve considerations that were not measured in trials, including feasibility, resources or unequal access. The important point is whether the authors identify those considerations and explain their influence. Evidence informs recommendations; it does not remove the need for judgement.

Read the strength and the wording

Guideline developers distinguish recommendations intended for most people in a defined situation from recommendations for which reasonable choices may vary. Read what the developer says about its grading and terminology systems. Do not assume that words, letters or numbers used in one organisation’s document have the same meaning in another’s.

Strength of recommendation and certainty of evidence describe different things. An apparently firm statement should still have a visible rationale. Where options have closely balanced advantages and disadvantages, the guideline should leave room for preferences and individual circumstances.

Check whether the recommendation is usable

Assess the resources, skills, equipment and follow-up needed to implement a recommendation. Consider whether the guideline addresses barriers to implementation and describes methods to assess it. Recommendations that assume immediate access to specialists may need careful interpretation in a service with different capacity.

Finish with a short appraisal of the specific recommendation: its intended population, the evidence supporting it, the main limitations and the conditions needed for use. That is more informative than calling an entire guideline good or bad. Where crucial development details are absent, make the uncertainty explicit instead of filling the gaps with assumptions.

Methodology references

  • AGREE II Users Manual and 23-item Instrument
  • Brouwers et al. AGREE II: advancing guideline development, reporting and evaluation in health care
  • NICE guideline manual, chapter 9: Interpreting evidence and writing the guideline