A practical critical appraisal checklist
Use a focused question, a design-specific checklist and a short evidence note to judge a research paper and choose a sensible next step.
Imagine you are reading a paper about text reminders for outpatient appointments. Its headline says reminders reduced missed appointments by a quarter. Your task is to decide whether the finding is credible enough to inform a small service pilot. A practical appraisal should help you answer that question and record what remains uncertain.
Write a short note on each paper. Include:
- The research question.
- The methods used.
- What the researchers found.
- How the findings relate to practice or policy.
The idea is to develop an argument about the evidence rather than simply accumulate ticks for an overall quality rating. When you have completed the notes, consider how much confidence you can place in the findings and what remains unclear.
Define the question before opening a checklist
Describe which patients were invited to participate in the reminder example; the reminder being tested; the comparison group (usual care, phone call, etc.) and the comparison itself; and the outcome(s) measured (attendance at one appointment vs. attendance over several months). The distinction is important because studies of reminders for existing appointments cannot directly establish whether reminders improve access to appointments in the first place. Write your question as a single sentence, referencing the appropriate patient population and timeframe, so you may compare the claims made by this study with those that have been actually measured.
Match the appraisal to the study design
Review the methods section to identify the study design. Randomised trials, before-and-after service evaluations and interview-based studies require different appraisal questions.
When reviewing a trial, assess allocation, comparison groups, follow-up and analysis. For a diagnostic study, assess participant selection and the reference standard. For a prognostic study, assess the initial cohort and subsequent follow-up. For qualitative research, consider how the methods address experiences or explanations. For a systematic review, assess how studies were identified, selected, appraised and combined.
An interview study may help explain why patients miss appointments but will not be able to estimate the causal effect of a reminder programme on its own. Choose a design-specific appraisal tool rather than applying trial questions to all kinds of papers.
Examine validity before precision
Re-look at your reminders study. Were the groups set up in a manner that would allow for an unbalanced comparison (e.g., two different booking systems)? Apart from receiving a reminder, did both groups have equivalent arrangements for their bookings? Were attendance records tracked similarly? Did the analysis retain people in the groups to which they were assigned?
Review the unit of analysis. If the same patient provided several appointments, those observations may be related. Treating each appointment as independent could give a misleading impression of precision.
A large sample or a narrow confidence interval does not correct a systematic difference between groups. First determine whether the methods support the claimed comparison. Then examine the size of the estimate and its uncertainty.
Separate missing reporting from a demonstrated problem
Suppose a paper calls allocation “random” without explaining the procedure. Record what is missing. Do not rewrite that uncertainty as proof that allocation was flawed, or assume that a suitable procedure must have been used.
Look for protocols, registration records, supplementary materials or companion reports and examine the information they provide. Distinguish between "not described", "described and appears acceptable" and "described with a problem". Missing information has varying implications depending upon the specific issues addressed and the contexts within which they were raised. Some omissions render one unable to draw a firm conclusion.
Each appraisal note should refer to the relevant evidence in the paper. Do not apply the same judgement indiscriminately to every outcome. For example, attendance records and self-reported satisfaction may have different risks of bias.
Put the result into usable terms
In a hypothetical result, missed appointments fall from 20% to 15%. The relative reduction is 25%, but the absolute difference is five percentage points: five fewer missed appointments per 100 people. Both describe the same comparison.
Find the confidence interval, denominators and follow-up period. Determine whether the result was the planned primary outcome and whether exclusions or missing data might change it. Statistical uncertainty and uncertainty about the methods should remain visible as separate concerns.
Decide what the paper supports next
Compare the participants and service with your own setting. Consider message delivery, language, phone access, staff workload and what an attendance improvement would mean locally. A credible effect elsewhere may need adaptation or further evaluation.
Finish your reading note with four short entries:
- The specific question the paper answered.
- The main result, including its absolute size and uncertainty.
- The concern most likely to change your interpretation.
- A proportionate next step.
For this example, that next step might be checking the trial protocol, looking for a systematic review or designing a monitored pilot. State what evidence would make you reconsider. The appraisal is most useful when another reader can follow the connection between your observations and your decision.
Methodology references
- Critical Appraisal tools, Oxford Centre for Evidence-Based Medicine
- Asking focused questions, Oxford Centre for Evidence-Based Medicine
- Cochrane Handbook, Chapter 8: Assessing risk of bias in a randomized trial
- Cochrane Risk of Bias 2: methodological guidance and FAQs