Tests and prediction

How to appraise a prognosis study

Read research about future outcomes by checking the starting population, follow-up, outcome definitions, missing data and survival estimates.

A prognosis study describes what happens after a defined starting point. This might be a diagnosis, an admission or the beginning of treatment. Possible outcomes include recovery, recurrence, hospital use or death. Each estimate needs a population and a time period to make sense.

“Ten per cent experienced the outcome” is incomplete. Ten per cent of whom, after how long, under what care, and with how much information missing? These questions turn a percentage into a result you can assess.

Make sure participants begin at a comparable point

Cohorts assembled near a recognisable beginning of a condition can describe its subsequent course. Mixing newly diagnosed people with those who have lived with the condition for years may answer a different question. People who died or recovered before recruitment will be absent.

Assess the eligibility criteria, recruitment settings and dates. A specialist service may have a different mix of disease severity from primary care. Care patterns also evolve over time, so an estimate based on an older cohort may not describe outcomes under current practice.

Prognostic factors are identifiable characteristics related to subsequent outcomes. The establishment of an association between a prognostic factor and a subsequent outcome does not imply causation. Prediction models frequently utilise multiple prognostic factors to estimate risk without implying that altering any single factor would alter the likelihood of developing an outcome.

Check how outcomes were established

Outcome definitions should be clear, and measurements should be consistent. An objective event, such as a recorded hospital admission, may still depend on local policy or access to care. A composite outcome may combine events of very different importance.

Check whether outcome assessors knew the prognostic information being examined, especially when judgement was involved. Was follow-up long enough to observe the outcome of interest? Could the frequency of assessments miss events or obscure when they occurred?

Follow the people who disappear from follow-up

Missing information can substantially alter a prognosis estimate. Consider a hypothetical cohort of 200 people recruited for a one-year yes-or-no outcome. At one year, outcome status is known for 160, and 16 have experienced the event. Forty cannot be contacted.

Among those with known outcomes, the event proportion is 16/160 = 10%. Dividing the 16 known events by all 200 gives 8%, but this treats every missing outcome as if no event occurred.

In the opposite extreme, suppose all 40 missing people experienced the event. There would then be 16 + 40 = 56 events among 200 people, or 28%. The range from 8% to 28% is a simple missing-data bound, not a confidence interval. It shows why incomplete follow-up deserves scrutiny.

Ask why people were lost and whether their characteristics differed from those retained. A large sample does not automatically overcome selective loss.

Read survival curves with their assumptions

Time-to-event methods allow people to contribute different amounts of follow-up. Someone whose follow-up ends before an event is observed may be censored at their last known time. Censoring does not mean they remained event-free forever.

Kaplan–Meier estimates depend on assumptions about censoring, including whether people no longer observed have comparable prospects to those still followed. Read the numbers remaining at risk beneath the curve. A dramatic late change based on very few people is less secure than an earlier estimate supported by more observations.

Competing events require particular care when estimating absolute risk. Death, for example, can prevent a later non-fatal outcome from occurring. Treating such deaths as ordinary censoring can overestimate that outcome’s cumulative incidence. The analysis needs to match the question being asked.

Keep absolute risk separate from relative measures

A hazard ratio compares event rates among people still at risk over time under a model. It is not a direct statement that an individual’s probability over a fixed period has doubled. Ask for absolute risks at relevant time points and the uncertainty around them.

Finally, compare the study population, care and follow-up with the setting you have in mind. A group average describes a distribution of possible outcomes. It does not give everyone in the group the same future or eliminate uncertainty about an individual person.

Methodology references

  • Prognosis research strategy (PROGRESS) 1: A framework for researching clinical outcomes
  • Prognosis and prognostic research: what, why, and how?
  • Survival probabilities: the Kaplan–Meier method
  • Bias by censoring for competing events in survival analysis