Tests and prediction

How to read a clinical prediction rule study

Distinguish model development from evaluation, check calibration and discrimination, and examine whether a prediction rule improves decisions.

Before you believe a prediction score, find out what it's predicting. That score may represent if a patient currently has a medical condition (current) or the likelihood of an occurrence in the future. Both calculations can produce scores that look simple, however there is a significant amount of work done before obtaining that score.

Understanding the work done to get that score requires understanding three components, which are: the target population, the timing of the use of the model, and the prediction period. When a model has performed well in terms of making predictions, it's difficult to understand the significance of that statement without knowing who was included in the model (population), what was predicted (the outcome), and at what point in time the outcome was measured.

How was the rule created?

Timing can be overlooked. All predictors must be available to assist during the time of decision-making. For example, suppose the model uses a discharge diagnosis to estimate what should have been recognised upon admission. However, if the discharge diagnosis isn't made until sometime later then the model is utilising information that wouldn't have been available when assistance was needed. That's referred to as "data leakage".

Missing records require review. Using only individuals with complete data can make analysis easier; however, selecting only those with complete data can change which people remain in the sample. Thus, the sample selected may no longer reflect the population in which the model will be utilised. Therefore, the author should describe how participants were recruited, how predictors and outcomes were measured, and how they handled missing data.

A flexible model has many more opportunities to identify random patterns. Thereby, this issue is most concerning in a very small development set. Factors that contribute to how confident one can be in results produced from both developing and testing a model with the same data are sample size, amount of outcome events collected and complexity of analysis.

Development vs Evaluation

Developmental studies develop or update a model while evaluative studies evaluate an existing model's predictive capability. Evaluating a model internally through methods like resampling provides insight into how dependent the apparent performance of a model is on the development data. While internal evaluation has merit for assessing performance, it does not replace evaluating a model externally with new participants or in a different relevant setting.

The differences between sites where predictive models are evaluated can have implications. Utilizing nearly identical processes in another hospital may yield differing results compared to an outpatient clinic that utilises different referral patterns. In addition, participants recruited at a later date may allow for evaluating predictive performance subsequent to changing circumstances.

An individual study demonstrating predictive capability cannot determine suitability for application to all places and times. Changes in case mix, measurements, care delivery and frequency of outcomes can occur post-implementation. The findings must therefore be interpreted within this framework.

Calibration in addition to Discrimination

Discrimination refers to a model's ability to differentiate individuals experiencing an outcome from those not experiencing an outcome. Researchers often utilise Area Under Receiver Operating Characteristic Curve (also referred to as c statistic) for binary outcomes to measure a model's discriminatory ability.

Calibration is concerned with whether the estimated probability values of an outcome correspond to the actual frequencies of occurrence. Although a model may rank individuals correctly regarding potential risk, it may systematically overestimate or underestimate risk. As a result, a good discriminatory result does not guarantee adequate calibration.

For example, assume a hypothetical model provided 100 participants with an average predicted one year risk of 10 percent. The model predicts approximately ten events among these participants based on their predicted risk levels. Assume twenty events actually occurred among these 100 participants. The observed ratio of events occurring among the sample is 0.2 or 20 percent. The predictions are less than the observed rate of events for this group. This type of comparison relates specifically to calibration. It offers no direct information regarding the model's ordering ability.

Sampling variability continues to influence the interpretation of this result regardless of calibration.

Calibration plots, uncertainty surrounding each estimate, and evaluations focused on subgroups offer additional information to support the interpretation.

A single accuracy percentage does not provide a complete assessment. An overall average can hide poor performance in some subgroups.

What occurs at the decision threshold

Decisions are typically made based on thresholds established by rules. These decisions involve sending individuals for further diagnostic procedures and outcome events that the rule would miss when its threshold is applied. Whether or not the trade-offs associated with these actions are acceptable depend upon the nature of the decision(s).

There are also issues related to staff response, implementation and effects on patients that require evidence of their own. Improved prediction does not necessarily indicate improved outcomes.

Provide appropriate utilisation for reporting and appraisal tools

TRIPOD+AI addresses reporting aspects relative to prediction models generated through regression-based techniques or machine learning techniques. TRIPOD+AI assists authors in providing essential information to facilitate reader comprehension. The completion of a reporting check-list does not assure low-bias risk.

PROBAST+AI provides appraisals of model quality, risk of bias, and applicability. PROBAST+AI expands upon and supersedes the original PROBAST tool. Reporting facilitates availability of prior work for critical examination; Appraisal provides assessment of validity of methodology and conclusions drawn.

Your reading note should provide clarity as to: population characteristics, available predictors, expected time frame for outcome event, location/site of evaluation, calibration, discrimination/ranking and expected decision-related consequences. These factors provide meaning for an otherwise promising score value.

Methodology references

  • TRIPOD+AI statement: updated guidance for reporting clinical prediction models
  • PROBAST+AI: an updated quality, risk of bias, and applicability assessment tool
  • PROBAST+AI tool