Research skills

How to interpret the results of a research study

Read research results through the estimand, absolute and relative effects, confidence intervals, p-values and practical importance.

In the results section, an investigator's original research question becomes an estimate. The label “significant” can make an estimate appear settled before the reader sees its size. Estimating and displaying estimates along with their uncertainty gives the reader a better foundation for assessing findings. Consider a hypothetical evaluation of an appointment reminder service. There are two groups; one group is assigned to use new reminders (n=400) and the other group is assigned to continue using regular reminders (n=400). After six months, 48 people assigned to new reminders missed their scheduled reviews and 60 people assigned to regular reminders missed their scheduled review. As mentioned earlier, we will assume complete data on outcomes.

define the effect being estimated

The estimand defines the specific effect being sought, while an estimator is the method used to estimate it. The definition includes target population, alternatives, outcome, summary measure and how relevant events after assignment are handled, and the specific type of analytical objective. The estimand defines the analytical objective but does not define how the answer was derived. The estimand helps determine study design & which method(s) should be used for estimation.

To evaluate the appointment reminder service, you could either evaluate whether assigning people to receive the new reminders regardless of their usage would impact the number of people who missed their scheduled review or you could ask what would happen if everyone used the new service as intended. That is a different target. If you drop those who did not use the new service, you will not automatically answer the second question. Changing the analysis population can change both your question and result.

Timing belongs within the outcome definition. Missing a scheduled review within six months represents a different outcome than the total number of missed appointments over several years.

Describe absolute and relative effects

The observed risk is 48 divided by 400 or 12% and 60 divided by 400 or 15%. The risk difference is −3 percentage points (new reminder minus usual reminder). Expressed as a reduction, that is 3 fewer missed reviews per 100 people over six months.

The risk ratio is 12 percent divided by 15 percent or 0.80. The corresponding relative risk reduction is 20 percent. Both descriptions are correct, however, the relative figure needs its baseline risk to convey the practical scale of the difference.

The above example data is artificial. It does not indicate whether a real appointment reminder service would work.

read confidence interval as well

Using a simple approximate calculation provides a 95 percent confidence interval for the risk ratio of 0.56 to 1.14. Although the point estimate favors the new reminders, the interval contains both a substantial decrease and a minor increase in missed reviews.

This leaves room for very different interpretations. We have not established that the reminders have identical effects. Neither has it been determined that we have a dependable 20 percent reduction.

With a 95 percent confidence procedure, repeated studies would produce intervals containing the true parameter about ninety-five percent of the time provided assumptions hold. Once this particular interval has been calculated, this does not provide a fixed parameter a 95 percent probability of lying inside it. Systematic bias and poor models can also create uncertainty that is not captured by the interval.

position p-value properly

For these data, an approximate log-risk-ratio test gives a p value around .22. This describes compatibility between data and specified no-effect model plus assumptions. This is not a twenty-two percent chance that reminders are equally effective, nor is it a measurement of the probability that finding occurred "by chance".

Sample size impacts interpreters. Large samples may make small differences statistically detectable while small samples may leave important differences uncertain. Keeping estimates and their confidence intervals together when evaluating authors' conclusions will help assess practical importance. Statistical significance alone does not establish practical importance.

identify number of results examined

A publication often has more than one outcome, follow-up periods and subgroup comparisons. A headline based upon smallest p value may over state evidence. Distinguish planned main analyses from exploratory findings, explain how multiplicity was addressed, and provide contextual information to assist reader judgment of findings.

Assume all outcome data was available in this example. If appointment records were incomplete in a real report, authors must explain what was missing and how it was handled. Counting people whose attendance records are missing as having attended creates unsupportable assumption. Alternative reasonable assumptions regarding missing data may affect interpretation. Narrow confidence interval from any single analysis does not render its missing-data assumptions harmless.

deciding what size means in context

Three fewer missed reviews per hundred people could mean something different in a small clinic and a larger service. Implementation costs, staffing requirements, discomfort created by inconvenience and any adverse effects also belong in discussion.

A good account keeps together the observed difference, the range of effects compatible with the data and the assumptions behind the calculation, so readers can judge practical importance while keeping uncertainty visible.

Methodology references

  • ICH E9(R1): Addendum on estimands and sensitivity analysis in clinical trials
  • Wasserstein and Lazar: The ASA Statement on p-Values, Context, Process, and Purpose
  • Cochrane Handbook, Chapter 15: Interpreting results and drawing conclusions