Inference

Choosing a statistical test from the question and study design

Start with the outcome and comparison. Ask how the observations are related, including whether they are paired. Those choices help determine the method.

Specify the question and outcome

Are you estimating a typical value, comparing group means, comparing event risks or examining an association? A descriptive question may need a summary and plot rather than a hypothesis test. Match the analysis to the quantity you want to learn about.

Continuous numerical measurements differ from binary outcomes, ordered ratings, counts and time-to-event outcomes. Choose the analysis from the measurement type and study design, rather than familiarity with a test.

Identify dependence

Before-and-after measurements from the same person are related. Separate groups have a different structure and may be independent. Measurements nested within households, clinics or schools can also be dependent because of clustering. Spreadsheet rows do not necessarily equal independent observations.

paired t-tests examine the difference of each pair; independent samples t-tests do not consider those pairs. More complex studies require other ways to represent their dependencies. The amount of detail necessary to be able to recreate the study design is far greater than what can be included by simply reporting summary statistics.

Common starting points

For suitable data, a one-sample t-test compares a mean with a reference; Welch’s t-test compares two independent means; a paired t-test analyses matched differences. Inspect a scatter plot before choosing correlation or regression.

For categorical associations, a chi-square independence test needs a suitable approximation based on expected counts. Sparse tables may require an exact method. Check missing data, outliers and assumptions. Rank-based methods do not automatically test arithmetic means. Report an estimate and its uncertainty along with any test.

Worked example: the same measurements, different design

You have scores that represent each participant's performance at two different times. Both columns contain data from the same set of 18 individuals.

Calculate each participant's difference first; analyse the 18 differences with a paired approach.

Treating the two columns as if they were independent sets of data would eliminate the pairing and alter how you calculate your uncertainty.

Research questions and dependencies typically precede what type of test you use.

Sources for further reading