P-values: what they answer and what they leave out
A p-value shows how compatible your data are with the statistical model you chose. It doesn’t tell you the probability that your hypothesis is correct.
Begin with a specified model
A hypothesis test begins with the null hypothesis, test statistic and a definition of more extreme results. A one-sample t-test could examine whether a population mean differs from 50. A two-sided alternative considers both positive and negative departures from that number.
The p-value is calculated assuming the null model and its other assumptions are correct. It is the probability, under that model, of a test statistic at least as extreme as the observed value in the direction specified by the alternative.
Do not reverse the probability
A p-value of .03 does not mean a 3% chance that the null model is true or a 97% chance that a preferred explanation is correct. It describes the probability, under the assumed null model, of a test statistic at least as extreme as the one observed.
Effect size and uncertainty still matter. Large studies can detect very small effects, while small studies may leave even important effects uncertain. Report estimates in meaningful units with confidence intervals.
Plan before seeing the answer
Define your primary research question and statistical methodology as far in advance as possible. However, if you have multiple potential research questions/outcomes/models, it can be tempting to search all of them, and report only the most statistically significant results. This will distort what was found. In this case, either use some type of multiplicity adjustment (e.g., Bonferroni), or clearly label these analyses as "exploratory".
Determine your appropriate directional hypothesis before viewing data to determine the direction of effect. Large p-values do not prove that the means are equal. A value slightly above .05 does not represent a clear or scientifically meaningful distinction from a value slightly below .05, even though the same decision rule for determining whether there is a difference at alpha = .05 would produce a different conclusion for each of those values.
Worked example: interpreting a reported value
A pre-specified two-sided test comparing average appointment times reports p = 0.03.
Under the null model, a test statistic this extreme or more extreme has probability 0.03.
Ask how large the estimated difference is, how uncertain it is, and whether the design supports the comparison. The p-value alone cannot answer those questions.
Use the p-value in conjunction with your estimate of the effect-size, and with your assessment of the precision of that estimate, and the quality of the data upon which it is based.