Appraising studies

How to appraise a qualitative research study

Assess the fit between a qualitative question, sampling, data collection, interpretation and the setting.

Qualitative studies can help researchers understand how individuals perceive an experience, why a service is difficult to use or how decisions are made. These questions differ from determining how frequently an event occurs or estimating the mean effect of an intervention.

Evaluating a qualitative study involves assessing whether the approach taken in collecting data corresponds to the question being investigated and whether the researcher's interpretations are based upon the evidence collected. Simply counting interviews or looking for a p-value cannot answer these questions on its own.

Start with the research question

Identify what the researchers wanted to understand. Interviews may suit questions about personal experiences. Observation may reveal what happens during a process that people find difficult to describe. Focus groups can explore discussion and disagreement, while also creating pressures that affect what participants say.

Review both the researcher's methodology description and their rationale for using this methodology. In addition to establishing relevance, the critical factor is whether the methodologies used make logical sense in relation to answering this specific question. Simply providing a title/label for a methodology does not equate to describing what was done.

Examine who took part

Qualitative sampling is often deliberate: researchers seek people with experiences or perspectives relevant to the question. That is different from selecting a probability sample to estimate a population percentage.

Ask who was invited, who agreed and whose perspective may be missing. Recruitment through a service may exclude people who stopped attending. An online interview study may miss people without dependable internet access or confidence using the required technology.

The number of participants (sample) has to be viewed within the context of the research questions asked, the variety of experiences represented by the sample and the amount, depth and quality of data generated. Claims that enough data were collected need to be explained. There is no single number of interviews that ensures a convincing study across all subjects and methods.

Look closely at collection and interpretation

A useful report describes the environment in which interviews or observations took place, how data were recorded and the procedures used to move from transcripts, field notes or other material to the final interpretations or themes. How did the researchers get from the data to their interpretations?

Supporting quotes help readers see the connection between the material collected and the researchers’ interpretation. They should support conclusions rather than simply embellish them. The researcher should demonstrate awareness of conflicting opinions, unusual case examples and quotes/data that conflict with their overall interpretation.

The researcher’s involvement in the research process also plays a significant part. Their past work, professional position and relationships with participants can affect what is stated and how it is interpreted. Reflexivity means openly examining these influences. This doesn’t imply eliminating all such influences from the research process.

Work through a practical example

Suppose an imaginary study interviews 18 people about missed outpatient appointments. All participants were recruited through an English-language text reminder, and interviews took place online.

The analysis identifies confusion about appointment times and difficulty arranging transport. Those findings could be useful, especially if the report shows how they arose from detailed accounts and how participants with different experiences were considered.

However, the recruitment method may exclude people who do not read the reminders, cannot use the online platform or prefer another language. The study cannot establish that 18 interviewees represent every person who misses an appointment. Nor can the number mentioning transport be treated as a reliable estimate of its prevalence in the whole service.

In appraising this study, keep the valuable insight about transport while considering which perspectives the study may not have captured. It may also suggest questions for further investigation or changes worth evaluating.

Judge relevance without demanding statistical representation

Transferability concerns whether the findings may be useful in another setting. Readers need enough information about the participants, service and context to make that judgement. A thoughtful study in one clinic may illuminate another clinic's problems without proving that the same explanation applies everywhere.

CASP provides a framework for asking questions about qualitative studies. COREQ addresses the reporting of interviews and focus groups, including the research team, study design and analysis. Clear reporting assists appraisal, but compliance with a reporting checklist does not establish that the interpretation is convincing.

Combining several qualitative studies into an evidence synthesis requires appropriate methods and confidence assessments. It should not be treated as an ordinary numerical meta-analysis.

A balanced final judgement describes what the study helps us understand, the evidence supporting that understanding and where the interpretation is less secure. Preserve uncertainty and context rather than simplifying a rich description into an easy pass or fail.

Methodology references

  • CASP Qualitative Studies Checklist
  • Tong et al.: Consolidated criteria for reporting qualitative research (COREQ)
  • Cochrane–Campbell Handbook for Qualitative Evidence Synthesis