Critical Appraisal: Quantitative - London Links · Research methods Quantitative ... Useful for finding precise answers to defined questions Objective Deductive reasoning Statistical

Critical Appraisal: QualitativeSian Aynsley

Information Skills Trainer

South London Healthcare NHS Trust

Queen Elizabeth, Woolwich

Outline of session

Qualitative research: what, why, how?

◦ Presentation

◦ “Qualitytative Street” exercise

◦ Appraisal of article

Questions and follow up

Objectives of session

Understand the difference between

quantitative and qualitative research

Become familiar with major features and

terminology of qualitative research

Apply critical appraisal to a qualitative

paper using the CASP checklist

Research methods

Quantitative

◦ Uses numbers to

describe and analyse

◦ Useful for finding

precise answers to

defined questions

◦ Objective

◦ Deductive reasoning

◦ Statistical sampling

Qualitative

◦ Uses words to

describe and analyse

◦ Useful for finding

detailed information

about people‟s

perceptions and

attitudes

◦ Subjective

◦ Inductive reasoning

◦ Theoretical sampling

Quan or qual?

Quan or qual?

How many parents would consult

their GP if their child had a mild

temperature?

Why do so many parents worry

so much about their child‟s

temperature?

What proportion of smokers

have attempted to give up

smoking?

What stops people giving up

smoking?

Qualitative

Qualitative

Quantitative

Quantitative

Quan or qual?

“a frequent criticism is that qualitative

data...amounts to little more than anecdotes,

personal impressions or conjecture” (Mays and Pope, 2000)

“Doctors have traditionally placed high value on

number based data, which may in reality be

misleading, reductionist and irrelevant to the

real issues” (Greenhalgh, 2010)

Quan or qual?

“Both methods are valid if applied to

appropriate research questions, and they

should complement each other” (Bowling, 2010)

“The view that the two approaches are

mutually exclusive has itself become

„unscientific‟” (Greenhalgh, 2010)

Qualitative research: why?

Different people have different ways of making

sense of the world

“Allows for the generation of rich data and the

exploration of “real life” behaviour, enabling

research participants to speak for themselves” (Kuper et al, 2008)

Results can explore issues that are poorly

understood or extend / modify existing

theories

Qualitative research: how?

Research questions have an open focus

A variety of methods and approaches can be

used which are rigorous and systematic

“A method of naturalistic enquiry which is

usually less obtrusive than quantitative

investigations and does not manipulate a

research setting” (Bowling, 2002)

◦ Natural settings are used

Results may be transferred to similar situation

but are usually not totally replicable

Components of qualitative

research: IMRAD

Introduction

◦ Why?

Methods

◦ How?

Results

◦ What?

Discussion

◦ Meaning

Types of question

What? How? Why?

“What is it like to be the mother of an

unborn baby?”

“How do smokers feel about the various

smoking cessation options available?”

“Why do people not take the medicine

prescribed to them?”

Theoretical frameworks

“Qualitative researchers pay attention to the

theoretical bases of their methodological

approaches, so their choices of what theory

and which methodology to use are related” (Kuper et al, 2008)

Qualitative methodologies

Discourse analysis

◦ Analyses language and its role in the social

world

Ethnography

◦ Studies a group‟s cultural beliefs and values

◦ Originates from anthropology

Grounded theory

◦ Generating a theory based on data collected

Qualitative research methods

Different research methodologies may

employ the same method

Most common methods:

◦ Observation

◦ Interviews

◦ Focus groups

◦ Questionnaires

Observation

Used most frequently by ethnographers

Participant observation: researcher

takes an active role in group interaction

Non-participant observation:

researcher takes unobtrusive role to

minimise bias

Interviews

Widely used research method, used in all

methodologies

Pre-set questions, broad list of topics or

mix of both

◦ Structured

◦ Semi-structured

◦ Unstructured

Focus groups

Group interview

Capitalises on communication between

research participants to generate data

Requires expertise to manage

Questionnaires

Can vary in structure

◦ Structured (Lickert scale)

◦ Unstructured (Market survey)

Content of question differentiates a

qualitative questionnaire from a

quantitative – more than yes / no

Free text questions

Sampling

Tends to be small

“Biased”: randomisation not an issue

Study must clearly define:

◦ Sample individuals - characteristics

◦ Study inclusion / exclusion criteria

◦ Recruitment

◦ Sampling strategies

Sampling strategies

Convenience sampling

◦ Subjects easy to recruit, likely to respond etc.

Purposive sampling

◦ Based on purpose of study

Snowballing

◦ Building a network of respondents

Theoretical sampling

◦ Based on emerging findings

Setting

Venue should be clearly described

Appropriate setting lessens chance of

performance bias

Familiar environment best

Data analysis

Interpretations made by researcher(s)

from descriptions / observations of

participants – inductive approach

Data should be analysed individually, then

compared with other data to identify

recurring elements

Content analysis: counting words, coding

data

Data saturation: themes start recurring

Data analysis

Any use of computer software for data

analysis should be stated (ex: NVivo)

Analysis should be repeated by more than

one researcher

Themes synthesised from raw data should

be described (thematic analysis)

Large amounts of data are usually

generated

Triangulation

Comparison of results from:

◦ Two or more different methods of data

collection (e.g. interview and observation)

◦ Two or more data sources (e.g. interviews

with members of different interest groups)

◦ Two or more theoretical approaches

Researcher bias / Reflexivity

To address performance bias, researchers

should:

◦ Keep a reflective component in their field

notes

◦ Engage in dialogue with the subjects to clarify

their views

◦ Consider what the biases could have been

Ethics

Ethical standards, approval, issues should

be discussed

◦ Anonymity (if possible)

◦ Confidentiality (always)

◦ Informed consent (always)

Follow up interview to check whether

findings are recognisable by the subject

◦ Respondent validation

Transferability

Can I apply these results to my own

practice?

◦ Participants‟ socio-cultural origin

◦ Age range

◦ How the researchers had access to this

group

◦ The researcher‟s relationship with the

subjects

Can be difficult to transfer results

exactly

Summary

Researchers should consider:

◦ How a sample is recruited

◦ Whether the setting for data collection is

appropriate

◦ How the data is analysed

◦ Whether the findings make sense to the

subjects

29

The “Qualitytative Street”

exercise*

Focus group study

Participants must have some experience

of eating Quality Street chocolates and

know which one is their favourite

Do you want to participate?

Is there anyone who does not wish to

participate?* Source: Jenny Tancock, former Clinical Librarian at University Hospitals of

Morecambe Bay NHS Trust

The “Qualitytative Street”

exercise: appraisal

How was the group “recruited”?

◦ Did anyone ask what the study was about

before agreeing to participate?

Non-participants: why did you choose not

to take part?

◦ What are some of the reasons participants

might choose to opt out?

The “Qualitytative Street”

exercise: appraisal

How were the data “collected”?

◦ Observation

◦ Interview

◦ Focus group

◦ Questionnaire

Sweet choice / Sweet preference

How did we know when to stop

collecting data?

The “Qualitytative Street”

exercise: appraisal

How were the data “analysed”?

◦ Content analysis

◦ Thematic analysis

◦ Use of computer software

How might the two methods of data

collection be compared?

◦ Triangulation

The “Qualitytative Street”

exercise: appraisal

Reflexivity

◦ How did you feel about participating in a

study without knowing the objectives?

What aspects of the study might have

affected how you participated?

◦ Setting

◦ Group

◦ Role of researcher

The “Qualitytative Street”

exercise: appraisal

How could the study have been

improved?

If this was a real study, what would be

good practice?

A final thought...

“Qualitative methods really come into

their own when researching uncharted

territory – that is, where the variables of

greatest concern are poorly understood,

ill-defined and cannot be controlled” (Greenhalgh, 2010)

Any questions?

References and further reading

Ajetunmobi, O (2001) Making sense of critical appraisal

Arnold, London

Bowling, A (2010) Research methods in health (2nd ed)

Open University Press, Buckingham

Crombie, I K (1996) The pocket guide to critical appraisal

BMJ Books, London

Dawes, M (2005) „Introduction to critical appraisal‟ in

Dawes, M et al Evidence-based practice (2nd ed)

Elsevier, Edinburgh

Greenhalgh, T (2010) How to read a paper: the basics of

evidence-based medicine (4th ed) BMJ Books, London

References and further reading

Kuper, A (2008) An introduction to reading and appraising

qualitative research, BMJ vol 337(7666) p404-9

Patton, MQ (2002) Qualitative research and evaluation

methods (3rd ed) Sage Publications, London

Mays, N and Pope, C (2006) Qualitative research in health

care (3rd ed) Blackwell, Oxford

Trochim, W (2006) Research methods knowledge base

[Online] Web Centre for Social Research Methods

http://www.socialresearchmethods.net/kb/index.php#about

[Accessed 24 Nov 2010]