Qualitative and Mixed Methods Annual Meeting Handouts/HANDOU… · Qualitative and Mixed Methods ... What is Qualitative Research? ... • Creswell JW, et al, for the Office of Behavioral

Qualitative and Mixed Methods

For Emergency Medical Services Research

January 8, 2013

Anne Lang Dunlop, MD, MPH

Emory University

Learning Objectives

After this presentation, learners will have knowledge of:

• Qualitative research methods and how they contribute to disaster preparedness and response research;

• Approaches for conducting rigorous mixed methods research;

• Strategies for combining methods to increase research depth and rigor.

Mixed Methods Research

What is Qualitative Research?

• Qualitative research is an approach which seeks to understand human experiences, perceptions, motivations, intentions and behavior.

• By employing interactive, inductive, flexible, and reflexive methods of data collection and analysis.

Qualitative Research: Goal

• To understand those being studied from their perspective;

• To develop concepts and explore themes in order to understand phenomena in natural settings in an observational manner, emphasizing the experiences and views of participants.

Qualitative Research: Goal

• To understand those being studied from their perspective;

• To develop concepts and explore themes in order to understand phenomena in natural settings in an observational manner, emphasizing the experiences and views of participants.

In contrast to study in controlled settingsin an experimental manner

Qualitative Research: Research Questions

Qualitative studies are concerned with research questions such as:

“What is X and how does X vary in different circumstances, and why?”

In contrast with research questions such as:

“How many X’s are there?”

“Not everything that can be counted counts and not everything that counts can be counted.”

- Albert Einstein

1) Exploration is the essential feature of qualitative approach in order to understand the perceptions and actions of participants.

Qualitative Research: Main features

2) Inductiveness: Focused on developing concepts and generating hypotheses. The researcher is open to ideas that emerge from listening and observing people.

The inductive approach is particularly useful when little is known about the topics one wants to study.


3) Interactive & Reflexive: Researchers use interaction between the research team and participants to get closer to the topic under study.

The researcher becomes an instrument of data collection. The researchers probe, facilitate, and note tone, hesitations and repetition in participants responses.


4) Holistic: Researchers are allowed to put their responses in context.


5) Flexible methods: Qualitative research relies on personal and individual world of participants. So flexible, imaginative, creative and varied strategies are used to facilitate this process.


Four Fundamental Tasks of Research

1. Selecting subjects to study

2. Interacting with subjects to gather data

3. Avoiding arbitrary findings

4. Convincing others of what you found

Quantitative & qualitative research

approach these tasks differently:

Quantitative Four R’s

Qualitative Four P’s

Research Task4 R’s

(Quantitative)

4 P’s

(Qualitative)

How do I select research subjects?

Representativeness:Random samples of pre-

determined groups

Purposefulness:Sites/subjects sampled

according to needs

How do I work with subjects to get data?

(non-)Reactivity: Fixed data collection instrument

Participation: Flexible data collection strategies

How do I avoid arbitrary findings?

Reliability: Hypothesis testing via statistical

inference

Process: Iterative coding, synthesizing

How do I convince others of my findings?

Replicability: Tables, figures of specific

categories

Particularity: Narrative reports of findings in context

Qualitative Methods: Sample Selection

Qualitative inquiry focuses on purposefully selecting samples (purposive sampling):

Pre‐specified sampling logic to select information‐rich cases for study in depth

Not intended to achieve statistically representative sample!

Specific sampling strategy dependent upon purpose of the evaluation: See Table.

Type Purpose

Extreme or deviant case sampling Learn from highly unusual manifestations of the phenomenon (outstanding successes, notable failures, crises).

Intensity sampling Learn from information-rich cases that manifest the phenomenon intensely but not extremely.

Maximum variation sampling Documents diverse variations that have emerged in adapting to different conditions.

Homogeneous sampling Reduce variation in different possible subtypes to explore in depth for one subtype.

Typical case sampling Illustrates or highlights what is typical.

Stratified purposeful sampling Illustrates characteristics of particular subgroups of interest; facilitates comparisons.

Snowball sampling Identifies cases of interest from people who know people who know what cases are information-rich.

Criterion sampling Picking all cass that meet some criterion.

Theory-based sampling Finding manifestations of a theoretical construct so as to examine the construct.

Opportunistic sampling Following new leads during fieldwork to take advantage of the unexpected.

Table. Types of Purposeful Sampling for Qualitative Research

There are no hard “rules” for sample size in qualitative inquiry:

– The validity, meaningfulness, and insights generated have more to do with the information‐richness of selected cases and the analytic capabilities of the researcher than the size of the sample.

• Sample needs to be large enough to achieve saturation (the point at which no new ideas are being generated).

Qualitative Methods: Sample Size

Strengths & Weaknesses of Data Collection MethodsMethod of Data

Collection

Strengths Weaknesses

Participant observation Allows observer to “see” what is

happening; Occurs in natural environment

Presence may bias behavior/practices

May present ethical challenges

Key informant interview In‐depth information from one person

Individuals may be more likely to share

sensitive info

Can probe emergent areas/themes

Single person’s perspective

Results highly dependent upon

interviewer

Focus group interview Multiple perspectives on focused topic

Dialogue among participants exposes

themes

Limited information from each person

Dissent/sharing in group may be

limited

Document review Provide insights into perspectives Can be difficult to analyze, interpret

Open‐ended survey Anonymity may make it easier to share

sensitive info

Limited training needed to distribute,

administer

Literacy may limit responses

Requires training to develop valid

survey

Thematic Analysis

Coding of content into relevant themes and patterns AFTER reviewing responses

– Key concept is that themes, and patterns of themes, emerge out of the data (rather than being imposed on them prior to data collection and analysis)

Can be done manually (color coding schemes, index cards) or via computer software (MAXqda, Nvivo,

Atlas‐ti, HyperResearch).

Thematic Analytic – the Process1. Transcribe interview data into text files

2. Actively read and review the textual data:

a) Initial reading – to allow for inductive, emergent themes

b) Re‐read – annotating thoughts in margin to generate open codes

c) Sort items into proto‐themes and themes

3. Examine proto‐theme defined themes Code Book

4. Re‐examine text applying coding schema (multiple coders)

5. Analysis: Reducing and synthesizing data into patterns

Reporting Results of Analysis

Narrative – summarizes key response themes:

– Make mention of tally or hierarchy of themes;

– Include particularly vivid examples verbatim to provide richness and depth.

Matrix – tabular representation of the key response themes by specific variables (e.g. gender or education):

– Neatly “packaged” for review.

Reporting Criteria

Consolidated criteria for reporting qualitative research

(COREQ):

– 32‐item checklist

– 3 Domains addressed by checklist items:

1. Research team and reflexivity

2. Study design

3. Analysis and findings

Validity in Qualitative Research

Addressed via:

1. Triangulation

2. Instrument validity

3. Documentation of processes

Validity: Triangulation

1. Using a rich variety of data– Multiple sources of information

– Multiple data collection techniques

– Multiple analysis techniques

2. Any stated conclusion must rely on multiple types of evidence

3. Contradictory evidence– emerging theory must be modified in the presence of evidence that contradicts the current version of the theory

– contradictory evidence must be sought out

Validity: Instrument Validity

Eliminating bias by:

– Ensuring interview guides and survey questions convey the intended meaning

– Ensuring observers understand and interpret what they see correctly

Terminology is HUGE

Pre test and Pilot test!

Validity: Documentation

Auditing – tracing all processes from data sources to collection methods to analysis (coding, code book development) to conclusions

The chain of evidence and logic that leads to any conclusion must be clear to any reasonable observer

Qualitative Methods: Emergency Svcs Research

Two special features of public health emergency research distinguish it from other types of research:

1. The rapidity and evolution of the occurrence of the event and the response to it;

2. The great importance of the special circumstances or context of the event.

These same two features make qualitative methods particularly appropriate.

Well‐conducted qualitative methods in public health emergency research contribute to a “cross‐hazards” repository of information…

….by providing a rich contextual foundation for a given disaster event AND the associated behaviors of individuals/organizations affected by or participating in the response.

Qualitative Methods: Emergency Svcs Research

Defining Mixed Methods

“The collection, analysis, and integration of

quantitative and qualitative data into a single or

multiphase study. Its central premise is that the

use of quantitative and qualitative approaches

in combination provides a better understanding

of research problems than either approach

alone.” – Hanson, Creswell, Plano Clark, Petska 2005

Rationale for Mixed Methods

Quantitative research can reveal generalizable information fora large group of people….BUT these data often fail to providespecific answers, reasons, explanations or examples

Qualitative research provides meaning and context regardingthe people and environments of study…BUT the findings areoften not generalizable because of the small numbers &narrow range of participants

Mixed Methods: When used together,

these methods can be complimentary

Purpose of Combining Methods

1. Develop and enhance the validity of scales, questionnaire and tools

2. Develop, implement and evaluate interventions

3. Confirm or cross‐validate data

4. Study different aspects of the same topics

5. Explore complex phenomena from different perspectives

6. Further explore or test findings of one method

Mixed Methods – To Explain

Quantitative: Develop

concepts & ideas

Qualitative: Explain

concepts & ideas

Primary focus is to explain a phenomenon

Dig deeper into results to contextualize

Mixed Methods ‐ Strategy A

QUANTITATIVE

• Closed‐ended survey to assess knowledge, attitudes, behavior

QUALITATIVE

• Focus group interviews to add depth to survey findings

Quant/Qual

• Site visit to enrich and confirm findings (participant observation, checklists)

Mixed Methods – To Explore

Qualitative: Develop

concepts & ideas

Quantitative: Explore

concepts & ideas

Primary focus is to explore a phenomenon

Generalize qualitative findings to projectable samples

Mixed Methods ‐ Strategy B

QUALITATIVE

• Key informant interviews to inform survey language and content

QUANTITATIVE

• Closed‐ended survey to rank factors

QUALITATIVE

• Focus group interviews to add depth and context to interpret survey findings

Key Take‐Home Messages

We need a range of methods if we are to understand the complexities of disaster preparedness and response

Qualitative methods are complementary to quantitative methods:

– As an essential prerequisite to quantitative work

– To validate quantitative work

– To explore complex and sensitive issues not approachable by quantitative methods.

“Best Practices” for Mixed Methods Research

From NIH Office of Behavioral & Social Sciences Research

Available at: http://obssr.od.hih.gov/mixed_methods_research

Purpose: To develop practices that ‐

1. Assist investigators using mixed methods as they develop competitive applications for support from NIH;

2. Assist reviewers and staff for review panels at NIH who evaluate applications that include mixed methods research;

3. Provide the NIH Institutes and Centers with "best practices" to use as they consider potential contributions of mixed methods research and set priorities.

References• Creswell JW. Qualitative Inquiry and Research Design. 2nd

Edition. SAGE Publications, Thousand Oaks, CA: 2006.

• Creswell JW, et al, for the Office of Behavioral and Social Sciences Research. Best practices for mixed methods research in the health sciences. August 2011. National Institutes of Health. http://obssr.od.nih.gov/mixed_methods_research

• Hesse‐Biber S. Qualitative Approaches to Mixed Methods Practice. Qualitative Inquiry 2010; 16(6): 455‐468.

• Johnson RB. Mixed Methods Research: A Research Paradigm Whose Time Has Come. Educational Researcher 2009; 33(7): 14‐26.

• Tong A. Consolidated criteria for reporting qualitative research (COREQ). International J Qual Health Care 2007; 19(6): 349‐57.