AARON W. BERNARD, MD JONATHAN FISHER, MD Qualitative Methods in Educational Research.

AARON W. BERNARD, MDJONATHAN FISHER, MD

Qualitative Methods in Educational Research

Objectives

Define qualitative researchContrast with quantitative researchDescribe how to conduct qualitative research

by using examples in the literature

Quantitative Research

Test well-specified hypothesis concerning some predetermined variables

Answer questions like; Whether…….. How much….

Quantitative Educational Research

Attempts to study links between teaching factors and learning outcomes

Improved Medical Student Satisfaction and Test Performance with a Simulation-based Emergency Medicine Curriculum: A Randomized Controlled Trial.

Ten Eyck. Ann Emerg Med. 2009;54:684-691

Qualitative Research

Qualitative research questions tend not to ask whether or how much but rather; What Why How

Qualitative research offers insight into social, emotional, and experiential phenomena.

Qualitative reports do not typically generate answers but rather generate narrative accounts, explanations, conceptual frameworks.

Qualitative Research

Qualitative research is inquiry-guided, inductive, and descriptive.

Similar to an anthropologic or sociologic descriptive study.

Data Collection Methods

Field observations

Interviews Acad Med. 2009.

Document analysis Teach Learn Med. 2004.

Data Analysis

Grounded theory

Coding

Iteration

Theoretical saturation

Qualitative Educational Research

Acad Emerg Med 2005;12:856-861.

What Do Emergency Medicine Learners Want from Their Teachers? A Multicenter Focus Group Analysis. Acad Emerg

Med 2005.

Background Effective teaching practices in the ambulatory setting

have been documented The ED is a different type of learning environment A qualitative study to determine what EM teachers

consider important has been done1

What do EM learners consider important to effective EM teaching? Quantitative methods vs.. qualitative methods

Bandiera . Ann Emerg Med 2005; 45:253-261

What Do Emergency Medicine Learners Want from Their Teachers? A Multicenter Focus Group Analysis. Acad Emerg

Med 2005.

Methods

Focus groups at Five EM programs in Ontario Multiple sites adds validity

Medical student, EM Resident, Off-service resident, Junior FRCP, Senior FRCP Purposeful sampling 28 participants total

What Do Emergency Medicine Learners Want from Their Teachers? A Multicenter Focus Group Analysis. Acad Emerg

Med 2005.

Methods

Semi-structure interviews by a trained moderator

Tape-recorded and transcribed by a third party

Participants used terms like “junior” to identify themselves to allow the data to be matched to type of participant

What Do Emergency Medicine Learners Want from Their Teachers? A Multicenter Focus Group Analysis. Acad Emerg

Med 2005.

Methods/Data Analysis Transcripts were reviewed by two independent

assessors and a coding framework was derived using grounded theory methods

Novel ideas were assigned a new teaching code or a new qualifier as they emerged

When a new code was added the entire transcript was reviewed

The two codes that emerged were compared and consensus was reached

Agreement on 41 coded items and disagreement on 9 5/9 were subthemes

What Do Emergency Medicine Learners Want from Their Teachers? A Multicenter Focus Group Analysis. Acad Emerg

Med 2005.

What Do Emergency Medicine Learners Want from Their Teachers? A Multicenter Focus Group Analysis. Acad Emerg

Med 2005.

Data analysis The transcripts were then reviewed against the final

code by one investigator to check for frequency of codes

A second investigator reviewed 20 percent of the transcript to allow for an estimation of interrater reliability Computer software exists to help with this!

What Do Emergency Medicine Learners Want from Their Teachers? A Multicenter Focus Group Analysis. Acad Emerg

Med 2005.

Summary Qualitative methods were most appropriate for

this investigation Focus group interviews are a common method to

obtain data Purposeful sampling (low numbers, key players) Grounded theory to create code Frequency of themes to assess importance of

themes to subject

Qualitative Educational Research

Academic Medicine 2010; 85:124-133

Medical student professionalism narratives: a window on the informal and hidden curriculum. Acad Med 2010

Background The informal and hidden curriculum drives

professionalism development There is a lack of evidence regarding what

students experience.

Objective To understand students experiences as they relate

to an understanding of professionalism

Medical student professionalism narratives: a window on the informal and hidden curriculum. Acad Med 2010

Methods Students on IM required to post 2 narratives over 2

months Retrospective review of narratives June – November

2007 272 narratives from 135 students

Medical student professionalism narratives: a window on the informal and hidden curriculum. Acad Med 2010

Methods/Data Analysis Thematic content analysis Immersion/crystallization method Reading and rereading narratives until themes

emerge Code was created with the first 50 narratives A narrative could be coded more then once

Medical student professionalism narratives: a window on the informal and hidden curriculum. Acad Med 2010

Reliability1. One author created the code based on the first 50

narratives then two others independently reviewed the work.

2. Created focus groups of students to review themes

Medical student professionalism narratives: a window on the informal and hidden curriculum. Acad Med 2010

Two major domains14 themes63.4% positive, 29.1%

negative, 7.5 % hybrid

Medical student professionalism narratives: a window on the informal and hidden curriculum. Acad Med 2010

Direct representative quotes

Smith AK, Fisher J. Ann Emerg Med. 2009.

Direct representative quotes

Direct representative quotes

Medical student professionalism narratives: a window on the informal and hidden curriculum. Acad Med 2010

SummaryQualitative methods were most

appropriate for this investigationDocument analysis is a common

method to obtain dataMethodological steps to improve

validity is importantThe use of direct representative

quotes

Qualitative Educational Research

Teach Learn Med. 2010;22(1):28-32

Medical student participation in a disaster seminar and drill: brief description of activity and report of student experiences. Teach Learn

Med 2010

Background Disaster training is relevant to medical students who

as physicians may be asked for guidance during training or real events.

Medical school offer little formal curriculum

Objective Provide 4th year medical students with a didactic and

clinically experience in disaster medicine Evaluate the effectiveness through lecture evaluations

and a qualitative analysis of focus groups

Medical student participation in a disaster seminar and drill: brief description of activity and report of student experiences. Teach Learn

Med 2010

Methods 43 students 4 hour didactic session the evening before the drill Training in the use of a standardized assessment tool

for disaster medicine drills

Evaluation and Focus Group 5 point likert scale surveys regarding the didactics 2 Focus groups after the drill

Semi-structured interviews

Medical student participation in a disaster seminar and drill: brief description of activity and report of student experiences. Teach Learn

Med 2010

Methods/Data Analysis Median, means, SD for the survey data Notes were taken in real time One researcher performed the qualitative analysis and

created the thematic domains of the code

Medical student participation in a disaster seminar and drill: brief description of activity and report of student experiences. Teach Learn

Med 2010

28 of the 42 students participated in both the didactic and the drill

17 agreed to participate in the focus group

Themes Changes in self-perceived

attitude toward DM Changes in student’s

ability to apply this knowledge in a simulated setting.

Medical student participation in a disaster seminar and drill: brief description of activity and report of student experiences. Teach Learn

Med 2010

Effective use of qualitative research to create a descriptive study of an educational curriculum that is publishable.

Combination of qualitative and quantitative methods

Final Conclusions

Qualitative research is a tool that can be used for educational investigations.

Qualitative research is inquiry-guided, inductive, and descriptive.

Data usually comes from field observations, interviews, or document analysis.

Grounded theory is used to review the data to create themes and subthemes. This is known as coding.

Frequency of themes is often used as a marker of importance of themes to subjects studied

Various techniques can be used to increase reliability of the analysis

References

Tehaerani A, O’Brien B, Masters DE, et al. Burden, responsibility, and reward: preceptor experiences with the continuity of teaching in a longitudinal integrated clerkship. Acad Med. 2009;84(10 Supp):s50-s53.

Alford CL, Currie DM. Introducing first-year medical students to clinical practice by having them “shadow” third-year clerks. Teach Learn Med. 2004;16(3):260-263.

Thurgur L, Bandiera G, Lee S, et al. What do emergency medicine learners want from their teachers? A multicenter focus group analysis. Acad Emerg Med 2005; 12:856-861.

Karnieli-Miller O, Vu R, Holtman M, et al. Medical students’ professionalism narratives: a window on the informal and hidden curriculum. Acad Med. 2010;85:124-133.

Giacomini MK, Cook DJ, et al. Users’ guides to the medical literature XXIII. Qualitative research in health care A. Are the results of the study valid? JAMA. 2000;284:357-362.

Smith AK, Fisher J, Schonberg MA, et al. Am I doing the right thing? Provider perspectives on improving palliative care in the Emergency Department. An Emerg Med. 2009;54:86-93.

Kaji AH, Coates WC, Fung C-C. Medical student participation in a disaster seminar and drill: brief description of activity and report of student experiences. Teach Learn Med. 2010;22(1):28-32