Page 1
7/29/2019 QualMethods minicourse
http://slidepdf.com/reader/full/qualmethods-minicourse 1/50
Using Qualitative MethodsUsing Qualitative Methodsin Empirical Studies of in Empirical Studies of Software EngineeringSoftware Engineering
University of Maryland Baltimore County University of Maryland Baltimore County Fraunhofer USA Center for Empirical Software Fraunhofer USA Center for Empirical Software
Engineering Maryland Engineering Maryland
ESELAW 2009 ESELAW 2009
12 November 2009 12 November 2009
São Carlos São Carlos- -SP, Brazil SP, Brazil
Page 2
7/29/2019 QualMethods minicourse
http://slidepdf.com/reader/full/qualmethods-minicourse 2/50
DefinitionsDefinitions
Qualitative data - data in the form oftext and pictures, not numbersQualitative analysis – analysis of
© Carolyn Seaman, 2009
trends, patterns, and generalizationsGrounded theory – theory formedbottom-up from the (usuallyqualitative) dataRich data – data that includes a lot of
explanatory and context information
Page 3
7/29/2019 QualMethods minicourse
http://slidepdf.com/reader/full/qualmethods-minicourse 3/50
Why Qualitative Methods?Why Qualitative Methods?
Problem : Difficult to answer complexSE questions with a purelyquantitative approach because
© Carolyn Seaman, 2009
Typically have small sample sizesExperiments are expensive to runNeed some support for a hypothesis
before investing effort in full experimentSolution : Use a qualitative approachthat includes a quantitative aspect
Page 4
7/29/2019 QualMethods minicourse
http://slidepdf.com/reader/full/qualmethods-minicourse 4/50
Types of resultsTypes of results
A qualitative study will result in:Propositions tied to a trail of“evidence”
© Carolyn Seaman, 2009
-Complex findings that incorporatethe messiness of the phenomenon
under studyExplanationsAreas for future study
Page 5
7/29/2019 QualMethods minicourse
http://slidepdf.com/reader/full/qualmethods-minicourse 5/50
Types of ResearchTypes of ResearchQuestionsQuestions
Qualitative methods are most appropriatewhen:Subject of study involves human
© Carolyn Seaman, 2009
e av orNo concrete hypothesesVariables hard to define or quantify
Little previous workQuantitative results may be hard tointerpret
Page 6
7/29/2019 QualMethods minicourse
http://slidepdf.com/reader/full/qualmethods-minicourse 6/50
Advantages to ResearchersAdvantages to Researchers
Richer resultsResults more explanatory
© Carolyn Seaman, 2009
Avoid errors in interpretation
Page 7
7/29/2019 QualMethods minicourse
http://slidepdf.com/reader/full/qualmethods-minicourse 7/50
Advantages to PractitionersAdvantages to Practitioners
Richer , more relevant resultsTerminology of results
© Carolyn Seaman, 2009
Opportunity to clarify and explainfindings
Page 8
7/29/2019 QualMethods minicourse
http://slidepdf.com/reader/full/qualmethods-minicourse 8/50
Overview of TechniquesOverview of Techniques
Data Collection
Prior Ethnography
Data Analysis
Codin
© Carolyn Seaman, 2009
ParticipantObservationInterviewing
SurveysDocument Analysis
ConstantComparison MethodCross-case analysis
Member checkingAuditing
Page 9
7/29/2019 QualMethods minicourse
http://slidepdf.com/reader/full/qualmethods-minicourse 9/50
Participant ObservationParticipant Observation
Definition: non-covert directobservation of phenomenon
© Carolyn Seaman, 2009
inspection meetingscollected both qualitative and quantitative
datadid not participate in the inspectionused data forms as well as field notes
Page 10
7/29/2019 QualMethods minicourse
http://slidepdf.com/reader/full/qualmethods-minicourse 10/50
Observation Data FormObservation Data FormInspection Data Form
Class(es) inspected Inspection date: Time:
Author:Moderator:Reviewers:
Name Responsibility Preparation time Present
© Carolyn Seaman, 2009
Amount of code inspected:Complexity of classes:Discussion codes:D = Defects Q = Questions C = Classgen defect U = Unresolved issues G/D = Globaldefects G/Q = Global questions P = Process issues A = Administrative issues
M = Miscellaneous discussion
Time logged (in minutes):D______ Q_____ C_____ U_____ G/D______ G/Q______ P______ A______ M______
Page 11
7/29/2019 QualMethods minicourse
http://slidepdf.com/reader/full/qualmethods-minicourse 11/50
Field Notes ExampleField Notes Example
The "step" function is a very important but complicated function.[Reviewer1] did not have time to review it in detail, but[Author] said he really wanted someone to go over it carefully,so [Reviewer1] said she would later.
-
© Carolyn Seaman, 2009
values. This is a problem because often the code is such thatthere is no way to tell what a particular variable was initializedto. [Reviewer2] said "I have no way to see initial value". Thiswas a global discussion, relevant to many classes, including
[Reviewer2]’s evidently.
Page 12
7/29/2019 QualMethods minicourse
http://slidepdf.com/reader/full/qualmethods-minicourse 12/50
InterviewingInterviewing
Interviews are good for gettingopinionsfeelin s
© Carolyn Seaman, 2009 12
goalsprocedures (both formal and informal)
not facts
Page 13
7/29/2019 QualMethods minicourse
http://slidepdf.com/reader/full/qualmethods-minicourse 13/50
Standard Interview FormatsStandard Interview Formats
Structured (standardized)Tightly scripted, almost verbalquestionnaire
© Carolyn Seaman, 2009 13
,Analyze like questionnaire“How many times a day do you access the internet? [0, 1-5, 5-10, 10-15, 15+]”
Page 14
7/29/2019 QualMethods minicourse
http://slidepdf.com/reader/full/qualmethods-minicourse 14/50
Standard Interview FormatsStandard Interview Formats
Unstructured(Open/Informal/Conversational)Guided by a very scant script.Rich but not re licable.
© Carolyn Seaman, 2009 14
Difficult to be systematic, problem ofcoverage.Minimize interviewer effects, preservesinterviewee point of view.Interviewee led, interviewer probes.“Please, tell me about your internet usage...”
Page 15
7/29/2019 QualMethods minicourse
http://slidepdf.com/reader/full/qualmethods-minicourse 15/50
Standard Interview FormatsStandard Interview Formats
Semi-structuredGuided by a script ( interview guide ), butinteresting issues can be explored in
© Carolyn Seaman, 2009 15
.Good balance between richness andreplicability.Mixed analysis techniques.
“In a typical day, how often do you use the internet?”
Page 16
7/29/2019 QualMethods minicourse
http://slidepdf.com/reader/full/qualmethods-minicourse 16/50
Interview questionsInterview questionsClosed
Predetermined answer format (e.g. Yes/No)Easier to analyze
OpenNo predetermined answer format
© Carolyn Seaman, 2009 16
More complete responseCombinationClosed, with opportunity to elaborate
Probes
Pitfalls :leading questionsdouble-barreled questions
judgmental questions
Page 17
7/29/2019 QualMethods minicourse
http://slidepdf.com/reader/full/qualmethods-minicourse 17/50
Interview GuideInterview Guide
A script for use by interviewer only“Wish list” vs. structuredFlow/direction to interview
© Carolyn Seaman, 2009 17
equ re top csTransitions between topic areasImportant for replicabilityWording and sequence are critical
Page 18
7/29/2019 QualMethods minicourse
http://slidepdf.com/reader/full/qualmethods-minicourse 18/50
Interview DesignInterview DesignConsiderationsConsiderations
Context switchingFlow between open and closed
© Carolyn Seaman, 2009 18
“Shape ” of interviewMost important stuff first
Wording
Page 19
7/29/2019 QualMethods minicourse
http://slidepdf.com/reader/full/qualmethods-minicourse 19/50
Interview ShapesInterview Shapes
FunnelBegin with open, gradually become more closedGood if you’re not sure what you’re going to get
Pyramid
© Carolyn Seaman, 2009 19
,Good with nervous interviewees
Hour glassBegin with open, gradually become more closed,then open up again at end to pick up things youmight have missedGood if you know what you want, but suspectthere are important things you don’t know aboutyet
Page 20
7/29/2019 QualMethods minicourse
http://slidepdf.com/reader/full/qualmethods-minicourse 20/50
Interview Guide 2a : In-depth project interviewsWho : Developers on [Project1], [Project2], [Project3]Subjects covered : general opinions of GSS processes and productsDuration : 60-90 minutes
What do you like about the current process using GSS?
Example Interview GuideExample Interview Guide
© Carolyn Seaman, 2009
at o you s e a out t e current process us ng
Do you depend on any other groups, either for information or help withGSS, or for work to be done related to GSS?
What do you like about the applications resulting from using GSS?
What do you dislike about the applications resulting from using GSS?
Have there been any problems with the interface between GSS and otherCOTS products?
What do you see as the top risks associated with the use of GSS? Howwould you mitigate these risks?
Page 21
7/29/2019 QualMethods minicourse
http://slidepdf.com/reader/full/qualmethods-minicourse 21/50
Interviewing PointersInterviewing Pointersgive clues about the level of detail you want
establish rapport , but be subject neutralavoid jargon , esp. academesedispel any notion of the “ right ” answer
la the novice when a ro riate
© Carolyn Seaman, 2009 21
probe , but do not leadalways be aware of your biasesbe sensitive to their work (environment/schedule)no more than 60 minuteslet interviewee know next stepsend with “ anything else I should know? ”say Thank you !
Page 22
7/29/2019 QualMethods minicourse
http://slidepdf.com/reader/full/qualmethods-minicourse 22/50
Recording of interviewsRecording of interviews
Audiorecording
© Carolyn Seaman, 2009 22
Scribing
Page 23
7/29/2019 QualMethods minicourse
http://slidepdf.com/reader/full/qualmethods-minicourse 23/50
AudiorecordingAudiorecording
Best memory mechanismFull transcription or just verbatim quotesStill take notes
Tapes fail, digital files are deleted
© Carolyn Seaman, 2009 23
Does not record all aspects (esp. context / facial expressions)Required consent
Always ask first.
Do NOT hide recorder, keep it visible at alltimes.Give the option to turn it off at any point.
Page 24
7/29/2019 QualMethods minicourse
http://slidepdf.com/reader/full/qualmethods-minicourse 24/50
NotetakingNotetaking
Very hard to take notes andinterview at the same timeThere are some super-
© Carolyn Seaman, 2009 24
researchers who can do itInevitably results in incompletenotesSlows down the interviewSometimes inevitable
Page 25
7/29/2019 QualMethods minicourse
http://slidepdf.com/reader/full/qualmethods-minicourse 25/50
ScribingScribing
Partner-based interviewingAdvantages of a single contact vs. trading-offCan share roles (interviewer/scribe)
© Carolyn Seaman, 2009 25
ta e notes, t oug to erentdegree
Group debrief: what did you get/miss?Synchronize notes: overlap and emphasis
Clarify while it is still in your head
Page 26
7/29/2019 QualMethods minicourse
http://slidepdf.com/reader/full/qualmethods-minicourse 26/50
Writing up the interviewWriting up the interview
© Carolyn Seaman, 2009 26
Page 27
7/29/2019 QualMethods minicourse
http://slidepdf.com/reader/full/qualmethods-minicourse 27/50
Interview NotesInterview Notes
Write it up immediatelyDescriptive vs. reflective notesUse Observer’s Comments
Impressions, state of mind, assumptions, notes to
© Carolyn Seaman, 2009 27
How detailed?Verbatim transcript
only possible with audiorecordingExtremely labor-intensive
Summaries with major points quotedOK, but use LOTS of quotesStart closer to verbatim at the beginning of a study
Page 28
7/29/2019 QualMethods minicourse
http://slidepdf.com/reader/full/qualmethods-minicourse 28/50
Interviewing ExerciseInterviewing Exercise
Background :The National Federation of Makers of Feijoada(FNFF ) is concerned that the nationalconsumption of feijoada is declining due to
© Carolyn Seaman, 2009
ecreas ng qua ty o e oa a .So they have asked us to interview the topfeijoada chefs in the country (as determined byregional competitions)The goal is to find out the secrets to masterfeijoada making, so that it can start to be taught inelementary schools .
28
Page 29
7/29/2019 QualMethods minicourse
http://slidepdf.com/reader/full/qualmethods-minicourse 29/50
Interviewing ExerciseInterviewing Exercise
Three versions of the interviewguide
© Carolyn Seaman, 2009
You will be the interviewees
So take a moment to think of your
favorite feijoada chef
29
Page 30
7/29/2019 QualMethods minicourse
http://slidepdf.com/reader/full/qualmethods-minicourse 30/50
Interviewing ExerciseInterviewing Exercise
RecapFirst interview: pyramid
Started with easy, closed questionsEnded with open-ended questions
1. What do you think makes your feijoada the best?2. What is special about your ingredients?3. What are the basic steps to making feijoada?4. Who taught you to make feijoada?5. How long does it take you to make a feijoada?
1. How often do you make feijoada and how long does ittake you?
2. What do you think makes your feijoada the best?3. Of course, you always wash your hands thoroughly
before you start, right?
© Carolyn Seaman, 2009
econ nterv ew: funnelStarted very broadly, with open questionsFollowed up with narrower, closed questions
Third interview: just bad
Leading, judgmental questionsDouble-barreled questionsSwitching from topic to topicSwitching between open and closed
30
1. What is your name, please?2. How often do you make feijoada?3. How long does it take to make feijoada?
4. What are the ingredients you use?5. What do you think makes your feijoada the best?
. the end of the cooking?
5. What kind of pot do you use?
Page 31
7/29/2019 QualMethods minicourse
http://slidepdf.com/reader/full/qualmethods-minicourse 31/50
Constant ComparisonConstant ComparisonMethodMethod
Qualitative analysis method
Meant to generate grounded theoryOperates on a set of field notesB sic rocess:
© Carolyn Seaman, 2009
codinggroupingwriting field memo
forming hypothesesRepeated periodically in parallel withdata collection
Page 32
7/29/2019 QualMethods minicourse
http://slidepdf.com/reader/full/qualmethods-minicourse 32/50
What’s a Code?What’s a Code?
A labelA concept
© Carolyn Seaman, 2009 32
A categoryA relationship
A theme
Page 33
7/29/2019 QualMethods minicourse
http://slidepdf.com/reader/full/qualmethods-minicourse 33/50
What’sWhat’s Coding?Coding?
Open coding - assigning codes topieces of textual dataCoded “chunks” can overlap
© Carolyn Seaman, 200933
One chunk can have several codesAxial coding - grouping, categorizing,combining coded chunks
Selective coding - making sense of it
Page 34
7/29/2019 QualMethods minicourse
http://slidepdf.com/reader/full/qualmethods-minicourse 34/50
What’s here? What are the pieces?
Identification/discovery of conceptsClassification labelin of henomena
Open CodingOpen Coding
© Carolyn Seaman, 200934
Abstraction (this is part of that)Comparative analysis (this is different fromthat)
Categorization (organization, grouping)Value-neutral, at least initially“complexity” not “high complexity” or “lowcomplexity”
Page 35
7/29/2019 QualMethods minicourse
http://slidepdf.com/reader/full/qualmethods-minicourse 35/50
Preparing for codingRead the dataRead background material and research designCreate pre-formed codes, if applicable
Coding by hand
Open CodingOpen Coding ProcessProcess
© Carolyn Seaman, 200935
, .
Photocopy, scissors, and envelopesMS Word commentsExcel
Coding tools – NVivo, Atlas TI
Coding schemePre formed or post formed codesConstant iterationStructure develops over time
Page 36
7/29/2019 QualMethods minicourse
http://slidepdf.com/reader/full/qualmethods-minicourse 36/50
• Background :
• Study of the role of documentation insoftware maintenance
• Interviews with experienced software
OpenOpen Coding ExerciseCoding Exercise
© Carolyn Seaman, 200936
ma nta ners n severa organ zat ons• Process :
• I’ll show you an example• Then you’ll try it – code one excerpt with
one code• Find a partner – compare your codings• I’ll show you my coding of the excerpt
Page 37
7/29/2019 QualMethods minicourse
http://slidepdf.com/reader/full/qualmethods-minicourse 37/50
Respondent BackgroundInformation Gathering
Transition to maintenanceTypes of documentationCharacteristics of DocumentationQuality of documentation
Coding SchemeCoding Scheme
© Carolyn Seaman, 200937
Properties of documentationMissing documentationCreating documentationLocation of documentationImportance of documentationHuman sources of informationQuality of ProcessGreat Quotes
Human Sources of Information
Page 38
7/29/2019 QualMethods minicourse
http://slidepdf.com/reader/full/qualmethods-minicourse 38/50
Open Coding andOpen Coding andQuantificationQuantification
One form of coding
Objective is to derive quantitative data from qualitative datafor future statistical analysisUsually involves counting
How many subjects said…?
© Carolyn Seaman, 200938
…How many times did …?
Or timingHow long did subjects spend doing…?How long did it take to …?
Inevitably loses richnessOften seems a little like missing the point
What’s the point of collecting rich data when you’re just goingto condense it down to numbers?
But often is an effective and necessary way to reduce thesize of the data
Page 39
7/29/2019 QualMethods minicourse
http://slidepdf.com/reader/full/qualmethods-minicourse 39/50
© Carolyn Seaman, 200939
Page 40
7/29/2019 QualMethods minicourse
http://slidepdf.com/reader/full/qualmethods-minicourse 40/50
© Carolyn Seaman, 200940
Page 41
7/29/2019 QualMethods minicourse
http://slidepdf.com/reader/full/qualmethods-minicourse 41/50
How are things related?
Initial process of reassemblingRelationships among categories and codes
Axial CodingAxial Coding
© Carolyn Seaman, 2009 41
Structure (why?)Process (how?)Explanations not causal prediction
Page 42
7/29/2019 QualMethods minicourse
http://slidepdf.com/reader/full/qualmethods-minicourse 42/50
How does it all fit together?
Also called sense makingRelationships among relationships
Selective CodingSelective Coding
© Carolyn Seaman, 2009 42
Theory constructionThe central categoryStoryline memosRole of literatureWrite, write, write!!!Field Memos
Page 43
7/29/2019 QualMethods minicourse
http://slidepdf.com/reader/full/qualmethods-minicourse 43/50
Field MemosField Memos
The “ single most powerful analytical tool ” forqualitative researchersSimply, a piece of writingMaybe will later become part of a report,
© Carolyn Seaman, 2009 43
Summarizes and synthesizes:
A propositionAn open questionA chain of evidence and logic
The complexity of a conceptRich descriptionVersion control and organization
Page 44
7/29/2019 QualMethods minicourse
http://slidepdf.com/reader/full/qualmethods-minicourse 44/50
Judging ValidityJudging Validity
Validity of methodsTriangulation
© Carolyn Seaman, 2009
Contradictory evidenceWeight of evidence
How much is enough?Variety as well as quantity ofevidence
Page 45
7/29/2019 QualMethods minicourse
http://slidepdf.com/reader/full/qualmethods-minicourse 45/50
Using Qualitative andUsing Qualitative andQuantitative MethodsQuantitative Methods
Together Together
Qualitative and quantitative methodsbest used in combination
© Carolyn Seaman, 2009
an s mp y e use n para e oaddress the same researchquestions
There are other strategies to betterexploit the strengths andweaknesses of the methods
Page 46
7/29/2019 QualMethods minicourse
http://slidepdf.com/reader/full/qualmethods-minicourse 46/50
Example Design 1Example Design 1 :: StatisticalStatisticalHypothesis Testing withHypothesis Testing withFollowFollow- -up Interviewsup Interviews
Classic design – often done withoutfully exploiting the interview data
© Carolyn Seaman, 2009
Example scenario:Blocked subject-project experiment toevaluate a new testing techniqueStatistical results show that technique ismore effective on some applications thanon othersQualitative results show why
Page 47
7/29/2019 QualMethods minicourse
http://slidepdf.com/reader/full/qualmethods-minicourse 47/50
Want to evaluate a new technique, butnot sure what the evaluation criteriashould be
Example Design 2 : Using GroundedTheory to Identify Variables
© Carolyn Seaman, 2009
Evaluating a collaborative design processUse participant observation of designmeetings to generate hypotheses aboutproperties of the resulting designsGrounded hypotheses are used to design aquantitative evaluation of the resultingdesigns
Page 48
7/29/2019 QualMethods minicourse
http://slidepdf.com/reader/full/qualmethods-minicourse 48/50
Example Design 3Example Design 3 :: Using Prior Using Prior Investigation toInvestigation to
OperationalizeOperationalize VariablesVariables
Relevant variables are known, but therange and types of values is difficult tospecify
© Carolyn Seaman, 2009
Example scenario:Want to study the relationship betweendeveloper experience and types of defectsFirst use interviews to identify the range of
developer experience (in its complexity) anda taxonomy of defect typesQuantitative study then is much moreeffective when using this operationalization
Page 49
7/29/2019 QualMethods minicourse
http://slidepdf.com/reader/full/qualmethods-minicourse 49/50
ConclusionsConclusionsEmpirical software engineering researchers areaddressing more and more complex research
questions that have increasingly humanelementsQualitative methods, usually in combination with
© Carolyn Seaman, 2009
,
this complexityQualitative methods are both flexible andrigorousQualitative analysis provides richer, morerelevant, and more explanatory resultsThe most effective research designs combinequalitative and quantitative methods
Page 50
7/29/2019 QualMethods minicourse
http://slidepdf.com/reader/full/qualmethods-minicourse 50/50
BibliographyBibliographyDay, I. Qualitative data analysis: A user-friendly guide for social scientists . New York: Routledge. 1993.Eisenhardt, Kathleen M. "Building theories from case studyresearch." Academy of Management Review, 14:4, 1989.Gilgun, J.F., "Definitions, Methodologies, and Methods inQualitative Family Research." Qualitative Methods in Family
Research . Sage, 1992. pp. 22-29Glaser, Barney G. and Anselm L.Strauss. The Discovery of Grounded Theory: Strategies for Qualitative Research. de
© Carolyn Seaman, 2009
, .
Jick, Todd D. "Mixing qualitative and quantitative methods:Triangulation in action." Administrative Science Quarterly,24:4, 1979.Judd, Charles M., Eliot R. Smith, and Louise H. Kidder.Research Methods in Social Relations. Harcourt, Brace,Jovanovich, 1991.Miles, Matthew B. and A. Michael Huberman. Qualitative DataAnalysis: An Expanded Sourcebook. Sage Publications, 1994.