Overcoming Methodological Challenges - Princeton University Press

Copyrighted Material

C H A P T E R O N E

OvercomingMethodologicalChallenges

Questions about the relativemeritsofalternative research strategiespervadethesocialsciencesWhatcountsasanadequateexplanationforsocialphenomenaHowcanweevaluatecompetingexplanationsWhatstandards should we apply when weighing evidence How much andwhat typesof evidenceare convincingCan socialphenomena relatedtopolicyareasbestudiedscientificallySomeeminentscholarsappeartoagreeonbroadmethodological goalsor criteria (BradyandCollier2004Gerring2001Lieberman2005)Explanationsshouldbegeneralyetpreciseaccurateandwell-specifiedEvidenceshouldbetheoreticallyrelevant and should identify mechanisms linking explanations to out-comesAbundantevidenceiftheoreticallyrelevantisvaluedbecauseitenhancesconfidenceinfindings

Despite theapparentcommongroundunderlyingtheworkofmanyscholarsmethodologicaldivideswithinthesocialsciencesalsorundeepAslamentedbyMahoneyandGoertz(2006)andEOstrom(2006)rivalcampsoftencastaspersionsoneachotherrsquosworkratherthanengageinconstructivedialogueTheacrimonyhasseveralsourcesThedisagree-mentshavebeenprovokedinpartbybattlesoverinductionversusde-ductionpoormethodologicalpracticebysomescholarsandalackofsensitivity to diverse research goals The stakes of the methodologicaldebateareincreasedbytheintertwiningofmethodologicalchoicewithontologicalnormativeandtheoreticalpositionsandwithcompetitionforprofessionalstatusandresources(MosesandKnutsen2007)Thesedynamicsencourageintenseandsometimesgrosslyunfaircritiques

Thesubstantivefocusofthisbookisoncollectiveactionandthecom-monsItisafieldofresearchthatutilizesmultiplemethodsextensivelyaswellasbeingtheonemostfamiliartotheauthorsofthisbookWebelievethatthediscussionoftheuseofmultiplemethodsinthisresearchfieldandthelessonswedrawfromourpracticalexperiencesapplymorebroadlytosocialscienceingeneralThereforewestartthisfirstchapterwithabroaderdiscussiononthemethodologicalchallengesinthesocialsciences

Examplesofpoormethodologicalpracticepervadesocialsciencere-search Often scholars follow ldquothe rule of the hammerrdquo and apply asingle method indiscriminately regardless of its suitability for a given

Copyrighted Material

4 bull Chapter1

research project Harmonization of research goals theory data andmethoddoesnothoweverguaranteesoundpracticeOnecanfindqual-itative studies thatoverstate either theuniquenessor the generalityofparticularcasesfailtoutilizerelevantconceptsandtheories inthelit-eratureorworkwithconceptsthatconflatemultipledimensions(Sartori1991 compareGoldthorpe1997)Quantitative studies sometimesuseinadequate data and do not always use appropriate diagnostic checksandtechnicalfixes(Jackman1985Scruggs2007Shalev2007)Formalmodelsoftenworkwithunrealisticassumptionswithoutaddressingthegapbetweenassumptionsandreality(Bendor1988GreenandShapiro1994)Nomethodisimmunetopoorapplications

Critics sometimes conflatemethodological practicewith themethoditselfarguing thatexamplesofpoorapplicationdiscredit themethodAmethodneednotbeabandonedbecauseithasbeenpoorlyutilizeditmakesmoresense toencouragegreatermethodologicalawarenessandbetterpractices(Geddes2003Jackman1985KingKeohaneandVerba1994 Scruggs2007)Others fail to appreciate that researchgoals arevariedandrequirediversemethodsMorethanthreedecadesagoRobertClark(197710emphasisinoriginal)stronglywarnedagainstrelianceonasinglemethod

Afirstruleshouldbetobewareofoneresearcheronemethodor oneinstrumentThepointisnottoprovethatthehypothesisiscorrectbuttofind outsomethingTorelyonasingleapproachistobeshackled

Indiscriminateapplicationofamethodmakes little sensebut com-plete rejectionof amethodbecause it is inappropriate in aparticularsettingorforaparticularpurposeisnotmoresensibleItisimportantfor social scientists to recognize thatallmethodsgenerate results thatcontain some level of uncertainty While multiple scientific goals andtrade-offsinachievingthosegoalsarewidelyacknowledged(Coppedge1999Gerring2001)littleconsensusexistsontherelativeimportanceofparticulargoalsSomescholarsprioritizeoneorafewgoalstosuchanextentthattheydismissasunscientificresearchthatprioritizesothergoalsForexampleGoldthorpe(1997)includesgeneralityasthemostimportantcriterioninhisdefinitionofcausalexplanationratherthanasoneofseveralcriteria(compareGerring2001)ConsequentlyheseesuniqueeventsandcontingencyasmarkingthelimitsofscientificinquiryBythisdefinitionanalysesofsucheventsarenotscientificandcannotsupportcausalinferencesProponentsofpath-dependentexplanationsanalyticnarrativesinterpretivemethodsandotherapproachesstronglydisagree(Batesetal1998BennettandElman2006Rogowski2004R Smith 2004) As in this example and as discussed further below

Copyrighted Material

OvercomingChallenges bull 5

methodologicalcontroversiesoftenreflectcompetitionbetweenresearchtraditions

Fortunately social scientists increasingly recognize trade-offs acrossmethods (Bates 2007 Brady and Collier 2004 Gerring 2001)1 KingKeohaneandVerba(1994)forexamplepointoutthatallmethodolo-gieshavelimitationsscholarsshouldbemoreawareoftheselimitsandmoretransparentaboutthe limitsaswellas thesolidcontributionsoftheir work To overcome the limits of any one method one needs todrawonmultiplemethods(Batesetal1998Coppedge1999GranatoandScioli2004Jackman1985KingKeohaneandVerba1994Laitin2003Lieberman2005Scharpf2000Tarrow2004)Ifsocialscientistshave shared standardsno singlemethod fullyaddressesall standardsMethods offer different strengths and weaknesses Rigorous researchthatcombinescomplementarymethodswillbesuperiortoresearchthatreliesonanysinglemethod(Grayetal2007)

Thepragmatismandrespectfordiversemethodologicaltraditionsinthese reflections are welcome Too often however the challenges in-volvedinusingmultiplemethodsarethemselvesoverlookedProponentsof mixed methods justify their preferred combination in logical termsandillustratetheapproachwithafewexamplesWithsomeexceptions(Lieberman2005Scharpf2000)thisliteratureoffersfewspecificpracti-calsuggestions

Practical challenges canbe formidableNotallmethodsare equallyfeasibleorevenappropriateforallresearchtopics(BennettandElman2006PoteeteandOstrom2008)Liebermanrsquos (2005)nestedanalysisfor example involves large-N analysis prior to any case study workTherearemany important topics forwhichbroadlycomparativedataare scarce difficult to access or of dubiousqualityLieberman how-everdoesnotaddressthesechallengesEvenifdataavailabilityisnotaproblemthevalueofamultimethodapproachrequiressufficientcom-mandofmultiplemethodsYet considerable investment is required togaincompetencyinanymethodologyandthebenefitsofmethodologi-calspecializationaresubstantialWhilethesechallengesaresometimesacknowledgedfewsocialscientistsmakepracticalsuggestionstoaddressthem

Thisbookfocusesonthepracticalchallengesthatinfluencemethod-ologicalchoiceWeareparticularlyconcernedwithresearchontopicsforwhichdataarescarcedifficulttocollectandnotreadilycomparableTheseconditionsaffect researchonawidevarietyof topics includingthose concerned with informal institutions subnational organizationsand nonelite populations We focus on collective action for the man-agementofnaturalresourcesanareaofresearchinwhichallofthese

Copyrighted Material

6 bull Chapter1

conditionsapplyForsuchtopicsdataforlarge-Nanalysisareneitheravailable nor readily accessible andfield research is unavoidableRe-searchersoftenneedconsiderablecontextualknowledgeeventorecog-nizethephenomenonofinterestTheneedtoconductintensivefieldworklimitsthepotentialforcollectingenoughdatatosupportbroadlycom-parativeanalysis

WehavebecomestronglyawareofthesechallengesthroughourownworkoncollectiveactionandnaturalresourcemanagementWefeelthatthe practical challenges of conducting rigorous social science researchontopicsforwhichdataarescarceordifficulttoaccessortointerprethavenotreceivedadequateattentionindiscussionsaboutsocialscienceresearchWehaveseenthebenefitsofcollaborationandthecombinationofmultiplemethodsinourresearchWealsohavefirsthandexperienceof the challenges involved in such research and we will discuss thesethroughoutthisbook

Inthischapterweintroducefourthemesthatrecurthroughthebook(1)theinterlinkingofmethodologicaldebateswiththeoreticaldevelop-ment (2) theadvantagesand limitationsofmultiplemethodsandcol-laborativeresearch(3)practicalconstraintsonmethodologicalchoicesand(4)theoftenproblematicinfluenceofcareerincentivesonmethod-ologicalpracticeInthisbookweexplicitlyacknowledgethepracticalchallengesthataffectmethodologicalchoicesevaluateseveralstrategiesforaddressingthesechallengesanddirectattentiontotheinfluenceofcareer incentives on methodological choices in social science researchWediscussarangeofoptionsforbalancingcompetingmethodologicaldemandsundertheinevitableconditionsoflimitedresourcesincludingavarietyoftechniquesthatwefeelhavebeenunderutilizedinthesocialsciencesWediscussthemeritsandlimitsofeachmethodaswellasthepossibilities forandconstraintsoncombiningvariousmethods Inourdiscussionofconstraintsonmethodologicalchoicewehopetostimulatea debate about professional incentives and other structural aspects ofacademiathatinfluencehowresearchisconducted

Thisbookismoreaboutmethodologicalpracticethanaboutmethod-ologicalidealsWethusbeginthischapterwithahistoricaloverviewofmethodologicaldebateshighlightinginteractionsamongmethodologicalpracticeschangingtheoreticalorientationsandcompetitionforprofes-sionalstatusandresourcesWethenlookmorecloselyatissuessurround-ingresearchthatusesmultiplemethodsanapproachthathasgainedinacceptanceinrecentyearsThisleadstoadiscussionofconstraintsonmethodologicalchoicebothpracticalandprofessionalWethenexplainhowoursubstantivefocusmdashthestudyofcollectiveactioninnaturalre-sourcemanagementmdashhelpsusaddressourfourthematicconcernsThechapterconcludeswithanoutlineoftherestofthebook

Copyrighted Material

OvercomingChallenges bull 7

Social Science Debates over the Superiority of Particular Methods

Thehistoryofthesocialsciencescanberecountedwithreferencetoma-jormethodologicalshiftsAninitialrelianceonqualitativeanalysisgavewaydramaticallytoquantificationintheearlytomid-twentiethcenturyWhenthistransformationbeganquantificationlargelymeantstatisticalanalysisoflarge-NdatasetsofpublicopinionsurveysThelastthirdofthetwentiethcenturysawasurgeintheuseofformalmodelsaswellDebatesabout the relativemeritsofqualitative statisticaland formalmethodscontributedtoseveraldevelopments inthe late twentiethandearlytwenty-firstcenturiesrefinementsofquantitativemethodsthatat-tempttobettermatchsocialconditionstheriseofformalmodelsgreaterappreciation for combiningmultiplemethods and the spreadofpost-positivistmethodssuchasdiscourseanalysis

Thequalitativeorientationoftheearlysocialsciencescanbeseenintheemphasisoncase studiesandparticipantobservation in sociologyethnographicfield-basedresearchinanthropologyanddescriptiveandnormativeanalysesofformallegalarrangementsIntheearlydecadesofthetwentiethcenturymanyscholarsembracedquantitativemethodsaspartofadrivetomakethesocialsciencesmorescientific2 Quantitativemethodsbegantogaincurrencyacrossthesocialsciencesinthe1920sand1930sTheadoptionofthesemethodsacceleratedatmidcenturyasconveyedbyreferencestothebehavioralrevolution

ThebranchesofthesocialsciencesdifferedintheirtimingpaceandpreferredformsofquantificationNonethelessthemethodologicalshiftfromqualitativetoquantitativemethodsinthesocialscienceswasdra-maticPsychologyrapidlyadoptedexperimentalandstatisticalmethodsQuantitativemethodsineconomicsencompassedformalmodelsaswellas experiments and statistics For sociology research activities duringWorldWarIImarkedtheascendanceofsurveyresearchexperimentsand statistical formsofanalysis (Platt1986)Postwarpolitical scienceshared the enthusiasm for survey research and statistical analysis butformalmodelingbecamewidespreadonly in the1980s and1990s Insocioculturalanthropologysomeinterestwasexpressedinmathematicalmodelsintheearlypostwarperiodbutmultivariatestatisticalanalysesremainedrelativelyrareuntilthe1970s(Chibnik1985)

Theroleofquantitativemethodsinthesocialscienceshasalwaysbeencontentious Current methodological debates echo those of a centuryago even if framed in somewhat different terms3 Scholars concernedwithmethodshavedisagreedover (1) thegoalsof social research (2)philosophical and theoretical issues and (3) practical considerationsespecially related to data quality Methodological choices should be

Copyrighted Material

8 bull Chapter1

drivenbytheoreticalandontologicalassumptions(Hall2003)buttheyalso reflectunderlying values andbeliefs (MahoneyandGoertz2006)andpractical considerations (Platt1986)Theontologicalandnorma-tive dimensions of methodological choices are not widely recognized(MahoneyandGoertz2006)Asa result social sciencedebatesaboutmethodsinvolvefrequentmisunderstandingswithproponentsofdiffer-entapproachestalkingpasteachother(EOstrom2006)Furthermorebecause methodological discussions rarely acknowledge practical andprofessionalconsiderationstheyofferlittleguidanceonhowtoaddresstheseconstraintsInthissectionwediscusscontroversiesoverthegoalsofsocialresearchandhowphilosophicalandtheoreticalissuesinteractwithprofessionalcompetitionWeexpandourtreatmentofpracticalandprofessionalconsiderationsinsubsequentsections

During the1920sand1930s the social sciencesbecamemore insti-tutionalized in North America The social sciences sought recognitionassciencesandeachdisciplinedevelopedamoreorlessdistinctprofes-sionalidentity(Guy2003Platt1986)ThisprocessofinstitutionalizationinfluencedmethodologicaldebatesDuringtheprewarperioddisagree-ments focused on the goals of social research Should sociological re-searchsupportsocialworktoimprovesocialconditionsseeksubjectiveunderstandingoflifeexperiencesorattempttoidentifygeneralpatterns(Platt1986)Shouldthestudyofpoliticsprovidenormativeandpracti-cal guidance for administratorsorobjectiveunderstandingofpoliticalphenomena(Guy2003Lasswell1951)Asuniversitiessetupschoolsofsocialworkpublicadministrationandbusinessadministrationalong-sidedepartmentsof sociologypolitical scienceandeconomicsdiffer-encesovergoalswerealleviatedmdashbutnotreallyaddressedmdashthroughtheinstitutionalizationofmorefocusedprogramsofstudy

Yetdifferencesovertherelativeimportanceoftheoryandpraxiscan-not fully account for methodological debates Scholars with commongoalsdisagreeovermethodsandscholarsdrawonthesamemethodstopursuedivergentgoalsAlackofconsensusonfundamentalphilosophi-cal issues contributes to disagreements over methods What counts asscienceWhatmodelormodelsofcausalityandexplanationmakesenseforsocialphenomenaInparticulardomodelsofscienceandexplana-tiondevelopedinthenaturalandespeciallythephysicalsciencesmakesenseforthesocialsciences

Overthepastcenturysomehaveembraceddeductivemodelsofscienceinspiredbythenaturalsciencesasawaytogainmorereliableinsightsaboutsocialprocesses(KingKeohaneandVerba1994PrzeworskiandTeune 1970) Deduction involves the logical derivation of universalis-ticlawlikestatementsofthesetsofconditionsassociatedwiththeout-comeofinterestfromtheoreticalassumptionsLawlikestatementsmay

Copyrighted Material

OvercomingChallenges bull 9

be derived from formal or mathematical models as in rational-choiceapproachesorlogicalanalysisasinsomequalitativestudiesEmpiricalevaluationsrelyontheanalysisofcorrelationasinbehavioralresearchor paired comparisons4 The journal Public Choice devoted a specialissue in December 2008 to the topic ldquoHomo Economicus and HomoPoliticusrdquo(editedbyGeoffreyBrennanandMichaelGillespie)withninearticlesaddressingthequestionofhowtoreconcilethebasicdifferencesbetweentheoriesofhumanbehaviorineconomicsandpoliticalscienceIntheintroductionBrennan(2008431)reflectsthat

theambitiontofindcommongroundonwhichpublicchoicescholarsandldquopoliticaltheoristsrdquoofamoretraditionalkindmighthaveprofit-ableexchangeisnotatrivialonewestartfromverydifferentconcep-tionsofwhatcountsas theorymdashevenofwhatcountsasworthwhilescholarshipmdashandfromratherdifferentdisciplinarypresuppositionsastohowdifferences inapproachcanmostprofitablybeengagedandresolved

Criticshoweverarguethatdeductivemethodsdonotallowforhu-managencyandreflexivitytheinfluenceofmeaningandinterpretationorcontingentrelationships(AlmondandGenco1977Hall2003Ragin1987seereviewinPlatt1986)Ifagencyistakenseriouslywemustal-lowforbothcreativityanddifferencesinperspectivesButcreativityanddifferencesininterpretationmeanthatlawlikesocialpatternsareunlikelytoariseContingentrelationshipsarepossibleevenifquestionsofagencyareputasideThesedifferencesoverthenatureofcausalityhavefueledheatedmethodologicaldebatesInpoliticalscienceboththebehavioralrevolutionoftheearlypostwarperiodandtheriseofrational-choicethe-oryinthe1980sand1990sassumedthevalueofdeductive-nomologicalreasoningScholarswhousedmethodsthatreflectedalternativeontologi-calassumptionshaddifficultygainingrecognitionfortheirworkTheirfrustrationgaverisetotherecentperestroikamovementinwhichcon-structivistsandotherschallengedboththeuniversalityofsocialpatternsassumedbyrationalchoiceandbehavioraltheoriesandthedominanceofstatisticalandformalmethodsassociatedwiththeseapproachesintheprofession(Monroe2005)Withineconomicstheconcernthatnarrowrational-choicemodelshavecometodominatemuchofeconomicschol-arshipisregularlyexpressedintheonlinejournalReal-World Economics Review5

Deductive-nomological reasoning suggests a mechanical view of theworldinwhichthesamestimulusproducesthesameeffectceterispa-ribusTheoriesthatviewsocialphenomenaasproductsofeitherevolu-tionaryprocessesor intentionalactionchallenge thismechanicalviewBothevolutionaryandintentionaltheoriesassumethatindividualsand

Copyrighted Material

10 bull Chapter1

organizationsadjusttheirresponsestosocialconditions(Alchian1950Brady2004EOstrom2000Thelen2003)Intentionaltheoriesofhu-manbehaviorassumethatadaptationoccursaspeoplestruggletosolvepuzzlesrelatedtothepursuitoftheirgoals(AlmondandGenco1977Elster1983Knight1992)While some intentional theories emphasizeroutines and heuristics there is always a possibility for creativity andinnovation (March andOlsen1984 Simon1955)Evolutionary theo-riesdonotrequire intentionalitybutdorequiresomesortofselectionmechanismsuchasmarketorelectoralcompetitiontodriveadaptationBothformsofadaptationimplythatthesamecircumstanceswillgener-atediverse responsesacrossactorsandchanges in individualbehaviorover timebut thatadaptationswill reflecthistorical trajectoriesThusthesamestimuluswillnot producethesameeffectonaverageandcon-stanteffectscannotbeassumedBothperspectivesraisequestionsaboutthesuitabilityof researchmethods thatassumeconstanteffects (Elster1998Hall2003Ragin19872000)

ThechoiceofmethodtendstosignalonersquostheoreticalperspectiveasdoesthenatureofmethodologicalcritiqueThosewhodiscountquali-tativemethodsasincapableofevaluatinggeneralrelationshipssignalabelief inbothlawlikesocialrelationsandtherelativeunimportanceoffactorssuchasagencyhistoryandinformalcontextNotsurprisinglycritiquesofquantitativemethodsoftenchargethattheydonotcapturethemost important aspects of social conditionsLikewise thosewaryofformalmodelsworryaboutthelevelofabstractionHowcanformalmodelsadequatelyrepresentthedensenetworksofformalandinformalinstitutionsandculturalunderstandingsinwhichhumanactionoccursNoneof these critiques really concerns themethodasmethod ratherthey target the theoretical assumptions as reflected in methodologicalchoicesWhatvariablesareimportantWhatistherelativeimportanceofformalinstitutionsculturesocialstructureorinformalinstitutionsHow important aremass beliefs andbehavior or individual interestsbeliefsandstrategicactionHowarethosevariablesrelatedWhilethebehavioralrevolutionduringthemid-twentiethcenturycertainlyfosteredtherapidspreadofquantitativeanalysisitalsoredirectedtheoreticalem-phasisfromformalinstitutionstothebehaviorandattitudesofindividu-als interactingwithinboth formal and informal institutionsLikewiserational-choiceanalysisoftenreliesongametheoryandothervarietiesof formal modeling but is defined by assumptions of methodologicalindividualismandintentionalaction

Yettheinfluenceoftheorymdashandtheimpliedinfluenceofontologymdashonmethodologicalpracticecannotbeassumedandshouldnotbeover-stated6 Theoreticalchangescananddooccurindependentlyofchangesinmethodologicalpractice(Hall2003Platt1986)Sometimesmethod-

Copyrighted Material

OvercomingChallenges bull 11

ologicalchallengesseemtodrivetheoreticalargumentsratherthantheotherwayaround(Lieberson1991318)Indeedsophisticatedmethodssometimescrowdouttheoryaltogether(Achen20022005)WearguethatmethodologicalchoicesareoftendrivenasmuchbydataavailabilityorcareerincentivesWhencareersurvivalisatstakepracticalconsider-ationscansqueezeoutconcernsaboutmatchingtheoryandmethodThelinkbetweenmethodsandcareerprospectscanhoweverbeexpectedtoinfluencethetenorofmethodologicalmdashandtheoreticalmdashdebates

SometimesmethodologicalandtheoreticaldebatestakeonexistentialovertonesWhenaparticulartheoryandassociatedmethodsbecomeex-tremelywidespreadforexampleproponentsofalternativeapproachesmayworryabouttheirownacademicsurvivalProponentsofnewtheo-riesmdashandassociatedmethodsmdashalsofaceanexistentialfightforrecogni-tionandsurvivalThedegreeof(perceived)existentialthreatdependsonthe extent to which fellowships job opportunities publishing outletsandresearchgrantsareopen(orclosed)todiversetheoriesandmethodsTherecentperestroikamovementinpoliticalscienceforexamplepre-sented itself asdefendingagainstmethodologicalhegemony conjuringimages of political scientists conspiring to control journal outlets andprofessionalassociations7ThiswasnotsimplyamethodologicalcritiquebutacalltoactionagainstpresumedtyrannyWhiletheinflammatorypublic languageassociatedwiththeperestroikamovementmaybeun-usualthelayeringofmethodologicaldebateswithvaluejudgmentsandcompetitionforprofessionalrecognitionandresourcesiscommonplace

Despite references to ldquorevolutionsrdquo and paradigm shifts new socialsciencetheoriesandmethodshavenotfullydisplacedtheirpredecessorsRathereachnewtheoryandmethodhasaddedanotherstrandConstruc-tivists institutionalists andpostmodernists coexistwithbehavioralistsandstructuralistsDespitethehistoryoftheoreticalandmethodologicalcompetitionandcritiquescholarsalsoengageincreativesynthesisThecurrentappreciationformethodologicalpluralismmaybeinterpretedasaproductofthesurvivalandadaptationofapproachesthatwereonceperceived to be under existential threat Promotion of methodologicalpluralismfavorsatheoreticaleclecticismthatshoulddecreaseconcernsaboutexistential threats toparticularapproachesandshouldthusde-creasetheintensityofmethodologicaldebates

Multiple Methods Promises and Challenges

Therearemanyreasonsforsocialscientiststowelcomemethodologicalpluralism and greater use of mixed methods No single method over-comes all challenges Case studies and small-N comparative research

Copyrighted Material

12 bull Chapter1

designs offer advantages for concept and theory development as wellasevaluationofhypothesizedcausalsequencesandmechanisms(Bates2008Batesetal1998CollierBradyandSeawright2004Coppedge1999 Lieberman 2005) Rich explanations of particular cases are of-tenvaluablesubstantivelyandtheoretically(MahoneyandGoertz2006Rogowski2004)Yetasiswidelyrecognizedsmall-Nstudiesofferanuncertainfoundationforpositingorevaluatinggeneralrelationships

Formalmethodsseektobuild logicallycoherentmodelsanddiscerntheir logical implicationsTheir emphasison logical consistency facili-tatesthedistillationofparsimoniousyetgeneralhypothesesandguidesthechoiceofstatisticaltechniques(Achen20022005Batesetal1998GranatoandScioli2004)8 Thehighlevelofabstractioninformalmod-elshoweverraisesquestionsabouttheirempiricalapplicability(GreenandShapiro1994)Thecontrolledconditionsinexperimentalresearchprovidegreaterconfidenceintheinternalvalidityofobservedrelation-shipsTheexternalvalidityofgeneralrelationshipscanbestbeevaluatedhoweverthroughanalysisofalargenumberofnonexperimentalobser-vations(Goldthorpe1997KingKeohaneandVerba1994)aswellasthroughfieldexperiments(seeCardenas2003CardenasStranlundandWillis2000Henrichetal2004List2004)

Small-Nqualitativestudiescansuggesttheplausibilityofformalmod-elsbutprovidelittleleverageinassessingthegeneralityofrelationshipsThebroad comparisons required to evaluate thegeneralityofhypoth-esizedrelationshipsdemandsomeformofquantitativeanalysisWherequantitativeanalysisoncemeantregression-basedanalysisoptions forquantitativeanalysisofempiricalsocialsciencedatanowincludeQuali-tative Comparative Analysis (QCA) and fuzzy-set Qualitative Com-parativeAnalysis(fsQCA)(Ragin19872000)aswellasprobabilisticlikelihood-basedandBayesianstatistics(Gill2004)9 Thismethodologi-calmenuincludesoptionsforscholarswhoholdvariedontologicalas-sumptionsaboutthesocialworld

MixedmethodstakeavarietyofformsAresearchermightusedifferentmethodstoaddressdifferentresearchquestionsorcontextsOrdifferentmethodsmightguidedifferentstagesofaresearchprogram(Lieberman2005)IncreasinglyscholarsstrivetousetwoormoremethodsateachstageofresearchThoseconcernedwithgeneralcausalpatternsdrawonquantitativeandqualitativemethods(Coppedge1999Lieberman2005Tarrow2004)Combinationsofformalandqualitativemethodsaddressconcernsaboutlogicalcoherenceandcausalprocessesincontingentrela-tionshipswherethereisnoexpectationofgenerality(Batesetal1998)Otherscontendthatscholarsshouldseeklogicalcoherenceandevidencefor causalprocesses and should test for thegeneralityof relationsbydrawingonformalqualitativeandquantitativemethods(Granatoand

Copyrighted Material

OvercomingChallenges bull 13

Scioli2004Laitin2003)Scholarswhodevelopagent-basedmodelsuserolegamesandexperiments tocollectdataaswellas involving stake-holdersinthevalidationoftheirmodels(BarreteauLePageandAquino2003Bousquetetal2002GurungBousquetandTreacutebuil2006)Otherscholars combine their formal models with ethnographic observations(Bharwanietal2005HuigenOvermarsanddeGroot2006)

Theuseofmultiplemethodshoweverdoesnotguaranteemethod-ologicallysuperiorsocialscienceresearchSomequestiontheextenttowhichformalqualitativeandquantitativeresearchmethodsareactuallycomplementary Several recent publications have argued that differentmethodsreflectdifferentassumptionsaboutthenatureofcausalityandhavecalledforgreatercareinmatchingmethodstoontologicalassump-tions(BennettandElman2006ClarkGilliganandGolder2006Hall2003Mahoney2003Ragin19872000)

Therearealso limits to the feasibilityofmultimethodresearchHy-potheses about complex causal relationships imply complex statisticalmodelsthatstretchthelimitsofavailabledataStatisticalanalysesoftenadd interaction termsordummyvariables tomodel contingent effectsandmultiplecausalpaths(ClarkGilliganandGolder2006Pontusson2007)but theseadditionalvariablesconsumedegreesof freedominacontextoflimiteddataavailability(Shalev2007)Othertechniquesde-velopedtoaddresscausalcomplexitysuchastheanalysisoftime-series-cross-sectional data and hierarchical models may strain the technicalskillsofboththeresearcherandtheaudience(Shalev2007)

MahoneyandGoertz(2006)contendthatinteractioneffectsdummyvariableshierarchicalmodelsandothersimilarstatisticalfixesdonotaccuratelyreflecttherelationshipspositedintheunderlyingtheoriesTheassumptionthatobservationsareindependentforexampleiscalledintoquestionbyglobalizationdiffusioneffectsandactor-centeredtheoriesthatemphasizestrategicinteractionsEvensomequantitativelyorientedscholarsquestiontheappropriatenessofstandardstatisticaltechniquesInrecentyearsnewtechniqueshavebeenproposedtoincorporateinter-dependence(Signorino1999)Bayesianstatistics(Dion1998Gill2004)andBooleanlogic(Braumoeller2003Ragin19872000)Theverdictisstilloutonwhetherthesenewtechniquesmatchunderlyingassumptionsbetterthandoesregression-basedstatistics

Too often the development of ever-more sophisticated techniquesseemstobeanendinitselfThelatesttechniquesaresometimesadoptedwithlittlereferencetotheoreticalconsiderationsorunderstandingoftheunderlyingassumptionsButmethodologicalsophisticationcannotsub-stitutefortheoryAchen(20022005)warnsthatquantitativeanalysesthat are not supported by theoretical microfoundations or careful ex-plorationofthedatayieldunreliableresultsandshouldnotbetrusted

Copyrighted Material

14 bull Chapter1

Scholarsmustdomoretodevelopexplicittheoreticalargumentsanden-surethattheirmethodsmatchtheirunderlyingassumptionsaboutcau-salityontologyandepistemology(Achen20022005BradyandCollier2004Hall2003)

Neither theorynormethodological techniques substitute for a thor-oughfamiliaritywiththedatagainedfromdiagnostictestsanddataex-plorationVisualizationtechniquessuchasgraphicalanalysisandsimplestatisticaltechniquessuchascross-tabulationsbringempiricalregulari-tiesandpatternedvariationintofocus(Achen20022005Shalev2007)Dataexplorationdrawsattentiontopotentialcausalheterogeneitynon-linearrelationshipsinteractioneffectsandotheraspectsofthedatathatareobscuredbymoresophisticatedmultivariatetechniquesThusthor-oughdataexplorationcontributestotheorytestinganddevelopmentbycomplementingmoresophisticatedformsofdataanalysisanddrawingattentiontoempiricalpatternsthatcalloutfortheoreticalexplanation(Achen20022005)

EvenifcausalepistemologicalandontologicalassumptionsposenobarrierpracticalconsiderationscomplicatemethodologicalchoiceThesepracticalchallenges largelyoverlooked in theexchangesregarding therelativemeritsofalternativeandmultiplemethodsstandatthecenterofouranalysisThegroundswellofinterestinmultiplemethodsdemandsmoreintensiveanddiversifiedformsoftechnicalskill-developmentYetindividualresearchersrarelymastermorethanacoupleofmethodolo-gies Even within a single research tradition technical language andefforts to solve technicalproblems threaten toobscureorovershadowsubstantiveissues(BeckandKatz1996GreenandShapiro1994Shalev2007) If thereare limitstothemethodsanyindividualresearchercanmasterwhatare the implications formultimethodresearchThenexttwosectionselaborateonsomeofthepracticalandcareer-relatedcon-straintsonmethodologicalpractice

Practical Challenges and Methodological Trade-Offs

Methodological debates in the social sciences have had at least threepositiveeffectsFirst steriledebatesover the superiorityofalternativemethodshavegivenwaytoanappreciationoftrade-offsandcomplemen-taritiesbetweenapproachesSecondthegoalsofqualitativeresearchandassociatedmethodsarereceivingmoreexplicitelaborationinresponsetoafeelingthattheywerewidelymisunderstood(BradyandCollier2004Coppedge 1999 Gerring 2001 2004 Goodwin and Horowitz 2002MahoneyandRueschemeyer2003)10 Thirdmoreconstructivecritiqueshave stimulated considerable innovation in techniques within specific

Copyrighted Material

OvercomingChallenges bull 15

methodologicaltraditionsandinstrategiesforcombiningmultiplemeth-odsinresearch(Batesetal1998Braumoeller20032004Gill2004Lieberman2005Ragin19872000)

NonethelessscholarsoftenstruggletomakefullandappropriateuseofavailableresearchmethodsAseachmethodologicaltraditionbecomesmoresophisticatedthetaskofmasteringmultiplemethodsalsobecomesmorechallengingWhenresearchdemandsintensivefieldworkandsub-stantiallocalknowledgeunavoidablylargeinvestmentsindatacollectionpresent additionalobstaclesAll toooftenmethodologicaldiscussionsoverlookthesepracticalconstraintsonmethodologicalchoice

Wepromotecollaborativeresearchasawaytoexpandthepotentialforusingmultiplemethodswellintheanalysisofbroadlycomparativere-searchCollaborationcanbringscholarsfrommultipledisciplinestogetheron the same research teamwith strengths in complementary methodsincreasingconfidencethateachmethodisappliedrigorouslyLikewisecollaborationthatbringstogetherexpertiseaboutdifferentcountriescanexpandthescopeofcomparisonInthisbookwewilldiscussavarietyofstrategiesforcollaborativeresearchandanalyzeobstaclestocollabora-tiveandbroadlycomparativeresearchButfirstweoutlinesomepracti-calconstraintsonmultimethodandcollaborativeresearch

Technological Development and the Costs of Border Crossing

ContemporarysocialsciencefeaturestremendousinnovationwithineachmethodologicaltraditionInnovationindicatesvitalitybutalsoincreasesthecostsofcompetencyinaparticularmethodHigherentrycostsraisethebarrierstomethodologicalbordercrossingAndyetthebenefitsofmultimethodresearchdependoncompetentapplicationofeachmethodOtherwise theuseofmultiplemethodsweakens rather than strength-ensconfidenceintheresearchTobetterillustratethechallengesletusconsiderwhatisrequiredforaresearchertogaincompetencyinseveralmethodsformalquantitativeexperimentalandqualitative

ThetechnicaldemandsofformalmodelingwereevidentevenasthisapproachspreadacrossthesocialsciencesFormalmodelingrequiresacommandofsettheoryandmathematicallogicoptimizationandothertechniquesfromeconomicsgametheoryandcomplexitytheoryCom-putationalmodelersrequireskills inprogrammingandalgorithmicde-signFormaltheoristsdevoteconsiderableenergytothedevelopmentofnewmodelingtechniquesandsolutionconcepts

IncreasinglysimilarconditionsprevailinquantitativemethodologyAsrecentlyasthe1980smanysocialscientistsequatedquantitativeresearchwithordinaryleastsquaresregressionTheassumptionsformultivariateregressionrarelyholdforsocialphenomenahoweverandmoresuitable

Copyrighted Material

16 bull Chapter1

statistical techniques exist ldquoStandardrdquo quantitative techniques nowencompassmaximum likelihood techniquesanalysisof cross-national-time-seriesdataandanalysesofeventhistoriesAvarietyofothertech-niquesincludingBayesianstatisticsandBoolean-basedmethodsarealsobecomingmorecommonComputationalpowerandstatisticalsoftwaremake it very easy toapplyadvanced statistical techniquesbutdonotguaranteeappropriateapplicationEachtechniqueinvolvesaparticularsetofassumptionsdiagnosticchecksandongoingdebatesabouttech-nologicalfixesAswithformalmethodsalargeinvestmentisrequiredoftheresearcherseekingtogainandmaintaincompetencyinevenasubsetofquantitativemethods

Ifresearchersaretoperformexperimentsitiscrucialthattheylearnthepracticeofexperimentaldesigninordertomeasuretherelevantattri-butesofdifferentexperimentaltreatmentsThisrequiresthedevelopmentof hypotheses related to outcomes expected from different treatmentsbased on formal models and statistical analysis on the data collectedfromtheexperimentstotestthesignificanceofdifferencesfoundacrosstreatmentsSomeoneonanexperimentalteamwillalsoneedprogram-mingskillstoenterandanalyzethedataandtoentertheexperimentalinstructions and response categories for experiments run in computerlaboratories

Themenuofqualitativemethodsofdatacollectiontechniquesincludesethnographyparticipantobservationinterviewsoralhistoriesandar-chivalresearchEachtechniqueinvolvesasetofissuesthatresearchersmustunderstandandaddresstoapplythemethodwell (egBurawoy1998Lustick1996Rocheleau1995)Manyofthesetechniquesrequirea substantial period of fieldwork keen observational skills thoroughrecord keeping and a high degree of self-awareness and ethical man-agementofsocialrelations11 Forfieldworkresearchersmusthaveap-propriatelanguageskillsandsufficientunderstandingofthelocalcontexttogainaccessrecognizeinformalinstitutionsandaccuratelyinterpretculturallycodedobservations

Fieldworkyieldsvoluminousdatabutthedatagenerallytakeformsthat are not easily processed (H Becker 1996) Thus the value of aqualitativestudyhingesondisciplineddataanalysisrelatedtotheoreti-calquestions (Campbell1975Lijphart1971)Qualitative researchershavedevelopedavarietyoftechniquestostructuredataanalysissuchascounterfactualanalysisprocesstracingstructuredcomparisonsandanalysisofdeviantcases(BennettandElman2006Fearon1991Gold-stone1997Tarrow2004)ThedevelopmentofsoftwareforComputer-AssistedQualitativeDataAnalysis(CAQDAS)expandsoptionsfordatamanagement There is considerable confusion however about whattheseprogramsdothedifferencesamongthemhowtomatchprograms

Copyrighted Material

OvercomingChallenges bull 17

and theoretical approaches and even whether CAQDAS makes senseforaparticularstudyorapproach(MacMillanandKoenig2004)Asinquantitativeresearchtheincreaseincomputationaltoolscanfacilitaterigorousdataanalysisbutitcanalsoproducemisleadingresultsifap-pliedinappropriately

Thus each method encompasses several sophisticated techniquesWhetheramethodyieldsanalytical insightsormisleadingfindingsde-pends on competency in recognizing appropriate techniques imple-mentingthemwellandmakingsenseofthedataAlargeandongoinginvestmentisnecessaryfortheresearchertogainandmaintaincompe-tencyinagivenmethodTheinvestmentrequiredtomasteranysinglemethodisnotexcessivebut it limits thenumberofmethods inwhichanyindividualcanbeexpectedtogainandmaintaincompetencyWhilescholarsshouldutilizediversemethodsaspossibleandappropriatemeth-odologicalspecializationandmultimethodresearchdesignspresentadi-lemmaCollaborationoffersapotentialsolutionScholarswithstrengthsincomplementarymethodscanworktogetherwithincreasedconfidencethateachmethodisappliedrigorously

Availability and Accessibility of Data

DependingontheperiodcountryandscaleofanalysisdatamightbeabundantandreadilyavailableorvirtuallynonexistentDifferentmeth-ods require different kinds and quantities of data Data compiled bynational and international agencies donot addressmany issues at thesubnational levelandareoftenblindtobothinformal institutionsandnoneliteactorsEveninindustrializeddemocraciesdataavailabilityandqualityvaryconsiderablyacrossstatesprovincescitiesandothersub-nationaljurisdictionsReliableandcomprehensivedatasourcesoftendonot exist for nongovernmental organizations informal institutions orcollectiveactionInpartthelackofreadilyavailabledataoninformalinstitutionssubnationalphenomenanoneliteactorsandothersimilartopicsreflectsthedifficultyofdatacollectionInformalityandnonelitestatusimplyaneedforlocalknowledgeandtrustIntheabsenceoftrustlocalactorsmayhesitate toprovideaccurate informationabout them-selvestheirpracticesorotherinformalinstitutions

Asthecostsofdatacollectionincreasesodotherestrictionsonmeth-odologicalchoiceRecommendationsthatqualitativeresearchersshouldgathermoredata(Goldthorpe1997KingKeohaneandVerba1994Lijphart 1971) ignore the difficulty of recognizing some types of phe-nomenainfieldsettingsthecostsofcollectingqualitativedataandthevoluminousyetdifficult-to-processdatayieldedbyqualitativeresearch(HBecker1996PoteeteandOstrom2004b)Theseconditionsmakeit

Copyrighted Material

18 bull Chapter1

moredifficulttobuildlargedatabasesforquantitativeanalysisevenifquantitativeanalysismakessenseforagiventheoreticalapproach

DataproblemsalsovaryinseverityIfdataavailabilityandaccesswereunproblematicthenscholarscouldchoosemethodologiesthatmatchedtheir causalandepistemologicalassumptionsScholarshave to choosefromasubsetoflessappropriatemethodshoweverwhendataarenotreadilyavailableAsaresultthecapacitytoengageinquantitativeanaly-sisandbroadcomparisonishigherforresearchonformalinstitutionssometypesofinternationalandnationalphenomenaandelitesBecausedataoninformalinstitutionssubnationalissuesandhistoricallydisad-vantagedpopulationsarelessreadilyavailableitisquiteachallengetoengageinbroadlycomparativeandquantitativesocialresearchonthesetopics

At least inprinciplecollaborativeresearchenhancesamoregeneralcomparativeanalysiswithoutsacrificingdataqualityCollaborativere-search offers the potential to collect larger quantities of data engagein more broadly comparative research and utilize a broader array ofmethodscompetentlyUnlikeanindividualresearcherwhoisexpectedtodoitallcollaboratorscanpooltheirdataanddrawoncomplementarymethodologicalskillsUsingformalmodelsScottPage(2007)foundthatgroupswithahigherdiversityofproblem-solvingapproachesaremoreeffectiveinovercomingdifficultproblemsThisgivesusevenmorecon-fidence in strongly recommending collaboration across methods as animportantfoundationforthefuturedevelopmentofthesocialsciences

InpracticecollaborativeresearchisitselfchallengingCollaborationisgenerallylimitedbydivergentresearchinterestsandtheoreticalorien-tations Inconsistency in conceptualization and measurement can be aproblemaswell(PoteeteandOstrom2004b)especiallyforqualitativeresearcherswhoworkhard todevelopcontextually suitablemeasuresYetthesechallengesarenotinsurmountableColleagueswithsharedin-terestsandtheoreticalperspectivescancollaborateonthefullresearchprocess from conceptualization through analysisAs discussedbelowhowever thesocialsciencesstill reward individualresearchmorethantheydo collaborative research Scholars concerned about their careersrecognize these incentivesand limit theirparticipation incollaborativeefforts

Career Incentives as Methodological Constraints

Ideallytraininginthesocialsciencesshouldencouragescholarstodevelopcompetencyinavarietyofmethodsandengageincollaborationsthatfur-

Copyrighted Material

OvercomingChallenges bull 19

therextendtheirmethodologicalrangeUniversitiesshouldfostermulti-methodandcollaborativeresearchbyencouragingcross-appointmentsandbycreatingandsustainingthematicresearchcentersandinitiativesFundingagenciesshouldofferlonger-termgrantstosupportthelongertimeframerequiredformultimethodandcollaborativeresearchInreal-ityacademiarewardsspecializationandindividualprojectsespeciallyinearlycareerstagesAlthoughcollaborativeandmultimethodresearchcanyieldbetterknowledgeindividualaccomplishmentsdomoretoadvancecareersThetenureclockalsogeneratesmorestimulusforrapidresearchoutputthanforthedevelopmentoflonger-termresearchprogramsAndfundingagenciesrarelyprovidelong-termsupport

Training

Graduateprogramcurriculaandprogramsforintensivemethodologicaltrainingprovidean indicationofdisciplinarysupport formultimethodandcollaborativeresearchTraininginquantitativemethodshasbeenastandardcomponentofgraduateprogramsineconomicspoliticalscienceandsociologythroughoutthepostwarperiodLikewiseopportunitiestosupplementin-housecourseswithintensivetraininginmorespecializedquantitativemethodshavebeenavailablefordecadesProbablythebest-knownsourceofspecializedquantitativetrainingforsocialscientiststheInteruniversityConsortiumforPoliticalandSocialResearch(ICPSR)attheUniversityofMichiganhasofferedanannual summer institute inresearchmethodssincethe1960s

BycomparisonoptionsfortraininginqualitativemethodswererareuntilrecentlyBeforetheturnofthe(current)centurymostsocialsciencedepartmentsofferednograduatetraininginqualitativemethodsbeyondacourseinresearchdesignOpportunitiesforintensivetraininginotherqualitativemethodsandinmultimethodresearchhaveexpandedoverthepastdecadeTheConsortiumonQualitativeResearchMethodsholdsanannual intensiveInstitute inQualitativeandMulti-MethodResearch12

The (US)NationalScienceFoundationhassupportedmethodologicaltrainingprogramsforthesocialsciencesincludingmonth-longsummerinstitutesonmultimethodresearchbeginningwiththeEmpiricalImplica-tionsofTheoreticalModels(EITM)programtheSummerInstituteonResearchDesigninCulturalAnthropologyShortCoursesonResearchMethods inCulturalAnthropologyandFieldTraining inMethodsofData Collection in Cultural Anthropology13 Even with these new op-portunities social science graduate students interested inmultimethodresearch find it difficult to gain adequate training in nonquantitativemethods(Siegeletal2007)

Copyrighted Material

20 bull Chapter1

Career Incentives and Specialization

SusanneLohmann (2007)argues forcefully that theprocedures for re-viewingmanuscriptsgrantapplicationsandapplicationsforacademicpositionsandpromotionsstronglyfavorspecializationAlloftheseformsofevaluationrelyonpeerreviewAsLohmannnotespeerreviewgener-allymeansreviewbyspecialistsTheworkofaspecialistwillbereviewedbyother specialists in the samemethodwith the sameareaexpertiseandorwiththesameorsimilarsubstantiveconcernsScholarswiththesamespecializationshareacommonunderstandingoftheirareaassumeitsvalueandarefamiliarwithpracticalchallengesfacedbytheirfavoredapproach

Scholars who engage multiple methods or disciplines on the otherhandwillmostlikelybeevaluatedbydisciplinaryspecialistsratherthanotherpractitionersofmultimethodorinterdisciplinaryresearchThere-viewersarenotlikelytofullyunderstandallofthemethodstherationaleformixingmethodsorthechallengesinvolvedinmultimethodresearchSpecialiststendtodiscounttheresultsofunfamiliarmethodsreferencestoworksinotherfieldspublicationsinjournalsoutsidetheirowndisci-plineandinterdisciplinarypublicationsThusscholarswhousemultiplemethodsanddrawonmultipledisciplines tend toget less enthusiasticandmorecontradictoryevaluationsOnlythebestscholarssurvivethisprocessAsaresultLohmannarguesasmallproportionofsocialscien-tistsaretop-notchscholarswhousediversemethodsandcrosssubfieldanddisciplinaryboundariesbut specialistsdominate thefieldnumeri-cally Despite increased interest in multimethod research hiring com-mitteesstillprefercandidateswhohaveastrongcommandofasinglemethod over candidates with more superficial competency in multiplemethods(Siegeletal2007)

Similar dynamics associated with career incentives constrain collab-orative researchHistorically as a profession the social sciences haverewarded individual innovation and individual accomplishments morethantheyhavecollaborativeresearchCommitteeschargedwithhiringand promotion typically give more weight to single-authored publica-tions than to multiauthored publications (Rothgeb and Burger 2009)Multiauthoredpublicationsareviewedwithskepticisminpartbecauseit is impossible to discern the individual contribution of each authorScholarsarewellawareoftheseissuesandrespondtothemwhenmak-ing decisions about how to pursue their research agendas Collabora-tivesocialscienceresearchhasbecomemorecommonbutpublicationsrarelyhavemore than threeauthors14 Thiscontrasts sharplywith thenaturalscienceswherepublicationsoftenincludethenamesofalloftheresearchersworkinginalaboratory

Copyrighted Material

OvercomingChallenges bull 21

Funding opportunities and career incentives that privilege particu-larmethodsalsoprivilege researchon topics forwhich thosemethodsarepossible(Lohmann2007)Onemightimaginethatrelativescarcityof data and greater practical difficulties in collecting comparable datawould merit higher levels of funding and institutional support for so-ciallyimportanttopicsOftenhoweverthisisnotthecaseScholarswhostudydata-scarcetopicscontendwithpracticalchallengesindatacollec-tionandanalysisthatlimittheirmethodologicaloptionsbutthentheirmethodologicalchoicesoftenlimittheirabilitytocompeteforfundingandgaincriticalappreciationfortheirworkasdiscussedabove

Funding agencies encourage fieldwork collaboration and multi-methodsocial science research to someextent15 Theprevalenceof in-tensivefieldworkhaswaxedandwanedreflectingvariablefinancialandinstitutionalsupportforlanguagetrainingandextendedperiodsoffield-basedresearchaswellasfluctuatingprofessionalappreciationforsuchresearchIntheUnitedStatesfield-basedresearchwasencouragedduringtheperiodimmediatelyfollowingtheworldwarsAsfinancialsupportforareastudiesdeclinedhoweverextendedfield-basedresearchbecameless common Theoretical and methodological trends favored broadlycomparativeanalysiswhichdampenedinterest inextendedfield-basedresearchEvenwhendonorsdosupportthesortofresearchrequiredforthe studyofdata-scarce topics theyrarelyprovide long-termsupportYetforresearchontopicswheredataarerelativelyscarceanddifficulttocollectlong-termsupportmayberequiredtofullyovercomepracticalobstaclestobroadlycomparativeresearchLong-termsupportcouldalsohelpovercomecollective-actionproblemsamongscholars

Careerincentivesdiscourage broadcollaborationandmultimethodre-searchinthesocialsciencesespeciallyforjuniorfacultyandexacerbatecollective-actionproblemsTheinfluenceofcareerincentivesonmethod-ologicalchoicesappearsasaleitmotifinthisbook16 Giventheunavoid-able influence of professional incentives and the other features of theacademicworldthisbookconsidershowfundingagenciesprofessionalassociationsuniversitiesandacademicdepartmentsandprogramscouldbetterencourageinnovativeeffortstotacklepracticalchallengesthatin-fluencemethodologicalchoicesandthusinfluencesubstantiveemphasesinsocialscienceresearch

Our Substantive Focus

We illustrate the challenges advantages anddisadvantages associatedwithparticularmethodswithreferencetoresearchoncollectiveaction

Copyrighted Material

22 bull Chapter1

for the regulation of natural resources In its contemporary form re-searchoncollectiveactionforthemanagementofnaturalresourcesre-spondstoHScottGordon(1954)andGarrettHardin(1968)bothofwhomemphasizedthedifficultyofmanagingsharednaturalresourcesOver the subsequenthalf century scholars fromacross the social andnaturalscienceshaveusedawidevarietyofresearchtechniquestoestab-lishthepossibilityofcollectiveactionfornaturalresourcemanagementidentifyconditionsassociatedwiththeemergenceanddurabilityofcol-lectiveactionandassesswhetherandwhencollectiveactioncontributestosustainablemanagementoftheresourcebase

Althoughwecouldhavedrawnonexamplesrelatedtodiverseresearchagendasfocusingonasinglewell-definedresearchstreamallowsustotracetheinteractionsbetweentheorymethodsandresultsbothintermsofhowtheoryguidesmethodologicalchoicesandhowvariousmethodscontribute to theoretical development Collective-action problems arepervasiveand importantTheyoccur in families theworkplace legis-laturesandinternationalrelationsTheyaffecttheprovisionofpublicgoodslikeinfrastructureandsocialmobilizationofgroupswithsharedpoliticalagendasProblemsofcollectiveactionhavecontributedtothecollapseoffisheriesdeforestationandclimatechangeFurthertheco-authorsof thisvolumehavethemselvesundertakenextensiveresearchusingmultiplemethodsoncollectiveactionforthemanagementofnatu-ralresourcesThuswecanspeakfromexperienceaswellasdrawingontheworkofothers

In approaching natural resource management as a question of col-lectiveactionwearemakinganumberofontologicalassumptionsInourviewtheoreticalexplanationmustidentifycausalmechanismsThetheoryofcollectiveactionassumesthatindividualbehaviorhasacriticalinfluenceoncollectiveoutcomesWearewellawareofthelimitsofratio-nalityhoweverandfavorabehavioraltheoryofindividualactionthatallowsforlimitedinformationattentionandcognitiveprocessingWealsoassumethatindividualbehaviorisstructuredbycontextFornaturalresourcesrelevantcontextualconditionsincludetheecologicalstructureoftheresourcesystemthesociopoliticalandeconomicstructureandanarrayof institutional arrangementsPerhapsmost importantly for thisvolumeweassumethatcomparisonisvaluablebutthatthereisrarelyasingleorlinearpatternWeexpectcausalheterogeneitythereismorethanoneroutetothesameoutcome

Theseassumptionsandconcernshaveinfluencedthetheoreticalpuz-zlesthatwechosetohighlighttheliteraturereviewedandthemethodsexaminedOurontologicalassumptionsaresharedbymanysocialscien-tistsbutnotallStructuralistsinterpretivistsandthosewhobelieveinlawlikesocialpatternsmayrejectoneormoreofourassumptionsWe

Copyrighted Material

OvercomingChallenges bull 23

hopethatthesescholarswillnonethelessbenefitfromthinkingaboutthepotentialtheoreticalcontributionsofavarietyofmethodsandthepracti-calchallengesthataffectmethodologicalpractices

Theresearchtraditiononcollectiveactionfornaturalresourcemanage-mentoffersagoodpointofreferencefordiscussing(1)theinteractionsbetween methodology and theory development (2) multiple methodsandcollaborativeresearch (3)practicalconstraintsonmethodologicalchoices and (4) the influence of career incentives on methodologicalpractice

Interactions between Theory and Methods

Scholarship on collective management of natural resources draws onawidevarietyof researchmethods including innovative strategies foraddressingpracticalmethodologicalconstraintsWewillshowhowdif-ferent methodsmdashabstract formal models case studies meta-analysescross-nationalcomparisonsandlaboratoryandfieldexperimentsmdashhavecontributedatdifferentpointsandindifferentwaystothedevelopmentofthisresearchagenda

Multiple Methods and Collaborative Research

PuzzlesrelatedtocollectivemanagementofnaturalresourcesspanthesocialandnaturalsciencesandinterdisciplinaryresearchisprominentThis research tradition features several innovative efforts toovercomepracticalchallengesandenablemorebroadlycomparativequantitativeandmultimethodresearchYetaswewilldocumentinchapter5collab-orativeresearchremainsrelativelyuncommonWedrawuponourownexperiencesaswellastheliteraturetohighlightboththepossibilitiesandthechallengesofcollaborativeandmultimethodresearch17

Practical Constraints on Methodological Choices

Problems with scarce and difficult-to-access data are rampant in thisresearch tradition Collective management of many natural resourcesoccursonasubnationalscaleandoftenentailsthedevelopmentofin-formalrulesforresourceusewithlittletonogovernmentinvolvementParticipantsinthedevelopmentandenforcementofthesearrangementsmayincludelocalbutnotnecessarilynationalelitesManyexamplesofcollectiveactionfornaturalresourcemanagementmdashoritsabsencemdashin-volveordinaryorhistoricallydisadvantagedpeopleBecauseinformalin-stitutionscanbedifficultforoutsiderstorecognizedataontheseeffortsare scarce and not readily accessible Qualitative field-based research

Copyrighted Material

24 bull Chapter1

isnecessarytosimplyidentifyrelevantcasesforanalysisThepracticalchallengesofsuchresearcharetypicalofworkontopicsforwhichdataarescarceanddifficulttoacquire

Career Incentives and Methodological Practice

Analysisofcollectivemanagementofnatural resourcesrequiresafirmunderstanding of the natural system institutional arrangements andhumanbehaviorArguablyresearchonthistopicisinherentlyinterdis-ciplinaryand requiresmultiplemethodsApatchworkofprojectsandresearchcentersprovidesinstitutionalandfinancialsupportthattosomeextent lowers the risksof interdisciplinaryandmultimethod researchbut thesedonot fullycompensate forsystemwidecareer incentives Ingeneral career incentives encourage either specialization or relativelynarrowformsofmultimethodresearchWedrawuponourownexpe-riences working with interdisciplinary and multimethod research cen-ters thatenjoyedstrong institutionalsupportaswellas inldquoordinaryrdquodiscipline-basedsettings

Outline of the Book

Wewelcometherecentturnawayfromrecurringdebatesoverthesu-periorityofparticularmethodsinthesocialsciencesWeconnectmeth-odologicaldebates todifferencesover theoryandontologyemphasizevariationinthecapacitytoengageinascendantmethodsanddrawouttheimplicationsforcompetitionforcareer-relatedresourcesforseveralperiodsofintensemethodologicalconflictacrossthesocialsciencesWeagree that theuseofmultiplemethods can improve research inmanysituationsbutalso stress that it isnotalwaysappropriateor feasibleThatpracticalconsiderationsconstrainmethodologicalchoicesisacen-tralpointEvenwhenscholarsareawareofandopentodiversemeth-odsmethodologicalchoicesareconstrainedbyspecializedtrainingdatascarcityandproblemsofdataaccessibilityCareerincentiveswithinaca-demiaunfortunatelybyencouragingspecializationanddoing little tofacilitatecollaborationmakeitmoredifficultforscholarstoovercomepracticalobstacles

InthisintroductorychapterwehavelaidoutourmethodologicalandpracticalconcernsandhaveindicatedthatwewillillustrateourpointswithreferencetoresearchoncollectiveactionandthecommonsPartsIIandIIIexamineseveralstrategiesutilizedinresearchoncollectiveactionrelatedtonaturalresourcesForeachresearchstrategy

Copyrighted Material

OvercomingChallenges bull 25

1 weprovideabroadoverviewofthemethod2 wereviewthecontributionsofthemethodtothestudyofcollective

actiononthecommonsand3 wediscussthemethodrsquosstrengthsandweaknesseswhenamethod

isparticularlyvaluableandrefertocomplementarymethods

Wealsodiscuss(inchapter8)somerelativelynewresearchapproachesthat combine formal theoretical methods with data derived from casestudies participatory research and experimental research so as to di-rectlyassessthecapabilityoftheformalmodeltogeneratesimilarpat-ternsofoutcomes

Thus we will address some of the basic concerns related to the useofaparticularmethod includingthefollowing theassumptionsusedtheanalyticalstrategywhetherthemethodhasinternalorexternalva-lidityandcanbereplicatedthepotentialcontributionsofthismethodtotheorydevelopmentandsomepracticalconsiderations(seetable11for anoverviewof these concerns)Weprovide references to textsonparticularmethodsandmethodologicalissuesandwediscusspragmaticconsiderations that influence methodological practices but we do notoutlinehowtoapplyanyspecificmethodInotherwordsthisbookisnotaldquomethodstextbookrdquoInsteadwefocusonwhathasbeenlearnedinabroadresearchprogramthroughtheuseofadiversityofmethodsTheunderlyingissuesarediscussedingeneraltermsexamplesaredrawnpri-marilyfromworkoncollectiveactionfornaturalresourcemanagement

Part II looksmorecloselyatmethodsused inempirical researchre-latedtonaturalresourcesincludingcasestudiesmeta-analysesofcasestudiesandlarge-Nandcollaborativefield-basedempiricalresearchInchapter2wefirstprovideabriefoverviewoftheconventionaltheoryofthecommonsandthenevaluatethecontributionsofcasestudiestotheo-riesofcollectiveactionanddiscusstheirlimitationsChapter3revealsthatdespiteimportantbroadlycomparativeandsyntheticpublicationsoncollectiveactionfornaturalresourcemanagementcasestudiesandsmall-Nstudiesdominatedarticlespublishedonthistopicbetween1990and2004Most large-Nstudiespublishedduring thisperiodanalyzedsurveydatadrawnfromasinglecountryorevenasinglesubnationalre-gionConsequentlytheselarge-Nstudiesofferscantimprovementinex-ternalvalidityandarenotwellsuitedforresearchrelatedtotheprospectsforcollectiveactionAsdiscussedinchapter4meta-analysisallowsformorebroadly comparativeanalysisbymaking structuredcomparisonsbasedonalargenumberofexistingstudiesMeta-analysisisconstrainedbythebodyofexistingempiricalresearchhoweverandcannotsubsti-tute forbroadlycomparativefield-basedresearchChapter5considers

Copyrighted Material

26 bull Chapter1

Table11Methodologicalconcernsandstrategies

Methodologicalconcerns Researchstrategies

Assumptions SingleormultiplecausalpathstooutcomeDeterministicorprobabilisticrelationshipsUniversalorcontingentrelationshipsComparabilityofcases(unithomogeneity)IndependentorinterdependentobservationsRandomassignmentorrepresentativesample

Analyticalstrategy(evidence Controlleddesignorstatisticalcontrolofcausality) Processtracing

Correlationsandanalysisofvariation

Formofvalidityand Internalorexternalvalidityreplicability Easeofreplicabilityoffindings

Potentialcontributionsto Conceptdevelopmenttheorydevelopment Deductiveorinductivetheorydevelopment

Practicalconsiderations Dataissuesaccessavailabilityoflargedatasetsgapsinsourcematerial(missingdata)dataqualitymdashconsistencyaccuracy

Coststravelmdashcostseasedifficultyofmovementfieldexpenseslabmdashavailabilitycosttorunpayoffshigh-endcomputer

Recognitionandorinterpretationofdata(egpotentialunitofcollectiveaction)

Skillslanguageskillslocal(case-specific)knowledgeanalyticalskills(includingQCAorComputer-AssistedQualitativeDataAnalysis[CAQDAS])programmingskillsstatisticalskills

Attributesofresearcherssizeofresearchteamcompositionmultipledisciplines

collaborationasastrategyforbroadlycomparativefield-basedresearchTherewasrelativelylimitedevidenceofcollaborativeresearchinarticlespublishedfrom1990to2004andcollaborativeresearchwasnotmorebroadly comparative than single-authored research was We examineseveral examples of collaboration highlighting strategies developed toovercomepracticalconstraintsandtheoreticalcontributionstothestudyofcollectiveactionfornaturalresourcemanagement

PartIIIdiscussesexperimentalresearchfieldlaboratoriesandformalmodeling approachesGame theoryhasbeen the formal approach for

Copyrighted Material

OvercomingChallenges bull 27

thestudyofcollectiveactioninthepastandwillremainanimportantmethodfortheyearstocomeChapter6focusesonexperimentstostudyhowsmallgroupsmakedecisionsincollective-actionsettingsEarlyex-perimentsshowedthatpredictionsfromgametheorywerenotconfirmedinmanysocialdilemmaexperimentsLaterexperimentshavestimulatedthedevelopmentofanupdated theoryofcollectiveactionChapters7and8discuss the emerginguseofagent-basedmodeling (ABM)asanalternative formalmodelingapproach forcollectiveaction (MillerandPage2007TesfatsionandJudd2006)Thebasicpremiseofagent-basedmodeling is that the macrolevel consequences of many microlevel in-teractionscanbe investigated ItputsmoreemphasisonheterogeneityamongtheactorscognitiveconstraintsandthetopologyofinteractionThefirstgenerationofagent-basedmodelsfocusedonbigquestionsinatheoreticalperspectivesuchasldquoHowdoessegregationemergerdquoandldquoWhendoegoistscooperaterdquo(Axelrod1984Schelling1978)Althoughmostmodelsofthefirstgenerationhavebeeninspiredbyobservationofrealbiologicalandsocialsystemsthemajorityofthesemodelsarenotrigorously tested on empirical data In fact the founding agent-basedmodelingeffortsdonotgobeyondaldquoproofofconceptrdquoHoweverthisischangingsinceanincreasingnumberofscholarsarestartingtoconfronttheirmodelswithempiricalobservationinmorerigorouswaysWedis-cussthesedevelopmentsespeciallyhowtheyarecombinedwithhumansubjectexperimentsandparticipatoryprocesses

Part IVoffersa synthesisChapter9distills lessonsaboutcollectiveactionrelatedtonaturalresourcesWeprovideatheoreticalframeworkofcollectiveactionandthecommonsbasedonthefindingsoverrecentdecadesofempiricalandtheoreticalresearchInchapter10weelaborateon thepractical implications for social science researchusingmultiplemethods

This book confirms that each method can make valuable contribu-tions if appliedappropriatelybutalsounderlines the limitsof relyingonanysinglemethodManyadvantagesexisttomultimethodresearchparticularly as researchmoves through successive stagesYetpracticalchallengesaresignificantThusthefinalchapterreturnstothestructuralfeaturesofacademiathatinfluencemethodologicalchoicesWeidentifyareaswhereincentivescouldbechangedtofostermoremultimethodandcollaborative research aswell as the sort of interdisciplinary researchthatissovaluableforstudyingthemanagementofnaturalresourcesandmanyotherpolicy-relevanttopicsUltimatelythenwehopethisbookwilldrawattentiontopracticalconstraintsonresearchmethods iden-tifystrategiesforovercomingtheseconstraintsandstimulatediscussionsabouthowtoencouragetheiradoption