Page 1
1
“Asking Interesting Questions” Prepared for Sage Handbook of Research Methods in Political Science and International Relations. Robert J. Franzese, Jr. and Luigi Curini, Editors. Forthcoming. ByWilliamRobertsClark1TexasA&MUniversity
Goodresearchisdrivenbyimpatiencewithbadanswerstointerestingquestions.But
wheredointerestingquestionscomefrom?Sincethisistheopeningchapterofahandbookon
researchmethods,itisimperativetopointoutatthestartthatthereisno“method”toasking
researchquestionsinthesenseofacookbookthatyoucanfollowthatwilllead,inexorably,to
scientificdiscovery.Theremaybeascientificmethodforevaluatinganswers,butthereis
certainlynoscientificmethodforaskingquestionsorgeneratinganswers.Andthereis
certainlyroomforalotofcreativityindevelopinginterestingandenlighteningresearch
designs,andseriousshortcomingsto“cookbook”approaches.2KarlPopper,(1962,2003)for
example,arguedthatsciencebeginsafterascientisthasconjecturedananswertoaquestion.
Thescientificmethod,therefore,ismore(perhapsonly)usefulinevaluatinganswersto
questions.Generatingquestionsandanswers,incontrast,isasmuchanartasitisascience.
Butthatisnottosaythattheprocessisrandomorlacksstructure.ThomasKuhn(1962)
saysepisodesofscientificdiscoverybeginwithanindividualwiththe“skill,wit,orgeniusto
recognizethatsomethinghasgonewronginwaysthatmayproveconsequential.”(p.763)But,
hehastenstoadd,“anomaliesdonotemergefromthenormalcourseofscientificresearch1TheauthorwishestothankBranislavSlanchevandLaurieClarkforthoughtfulcommentsandusefulsuggestionsonanearlierdraftofthispaper.2Thefadaround“cleveridentificationstrategies”isbutthemostrecentinstantiationofthisphenomenon.
Page 2
2
untilbothinstrumentsandconceptshavedevelopedsufficientlytomaketheiremergencelikely
andtomaketheanomalywhichresultsrecognizableasaviolationofexpectation.”(p.763)
Intheparlanceofsocialmedia,scientificdiscoverybeginswitha“WTF”moment.
Scientificdiscoverybeginswhenascholarobservessomethingcontrarytoexpectationsand
recognizesthatthisanomalousobservation“mayproveconsequential.”Notethatthe
motivatingfactmaybeanobservationabouttheworld,butitmayalsobeaboutwhatothers
havesaidabouttheworld.3
Butnotjustanysurprisewilldo.Anyonewhohaseverparentedayoungchildisfamiliar
withthequestions,bornoutofwonder,suchasthosethatourchildrenaskedmypartnerand
me:“whyistheskyblue?,”“wheredoesthesungo(attheendoftheday),”?or,“ifmybrain
controlsmybody,whydoIhavetogotothedoctortofindoutwhat’swrongwithmewhenI
amsick”?Answerstoallofthesequestions(assumingtheyareconsistentwithwhatscientists
currentlybelieve)arediscoveriesfortheinquirerbecausetheychangewhattheyknow,but
theydonotleadtoscientificdiscoveriesunlesstheychangewhatweknow.Thefieldsof
optics,astronomy,andneurosciencehavetheirrespectiveanswerstothequestionsabove
(althoughthelastquestionisprobablylesssettledthantheothertwo).
So,questionsoftenbeginwithsurprise,butgoodresearchquestionsbeginwithwell-
informedsurprise.Ifyoualonearesurprisedbyanobservation,theanswertoyour“WTF
3InthewordsofBranislavSlantchev,(personalcommunication)“theoreticinnovationdoesnothavetobeginwithanempiricalobservationbutwithapotentialflawinthelogic,inconsistencyoftheassumptions,oraninsightaboutageneralclaim(e.g.,theimpossibilityresults)”.
Page 3
3
moment”islikelytobepersonallyrewarding.Ifmostwell-informedobserversaresurprisedby
anobservation,thenananswerislikelytobesocially,and,therefore,scientificallyvaluable.4
Butsometimes,scienceproceedswhenanindividualrecognizesthattheanswers
embodiedinwhat“weknow”aboutasubjectarenotverygood.Forexample,formillennia
“we”knewthattheanswertothequestion“wheredoesthesungo”tobesomethinglike“the
suncirclesastationaryearth,soatacertainpointeachdayitleavesoursightwhileshinningon
theotherhalfoftheplanetonlytoreturnthenextmorning.”Eventually,however,scientists
with“theskill,wit,orgenius”torecognizethemountinganomaliescreatedbymodelsbasedon
ageocentricviewoftheuniversecametotheconclusionthatabetteranswerwasneeded.At
firstthesebetteranswerscameinattemptstomodifytheheliocentricviewwithelaborate
patchesmeanttoexplainawayanomalousobservations.Inaddition,to“skill,wit,andgenius”
itrequiredagreatdealofcouragetochallengetheexistingviewinamorefundamental
fashion.
So,goodquestionscomefromknowingwhat“we”know.Buttheyalsocomefrom
thinkingdeeplyaboutwhatweknowandbeingsufficientlyunsatisfiedwithbadanswersto
taketheriskofthinkingdifferentlyaboutaproblem.Aswithallthearts,goodscienceseems
tocomefromindividualsandgroupsthatengageinacertainkindofpractice.Iwouldliketo
beginthisessaybycommentingonwhatIseeasacommonstructureofmanygreat
contributionstopoliticalscienceandinternationalrelations.Specifically,Iwillputforwardalist
4Itisfashionableinmanytopgraduatepoliticalscienceprogramsforfacultytosaythat“substantivecourses”areawasteoftimeandenterprisingstudentsshouldhaveanalmostsingle-mindedfocusonmethodstraining.Itisalsocommonplaceforprofessorstocomplainthattheirstudentsarenotadeptatidentifyinginterestingquestions.Isuspectthatthesephenomenaarenotunrelated.
Page 4
4
offivequestionsthat,whenansweredwell,arelikelytoproduceworkthatasksandanswers
interestingandimportantquestionsandgivesusareasontobeconfidentinthoseanswers.In
thesecondhalfoftheessayIwillruminateonthekindofpracticethatIexpecttoleadtogood
questionaskingandgoodanswergiving.
FiveQuestions
WhenIwasingraduateschool,oneofmyprofessors,D.MichaelShafer,taughtme
howtoread.Hedidsobyencouragingmetoemployatemplatehecreatedsostudentscould
recordthekeypartsofwhattheyread:“Whatisthedependentvariable?”“Whatarethe
independentvariables?“Whatisthelogicthattiesthemtogether?Etc.”Ifoundthis
enormouslyhelpfulingettingthroughtheridiculousamountofreadingrequiredinmy
graduateclasses.WhenIbeganteachingIsharedthislistwithmystudentsandovertheyearsI
haverefineditforvariousreasons.Ihavecometobelievethatthislistofquestionsisuseful
notjustinfocusingourreadingefforts,butalsoinourresearchefforts.Ifyouaskwhatthe
author’sanswertoeachofthefollowingquestionsis,youwillhaveagoodsummaryofmost
articlesorbooksinourdiscipline.5Ifyouaskwhethertheauthorhasagoodanswertoeachof
thequestions,youwillhaveagoodcritiqueofthepaperinquestion.Andifyouareimpatient
withanybadanswersprovidedbytheauthor,anddevelopbetterones,youwillbeonyourway
tomakingyourowncontributiontotheliterature.Consequently,Ihavecometobelievethat
thesequestionscanalsoserveasanexcellentguidewhendesigningaresearchproject.Ifyou
havegoodanswerstothesefivequestions(andatleastoneoftheseanswersisan5Thequestionswouldhavetobeadaptedtoservethispurposeforliteraturereviews,methodspapers,andpurelytheoreticalpapers.
Page 5
5
improvementoverexistingwork),youwillhaveagoodpaper,dissertation,orbook.These
questionsalsocorrespondtotheorganizationofthemodalpaperinourdiscipline:
“Introduction”,“LiteratureReview”,“Theory”,“ResearchDesign”,and“Findings”.
Itisimportanttoaddthatresearchquestionsneednotbegeneratedbyreading.They
canjustaseasily,andperhapsmoreprofoundly,beprovokedbyourinteractionand
observationofthesocialworld.Wemightobservebehaviorandsay“whydoesthathappen”?
Itisgoodpracticetoofferone’stentativeanswertosuchaquestionunencumberedby“the
literature.”Butitisimprudenttospendverymuchtimeonsuchactivitybeforeevaluating
existinganswerstoyourquestion.
Question1:WhatdoIwishtoexplain?(TheIntroduction)
FollowingKuhn’sdescriptionofscientificrevolutions,mostgoodworkbeginswitha
puzzlingobservation.Beginningwithobservationisimportantbecausegoodreaderswouldlike
tobeconvincedthatthephenomenonyouareexplainingactuallyoccurs(thoughitis
frequentlyfruitfultoengageinthoughtexperimentsaboutthingsthathavenotoccurred).
Thisstepisbynomeanstrivialandconsiderablemethodologicalsophisticationmaybe
necessarytoaccuratelydescribetherealworldeventsor,betterstill-patternsofevents-which
youwishtoexplain.
SamuelHuntington’sclassicPoliticalOrderinChangingSocieties(1968)seekstoexplain
therisingpoliticalinstabilityheobservedaroundtheworld.Asevidenceofthisrising
instability,onpagefourofthis462pagebook,theauthorpresentsU.S.DepartmentofDefense
datashowingthatthenumberofnationsaroundtheworldexperiencingmilitaryconflictsof
Page 6
6
varioustypesrosealmostmonotonicallyfrom34in1958to57in1965(Table1.1).Thisisa
dramaticincrease:inlessthanadecadethenumberofconflictsnearlydoubled!Theproblem,
however,isthat,asaresultofdecolonization,thenumberofindependentcountriesinthe
worldalsogrewrapidlyduringthisperiod.IfonetakesHuntington’snumbersanddividesthem
bythenumberofindependentcountriesineachyear(asameasureoftheopportunityfor
militaryconflict),therelativefrequencyofmilitaryconflictactuallydeclinedoverthisperiod.
Sincemilitaryconflictwasjustoneproxyforpoliticalinstability,itisentirelypossiblethat
politicalinstabilityactuallyincreasedduringtheobservedperiod.Butifyoubelievethat
therelativefrequencyofconflictisabetterindicatorofpoliticalinstabilitythantheraw
frequency,youwouldbejustifiedinwonderingifthephenomenonexplainedinthesubsequent
fourhundredorsopagesactuallyoccurred.
Thefirstorderofbusiness,therefore,indemonstratingthatsomethingthatmayprove
consequentialhashappened;istodemonstratethatthatthinghashappened.Thiscrucialtask
Page 7
7
isoftenbestaccomplishedwiththepresentationofclearlypresented,wellthoughtout,
descriptiveevidence.Whilethisoftenrequiresafairamountofmethodologicalskill,
sometimesitsimplyrequiresnumeracy–which,unfortunately,isofteninshortsupply.
Effectivelypresentingevidenceforone’sexplanandumis,perhaps,bestdescribedinthe
breach.Forexample,youcanreadnewspaperheadlinesonalmostadailybasisthatpurportto
capturesomeimportantchangeintheworldthatis,infact,notsupportedbythetextofthe
accompanyingarticle.Wouldthatitwerethecasethatthesemistakeswererareinacademic
work.
Onecommonmistakeistomakeaclaimaboutinter-temporalchangeinavariableby
citingonlycurrentvaluesofthatvariable.“Tenure-trackjobsaredisappearing”readsthetitle
ofanarticle,butthearticlemakesnoreferencetothenumberofsuchjobsthatwereavailable
inthepast.Howdoweknowthatchangehasoccurred?Arelatedissuethatrequiresabit
moremethodologicalskilltoavoidistopointoutadifferencebetweenthevaluesofafew
recentvaluesofavariablefromprecedingvaluesandclaimthattheyareevidenceofanew
trendwithoutcomparingthenewobservationswithalongenoughtrendofdatatodetermine
whethertheyrepresentameaningfuldeviationfromthetrendor,asisoftenthecase,just
typicalvariationwithinthetrend.
Anothercommonerroriswhatmightbecalled“thedenominatorproblem”–thefailure
tochooseadenominatorthatwouldtransformthedataintoavariableappropriateforthe
conceptualcomparisonrelevanttothediscussionathand.Wealreadysawanexampleofthis
whenHuntingtonconfusedatrendintherawfrequencyofavariableforatrendintherelative
frequencyofthedata,whichIarguedwouldhavebeenmoreappropriate.Butitisalsopossible
Page 8
8
thattherawnumberiswhatmostinterestsus–inwhichcaseweshouldnotbedistractedby
anapparentlyrelatedratio.Toreturntothe“disappearingtenuretrackjobs”problemwe
oftenhearaboutinthepopularpress,intherareinstanceswhereinter-temporaldatais
presentedinanattempttoestablishthistrend,thequantitypresentedistypicallytheratioof
tenuretrackjobstothetotalnumbercollegeteachingjobs.Thisisproblematicbecauseitis
entirelypossiblefortheshareoftenuretrackjobstobedecliningwhenthenumberoftenure
trackjobsisincreasing(ashasbeenthecaseintheUnitedStatesfordecades).Anditis
probablythelatternumberthatisofinteresttomostreaders(forexample,currentdoctoral
studentshopingtoforecastfuturedemandforpeoplewiththecredentialstheyareworking
hardtoobtain).
Question2:Whydoesitneedtobetoexplained?(TheLiteratureReview)
Havingexplainedthatthisthinghasoccurred,itisimportantforauthorstodemonstrate
thata)thisthingviolatesexpectationsinsomeway(i.e.“somethinghasgonewrong”)andb)
thatthisviolationmay“proveconsequential.”Inotherwords,inthewordsofMilesDavis,“so
what?”
Onceagain,itmightbeeasiertosaywhatoneshouldnotdo.Ionceattendedapractice
jobtalkwhereasmart,hard-workingand,subsequently,verysuccessfulscholar,whenpressed
tosaywhathewastryingtoexplain,saidthathewastryingtoexplainwhyaparticularvariable
varies.BeinglesssupportivethanIshouldhavebeen,Iasked,“doyouhaveatheorythatleads
ustoexpectthisvariabletobeaconstant?”Variablesvary.Itisevenintheirname.
Page 9
9
Observingthatvariation,therefore,hardlyconstitutesasurprise.Soifvariationinavariable
doesnotconstituteaviolationofexpectations,whatdoes?
Asacomparativepoliticsscholar,itpainsmetosaythatIhaveattendedmanyseminar
talksoverthelastfewdecades,mostgivenbysuccessfulandinfluentialseniorscholars,where
theworkinprogressismotivatedbyanassertionthatissomevariantofthefollowing“puzzle”:
theoryQclaimsthathighlevelsofvariableXshouldcauseYtohappen,butincountryiattimet,Xwasveryhigh,andYdidnotoccur.
Theproblemwiththis“puzzle”isthatoncethemisunderstandingitisbasedonisclearedup,it
isnolongerapuzzle.Themisunderstandingisthis:withveryfewexceptions(Icannotthinkof
one)theempiricalimplicationsofsocialscientifictheoriesarebesttreatedasprobabilistic
(Lieberson1991).Whetheronetracesthereasonstotheintrinsicallyprobabilisticnatureofall
humanbehaviorderivingfromhumanagency,thelimitationsofourunderstanding,thefact
thatmost(all?)socialphenomenahavemultiple,context-dependentcauses,orthepossibility
ofclassificationerror(didYoccurordiditnot?wasXreallyhighorlow?andcomparedto
what?)itisbesttothinkofourhypothesesasprobabilistic.WhichmeansthemosttheoryQ
canclaimisthat“highlevelsofvariableXshouldmakeYmorelikelytohappen.”
Consequently,thefactthatYdidnotoccurincountryiattimet,despitethefactthatXwas
veryhighisnot,atleasttomyear,particularlypuzzling.Unlikelyeventsareexpectedtohappen
occasionally.Consequently,onecannotreasonablycallaprobabilisticconjectureintoquestion
withasinglenullcase.Doingsoislikebeingpuzzledaboutone’sunclewholivedtoaripeold
agedespitebeingaheavysmoker.Thisisnotpuzzlingbecausethebestscientificevidenceis
thatsmokingincreasesthelikelihoodofcancer,notthatitalwaysleadstocancer.Incontrast,it
Page 10
10
wouldbesurprisingtofindanentiresub-sampleofthepopulationthatappearstobeimmune
tothedeleteriouseffectsofsmoking,or,thataftercontrollingforincomeoreducation(orany
otherpotentialconfound),smokersarenotmorepronetocancerthannon-smokers.Insum,
sinceourtheoriestypicallyjustifyexpectationsaboutpatternsofdata,ittakesobservations
aboutpatternsofdata,notdiscreetdatapoints,toviolatethoseexpectations.
Whilerecognizingapatterninthedataisoftennecessaryforgeneratingsurprise,itisbyno
meanssufficient.GoingbacktothemanycomparativepoliticsseminarsIhaveattended,be
waryofthescholarwhoselectsasmallsampleofobservationsanddemonstratesthatawidely
corroboratedempiricalregularity,suchastheincumbencyadvantage,thedemocraticpeace,
Gamson’sLaw,Duverger’sLaw,ortheresourcecurse,“doesn’thold”inthatsub-sample.Why?
Becausesocialbehaviorisprobabilistic,soevenhighlypredictiveempiricalmodelsyield
predictionswithnon-zeroerrors.Asaresult,onecanalwaysfindasubsampleofdatawhere
thebroaderpatterndoesnothold.Takeany“footballshaped”scatterplot,suchasthefamous
scatterplotshowinFigure1.1.6Onecanselectoutasub-sampleofcases,suchasthoseinthe
ellipse,tosuggestthattheregressionlineisflatorevennegativeeventhoughthereisclearlya
positiverelationshipinthesampleonthewhole.
6ThescatterplotisbasedondatafromanexamplefromPearsonandLee’s(1903)examinationofthecorrelationbetweentheadultheightsoffathersandsons.
Page 11
11
Figure1.1RelationshipBetweentheHeightofFathersandSons(DataSource:Freedman,Pisani,andPurves(2007)addedrandomnoisetodatafromPearsonandLee(1903)whoonlyhaddatatonearestinch,)http://myweb.uiowa.edu/pbreheny/data/pearson.html
Page 12
12
RecallthatIsaid,“bewary”ofascholarwhomotivatestheirstudywithasub-sampleof
casesthatappeartoruncontrarytoawell-corroboratedsetofexpectations.ButIwouldnot
encourageyoutodismisssuchascholar.Itis,forexample,entirelyappropriatetoshowthat
thereareboundaryconditionsoneventhemostwell-corroboratedempiricalregularities.But
themereexistenceofsuchasub-sampledoesnotconstituteapuzzleuntilonecanconvince
thereaderthatthesub-sampleconstitutesacomprehensiblecategoryandisnotjusttheresult
offelicitous(fromthestandpointoftheauthorseekingsomethingtowriteabout)case
selection.Further,ifonedoestakeastheirprojectthetaskofexplainingwhyawell-
corroboratedregularitydoesnotapplytoaparticularsub-sample,itisincumbentuponthemto
developanexplanationforwhythesub-sampleisdifferentthatyieldspredictionsotherthan
thefactthatthesub-sampleisdifferent.Otherwise,theyareengagedinbothpost-hocandad-
hocreasoning.
Yetanotherproblemcanarisewhenonegeneratestheirresearchprojectbygazingata
scatterplot.ManywilllookatalikeFigure1.1afterestimatingaregressionlineandbe
disturbedthatsomanyobservationsfallfarfromtheregressionline.Itisokaytowantthe
modeltofitthedatawell,butgiventheprobabilistic,multi-causalnatureofourhypotheses,it
isnotpuzzlingthatsomeobservationsfallfarfromtheregressionline.Myfatherwassixfeet
tall,whileI,ahem,amnot.Thatisnotsurprisingbecauseotherfactorsenterintoheightat
adulthoodotherthanmygeneticinheritancefrommyfather–dietandcontributionsfrommy
mother’sgeneticmake-upcometomind.Beingpuzzledinthiswayisaslightlymore
sophisticatedversionofthe“ifXishighincountryiattimet,whydowenotobserveY”
problem.Bothmethodsarefrequentlyusedtojustifytheclaimthat“existingexplanationsare
Page 13
13
incomplete.”Theproblemisthatanyexplanationtheauthorcomesupwithislikelytobe
susceptibletothesamecriticism.
Iwanttobeclear,thereisnothingwrongwithbeingunsatisfiedwithexplanationsthat
donotfitthedatawell.However,iftheonlyresultofpointingoutobservationsthatfalloffthe
regressionlineisanewmodelthatmarginallyincreasesmeasuresofgoodnessoffit,donotbe
surprisedifreadersfailtoseethisas“consequential.”Ceterisparibus,papersthatare
motivatedbytheidentificationofunclear,misleading,orincorrectunderstandingsinthe
existingliteraturearemoreconsequentialthanthosethatpointtomerely“incomplete”
understandingsbecausetheformercausesustorevise(thatisto“lookatagain”)ratherthan
merelysupplementourcurrentunderstanding.
Sofar,wehavebeenseekingtoidentifyviolationsofexpectationsthatare
consequentialforourunderstandingoftheworld,butonemightalsoplaceapriorityon
consequencesthataremorepractical.Onewayofaskingthe“sowhat”questionistoask,“if
youweresuccessfulinexplainingyouranomalousobservation,howwouldtheworldbe
different?”Unlessoneisentirelynaïve,thisisaverytoughquestiontoanswer.Butsincemost
ofusbecamepoliticalscientistsandinternationalrelationsscholarsbecausewewantedto
maketheworldabetterplace,itisstillworthwhile.Onereasontothinkaboutthe“normative”
implicationsofthequestionsweaskisthatanevenpassingfamiliaritywiththeliteraturein
politicalscienceandinternationalrelationsisenoughtounearthaseeminglyendlesssupplyof
unclear,misleading,orincorrectunderstandings.Inlightofthis,itisnotunreasonabletotryto
tacklefirstthosethataretiedtoissueswecaredeeplyabout.
Page 14
14
NobellaureateRobertLucassaid“onceyoustartthinkingabouteconomicgrowth,itis
hardtothinkaboutanythingelse.”Isuspectthatisbecauseitisnothardtoseetherealworld,
sticktoyourribs,consequencesofeconomicgrowth.Likewise,immigration,politicalviolence,
economicinequality,governmentcorruption,racialandethnicdiscrimination,financial
instability,authoritarianism,genderbias,illiteracy,failingschools,orahostofotherpolicy
issuesareofinterestbecauseoftheirimpactonmattersofjusticeandhumanwell-being.
Explainingobservationsthatviolateourexpectationscanbequiteconsequentialwhendoingso
shedslightontheseandothersocialproblems.
Marx’slastandmostfamousthesesonFeuerbachisthat“thephilosophershaveonly
interpretedtheworld,invariousways.Thepoint,howeveristochangeit”anditisinteresting
thatitisetchedonhistombdespitehavingneverbeenpublishedwhilehewasalive.8It
capturesthefrustrationofmanyscholarswhowouldliketo“makeadifference.”Itcertainly
capturedmyromanticheartwhenIfirstreaditasayoungman(notmuchyoungerthanMarx
waswhenhewroteit)atthestartofgraduateschool.ButIwasnotingraduateschoollong
beforeIrealizedthecomplexityof“interpreting”theworldandthedangersthatcouldresultif
onesoughttochangetheworldwithouthavinginterpreteditcorrectly.Understandingthe
worldisaprerequisiteforchangingitinaresponsiblemanner.
Whileitisdesirable,perhapsevennoble,bridgingthegapbetweenstudyingtheworld
thewayitisandusingthisinformationtoimprovesocialconditionsisdifficult-particularly
whenpeople,and,therefore,politicsareinvolved.Oneproblemisthatifsocialillshave
politicalrootsevenaccurateexplanationsoftheircausesarelikelytobeinsufficientfor
8Marx(1888).
Page 15
15
mitigatingthem.Onereasonforthisisthefactthatthehallmarkofpoliticsisconflictingvalues.
Explainingtoprisonersconfrontedwithpleadealsthatrewardthemforincriminatingeach
otherthattheycollectivelybenefitbykeepingmumwillnotsolvetheprisoner’sdilemma
becausetheywillstillhaveindividualincentivestoratontheirco-conspirators.9
So,whileunderstandingtheworldmaybeanecessaryconditionfor(responsibly)
changingit,itisnotlikelytobesufficient.And,conversely,changingtheworldcanmakeitalot
hardertounderstand.Oneofthethingsthatmakessocialsciencedifficultisthattheentities
westudycanreadwhatwewriteandchangetheirbehaviorinwaysthatmakeourmodelsless
predictivelyaccurate.10
SomethinglikethismayhavebeenatworkinthewritingsofMarx.Thephrase
“workersofalllands,unite!”alsoappearsonMarx’stomb.Incontrasttohistheseson
Feuerbach,thisphrasewaspublishedduringhislife-time.Threeyearsafterbemoaningthe
irrelevanceofpriorphilosophersMarxandEnglesclosedoneofthemostinfluentialpolitical
pamphletseverwrittenwithit.11Inan1890appendixtoTheCommunistManifestoEngels
admitsthatfewheededthecallin1848butsuggestsmanyeventuallydidsoovertime,
includingthosewhowereorganizinginsupportoftheeight-hourworkday1890.Itisnot
unreasonabletosuggestthatMarx’sanalysisofaninternallogictocapitalism(thatthe
9Incontrast,iftheonlyproblemisaco-ordinationproblemthenthemeredisseminationofinformationislikelytobesufficient.Butsuchproblemsareaboutaspoliticalasgettingdriverstostayontheirsideoftheroad.10Thoughsometimesthisworksintheoppositiedirection.Forexample,experimentshaveshownthatstudentswhotakeeconomicsclassesbehavemuchlesscooperatively,and,therefore,moreinlinewiththemodelslearnedinthesecourses.11TheCommunistManifesto(MarxandEngels,1996)hadlittleimmediateimpactonembryonicsocialistmovements,butitslongruninfluenceisundeniable.
Page 16
16
inexorableimmiserationoftheproletariatwouldleadtorevolution)helpedfueltheformation
oflaborunionsandthecreationofsocialprogramsthatimprovedthematerialconditionsof
workers.Butindoingso,thismadethemlessrevolutionary–therebyreducingtheprobability
oftherevolutionhepredicted.
Anotherexampleofhowitishardtohavebothinfluenceintherealworldand
predictiveaccuracycomesfromtherecentliteratureon“thehappinesscurve”–therobust
empiricalregularitythatreportedlifesatisfactiontendstodeclinewhenpeopleareintheir
fortiesandriseconsistentlystartingintheirearlyfifties(Rauch2018).Oneexplanationforthis
empiricalregularityisthatbecausehumanpsychologyisbiasedtowardsoverlyoptimistic
forecasts,youngpeopleover-estimatehowmuchtheirliveswillimproveintheirthirtiesand
forties.Thisresultsindisappointmentduringtheirmiddleyearsevenifindividuals’liveshave
improvedconsiderably,butnotasmuchastheyexpected.Thisdisappointmentalsoleads
peopletoupdatetheirexpectationsandmakegrimforecastsforthefuture.Consequently,
whenlifeintheirfifties,sixties,andbeyondturnsouttobenotasbadasexpected,theyreport
highlevelsoflifesatisfaction.Ifthisprocessistrulyatwork,peoplewhoreadthisliterature
mightbeinclinedtomakemorerealisticpredictionsaboutfuturelifesatisfaction.Iftheydidso
inlargenumbers,the“happinesscurve”coulddisappear.
NoticethattotheextentthatMarxchangedhistoryitmayhavebeeninwaysthat
frustratedbothhispredictiveaccuracyandhissocialdesires(forrevolution)butifhappiness
researchersturnouttohavethesamedegreeofimpactonsocietytheymightbeperfectly
willingtotradepredictiveaccuracyfortangibleimprovementsinpeople’slifesatisfaction.
Page 17
17
Insum,wewouldliketoanswerquestionsthat,whenanswered,wouldprove
consequential.Theseconsequencescanbeeitherforthewaywethinkabouttheworld,orthe
waypeoplebehave.While,allelseequal,wewouldlikeourresearchtoleadtoimprovements
inhumanwell-being,thestrategicnatureofpoliticsmeansthatevenwhenweprovidegood
answerstoquestionsthatareimportanttousitmaynotleaddirectlytoimprovementsinsocial
outcomes.Thatisnottosuggestweshouldstoptrying.
Question3:Whatistheexplanation?(Theory)
Agoodexplanationwilltakeanobservationthatissufficientlysurprisingthatitjustifies
yourstudy,andturnitintosomethingthat,inretrospect,shouldhavebeenexpectedallalong.
InwhatremainsoneofthefewbooksIknowofthatattemptstoteachpeoplehowtoexplain
things,theauthorsofAnIntroductiontoModelsintheSocialSciences(LaveandMarch,1975)
describeexplanationasaprocessinwhichoneimaginesapriorworldsuchthat,ifitexisted,
thesurprisingfact(s)wouldhavebeenexpected.Technically,anysetofstatementsthat
logicallyimplytheoccurrenceoftheanomalousobservationconstituteanexplanation.But
goodexplanationshaveadditionalattributes,andwewouldliketoproducethebest
explanation.Asatisfyingexplanationwillgivethereaderanunderstandingoftheprocessor
mechanismthatislikelytoproducethepreviouslyanomalousobservation.Readerswantto
knowhowsurprisingeventscameabout,andexplanationsshouldtellthem.Good
explanationsareefficient–theratioofthingstheyexplain(implications)tothingstheyrequire
youtobelieve(assumptions)ishigh.
Page 18
18
Thereisanoptimaldegreeofnoveltytoanexplanation.Anexplanationshouldbe
interesting,yetsound.By“interesting”Imeanthatanexplanationshouldcauseustoseethe
worldinanewway.By“sound”Imeananexplanationshouldfitinwithotherthingsweknow
abouttheworld.Anexplanationthatcausesustoseeeverythinginanewwayislikelytobe
wrong.Anexplanationthatdoesnotrequireustochangeourmindatallisprobablyjusta
corollaryofthingswealreadyknew(and,byextension,ourmotivatingpuzzlemustnothave
beenmuchofapuzzle).
Finally,explanationsmustbelogicallyconsistent.Ihavehadempiricallyminded
politicalscientistsandinternationalrelationsscholarstellmethatformaltheoryisnot
importantbecausetheyaresophisticatedenoughtolivewiththeoriesthatcontain
contradictions.Thisisnonsense.Itcanbeshownwithelementarylogicthatanythingfollowsa
contradiction.Consequently,ifyourtheorycontainsacontradiction,anythingcanbesaidto
followfromit.Asaresult,acontradictorytheoryrulesnothingoutand,therefore,noamount
ofempiricalinformationwillbesufficienttofalsifyit.Sincepotentialfalsificationisthe
hallmarkofscience,atheorythatcontainsacontradiction,therefore,isnotascientific
theory.12
Onewaytoincreasethelikelihoodthatyourexplanationislogicallyconsistentistotry
tocaptureitwithaformalmodel.Formalmodelsallowustodemonstratethatour
explanation’sconclusionsfollowfromitsassumptions-mostimportantly,thatourpreviously
puzzlingobservationisnotsurprisinginlightoftheworldthatourexplanationposits.Also,by
12InthepossiblyapocryphalwordsoftheoreticalphysicistWolfgangPauli,itis“notevenwrong.”
Page 19
19
makingtheassumptionsofourexplanationexplicitwearemorelikelytonoticeifthey
contradicteachother.
Whilethesebenefitsofformalizationareundeniable,itdoesnotfollowthatevery
explanationshouldbeformalized.Itypicallyencouragemystudentstofirstarticulatetheir
explanationsasastorythatrevealsaprocessthatproducesthepreviouslyunexpected
observation.Formalizationisonlynecessarywhenonehearssuchastoryandasks,“whywould
peopledothat?”or,equivalently,“Thatdoesn’tsoundlikeanequilibrium,”or,“isn’ttherea
tensionbetweenthispartofthestoryandthatpartofthestory?”Whenoneisconfronted
withsuchquestions,agoodformalmodelcanoftenprovideanswers.Thus,Itellmystudentsto
learnhowtowritedownformalmodelsnotbecausetheywillalwaysneedone,butlikefire
insurance,theyarealwaysatriskofneedingone.
Anotherreasontobeginwithaninformalstatementofone’stheoryistoavoidthetrap
ofthinkingthatagametheoreticmodelwillgenerateatheoryforus.Formalmodelshelpus
interrogatecertainaspectsofourtheory,theydonotproducethetheoryforus.Wemust
beginwithsometheoreticalintuitionaboutwhatexplainsthephenomenoninquestionbefore
wecanbegintomodeltheprocess.
Question4:Iftheexplanationistrue,whatelseshouldweobserve?(ResearchDesign)
Ifyouofferaviewofatheoreticalworldthathasthepreviouslypuzzlingobservationas
oneofitsimplications,youhaveofferedanexplanation.Andwhiletherearevariouswaysto
evaluatethatexplanation,tobescientific,youranswertoyouroriginalquestionmustprovide
ananswertothefollowingquestion:“ifyourexplanationiscorrect,whatelseoughttobe
Page 20
20
true?”Goodscientificexplanationsprovidelotsofanswerstothisquestion.Ifyour
explanationonlyimpliesthefactsthatyousetouttoexplain,thenthereisnowayto
empiricallyevaluateyouranswer.Youcannotusethefactthatdemocraciesseldomfighteach
other,orthefactthatthereisalotofcorruptioninpresidentialdemocraciestoevaluateyour
explanationofthesethingsbecauseitwasthosefactsthatledyoutodevelopyourexplanation
inthefirstplace.
Thispartoftheresearchprocessisastumblingblockformanyresearcherswhenthey
areattractedtoasubject,ratherthanaquestion.IoncehadastudentwhovisitedBraziland
wasshockedbythelevelofcorruptioninthegovernmentthereanddevelopedanexplanation
thatpointedtoaspectsofthelargedistrictmagnitudeproportionalrepresentationelectoral
systemasacause.ThestudentwassurprisedwhenIsaidIthoughttheargumenthadmerits,
butthatreturningtoBraziltocollectdatawasnotapromisingavenueforevaluatingthe
argument:wealreadynewthatBrazilfittheargument!Perhapsdataoncorruptionlevelsin
countrieswithdifferentelectorallaws(suchastheUnitedStates)wouldbemoreuseful.The
student,however,respondedthathedidnotwanttostudycorruptioninothercountries,after
allhewasinterestedinBrazil!
AsimilarproblemisfoundinaveryfamousbookbyThedaSkocpol,StatesandSocial
Revolutions(1978).Init,theauthorwishestoexplaintheoccurrenceofsocialrevolutionsand
shearguedthathersubjectdictatedherempiricalstrategy.Givenherdefinition,thereareonly
fivehistoricalcasesofsocialrevolution.Shearguedthatasaconsequenceofthisfact,
structuredfocusedcomparison(specifically,Mill’sMethodofAgreement)wastheonlypossible
methodforevaluatingherexplanation.Thatisnottrue.
Page 21
21
Thechiefproblemhereisthatifanexplanationforasetofrareeventsonlyhas
implicationsaboutthoserareevents,theauthordoesnothaveadataproblem,theyhavea
theoryproblem.Ifanexplanationforglobalwarmingonlypredictsthegeneralriseinthe
temperaturethatmotivatedtheexplanation,thenitisnotaveryusefulexplanation.
Cosmologistshaveofferedexplanationsforthecreationoftheuniverse,buttheydonotchoose
theirmethodologyforevaluatingtheirexplanationsbasedonthefactthattheobjectoftheir
studyonlyhappedonce.Instead,theyask,“ifmyexplanationforthisuniqueeventiscorrect,
whatelseoughttobetrue?”Theythenthinkabouthowbesttocarefullyobservethe
implicationsoftheirargument.
Thegoalofempiricalresearch,therefore,shouldbetoexamineasmanyimplicationsof
one’sexplanationaspossible.Becausemany,manyscholarsrestricttheirattentiontothe
empiricalpuzzlethatmotivatedtheirstudytobeginwith,manyimportantpaperscanbe
writtenbysimplyaskingofexistingexplanations,“ifthisargumentistrue,whatelseoughtwe
observe?”
Onereasonwhyscholarsoftenrestricttheirattentiontothedatathatgeneratedthe
questionisthatitcanoftentakeconsiderablecreativitytothinkabouttheimplicationsofan
explanation.Thereisnocookbook-likeapproachthatcanbeappliedthatwillautomatically
revealtothescholarthatseeminglyunrelatedeventsmightbeinstantiationsofasinglesocial
process.ButonepracticeLaveandMarchrecommendistotrytoseeyouranswertoa
particularquestionasrelatedtoamoregeneralprocess.
Forexample,inhercriticalreviewofSkocpol’sbook,BarbaraGeddes(2003)suggests
thatoneelementofSkocopol’sexplanationofraresocialrevolutionshadimplicationsforthe
Page 22
22
occurrenceofpeasantrevolts.Geddessuggestthatastatisticalmodelexaminingtheconditions
underwhichpeasantrevoltsdoanddonotoccurwould,therefore,beusefulinevaluatingthe
empiricalrelevanceofSkocpol’sexplanationofsocialrevolutions.
Notice,thatwhenweask“whatelseoughttobetrue”weseparatethequestionof
“whatistheauthorsexplanandum?”from“whatistheauthor’s“dependentvariable?”The
explanandumisastatementofwhattheauthordevelopsatheorytoexplain.The“dependent
variable”istheendogenousvariableinamodeltestingoneormoreoftheimplicationsofthe
author’stheory.Therearetimeswhenthesemightbethesame,butthereisnoreasonto
assumetheywillbe.Infact,whentheyare,weshouldwonderiftheauthorisengagedinpost-
hocreasoning–“havetheyobservedthedependentvariableanditscovariatesandconstructed
acausalstoryafterthefact?”Doingsowouldconstitutea“test”ofthetheoryonlytothe
extentthatthelion’sshareoftheobservationscouldbethoughttohavebeenappreciably
differentfromthosethatwereobservedbeforethetheory’sformulation.Conversely,atheory
thatproducesalotofnovelimplicationshelpsassuagethereader’ssuspicionthattheauthoris
merelyengagedinacurve-fittingexercise.
Insum,itistypicallymorehelpfultothinkofempiricalworkastestingtheimplications
ofatheory,ratherthantestingthetheorydirectly.Onereasonthisistrueisthattestingthe
theorydirectlycaneasilydescendintomoreorlesscomplicatedversionofcurve-fittingand
post-hocreasoning.Instead,spendtimethinkingabouttheimplicationsofyourexplanationfor
observationsotherthanthosethatmotivatedyourquestioninthefirstplace.Themorevaried
thoseimplicationsthebetter,becauseitisonlythoseobservationsthataremadeafterthe
constructionofyourtheorythatruntheriskofbeingfalseandthereforeactuallyconstitutean
Page 23
23
empiricalcheckonyourexplanation.Andremember:ifyourtheoryonlyhasimplicationsfora
setofeventstoosmalltousestandardinferentialtoolstoevaluate,youdonothaveadata
problem-youhaveatheoryproblem.
Question5:Doweobservetheimplicationsofourexplanation?(Findings)
Determiningifevidenceisconsistentwithone’stheoreticalexpectationsistheprimary
focusofresearchmethodologyand,so,isthecentralfocusoftheremainderofthisvolume.
HereIwillmerelystressthefollowing:many,manystudiespresent,oftenindizzyingdetail,
reamsofinformationthatiseitherirrelevanttoorinconsistentwiththeoreticalexpectations.
Typically,however,itispresentedinamannerthatsuggeststhatthisinformationconfirmsthe
author’sexpectations.Distinguishingwhenthisisthecaseisalargepartofwhatismeantby
learningtoreadcritically.
AsIsaid,allofthecollectivewisdomofresearchmethodologistsisrelevantfor
becomingacriticalreaderandproducerofknowledgebutIwillfocusononeadmonition:
presentclearestimatesofthequantitiesofinterestaswellasastatementaboutthedegreeof
confidenceonehasinthoseestimates.13Thereareafewwaysinwhichthisadmonitionis
frequentlyviolated,andIwouldliketobrieflydrawyourattentiontothem.
AtleastinthesocialscientificpapersIread,explanationstypicallyproduceclaimsabout
theassociationbetweenvariables.Evenwhenoneisengagedinwhatlookslikeadescriptive
exercise,likeHuntingon’sattempttodemonstraterisingpoliticalinstability,oneisengagedin
demonstratingthatvaraiblesarerelatedtoeachotherinaparticularway.Ifonewantsto
13King,Keohane,andVerba(1994).
Page 24
24
demonstratethataphenomenonischangingovertime,onemustlookattherelationship
betweenthatvariableandtime.Ifonewantstodemonstratethataparticularbehavioror
attitudeismoreprevalentinsomeplacesoramongsomegroups,onemustlookatthe
relationshipbetweenthatvariableandgroupmembershiporspatiallocation.Consequently,
mostofourempiricalclaimsareabouttherelationshipbetweenvariables.Inalinearmodel
wethinkofthisquantityofinterestasaslopecoefficient,soIwillusethatterminologyhere,
thoughtheterm“derivative”mightbeevenmoreappropriate.
Acommonwayinwhichscholarsbecomedistractedfrompresentingthequantityof
interestisbypresentingsomethingotherthananestimateofaslope,whenthatisthequantity
theyareconcernedwith.Forexample,ithasbecomecommonforscholarstoplotthe
predictedprobabilitiesfromalogitmodelonthey-axiswithsomevariableofinterestonthex-
axiswhenthequantitiyofinterestistheassociationbetweenachangeinthatpredicted
probabilityandameaningfulchangeinsomevariableofinterest.Theproblemwithdoingsois
thatitrequiresthereadertoinfertheslopeofthatrelationshipfromthepicture.Whileitis
truethatslopesarenotconstantinnon-linearmodelssuchaslogit,and,thereforethequantity
ofinterestdoesnotreducetoasinglenumber,itwouldbebettertoplotthemarginaleffectof
thevariableofinterestacrossameaningfulsetofvaluesofthatvariableofinterest.14Adding
confidenceintervalsaroundthepredictedprobabilitydoesnothelpbecausethattellsthe
readerifthepredictedprobabilityissignificantlydifferentfromzero,whichistypicallynotthe
hypothesisbeingtested.
14Inthelanguageofcalculus:ifthequantityofinterestisdy/dx,thenplotdy/dxagainstx,notyagainstx.Theformertellsthereaderwhattheyneedtoknow.Thelattermakesthereadertrytoinferwhattheyneedtoknowfromthepicture.
Page 25
25
Forexample,Hellwig,Ringsmuth,andFreeman(2008)presentthegraphsinFigure1.2
asevidenceinthatthepropensityforcitizenstobelievegovernmentshavelittleroomto
maneuverpolicyinaglobalizedeconomy.Eachpanelplotsthepredictedprobability(and90%
confidenceintervals)thatasurveyrespondentsaidtheydidnotbelievetheU.S.government
retainsthe“roomtomaneuver”policyagainsttherespondent’spartisanship.Theauthors
interprettheapparentdifferencebetweentheslopeoftheplotsinthelefthandpanelfromthe
righthandpanelasevidencethatpartisanshiphasaneffectonrespondentbeliefsamong
respondentswithCollegeDegrees(panela)butnotwithHighSchoolDegreesorLess(panelb)
andamongrespondentsabovetheageoffifty-nine(panelc)butnotbelowtheageofforty
(paneld).Butwhatisthebasisofthisconclusion?Theslopesontherightclearlylooktobe
closetozeroand,incomparison,theslopesontheleftappeartobepositive.Butweare
offeredneitheranestimateoftheslopesforanydegreeofpartisanship,noranestimateofour
uncertaintyaboutthatestimate.Wecantrytocalculatetheslopeatdifferentpointsontheline
byestimatingthe“riseoverrun”andwecankindofcomparethatestimatewiththe
uncertaintyimpliedbytheerrorbars,butwhymakethereaderconstructat-testfromthe
pictureratherthanpresentthatinformationforthereaderbyplottingmarginaleffectswith
theirassociatedconfidenceintervals?Neitherdotheauthorsprovideanyevidencewhether
theslopesintheleft-handpanelsaredifferentfromtheslopesintherighthandpanels.Asa
consequence,thesepictures,andoneslikethemthatappearfrequentlyintheliterature,
providealmostnoquantitativeevidenceaboutthequantityofinterest(underwhatconditions,
ifany,achangeinpartisanshipassociatedwithachangeincitizenbeliefsaboutthe
government’s“roomtomaneuver”).
Page 26
26
Figure1.2PartisanshipandBeliefsabout‘RoomtoManeuver:TheConditionaleffects
ofKnowledgeandAge.SourceHellwig,Ringsmuth,andFreeman(2008,Figure2,p.875.)
Page 27
27
Anothercommonwayofobscuringthequantityofinterestisinpresenting“marginal
effects”thatarenotmarginal.Itiscommonplaceforauthorstosaythingslike“togainsome
substantiveunderstandingoftheseresults,InotethataonestandarddeviationchangeinXis
associatedwitha0.056changeinY.”Theproblemwiththisisthatthereisnothingtypicalor
representativeaboutastandarddeviation–indataapproximatinganormaldistributionabout
two-thirdsofallobservationswillbelessthanastandarddeviationawayfromthemean.Asa
consequence,achangeofastandarddeviationinthevariableofinterestisnotaparticularly
meaningfulcounterfactualtoconsider.Thisisparticularlytruewherethispracticeismost
frequentlyfound–wheninterpretingtheresultsofanon-linearmodel.Underthis
circumstance,themarginaleffectofavariableisextremelysensitivetowhereitisbeing
evaluated.Theslopedescribedbya“marginaleffect”thesizeofastandarddeviationislikely
tobeveryfarfromtheslopeofanyestimatedmarginaleffectwithinthisinterval.Another
reasonwhythisisnotaparticularlyusefulcounterfactualcomparisonisthatmarginaleffects
areinterpretedunderaceterisparibusclausewhereotherfactorsareheldconstant–
somethingwhichisnotlikelytobeapproximatedintherealworldwhenthevariableofinterest
experiencesanunusuallylargechangethesizeofastandarddeviation.15
Anothercommonwayscholarspresentinformationthatisnotthequantityofinterestis
whentheyhaveahypothesisthatisconditionalinnatureandeitherpresentresultsfroman
unconditionalmodel,or,equallycommon,estimateaconditionalmodelbutgoontointerpret
someofitsresultsasiftheywereunconditional.16
15SeeKingandZeng(2006)on“TheDangersofExtremeCounterfactuals.”16SeeBrambor,ClarkandGolder(2006)orKamandFranzese(2007)forafullerdiscussion.
Page 28
28
Summary
Myclaim,uptothispoint,isthatapaper,book,ordissertationthathasgoodanswers
tothefivequestionsabovewillbeausefulpaper,book,ordissertation.Itdoesnotfollowthat
apaper,book,ordissertationmusthaveaninnovativeanswertoallfiveofthosequestions.
Progresscanbemadeaslongasoneoftheanswersisbetterthanexistinganswersandnone
areworse.
Whichquestionsare“mostimportant”and,therefore,whichonesshouldbethefocus
ofyoureffortstoinnovate?Itishardtosay.ThoughIbelievethatitisprobablynotbesttotry
toexplainsomethingthatnoonehasexplainedbefore.Thisisanimportantpoint.Ihavehad
manygraduatestudentsinformmegloomilythatsomeonehasbeatenthemtotheir
“question.”Mystandardreactionistosay,“well,Idoubttheyhavecomeupwiththedefinitive
answer,sowhatareyouworriedabout?”Sinceanyquestionworthaskingislikelytobe
difficulttoanswer,itishighlyunlikelythatanotherscholarislikelyto“beatyoutothepunch”
andstatethe“lastword”onasubject.Indeed,ifyouareaskingaquestionthatnooneelse
hasasked,itshouldgiveyoupause.Maybeitisnotaveryinterestingquestion:ormaybethere
issomethingaboutaskingthequestioninthatwaythatledotherscholarstobelieveproductive
answerswerenotforthcoming.Thatsaid,themerefactthatothersmartpeoplehaveasked
thequestiondoesnotmeanitisagreatideaforyoutotrytoanswerit.
Graduatestudentsaretoldthattheyneedtomakeanoriginalcontributionwhichleads
themtobelievethattheymustaskaquestionthathasneverbeenasked,oratleast,never
beenansweredbefore.Thatisnottrue.Rather,an“originalcontribution”requiresonlythat
thestudentprovideabetteranswertoatleastoneofthequestionsmentionedabove.So,ifa
Page 29
29
studentattheprospectusstageisgoingtoattempttoofferanovelexplanation,thenpartof
theiranswertoquestion2shouldcontainastatementaboutwhattheybringtothetablethat
mightallowthemtomakeprogresswhereothershavefailed.Whattheoreticalinsight,
methodologicaladvantage,orhistoricalknowledgeputstheauthorinapositionto
simultaneouslyrecognizethat“thingshavegonewrong”withexistingexplanationsandoffera
solutionthatpushesthefieldinapromisingdirection?
Since“theoreticalinnovation”isoftenthoughttobethemostprizedcontributiona
politicalscientistcanmake,scholarsoftenbelievethatagoodpapershouldofferanovel
explanation.Ibelievethiscomes,inpart,fromphysicsenvycombinedwiththenotionthat
theoreticalphysicistshaveahigherstatusthanexperimentalists.Ibelievetheideathatevery
importantcontributionmustcontainatheoreticalinnovationhasgreatlyhamperedthe
progressofourdiscipline.Howistheaccumulationofknowledgepossibleifeverytimea
scholarputspentopapertheyhavetoofferanewexplanation?Givenfrequentlyimperfect
researchdesignsandflawedempiricalmethods,Ioftenthinktheoppositeistrue.Wemightbe
temptedtodeclareamoratoriumonthedevelopmentofnewexplanationsuntilthediscipline
hasreachedconsensusaboutempiricaltestsoftheimplicationsofexistingexplanations.Asmy
critiqueofHuntingtonsuggests,ifwedonotgetatleastsomeoftheempiricsright,howdowe
evenknowifourobservationsviolatecurrenttheoreticalexpectationsenoughtowarrantnew
explanations?Onereasontoresistsuchatemptationisthatnewtheoriesdomorethan
explainanomalies.Foroneexample,theyalsoaddressconceptualandlogicalproblemswith
existingexplanations.
Page 30
30
Practicesthatencouragegoodquestionasking
FollowingKuhn’slineofreasoningabove,itisworthaskingwhatislikelytopromote
“theskill,wit,andgenius”capableofrecognizingwhenthingshave“gonewronginwaysthat
mayproveconsequential.”OfKuhn’sthreedesiderata,“skill”seemstheleastconstrainedby
naturalabilityand,therefore,mostresponsivetotheenvironmentswecreate.Whileartistic
creationinvolvesmanyaspects,adegreeofcraftsmanshipistypicallyinvolvedand
craftsmanshipisderivedlargelyfrompractice.Extensivetrainingingametheoryandstatistics
isnowcommonplaceinmostgraduate(andsomeundergraduate)programsinpoliticalscience
andinternationalrelationsandthisiswhatistypicallythoughtofwhenscholarsevaluatethe
“skills”ofjobapplicants.Theseskillsareimportantbecausewithoutthem,scholarsmightask
questionsbasedonfaultyreasoningbasedonformalorinformalfallaciessuchastheecological
fallacy,adhominemattacks,hastygeneralization,confusingcorrelationwithcausation,
ignoringstrategyinducedselectioneffects,andfailingtorecognizethepresenceofconfounds.
Butwhilemethodstrainingisextremelyhelpful,itisnotsufficienttoproducescholars
whoaskandanswerinterestingquestions.Theproblemsetstypicallyassignedinquantitative
methodsandformaltheoryclassesdohelpbuildtheskillsnecessarytoexecutesophisticated
research.Justasplayingscalesandarpeggiosbuildthetechniquesnecessarytoexecute
sophisticatedmusic.Butthereismoretotrainingamusicianthanplayingscalesandarpeggios
becauseasimportantasscalesandarpeggiosare,theyarenotmusic.Ihaveheardmusicians
criticizedforhavingsufficienttechniquethatthey“knowhowtosaythingsontheir
instruments,buttheydonotseemtohaveanythingtosay.”Theanalogouscriticismis
frequentlyleveledatnewlytrainedpoliticalscientistsandinternationalrelationsscholars.
Page 31
31
So,whatistobedone?Toplaygoodmusic,studentshavetolistentogoodmusicand
theyhavetohavealotofexperiencemakinggoodmusic.Mostgraduateprogramsprovide
studentswiththeequivalentoflisteningtomusic.WhenIwasanewlymintedPh.D.Iheard
BruceBuenodeMesquitagivealectureattheHooverSummerPrograminGameTheoryand
InternationalPoliticsatStanfordUniversity.Hebuiltagametheoreticmodelbasedonthe
assumptionsofhegemonicstabilitytheory–seeminglyontheflybasedoncommentsshouted
outbymyclassmates.Ihadanepiphany.Ofcourse,ifdevelopingsocialscientific
explanationsisanart,thenitmustbetaughtastheartsaretaught!Iwaswatchingthemaster
attheeasel–engagedintheverycraftIwastryingtolearn.Itsuddenlyoccurredtomethat
muchofmygraduatetrainingamountedtotheequivalentofsittinginaroomlisteningto
recordingsofmusic,thenwhenitwastimetowritemydissertationitwasasifadoorhadbeen
flungopen,IwashandedaninstrumentIhadneverplayed(Iimaginedacello)andpushedout
ontoastagewhereIwasexpectedtoperform.Mostgraduateprogramsinpoliticalscience
teachpeopletheequivalentofplayingscalesinmethodsclassesandmusichistoryor
appreciationinsubstantiveclassesandarelefttofigureoutontheirownhowtoputthis
togethertomakemusic.
Themissingpieceinmostofourgraduateeducationiswhatmusicianscall“etudes.”
Theseareexercisesdesignedtobemusic-like(sostudentscanbegintothinkabout
interpretationandexpression)butareartificiallydesignedtoallowforadegreeofrepetitionof
particulartechniques(articulation,vibrato,dexterity)thatallowsthoseskillsnecessaryfor
musicalexpressiontoseepintothestudent’smusclememory.Manydoctoralprograms
emphasizethatstudentsshouldwritepublishablepapers,butIbelievethatsuccessisunlikelyif
Page 32
32
thisisattemptedbeforestudentshaveengagedinmanyrepeatedattemptstoexplainthingsor
thinkaboutwhatobservationsareimpliedbytheirexplanations.Studentsneedtopractice
askingandansweringthefivequestionsoutlinedaboveandwritingasinglepaperineach
seminardoesnotgivethemthe“reps”todevelopmusclememory.Virtuallynoskillworthyof
thenamecanbedevelopedafteradozenorsoattempts.
Consequently,Ihavearguedthatproblemsetsin“substantiveclasses”canhelp
studentsbecomeproficientataskingandansweringthequestionsthatwillmakeforinnovative
research.Ananalogytothevisualartsmightbeuseful.Whenstudentsarelearningtodraw,
theyarenothandedablanksheetofpaperandtoldto“thinkofsomethinginterestingtodraw,
thatnooneelsehasdrawn.”Rather,abowloffruit,orperhapsawoodenmodelofahuman
figureisplacedonatable.Then,everyoneintheclassdrawsthesamethingafterreceiving
instructionfromtheinstructorabouthowtodoso.Incontrast,manypoliticalscience
departmentsdotheequivalentofhandingtheirstudentsablanksheetofpaperandaretoldto
“drawsomethinginteresting.”Problemsetsinsubstantiveclassescanbetheequivalentofa
bowloffruit.Theinstructorcanassignstudentstoaquestionrelatedtoaparticularresearch
area.“ExplainwhyXoccursunderZcircumstances.”“IfPexplainsY,whatelseoughtwe
observe?”“WhyisQaninterestingquestion?”“DoesFigure2countasconfirmingor
disconfirmingevidenceforhypothesis2,andwhy?”
Studentsneedalotofexperience“makingmusic”beforethey“havesomethingtosay.”
Iftheanalogytotheartsdoesnotresonatewithyou,considerthefollowing.Politicalscience
andinternationalrelationscantakealessonfromtheso-called“benchsciences”where
studentsworkonmanyprojectsasmembersoflargeteamsbeforetheyaretaskedwiththe
Page 33
33
responsibilityofdecidingonthetopicofthegroup’snextproject.Experienceandrepetition
helpsstudentslearnwhatworksandwhatdoesnot.
Whilegraduatepedagogyisimportantforstimulatingcreativequestionaskingand
answering,thebroaderclimateandculturewecreateinourdepartmentsandresearchcenters
isequallyimportant.Inparticular,itisextremelyimportanttocreateanenvironmentwhereit
issafetoplaywithideasandchallengeorthodoxy.Ioncehadacolleaguewho,whilewalking
downthehallreadapassagefromabookthathethoughtwasincorrectandloudlydeclared
theauthoran“idiot.”Creativityandrisktakingisnotencouragedbyaculturethatsuggests
thatonlystupidpeoplesaystupidthings.Instead,itisimportanttocreatetheideathatthe
smartestamongusarecapableoferrorandthatthereisabigdifferencebetweensaying
somethingthatstupidandbeingstupid.Tothatend,Ithinkitisextremelyimportantforsenior
scholarstobetransparentabouttheerrorstheyhavemade.Youngscholarsneedtolearnthat
ifthey’vemadeamistake,theyareinverygoodcompanyandiftherequirementforadmission
wasnevermakingamistakethebuildingwouldbeempty.
Whileacultureofsupportforindividualrisktakingisvitaltoanyscientificorartistic
community,thereisanoptimaldegreeofindividualismbehindscientificdiscovery.Ifyoudon’t
readwhateverybodyelsereadsandfailtotrainlikeeveryoneelsetrains,youwillasknaïve
questionsthattherestofyourcommunityknowstheanswersto.Butifyouonlyreadwhat
everyreads,andonlytrainlikeeveryoneelsetrains,youareunlikelytoexperiencethat
momentwhenyouseesomethingthathasgonewrongthatnooneelsesees.
JazzbassistScottLaFarostartedplayingthebassin1954whenhewas19yearsoldand
inthefewshortyearsbeforehewaskilledinatragiccaraccidentin1961,hecompletely
Page 34
34
changedtheworld’sconceptionofwhatcouldbeaccomplishedonadoublebassandwhatrole
theinstrumentcouldplayinapianotrio.Priortoplayingthebasshehadplayedtheclarinet
andsaxophoneforyearsandmanyhaveattributedhisphenomenaltechnicalprowesstothe
factthathepracticedthebassbyplayingetudescomposedfortheclarinetbyHyacintheKlosĕ
inthe19thcentury(LaFaro-Fernandez,2009).ThelessonLaFarotaughttheworld,inadditionto
thegeneralbenefitsofinter-disciplinarily,was“ifyouwanttosoundlikeeveryoneelse,
practicelikeeveryoneelse;butifyouwanttosoundlikeno-oneelse,practicelikenoone
else.”17
Justasthereisanoptimaldegreeofindividualitythatislikelytoproducescholarswith
theskill,wit,andgeniustodeterminewhensomethinghasgonewronginwaysthatmayprove
consequential,communitiesthatstriketherightbalancebetweenconformityanddiversityare
likelytoencouragethehabitsthatleadtoscientificbreakthrough.
Ontheonehand,itisimportantforascientificcommunitytoshareacommitmentto
thegrowthanddisseminationofknowledgeandacommonunderstandingofthelogicof
inferenceandthestandardsofevidence.Withoutthissharedunderstanding,criticismislikely
tofallondeafears.Butontheotherhanditisimportantforacommunitytobeasdiverseand
eclecticaspossible.Peoplefromdifferentcultural,class,linguistic,andreligiousbackgrounds
arelikelytoseethesocialworlddifferentlybecausetheyarelikelytohavehaddifferent
experiences.Thesedifferentexperiencesarelikelytoleadtodiversemoral,political,andsocial
intuitionsthatleadthemtoraisequestionsthatamorehomogeneousgroupmightnot(Page,
2007).17Atthesametime,nearlyeveryinnovativejazzmusicianlearnedtheircraftbymemorizingperformancesofmusiciansthatcamebeforethem.
Page 35
35
Inaddition,diversegroupsarelesslikelytofallpreytowhatIcall“strategic
confirmationbias.”Confirmationbiasoccurswhenanindividualembracesanideauncritically
becauseitconformstotheirpriorbeliefs.Whenconfirmationbiasisatwork,peopleareless
likelytoscrutinizetheresearchpracticesthatproducedtheclaiminquestion.Theyareless
likelytolookforconfounds,toaskaboutthedetailsofdatacollection,ortothinkcritically
abouteitherthemicro-foundationsormoralimplicationsofaclaimbecausetheresultsconfirm
whattheyhavelongsuspectedabouttheworld.
Butstrategicconfirmationbiasoccurswhenanindividualisabletoovercomefirst-order
confirmationbiasandthinkcriticallyabouttheclaimbeingmade,butisdeterredfromvoicing
thecriticismbecausetheybelieveothersarerefrainingfromcriticismasaresultof
confirmationbias.Undersuchcircumstances,criticallyengagingtheclaiminpublicmightsignal
toothersthatthecriticdoesnotsharetheirbeliefsonthematter.
Strategicconfirmationbiasismostlikelytobeaproblemincommunitieswhere
“everybody”sharesparticularbeliefs.Insuchanenvironment,thinkingcriticallyaboutaresult
thatconfirmsthecommunity’sbeliefscouldresultinostracism,orattheveryleast,fewer
dinnerinvitations.Acommunitycomprisedofindividualsfromdiverseeducational,class,
religious,andideologicalbackgroundsislesslikelytoproducethekindofmonolithicviewsthat
encouragestrategicconfirmationbias.Individualsaremorelikelytosaysomethingwhenthey
seesomethingwrongthatmayproveconsequentialbecausethesetoftakenforgranted
sharedbeliefsislikelytobesmaller.Diversityismostlikelytobehelpfulinthisregardwhen
themultipledimensionsofidentityarerelativelyuncorrelated.Ifgender,race,orideologyare
heavilycorrelated,thendissentononedimensioncanbeseenasdefectiononanother.Thus,
Page 36
36
inidealcircumstancescommunitieswouldhaveasmuchwithingroupdiversityasbetween
groupdiversity.18Ofcourse,diversityhastobesufficientlydevelopedtogiveindividuals
confidencethatspeakingupundersuchcircumstanceswillnotsimplyconfirmthatoneisan
“outsider.”Ifacommunitypromulgatesthenormthatinamultidimensionalspaceweareall,
ononedimensionoranother,outsiders,thecostofrevealingthatone“thinksdifferently”
aboutsomethingislikelytobelesscostly.Thedauntingthingaboutstrategicconfirmationbias
isthatitismostlylikelytooccuraroundissuesscholarsfeelpassionateabout.Asaresult,
thereisadangerthataresearchcommunitywillbeleastscientificaboutthemattersthatit
caresmostdeeplyaboutandmostscientificaboutmattersitsparticipantsviewaslargely
inconsequential.
CONCLUSION
Goodscientistsaskinterestingquestionsandareunsatisfied,evenimpatient,withbad
answers.Ihavearguedthatmostworkinpoliticalscienceandinternationalrelationscanbe
understoodthroughthelensoffivequestionsandthatcontributionscanbemadetothe
literaturebyimprovingonaresearchcommunity’sanswertoanyofthefivequestions.
Sincecomingupwithbetteranswerstoquestionsisasmuchart,asitisscience,Ihave
arguedthatthebestwaytotraingoodsocialscientistsistolearnfromthewayartistsare
trained.Musicalandvisualartistslearntheircraftsthroughstructuredrepetitivepractice.
Theimplicationofthisinsightforthesocialsciencesisthatscholarsshouldbegivenmaterials
toworkwiththatallowthemtoengageinthedailypracticeofaskingandansweringthefive18Theconnectionbetween“intersectionality”andcross-cuttingcleavagesshouldbeexploredfurther.
Page 37
37
questionsoutlinedinthefirstsectionofthepaper.Ihavesuggestedthatthebestwayto
encouragethisisthroughtheuseofproblemsetsinoursubstantivecourses.Ihavealsohinted
thattherearegreatbenefitstointer-disciplinarity.Bybringhabits,techniques,andinsights
thatarenormalinonedisciplinetoasettingwheretheyarerare,individualsaremorelikelyto
recognizewhensomethinghas“gonewronginwaysthatmayproveconsequential.”Finally,I
havearguedthatdiversecommunitiesaremorelikelytoproducegoodquestionaskers,inpart
becausetheyarelesslikelytofallpreytostrategicconfirmationbias.
Page 38
38
References
Brambor, Thomas, William Roberts Clark and Matt Golder. 2006. “Understanding Interaction Models:
Improving Empirical Analysis” Political Analysis. 14(1): 63-82.
Cindy D. Kam and Franzese, Robert J. Jr. 2007. Modeling and Interpreting Interactive Hypotheses in
Regression Analysis. (Ann Arbor: University of Michigan Press).
Friedman, David, Robert Pisani and Roger Purves. 2007. (New York: W.W. Norton).
Geddes, Barbara. 2003. Paradigms and Sand Castles: Theory Building and Research Design in
Comparative Politics.
Hellwig, Timoth T, Eve M. Ringsmuth, and John R. Freeman. 2008. “The America Public and the Room
to Maneuver: Responsibility Attributions and Policy Efficacy in an Era of Globalization.” International
Studies Quarterly 52(4).
Huntington, Samuel. 1968. Political Order in Changing Societies. (New Haven: Yale University Press).
King, Gary, Robert O. Keohane, and Sidney Verba. 1994. (Princeton: Princeton University Press).
King, Gary and Langche Zeng. 2005. “The Dangers of Extreme Counterfactuals,” Political Analysis
14:131-159.
Kuhn, Thomas S. 1962. “Historical Structure of Scientific Discovery,” Science 136(3518):760-764.
Lafaro-Fernandez, Helene. 2009. Jade Visions: The Life and Music of Scott LaFaro. (Denton, Tx:
University of North Texas Press)
Lave, Charles A. and James G. March. 1975. (New York: Harper and Row).
Lieberson. Stanley. 1991. “Small N’s and Big Conclusions: An Examination of the Reasoning in
Comparative Studies Based on a Small Number of Cases.” Social Forces 70: 307-320.
Page 39
39
Marx, Karl. 1888. “Theses on Feuerbach” in Friedrich Engels, editor, Ludwig Feuerbach and the End
of Classical German Philosophy.
Marx, Karl and Friedrich Engels. 1996. The Communist Manifesto. (London: Pluto Press).
Page, Scott. E. 2007. The Difference: How the Power of Diversity Creates Better Groups, Firms, Schools,
and Societies. (Princeton: Princeton University Press).
Pearson, K. and Lee, A. (1903). On the laws of inheritance in man: I. Inheritance of physical characters. Biometika, 2(4), 357-462.
Popper, Sir Karl. 2003 [1959]. The Logic of Scientific Discovery. New York: Routledge.
Popper, Sir Karl. 1962. Conjectures and Refutations: The Growth of Scientific Knowledge. New York:
Basic Books.
Rauch, Jonathan. 2018. The Happiness Curve: Why Life Gets Better After 50. (New York: St. Martin’s
Press).
Skocpol, Theda. 1979. States and Social Revolutions: A Comparative Analysis of France, Russia, &
China. New York: Cambridge University Press.