Vindicating Methodological Triangulation Remco Heesen 1 Liam Kofi Bright 1 Andrew Zucker 2 1 Department of Philosophy Carnegie Mellon University 2 Department of Mathematical Sciences Carnegie Mellon University Scientific Discovery in the Social Sciences London School of Economics and Political Science 30–31 January 2015
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Vindicating Methodological Triangulation
Remco Heesen1 Liam Kofi Bright1 Andrew Zucker2
1Department of PhilosophyCarnegie Mellon University
2Department of Mathematical SciencesCarnegie Mellon University
Scientific Discovery in the Social SciencesLondon School of Economics and Political Science
30–31 January 2015
Introduction The Model Conclusion References
The Authors
Remco Heesen Liam Kofi Bright Andrew Zucker
Heesen, Bright, and Zucker Vindicating Methodological Triangulation LSE, 30 January 2015 2 / 12
Introduction The Model Conclusion References
Methodological Pluralism and Triangulation
“The best available methods of sociologicalresearch are at present so liable to inaccuraciesthat the careful student discloses the results ofindividual research with diffidence.” (Du Bois1996 [1899], p. 2)
“The use of [multiple methods] may perhapshave corrected to some extent the errors ofeach.” (Du Bois 1996 [1899], p. 3)
Heesen, Bright, and Zucker Vindicating Methodological Triangulation LSE, 30 January 2015 3 / 12
Introduction The Model Conclusion References
Methodological Pluralism and Triangulation
“The best available methods of sociologicalresearch are at present so liable to inaccuraciesthat the careful student discloses the results ofindividual research with diffidence.” (Du Bois1996 [1899], p. 2)
“The use of [multiple methods] may perhapshave corrected to some extent the errors ofeach.” (Du Bois 1996 [1899], p. 3)
Heesen, Bright, and Zucker Vindicating Methodological Triangulation LSE, 30 January 2015 3 / 12
Introduction The Model Conclusion References
Methodological Purists
Arguments against triangulation:
I “Research methods are often developed within differing researchtraditions carrying varying epistemological and theoreticalassumptions with them. Thus the combination of methods [will] notlead to more valid results.” (Kelle 2005, p. 99)
I There is One True Method (e.g., Tewksbury 2009)
Heesen, Bright, and Zucker Vindicating Methodological Triangulation LSE, 30 January 2015 4 / 12
Introduction The Model Conclusion References
Methodological Purists
Arguments against triangulation:
I “Research methods are often developed within differing researchtraditions carrying varying epistemological and theoreticalassumptions with them. Thus the combination of methods [will] notlead to more valid results.” (Kelle 2005, p. 99)
I There is One True Method (e.g., Tewksbury 2009)
Heesen, Bright, and Zucker Vindicating Methodological Triangulation LSE, 30 January 2015 4 / 12
Introduction The Model Conclusion References
Related Issues
I A pluralism of models in the natural sciences (e.g., climate science,evolutionary biology)
I (Bayesian) confirmation theory and the “variety of evidence thesis”(Fitelson 2001)
Heesen, Bright, and Zucker Vindicating Methodological Triangulation LSE, 30 January 2015 5 / 12
Introduction The Model Conclusion References
Example: Housing Policy
I Four answers: Great, Good, Bad, Terrible
I Three methods: Ethnographic participant observation, sociologicalsurvey, rational choice models
Heesen, Bright, and Zucker Vindicating Methodological Triangulation LSE, 30 January 2015 6 / 12
Introduction The Model Conclusion References
The Triangulator and the Purist
I Each method: Pr(Good) = 1/3, Pr(other answer) = 2/9
I Purist believes correct answer w.p. 1/3
I Triangulator believes correct answer w.p. 29/81 > 1/3
Heesen, Bright, and Zucker Vindicating Methodological Triangulation LSE, 30 January 2015 7 / 12
Introduction The Model Conclusion References
A First Result
I n answers, one correct
I m methods: probability distribution over answers
Theorem 1 (List and Goodin (2001))
If all methods have the same probability distribution, which is more likelyto pick the correct answer than any given alternative, then the triangulatoris more likely to believe the correct answer than the purist (providedn ≥ 2, m ≥ 3).Moreover, the probability that the triangulator picks the correct answer isincreasing in m, and approaches 1 as m→∞.
Note: Reflects Du Bois, but will not convince the purist
Heesen, Bright, and Zucker Vindicating Methodological Triangulation LSE, 30 January 2015 8 / 12
Introduction The Model Conclusion References
A First Result
I n answers, one correct
I m methods: probability distribution over answers
Theorem 1 (List and Goodin (2001))
If all methods have the same probability distribution, which is more likelyto pick the correct answer than any given alternative, then the triangulatoris more likely to believe the correct answer than the purist (providedn ≥ 2, m ≥ 3).Moreover, the probability that the triangulator picks the correct answer isincreasing in m, and approaches 1 as m→∞.
Note: Reflects Du Bois, but will not convince the purist
Heesen, Bright, and Zucker Vindicating Methodological Triangulation LSE, 30 January 2015 8 / 12
Introduction The Model Conclusion References
An Extreme Case
I The One True Method: Ethnographic participant observation
I Other methods merely guesswork
Theorem 2If The One True Method always picks the correct answer, and othermethods pick randomly, then the triangulator is more likely to believe thecorrect answer than the (diffident) purist (provided n ≥ 2, m ≥ 3).
Note: Non-diffident purist may do better than the triangulator
Theorem 3If The One True Method is more likely than 1/n to pick the correctanswer, and other methods pick randomly, then the triangulator is morelikely to believe the correct answer than the (diffident) purist (providedn ≥ 2, m ≥ 3).
Heesen, Bright, and Zucker Vindicating Methodological Triangulation LSE, 30 January 2015 9 / 12
Introduction The Model Conclusion References
An Extreme Case
I The One True Method: Ethnographic participant observation
I Other methods merely guesswork
Theorem 2If The One True Method always picks the correct answer, and othermethods pick randomly, then the triangulator is more likely to believe thecorrect answer than the (diffident) purist (provided n ≥ 2, m ≥ 3).
Note: Non-diffident purist may do better than the triangulator
Theorem 3If The One True Method is more likely than 1/n to pick the correctanswer, and other methods pick randomly, then the triangulator is morelikely to believe the correct answer than the (diffident) purist (providedn ≥ 2, m ≥ 3).
Heesen, Bright, and Zucker Vindicating Methodological Triangulation LSE, 30 January 2015 9 / 12
Introduction The Model Conclusion References
An Extreme Case
I The One True Method: Ethnographic participant observation
I Other methods merely guesswork
Theorem 2If The One True Method always picks the correct answer, and othermethods pick randomly, then the triangulator is more likely to believe thecorrect answer than the (diffident) purist (provided n ≥ 2, m ≥ 3).
Note: Non-diffident purist may do better than the triangulator
Theorem 3If The One True Method is more likely than 1/n to pick the correctanswer, and other methods pick randomly, then the triangulator is morelikely to believe the correct answer than the (diffident) purist (providedn ≥ 2, m ≥ 3).
Heesen, Bright, and Zucker Vindicating Methodological Triangulation LSE, 30 January 2015 9 / 12
Introduction The Model Conclusion References
Summary
I Voting theory framework shows benefits of triangulationI From an optimist perspectiveI From a pessimist perspective
Heesen, Bright, and Zucker Vindicating Methodological Triangulation LSE, 30 January 2015 10 / 12
Introduction The Model Conclusion References
Future Research
I Correlated errorI Generalizing the resultsI Individuating methods
I Social epistemology (Mayo-Wilson et al. 2011)
Heesen, Bright, and Zucker Vindicating Methodological Triangulation LSE, 30 January 2015 11 / 12
Introduction The Model Conclusion References
Thank You!
Thank you for your attention!
Questions?
Heesen, Bright, and Zucker Vindicating Methodological Triangulation LSE, 30 January 2015 12 / 12
Introduction The Model Conclusion References
References I
W. E. B. Du Bois. The Philadelphia Negro: A Social Study. University ofPennsylvania Press, Philadelphia, 1996 [1899].
Branden Fitelson. A Bayesian account of independent evidence withapplications. Philosophy of Science, 68(3):S123–S140, 2001. ISSN00318248. URL http://www.jstor.org/stable/3080940.
Udo Kelle. Sociological explanations between micro and macro and theintegration of qualitative and quantitative methods. Historical SocialResearch / Historische Sozialforschung, 30(1 (111)):95–117, 2005. ISSN01726404. URL http://www.jstor.org/stable/20762014.
Christian List and Robert E. Goodin. Epistemic democracy: Generalizingthe Condorcet Jury Theorem. Journal of Political Philosophy, 9(3):277–306, 2001. ISSN 1467-9760. doi: 10.1111/1467-9760.00128. URLhttp://dx.doi.org/10.1111/1467-9760.00128.
Heesen, Bright, and Zucker Vindicating Methodological Triangulation LSE, 30 January 2015 13 / 12
Conor Mayo-Wilson, Kevin J. S. Zollman, and David Danks. Theindependence thesis: When individual and social epistemology diverge.Philosophy of Science, 78(4):653–677, 2011. ISSN 00318248. URLhttp://www.jstor.org/stable/10.1086/661777.
Richard Tewksbury. Qualitative versus quantitative methods:Understanding why qualitative methods are superior for criminology andcriminal justice. Journal of Theoretical and Philosophical Criminology, 1(1):38–58, 2009. ISSN 21668094. URLhttp://search.ebscohost.com/login.aspx?direct=true&db=
i3h&AN=57233244&site=ehost-live.
Heesen, Bright, and Zucker Vindicating Methodological Triangulation LSE, 30 January 2015 14 / 12