Vindicating Methodological Triangulation

Vindicating Methodological Triangulation

Remco Heesen1 Liam Kofi Bright1 Andrew Zucker2

1Department of PhilosophyCarnegie Mellon University

2Department of Mathematical SciencesCarnegie Mellon University

Scientific Discovery in the Social SciencesLondon School of Economics and Political Science

30–31 January 2015

Introduction The Model Conclusion References

The Authors

Remco Heesen Liam Kofi Bright Andrew Zucker

Heesen, Bright, and Zucker Vindicating Methodological Triangulation LSE, 30 January 2015 2 / 12

Introduction The Model Conclusion References

Methodological Pluralism and Triangulation

“The best available methods of sociologicalresearch are at present so liable to inaccuraciesthat the careful student discloses the results ofindividual research with diffidence.” (Du Bois1996 [1899], p. 2)

“The use of [multiple methods] may perhapshave corrected to some extent the errors ofeach.” (Du Bois 1996 [1899], p. 3)

Heesen, Bright, and Zucker Vindicating Methodological Triangulation LSE, 30 January 2015 3 / 12

Introduction The Model Conclusion References

Methodological Pluralism and Triangulation

“The best available methods of sociologicalresearch are at present so liable to inaccuraciesthat the careful student discloses the results ofindividual research with diffidence.” (Du Bois1996 [1899], p. 2)

“The use of [multiple methods] may perhapshave corrected to some extent the errors ofeach.” (Du Bois 1996 [1899], p. 3)

Heesen, Bright, and Zucker Vindicating Methodological Triangulation LSE, 30 January 2015 3 / 12

Introduction The Model Conclusion References

Methodological Purists

Arguments against triangulation:

I “Research methods are often developed within differing researchtraditions carrying varying epistemological and theoreticalassumptions with them. Thus the combination of methods [will] notlead to more valid results.” (Kelle 2005, p. 99)

I There is One True Method (e.g., Tewksbury 2009)

Heesen, Bright, and Zucker Vindicating Methodological Triangulation LSE, 30 January 2015 4 / 12

Introduction The Model Conclusion References

Methodological Purists

Arguments against triangulation:

I “Research methods are often developed within differing researchtraditions carrying varying epistemological and theoreticalassumptions with them. Thus the combination of methods [will] notlead to more valid results.” (Kelle 2005, p. 99)

I There is One True Method (e.g., Tewksbury 2009)

Heesen, Bright, and Zucker Vindicating Methodological Triangulation LSE, 30 January 2015 4 / 12

Introduction The Model Conclusion References

Related Issues

I A pluralism of models in the natural sciences (e.g., climate science,evolutionary biology)

I (Bayesian) confirmation theory and the “variety of evidence thesis”(Fitelson 2001)

Heesen, Bright, and Zucker Vindicating Methodological Triangulation LSE, 30 January 2015 5 / 12

Introduction The Model Conclusion References

Example: Housing Policy

I Four answers: Great, Good, Bad, Terrible

I Three methods: Ethnographic participant observation, sociologicalsurvey, rational choice models

Heesen, Bright, and Zucker Vindicating Methodological Triangulation LSE, 30 January 2015 6 / 12

Introduction The Model Conclusion References

The Triangulator and the Purist

I Each method: Pr(Good) = 1/3, Pr(other answer) = 2/9

I Purist believes correct answer w.p. 1/3

I Triangulator believes correct answer w.p. 29/81 > 1/3

Heesen, Bright, and Zucker Vindicating Methodological Triangulation LSE, 30 January 2015 7 / 12

Introduction The Model Conclusion References

A First Result

I n answers, one correct

I m methods: probability distribution over answers

Theorem 1 (List and Goodin (2001))

If all methods have the same probability distribution, which is more likelyto pick the correct answer than any given alternative, then the triangulatoris more likely to believe the correct answer than the purist (providedn ≥ 2, m ≥ 3).Moreover, the probability that the triangulator picks the correct answer isincreasing in m, and approaches 1 as m→∞.

Note: Reflects Du Bois, but will not convince the purist

Heesen, Bright, and Zucker Vindicating Methodological Triangulation LSE, 30 January 2015 8 / 12

Introduction The Model Conclusion References

A First Result

I n answers, one correct

I m methods: probability distribution over answers

Theorem 1 (List and Goodin (2001))

If all methods have the same probability distribution, which is more likelyto pick the correct answer than any given alternative, then the triangulatoris more likely to believe the correct answer than the purist (providedn ≥ 2, m ≥ 3).Moreover, the probability that the triangulator picks the correct answer isincreasing in m, and approaches 1 as m→∞.

Note: Reflects Du Bois, but will not convince the purist

Heesen, Bright, and Zucker Vindicating Methodological Triangulation LSE, 30 January 2015 8 / 12

Introduction The Model Conclusion References

An Extreme Case

I The One True Method: Ethnographic participant observation

I Other methods merely guesswork

Theorem 2If The One True Method always picks the correct answer, and othermethods pick randomly, then the triangulator is more likely to believe thecorrect answer than the (diffident) purist (provided n ≥ 2, m ≥ 3).

Note: Non-diffident purist may do better than the triangulator

Theorem 3If The One True Method is more likely than 1/n to pick the correctanswer, and other methods pick randomly, then the triangulator is morelikely to believe the correct answer than the (diffident) purist (providedn ≥ 2, m ≥ 3).

Heesen, Bright, and Zucker Vindicating Methodological Triangulation LSE, 30 January 2015 9 / 12

Introduction The Model Conclusion References

An Extreme Case

I The One True Method: Ethnographic participant observation

I Other methods merely guesswork

Theorem 2If The One True Method always picks the correct answer, and othermethods pick randomly, then the triangulator is more likely to believe thecorrect answer than the (diffident) purist (provided n ≥ 2, m ≥ 3).

Note: Non-diffident purist may do better than the triangulator

Theorem 3If The One True Method is more likely than 1/n to pick the correctanswer, and other methods pick randomly, then the triangulator is morelikely to believe the correct answer than the (diffident) purist (providedn ≥ 2, m ≥ 3).

Heesen, Bright, and Zucker Vindicating Methodological Triangulation LSE, 30 January 2015 9 / 12

Introduction The Model Conclusion References

An Extreme Case

I The One True Method: Ethnographic participant observation

I Other methods merely guesswork

Theorem 2If The One True Method always picks the correct answer, and othermethods pick randomly, then the triangulator is more likely to believe thecorrect answer than the (diffident) purist (provided n ≥ 2, m ≥ 3).

Note: Non-diffident purist may do better than the triangulator

Theorem 3If The One True Method is more likely than 1/n to pick the correctanswer, and other methods pick randomly, then the triangulator is morelikely to believe the correct answer than the (diffident) purist (providedn ≥ 2, m ≥ 3).

Heesen, Bright, and Zucker Vindicating Methodological Triangulation LSE, 30 January 2015 9 / 12

Introduction The Model Conclusion References

Summary

I Voting theory framework shows benefits of triangulationI From an optimist perspectiveI From a pessimist perspective

Heesen, Bright, and Zucker Vindicating Methodological Triangulation LSE, 30 January 2015 10 / 12

Introduction The Model Conclusion References

Future Research

I Correlated errorI Generalizing the resultsI Individuating methods

I Social epistemology (Mayo-Wilson et al. 2011)

Heesen, Bright, and Zucker Vindicating Methodological Triangulation LSE, 30 January 2015 11 / 12

Introduction The Model Conclusion References

Thank You!

Thank you for your attention!

Questions?

Heesen, Bright, and Zucker Vindicating Methodological Triangulation LSE, 30 January 2015 12 / 12

Introduction The Model Conclusion References

References I

W. E. B. Du Bois. The Philadelphia Negro: A Social Study. University ofPennsylvania Press, Philadelphia, 1996 [1899].

Branden Fitelson. A Bayesian account of independent evidence withapplications. Philosophy of Science, 68(3):S123–S140, 2001. ISSN00318248. URL http://www.jstor.org/stable/3080940.

Udo Kelle. Sociological explanations between micro and macro and theintegration of qualitative and quantitative methods. Historical SocialResearch / Historische Sozialforschung, 30(1 (111)):95–117, 2005. ISSN01726404. URL http://www.jstor.org/stable/20762014.

Christian List and Robert E. Goodin. Epistemic democracy: Generalizingthe Condorcet Jury Theorem. Journal of Political Philosophy, 9(3):277–306, 2001. ISSN 1467-9760. doi: 10.1111/1467-9760.00128. URLhttp://dx.doi.org/10.1111/1467-9760.00128.

Heesen, Bright, and Zucker Vindicating Methodological Triangulation LSE, 30 January 2015 13 / 12

Introduction The Model Conclusion References

References II

Conor Mayo-Wilson, Kevin J. S. Zollman, and David Danks. Theindependence thesis: When individual and social epistemology diverge.Philosophy of Science, 78(4):653–677, 2011. ISSN 00318248. URLhttp://www.jstor.org/stable/10.1086/661777.

Richard Tewksbury. Qualitative versus quantitative methods:Understanding why qualitative methods are superior for criminology andcriminal justice. Journal of Theoretical and Philosophical Criminology, 1(1):38–58, 2009. ISSN 21668094. URLhttp://search.ebscohost.com/login.aspx?direct=true&db=

i3h&AN=57233244&site=ehost-live.

Heesen, Bright, and Zucker Vindicating Methodological Triangulation LSE, 30 January 2015 14 / 12