Evidence-based education as paradox: a critique

Evidence-based education as paradox: A critique

By Paul Prinsloo

Annual conference of the Higher Education Learning and Teaching Association of South Africa (HELTASA), 18-21 November, Bloemfontein,

South Africa

I do not own the copyright of any of the images in this presentation. I hereby acknowledge the original copyright and licensing regime of every image and source I’ve used. Images used in this presentation have been sourced from Google labeled for non-commercial reuse, or from Flickr published under a CC license. Where no ownership or license could be established, I indicate the hyperlink address.

This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License

“In an age of advanced technology, inefficiency is the sin against the Holy Spirit” (Morozov, 2013b, quoting Aldous Huxley, p. ix).

“ The strong desire for proof burns bright in education” (Wagner & Ice, 2011, p. 36)

“…we optimise the measurable at the risk of neglecting the immeasurable” (Richardson, 2012)

Purpose of this presentation

My purpose is not to debunk evidence-based education as such but to assemble a number of voices and histories in a reflective caring but critical space (e.g. Latour, 2004). No one disagrees that evidence is important.

… a first and necessary step in counteracting the force of any discourse is to recognise its constitutive power, its capacity to become hegemonic, ‘to saturate our very consciousness, so that the … world we see and interact with, and the commonsense interpretations we put on it, become the world tout court, the only world’ (Apple, 1979, p. 5)”

(Davies, 2003, p. 102)

http://wrexhamfan.wordpress.com/2014/03/23/the-small-elephant-in-the-room/

The elephant in the room… (Denzin, 2009)

The signal and the noise

“Most of it is just noise, and the noise is increasing faster than the signal”

(Silver, 2012, p. 13)

The “problem with predicting the future is rarely the predictions themselves, but rather the base assumptions that make it the logical progression”

(Tweet: InfoSec Taylor Swift, 2014)

Disruption

Situating evidence-based education (1) – the higher education landscape: looking for a center that holds

Innovation

Unbundling and unmooring

Revolution

Crisis

Increasing

casualisation

of facultyPrivatisation of higher education

Doing more with lessAccountability

Rankings

Quantification fetish

Technosolutionism

The data deluge

We need evidence of

what works…

Disaggregation

Situating evidence-based education (2) – the higher education landscape: data

Learning analytics as the “new black” (Booth, 2011)

“…the claims about big data and education are incredibly bold, and as yet, unproven” (Watters, 2013, par. 17)

Student data as “the new oil” (Watters, 2013)

Learning analytics as “the new black” (Booth, 2011)

“Most of it [the data] is just noise, and the noise is increasing faster than the signal” (Silver, 2012, p. 13)

Image credits: https://flic.kr/p/dSHr87

Hartley (1995) - McDonaldisation of higher education• Impact of external scrutiny,

inspection, increasing emphasis on efficiency, calculability, predictability, and control

• Doing more with less• Funding following performance

rather than preceding it

Commentators “claim that we [higher education] cost too much, spend carelessly, teach poorly, plan myopically and when questioned, act defensively (Lagowski 1995, p. 861)

http://commons.wikimedia.org/wiki/File:Mcdonalds-90s-logo.svg

Overview of the presentation

1. Problematisingevidence/data/accountability/quantification

2. Roots of evidence-based education• Colonialist roots• Neoliberalism and managerialism• The ‘gold standard’ of evidence in medicine• Concerns about educational research

3. Problematising evidence-based education• Epistemological concerns• Ontological concerns• Evidence and power

4. Towards value-based education

Problematising evidence evidence/data/accountability/quantification

Data cannot and do not speak for itself (Boyd & Crawford, 2013; Gitelman, 2013)

“…data are political in nature – loaded with values, interests and assumptions that shape and limit what is done with it and by whom” (Selwyn, 2014, p. 6)

http://en.wikipedia.org/wiki/British_Empire

Roots 1: Colonialist roots (Shahajan, 2011)

http://en.wikipedia.org/wiki/Slave_breeding_in_the_United_States

From a “masculinist epistemology of science” and epistemologies of control it is necessary to “systematise everything, reducing them to manageable questions and subjects, and then find some causal links between them” (Shahjahan, 2011, p. 188)

“Students are reduced to test scores, future slots in the labor market, prison numbers, and possible cannon fodder in military conquests. Teachers are reduced to technicians and supervisors in the education assembly line – ‘objects’ rather than ‘subjects’ of history. This system is fundamentally about the negation of human agency, despite the good intentions of individuals at all levels” (Lipman, 2004, as quoted by Shahjahan, 201, p. 196).

Evidence-based education therefore becomes synonymous with the “thingification” of education (Shahjahan, 2011, p. 197).

Roots 2: Neoliberalism & managerialism

Neoliberalism and managerialism heralds the most significant shift in “the discursive construction of professional practice that any of us will ever experience”

(Davies, 2003, p. 91) https://openclipart.org/detail/170056/they-are-watching-you-by-asrafil

“ … as long as the objectives have been specified and strategies for their management and surveillance put in place, the nature of the work itself is of little relevance to anyone.”

“If the auditing tools say that the work has, on average, met the objectives, it is simply assumed that the work has been appropriately and satisfactorily tailored according to the requirements of the institution (and often of the relevant funding body).”

(Davies, 2003, p. 92; emphasis added)

Those who resist “the terms of auditors and economists” are identified for re-education (Davies, 2003, p. 93).

htt

ps:

//w

ww

.flic

kr.c

om

/ph

oto

s/1

34

76

48

0@

N0

7/8

68

00

70

93

8

Roots 3: The ‘gold standard’ of evidence in medicine

We need to question the “homology between education and medicine” and Biesta points to “the different meanings of evidence in these fields” (Biesta, 2007, p. 4).

Students are not patients or ill and education is not a cure (Biesta, 2007, 2010)

(Also see Simons, 2003)

Roots 4: Concerns about educational research

Current educational research and professional practices don’t provide the answers government is looking for (Biesta, 2007, 2010)

Experimental research is seen as “the only method capable of providing secure evidence about ‘what works’ (Biesta, 2007, p. 3; emphasis added)

Problematising evidence-based education

1. Epistemological concerns2. Ontological concerns3. Concerns re the relationship between evidence and

power

Epistemological concerns

“… research knowledge is always fallible, even if it is more likely to be valid than knowledge from other sources” (Hammersley, 2001, par. 7).

htt

ps:

//fl

ic.k

r/p

/49

32

J1

Our increasing assumptions about and reliance on algorithms resemble a possible “gnoseological turning point” in our understanding of knowledge, information and faculties of learning where bureaucracies increasingly aspire to transform and reduce “ontological entities, individuals, to standardised ones through formal classification” into algorithms and calculable processes (Totaro & Ninno, 2014, p. 29).

Experimentation and interventions are not disentangled but embedded and part of the system

under investigation(Barad, 2007; Biesta, 2010, ).

Ontological concerns

Education is not a closed system or a “causal technology” but an “open and recursive system” (Biesta, 2007, p. 8)

Research and evidence “can tell us what worked but cannot tell us what works (Biesta, 2007, p. 16).

In an open and recursive system it is highly improbable that the same solution will work again…

http://commons.wikimedia.org/wiki/File:Cynefin_framework_Feb_2011.jpeg

The Cynefinframework (Snowden & Boone, 2007)

https://flic.kr/p/6hv3zZ

Data, vidence and power

Data, vidence and power

htt

p:/

/ww

w.w

ebso

ph

ist.

com

/Pro

filin

g_TS

A_T

oo

n.jp

g

“… data are political in nature – loaded with values, interests and assumptions that shape and limit what is done with it and by whom” (Selwyn, 2014; p. 6).

As data sets are increasingly combined and reused, it is important to acknowledge that “data itself can take on its own life” where the original context and intention of harvesting, as well as the original assumptions informing the parameters are lost (Selwyn, 2014, p. 7).

Towards a value-based education

We accept that data and evidence are• Anything but neutral or speak for itself (Gitelman,

2003)• Embedded in historical and present socio-

economic, ideological and geopolitical power relations (Apple, 2010; Castells, 2009; Henman, 2001; Selwyn, 2014)

• Not the whole/real picture(Mayer-Schönberger, 2009)

• Precariously temporary, fragile and mutable (Fenwick & Edwards, 2014)

Noddings (1999) moots the interesting point that “when a just decision has been reached,

there is still much ethical work to be done” (p. 16).

So while it is important to ask whether an intervention is effective, it is even more

important to ask whether it is appropriate, caring and interrupt cycles of inequity and

injustice

Thank you. Baie dankie. Ke a leboga

Paul PrinslooResearch Professor in Open Distance Learning

3-15, Club 1P O Box 392

Unisa0003

[email protected]://opendistanceteachingandlearning.wordpress.com

Twitter: 14prinsp

Office: +27 12 433 4719

References and reading listAltbach, P.G. (1999). The logic of mass higher education. Tertiary Education and

Management, 5(2), 107—124. Altbach, P.G., Reisberg, L., & Rumbley, L.E. (2009). Trends in global higher education:

tracking an academic revolution. A report prepared for the UNESCO 2009 World Conference on Higher Education. Retrieved from http://graduateinstitute.ch/files/live/sites/iheid/files/sites/developpement/shared/developpement/cours/E759/Altbach,%20Reisberg,%20Rumbley%20Tracking%20an%20Academic%20Revolution,%20UNESCO%202009.pdf

Andrejevic, M. (2014). The Big Data divide. International Journal of Communication,8, 1673—1689 .

Apple, M.W. (2010). Global crises, social justice, and education. New York, NY: Routledge.

Barad, K. (2007. Meeting the universe halfway. Quantum physics and the entanglement of matter and meaning. London, UK: Duke University Press.

Beer, C., Jones, D., & Clark, D. (2012). Analytics and complexity: Learning and leading for the future. In Proceedings of the 29th Annual Conference of the Australasian Society for Computers in Learning in Tertiary Education (ASCILITE 2012) (pp. 78-87). Australasian Society for Computers in Learning in Tertiary Education (ASCILITE).

Berliner, D.C. (2002). Comment: Educational research: The hardest science of all. Educational Researcher, 31, 18—20. DOI: 10.3102/0013189X031008018.

Biesta, G. (2007). Why “what works” won’t work: evidence-based practice and the democratic deficit in educational research. Educational Theory, 57(1), 1–22. DOI: 10.1111/j.1741-5446.2006.00241.x.

Biesta, G. (2010). Why ‘what works’ still won’t work: from evidence-based education to value-based education. Studies in Philosophy of Education, 29, 491–503. DOI 10.1007/s11217-010-9191-x.

Boyd, D., & Crawford, K. (2013). Six provocations for Big Data. Retrieved from http://papers.ssrn.com/sol3/papers.cfm?abstract_id=1926431

Carr, N. (2012). The crisis in higher education. Technology Review. Retrieved from http://www.technologyreview.com/featuredstory/429376/the-crisis-in-higher-education/

Castells, M. (2009). Communication power. Oxford, UK: Oxford University Press. Christensen, C. (2008). Disruptive innovation and catalytic change in higher education.

Forum for the Future of Higher Education. EDUCAUSE,[Online].Retrieved from https://net.educause.edu/ir/library/pdf/ff0810s.pdf

Clegg, S. (2005). Evidence-based practice in educational research: a critical realist critique of systematic review. British Journal of Sociology of Education, 26(3): 415-428. DOI: 10.1080/01425690500128932

Clow, D. (2013). An overview of learning analytics. Teaching in Higher Education, 18(6), 683—695. DOI: 10.1080/13562517.2013.827653

Davies, P. (1999). What is evidence-based education? British Journal of Educational Studies, 47(2): 108-121.

Davies, B. (2003). Death to critique and dissent? The policies and practices of new managerialism and of 'Evidence-based Practice', Gender and Education, 15(1), 91-103, DOI: 10.1080/0954025032000042167

Elliot, J. (2001). Making evidence-based practice educational. British Educational Research Journal, 27(5): 555-574. DOI: 10.1080/01411920120095735

Eynon, R. (2013). The rise of Big Data: what does it mean for education, technology, and media research? Learning, Media and Technology, DOI: 10.1080/17439884.2013.771783

Fenwick, T., & Edwards, R. (2010). Actor-network theory in education. London, UK: Routledge.

Fenwick, T., & Edwards, R. (2014). Networks of knowledge, matters of learning, and criticality in higher education. Higher Education, 67: 35-50. DOI: 10.1007/s10734-013-9639-3

Francis, B. (2006) Heroes or zeroes? The discursive positioning of ‘underachieving boys’ in English neo‐liberal education policy, Journal of Education Policy, 21(2),187-200, DOI: 10.1080/02680930500500278

Gitelman, L. (Ed.). (2013). “Raw data” is an oxymoron. London, UK: MIT Press.Hammersley, M. (2001). Some questions about evidence-based practice in education.

Retrieved from http://www.leeds.ac.uk/educol/documents/00001819.htmHargreaves, A., & Braun, H. (2013). Data-driven improvement and accountability. National

Education Policy Centre. Retrieved from http://co.chalkbeat.org/wp-content/uploads/sites/2/2013/10/PB-LB-DDIA-POLICY-FINAL-EMBARGOED.pdf

Hartley, D. (1995). The ‘McDonaldisation’ of higher education: food for thought? Oxford Review of Education, 21(4): 409—423.

Henman, P. (2004). Targeted!: Population segmentation, electronic surveillance and governing the unemployed in Australia. International Sociology, 19, 173-191. DOI: 10.1177/0268580904042899

InfoSec Taylor Swift. [@SwiftOnSecurity] (2014, 8 November). Problem with predicting the future is rarely predictions themselves, but rather the base assumptions that make it the logical progression. [Tweet]. Retrieved from https://twitter.com/SwiftOnSecurity/status/530831217227497472

Kitchen, R. (2014). The data revolution. New York, NY: SAGE. Knight, S., & Buckingham Shum, S., & Littleton, K. (2014). Epistemology, assessment,

pedagogy: where learning meets analytics in the middle space. Journal of Learning Analytics. (In press). Retrieved from http://oro.open.ac.uk/39226/

Lagowski, J.J. (1995). Higher education: a time for triage? Journal of Chemical Education, 72(10): 861.

Lather, P. (2004). This is your father's paradigm: government intrusion and the case of qualitative research in education, Qualitative Inquiry 10: 15-34. DOI: 10.1177/1077800403256154

Latour, B. (2004). Why has critique run out of steam? From matters of fact to matters of concern. Critical Inquiry, 30: 225-248.

Lockyer, L., & Dawson, S. (2011). Learning designs and learning analytics. LAK '11 Proceedings of the 1st International Conference on Learning Analytics and Knowledge, pp. 153—156. Retrieved from http://dl.acm.org/citation.cfm?id=2090140.

Mayer-Schönberger, V. (2009). Delete. The virtue of forgetting in the digital age.Princeton, NJ: Princeton University Press.

Mayer-Schönberger, V., & Cukier, K. (2013). Big data. A revolution that will transform how we live, work, and think. New York, N.Y.: Houghton Miffling Harcourt Publishing CompanyMorozov, E. (2013). To save everything, click here. London, UK: Penguin Books.

Nutley, S., Walter, I., & Davies, H. T. (2003). From knowing to doing a framework for understanding the evidence-into-practice agenda. Evaluation, 9(2): 125-148.

Oliver, M., & Conole, G. (2010). Evidence-based practice and e-learning in higher education: can we and should we? Research Papers in Education, 18(4): 385-397. DOI: 10.1080/0267152032000176873

Olssen, M., & Peters, M.A. (2005) Neoliberalism, higher education and the knowledge economy: from the free market to knowledge capitalism, Journal of Education Policy, 20(3): 313-345, DOI: 10.1080/02680930500108718

Pirrie, A. (2001). Evidence-based practice in education: The best medicine? British Journal of Educational Studies, 49(2), 124—136. DOI: 10.1111/1467-8527.t01-1-00167.

Prinsloo, P. (2014). A brave new world: student surveillance in higher education. Paper presented at the 21st Southern African Association for Institutional Research (SAAIR), Pretoria, 16-18 September. Retrieved from http://www.slideshare.net/prinsp/a-brave-new-world-student-surveillance-in-higher-education.

Prinsloo, P., & Slade, S. (2014). Educational triage in higher online education: walking a moral tightrope. International Review of Research in Open Distance Learning (IRRODL), 14(4): 306-331. http://www.irrodl.org/index.php/irrodl/article/view/1881

Reeves, T. (2011). Can educational research be both rigorous and relevant? Educational Designer. Journal of the International Society for Design and Development in Education. Retrieved from http://www.educationaldesigner.org/ed/volume1/issue4/article13/index.htm

Richardson, W. (2012, July 14th). Valuing the immeasurable. [Web log post]. Retrieved from http://willrichardson.com/post/27223512371/valuing-the-immeasurable

Schildkamp, K., Lai, M.K., & Earl, L. (eds). (2013). Data-based decision making in education. Challenges and opportunities. London, UK: Springer.

Selwyn, N. (2014). Data entry: towards the critical study of digital data and education. Learning, Media and Technology, 1-19. DOI: 10.1080/17439884.2014.921628

Simons, H. (2003). Evidence-based practice: panacea or over promise? Research Papers in Education, 18(4): 303-311.

Silver, N. (2012). The signal and the noise. The art and science of prediction. London, UK: Penguin.

Shahjahan, R.A. (2011). Decolonizing the evidence‐based education and policy movement: revealing the colonial vestiges in educational policy, research, and neoliberal reform, Journal of Education Policy, 26(2): 181-206, DOI: 10.1080/02680939.2010.508176

Siemens, G. (2011). Learning analytics: a foundation for informed change in higher education. Retrieved from http://www.educause.edu/library/resources/learning-analytics-foundation-informed-change-higher-education

Simons, H. (2003). Evidence-based practice: panacea or over promise? Research Papers in Education, 18(4): 303-311. DOI: 10.1080/0267152032000176828

Snowden, D.J., & Boone, M.E. (2007). A leader’s framework for decision making. Harvard Business Review, November: 1-9.

Solove, D.J. (2001) Privacy and power: Computer databases and metaphors for information privacy. Stanford Law Review, 53(6), 1393—1462.

Swain, H. (2013, August 5). Are universities collecting too much information on staff and students? [Web log post]. The Guardian. Retrieved from http://www.theguardian.com/education/2013/aug/05/electronic-data-trail-huddersfield-loughborough-university

Sykes, G., Schneider, B., & Plank, D.N. (2009). Handbook of education policy research.New York, NY: Routledge.

Totaro, P., & Ninno, D. (2014). The concept of algorithm as an interpretive key of modern rationality. Theory Culture Society 31, pp. 29—49. DOI: 10.1177/0263276413510051

Wagner, E., & Ice, P. (2012, July 18). Data changes everything: delivering on the promise of learning analytics in higher education. EDUCAUSEreview, [online]. Retrieved from http://www.educause.edu/ero/article/data-changes-everything-delivering-promise-learning-analytics-higher-education

Waitere , H.J., Wright, J., Tremaine, M., Brown, S., & Pausé, C.J. (2011). Choosing whether to resist or reinforce the new managerialism: the impact of performance‐based research funding on academic identity, Higher Education Research & Development, 30(2): 205-217, DOI: 10.1080/07294360.2010.509760

Watters, A. (2012). Unbundling and unmooring: technology and the higher ed tsunami. EDUCAUSE review, [online]. Retrieved from http://www.educause.edu/ero/article/unbundling-and-unmooring-technology-and-higher-ed-tsunami

Watters, A. (2013, October 13). Student data is the new oil: MOOCs, metaphor, and money. [Web log post]. Retrieved from http://www.hackeducation.com/2013/10/17/student-data-is-the-new-oil/

Wiley, D., & Hilton III, J. (2009). Openness, dynamic specialization, and the disaggregated future of higher education. International Review of Research in Open and Distance Learning, 10(5), 1—16. Retrieved from http://www.irrodl.org/index.php/irrodl/article/view/768/1414

Wishon, G.D., & Rome, J. (2012, 13 August). Enabling a data-driven university. EDUCAUSEreview, [online]. Retrieved from http://www.educause.edu/ero/article/enabling-data-driven-university