The Ethics of Accountability in Education Assessment: Ofqual ethics symposium

EUROPEAN STUDIES CENTREST. ANTONY’S COLLEGE, OXFORD

The Ethics of Accountability in Education Assessment

Professor Paola MatteiAssociate Professor in Comparative Social PolicyUniversity of [email protected]

Symposium: Teacher Ethics in AssessmentOxford University Centre for Educational Assessment (OUCEA) & Ofqual

26 March, 2015St Anne’s College, University of Oxford

The Wire—Louie and Jane

“ I don’t want to go to school…It’s not good…the teachers don’t know

anything, they’re mean and tired…like why is there even an

America?” (Jane)

Oxford 26 March, 2015

2

Public Accountability: why bother?

Why is accountability important?

Public Accountability and democratic theory (Waldron, 2014)

Accountability essential for effective public-private partnership

and for market-based collaboration and new tools of

government (John, 2011; Ranson, 2003; Christensen and

Laegreid, 2007)

Central elements of accountability (Mulgan, 2014; Finer, 1941;

Mattei, 2012)

Delegation of authority from accountor to accountee (agency)

Public process (transparency)

Consequences (responsiveness)


3

Accountability as institutional mechanism

Five formally structured relationships that influence public

organization functions and performance

Political

Principal-agent delegation to elected officials

Administrative/managerial

Ministerial accountability (pre-NPM); financial auditing; top political executive

controls; accountability for performance and results (“managerial

accountability”)

Professional

Codes, standards, norms

Legal

Courts

Social

Clients, customers, interest groups through media, public panels


4

Multiple and competing accountabilities

M


5

Types of accountability relationships

Political Managerial Professional

Direction Clear democratic

accountability lines from

electorate to elected

politicians

Accountability to

owners/shareholders (private)

or autonomous agencies if

public.

Accountability primarily to

professional forums and peers

Logic Emphasis on broader public

good/interest

Emphasis on “value for

money”

Emphasis on medical/

educational evidence

Focus Process dimensions

(openness, involvement, due

process etc.) and politically

determined goals

Output dimensions: bottom

line, business strategy; market

based coordination

Clinical output/outcome

Source: Mattei et al, 2013; Mattei, 2012

Measured Performance Indicators:

New Accountability in Government

A new approach to public services governance in the 2000s: targets and measured performance indicators linked to negative feedback/rewards (Osborne and Gaebler, 1992; Chubb and Moe, 1990)

Was it a decisive breakthrough in governance – or a partial repeat of the some of the history of the Soviet Union (Bevan and Hood, 2006)?

Key question: how far is the world of cheating and output distortions “unethical”?


6

Governance by league tables and targets:

a necessary evil?

Targets threshold standards that a person, organization, country is expected

to reach at a specific time

They have powerful incentive effects in organizations

They help organizations to focus on performance deficits

League TablesUse of indicators to compare the performance of different

organizations

They attract media attention

Can encourage good performers to continue

Should they be scrapped all together (as in Scotland and Wales in 2001-2002)?


7

Performance management systems: does the

removal of league tables matter?

1979-1997 Quasi-

markets

Inspection League

Tables

Targets

England V V V X

Wales V V V X

Scotland X X X X

1997-2009 Quasi-

markets

Inspection League

Tables

Targets

England V V V V

Wales V V X( abolished

in 2001)

V

Scotland X V X (in 2002) V

England, Wales and Scotland: Pupils with 5+ A* to C

GCSEs and SCQF at Level 5 (Mattei, 2012)

20%

25%

30%

35%

40%

45%

50%

55%

60%

65%

1993 1995 1997 1999 2001 2003 2005 2007

Wales England Scotland

Assessment as an accountability tool

1960s: “secret garden of the curriculum” (David Eccles,

Conservative Secretary for Education, cited in Timmins, 1995)

Late 1980s onwards: accountability through measured

assessment and performance indicators (Baird, 2014)

2000s: output targets (based on exam results) and new

performance management systems (Lawn, 2014; Mattei, 2012) to

address underperforming schools


10

Performativity accountability regimes and controversies

Pro-reform claims External public scrutiny of the teaching

profession through measurable outcomes

Information available to parents to bring

the sanction of ‘exit’ (market

accountability)

Learning and professional self corrective

measures

Minimize teacher assessments and

stereotyping against ethnic minorities

(Burgess and Greaves, 2009)

Anti-reform claims Narrowing of the curriculum

Teaching to the tests and negative

curriculum reallocation

Gaming and cheating by teachers

Cream skimming (entry selection)

Schools give up on low performing

students and focus on those on the

margins

Need for broader indicators (child

wellbeing)


11

Some problems with targets and PIs

Threshold effect incentive to concentrate on meeting the minimum target

e.g. teachers concentrate on narrow band of students on the margins to achieve targets

Output distortion incentive to those subject by targets to concentrate on

achieving success at the expense of other factors which are not measured by the target

e.g. teaching to the test (at the expense of sports, arts)


12

Output distortions and cultural conditions

Role of culture (Mary Douglas, 1966; Christopher Hood, 2000; Dan Kahan, 2006) in policy making and administrative process.

Design performance indicators sensitive to organisational culture. Why?

How individuals and groups respond to measures depends on four

types of control systems/cultures

Strength of distortions

Ofqual survey questionnaire on acceptability of cheating behaviour is a measure of “culture”


13

Grid-Group cultural theory and four control systems


14

Fatalism

Low group cohesion,

apathy, sense of disorder

and distrust

Hierarchy

Oversight through within a

hierarchy characterised by

strong regulation and rule-

bound institutions

Individualism

Privileges markets,

unbridled

entrepreneurialism

competition, and

deregulation

Egalitarianism

Mutuality, solidarity,

communal governance,

participative decision-

making

GRID

GROUP

Types of actors/motivation (Hood 2007)

1)‘Saints’: who may not share mainstream goals, but whose public

service ethos is so high that they voluntarily disclose shortcomings to

central authorities

2) ‘Honest triers’: who broadly share mainstream goals and do not

voluntarily draw attention to their failures, but do not attempt to spin or

fiddle data in their favour

3)‘Reactive gamers’: who broadly share mainstream goals, but aim to

spin or fiddle data if they have a motive or opportunity to do so.

4)‘Rational maniacs’: who do not share mainstream goals and aim to

manipulate data to conceal their operations (gross misconduct)


15

Illustration from another field: gaming the transplant system in the United States

For a patient in need of an organ transplant, life is a waiting game!

Low Group Low Grid—Individualist control system and cultural conditions

Majority of hospitals: private non-profits

78% of all medical procedures performed on a fee-for-service basis (incentives

volume targets and rewards)

Most remunerated procedure (transplant), highest DRGs for hospitals

High transplant prices – average liver transplant now more than $577,000 and

bonuses for surgeons

Policy challenge: do market pressures combined with high prices

transplant surgeons command, and financial incentives and bonus

rewards encourage unethical cheating behaviour?


16

Cases of cheating by surgeons: control

systems or motivation?

2002: University of Illinois teaching hospital

Federal lawsuits against three centres in Chicago ($2mil to settle the lawsuit)

Misleading medical information and inappropriate hospitalisation

Admission to Intensive Care Unit to jump the transplant waiting list in the

region (Department of Justice, 2003)

MELD introduced in 2002 (new medical standards)

sharp decrease in admissions to ICUs (Snyder, 2010)

Largest decrease in ICU admissions in markets with highest provider

competition (controlling for surgeons’ rewards system)

Findings: cheating associated with type of target (volume

expansion, 12 procedures per year) and type of reward (bonus)


17

Public trust (O’Neill, 2011) and morality in public policy

Public policy discourse of efficiency/accountability is now

increasingly tied to morality and trust (Simpson and Baird, 2013)

Malpractices in public services may diminish public trust

In live organ transplant (scarce supply), this is detrimental-- drop in

organ donation has severe implications for patients

Strengthening “defensive medicine” and “defensive teaching”

Erosion of professional ethos and demoralised professions


18

Future research and policy challenges:

“Ethics and Gaming in Education”

Is cheating in the education system caused by regulatory failure or individual human action?

What is the relationship between cultural conditions (Grid-Group theory) and educational assessment?

Do some control systems deliver a better game-proof design?

What are the potential effects of changing administrative values and cultures (e.g. increase relational distance between teachers and regulators) on the strength of output distortions?


19

Why ethics and accountability?

Move beyond institutional mechanisms and formal arrangements in the field of accountability

Understanding accountability through human relationships and social

interactions (Bovens, 2002; Dubnick, 2006)

Multiple diverse conflicting expectations (MDCE) from policy makers who face dilemmas and make choices among opposing values (Dubnick and Romzek, 1993; Mattei, 2015)

Accountability is intrinsically an ethical question.

Prestige management increasingly significant with PIs governance

Achieve moral status in the eyes of others (not only about bonuses)

Avoid moral blame that might result from wrongdoing


20

Nozick’s ethical theory and public policy:

moral pushes and pulls

Action as outcome of tensions between moral push and moral pull

Moral push—my own values

internal motivation, individual, that determines own moral conduct

based on self-worth

Internalisation of values

Moral pull—the others’ values

external, institutional values and demands, conduct based on the

others’ values (structures and procedures)

Moral pull of ‘A’ puts a moral constraint on ‘B’ and determines the

behaviour of ‘B’ in accordance to ‘A’ values

When does ethical action occur?

Moral push is equal or greater than the moral pull


21

Conclusion


22

If

production

and

economic

values

Then, conflict

between compliance and fidelity

The Ethics of Accountability in Education Assessment: Ofqual ethics symposium

Education

public accountability

ethics of accountability

accountability essential

accountability important

public panels oxford

public services governance

public organization

governmenta new approach