going to cover:

... ‘ Messages for QA agencies ’ ....

an overview of the ‘messages’ from ENQA review reports (2005 to December 2009)

Nick Harris

[and based on a presentation to ENQA GA 20010]

going to cover:

1. the project – an overview (+ a little detail)2. some predictable messages 3. some awkward issues (+ a little detail)4. a brief reminder5. a few conclusions and where now for the ESG, ENQA and its members?

.. and a recurring theme .. who’s doing what? who wants what?

(but should that really be .. who needs what?)

1. The project – an overview

Based on reports for 26 ENQA reviews (2005 to Dec 2009)

Purpose a ‘self evaluation’ by ENQA itself to:

• support an analysis and reflection on the work to date• provide collated (evidence-based) information about the

progress/impact of the Bologna action line:Promotion of European co-operation in Quality

Assurance• inform the second cycle of ENQA reviews• inform any review / revision of the ESG

1. The project – a bit more detail

Method:• co-ordinated by 3 ENQA Board members • thematic sorting and collation of material in the reports

– by a team at QAAHE – UK – involves ‘coding’ and sorting/aggregating of evidence using N4 software

• analyses and comparisons .. – with 2 external experts .. who had each chaired several reviews

• ENQA publication (xxx xxxxx )

• BUT still under consideration – very much work in progress

1. The project – a little bit more detail

Method:analyses collated and compared .. • by each standard .. 3.1 to 3.8

– and for 3.1 (where available detail) analysis of references to ESG Part 2

• standards ‘clustered’ by ‘themes’– ‘legitimacy’ ..

3.2 Official status; 3.5 Mission statement; 3.6 Independence

– ‘activities’ .. 3.1 Use ext QA procedures for HE; 3.3 Activities;

3.7 Ext QA criteria and processes; 3.8 Accountability procedures

– ‘sustainability’ .. 3.4 Resources; 3.8 Accountability procedures

but – too late – realised something was missing !

‘responsibility’ !

perhaps would have been better to have had‘legitimacy’‘responsibility’

‘activities’‘sustainability’

with responsibility looking at / comparing mission statement (what is intended) and accountability (evidence of what has been achieved

.. and who benefitted in what ways ?!)

2. some predictable messages (and successes)

from the ‘headlines’ .. it is clear that• the ESG (parts 2 and 3)

– have had a very significant impact .. – have been a significant success in providing

• support for the Bologna action line on QA• the ESG were drawn from/on sensible principles

– that have led to shared / improving practice and successful agency reviews

BUT ... • all is not ‘perfect’ ..

– there are some (recurring) problems and difficulties

2. some predictable messages (cont)

• most agencies were found to be compliant with the standards .. (approx %with substantial/partial compliance)3.1 use of external QA procedures for HE (30% S/P) **3.2 official status (5% S/P)3.3 activities (5% S/P)3.4 resources (30% S/P) *3.5 mission statement (30% S/P) *3.6 independence (35% S/P) *3.7 external QA criteria and processes used by agencies (35%) *3.8 accountability procedures (30% S/P) *

... BUT ... are panel judgements consistent !!??

2. some predictable messages (cont)

• the ‘understanding’ and use of language in the USG

does not appear to be absolutely consistent ! .. are there consequences ? do they matter ?

But do remember .... • agencies are different ! in terms of their – the background contexts in which they work– their constituencies and responsibilities– size (is important)– ‘maturity’

2. some more predictable messages

• ‘QA agencies ask for too much’ / ‘HEIs provide too much’ but .!• Self Evaluation Reports/ Documents don’t .. ‘self evaluate’

• there are huge inefficiencies/duplications / omissions in what is asked for / provided

is QA focussing on the necessary and the sufficient ?

trust building is essential

but this means recognising some ‘awkward issues’

3. some awkward ‘issues’ .. some specifics

3.1 use of external QA procedures for HE (30% S/P) how can an agency that gets very few ‘Compliants’ in Part 2 still manage to get a ‘Compliant’ overall for 3.1 ?!3.4 resources (30% S/P)

concerns about changing (expanding) missions and roles – at a time of financial uncertainty3.5 mission statement (30% S/P) the ‘context thing’ .. often only implicit and / or not succinct

often aspirations for future rather than the reality of now wording of the Standard – is an MS ok even if it is inaccurate?!

3. some awkward ‘issues’ .. some more specifics

3.6 independence (35% S/P) a ‘BIGGY’ .. and a problematic one at that ! overlap with 3.2 Official status

Analysis included looking at coverage of organisational independence .. heavily context dependent operational independence .. more often OK financial independence .. ‘he who pays the piper’ !?

and ... the ‘conflict of interest’ issues .. (the reports / the grapevine gossip / what’s the reality?)

3. some awkward ‘issues’ .. some more specifics

3.7 external QA criteria and processes used by agencies (35%) *another ‘biggy’ .. and a problematic one at that !

contexts are again important .. including• the legal context• the age / ‘maturity’ thing• structural aspects

on the ‘plus side’ .... agencies recognise the difficultiesbut ... panels vary in their application of ‘mitigating circumstances’

3. some awkward ‘issues’ .. some more specifics

3.8 accountability procedures (30% S/P)

Oh dear ! a classic example of standards written ‘by the body for the body’

the hypothesis seems to be : ‘good practice’ comes from .. better procedures + more data = ‘greater transparency’

but surely what is needed is ... CLEAR(ER) INFORMATION - that is audience specific

3. some awkward ‘issues’ .. some general

It would appear that• there is context dependent interpretation of the ESG

– not necessarily ‘a bad thing’ .. BUT

• panels vary in the extent to which they apply ‘the context argument’– could/should panel judgements consistent !!??

• despite such variation - a secure basis for EQAR (?)

4. a brief reminder

• Bologna Process Berlin Communique (2003): invited states to develop “an agreed set of standards, procedures and guidelines on quality assurance” and explore ways “of ensuring an adequate peer review system for quality assurance and / or accreditation bodies”

• Christian Thune (2005) “it must be emphasised that the report [on the ESG] is no more than a first step in what is likely to be a long and possibly arduous route to the establishment of a widely shared set of underpinning values, expectations and good practice in relation to quality and its assurance”

5. where now for ENQA – and its members?

“ .. a long and possibly arduous route ..” [CT, 2005]

the ESG clearly ‘WORK’ !! .. they will be reviewed - should they be revised?

is there a reasonable level of consistency in ENQA reviews ? – how best to secure and demonstrate that?• training ? or a more radical approach ?

5. where now for ENQA – and its members?should the ESG be REVIEWED ?

YES ! but ..that is NOT the same as revising them !

( accept that they will never be perfect for all )

And ... if they are reviewed .. then with what / whose foci ?

5. where now for ENQA foci for review of the ESG (2 of 2)

• the Bologna Process – certainly .. but only that directly linked to the QA of HE ! NOT the ‘social engineering’ (‘political’) aspects

• the public information agenda – but be clear about providing Information! (not the T word)

• clarifying the bases of standards .. (inputs and / or outputs?)– the Learning Outcomes / Competencies conundrum .. – QA includes external moderation of assessment (oh dear!!)

• the improvement / enhancement agenda .. – but of what ?! the QA of HE or HE itself (they are different)?

5. where now for ENQA foci for review of the ESG .. and their application (2 of 2)

• making the ESG and ENQA review more effective – even if ESG unchanged better emphasis on how they should

be used .. and not used !!

• is there a reasonable level of consistency in ENQA reviews ? – how best to secure and demonstrate that?• training ? or a more radical approach ?

• greater [overt] emphasis on independence – reduce the (perhaps only perceived) conflicts of interest of

peer review

5. where now for ENQA, the ESG and QA agencies ?

the importance of ‘context’ in QA has been a recurring theme the HE systems of Europe have long and rich and similar but distinctly different heritagesbut QA (and ENQA) has moved into the ‘political arena’

is our heritage to be abandoned .. in search of some harmonised, mutually-recognisable transparency that

may not be wanted by (or benefit) the ‘real stakeholders’?!

Globalisation has brought us ... ?

[email protected]

The disclaimer: whether ‘couture’ or just ‘fit to flush’ (down the drain) the opinions expressed in this presentation are mine and mine alone and do not represent the views of the ENQA Board or any of its members The thanks: to the ENQA Board for inviting me to join the project, the QAA (UK) Outcomes team and Professor Thierry Malan .. my co conspirator

Madrid, Tetbury