Primary Processes and the Pursuit of Quality: Past and Present Perspectives

www.nifustep.no Page 1

Primary processes and the pursuit of quality: Past and present perspectives

Bjørn Stensaker

www.nifustep.no Page 2

Past perspectives and debates….

Evaluation as a phenomenon in general emerging after WWII

a) Overarching belief: ”Social engineering”, positivism, quantification

b) Two forms of evaluation schemes:

- PPBS (future oriented, policy-relevant techniques: cost-benefit analysis, system analysis, etc)

- Social programme evaluation (the controllable experiment)

www.nifustep.no Page 3

Past perspectives and debates cont.

Critique of positivism (is man really rational? Is generalisation possible when the context change?

Change of perspective:

- from a belief that those in charge knows best..(”outside perspective”)

- to a belief grounded in enlighenment of those implementing reforms/practitioners (interpretative, hermeneutic perspectives -> extreme case: social constructivism) (”inside perspective”)

- from that time, one can identify a continuing debate between proponents of the two perspectives

www.nifustep.no Page 4

Current perspectives and debates

Central issue in the debate: how to deal with values? (what criteria should be used as ”benchmark”?)

- validisation: is it true?

- valorisation: is it good?

Dominant evaluation perspectives in use:

- Evaluation ”in context”

- From summative to formative evaluation

- Re-instate rationality (evidence-based approach)

- Pluralist approach (using competing theories and perspectives)

www.nifustep.no Page 5

Evaluation in Higher Education today

Evaluation schemes a reflection of a changing policy context:

- downsizing and transforming welfare states (relevance, ”the evaluative state”)

- resource problems (growth in number of HE-students) (efficiency)

- accountability issues (effectiveness)

Quality = relevance, efficiency and effectiveness put together?

www.nifustep.no Page 6

Evaluation: a key to understand the primary processes of HE

Evaluation used to:

- determine reputation (rankings..)

- resource allocation

- appraisal of knowledge (peer review)

- certification of students

- legitimation of academics

- ranking of students, academics and institutions

- promotion to academic posts

- etc, etc….

www.nifustep.no Page 7

The different underpinnings of peer review

Is peer review:

- based on general consensus of values and understandings (within the discipline)

- just a process to enforce the autonomy, creativity and integrity of the individual academic?

- a conservative process emphasising values of the past, unable to recognise and appriciate innovation?

- or, a power struggle between competitive groups within certain disciplines or fields

www.nifustep.no Page 8

The development of peer review:

”extended peer review”

- new groups of academics involved (need for an ”outside view” of new academic spesialities/tribes and territories)

- people from business, industry or society more involved

- students are given a voice as members of evaluation panels

- ”the ordinary man” included in evaluations where ”big issues” are dealt with (technology assessment, etc)

More focus on legitimacy of the process? Evaluation as a political process

www.nifustep.no Page 9

From Method (evaluation) to Content (quality)

How to evaluate what academics do? (directly, indirectly, quantitative, qualitative)

What form of QA is most appropriate for HE? (control, enhancement, finding a balance?)

How can academic standards be secured and assessed? (examinations, relevance)

How can the outcome of HE be assessed? (employment, relevance, ”happiness”, democracy)

What are the links between QA and the improvement of teaching and learning? (is there a causal model, poor methodology)

www.nifustep.no Page 10

Studies of quality focuses on four central themes:

Course evaluation

Grading and outcomes

National monitoring practices

System standards

www.nifustep.no Page 11

Course evaluations

Are current course evaluation techniques effective?

- ”happy sheets” (problem with response rates)

- ”student engagement questionnaires”

What is measured in student assessment of teaching?

Shevlin et al (2000): is there a ”halo effect” in student evaluation of their teachers? Charismatic teachers receive better assessments…(but, can this research be trusted, or is the result dependent on the design of the study..?)

Wiers-Jenssen et al (2002): what trigger student satisfaction? Academic quality..(but also institutional size, buildings, etc..)

www.nifustep.no Page 12

Grading and outcomes

How do/should lectures assess their students?

- coursework receive higher grading than formal examinations

- differences between academic fields

- argument for external examiner schemes (Norway, Denmark and the UK)

- cultural differences between countries (Bologna)

- In general, grading not high on the institutional/public policy agenda (how are teachers trained in grading procedures…?)

www.nifustep.no Page 13

National monitoring practices (NMP)

Research and teaching monitored independently, and with very different consequences

What is the effect of NMP?

- shifts in power distribution

- bureaucratisation

- permeability (exposure of HE)

- public relations (active use of rankings to establish a positive image externally)

Very few studies show a link between NMP and improvement in teaching and learning

www.nifustep.no Page 14

System standards

What is the role of evaluation in HE?

- an instrument to instigate change?

- an instrument to monitor change?

Why this focus on ´quality´?

- how should ”good” QA-systems look like?

- what is the role of various actors in such systems?

- how can current QA-systems be transformed?

Conclusion: quality is an integrated part of the governance of the HE-sector