Rafael Llavori de Micheo Head of the International Relations Unit ANECA Case study 4: Spain ECA Seminar Changing Systems: achievements and new develop,ments.

Rafael Llavori de Micheo

Head of the International Relations Unit

ANECA

Case study 4: Spain

ECA Seminar

Changing Systems: achievements and new develop,ments in national accreditation systems

The Hague, 9th December 2009

Adapting the national system to the EHEA

The Spanish case study

The accreditation system

The institutional perspective: AUDIT

Open questions for the future

ContentsContents

1. Adapting the national Adapting the national system to the EHEAsystem to the EHEA

The Spanish HE system

UniversitiesUniversities

7575 universities universities

(50 public – 25 private)(50 public – 25 private)

Public universities – 1.326.734

Private Universities – 134.743Number of students enrolled 1.461.477

Adapting the HE system to the EHEA

Bologna process and quality assurance developments

1999

Bologna

2001

Prague

2003

Berlin

2005

Bergen

2007

London

European cooperation in quality assurance

Primary responsibility of HE institutions

for quality

European Standards and Guidelines

European RegisterEQAR

Cooperation of QA agencies and HE

institutions

E4 Group

1998 Reco. 98-561-CE 2006 Reco. 2006-143-CE

Source : Colin Tück, E4, B. Curvale

04/2008

2009

Leuven

Louvain-la-Neuve

Evaluation of EQAR



New Spanish Legal FrameworkNew Spanish Legal Framework

European HigherEuropean Higher

Education AreaEducation Area

•University Act (December 2001) and its reform (April 2007).

•Real Decreto 1044/2003, 1 August, European Degree Supplement

•Real Decreto 1125/2003, 5 September, ECTS

•Real Decreto 56/2007, 21 January, Postgraduate Official Programmes

•Real Decreto 1393/2007, 29 October, New Organisation of Programmes

•Real Decreto 1509/2008, 12 de September, Universities, Centres and Degrees Register

•University Act (December 2001) and its reform (April 2007).

•Real Decreto 1044/2003, 1 August, European Degree Supplement

•Real Decreto 1125/2003, 5 September, ECTS

•Real Decreto 56/2007, 21 January, Postgraduate Official Programmes

•Real Decreto 1393/2007, 29 October, New Organisation of Programmes

•Real Decreto 1509/2008, 12 de September, Universities, Centres and Degrees Register

•European Standrads and Guidelines for QA of HE•European Standrads and Guidelines for QA of HE

CHANGES FOR THE UNIVERSITY SYSTEMCHANGES FOR THE UNIVERSITY SYSTEM

Changes in the legal frameworkChanges in the legal framework


2. The Spanish case studyThe Spanish case study

The Spanish HE system: the actors

Ministry of Education

Regional governments

Universities

• National legal framework: LOMLOU

• Programme accreditation

• Eval. and Accreditation of Teaching Staff

• Regional legal framework

• Funding public HEIs

• Follow-up and ex-post accreditation

• Evaluation of teaching staff (hired)

• Programme offer

• Selection of teaching staff

• Selection of students


University Council/Reg. Gov.

Universities

• Evaluation for accreditation (ANECA)

• Follow-up and ex-post accreditation (both)

• Teaching staff accreditation (ANECA) and Evaluation of Teaching staff (hired) (both)

• Institutional Evaluation practices DOCENTIA y AUDIT

• Internal QA systems

• Evaluation of Teaching activity

• Selection of teaching staff

ANECA/Regional Agencies


o The new legal framework assigns

ANECA and the regional agencies a

concrete role to be played

ANECAANECA Autonomous Communities AgenciesAutonomous Communities Agencies

o RD 1393/2007 establishes that ANECA: Prepares the procedures Elaborates the evaluation report for the ex

– ante accreditation Mutual recognition between ANECA and

Spanish ENQA full members / EQAR agencies

• ANECA and the A.C. agencies

• Do the follow-up procedure of the accredited programmes

• Elaborate a previous report for re-accreditation

Regions with regional agencies Regions without regional agencies

QA in Spain at the national level

BEFORE EHEA WITHIN EHEA

FOCUS Contents Competences (LO)

AUTONOMY Less Higher

WHO DESIGN THE PROGRAMMES?

The Ministry University

FRAMEWORK Catalogue Register (Flexible)

DEGREES OFFERED

Limited number Depending on the universities

QUALITY Important Key

EXTERNAL QUALITY

ASSURANCE

Voluntary Mandatory (Ex-ante, Follow-up

and Re-accreditation)


3. The accreditation The accreditation procedureprocedure

Design

Programme design

UNIVERSITY

The programme accreditation processThe programme accreditation process

Accreditation

Fulfilment evaluation

Council Univ.

Follow-up

Implementation

UNIVERSItY

ANECA and Reg. Agencies

Ex – ante Evaluation

Accreditation

Council Univ.

ANECARegional Government

The accreditation procedure

The ex ante programme accreditation focuses on bachelor, master and doctorate level

Apply to all both existing and new programmes.

ScopeScope

The Council of Universities ministerial body, based upon the evaluation report of ANECA

ANECA has set up recognition agreements with regional agencies which fulfil the European standards (external reviewed by ENQA) to take part in the accreditation procedure.

Accrediting bodyAccrediting body


The standards are established in the legal framework that defines the features of the official degrees (Real Decreto 1393/07).

It includes broad references to justify a new programme.

It establishes general descriptors for bachelor, master and doctorate based on the Dublin Descriptors and related to the national QF (MECES).

Academic standards for learning outcomes/specific competences of a particular programme still pending. They are evaluted according to external (international reference points).

Academic standardsAcademic standards


Verification procedure of a programme/degree:Verification procedure of a programme/degree:

What is evaluated?

6.6. Personal académicoPersonal académico

7.7. Recursos materiales y serviciosRecursos materiales y servicios

8.8. Resultados previstosResultados previstos

9.9. Sistema de garantía de calidadSistema de garantía de calidad

10.10. Calendario de implantaciónCalendario de implantación

1.1. Descripción del títuloDescripción del título

2.2. JustificaciónJustificación

3.3. ObjetivosObjetivos

4.4. Acceso y admisión de estudiantesAcceso y admisión de estudiantes

5.5. Planificación de las enseñanzasPlanificación de las enseñanzas


Evaluation committees by field of knowledge (5)

ANECA selects, appoints and trains

A chairperson.chairperson.

A different number of academic peersacademic peers according to the numer of proposals.

One of the experts belongs to the professional fieldprofessional field of the corresponding field of knowledge.

A studentstudent.

A secretarysecretary from ANECA

Who evaluates?


Comisión de Rama de Conocimiento A

VOCAL 1

VOCAL 2

VOCAL 3

PRESIDENTE

SECRETARI O

Fase de alegaciones

I nforme de Evaluación Definitivo de cada Título

Comisión de Rama de Conocimiento A

Propuesta de Informe para cada Título

COMISIÓN DE EMISIÓN DE INFORMES

Estudiante

Valoración

VOCAL 4

VOCAL 5

VOCAL 6

VOCAL N

Comisión de Rama de Conocimiento B

PRESIDENTE

SECRETARI O

Estudiante

Valoración

Comisión de Rama de Conocimiento n

PRESIDENTE

SECRETARI O

Estudiante

Valoración

*Cada expediente será asignado a una Comisión de Rama de Conocimiento.

Comisión de Rama de Conocimiento nComisión de Rama de Conocimiento B



VOCAL 1

VOCAL 2

VOCAL 3

VOCAL 4

VOCAL 5

VOCAL 6

VOCAL N

VOCAL 1

VOCAL 2

VOCAL 3

VOCAL 4

VOCAL 5

VOCAL 6

VOCAL N

How it is evaluated?


Evaluation committees for field of study

Committee for Delivering Reports

ANECA’s double contrast

The evaluation of the programmes are conducted by the Evalluation Committees for Field of StudyEvalluation Committees for Field of Study. ANECA will submit the evaluation reports to the Council of Universities through the

Committee for Delivering ReportsCommittee for Delivering Reports.

Consistency ad intra and

inter- committees


How is it the

decision made?

ANECA delivers an evaluation ANECA delivers an evaluation reportreport either positive or negative. The report either positive or negative. The report is delivered following two steps:is delivered following two steps:

1. ANECA submits to the university a 1. ANECA submits to the university a provisional report including those provisional report including those aspects to be changed to obtain a aspects to be changed to obtain a positive decision proposals for positive decision proposals for enhancement. enhancement.

2. ANECA submits a reasoned report to 2. ANECA submits a reasoned report to the Council of Universities which is in the Council of Universities which is in charge of making the accreditation charge of making the accreditation decision.decision.

Once the programme has been Once the programme has been accredited and is included in the accredited and is included in the Register, ANECA publishes the report Register, ANECA publishes the report in its web-site.in its web-site.


The procedure consists of the following stages:

1.1. Self-evaluationSelf-evaluation

2.2. External reviewExternal review

3.3. ReportReport


ECA Principles for accreditation procedures

regarding joint programmes


4. The institutional perspective: The institutional perspective: AUDITAUDIT

Recognition of Internal Quality Assurance

Systems in University Education Programme

The aim of this programme is to enhance and

strengthen the development and implementation of

Internal Quality Assurance SystemsInternal Quality Assurance Systems in all center

(schools and faculties) that offer university

education in Spain. Another objective is to create a

procedure for the recognition of such systems.

Voluntary participation of institutions (faculties/universities)

Open to all Spanish Universities

First call for proposals started in September 2007First call for proposals started in September 2007

Voluntary participation of institutions (faculties/universities)

Open to all Spanish Universities

First call for proposals started in September 2007First call for proposals started in September 2007

•Agency for the Galician University System Quality (ACSUG)

•Agency for the Catalonian University System Quality (AQU)

The institutional perspective: AUDIT

To draw up Criteria and Guidelines for IQAS.

To evaluate HEI’s IQAS design.

To write an Interim and a Final Reports with an advisory function.

AUDIT Evaluation Methodology

QAAs

To design IQAS, according to Criteria and Guidelines.

To improve their IQAS according to recommendations in the Interim and Final reports.

To implement their IQAS.

HEIs

Focuses on teaching and learning processes

MILESTONES MILESTONES of the AUDIT programme

1. To orientate in the designdesign of the Internal Quality Assurance Systems (IQAS) at University faculties

2. Certification of the designdesign of the IQAS

3. Certification of the Implemented SystemsImplemented Systems


1. To orientate in the designdesign of the Internal Quality Assurance Systems (IQAS) at University faculties

Compromise

Planning

Diagnosis

Definition and documentation

Design guidelines

•Design guides

•Guidelines

•Tools

Schools, Faculties/ UniversitiesAgencies


Participation in the accreditation

Improvement and implementation On the basis of the given

documentation if verifies the

fulfillment of the IAQS

requirements

Design verification report

2. Verification of the designdesign of the IAQS



Implementation of IAQS Visit to the faculty /university

IAQS certification report

3. Certification of the implemented systemsimplemented systems


Internal Quality Internal Quality Assurance SystemAssurance System

Stakeholders Examples of aspects considered in the development of quality assurance mechanisms

StudentsStudents Selection and admission of students, programme profile, organization and development of the studies, learning support systems, educational outcomes and response of the labour market...

University University (in its different levels, i.e. faculty, managers and support staff)

Selection and admission of students, programme profile, organization and development of the studies, learning support systems, faculty and support staff, resources, academic progress, educational outcomes and response of the labour market, information systems...

EmployersEmployers Educational offer, programme profile, information quality, response of the labour market…

Public Public administrationsadministrations

Educational offer, programme profile, information quality, response of the labour market, education quality and response of the labour market, costs…

Society in generalSociety in general Supply and demand of education, academic progress and results, response of the labour market…

FFocus on stakeholdersocus on stakeholders


IInternal quality assurance guidelines

1. Quality policy and objectives

2. Design of educational offer

3. Development of the studies and other actions addressed to the students.

4. Faculty and support staff

5. Material resources and services

6. Educational outcomes

7. Public information


1.4. How the HEI manages and improves its physical resources and services

The HEI/university must be provided with mechanisms so it can design, manage and improve its services and physical resources in order for student learning to develop appropriately.

1.5. How the HEI analyses and takes into consideration the outcomes.

The HEI/university must be provided with procedures to ensure that outcomes (learning, graduate employment and the satisfaction of the different interest groups) are measured, analysed and used[1] for decision-making and to enhance the quality of degree programmes.

1.6. How the HEI publishes information on degree programmes.

The HEI must be equipped with mechanisms to ensure that updated information on degrees and programmes is published periodically.

EEvaluation criteria

• Organisation and structure, with a well-defined scope and a meaning that is clear.

• All guidelines must be dealt with.

• Existence of a comprehensive process map (or similar) describing what the process is, who is involved, and how each process is carried out.

• Examples of procedures and indicators that are included in IQAS.

AUDIT Evaluation Methodology

TThe review panels

• The same kind of review panel has been developed by the three QAAs involved in the AUDIT initiative.

• Panel members:• Three academic staff members with experience in QA.• An QA expert, staff member of a HEI.• An QA professional. • A staff member of QAA.

Participation in AUDIT

ANECA : 71

ACSUG: 35

AQU: 20

Faculties and collegesQAAs Universities

43

2

8

• 53 Spanish Universities are participating in the programme.

• Majority of these universities are public.

• For most universities, 2 or 3 faculties or colleges.

• 10 universities have designed an IQAS for all their faculties or colleges.

• 52 Universities have completed IQAS design.

Source: Rodriguez et al. EQAF 2008

• High degree of involvement by the universities and a high demand from new universities to take part in the 2008 call.

• QA experts from different universities and regions of Spain set up workgroups to jointly learn about and draw up improved proposals for their universities.

• The AUDIT initiative is contributing to the updating of the QA culture in Spanish universities by encouraging the development of more systematic initiatives for QA.

• Universities have designed their IQAS following the criteria and guidelines laid down in AUDIT.

• Mutual recognition between agencies is an ongoing process.

• Project developed by ANECA to train experts to take part in the evaluations: 2009-2010

Conclusions and Lessons Learned


• Inadequate specification of the stakeholders and how they are involved in identifying the requirements of the IQAS;

• Imprecise aims for quality

• Incomplete definition of the bodies responsible for the management of quality in faculties and colleges

• Lack of specific accountability procedures

• Incomplete mechanisms for decision-making in the IQAS, especially in relation to decisions subsequent to self-evaluation processes.



The AUDIT initiative is very useful in HEIs, specially in HEIs without an IQAS or where the internal quality assurance procedures and mechanisms have still not been systematised.

A university’s mission, vision and values should serve as the basis for producing an IQAS, and consideration also needs to be given to strategic issues.

In an initial stage of the development of IQAS, the agencies need to play an advisory role as well as that of external review to encourage the growth of a true quality culture in the universities.


Lessons Learned

It is necessary to involve university QA experts, professionals and academics to create synergies between those who play key roles in the design, development and enhancement of IQAS.

It is necessary to develop standard procedures between the different peer reviews that are carried out to ensure that results between the various review panels and committees are coherent.

Cooperation by the QAAs in joint projects is beneficial to the management of peer review processes.


Is IQAS recognition really an added value for HEIs? and for society?

Can the IQAS guarantee the quality of degree programmes in an external review?

What differences are there between the IQAS design and the usual QA activities in HEIs? Are they similar or not?

What is the real involvement of HEIs policy and decision-makers in a QA system such as AUDIT?

How can a QAA undertake a follow-up process with the engagement of the university? What is the effective role of the QAA in this process?

Key Issues To Be Mastered

There are various issues that still need to be addressed:


5. Open questions for the futureOpen questions for the future

AUDIT programme / follow-up:The follow-up procedure of the accreditation procedure will be the key part of the process

The follow-up stems very much to the IQAS of HEIs AUDIT was crucial to strengthen Q culture within HEIs The role of the QA Units within HEIs is crucial to support

programme and institutional co-ordinators Information tools generated by the HEIs are at the centre of the

procedure It will result in non-intrusive and non-beurocratic procedures for

HEIs Student protection in terms of the fulfilment of the programme

proposal accredited must be assured

Open questions for the future1

Future trends in QA and accreditation in Spain

Open questions fofr the future 2

On-going approach to quality: Quality and transparency of the information Reforms in the training plan focused on :

– Student learning

– Flexible and student-needs tailored curricular paths

– Development towards learning outcomes

– Internal quality assurance systems focused in the ESG

– Highly qualified teaching staff


Open questions for the future 3

The instituional approach will be therefore tested through the follow-up procedure. It must show adequacy and fitness for purpose.

The meta-evaluation of the procedure will be decisive in a future trethinking of the the whole accreditation procedure


Challenges

Multidimensional Transparency tools: Qrossroads (Proyect TE@M ECA) Rankings Joint evaluations reducing procedures through standards shared

by the agencies avoiding duplication of work. How interacting with project led by associations of disciplines to

define learning outcomes at the cycle level evaluating those programmes by means of labels aside the national QA/accreditation frameworks.


• The joint evaluation reducing the procedures by means of ddefining the standards shared by agencies in order to reduplicate proceses.

• How interact with projects led by associations of disciplines to define learning outcomes and giving quality labels apart from the national accreditation framework and agencies.

Challenges

Priorities for the next decade


[email protected]@aneca.es

Thank you for your attention

Rafael Llavori de Micheo Head of the International Relations Unit ANECA Case study 4: Spain ECA Seminar Changing Systems: achievements and new develop,ments.

Documents

accreditation procedure

ehea slide

accreditation regions

re accreditation

system universities

universities evaluation

national accreditation

national system