New trends in university rankings 2011

www.che.de

Gero Federkeil,CHE Centre for Higher Education, Germany

New Trends in University Rankings

1st International Seminar on Higher Education Rankings and E-Learning,

22-23 September 2011, Barcelona

The CHE – A Short Introduction

The traditional ranking approach

Überschrift Nummer 3New trends: Multi-dimensional rankings

Presentation

2Barcelona, September 2011

The U-Multirank Project

Conclusions

Private, not-profit organisationFounded in 1994 by Bertelsmann Foundation and German Rectors Conference

The CHE - Centre for Higher Education

Goal: Initiate and promote of reforms in German higher education

Activities:HE policy issues (e.g. Bologna, funding, …)ConsultingCommunication & trainingRanking


The CHE - Centre for Higher Education

Ranking of German universities among founding tasks of CHEFirst ranking published in 1998

Extension of fields and indicatorsContinuous further development of methodology

InternationalisationExtension of CHE Ranking: Austria, Switzerland, NetherlandsCooperation with Fundación CYD to develop a ranking in SpainU-Multirank project to “develop the concept and test the feasibility of a global multi-dimensional university ranking” Founding member of IREG –Observatory on Academic Rankings and Excellence (“Berlin Principles”)





Presentation



Conclusions

Barcelona, September 2011 6

The growing importance of rankings

Rankings have become a wide spread phenomenon now:about 10 global rankingsnational rankings in about 50 countries

Rankings have an impact onstudentsstrategies of universities (e.g cooperations, mergers)politics (e.g. Excellence / exchnage programmes)

Yet they remain controversial

The Traditional Ranking Approach


Ranking of wholeuniversities

Composite indicator

League table


The critique of the traditional ranking model

Ranking of whole institutions

Composite overall indictor

League table approach

• Composite indicators blur profiles and strengths & weaknessesThere are neither theoretical nor empirical arguments for assigning specific pre-defined weights to single indicators

Most users are interested in information about “their” field”Institutional rankings give misleading averages across fields/units

Barcelona, September 2011

Shanghai Jiaotong Ranking QS Indicator Weight Indicator Weight SCI publications 20 % Reputation among scholars 40 %

Publications Science & Nature 20 % Reputation among employers 10 %

Highly cited authors 20 % Citations 20 %

Nobel Prizes & Field Medals 20 % Student-staff-ratio 20 %

Alumni with NobelPrizes 10 % International students 10 %

Size 10 % International staff 10 %

World Rankings: Indicators & Weights

9

http://www.arwu.org/index.jsp


The critique of the traditional ranking model

Ranking of whole institutions

Composite overall indictor

League table approach

• Composite indicators blur profiles and strengths & weaknessesThere are neither theoretical nor empirical arguments for assigning specific pre-defined weights to single indicators

• Small differences in the scores of indicators lead to big differences in league tableGive false impression of exactness (“Number 123 is better than number 127”)

Most users are interested in information about “their” field”Institutional rankings give misleading averages across fields/units




Presentation



Conclusions


New trends in rankings

Although this is still the dominant model there are new trends:

Alternative approaches: Multi-dimensional rankingsChanges within existing rankings


Changes in existing rankings

Introduction of field based-rankings in addition to institutional rankings

At least in lower part of ranking: broad groups instead of league tables (e.g. 51 – 75, 76 – 100 ...),


A new approach: Multi-dimensional rankings

There is a number of rankings with a different approach:

National rankings, e.g.:CHE ranking (Germany, Austria, Switzerland, Netherlands)College Navigator TaiwanGood University Guide AustraliaRanking Project Spain (Fundación CYD – CHE)

International rankings, e.g.Leiden RankingU-Multirank Project


Multi-dimensional rankings; Basic concept

Basic assumption: There is no objective ranking as there is no single, objective concept of quality (“Quality is in the eye of the beholder”)Different users of rankings have different ideas about quality & different preferences with regard to indicatorsEach ranking reflects the views of the producers of rankingsAs the producers of rankings have to define a set of indicators, butthis set should be broad and be based on stakeholder consultation, and,the decision about the relevance of indicators should be left to users

Multi-dimensional rankings do not calculate

composite indicatorswith pre-defined weights

of single indicators !

HE Conference 2011 | Gero Federkeil | Canberra 2011/03/02 16

Multi-dimensional ranking : CHE

Field based ranking

Multi-dimensional ranking

Rank group approach

• ~20 to 25 indicators, each stands for itself• Show profiles, allow for an anaylsis of strengths

& weaknesses at relevant levelRefer to a user-focused concept of good performance /quality

• Top, Middle, Bottom group• Avoids false impressions of exactness• Takes serious limitations in data (quality)

Inclusion of 34 fieldsMore meaningful information to usersReflects internal heterogeneity of universities

+

+

+




Presentation



Conclusions

The project• Commissioned by the European Commission• 2‐year project, 2009 – June 2011• Report now available:

http://ec.europa.eu/education/higher-education/doc/multirank_en.pdf

• Ján Figel, the former European Commissioner for Education, Training, Culture and Youth:

“‐ to allow stakeholders to make informed choices;‐ to help institutions to position themselves andimprove their performance”

• Two phases:o Design of new instrumento Testing the feasibility of new instrument


http://ec.europa.eu/education/higher-education/doc/multirank_en.pdf

Specification of U‐Multirank

• Five dimensions:o Teaching & learningo Researcho Knowledge transfero International orientationo Regional engagement

• Long list of indicators to be tested in pilot project

• development of data collection tools and processes (question‐naires, definitions, FAQs, communication + feedback processes)

• methods for building ranking groups instead of league tables


Testing U‐Multirank

• Two levels:• Institution (FIR)• Field (FBR)

• Global sample of higher education and research institutions: 159 (target: 150), 2/3 Europe, 109 completed institutional questionnaires

• Two fields: • Business studies• Engineering (electrical and mechanical)


Classification and Ranking: Mapping diversity

Diversity of higher education institutions in Europe & theworld

Identifying comparableinstitutions that can be

compared in one ranking

Description of horizontal diversity

Types/profiles

Assessment of verticaldiversity

Performance

Complementary instruments of transparency

+

M lti di i l l b l i it ki


First element: Selection of a comparable set of universities based on institutional profiles

Teaching andlearning

Research involvement

Knowledgeexchange

Regional engagement

International orientation

Student profile

Example:

• Comprehensive, teaching oriented institution• Mainly undergraduate education• Low research orientation• Some activities with regard to knowledge transfer• Low international orientation• Regionally embedded (e.g. recruiting)

Comparison / ranking among this particularprofile of institutions


Second element: ranking – Multi-dimensional ranking


Second element: Selection of indicators according to user‘s preference

Selection of (upto) 5 indicators

Priorisation ofindicators



The result is a personalised

ranking



Looking different with different

indicators


Dimension Total # indicators

After pilot

A: need no/minor modification

B: need further work C: discarded

Teaching & Learning

FIR 5 0 1 0

FBR 8 4 4 0

FBR (student survey) 13 13 0 0

Research

FIR 9 3 5 1

FBR 7 6 1 0

Knowledge transfer

FIR 8 3 5 0

FBR 7 1 2 4

International orientation

FIR 8 6 2 0

FBR 9 6 3 0

Regional engagement

FIR 4 1 3 0

FBR 5 1 4 0

feasibility of dimensions and indicators


Teaching & Learning Teaching & Learning

U‐Multirank ‐ Indicators


Research Knowledge Transfer



Regional EngagementInternational Orientation



EU Commission: Communication on the Modernisation of Higher Education (20 Sept.2011):

Future Prospects

http://ec.europa.eu/education/higher-education/doc/com0911_en.pdf

http://ec.europa.eu/education/higher-education/doc/com0911_en.pdf




Presentation



Conclusions

Conclusions


Rankings are here to stay, their relevance will rather increaseThere is a need for transparency about European Higher Education Aera

Multi-dimensional rankings are a new approach which

is user-driven, taking into account that there is no single objective ranking, and,

is able to make visible the diversity of higher education institutions by showing different profiles, and

is looking beyond research excellence only (teaching & learning, transfer ...)

Conclusions


Within the context of U-Multirank there could be one particular ranking among Open Universities /distance education institutions.

Some indicators of U-Multirank are applicable,

the appropriateness of others would have to be discussed, and

there is surely a need for new, particular indicators


There might be some limits to rankings in general

„You‘re kidding! You count publications?“

37

More information:

www.che-ranking.dewww.u-multirank.eu

[email protected]

http://www.che-ranking.de/

http://www.u-multirank.eu/

mailto:[email protected]

www.che.de

Gero Federkeil,CHE Centre for Higher Education, Germany

New Trends in University Rankings

1st International Seminar on Higher Education Rankings and E-Learning,

22-23 September 2011, Barcelona

New trends in university rankings 2011

Education

che ranking

number of rankings

fieldinstitutional rankings

academic rankings

world rankings

existing rankings

objective ranking

indicatorseach ranking