Positional impact of reputation in global university rankings

IREG Forum University Rankings, Warsaw 16 -17 May 2013

Positional impact of reputation

in global university rankings

Philippe Vidal and Ghislaine Filliatreau

Observatoire des Sciences et Techniques, Paris

IREG Forum University Rankings, Warsaw 16 -17 May 2013

Reputation is an idle and most false imposition, oft got without

merit, and lost without deserving

Shakespeare, Othello, II, 3

IREG Forum University Rankings, Warsaw 16 -17 May 2013

In Global university rankings and their impact. Report II, EUA Report on rankings 2013

• 2.3 Superficial descriptions of methodology and poor indicators

• …. Use of poor indicators also persists. In spite of widespread criticisms, reliance on reputation is becoming more and more widespread…..

IREG Forum University Rankings, Warsaw 16 -17 May 2013

Rankings graphical comparison as an approach

for deconvoluting ranking patterns

Reputation surveys: ARWU: 0%, THE: 34.5%, QS: 40%

Webometrics as a proxy for reputation

Facts vs reputation

IREG Forum University Rankings, Warsaw 16 -17 May 2013

Methodology 1Ternary diagrams

IREG Forum University Rankings, Warsaw 16 -17 May 2013

Methodology 2Ranking A vs Ranking B

0

50

100

150

200

250

300

350

400

450

500

0 50 100 150 200 250 300 350 400 450 500

rank

ing

A

ranking B

Ranking A vs Ranking B - Europe

IREG Forum University Rankings, Warsaw 16 -17 May 2013

Methodology 3Absolute and relative differences

Diagrams

(ARWU-Y) vs ARWUand

(ARWU-Y)/ARWUvs ARWU

IREG Forum University Rankings, Warsaw 16 -17 May 2013

Methodology 4Diagram ARWU/X vs ARWU

A

B

C

0

0,5

1

1,5

2

2,5

0 100 200 300 400 500

Ran

king

AR

WU

/Ran

king

X

Ranking ARWU

forbidden zone

A B

C

0

100

200

300

400

500

0 100 200 300 400 500

Ran

king

X

Ranking ARWU

IREG Forum University Rankings, Warsaw 16 -17 May 2013

Methodology 5Rankings reproducibilities

- years selected: 2010 - 2011 - 2012

- average deviations to the average from one year to next:

ARWU: 2-3 %

THE: 10-12 %

QS: 6-7 %

Webometrics: 25-30 %

IREG Forum University Rankings, Warsaw 16 -17 May 2013

Methodology 6 Filtering rules

90 % of the universities (259) selected

62 % ranked in 4 rankings6 % in 3 rankings22 % in 2 rankings

First 250 ARWU ranks: 131 universities

IREG Forum University Rankings, Warsaw 16 -17 May 2013

Targets

United States

United Kingdom

Northern Europe (Germany, Scandinavia, Netherlands)

Southern Europe (France, Spain, Portugal, Italy)

IREG Forum University Rankings, Warsaw 16 -17 May 2013

Results 1 Ternary diagrams

IREG Forum University Rankings, Warsaw 16 -17 May 2013

Results 2 Ternary diagrams

US

UK

Northern Europe

Southern Europe

ARWU

Webometrics QSbetter ARWU score

better QS score

better Webometrics score

ARWU

Webometrics QS

better ARWU score

better QS score better Webometrics score

IREG Forum University Rankings, Warsaw 16 -17 May 2013

Results 3 ARWU vs THE 1

(ARWU - THE) vs ARWU

-250

-200

-150

-100

-50

0

50

100

150

200

0 50 100 150 200 250

ARWU

AR

WU

- TH

E

USA

United Kingdom

Northern Europe

Southern Europe

IREG Forum University Rankings, Warsaw 16 -17 May 2013

Results 4 ARWU vs THE 2

(ARWU/THE) vs ARWU

0

1

2

3

4

5

6

7

0 50 100 150 200 250

ARWU

AR

WU

/ TH

E

USA

United Kingdom

Northern Europe

Southern Europe

forbidden zone

IREG Forum University Rankings, Warsaw 16 -17 May 2013

Results 5 ARWU vs QS 1

(ARWU - QS) vs ARWU

-350

-300

-250

-200

-150

-100

-50

0

50

100

150

200

0 50 100 150 200 250

ARWU

AR

WU

- Q

S

USA

United Kingdom

Northern Europe

Southern Europe

IREG Forum University Rankings, Warsaw 16 -17 May 2013

Results 6 ARWU vs QS 2

(ARWU / QS) vs ARWU

0

0,5

1

1,5

2

2,5

3

3,5

4

4,5

0 50 100 150 200 250

ARWU

AR

WU

/QS

USA

United Kingdom

Northern Europe

Southern Europe

forbidden zone

IREG Forum University Rankings, Warsaw 16 -17 May 2013

Results 7 ARWU vs Webometrics 1

-500

-400

-300

-200

-100

0

100

200

0 50 100 150 200 250

ARWU

AR

WU

- W

ebo

met

rics

US

UK

Northern Europe

Southern Europe

IREG Forum University Rankings, Warsaw 16 -17 May 2013

Results 8 ARWU vs Webometrics 2

(ARWU/Webometrics) vs ARWU

0

0,5

1

1,5

2

2,5

3

3,5

4

4,5

0 50 100 150 200 250

ARWU

AR

WU

/ W

ebo

met

rics

USA

United Kingdom

Northern Europe

Southern Europe

forbidden zone

IREG Forum University Rankings, Warsaw 16 -17 May 2013

Results 9

US: Relative differences

US: (ARWU - THE, QS, Webometrics) / ARWU

-7

-6

-5

-4

-3

-2

-1

0

1

2

0 50 100 150 200 250

ARWU

(AR

WU

- TH

E, Q

S, W

ebom

etric

s)/A

RW

U

THE

QS

Webometrics

IREG Forum University Rankings, Warsaw 16 -17 May 2013

Results 10 Europe: Relative differencesTHE-QS: Continental Europe Webometrics=whole Europe

Europe: (ARWU -THE, QS, Webometrics) / ARWU vs ARWU

-12

-10

-8

-6

-4

-2

0

2

0 50 100 150 200 250

ARWU

(AR

WU

- TH

E, Q

S, W

ebom

etric

s)/A

THE

QS

Webometrics

IREG Forum University Rankings, Warsaw 16 -17 May 2013

Summary

Underestimatation of universities best ranked for researchOverestimation of universities less well ranked

for researchClear for THE and QS, even clearer for Webometrics

And for the best research universities:

US: underestimation THE/QS, overestimation WebometricsUK: overestimation THE/QS, underestimation WebometricsN + S Europe: underestimation THE/QS/Webometrics

IREG Forum University Rankings, Warsaw 16 -17 May 2013

Conclusions

Reputation concept still on its infancy

Necessity to do research on the concept as well as its modalities of implementation