IREG Forum University Rankings, Warsaw 16 -17 May 2013 Positional impact of reputation in global university rankings Philippe Vidal and Ghislaine Filliatreau.

IREG Forum University Rankings, Warsaw 16 -17 May 2013

Positional impact of reputation

in global university rankings

Philippe Vidal Philippe Vidal andand Ghislaine Filliatreau Ghislaine Filliatreau

Observatoire des Sciences et Techniques, ParisObservatoire des Sciences et Techniques, Paris


Reputation is an idle and most false imposition, oft got without

merit, and lost without deserving

Shakespeare, Othello, II, 3


In Global university rankings and their impact. Report II, EUA Report on rankings 2013

• 2.3 Superficial descriptions of methodology and poor indicators

• …. Use of poor indicators also persists. In spite of widespread criticisms, reliance on reputation is becoming more and more widespread…..


Rankings graphical comparison as an approach

for deconvoluting ranking patterns

Reputation surveys: ARWU: 0%, THE: 34.5%, QS: 40%

Webometrics taken as a proxy for reputation

Rankings graphical comparison as an approach

for deconvoluting ranking patterns

Reputation surveys: ARWU: 0%, THE: 34.5%, QS: 40%

Webometrics taken as a proxy for reputation

Facts vs reputationFacts vs reputation


Methodology 1Ternary diagrams


Methodology 2Ranking A vs Ranking B

0

50

100

150

200

250

300

350

400

450

500

0 50 100 150 200 250 300 350 400 450 500

rank

ing

A

ranking B

Ranking A vs Ranking B - Europe


Methodology 3Absolute and relative differences

Diagrams

(ARWU-Y) vs ARWU

and

(ARWU-Y)/ARWU vs ARWU

Diagrams

(ARWU-Y) vs ARWU

and

(ARWU-Y)/ARWU vs ARWU


Methodology 4Diagram ARWU/X vs ARWU

A

B

C

0

0,5

1

1,5

2

2,5

0 100 200 300 400 500

Ran

king

AR

WU

/Ran

king

X

Ranking ARWU

forbidden zone

A B

C

0

100

200

300

400

500

0 100 200 300 400 500

Ran

king

X

Ranking ARWU


Methodology 5Rankings reproducibilities

- years selected: 2010 - 2011 - 2012

- average deviations to the average from one year to next:

ARWU: 2-3 %

THE: 10-12 %

QS: 6-7 %

Webometrics: 25-30 %


Methodology 6 Filtering rules

90 % of the universities (259) selected

62 % ranked in 4 rankings6 % in 3 rankings22 % in 2 rankings

First 250 ARWU ranks: 131 universities


Targets

United States

United Kingdom

Northern Europe (Germany, Scandinavia, Netherlands)

Southern Europe (France, Spain, Portugal, Italy)


Results 1 Ternary diagrams


Results 2 Ternary diagramsleft: ARWU 1->500, right:1->250

US

UK

Northern Europe

Southern Europe

ARWU

Webometrics QSbetter ARWU score

better QS score

better Webometrics score

ARWU

Webometrics QS

better ARWU score

better QS score better Webometrics score


Results 3 ARWU vs THE 1

(ARWU - THE) vs ARWU

-250

-200

-150

-100

-50

0

50

100

150

200

0 50 100 150 200 250

ARWU

AR

WU

- TH

E

USA

United Kingdom

Northern Europe

Southern Europe


Results 4 ARWU vs THE 2

(ARWU/THE) vs ARWU

0

1

2

3

4

5

6

7

0 50 100 150 200 250

ARWU

AR

WU

/ TH

E

USA

United Kingdom

Northern Europe

Southern Europe

forbidden zone


Results 5 ARWU vs QS 1

(ARWU - QS) vs ARWU

-350

-300

-250

-200

-150

-100

-50

0

50

100

150

200

0 50 100 150 200 250

ARWU

AR

WU

- Q

S

USA

United Kingdom

Northern Europe

Southern Europe


Results 6 ARWU vs QS 2

(ARWU / QS) vs ARWU

0

0,5

1

1,5

2

2,5

3

3,5

4

4,5

0 50 100 150 200 250

ARWU

AR

WU

/QS

USA

United Kingdom

Northern Europe

Southern Europe

forbidden zone


Results 7 ARWU vs Webometrics 1

-500

-400

-300

-200

-100

0

100

200

0 50 100 150 200 250

ARWU

AR

WU

- W

ebo

met

rics

US

UK

Northern Europe

Southern Europe


Results 8 ARWU vs Webometrics 2

(ARWU/Webometrics) vs ARWU

0

0,5

1

1,5

2

2,5

3

3,5

4

4,5

0 50 100 150 200 250

ARWU

AR

WU

/ W

ebo

met

rics

USA

United Kingdom

Northern Europe

Southern Europe

forbidden zone


Results 9

US: Relative differences ordinate scale: 1 = 100%

-7

-6

-5

-4

-3

-2

-1

0

1

2

0 50 100 150 200 250

ARWU

US: (ARWU - THE, QS, Webometrics) / ARWU

THE

QS

Webometrics


Results 10 Europe: Relative differences

THE-QS: Continental Europe Webometrics=whole Europe

ordinate scale: 1 = 100%

-12

-11

-10

-9

-8

-7

-6

-5

-4

-3

-2

-1

0

1

2

0 50 100 150 200 250

ARWU

Europe: (ARWU -THE, QS, Webometrics) / ARWU vs ARWU

THE

QS

Webometrics


Summary

Underestimatation of universities best ranked for researchOverestimation of universities less well ranked

for researchClear for THE and QS, even clearer for Webometrics

And for the best research universities:

US: underestimation THE/QS, overestimation WebometricsUK: overestimation THE/QS, underestimation WebometricsN + S Europe: underestimation THE/QS/Webometrics


Conclusions

Reputation concept still on its infancy

Necessity to do research on the concept as well as its modalities of implementation

IREG Forum University Rankings, Warsaw 16 -17 May 2013 Positional impact of reputation in global university rankings Philippe Vidal and Ghislaine Filliatreau.

Documents

ireg forum university

arwuyarwu vs arwu slide

arwu vs webometrics

arwu vs qs

rankings graphical comparison

diagram arwux vs arwu

vs ranking b slide

arwu ranks