Mauro Degli Esposti Towards a new model for evaluating the merit of Italian academic scientists and their institutions Ferrara, Jan 2011.

Faculty ofLife SciencesFaculty ofLife Sciences

Mauro Degli Esposti

Towards a new model for evaluating the merit of Italianacademic scientists and their institutions

Ferrara, Jan 2011

A new analysis of Italian universities – the start

From The Times October 9, 2008

Whatever happened to the University of Bologna?

The University of Bologna, which received its charter from Frederick I Barbarossa in 1158, was probably the first university founded in the Western world. It became a renowned home to scholars of great distinction in both the sciences and the humanities. The Times Higher Education-QS World University Rankings survey was published this week. It rates the performance of universities according to the quality of their research, the employment rates of graduates and their international profile.

The University of Bologna does not feature in the top 100!

Manchester = 29 – now 26

Harvard = 1, Cambdrige = 3, Oxford = 4, UCL = 7...

M Degli Esposti, presented at the first conference of Via-academy, 21 September 2009 http://via-academy.org/Relazione_Prima_Conferenza_della_VIA_a_Manchester.aspx

MERIT MERIT – – a new approach to evaluation a new approach to evaluation

M Degli Esposti, presented at the first conference of Via-academy, 21 September 2009 http://via-academy.org/Relazione_Prima_Conferenza_della_VIA_a_Manchester.aspx

The members of the Via-academy network who work in UK have been selectedon the basis of merit, but educated in Italy.

The number of academic links of Via-members with Italian institution will be proportional to the relative teaching capacity of these institutions – i.e. they will be an output measure of their quality and relative standing.

Diritto di cronaca

12/10/2009

Un ateneo per scappare

di FLAVIA AMABILE

http://via-academy.org/Relazione_Prima_Conferenza_della_VIA_a_Manchester.aspx

Italian University Times QS Leiden 250 VIA members

Bologna 1 3 3

Roma 1 ‘La Sapienza’ 2 2 2

Padova 4 4 4

Pisa 5 7 6

Firenze 6 8 8

Napoli ‘Federico II’ 7 5 5

Milano 15 1 1

Kendall tau correlation coefficient

0.31with Leiden

1With Leiden

0.79**with Leidenby Marco Geraci

M Degli Esposti, presented at the first conference of Via-academy, 21 September 2009

by Marco Geraci

Italian University VIA25 Gelmini virtuose (27+)

Padova 1 14

Bologna 2 16

Milano 3 11

Roma 1 ‘La Sapienza’ 4 (42)

Torino 5 8

Pisa 6 22

Firenze 7 27 (later)

Pavia 8 25

Trento 9 1

Napoli ‘Federico II’ 10 (38)

Kendall tau correlation coefficient

0.63**

with Leiden

0.09

with Leiden

VIA25 - cumulative ranking from 10 separate rankings

http://rpc264.cs.man.ac.uk/VIA/index.php/VIA25

October 2009

http://rpc264.cs.man.ac.uk/VIA/index.php/VIA25

parere sull'iniziativa 'top italian scientists'

0

10

20

30

40

50

60

70

80

positivo neutro negativo ISI negativo

nu

me

ro

parere sull'iniziativa 'top italian scientists'

0

10

20

30

40

50

60

70

80

positivo neutro negativo ISI negativo

nu

me

ro

M Degli Esposti, presented at the second conference of Via-academy, 18 September 2010

From ‘top Italian scientists’: ranking of Italian institutes

La Stampa 30/8/10 Rank Italian Institution Sum of H-Index N. of TIS

1 Bologna 3019 712 CNR, all 2524 633 Milano 2113 464 Padova 2060 465 Roma 1713 416 Torino 1619 387 INAF, all 1420 388 San Raffaele Milano 1333 289 Firenze 1205 29

10 Ist. Naz. Tumori Milano 1106 2511 Pisa 1049 2312 Ferrara 998 2613 Napoli 959 2314 Mario Negri Miano 931 2015 Genova 891 2016 Roma 2 880 2117 Perugia 804 1918 Pavia 712 1519 Brescia 610 1520 Trieste 585 1321 Modena 489 1222 Normale Pisa 478 1123 Milano Bicocca 436 824 Parma 433 1025 IEO Milano 427 926 Sanita' Roma 423 1027 IFOM Milano 399 828 Bari 388 929 Verona 386 930 SISSA Trieste 374 1031 Siena 374 932 Trento 373 933 Humanitas Milano 336 634 IIT Genova 326 835 Roma 3 326 836 Udine 297 737 Cagliari 250 538 Cattolica Roma 244 639 Bocconi Milano 222 540 Ancona 217 641 Gaslini Genova 186 342 Palermo 182 5

M Degli Esposti, December 31, 2010 – retrieved from: http://www.topitalianscientists.org/Top_italian_scientists_VIA-Academy.aspx

400 TIS

1230 TIS La Repubblica 5/1/2011

The media resonance of ‘top Italian scientists’

Italian biochemists and their h-index

NoteValori di h-index sonostati calcolati cosi’:

X, con Google Scholaradd-standard, con iniziale,ristretto alle aree ‘chemistry’,‘biology’ and ‘medicine’

Y, con ISI Web of Science,In genere ‘all years’ nei settori di bio-medicina (non clinici) e chimica(e.g. medicinal chemistry)

Correlation h-index for biochemists, 54 names

y = 0.97

R2 = 0.973

0

10

20

30

40

50

60

70

80

90

100

0 20 40 60 80 100

h-index Google Scholar add-on

h-in

dex

ISI W

oS

Mauro Degli Esposti La Voce.it, 14-9-10

http://www.topitalianscientists.org/downloads/Manualetto_del_Ricercatore_Provetto_ver1.2.pdf

mean for full professors = 19

Italian scientists: home and abroad

Abroad (50)Home - in Italy (50)

>70 >70

over 70

under 50

female

prize winners

cell biologists/cancer

physics

chemistry & biochemistry

computer sciences

economics

over 70

under 50

female

prize winners

cell biologists/cancer

physics

chemistry & biochemistry

computer sciences

economics

average h-index computer scientitsts

0

10

20

30

40

50

60

70

abroad Italy

aver

age

h-i

nd

ex

TIS Reports 001http://tisreports.com/fullreport/1-Top_100_-_home___abroad.aspx

1118 names

all top updated

ID h index Area1 125 cancer - medicine2 119 cancer - medicine3 109 immunology - medicine4 109 immunology - medicine5 102 immunology - medicine6 99 nephrology - medicine7 98 immunology - medicine8 98 computer, neurosciences, social sciences

9 95 immunology - medicine

ultimo aggiornamento: 09/11/2010

Top Italian Scientists List

Academic Nation Italian Institution noteCarlo Croce USANapoleone Ferrara USA Lasker award 2010Alberto Mantovani Italy Humanitas e MilanoGiorgio Trinchieri France/USAEttore Appella USAGiuseppe Remuzzi Italy Mario NegriAlex Sette USA youngest topTomaso Poggio USA most eclectic

Lorenzo Moretta Italy Gaslini GE

Top Italian Scientists – towards the saturation of names?

83% of the top100 work inBio-medicine!Bio-medicine!


Evolution TIS List

0

200

400

600

800

1000

1200

1400

10-Aug 30-Aug 19-Sep 09-Oct 29-Oct 18-Nov 08-Dec 28-Dec

time (date, 2010)

nu

mb

er

of

na

me

s

Top Italian scientists abroad(total = 363)

0

50

100

150

200

USAUK

Swizerla

nd

Franc

e

Germ

any

Sweden

Spain

Austri

a

Vatica

n

Canad

a

Belgium

Other

nat

ions

nu

mb

er

% total Top Italians

0

20

40

60

80

100

Ital

y

abro

ad

doub

leaf

filia

tion

M Degli Esposti (2010), retrievedJanuary 3, 2011 from: http://www.topitalianscientists.org/Top_italian_scientists_VIA-Academy.aspx

http://www.topitalianscientists.org/Top_italian_scientists_VIA-Academy.aspx

Fonte : OCSE

Nel 2006 Italia = 93000 ricercatori

0

0.5

1

1.5

2

2.5

3

3.5

4

haem

atolo

gy

aging

endo

crino

logy

canc

er re

searc

h

neph

rolog

y

haep

atolo

gy

onco

logy

neur

olog

y

inte

rnal m

edici

ne

imm

unolo

gy/a

llerg

y

neur

oscie

nces

micr

obiolog

y

biolo

gy

cell b

iolog

y

bioc

hemist

ry

phar

maco

logy

phys

ics-a

stro

physic

s

nucle

ar p

hysic

s

mat

hem

atica

l phy

sics

mat

hem

atics

econ

omy

chem

istry

vete

rinar

y

no

rmal

ized

ran

kin

g v

s. m

edic

ine

haematology

immunologybiochemistry

cancer research

0

0.5

1

1.5

2

2.5

3

3.5

4

haem

atolo

gy

aging

endo

crino

logy

canc

er re

searc

h

neph

rolog

y

haep

atolo

gy

onco

logy

neur

olog

y

inte

rnal m

edici

ne

imm

unolo

gy/a

llerg

y

neur

oscie

nces

micr

obiolog

y

biolo

gy

cell b

iolog

y

bioc

hemist

ry

phar

maco

logy

phys

ics-a

stro

physic

s

nucle

ar p

hysic

s

mat

hem

atica

l phy

sics

mat

hem

atics

econ

omy

chem

istry

vete

rinar

y

no

rmal

ized

ran

kin

g v

s. m

edic

ine

haematology

immunologybiochemistry

cancer research

Areas of relative excellence for Italy

SCImago. (2007). SJR — SCImago Journal & Country Rank.Retrieved November 10, 2010, from http://www.scimagojr.com

top scienziati per discipline

050

100150200250300350400450500

med

icina

biolog

iafis

ica

astro

fisica

chim

ica

infor

mat

ica

farm

acolo

gia

econ

omia

mat

emat

ica

tech

nolog

y

nu

me

ro d

i s

cie

nzi

ati


Fonte : ISTAT

0

100

200

300

400

500

600

700

TIS

Ranking di universita’ statali e non – problems

L’università italiana ha finito anche la fruttadi Andrea AparoIL Fatto Quotidiano, 12 Novembre 2010

Ricercatori italiani uguale fannulloni?di Pietro GrecoScienza in Rete (http://www.scienzainrete.it)04/05/2009

Ranking for Italian universities – various and different

Kendall corr. coefficientVIA25_2 VIA_membtimes_09 iarwu_09 webo_09 leid_orangecampus campus_supersole_09 censis_09 censis_10 social_cap_gategelmini_09 udu_09 MIUR_09

VIA25_1 stampa 0.88** (25) 0.53** (16) 0.42* (19) 0.65** (18) 0.68** (19) 0.63** (18) 0.39* (18) 0.54** (21) -0.12 (17) -0.19 (14) 0.08 (13) 0.65** (22) -0.04 (17) 0.57** (20) 0.59** (20)VIA25_2 0.51** (16) 0.46* (19) 0.7** (18) 0.73** (19) 0.63** (18) 0.42* (18) 0.52** (21) -0.12 (17) -0.25 (14) -0.1 (13) 0.69** (22) -0.03 (17) 0.59** (20) 0.6** (20)VIA_memb 0.43 (12) 0.66** (12) 0.49* (13) 0.79** (13) 0.29 (11) 0.46* (15) -0.36 (10) -0.03 (9) -0.6 (6) 0.49* (13) -0.02 (10) 0.64** (13) 0.71** (14)times_09 0.24 (14) 0.49* (16) 0.31 (16) 0.08 (14) 0.25 (17) 0.02 (12) -0.02 (11) -0.24 (10) 0.46* (18) -0.1 (13) 0.3 (17) 0.35 (16)iarwu_09 0.56** (15) 0.64** (15) 0.33 (13) 0.45* (15) -6.16 -0.28 (9) -0.57 (8) 0.53** (17) -0.24 (11) 0.57** (17) 0.57** (18)webo_09 0.49* (14) 0.12 (13) 0.31 (15) -0.35 (12) -0.22 (9) -0.11 (8) 0.81** (19) -0.4 (13) 0.48** (17) 0.49** (16)leid_orange 0.08 (13) 0.42* (16) -0.64** (12)-0.06 (9) 0.07 (8) 0.53** (16) 0.09 (12) 0.67** (16) 0.75** (17)campus 0.54** (14) 0.21 (14) 0 (12) -0.24 (10) 0.22 (15) 0.28 (15) -0.01 (14) 0.03 (14)campus_super 0.05 (14) 0.03 (13) -0.05 (11) 0.35* (18) -0.05 (13) 0.53** (17) 0.41* (17)sole_09 0.26 (13) 0.13 (11) -0.25 (14) 0.01 (15) -0.3 (12) -0.42 (12)censis_09 0.73** (10) -0.38 (11) 0.09 (12) -0.24 (10) -0.07 (10)censis_10 -0.09 (11) 0.06 (9) -0.2 (10) -0.06 (9)social_cap_gate -0.16 (14) 0.49** (19) 0.53** (18)gelmini_09 -0.15 (12) -0.12 (12)udu_09 0.75** (19)

VIA25-1 s VIA25_2 VIA_membtimes_09 iarwu_09 webo_09 leid_orangecampus campus_supersole_09 censis_09 censis_10 social_cap_gategelmini_09 udu_09 MIUR_09

Marco Geraci e Mauro Degli Esposti, unpublished results, October 2010

Ranking From source nature

Times QS - THE UK QS Agency and Times Mix of 6 parameters

ARWU China Jiao Tong Shanghai University

Mix of 6 parameters (different)

Leiden top 250 Holland CSTS Leiden University Bibliometric (normalized)

Webometrics Spain Cybermetrics lab, CISC Based on data of web visibility

Vision UK - Italy Vision think-tank Mix of 5 parameters (2009)

Campus super Italy Campus magazine Average of 8 rankings

Sole 24 Ore Italy Sole 24 Ore newspaper Mix of national parameters

Repubblica Censis Italy Censis Institute Mix of national parameters

MIUR Italy CIRV, VTR2001-2003 Bibliometric (selected)

Marco Geraci e Mauro Degli Esposti, November 2010, in preparation

In depth analysis of all ranking – recent advances

GRAND VIA - a new meta-ranking combining 5 international and 4 national ones

From co-variance patterns – principal components analysis, PCA (Pearson, 1901)

Ranking of Italian universities: 2 principal components

Principal component analysis of different rankings of Italian universities

International impact and visibility

Na

tio

na

l p

erf

orm

an

ce

-0.5 0.0 0.5 1.0

-0.5

0.0

0.5

1.0

Rankings map

PC1 (39.7%)

PC

2 (

25

.9%

)

Times-QS

iARWU

Leiden OrangeVIA-Member

Webometrics

Campus

Sole

Censis-Repubblica

MIUR

Grand VIA

VIA-TopH

Campus Super

UDU

Gelmini

Top Italian Scientists

Vision

Marco Geraci e Mauro Degli Esposti, November 2010 in preparation

Grand VIA

Ranking from ‘top Italian scientists’ – better than others!

Marco Geraci e Mauro Degli Esposti, November 2010

cumulative correlation with rankings

0

0.5

1

1.5

2

2.5

3

3.5

Times QS ARWU Grand VIA Top HItalians

cu

mu

lati

ve

co

rre

lati

on

Computed over 4 different international rankings

Vision

Evaluation landscape of state Italian universities

Marco Geraci e Mauro Degli Esposti, submitted

Individuals factor map

International impact and visibility (PC1) +-

Nat

ion

al p

erf

orm

ance

(P

C2)

+-

International impact and visibility (PC1) +-

Nat

ion

al p

erf

orm

ance

(P

C2)

+-

Padova

Bologna

Roma 1

Torino

Milano

Politecnico MI

Napoli

Pavia

Trento

Perugia

Parma

average

average

Roma 2

Pisa

Firenze

Landscape of state Italian universities - all

Marco Geraci e Mauro Degli Esposti 2010, submitted…

Ferrara

Ferrara university – ranking among top institutes in Italy: before 15-12

+ 11 new TIS = 26Total TIS = 15

and after systematic analysis 31-12

Rank Italian Institution Sum of H-Index N. of TIS

1 Bologna 3019 712 CNR - global 2524 633 Milano 2113 464 Padova 2060 465 Roma 1713 416 Torino 1619 387 INAF - global 1420 388 San Raffaele Milano 1333 289 Firenze 1205 29

10 Ist Naz Tumori Milano 1106 2511 Pisa 1049 2312 Napoli 959 2313 Mario Negri Milano 931 2014 Genova 891 2015 Roma 2 880 2116 Perugia 804 1917 Pavia 752 1618 Brescia 610 1519 Ferrara 589 1520 Trieste 585 1321 Modena 489 1222 Normale Pisa 478 11

Rank Italian Institution Sum of H-Index N. of TIS

1 Bologna 3019 712 CNR, all 2524 633 Milano 2113 464 Padova 2060 465 Roma 1713 416 Torino 1619 387 INAF, all 1420 388 San Raffaele Milano 1333 289 Firenze 1205 29

10 Istit. Naz. Tumori Milano 1106 2511 Pisa 1049 2312 Ferrara 998 2613 Napoli 959 2314 Mario Negri Miano 931 2015 Genova 891 2016 Roma 2 880 2117 Perugia 804 1918 Pavia 712 1519 Brescia 610 1520 Trieste 585 1321 Modena 489 1222 Normale Pisa 478 11

Ferrara university – ranking among state universities: before

Total TIS = 15

Ranking = 15


+ 12 new TIS = 27

and after systematic analysis

Ranking = 10

M Degli Esposti, January 5, 2011 - retrieved from: http://www.topitalianscientists.org/Top_italian_scientists_VIA-Academy.aspx

Different Area – different impact(publications and citations)

•Normalization? By which criteria? •Which database to use for reference?

M Degli Esposti, September, 2010 – retrieved from: Manueletto del Ricercatore Provetto

NOT ISI WoS!

Scientometrics 73 (2007)

The normalization factorfollows a ‘stretched exponential’;it becomes bigger by increasing the number of published articles

For scientists with less than 50 published articles, the field-specific differences are small

NoteValori di h-index sonostati calcolati cosi’:

X, con Google Scholaradd-standard, con iniziale,ristretto all’area ‘egineering and mathematics’

Y, con ISI Web of Science,all years, ristretto ai settori matematici

Matematici con nomiCompositi (e.g. De Giorgi)sono esclusi - ISI non li trova!

Mauro Degli Esposti – Via-academy.org, 6-9-2010

Correlation h-index mathematicians, 24 names

y = 0.599

R2 = 0.83

0

10

20

30

40

0 10 20 30 40 50 60

h-index Google Scholar add-on

h-i

nd

ex

ISI W

oS

h-index - discrepancies between different databases

NOTA: usando Google Scholar si normalizza gia’ le citazioni dei matematicirispetto a quelle che si computano con ISI WoS.Questo risulta in una normalizzazione automatica di un fattore = 1.67,Simile al fattore di nromalizzatione proposto da Iglesias (2007) ca. = 1.83

Discrepancies between different databases for h-index: the problem with chemistry

Add-on

PorP

ISI WoS

Av

era

ge

h-i

nd

ex

Google Scholar

0

5

10

15

20

25

database

ISI

All chemists of Ferrara university

Correlation h-index for chemists

y = 1.379

R2 = 0.802n = 38

0

10

20

30

40

50

60

70

80

0 20 40 60

PorPIS

I Wo

S

Mauro Degli Esposti, 21-12-2010Correlation factor = 1.38

top areas in Ferrara

0

10

20

30

40

50

60

70

Cardiol

ogia

Patolo

gia

Anato

mia

Uman

a

Farmaco

logia

Istolo

gia ed

Em

briol

ogia

Med

icina

Inter

na

Micr

obiol

ogia

Scienze

Far

mace

utiche

Emato

logia

Anato

mia,

Isto

logia,

Cito

logia

Fisica

Chimica

Biolog

ia Evo

lutiv

a

Fisiolo

gi Um

ana

Scienze

dell

a Ter

ra

Mat

emat

ica

Info

rmat

ica

Average h-index Total citations Number of TIS

Mauro Degli Esposti, 5-1-2011

average h-index Ferrara departments

y = 0.9086xR

2 = 0.9325

0

5

10

15

20

25

30

35

40

0 5 10 15 20 25 30 35 40

PorP (initial)

Ad

d-O

n (

full

na

me

)

Fonte : National Science Board

Biomedicine: area variation of publication rateP

ub

lica

tion

s p

er

yea

r x

10

-3

Relative disease burden (Log %)

Correlation, RCorrelation, R2 2 = 0.72= 0.72 Neoplasms

6

4

2

%articles that arepatent-associated[

biomedical and clinical scientists in Ferrara

0

5

10

15

20

25

30

Anato

mia

Com

para

ta

Anato

mia

Um

ana

Anato

mia

, Isto

logia

, Cito

logia

Biochi

mica

Biochi

mica

Ese

rcizi

o Fisi

co

Biolog

ia A

mbi

enta

le

Biolog

ia C

el/Gen

etica

Mol

Biolog

ia E

volut

iva

Cardio

logia

Chirur

gia G

ener

ale

Clinica

Chir

urgic

a

Clinica

Der

mat

ologi

ca

Clinica

Med

ica

Clinica

Oste

tricia

Emat

olog

ia

Endoc

rinol

ogia

Farm

acol

ogia

Fisiolo

gia

e Biof

isica

Fisiolo

gia

Gener

ale

Fisiolo

gia

Uman

a

Genet

ica M

edica

Igie

ne

Istolo

gia

ed E

mbr

iolog

ia

Mal

attie

Infe

ttive

Med

icina

Inte

rna

Med

icina

Leg

ale

Med

icina

Nuc

leare

Micr

obiol

ogia

Nutriz

ione

clinic

a

Patolo

gia

Pedia

tria

average h-index

total citations x 10-3

5-1-2011

Ordine alfabetico

Fonte : OCSE

MERIT MERIT -- evaluationevaluation

Let’s the discussion begin!

Fondazione CRUI luglio 2010

CAF = Common AssessmentFramework

H-index, pagina 65 (su 85)

Evaluate university institutions and individuals:

The VIA-Ferrara model

A 3-step new evaluation model

Mauro Degli Esposti, 6 December 2010

TIME FRAME: end of 2010, beginning of 2011


The Manchester model


Total weightPublications = 40% of RQI


The Manchester model


Before RAE 2009:Category - weight Esteems 5Publications 75Environment* 25

PROBLEMS:1. Evaluation of Publication quality byInternal Committee – prone to politics!2. No consideration of international standing and individual’s scientific impact, e.g. h-index

*including research income

Research Quality Indicator = RQI


The VIA-Ferrara model

A 3-step new evaluation model

1. Via-academy •Quantitative evaluation of the scientific impact of all Ferrara scientists based on the systematic analysis of h-index.•Area-specific analysis of h-index using average values of top Italian scientists as reference

2. Ferrara All scientists to provide by e-mail:• a list with their top 4 papers in

the last 2 years;• the number of Ph.D. and Master

students supervised;• the total grant income obtained

from external sources.

TIME FRAME: end of 2010

3. Combine the analysis 1 with data in 2 to define the basic format (like in Manchester) and quantify individual profiles - with 1 at least counting for 50%


Questions?

Simple model for auto-evaluation in specific disciplines within a universitySimple model for auto-evaluation in specific disciplines within a university

Set a threshold of intrinsic (local) excellence by applying this formula:

Threshold = average + SD + 1 (in absolute value of h-index)

For example, Ferrara chemists:Threshold = 15.6 + 7.5 + 1 = 24

Italian scientists: home and away

0

10

20

30

40

50

60

70

rettori edirettori

biologi astrofisici economisti

h-i

nd

ex

me

dio

Nota: la barra corrisponde alla deviazione standard.

Media dei valori di h-index per gruppi di scienziati ed accademici italiani che lavorano in Italia. I valori sono simili a quelli degli Italiani che lavorano all’estero.

Mauro Degli Esposti – La Voce.it, 14-9-2010

Valore sogliaper entrare in lista

Top Italian scientists per regione Lombardia

Emilia-Romagna

Lazio

Toscana

Veneto

Piemonte

Friuli

Campania

Liguria

Umbria

Sicilia

Trentino-Alto Adige

Puglia

Marche

Sardegna

Calabria

Abruzzi

Su un totale di 1188 scienziati nella TIS al 15-12-2010,Di cui 834 in Italia – escluso CNR ed INAF


Mauro Degli Esposti Towards a new model for evaluating the merit of Italian academic scientists and their institutions Ferrara, Jan 2011.

Documents

italian scientists

manch slide

university of bologna

italian institutes

italian biochemists

italian institution

academy network

flavia amabile slide