June 2006 How good is our research? New approaches to research indicators.

June 2006

How good is our research?

New approaches toresearch indicators

June 2006

Average is a metric; distribution is a picture

• ‘Average impact’ is a good bibliometric index but not sufficient– A tool for reporting but not for action

• Data are skewed, so average is not central– Many papers are uncited and a few papers are very highly cited

• New approach looks at where the spread of performance falls– Activity is located within distribution by more than a single metric

– Thresholds help in describing peak of performance.

• This improves descriptive power, information content and management value

June 2006

Traditional impact indicatorsRebased average impact (world = 1.0) for G8 nations

0.2

0.4

0.6

0.8

1.0

1.2

1.4

1.6

1995 1996 1997 1998 1999 2000 2001 2002 2003 2004

Year

Reb

ased

Imp

act

UK

USA

CANADA

FRANCE

GERMANY

ITALY

JAPAN

RUSSIA

EUROPEAN UNION

Data: Thomson ISI® National Science Indicators

June 2006

Distribution of “research performance”

Performance measure of 'research quality'

Fre

qu

ency

of

ach

ieve

me

nt

Benchmark e.g. 'world average'

Threshold of 'excellence'

June 2006

Distribution of “research performance”

Performance measure of 'research quality'

Fre

qu

ency

of

ach

ieve

me

nt

Benchmark e.g. 'world average'

Threshold of 'excellence'

A

B

Overall volume A = B

Average quality A = B

Activity above threshold B > AA good ‘indicator’ should capture and reflect this in some meaningful way

Do current metrics do this?

June 2006

Distribution of data values - income

0

5

10

15

20

Income category

Fre

qu

ency

Income per FTE Gross income

RAE2001 - research income for units in UoA14 Biology

£10m per unit

£250k per FTE

MaximumMinimum

June 2006

0

100

200

300

400

Impact category (normalised to world average)

Fre

qu

ency

UK Physics papers for 1995 = 2323

World average

Maximum0

Distribution of data values - impact

The variables for which we have metrics are skewed and therefore difficult to picture in a simple way

June 2006

Simplifying the data picture

• Scale data relative to a benchmark, then categorise– Could do this for any data set

• All journal articles– Uncited articles (take out the zeroes)

– Cited articles• Cited less often than benchmark• Cited more often than benchmark

– Cited more often but less than twice as often– Cited more than twice as often

» Cited less than four times as often» Cited more than four times as often

June 2006

Categorising the impact data

All papers

Uncited papers

Cited papers .

Papers cited less often than

benchmark Papers cited more often than benchmark

Papers cited more than

benchmark, but less than four times as often

Papers cited more than four times as

often as benchmark

= 0 >0 >0.125 >0.25 0.5 < 1 1 < 2 2 < 4 4 < 8 > 8

This grouping is the equivalent of a log 2 transformation. There is no place for zero values on a log scale.

June 2006

UK ten-year profile 680,000 papers

0

5

10

15

20

25

RBI = 0 RBI >0 - 0.125 RBI 0.125 - 0.25 RBI 0.25 - 0.5 RBI 0.5 - 1 RBI 1 - 2 RBI 2 - 4 RBI 4 - 8 RBI > 8

Per

ce

nta

ge

of

ou

tpu

t 1

995

-200

4

% of UK output over decade

AVERAGERBI = 1.24

MODE (cited)

MEDIAN

THRESHOLD OF EXCELLENCE?

MODE

June 2006

Implications

• Is the UK as good as we thought?– YES - the average is unchanged– What lies beneath just became apparent

• The effective peak is very concentrated

– Other countries would probably look similar

• New metrics are needed– Average impact not indicative of distribution– Need to add median, mode– Proportion of activity at thresholds of excellence

• Above world average, More than 4 x world average, etc

• Evaluate methodology– Does it work by year and by subject?– How can we apply it?

June 2006

Time profile

0

5

10

15

20

25

30

RBI = 0 RBI >0 - 0.125 RBI 0.125 - 0.25 RBI 0.25 - 0.5 RBI 0.5 - 1 RBI 1 - 2 RBI 2 - 4 RBI 4 - 8 RBI > 8

Group

Per

cen

tag

e o

f o

utp

ut

for

per

iod

1995-1997 1998-2000 2001-2003

June 2006

Subject based curvesUK - Biology & Biochemistry

0

5

10

15

20

25

RBI = 0 RBI 0 - 0.125 RBI 0.125 - 0.25 RBI 0.25 - 0.5 RBI 0.5 - 1 RBI 1 - 2 RBI 2 - 4 RBI 4 - 8 RBI > 8

Per

cen

tag

e o

f o

utp

ut

1995

-200

4

% of UK for Year and Subject

UK - Chemistry

0

5

10

15

20

25

RBI = 0 RBI 0 - 0.125 RBI 0.125 - 0.25 RBI 0.25 - 0.5 RBI 0.5 - 1 RBI 1 - 2 RBI 2 - 4 RBI 4 - 8 RBI > 8

Per

cen

tag

e o

f o

utp

ut

1995

-200

4

% of UK for Year and Subject

Median RBI - 0.627

UK - Clinical Medicine

0

5

10

15

20

25

RBI = 0 RBI 0 - 0.125 RBI 0.125 - 0.25 RBI 0.25 - 0.5 RBI 0.5 - 1 RBI 1 - 2 RBI 2 - 4 RBI 4 - 8 RBI > 8

Per

cen

tag

e o

f o

utp

ut

1995

-200

4

% of UK for Year and Subject

Median RBI - 0.450

UK - Social Sciences, general

0

5

10

15

20

25

30

35

40

RBI = 0 RBI 0 - 0.125 RBI 0.125 - 0.25 RBI 0.25 - 0.5 RBI 0.5 - 1 RBI 1 - 2 RBI 2 - 4 RBI 4 - 8 RBI > 8

Per

cen

tag

e o

f o

utp

ut

1995

-200

4

% of UK for Year and Subject

Median RBI - 0.372

June 2006

Subject & site profiles – molecular biology

0

5

10

15

20

25

30

RBI = 0 RBI >0 - 0.125 RBI 0.125 - 0.25 RBI 0.25 - 0.5 RBI 0.5 - 1 RBI 1 - 2 RBI 2 - 4 RBI 4 - 8 RBI > 8

Pe

rce

nta

ge

of

ou

tpu

t 1

99

5-2

00

4

% of UK for subject and time period shown as a

smoothed line

UK average shown as red

symbol

June 2006

HEIs – 10 year totals - 1

0

5

10

15

20

25

30

RBI = 0 RBI >0 - 0.125 RBI 0.125 - 0.25 RBI 0.25 - 0.5 RBI 0.5 - 1 RBI 1 - 2 RBI 2 - 4 RBI 4 - 8 RBI > 8

Per

cen

tag

e o

f o

utp

ut

1995

-200

4

Leading research university Big civic 'Robbins' type university Former Polytechnic

June 2006

HEIs – 10 year totals - 2

0

5

10

15

20

25

30

RBI = 0 RBI >0 - 0.125 RBI 0.125 - 0.25 RBI 0.25 - 0.5 RBI 0.5 - 1 RBI 1 - 2 RBI 2 - 4 RBI 4 - 8 RBI > 8

Per

ce

nta

ge

of

ou

tpu

t 1

995

-200

4

Leading research university Big civic 'Robbins' type university Former Polytechnic

June 2006

HEIs – 10 year totals – 4.1

0

5

10

15

20

25

30

RBI = 0 RBI >0 - 0.125 RBI 0.125 - 0.25 RBI 0.25 - 0.5 RBI 0.5 - 1 RBI 1 - 2 RBI 2 - 4 RBI 4 - 8 RBI > 8

Per

cen

tag

e o

f o

utp

ut

1995

-200

4

Leading research university Big civic 'Robbins' type university Former Polytechnic

Smoothing the lines would reveal the shape

of the profile

June 2006

HEIs – 10 year totals – 4.2

Absolute volume would add a further element

for comparisons

0

5

10

15

20

25

30

RBI = 0 RBI >0 - 0.125 RBI 0.125 - 0.25 RBI 0.25 - 0.5 RBI 0.5 - 1 RBI 1 - 2 RBI 2 - 4 RBI 4 - 8 RBI > 8

Per

cen

tag

e o

f o

utp

ut

1995

-200

4

Leading research university Big civic 'Robbins' type university Former Polytechnic

June 2006

HEIs – 10 year totals – 4.3

0

2000

4000

6000

8000

10000

RBI = 0 RBI >0 - 0.125 RBI 0.125 - 0.25 RBI 0.25 - 0.5 RBI 0.5 - 1 RBI 1 - 2 RBI 2 - 4 RBI 4 - 8 RBI > 8

Nu

mb

ers

of

arti

cles

199

5-20

04

June 2006

What next?

• Profiles– Create a view of the distribution of performance– Provide more information useful to management– Require a change in metrics

• Applications– Disaggregate the components of the research base– Track institutional profiles against benchmark– Evaluate the link between platform and peak– Track papers through time: e.g. leaders vs. climbers

June 2006

How good is our research?

New approaches toresearch indicators