Australia’s Research Assessment Arthur Sale Emeritus Professor of Computer Science University of Tasmania [email protected] .

Australia’s Research Assessment

Arthur Sale

Emeritus Professor of Computer ScienceUniversity of Tasmania

[email protected]://eprints.utas.edu.au/8695

History : RQF ERA

Australia had embarked on planning a national assessment of the research of its 39 universities – called the Research Quality Framework (RQF)

In November 2007, the Federal Government changed as the result of an election.

The incoming Government had promised a lower cost, simpler evaluation scheme – the ERA, announced February 2008.

UK Background - RAE

The UK has long operated a Research Assessment Exercise (RAE) which used extensive panel assessment, occurred at six-year intervals, and which determined the research funding of universities for six years.

The very last RAE was completed in 2008. The UK government announced a change of policy prior to that round of assessment.

UK Background - REF

The UK will replace the RAE by a mostly metric-based process, the Research Excellence Framework (REF).‘Assessment will combine quantitative indicators - including bibliometric indicators wherever these are appropriate - and expert review. Which of these elements are employed, and the balance between them, will vary as appropriate to each subject. The economic and social impact of research, as well as impact upon public policy, will also contribute to the overall assessment of quality.’

Ironic Fun!

RAE REF

RQF ERA

Excellence in Research for Australia

The RQF was based solidly on expert assessment of discipline groups like the RAE: expensive

ERA uses the experience of the UK’s RAE but does not copy it, and is biased towards metrics, with concessions to panel assessment where essential.

ERA is managed by the Australian Research Council (ARC) and reports to the Minister for Innovation, Industry, Science & Research.

Rolling assessment

In another major break with the RAE/REF, the ERA assessment will be implemented in a rolling stream two discipline clusters at a time, rather than all at once in a snapshot year.

Also the Minister has stated that assessment will not lead to direct financial outcomes for universities, though this remains to be tested.

Instead it will create data that could lead to ‘league tables’.

Background

Scholarly research is divided into eight discipline clusters (see next slide).

The groups are defined by the first four digits of the six digit Australian and New Zealand Standard Research Classification (ANZSRC).

All research outputs are classified by this scheme and have been for many years (under an older version). All Australian repositories use this scheme (not LC or Dewey).

Discipline clusters

1 Physical, Chemical and Earth Sciences (PCE) – 20102 Humanities and Creative Arts (HCA) – 20103 Engineering and Environmental Sciences (EE) – 2011?4 Social, Behavioural and Economic Sciences (SBE)5 Mathematics, Information and Communication Sciences

(MIC) – 2012?6 Biological Sciences and Biotechnology (BSB)7 Biomedical and Clinical Research (BCR) – 2013?8 Public and Allied Health and Health Services (PAHHS)

ANZSRC (sample) PrefaceFields of ResearchDIVISION 01 MATHEMATICAL SCIENCES

GROUP 0101 PURE MATHEMATICS GROUP 0102 APPLIED MATHEMATICS GROUP 0103 NUMERICAL AND COMPUTATIONAL MATHEMATICS GROUP 0104 STATISTICS GROUP 0105 MATHEMATICAL PHYSICS GROUP 0199 OTHER MATH SCIENCES

DIVISION 02 PHYSICAL SCIENCES GROUP 0201 ASTRONOMICAL AND SPACE SCIENCES GROUP 0202 ATOMIC, MOLECULAR, NUCLEAR, PARTICLE & PLASMA PHYSICS GROUP 0203 CLASSICAL PHYSICS GROUP 0204 CONDENSED MATTER PHYSICS GROUP 0205 OPTICAL PHYSICS GROUP 0206 QUANTUM PHYSICS GROUP 0299 OTHER PHYSICAL SCIENCES

DIVISION 03 CHEMICAL SCIENCES GROUP 0301 ANALYTICAL CHEMISTRY GROUP 0302 INORGANIC CHEMISTRY GROUP 0303 MACROMOLECULAR AND MATERIALS CHEMISTRY GROUP 0304 MEDICINAL AND BIOMOLECULAR CHEMISTRY GROUP 0305 ORGANIC CHEMISTRY GROUP 0306 PHYSICAL CHEMISTRY GROUP 0307 THEORETICAL AND COMPUTATIONAL CHEMISTRY GROUP 0399 OTHER CHEMICAL SCIENCEs

DIVISION 04 EARTH SCIENCES GROUP 0401 ATMOSPHERIC SCIENCES GROUP 0402 GEOCHEMISTRY GROUP 0403 GEOLOGY GROUP 0404 GEOPHYSICS GROUP 0405 OCEANOGRAPHY GROUP 0406 PHYSICAL GEOGRAPHY AND

ENVIRONMENTAL GEOSCIENCE GROUP 0501 ECOLOGICAL APPLICATIONS

Rolling implementation

The first two clusters to be assessed are– PCE = Physical, Chemical and Earth Sciences– HCA = Humanities and the Creative Arts (for

example literature, history, architecture, drama and music)

July 2009 will see a trial evaluation, with the ‘real ERA’ to commence in 2010. It is expected that two clusters will be assessed per year giving a four year cycle of evaluation.

2009 Trial

• PCE and HCA are extremes. PCE will be evaluated mainly by bibliometrics, whereas HCA won’t use bibliometrics at all but will use panel evaluation.

• Both will use applied research metrics and staffing data. This will provide a basic for future evaluations which will probably be mixed and more complex.

PCE SubmissionEach university will provide an XML document

describing its publication, grant and application record for the cluster. This will include a citation of each publication and its Scopus ID.

The ARC will look up each publication and compile metrics for the university’s cluster (the SEER system).

The panel will review the metrics and argue around the margins to arrive at an assessment.

Important Note

Evaluations are not by a research group, but by all the university’s research output in that discipline cluster.

Universities must also report their total FTE staff count in that cluster, and the researcher count (which includes adjunct and honorary staff). These are also taken into account in assessing quality. Research by adjunct staff is also counted.

Databases

After a tender process, Scopus was selected as the journal citation data supplier for PCE.

ARC has also created a public tier list of 21,000 journals, ranked by the various professional societies. The four tiers are– A* top 5% (top international standard)– A next 15% (excellent, but not in the top tier)– B next 30% (good, perhaps local, often PhD

outlets)– C remaining 50% (the rest)These will also be used to create metrics.

Ranked Tier List (sample)18134 B Software Testing Verification & Reliability 0960-0833 0803 COMPUTER SOFTWARE5270 B Soil & Tillage Research 0167-1987 0503 SOIL SCIENCES3554 B Soil Dynamics and Earthquake Engineering 0267-7261 0905 CIVIL ENGINEERING4383 B Soil Mechanics and Foundation Engineering 0038-0741 0905 CIVIL ENGINEERING5277 B Soil Use and Management 0266-0032 0503 SOIL SCIENCES

10943 B Sojourn 0217-9520 gen ss/humMULTIDISCIPLINARY - SOCIAL SCIENCES/HUMANITIES1084 B Solar Physics 0038-0938 0201 ASTRONOMICAL AND SPACE SCIENCES4839 B Soldering & Surface Mount Technology 0954-0911 0914 RESOURCES ENGINEERING AND EXTRACTIVE METALLURGY1141 B Solid State Communications 0038-1098 0206 QUANTUM PHYSICS1142 B Solid State Ionics 0167-2738 0206 QUANTUM PHYSICS1142 B Solid State Ionics 0167-2738 0306 PHYSICAL CHEMISTRY (INCL. STRUCTURAL)1143 B Solid State Nuclear Magnetic Resonance 0926-2040 0202 ATOMIC, MOLECULAR, NUCLEAR, PARTICLE AND PLASMA PHYSICS1143 B Solid State Nuclear Magnetic Resonance 0926-2040 0206 QUANTUM PHYSICS1144 B Solid State Phenomena 1012-0394 0206 QUANTUM PHYSICS1144 B Solid State Phenomena 1012-0394 0912 MATERIALS ENGINEERING7701 B Soudobe Dejiny 2103 HISTORICAL STUDIES 3718 B Sound and Vibration 1541-0161 0913 MECHANICAL ENGINEERING8253 B South African Archaeological Bulletin 0038-1969 2101 ARCHAEOLOGY7702 B South African Historical Journal 0258-2473 2103 HISTORICAL STUDIES

18235 B South African Journal of Economics 0038-2280 14 ECONOMICS20079 B South African Journal of Education 13 EDUCATION1861 B South African Journal of Geology 1012-0750 0403 GEOLOGY

31493 B South African Journal of Italian Studies 2005 LITERARY STUDIES10178 B South African Journal of Musicology 1904 PERFORMING ARTS AND CREATIVE WRITING21712 B South Asia Research 0262-7280 gen ss/humMULTIDISCIPLINARY - SOCIAL SCIENCES/HUMANITIES12186 B South Atlantic Review 0038-2868 2005 LITERARY STUDIES30605 B South Carolina Historical Magazine 2103 HISTORICAL STUDIES 12187 B South Central Review 2005 LITERARY STUDIES33794 B South Dakota Law Review 1801 LAW

HCA Submission

Like PCE, the university will provide an XML submission, but each publication or piece of research output will be linked (via URL) to a document in the university’s repository.

The panel will evaluate the research outputs on SEER from face evidence and look at a nominal 20% of the documents.

This will form part of the HCA assessment.

System to Evaluate the Excellence of Research

The ARC’s SEER software provides the central glue to make this all work.

It will automatically extract data from the XML submissions and databases and create summary metric data for review panels.

Expert reviewers will access the university repositories (eg HCA) through SEER which

(a) will anonymize them, and (b) has permission to look at restricted documents.

Sampled periodsIt is expected that for the 2010 ERA evaluations the census date

and the dates for the reference periods will move forward one year compared to the trial. That is, the staff census date would be 31 March 2009, and the reference periods would be as follows:

Research outputs - 1 January 2003–31 December 2008 (6 years)Citation analysis - 1 January 2003–1 March 2010 (6 years)Research income - 1 January 2006–31 December 2008 (3 years)Applied measures - 1 January 2006–31 December 2008 (3 years)

Summary

• ERA is planned to be low-cost and rolling.• It is adapted to clusters, and within them to

disciplines.• Extensive consultation is taking place• It will reshape Australian research• I am sure that Australia is willing to share its

experiences with any other country.• Read the documentation, watch the space...

By now you probably deserve a picture of me. I am really sorry not to have been with you.

Best wishes, Arthur

Questions?

© Copyright 2009 Arthur Sale

All rights reserved

Arthur Sale asserts the right to be recognized as author of this work

Contact: [email protected]

Links

• ERA (Aus)• ANZSRC• REF (UK)