RESEARCH EVALUATION WORKSHOP UNITED KINGDOM OCTOBER 2010
Jan 17, 2016
RESEARCH EVALUATION WORKSHOP
UNITED KINGDOM
OCTOBER 2010
WHY EVALUATE RESEARCH PERFORMANCE?
• Quantitative analysis is the main tool of science
• Communicating research results complex
• Personal knowledge no longer sufficient for decision making
• Need to be selective over support for research projects
• Peer review was foundation of policy decisions– Library collection decisions
– Foundation allocating limited funding
– Government office weighing national research needs
WHY EVALUATE RESEARCH PERFORMANCE?
• Evaluation and strategic planning– Periodic evaluation of research performance
– Institution, departmental or researcher level assessments• Accreditation, tenure, faculty review
– Performance indicators• Used in strategic planning
• Reporting to government bodies, boards of directors/trustees
• Research Centers– Find new staff
– Develop lines of investigation
– Compete for funds
HOW IS RESEARCH EVALUATED?
• Research– Volume, income, reputation
• Prestigious awards – Nobel Prizes
• Innovation– Industry income and patents
• Teaching– Academic Reputation Survey, higher degrees
• International Mix– National / International staff and students
• Citation analysis– Normalised for volume and subject area
• Peer Evaluation– Reputational survey
HOW IS RESEARCH EVALUATED?
5
HOW IS RESEARCH EVALUATED?
6
HOW IS RESEARCH EVALUATED? SOUTH AFRICA
7
THE GROWING USE OF BIBLIOMETRICS
• Nations with significant science enterprises have embraced bibliometrics
• Today, bibliometrics programs with large teams of analysts are firmly established in many nations– These groups issue bibliometric reports, often called
science indicators studies, at regular intervals
• In almost all cases, the publication and citation data of Thomson Reuters form the basis of their bibliometric analyses
INSTITUTIONS USING WEB OF SCIENCE CITATION DATA FOR EVALUATION (INCL.)
• United Kingdom: KCL, HEFCE, St. Andrew’s
• Germany: IFQ, Max Planck Society, DKFZ, MDCUS
• Netherlands: NWO & KNAW
• France: Min. de la Recherche, OST - Paris, CNRS
• European Union: EC’s DGXII(Research Directorate)
• US: NSF: biennial Science & Engineering Indicators report (since 1974)
• Canada: NSERC, FRSQ (Quebec), Alberta Research Council
• Australian Academy of Science, gov’t lab CSIRO
• Japan: Ministry of Education, Ministry of Economy, Trade & Industry
• People’s Republic of China: Chinese Academy of Science
• Multiple rankings agencies
9
THE DATA
A BRIEF HISTORY OF THE CITATION INDEX
• Concept first developed by Dr Eugene Garfield – Science, 1955
• The Science Citation Index (1963)– SCI print (1960’s)– On-line with SciSearch in the 1970’s – CD-ROM in the 1980’s– Web interface (1997) Web of Science
• Content enhanced:– Social Sciences Citation Index (SSCI)– Arts & Humanities Citation Index (AHCI)
• The Citation Index – Primarily developed for purposes of information retrieval– Development of electronic media and powerful searching tools
have increased its use and popularity for purposes of Research Evaluation
THE VALUE OF A CITATION
• Why do people cite?– Pay homage / give credit to pioneer
– Identifying a methodology
– Provide background reading
– Quotations
– Authenticating data, reproducing work etc
– Corrections
– Criticizing/Disclaiming someone's work/opinions
• Citations are an indicator of an article’s impact and usefulness to the research community; they are the mode by which peers acknowledge each other’s research.
• The value of a citation is only as important as its source. – Clearly a citation from a prestigious peer review journal has more value than a
citation from non-scholarly material.
– How can you be sure that the citing source is reputable?
“When to Cite”, E. Garfield, Library Quarterly, v66, p449-458, 1996
WHY NOT INDEX ALL JOURNALS?
0
20
40
60
80
100
120
0 1000 2000 3000 4000 5000 6000
# of journals
% o
f dat
abas
e
Articles Citations
40% of the journals:
• 80% of the publications
• 92% of cited papers
4% of the journals:
• 30% of the publications
• 51% of cited papers
HOW DO WE DECIDE WHICH JOURNALS TO INDEX?
• Approx. 2.500 journals evaluated annually
– 10-12% accepted
• Thomson Reuters editors– Information professionals
– Librarians
– Experts in the literature of their subject area
Web of Science
Journals under evaluation
Journal ‘quality’
THOMSON REUTERSJOURNAL SELECTION POLICY
• Publishing Standards– Peer review, Editorial conventions
• Editorial content– Addition to knowledge in specific subject field
• Diversity– International, regional influence of authors, editors, advisors
• Citation analysis– Editors and authors’ prior work
Region # Journals from Region in Web of Science
Europe 5,573 49%
North America 4,251 38%
Asia-Pacific 965 9%
Latin America 272 2%
Middle East/Africa 200 1%
Language # Journals in Web of Science
English 9114 81%
Other 2147 19%
GLOBAL RESEARCH REPRESENTATIONWEB OF SCIENCE COVERAGE
Full range of scholarly research disciplines
Adheres to a consistent selection policy Ensures that publications and citations are comparable
Consistent indexing Cover-to-cover indexing
All author names
All author addresses
CONSISTENCY IS THE KEY TO VALIDITY- COMPARE APPLES WITH APPLES
Government agencies/funding organizations
IndividualsFaculty, staff, students
University DepartmentsInstitutional research, academic
affairs, tech transfer, etc.
External Entities
University ManagementManagement, including committees,
provost, vice provosts
PRIMARY USERS OF CITATION DATA INRESEARCH EVALUATION
Government agencies/funding organizations
External Entities
EXTERNAL ENTITIESRESEARCH EVALUATION
– Higher Education Funding Council for England, UK
– National Science Foundation (USA)
– European Commission (EU)
– L’Observatoire des Sciences et des Techniques (OST )
– National Institute for Science and Technology Policy (NISTEP ), Japan
– Human Sciences Research Council, South Africa
University ManagementManagement, including committees,
provost, vice provosts
UNIVERSITY MANAGEMENTRESEARCH EVALUATION
INSTITUTIONAL LEVELRESEARCH EVALUATION
University ManagementManagement, including
committees, provost, vice provosts
Source: Thomson Reuters U.S. and Canadian University Science Indicators
Number of citations to North American scientific papers
NUMBER OF RESEARCHERS BY DEPARTMENT AND ROLE
PRODUCTIVITY BY UNIVERSITY DEPARTMENT
Allows user to analyse output and performance based on their institution’s departments
University Departments
Institutional research, academic affairs, tech
transfer, etc.
SUBJECT CATEGORY LEVELRESEARCH EVALUATION
DOCUMENT TYPE BY DEPARTMENT
IN WHICH JOURNALS HAS OUR CHEMISTRY DEPT. PUBLISHED?
HOW HAVE THOSE PAPERS PERFORMED?
27
HOW MANY CITATIONS HAS THE CHEMISTRY DEPT. RECEIVED?
28
IN WHICH JOURNALS ARE THOSE CITATIONS PUBLISHED?
29
Individuals
Faculty, staff, students
INDIVIDUAL LEVELRESEARCH EVALUATION
EVALUATING INDIVIDUALS
Nancy J Rothwell, DBE, FRS
President & Chancellor, Univ Manchester
Dame Commander Order of the British Empire
Fellow of the Royal Society
Research Chair, Medical Research Council
EVALUATING INDIVIDUALS
32
• Number of articles : 485
• Sum of the Times Cited : 18,943
• Average Citations / Item : 39.06
• h-index : 70
WHO CITED THIS AUTHOR’S RESEARCH?
A very international profile illustrating the global impact of Prof. Rothwell’s research
EVALUATING INDIVIDUALSINDIRECT INFLUENCE
Full range of scholarly research disciplines
Adheres to a consistent selection policy Ensures that publications and citations are comparable
Consistent indexing Cover-to-cover indexing
All author names
All author addresses
CONSISTENCY IS THE KEY TO VALIDITY