ACS 243rd National Meeting San Diego, CA Mar 28, 2012 Atyab Tahir SciVal Consultant A & G Institutions www.elsevier.com The Next Era of Research Productivity Evaluation: The Multi-Dimensional Research Assessment Framework
ACS 243rd National Meeting
San Diego, CA
Mar 28, 2012
Atyab Tahir
SciVal Consultant
A & G Institutions
www.elsevier.com
The Next Era of Research Productivity Evaluation:
The Multi-Dimensional Research Assessment
Framework
Who we are: A global company based in Amsterdam, employing 7,000+ people in 24 countries Partners: A global community of 7,000 journal editors, 70,000 editorial board members, 300,000 reviewers and 600,000 authors In the community: We are a founding publisher of global programs that provide free or low-‐cost access to science and health information in the developing world History: Elsevier's roots are in journal and book publishing, where we have fostered the peer-‐review process for more than 125 years. Products: Elsevier publishes around 2,000 journals and close to 20,000 books and major reference works as well as a variety of scholarly databases and other solutions.
Elsevier serves more than 30 million scientists, students and health and information professionals worldwide.
Elsevier at a glance
2
A framework to characterize and position bibliometric indicators and methods
3
The multi-dimensional research assessment framework Top-down versus bottom-up delimitation methods Three generations of bibliometric indicators
The future of research assessment exercises lies in the intelligent combination of metrics and peer review
THE MULTI-DIMENSIONAL RESEARCH ASSESSMENT FRAMEWORK
Source: Expert Group on the Assessment of University-Based Research (AUBR, 2010)
Multi dimensional Research Assessment Framework Unit of assessment
Purpose Output dimensions
Bibliometric indicators
Other indicators
Individual Allocate resources
Research productivity
Publications Peer review
Research group
Improve performance
Quality, scholarly impact
Journal citation impact
Patents, licences, spin offs
Department Increase multi-discipl. research
Innovation and social benefit
Actual citation impact
Invitations for conferences
Institution Increase regional engagement
Sustainabi-lity & Scale
Internat. co-authorship
External research income
Research field
Promotion, hiring
Research infrastruct.
citation
PhD com-pletion rates
READ COLUMN
WISE
Multi dimensional Research Assessment Framework Unit of assessment
Purpose Output dimensions
Bibliometric indicators
Other indicators
Individual Allocate resources
Research productivity
Publications Peer review
Research group
Improve performance
Quality, scholarly impact
Journal citation impact
Patents, licences, spin offs
Department Increase multi-discipl. research
Innovation and social benefit
Actual citation impact
Invitations for conferences
Institution Increase regional engagement
Sustainabi-lity & Scale
Internat. co-authorship
External research income
Research field
Promotion, hiring
Research infrastruct.
citation
PhD com-pletion rates
7
Types of Research Output
Impacts Publication/text Non-publication
Scientific-scholarly Journal paper; book chapter; monograph
Research data file; video of experiment
Educational Teaching course book; syllabus Skilled researchers
Economic Patent Product; process; device; design; image; spin off
Cultural Newspaper article Interviews; events; performances; exhibits
Example 1
Individual Hiring/promotion Productivity &
impact PhD date, place,
supervisor; Invitations for
conferences
Publications in international jrnls;
Actual citation impact
Example 2
Research group Allocate resources Res. productivity &
impact Competitive
research income; Ratio research
active/total staff
Publications in international jrnls;
Actual citation impact
Example 3
Department Increase regional
engagement Social benefits &
innovation % Funding from
endusers; Licenses, spin
offs
% Industry co-authored papers;
Patents Res. reports
Example 4
Institution Increase multi-
disciplinary research Res. productivity &
impact Inter-discipl.
teaching progr; Funding from
multi-disc. res. progr
% Papers in multi-disciplinary fields;
Co-authorships from different discipl
Assessment Solutions Unit of assessment
Purpose Aspect SciVal Solutions
Research group
Improve performance; Allocate resources
Productivity/ impact
Strata
Department; institution
Improve performance; Allocate resources;
Research infrastructure and potential
Experts
Institution Stimulate multi-disciplinary research; Improve performance;
Social benefits; sustainability; Prod./impact
Spotlight
Group, Dept, Institution
Improve performance & regional engagemt; Allocate resources;
Productivity/ impact; social impact; infrastr.
Custom Analytics
TOP-DOWN VS. BOTTOM-UP INSTITUTIONAL DELIMITATION
Top-down institutional analysis
author affiliations (incl. verification)
Categorize articles into research fields
Calculate indicators
Compare with benchmarks
Bottom-up institutional analysis
Compile a list of researchers
Compile a list of publications per researcher (incl. verification)
Aggregate researchers into groups, departments, fields, etc.
Calculate indicators; compare with benchmarks
TOP-DOWN VS. BOTTOM-UP SUBJECT FIELD DELIMITATION
Top-down field analysis
(Pre-‐)define a field categorization
(Pre-‐)allocate papers (journals) to categories
Calculate indicators per category
Bottom-up field analysis
Create a (large) dataset of publications
Cluster papers on the basis of
(co-‐) citation or word analysis
Name the clusters
Calculate indicators by cluster
Chemistry Competency Map of US
22
THREE GENERATIONS OF BIBLIOMETRIC INDICATORS
Three generations of bibliometric indicators
1 Basic No. publications No. Citations Cites per paper (e.g., Jrnl Impact Factor)
2 Normalized Relative Citation Rate No. publ. in top citation percentiles
3 Contextual Entire citation network based (e.g., Eigenfactor, SJR, Jrnl Prestige); Citation context based
ACS 243rd National Meeting
San Diego, CA
Mar 28, 2012
Atyab Tahir
SciVal Consultant
A & G Institutional Markets
www.elsevier.com
The Next Era of Research Productivity Evaluation:
The Multi-Dimensional Research Assessment Matrix
Thank You!