Staying Ahead of The Curve: Rethinking Research Performance Yukun Harsono Vice President Product Marketing IREG-4 Astana, Republic of Kazakhstan
Jan 20, 2016
Staying Ahead of The Curve:Rethinking Research Performance
Yukun Harsono
Vice President Product MarketingIREG-4
Astana, Republic of Kazakhstan
2
LEAN TIMES ARE HERE
3
THE NEW REALITY…
4
OTHER FORCES THAT SHAPE LEAN RESEARCH
Trend Exacerbated by Economic Downturn
GOVERNMENT POLICIES ON RESEARCH
Research Program Sponsor Description
Governments are actively guiding their national research agenda
Assess the quality of research in universities and colleges in the UK
Enable funding bodies to determine how to allocate grants across research projects
Detail by institution and by discipline those areas that are internationally competitive, together with emerging areas
Identify thematic domains for future European support
Part of EU’s strategy to become “the most dynamic competitive knowledge-based economy in the world”
................................................................................................................................................................................................................................................
.
5
................................................................................................................................................................................................................................................
.
6
SCOPUS POWERING PERFORMANCE ANALYSIS
7
CWTS found that, in general, Scopus covers much of the same material as the Web of Science (WoS) and ISI Proceedings and in some areas provides a higher level of coverage. Overall, the report found that Scopus should be considered as a potential alternative to the WoS for the purposes of the REF.
“The overall percentage of WoS-covered, science-related papers found in Scopus increased over the years, from 89 per cent in 1996 to 97 per cent in 2005. In other words, in these fields and for published articles and reviews the WoS now constitutes almost a complete subset of Scopus.”
CWTS found that, in general, Scopus covers much of the same material as the Web of Science (WoS) and ISI Proceedings and in some areas provides a higher level of coverage. Overall, the report found that Scopus should be considered as a potential alternative to the WoS for the purposes of the REF.
“The overall percentage of WoS-covered, science-related papers found in Scopus increased over the years, from 89 per cent in 1996 to 97 per cent in 2005. In other words, in these fields and for published articles and reviews the WoS now constitutes almost a complete subset of Scopus.”
"Science and technology play a crucial role in a country's economic growth," said Mr. Hiroyuki Tomizawa, Principal Administrator, Economic Analysis and Statistics Division, Directorate for Science, Technology and Industry, with the OECD. "We selected Scopus for its breadth of coverage including journal titles from over 100 nations as well as its advanced features. Together, these advantages will enable the OECD to execute more sophisticated statistical analyses to guide our member countries."
"Science and technology play a crucial role in a country's economic growth," said Mr. Hiroyuki Tomizawa, Principal Administrator, Economic Analysis and Statistics Division, Directorate for Science, Technology and Industry, with the OECD. "We selected Scopus for its breadth of coverage including journal titles from over 100 nations as well as its advanced features. Together, these advantages will enable the OECD to execute more sophisticated statistical analyses to guide our member countries."
Using Scopus in 2007 has resulted in our being able to track for 127 more institutions than in 2006 resulting in better showing for many universities. A better representation from non-English language journals has led to better results from many countries where English is not the first language
Using Scopus in 2007 has resulted in our being able to track for 127 more institutions than in 2006 resulting in better showing for many universities. A better representation from non-English language journals has led to better results from many countries where English is not the first language
RESEARCH ASSESSMENT EXERCISEHIGHER EDUCATION FUNDING COUNCIL FOR ENGLAND (HEFCE)
Using Scopus in 2007 has resulted in our being able to track for 127 more institutions than in 2006 resulting in better showing for many universities. A better representation from non-English language journals has led to better results from many countries where English is not the first language
Using Scopus in 2007 has resulted in our being able to track for 127 more institutions than in 2006 resulting in better showing for many universities. A better representation from non-English language journals has led to better results from many countries where English is not the first language
"Science and technology play a crucial role in a country's economic growth," said Mr. Hiroyuki Tomizawa, Principal Administrator, Economic Analysis and Statistics Division, Directorate for Science, Technology and Industry, with the OECD. "We selected Scopus for its breadth of coverage including journal titles from over 100 nations as well as its advanced features. Together, these advantages will enable the OECD to execute more sophisticated statistical analyses to guide our member countries."
"Science and technology play a crucial role in a country's economic growth," said Mr. Hiroyuki Tomizawa, Principal Administrator, Economic Analysis and Statistics Division, Directorate for Science, Technology and Industry, with the OECD. "We selected Scopus for its breadth of coverage including journal titles from over 100 nations as well as its advanced features. Together, these advantages will enable the OECD to execute more sophisticated statistical analyses to guide our member countries."
CWTS found that, in general, Scopus covers much of the same material as the Web of Science (WoS) and ISI Proceedings and in some areas provides a higher level of coverage. Overall, the report found that Scopus should be considered as a potential alternative to the WoS for the purposes of the REF.
“The overall percentage of WoS-covered, science-related papers found in Scopus increased over the years, from 89 per cent in 1996 to 97 per cent in 2005. In other words, in these fields and for published articles and reviews the WoS now constitutes almost a complete subset of Scopus.”
CWTS found that, in general, Scopus covers much of the same material as the Web of Science (WoS) and ISI Proceedings and in some areas provides a higher level of coverage. Overall, the report found that Scopus should be considered as a potential alternative to the WoS for the purposes of the REF.
“The overall percentage of WoS-covered, science-related papers found in Scopus increased over the years, from 89 per cent in 1996 to 97 per cent in 2005. In other words, in these fields and for published articles and reviews the WoS now constitutes almost a complete subset of Scopus.”
8
AUSTRALIAN RESEARCH FRAMEWORK
Publication Data for National Assessment
Elsevier provides citation information for the Excellence in Research for Australia (ERA) initiative Cluster One evaluation
Refine affiliation data: “In order for the ARC to process publication records for citation analysis, all publication outputs must be submitted with a Scopus unique article identifier. This allows the ARC to match the citation numbers for each publication outputs quickly and accurately post-submission.”
9
GLOBAL RESEARCH LANDSCAPE CHANGING
Countries ranked by published output in 2007
10
GLOBAL RESEARCH LANDSCAPE CHANGING
Countries ranked by output growth 1997-2007
Percent
11
RESEARCH RESPONDS TO GLOBALIZATION
Research now...
Crosses national boundaries Exhibits high mobility of resources, people,
ideas, technologies and infrastructure Overarches multi-jurisdictional regulation Responds and reacts to public and private
pressures Networks energetically in real and virtual
space/time Balances on the knife edge of Competition
and Collaboration
Source: Prof. Daryl Le Grew, University of Tasmania
12
RESEARCH MORE EXCITING BUT ALSO MORE CHALLENGING
Q: How many hours per week searching and gathering information?
13
Researchers spend more time looking for information than analyzing and applying it
Q: How many hours per week organizing, analyzing and applying information
Source: 2007 Survey by Outsell Inc. on 6,300 knowledge workers
A:
FUNDING PRESSURES
Q: What do you estimate is the approval rate for National Science Foundation grant applications by new researchers?
14
Source: American Academy of Arts and Sciences (www.scienceprogress.org)
Competition for funding is intense and will continue to intensify
Q: What do you estimate to be the average age when biomedical researchers receive their first grant from the National Institute of Health?
A:
15
Increased focus on EFFECTIVENESS & EFFICIENCY
IMPROVE PERFORMANCE across all workflows that make up the
research enterprise
16
RESEARCH EXECUTIVES: KEY CHALLENGES
Need constant flow of information and intelligence about internal performance, benchmarks and comparative performance
Require matrix of tools linking:- Reputation and Ranking- Growth and Development
with - Research Performance - Research Strategy
Implications “Have my strategic decisions
been effective?” “Are we capitalizing on new hot
areas e.g. stem-cells?” “Where should my strategic
focus be?” “Who are my true competitors
per competency?” “How is my competency
portfolio performing?”
Questions
Source: Prof. Daryl Le Grew, University of Tasmania
HOW WE MEASURE TODAY
Total number of papers Total number of citations Average number of citations per author Average number of papers per author Thomson Reuters’ Impact Factor Bergstrom’s Eigenfactor Egghe's g-index SCImago’s Journal Rank Hirsch's h-index
17
18
SAMPLE MEASURE: PUBLICATION OUTPUT
Drawbacks Bias against smaller institutions Quantity ≠ Quality
19
SAMPLE MEASURE: REFERENCE VS OUTPUT
Drawbacks Inter-disciplinary differences Not granular enough
20
NEW MEASURE FOR A NEW REALITY
New bottom-up approach needed for identifying scientific leadership
EXISTING APPROACH TO PERFORMANCE MEASUREMENT
ISSUES
Journal-based classification
Assign journals to categories
Science categorized into small number of fields and subfields (i.e. highly aggregated classification system)
Example: National Science Foundation groups science into 13 fields and 127 subfields
Number and scope of journals too limitingRisk: Too few journals included in analysis & potential for English-speaking bias in journal selection
Level of aggregation too high-levelRisk: Limited view of performance data on the level where it counts – that of school or large focused laboratory
Inter-disciplinary boundaries changingRisk: Not understanding how different disciplines interact with one another on the institutional level
Risk: Not seeing new emerging research areas
21
A NEW APPROACH
22
Partitioning the literature
This literature is partitioned into ~40,000 highly detailed subject areas.
Data
We start with the Scopus literature database, which indexes over 1.2 million articles per year, and covers a broad set of scientific areas.
Brain Research
Biotechnology
Earth Sciences
Math & PhysicsChemistry
Biology
Infectious DiseaseMedical SpecialtiesHealth Services
Social Sciences
Engineering
Computer Science & EE
Humanities
# of papers
23
Additional information
Strengths can be identified for: Universities Regions Nations
Details about strengths: Market shares Growth rates Top competitors Top researchers
Identifying areas of strength
24
DISTINCTIVE COMPETENCY #26
25
Math & PhysicsChemistry (1)
Earth Sciences
Biology
Biotechnology
Infectious Disease (1)
Medical Specialties (27)
Health Services (1)Brain Research (2)
Social Sciences (1)EngineeringComputer Science
Humanities
Description:
Clinical cancer researchMolecular endocrinologyMolecular neuroscience
DISTINCTIVE COMPETENCY #26
26
THE ADVANTAGES?
Bottom-Up Cross-Silos Granular
Identify “Emerging Areas”
Identify “Super Connectors”
Identify “Rising Stars”
1 2 3
28
IDENTIFYING NATIONAL STRENGTHS
Source: Klavans & Boyack, “US vulnerabilities in science and engineering,” 10th International Conference on Science and Technology Indicators, Vienna, Austria, Sept. 17-20, 2008
Strengths – Top 40 NationsStrengths – USA
US strengths (transparent) are overlaid here on the strengths (dark) from another 40 nations
Strengths – Overlaid
29
ELSEVIER?
We own 25% of the world’s scientific literature and we are usually perceived as simply a STM publisher…
But the reality is that...
30
We have been indexing and analyzing the world’s research output over the last few years, not just Elsevier-owned content
We have been spotting and building new advanced tools on top of our multi-publisher citation dataset to help universities take the guesswork out of research performance assessment
We solicited help from key partner institutions to co-define what performance intelligence needs to bring to light in order to help drive success for universities
SUMMARY
Efficiency and effectiveness are realized by taking a deeper look at the research activity workflows and identifying improvement opportunities
Progressive institutions will demonstrate lean research by adopting new performance intelligence tools to identify research strengths to focus on
With more rigorous analytics and better insights into research performance these institutions will be better positioned to gain standing among their peers
31
Economic downturn has intensified focus on research effectiveness and efficiency, bringing a new reality for us all…
Thank You For Listening