Evaluation and Ranking Evaluation and Ranking of National Research of National Research Institutes Institutes Project Group of National Research Institu te Ranking Institute of Policy and Management, Chines e Academy of Science Presented by Prof. WB Liu Kent Business School, University of Ke nt, UK
42
Embed
Paper 5: Rankings of International Research Institutes (Liu, Wenbin)
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Evaluation and Ranking Evaluation and Ranking of National Research of National Research
InstitutesInstitutes
Evaluation and Ranking Evaluation and Ranking of National Research of National Research
InstitutesInstitutes
Project Group of National Research Institute Ranking
Institute of Policy and Management, Chinese Academy of Science
Presented by Prof. WB Liu
Kent Business School, University of Kent, UK2010-07-01 , Canterbury, UK
MotivationMotivation CAS, the largest independent research institutions in Chin
a , with almost 100 research institutions involving all the disciplines of natural sciences, about 30,000 permanent research staff
Quantitative expansion moved to qualitative improvement Crucial to enhance competitiveness and sustainability in i
nternational level Need to know the gap comparing with relevant institution
s in the word, relative positionStrength and weaknessHow to improve
Evaluation and rankingEvaluation and ranking
Evaluation and Ranking of National Evaluation and Ranking of National Research InstitutesResearch Institutes
CAS needs to compare its performance with those of leading national research institutes
There are international evaluation and ranking reports that contain R&D evaluations for a country or university
But there exist no reports tailor-made for national research institutes
Also these reports cannot be used to diagnose weakness of CAS
What is a national research institute? They are ( 1 ) Large multi-discipline research i
nstitutes adm and funded by a country ( 2 ) Research institutes adm. and fu
nded by ministries or public bodies ( 3 ) Research institutes adm. and fu
nded by local government or mixed
Evaluation and Ranking of National Evaluation and Ranking of National Research InstitutesResearch Institutes
Research of a national research institute often covers many different subjects’ areas and has multiple-purposes for basic research, applied research, and R&D.
The key issue is how to evaluate its overall capability and performance.
In 2006 this study first started in the Evaluation Centre of CAS, IPM, CAS. Then CAS Library joined.
A project group was formed –key members including Li X.X, Wei Meng, Ms Liu, Z.Y, Ms Fang Xu (Kent), Mr Y.G Yang Prof. WB Liu was in charge of methodology development
The project was finished in 2009.
What are their missions?
R&D Development need of a country This part is difficult to compare and evaluate Basic and applied scientific research For this part, it is possible to compare their perfo
rmance, at least by subject. We developed 3E measurement for this purpose:
E1 – Research Efficacy E2 – Research Efficiency E3 - Research Effectiveness
Evaluation and Ranking of National Evaluation and Ranking of National Research InstitutesResearch Institutes
Based on Soft System Methodology, we developed a methodology in building indicators system to measure efficacy, efficiency; effectiveness of a national research institute.
We have built the theoretical 3E indicators system for the overall capability and performance of national research institutes, based on the SSM models of basic research, applied research
We have developed evaluation methods to aggregate the data in order to produce ranks for the institutes 。
Hierarchical Structure of Indicator
Output or effectiveness were divided into several levels
Example: High quality Pub has Top0.1%,Top1%, Top10%, and SCI, with decreasing weights
SCI editorship is divided into 1% 、 Top5%,Top10%,
Awards were similarly divided. – cross subjects, subjects, sub-subjects
TOP10% Pub
TOP0.1%Pub
TOP1% Pub
Approach Samples Indicators Evaluation Method Case Further work – data reliability and
methodology robust International Discussion Group
Evaluation and Ranking of National Evaluation and Ranking of National Research InstitutesResearch Institutes
ApproachApproach Subject evaluation first -then cross subject evaluation
Theoretical base: SSM and 3E indicator theory
Research Output(E1), Reproach Efficiency (E2), Research Effectiveness (E3)
Overall by aggregating sub-rankings of subjects
ApproachApproachOverall
Subject 1 …Subject N
E1 output E2 efficiency E3 E2 E1
Subjects Ranking and then Overall Ranking
SSM ANALYSIS
E3 effectiveness
Ranking of RIsRanking of RIs
Overall performanceOverall performance
Functions analysisFunctions analysis
Single Single disciplinediscipline
Multi-Multi-disciplinesdisciplines
BasicBasic researchresearch
BasicBasic researchresearch
AppliedAppliedResearchResearchAppliedApplied
ResearchResearch
indicators
•Sample comparison, feedback and discussion Sample comparison, feedback and discussion •Define and select indicators, build indicators systemDefine and select indicators, build indicators system•Aggregating, preliminary calculation, further adjustingAggregating, preliminary calculation, further adjusting
indicators
Sample Selection - basic research Generating new knowledge Undertaking state projects or programme, contribute to h
igh-tech development, finally, social and economic development
Total publications in top 100
Total citations in top 100
Citations per article in top 100
Discipline
Based on ESI ( Essential Science Indicators), 20 disciplines, also EI,..
Selected 89 organization world-wide
Samples pool
Selected samples
Sample-Sample-Scale difference Scale difference
6, 10%
14, 23%
25, 42%
15, 25%
>20,000 10,000-20,000 1,000-9,999 <1,000
Sample-Sample- subjects distribution subjects distribution
Present overall national or local pictures in terms of R&D activities
Linkage with organizational function is weak, hard to use to evaluate research institutions directly
Satisfy specific assessment objective : (University ranking)
As least as possible to achieve evaluation objective More or less, lack of systemic inter-functional analysis
of research institutions Indicator definition and selection : expert experiences 、
questionnaire
Indicators comparisonIndicators comparison
Indicator System for Research Institutes
Only partial functions
Too many
No explicit relationship to internal functions
Mixed use of different type indicator
Scores always from weighed averaging
……
Indicators systemIndicators system 3Es theory3Es theory
InputsInputs
OutputsOutputs
internaloperationalsystem
External External environmentenvironment
Why needs system analysisWhy needs system analysisWhy needs system analysisWhy needs system analysis
How to build a complete 3E indicator system?
1. Complete indicator systems can only be based on good understanding of internal processes
2. A good SSM system analysis will not only bring good understanding of internal processes but also structured 3E indicator data-bases
3. Good combinations of existing techniques
SSM and 3EsSSM and 3Es
2.Express messy system
3.Build RD of 3.Build RD of relevant purposeful relevant purposeful
activity systemactivity system
1.Complex situation inspect
6.Debate with systematically desirable
and culture feasible
7.Take into actions to improve
5.Compare CM with 2
4.Formulate CM4.Formulate CM
Real worldReal world
system thingssystem things
Activities inmodel
Exist? How? Who? Comments
-Obtain-Understand-Determine–Decide-
RD1
RD2
RD3
The spirit of SSM is that any system can be analysed by answering three questions as:
1) What to do (P) - Efficacy relate, measures the outputs of the system
2) How to do (Q) - Efficiency relate, evaluates whether minimum resources are used.
3) Why to do (R) - Effectiveness relate, assesses whether the system outputs are meaningful or useful to higher level (wider) system.
Evaluation Framework:Evaluation Framework:System monitored and controlled by 3EsSystem monitored and controlled by 3EsEvaluation Framework:Evaluation Framework:System monitored and controlled by 3EsSystem monitored and controlled by 3Es
Based on 3E’ Efficacy
Outputs of a system, or value-add
Efficiency Outputs/resources, or value-add/resources
Effectiveness Desirable impact of Efficacy to aims of higher systems
Example:Example: A production-line can produce goods efficiently but make no profits. So
effectiveness is 0 for the company. Or it may make profits but with 0 effectiveness for the state.
4E’ or 5E’ …
Basic research evaluation in CASBasic research evaluation in CASBasic research evaluation in CASBasic research evaluation in CAS
Clients:
•CAS
•REC (PO)
Problem Owner (PO)
•institutes
A state-owned system to improve the originality, significance, reputation and sustainability of CAS basic research in the natural sciences by developing the research capabilities and infrastructure, and by improving resources utilisation of CAS in order to benefit Chinese social and economic development and to enrich human knowledge.
C: World scientific community A: CAS headquarters and institutesT: Improvement in significance, reputation and sustainabilityW: Goal of CAS is improvement in research outcomesO: Chinese state councilE: Economic and cultural constraints, current research infrastructure(R): Institutes, scientists, money, equipment
Basic research in CASBasic research in CAS— — CAS as a CAS as a wholewholeBasic research in CASBasic research in CAS— — CAS as a CAS as a wholewhole
A CAS-owned system to enrich the world’s knowledge in a particular scientific domain with original and significant research by identifying potential areas of discovery, developing the capabilities to undertake appropriate research, carrying out the research and disseminating it through prestigious channels in order to enhance the reputation, significance and sustainability of CAS and the Institute.
C: World scientific community in discipline, CAS (reputation)A: Institute personnelT: Generating significant new knowledgeW: Role of an Institute within CAS is to generate significant new knowledgeO: CASE: CAS procedures, funding(R): Scientists, other staff, funding, equipment
Basic research in CASBasic research in CAS——Generic Generic InstituteInstitute Basic research in CASBasic research in CAS——Generic Generic InstituteInstitute
Monitor
ControlCriteria forEfficacyEfficiency
Monitor
Control
Criteria forEffectiveness
OperationalControl
Strategic Control
1. Identify potentially significant areas of discovery
2. Appreciate current resources, capabilities and projects
3. Decide which opportunities to pursue
4. Obtain necessary resources
5. Conduct research
6. Disseminate results through prestigious channels
7. Contribute to the reputation,
resources and sustainability
Generic InstituteGeneric Institute Conceptual Model
Example:
Areas of possible research in related disciplines
Areas of significance identified
RD1 A system to identify research opportunities for institute X that are significant and original bearing in mind the resources and capabilities needed, by effective external scanning and by improving internal discussion and communication in order to contribute to decide which opportunities to pursue which contribute to enhance reputation, resources and sustainability of the Institute.
Resources: Funding, hardware investments, staff
Research levelsResearch reputationCAS procedure
1.1 Scan the external environment for research opportunities
1.2 consider current resources and capabilities
1.3 Encourage discussion and communication within and among institutes
1.4 decide on those which are potentially possible
1.5 specify extra requirements for potentially possibilities
E1: does it produce possible further research topics?
E2: E1/resources (eg time, funds, people)
E3: do we get significant research done which contributes research reputation and resources enhancement?
CM1. Identify potentially significant areas of discovery
CM4 necessary resources obtaining system
4.1 Know possible research can be conducted
4.2 Appreciate to extra resources that will be needed•Human resources•Equipment•Competence•funds
4.3 Decide how that can be obtained by
4.5 Make appreciated application and proposal
4.6 Check that necessary resources are obtained
4.4 Provide institutional support
4.3.1 Human resources: 4.3.2 Equipment: by applying from headquarter based on current research capability and further development4.3.3 Funding: by applying for external research projects, or internal projects 4.3.4 Others: by applying for extra investment to improve research infrastructure, such as building, decoration
3.1 Citation Total (1/6) Average (1/6)3.2 Position Post in Inter-Oragn ( 1/6 ) Editorship TOP1% editor (1/18) TOP5% editor (1/18) TOP10% editor (1/18)3.3 Award cross subject (1/6) subject +.. (1/6)
E3 : (1/3) 3.1 Citation Total (1/6) Average (1/6)3.2 Editorship TOP1% editor (1/9) TOP5% editor (1/9) TOP10% editor (1/9)3.3 honor Awards – cross (1/12) subject+subject (1/12) Post in Inter Organ ( 1/6 )
E3 Plan 2
Random Sensitivity Analysis E3(Max)
Change (5%)
Change (10%)
NO. 机构名称 Plan1 Plan 2 Plan 1 Plan 2 Plan 1 Plan 2
Average Change 0.0978
0.0477
0.1363
0.0666
1 美国农业部 0.877238
0.8772
0 0 0 0
2 法国农业科学研究院 0.192529
0.2536
0.0257
0.0097
0.0247
0.0093
3 西班牙科学研究委员会 0.150366
0.1796
0.0317
0.0132
0.0324
0.0135
4 加拿大农业及农业食 0.110103
0.1493
0.0302
0.0111
0.0247
0.0091
5 日本农林水产省 0.050875
0.0518
0.0490
0.0240
0.0829
0.0407
6澳大利亚联邦科学与工
业研究组织0.11267
40.151
20.055
40.020
60.059
00.022
0
7 中国科学院 0.116528
0.1735
0.0235
0.0078
0.0482
0.0161
8 印度农业研究理事会 0.021313
0.0213
0.1763
0.0881
0.4217
0.2108
10 英国研究理事会 0.111147
0.1403
0.0426
0.0168
0.0572
0.0226
11 印度科学与工业研究理 0.051480.055
00.130
00.060
70.144
80.067
6
Weights and Aggregation Weights and Aggregation
Each of the subjects has the same importance
Using weighted average at all levels inside a subject. All the indicators at the same level have the same weights
Thus those with a fuller arrange of research subjects and good quality research will perform better
Aggregation techniquesAggregation techniques
Subject Ranking: For each ESI21 subject ,compute E1 and E3.
Institute Ranking: Based on these, compute the total scores or ranks of E1 and E3 to rank institute (in terms of E1 and E3)
Difficult to obtain E2 data.
Aggregation techniquesAggregation techniques
Method One: Compute the total of E3 from all those of
subjects Method Two: Calculate total ranks of E1 or E3 Method Three: Count number of gold, silver, and brown
Aggregation techniquesAggregation techniques Based on each discipline, Normalization E1-E3 based on average value Each discipline are equally important Methods:
Rank-added method
Score-added method
13
1
( ), , , 1,...,54 k k kj j j j
k
R Rank E then R R j
13
1
, 1,...,54kj j
k
R E j
Case study - aggregation method based on rank-added method
Score Rank Agr. BioS Che Comp Eng. Envi Geo Mate math Med Multi- Phys Soci