X ING Huaibin SHI Xiaoyong China National Center for S&T Evaluation (NCSTE)
Post on 14-Jan-2016
39 Views
Preview:
DESCRIPTION
Transcript
国 家 科 技 部 评 估 中 心National Center for S&T Evaluation
Recent Experiences and Challenges of Research
Program Evaluation in China: An Introduction to the International Evaluation of
NSFC Funding and Management Performance
XING Huaibin
SHI XiaoyongChina National Center for S&T Evaluation (NCSTE)
25 October 2012, Minneapolis
国 家 科 技 部 评 估 中 心National Center for S&T Evaluation
• Provides a successful case of R&D program evaluation in China which has been done from 2010 to 2011
• CASE: International Evaluation of NSFC’s Funding and Management Performance
• To show some recent experiences and challenges
THIS PRESENTATION
国 家 科 技 部 评 估 中 心National Center for S&T Evaluation
• Founded in 1997, more than 50 staff now.
• www.ncste.org
ABOUT NCSTE
国 家 科 技 部 评 估 中 心National Center for S&T EvaluationOUTLINE
• Background
• Methodologies
• Implementation
• Future and Challenges
国 家 科 技 部 评 估 中 心National Center for S&T EvaluationBackground
Why to conduct this evaluation?
■ Performance of the government, including of the public funding on R&D have recieved more corcern in recent years in China
■ 2011 marks the 25th anniversary of NSFC
■ The Ministry of Finance and NSFC jointly commissioned this evaluation in Feb, 2010
国 家 科 技 部 评 估 中 心National Center for S&T EvaluationBackground
Purpose of This Evaluation
■ The Evaluation will, from historical angles, comprehensively examine NSFC’s funding and management performance, its strategic positioning over the past 25 years, and from an international perspective, assess its strengths, weaknesses as well as the challenges it faces.
国 家 科 技 部 评 估 中 心National Center for S&T EvaluationOUTLINE
• Background
• Methodologies
• Implementation
• Future and Challenges
国 家 科 技 部 评 估 中 心National Center for S&T Evaluation
Firstly,domestic preparation, international evaluation.
Secondly, Issue and Key Question-Oriented
Thirdly, Evidence-Based
Methodologies
国 家 科 技 部 评 估 中 心National Center for S&T Evaluation
Firstly ,“ domestic preparation, international evaluation”.
□ Domestic Preparation NCSTE, an independent and professional evaluation organization, will design the overall evaluation plan and prepare evidence needed for the Evaluation, with the support of NSFC.
□ International Evaluation International Evaluation Committee (IEC) will evaluate the overall performance of NSFC’s funding and management based on the evidence prepared by NCSTE and its own investigations and international comparisons
Methodologies
国 家 科 技 部 评 估 中 心National Center for S&T Evaluation
Why need domestic preparation
■The Evaluation covers a long time span and a thorough understanding of Chinese context are needed;
■There are no systematic evaluation reports on NSFC;
■Source materials are mostly in Chinese. Useful information needs to be selected from large amount of materials.
■Most workshops and interviews should be implemented by NCSTE.
Methodologies
国 家 科 技 部 评 估 中 心National Center for S&T Evaluation
domestic preparation conducted independently
Independence of domestic preparation is reflected in
■Institutional arrangement in NSFC ■ Ensure NCSTE team had full independence to conduct their work without interference
■ NSFC provided coordination and support request by NCSTE team
■ NCSTE team have the independence to:■ Access to all relevant information■ Select the target group for interview or focus group meeting■ Conduct Questionnaire surveys ■ Present evidences and evidence based findings■ Communicate with the IEC directly
Methodologies
国 家 科 技 部 评 估 中 心National Center for S&T Evaluation
12
Methodologies
Secondly, Issue and Key Question-Oriented
□Strategic Positioning
Issue 1 NSFC’s strategic positioning Q1-Q3Issue 2 Funding strategy Q4-Q6
□Funding Performance
Issue 3 Contributions to originality and novelty in basic research
Q7-Q8
Issue 4 Promoting the balanced development of scientific disciplines
Q9-Q11
Issue 5 Fostering innovative talents Q12-Q14Issue 6 Supporting the national demands and challenges
Q15-Q17
□Management performance
Issue 7 Modality of funding management Q18-Q20Issue 8 Funding instruments Q21-Q23Issue 9 Peer Review System Q24-Q26
□Impact Issue 10 Impact of NSFC’s funding Q27-Q30
国 家 科 技 部 评 估 中 心National Center for S&T EvaluationMethodologies
Thirdly, Evidence-Based
Ensure that the IEC has an adequate evidence base available for future evaluation
Focus on answers to the evaluation questions defined in the TOR
Both qualitative and quantitative evidences; multiple approach for evidence collection
国 家 科 技 部 评 估 中 心National Center for S&T Evaluation
14
□Strategic Positioning
Issue 1 NSFC’s strategic positioning
Q1-Q3 5 Evidences
Issue 2 Funding strategy Q4-Q6 10 Evidences
□Funding Performance
Issue 3 Contributions to originality and novelty in basic research
Q7-Q8 4 Evidences
Issue 4 Promoting the balanced development of scientific disciplines
Q9-Q11 12 Evidences
Issue 5 Fostering innovative talents
Q12-Q14 14 Evidences
Issue 6 Supporting the national demands and challenges
Q15-Q17 6 Evidences
□Management performance
Issue 7 Modality of funding management
Q18-Q20 11 Evidences
Issue 8 Funding instruments Q21-Q23 13 EvidencesIssue 9 Peer Review System Q24-Q26 13 Evidences
□Impact Issue 10 Impact of NSFC’s funding
Q27-Q30 8 Evidences
Evidence-Based
Methodologies
国 家 科 技 部 评 估 中 心National Center for S&T Evaluation
Evidence Collection■Cases prepared by the NSFC■Other evidence prepared by the NCSTE ■ Review of relevant policy documents
■ Analysis of NSFC documents and statistics
■ Bibliometric study on the SCI paper with NSFC
■ Interviews and focus groups with government officials, beneficiary institutions, Principal Investigators, peer reviewers and NSFC personnel
■ Questionnaire surveys of supporting institutions, Principal Investigators and peer reviewers of NSFC projects
■ Case analysis base on the Case Collection of the NSFC
Methodologies
国 家 科 技 部 评 估 中 心National Center for S&T EvaluationOUTLINE
• Background
• Methodologies
• Implementation
• Future and Challenges
国 家 科 技 部 评 估 中 心National Center for S&T Evaluation
•organizational structure of the Evaluation
Leading Group
Evaluation OfficeIEC NCSTE
Working Group
Implementation
国 家 科 技 部 评 估 中 心National Center for S&T Evaluation
Files and Reports
Implementation
International Evaluation Report
Synthesis Evidence Report
10 issues study reprots
cases report
SCI papers report
Questionnaire Survey report
Interviews and Focus Group Meeting report
Terms of Reference
国 家 科 技 部 评 估 中 心National Center for S&T Evaluation
Type of Interviewees Number of Interviewees
S&T management officials at the central government level
3
Academicians of CAS and CAE 6
Former and current presidents and vice-presidents of NSFC
4
Staff of NSFC 10
Total 23
Interviews-4 types
Implementation
国 家 科 技 部 评 估 中 心National Center for S&T Evaluation
Focus Group Meeting-7 types
Implementation
Type of Focus Group Meeting Number Number of Participants
With research managers of supporting institutions
5 40
With PIs of NSFC projects 3 24
With academic leaders of related discipline 1 4
With researchers of local colleges and universities
11 94
With reviewers of proposals 1 22
With rejected applicants 6 42
With staff of NSFC 3 23
Total 30 249
国 家 科 技 部 评 估 中 心National Center for S&T Evaluation
Questionnaire Survey- 6 types
Implementation
Questionnaire Survey No. of Questionnaire Delivered
No. of Questionnaire Returned
Respond Rate
supporting institutions 2,257 841 37.3%
PIs of the General Program projects 43,412 10,228 23.6%
PIs of the Young Scientists Fund projects
16,092 6,104 37.9%
Grantees of the National Science Fund for Distinguished Young Scholar
2,164 445 20.6%
reviewers 6,036 1,232 20.4%
rejected applicants 7,838 1,371 17.5%
Total 77,799 20,221 26.0%
国 家 科 技 部 评 估 中 心National Center for S&T Evaluation
Contents and structure of the Synthesis Evidence Report by NCSTE
■Full coverage of the requirements defined in the ToR
■The logical framework: 4 dimensions, 10 key issues,30 questions and 96 evidences
■The structure: Part A (Introduction), Part B (Background), Part C (Key issues and Evidences) and Annex
Implementation
国 家 科 技 部 评 估 中 心National Center for S&T Evaluation
• The Evaluation Report and Synthesis Evidence Report can be download at
• http://www.nsfc.gov.cn/nsfc/cen/pgbg/index.html
国 家 科 技 部 评 估 中 心National Center for S&T EvaluationOUTLINE
• Background
• Methodologies
• Implementation
• Future and Challenges
国 家 科 技 部 评 估 中 心National Center for S&T Evaluation
Evaluation increasingly received attention from central government but is still a lower priority.
Evaluation of R&D programs and projects has not been institutionalized in related ministries and agencies.
Evaluation results are seldom used in the research priority setting.
Lack of indicators to reflect the performance of different research program.
……
Future and Challenges
国 家 科 技 部 评 估 中 心National Center for S&T Evaluation
Many tMany thankhanks for your attention!s for your attention!
top related