FP7 /1 EUROPEAN COMMISSION – DG Research – 12 November 2009 EVALUATION 2009 Orlando, Florida, USA 2 November 2009 Neville Reeve European Commission DG RTD Multi-actor, multi-component evaluation of the EU Framework Programmes
Feb 13, 2016
FP7 /1 EUROPEAN COMMISSION – DG Research – 12 November 2009
EVALUATION 2009Orlando, Florida, USA
2 November 2009
Neville Reeve European Commission DG RTD
Multi-actor, multi-component evaluation of the EU Framework Programmes
FP7 /2
European Commission
Council of the
European Union
European Parliament
European Court of Auditors
Court of Justice
European Union InstitutionsEuropean Commission and Directorates General
Secretariat General
DG Research DG Information Society and
Media
European Commission
DG Budget
DG Energy & Transport
DG External Relations
Joint Research Centre
DG Enterprise
DG Agriculture & Rural dev’t
DG Regional Policy
DG External Relations
DG Competition
Managing the Framework Programme
FP7 /3
EU research
Framework Programme (FP) is a multi-annual research funding programme implemented at EU level
7th Framework Programme (2006-2013) - €50 billion – 27 Member States – more than 100 participating countries – tens of thousands of transnational co-funded projects
FP7 /4
EU Research Framework ProgrammesAnnual Budgets between 1984 and 2013
NB: budgets in current prices. Source: Annual Report 2003, plus FP7 revised proposal
0
2
4
6
8
10
12
1984 1986 1988 1990 1992 1994 1996 1998 2000 2002 2004 2006 2008 2010 2012
€ bi l l i on
FP7 /5
3.270
5.360
6.600
13.120
14.960
17.500
50.521 2.7
0 10 20 30 40 50 60
1984-1987
1987-1991
1990-1994
1994-1998
1998-2002
2002-2006
2007-2013
Billion euros
Budgets of the EU Framework Programmes
FP7 /6
EU research: changing priorities
01020
3040506070
8090
100
FP1
FP2
FP3
FP4
FP5
FP6
FP7
%
OtherBasic researchCoordination & developmentSpaceScience and societyTraining of researchersDissemination & exploitationInternational cooperationSocio-economicTransportEnergyLife sciencesEnvironmentIndustrial & materials technologyIT and Communications
FP7 /7
FP7 – Indicative breakdown (€ million)
FP7 /8
Multi component evaluation system
FP8 Ex ante Impact Assessment
Annual monitoring
1998 2000
FP 5FP 6
2002 2004 2006 2008
Thematic level evaluations
5 year assessment
National Impact studies
FP7 Ex ante Impact Assessment
FP 7
FP6 ex post
FP7 interim evaluation
FP7 /9
Evaluation Actors, Networks & Standards
Actors Political level – Council, European Parliament Commission level – Secretariat General DG level – central units, operational level Member States evaluation
Networks DG Research Evaluation Network European RTD Evaluation Network Commissions internal Evaluation Network
European Commission evaluation standards
FP7 /10
Indicator / issue Sub-indicator Main Data Source
Promotion of FP7
1.1 Number of information days Annual NCP Survey
1.2 Number of attendees at information days Annual NCP Survey
1.3 Commission organised meetings of NCPs DG RTD
Performance of the calls
2.1 Success rate (overall) by priority area and funding scheme CORDA
2.2 Success rate for different types of organisation by priority area and funding scheme CORDA
2.3 Success rate for different types of organisation by priority area and funding scheme & success rates per country CORDA
Performance of the proposal evaluation and redress procedure
3.1 Overall quality assessment of the proposal evaluators on the FP proposal evaluation process (evaluators survey)
Annual Evaluators' Survey
3.2 Assessment of quality by the evaluators between the FP evaluation process and other equivalent systems (evaluators survey)
Annual Evaluators' Survey
3.3 Time to grant agreement CORDA
3.4 Percentage of experts reimbursed within the specified 45 days DG RTD/PMO
3.5 Redress cases upheld (i.e. leading to a re-evaluation) – numbers and percentages DG RTD
Quality of on-going research projects
4.1 Average results of independent project review process by priority area Data from new reporting system (not existing yet for 2008)
4.2 Percentage of projects by priority area covered by reviewsData from new reporting system (not existing yet for 2008)
Project performance by outputs
5.1 Average number of project publications per project by priority area and funding scheme
Data from new reporting system (not existing yet for 2008)
5.2 Average number of other forms of dissemination activities per project by priority area and funding scheme
Data from new reporting system (not existing yet for 2008)
5.3 Average number of different types of intellectual property protection per project by priority area and funding scheme
Data from new reporting system (not existing yet for 2008)
Indicators for FP Monitoring (1)
FP7 /11
FP activity
6.1 Total number of active projects by priority area CORDA
6.2 Average financial size of projects by priority area and funding scheme CORDA
6.3 Participation by types of organisation by priority area funding scheme CORDA
6.4 Participation totals per country CORDA
Achieving gender equality
7.1 Number of male and female coordinators in proposals CORDA
7.2 Number of male and female coordinators in projects CORDA
7.3 Gender breakdown (by seniority) of project participants CORDA
7.4 Percentage of male and female members in Advisory Groups and Programme Committees DG RTD
Observing sound ethical principles in FP research
8.1 Number of projects going through the review process/ % by area/ programme DG RTD
8.2 Number of ethics reviews where the result showed insufficient attention had been given in proposal DG RTD
8.3 Number of projects stopped as a results of the ethics review DG RTD
8.4 Number of ethics screenings DG RTD
Performance International Cooperation activities
9.1 Total numbers of participations of Third Countries by priority area and funding scheme CORDA
9.2 Success of Third Countries in calls by priority area and funding scheme CORDA
9.3 EC contribution to Third Countries CORDA
9.4 Number of international outgoing/incoming fellowships DG RTD
Simplification 10.1 Do stakeholders perceive that the FP is getting simpler to use in terms of financial and administrative procedures? Annual NCP Survey
Indicators for FP Monitoring (2)
FP7 /12
Evaluation Roadmap
FP6 Ex-post evaluation in 2008 FP7 Progress report in 2009 ERC review 2009 FP7 interim evaluation in 2010 Ex ante Impact Assessment (FP8) in 2011/12 Ex evaluation FP7 in 2015
FP7 /13
EXPERTGROUP
Supporting Experts analyses
Stud
ies
at
Them
atic
lev
elStudies at FP level
Self-
asse
ssm
ents
; in
terv
iew
s;
cons
ulta
tions
Impa
ct s
tudi
es b
y M
embe
r Sta
tes
Statistical analysis of FP implementation and annual Monitoring Reports
Evaluation of the 6th FP - Evidence and Analysis
FP7 /14
Evaluation study of FP structuring – (1)
FP6 networks are characterised by a core periphery structure dominated by a small number of closely-knit organisations.
Industry plays less of a coordinating role and is weakly embedded in the FP6 networks with the exception of IST
FP6 has resulted in more upstream than downstream outputs
Performance decreases as project size increases New Member State participation is dominated by a few
actors in Poland, Hungary and the Czech Republic.
FP7 /15
Evaluation study of FP structuring (2) - Network structure of the FP6 thematic areas
FP7 /16
Evaluation study of FP structuring (3) - Who coordinates whom in FP6 (Universities, Industry, Institutes)
FP7 /17
Evaluation study of FP structuring (4) - Who coordinates whom in FP6 (New Member States)
FP7 /18
0 0.5 1 1.5 2
Total
POLICY_SUPPORT
INCO
EURATOM
7_CITIZENS
6_GLOBAL
6_ENERGY
5_FOOD
4_AERO_SPACE
3_NMP
2_IST
1_LIFE_SCIENCE_HEALTH
200620042002
Using bibliometrics to assess participation (1) - Relative citation rate of lead scientists, by FP priority – year 2002, 2004, 2006
FP7 /19
0,000,020,040,060,080,100,120,140,16
Wor
ld sh
are
(%)
Lead Scientist Pub world share (%) Lead Scientist Citation world share (%)
0,0
0,5
1,0
1,5
2,0
2,5
Rela
tive I
mpa
ct In
dex
Lead Scientist Relative Impact ratio WOS_Relative Impact ratio
0,00,20,40,60,81,01,21,41,61,82,0
Expe
cted
Impa
ct In
dex
Lead Scientist Expected Impact ratio WOS_ Expected Impact ratio
0,00,20,40,60,81,01,21,41,6
Rela
tive C
itatio
n Ra
tio (R
CR)
Lead Scientist RCR WOS_RCR
Using bibliometrics to assess participation (2) - Lead scientists’ 2–year Relative Citation Rate (RCR), all priorities taken together
FP7 /20
http://ec.europa.eu/research/evaluations/
FP7 /21
FP7 /22
FP7 /23 EUROPEAN COMMISSION – DG Research – 2 November 2009
Dr Neville ReeveDirectorate-General for Research Unit for Evaluation and Monitoring of ProgrammesPhone +32-2-298 93 29Fax +32-2-295 40 [email protected]://ec.europa.eu/research/evaluations