1 Guidance document on ex-ante evaluation 15 March 2012 Joint presentation to the ERDF and ESF Evaluation Partnership Meeting.

1

Guidance document on ex-ante evaluation

15 March 2012Joint presentation to the ERDF and ESF Evaluation

Partnership Meeting

Ex ante evaluation: our approach

• What is different?• Common Regulation & multi-fund programmes• Stronger focus on results / Europe 2020 strategy• More detailed Article 48• Links with Partnership Contracts, SEA

• Our common approach:• Stick to Regulation (article 48 and related articles)-as

proposed by Commission• Focus on ex ante evaluators' role• When necessary, give examples of differences between

ESF/ERDF

3

Ex ante evaluation Guidance• Programme strategy

Q&A

• Indicators, monitoring and evaluation • Consistency of financial allocation• Contribution to Europe 2020

Q&A

• Strategic Environmental Assessment • Evaluation Process

Q&A

4

Programme strategy

• Consistency of programme objectives

• Coherence

• Horizontal principles

• Linkage between supported actions, expected outputs and results

Q&A

5

Programme strategy – key reference points

Art. 24(1) CPR:• Europe 2020 strategy • Common Strategic Framework • Partnership Contract

Art. 87(2)(a)(i) CPR:• Country specific recommendations• national and regional needs

6

Consistency of programme objectives

• Step 1: The ex ante evaluator should assess the challenges and needs identified by the programme.

• Step 2: The evaluator should examine if these challenges and needs have been consistently translated into thematic objectives, priorities, investment priorities and corresponding specific objectives.

7

Step 1: Challenges & needs (1)• National & regional programmes: the evaluators should primarily base their

assessment on the NRP, country-specific recommendations, European semester analyses

Justification of specific regional challenges and needs

In addition:

• National ERDF sectoral programmes: specific sectoral analysis & territorial priorities

• Regional programmes: specific regional situation and needs

• Specific challenges of sub-regional or functional areas or of specific target groups

• Horizontal principles (Art. 7 & 8 CPR)

8

Step 1: Challenges & needs (2)

• The ex ante evaluator should assess whether the identified challenges and needs are consistent with the analysis of disparities and development needs in the Partnership Contract.

• The evaluators should also base their assessment on the Common Strategic Framework.

• They should examine the prioritisation of the identified challenges and needs and suggest reviewing it, if appropriate.

9

Step 2: Programme objectives

• While appraising the consistent translation of the identified challenge & needs into programmeobjectives, the evaluators should focus on:

• Their appropriate weight in the investment priorities

• Precise formulation of specific objectives

• Rationale and evidence for omitting major challenges or needs

• The evaluators may recommend complementary analyses

thematic objectives

investment priorities

Specific objectives

10

Coherence

Internal coherence

The evaluators should assess:

• Relationship between specific objectives

• Coordination mechanisms in multifund programmes

External coherence

The evaluators should examine:

• Analysis of programme contribution to other strategies/programmes and of the influence of these on the expected results of the programme

• Planned integrated territorial approaches

Horizontal principles•The evaluators should assess the following elements under each principle (Art. 87(3) CPR):

•Equality between men an women• Arrangements to integrate the gender perspective at the level of the programme and operations

• Contribution to its promotion

•Anti-discrimination• Specific actions in preparation and

implementation of programme

•Sustainable development• Specific actions in the selection of operations

11

Opinion of national equality bodies

Linkage between outputs and results

• Clear intervention logic: condition for good programming and evaluations!

• Recommendation: use a logical framework for each priority axis or investment priority

• The evaluator should examine:- What is the expected change?- How will outputs contribute to intended results?- Will the proposed actions effectively lead to these outputs? - What other factors could influence the expected results?- Would evidence suggest other approaches?

Linkage between outputs and results

• Are the planned form of support the most effective? What is the rationale?

(grants, prizes, repayable assistance and financial instruments and a combination: Art. 56)

• Will the actions effectively meet the needs of specific territories or target groups?

Indicators monitoring and evaluation

• Relevance and clarity of proposed indicators• Baseline and target values• Suitability of milestones• Administrative capacity • Data collection• Evaluation• Consistency of financial allocation• Contribution to Europe 2020

Q&A

Indicators, monitoring and evaluation

Art. 24 CPR- For each priority axis, indicators to assess progress towards achievement of objectives - Basis for monitoring, evaluation, review of performance

•Increased focus on results: role of well designed indicators and evaluations•New focus on territorial integrated approaches: reflected in monitoring and evaluations where relevant•Common indicators compulsory

Indicators, monitoring and evaluation

• Relevance of indicators- Output indicators reflect the main operations - Result indicators are responsive to the policy (values evolve

with development of operations) - Result indicators reflect the most important intended

change - Common indicators - ERDF: used where relevant to

operations and specific objectives / ESF: monitored under all investment priorities

- Results may be close to the intervention, short or long term: depends on the intervention logic, on the intended change

Indicators, monitoring and evaluation  Intervention Output indicator Result indicator

ERDF Incubator services for start ups Number of incubated start ups

Increase of the share of start ups in regional statistics

Increase of employment in young enterprises (3 years after incubation)

  Intervention Output indicator Immediate result indicator

Longer-term result indicator

ESF Training of individuals who are inactive, not in education or training  

Number of supported participants (who are inactive, not in education or training)

Number of inactive participants who are newly engaged in job searching upon leaving the intervention

Participants who are in employment 6 months after leaving

Indicators, monitoring and evaluation

• Clarity of indicators- Unequivoqual title, clear definition - Normative interpretation• Robustness and transparence- Robustness: reliability of data source,

representativeness of samples (survey)- Data sources for baselines and target values +

definitions are publicly available- See ex ante conditionality (Annex IV CPR)

Indicators, monitoring and evaluationBaseline and target values

•Availability of baseline data for result indicators? -latest available data (ERDF and ESF Regulations)-advice on sources and methods to collect data when necessary, or research to establish baselines

•Realistic targets? -Compared to actions and form of support, financial allocations -For output indicators, compared to references (previous programmes or sectoral norms) -For result indicators, taking into account other factors

Indicators, monitoring and evaluation

• Performance framework: a subset of indicators for each priority axis

• Suitability of milestones - Do they capture essential information on the

progress of a priority?- Can they be achieved at the review points? - Will data be available for 2017/2019 progress

reports?

Administrative capacity

• The ex ante evaluator should assess whether adequate human resources and administrative capacity for the management of the programme are in place • Including: Have possible previous bottlenecks at

the level of MA and IB been tackled?

• The evaluator may advice on how to address any issues

Data collection

The ex ante evaluator shall assess whether the data collection procedures are suitable. Whether they:

• Respond to all requirements for monitoring• including: is data available on time, in the

required form and quality? How will the data be collected and stored? Will administrative data bases be used to reduce admin. burden? Etc.

• Provide the data needed for evaluations• including: data to i.a. assess effectiveness,

efficiency and impact? What are the mechanisms to ensure good quality data? Etc.

Evaluation

The ex ante evaluator may advice:• On possible evaluation needs for on-going

evaluations • On evaluation methodologies and resulting data

needs • On setting up the evaluation plan

Consistency of financial allocation

• The ex ante evaluator shall assess the consistency of the financial allocation on the basis of the identified challenges and needs and the proposed actions and selected forms of support

• The evaluator may assess whether the different funds are adequately combined and contribute to integrated approaches

Contribution to Europe 2020

• Taking account of the proportionality principle and against the background of national and regional needs

• the ex ante evaluator shall assess the programme’s potential contribution to Europe 2020 in the light of its thematic objectives and investment priorities.

Process

• Strategic Environmental Assessment• Timing• Interactive and iterative process• Partnership and multi-level governance• Evaluation methods and financing• Final report and publication• An independent process• What requirements for evaluators?

Q&A

Strategic Environmental Assessment

• A new link with Ex ante: "shall incorporate, where appropriate, the requirements for SEA…"

- "Where appropriate": direct impact on environment- "Incorporate": does not mean that the ex ante

evaluator has to do SEA- "Requirements": environmental report,

consultations, information, monitoring• How to incorporate SEA: still under discussion

Timing• Ex ante evaluation & Partnership contractsPC includes (Art. 14):- Summary analysis of ex ante evaluations justifying selection

of the thematic objectives and the indicative allocation of Funds

- Summary of main results expected for each thematic objective

• Developping ToR … now- PC and OPs adopted Oct.-Dec. 2013- 6 months negotiations: draft PC & OP April-June 2013- Programme elaboration 6-8 months: starts Aug.-Oct. 2012- Calls for tender launched mid 2012

Interactive and iterative process

• Work in stages

• Timing and interaction

• Steering group

29

Partnership and multilevel governance

Art. 5 CPR• Strengthening with the objective of:

• Increasing legitimacy

• Build on wider expertise and knowledge

• Ensure collective commitment and shared understanding of expected results

• The evaluators should review if the relevant stakeholders are involved in the preparation of the programme as well as in its implementation, monitoring and evaluation

• The evaluators should base their assessment on the Commission Staff Working Document “Elements for a Code of Conduct on Partnership”

30

Evaluation methods and financing

• Mix of methods: - TBE: literature review, workshops, interviews,

focus groups, peer reviews- Involve partners in programme design

• Cost:- number of evaluation questions- methods - Extra tasks (quantification of baselines,

complementary analyses)

Final report and publication

• Submitted with the OP to the Commission - Main evaluation methods used- Changes and improvements to programme- Final assessement of OP- Strategic Environmental Assessment - Executive summary (+ in English)

• Made available to citizens

An independent process

• Article 47 CPR: "carried out by experts functionally independent of the authorities responsible for programme implementation"

• No hierarchical link with the entity responsible for implementation: internal expert may contribute

• Balance between an interactive/iterative and independent process: critical jugements on the different elements of the programme and its coherence

What requirements for evaluators?

• Knowledge of EU and national documents • Deep knowledge of context- Situation of areas or target groups with specific

needs- Situation of economic sectors- Able to judge quality of socio-economic analysis• Experience in evaluations- Data needs for future impact evaluations- Relevance of indicators within the intervention

logic (thematic evaluations)