“The organization of the on-going evaluation of rural development policy in Italy” 122 nd EAAE Seminar “Evidence-Based Agricultural and Rural Policy Making:

“The organization of the on-going evaluation of rural development policy in Italy”

122nd EAAE Seminar

“Evidence-Based Agricultural and Rural Policy Making: Methodological and Empirical Challenges of Policy

Evaluation”

Ancona ,18 febbraio 2011

Simona Cristiano, Alessandro Monteleone

Aim: Modelling the governance of the on-going evaluation in view of being influential for evidence-based policies.

Methodology:•Analysing the state of play of on-going-evaluation in Italy: the context, the framework, the governance models settled;•Analysing the first findings •Get to some reflections and insights on the way forward to a better use of the evaluations as based on a theoritical framework on influential evaluations

Aim of the study and methodology

Timely establishing and quantifying baseline indicators and target levels

Strengthening the linkage monitoring with evaluation in terms of data collection/provision

Ensuring capacity building early onEnsuring continuity of evaluation activities, i.e. regular

assessment of progress, annual reportingSupporting the establishment of good practicesPreparing the ground for the mid-term and ex-post

evaluations (2010, 2015)

Objectives & approach of the EU ongoing evaluation

Overall Objective: feeding the policy- makers;

Quality of the implementation & and review of RDPs

– Responsibilities: MAs, M&E system and independent evaluators – Burdens: setting up a M&E system at Programme level (Italy: 21+1) – Deadlines: ex-ante, mid-term and ex-post.

– A Common Monitoring and Evaluation Framework (CMEF)

– Programme evaluation: Measure-driven overall approach

– Steps of the evaluation process

– Organization: stakeholders, steering group, evaluation need/services (ToR)

–Tools: Common evaluation questions; Common indicators to be complemented by programme-specific ones (baseline-targets)

Accountability 4

On-going Evaluation: EU framework

I. Improving the competitiveness of rural areas (17 measures)

II. Maintaining and preserving the environment and rural heritage (13 measures)

III. Diversification of rural economy and proving the quality of life in rural areas (8 measures)

IV. LEADER approach (5 measures)

The RDPs: three strategic objectives +a delivering axis

5

In Italy

The context of the on-going evaluation in Italy

+ 21 Rural Development Programmes (regional level)

+ 1 National Rural Development Network Programme (national

level)

• Clair identification of evaluation need: beyond the requirerments; addressing the specific needs for knowledge; territorial approach (evaluation plan).

• Learning process (mutual): implementation and effects of the policy; feeding the policy decision –maker (review and future programming period).

• Evaluation Culture & Ownership, through participative evaluation process and vocational activities

The Italian approach

6Ownership, Capabilities, Utilization-focused

… through …

• Governance: setting up proper structures (M&V unit and steering group); definition of the evaluation need; adequateness of financial resources.

• Communication: transaparency on results and participative dissemination.

• Participative process: let emerging the evaluation need and governing the evaluation process.

• Identify and sharing practises • Coordination: coordination with other territorial policies.• Guiding role for the National Monitoring and Evaluation

Network 7

Italy: the Evaluation budget (%)

Indipendent evaluators in Italy

ESA-AGER (1)

Agriconsulting (8) Vary timing and Delays in selecting 21 RDPs/10 evaluators; 7 Partnerships Evaluation “market” Variety in budgets Relevant Expertise:

Regional policy; Technical Assistance on Rural

policy

Ecosfera (5+NRN)

Agrotec – Disamis (1)

ISRI (1)

In house (1)

ESA (1)

IZI – Appollis (1)

AGER – Starter (1)

Agrotec–Rina Value (1)

10

Steering groups: setting up

10

Set up (6)Foreseen (10)Not esplicitely foreseen (5)

Structure Responsible for the on-going evaluation of RDP M&E Unit Steering Group

Role Manager - Administrative management Manager - Administrative

management Advocacy/Rapresentativiness Support to the MA

Technical and Scientific support

Function

Setting up the evaluation system Setting up the evaluation

system Evaluation needs' assessment Preparation of the tender

Interfacing the indipendent evaluator

Preparation of the tender and management of the indipendet evaluator

Compliance Control of services and products

Refining the evaluation demand

Compliance Control of services and products

Setting up the relations between the stakeholders

Interface with Monitoring unit Preparation of the tender and

management of the indipendet evaluator

Facilitating of evaluation's results utilization

Quality and utilization of data

Facilitating of evaluation's results utilization

Compliance with the regulation Compliance with the

regulation Setting up the relations

between the stakeholders Communication of the

evaluation's results Refining the evaluation

demand

Compliance Control of services and products - Validation

Quality and utilization of data Facilitating the dialogue

between MA and indipendent evaluator

Compliance with the regulation

Compliance with the regulation

Representativeness Low Low Large Low/Medium Low/Medium Participation Low Low Variable Low/Medium High

Governance Models in Italy

11

First findings Evaluation Culture ⇒ Utilization-focused evaluations ⇒ Evidence-

based policies Increasing consciousness on the evaluation needs;Participative

process: “accountability” vs. “learning process” Steering group as pathawy of influence; Continuity: the role of the

evaluation unit; Discussion on the results and Reviewing the programmes

Evaluation capabilities ⇒ Innovation ⇒ more Targeted

Evaluations Competitive evaluation market; Innovative approaches and

methodologies; Increasing the skills and the exchange of practieses Improve the avalability of data and their evaluation-utility

Awarness and communication ⇒ Rural Activicitizenship ⇒ Need Assessment/Responsiveness Reaching the territory: stakeholders & rural population Communication: innovative activities; Tailor-made Sharing practises; second language; dialogue

European and National Evaluation Networks as boosters

Reconducting the empirical evidence to the theoretical framework: evidence-based policy & evaluation influence (Patton, (1998, 2007, 2008); Preskill (1994, 1997, 1998); Kirkhart (2000); Mark & Henry (2003, 2004); Cousins (2005); Barca (2009); Verdung (2009):

The way forward …

Evidence-based policy and

programmes

Use of the evaluations and follow-up actions

Utilization-

focused

evaluations

Thank you for attention

• Simona Cristiano: [email protected] • Alessandro Monteleone: [email protected]

For further information and updatings on the research, please take a look on the website:

www.reterurale.it

14