Evaluation within the context of the intervention of the Structural Funds in Portugal IFDR 1 October 2008.

Evaluation within the context of the intervention

of the Structural Funds in Portugal

IFDRIFDR1 October 2008

General contents of the presentation

A. Fundamental issues for the evaluation of public interventions

B. Evolution of evaluation processes over the CSF

C. Evaluation in the NSRF

Evaluation

Fundamental issues for the evaluation of public interventions

Evaluation


Relevance – Evaluation of the closeness of the strategy to reality

Efficiency – Evaluation of the way in which resources were transformed into output and results

Efficacy – Evaluation of the way in which the resources contributed towards realising the objectives

Utility – Evaluation of the benefits for recipients

Sustainability – Evaluation of the extent and durability of the alterations made

Aspects to take into consideration in the evaluation:


Relevance

Evaluation of the closeness of the strategy to reality


Efficiency

Evaluation of resources and its outputs


Efficacy

Evaluation of the way in which resources were applied


Utility

Evaluation of the benefits for recipients


Sustainability

Evaluation of the extent and durability of the alterations made


– Proportionality – Partnership – Independence

– Transparency

Principles of Evaluation

Principle of Proportionality Principles of Evaluation

Financial and administrative resources of evaluations

Proportional to the total amount of the expenses allocated to the OP

Principle of Partnership

Cooperation of the major agents throughout the planning process and implementation of evaluations

Principles of Evaluation

Principle of transparency Disclosure of the main results of the evaluations, as well as

their use as a resource for the qualification of public debate ACCOUNTABILITY

Evaluation exercises by entities (internal or external to the Public Administration) functionally independent of the

Management Authorities and of the entities with responsibilities in the NSRF and of the OP

Principle of independence

A. Fundamental issues for the evaluation of the public interventions

B. Development of Evaluation processes over the CSF



Evaluation throughout the CSF

1. Evaluation in the CSF I (1989-1993)

2. Evaluation in the CSF II (1994-1999)

3. Evaluation in the CSF III (2000-2006)

CSF I Evaluation1989-1993

CONCEPTUAL MODEL

Institucional and Functional independence between Management & Evaluation

Different Structures methodologies

Ex-ante, ongoing e ex-post Evaluations In collaboration: Commission - MemberState Monitoring and evaluation functionally & institucionally independent from Management

Observatory of CSF I

Political Decision Makers

Independent Experts Ex-post Evaluation “IFDR” Coordination

Terms of

Reference

CSF II Evaluation

Main changes:

Increasing importance of the ex-ante appreciation and ex-post evaluation

Evaluations were carried out by each Operacional Intervention.

The Commission encouraged the ex-post evaluation of the CSF II.

1994-1999

CSF II Evaluation

Main conclusions:

High performance

Around 7.7% of the GFCF in the period 1994-2000 was directly induced by the CSF II

Around 77.000 jobs were created and maintained up to 1999 directly induced by the investments made in the CSF II

Ex-post evaluation 1994-1999

CSF II Evaluation

Changes into the Information System

Ex-post evaluation 1994-1999

creation of a single Information System Incorporating all of the Funds.

Improve the efficacy of the programmes

Improve monitoring functions

Separation between the activities of project control and monitoring

implementation of a system of management by objectives

Control Monitoring≠

Main Recommendations

CSF III EvaluationEvaluation Phases (2000-2006)Ex-ante Evaluation from the Member State responsibility carried out by independent evaluators

Mid Term Evaluation – December 2003 carried out by independent evaluators - responsibility of the MA allocation of the Performance Reserve

Mid Term Evaluation Update – December 2005

Ex-post Evaluation – 3 Years after programming period carried out by independent evaluators - responsibility of the EC

Diagnosis of the departure situation and analysis of the expected results.

Analysis of the first results of the interventions, taking into account the Ex-Ante Evaluation.

CSF III Evaluation

Ex-ante evaluation = basis for the preparation of the interventions

* being the responsibility of the Member StateChecksstrategy and objectivescoherence execution and monitoring rules planned

3 different moments of evaluation

Mid Term ReviewEx-ante Ex-Post

results of the evaluations

Important input

CSF III Evaluation

Enables:

the knowledge and analysis of the first results

the pertinence and the implementation of the objectives

the follow-up of financial allocations

the functioning of the monitoring and the execution of activities.

* being the responsibility of the Management Authorities in collaboration with the Commission

Mid Term Review

Each OP CSF

Ex-post Evaluation

CSF III Evaluation

Independent evaluators 3 years after end programming period Commission in collaboration with Member State

Intended to report on achievements and effects, the use of resources, the efficacy

and efficiency of the OP, The analysis of the causes of inefficacy, the impacts

Included results of the evaluation of macro-economic impacts

through the analysis of different reports cross-referencing them with the results of the

evaluation studies Performed by independent evaluators – until Dec 2003

Mid Term Review

CSF III Evaluation

Recommended

Changes in OP Mid Term Review

redefinition of strategies and priorities

redistribution of financial allocationsFundsOPPriorities

Performed by independent evaluations

Mid Term Review

CSF III Evaluation

Changes suggested

Intermediate reprogramming of 2004

Update of evaluation in 2005

until Dec 2003

Preparation of the subsequent interventions indicating priorities for the following

programming period

Mid Term Review

CSF III Evaluation

quality of the evaluation exercise

results of the Mid Term Review

4% of the total allocations planned

for each Member State

To reward the more efficient OP

attribution of the efficiency reserve

Reprogramming of 2004

Attribution of the Efficiency Reserve

CSF III Evaluation

Stage 1Stage 1

Selection of the OP which meet the criteria

Stage 2Stage 2

Determination of the amount of the efficiency reserve to be awarded

Efficiency Reserve

CSF III Evaluation

Criteria for all OP - Considering the fulfilment by the OP of good management practices

Man

agem

ent

crit

eria

Management System QualityQuality of the system of controlQuality of the selection criteriaQuality of the evaluation system

Fin

an

cia

l cri

teri

a Global financial execution Financial execution in LVT Leverage effect

Eff

ica

cy

c

rite

ria

Measures the effects of the Funds through output of result indicators

Attribution of the Efficiency Reserve

CSF III Evaluation

Common Management

Criteria

Very efficient: 5 of 6 criteria Efficient: 4 of 6 criteria Not Efficient: 3 criteria

Financial Criteria

Very efficient: at least 80% of output & result IndicatorsEfficient: at least 60% of output & result IndicatorsNot efficient: < 60%

Very efficient: reaches 90% of targets Efficient: 75 – 90% of targets Not Efficient: < 75%

Efficacy Criteria

Results of The Mid-Term Review

CSF III Evaluation

Main Conclusions – 2003

Relevance of the European Strategy For Employment

Reduction of unqualified employment

Reduction of early school leavers

Integration of education investment in companies

Modernization of Public Administration

Focus on Telecommunications and Energy (with direct impact of the Portuguese economy)

Results of The Mid-Term Review

CSF III Evaluation

Main Conclusions - 2005

Slow but progressive increase in the qualification of labour supply

Reduced expression of entrepreneurship in Portugal

High level of performance of OP in terms of approval, without perfect correspondence in terms of financial execution

Mid-Term ReviewCSF III Evaluation

Main Conclusions for 2007 – 2013

Integration of policies of innovation and

productivity

Development of territorial

competitiveness

Active employment policies bringing them closer to the regional

and local dimensions of structural

unemployment

Develop articulation of institutions and

organizations as to avoid losses in the

efficiency

Improve global coordination

Analysis of the evaluation exercises

In the whole period 2000 – 2006In the whole period 2000 – 2006

133 evaluation studies133 evaluation studies

39 = evaluations of an obligatory nature 39 = evaluations of an obligatory nature

One significant investment – but only 3% of the global One significant investment – but only 3% of the global financial allocations of technical assistance of the CSF III financial allocations of technical assistance of the CSF III

44% of the financial resources were intended for 44% of the financial resources were intended for evaluations of an obligatory natureevaluations of an obligatory nature

Evaluation studies of an obligatory nature


20

Mid-Term Review

N.º ofstudies

19 20

Total Cost l (1000 €)

4 086 2 293

o 15 Consultant Companies

o 15 Consultant Companies

Consultant Companies

o 1 Public Administration

o 1 Public Administration

Institutions

o 3 Research Universit. Centers

o 3 Research Universit. Centers

Follow up of Mid Term-Review


Evaluation studies of an obligatory nature

Strong points

Overall quality: credibility and utility Significant progress on analysis of implementation processes; global

rationality of the interventions; quantification of outputs; Institutional involvement and use of the evaluation as a support tool

for decision making

Weak points

Rigid calendars Exaggerated scope Weaknesses in quantification or estimate of impacts


Evaluation studies of an obligatory nature: Recommendations

Good practices of

Generalization of exercises of this nature in management practice

Flexibility and subsidiarity

Disclosure

A. Fundamental issues for the evaluation of the public interventions

B. The evolution of the evaluation processes throughout the CSF

– Evaluation in the CSF I (1989-1993) and CSF II(1994-1999)

– Evaluation in the CSF III (2000-2006)



Evaluation in the NSRF

2007-2013


2007-2013

Strategic purpose:

Operational Programmes of three large Thematic Agendas

knowledge science, technology innovation promotion of high and sustained levels of economic and socio-cultural development and of territorial qualification

EDUCATION and QUALIFICATION

Through:

Equal opportunities.

Structural Funds and Cohesion Fund

Increased efficiency and quality of public institutions.

Promoted by

Short presentation of the NSRF

National strategic goals and priorities

NSRF OP

Renewed Lisbon Agenda

National Public

Policies

EU Strategic Guidelines

Growth and Employment

OUTPUTS


Thematic Operational Agendas

NSRF

Operational Programme

s

Human Potential

Factors ofCompetitivity


Enhancement of the Territory

Articulation of instruments of national public intervention ENDS

PNACE PNPOT

National StrategyFor Energy

National Employment Plan

National Programme for Inclusion

National Plan for Equality

Cultural Development Plan

National Strategic Plan for Tourism

Technological Plan

NSRF

Territorial Development Plans

Autonomous Regions Plans

Nat. Strat. Rural Develop. Plan / FEADER

National Nature and Biodiversity Conservation

Strategy

National Strategy for the Oceans

PEAASAR

Cities Policy

Nat. Forestry Plan

National Water Plan:Hydrographic Basins and

Efficient Use of Water

National Coastal Zone Integrated Strategy

National Waste Plan

PNAC


Characteristics

Before the start of the programming period

During the programming period

After the programming period

Ex anteEx ante

Ex postEx post

On goingOn goingTIM

ING

OB

JEC

TIV

E

To improve the quality, efficacy, efficiency and coherence of the intervention of the funds and of the strategy and execution of the OP.

Evaluation

Evaluation

Strategic

Operational

NA

TU

RE

RE

SP

ON

SI

BIL

ITY EC Ex-postEx-post

MS Ex-anteEx-ante

On-goingOn-going

Analysis of the evolution of an OP or group of OP in relation to the community and national priorities

To support the monitoringof an OP

Characteristics

Environmental Strategic

Evaluation of the effects of the Interventions on the

Environment

Institutional Framework

Evaluation – Responsibility

Institutional FrameworkOn-going Evaluation Responsibility

Strategic Monitoring

Financial and Operational Monitoring

Management, Evaluation,Internal Control

To ensure the efficient use of the Evaluations

To ensure that the evaluations are integrated and considered as a management tool during the implementation of the OP

Evaluation Plan

Covers the whole programming period

Nature of the evaluations

Description of the coordination and articulation mechanisms

An indicative list of Evaluation exercises

Foreseeable calendar

Evaluation PlanIncludes

Strategic evaluations

Operational evaluations

Level :

NSRF

OP/Fund

Parts:

I - Coordination

II – Evaluation activities and reports

From the Mid Term Review to the

on-going Evaluation

Regulation no. 1083 allows for a NEW CONCEPT of Evaluation:

More flexible

In line with the needs of the policy decision-making process and with the more efficient management of resources

More Flexible

By TopicBy OPBy PriorityBy Actions

Major projects

For all OP For one OP

At NSRF level


on-going Evaluation

Two closely related concepts, but with different objectives and functions

Monitoring analysis, monitoring and verification of results

Evaluation analysis and interpretation of the information obtained through the

monitoring, and other sources in order to find and explain the effects of the interventions.


on-going Evaluation

Monitoring Evaluation

Provides quality information and Provides quality information and respective analysesrespective analyses

Helps the decision-making process


on-going Evaluation

Regular monitoring Alerts for the need for Evaluation

Regular follow up of the Evaluation

Ensures implementation of reccommmendations


on-going Evaluation


Practical applications


Practical applications

Reflections of the Evaluation of the CSF III on the NSRF

CSF III NSRF 2007-2013

Insufficient concentration of financing options on the key areas corresponding to the major development problems of the country

Reduction in the no. of operational interventions More refined criteria in the prioritization of projects

Insufficient strategic alignment of operational instruments and of projects supported

On-going strategic monitoring mechanisms Consolidation of rationality centers Articulation between strategic objectives and financing models

Reflections of the Evaluation of the CSF III on the NSRF

CSF III NSRF 2007-2013

Difficulty in making the innovative potential of agents emerge

Dissemination of good practices, monitoring systems, Evaluation and benchmarking, inter-institutional coordination and innovative approaches

Insufficient attention to bolstering the institutional capacity of the Public Administration

Notion of Strategic State:Strategic planning Inter-sectorial coordination Monitoring and Evaluation

Insufficient focus on the quality of the effects, efficiency and sustainability of co-financed operations

Quality of the expense: Relevance of the investments and economic and financial sustainability Results to be achieved and the effects this will bring

SWOT Analysis of the Ex-ante Evaluation of the NSRF

What is it?

It defines a strategy through the relationships existing between the strong and weak points with the most important trends noted in the global external context, whether due to the economy, legal impositions, etc.


The term SWOT comes from the initials of the words:

Strengths

Weaknesses

Opportunities

Threats

What is it?


Weaknesses

Strengths

External context

Threats

Opportunities

Strategy

Internal context

SWOT Analysis of the Ex-ante Evaluation of the NSRFWhat is it?

Opportunities Threats

Limiting exogenous Limiting exogenous factorsfactors

(external analysis)

Strong PointsWeak Points

Existing applicationsExisting applicationsOn-going dynamicsOn-going dynamics(internal analysis)

For a good strategist there are no threats, only opportunities


How?

Strong Points

Weak Points

Opportunities

Threats

Build a table with the four elements: with the strong and weak points on one side and the opportunities and threats on the other

Identify the key elements which help to establish priorities and take strategic decisions


Opportunities Threats

Broadening of the process of integration of Iberian economies

Completion of the High Speed Lisbon – Madrid project

Financial difficulties in realising a fundamental project for the interna-tional connectivity of the Portuguese economy – high speed trains

Extending the market ado to companies which until now have been more focussed on the domestic market (potentially interesting for Portuguese SME in industry and services), within the space of proximity which is the Spanish economy

Growing affirmation of Spain as an European force, affecting the balanced development of Luso-Spanish trade and the maintenance of national decision-making centres in strategic sectors

Analysis of the opportunities and threats which confront the development process, as well as of the strengths and weaknesses of the Portuguese situation.

Strengths Weaknesses

Environment and natural heritage

Generalised coverage of the population in terms of water supply

Insufficient levels of service in basic areas with emphasis on the drainage and treatment of waste waters

Great diversity of natural heritage with high conservational value; 22% of the national territory is calssified with the status of nature protection and conservation

Insufficient protection and enhancement of natural heritage, frequently associated to the lack of knowledge for the management of protected species and habitats


SMART Criteria in the preparation of the Indicator File

The definition of the objectives may follow SMART criteria

Specific

Measurable

Attainable

Realistic

Timely

What is it?

Specific

They should clearly define what is intended

What?

Why?

How?


What is it?

Measurable

If it cannot be measured, it cannot be managed!!

The objectives should be quantifiable


What is it?

Attainable

The objectives should be achievable


What is it?

Realistic

Objectives should be defined taking in to account the effort required to achieve them


What is it?

Timely

Deadline - clarifies when it is intended for


What is it?

Indicator File

S

M

A

R

T


Identification

Planning and Execution

Characterization

Appreciation of the Quality of the EvaluationMid Term Evaluation CSF III

Excellent Good Sufficient Unacceptable

1. Information needs are satisfied

2.Pertinence of the scope of theEvaluation exercise3. Appropriate character of theMethodology

4.Reliability of the Data

5.Validity of the Analysis

6.Credibility of the Results

7.Impartiality of the Conclusions

8.Utility of the Recommendations

9.Clarity of the Report

Qualitative Appreciation Criteria

Thank you for your attention!

Thank you for your attention!

Evaluation within the context of the intervention of the Structural Funds in Portugal IFDR 1 October 2008.

Documents

relevance evaluation

csf i evaluation

reality efficiency evaluation

results efficacy evaluation

objectives utility evaluation

nsrf slide

presentation slide

recipients fundamental