Prof.dr. Branko Kontić, dr. Davor Kontić CIVITAS Elan Study Tour Ljubljana, 23 October 2013 Role of evaluation in the stage of measures and project planning.

prof.dr.prof.dr.Branko Kontić, Branko Kontić, dr.dr.Davor KontićDavor Kontić

CIVITAS Elan Study CIVITAS Elan Study TourTourLjubljana, 23 October Ljubljana, 23 October 20132013

prof.dr.prof.dr.Branko Kontić, Branko Kontić, dr.dr.Davor KontićDavor Kontić

CIVITAS Elan Study CIVITAS Elan Study TourTourLjubljana, 23 October Ljubljana, 23 October 20132013

Role of evaluation Role of evaluation in the stage of in the stage of measures and measures and project planningproject planning

Aim of this discussionAim of this discussionAim of this discussionAim of this discussion

Key message to the EC – over VANGUARD (why not POINTER?; expecting synthesis report) – for improving the CIVITAS strategic evaluation system (top responsibility for quality projects and efficiency of budgeting)

Potential for contributing to the benefits of CIVITAS projects at operational level (e.g., city mobility level, measures level, common measures, transport policy development)

Key message to the EC – over VANGUARD (why not POINTER?; expecting synthesis report) – for improving the CIVITAS strategic evaluation system (top responsibility for quality projects and efficiency of budgeting)

Potential for contributing to the benefits of CIVITAS projects at operational level (e.g., city mobility level, measures level, common measures, transport policy development)

CIVITAS FameworkCIVITAS FameworkCIVITAS FameworkCIVITAS Famework

How does EC know that How does EC know that the project will meet the the project will meet the goals?goals?

How does EC know that How does EC know that the project will meet the the project will meet the goals?goals?

DIS

SE

MIN

AT

ION

METEOR

CIVITAS I GUARD

CIVITAS II POINTER

CIVITASPlus

DIS

SE

MIN

AT

ION

METEOR

CIVITAS I GUARD

CIVITAS II POINTER

CIVITASPlus

Impact evaluationImpact evaluation

Process evaluationProcess evaluation

Integrated package Integrated package evaluationevaluation

City level City level evaluationevaluation

Cost-Benefit Cost-Benefit AnalysisAnalysis

CIVITAS Evaluation Objectives

Impact Evaluation Guidelines for defining

evaluation scenarios Common core indicators Guidelines for measurement Guidelines for up-scaling Guidelines for reporting

Process Evaluation Collecting information on

all stages of the measure Feedback and support Factors of success and

unexpected barriers Input for policy

recommendations

Evaluation at Project and City Levels

Cross-site Evaluation Comparing results across sites Assessing transferability Conclusions and recommendations

CIVITAS Measures

Reporting Evaluation Results

CIVITAS Evaluation Objectives

Impact Evaluation Guidelines for defining

evaluation scenarios Common core indicators Guidelines for measurement Guidelines for up-scaling Guidelines for reporting

Process Evaluation Collecting information on

all stages of the measure Feedback and support Factors of success and

unexpected barriers Input for policy

recommendations

Evaluation at Project and City Levels

Cross-site Evaluation Comparing results across sites Assessing transferability Conclusions and recommendations

CIVITAS Measures

Reporting Evaluation Results

Evaluation Framework Evaluation Framework (by POINTER, 2009)(by POINTER, 2009)Evaluation Framework Evaluation Framework (by POINTER, 2009)(by POINTER, 2009)

Impact Impact Evaluation Framework Evaluation Framework (by POINTER, 2009)(by POINTER, 2009)Impact Impact Evaluation Framework Evaluation Framework (by POINTER, 2009)(by POINTER, 2009)

Detailed Measure Descriptions

Effects/Impacts Indicators

Evaluation Plans

Measure or Grouped Measure level Evaluations

Project & City understandings

Cross-site Evaluation and Transferability

POINTER

‘Baseline’

‘Business-as-Usual’

‘After’

Support for Recommendations to other

EUROPEAN CITIES

BAU and CBA BAU and CBA Evaluation Evaluation (by POINTER, 2009)(by POINTER, 2009)BAU and CBA BAU and CBA Evaluation Evaluation (by POINTER, 2009)(by POINTER, 2009)

Process Evaluation FrameworkProcess Evaluation Framework(by POINTER, 2009)(by POINTER, 2009)Process Evaluation FrameworkProcess Evaluation Framework(by POINTER, 2009)(by POINTER, 2009)

Measure Measure level:level:336 336

measuresmeasures(100%)(100%)

City City level:level:

26 cities26 cities

ImplementatiImplementationonPreparationPreparation OperationOperation

Selection Selection

POINTER POINTER SupportSupport

Measure Process Measure Process Evaluation Form Evaluation Form

Process Topics & Process Topics & Issues FormIssues Form

FocusedFocused Measure Measure Process Process

Evaluation Form Evaluation Form

Selection Selection

Measure Measure level:level:

Ca. 120 Ca. 120 measuresmeasures

(Ca. (Ca. 30%) 30%)

Measure Process EvaluationMeasure Process Evaluation

Focussed Measure Process EvaluationFocussed Measure Process Evaluation

Process Process Topics & IssuesTopics & Issues

(by POINTER, (by POINTER, 2009)2009)

Overview - LCA of measures and evaluationOverview - LCA of measures and evaluationOverview - LCA of measures and evaluationOverview - LCA of measures and evaluation

Phases/activities of a measurePhases/activities of a measure Evaluation of measure’s Evaluation of measure’s and project goalsand project goals

Transfer&upgradeTransfer&upgrade

Process evaluationProcess evaluation

planningplanning

implementationimplementation

operationoperationImpact evaluationImpact evaluation

CBACBA

Missing componentsMissing componentsMissing componentsMissing components

Strategic view in the overall concept (not transparent and explicit; expectations to be provided by POINTER and Project Manager, however not clearly presented/formulated)

Loop, feedback, monitoring (possibility for consistent adaptation in terms of project goals – need for additional evaluation approach/understanding/tools in the planning stage of the project)

Strategic view in the overall concept (not transparent and explicit; expectations to be provided by POINTER and Project Manager, however not clearly presented/formulated)

Loop, feedback, monitoring (possibility for consistent adaptation in terms of project goals – need for additional evaluation approach/understanding/tools in the planning stage of the project)

Missing elements in the planning Missing elements in the planning stage 1/2stage 1/2Missing elements in the planning Missing elements in the planning stage 1/2stage 1/2

Role of the Evaluation Plan• Purpose (holistic view); it is not only a deliverable

• Project management context (consultation); better cooperation needed

• Changes in project implementation (measures’ design, implementation, operation) and links to the evaluation plan

Agreements between MLs and Evaluation Team (objectives, data collection, etc.); see next slide

Performance assessment (PA) of a measure in the context of evaluation; see next slide; PA follows all (internal and external) barriers and drivers, process evaluation follows only outside barriers and drivers

Role of the Evaluation Plan• Purpose (holistic view); it is not only a deliverable

• Project management context (consultation); better cooperation needed

• Changes in project implementation (measures’ design, implementation, operation) and links to the evaluation plan

Agreements between MLs and Evaluation Team (objectives, data collection, etc.); see next slide

Performance assessment (PA) of a measure in the context of evaluation; see next slide; PA follows all (internal and external) barriers and drivers, process evaluation follows only outside barriers and drivers

Missing elements in the planning Missing elements in the planning stage 2/2stage 2/2Missing elements in the planning Missing elements in the planning stage 2/2stage 2/2

Links between evaluation and dissemination: evaluation of dissemination activities, dissemination of evaluation results

Links and preparations to strategic and management role of the evaluation, specifically PA and CBA: Is/was money being reasonably spent? Was it worth doing? What could be done better? Which mistakes we should not repeat? Post “lessons learned” approach is less effective!

Links between evaluation and dissemination: evaluation of dissemination activities, dissemination of evaluation results

Links and preparations to strategic and management role of the evaluation, specifically PA and CBA: Is/was money being reasonably spent? Was it worth doing? What could be done better? Which mistakes we should not repeat? Post “lessons learned” approach is less effective!

A sample of A sample of an an agreement between ML agreement between ML and SEMand SEM

A sample of A sample of an an agreement between ML agreement between ML and SEMand SEM

A sample of the PA sample of the Performance erformance AssessmentAssessment table tableA sample of the PA sample of the Performance erformance AssessmentAssessment table table

Possible improvements Possible improvements 1/31/3Possible improvements Possible improvements 1/31/3

Within the hierarchy the areas of responsibility are not defined between measure, city and project level. Especially the role of the measure leader with regard to the evaluation needs to be defined clearly.

Not everyone knows how evaluation works and what is needed. The measure leaders are not experts in evaluation.

The planned evaluation procedure does not always work according to plan, it needs to be flexible.

Ljubljana SEM signed agreements with measure leaders on the collection of data, this is a good tool to clarify the role of the measure leaders, asking them to provide data on a regular basis.

There is a pressure from the PM (Project Management) regarding ticking boxes only without checking the content of the evaluation and its wider/strategic purpose.

Measure leaders need to understand what evaluation is about!! Evaluation should be continuous activity in line with measure planning and implementation.

Within the hierarchy the areas of responsibility are not defined between measure, city and project level. Especially the role of the measure leader with regard to the evaluation needs to be defined clearly.

Not everyone knows how evaluation works and what is needed. The measure leaders are not experts in evaluation.

The planned evaluation procedure does not always work according to plan, it needs to be flexible.

Ljubljana SEM signed agreements with measure leaders on the collection of data, this is a good tool to clarify the role of the measure leaders, asking them to provide data on a regular basis.

There is a pressure from the PM (Project Management) regarding ticking boxes only without checking the content of the evaluation and its wider/strategic purpose.

Measure leaders need to understand what evaluation is about!! Evaluation should be continuous activity in line with measure planning and implementation.

Possible improvements Possible improvements 2/32/3Possible improvements Possible improvements 2/32/3

Process evaluation is important for all measures not only focus ones; it clarifies whether implementation is progressing well or not.

Ljubljana introduced performance assessments from the early beginning. It was defined as an addition to the process evaluation – covers also internal barriers and drivers, and monitors some key components of a ML's work and performance of the measure itself (e.g. management, implementation, evaluation). The results are aggregated in a score (colour score system with comprehensive explanation of the rules for assigning a specific score/colour).

Internal Progress Report (IPR) serves primarily for management purposes; evaluation results loose their strength in the context of this reporting. Transposing performance assessment scores into "traffic light system” of the IPRs should be consistent.

Process evaluation is important for all measures not only focus ones; it clarifies whether implementation is progressing well or not.

Ljubljana introduced performance assessments from the early beginning. It was defined as an addition to the process evaluation – covers also internal barriers and drivers, and monitors some key components of a ML's work and performance of the measure itself (e.g. management, implementation, evaluation). The results are aggregated in a score (colour score system with comprehensive explanation of the rules for assigning a specific score/colour).

Internal Progress Report (IPR) serves primarily for management purposes; evaluation results loose their strength in the context of this reporting. Transposing performance assessment scores into "traffic light system” of the IPRs should be consistent.

Possible improvements 3/3Possible improvements 3/3Possible improvements 3/3Possible improvements 3/3

Role of the evaluation should be somehow integrated in the project approval procedure by the EC!

Justification: In the case of ELAN more than a year after official start of the project the evaluators were working with MLs and PEM on "what is to be done in the framework of a certain measure and how will the work/measures be evaluated". The consolidation of DoW in terms of evaluation activities is necessary. This could be formalized by a requirement that a DoW quality check is done by the evaluators too, not only by the project manager and project coordinator.

Role of the evaluation should be somehow integrated in the project approval procedure by the EC!

Justification: In the case of ELAN more than a year after official start of the project the evaluators were working with MLs and PEM on "what is to be done in the framework of a certain measure and how will the work/measures be evaluated". The consolidation of DoW in terms of evaluation activities is necessary. This could be formalized by a requirement that a DoW quality check is done by the evaluators too, not only by the project manager and project coordinator.

Improved organization of Improved organization of CIVITAS ElanCIVITAS ElanImproved organization of Improved organization of CIVITAS ElanCIVITAS Elan

European CommissionEuropean CommissionEuropean CommissionEuropean Commission

Political Steering GroupPolitical Steering Group

Project Consortium Project Consortium MeetingMeeting

PartnersPartners Site Site CoordinatorsCoordinators

Evaluation Evaluation DisseminationDissemination Scientific Scientific CoordinatorCoordinator

Project CoordinatorProject Coordinator Project ManagerProject Manager

Project Management GroupProject Management Group

Fin

anci

al &

Ad

min

istr

ativ

e

Fin

anci

al &

Ad

min

istr

ativ

e

Rep

ort

ing

Rep

ort

ing

European CommissionEuropean CommissionEuropean CommissionEuropean CommissionEuropean CommissionEuropean CommissionEuropean CommissionEuropean Commission

Political Steering GroupPolitical Steering GroupPolitical Steering GroupPolitical Steering Group

Project Consortium Project Consortium MeetingMeeting

PartnersPartners

Project Consortium Project Consortium MeetingMeeting

PartnersPartners Site Site CoordinatorsCoordinators

Site Site CoordinatorsCoordinators

Evaluation Evaluation DisseminationDisseminationEvaluation Evaluation DisseminationDissemination Scientific Scientific CoordinatorCoordinator

Scientific Scientific CoordinatorCoordinator

Project CoordinatorProject Coordinator Project ManagerProject Manager

Project Management GroupProject Management Group

Project ManagerProject Manager

Project Management GroupProject Management Group

Fin

anci

al &

Ad

min

istr

ativ

e

Fin

anci

al &

Ad

min

istr

ativ

e

Rep

ort

ing

Rep

ort

ing

ManagerManagerManagerManager ManagerManagerManagerManager ManagerManagerManagerManager

MeasureMeasure

MeasureMeasure

MeasureMeasure

MeasureMeasure

MeasureMeasure

MeasureMeasure

Evaluation Evaluation ManagerManager

Evaluation Evaluation ManagerManager

Evaluation Evaluation ManagerManager

LocalLocal

DisseminationDisseminationManagerManager

DisseminationDisseminationManagerManager

DisseminationDisseminationManagerManager

LocalLocal

Improved organization of Improved organization of CIVITAS ElanCIVITAS ElanImproved organization of Improved organization of CIVITAS ElanCIVITAS Elan