TCA VENet | project evaluation Gerd Beidernikl Center for Education and Economy, Research & Consulting Graz, Austria 1st project meeting 11. September 2005, Cyprus
Dec 10, 2015
TCA VENet | project evaluation
Gerd Beidernikl
Center for Education and Economy, Research & Consulting
Graz, Austria
1st project meeting
11. September 2005, Cyprus
2
the goals of this session
To introduce myself and the ZBW
To develop a common understanding WHY we are evaluating our work and HOW we are going to do it
To present the (preliminary) evaluation concept and to get feedback on it
To make sure the words and concepts used are familiar to everybody
To develop a common understanding of how to develop criteria that could be used as criteria for success
Overview on the upcoming steps
3
Introduction
4
my background
Profession: Sociology, further training in the field of evaluation and quality management
Researcher in the „Center for Education and Economy“, Graz, Austria
5 full time employees
Public and private clients
My main working areas: labour market research, research on vocational education and training, evaluation, statistics and surveys
Ca. 20 evaluation projects in the last 4 years
5
project references on EU level
Evaluation of the ongoing Leonardo project „inclusive training for disability care workers“
Self-evaluation of the Equal project „girls crack IT“
Evaluation of the Styrian Territorial Employment Pacts
Evaluation of the article VI project „GO BEST“
Project management of an ongoing Leonardo project „online consulting tool for IT professions“
Member of a proposal development group in the 6th framework programme, priority 7
Reasearch activities for the CEDEFOP: reports on the Austrian training system, comperative country studies, administrating the ERO database (European Research Overview)
6
workflow of this presentation
Evaluation
What does it mean?
What can it do for us?
What do we need?
Concept
Log FrameElements of tender
Gender Mainstreaming
Defining the Log FrameIndicators
Terms of Reference
Next steps
Timeline
At this meeting
After the meeting
Feedback
Feedback
7
Evaluation
>>There are as many different definitions of „evaluation“ as there are evaluators.<<
8
what is “evaluation”
>>Evaluation is the systematic investigation of the merit or worth of an object or process for the purpose of reducing uncertainty in decision making.<< (Mertens, 1998)
3 types of evaluation – regarding the point in time
Ex-ante or predictive evaluation concept
Acompanying or formative evaluation processes
Ex-post or summary evaluation results
Different context, different focus
Generating and securing knowledge
Controlling
Legitimizing actions
Dialogue with stakeholders, etc…..
9
why evaluation? what’s the benefit?
An evaluation offers a systematic way of approaching a topic benefits for project planning
Gives neutral feedback for all project partners
Informs about performance, efficiancy, effectiveness and quality of the project
Makes the project results visible marketing, proof
Possibility to learn in the runtime of a project
Possibility to counteract in case of anticipated problems
Secure Knowledge
Identify best practice and transferable models
10
what this evaluation should be
Support for the transnational cooperation
Answering the question: Do we deliver what we are supposed to deliver?
Neutral reflexion of what is going on
Suggestions for changes and improvement
Ensuring positive project development by giving periodic input
Fostering the coperation and communication
11
what an / this evaluation should not be
A judgement of good and bad, of right or wrong
Evaluators are no project outsiders, they are project partners
Evaluators are no pitbull terriers, lurking for any mistake
Evaluation ≠ project controlling
No analysis of cost efficiancy
No evaluation of the benefits for the target groups
12
What are your expectations?
13
Evaluation concept
14
logical framework
One commonly used apporach: logical framework = LogFrame
The LogFrame is the starting point of the evaluation
Displays the main logic of the evaluation subject
A skeleton of the project that has to be evaluated
Represents the causal relations between different levels of objectives, actions and results
15
LogFrame
GoalsNeeds Ressources / Input
Processes / Mechanisms
Effects / results
Overall objectives
Indikators
Specific objectives
Expected Products
Impact
Outcome
Output
Performance
Relevance
Effectiveness
Efficiency
Secondary effects - unexpected
sustainability
Context effects
Structure
Concept
16
preconditions
There are commonly shared, defined and agreed goals
Goals have to be measureable
There are defined ressources
There are defined processes
The skeleton of the log frame has to be filled based on information in the kick off stage of the project.
Should be kept in mind in every working session of this meeting.
Will be worked out by the evaluation team after the meeting in cooperation with the project management
17
call for tenders
The call for tender asked for
Evaluation of the agreed procedures, tasks and responsibilities formal evaluation
Evaluation of the results and the processes to reach these results content evaluation
Participation at every meeting
Monitoring of the Gender Mainstreaming Process
Periodically reporting
18
main elements of the tender
Evaluation TCA VENet
Formal Evaluation Content Evaluation
Concept Evaluation
Structural Evaluation
Gender Mainstreaming
Process Evaluation
Result Evaluation
VENet ModelEuropean Mainsteaming
Spport SystemInternet Platform
Final Activity
Gender Mainstreaming – cross-sectional topic
19
formal evaluation - concept
Concept analysis
Assessment of the codified project structure and plan
Assessment of management tools
Assessment of communication tools
Data
project documents (e.g. proposal)
meetings
protocols (e.g. steering group)
interviews
20
formal evaluation - structure
Structural analysisFocus: transnational cooperation als system of interacting
organizations resp. individuals
Assessment of flow of communication
Assessment of the cooperation patterns
DataHard facts: contribution on formal level (reports, website,..)
Soft facts:Social network survey (x2, email)
Satisfaction survey (x2, email)
Event questionnaire (x5, at every project meeting)
Interviews
Documents, Internet Platform (?),….
21
Social network analysis (SNA)
SNA – research method that focuses on the relation (communication, cooperation, exchange of information,…) between any given set of actors.
A SNA survey asks the partners to rate their own network behaviour according to different criterias (frequeny of interaction, channels of interaction, topics of interaction,…).
The answers on these questions are producing a picture of the overall network. All individual answers are displayed on the network level.
Benefit
You get SNA plots of the network structure
Key fiugures on well and not well established elements
22
network plots
23
Gender Mainstreaming
GM are the efforts we make to ensure that gender aspects and gender equality is considered in all parts and stages of the project (working plan, objectives, actions,…).
A cross sectional topic
GM aspects are considered in every part of the evaluation
and the evaluation itself
3 main GM approaches
Equal opportunities approach neutral treatment
Diversity approach awareness of the different needs
Affirmative action approach focus on women
24
Gender Mainstreaming
Formal Evaluation Content Evaluation- Gender sensitive language- Roles and functions- GM as a topic at the meetings- GM as cross sectional topic in the project- Gender aspects in documents- Treatment of GM in general- GM experience of the partners-…
- Is GM considered by the partners in
- the analysis of underling
problems
- the definition of goals and
target groups
- the design of the processes
- the implementation
- Do the products reflect GM principles?
- Design / content
- Dissemination
- Benefit
Data: protocols, interviews, survey questions, products,…
25
Content evaluation
Two levels
Processes
Results
Main questions
Have the intended results been reached (quantity and quality)?
Have the processes to reach these results been carried out as intended?
Did the partners contribute as scheduled?
Are there any positive or negative deviations? If so, why?
Which deductions for the following project work can be made?
26
Content evaluation
For each of the five main products
VENet model, internet platform,…..
Data
Documents
Interviews
The products themselves
Check lists resp. progress reports for the project (in cooperation with the project management)
Questions in the email survey
27
deliverables of the evaluation
Short reports before each meeting
Distributed electronicly (internet platform)
Participation at every transnational meeting (1 w.d.)
Presentation of evaluation results and workshop on the findings (1-2,5h)
1 mid-term report
1 final report
28
Feedback on the evaluation model What is your impression?
Does the evaluation provide the information we need?GM in our project?
How do we take the evaluation results up?
29
Defining the LogFrame
30
Defintion
By now the LogFrame is empty
Will be based on the transnational project plan with defined goals, milestones and responsibilities
Next Step: Filling the LogFrame with information
Results from this meeting (e.g. working plan,…)
Documents
In cooperation with transnational coordination
The rough LogFrame will increase in its density as the project goes on
31
Definition
The defined aims of the project are crucial for the evaluation!
Important: If you set up any aims, goals and objectives:
be SMART
S…specific: general aims must be broken down
M…measurable: no untouchable aims
A…acceptance: agreement among all partners
R…realistic: no trivial aims, no unrealistic visions
T…timely fixed and with fixed responsibilities
32
Definition
The LogFrame is the catalogue of the evaluation.
Beneath this we will set up the so called Terms of Reference
Kind of evaluation handbook, the practice version of the tender
Puts down the procedures of the evaluation
Commonly shared evaluation principles
Will beset up by the ZBW
33
Next Steps
34
At this meeting
Consider the LogFrame allready when working out objectives (SMART), products and processes
Evaluation grows as the project grows.
Some short interviews on the 2nd resp. 3rd day of the meeting – transnational coordinators
A concluding event questionaire in the last session of the meeting
Define one main contact person regarding the evaluation in each partner organization
35
After the meeting
ZBW will work out the LogFrame in cooperation with the transnational coordination
Set up the Terms of Reference
Set up dates for the next evaluation steps
Inform the partners about the evaluation
36
Timeline
37
Your responsibilities!
To commit and take active part in the evaluation process
To fill in questionnaires carefully
To submit documents and information when asked for
To return you contribution to the evaluation in time – respect deadlines
To evaluate the evaluation! Feedback is welcome!!
38
Contact information
Thank you!Let‘s get it on!
Mag. Gerd BeiderniklZentrum für Bildung und Wirtschaft
Joanneumring 5/4, 8010 Graz, Austria (valid 1st October 2005)
Mail: [email protected]: +43 / 316 / 72 17 44 -13
Fax: +43 / 316 / 72 17 44 -21Mobile: +32 / 650 / 46 21 061