Evaluation – the why’s, the what’s and the how’s 2014 Dr Basma Ellahi (PI) Cat Crum (PhD Student)

Evaluation – the why’s, the what’s and the how’s

2014

Dr Basma Ellahi (PI)Cat Crum (PhD Student)

What is evaluation?

• Evaluation is a process that critically examines a program. It involves collecting and analysing information about a program’s activities, characteristics, and outcomes. Its purpose is to make judgments about a program, to improve its effectiveness, and/or to inform programming decisions (Patton, 1987).

Why evaluate?

• Answer the objectives of the scheme/project

• Demonstrate change due to the scheme/project

• Determine effectiveness i.e. causality or association

• Inform improvements

‘dull but important

When compared with the creative and exciting process of conceiving and initiating a project, evaluation can often be forgotten and be perceived as dull! However, some form of evaluation or formative feedback is the only thing that will show the effectiveness of the project

• Evaluations fall into one of two broad categories: formative and summative.

• Formative evaluations are conducted during program development and implementation and are useful if you want direction on how to best achieve your goals or improve your program.

• Summative evaluations should be completed once your programs are well established and will tell you to what extent the program is achieving its goals.

Type of Evaluation Purpose

Formative

1. Needs Assessment

Determines who needs the program, how great the need is, and what can be done to best meet the need. A needs assessment can help determine what audiences are not currently served by programs and provide insight into what characteristics new programs should have to meet these audiences’ needs.

2. Process or Implementation Evaluation

Examines the process of implementing the program and determines whether the program is operating as planned. Can be done continuously or as a one-time assessment. Results are used to improve the program. A process evaluation of a program may focus on the number and type of participants reached and/or determining how satisfied these individuals are with the program.

Summative

1. Outcome Evaluation

Investigates to what extent the program is achieving its outcomes. These outcomes are the short-term and medium-term changes in program participants that result directly from the program. For example, outcome evaluations may examine improvements in participants’ knowledge, skills, attitudes, intentions, or behaviours.

2. Impact Evaluation

Determines any broader, longer-term changes that have occurred as a result of the program. These impacts are the net effects, typically on the entire school, community, organisation, society, or environment.

Adapted from Norland (2004), Pancer and Westhues (1989) and Rossi et al. (2004).

Aims of evaluation

• How has the pilot been received by the different participants in each of the health communities/

• What have been the successes and the issues – lessons learned?

• How successful has the pilot been in moving the health communities towards working on the basis of the 5 key principles?

• How successful has the pilot been in helping the health communities meet their objectives?

Evaluation Objectives of the Malnutrition Prevention Pilot Programme

• Confirming outcomes and impact indicators

• Inputs and activities

• Appraising progress

• Identify limiters

• Identify unintended consequences of implementation

• Generate evidence base

• Use self generated data

Methodology

Impact Evaluation Framework

Theory of change

Logic Model

Linkages between inputs, activities, outputs and outcomes

Evaluation Plan

• Why?

• What?

• Who?

• How?

• When?

• Where?

Complex Evaluation

Range of activities-One-off information and/or taster session -Promotional stand-Promotional Campaigns-Health days-Demonstrations-Cooks and eat-Community Enterprise-Training courses

• Combination of evaluation methods

• Qualitative

• Quantitative

The tools of the evaluator

• Quantitative• Monitoring Information• Questionnaire/survey• Experimental evaluation – RCT, case control, cohort etc

• Qualitative• Observation• Interviews• Focus Groups• Case study• Documentation

Methods• Semi-structured interviews

(telephone)

• Action Learning sessions

• Audit - baselines and MUST

• Sampling strategy – purposive sampling

• Pluralistic model

• Ethics

• Frame of reference

What does good look like?

• Good evaluation is tailored to your program and builds on existing evaluation knowledge and resources.

• Good evaluation is inclusive• Good evaluation is honest. • Good evaluation is replicable and its

methods are as rigorous as circumstances allow.

Common dilemmas

• Intellectual property rights / Data protection and data sharing

• Be conscious of multiple roles• Follow informed-consent rules• Respecting confidentiality and privacy• Ethics• Complexity of data collection sites

How do I make evaluation an integral part of my program?

• Making evaluation an integral part of your program means evaluation is a part of everything you do. You design your program with evaluation in mind, collect data on an on-going basis, and use these data to continuously improve your program.

To build and support an evaluation system:

Couple evaluation with strategic planning.

Revisit and update your evaluation plan and logic model to make sure you are on track.

Build an evaluation culture

What are the benefits?

better understand your target audiences' needs and how to meet these needs

design objectives that are more achievable and measurable

monitor progress toward objectives more effectively and efficiently

learn more from evaluation

increase your program's productivity and effectiveness

10 reasons to evaluate your project

1. So you know whether it’s working

2. So you can be adaptable

3. To know how things are working

4. So you’re aware of unintended outcomes

5. To be able to better communicate the value of your work

10 reasons to evaluate your project cont.

6. To focus your work

7. To help look after the people you work

with

8. Build organisational resilience

9. Know why things are working

10. Life is complicated

Make evaluation part of your program; don’t tack it on at the end!

Resources

• Magenta Handbook - https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/220542/magenta_book_combined.pdf

• The Magenta Book: guidance notes for policy evaluation and analysis - http://www.civilservice.gov.uk/wp-content/uploads/2011/09/the_complete_magenta_book_2007_edition2.pdf

Contact details

Centre for Ageing Studies

Faculty of Health and Social Care

University of Chester

Riverside Campus

Chester

Dr Basma Ellahi – Reader in Food and Nutrition

Email: [email protected]

Professor Paul Kingston – Director of The Centre for Ageing Studies, Professor of Mental Health and Ageing.

Email: [email protected]

Cat Crum (PhD Student – sponsored by Age UK South Staffordshire)

Email: [email protected]