Evaluating Communication Programmes, Products and Campaigns: Training workshop
Post on 06-May-2015
20050 Views
Preview:
DESCRIPTION
Transcript
Evaluating Communication
Programmes, Products and
Campaigns:
1 day training workshop for
communication professionals
Glenn O’Neiloneil@owlre.comwww.owlre.com
Workshop originally conducted for Gellis Communications (www.gellis.com) in Brussels on 30 October 2009
2
1. Introduction & definitions
2. Five steps of evaluation
3. Campaign evaluation methodology
4. Programme evaluation methodology
5. Product evaluation methodology
6. Reporting on communications evaluation
Schedule
3
Communication professionals understand the
key concepts of communications evaluation
and thus increase their effectiveness in
managing evaluation aspects of their projects!
Training objective
4
“Evaluation is the systematic assessment of the operation and/or the outcomes of a program or policy compared to a set of explicit or implicit standards, as a means to contributing to the improvement of the program or policy”source: “Evaluation”, by Susan Weiss, (1998)
“A form of research that determines the relative effectiveness ofa public relations campaign or program by measuring program outcomes (changes in the levels of awareness, understanding, attitudes, opinions, and/or behaviours of a targeted audience orpublic) against a predetermined set of objectives that initiallyestablished the level or degree of change desired”source: “Stacks, D. (2006). Dictionary of Public Relations Measurement and Research. Institute for Public Relations.”
What is evaluation?
5
Programmes, projects, campaigns and activities that are dedicated to the management of communications between an organisation and its publicsSource: Grunig, J. (ed.) (1992). Excellence in Public Relations and Communications
What is communications?
6
– A programme is an organised set of communication
activities based on target audiences, themes or functions running continuously or for long periods
– A campaign is an organised set of communications activities, directed at a particular audience usually
within a specified period of time to achieve specific
outcomes
– A product is an individual object, such as a
publication, website or video created to support a communication activity
What is communications?
7
Aim of communications?
8
– What is communications attempting to
change? What are the effects desired?
– How can these effects be categorised?
– How can these effects be measured?
What can communications change?
9
When to evaluate?
Monitoring (not evaluation!)
Summative evaluationOutcome evaluationEx-post evaluation
Formative evaluationProcess evaluationIntermediate evaluationMid-term evaluation
Formative evaluation
Baseline evaluationEx-ante evaluationAppraisal
Impact evaluation
Activity
Different terms that mean the same thing!
10
EPost only
TEET
TE
E
E
E
ETE EE
ETE
EE
Pre-post
True & constructed
cohort studies
Time series
Field
experiments
Meta-analyses TE
E
T E
TE T EE
E
Communications evaluation design
Evaluating Communication
Programmes, Products and
Campaigns
2. Five steps of evaluation
Glenn O’Neil
12
All evaluations, communications or otherwise,
follow similar steps in how they are carried
out:
Five steps of an evaluation
Planning Creation Management Analysis Presentation
13
Five steps of an evaluation
Planning
Creation
Management
Analysis
Presentation
• Creating inception report
• Creating the evaluation framework
• Selecting the evaluation methods
& sample
• Consulting with client
• Establishing the team
• Determining the timeframe
• Determining the deliverables
• Creating the workplan
• Preliminary desk review
• Designing the evaluation methods
• Designing the templates and tools
• Designing final report outline• Refining inception report
• Testing the templates and tools
• Collecting the data
• Managing the data
• Analysing and interpreting the results
• Formulating conclusions
• Creating recommendations
• Writing the final report
• Submitting the final report• Presenting the final report• Disseminating the final report
• Promoting the final report
• Creating follow-up mechanisms (e.g. steering group)
14
– Which steps are usually done well?
– Which steps are usually done less well or
skipped over?
Five steps of an evaluation
Evaluating Communication
Programmes, Products and
Campaigns
3. Campaign evaluation
methodology
Glenn O’Neil
16
Public information campaign
Advocacy campaign
Two types of campaigns
17
– In theory, campaigns are easier to evaluate
than programmes
– Where do you start if you have been asked to
evaluate a campaign?
Evaluating campaigns
18
– A good starting point is to map out the “theory
of change”
– The theory of change shows the pathway
from inputs to impact
What was this campaign trying to achieve?
Theory of change
Activities OutcomesInputs Impact
19
Partners-led
Training programme
Example: Theory of change
ActivitiesInputs
Planning of campaign goals and activities
Preparation of campaign materials
Consultation and briefing of campaign partners
Special events & conferences
Grassroots mobilisation
Media campaign
Web campaign
Adaptation & production of campaign material
Activities
Organisation-led
Special events & conferences
Global day
Web campaign
Artistic projects (film, cartoon, book, poster)
Communication tools
Media campaign
20
Outcomes Impact
Alliances
Network actively participated in the campaign
Engaged multiple stakeholders in the campaign at the country level & globally
Awareness
Increased awareness of human rights in general amongst rights holders
Action
Stimulated debate, spurred action and reaffirmed commitment of governments, civil society, educational, cultural and human rights institutions
Helped bridge gaps in HR implementation at the national level
Garnered further support for the organization
Outcomes Outcomes
People are
protected and
empowered to
realise their
rights
Example: Theory of change
21
– The “theory of change” assists in clarifying
the objectives of the campaign
– The next step would be to create the
evaluation framework – the link from
objectives/outcomes to indicators to
evaluation methods
Evaluation framework
22
Evaluation framework - exampleCampaignOutcomes
Proposed Indicators
Means of verification(evaluation tools)
Selection frame
6. Increased association of organisation as key actor for today’s ecological challenges
- Change to level of association of individuals
-Change to level of visibility in the media of the organisation
-Online panel study of individuals to assess changes of association
- Street polls in major cities(voxpop)
- Event attendance statistics and feedback
- Number of mentions of organisation and other key words in the media and online
- Number of visitors to campaign portal
-individuals recruited online
- urban population
- key events
- Selection of print and online media
- Campaign portal
23
– Standard evaluation methods are used in
campaign evaluation…in combination with
methods particularly adapted for campaigns
and communications programmes
– A combination of qualitative and quantitative
methods is recommended
Evaluation methods
24
Standard
Surveys
Interviews
Panels
Focus groups
Case studies
Observation studies
Evaluation methodsAdapted
Expert reviews
Content analysis
Media monitoring
Web metrics
Tracking mechanisms
Network mapping
We focus on these methods as they are special to communications!
25
– A specialist examines a communication
activity or product and provides an
assessment
– Assessment is made often against best
practices or standards
A brochure is compared to the corporate
identity guidelines of an organisation
A website is measured against usability
standards
Expert review
26
Expert review - exampleYour organisation
Comparative organisation A
Comparative organisation B
Comparative organisation C
High-level design decisions and strategy
57% 71% 71% 71%
Content design62% 85% 73% 93%
Navigation and search59% 87% 84% 94%
Content presentation59% 86% 86% 93%
General design aspects45% 95% 90% 92%
Overall compatibility58% 86% 83% 92%
Compatibility of organisation web tool and comparative tools with ISO usability standard
27
– Media reports, documents or other
sources are analysed and categorised to
identify trends and patterns
– Content analysis assists in identifying
preference, priorities, trends, etc.
Content analysis
28
Content analysis - example
Theme (Question) Category No. of posts KeywordsPolicy in the Middle East research 110 Middle East, Jordan, Iran, participation
Policy in Rwanda research 89Rwanda, research, quotas, gender, leadership
Policy and gender research 88 gender, legislation
Careers in policy work advise 55 consultancy, careersHeads of state and policy research 22 research, leadership
Upcoming elections & policy advise 18 Ecuador, elections, quotas
Indicators for measuring policy advise 13 policy, research, evaluation, PeruPolicy and representation advise 5 Congo, representation
Policy development advise 1 New Zealand, development
Content analysis of postings in online forum
29
Content analysis - example
Comparison of media releases & updates by crises
Crisis one
Crisis two
Crisis three
Day of crisis 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35
1 square = 1 media release or update 2 squares = 2 media releases or updates
issued on that day issued on that day
30
– Media monitoring measures visibility of an
issue or organisation in the media
– Most monitoring counts mentions of
keywords in a pre-selected group of media
using automated software
– Media monitoring can be an indication of
levels of awareness amongst publics – but
it is not a replacement!
Media monitoring
31
Media monitoring - example% of coverage
32
Media monitoring - exampleNo. of articles
0
500
1000
1500
2000
2500
nov
dec jan
feb
mar
april
may
june
july
aug
sept
octnov
dec jan
Campaign 2008/9 Campaign 2005/6
33
– Web metrics is data collected by
automated software on visits and other
actions on web sites
– This can be both for an organisation’s
website or a sector
– Web metrics can measure different
variables including interests, preferences,
interaction and online behaviors
Web metrics
34
Web metrics - example
Language – no. articles (%) Language of visitors (%)
English 90 69
Chinese 6 15
Russian 3 4
French 0.5 3
German -- 2
Spanish
--
2
Other -- 5
Combination of content analysis (language content) with web metrics (language of visitors per computer settings) for online portal
35
– Tracking mechanisms record actions
taken on issues, policies, legislation, etc.
– Tracking mechanisms are usually
manually tracked on standard forms in a
systematic manner
Recording how many partners join a
campaign
Tracking and recording the number of
business leaders that speak out on an
issue
Tracking mechanisms
36
Tracking mechanisms - exampleCampaign year
37
– Network mapping measures the relations
and flow between people, ideas and
organisations
– Network mapping is useful in measuring
growth of networks and interconnectivity
between publics and issues
Network mapping
Network mapping - example
Before After
Conference participants – networks
Network mapping - example
No content
Legend
Mostly out-of-date Mostly up-to-date
Size of square indicates number of visits Connecting lines indicate users have visited both directories
Thickness of connecting line indicate number of users that have visited both directories
Network map of directories of online portal combining web metrics (number of visits per directory), content analysis (data updated or not) and user survey data (visits to which
directories)
Evaluating Communication
Programmes, Products and
Campaigns
4. Programme evaluation
methodology
Glenn O’Neil
41
– This type of evaluation examines a series
of communication activities grouped under
a programme, for example:
By target audience: communication
programme aimed at young people
By function: online communication
programme
By theme: communication programme on
health policy
Programme evaluation
42
– The evaluation steps are the same as for
campaign evaluation, but will normally be over
a longer period (2 – 6 months)
– More consultation and interim meetings with
the client are usually built into the evaluation
planning
Evaluation steps
Planning Creation Management Analysis Presentation
43
Programmes are typically more difficult
to evaluate than campaigns because:
– They are often on rolling timeframes with
no clear end
– They often have unclear or very broad
objectives
– They often lack an institutional memory on
past activities and achievements
However, organisations increasingly need to
evaluate such programmes!
Programme evaluation
44
– A similar methodology can be applied as
to campaigns
– Determining the programme’s objectives
and defining the evaluation framework are
key
Programme evaluation
45
Programme evaluation can focus on three
distinct areas:
– Process: how has the programme been
managed?
– Outcomes: what has the programme
achieved?
– Impact: what has the programme contributed
to overall?
Programme evaluations can combine
elements of all three!
Programme evaluation
46
Questions for completing the evaluation
framework:
– Are programme objectives documented?
– Does any baseline data exist?
– Has any programme monitoring being done?
– What is the balance between “outcome”,
“process” and “impact” evaluation questions?
Evaluation Framework
Indicators ToolsObjectives /Questions Source
47
Standard
Surveys
Interviews
Panels
Focus groups
Case studies
Observation studies
Onsite visits
Evaluation methodsAdapted
Expert reviews
Content analysis
Media monitoring
Web metrics
Tracking mechanisms
Network mapping
48
– In ongoing programmes operating across
multiple countries, onsite visits are often
included as an evaluation method
– Onsite visits involve a combination of
observation, interviews and discussions
– An evaluator can observe directly a
programme’s activities, discuss with its
implementers and gain in-depth knowledge
– Onsite visits add credibility to the evaluation
findings
Onsite visits
Evaluating Communication
Programmes, Products and
Campaigns
5. Product evaluation methodology
Glenn O’Neil
50
– Product evaluation is a more narrower
approach focusing on an individual item
(or series of items)
– This evaluation provides feedback on a
product’s use and its contribution to a
communication programme (or other type
of programme)
Product evaluation
51
– Different types of products can be
considered including:
• Promotional videos
• Publications
• Websites and online tools
Product evaluation
52
– Evaluation questions often include:
• Is the product considered to be of high quality in
terms of design, usability and content?
• Is the product targeted to the right audiences?
• Is the product available, accessible and
distributed to the intended target audiences?
Criteria for evaluation
53
– Evaluation questions often include (cont.):
• Is the product used in the manner for which it
was intended - and for what other unintended
purposes?
• What has the product contributed to broader
communication and organizational goals?
• What lessons can be learnt for improving future
editions of the product and design, distribution and promotion in general?
Criteria for evaluation
54
Surveys
Interviews
Focus groups
Case studies
Observation studies
Evaluation methods
Expert reviews
Content analysis
Web metrics
Tracking mechanisms
Distribution statistics
The evaluation methods have to be adapted to
the type of product and can include:
55
Distribution statistics - example
Student / teacher
Fax
orders
5%
National
offices
30%
Promotional
distribution
40%
Web
orders
25%
Local partners
NGOsMedia
Student / teacher
Fax
orders
5%
National
offices
30%
Promotional
distribution
40%
Web
orders
25%
Local partners
NGOsMedia
56
Mapping use - example
Training
Resource Working tool
Policy support
Used for training of
national partners
Used for staff training
Develop teaching
materialsCreate presentations for clients
Charts & tables used in
production
Used as guidelines
for product design
Used by NGOs to
influence debates on regulations
Used by authorities
to revise guidelines
Product
57
Mini case study - exampleCapacity building for women in Uzbekistan, Central Asia
Nargiz, portal member, Uzbekistan
In Uzbekistan, Nargiz, a portal member is part of a group of 50 women who were preparing to run in parliamentary elections. For her, the portal has been a valuable source of support and information.
“In the e-discussions I got important feedback on fundraising strategies and financing of campaigns. This information will be used!”
Nargiz especially mentions an interesting experience from Mauritius, shared on the portal, which she can apply in her daily work at the Women’s NGO Forum of her country
“The material on capacity building is also very useful for us. I hope that in the future, we can share more of our own resources with the network.”
58
Devising precise questions
• In all communications evaluations, if an evaluation
framework exists, it should be relatively easy moving from indicators to questions or criteria for collecting data
• But these questions and criteria must be created, documented and shared with the persons undertaking
the evaluation
• Questions and criteria would normally be documented in
templates and guides
59
Portal facilitatesa global
exchange ofresources
Are resources
being exchanged?
Objectives Evaluation questions Indicators Precise questions
Devising precise questions
By whom, what type,
within what regions and at which frequency?
- Level of usage of resource areas of website
- Frequency and type of
resources exchanged
- Instances of
uses of resources
How often do you use the
resource section?
Have you contributed
resources?How have you used the
resources found on the
Portal?
- etc.
Extending the Evaluation Framework
We are here!
Evaluating Communication
Programmes, Products and
Campaigns
6. Reporting on communications
evaluation
Glenn O’Neil
61
– We have now jumped to the final phase of the
evaluation
– The presentation phase is often the most
neglected of all the phases
– Evaluation regularly fails in ensuring that
people know of the findings and take action
Reporting and presenting
Planning Creation Management Analysis Presentation
62
Different presentation formats
– Written report
– Written summary (Word or PPT)
– Scorecard or findings table
– Video report
63
Creating readable reports
A Good Evaluation Report is… A Weak Evaluation Report is…
• Impartial • Credible • Balanced • Clear and easy to understand • Information rich • Action oriented and crisp • Focused on evidence that supports conclusions
• Repetitious • too long • Unclear and unreadable • Insufficiently action oriented • Lacking hard data and relying on opinion • Poorly structured and lacking focus on key findings• Lacking comprehension of the local context • Negative or vague in its findings
64
Summary sheet: an example
65
Scorecard:an example
66
Findings table:an example
Summary of the review’s key findings
Expected Results Rating
Outputs
Eight directories of the CR established and
accessible to potential users from the disaster
management community worldwide.
Largely achieved
Eight directories of the CR stocked with relevant,
appropriate and up-to-date information on disaster
management capacities.
Only partially achieved
Outcomes
Potential users from the disaster management
community worldwide learned of the CR.
Very limited achievement
Potential users from the disaster management
community worldwide visited the CR and registered
Very limited achievement
Users obtained information of use to them in one or
more of the eight directories of the CR.
Only partially achieved
Users contributed information from their
organisations to one or more of the eight directories
of the CR.
Very limited achievement
Information found on the CR facilitated the rapid
identification of appropriate disaster management
services.
Very limited achievement
Information found on the CR contributed to the rapid
delivery of humanitarian emergency assistance.
Not achieved
Impact
Delivery of humanitarian emergency assistance
improved. Not measured in this review
67
Video report: an example
http://www.youtube.com/watch?v=q6nKXcUrNXA
68
Follow-up mechanisms
Evaluations may require follow-up
mechanisms to ensure that the findings are disseminated and acted upon, including:
– Workshops with staff and donors to discuss findings
– Steering committees to discuss findings and implementation
– Plans of action based on findings and recommendations of the evaluation
69
A parting quote
Scientific quality is not the principle standard; an
evaluation should aim to be comprehensible, correct and complete, and credible to partisans on all sides
Professor Lee Cronbach
oneil@owlre.com
www.owlre.com
top related