Evaluating Communication Programmes, Products and Campaigns: Training workshop

Evaluating Communication

Programmes, Products and

Campaigns:

1 day training workshop for

communication professionals

Glenn O’[email protected]

Workshop originally conducted for Gellis Communications (www.gellis.com) in Brussels on 30 October 2009

2

1. Introduction & definitions

2. Five steps of evaluation

3. Campaign evaluation methodology

4. Programme evaluation methodology

5. Product evaluation methodology

6. Reporting on communications evaluation

Schedule

3

Communication professionals understand the

key concepts of communications evaluation

and thus increase their effectiveness in

managing evaluation aspects of their projects!

Training objective

4

“Evaluation is the systematic assessment of the operation and/or the outcomes of a program or policy compared to a set of explicit or implicit standards, as a means to contributing to the improvement of the program or policy”source: “Evaluation”, by Susan Weiss, (1998)

“A form of research that determines the relative effectiveness ofa public relations campaign or program by measuring program outcomes (changes in the levels of awareness, understanding, attitudes, opinions, and/or behaviours of a targeted audience orpublic) against a predetermined set of objectives that initiallyestablished the level or degree of change desired”source: “Stacks, D. (2006). Dictionary of Public Relations Measurement and Research. Institute for Public Relations.”

What is evaluation?

5

Programmes, projects, campaigns and activities that are dedicated to the management of communications between an organisation and its publicsSource: Grunig, J. (ed.) (1992). Excellence in Public Relations and Communications

What is communications?

6

– A programme is an organised set of communication

activities based on target audiences, themes or functions running continuously or for long periods

– A campaign is an organised set of communications activities, directed at a particular audience usually

within a specified period of time to achieve specific

outcomes

– A product is an individual object, such as a

publication, website or video created to support a communication activity

What is communications?

7

Aim of communications?

8

– What is communications attempting to

change? What are the effects desired?

– How can these effects be categorised?

– How can these effects be measured?

What can communications change?

9

When to evaluate?

Monitoring (not evaluation!)

Summative evaluationOutcome evaluationEx-post evaluation

Formative evaluationProcess evaluationIntermediate evaluationMid-term evaluation

Formative evaluation

Baseline evaluationEx-ante evaluationAppraisal

Impact evaluation

Activity

Different terms that mean the same thing!

10

EPost only

TEET

TE

E

E

E

ETE EE

ETE

EE

Pre-post

True & constructed

cohort studies

Time series

Field

experiments

Meta-analyses TE

E

T E

TE T EE

E

Communications evaluation design



Campaigns

2. Five steps of evaluation

Glenn O’Neil

12

All evaluations, communications or otherwise,

follow similar steps in how they are carried

out:

Five steps of an evaluation

Planning Creation Management Analysis Presentation

13


Planning

Creation

Management

Analysis

Presentation

• Creating inception report

• Creating the evaluation framework

• Selecting the evaluation methods

& sample

• Consulting with client

• Establishing the team

• Determining the timeframe

• Determining the deliverables

• Creating the workplan

• Preliminary desk review

• Designing the evaluation methods

• Designing the templates and tools

• Designing final report outline• Refining inception report

• Testing the templates and tools

• Collecting the data

• Managing the data

• Analysing and interpreting the results

• Formulating conclusions

• Creating recommendations

• Writing the final report

• Submitting the final report• Presenting the final report• Disseminating the final report

• Promoting the final report

• Creating follow-up mechanisms (e.g. steering group)

14

– Which steps are usually done well?

– Which steps are usually done less well or

skipped over?




Campaigns

3. Campaign evaluation

methodology

Glenn O’Neil

16

Public information campaign

Advocacy campaign

Two types of campaigns

17

– In theory, campaigns are easier to evaluate

than programmes

– Where do you start if you have been asked to

evaluate a campaign?

Evaluating campaigns

18

– A good starting point is to map out the “theory

of change”

– The theory of change shows the pathway

from inputs to impact

What was this campaign trying to achieve?

Theory of change

Activities OutcomesInputs Impact

19

Partners-led

Training programme

Example: Theory of change

ActivitiesInputs

Planning of campaign goals and activities

Preparation of campaign materials

Consultation and briefing of campaign partners

Special events & conferences

Grassroots mobilisation

Media campaign

Web campaign

Adaptation & production of campaign material

Activities

Organisation-led

Special events & conferences

Global day

Web campaign

Artistic projects (film, cartoon, book, poster)

Communication tools

Media campaign

20

Outcomes Impact

Alliances

Network actively participated in the campaign

Engaged multiple stakeholders in the campaign at the country level & globally

Awareness

Increased awareness of human rights in general amongst rights holders

Action

Stimulated debate, spurred action and reaffirmed commitment of governments, civil society, educational, cultural and human rights institutions

Helped bridge gaps in HR implementation at the national level

Garnered further support for the organization

Outcomes Outcomes

People are

protected and

empowered to

realise their

rights

Example: Theory of change

21

– The “theory of change” assists in clarifying

the objectives of the campaign

– The next step would be to create the

evaluation framework – the link from

objectives/outcomes to indicators to

evaluation methods

Evaluation framework

22

Evaluation framework - exampleCampaignOutcomes

Proposed Indicators

Means of verification(evaluation tools)

Selection frame

6. Increased association of organisation as key actor for today’s ecological challenges

- Change to level of association of individuals

-Change to level of visibility in the media of the organisation

-Online panel study of individuals to assess changes of association

- Street polls in major cities(voxpop)

- Event attendance statistics and feedback

- Number of mentions of organisation and other key words in the media and online

- Number of visitors to campaign portal

-individuals recruited online

- urban population

- key events

- Selection of print and online media

- Campaign portal

23

– Standard evaluation methods are used in

campaign evaluation…in combination with

methods particularly adapted for campaigns

and communications programmes

– A combination of qualitative and quantitative

methods is recommended

Evaluation methods

24

Standard

Surveys

Interviews

Panels

Focus groups

Case studies

Observation studies

Evaluation methodsAdapted

Expert reviews

Content analysis

Media monitoring

Web metrics

Tracking mechanisms

Network mapping

We focus on these methods as they are special to communications!

25

– A specialist examines a communication

activity or product and provides an

assessment

– Assessment is made often against best

practices or standards

A brochure is compared to the corporate

identity guidelines of an organisation

A website is measured against usability

standards

Expert review

26

Expert review - exampleYour organisation

Comparative organisation A

Comparative organisation B

Comparative organisation C

High-level design decisions and strategy

57% 71% 71% 71%

Content design62% 85% 73% 93%

Navigation and search59% 87% 84% 94%

Content presentation59% 86% 86% 93%

General design aspects45% 95% 90% 92%

Overall compatibility58% 86% 83% 92%

Compatibility of organisation web tool and comparative tools with ISO usability standard

27

– Media reports, documents or other

sources are analysed and categorised to

identify trends and patterns

– Content analysis assists in identifying

preference, priorities, trends, etc.

Content analysis

28

Content analysis - example

Theme (Question) Category No. of posts KeywordsPolicy in the Middle East research 110 Middle East, Jordan, Iran, participation

Policy in Rwanda research 89Rwanda, research, quotas, gender, leadership

Policy and gender research 88 gender, legislation

Careers in policy work advise 55 consultancy, careersHeads of state and policy research 22 research, leadership

Upcoming elections & policy advise 18 Ecuador, elections, quotas

Indicators for measuring policy advise 13 policy, research, evaluation, PeruPolicy and representation advise 5 Congo, representation

Policy development advise 1 New Zealand, development

Content analysis of postings in online forum

29

Content analysis - example

Comparison of media releases & updates by crises

Crisis one

Crisis two

Crisis three

Day of crisis 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35

1 square = 1 media release or update 2 squares = 2 media releases or updates

issued on that day issued on that day

30

– Media monitoring measures visibility of an

issue or organisation in the media

– Most monitoring counts mentions of

keywords in a pre-selected group of media

using automated software

– Media monitoring can be an indication of

levels of awareness amongst publics – but

it is not a replacement!

Media monitoring

31

Media monitoring - example% of coverage

32

Media monitoring - exampleNo. of articles

0

500

1000

1500

2000

2500

nov

dec jan

feb

mar

april

may

june

july

aug

sept

octnov

dec jan

Campaign 2008/9 Campaign 2005/6

33

– Web metrics is data collected by

automated software on visits and other

actions on web sites

– This can be both for an organisation’s

website or a sector

– Web metrics can measure different

variables including interests, preferences,

interaction and online behaviors

Web metrics

34

Web metrics - example

Language – no. articles (%) Language of visitors (%)

English 90 69

Chinese 6 15

Russian 3 4

French 0.5 3

German -- 2

Spanish

--

2

Other -- 5

Combination of content analysis (language content) with web metrics (language of visitors per computer settings) for online portal

35

– Tracking mechanisms record actions

taken on issues, policies, legislation, etc.

– Tracking mechanisms are usually

manually tracked on standard forms in a

systematic manner

Recording how many partners join a

campaign

Tracking and recording the number of

business leaders that speak out on an

issue

Tracking mechanisms

36

Tracking mechanisms - exampleCampaign year

37

– Network mapping measures the relations

and flow between people, ideas and

organisations

– Network mapping is useful in measuring

growth of networks and interconnectivity

between publics and issues

Network mapping

Network mapping - example

Before After

Conference participants – networks

Network mapping - example

No content

Legend

Mostly out-of-date Mostly up-to-date

Size of square indicates number of visits Connecting lines indicate users have visited both directories

Thickness of connecting line indicate number of users that have visited both directories

Network map of directories of online portal combining web metrics (number of visits per directory), content analysis (data updated or not) and user survey data (visits to which

directories)



Campaigns

4. Programme evaluation

methodology

Glenn O’Neil

41

– This type of evaluation examines a series

of communication activities grouped under

a programme, for example:

By target audience: communication

programme aimed at young people

By function: online communication

programme

By theme: communication programme on

health policy

Programme evaluation

42

– The evaluation steps are the same as for

campaign evaluation, but will normally be over

a longer period (2 – 6 months)

– More consultation and interim meetings with

the client are usually built into the evaluation

planning

Evaluation steps


43

Programmes are typically more difficult

to evaluate than campaigns because:

– They are often on rolling timeframes with

no clear end

– They often have unclear or very broad

objectives

– They often lack an institutional memory on

past activities and achievements

However, organisations increasingly need to

evaluate such programmes!


44

– A similar methodology can be applied as

to campaigns

– Determining the programme’s objectives

and defining the evaluation framework are

key


45

Programme evaluation can focus on three

distinct areas:

– Process: how has the programme been

managed?

– Outcomes: what has the programme

achieved?

– Impact: what has the programme contributed

to overall?

Programme evaluations can combine

elements of all three!


46

Questions for completing the evaluation

framework:

– Are programme objectives documented?

– Does any baseline data exist?

– Has any programme monitoring being done?

– What is the balance between “outcome”,

“process” and “impact” evaluation questions?

Evaluation Framework

Indicators ToolsObjectives /Questions Source

47

Standard

Surveys

Interviews

Panels

Focus groups

Case studies

Observation studies

Onsite visits

Evaluation methodsAdapted

Expert reviews

Content analysis

Media monitoring

Web metrics

Tracking mechanisms

Network mapping

48

– In ongoing programmes operating across

multiple countries, onsite visits are often

included as an evaluation method

– Onsite visits involve a combination of

observation, interviews and discussions

– An evaluator can observe directly a

programme’s activities, discuss with its

implementers and gain in-depth knowledge

– Onsite visits add credibility to the evaluation

findings

Onsite visits



Campaigns

5. Product evaluation methodology

Glenn O’Neil

50

– Product evaluation is a more narrower

approach focusing on an individual item

(or series of items)

– This evaluation provides feedback on a

product’s use and its contribution to a

communication programme (or other type

of programme)

Product evaluation

51

– Different types of products can be

considered including:

• Promotional videos

• Publications

• Websites and online tools

Product evaluation

52

– Evaluation questions often include:

• Is the product considered to be of high quality in

terms of design, usability and content?

• Is the product targeted to the right audiences?

• Is the product available, accessible and

distributed to the intended target audiences?

Criteria for evaluation

53

– Evaluation questions often include (cont.):

• Is the product used in the manner for which it

was intended - and for what other unintended

purposes?

• What has the product contributed to broader

communication and organizational goals?

• What lessons can be learnt for improving future

editions of the product and design, distribution and promotion in general?

Criteria for evaluation

54

Surveys

Interviews

Focus groups

Case studies

Observation studies

Evaluation methods

Expert reviews

Content analysis

Web metrics

Tracking mechanisms

Distribution statistics

The evaluation methods have to be adapted to

the type of product and can include:

55

Distribution statistics - example

Student / teacher

Fax

orders

5%

National

offices

30%

Promotional

distribution

40%

Web

orders

25%

Local partners

NGOsMedia

Student / teacher

Fax

orders

5%

National

offices

30%

Promotional

distribution

40%

Web

orders

25%

Local partners

NGOsMedia

56

Mapping use - example

Training

Resource Working tool

Policy support

Used for training of

national partners

Used for staff training

Develop teaching

materialsCreate presentations for clients

Charts & tables used in

production

Used as guidelines

for product design

Used by NGOs to

influence debates on regulations

Used by authorities

to revise guidelines

Product

57

Mini case study - exampleCapacity building for women in Uzbekistan, Central Asia

Nargiz, portal member, Uzbekistan

In Uzbekistan, Nargiz, a portal member is part of a group of 50 women who were preparing to run in parliamentary elections. For her, the portal has been a valuable source of support and information.

“In the e-discussions I got important feedback on fundraising strategies and financing of campaigns. This information will be used!”

Nargiz especially mentions an interesting experience from Mauritius, shared on the portal, which she can apply in her daily work at the Women’s NGO Forum of her country

“The material on capacity building is also very useful for us. I hope that in the future, we can share more of our own resources with the network.”

58

Devising precise questions

• In all communications evaluations, if an evaluation

framework exists, it should be relatively easy moving from indicators to questions or criteria for collecting data

• But these questions and criteria must be created, documented and shared with the persons undertaking

the evaluation

• Questions and criteria would normally be documented in

templates and guides

59

Portal facilitatesa global

exchange ofresources

Are resources

being exchanged?

Objectives Evaluation questions Indicators Precise questions

Devising precise questions

By whom, what type,

within what regions and at which frequency?

- Level of usage of resource areas of website

- Frequency and type of

resources exchanged

- Instances of

uses of resources

How often do you use the

resource section?

Have you contributed

resources?How have you used the

resources found on the

Portal?

- etc.

Extending the Evaluation Framework

We are here!



Campaigns

6. Reporting on communications

evaluation

Glenn O’Neil

61

– We have now jumped to the final phase of the

evaluation

– The presentation phase is often the most

neglected of all the phases

– Evaluation regularly fails in ensuring that

people know of the findings and take action

Reporting and presenting


62

Different presentation formats

– Written report

– Written summary (Word or PPT)

– Scorecard or findings table

– Video report

63

Creating readable reports

A Good Evaluation Report is… A Weak Evaluation Report is…

• Impartial • Credible • Balanced • Clear and easy to understand • Information rich • Action oriented and crisp • Focused on evidence that supports conclusions

• Repetitious • too long • Unclear and unreadable • Insufficiently action oriented • Lacking hard data and relying on opinion • Poorly structured and lacking focus on key findings• Lacking comprehension of the local context • Negative or vague in its findings

64

Summary sheet: an example

65

Scorecard:an example

66

Findings table:an example

Summary of the review’s key findings

Expected Results Rating

Outputs

Eight directories of the CR established and

accessible to potential users from the disaster

management community worldwide.

Largely achieved

Eight directories of the CR stocked with relevant,

appropriate and up-to-date information on disaster

management capacities.

Only partially achieved

Outcomes

Potential users from the disaster management

community worldwide learned of the CR.

Very limited achievement

Potential users from the disaster management

community worldwide visited the CR and registered


Users obtained information of use to them in one or

more of the eight directories of the CR.

Only partially achieved

Users contributed information from their

organisations to one or more of the eight directories

of the CR.


Information found on the CR facilitated the rapid

identification of appropriate disaster management

services.


Information found on the CR contributed to the rapid

delivery of humanitarian emergency assistance.

Not achieved

Impact

Delivery of humanitarian emergency assistance

improved. Not measured in this review

67

Video report: an example

http://www.youtube.com/watch?v=q6nKXcUrNXA

68

Follow-up mechanisms

Evaluations may require follow-up

mechanisms to ensure that the findings are disseminated and acted upon, including:

– Workshops with staff and donors to discuss findings

– Steering committees to discuss findings and implementation

– Plans of action based on findings and recommendations of the evaluation

69

A parting quote

Scientific quality is not the principle standard; an

evaluation should aim to be comprehensible, correct and complete, and credible to partisans on all sides

Professor Lee Cronbach

[email protected]

www.owlre.com

Evaluating Communication Programmes, Products and Campaigns: Training workshop

Business

evaluation methods

evaluation creation

evaluation framework

evaluation aspects

product evaluation methodology

communications change

management of communications

public relations campaign