Top Banner

Click here to load reader

Evaluating Communication Programmes, Products and Campaigns: Training workshop

May 06, 2015




A one day workshop on evaluating communication programmes, products and campaigns. The main steps and methods are covered with real life examples given. This workshop was originally conducted by Glenn O'Neil of Owl RE for Gellis Communications in Brussels in October

  • 1.Evaluating Communication Programmes, Products andCampaigns: 1 day training workshop for communication professionals Glenn [email protected] www.owlre.comWorkshop originally conducted for Gellis Communications ( in Brussels on 30 October 2009

2. Schedule1. Introduction & definitions 2. Five steps of evaluation 3. Campaign evaluation methodology 4. Programme evaluation methodology 5. Product evaluation methodology 6. Reporting on communications evaluation 2 3. Training objectiveCommunication professionals understand the key concepts of communications evaluation and thus increase their effectiveness in managing evaluation aspects of their projects! 3 4. What is evaluation? Evaluation is the systematic assessment of the operation and/or the outcomes of a program or policy compared to a set of explicit or implicit standards, as a means to contributing to the improvement of the program or policy source: Evaluation, by Susan Weiss, (1998)A form of research that determines the relative effectiveness of a public relations campaign or program by measuring program outcomes (changes in the levels of awareness, understanding, attitudes, opinions, and/or behaviours of a targeted audience or public) against a predetermined set of objectives that initially established the level or degree of change desired source: Stacks, D. (2006). Dictionary of Public Relations Measurement and Research. Institute for Public Relations.4 5. What is communications?Programmes, projects, campaigns and activities that are dedicated to the management of communications between an organisation and its publics Source: Grunig, J. (ed.) (1992). Excellence in Public Relations and Communications5 6. What is communications? A programme is an organised set of communication activities based on target audiences, themes or functions running continuously or for long periods A campaign is an organised set of communications activities, directed at a particular audience usually within a specified period of time to achieve specific outcomes A product is an individual object, such as a publication, website or video created to support a communication activity 6 7. Aim of communications?7 8. What can communications change? What is communications attempting to change? What are the effects desired? How can these effects be categorised? How can these effects be measured? 8 9. When to evaluate? Formative evaluation Process evaluationSummative evaluation Intermediate evaluation Outcome evaluation Mid-term evaluation Ex-post evaluationActivity Impact evaluation Formative evaluationMonitoring (not evaluation!) Baseline evaluation Ex-ante evaluation Appraisal Different terms that mean the same thing! 9 10. Communications evaluation designPost only T EPre-post E T E True & constructed E E cohort studiesTE ETime seriesE ET EEField ET EexperimentsEEMeta-analyses E TE TE E T E E T E10 11. Evaluating Communication Programmes, Products and Campaigns2. Five steps of evaluationGlenn ONeil 12. Five steps of an evaluation All evaluations, communications or otherwise, follow similar steps in how they are carried out:Planning Creation Management Analysis Presentation12 13. Five steps of an evaluation Creation Designing the evaluation methods Designing the templates and tools Designing final report outline Planning Creating inception report Refining inception reportManagement Creating the evaluation framework Testing the templates and tools Selecting the evaluation methods Collecting the data & sample Managing the data Consulting with client Establishing the team Determining the timeframe Determining the deliverables Creating the workplanAnalysis Preliminary desk reviewAnalysing and interpreting the resultsFormulating conclusions Presentation Creating recommendations Presenting the final reportWriting the final report Disseminating the final report Submitting the final report Promoting the final report Creating follow-up mechanisms (e.g. steering group)13 14. Five steps of an evaluation Which steps are usually done well? Which steps are usually done less well or skipped over? 14 15. Evaluating Communication Programmes, Products and Campaigns 3. Campaign evaluation methodology Glenn ONeil 16. Two types of campaigns Advocacy campaignPublic information campaign16 17. Evaluating campaigns In theory, campaigns are easier to evaluate than programmes Where do you start if you have been asked to evaluate a campaign? 17 18. Theory of change A good starting point is to map out the theory of change The theory of change shows the pathway from inputs to impactWhat was this campaign trying to achieve? InputsActivitiesOutcomes Impact18 19. Example: Theory of change Inputs Activities ActivitiesOrganisation-ledPartners-led Planning ofCommunication toolsAdaptation & productioncampaign goals andof campaign material Artistic projects (film,activities cartoon, book, poster) Special events &conferencesPreparation ofSpecial events &campaign materialsconferencesGrassroots mobilisation Consultation and Global day briefing of campaignMedia campaign partnersMedia campaign Web campaign Web campaignTraining programme19 20. Example: Theory of change Outcomes Outcomes OutcomesImpactAlliancesAwareness ActionPeople are Stimulated debate,protected andIncreasedawareness of spurred action andempowered to Network activelyhuman rights inreaffirmed commitment realise their participated in thegeneral amongstof governments, civil rights campaignrights holders society, educational, cultural and humanEngaged multiple rights institutionsstakeholders in thecampaign at thecountry level &Helped bridge gaps inglobally HR implementation at the national levelGarnered further support for the organization20 21. Evaluation framework The theory of change assists in clarifying the objectives of the campaign The next step would be to create the evaluation framework the link from objectives/outcomes to indicators to evaluation methods21 22. Evaluation framework - exampleCampaignProposedMeans of Selection frameOutcomesIndicatorsverification(evaluation tools) 6. Increasedassociation of- Change to level of-Online panel study oforganisation as key association ofindividuals to assess-individuals recruitedactor for todays individuals changes of onlineecologicalassociationchallenges-Change to level ofvisibility in the media - Street polls in major - urban populationof the organisation cities(voxpop) - Event attendance - key eventsstatistics andfeedback - Number of mentions - Selection of printof organisation andand online mediaother key words inthe media and online - Number of visitors - Campaign portalto campaign portal22 23. Evaluation methods Standard evaluation methods are used incampaign evaluationin combination withmethods particularly adapted for campaignsand communications programmes A combination of qualitative and quantitative methods is recommended23 24. Evaluation methods Standard Adapted SurveysExpert reviews Interviews Content analysis Panels Media monitoring Focus groups Web metrics Case studies Tracking mechanisms Observation studiesNetwork mappingWe focus on these methodsas they are special tocommunications!24 25. Expert review A specialist examines a communication activity or product and provides an assessment Assessment is made often against best practices or standards A brochure is compared to the corporateidentity guidelines of an organisationA website is measured against usabilitystandards 25 26. Expert review - example Your ComparativeComparativeComparative organisation organisation A organisation B organisation CHigh-level design decisions and 57%71%71%71% strategyContent design62%85%73%93% Navigation and search 59%87%84%94% Content presentation59%86%86%93% General design aspects45%95%90%92% Overall compatibility 58%86%83%92%Compatibility of organisation web tool and comparative tools with ISO usability standard26 27. Content analysis Media reports, documents or other sources are analysed and categorised to identify trends and patterns Content analysis assists in identifying preference, priorities, trends, etc.27 28. Content analysis - example Theme (Question)Category No. of postsKeywords Policy in the Middle Eastresearch 110 Middle East, Jordan, Iran, participationRwanda, research, quotas, gender, Policy in Rwandaresearch 89leadership Policy and gender research 88gender, legislation Careers in policy workadvise 55consultancy, careers Heads of state and policyresearch 22research, leadership Upcoming elections & policyadvise 18Ecuador, elections, quotas Indicators for measuring policy advise13policy, research, evaluation, Peru Policy and representation advise5 Congo, representation Policy development advise1 New Zealand, development Content analysis of postings in online forum28 29. Content analysis - exampleComparison of media releases & updates by crisesCrisis oneCrisis twoCrisis threeDay of crisis 12 3 4 5 6 7 89 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 1 square = 1 media release or update2 squares = 2 media releases or updates issued on that dayissued on that day 29 30. Media monitoring Media monitoring measures visibility of an issue or organisation in the media Most monitoring counts mentions of keywords in a pre-selected group of media using automated software Media monitoring can be an indication of levels of awareness amongst publics but it is not a replacement! 30 31. Media monitoring - example % of coverage 31 32. Media monitoring - example No. of articles 2500200015001000 5000ly vvnng bayt ccnear rilpt ocnono jaja dede feauju apm se mju Campaign 2008/9Campaign 2005/6 32 33. Web metrics Web metrics is data collected by automated software on visits and other actions on web sites This can be both for an organisations website or a sector Web metrics can measure different variables including interests, preferences, interaction and online b