A
• Welcome & Introductions
• GCS Mandatory Evaluation Project (MEP) - Overview
B• Why evaluate
• Barriers to evaluation
C
• How to go about it• 4 stage evaluation process - The big IDIA• Main performance categories - KPI indicators sources• Take Away - Tips
D• GCS Mandatory Evaluation Project (MEP) - Update & Next steps
Agenda
Group interactive session
Government Communications Review: Summary findings
• Tactically strong, strategically weak Half of departments had no communication strategy
• Poor links between policy and communications
• Major activity rarely evaluated
• Inconsistent and variable standards and approaches
• Little join up across departments
• Digital skills falling behind private sector
GCS Mandatory Evaluation Project – Overview
Mandatory Evaluation Project (MEP) is a key part of Government Communication reform programme.
Objective: To create a ‘step change’ in approach to evaluation of communication activities across departments.
Four Key deliverables:
1.Develop high-level performance frameworks and dashboards.
2.Introduce mandatory evaluation training across all comms disciplines.
3.Enhance evaluation capability across the GCS.
4.Develop an evaluation ‘centre of excellence’ & promote best-practice.
A
Agenda13.45 – 14.00 Welcome and introduction – Paul
14.00 – 14.10 Scene setting: Conference theme, aims and objective
14.10 -14.30 Guest Speaker: Selvin Brown - DWP Testimony
14.30 – 14.50 Vision, objectives and challenges for Government Communications
Alex Aiken, Exec Director of Government Communications
15.00 -15.45 Group Exercise
15.45 – 16.00 TEA BREAK
16.00 – 16.30 Guest Speaker: Spenser Fox (CEO, Reputation Institute
Topic: Significance of Reputation as a key outcome measure
16.30 – 16.50 Guest Speaker: Gareth Evans (Independent Consultant-MR2)
Topic: Communication ROI case study for public sector
16.50 – 17.05 Guest speakers: Q & A Panel
17.05 -17.15 Evaluating communication activity – Never an optional extra
Sean Larkin, Head of Government Communications Policy and Capability
17.15 -17.20 Next Steps & Close
Departments & ALBs - Membership
40 Depts. /ALBs
80 ‘evaluation champions’
A1
ST
EP
CH
AN
GE
No Evaluation happening at Campaign / Project level
Evaluation @ Project / Campaign level but no overall Comms KPI’s
(Outcome measures)
Evaluation & Comms KPI’s exist but partially aligned
with dept. strategic priorities
Fully functional Comms performance centre (hub) aligned
with dept. strategic priorities
Stage 1
Stage 2
Stage 3
Stage 4
Dept. Outcomes measures aligned
with Govt. priorities
TIME
Journey (start)…
39%
5%
56%
A2
John WanamakerInventor of mass
retailing in the United States
"Half the money I spend on advertising is wasted;
the trouble is I don’t know which half."
Context• Austere times
Need to make every £ count View as an investment not an expense
• Media landscape & consumption patterns
Evidence of what works and what does not Optimise use of scarce resources
• The role of communications under scrutiny
How it supports achievement of policy outcomes Business planning & activity prioritisation Sharing best practice
B1
1111
Barriers – stopping it happening
Insufficient time / resource / budget
Lack of SMART or unrealistic policy
objectives / targets
Difficulty accessing the right data / tools
Culture & entrenched behaviours
Gaps in evaluation standards & capability
Lack of integrated communication
strategy linked to policy objectives
B2
12
Strategic Alignment
Start with the policy aim, then develop communications objectives that help deliver this.
Objectives should be measurable, focused on outcomes not outputs, and related to changing attitudes and/or behaviour.
Setting the right objectives is crucial
C1
Evaluation stages –The Big IDIAC3
IdentifyThe scope of your project
DevelopYour evaluation plan
ImplementGather data to measure
performance
Analyse & reportPerformance against plan
1
2
3
4
Task 1: Define what you need to evaluate by asking:• What activity am I evaluating?• What do I know & what factors could affect the outcome?• What is my evaluation expected to achieve?Output: Summary of your proposed evaluation approach
Task 2: Define how you’ll measure success:• Set SMART objectives & define your target audience• Map out how activity will work• Set performance metrics (KPIs) & agree baseline / targets Output: Draft evaluation plan
Task 3: Identify and gather evaluation data:• Make most of existing data• Gather additional data (research, feedback & proxies) • Review data gaps (more budget ?) manage expectations Output: Completed evaluation plan
Task 4: Assess the success of your activity:• Analyse effectiveness & provide insights for future• Demonstrate efficiency and value for money • Demonstrate role of communications in supporting
policy objective delivery (outcome)Output: Final evaluation report
1. Strategic alignment – Ensure activity objectives are SMART, aligned and supports policy delivery.
2. Business impact – Always measure true business impact (outcome) rather than outputs and out-takes
3. Big IDIA – Try to adopt the suggested frameworks and follow the big IDIA stages.
4. Continuous improvement – Ensure results drive appropriate actions and any learnings inform future activities.
5. Best practice – Be objective, share results and make evaluation an integral part of your communications approach.
Top TipsC5
D1: Performance Frameworks & Dashboards
1. Performance frameworks /dashboards (Phase one: 37 departments /ALBs) 33 depts. /ALBs (89%) with complete or near complete frameworks. 27 depts./ALBs (73%) expected to have fully functional dashboards by April 14.
Headline measures Audience Team activity / Performance indicators Resource Risks / Concerns
Stakeholders
Partners
Staff
Claimants
Staff
People • FTE (actual vs. plan)• FTE (team and profiles)• Flexible resource stats
Financials • Allocations (Staff vs. Non-staff costs)• Spend vs. Forecast• ERF submissions / spend (teams)
1 2 3 4 5
Directorate Targets
• Sickness levels: AWDL (teams)• Directorate engagement score• HSE incident:• Compliance targets (e.g.) mandatory training performance discussions held Skills manager
Risk Impact Action Lead
1 H
2 M
3 L
Risk register
No Raised by date Status
1
2
3
3 C’s / Hot Topic register
Press office ‘Forward Look’ – up coming events
12 months rolling activity communications plan
Prioritisation activity list
Planning Section
New
Existing
6
SRP1: Welfare Reform
Business Objective: Introduce the Universal Credit and other reforms to simplify the welfare system; Ensure that the system always incentivises work and work always pays; Help to make the welfare system affordable longer term. Comms Objective / Outcome:
WSR
Team / Area Input / output / outcome indicators)
Universal Credit
• Milestones – performance up date• Nos. engaged / events held• No. of web visits / downloads. • No. of comms issued• Staff wavelength / People survey• % Net positive media coverage
SRP2: Labour Market (GBW)
Business Objective: Implement and manage the Work Programme; Deliver personalised package of support to get people into work. Comms Objective / Outcome:
Stakeholders
Employers
Individuals
Staff GBW
Team / Area Input / output /Outcome (indicators)
Labour Market
• Milestones – performance up date• Nos. events held / attendance levels• No. of web visits / downloads. • No. of comms issued• Staff wavelength / People survey• % Net positive media coverage
SRP3: Tackling Poverty
Business Objective: Develop a welfare system that recognises work as the primary route out of poverty; Implement a the Social Justice and Child Poverty Strategies that focuses on eradicating child poverty by 2020. Comms Objective / Outcome:
POVERTY
Team / Area Input / output metrics (indicators)
Social Justice
• Milestones – performance up date• Nos. events held / attendance levels• No. of web visits / downloads. • No. of comms issued• Staff wavelength / People survey• % Net positive media coverage
Stakeholders
OGDs
General public
Staff
SRP4: Pensions Reform (PR)
Business Objective: Provide decent State Pensions; Encourage employers to provide high quality pensions and make automatic enrolment and higher pension saving a reality; Phase out the default retirement age
Comms Objective / Outcome: PR
Team / Area Input / output metrics (indicators)
Pensions
• Milestones – performance up date• Camp tracker data (Ind / employers)• No. of web visits / downloads. • No. of comms issued• % Net positive media coverage• Auto enrolment data
Stakeholders
OGDs
General public
SRP5: Disability Equality
Objective: Develop and implement a disability strategy to enable disable people fulfil their potential; Support more independent living for disable people through an effective strategy and evidence-based policies for disability benefits; Support disabled people take up employment opportunities through specialist programmes. Comms Objective / Outcome:
DISABILTY
Team / Area Input / output metrics (indicators)
A2 work
DLA
PIP
ESA
• Milestones – performance update• No. of events (dialogues) carried out• No. of partnerships established• No. of web visits / downloads. • No. of comms issued• % Net positive media coverage• % take up rates (PIP)
Stakeholders
OGDs
General public
Disabled people
Claimants
Staff
SRP6: Improving our services
Objectives
• Effective communications delivery
• Staff engagement improvements
• Public and Corporate info delivery
• Secure and effective digital comms
• Resource management / R&E
Outcomes:
People Survey – Headline scores
People survey scores for: B31,41,47,50-54, B06-08.
SERV I CES
Teams Input / output metrics (indicators)
Internal Comms
• Milestone – performance update• No of staff comms issued• No. of proactive briefings (SCS & Line mgrs)• No. of bright ideas (submitted vs. Implemented)• Intranet staff visits to: Home page / DWP Story pages Have your say / forum discussions• Wavelength scores (QB1, QB5,QC1,QE1, QE3)
Corp Info
• No. of leaflets / forms delivered or reduced• No. of evaluation reports requested / delivered• No. of jobs successfully delivered• L&D core training delivered / attendance levels• People survey scores for: B22, B25-26
R&E
Business MI*
Existing
D2: Mandatory Evaluation Training
2. Evaluation training - Free half-day courses: 34 training sessions delivered. 541GCS staff trained to date. 4,402 unique page views to our online evaluation guide.
Evaluation Training (400 free places)*
• Introduction to Evaluation for Marketing
• Introduction to Evaluation for Press & Media
• Guest speaker Events
* NB: The nature of the programme this year 2014/15 under review (TBC)
D3: Evaluation Standards & Capability
• New evaluation standards – Three levels
• Capability audit to start soon - April / May 2014
• Tailored package of support & training for each level
Advance
Practitioner
Foundation
D4: Evaluation ‘ centre of excellence’
• Established an Evaluation Council - Industry expert who meet regularly and act as a sounding board for new evaluation thinking as well as to review and validate government evaluation plans and outputs.
• Creating a repository for:
Case studiesEvaluation GuidePerformance Framework Top tipsStandard Metrics
What Next
• GCS Live - 1st April 2014
• Govt. Comms Plan launched – 12th May
• Capability Audit – April /May 2014
• Developing a tier package of support for each level
• Phase 2 roll-out across all departments / ALBs (330)
• Regular updates to GCS Programme board and Council Invite departments / ALB to share their experience & approach
• Lots more events planned – Register on GCS now!
24
Thank you!
Contact: Paul NjokuEmail : [email protected] link to guide: https://gcn.civilservice.gov.uk/guidance/evaluation/
ProductsCustomer Service
InvestmentsEmployment
BrandingPublic Relations
MarketingSocialResponsibility
MEDIA (Traditional, Social)
Topic Experts, Leaders,Friends/Family
Perceptions & expectations
Supportive Behaviour
Results
What your departmentSays/Does
Direct Experience
What Others Say
How reputation is created
The attributes and dimensions have different meanings and importance for different stakeholders. Beneath the 7 dimensions, 32 attributes underpin the individual dimension themes. Different stakeholder groups typically have unique attributes that are found more important than others (reputation drivers).
Reputation Attributes
Reputation Dimensions
The seven dimensions specify at a more operational level, which aspects are most important for stakeholders’ perceptions and expectations – i.e. what’s driving a company’s reputation
Reputation StrengthA measure of the emotional connection.
Reputation has a positive/negative impact on support. An increase in reputation = an increase in support. Support (such as buying products and services, saying something positive, giving the benefit of doubt in times of crisis (etc.) leads to increased business results
Supportive Behaviour
Reputation drivers & dimensions
Measuring outputs
28
Content metricsFavourability
MessagesThemes / Issues
CampaignsArticle type
Proactive / reactiveProminence
Calls to action
Delivery metricsVolume
Opportunities to seeReach and Frequency
Media typeKey publications
JournalistsSpokespeople
Regional breakdownImpact measures
Output Metrics
Are we saying the right thing... ....to the right audience?
Activity
Intermediary Effect
Audience Effect
‘Earned’ mediaModel
Amplification from social media
Government analysis audit
29
Are we saying the right thing...
....to the right audience
Metric% of projects
Favourability 100%Message delivery 91%Campaign tracking 48%Theme / issues tracking 57%Spokespeople / commentators 78%Prominence 57%Article type 43%Proactive / reactive 22%Calls to action 39%Volume (number of articles) 100%Opportunities to see (OTS) 74%Reach and frequency 78%Impact measures 26%Media type 83%Regional split 57%Publications 87%Journalists 87%Advertising value equivalent AVE) 57%Cost per thousand (CPM) 22%Linking ouput to outcomes 22%
Content Metrics
Delivery metrics
Efficiency / effectiveness
metrics