PR Measurement Summit 2016: Jim Macnamara's Keynote Speech Presentation

Latest International Developments In Standards And Models For Evaluation Prof.Jim Macnamara PhD, FAMI, CPM, FAMEC, FPRIA Associate Dean, University of Technology, Sydney - Link

Latest International Developmentsin Standards and Models for Evaluation

Breakthroughs at Last!

Professor Jim Macnamara PhD, FAMEC, FAMI, CPM, FPRIAProfessor of Public Communication, University of Technology SydneyVisiting Professor, London School of Economics and Political Science

Measurement and Evaluation – The Answer!

This Presentation■ Review some fundamentals

―Key principles

―The basis of evaluation frameworks and models – where do they come from?

■ New frameworks for evaluation―EU evaluation framework

―US Task Force on Standards

―UK Cabinet Office evaluation framework

―AMEC Integrated Evaluation Framework

■ A taxonomy of evaluation – mapping the stages, steps, metrics, and methods ―A tool to apply evaluation

This presentation

• Goal setting and measurement are fundamental for communication and PR

• Measuring communication outcomes is recommended versus only measuring outputs

• The effect on organizational performance can and should be measured where possible

• Measurement and evaluation require both qualitative and quantitative methods

• AVEs are not the value of communications

• Social media can and should be measured

• Measurement and evaluation should be transparent, consistent and valid

MEASUREMENT

• The taking of measures• The collection and analysis of data in relation to

a particular object, process, or condition• SO WHAT?

EVALUATION

• “Making a judgement” about the value or significance of something (Oxford and Merriam-Webster dictionaries)

• Evaluation is the systematic acquisition and assessment of information to provide useful feedback about some object” (Trochim, 2006, para, 3)

• “The systematic application of research procedures to understand the conceptualization, design, implementation, and utility of interventions (Valente, 2001, p. 106)

• Assessing results against objectives

SMART objectives

■ Contain numbers, percentages, dates

■ Design measurement in the planning stage

■ Be realistic

■ Linked to organisational objectives and goals

■ Achieved by a specific date

Three Types of Evaluation

FormativeProcessSummative

• Baseline (e.g., what is the current level of awareness)

• Precedents (have other similar programs worked in the past?)

• Pre-testing (will what we propose work?)

• CEA (is it the most cost-effective approach?)

• Insights / learning (what has been learned that can inform future strategy, policy, etc?)

• Impact (what has happened as a result of the communication?)

• Outcomes – intermediate and long-term (what attitude or behaviour change has occurred?)

• Monitoring and tracking outputs (e.g., are we reaching the audience?)

• Monitoring and tracking immediate outcomes (e.g., what response are we getting?

Why Stages?

Six stages of communication (W. J. McGuire, 1968, 1969)

ComprehensionPresentation Change/actionRetentionAttention Acceptance

Stages of communication

The AIDA model of advertising (Strong, 1925)

AWARENESS

INTEREST

DESIRE

ACTION

Stages of communication■ Exposure

■ Attention

■ Interest and/or liking

■ Comprehension

■ Cognition, particularly cognitive elaboration (thinking about the message)

■ Acquiring skills or knowledge required to deal with the issue (if necessary)

■ Attitude change, particularly to agreement (what McGuire calls “yielding”)

■ Storing information in memory (retention)

■ Retrieving information (i.e., recall)

■ Deciding to act in accordance with information (intention)

■ Action/behaviour

■ Cognitive integration of behaviour (e.g., reinforcement)

■ Encouraging others to behave similarly – what McGuire called “proselytizing” and what is commonly regarded in modern marketing as ‘advocacy’ (McGuire, 1999, 2001)

Frameworks/models of evaluation

The PII model of evaluation (Cutlip, Center, & Broom, 1985)


Macro model of evaluation (Macnamara, 1992)


Pyramid model of PR research (Macnamara, 2002, 2005)

OUTCOMES(Functional & organisational evaluation)

OUTPUTS(Process & programevaluation)

INPUTS(Formativeresearch)

Number who . change ..…..……………………………………………….. Quantitative surveys (large scale structured)behaviour …...…………………………………………….. Sales; Voting results; Adoption rates; Observation

Number who ….. changeattitudes ……….…………………………………………… Focus groups; Surveys (targeted) (eg Customer, Employee

Number who understand messages..……………………………… Focus groups; Interviews; Complaint decline; ExperimentsNumber who retain messages …….………………………………… Interviews; Focus groups; Mini-surveys; Experiments

Number who consider messages ……....….. ………………………… Response mechanisms (1800, coupons); InquiriesNumber & type of messages reaching target audience ……………………… Media Content Analysis; Communication Audits

Number of messages in the media ……………..….`…………………… Media Monitoring (clippings, tapes, transcripts)Number who received messages ………………...……………………….. Circulations; Event attendances; Web visits & downloads

Number of messages sent …………………………………..……………….. Distribution statistics; Web pages posted

Quality of message presentation …………………………………………………. Expert analysis; Peer review; Feedback; Awards

Appropriateness of message content …………………………………….………….. Feedback; Readability tests (eg. Fog, Flesch); Pre-testing

Appropriateness of the medium selected ………………………………………….…….. Case studies; Feedback; Interviews; Pre-testing (eg. PDFs)

How does target audience prefer to receive information? …………….…………..……… Academic papers; Feedback; Interviews; Focus groupsWhat does target audience know, think, feel? What do they need/want? ………………… Observations; Secondary data; Advisory groups; Chat rooms

& online forums; Databases (eg. Customer complaints)

or Shareholder Satisfaction); Reputation studies

Measurement Methodologies: (formal & informal)

Key Steps/Stages in Communication:

OUT-TAKES(Proposed by some as a 4th stage)

Frameworks/models of evaluation■ PR effectiveness yardstick (Lindenmann, 1993)

Measuring:Behaviour changeAttitude changeOpinion change

ADVANCED

Measuring:RetentionComprehensionAwarenessReception

INTERMEDIATE

Measuring:Target audiencesImpressionsMedia placements

OUTPUT

LEVEL #3

LEVEL #2

LEVEL #1

PUBL

IC R

LEAT

ION

S EF

FECT

IVEN

ESS

YARD

STIC

K

OUTCOMES

OUTGROWTHS

OUTPUTS


Noble & Watson’s united model (Noble & Watson, 1999)

INPUT STAGEPlanning & Preparation

OUTPUT STAGEMessages & Targets

IMPACT STAGEAwareness & Information

EFFECT STAGEMotivation & Behaviour

Tactical feedback

Management feedback


Communication controlling (DPRG/GPRA, 2000; DPRG/IPV, 2009)

Frameworks/models of evaluation■ The European Commission’s (EC) Better Regulation Guidelines uses the terms:

― Inputs―Outputs―Results― Impact

■ European Commission model (EC, 2015)

10 political priorities

Communication objectives

Activities Relevance Output Outtake Outcome

Recall indicators measuring the extent

to which the communication

activity is likely to have satisfied,

captured the attention of audiences, or raised

knowledge and awareness focussing

on the audience directly reached

Political matchOf the communication activities undertaken with the 10 political

priorities

• Organising events• Providing

information• Working with

information networks and desks

Reach indicators measuring the extent to which extent the

communication activity is likely to

have reached the right target audience

quantitatively and qualitatively

Engagement Trust indicators measuring the extent to which the communication

activity led to either a discernible action being taken or the

desired change in the target audience’s

perception

Inputs Results

Did the activity match the 10 priorities?

Did the activity reach the target?

Was the message received?

Did the activity meet the expectations of the target audience?

Did the action lead to the expected result?

Did the action change the perception of the EU?

Q.

Terms• Inputs / preparation

• Activities / implementation

• Outputs

• Outtakes

• Outcomes / outflows / outgrowths

• Results / effects / impact

Program theory and program logic models

Basic program model evolved since the 1970s (Kellogg Foundation, 2004, p. 1)

Program theory and program logic models

University of Wisconsin University Cooperative Extension Program (UWEX)(Taylor-Power & Henert, 2008, p. 5)

What are the standards for stages/terms?Program and PR evaluation models (1985–early 2000s)Basic

program logic model – UWEX (Taylor-Power & Henert, 2008)

Expanded program logic model – UWEX (Taylor-Power & Henert, 2008)

Classic Program Logic Model (e.g., Kellogg Foundation (1998/2004)

Cutlip, Center & Broom (1985) ‘PII model’

Macnamara’s Macro model’ (1992)

Lindenmann’s Effectiveness Yardstick (1993, 1997a)

IPRA Gold Paper on Evaluation (1994, pp. 10, 18–19)

Fairchild (1997, 2001); Fairchild & O’Connor, IPR Toolkit ( 1999)

Lindenmann Standards and Guidelines, 1997b, 1997c)

Noble & Watson’s ‘Unified Model’ (1999)

Grunig & Hon ‘relatinships’ model

DPRG/ GPRA (2000), DPRG/ICV (2009) models

Likely Performance Measurement Framework (2000)

Macnamara’s ‘Pryamid’ model (2000, 2002a, 2002b)

Lindenmann’s ‘guidelines' (2002/2003)

Input Inputs Inputs Inputs Inputs Inputs (hinted)

Input Input Inputs

Activities Planning

Output Outputs

Activities Participation

Outputs Implement-ation

Outputs Outputs (basic)

Outputs Output Outputs Output Outputs Output Outputs Outputs PR Outputs

Outtake Outtakes Outtakes PR Outtakes

Outgrowths

(inter-mediate)

Outcomes Outcomes Short-term Intermediate Long-term

Outcomes Outcomes (advanced)

Outcomes Outcome Outcomes Outcomes Outcome Outcomes Outcomes PR Outcomes

Impact Impact Results Business / organization outcomes

Impact Relationships Outflow Outgrowths Business / organization outcomes

Effect

9

14

41

125

1(Macnamara, 2016)

Theory of Change

INPUTS ACTIVITIES OUTPUTS OUTCOMES IMPACT

Communication• Objectives• Target audiences• Strategy

Actions

Short | Medium | Long term

The OrganisationOrganisation goals and objectives

Feedback loops to monitor and adjust strategy and tactics if required

Program Theory

INPUTS ACTIVITIES OUTPUTS OUTCOMES OUTCOMES OUTCOMES Short term Intermediate Long term OUTTAKES OUTCOMES IMPACT


Feedback loops to planning

Preparation Production Distribution Exposure/Reception Response Effects Results Organisation/Stakeholder Impact

ORGANISATIONOrganisation goals and objectives

Theory of ChangeProgram Theory

AMEC Integrated Evaluation Framework (AMEC, 2016)

INPUTS ACTIVITIES OUTPUTS OUTCOMES OUTCOMES OUTCOMES Short term Intermediate Long term OUTTAKES OUTCOMES IMPACT


Feedback loops to planning

Preparation Production Distribution Exposure/Reception Response Effects Results Organisation/Stakeholder Impact

ORGANISATIONOrganisation goals and objectives

Theory of ChangeProgram Theory

? ? ? ? ? ?

Inputs Activities Outputs Outtakes Outcomes Impact





Communication• Objectives• Target audiences• Strategy Theory of Change

Program Theory



• Budget• Personnel• Expertise• Other resources

• Formative research• Creative design• Pre-testing• Planning• Production• Media relations

• Paid advertising• Media publicity• Publications (e.g.,

newsletters, reports)• Web sites• Social media posts• Events• Sponsorships• Community projects


Preparation Production Distribution Exposure/Reception Response Results


• Recall• Awareness• Interest (e.g., follows,

likes, retweets)• Engagement• Satisfaction• Attitude change• Trust• Preference• Intentions• Inquiries / leads• Advocacy

• Reputation• Relationships• Public support• Sales / revenue• Donations• Cost savings• Staff retention• Customer retention• Complying behaviour

(e.g., give blood; drive safely, get fit)

• Quality of life / well-being improvements


Program Theory



• Budget• Personnel• Expertise• Other resources

• Formative research• Creative design• Pre-testing• Planning• Production• Media relations

• Paid advertising• Media publicity• Publications (e.g.,

newsletters, reports)• Web sites• Social media posts• Events• Sponsorships• Community projects

Stakeholders, Publics, Society


• Sales/revenue data• Behaviour tracking• CRM data• Surveys• Cost benefit / ROI• Well-being metrics

• Surveys (awareness, reputation, trust)

• Interviews• Social media qual

analysis (likes, shares, comments)

• Net Promoter Score

• Media metrics (reach, impressions, OTS, TARPs, CPM)

• Content analysis (e.g., messages, tone/sentiment)

• Web site data• Social media stats• Attendance figures• Reader surveys

• Pre- surveys• Focus groups• Interviews• Baseline data

collection (e.g., databases)

• Pre-testing• Peer review / expert

review

EVALUATION METHODS

Preparation Production Distribution Exposure/Reception Response Results


• Recall• Awareness• Interest (e.g., follows,

likes, retweets)• Engagement• Satisfaction• Attitude change• Trust• Preference• Intentions• Inquiries / leads• Advocacy

• Reputation• Relationships• Public support• Sales / revenue• Donations• Cost savings• Staff retention• Customer retention• Complying behaviour

(e.g., give blood; drive safely, get fit)

• Quality of life / well-being improvements


Program Theory

Questions

PR Measurement Summit 2016: Jim Macnamara's Keynote Speech Presentation

Data & Analytics