Which Method of Evaluation is Most Accurate in the Public Relations Industry

Suraj Mashru Bill Nichols

Second Marker: Christine Parsons

20923417

Buckinghamshire New University

Faculty of Design, Media and Management

School of Applied Business Management

B.A.HONS Public Relations & Marketing Communications

Which evaluation method is most accurate in the PR Industry?

Dissertation

9th May 2013

Word Count: 11,152

I certify that this submission is my own work and understand Buckinghamshire New Universities

regulations about, and the consequences of plagiarism and cheating.

Signed:

Date: 9th May 2013

2 | P a g e

Contents Acknowledgement .............................................................................................................................. 4

Abstract ............................................................................................................................................... 5

1.1- Introduction ................................................................................................................................. 7

1.2 - Research Objectives .................................................................................................................... 8

1.3 - Research Context ........................................................................................................................ 8

2.1 - What is Evaluation? .................................................................................................................. 10

2.2 - What is Evaluation in Public Relations? .................................................................................... 10

2.3 - Business Benefit & Objectives .................................................................................................. 13

2.4 - Historical Context ...................................................................................................................... 15

2.5 - Input vs. Output vs. Outcome ................................................................................................... 16

2.6 - Placement vs. Effects ................................................................................................................ 18

2.7 - Output Methods ....................................................................................................................... 19

2.8 - Outcome Methods .................................................................................................................... 19

2.9 - Pros and Cons of Current Evaluation ........................................................................................ 21

2.10 - Best Practice Evaluation ......................................................................................................... 21

3.1 -Research Philosophy .................................................................................................................. 24

3.2 - Research Approach ................................................................................................................... 24

3.3 - Research Strategy ..................................................................................................................... 25

3.4 - Research Choices ...................................................................................................................... 26

3.5 - Research Time Horizon ............................................................................................................. 27

3.6 - Critique...................................................................................................................................... 27

4.1 - Introduction .............................................................................................................................. 29

4.2 - Agency Opinion on AVE ............................................................................................................ 30

4.3 - The Need of Setting Goals & Objectives ................................................................................... 31

4.4 - Multiple Medium Measurement .............................................................................................. 32

4.5 - International Evaluation Culture............................................................................................... 33

4.6 - Overspending on Evaluation ..................................................................................................... 34

4.7 - Key Findings Conclusion ............................................................................................................ 35

5.0 - Conclusion: ................................................................................................................................ 37

6.0 - Recommendations: ................................................................................................................... 40

6.1 - Goals & Objectives Recommendation ...................................................................................... 40

6.2 - Overspending Recommendation .............................................................................................. 41

6.3 - Multi Medium Evaluation Recommendation ........................................................................... 42

3 | P a g e

20923417

6.4 - International Evaluation Culture Recommendation ................................................................. 43

6.5 - Recommendation Conclusion ................................................................................................... 44

Reference List & Bibliography ........................................................................................................... 46

Appendix 1.1 - Pyramid of PR Research ............................................................................................ 52

Appendix 1.2 - Interviewee Profile Table .......................................................................................... 52

Appendix 1.3 - Coding Table ............................................................................................................. 52

Appendix 1.4 - Sample ...................................................................................................................... 74

Appendix 1.5 - Managerial Recommendations ................................................................................. 75

Appendix 2.1 - Transcript - Interviewee Coding A (two respondents) ............................................. 76

Appendix 2.2 - Transcript - Interviewee Coding B ............................................................................ 84

Appendix 2.3 - Transcript - Interviewee Coding C ............................................................................ 86

Appendix 2.4 - Transcript - Interviewee Coding D ............................................................................ 94

Appendix 2.5 - Transcript - Interviewee Coding E ........................................................................... 100

Appendix 2.6 - Transcript - Interviewee Coding F ........................................................................... 102

Appendix 2.7 - Transcript - Interviewee Coding G .......................................................................... 104

Appendix 2.8 - Transcript - Interviewee Coding H .......................................................................... 106

Appendix 2.9 - Transcript - Interviewee Coding I ........................................................................... 109

Appendix 3.1 - Permission Letter - Coding A .................................................................................. 113

Appendix 3.2 - Permission Letter - Coding B .................................................................................. 113

Appendix 3.3 - Permission Letter - Coding C................................................................................... 114

Appendix 3.4 - Permission Letter - Coding D .................................................................................. 114

Appendix 3.5 - Permission Letter - Coding E ................................................................................... 115

Appendix 3.6 - Permission Letter - Coding F ................................................................................... 115

Appendix 3.7 - Permission Letter - Coding G .................................................................................. 116

Appendix 3.8 - Permission Letter - Coding H .................................................................................. 116

Appendix 3.9 - Permission Letter - Coding I .................................................................................... 117

Appendix 4.1 - Supervisor Meetings ............................................................................................... 117

4 | P a g e

Acknowledgement

I would like to thank the following people for helping me through this

dissertation.

I would like to begin by thanking my supervisor and course tutor Dr Bill Nichols,

for his support and guidance through the research process, as well as through

my PR training at university.

I would also like to thank the entire Marketing faculty at Buckinghamshire New

University, in particularly John Hathaway, for his good humoured support

whenever it was needed, Vic Davies, for helping me to find primary research

respondents and Ruth Hickmott for helping me choose PR as a profession.

I would further like to thank my family and friends for their support through

this process.

-Suraj Mashru

5 | P a g e

20923417

Abstract

This dissertation was written to evaluate the measurement of contemporary Public

Relations. This topic was chosen as it questions a long standing industry issue, which not

only affects the UK PR industry but the industry across the globe.

It was found from the literature review that many academic texts are arguing the point that

PR should be measured using methods such as AVE (Advertising Value Equivalent) or OTS

(Opportunity to See), which whilst providing figures for evaluation, only presents figures

based on PRO (Public Relations Officers) outputs, as opposed to their client's business

outcomes.

The primary research data, collected through face to face semi structured interviews, which

allow for free speech during the interview. This method was selected as it allows for the

respondent and interviewer to discuss items that may have come up during the stages of

the interview, which might not have been identified at the planning stage. The second

selected method was e-mail interviews. This method was selected for respondents who

were very busy, abroad or a great distance away from the researcher. The format for these

was structured written interviews, as there was no opportunity for conversations between

the researcher and interviewee to take place.

The questions selected for the interviews were chosen at the end of the literature review, as

these demonstrated the issues that are affecting the industry. Therefore, the author of this

dissertation chose to select these, to form key findings. These questions were chosen to

assist in attaining key findings for the dissertation.

This dissertation produced five key findings, which are that the AVE methodology is not the

most commonly used method of evaluation, as the literature review stated. Secondly, this

dissertation found that in order to evaluate PR effectively, there is a need for PRO's to set

communications goals and objectives at the start of a campaign. Third, the dissertation

found a split in the opinion on how different forms of PR should be measured. The fourth

finding evaluated how international PR is measured, to establish if local culture affects the

process. The final key finding is one that was not anticipated, which is overspending on

evaluation.

6 | P a g e

The dissertation proceeded by making four strategic recommendations, the first, to set

communications goals and objectives at the start of every campaign or quarter, secondly, to

monitor how long is spent on evaluation, the third recommendation was to measure

different forms of PR differently, as they require different techniques, making their culture

different. The final recommendation is regarding international PR, and this dissertation

recommends that international PR, should, be measured in accordance with the local

cultured taking priority.

7 | P a g e

20923417

1.1- Introduction

‘Measure what is measurable, and make measurable what is not so.’ (Phillips, D, 2001)

This dissertation has been written to identify the most effective method of evaluation in

contemporary Public Relations. The study will be analysing various methods of evaluation

used within the industry.

By holding interviews with various people within the industry, from academics, practicing

PRO's (Public Relations Officers), from both agencies and in house teams around the UK and

abroad, members of trade bodies such as the PRCA, members of teams whom specialise in

PR evaluation, this dissertation will be able to obtain a rounded view of what the industry is

currently doing, and how evaluation methods are being changed or altered for the future, to

incorporate social media and an increase in the use of digital PR, such as blogs and online

news websites.

Alison Theaker has stated that 'measurement and evaluation in the public relations field is

not a brand new issue or topic that has suddenly emerged in the past few years. It is an

issue and topic that has been widely discussed, actually carried out and grown and evolved

over a 60-year period of time.' (Theaker A 2012: 196)

8 | P a g e

1.2 - Research Objectives The objectives set in this dissertation were to:

Research Objectives

Establish if AVE is still the most widely used method of evaluation in the industry, both in house and in agencies

Critically discuss the requirements of how to set up an effective PR evaluation process

Critically discuss the culture of evaluation in the UK and Abroad, as well as discuss whether evaluation is something that PRO's do because, they have to, or because, they want to

Make recommendations as to how online evaluation (including social media) can take place

Make recommendations on how the Public Relations agencies can use evaluation to its fullest extent by giving clients accurate data.

These objectives have been set in line with the industry issues, of PRO's measuring outputs,

as opposed to outcomes.

The above research objectives will be used in order to coordinate the primary research

questions.

1.3 - Research Context

The aims of this study were to identify the strengths and weaknesses, of the various

evaluation methods, which are used to evaluate the results of public relations. The

Advertising Value Equivalent (AVE) system is the unwritten evaluation method, however,

not always accurate. This system is used with a multiplier of normally, 3 or 5, depending on

the agency.

However, even though this method is used on a regular basis, by many PRO's in the

industry, it is a method that only measures outputs, as opposed to the outcome that a

client's business will benefit from .

9 | P a g e

20923417

This dissertation aimed to seek out methods used by PRO's, to analyse whether the

industry has any form of evaluation, which accurately measure business outcomes, and if so,

how does the measurement process work.

This paper was written, as the topic of evaluation in Public Relations, is one that is very

important in the industry, as without having accurate methodologies to measure work in, It

would be very difficult for PRO's to justify their existence in the work place.

The dissertation looked at the various PR evaluation methods, including, AVE, clip and

message counting, as well as others and then, critically evaluating the methods, to establish

which would give clients, the most accurate figure for their PR activities. Tench & Yeomans

describes evaluation as ‘the evergreen topic of entire practice and one of the areas where

both practitioners and academics have a vast common interest’ (Tench and Yeomans 2006:

P213).

This study focused on evaluating the accuracy of these methods, by comparing the various

models, and analysing which will in turn, bring the agency and client, a form of win win. Win

for the client, will be having a successful campaign, resulting in an increase of sales, and for

the agency, win, would result in an increase of business from that client.

There are many different texts available on the topic of the PR evaluation, there is also a

heavy online presence on the latest trends and developments on this topic. However, a high

amount of online sources, are proving to be arguing the same point in the same way as

others. Therefore proving only one angle of evaluation in PR. Therefore not showing the

true story of what is going on in the industry.

10 | P a g e

Literature Review

2.1 - What is Evaluation?

Evaluation is the process of measurement of any form.

Collins Dictionary, have defined the verb 'evaluation' as 'to ascertain or set the amount or

value of' (Collins Dictionary 2013).

Evaluation is a process which is used in everyday life, not only in the Public Relations

industry, but also stems into all parts of business and academic life. The need to evaluate

ones work in every industry is very important, as it allows organisations to ensure they are

meeting their goals and objectives.

2.2 - What is Evaluation in Public Relations?

Evaluation in the PR industry is used for PRO’s to give an ROI for their efforts. By giving

information on the ROI for PR campaigns, the PRO can present back to the client, proving

that PR has a valid place in a modern business.

Evaluation in public relations has been described as ‘as elusive as finding the Holy Grail’

(Pavlik, 1987: 65). This comment has been made because a high number of agencies and in-

house teams which use their own methods of evaluation, it has been found that one

accurate key method, which looks at outcomes, compared to outputs, is very difficult to find

(Watson 2012: 19). A further reason for the industry not having any accurate models,

compared to advertisers, is that many agencies use their own internal forms of

measurement, therefore, the industry is not able to analyse them all, in order to find an

accurate and effective methodology (Watson and Noble 2007).

RD Smith has stated that there are many different forms of evaluation in Public Relations,

from 'output evaluation, 'message production' and 'message distribution' (Smith 2009:

P245).

11 | P a g e

20923417

Watson and Noble’s book has stated over the years, there have been many ideas thrown

about the industry. During the transition to the digital world, it has been found that word of

mouth, or interpersonal influence has become more common with the use of social media,

allowing the public to talk to their peers about a certain product or company easily, from

almost any location worldwide (Watson and Noble, 2007, P14). However, as the authors

have not discussed specific models or theories regarding social media, it is not possible to

establish whether any methods were approved or altered to help the evaluation of digital

PR. In 2012, CIPR launched ‘Share This’, a book about the online world of PR, where the

authors state that online PR measurement should be completed using the following

framework.

Factor Meaning

Influence Is there a best practice way to identify who

influencers are?

Sentiment Has the data been analysed by human or

machine? What is the error rate?

Engagement

Understand exactly what engagement

means, a follower on twitter does not mean

engagement.

Monitoring Content

How fast are the researchers at finding

relevant content, how broad are searches,

how much data returned is valid?

Demographics Are you sure that the correct target

audience is being met?

Source: CIPR 2012: P169

Tench and Yeomans have expressed that evaluating Public Relations efforts, is extremely

important, as it reflects effectiveness, professionalism, whilst giving accountability for the

efforts (Tench and Yeomans 2006: P224). This view has been reflected Anne Gregory, who

has described evaluation as the driving force which helps PR prove its effectiveness, and its

position at the board. Without doing accurate evaluation, PR will be at risk of being a

12 | P a g e

communicator that is intangible or ‘far too important to be measured’. Something, which

many board members, do not believe in (Gregory 2010: P164).

An interview article in PR Week, from July 2012, with Philip Sheldrake & Mark Westaby,

states that PR measurement has not changed in the last 20 years. The article quotes

Westaby saying ‘A real problem is the PR industry thinks its job is to get media coverage or

run a good event – that’s not the case. The PR industry’s job is to influence and change

behaviour of consumers and stakeholders.’ (PR Week 2012).

Sheldrake comments that the industry is wasting time trying to find a universal solution to

evaluation, as it is something that he feels will never be found (PR Week 2012). The PR

Week article agrees with Sheldrake’s book, as he states that PR should measure what it

needs to, and current methods used are doing. Therefore, until PRO’s learn they only need

to evaluate their work on its own, PR evaluation will never be accurate (Sheldrake 2012:

P47).

Tench & Yeomans have incorporated a survey carried out by PR Week in 1999 that PRO’s

prefer to evaluate their work through press cuttings (Tench and Yeomans, 2006, P226). This

system is AVE, which looks at the cost of the space taken by PR efforts in terms of

advertising. Which only provide figures of output, compared, to the actual result of the

campaign to the company (RomyR PR, n.d).

The author of this dissertation agrees with this comment, as the industry needs to focus on

this, by having a clear understanding of the work, measuring success would become easier,

as there is a set bench mark for the outcome to live up to.

There are a handful of external organisations who offer accurate media evaluation, by

looking at a number of different parameters, including position on the page, the wording

used and whether the coverage from the campaign is solely on the PRO's client, or its

competitors also. The more parameters that are met the higher, the evaluation will be.

However, this process is very costly, therefore, not many PR agencies would not be able to

afford this, and also clients would expect this to be included in the price. Therefore, if this

process is carried out for each client, the agency would not make any money from their

client.

13 | P a g e

20923417

There are many external organisations that offer this service, examples of these include

Kantar Media, Mantra, Metrica (part of the Gorkana group) and Apollo.

2.3 - Business Benefit & Objectives

From the side of a PR agency, the organisation will have their own set of brand values and

will also have their own business objectives - which will include ensuring all clients are

profitable to the company.

However, as a client of a PR agency, the values and business objectives will be very

different, the clients business objectives may include increasing market share, through a

mixture of marketing communications including PR. Therefore, the biggest job for the PRO,

is to prove to the client, that the medium of PR has proven effective to the overall business.

The pitch to prove the effectiveness, must be successful, as some larger corporate

companies may have internal marketing managers, who need to protect their budgets,

therefore passing the information onto their organisations boards of directors.

Therefore, from a PR agency's perspective, the benefit of PRO’s carrying out evaluation is

important, by allowing clients to have a tangible figure to an intangible operation, this

allows for a greater understanding, allowing for PR to obtain a greater power in the modern

business place.

Tom Watson found from a 1994 survey, that many managers believe, the best form of

evaluation, is when clients renew contracts (Watson, 2012:19). While this can be considered

to be true, clients would only renew if they are happy with the service, many larger

cooperation’s still require a higher level of information, of where funds are going, and what

their ROI is. From a Public Affairs consultancy approach, this is considered true, due to the

need of concrete objectives at the outset, it is somewhat easier to measure a successful

outcome to a campaign, however, there will also be some form of ongoing evaluation which

will take place during the campaign to measure outcomes, this is something that is more

difficult compared to the end result which is essentially easier to measure (Underwood, J

and Stevenson J, 2013).

14 | P a g e

Anne Gregory has endorsed the view by Tom Watson, that many PRO’s are reserved,

regarding evaluation, as it is believed to be done to make themselves look better, rather

than giving an answer of how the media relations worked (Gregory, 2010, P158). This maybe

as PRO’s could give the impression to clients, that their job is easy, potentially meaning, the

client takes their PR in house.

Gregory also states that advising clients on how well the media relations worked, it would

entice them to invest more time and money into PR efforts, resulting in PRO’s becoming

more profitable for their company, as they will be promoting brands and services through

methods of communications, which has proved to give a high ROI (Gregory 2010: P159). This

may have an adverse effect, as not all campaigns are guaranteed to be successful, meaning

that sometimes PR can also be unhelpful for an organisation.

The author has discussed the issues regarding evaluating PR campaigns, as many firms may

run campaigns alongside sales promotions, therefore causing the PRO to find a balance

between the two campaigns, as both are triggers which potentially cause the preferred

outcome.

Tench and Yeomans have also stated, that PRO’s have been known to overemphasise

evaluation of print media, and struggling to keep up with the fast paced technological world,

as well as finding ways to evaluate communications, such as TV and images (Tench and

Yeomans 2006: P224). PR, also has a need to evaluate live PR stunts, in order to give an

accurate measurement of the work (Tench and Yeomans 2006: P225). Should this not be

done, it will be difficult to prove its worthiness, as there would be no results for the various

tasks carried out. Potentially making it unbeneficial for companies to invest in.

15 | P a g e

20923417

2.4 - Historical Context

The 1940’s saw evaluation become a big part of PR strategies, with various systems being

used at government level, however, Tom Watson's article in the Journal of Public Relations,

2013, does not talk about agency use with the activities similar of Ivy Lee and Edward

Bernays. Nor does the article discuss any methods used, making it difficult for the reader to

understand what methods were available. This issue has been reflected in the majority of

texts read during the research stage, of this section.

1949 saw the birth of the UK’s IPR (Institute of Public Relations), which was mainly run by

the Governments communicators, who saw the need for a good method of evaluation for

their work, something which was discussed in their first book, which helped train

communicators on how to evaluate (IPR 1958). The above mentioned book, is one that is

very difficult to obtain, therefore no analysis can be made to the ideas put forward by the

authors in 1958, except from the content which has been delivered in this journal, mixed

with the perception of Tom Watson.

This article goes onto to compare and contrast comments made over the years, regarding

whether PR is measureable or not. Watson compares quotes made by Ivy Lee, and the

quote of John Marston in 1963, where the two men, have taken two very different

approaches to the issue of evaluation in this industry. As the quotes are around 50 years

apart, it is possible to understand, how over the years impressions and opinions of

measurement in the PR industry have changed. Ivy Lee has claimed PR cannot be measured,

whereas John Marston, in ‘Nature of Public Relations’, has claimed that they cannot be

bothered to measure success (Watson, 2012:19). This is an interesting angle, as it opens a

new route of investigation, something which the primary research will also be asking, as it

will help establish if the evaluation reports made by PRO’s, are accurate, or altered to look

good for the client.

Watson goes on to discuss how the rise of the service agencies, such as Hill & Knowlton, and

Burson-Marsetller, helped kick the start the consumer public relations sector, increasing the

demand for more university programmes (Watson, 2012:11). As these two agencies became

international very quickly, there was an urgent need to send evaluation reports, back to the

16 | P a g e

US headquarters, for CEO’s to see what was going on in their other offices (Watson,

2012:11). This would help the directors of the company, to monitor business and results

given to clients, regardless of where they are located in the world. However, over the years,

many agencies have created their own theory behind evaluation, and use these systems

effectively. While all maybe accurate, questions still need to be asked as to how accurate

they really are compared to competitors systems.

2.5 - Input vs. Output vs. Outcome

The PRO's input to the campaign is dependent, as the brief, given to them by clients should

be clear, where PRO’s can decipher the issues with clarity, allowing the campaign in which

they plan to contain greater insights, allowing for better targeting, placing and content.

Anne Gregory has described evaluation, as being an on-going process, from start to finish,

and should give live figures, allowing for any changes to be made, should the campaign not

perform in the intended manner. She also states the importance of evaluating the final

results, in order to look back at the campaign on a time scale, to find the most effective

aspects (Gregory 2010: P161).

This is something that is being echoed in other author’s works. Experts such as Tom Watson,

have discussed the need to start evaluation methods at the start of the campaign, by setting

goals (Watson and Noble 2007: P176), however Anne Gregory has taken this further, by

evaluating campaigns while they are running (Gregory 2010: P161). AMEC have also

endorsed this as the Barcelona Principles, which start with the importance of setting goals

(AMEC, 2012). However, this is not a new idea, as Smiths 9 steps to PR success stated in the

2009 book, Strategic Planning for Public Relations, where he states that any PR campaign

should start with goal setting at the top of the agenda (Watson and Noble 2007: P55).

Watson and Noble discuss the use of non-predictive methods. This would work well, as it

would not give a fixed route that PRO’s need to follow, to establish if their campaigns have

been successful (Watson and Noble 2007: P41) For example, if a campaign has been started

to change the perception of a local government agency, then AVE would not give a relevant

answer, as it is being based upon the advertising value, rather than the opinions of the

people. Therefore, the more positive coverage and online comments, which are being

17 | P a g e

20923417

made, would prove whether the campaign has been a success, as the public’s perception

has changed, therefore giving the PRO a choice of methods of evaluation which is most

suited to the campaign.

Philip Sheldrake has endorsed in his book, ‘The Business of Influence’, the comments made

by The Holmes Report and CIPR, one of the industry bodies for PR, as they state that PRO’s

should focus more, on measuring outcomes, compared to outputs (CIPR,2010). The reason

this is being agreed with, is because the AVE method only allows for the PRO to measure the

value of their success, thus only allowing for output, whereas the outcome, for the business

can be extremely different.

Appendix 1.1, shows the PR Research Pyramid, (Watson and Noble 2007: 85) which shows

the questions that should be asked, at the start of a PR campaign, by the PRO, in order to

gain a greater understanding, of what it is that the client wishes to achieve from this

campaign. By understanding this, the campaign will have the ability to target the correct

audience, as well as sending the correct messages to this audience.

As the pyramid demonstrates, the starting process starts with the formative research, which

in turns leads and the creation of the inputs and the pilot testing of messages, in order to

ensure that the correct messages are taken out of it. Once this stage is completed, the PRO

beings to talk to the media, and sell their inputs, to lead to outputs. The current industry un

written standard, AVE, is the method which measures outputs, compared to the actual

outcomes for the client.

A real life example of this would be of an online retailer, who has used a PRO to create a

campaign for a product. The inputs would be from the client and the PRO, following this

outputs would include, the press releases and the number of pieces of coverage obtained.

Finally, the most important part of the process, is analysing what this coverage means for

the client. To establish this, it is possible to obtain website visitor reports, which can be

compared, with the previous 12 months, to establish if there has been an increase in the

number of visitors on the site, during the time of the campaign. If there has been an

increase, this can be put down to the PR campaign, to analyse this fully, it is possible to ask

customers when they visit the site, as to how they learnt of the company. This will provide

insights into how effective the campaign has been.

18 | P a g e

Watson and Noble, have stated in the chapter on evaluation in their academic book, that

many PRO's often opt to use the simplified method of evaluation, rather than any other

form (Watson and Noble 2007). This shows how the industry is viewing evaluation, as

something that is only done, because it has to be. Therefore, hindering the quality of the

evaluation results.

2.6 - Placement vs. Effects

The PR industry has seen two different approaches to evaluation. The first is placement,

where the PRO would look at press cuttings, and quantity, compared to quality of where the

article is being published. However, as placement looks at quantity, it is unable to address

the key messages, and what context they were used in as they ignore the fact, whether they

are positive or negative. Placement offers figures on the space used, and in which

publication, therefore, again referring to outputs, not outcomes. This does not give answers

as to how good the coverage received was, it only gives a reassuring number for the client

to see (Nichols, 2012).

The second method is measuring effects, this is where the change in perception, consumer

behaviour, or opinions are measured. In order to get an accurate measurement, it is

required for the PRO, to carry out qualitative research, such as opinion polls and survey’s, to

measure the effectiveness of any changes, that the campaign has made (Nichols, 2012).

Whilst these two methods of evaluation, will have strengths and weaknesses, which will be

found by the industry, as stated above, the first methodology does not answer the question

of evaluation, as it only looks at the outputs. Also, the second methodology, would be very

costly for PRO's, as, it requires a series of opinion studies on the topic, to ensure that the

objectives have been met.

19 | P a g e

20923417

2.7 - Output Methods

There are many methods in the PR industry, which allow for a PRO to evaluate their work in

their terms.

The most renowned method, is the Advertising Value Equivalent method, or AVE, this is

where the PRO measures the value of the coverage, by looking at how much it would have

cost, should that space have been purchased at rate card price. All texts have clearly stated

that PRO's add a multiplier onto this value, thus further increasing the figure. However,

AVE's do not measure the value against the value of advertising, they only measure against

the cost (Sheldrake 2011).

Watson discusses measurement of outputs, through, clip counting, clip size and message

direction. The method of clip size lead to AVE, as well as the system of ‘pepping up’, this was

used, when images had been published, with the PRO enlarging the image on the clip, thus

claiming more space, making it seem the coverage was much larger than it was (Harlow,

1942). This is the issue that AMEC are trying to resolve today, with a method ,which reflects

PR activities effectively and accurately. For more information on the output process, please

see Appendix 1.1.

Other options of PR models, which evaluate outputs, include the use of the Opportunity to

See method, where an ROI can be given on the circulation figures established by the PRO, in

relation to how many people the campaign theoretically reached (Tench and Yeomans

2006).

2.8 - Outcome Methods

In order to measure Public Relations effectively, it is essential for PRO's to measure the

outcome to an organisation, or their client, as opposed to measuring outputs. In section 2.7

of this dissertation, it was stated that measuring outputs, only allowed the PRO, to report to

their clients about the value of their work, not the valuable or tangibility of the campaign to

their clients organisation.

20 | P a g e

In order to measure outcomes, it has been found, that it is essential to set communications

goals and objectives in order for the PRO to have a fixed protocol to measure against. An

example of this would be, a client wanting to increase enquires of a certain product by 15%,

by the end of the month, or increase traffic to a webpage or website by 15%, by the end of

the quarter.

By having such clear targets set, it is possible for PRO's to use the Yardstick Model,

developed in 1998, by American PRO and research, Walter Lindenmann, the method has

three stages, similar to set up of Appendix 1.1, this model also endorses the need to set

goals and objectives at the start of campaign, the second is known as PR Outgrowth, this is

where it is analysed, whether the target audience attended to the campaign and if they

understood and retained the key messages. Finally, the third stage of the process measures

the outcomes to the campaign, by analysing opinion, attitude and behavioural changes

within the target market(Theaker 2012: 200).

In 2010, AMEC teamed up with the CIPR, as well as other international PR representative

bodies, to try find an industry wide solution, to this issue.

Seminars around the world led to the creation of the Barcelona Principles, seven steps that

will allow for PRO’s to report accurate ROI figures. The stages of this evaluation process

include, setting goals before any measurement should start, measure media with quantity

and quality metrics, understand the value of change in relation to PR for the public and

business, use the same measurement for social media as efforts are still the same and

ensure that all measurement is transparent (AMEC, 2012).

However, no information has been announced as to how the principles will work, and how

they will be monitored, as the step regarding quality can vary from person to person.

There has been much discussion in the industry, regarding this new system of evaluation.

Peter Sheldrake does not endorse the Barcelona Principles, claiming that they are too basic.

The author endorses the Holmes Report, as AVE’s do not measure the value of PR, it only

analyses the cost, in relation to advertising, at the un-negotiated rate card price (Holmes

Report, 2010).

21 | P a g e

20923417

AMEC’s Executive Director, Barry Leggetter, has described the Barcelona Principles as ‘baby

steps towards a unified solution’ (Leggetter, 2010).

2.9 - Pros and Cons of Current Evaluation

The current methods of evaluation in the PR industry are extremely varied, as this literature

review has expressed from the start, the biggest weakness of current methods stated in

academic textbooks, industry press and journals, are that many PRO's are only evaluating

because they have to, and by doing so, only using the most simple methods available.

By using such a simple method, PRO's, are not doing the industry any good, as they are

potentially giving the impression that PR, can only be measured in terms of outputs,

compared to the all important outcomes.

The pro's of PR evaluation, are that, it can be proven, how cost effective the medium is,

compared to others, such as advertising, or digital marketing. Many business owners know

that PR is an effective medium, as they understand, that when a client has been included in

an article, it is simply because the journalist has chosen that company, over a competing

firm, thus delivering the message through a third party, whom the public trust and connect

with.

2.10 - Best Practice Evaluation

This dissertation, was written to establish the most accurate method of evaluation in the

Public Relations industry. Following the above reading, as well as much other background

information research, the author has established, that evaluation in PR is something which

is considered a dark subject. This means that the measurement results can be increased,

thus causing the client, to believe they are getting a higher ROI. It was also found that there

has been many ideas thrown around the industry over the years, however, very few have

been put into practice.

The last few years have seen the Barcelona Principles, something which is aiming to combat

the negative side of evaluation in PR, however, as stated above, some academics and

22 | P a g e

professionals, believe it is too little too late, as the industry has created the illusion of

greater ROI's (Holmes Report 2010).

It has been found from the above literature, that best practice would involve the need to set

goals and objectives, prior to the start of the campaign, to give the PRO a metric to measure

against.

Watson and Noble endorsed this, as it allows the client to obtain an accurate evaluation

figure, illustrating whether the goals set have been met effectively (Watson and Noble 2007:

48) - A view, which has been heavily recommended, by the other authors. Watson and

Noble, endorsed the theory of McCoy & Hargie, that PR has to create awareness, leading to

public knowledge, aiming to form a favourable attitude change, leading finally, to changes in

behavioural patterns (McCoy and Hargie 2003:305).

Smaller agencies, and in house PR teams, use the most common form of evaluation, AVE

(Tench and Yeomans 2006: P229). However, as this measures output, the true value of PR is

not being discovered.

Following this literature review, primary research aimed to identify a method of evaluation,

which will allow PRO’s to accurately uncover the value of their outcomes, compared to

outputs.

It was vital to understand from PRO’s, if they felt the mass media impacts the public

behaviour, and how the impact of digital, made them alter their evaluation methods to

incorporate this.

The research programme, will also look to establish, whether PRO’s feel that PR is

intangible, in terms of results, and how important evaluation is for them. It is also

imperative to establish, how, they evaluate other methods of PR, including radio, TV and

online, as whether they feel this is accurate. The study, will look at evaluation cultures in

agencies, and in-house teams, the study will also aim uncover, if PRO’s are being trained in

AVE or other methods.

23 | P a g e

20923417

By looking at various methods, used by PR agencies, and in-house teams, it will be vital to

establish, whether the process requires goals to be set at the start of the campaign, and if

they require measurement, to run simultaneously with the campaign.

The research also needs to establish how international agencies evaluate PR abroad, to see

if they are using one unified system, or if local culture comes into effect.

Finally, it would also be important, to establish, whether PRO’s feel that methods used

within their organisation, give accurate figures, while trying to understand what they would

do differently for clients, or, if they are being told by their superiors to make evaluation

reports look better, or, if they are aware of this malpractice.

Below is the methodology for the primary research, where the research objectives, as stated

above will be answered.

24 | P a g e

Methodology

3.1 -Research Philosophy

The philosophy chosen for this study was critical realism. This was selected, as it reflects the

topic of this dissertation as well as the conclusion to the literature review, as each source

reviewed gave a different opinion, on which method of PR evaluation, gives the most

accurate result.

The realism philosophy, is based on the belief that reality is independent of human thoughts

and beliefs, thus meaning, each person has their own views, even though their industry may

suggest another way (Saunders, Lewis, & Thornhill 2003: P81).

3.2 - Research Approach

The research approach to this dissertation, was deductive, as this study looked at the

various methods of evaluation in PR, and which are the most accurate, rather than methods

such as AVE, and its associated multipliers, which vary from agency to agency. This method,

also, ensured that by proposing a relationship, between two variables that were not fixed,

can be changed, after the primary research results have been collected and analysed

(Milyankova n.d).

Dr Burney, has stated that a deductive approach, starts, with general content, and then

follows down to specifics, where as an inductive approach, can be used to create a theory,

from looking at specifics (Burney 2008).

In order to gain the data, it was vital to speak to practicing PRO’s, both in house and

consultancy, to discuss the various ways which their departments evaluate their work. This

was conducted, by holding face to face interviews, as well as some email interviews with

PRO’s, who have worked in the UK and possibly abroad, to discuss how the culture of

evaluation varies, in the countries that they have been working in.

25 | P a g e

20923417

Wilson, has described face to face surveys, as a form of research which involves meeting

respondents face to face and interviewing them, using paper or digital methods (Wilson

2012). This will provide a form of qualitative data, as it will give insight into what PRO’s and

academics feel on the subject.

Bernard further quotes, that 'if a respondent doesn't understand a question in a personal

interview, you can fill in, and, if you sense that the respondent is not answering fully, you

can probe for complete data' (Bernard, H 2006: 256).

This required an outline plan, of what the interviewer wished to establish from the

interview, in order to lead the conversation effectively (Lindlof and Taylor, 2002). This is also

further supported, by Bernard's quote above, that the interviewer is able to lead the

conversation, if the respondent is not answering the questions in the correct manner. It also

allowed the interviewer, to take the discussion into an area that may have not been

anticipated at the planning stage of the study, depending on the responses given.

3.3 - Research Strategy

As there are various theories, surrounding the topic of evaluation in Public Relations, the

strategy for the research, was based around a grounded theory, as this dissertation

discussed various models, in order to ascertain which method will allow the PR industry to

give the most accurate return on investment figures.

A ground research, is a method, which works in a systematic manner, while still having

flexible guidelines, allowing for a free flowing conversation, with the respondent able to

mention any facts, that will reinforce the point behind their answer. This allowed the

interviewer to take the respondent, into a feeling of freedom, by asking open questions,

allowing for a free flowing conversation (Charmaz K, 2006).

The British Medical Journal, stated that grounded theory studies, are appropriate when the

study aims to explain a process, and not to test, of verify, any existing theory (British

Medical Journal 2008).

26 | P a g e

The research method, which was undertaken for this dissertation, were semi structured

interviews, and structured email interviews, with persons from the PR industry, including

PRO’s in house and in agencies from the UK and the US. The interviewer, also spoke to

academics about their views, on the accuracy of evaluation in the industry, as well as the

various methods available, finally, the dissertation approached various PRO's who were

apart of trade bodies, such as the PRCA.

'First, they are well suited for the exploration of the perceptions and opinions of

respondents regarding complex and sometimes sensitive issues and enable probing for

more information and clarification of answers.' (LK, 1994)

The target number of interviews stood at 12, as this was felt that, it would allow for a

overall rounded opinion to be formed on the topic.

3.4 - Research Choices

The choice of research chosen for this dissertation, was a qualitative mono-approach.

This was chosen, as it enabled the dissertation to obtain a rich insight, into the minds and

emotions, of PRO’s and academics, to understand exactly what it is, that they feel about the

evaluation processes, that are being used in the industry today.

The result of this study, gave the dissertation a form of qualitative data, providing a high

level of insight, to the topic of this dissertation. By gaining insight, from the target audience

outlined above, the study was able to gain opinions and ideas, of what the industry thinks

and feels about the subject of evaluation.

This allowed the author, to add this to the review of literature above, allowing for further

information to be uncovered, giving the industry a better insight. By understanding what the

industry is doing, it will be possible, to establish a common ground, on what it is that PRO’s

are doing in terms of evaluation. As this study set out to talk to people, who work with

public relations, it is only these people who will need to be interviewed.

27 | P a g e

20923417

The reason that this target audience was chosen, was due to active PRO’s holding insights,

into how their organisation undertakes evaluation, and its culture within their company. The

second target group were academics, by interviewing this group, insight were found of how

evaluation is being taught in universities, and how the lecturers feel about the Barcelona

Principles. Finally the researcher, contacted those in trade bodies and those who work in

evaluation companies. Please see appendix 1.2 for the profile of those interviewed.

3.5 - Research Time Horizon

The time horizon chosen for this dissertation, was a cross-sectional study. This type of study

takes place over a fixed period of time, rather than being longitudinal, where a study would

be monitoring changes, and developments around the research topic. The cross sectional

method, works differently, as the dissertation looked to establish the differences between

agency cultures, in terms of evaluation of public relations.

Cross Sectional Study – A study based on observation represented at a single point in time

(Babbie E, 2010).

This strategy was chosen, as it will be possible to complete the study in the limited time

scale available.

3.6 - Critique

Following the study, it was found that research methodology chosen worked well for this

study.

The dissertation interviewed ten people from the PR industry, please see appendix 1.2 for

the profiles of those interviewed.

By holding a semi structured interview, it was possible for the interviewer and respondents

to hold an open conversation, rather than a strict conversation where the interviewee can

only answer the questions they've been asked (Sage Research Publications, n.d).

Having completed the study, a reflective summary can be created.

28 | P a g e

It was found that access to people in the industry was the biggest barrier which was faced.

As many PRO's work to deadlines, agreeing a suitable time and date to interview them was

very difficult, as many of the times offered by the interviewees clashed with core lectures

that the author of this dissertation had to attend.

If this study was repeated, it would be recommended to allow a longer period of time to

access PRO's in the industry. An appropriate time period would be an extra four weeks, as

this will allow for drop outs to be replaced.

Other than this, it has been found that the study carried out was very successful and helped

to prove the research objectives.

Please see appendix 1.2 for a table which has the profiles for those interviewed.

These people were selected to be part of the interview, as the researcher wished to obtain a

rounded view from people from all corners of the industry, as well as those from the UK and

America, incorporating those working in agencies, in house and from measurement

organisations.

However, the limitations are that the research project is only talking to very few people in a

mixture of position, regardless of agency or in house. A second research limitation is the

study is only talking to two respondents based in the US, with the remainder in the UK.

The data capture methods used, was extremely effective, as it allowed for the researcher to

probe into the mind and emotions of PRO's, regarding, evaluation and measurement in

Public Relations.

By holding semi-structured interviews, it was possible to discuss in great depth around the

topic, also allowing respondents to bring new parts of the issue to the table which they feel

are important.

29 | P a g e

20923417

Key Findings

4.1 - Introduction

This section of the dissertation, analyses, key findings below. The findings are based on the

research objectives set with the key findings, which are:

Research Objectives Key Findings


To explain the findings on the subject of agencies opinions on AVE


To discuss an industry issue of over servicing clients in terms of evaluation


To explain the findings on the subject of international evaluation culture


To explain the findings on the way that various mediums are evaluated


To explain the findings on the needs to set goals and objectives at the start of a

campaign

These five points were chosen to be analysed, as it was these that allow for the author to

analyse, what is best practice, and what steps that can be taken, to ensure that evaluation in

Public Relations can be carried out accurately.

A coding table (appendix 1.3) has been used, as it enables the research to see they key data

in one table, allowing for trends to be identified, thus engaging with the data (Miles and

Hubberman, 1994; Yin, 1994).

30 | P a g e

4.2 - Agency Opinion on AVE

It was found that many agencies, while offering the AVE method, do not recommend this to

clients, as it refers only to outputs, not outcomes. This is something which links to the

literature review, as it was found that many authors stated, that AVE was an outdated

methodology, as well as being highly inaccurate, thus not giving the true value of PR.

Respondent C stated in their interview that 'it doesn't answer the problems, it is very

outdated.'

This quote, was countered by respondent A, with the comment 'you're talking their

language.' To put this comment in context, respondent A, was discussing the use of AVE, as

a language that many small business owners understand, as they may have commissioned

advertising in the past, therefore understand the currency.

It was found from the primary research, that out of the people interviewed, only two PR

organisation used AVE on a regular However, other agencies, offering it as a methodology to

clients , other methods found included cost per 1000, where the agency is able to work out,

how many people were reached, using the allocated budget. This again helps to measure

outputs, with a slight amount of outcomes, as the PRO is able to put a figure on readership,

this allows clients to understand exactly how many potential future customers they are

reaching. It was also found from the primary research that, many agencies do not stick to

one method, the methods they use, are dependent on the client's needs and wants from

their campaign, and what they want to know from the evaluation.

Other methods to measure good or bad PR, is by measuring reputation, as the fundamental

task of PR, is to change opinion of the public.

However, respondent D stated in the interview, that ' What we haven't done yet is make

that link, to show how things that affect reputation can affect share price can be affected by

good PR.' This quote discussed that while the industry has moved forward, in terms of

accurate evaluation, there is still a long way to go, before the answer will be around.

31 | P a g e

20923417

Following the literature review, it was noted, that many authors have shown in their work,

that finding one methodology, for evaluating public relations, is as elusive as the holy

grail(Pavlik, 1987:65).

As PR covers such a vast part of the business place, the requirement, to measure their

outcomes are extremely important. As quoted by respondent F, who stated very clearly that

evaluation in PR is 'very' important.

4.3 - The Need of Setting Goals & Objectives

The second main theme, found from the primary research stage, was the need to set goals

and objectives at the start of a campaign, this is vital, as it allows for the PRO to have some

of parameter to measure against. As respondent B quoted 'just because we have 250 press

cuttings, what does that mean to a client?' This quote states, that without communication

goals and objectives, numbers are simply meaningless.

It was found from the research carried out, that this was fundamental, with some agencies

not doing this as much, and simply agreeing to 'just market' clients products and hope for

the best.

The research found, smaller agencies do not do this on a regular basis, with respondent A

claiming that by setting goals and objectives, 'it can allow the PRO to be more realistic in

what they need to achieve', and not to over promise the client.

Also by setting goals and objectives, the client will understand, what they the aims are from

the campaign, and can also be assured, that the PRO is working at an acceptable speed with

their deliverables.

One reason why this may not happen as often, was found from another question in the

interview process, regarding the difference, between the in house and agency culture. It

was found that in house teams, may be prone to have more information, than would

normally be released to an agency.

Again, by quoting respondent A, it was found that 'Because they can be much more open

about their business plans and objectives than they can be to perhaps an outsider.' This is an

32 | P a g e

interesting quote, as many businesses, may not wish to publicise their plans, in fear or

having them copied, or being second to the mark with their own plans, as competitors many

have beaten them to the finish.

It was found from respondent G, who is in support of the AVE system, that this methodology

cannot use goals and objective at the start, with the comment 'No, not at all.'

It was stated by respondent H, that on social media, their agency is able to set a numeric

targets on how much engagement is required from the campaign. 'we will set a numerical

goal in terms of retweets, likes, hashtag uses.'

Writings such as those of Tom Watson, who have used the Pyramid Model of PR Research

(see appendix 1.1), which shows the process that PRO's should undertake this, when

creating a campaign, this is similar to the model of Smith's 9 step process, which shows, that

when the client is open and willing to share information with the PRO to gain a better

insight, the outputs given, will be very different, as the PRO is able to understand, exactly,

what is wanted to be achieved from the campaign in question.

4.4 - Multiple Medium Measurement

This dissertations third key finding related to the evaluation of various media's, as each

media is different, with different media's, having different capabilities and different

parameters to measure, it is has been felt by the author of this dissertation that different

ways are required to evaluate the PR results on different mediums.

Respondent E, who chooses to evaluate various mediums, with different methods, claimed

that 'Clients want to measure click throughs but only some clients have this technology to do

so.' Therefore, although the respondent prefers to measure different mediums, with

different techniques, it is not always possible, thus meaning each client is evaluated

differently, depending on needs, resources or budget.

Whereas, respondent C, stated that they would agree with the need, to evaluate different

mediums in different ways.

33 | P a g e

20923417

This was further confirmed by the quote of respondent G, that 'I think evaluation methods

do differ for different media.'

The literature review again has a mixed perception over this, with various authors arguing

their personal views on the subject, with many saying that PR, should have one form of

measurement, for the client to understand the value, with others arguing, that different

media's require different things, therefore, requiring different methods of evaluation. The

CIPR have stated, that online PR evaluation, should follow one protocol. Again, as

mentioned above, the issue with evaluation, is that PRO's attempt, to measure what they

need, as well as what they don't need to. This is something, that as outlined above, is a big

issue in the industry, as it means that PRO's, may be preoccupied, measuring the

unimportant aspects of the campaign, compared to ones which are going to make a

difference, to the clients business.

4.5 - International Evaluation Culture

The fourth key finding, discussed the need to understand, the local culture of international

agencies, when a UK agency, is acting as a lead agent. This happens when working with

clients, who require international PR.

It was discovered from the primary research, that a 66% of respondents, felt, that as long as

the media allowed the same methods of evaluation, including multipliers in AVE, methods

used by the lead agency should stand. However, as each media landscape is different,

respondent B correctly discussed the way media culture in different countries vary, thus

meaning evaluation methodology would need to be altered to reflect this.

Respondents quotes to this point, include respondent F, who stated 'For a Global Comms

teams, one method is important in terms of framework and success metrics. The targets

however have to adjust to take into account market differences.'

However, respondent D, gave a different opinion. As they stated that, 'one that is set in the

home country is the one that resonates around the world, if they can afford it, if the media in

the country allows it.'

34 | P a g e

Nevertheless, a third quote, from the primary research came from respondent B, who states

that ' Yes - for example, in China, where press release reprints are worth a lot , here they are

not, and if you tell a Chinese PRO this, they will be very offended as it is their system.'

Respondent B's quote came with local culture in mind, as the media landscape is different

compared to the one which operates in the UK, PRO's in China, would have to evaluate with

a different methodology. By altering methodologies for different countries, international

clients will be able to monitor their progress, in various countries around the world.

However, as stated by respondent I 'Global media trends'. This quote shows how many

media corporations will use the same stories in the same manner, however, it may not

always work this way, as different organisations use different methods of persuasion

depending on their agenda and audience.

This point was further confirmed by respondent H, with the comment 'I definitely think that

local culture and ethics make a huge difference. There may be an agency out there who

doesn’t properly measure their PR results, and in result, they may continue making a mistake

over and over again with different clients.'

The reason as to why agencies feel that the methodologies should remain the same, was

discovered in the literature review, as Watson described, by explaining, that as agencies

such as Burson Marstellar and Hill & Knowlton become international very quickly, there was

a need for PRO's to send evaluation figures to their head office, in order for CEO's to

monitor the progress.

4.6 - Overspending on Evaluation

An issue that was found from the primary research, is that many agencies regularly,

overspend on evaluation, with some agencies spending up to 40%, of the budget on this

task. It is important for agencies to address this point, as it is currently reducing outputs, for

example, a client budget of £1 Million, £400,000 is being spent on evaluating £600,000's

worth of PR. If clients became aware that such high proportions of the budget, was being

spent on evaluation, they may feel that their money is not causing a valuable difference to

their business, potentially scrapping their PR programme.

35 | P a g e

20923417

This point was raised by respondent B, who claimed that 'It is important not to over spend

on evaluation, many clients want figures, but they don't realise it can be costly and effective,

or cheap and effective, in this case cost doesn't really impact results. I feel the average spend

for evaluation should not exceed 15% of the overall budget, where as many cases are over

40%!'

This was also resonated in a quote made by respondent C, who stated, over servicing is a

industry wide issue. 'A PR agency will very rarely make money out of an account for the first

quarter, it's only because of the amount of money you put into winning that account, time

and effort, if you take that away from the fees, and in fact in that first quarter, you over

service it, you're getting up to speed, there is a lot of content, you've got to hit the ground

running'.

This is something which was not covered in the literature review, therefore, a new find,

which the industry may not have considered. However, due to the nature of PR, when

starting with a new client, the first and second quarter, working with the client may not be

profitable for the agency, due to the vast amount of time and money put into winning the

business, prior to the client signing up with the agency. Therefore over servicing is an issue

in the industry which cannot be resolved, as from the first day, the PRO is having to evaluate

their work.

4.7 - Key Findings Conclusion

In conclusion to this section, it was established, that while PRO's offer AVE as a form of

evaluation, it was not a method which a majority of respondents choose to use. This

therefore disregarded the literature review, as many authors have stated that PRO's use the

AVE methodology to evaluate their work on a regular basis.

The second key finding, was regarding the needs to set goals and objectives, prior to the

start of a PR campaign. This was found to be an important stage in the evaluation process,

as many people interviewed have claimed that it is not possible to evaluate PR, if there are

no communications goals or objectives. The literature review also showed the importance

of this, with Smiths 9 step model, which incorporates this into the planning process of PR.

36 | P a g e

Thirdly, multiple media evaluation is one topic, which the literature review and primary

research has provided mixed views on. Many PRO's, and academics argue that PR should

have one form of measurement, in order to use the same metrics, however, others argue

that different mediums require different methods of evaluation to be accurate and efficient.

The author of this dissertation believes that multiple mediums, should be measured in

different ways, as each have their own pros and cons.

The fourth recommendation listed, was based around international evaluation culture, and

whether local culture comes into consideration, when evaluating international PR

campaigns when partner agencies are employed.

It is believed by the author of the dissertation, that as the local culture and media landscape

may not always operate in the same manner as the UK media landscape, the way

international PR is evaluated should differ to account for this.

The literature review, showed little information on this topic, with one source claiming that

the early PR agencies commissioned one method of evaluation, as it allowed for PR agency

CEO's to monitor how regional offices are performing.

The final key finding was on overspending of evaluation in the industry. While not many

respondents discussed this, it was found that two respondents are concerned about over

servicing in the industry.

37 | P a g e

20923417

5.0 - Conclusion:

In conclusion to this dissertation, it was found that due to the high number of internal

methods of evaluation, this dissertation has only been able to evaluate a small number of

these methods.

Following the literature review, the key issues identified, were that many PRO's attempted

to measure too much, this means, they are measuring things that would not have made a

difference to their client, in order to make themselves look better. It was also found that

many PRO's, actively increased the value of their evaluation figures, again to impress clients

and giving them the perception that PR was much more cost effective than it actually was.

Questions also remained unanswered, regarding the evaluation in international agencies, as

well as the cultural differences, between in house and agency staff, as the business

objectives and goals are different. This may influence the way that agencies would like to

evaluate their work, as it was found from the primary research, there was a need to

evaluate outputs in agencies, but with in-house teams, the need to evaluate outcomes to

the business were much greater.

Part of the research methodology used, was a deductive semi structured interview

approach, this allowed the interviewer, and interviewee, to hold a free discussion on the

topic, compared to a structured interview, where the interviewer takes full control in the

conversation, not allowing the respondent to give varied details.

The research was carried out, using face to face and e-mail interviews, which were offered

to PRO's, both agency side and in house, however, as explained in the critique section,

people had to drop out due to business related issues. Therefore finding respondents who

kept their appointments was an issue.

38 | P a g e

The objectives set were to:

Research Objectives






By comparing the primary research findings with the research objectives, it was been found

that some objectives have been disproven, while others had been confirmed, potentially

leading to some industry wide changes in terms of online evaluation in PR.

It was discovered from the primary research, that the AVE system, is not the most common

method used in the PR industry, therefore this objective had been disproven. This was

proven extensively, as all but one respondent claimed they use various methods, depending

on the clients wants, and their budget.

It was found that the main theme from the literature review, was to set goals and objectives

at the start of any campaign. This was a trend, which was heavily promoted in the primary

research, with many respondents stating, that without this happening, it would not be

possible to measure accurately, thus returning an issue highlighted in the literature review.

Where PRO's have been known for evaluating much more than they need to.

It was also planned to discover, whether the culture of international agencies made a

difference, in the way that international colleagues evaluated work. It was found that there

39 | P a g e

20923417

was a split approach in the way that UK PRO's see the culture and methodologies, with

some believing that methodologies, should be the same in partner agencies, rather than

being different in other cultures.

40 | P a g e

6.0 - Recommendations:

It was established from the primary research data, in section four of this dissertation, that it

was not possible to ascertain a unified method of evaluation, for the Public Relations

industry, which was cost effective and accurate. However, this section of the dissertation,

will make strategic recommendations, as to how evaluation in the industry, can be

improved, in a cost effective manner, with little risk to the industry.

6.1 - Goals & Objectives Recommendation

The first recommendation, was based on the finding of the importance, to agree a set of

communication goals and objectives, at the start of the campaign, or at the start of a

quarter. This is an essential process, as it will allow PRO's to understand, how to target the

media outlets, and with which stories. By understanding what the client wants to achieve, it

is possible to plan a campaign around that, thus, giving real parameters for PRO's to

measure against.

By having such solid parameters, such as, driving 20% more traffic to a website, or gaining

35% more inquires, it will possible for PRO's to work with clients, to measure this, by

monitoring website traffic, or enquiries, it can be found out whether the campaign has

generated this increase. If it is found that the campaign has provided this increase, then the

campaign has been a success, however, it is important for the campaign, to run side by side,

with the monitoring process, as it will enable the PRO, to make alterations, should things go

wrong.

This recommendation, is one that should be spread out across the industry, as there are

many different forms of evaluation, used in the industry, it is felt by the author of this study,

that this will ensure accuracy of evaluation, can be improved.

41 | P a g e

20923417

6.2 - Overspending Recommendation

The second recommendation made, is for agencies to monitor expenditure, in terms of how

many hours are being spent on evaluation. As outlined in section 4.6, it was found that PR

agencies, regularly over spent on evaluation. In order to combat this, it is recommended,

that PR agencies take steps, at the start of the campaign to ensure that they are only

evaluating what is needed, and not anything more.

As PR is a medium that is constantly trying to increase its presence, as a form of marketing,

which is better value for money than advertising, many agencies, are spending too many

hours, simply trying to accomplish this, therefore, effectively wasting money, on tasks

where cost, does not affect the quality of the final figure.

Therefore, this key recommendation, is for account executives and managers, to monitor

how long PRO's are spending on evaluation work. In order to help PRO's measure what is

needed, it is important for the agency, at the start of the campaign, to sit down with their

client, to establish which publications, websites, and blogs, are the most influential in the

clients business sector.

By establishing this, not only can the correct media be targeted, but the PRO can then make

a decision, whether other media outlets will follow suit. Should this be the case, it will be

possible, for the PRO, to only measure the effectiveness of the coverage, from the media

outlets that will be most influential, thus making a larger difference to the clients business,

compared to the smaller publications which the clients customers, may not read.

This will also require the PRO, to get a greater understanding of the clients business sector,

allowing them to learn more about the client's choice of media. By understanding this, it will

be possible for the PRO, to understand the client's business much clearer, by having such a

clear comprehension, they will be able to provide better outputs for their clients, as they

will be able to sell PR to journalists with greater ease.

42 | P a g e

6.3 - Multi Medium Evaluation Recommendation

The third recommendation, from this dissertation, is to advise PRO's of the need to have

different evaluation models, for different types of media. As stated in the literature review,

the CIPR have provided a template of parameters to follow. By following this table, it will be

possible to establish exactly what result, a digital campaign has achieved.

It is important that the first step for PRO's, is to divide their evaluation, from online and

traditional. Following this, the online method can be further adapted for the use of social

media.

Online PR can be tracked by the use of Google Analytics, or other tracking services, this will

allow for the PRO, or client to monitor what traffic is going to a web page, and where it is

coming from. By analysing this, a PRO would be able to measure the success of a campaign,

depending on the objective set, as, if the objective was increase traffic to a specific

webpage, this can be monitored with ease as it is a solid objective, with a clear outcome.

However, if page visitors are not interacting with the website, and are simply spending time

on the webpage, the ROI value may not be as high as a client would have liked. However, on

sources such as Google Analytics, the user is able to set a conversation rate, which is in line

with their business.

The tracking software, will then provide reports as to what sources, are providing the best

form of ROI for a client. This is where the client will be able to see, where their website

traffic and engagement is coming from, and with what value. The value is something, which

is advisable to be preset with the PRO, before the campaign. It is possible to find

information on bounce rates, this provides a value of how many people have dropped off

the website without engaging in any form. The industry average for this stands at between

30 and 40% of visitors (Marketing Journal Blog, 2012).

By allowing for this drop off rate, a value can be put onto the value of online PR.

43 | P a g e

20923417

It is recommended, that social media PR, should be measured by the number of

engagements which happen per post, it is recommendable, for PRO's sit with clients, to

discuss the business's financial value for each engagement, following this, an accurate and

personal ROI can be given to each client, as to what each social platform is providing.

The further recommendation, is for agencies to do with this all clients, to calculate an

average value overall. By doing this, the industry can determine an industry standard. This

will allow for the social media platform to be evaluated effectively.

This will allow for businesses, to see a financial tangible value on their social media efforts,

helping them to analyse whether or not social media is working for them.

6.4 - International Evaluation Culture Recommendation

The forth recommendation, is working with international agencies, it was found from the

primary research, that 50% of respondents, felt there was a difference in culture, and that

local culture and ethics, influence the way in which media is evaluated abroad. Also, as

media is different in various countries, evaluation would have to be undertaken in a

different manner, as values and landscape would be different. Therefore, models used in

the United Kingdom, and the United States of America, may not be valid in China, or Russia.

Due to this fact, international agencies, or agencies with partner agencies abroad, will need

to discuss ways to evaluate work effectively and accurately, depending on the local culture.

It was found from the primary research, that one respondent, clearer stated in China, press

release reprints or 'lazy editing' are normal, however, in the UK, this is not normal practice,

therefore, the value, could be considered to be less, as the journalist has not written the

article in the style they would normally have, nor may it be in the personal tone that the

readers are used to.

As this is the norm in other counties, PR colleagues based in these countries, may become

very offended, should a UK PRO discuss the value of their outputs, and claim they are very

low.

In order to combat this, PRO's need to discuss the local culture, and norms with agencies

abroad, in order for a seamless evaluation process to take place, as this will allow the lead

44 | P a g e

agency, to inform their client of the various ROI's abroad. This can help the client, as they

can decide exactly what it is that is going on around the world, in their PR efforts, while

obtaining accurate ROI figures for each nation they are operating PR in.

Should lead agencies expect partner agencies to evaluate using the same methods,

inaccurate figures can be reached. Therefore, making the ROI figures produced inaccurate.

6.5 - Recommendation Conclusion

In summary, the recommendations proposed in this section, are advising PRO's, to monitor

the time they spend on evaluation, in order to give clients the best possible value of money,

with around 10 to 15% of the overall budget, being spent on evaluation purposes.

The second recommendation proposed, to set communication goals and objectives at the

start of a campaign, in order to allow for the PRO to have a set of solid targets, they will be

able to accurately measure their work.

Thirdly, it is proposed, that PRO's should use two methods of online evaluation, first is the

use of Google Analytics and bounce rates, by setting an ROI figure for each type of

conversion, from key word searches, to click throughs, it will be possible to establish how

much ROI a campaign has given a company, as mentioned above, the PRO and client would

need to decide a financial figure for each conversion method. This can then be monitored

throughout the campaign, to ensure the objectives are being met.

Secondly, a similar approach is proposed for social media, with every form of engagement

given a personal value for each client, in regards to how much value, the client feels they are

getting from engaging with a member of public in a certain way, through their social

network portal.

By setting individual figures for clients, agencies will be able to set agency standards, which

in time will lead into industry standards. This is something that can be done, for both social

media, as well as other online methods used.

The fourth and final recommendation proposed in this dissertation, is to ensure that all

PRO's understand the culture and ethics of partner agencies, who operate abroad, in order

45 | P a g e

20923417

to understand how they undertake their evaluation, as media landscapes are different in

every country, where some countries, such as China, still operate a Government controlled

media, where as others, such as the UK operate free media.

It is felt that these recommendations will help the PR industry, to improve the accuracy of

evaluation while providing an initial answer to working with agencies abroad, as some

agencies are still naive in believing local culture and media landscapes vary, the

recommendations will also allow a frame work for evaluating online PR and social media, as

this is something which is currently seen to be evaluated in the same way as traditional PR,

even though the landscape of the outlet is very different.

By carrying out the above recommendations, it will be possible for PRO's to offer a cheap

and effective method of evaluation, in both traditional and online PR, including social

media.

46 | P a g e

Reference List & Bibliography AMEC, 2012, AVEs not an awards answer, [Online], AMEC, Available at:

http://amecorg.com/2012/06/ave%E2%80%99s-not-an-awards-answer/, [Accessed

13th December 2012]

AMEC International Association for the Measurement and Evaluation of

Communications, (2010). Barcelona Declaration of Measurement Principles, [Online],

London, AMEC, Available from:

http://www.amecorg.com/newsletter/BarcelonaPrinciplesforPRMeasurementslides.

pdf. [Accessed 31 December 2012]

AMEC, n.d, International association for the measurement and evaluation of

communication, [Online], AMEC, Available at: www.amecorg.com, [Accessed 24th

September 2012]

AMEC, 2011, Valid metrics for pr measurement, [Online], AMEC, Available at:

http://ameceuropeansummit.org/amecorgdocs/ValidMetricsFramework7June2011P

rintVersion.pdf, [ Accessed 14th October 2012]

Anderson FW, Hadley L, Rockland D & Weiner M, 2009, Guidelines for setting

measureable public relations objectives: an update, [Online], Anderson FW, Hadley L,

Rockland D & Weiner M, Available at:

http://www.institueforp.org/files/uploads/Setting_PR_Objectives.pdf, [Accessed

13th April 2013]

Babbie E, 2010, The practice of social research, 12th ed, Canada, Nelson Education

LTD

Baran SJ, Davies DK, 2008, Mass communication theories: foundations, ferment and

future, Boston, Cengage Learning.

Barry A, 2004, PR power, London, Virgin Books

Baskin O, Hahn J, Seamen S and Reines D, 2010, Perceived effectiveness and

implementation of public relations measurement and evaluation tools among

European providers and consumers of PR services, Public Relations Review 36 (2), PP

105-111

47 | P a g e

20923417

Bengston DN, Fang DP, 1999, An innovative method for evaluating strategic goals in

a public agency, Evaluation Review, 23, (1), pp77-100

Bernard HM, 2006, Research methods in anthropology: qualitative And quantitative

approaches, Oxford, AltaMira Press

Bernays E, 1923, Crystallizing public opinion, New York, Boni and Livenight

Botan CH, Hazleton V, 1989, Public relations theory, Hillsdale, Lawrence Erlbaum

Associates.

British Medical Journal, 2008, Qualitative Research, [Online], British Medical Journal,

Available at: http://www.bmj.com/content/337/bmj.39602.690162.47, [Accessed

28th April 2013]

Broom GM, 2009, Cutlip and centre's effective public relations, 10th Ed, Upper

Saddle River, Pearson

Bryman A, 2008, Social research methods, 3rd ed, Oxford University Press, Oxford

Burney, SM, 2008, Inductive and deductive research approach [Online], Dr SM Aqil

Burney, Available at:

http://www.drburney.net/INDUCTIVE%20&%20DEDUCTIVE%20RESEARCH%20APPR

OACH%2006032008.pdf, [Accessed 30th April 2013]

Carey, B, 2013, Active PRO's, S Mashru, 10th January 2013, 17.30

Charmaz K, 2006, Constructing ground theory, a practical guide through qualitative

analysis, London, Sage Publications.

CIPR, 2010, Barcelona principals, the end of ave?, [Online], Jay O’Connor, Available

at: http://www.cipr.co.uk/content/news-opinion/presidents-blog/4912/barcelona-

principles-the-end-of-ave-, [Accessed 12th October 2012]

CIPR, 2012, Share this, West Sussex, John Whiley.

Collins Dictionary, 2013, Definition of evaluation, [Online], Collins Dictionary,

Available at:

http://www.collinsdictionary.com/dictionary/english/evaluation?showCookiePolicy=

true, [Accessed 24th April 2013]

48 | P a g e

Cutlip, S.M. (1994) The unseen power: Public relations, a history, Hillsdale, N.J,

Erlbaum Associates.

Dey ,I, 1993, Qualitative data analysis, Routledge, London

Gregory, A. (2010) Planning and managing Public Relations Campaigns. 3rd Ed,

London UK: Kogan Page.

Guardian, 2012, Eastenders wins Christmas day ratings, [Online], John Plunkett,

Available at: http://www.guardian.co.uk/media/2012/dec/26/eastenders-wins-

christmas-day-ratings, [Accessed 30th December 2012]

Harlow, R.F. (1942) Public relations in war and peace, New York, Harper and

Brothers.

Heath, R. (2000), Handbook of public relations, Sage Publications, London

Holmes, P, 2010, A stake through the heart of advertising value equivalency,

[Online], Paul Holmes, Available at:

http://www.holmesreport.com/blog/index.cfm/2010/6/21/A-Stake-Through-the-

Heart-of-Advertising-Value-Equivalency, [Accessed 22nd December 2012]

IPR Institute of Public Relations. (1958) A guide to the practice of public relations.

London, Newman Neame.

Jefkins F, 1992, Public relations technique, Butterworth-Heinemann LTD, Oxford

Kimmel A, 1996, Ethical issues in behavioural research, Blackwell Publishers, Oxford

L’Etang, J. (2004) Public relations in Britain, Mahwah, NJ, Lawrence Erlbaum

Associates.

Lincoln Y, Denzin N, 2003, The landscape of qualitative research, 2nd ed, Sage

Publishers, London

Lindlof T.R. and Taylor B.C., 2002, Qualitative communication research methods 2nd

ed, Sage Publications, Thousand Oaks, CA

Marketing Journal Blog, 2012, What is the average bounce rate?, [Online], Marketing

Journal Blog, Available at: http://www.marketingjournalblog.com/2012/01/what-is-

average-bounce-rate.html, [Accessed 21st April 2013]

Marston, J. (1963) The nature of public relations, New York, McGraw-Hill

49 | P a g e

20923417

Maxfield M, Babbie E, 2012, Basic research methods, Cengage Education, USA

McCoy, M, Hargie, O. (2003) Implications of mass communication theory for

asymmetric public relations evaluation, Journal of Communication Management, (4),

pp 304–16

McGuire, WJ. (1984) Attitudes and attitude change, in Handbook of Social

Psychology, II, 3rd ed, ed.

Merriam S, 2009, Qualitative research, Jossey Bass, USA

Miles MB, Hubberman AM, 1994, Qualitative data analysis, 2nd ed, Sage

Publications, London

Milyankova, R,(n.d), Choosing research & approach, [PowerPoint Slides] Russia:

International University College, Available at: www.iuc-

edu.eu%2Fgroup%2Fsem1_L2%2F2012%2520PDEVR%2F_7_PDR_Choosing%2520str

ategy_method.ppt&ei=vq3uUP3VBseY1AW27oDACg&usg=AFQjCNEQBgN32qYP3KAk

_ONUIzUPYL32pw&bvm=bv.1357700187,d.d2k, [Accessed 10th January 2013]

Morehouse R, Maykut P, 1994, Beginning qualitative research, Falmer Press London

Nichols, B. (2013) PR strategy part V: research & evaluation 2, [PowerPoint Slides],

Presented at MG606 at Buckinghamshire New University.

Paine KD. (2011) Measure what matters: online tools for understanding customers,

social media and key relationships, West Sussex, John Wileys & Sons LTD

Palvik, J, V. (1987) Public Relations: what research tells us, Newbury Park, NJ, Sage

Philips, D. (2001) The PR evaluation evaluationists, 6 (14), p2

PRCA, 2004, What is PR, London, PRCA

PRmoment.com, 2010, The AMEC principles of PR evaluation, [Online], Ben Smith,

Available at: http://blog.prmoment.com/the-amec-principles-of-pr-evaluation/

[Accessed 13th December 2012]

50 | P a g e

PR Week, 2012, Measurement in pr has ‘not changed in 20 years’, [Online], Sara

Luker, Available at: http://www.prweek.com/uk/news/1139832/, [Accessed 3rd

January 2012]

Riley J, 1996, Getting the most from your data, a handbook of practical idea on how

to analyse qualitative data, 2nd ed, Bristol, Technical & Educational Services

Robson C, 2002, Real world research, 2nd ed, Blackwell Publishing, Oxford

Romy R PR,(n.d), How to measure PR effectiveness, [Online], RomyR PR, Available at:

http://www.romyr.com/PRinsights/PR_effectiveness?cs_lang=en, [Accessed 28th

January 2013]

Sage Research Publications, n.d, Semi-structured interviews, [Online], Sage

Publications, Available at: http://srmo.sagepub.com/view/the-sage-encyclopedia-of-

social-science-research-methods/n909.xml , [Accessed 29th April 2013]

Saunders M, Lewis P & Thornhill A, 2003, Research methods for business students, 3rd

ed, Harlow, Pearsons Education LTD

Scott, DM, 2010, The new rules of marketing & pr, 2nd Ed, New Jersey, John Whiley &

Sons

Sheldrake, P, 2011, The business of influence, New Jersey, John Whiley & Sons

Silverman D, 2010, Doing qualitative research, 3rd ed, Sage Publications, London

Strauss A, Corbin J, 1998, Basics of qualitative research, 2nd ed, Newbury Park CA,

Sage

Tench R, Yeomans L. (2006) Exploring public relations, 2nd Ed, Harlow, Essex,

Pearsons Education LTD

Theaker A.2012, The public relations handbook, 4th Ed, Oxon, Routledge

Turner S, 2011, Why PR evaluation needs objectives to succeed, [Online], Freshfield

Communications, Available at: http://www.freshfield.com/pr-evaluation-debate-takes-

new-direction/, [Accessed 3rd October 2012]

Underwood J and Stevenson J, 2013, Public Affairs, S Mashru, 19th March 2013,

10.45am

51 | P a g e

20923417

Watson, T, Noble, P. (2007) Evaluating Public Relations. 2nd Ed, London, Kogan Page.

Watson, T. (1994) Public relations evaluation: nationwide survey of practice in the

United Kingdom, Paper presented to the International Public Relations Research

Symposium, Bled, Slovenia, July 1994.

Watson T. (2012) The evolution of evaluation – the accelerating march towards the

measurement of public relations effectiveness. Public Relations Review. 38 (3), p390-

398.

Wilson, Alan, 2012, Marketing research an integrated approach, 3rd ed, Harlow,

Pearsons Education LTD

52 | P a g e

Appendix 1.1 - Pyramid of PR Research

Watson and Noble, 2007, P85 1

Appendix 1.2 - Interviewee Profile Table Interviewee Title Gender Years in

Industry

A 2 Respondents: 1, 2

1) Managing Director (PR)

2) Account Manager (PR)

1) F 2) F

1) 10 Years 2) 5 Years

B B) Vice President (PR) B) M B) 13 Years

C C) Board Director (PR) C) M C) 13 Years

D D) Corporate Affairs Partner (PR)

C) M C)24 Years

E E) Junior Account Director (PR)

E) F E) 6 Years

F F) Head of Analysis and Insight

F) F F) 11 Years

G G) Director of Communications

G)M G) 6 Years

H H) PR & Comms Specialist.

H) F H) 7 Years

I I) Head of Social Media I)M I) 12 Years

Appendix 1.3 - Coding Table

53 | P a g e

20923417

Code Where Quote to use Notes Linked Reference

Methods of Evaluation

A- Line: 11 B- Line:3-4 C- Line: 7, 10-12, 14-15, 17,20, 27, 44-45, 49, 55, 64,-67 D- Line: 7-10,12 E- Line: 2-3 F- Line: 2-6 G - Line: 2

A- AVE - Advertising Value Equivalent. B- Depends on client needs, measure what they want, not what we want. C- one size doesn't fit all, what methodology suits your business, we use cost per 1000, Apollo measures share of voice, Metrica is very expensive. D- Media monitoring, measuring sentiment, and reputation changing studies. E - AVE, Circulation, total number of cuttings. F- Start with what is cheap and easy, media content analysis is often used as a basis for the success of PR. G - AVEs probably, although I have no evidence of this.

Interviewee A and B have taken opposing views on the requirements of evaluation. Interviewee C has taken a similar approach to that of

agency B where they measure on the client's needs.

Specialist agencies who evaluate work are very

expensive, PR's need to be flexible on the methods they

use for clients

Interviewee D has worked from a corporate angle, where

the need to measure reputation is key.

Interviewee E offers few forms

of evaluation, with basic results monitoring outputs. Interviewee F works for a

media evaluation company who feels that the delivery of

the content is key.

Interviewee G states that they feel AVE is the most common

method used.

Watson, 2012: It has been found that one accurate key method which looks at outcomes compared to outputs is very difficult to find.

Gregory, 2010: Without doing accurate evaluation, PR will be at risk of being a

communicator that is intangible or ‘far too important to be measured’.

Nothing Obvious

54 | P a g e

Methods of Evaluation

H - Line: 2,3 I - Line: 2-5

H - Google analytics for the website,

retweets on Twitter, likes on

Facebook, Google alerts, hash-tags on

Twitter and Instagram.

I - The most common method of

evaluation unfortunately is still based

on OTS (opportunity to see). This is

probably based on clients’ familiarity

with the notion of “reach” from ad

campaigns and a requirement to

translate PR into comparable return

on investment.

Interviewee H specialises in digital and social media PR,

they analyse the engagement of social media.

Interviewee I feels that the OTS method as it can give the client an understand of who

they have spoken to.

Do your methods

require goals and

objectives to be set

at the start of the

campaign?

A- Line:16 B - Line: 7-8 C - Line 74,79-80 D- Line 17 E- Line: 7-8

A - Yes B- Yes- if you don't set targets, what are you measuring? C – Absolutely, if not you are going blind...

D - When running specific campaigns yes, but when in corporate there is a continuous measurement looking at reputation changes. E - Set with clients in accordance with business goals.

All interviewees are in agreement here, except

interviewee G who feels that even AVE's do not need goals

and objectives.

Need solid parameters to measure against.

Not all campaigns need to

have set goals and campaigns.

Tench and Yeomans, 2006 Expressed that evaluating Public Relations efforts is extremely important, as it reflects

effectiveness, professionalism, whilst giving accountability for the efforts.

55 | P a g e

20923417

F - Line 9-13 G- Line: 5 H - Line: 6,8 I- Line: 8-14

F - If done properly, depends on the type of campaign. G- No, not at all. H - Yes. For example, if we begin a

social media campaign and want to

build a buzz about a new product, we

will set a numerical goal in terms of

retweets, likes, hashtag uses.

I- It often does, although OTS can be

easily gamed by securing online

coverage. If OTS is a key

measurement then demographic

should be part of this from the outset

(i.e. the right people see the

coverage). Also a list of key media

should be defined up front to avoid

coverage being secured in non-

relevant media and being counted in

the final evaluation tally. Often, there

is a volume target set at the start of a

campaign (e.g. 15 x pieces of national

coverage, 25 x pieces of regional, 2 x

broadcast, etc.).

Social Media means that it is possible to set numeric goals

in terms of engagement.

The OTS system is required to set goals and objectives, as

the target audience will need to be addressed effectively.

56 | P a g e

Campaign

Failure

Recalls

A- Line:35-39 B - Line: 13 C- Line 93-97, 99-103 D- Line 41-46 E- Line: 13 F- Line: 18-21

A- Product PR is hard to get negative

PR, generally always positive, if

something doesn't work it is replaced,

not recalled.

B- Only add if the content is suitable.

C- Not so much gone wrong, but more of looking at it around a month in and saying, this will not achieve what we want to, let's review it. It's about understand what success looks like. D- I've advised the client to stop doing

something, because the original idea

was not a good one.

So we advised them to stop running

the campaign and re think the offer to

the market.

E - No

F- PR doesn't work with recalling, it

reprioritises, if something doesn't

work it should turn to damage

limitation.

Interviewee A & B have

agreed that as long as the

plan was still valid the content

can be tweaked.

Interviewees C & D have

stated that if something does

not work how they wanted it

to or if something else is

happening that will affect the

outcome, they will advise

their clients put their plans on

hold.

Interviewee E has not had to

recall or alter any campaign.

Interviewee F claims that

should things go wrong, PR

should concentrate on

containing the damage, not

Nothing Obvious.

PR Week, 2012 quote ‘A real problem is the PR industry thinks its job is to get media

coverage or run a good event – that’s not the case. The PR industry’s job is to influence and

change behaviour of consumers and stakeholders.’

57 | P a g e

20923417

Campaign

Failure

Recalls

G- Line: 10 H - Line: 15 I- Line: 22-25

G- No

H - No

I- No. Often, the strategy or KPIs are

revised depending on initial results.

So, if a brand has a new product

range, KPIs could be increased.

Likewise, if a key component of a

campaign is no longer possible to

activate, KPIs can also be rejigged to

make the most of the campaign fee.

worry about fixing it.

Interviewee G and H has

never across this in their

career.

Interviewee I - has not had to

recall a campaign, as the KPI's

are active and can be

increased should the

campaign fail.

Mass Media affect on way PR is seen in the business place?

A- Line:44-47, 54-55 B - Line: 16-18 C - Line: 117-121, 123-124, 128-129

D - Line: 57, 59-60

A- Some clients want to be everywhere, and they want to be where you can't always get them. B- Mass Media here is not as large as others abroad. C- Target specific titles, choose them as clients customers are reading them, Germany has much more media than us, doesn't impact it. D- People are more aware of what can go wrong and what can go right. Globalisation of media has made PR an important.

A believes there is a mass

media

Interviewee B & C do not feel

that the UK has a mass media

scheme anymore.

Interviewee D claims that due

to the mass media effect in

the UK and around the world,

if things go wrong, they go

wrong quickly, however, good

news also travels very quickly.

Tench & Yeomans, 2006 state that PR's overemphasize the value of print media to

make it look competitive with digital sources.

CIPR Share This, 2012, has stated that there

are five steps that should be taken when it

comes to carrying out digital PR evaluation,

these steps include: identify the influences,

check for the current evaluation and error

rates, what and when engagement occurs,

58 | P a g e


E- Line: 16-17 F - Line: 24-31 G- Line: 13-15 H - Line: 18,22

E- I think the large choice of media

allows businesses to reach people in

different ways and allows PRs to be

truly creative and proactive in their

pitching of ideas.

F- Mass media is not connected, social

media is more key, people can talk to

a business directly.

G- Not sure what you mean - I do

think that the media is influential on

the way PR is seen in a number of

ways. If you mean with regards to

evaluation, I do not see any

connection.

H - The industry of PR can have a

negative reputation because many

people who are not a part of the

industry think we “spin” stories. This

is not true, because most PR

professionals are well-versed in

what’s ethically right in terms of

communication with the public and a

Interviewee E feels that the

Mass Media acts as a positive

for the industry.

Interviewee F feels that there

is not a mass media, and that

it is more important to

concentrate on social media

and digital forums, as this

affects the way PR is seen in

the business place.

Interviewee G feels the mass

media does not affect the way

PR is seen.

Interviewee H feels that PR

will never have a negative

image, as PR is not about spin,

it is, it is about being ethical.

looking at the number of data sources picked

up, and understanding how many could be

missed, and finally, ensuring the correct

target is being met.

59 | P a g e

20923417


I- Line: 28-31

private organization. I do not call it

spinning, it is more like finding what’s

newsworthy and running with it.

I- There is still a misconception of

what PR actually is both within the

wider public and within businesses in

the UK. Particularly in the small

business space, PR is seen as a poor

relation of advertising whereas the

reality is that it can often deliver

better long-term results for

companies when compared against

paid-for media.

Interviewee I feels that PR

gives better long term results

compared against paid for

advertising.

Unified Method of evaluation for all

clients?

A- Line:76-77 B - Line:20-23 C- Line: 172 D-Line: 93-96

A- No, there is room for manoeuvre, but no generally no. B - No silver cloud, everyone is different, where many agencies go wrong. C- It depends on the clients budgets and needs. D- I got very involved with evaluation from the start and try to push for metrics that allow reputation change tracking as well.

Interviewee A treats all clients equally, giving them one form of measurement which they say their clients know and

use.

Interviewee B and C base their plans on what the client

wants to achieve and how much they want to spend.

Interviewee D used evaluation

to measure the change in share prices following a PR

campaign by looking at

Nothing obvious

60 | P a g e

Unified Method of evaluation for all

clients?

E- Line: 20 F- Line: 33-34 G- Line: 17-18 H - Line: 24,25 I-Line: 42-46

E- No, we tailor our methods to their

reporting needs and business

processes.

F- Smart PROs are using best practice

frameworks fitting into clients needs.

G - No, there are many methods, and many agencies have their own methods that they have used themselves. AVEs do seem to be the most prevalent however. H - Yes. We have our clients fill-out a

survey about their experience, and if

we delivered the results they wanted

and expected.

I - Not in my experience. I know that

we try to use qualitative and

quantitative methods of evaluation

for all clients but have to tweak our

evaluation methodology to align them

with our clients’ business objectives.

So, if a client is interested purely in

inbound clicks to their website(s), our

reputation change.

Interviewee E is able to measure what the client

wants measured.

Interviewee F feels that some PRO's do have one method,

but the better ones can measure anything.

Interviewee G feels that the AVE system is still the widest

used.

Interviewee H's agency uses surveys to establish what

clients want from their evaluation results.

Interviewee I believes that PR should have different

methodologies for evaluation depending on the mediums.

61 | P a g e

20923417

evaluation must prove that we have

delivered on this regardless of how

many pieces of coverage we secured.

PRO's increase the

value of evaluation

to make themselves

look good.

A- Line:89-95 B - Line:26 C- Line: 145, 149-151, D- Line: 67-68 E- Line: 23 F - Line: 37-43 G - Line: 21-24

A- Yes, few tools available, estimate, language business people understand, they've used advertising so they know how it works. B- Yes they do, and they have been caught for doing this. C: I'm sure there are some people like this, but we often report to a marketing manager, so we try to make them look good, not us. D-Agencies & In house are guilty of this, over enthusiastic, media volumes with multipliers. E- No F - The one way to do this is by using AVE's, which are nonsensical, does not answer the question, no research can be done to find out what multiplier used for AVE is accurate due to too many parameters. G- There is one example of how this seems to happen - many PROs multiply the value of their AVEs by 3,

All interviewees are in

agreement here, all agree that PRO's are being too kind by

giving figures that are increased, meaning that they are not giving clients the true

value of the PR to their business .

When reporting to a

marketing manager, they need to make sure their budgets

don't get cut, so a degree of reality needs to be kept.

Interviewee F feels this is only done if PROs use the AVE system, and believes that PR holds more value

than advertising, but is not sure how much.

Interviewee G has stated the AVE method, allows for this,

therefore inaccurate.

Harlow, 1942 states that the system of ‘pepping up’, this was used when images had been published, with the PRO enlarging the image, thus claiming more space, making it seem the coverage was much larger than it

was.

62 | P a g e

PRO's increase the

value of evaluation

to make themselves

look good.

H - Line: 28 I - Line: 49,50

justifying this by claiming that PR is 3 times as effective as advertising. This seems to be completely unscientific and just there to increase the value. H - Not that I know of, that would be unethical.

I - Part of our job is to work to

whatever methods of presentation

best suit our clients. That said, we

would never fabricate results.

Interviewee H feels that ethics comes into this in a big way,

therefore, ethical issues have been identified.

Interviewee I believes that this is not an industry issue.

How and why would

you change

evaluation models?

A- Line:98-110 B - Line:29-34 C- Line: 193-194

A- See it computerised, complicated due to high number of parameters. B- They can't be changed, only improved, putting methods together, altering methods to suit client needs and wants, make it cheap as possible, cost does not affect effectiveness. Also the need for human interaction is vital, a computer can't read the tone of an article. C- You can't change them, merging them will help make it clearer to clients.

This question was asked to establish what would be done

differently in possible. Interviewee A believes that if

possible computerisation should take place to make life

easier.

Interviewee B believes that this shouldn't happen as

computers can't be as accurate as humans. Whereas

interviewee A would like to see computerisation take

place.

Nothing obvious

Recommendation: Keep AVE to measure

outputs internally, but give clients

information on how the outcomes will affect

63 | P a g e

20923417

How and why would

you change

evaluation models?

D- Line: 101-107 E- Line: 26-28

F - Line: 45-48 G - Line: 26-29 H - Line: 31 I - Line: 52-58

D- Need to find the link between good PR and reputation change, leading to the ultimate bottom line to the client. E- More emphasis on sales as a result of a click through from online coverage. F - By looking at how PR can help the business, thinking upfront means you don't need to be spurious with evaluation. G - A method based on the full length of the campaign, which focuses on the initial requirements, and whether the right people we actually reached. The AMEC meeting in June is expected to put forward a new form of evaluation which I would suggest will be useful to have a look at. H - I would not, I think they do a great job.

I- I would fully embrace a uniform

system that takes on board the

Barcelona Declaration

of Measurement Principles that set

out to include volume as well as

impact of campaigns. These principles

go beyond pure OTS and offer a

Interviewee C and D agree that merging multiple

methods together will allow for further clarification of the

outcome for the client.

Interviewee F feels that the PR industry should start with brain storming, to understand

every business is different with different needs.

Interviewee G would like to see evaluation based on

whether the correct people were reached.

Interviewee H believes that the methods used in their

agency answer the question effectively.

Interviewee I feels that PR

should have a unified evaluation around the Barcelona Principles.

their business, product PR - check to see if

any enquiries are being received for

campaigned products.

64 | P a g e

better 360 degree understanding of

how effective a PR campaign is. It also

takes into consideration social media

and online coverage which goes

beyond the pure vanity of the number

of Likes and Followers (which can be

unscrupulously bought) and focuses

more on actions such as engagement.

Importance of

Evaluation

A- Line:123 B - Line:40-44 C- Line: 82,83 D-80-81 E- Line: 32-33 F - Line: 50

A- Yes B- It is very important, prove our existence, also CEO's do not have time to read a 140 page report, they need a simple dashboard with the pros and cons to make their decision. C- If you are sure what success looks like, it is possible to evaluate effectively. D- : Yes we have very few tools, and because we haven't found, it's called scientific methodology yet, that is cheap and easy to apply. E-Very Important - allows for people to work to their fullest. F - Very

Both interviewees A & B see evaluation as an important aspect, however, agency B

have gone into evaluation in a way t hat allows their clients to see the bottom line, only

knowing the main information.

Interviewee C and D are

discussing the significance between finding out what success looks like and how much clients want to spend

on evaluation.

Interviewee E feels that evaluation makes people work

even harder.

Interviewee F is very upfront about their feelings on the

Tench and Yeomans, 2006:

Shows professionalism and gives accountability.

No further links have been found.

65 | P a g e

20923417

Importance of

Evaluation

G - Line: 31-33 H - Line: 33-37 I - Line: 60,62

G- Quite important, but perhaps its importance is over-stated. The PR industry does very good work and that should not be forgotten just because the means of measuring it are not totally accurate. H - Of course it has to be one of the

most important aspects of a campaign

because you need to be able to see

what type of impact your campaign

initiatives had on the public. Without

evaluation, it would be impossible to

measure return on investment as

well. Not to mention, the client needs

to see proof that you are doing

something right with their campaign.

I- Incredibly important. Without

measuring success, you can’t justify

spend. That said, some things are

more difficult to evaluate than others.

For example, measuring people’s

warmth towards a brand (beyond £) is

difficult, although not immeasurable.

subject.

Interviewee G feels that evaluation is important, but there are more important

tasks in PR.

Interviewee H feels that it is essential to understand what

the client needs. This will show in the understanding of the impact on the public and

clients.

Interviewee I feels evaluation is very important, as it allows the PRO to measure success.

However, things are more difficult to evaluate than

others, such as reputation change and intangible factors.

66 | P a g e

Understanding

of

Evaluation

A- Line: 128-129 B- Line:47 C- Line: 205-206, 211-214 D - Line: 125-136 E: Line - 36-37 F - Line: 53-54 G - Line: 36,37 H - Line:40,42

A- Yes, if you have an understanding of you, you will do it better, as you know why it needs to be done. B- If you don't understand something, you won't do it well. C- If you don't know what you're doing you are going blind. D- It depends on how good their degree is, experience counts as some journalists might be negative, or not as negative as normal, making the coverage good, but new PRO's may not understand that. E - If PROs understand their client's needs they can provide a better all round service. F- Results that show better what PR has done to move the needle for the business? Yes. G - I suspect that this may not be the case, and that good PR skills are not linked to evaluation.

H - Well, if you are a professional, you

must know what evaluation your

work is a must. For example, if a pro

All interviewees agree that if a person doesn't understand

why evaluation is being done, they won't be able to do it

properly.

It is down to agency to teach their staff how to evaluate properly, and look at the

trends that are given from the journalists, in terms of their

language and tone.

Interviewee E has discussed the industry as a whole as it is believed that the industry

can work better with a greater understanding.

Interviewee F has stated that

if everyone understands evaluation, everyone can

work to a better goal. Interviewee G feel that PRO's should not need to be good at

evaluation.

Interviewee H states that if a PRO is professional, it will be

possible to understand

‘A real problem is the PR industry thinks its

job is to get media coverage or run a good

event – that’s not the case. The PR industry’s

job is to influence and change behaviour of

consumers and stakeholders.’

(PR Week, 2012).

67 | P a g e

20923417

Understanding

of

Evaluation

I - Line: 65,65

doesn’t evaluate their process, they

will not know what worked and what

did not work.

I - Yes. Evaluation isn’t an end-process

that you do post-campaign. It should

be an ongoing process that is central

to the execution of PR strategy. PROs

that get this, will make sure that

everything they do has a positive

impact on evaluation.

evaluation and the process. If the PRO is not professional, they will not understand the

pros and cons of it.

Interviewee I feels that evaluation should be the most important process, in PR, as it

helps them show that everything they do has a

positive impact to the company.

Various

Media

Evaluations

A- Line: 143-147 B- Line: 50-51

C-Line: 227

D- 139-141

A- Same way, debate on how much AVE should be worth, depends on unique visitors which changes monthly. B- No, as with some clients, they like to see where all their links have come from, so you need one form of evaluation to make sense of it all. C- Yes I'd agree that different medias have to be evaluated in different ways. D- Different media's give different things, outcomes may be the same but they have to monitored separately.

Interviewees A and B recommend it is best to

evaluate PR with one methodology for consistency.

However, interviewees C and D take opposing views, with them recommending that different media's require

different methodologies as they are achieving different

tasks.

Interviewee E claims that they

can measure click throughs

During the transition to the digital world, it

has been found that word of mouth, or

interpersonal influence has become more

common with the use of social media,

allowing the public to talk to their peers

about a certain product or company easily,

from almost any location worldwide.

(Watson and Noble, 2007, P14)

68 | P a g e

Various

Media

Evaluations

E: Line - 41-

42

F - Line 57-58

G - Line: 40,4

H - Line:45-47

I - Line: 70-73

E - Measure click throughs but only some clients have this technology to do so. F - Best practice should not allow for changes, even though metrics change. G - I think evaluation methods do differ for different media. H - No, because we mainly

concentrate on digital marketing and

PR and most of our impressions come

digitally. Unless a client asks for a

traditional PR route for their

campaign.

I- For clients that have an online-

centric model, it is easier to show

impact of PR through clickable links in

coverage that can be measured on a

source by source basis. These clients

seem to be more switched on to

other forms of quantitative evaluation

(SEO, PPC, email newsletters, etc.)

online but only with certain clients.

Interviewee F feels that PRO's

should measure everything the same way.

Interviewee G feels that

evaluation already does differ.

Interviewee H feels that as they concentrate on online PR they use one method, unless

clients ask for their evaluation to be done in a traditional

manner.

Interviewee I feels that clients with online campaigns can obtain accurate and easy

measurements, as there are more methods to measure

this accurately.

69 | P a g e

20923417

In house and agency differences

A- Line: 169-170, 172 B- Line: 54-56 C- Lines: 244-253 D- Lines: 155-156 E- Line: 46 F - 61-65 G - Line: 44,45 H - Line: 50-56

A- Still have to justify existence, but more defined targets, with more openness. B- Yes, we have to make ourselves look good to clients and our bosses, In house teams have to make themselves look viable to their bosses. C- Massive Difference, marketers have 20 things to do a day and PR is just one of these things, we make their lives easier. D- Simply - agencies look for outputs, In house looks for outcomes. E - Only worked in agency F - No clear distinction - good and bad in both. G - I think that agencies ‘own’ evaluation and that in house teams leave it to them to do most the evaluation work.

H- I work in an agency type of

environment, so I am used to dealing

All interviewees agree with

the question, with the general consensus being that in house teams still take up space and

need to prove their worthiness in the

organisation.

However, due to the cultural difference, it is possible to

assume that in house teams undertake a more intensive

approach to PR, in terms that the work that they achieve will be of a higher standard and specialism compared to

agencies.

Interviewee F feels that both in house teams and agencies would encourage businesses

by measuring accurately, however, some still try to impress with big figures.

Interviewee G feels that in

house teams do not evaluate, but it is only agencies that do

so.

Anne Gregory, 2010, states that if PRO's are too open about evaluation, clients may feel it is easy and then take the

account in house and do it themselves.

70 | P a g e

In house and agency differences

I - Line: 76-78

with a few clients at a time. It’s

important to discuss and understand

what the client wants and execute

what they need. These are clients

who demand results and rely on us to

make a PR buzz in any industry they

are a part of. We survey the client, we

measure the “buzz” digitally, and

show the clients what worked and

what didn’t work based off of our PR

measurements. I feel like in house PR

also use evaluations to show profit to

their seniors. That would be the

difference.

I - In-house teams tend to have more

pressure on delivering comparable

pound for pound ROIs on different

marketing disciplines (SEO, Direct

Marketing, advertising, PR, etc.)

whereas agencies will focus on

delivering excellent PR results.

Interviewee H feels that agencies need to explain

evaluation to clients, and for PRO's to understand what

clients want from evaluation.

Interviewee I feels that agencies have more pressure

to prove every pound they spent works, compared to

agencies.

International

Evaluation

Culture

A- Line: 183-186 B- Line:59-61

A- May use the same idea's but in a different method, using different multipliers, but culture can come into it. B- Yes, as different media's have

Interviewees A, C and D all agree that local culture and ethics affect the way that

evaluation takes place.

As these two agencies became international very quickly, there was an urgent need to

send evaluation reports back to the US headquarters for CEO’s to see what was going

71 | P a g e

20923417

International

Evaluation

Culture

C- Lines: 276-278 D-Lines: 168-169, 172-174 E - Line: 49-50 F - Line: 68-70 G - Line - 48,49

H - Line: 59-62

different values that we may not have here, and if you insult a colleague abroad by saying this they won't be happy. C- Some cultures are more laid back so they don't do it on time, but the methodology is generally the same around the world. D- If the media allows it they will use one, if you want to compare two different countries, you need to be using the same yardstick. E - Only one client US based, use similar methods. F - It is important to use one method however, you have to allow for the local market adjustments. G - I think it depends from one company to another, but there will be some instances of international companies using one method across the world.

H - I definitely think that local culture

and ethics make a huge difference.

There may be agencies who doesn’t

properly measure their PR results,

and in result, they may continue

However, interviewee C is part of a large network which

makes partner agencies use their methods of evaluation

when it comes to work sharing.

Interviewee E uses the same methods for all clients,

regardless.

Interviewee F feels that the framework should remain the

same, however, with local culture coming into play.

Interviewee G feels that one method should be used, but

there is room for other methods if needs be.

Interviewee H feels that local

culture does make a difference, and agencies

which do not account for this, is making a mistake over and

on in their other offices (Watson, 2012:11)

Nothing obvious to promote the cultural differences.

72 | P a g e

International

Evaluation

Culture

I - Line:81-83

making a mistake over and

over again with different clients.

I - Global media trends and localised

news will affect the effectiveness of

PR but general universal PR

evaluation methods should be applied

to all campaigns (see above:

Barcelona Principles)

over again.

Interviewee I feels that one method should be used internationally, as global

media trends, however, the Barcelona Principles should be

considered.

Client

Contract

Renewals

A- Line: 199-203 B- Line: 63 C- Lines:281-286 D-Lines: 185-190

A- Best form of confirmation that things are going well and they value the work that has been carried out. B- Yes! Nothing is better than a good pat on the back for your work.

C - Definitely, A pat on the back is the best form of evaluation, it means your work is not going unnoticed. D- Client doesn't need to understand

everything, that is why they hire an

agency, as long as the agency is

making money for the client they are

happy.

All interviewees agree that

this is the best form of

evaluation, however, it still

does not answer the

questions, as some clients

may change their agency to

get a fresh perspective on

their business.

Interviewee E feels that

Tom Watson found from a 1994 survey, that

many managers believe the best form of

evaluation is when clients renew contracts.

(Watson, 1004)

73 | P a g e

20923417

Client

Contract

Renewals

E- Lines: 53-56 F - Lines: 72-79 G- Line: 51 H - Line: 64 I -Line: 85

E- Best form as owners see the

results, changing beliefs is more

important though.

F - Yes, also when the client expands

their network with the PRO, It doesn't

mean you are necessarily evaluating

properly or affecting the target

audience. Chemistry could come into

play.

G - Probably not. No!

H - Well, they renewed the contract!

So that should say it all.

I - No, but it always helps

changing beliefs is more

important

Interviewee F feels that the

PRO can keep contracts by

having a good relationship

and understanding with the

client, and not always about

finances.

Interviewee G feels that the

clients renewals would not be

the best form of

measurement.

Interviewee H agrees with this

comment.

Interviewee I feels that client

renewals are good for

business, but not a good

source of evaluation.

74 | P a g e

Appendix 1.4 - Sample The research sample for this study, was selected from PR agencies and in house teams,

which have a population of around 9500 (Carey B, 2013) CIPR members, who will be based

around the United Kingdom, as well as those from America, who were able to provide

international insights, the study also contacted members of trade bodies and evaluators.

This sample was created, by using the non-probabilistic, purposeful sampling method, as it

allowed for the researcher, to select who to talk to. This study spoke to in house and agency

PRO’s, who are CIPR members, and located in the South East of England. However, it

excluded free-lance PRO’s, as they may not have the funding or skill set to use more

accurate models.

The sample also contained academics, who were teaching Public Relations at

Buckinghamshire New University at the time of the study.

Non Probability - This is where the respondents are chosen without the need of working out

a sample size, for example, establishing to choose every 3rd male on the street. This is

because the study will need respondents, who are from a specific sector, regardless of age,

gender, or level of career (Maxfield MG, Babbie E, 2011).

Purposeful Sampling Method - This is where the population of the sample, have been

chosen from specific target, in order to gain a purposeful and meaningful result (Merriam, S,

2009).

The limitations from this primary research were, that due to the allocated time scale, not all

sizes of PR agencies and in house teams, could be interviewed, also, as some PRO’s may not

have been available to answer questions, the interviewer, had to search for new

respondents, the final limitation, was that the academics interviewed, will only be from

Buckinghamshire New University.

75 | P a g e

20923417

Appendix 1.5 - Managerial Recommendations

The managerial recommendations, which are made from this dissertation, are

To educate PRO's on the importance to set goals and objectives with clients at the

start of every campaign, or client meeting;

To allow for PRO's working abroad to evaluate Public Relations in a manner which

allows for local media culture and landscape to be accounted for;

To monitor how many hours are being spent on evaluation, to ensure that clients are

not overspending on evaluation; and

To put the digital media evaluation process from this dissertation to be put into

practice, as this is a low cost method, which can be trailed risk free.

76 | P a g e

Appendix 2.1 - Transcript - Interviewee Coding A (two

respondents) R: This is the recording (erm) at 2.15 on the 20th March (erm) for the dissertation: 'Which evaluation

method is the most accurate in the PR industry. (erm) Respondent 1 and respondent 2. Do you agree

to be recorded? Respondent 1?

1. Yes

R: Respondent 2

2. Yes

R: Ok, firstly we will start with (erm), what method of evaluation does your company use -

(erm)respondent 1?

1.AVE - Advertising Value equivalent

R: Ok and respondent 2 works for the same company so it's going to be the same?

2. Yes

R: Respondent 2, do you feel this method requires goals and objectives to be set at the start of the

campaign ?

2. (pause) Yes

R:Ok, do you feel that you can measure against the during the campaign as well?

2. (erm)Yes (pause) because all publications, depending on their position of the magazine and things

varies, so if we know what we are aiming for.

1. Yes definitely, although, I have to say, it is something we should do more of, the danger (pause) is

that if you set targets of advertising standard, advertising value equivalent, at the beginning of a

campaign, you might tend to be slightly more realistic and achieve a lot more than you target

yourself to do, because you are worried of falling short.

R: Ok

1. So I think it is a good thing to put in as an objective and KPIs, it could be, it could backfire on the

client, so we don't always do it. Unless we have a very reluctant client and we want to convince

them to have more confidence.

R: Ok, and secondly, does (erm) your evaluation process run from the start of the campaign to the

end?

1 & 2: Yes (Simultaneously)

R: It does, ok, Say (pause) you send out a campaign to a journalist of group or journalists, it goes into

the public domain, and it doesn't (erm) react in the same way that you've hoped, and its getting bad

77 | P a g e

20923417

evaluation reports and advertising value equivalent figures, have you ever had to recall a campaign

because of this.

1: No, I think because we are working with mainly product PR and not opinion PR, it's difficult to get

bad PR.

2: Its generally always positive

1: Yeah, and if a particular product has been used a press release or campaign, doesn't work very

well, we will replace it with something else , but we wouldn't recall it.

R: Respondent 2, do you feel that the mass media (erm) has an impact on the way PR is seen in the

clients eyes, do you feel, that its, seen that, the mass media makes PR a viable medium?

2: As in the reputation of PR?

R: Yes

2. (pause) You do get a lot a lot of dubious people when it comes to PR, but I think it depends what

sort of industry you are in, because, when it comes to people, celebrities and things, it is quite a

negative thing, politics as well. But (erm), when it comes to products and brands, I think it's generally

positive.

R: And yourself (To 1)

1: Would you mind repeating the question again, it was quite a long one wasn't it.

R: Yeah, the (pause) mass media, do you feel that it has an impact on the way that see PR and how it

relates to the business objectives as such?

1: Right so because there is so much media?

R: Yes

1: Because there is so much, some clients would want to be everywhere, which is unrealistic, and

sods law is that they will want to be in the publications that you haven't managed to get them into.

R: Yeah

1: So I think it can detrimental actually, in some countries there is nothing like the number of

publications, and I think it's something easier there. Having said that, it does give us better scope.

R: Yeah (erm), and also, going back to you saying the clients want to be where you can't get them at

times, do you feel that clients are not disappointed with the results they get, but (pause), they're not

as happy as they would be?

1: if they are not in certain publications?

R: Yeah

1: Well I think most clients have a list of publications they want to be in, and that is part of our key

performance indicators or brief at the beginning of the campaign. So we would try very hard to get

78 | P a g e

them into them, so they're not disappointed. However (pause), nature of the PR consultants, is that

you are good at spin, you can always spin the value of what, and perhaps change their mind and turn

their disappointment into contentment.

R: Ok, and also do you feel that if your client is in a certain regional of the country and you can get

them into that Regional publications, do you feel that perhaps it's worth more than being in an

national paper?

1: It depends how they sell, if they sell online, it is worth less, if they have a local business (2: Shops),

yeah, then it is obviously worth, yes I think it is worth not more, but equal.

R: Equal, ok, (erm) and also as the agency do you have one unified evaluation method for all of your

clients, or does it vary from client to client?

2: We have a method as a rule, occasionally there is room for manoeuvre, but generally (erm) it is

three times the AVE.

R: Have any clients ever asked you to edit or alter the method.

1: No

2: No

R: No, ok

1: Some are a little bit cynical about it and ask us to justify it.

R: Yeah,

1: Especially in the current climate, when advertising rates are heavily discounted, so you have,

before you start on this, you have to say, of course, we know no one pays rate card at the moment,

so the advertising equivalent, is you know perhaps a little bit inflated, however, you know, we use it.

R: (erm) Do you agree, that some PR officers, use evaluation reports and evaluation figures, to

impress clients?

1: Yes!

2: Yeah

1: Because it is one of the few tools you have really.

2: It is a good way of proving value of PR, although it is an estimate (pause, erm), you know,

everyone will see facts and figures, especially business owners, because they're all sales minded, but

it's also a good back up.

1: And quite often they will advertise as well, so it's a currency they understand.

R: (pause) If you could change the way that you measure PR, how would you, what would you do

and how would you do it?

79 | P a g e

20923417

1: Well, I'd love to see it change really, because I do feel that this is quite an old fashioned way of

doing it, but I can't see how you could change it. Ideally, you could, you know, something that is set

up, computerised perhaps, to say this is, your client has this bit of coverage and it is so this on the

page, or it's only one of three, but your client has a web address and the others don't or your client

has something positive said about it and the others don't or whatever, so the value then is estimated

on what that is worth, to the client (pause) and not just in what it would have cost to have bought

that space. Because it is catch all, and I don't think it really works, but nobody has come up with

anything better, especially for the type of PR that we do, which is product PR (pause) and I think it

would be great if somebody could come up with some software or whatever.

2:Because it would get incredibly complicated, because you would have to have a value for an

image, a value for a text and a value for the website, stockist details.

1: If it's positive or negative, if its glowing, if it's an endorsement or if it is just stating facts.

2:And its position in the newspaper or magazine.

R: Ok, so you feel almost opinion polls, almost swaying towards that side, the opinion of the

journalists how they write resonates, that sort of thing would impact?

1: Oh I see, I think that would help, again because of the type of PR that we do, nearly it's all

positive, because by default the journalists are endorsing it, because they are using in the feature, its

great if they say something really positive about the client.

R: Yeah.

1: Specifically, which doesn't happen very often, if it does then that coverage is definitely worth a lot

more than three times the advertising equivalent. But because the coverage is analysed and picked

up by computer, (erm) it is very difficult to do anything more, and we could o it ourselves but we

are not independent or unbiased it would be quite difficult to do.

R: Yeah, ok so (erm), from what you've been saying, I understand that evaluation is very important

to both of you.

Both: Yes

R: Do you feel that (pause), as you understand evaluation quite well, you provide better and more

accurate results, compared to a colleague who doesn't have so much of an understanding, so like a

someone who is new in the industry maybe or someone who is naive and does not want to

understand it.

2: Yes, I think you need to have a good understanding of why it is worth what it is and its, you know

(pause) how you work it out, the technical aspect, is important and also the reasoning behind it,

because (pause) if they understand why they're doing it, then they are going to read it in a different

way, because you have always got to (pause) take different things into account, like it might only be

a 3cm mention, however, it's part of a big feature and it's the feature as a whole that has the power.

1: Definitely, I think also, it's great for trainees and graduates to look at coverage (pause), not only

do you learn to analyse the evaluation, but you see the circulation of a title, so you see that

80 | P a g e

something that has been in the Sunday Times Style Magazine is going to be worth more than

something that has been in the Daily Express or whatever, so you suddenly get a much clearer view

and a more perspective view of the current media landscape and if the circulations have gone down

over the last few years and they fluctuate quite a lot, so I think it's a really good thing for trainees to

do, as long as they are supervised, as they learn a lot about the industry as well.

R: Ok, (erm) and also (erm), When you are working with clients who gain coverage online and

offline, (erm, pause) do you measure it in a different way, are there any other ways you measure

online PR?

1: There isn't at the moment is there?

2: No, we always (erm), have a little debate when we have online PR, because it is difficult, I mean

you have your advantages because you have a direct link to the website, so people are directed to

the client, (erm) and you have a mass audience (pause), but then are they actually just clicking on it

and not reading it, you don't know. So it's sort (erm) of a difficult one.

R: Yes, that's almost similar to traditional PR, where if someone has bought a newspaper, it doesn't

necessarily mean they are going to read the article, so you would say it's the same issue really with

online. It's harder to control?

1: It's harder to get the data isn't it, with the Daily Mail, you know that their circulation is 1.9million

or whatever, but if it is on I don't know, House to Home or whatever, you know what their unique

visitors monthly, what do you gauge it on, is it, unique users, visitors, is it, how do you measure it,

you know it's very difficult. Is it unique number of click through's, you know you can't get that

information. So it really is a difficult one I think,(erm)and also our cuttings agency don't give us the

value of online coverage.

2: No they don't seem to value online or give you any details, so I think with online it can fluctuate so

quickly.

1: Yeah

2: So one month could be 1000's higher than another month

1:Yeah, with national newspapers and magazines, you have the National Audit Circulation, which

audits the national newspapers and magazines and whatever, so that you know exactly how many

people should be buying that newspaper or magazine at any one time. So you have a much more

accurate figure, and with online you don't have that. Somebody needs to work on this I think! It

would be great!

R: Ok, nearly done now, do you feel that there is a difference in the culture of evaluation between

agency and in house PR teams?

1: I've never worked in house, they probably, could say it's quite similar Suraj, because they have to

justify their existence, in the same way as external agencies do . They are probably given quite more

defined targets though.

R: Yeah

81 | P a g e

20923417

1: Because they can be much more open about their business plans and objectives than they can be

to perhaps an outsider, maybe they put a lot more value if their target is to get into The Sunday

Times Style, if they achieve that then maybe the value is inflated, if they get into the Hemel

Hempsted Gazette, that doesn't have the same impact, so I don't know is the long winded answer to

that, it would be interesting to find out.

R: Yeah

2: I've never worked in house either

R: You've never worked in house, ok, this one might be a little bit tricky to answer as well,

International PR agencies, do you feel that they evaluate using the same method in all of their

offices around the world, or do you feel local culture comes into it, I mean culture comes into

everything in business really.

1: I'm sure it must do! I'm sure it must do, and they might have an advertising, they might use the

advertising value equivalent, and might be 5:1 or 10:1 or 2:1 other than ours which is 3:1. We use

which has been around for ages (pause), so they probably do, I'm sure they do, but it's probably not

the same formula.

R: Have you ever worked with any clients who are offer your PR services to, to market their products

outside the UK?

1: Yes, we are at the moment actually.

R: Ok, and how are you planning on evaluating that, is it the same way or??

1: I think it will be the same way, because it is the way we work, so I think it's the way we

understand and the way the client understands.

R: Ok (erm), and finally, bit of a cheeky question this, but do you feel that the best form of

evaluation for a PR agency is when the client renews the contract?

1:Sorry?

R: When the client renews their contract, do you feel that this is the best form of evaluation,

because they come back to you?

1: Oh I see.

2: It's the best form of confirmation (pause, erm), to say well done you've done a good job,

continue!

1: (pause)Yeah, I think they've valued your work and your worth and they've said yes its working so

we will renew, (pause, erm) I think we are reporting back all the time to our clients its almost an

ongoing confirmation actually.

2: And ultimately our evaluation results and our coverage, and everything that works towards

getting them to confirm it.

82 | P a g e

R: Yeah

1: One thing you could perhaps think about Suraj in all of this is what targets for evaluation there is,

you asked us that, there is sort of an industry standard of return on investment, for every pound you

spend you get I think it's nine at the moment, eight or nine.

R: Ok

1: Back, and its only setting targets at the beginning, so rather than saying we want to get £500 or

£50,000 of coverage, what we want to say is we want an investment of 20:1 or something like that

and you can really work towards that and keep a check on it every month.

R: Yeah

1: And the other thing that we don't use very often, but I know other people do, is the opportunity

to see target.

R: Ok

1: Rather than just the financial (pause) value, so I would like this campaign to be seen by 3.5 million

people and you know that might be possible within a couple of months or whatever (erm). So that is

always very useful, but we don't always use it in our reporting, we are beginning to use it more

actually. So it's not financial but it's another return.

2: Return on income is something

1: Return on investment

2: Return on investments is something that the clients really listen to as well, as it's

R: It's hard facts and figures, you're talking their language.

2: And you're saying don't worry, we know what you've spent (pause) and this is what you're getting

for it, so they know it's worth it.

R: Yeah

2:So they know it's worth 10 times, or 20 times or even 30 times on what they've spent with us.

R: Obviously you have clients who would potentially have internal marketing manager or internal PR

manager, who has to report to their CEO's and CFO's, (pause) so would you have a hard time

justifying (erm) your worthiness to them, so they can go back and really sound positive about it so

they don't have their budgets cut?

1: We've got two or three clients like that haven't we.

2: Yeah, it's sort of just an (pause) age old thing with business isn't it, you always have to justify.

1: They have to justify their position, in the main they want to use a PR company, because, they

don't really have the time or inclination to do it themselves, and the danger in the recession is that

clients think ok we have so and so in our marketing department, they can do the PR, its really easy,

83 | P a g e

20923417

but they don't have any tools or resources that we have, so yeah I think in the main they are happy

to use us. So they try and positively (pause), what's the word, sort of sell us in I suppose, I think they

are on our side.

R: Because you're making their lives easier?

1: Yeah

2: And PR is a valuable part of a marketing plan

1: Its essential, (pause) it's the only important part.

R: Ok, that's great! Thank you for your time (pause), I don't know if you want to add anything else?

1: It's a tricky but really interesting subject.

R: Yea, it is very, because there are so many internal agency methods, so it's difficult to find out

which one is going to be the most accurate.

1: Ours is a bit different because it is product PR, but I don't know if you were doing Hoover or

Panasonic or something, which I know are products as well, there is good and bad press.

2: It's such a huge brand that people talk about them in general and their products .

1: it might be quite interesting to talk to, I don't know if it's possible, but someone who does PR for a

political party.

R: Yes, I am actually speaking to some people , another agency who deal with public affairs .

1: Excellent!

R: I am talking to them

1: That would be a good way.

R: Definitely, because they evaluate it very differently.

1: There is obviously loads of coverage they put no value on at all because it is negative

R: Yes , yeah

1: Is that ok?

R: Yeah, that is great, thank you for your time and thank you for your insights.

1: Thank you

84 | P a g e

Appendix 2.2 - Transcript - Interviewee Coding B 1. What method of evaluation does your company use?

My organisation does not use a single type of evaluation, it is dependent on the client's

needs, it is possible to measure what the clients, not what we want.

1a. Does this method require goals and objectives to be set at the start of the campaign?

Yes! If you don't set targets, you could be measuring anything, just because we got 250

pieces of coverage, what does it mean in a business sense?

1b. Does the evaluation process run from the start of the campaign to the end?

We begin our evaluation process in around the first month until the campaign ends.

1c. Have you ever had to re call a campaign as measuring it when it was live gave a negative

result?

No, we would add to it, but that is only if the content is not suitable.

2. Do you feel that the Mass Media or the large choice of media has an effect on the way that

PR is seen in the business place?

There is not really such a thing as mass media in the UK anymore, compared to most other

countries including Eastern European Countries, China, Russia, India, USA as well as many

others have much more choice of media outlets compared to ours.

3. Does your company use one unified method of evaluation for all clients?

Again - no, our evaluation model depends on the campaign and what the client is looking

for... There is no silver cloud when it comes to this, and this is where many agencies go

wrong, as they are trying so hard to justify their existence figures they give may not mean

anything to the client.

4. Do you agree that some PRO's increase the value of their evaluation reports to impress

clients?

Of course. However, many PRO's have been found out for doing this.

5. If you could change your company's method of evaluation what would you do and why?

I don't think we can change them, they can be improved and that is by putting more than

one method together to make results stronger. Also, it is important not to over spend on

evaluation, many clients want figures, but they don't realise it can be costly and effective, or

cheap and effective, in this case cost doesn't really impact results. I feel the average spend

for evaluation should not exceed 15% of the overall budget, where as many cases are over

40%!

The way we tend to work is by sitting down with a client, and establishing which 50 print

titles are most important to them and why. By doing this, we can then establish which ones

are the most influential, this means that other publications will follow what they say,

therefore there is only the need to evaluate these top 50 titles.

6. How important do you feel evaluation is?

It is very important, it is there as we do need to prove our existence to clients, but it has to

be done realistically. It is important as many CEO's don't want a 140 page report on how

85 | P a g e

20923417

campaigns have been, they want a simple dashboard where they can see what it is that is

going on, with key facts and figures saying what went well and what didn't.

6a. Do you feel that PRO's who understand evaluation get more accurate results than their

colleagues who don't have such an understanding?

Yes - if you don't know what you're doing, you won't do it very well.

7. When working with clients who gain coverage on various media's away from print does your

evaluation model differ?

No, as with some campaigns, such as online, you can see where click through's come from.

8. Do you feel there is a difference between the culture of in house and agency evaluation?

There is, we have to report to our bosses and clients, who have to report to their bosses, so

we have to help them to look good rather than making us look good. IN House teams need

to make themselves looks financially viable to their bosses.

9. Do international companies use one method across the world, or does local culture and

ethics make a difference?

Yes - for example, in China, where press release reprints are worth a lot , here they are not,

and if you tell a Chinese PRO this, they will be very offended as it is their system.

10. Is the best form of evaluation when the client renews the contract?

Yes! Nothing is better than a pat on the back for good work.

86 | P a g e

Appendix 2.3 - Transcript - Interviewee Coding C

R: Ok, this is the recording for (erm) the evaluation in public relations dissertation, (erm) it's the 5th

April at 11 o clock, (erm) firstly (pause), do you give your consent to be recorded.

1: Absolutely

R: Ok that's great! (erm) firstly if we begin with (erm) what method of evaluation your company

uses.

1: Ok, so the reality is, we don't just use on form of evaluation, we've got a variety of clients, from

B2B, to tech, and consumer, so they way that say Go Ape Tree Top Adventure would evaluate their

PR is very very different to Information security clients Kaspersky, if that makes sense. (Pause) What

I've done, I've listed down a couple of the most popular, and what we do, is we sit down with the

client, either when we're pitching or we've won the business and say, there isn't one size fits all, but

we'll work to whatever, methodology suits your business.

R: Yeah

1: We use (pause) quite frequently, opportunity to see, so OTS, looking at the circulations, the

amount of online traffic and stuff like that.

R: Ok

1: We use AVE, which I'm sure you've come across

R: Yes

1: Which is now being see as quite a dated methodology, but it is still a way that some clients like to

work, they feel familiar with it, they understand it, we (erm) frequently use cost per 1000

R: OK

1: So we will look at the actual spend that (erm) a client is spending, and then look at the amount of

people that they've managed to get in front of, then you can start wheeling it down to every 1000 it

costs 23p or something like that. That (pause) proves reasonably popular, because (erm) it means

that the client can go back and actually give a bit of ROI to their board, if that makes sense.

R: Yeah

1: A lot of them, just simply want to go by volume, volume of coverage in their (pause, erm) tiered

titles. So when we start the campaign we create a press list or the (erm) online titles that we want to

go in for, and they will agree a goals list. So these are like number one this is where we are aspiring

to be, so some clients might say, we want to get five pieces of coverage in that every quarter, we

expect 20 in the next layer, and 30 in the layer below that.

R: Ok

87 | P a g e

20923417

1: So, (erm) again targets are agreed, it's important they get, likewise you know, and we're not just

talking, say just like the, (pause, erm) biggies, we want to be on the BBC website, or broadcast or

anything like that. For some clients it's just really really important that they are in their tier, sorry,

their trade publication.

R: Yeah

1: Every single issue.

R: I suppose that that would be worth more to them.

1: It's worth more to them, it gets in front of all of their customers, if they're not there, then that's

when they see an actual dip in the amount of phone calls they have.

R: Ok

1: Increasingly we're seeing traffic, as a massive thing, so people say, I don't actually care what

pieces of coverage you get me, or where it is, all I'm interested in is the amount of traffic you can

drive to my site or you can drive to this specific page. Because that is giving me leads.

R: Ok

1: We quite like that, because, it starts making it a bit more accountable, (pause) you can sit there

and say, you spent X amount on PR, and we can actually prove that PR has driven X amount of leads,

because using Google Analytics, they will be able to see where the links have come from, and they

will be able to link them to the coverage that we've got, And hopefully we will also be able to see

how many of those leads have been converted, so you can actually start to get a price for the leads

you're getting.

R: Sure

1: Then there are two paid for methodologies that we, some of our clients us, or that we put

forward, one is Apollo (erm) and they measure share of voice, so, you can go onto Apollo and

(pause) and they give you a free overview of the top 100 tech companies in the world, and it gives

you a flavour of, ok if say we had, I'll use Kaspersky as a good example, and they actually use this, so

the antivirus market is quite busy, there's Symantec, McAfee, Sophos, there's a wide variety of them,

when we started working with them, they were a distant thif, in terms of the share of voice every

month. There were a lot of other companies making a lot more noise, they were just another one,

not really getting their voice out there, they are now number two or number one, two years later,

and that is something that they are very proud of And that is something that is very important to

them. That they are seen out there as very visible, it helps with their thought leadership strategy.

The other one is Metrica, Metrica is (erm) extremely expensive, but it is (erm) run by a company that

does clicking analysis, and they basically take all of the coverage and they analyse it all on a variety

of different things, key messages that are in there, who has been spoken to, who's the

spokesperson, imagery, all of this stuff, and the figure comes from there. It is quite pricey.

R: Sure

88 | P a g e

1: The reality is, what method do we use, we use a lot, there isn't one size fits all, you do what suits

the client.

R: Yes, so, I mean, as you were just explaining, I gather that a lot of your methods do require goals to

be set at the start, so to drive traffic, you want X amount of people to visit this page or this website

in the period of time.

1: Absolutely

R: Do you feel that that is the most important aspect (erm) when it comes to measuring?

1: I think it's not just measuring, it's actually one of the most fundamental key parts of any PR

campaign, whether it's traditional or using social media, or if you're doing digital such as video, any

project or campaign you're going to do if you just start working and you haven't sat down and

thought why are we actually doing this, what are we wanting to achieve, and what will success look

like, you haven't actually got that, then you're shooting yourself in the foot.

R: Yes

1: You have to be a 100% clear what success is going to look like, then you can come up with the

evaluation process to actually fit that. The reality is that some clients sit down and say yeah, we're

really interested in, (pause) seeing the volume of our coverage go up over the next three quarters,

but if you get to the end of those 3 quarters and the volume has gone up but sales have not gone up,

and that's what they're actually really being judged on, that's what they really what success is going

to be measured against, you've actually failed. So it's really important as PR agency, to sit down and

actually really dig deep and find out what success is going to look like, and how we can actually help

them with that.

R: (erm) Next question was (pause) sort of about, a campaign that might have gone wrong, once

you've sent it out, I don't know if this has ever happened to you in your career before.

1: Less than its gone wrong, a good PR agency is always evaluating, so you don't wait until the end.

And again, this is a (pause) key part to that first conversation at the start, about understanding what

success is going to look like, you have to be flexible enough to be say, a month into a campaign, to

be able to say what we are doing isn't going to achieve what we need to do, what we need to do is

to address this, we need to change, if you don't have that flexibility, you are going to fail and the

client isn't going to stay with you, it's as simple as that really. So yeah, I've got (pause) I can't give

you specific examples, because the client won't be happy about me sharing that. But yeah we have

had examples of, you know we've had a product launch and we have been working towards a certain

date, and actually we've turned round and said look, this won't work, because (pause) there's

another conference going on, or there is something else, so we're going to have to change the

deliverables on that. So yeah (pause) it often happens, but quite a lot of campaigns change, you

have to think on your feet.

R: Yeah, I suppose public holiday's, that comes into everything, and even holiday's abroad or

something.

89 | P a g e

20923417

1: The big thing is finding out if there is a big thing happening, at the same time, as a major launch.

So you have to look at when exhibitions are happening, you need to look at, not just players in your

market, but major players, if Apple, is having the launch for a new iPhone, on the same day that you

want to launch your product, you're probably not going to get a lot of coverage, because someone

else is making a lot more noise than you, and is a lot more interesting. It might sound harsh, but that

is the truth, sometimes you have to sit down with a client and say, all the press are going to be

going to that, so no one is going to come to your event if you hold it on the same day.

R: Ok, (pause) do you think that, because we have such a large choice of media, especially compared

to other countries, do you think that it has an effect on the way PR is seen by clients, in a sense that,

oh there is so much media, will we get into the right publications. Or do you feel that the PR agency

can combat that?

1: I think that that is a responsibility of the PR agency, to make sure that you're speaking to the right

people, and sending out news scatter gun to everyone and their dog, to try and get everything,

you're not doing the right job, what you actually need to be doing is targeting specific titles, tailoring

your pitches for them, building relationships with them, and you choose those ones based on clients

customers are reading it, or that's something they watch or that is somewhere that they go, so you

need to understand who your clients customers are, and then you can get to the right titles, so the

fact that there is a lot of media in the UK, no that doesn't impact it, in actual fact, yeah, there is a lot

of media, but if you compare, it say Germany, WOW, we are nowhere near them! Nowhere near! So

I run, international campaigns, and that is our biggest challenge, for this specific client, that volume

is and the OTS is a driving factor for them to evaluate their public relations. Germany has five or six

times as many titles, so you're always struggling to, on that front. And if you beat them, wow you've

obviously had a good month! So, I don't think it impacts it, I think there is a big element, that it is on

the shoulders of the PR agency to be targeting the right people. The industry has changed, if you

send out a press release, and send it out to 200 people, the reality is that there is every chance you

will get named and shamed! Journalists are so fed up with that approach, of it being untailored, its

spam, so you are sending something out and it's just no relevance to their publication. There is every

chance that they are going to go to twitter, they are going to email people, and go, what the hell is

this agency doing?!, they are just amateurs, this is the second time they have done this! I will never

deal with them or their clients again! Now if your clients see that, they will not be happy! As you are

losing publications, and they don't want to be represented by someone who is doing shoddy work.

R: I mean, also, (pause) for example, the AVE system, I don't know if your agency uses a multiplier,

on that.

1: We very rarely use the AVE system now, it (pause) you go on to any sort of PR Week, PR

Moments, anything like that, you will see that there a school of thought, that it is out dated, it isn't

answer the problems, it just doesn't the evaluation questions of PR anymore. We really, we sort of

put it forward to clients, but we rarely use it.

R: Do you feel that some PR people, actually increase the value of the coverage, just to impress the

clients?

1: Yeah, I'm sure there are some unscrupulous people, who (pause), yeah I think there is an element

of it, the other part is, actually, you shouldn't be about, impressing yourself to the client, you

90 | P a g e

actually need to understand, (pause) so if it take a step back, so say for us for example, we win an

account, more often than not we will be dealing with a marketing manager, or marketing director,

they actually have to report their board, or a VP of marketing, or something like that. What is

actually more important to us, is not making ourselves look good, it's actually making that

marketing, our contact, at the client look good. So we need to understand what they need to say, to

their bosses if that makes sense. So if you know that the VP is looking solely on volume, if the VP is

looking solely on traffic, or something like that, you are going to make sure, you're not going to

doctor them, you are not going to falsify them, you're going to make sure that, you give the

information to our contact at the client, that is going to help make them look good, and stop them

getting pressure from above as to what the hell is going on with t his PR agency, they are not

delivering? So (pause) I'm sure there are some unscrupulous ones, who have gone out there and

been slightly dodgy with numbers, but you know what, it's going to come back and bite you, for us,

the amount of time, energy and money you put into winning a piece of business, you've, the only

reason you would do is that for a long term relationship, you want to keep that client, two, three or

even longer years. If you go in there and start lying about the amount coverage you got, or the

amount of circulations or the AVE, and you get found out, your gone! To put it into context, the new

business process, a PR agency will very rarely make money out of an account for the first quarter,

and it can impact the second quarter as well, it's only because of the amount of money you put into

winning that account, time and effort, if you take that away from the fees, and in fact in that first

quarter, (pause) you over service it, you're getting up to speed, there is a lot of content, you've got

to hit the ground running, because you've got to be evaluating it in three months time, so you need

some coverage.

R: Ok, I haven't actually thought of that point, so (pause) quite a good thought there, I mean,

(pause), which evaluation method say for yourself, would you recommend out of the ones you

mentioned to a client?

R: (pause), again it depends on their budget, it depends, there really isn't one I would put my hat on,

more often than not, cost per 1000 can be quite good, again, crude but it can give an ROI figure,

which they can give to their boss, and say from this amount of money that you spent on PR, this is

how many people we got in front of, online is very exciting to me, because there you can start

putting a value against PR, you can actually say for your spend, we sent you these links, this many

have been converted, so for this money, you have generated X amount, but there needs to be a level

of sophistication at the client side as well to understand that. A lot of them are starting to get there,

but a lot don't have that yet.

R: With the cost per 1000, do you find that clients ask you, just because for example I'm in the daily

mail today, and they have a circulation of 1.9million or whatever, how many people actually read it?

Do you get that at all?

1: Of course, you are going to get that, I think, that's where you step back and it's important that you

are targeting the right titles, you know, you got to make sure you are in the right publications, and

the other part is they will start to see how effective it was as well from the amount of traffic that is

going to their site, how many people have logged in, how many people have clicked through to the

site, how many people have searched on them based upon something that was in there, more often

than not if you hit the Daily Mail, the next day the client's website will see a massive spike, it will be

91 | P a g e

20923417

the client name and the spokesperson, and you know where it has come from, because of that, so

you can just about justify it.

R: Also, out of any of the systems that your agency uses, if you could change one, how would you

change it and why?

1: (pause) I don't think you can change any of them, I think they are what they are, I think it's less

about changing them, and more about merging a couple of them, so on their own, they don't work

particularly well, but alongside a couple of the others ,then that does work, say, you've got your

opportunity to see, you're measuring the amount of volume you've got, so you know that the vast

majority of your coverage, is in the titles that your customers are reading, and your opportunity to

see in them is X, you can start doing it, but an element of that and an element of digital, I think you

need to have, there isn't one size fits all, so many three or four of them inclined together. Does that

make sense?

R: Yeah, (pause) as a graduate myself, evaluation in PR in our degree is something that is touched

upon, but we don't spend too much time on it, so do you feel that people who don't have a full

understanding of evaluation, tend to not get as accurate results as those who do, because they don't

see the point of it, they don't understand it in that sense as well?

1: Yeah, you're going blind, if you haven't got a way of evaluating what you are doing, you are never

going to know, whether you have achieved success, and I suppose it is the same as doing your

course, if you are not evaluated on that every term, as to how you're doing, or end of each year, you

don't know how much harder the following term or year, to get your grades up, or you can sit there

and say actually, I'm doing alright, my coursework is on track, I can concentrate on improving X over

there, so it's exactly the same in the PR world, if you're not evaluating or evaluating properly, you're

kind of going blind as you're not helping your clients, as you are just taking their money and running,

you're not actually helping to improve it, you should always be striving to improve it, whether that's

the results or the amount of sales they are getting, and it's only by doing that you will have a long

term and successful relationship with them.

R: OK (pause)

1: Evaluation, partially its hasn't been touched on so much, doesn't surprise me enormously, it's

something I think, it's important to talk about it, but the reality is, the methodology for evaluating

changes, it changes yearly, and it is rapidly changing with the impact that digital is having on PR, so

the fact that it is not being in the curriculum, I kind of think it is the right thing, because, once that

curriculum is set, it is out dated, as long as I think people on the courses understand the

importance, it is actually the responsibility of the agencies you go into for work experience or your

first job as a trainee to help you along those roads, and they will show you their different methods to

understand it better if you are doing it.

R: And also you feel that online and offline evaluation of PR should have a mix of all the methods

really, so you can have an understanding of them all. So for examples, cost per 1000 and Google

Analytics, would you agree?

1: Yes, I'd agree with that, I think, obviously, the world is moving to a digital phase, that is where a

lot of reading is being done, it is where a lot of people click through and see stuff, but there still very

92 | P a g e

much is a place for the hard copy, (pause), the vast majority of the news I read is online, because I

like to know it as it's happening, but I still go out and buy Empire magazine, as I like to read big

features about movies and things like that. I mean, for the reality is I am actually going to see a film

or buy a DVD based upon the reviews and the big features I've read, and I still like that in a hard

copy, but I check out the news of what's coming out online, So I'm not alone in that I'm sure that will

change again in the next few years, but I really hope it doesn't ever die off hard copy, because, there

is a place where you have to measure both, and they are slightly different.

R: Ok, I don't know if you have worked in house at all or if you know anyone who has?

1: I haven't but I know a lot of people that have and do, and also at Berkeley, we like to employ

people who have been in house as some point, as well as the people who have come straight from

university, or just worked in house, because it gives us a different perspective.

R: Do you feel that there is a cultural difference in evaluation, as an agency, you have your own

business objective and goals and when reporting to a client they have their own, as well. So do you

feel that although they are in house and may have their own department objectives, that the culture

will be different?

1: There is a massive difference, in that we are a PR agency, we are totally blinkered, its PR and

generating coverage and noise, and helping with crisis management and all of this stuff, that is all we

think about. The reality is, in house, they very rarely, unless it is a massive company have a PR

manger, you are actually reporting to a Marketing manager, and PR is a small mix of what they are

responsible for, so where as it is all day every day for us, it's probably a day's worth of activity a

week, because they also have all of their other stuff which they have to do. So for us it's all

important, and for them it is just about results and it is an ease of life, they want to be seen that

they are improving things to their boss, and also, it is making sure that it doesn't eat up too much of

their time, so yes, there is a massive difference between the two. And the reason that we like to

have people who have worked in house is so that we can understand that.

R: Right sure.

1: But PR people go, oh god why is so and so not answering my calls, they're not sending this

through, how are we going to achieve this? and it is actually because that is one of 20 other things

they have to do that hour, and the other things are more important, at that point in time. Yes it is

the most important thing in our world, but it is just critical at their end.

R: Ok, obviously you deal with international clients, (pause) do you deal with partner agencies in

other parts of the world?

1: Yes, So we have a network, called the Berkeley 360, which we are the lead agency of, that we

have agencies, we just held our summit actually in February, and they came over to Reading, we

have agencies in the Netherlands, Germany, France, Spain, Italy, Nordics, and there are a couple of

other Europeans, I actually can't remember, we also have Singapore, which covers Asia Pacific, we've

also got Australia and America.

R: So you quite a large span network.

93 | P a g e

20923417

1: Those are agencies we share work through, but we also have a certain way of working and we do

work on international clients together.

R: do you feel that when it comes to measuring, when working with other agencies, the local culture

at their end plays a role in how they evaluate their work, or are they open to use the methods you

use here?

1: No they generally, more often than not, open, I think the way. Evaluation methods that are used

in the UK are pretty much used around the world, so more often than not, stuff comes from the US

first, then comes to the UK, so if you are looking at in terms of where new ideas and techniques

come from, that's usually a rule of thumb, but some people might disagree with me, but I believe

that. In terms of evaluation, it's pretty much the same around the world, Its more along the lines of

getting people to do the evaluation and do it on time. It is sometimes that some cultures are a little

more laid back about it. But yeah, the actual methodology is the same.

R: Ok, and finally, a bit of a cheeky question this - do you feel the best form of evaluation is when the

client comes back and says we want another 12 months with you?

1: Absolutely! Absolutely! That is not a cheeky question, it goes back to what I said, you need to

understand what success is going to be like, there is no point of us kicking off a campaign and then

sitting in a board room, six months later, look the volume of coverage, has gone up every month,

and they go yeah but it is all in magazines that have no relevance to us, we've got no sales from this,

your fired! That is what will happen, it is as simple as that, you have to make sure it starts, so yeah

the biggest pat on the back for the work that your doing is that you are not going unnoticed.

R: Ok,

1: Does that all make sense?

R: Yeah, that is great thank you for your time.

1: No Worries and thanks for your questions.

R: It's been great talking to you

1: Cool, perfect.

94 | P a g e

Appendix 2.4 - Transcript - Interviewee Coding D R: Firstly do you give your consent to being recorded?

1: I do

R: Ok, that is great, thank you. (erm) Firstly, what evaluation method did you use when you were in

the industry?

1: Erm, well, in industry, I mainly used more, 2 things, media evaluation, media evaluation, to

monitor media sentiment and volumes and (pause) which way opinion was going, but I also did a lot

of stakeholder opinion research, to see what stakeholders knew what our reputation was and how it

was changing.

R: Right, so sort of opinion polls.

1: It was opinion polls, but very specific commissioned research, because for example, investors,

there was probably never enough, investors to do proper quantitative research, in mass market

opinion polls, so we do them with target research, with almost full population samples .

R: Right, these methods that you were saying, do they require goals and objectives to be set at the

start of the campaign?

1: (erm) Yes, I mean, if we were doing specific campaigns then yes, but very often because these

were very big organisations that I was working with, we would usually have a continuous research

programme and a reputation monitoring programme running, so you would probably see what the

results were within that, so if you did a twice annually, yearly, reputation survey, and if you did

monthly based media evaluation, then you can see how (pause) the media opinion was going, how

sentiment was going and you would fit campaigns into that. We did do a specific campaign

monitoring as well.

R: Ok

1: If I might just say, the role I am doing now, in consulting, much more of what I do is strategic

consulting, so I am much more concerned with outcomes rather than outputs. In other words, has it

affected the business? The continuity of a business, the probability of business growth and so on.

R: Sure, so before you started the consultancy, your evaluation process ran from the start to the end

of a campaign, and continued all the way through.

1: Yeah, I mean, the evaluation process ran all the time, and we would fit campaigns in it, for

example, when I was with Rover Group, for seven years, whatever, nine years, we would launch

usually a couple of products a year, so you would know within the years analysis, how coverage of

that product had gone, and whether the media thought it was good or bad, how that affected the

reputation of the company (pause) and you know, how other things were affected because of that.

But what we didn't do very well in those days, was monitor how that coverage affected sales.

R: Right sure. I mean, this might not be very relevant to your previous career in PR, but have you

ever had to send out a campaign, say for a product launch, which backfired, or didn't work in the

95 | P a g e

20923417

way that you wanted to? So you had to recall the campaign, or you had to edit it while in the public

domain?

1: (erm) I can't recall doing it, necessarily because the campaign failed, sometimes, particularly

since I've become a consultant, I've advised the client to stop doing something, because the original

idea was not a good one. An example of that was an IPO, I was trying to gain awareness of an IPO, so

an offering to the market, (pause), it became very clear when explaining it to the media, that the

media did not think that it was a very good idea. This particular entrepreneurial business. So (pause)

we knew they were not going to get good coverage, so we advised them to stop running the

campaign and re think the offer to the market.

R: Right

1: So, generally, no, in corporate life and in most consultancy work that I've done, I can't recall

recalling a campaign.

R: Sure, Also, we have quite a large choice of media outlets in the UK, and increasingly in other

countries across the world, there's are growing as well, say China, India, Russia, Eastern Europe, they

are very quickly catching up and overtaking us on the number of media outlets they have. Do you

feel that this has an effect in the way that PR is seen in the way business place as there is so much

media to choose from that PR professionals, they should know which publications to target?

1: Yeah (pause), I think, the proliferation of media, particularly social media, or online media let's call

it as well as social media, has raised the profile of PR generally full stop. I think people are more and

more aware now that things that can go wrong, as well as things that can go right, and need people

in organisations or consultancies, that can advise on how best to handle that. So I think overall, the

growth in and proliferation of media (pause) and the globalisation of media as you were saying has

made PR a more important role in an organisation that it was 20 years ago.

R: Sure, I mean, this might not have really affected you, as you were in house, but it is something

that agencies, sometimes do tend to do, to justify their existence, to their clients, that they increase

the value of their outcomes, to make themselves look good. I mean being in house, you would

probably, your core business objectives are the same as the companies, you may have different

departmental ones, but you are part of the same company, therefore potentially you wouldn't have

to do that?

1: I think, in (pause) in house or externally, anybody in any business always tries to make themselves

look better than a perhaps be seen by others. I think in PR we have the dual task, of convincing

people that what we do is important, in managing reputations as opposed to managing media, so we

had that task to do, then we had the task of persuading them that what we do in managing

reputation, can be effective, so yes I think agencies and in house people can be guilty of over

emphasising that and it is something that we do with our enthusiasm to get the message across in

the first place, rather than just pretend to do better than we are, I mean in simple measurement

terms, a lot of people over emphasise media volumes by multiplying by three of five when doing

advertising value equivalents, or simply column inches and weighting column inches. To the extent

that it becomes meaningless to the audience, (pause) I think the answer to your questions is yes, but

sometimes with the right reasons and with the wrong reasons.

96 | P a g e

R: Right sure, I mean do you feel that evaluation is important in PR, because it is one of the very few

tools we have to justify our existence to clients?

1: Yes, exactly that, (pause) and because we haven't found, it's called scientific methodology yet,

that is cheap and easy to apply, then we haven't got that easy route to persuading people, but I do

think that the profession through the leadership of the CIPR and PRCA and other bodies, who are

moving more and more to outcomes, rather than outputs, (erm) we are going to get closer to that

end and getting the management of organisations to appreciate what we are doing is not just trying

to get the biggest amount of coverage, or even the most positive amount of coverage, but trying to

affect the bottom line, I think once we get into those conversations with people, they will start to set

us objectives that will start to match the corporate objectives. (pause) I mean one of my current

clients has done just that, we are trying to start with the corporate objectives, and then work out a

PR programme that will help achieve those. One thing leads another - we hope.

R: I mean (pause) over your career, you must have come across loads and loads of different methods

of evaluation, if you could change any of the ones you have used, or merge them together, how and

why would you do that?

1: (pause), well with my other hat on, in my other life as an economist - a statistician, I've got very

involved from the beginning of the evaluation process with agencies that do the evaluation process,

and pushed them a lot to try and become more aware of how we measure the factors that affect

reputation, because we can see how a good media report, or a good media campaign, will affect

some of those factors, but some of them are about relationships some are about other departments

and how they work but they all affect reputation, very recently we've now got companies that are

being very effective in measuring the impact of those factors on share price for example, which can

be a good way of evaluating if something has worked, in changing the opinion, if it has changed the

opinion of investors, it has usually been very effective in other areas as well. What we haven't done

yet is make that link, to show how (pause) things that affect reputation can affect share price can be

affected by good PR, so we are measuring the end result, we can find ways of measuring the end

result but we can't necessarily find ways to measure the links in the chain that lead to that, so if I

could (pause) take good media analysis, and project output analysis, and link that to the factors that

affect reputation, then I think we would be in heaven in terms of persuading the boards that we

know what we are doing, we know what works and this is how it works.

R: I mean, also one thing that I've found through primary and secondary research is that, a lot of

measuring agencies as such, or monitoring agencies, using technology more and more, but do you

feel that computers can't read an article in a newspaper and measure the tone?

1: Yeah, I mean I'm using, doing some of that at the moment, I have a client who I am doing some

media analysis for, and they've used a computerised system up until now, and it doesn't tell them

much as you say, it reports words and when words are used, and they might have somebody who is

reading the coding content analysis, to see whether it is positive or negative, but I think it does need

to be a human being in my view, to read that sort of stuff, to really interpret the way things are

going, because you can get a lot of coverage, you can get a lot of positive coverage, but the overall

mood of the report when you read them together, and the impact of saying you've done a press

release and it gets covered or events happen and it gets covered in the next day's report, you can

read them together and take out a very negative view from them, rather than a computerised media

97 | P a g e

20923417

analysis look at it and suggest a positive, so you're right, I do think you need it - there will always be

a place for humans. It is what makes us human!

R: Definitely! So do you feel that a person with a greater understanding of evaluation, would get or

arrive a better result or better figure ROI figure, compared to a colleague who is either new in the

industry and doesn't have a good understanding of why it is being done?

1: Yeah, it depends if the person who is new in the industry came from a good university degree,

where they got access to the latest methodology and so on, (pause) then they would obviously be

better, but I think sometimes there can be very strange nuances, sometimes you can rejoice that a

journalist that has been negative is slightly less than negative, and you feel that you have achieved

something, whereas in the straight forward physical response, or as you say, someone who is new to

the industry or new to the company, looked at it, they would think it was a negative report.

Somebody with experience might be happy that it is not as negative as it could be. And you need

experience to understand those nuances, sometimes it can be how a media reports, Top Gear is

famous for that in the car industry, they will look at some things in a way that its public although

they know they are taking the mickey, they actually like the thing, but you might look at it from a

straight forward use of words point of view and take out a negative view. So experience counts but

new ideas are always welcome.

R: Ok, And also, when you are working with clients who are using various media's, say digital and

traditional, in one campaign or in one hit, would you alter your evaluation model to affect that?

1: Yeah, I think with different media's you are looking for different things, the end results, the

outcomes and the reputation scores will stay constant, the way you measure those and monitor

those. But the way you measure outputs and(pause) and activity levels in media will have to be

different and the way you measure even Facebook from Twitter is different, you can measure likes

on Facebook, but that doesn't really tell you very much at all, you know, you need to understand

which part of the message people responded to, for example, measuring and understand number of

likes is one thing, but what people did about it is something else. So on monitoring website hits,

length of time people stay on pages, all of that sort of stuff, we need to develop a different

monitoring mechanism, to measure social media, more importantly to understand what they means,

for whether our campaign has been a success or not, people might the joke you made in an ad or in

a Facebook video or whatever, but not really, but does that mean they like the product, they like the

brand or the campaign, not necessarily, we need more research than just simple counting.

R: And also, having worked in house, have you ever worked in an agency?

1: Yeah.

R: You have, so what do you feel the main differences are between evaluation culture in in-house

teams and agency teams?

1: agencies are much more focused on outputs, in house are much more focused on outcomes. I'll

keep it that simple.

R: Right

98 | P a g e

1:It is not that easy, a lot of in house teams are still very old fashioned and count and weigh things

rather than understand the impact on a business, but what I said before about in house teams and

PR generally wants to get up higher up the pecking order, the executive pecking order, by getting the

executive to understand the role that PR plays in the business in building reputation, and I think that

most of the big companies that I've worked with are geared that way, but most of the consultancies

tend to be geared to either outputs or either campaign objectives achieved, you know where

campaign objectives are measured by outtakes and outputs rather than outcomes.

R: Ok, also when working with a PR agency or a Corporate who has companies or offices across the

world, do you feel that their local culture and ethics come into play when it comes to evaluation or

do you feel that the model set in the home country is the one that resonates around the world?

1: (erm pause) Usually, I think in big companies the one that is set in the home country is the one

that resonates around the world, if they can afford it, if the media in the country allows it, trying to

understand sentiment in Japanese or Chinese is very very difficult to get a computer to analyse that.

Yes I think it is centrally driven, but in like all PR local cultures, but I do think you want to be able to

compare one countries results to another. You want to compare one campaign with another, and

how brands are doing in different market places, so you need to use the yard stick, the same statistic

to measure that.

R: Ok, last question - do you feel that the best form of evaluation is when a client comes to you and

says ok I'm happy with the word you have done, I want you, to sign you for another year?

1: If you are an agency then yes, but, I'll say but, clients are most likely to do that if the agency can

prove and demonstrate that their work has been effective, rather than just having a good

relationship, and the client doesn't think that anybody else could do any better, the agencies need to

demonstrate more and more that they are effective, and that does involve them getting monitoring

outputs rather than simple outcomes.

R: Sure

1: Sorry outcomes rather than outputs, got them the wrong way round!

R: So if an agency promises you X Y and Z, they have to prove they have got X Y and Z.

1: Yeah, I mean the client, a good client would demand more of the agency all the time, and

because, one of the things is, agencies have more of a broader range of staff, so have more capacity

to do things that clients do, so although clients have the money, the clients don't necessarily

understand the latest analysis techniques for evaluation so they could, agencies could demonstrate

their worth, by bringing those to the attention of the client, especially social media evaluation, and

offering that, to do that for the client, and grow the business as well.

R: Ok, that is great! Thank you for your time and your insights!

1: Ok, My pleasure, I'm sorry I'm a bit half way between the two here. Former client and running a

small agency.

99 | P a g e

20923417

R: No it has been really helpful, as the people I've spoken to so far have all been agency staff, so

having someone who has actually done both, and you've done the corporate side and the consumer

side erm, it has put a whole new spin on things! Thank you very much.

100 | P a g e

Appendix 2.5 - Transcript - Interviewee Coding E

1. What method of evaluation does your company use?

We currently use a spread of evaluation methods including EAV, ROI, circulation stats and

total pieces of coverage.


We always set targets at the beginning of a campaign, in partnership with clients to align

with their business objectives.


Yes

1c. Have you ever had to re call a campaign as measuring it when it was live gave a negative

result?

No



I think the large choice of media allows businesses to reach people in different ways and

allows PRs to be truly creative and proactive in their pitching of ideas.


No, we tailor our methods to their reporting needs and business processes


clients or their board of directors?

No

5. If you could change your company's method of evaluation what would you do and why?

I would place more emphasis on 'impact on the ground' for example the number of sales as

a result of click through a from online coverage or the number of tickets sold/entries to a

race.


Evaluation is very important. It makes everyone work to their best ability and ensures

efficient and tailored activity



I believe PROs who understand their client's needs and targets will achieve quality results

e.g. Key press and USP inclusions

101 | P a g e

20923417

7. When working with clients who gain coverage on various media's away from print does your

evaluation model differ?

Yes, for online we measure click throughs but only a few clients have systems in place to

monitor this on their websites.

8. What are the main differences do you feel that there are between evaluation culture of in

house teams to agency teams?

I'm not sure as I've only worked agency side

9. Do international companies use one method across the world, or does local culture and

ethics make a difference?

Can't really comment on this as the large majority of clients are UK based. We have one

client in US and we use similar process of evaluation


Renewal of a contract is very positive as it demonstrates the value of PR and relationships

however I feel the best form of evaluation is business success and impact on the ground,

with the consumers e.g changing beliefs, awareness of a product or service.

102 | P a g e

Appendix 2.6 - Transcript - Interviewee Coding F 1. What method of evaluation do you feel is most commonly used in the PR industry?

As with most forms of measurement, people start with what is both easy and cost effective to

measure. For PR this starts with a focus on media relations, which has an output that is easy to both

capture and count. Media content analysis is often used as a basis for the success of PR, despite the

fact that PR involves so much more than media relations. Basic output measures based on media

content certainly dominate PR measurement.


It can do if it is done properly. Of course, perfect measurement involves climbing the ladder

from outputs, to outtake and ultimately outcomes but any form of measurement can be

rooted in best practice and begin with the setting of goals and objectives. Whether media

content analysis is the right measure to demonstrate that those goals and objectives have

been met is another matter.


Often is does, yes, as typically the work we do with clients is ongoing.

1c. Have you ever had to or heard of examples that a campaign has had to be recalled due to

negative evaluation, whilst the campaign was in the public domain?

PR doesn’t really work with the concept of recalling. Adapting or reprioritising might take

place if a campaign does not gain traction. If something elicits an immediate negative

response which is picked up by us, there may be the need for damage limitation.



I don’t think these things are connected. I’m not sure mass media is the right phrase. Social

media specifically, and the ability for the consumer to publically communicate back to an

organisation, alongside the effect this can have on search e.g. my Dell Hell / Dell sucks

coming top when you searched for Dell online back in 2005, made CEOs sit up and take

notice. A lot of customer service problems were landed on PR in the early days, now there

is the realisation that good PR cannot fix a fundamental problem in the business. The

concept of social business is one that is really changing the way businesses operate.

3. Do you feel that PRO's use one unified method of evaluation for all clients?

No. Smart PR agencies are using framework based in best practice and then selecting the

right measures to fit client goals and strategies.


clients or their board of directors?

The only way for a PR person to increase the ‘value’ of evaluation would be to start by using

a value in the first place which I take to mean AVE. I have seen lots of PR people over the

years apply a multiplier to AVE on the basis that editorial holds more value than advertising.

Although third party endorsement does add value, it has never been calculated how much

this should be (because it’s an impossible calculation dependent on so many variables). On

top of that AVE is a nonsensical number that talks to nothing PR does for a business.

103 | P a g e

20923417

5. If you could change evaluation in PR, what would you do and why?

I would ensure that every programme began with some brilliant thinking as to what the

business was trying to achieve, how PR was helping this happen, what measures would

accurately show PR was doing this successfully. Thinking upfront destroys the need for

spurious measures such as AVE and basic counting of outputs.


Very.



I’m not sure what you mean by accurate results? Results that show better what PR has done

to move the needle for the business? Yes.

7. When working with clients who gain coverage on various media's away from print does how

do you feel evaluation models differ?

Such as online and social? The model of best practice should not differ, the metrics used

though can differ.



Great agencies are taking the lead and pushing clients on setting measureable objectives

and measuring them credibly. On the other end of the scale, some agencies use big figures

to try and impress clients. Exactly the same is true of clients in-house. There’s no clear

distinction between in-house and agency, there’s good and bad in both.

9. Do you feel international companies use one method across the world, or does local culture

and ethics make a difference?

For a Global Comms teams, one method is important in terms of framework and success

metrics. The targets however have to adjust to take into account market differences.


For an agency? Yes. And whether you grow that account into other territories or business

areas etc. Retention, growth and recommendation are all indictors of whether a client is

happy with you. It doesn’t necessarily mean you are evaluating properly or having an effect

on the target audience. Chemistry and many other factors could be make this happen. It

would be interesting to do some quantitative research to understand if there is a link

between agencies evaluating properly and the retention / growth of their client base to see

if there is a causal link. If you do, I’d love to see it!

104 | P a g e

Appendix 2.7 - Transcript - Interviewee Coding G

1. What method of evaluation do you feel is most commonly used in the PR industry?

AVEs probably, although I have no evidence of this. 1a. Does this method require goals and objectives to be set at the start of the

campaign?

No, not at all.


No, it only covers the measurement of the press coverage.

1c. Have you ever had to or heard of examples that a campaign has had to be

recalled due to negative evaluation, whilst the campaign was in the public domain?

No

2. Do you feel that the Mass Media or the large choice of media has an effect on the

way that PR is seen in the business place?

Not sure what you mean - I do think that the media is influential on the way PR is

seen in a number of ways. If you mean with regards to evaluation, I do not see any

connection.


No, there are many methods, and many agencies have their own methods that they have used themselves. AVEs do seem to be the most prevalent however.

4. Do you agree that some PRO's increase the value of their evaluation reports to

impress clients or their board of directors?

There is one example of how this seems to happen - many PROs multiply the value of their AVEs by 3, justifying this by claiming that PR is 3 times as effective as advertising. This seems to be completely unscientific and just there to increase the value.


A method based on the full length of the campaign, which focuses on the initial requirements, and whether the right people we actually reached. The AMEC meeting in June is expected to put forward a new form of evaluation which I would suggest will be useful to have a look at.

105 | P a g e

20923417


Quite important, but perhaps its importance is over-stated. The PR industry does very good work and that should not be forgotten just because the means of measuring it are not totally accurate. 6a. Do you feel that PRO's who understand evaluation get more accurate results

than their colleagues who don't have such an understanding?

I suspect that this may not be the case, and that good PR skills are not linked to evaluation.

7. When working with clients who gain coverage on various media's away from print

does how do you feel evaluation models differ?

Don't understand the question - I think evaluation methods do differ for different media.

8. What are the main differences do you feel that there are between evaluation culture

of in house teams to agency teams?

I think that agencies ‘own’ evaluation and that in house teams leave it to them to do most the evaluation work.

9. Do you feel international companies use one method across the world, or does local

culture and ethics make a difference?

I think it depends from one company to another, but there will be some instances of international companies using one method across the world.


Probably not, no!

106 | P a g e

Appendix 2.8 - Transcript - Interviewee Coding H

1. What method of evaluation does your company use?

Google analytics for the website, retweets on Twitter, likes on Facebook, Google

alerts, hashtags on Twitter and Instagram

1a. Does this method require goals and objectives to be set at the start of the

campaign?

Yes. For example, if we begin a social media campaign and want to build a buzz

about a new product, we will set a numerical goal in terms of retweets, likes,

hashtag uses.


Yes. Absolutly. This is a must because one should be able to track and measure

during what time in the campaign there were more impressions in the entire

digital space.

1c. Have you ever had to re call a campaign as measuring it when it was live gave a

negative result?

No.



The industry of PR can have a negative reputation because many people who are

not a part of the industry think we “spin” stories. This is not true, because most PR

professionals are well-versed in what’s ethically right in terms of communication

with the public and a private organization. I do not call it spinning, it is more like

finding what’s newsworthy and running with it.


Yes. We have our clients fill-out a survey about their experience, and if we

delivered the results they wanted and expected.



Not that I know of. This would be unethical.

107 | P a g e

20923417

5. If you could change your company's method of evaluation what would you do and

why?

I would not, I think they do a great job.


Of course it has to be one of the most important aspects of a campaign because

you need to be able to see what type of impact your campaign initiatives had on

the public. Without evaluation, it would be impossible to measure return on

investment as well. Not to mention, the client needs to see proof that you are

doing something right with their campaign.

6a. Do you feel that PRO's who understand evaluation get more accurate results


Well, if you are a professional, you must know what evaluation your work is a

must. For example, if a pro doesn’t evaluate their process, they will not know what

worked and what did not work.


does your evaluation model differ?

No, because we mainly concentrate on digital marketing and PR and most of our

impressions come digitally. Unless a client asks for a traditional PR route for their

campaign.



I work in an agency type of environment, so I am used to dealing with a few clients at a

time. It’s important to discuss and understand what the client wants and execute what

they need. These are clients who demand results and rely on us to make a PR buzz in any

industry they are a part of. We survey the client, we measure the “buzz” digitally, and

show the clients what worked and what didn’t work based off of our PR measurements. I

feel like in house PR also use evaluations to show profit to their seniors. That would be the

difference.

9. Do international companies use one method across the world, or does local culture

and ethics make a difference?

108 | P a g e

I definitely think that local culture and ethics make a huge difference. There may

be an agency out there who doesn’t properly measure their PR results, and in

result, they may continue making a mistake over and over again with different

clients.


Well, they renewed the contract! So that should say it all.

109 | P a g e

20923417

Appendix 2.9 - Transcript - Interviewee Coding I 1. What method of evaluation do you feel is most commonly used in the PR industry?

The most common method of evaluation unfortunately is still based on OTS

(opportunity to see). This is probably based on clients’ familiarity with the notion of

“reach” from ad campaigns and a requirement to translate PR into comparable

return on investment.

1a. Does this method require goals and objectives to be set at the start of the

campaign?

It often does, although OTS can be easily gamed by securing online coverage. If OTS

is a key measurement then demographic should be part of this from the outset (i.e.

the right people see the coverage). Also a list of key media should be defined up

front to avoid coverage being secured in non-relevant media and being counted in

the final evaluation tally. Often, there is a volume target set at the start of a

campaign (e.g. 15 x pieces of national coverage, 25 x pieces of regional, 2 x

broadcast, etc.).


In our agency, evaluation is done in real-time, so that we can spot trends, patterns

and successes as we move along. This can be shared with clients on a weekly basis or

reviewed at a 6 month evaluation meeting. Usually, the campaign has a main

evaluation at the end of the year (for retained clients) or at the end of the project.

1c. Have you ever had to or heard of examples that a campaign has had to be

recalled due to negative evaluation, whilst the campaign was in the public domain?

No. Often, the strategy or KPIs are revised depending on initial results. So, if a brand

has a new product range, KPIs could be increased. Likewise, if a key component of a

campaign is no longer possible to activate, KPIs can also be rejigged to make the

most of the campaign fee.



There is still a misconception of what PR actually is both within the wider public and within

businesses in the UK. Particularly in the small business space, PR is seen as a poor relation of

110 | P a g e

advertising whereas the reality is that it can often deliver better long-term results for

companies when compared against paid-for media.

The general public often confuse or mistake PR as being synonymous with political spin-

doctoring or Champagne-swilling at parties (á la Absolutely Fabulous). PR is also seen as

being similar to the services provided by celebrity publicists like Max Clifford which is a small

part of what the industry actually has to offer.

In fact, now that the media landscape has become fragmented and much larger, the

opportunity to secure good PR has also grown. In addition to this, the way in which

consumers and businesses actually engage with media has changed so much that it is now

essential that PR takes on a bespoke strategy to ensure key audiences are reached via the

media platforms that they consume.


Not in my experience. I know that we try to use qualitative and quantitative methods of

evaluation for all clients but have to tweak our evaluation methodology to align them with

our clients’ business objectives. So, if a client is interested purely in inbound clicks to their

website(s), our evaluation must prove that we have delivered on this regardless of how

many pieces of coverage we secured.



Part of our job is to work to whatever methods of presentation best suit our clients. That

said, we would never fabricate results.


I would fully embrace a uniform system that takes on board the Barcelona Declaration

of Measurement Principles that set out to include volume as well as impact of

campaigns. These principles go beyond pure OTS and offer a better 360 degree

understanding of how effective a PR campaign is. It also takes into consideration social

media and online coverage which goes beyond the pure vanity of the number of Likes

111 | P a g e

20923417

and Followers (which can be unscrupulously bought) and focuses more on actions such

as engagement.


Incredibly important. Without measuring success, you can’t justify spend. That said, some

things are more difficult to evaluate than others. For example, measuring people’s warmth

towards a brand (beyond £) is difficult, although not immeasurable.

6a. Do you feel that PRO's who understand evaluation get more accurate results


Yes. Evaluation isn’t an end-process that you do post-campaign. It should be an ongoing

process that is central to the execution of PR strategy. PROs that get this, will make sure

that everything they do has a positive impact on evaluation.


does how do you feel evaluation models differ?

For clients that have an online-centric model, it is easier to show impact of PR through

clickable links in coverage that can be measured on a source by source basis. These clients

seem to be more switched on to other forms of quantitative evaluation (SEO, PPC, email

newsletters, etc.).

8. What are the main differences do you feel that there are between evaluation culture

of in house teams to agency teams?

In-house teams tend to have more pressure on delivering comparable pound for pound

ROIs on different marketing disciplines (SEO, Direct Marketing, advertising, PR, etc.)

whereas agencies will focus on delivering excellent PR results.

9. Do you feel international companies use one method across the world, or does local

culture and ethics make a difference?

Global media trends and localised news will affect the effectiveness of PR but general

universal PR evaluation methods should be applied to all campaigns (see above: Barcelona

Principles)

112 | P a g e


No. But it always helps!

113 | P a g e

20923417

Appendix 3.1 - Permission Letter - Coding A

Appendix 3.2 - Permission Letter - Coding B

114 | P a g e

Appendix 3.3 - Permission Letter - Coding C

Appendix 3.4 - Permission Letter - Coding D

115 | P a g e

20923417

Appendix 3.5 - Permission Letter - Coding E

Appendix 3.6 - Permission Letter - Coding F

116 | P a g e

Appendix 3.7 - Permission Letter - Coding G

Appendix 3.8 - Permission Letter - Coding H

117 | P a g e

20923417

Appendix 3.9 - Permission Letter - Coding I

Appendix 4.1 - Supervisor Meetings

Date Discussion Actions Agreed

October 15th Discuss topic (too broad), initial proposal content

and structure.

Agreed new question and discussed

initial proposal structure and

content.

November 15th Initial feedback discussion, improve research

objectives, research question.

Improve research objectives for

next meeting, think of ways to

improve question.

January 8th Discuss progress on literature review and

methodology, structure, sources, methodology

approach (onion)

Agreed other books to be used,

finish methodology, brief outline of

philosophy, aim to finish for

original deadline.

January 20th Literature Review and Methodology 1st draft,

discuss structure.

Alter structure with proposed one,

insert more content for submission

day.

118 | P a g e

February 20th Discuss feedback from literature review and

methodology to establish where mistakes were and

how to improve on them

Improve structure, find more

academic sources, Make

methodology and sample clearer.

Improve primary research

questions.

April 17th Discuss Coding with John Hathaway - Not sure if I

was doing it correctly.

Coding done correctly. No action

Key Skills Review

Through my time at Buckinghamshire New University, I have developed many new skills,

from writing skills, presentation skills, public speaking skills as well as many others that will

aid me in the transition from education to working in the real world.

Firstly I will discuss my writing skills. During my three years at university, I have noticed my

writing skills improved, as this is a fundamental part of working in Public Relations. My

tutors helped me get to this stage by ensuring that tasks are set to help me to practice my

writing skills, while providing feedback for the same.

Following my internship, which was arranged by my tutor again, I have been actively writing

press releases, blogs and social media postings on a regular basis, further improving my

skills by obtaining further feedback from my employer.

By having this opportunity, I was able to really stamp out bad habits I had when writing,

such as, using commas incorrectly, while improving on grammar and increasing vocabulary

with industry standard terms, which I feel will allow me to increase my chances in finding

future employment.

119 | P a g e

20923417

The second skill which I feel has improved in my time at university, is the one of speaking in

front of members of the public. Having completed public speaking examinations when

younger, I had got out of the habit of speaking in front of large groups of people, thus

making me feel very nervous, since being at university, and having to present in front of

peers and external persons, who are industry experts helped me to improve my spoken

presentations as well as teaching me to maintain eye contact with people while talking to

them.

I now feel extremely confident in approaching people and comfortably and confidently

speaking with them, while having an insightful conversation. This has helped me from not

only approaching lectures in university, but also with networking and generally in

approaching people whom I have not spoken to previously.

Now, I feel, if put on the spot to discuss a subject with people, I will be able to discuss the

topic with ease, whether I know a vast amount on the subject or not.

I feel that is skill will help me the most out of all of the skills in my chosen career, as in PR is

important to be able to confidently speak and engage with people.

The third skill that I have developed, is my ability to carry out research quickly, accurately

and effectively, I feel that having access to a wide range of sources has helped me to carry

out various analysis's on briefs for assignments. I feel that this was possible, due to the

number of interesting briefs provided by the teaching staff, to help put the classroom and

academic content into a real life perspective.

The fourth main skill that I feel I have learnt during my time at university is the skill that has

allowed me to further my career opportunities, was the presentation skills that we were

taught. By having a knowledge in how to present effectively, it was possible to add this to

my public speaking skill, meaning I can stand in front of a panel of professionals and discuss

my idea's and plans with great ease.

I feel that this will help me in my career as it will allow me to present to clients with pitch

idea's and to keep them enticed in what I am saying by creating a mini play and maintaining

eye contact with all members in the room.

120 | P a g e

Finally, my final main skill, is leadership, I feel that this skill is going to help me take control

in a work environment if needs be. This skill will help me show future employers that I am a

responsible individual, who is able to organise teams to carry out tasks, whilst ensuring that

I still have work to carryout to help make a difference to the outcomes.

I feel that the above skills outlined will help me to improve my work life as I feel I have

developed well as a human being to become an excellent PR officer and team player.

Which Method of Evaluation is Most Accurate in the Public Relations Industry

Documents

research objectives

research context

research philosophy

research approach

research strategy

research choices

evaluation method

cons of current evaluation