Orientation Project Implementation Reporting & Sustainability Testing Efficacy To Ensure Applicability and Impact Module 3, Unit 3, Sub-Unit 1, Session.

OrientationProject

Implementation

Reporting&

Sustainability

Testing Efficacy To Ensure Applicability and Impact

Module 3, Unit 3, Sub-Unit 1, Session 5

Failed in Focus Group Example:

http://www.youtube.com/watch?v=BmykFKjNpdY

2

Module 3, Unit 3, Sub-Unit 1, Session 5

3

Pretesting

• The process of systematically gathering target audience reactions to messages and materials before they are produced in final form

Module 3, Unit 3, Sub-Unit 1, Session 5

4

Use Pretesting To:

• Determine which of multiple alternative creative directions or messages may be most effective

• Identify strengths and weaknesses in a single execution (e.g., brochure, poster, etc.)

Module 3, Unit 3, Sub-Unit 1, Session 5

5

Reasons For Conducting Pretest

• Identify the benefit that resonates most with the target audience

• Ensure that audience understands the message• Identify any “red flags”• Strengthen your message• Conduct “real world” check• Make the materials more appealing• Identify weak and strong points

Module 3, Unit 3, Sub-Unit 1, Session 5

6

Which Materials Do You Pretest?

• Pamphlets, posters, brochures, booklets, flyers, print ads, and TV and radio spots and dramas

• No need to pre-test:– Bumper stickers or badges– Derivatives of already pretested materials

Module 3, Unit 3, Sub-Unit 1, Session 5

7

Which Materials Do You Pretest?

• Multiple versions (3 to 4) of creative executions with core messages communicating various benefit exchanges

• More elaborated version of the materials – containing supporting facts and information

Module 3, Unit 3, Sub-Unit 1, Session 5

8

Criteria Evaluated During Pretesting• Comprehension – does the Target Audience (TA)

understand the materials?

• Relevance – Is the material and the message important to TA?

• Credibility – have trust in the message

• Acceptability – fitting with the target audience value and culture?

• Attraction – will it hold the audience’s attention?

• Persuasion – is it convincing?

Rate this Ad:

Comprehension:

Relevance:

Credibility:

Acceptability:

Attraction:

Persuasion: 9

Rate this Ad:

Comprehension:

Relevance:

Credibility:

Acceptability:

Attraction:

Persuasion: 10

Rate this Ad:

Comprehension:

Relevance:

Credibility:

Acceptability:

Attraction:

Persuasion: 11

12

Module 3, Unit 3, Sub-Unit 1, Session 5

13

Pretesting Process

1. Determine test objectives and choose research methods

2. Draft research instruments (e.g., discussion guides, questionnaires)

3. Secure interviewers and space for conducting research (if required) and identify, screen and recruit research participants

4. Conduct testing and analyze results5. Based on consumer feedback revise concepts

and materials

Module 3, Unit 3, Sub-Unit 1, Session 5

14

Determine Test Objectives and Choose Research Methods

• Sample Objectives:• Ensure that the main message of a [poster,

brochure, etc.] is readily understood• Ensure that the language and images used in

a [poster, brochure, etc.] are clear • Select the concept that best communicates the

intended message• Determine what was unclear or hard to

understand• Identify the concerns of respondents that need

to be addressed by the materials

Module 3, Unit 3, Sub-Unit 1, Session 5

15

Determine Test Objectives and Choose Research Methods

• Methods:– Expert and gatekeeper review– Focus groups– In-depth interviews– Central location intercept interviews– Self-administered questionnaires

Module 3, Unit 3, Sub-Unit 1, Session 5

16

Expert and Gatekeeper Review

• Purpose:– Ensure technical accuracy

and acceptability

• To Be Tested:– Content and artistic elements– Print and audio-visual

• Number:– 5 to 10

• Pros and Cons

Module 3, Unit 3, Sub-Unit 1, Session 5

17

Focus Groups• Purpose:

– Group interaction; idea generation; materials evaluation; problem identification

• To Be Tested:

– Print, radio, or TV message concepts, slogans/-tag-lines

– Logos and rough artwork

• Number:– 8 to 10 per group– 2 groups per segment

• Pros and Cons

Module 3, Unit 3, Sub-Unit 1, Session 5

18

In-depth Interviews• Purpose:

– No peer pressure; sensitive issues or complex draft material; hard-to-reach audiences

• To Be Tested:– Draft brochures/pamphlets, including on sensitive or

emotional subjects– Message concepts

• Number:– Minimum 12 to 24

• Pros and Cons• Variations

Module 3, Unit 3, Sub-Unit 1, Session 5

19

Central Location Intercept Interviews

• Purpose:– Short time period and large number of

respondents• To Be Tested:

– Message concepts; print materials (e.g., leaflets, fact sheets, posters); print or radio ads

• Number:– Minimum 50 - 100

• Pros and Cons

Focus Group on Tape:

20

http://www.youtube.com/watch?v=m4qJbW9CDOY

Module 3, Unit 3, Sub-Unit 1, Session 5

21

Self-Administered Questionnaires

• Purpose:– Reactions to materials; large number of respondents

• To Be Tested:– Print materials such as book-

lets, brochures, pamphlets

• Number:– 50 - 100

• Pros and Cons

Module 3, Unit 3, Sub-Unit 1, Session 5

22

Draft Research Strategy

• Outline main features of pretest:– Materials to be tested – Variables to be measured – Method – Sample – Estimated length of each interview – Description

• Control for presentation• Pretesting brochures and radio spots

Module 3, Unit 3, Sub-Unit 1, Session 5

23

• Secure interviewers and space for conducting research (if required) and identify, screen, and recruit research participants. – “Facilities”– Moderators

Pretest Logistics

Module 3, Unit 3, Sub-Unit 1, Session 5

24

Conduct Testing and Analyze Results

• Observers

• Note taking

• Data analysis

• Pitfalls of data analysis

• Report writing

Module 3, Unit 3, Sub-Unit 1, Session 5

25

Revise Concepts and Materials

• Change

• Possibly change

• Keep

Module 3, Unit 3, Sub-Unit 1, Session 5

26

Reporting the data

• Background: research objectives and description of materials tested.

• Method: include sample size, sample description, how the respondent were recruited)

• Findings: included reactions and qoutes

• Conclusions and Recommendation:

Module 3, Unit 3, Sub-Unit 1, Session 5

27

A Word of Caution AboutPretesting

• No guarantees

• Controlled environment

• Depends on execution

• Biases introduced by participants

• Statistically imprecise

• Bad advice?

Module 3, Unit 3, Sub-Unit 1, Session 5

28

Myths and Misconceptions About Pretesting

• “I don’t have the time or money”

• “I can tell the difference between good and bad materials, so I don’t need to pretest”

• “Our artist/producer says that pretesting can’t be used to judge creativity”

• http://www.youtube.com/watch?v=BmykFKjNpdY

29