The Effect on Differential Mailing Methodologies on ...Postcards 13.6 1, 2, 3 Official Package 10.4 3 Overall 14.3 Household Response Rates by Treatment 1 Statistically significant

The Effect on Differential Mailing Methodologies on Response RatesTesting Advanced Notices, Package Design, Postage, and Other

Treatments

American Association for Public Opinion ResearchOrlando, Florida

May 18, 2012

Presented by: Yelena Pens, Robin Gentry

© 2012 Arbitron Inc.

Arbitron Diary Methodology

» Arbitron Radio Ratings recruits all household members (aged 12+) for a one-week diary survey of radio listening.

» Hybrid RDD and ABS sample.

» Once the person who answered the recruitment call agrees to participate in the radio ratings, the household is mailed a box.

Diary

Box

Brochure

Multi Person Flyer


Online Diary Vision

»Objective• To increase sample sizes while reducing costs • Improve representation for young adults and hard to

reach demos

»New Methodology• Primarily a web-based survey• Address based sample frame• More relevant to web-savvy younger adults and hard to

reach demos


Online Diary Field Test Methodology

»Sample: • ~11,000 households• ABS matched and unmatched telephone numbers• Five markets

»Methodology:• Fielded in Summer 2011• Tested several different mailing approaches, design, and personalization.• New online diary application (registration and radio listening portions)• Household flooding approach• Mobile option (iPhone and Android)• Emails (registration confirmation, reminders, thank you/incentive)• Promised Incentive choices (Amazon or Visa Gift Cards)


Treatments and Test Strategy

Control Test #1 Test #2 Test #3

Address Mail Label (Personalization Test)*Name Arbitron Radio Household

*See session The Effect on Personalized Address Labels on Response Rates and Postal Deliverability Rates by Vrinda Nair and Yelena Pens, Friday, May18, 2012 at 10:00 AM-11:30 AM

Box Invitation

6x9 envelope

Note Card

#10envelope

Snap Pack #2

Note Card

Snap Pack #3

Pre-alertPost Card

Pre-alert Phone

Message

Pre-alert: Post Card

Pre-alert Post Card

Promotional Package Official Package

Pre-alertNone

+

Snap Pack #1


Mailing Materials ( Promotional Look)

Wave 1: Pre-Alert postcard (Oversized)

Wave 2: Initial Invitation Box Mailer

Wave 3: 6x9 “Still Time” Reminder

Wave 5: Note card “Extended Deadline” Reminder

Wave 4: #10 “Last Chance” Reminder

$2 incentive


Mailing Materials (Official Look)

Wave 1: Pre-Alert postcard

Wave 2: Initial Invitation

Wave 3: “Still Time” Reminder

Wave 4: “Last Chance” Reminder

Wave 5: Note card “Extended Deadline” Reminder


Research Questions

» Mailing Treatments• Do pre-alert mailers alone increase response rates?• Do pre-recorded telephone messages enhance the effect of the

pre-alert mailers?• Does the official looking package increase response rates?


Response Rate Calculation

» Adjusted Response Rate:

» Registered Households• Households who successfully registered for the online diary• Not including completion of the online diary• This is a household level not person level statistic

» Estimated Usable Households• Starting sample • Removing the following:

– Non-deliverable Unusable Households (Vacant, No Such Number, Insufficient Address, or No Such Street)

– Media affiliated households– Group Quarters households (10+ persons living in a household such as

dormitories, military barracks, etc.)

= Registered Households

Estimated Usable Households÷(Also called registration rate)


Advanced Notices and Package Redesign

» Official looking package had a substantially lower rate compared to promotional mailers, thus the package did not increase response rates

» Pre-alert mailers alone did not increase response rates. • Surprising results as research

shows advanced notices help to increase response.

• This could be the design of mailer.

Treatment Groups % Sig.

Control 16.3 1

Phone and postcard 16.8 2

Postcards 13.6 1, 2, 3

Official Package 10.4 3

Overall 14.3

Household Response Rates by Treatment

1 Statistically significant differences between postcards and no postcards

2 Statistically significant differences between pre-recorded phone with postcards and postcards alone.

3. Statistically significant differences between promotional look and official look

Note: Rates based on a matched sample.

The pre-recorded telephone messages had a positive impact on response and enhanced the effect of the pre-alert postcards.


Advanced Notices and Package Redesign (Continued)

» Results of Pre-alert Postcard • Design (oversized postcard)

– A large majority of respondents identified it as junk mail and would throw it away.– Many said it appeared to look like something they would receive from a healthcare,

insurance or pharmaceutical company.– Considered to have too much text and unappealing pictures.

• Headlines– “Represent your community” – Some respondents thought “community” was having

to go somewhere and work or donate their time with others.– “You listen to the radio, now the radio listens to you!” – Most recognized that it was

too vague – they weren’t clear on what it was asking of them.– “Be part of the Arbitron radio ratings” – clearly states what they are being asked do.

» Focus Groups (Winter 2012)• Hispanic demos in three markets• Pre-alert postcard, box mailer, and

website


Postage

» The promotional mailers used first-class postage, while the official mailers used standard mail.

» First-Class Mail • Delivered within 2-3

days• Return of mail• Higher Cost

» Standard Mail• Delivered within 7-10

days• No return of mail• Lower cost

45.3%

33.1%

25.5%

31.0%

19.1%16.9%

10.1%

19.0%

0.0%

5.0%

10.0%

15.0%

20.0%

25.0%

30.0%

35.0%

40.0%

45.0%

50.0%

Promotional Official

InvitationStill TimeLast ChanceNote Card

Distribution of Total Response by Wave and Mailer

Note: The note cards were designed to have a flashy appearance and were mailed using first-class live stamps.

Unlike the promotional mailers where the note cards had the smallest impact, the official mailers had an increase in response.


Budget (Package Design and Postage)

» Promotional mailers had a positive impact on response compared to the official mailers.

» Promotional mailers have higher mailing costs compared to the official mailers with postage driving the increase.

» Is the increase in RR worth the increase in cost?

Promotional Package

Official Package

Total Mailers $0.288 $0.68

Pre-alert postcard $0.05 $0.015

Initial Invitation $0.155 $0.31

“Still Time” Invitation $0.025 $0.21

“Last Chance” Invitation $0.023 $0.11

“Extended Deadline” $0.035 $0.035

Total Postage $3.75 $1.79

Total (per household) $4.04 $2.47

Note: Labor costs for vendors are not included in the pricing; excludes incentives.


Budget (Package Design and Postage)(Continued)

» Higher response rates, lower costs.• More sample will have to be

mailed of the official items to reach the same targeted sample with promotional mailers.

» Sample representation better with promotional items.• Older demos had higher rates

for the official materials.• Hard to reach demos had

higher rates for the promotional materials.

Promotional Package

Official Package

Targeted Completes 10,000 10,000

Response Rate 16.8% 10.4%

Mailed Sample 59,524 96,154

Incentives ($2) $119,048 $192,308

Materials/Postage $240,357 $237,500

Total Costs $359,405 $429,808


Multiple Mailings

» If multiple mailings are being mailed to the same address, non-deliverable mail may differ by packages/waves.

» Reasons for inconsistencies:• When a mail piece is returned to the sender for any reason, the piece

is endorsed by the carrier.• There many be different carriers on a route on any given day, so

endorsements may vary.

Promotional Mailerpre-alert box 6x9 #10

Usable 84.0 76.7 82.4 82.8

Unusable 16.0 23.3 17.6 17.2

Non-Deliverables by Mailer

The box which is a bulky mailer has the highest unusable rate.

Note: Usable: Attempted Not Known , Illegible, Unable to Forward/Forwarding Order Expired, Deceased, Temporarily Away/hold mail, Refused, No Mail Receptacle, In Dispute, or Unclaimed; Unusable: Vacant, No Such Number, Insufficient Address, or No Such Street.


Postal Day Mailing» The promotional packages as well as the official packages were

mailed each postal day (except for Saturday mailings for official packages because of logistic reasons).

17.5

10.0

14.5

9.6

15.6

10.3

15.7

9.7

14.8

8.1

14.3

0.0

2.0

4.0

6.0

8.0

10.0

12.0

14.0

16.0

18.0

20.0

Promotional Official

MondayTuesdayWednesdayThursdayFridaySaturday

Response Rate by Day of Week

Monday mailings performed better than any other mailing of the week.


Additional Treatments (Effect on Response Rates)

» Incentive Choice (Visa vs. Amazon)• Mailed Visa gift cards were the preferred choice over online,

instantly available, Amazon gift cards (77% vs. 23%).

» Choice of Survey Start Date• About 93% of registrants selected a diary start date on the same

day as registration for the diary.

» Mobile Option• About 11% of new visitors going to the site were mobile users.

Desktop users were more likely to come back to the website as compared to mobile users.


Post Test (Effect on Response Rates)

» Emails• About half (46%) of the pre-registered household members did not have

an email address. Also, 93% of those pre-registered household members without an email address did not register for the online diary.

» Post Test• Pre-registered household members were mailed a package with a

personalized invitation to register for the online diary as well as a small token of appreciation.

• The packages were addressed to the original registrant with instructions to distribute the packages to unregistered household members.

• Results…– The household member registration rate was 36%, and after the Post Test it

was 47%, so the mailers had increased overall household member registration by over 10 points.

• It appears direct communication such as emails and mailers are important to response rates.

– Possibly the original registrant is a gatekeeper for other household members.


Conclusions

» The official looking package had a substantially lower household registration rates compared to the promotional, direct mail marketing materials.

» The pre-alert postcards plus pre-recorded telephone messages performed the best.

» On the other hand, the postcards alone did not perform better than no postcards.

• The results from a focus group show that design and messaging has an effect on response.

» Monday First Class mailings performed the best.

Advanced Notices

Pre-recorded telephone messages enhance mailers

Package Design

Promotional, direct marketing, bulk boxes work

Postage First Class Mail

Mail Date Monday, Received on Wednesday


Next Steps

» Planning a third Pilot Test in Fall 2012• Focus on collecting email addresses from all household members• Solving usability issues from previous tests• Testing pre-recorded telephone messages alone as a pre-alert• Testing different pre-alert designs and messaging• Testing a pre-alert incentive


Thank You

» For more information please contact:• Yelena Pens

– [email protected]• Robin Gentry

– [email protected]

The Effect on Differential Mailing Methodologies on ...Postcards 13.6 1, 2, 3 Official Package 10.4 3 Overall 14.3 Household Response Rates by Treatment 1 Statistically significant

Documents