10 Tips to Improve Testing

10 Tips for Better Testing

2

About Us

Jen Boland

Rosa Del Angel

Data Analyst@jenboland

Senior Marketing Consultant

3

About Beaconfire RedEngine

Integrated Fundraising & Marketing

Web Design & Development

Strategy, Testing & Analytics

Acquisition, Cultivation

and Retention

Team of 40+ fundraising, marketing, UX and technology experts focused on optimizing clients’ digital programs.

#1 Run an A/A Test

4Confidential - Presented to [Client Name] - ©2014

#2 Statistical Significance

• Make sure your results are statistically significant.

Sample Size Calculator

“A minimum detectable effect (MDE) is the smallest true effect that has a ‘good chance’ of being found to be statistically significant.”

Minimum Detectable Effect

Sorry! – You’re not Amazon

Minimum Detectable Effect

10

11

12

Test across thank you pages and emailsC: Honor Gift

B: Sustainer w/ Premium

A:Sustainer

“ I hate math. But I love counting my money.”

- unknown

#3 Use the right stats

Online Calculator

Statistical Significant Gift Array Test Similar conversion rate, but

55% higher

avg. gift and 13%

more revenue

T-test

Using Excel to Conduct t-Testt-Test: Two-Sample Assuming Unequal Variances

Variable 1 Variable 2Mean 96.69892 113.956Variance 15711.8 70387.51Observations 93 91Hypothesized Mean Difference 10df 128t Stat -0.88788P(T<=t) one-tail 0.188135t Critical one-tail 1.656845P(T<=t) two-tail 0.37627t Critical two-tail 1.978671

Results are significant if p < .05.

“You Need a Hypothesis. Period.”

#4 Hypothesis

18Confidential - Presented to [Client Name] - ©2014

Test against your control

2X Conversio

n Rate22% More Revenue

CONTROL VARIANT

20

Hypothesis: By improving the user experience and integrating the memorial and honor giving options into one donation form, we can improve conversion.Control Test

Hypothesis: Sending paid media traffic straight to donation form will generate more conversions, more revenue

Visitors sent to the splash page were 47% more likely to donate

and had an average gift $69

higher than visitors sent

straight to the donation form

22

Keep variables the same7% Higher Revenue 11% Higher Avg. Gift

23

#5 Segment your results

New Users vs Returning

Mobile vs Desktop

Donors vs Non Donors

Acquisition vs Cultivation

#6 Validate Your Results

Keep an eye on your conversion rates.

Make sure your leads convert

27

• Global Navigation Menu

• Carousel Banners and Sliders

• Stories Strip• Header and

Footer Navigation• Donate Page and

Quick Donate boxes

• CTA buttons within content

#7 Advanced Analytics

28

HRW form validation tracking

29

Become an EX registration form

30

AfterReduced error rate by

28%

#8 Learn from your failures

32

Winning Search Text Ad Facebook Ad

#9 Use test results to inform other channels

Nearly 30 Million Battle Diabetes. Your Donation Will Improve Lives. Donate Today.

33

#10 Test across channels

Supporters who received integrated emails turned out to be three times more likely to donate compared to those who did not. Also, a 5% lift was seen in offline giving for people who received the integrated emails.

Connecting Offline & Online Audiences

• Special Olympics EOY Facebook Custom Audience Test • Facebook CRM ads targeting direct mail donors, lapsed donors

and non-donors during EOY matching gift campaign • Test Group A featured donors who received a Matching Gift direct

mail package AND were served Facebook ads• Test Group B included lapsed donors and non-donors who

received a non-Matching Gift direct mail piece AND were served Facebook ads

• Match rate from Facebook was 50%

• DM lapsed and non-donors who saw FB ads (test) and who did not see ads (

Facebook EOY CRM Retargeting

• Facebook CRM ads targeting direct mail donors, lapsed donors and non-donors

• Test Group A featured donors who received a Matching Gift direct mail package AND were served Facebook ads

• Test Group B included lapsed donors and non-donors who received a non-Matching Gift direct mail piece AND were served Facebook ads

• 2 Control groups (donor and non) who did not see Facebook ads

Facebook EOY CRM Retargeting• For donors, FB ads generated 153% ROI

with online gifts, plus small avg gift increase in mail

• For non-donors, FB ads lifted avg gift in mail by nearly 20% and generated 21% more revenue than DM group who did not see ads. For every $1 spent online, we raised $8.50 in the mail.

• Match rate from Facebook was 50%

37

1. Run an A/A test2. Make sure your test results are statistically

significant3. Use the right stats4. You need a hypothesis5. Segment your results6. Validate your results7. Use Advanced Analytics8. Learn from your failures9. Use test results to inform other channels10.Test across channels

Recap

38

Questions?www.beaconfire.com | @beaconfirered

Jen Boland

Rosa Del Angel

Data [email protected]@Beaconfire-red.com

Senior Marketing [email protected]@Beaconfire-red.com