Is This Normal? Common Challenges Testing Programs Face

Is This Normal? Debunking myths & getting real about the challenges optimization programs face.

Ryan Garner for Clearhead Jessica Vasbinder for Warner Music Group

Hi. I’m Ryan Garner. !This is Jessica Vasbinder. !!This is the part where we tell you about ourselves. !!!!

(@ryantgarner)

(@entweetment)

You’ve seen these examples, right?

+125% lift!

control variation

+5,500% lift!

control variation

And we’ve all played

Jocelyn Michael

Winner! Good for you!

Jocelyn Michael

Now pick which is

A. B.

Everything is awesome!

This stuff is easy,

Let’s get real.

And we are going to do this by asking 1 simple question along the way... !

!

!

!

!

Is this normal?

We ran our first set of tests. Made buttons bigger, changed copy, tested different colors. And zip, zero… !

Is this normal? !

!

!

!

Yes!

• It’s okay to crawl before you walk, but set expectations. – You may (probably will) NOT materially improve your bottom

line. !

• Risk (Disruption) & Reward (Lift) are correlated. – Small tests often provide small returns…especially for smaller

businesses!

We don’t know where to start testing or which ideas are good and which are bad or how to keep momentum. !

Is this normal? !

!

!

!

Very!

You have no doubt heard..

• “Always start at the bottom of the funnel.”

• “Test the whole funnel simultaneously.”

• “NO, Find the bottlenecks IN the funnel.”

• “Navigation is always a good place to test.”

• “Forms, baby, forms.”

We say...start with the best ideas!

And where do the best ideas come from?

Look Inward Look Outward

Your product roadmap User tests!

Your customers’ feedback Competitors

Ideas from around the company Vendors trying to sell you stuff you

Your previous test results Companies you admire

OMFG! We have 1,000+ testing ideas. Lots and lots of them seem good. Now where do we start? !

Is this normal? !

!

!

!

Normal? No. Awesome? Hell Yes!

• Create a simple scoring system and get the cream to the top. – It should need no real explanation. – No new acronyms! –Why? No hurt feelings & better ideas!! !

• Then use your own judgment. !• Repeat regularly.

We launched a test that broke our site! I thought building tests was easy. !

Is this normal? !

!

!

!

No comment.

• Let’s be real here. When you launch a test, you are pushing new code onto your production website. – Should just anyone be doing this? !• Do you really want a high velocity, high returning optimization program? – Of course you do! Otherwise it’s just your normal dev process. !

• So mitigate the risks by.. – Using your best developers and designers. Not your marketers –Monitor, Monitor, Monitor.

Leadership wants a growth estimate from all our winning tests, but when we add it up, it seems too good to be true. !

Is this normal? !

!

!

!

!

Yes, but…

• There is no perfect way to do this. !• Stress the difference between statistical significance and

observed lift. !• Trust that there is improvement and move on.

– The only accurate way to confirm lift is to re-test. – Do you really want to do that? !• And if you must show something…

***

So what did we learn? You are normal if…

• It’s not as easy as you though it was going to be.

• Your initial efforts don’t double your business.

• You struggle to consistently find great testing ideas.

• You break stuff.

• You are not sure how to accurately measure your return.

And remember testing is awesome!

Questions?

Thank you.Ryan Garner [email protected] www.clearhead.me

Jessica Vasbinder [email protected]