WTW TLE 2015 Powerpoint_Hua_Charles-final
Post on 17-Aug-2015
14 Views
Preview:
Transcript
© 2015, WhichTestWon, a division of Anne Holland Ventures Inc. All rights reserved. #TLE2015
How To Win Friends & Influence People With A/B Testing
@
Charles Hua Director of Optimization & AB Testing
© 2015, WhichTestWon, a division of Anne Holland Ventures Inc. All rights reserved. #TLE2015
Real-Life Test: Which is True About Charles?
I have 2 kids:(older kid’s birthday
month/day/year) x 2 =(younger kid’s birthday
month/day/year)
Version A
At one point, I considered a career as an accountant
Version B
How good is your gut?
© 2015, WhichTestWon, a division of Anne Holland Ventures Inc. All rights reserved. #TLE2015
Web Analyst
Project Manager
QA
UX Designer
Tagging Eng
Frontend Developer
Statistician
Politician
Mom
Experiment Designer
© 2015, WhichTestWon, a division of Anne Holland Ventures Inc. All rights reserved.
I perform experiments on people…
#TLE2015
© 2015, WhichTestWon, a division of Anne Holland Ventures Inc. All rights reserved.
What the heck is RetailMeNot ?
#TLE2015
Multinational coupon company with the largest coupon sites in US and UK
We are here to save the world… some money!
© 2015, WhichTestWon, a division of Anne Holland Ventures Inc. All rights reserved. #TLE2015
2013 2014
30100
700+
Testing VolumeVeg CQMS
© 2015, WhichTestWon, a division of Anne Holland Ventures Inc. All rights reserved.
+12M Annualized incremental revenue with just one algorithm test
#TLE2015
© 2015, WhichTestWon, a division of Anne Holland Ventures Inc. All rights reserved.
Making sound business decisions with DATA
What Is The Purpose Of Testing @RMN?
#TLE2015
© 2015, WhichTestWon, a division of Anne Holland Ventures Inc. All rights reserved.
Build a solid foundation for success with Tools Process and People
#TLE2015
Get your House in order
1. Involve Others
Know what They are thinking
© 2015, WhichTestWon, a division of Anne Holland Ventures Inc. All rights reserved.
Step 1: Get The House In Order
#TLE2015
© 2015, WhichTestWon, a division of Anne Holland Ventures Inc. All rights reserved.
Right Tools For The Right Job
#TLE2015
Testing Tool Project Management
© 2015, WhichTestWon, a division of Anne Holland Ventures Inc. All rights reserved.
If You Don’t Do These, You Are Heading For Trouble
#TLE2015
© 2015, WhichTestWon, a division of Anne Holland Ventures Inc. All rights reserved.
Calibrate The Testing Tool Regularly
#TLE2015
AA or Blank Test
© 2015, WhichTestWon, a division of Anne Holland Ventures Inc. All rights reserved.
Filter Out Bots & Other Traffic Noise
#TLE2015
Keep the IP filter list up to date
© 2015, WhichTestWon, a division of Anne Holland Ventures Inc. All rights reserved.
Create A Framework For Outliers Then Monitor Them
#TLE2015
3 Standard Deviations
© 2015, WhichTestWon, a division of Anne Holland Ventures Inc. All rights reserved.
Check page load speed
#TLE2015
100mS increase in load time ~2% increase in bounce
© 2015, WhichTestWon, a division of Anne Holland Ventures Inc. All rights reserved.
Create a Process
#TLE2015
1 Define roles and responsibility
2 Set scope and cadence
3 Create test threshold
4 Know when to stop a test
5 Retrospective
© 2015, WhichTestWon, a division of Anne Holland Ventures Inc. All rights reserved.
Set Expectations w/ Test Spec
#TLE2015
1 Hypothesis
2 Decision KPI
3 Sample size needed
4 Experiment design
5 What to do with win?
© 2015, WhichTestWon, a division of Anne Holland Ventures Inc. All rights reserved.
Importance Of Both Quantitative And Qualitative Kpis
#TLE2015
-18% in NPS due to Drop in Promoter
© 2015, WhichTestWon, a division of Anne Holland Ventures Inc. All rights reserved.
How To Monitor A Test In Flight?
#TLE2015
1. Create a Threshold2. Don’t panic
© 2015, WhichTestWon, a division of Anne Holland Ventures Inc. All rights reserved.
Do You Have The Confidence & Power?
#TLE2015
90% Confident
80% Power
© 2015, WhichTestWon, a division of Anne Holland Ventures Inc. All rights reserved.
Why Not Peek Before Hitting Sample Size?
#TLE2015
Control Test
Order 1 2
Visits 5 5
Conversion 20% 40%
Lift 100%
Confidence 90%
Power 16%
64 samples needed
© 2015, WhichTestWon, a division of Anne Holland Ventures Inc. All rights reserved.
How Is It Trending?
#TLE2015
Look for diverging or parallel lines
© 2015, WhichTestWon, a division of Anne Holland Ventures Inc. All rights reserved.
Congrats! 15% of your work is done.
Now you have to deal with PEOPLE…
#TLE2015
© 2015, WhichTestWon, a division of Anne Holland Ventures Inc. All rights reserved.
Step 2: Involve Others
#TLE2015
© 2015, WhichTestWon, a division of Anne Holland Ventures Inc. All rights reserved.
Testing
Product
Eng
UX / Design
Ops / MerchFinance
Analyst
SEO
Informed Decision Making
Form Core Testing A-Team & Educate The Process
© 2015, WhichTestWon, a division of Anne Holland Ventures Inc. All rights reserved.
LIFT Brainstorm Sessions
#TLE2015
Structured and Focused
© 2015, WhichTestWon, a division of Anne Holland Ventures Inc. All rights reserved.
Turn Ideas Into Experiment Design
#TLE2015
Combine ideas into MVT
Element 1 Element 2
Variation 1 Variation 1
Variation 2 Variation 2
Variation 3 Variation 3
2 X 3 = 6 Variations
© 2015, WhichTestWon, a division of Anne Holland Ventures Inc. All rights reserved.
Build An Efficient Roadmap
#TLE2015
1 EVFPIE (Potential Importance Ease)
2 Room for the unexpected and HiPPOs
© 2015, WhichTestWon, a division of Anne Holland Ventures Inc. All rights reserved.
Be Transparent
#TLE2015
Communicate
Communicate
Communicate
© 2015, WhichTestWon, a division of Anne Holland Ventures Inc. All rights reserved.
Over-communicate, Share & Be Transparent
#TLE2015
WIKIEmail
Notifications
© 2015, WhichTestWon, a division of Anne Holland Ventures Inc. All rights reserved.
Discuss the results before making decisions…
#TLE2015
© 2015, WhichTestWon, a division of Anne Holland Ventures Inc. All rights reserved.
Promoting A Testing Culture – Make It Fun
#TLE2015
© 2015, WhichTestWon, a division of Anne Holland Ventures Inc. All rights reserved.
Step 3: Know What They Are
Thinking
#TLE2015
© 2015, WhichTestWon, a division of Anne Holland Ventures Inc. All rights reserved.
Who are they and what are they thinking?
Decision Makers?
#TLE2015
© 2015, WhichTestWon, a division of Anne Holland Ventures Inc. All rights reserved.
Questions To Ask
#TLE2015
1 What is the purpose of testing?
2 Who are the supporters?
3 How is the incentive structure set up?
4 Where is the testing team in the org?
© 2015, WhichTestWon, a division of Anne Holland Ventures Inc. All rights reserved.
Common Reasons People Push Back On Testing
#TLE2015
It slows me down
I’m afraid to know
I’m confident it’s
a win
© 2015, WhichTestWon, a division of Anne Holland Ventures Inc. All rights reserved.
Prep The Sirens
#TLE2015
© 2015, WhichTestWon, a division of Anne Holland Ventures Inc. All rights reserved.
Homepage Test
#TLE2015
Merchandising Team (Offers)
Design Team (Aesthetics, UX)
Sales Team($$$)
Operations Team(No Issues)
Marketing Team(Traffic)
© 2015, WhichTestWon, a division of Anne Holland Ventures Inc. All rights reserved.
Measure ROI
#TLE2015
1 Make friends at Finance Dept
2 Define measurement framework
3 Set up goals
4 Show sequential performance
© 2015, WhichTestWon, a division of Anne Holland Ventures Inc. All rights reserved.
Show them you care and make them feel important
#TLE2015
© 2015, WhichTestWon, a division of Anne Holland Ventures Inc. All rights reserved.
What’s HOT?
Get your HOUSE in order
Involve OTHERS
Know what THEY are thinking
#TLE2015
© 2015, WhichTestWon, a division of Anne Holland Ventures Inc. All rights reserved.
Continuous Innovation
#TLE2015
© 2015, WhichTestWon, a division of Anne Holland Ventures Inc. All rights reserved.
Sequential Likelihood Ratio Test – Bayesian Stats To complement Student T-Test Stats
#TLE2015
Test End (T-Test)Test End (SS)
© 2015, WhichTestWon, a division of Anne Holland Ventures Inc. All rights reserved.
Flexible Testing Platform
#TLE2015
Easy Integration With Any Testing Tools
© 2015, WhichTestWon, a division of Anne Holland Ventures Inc. All rights reserved.
Questions?
#TLE2015
© 2015, WhichTestWon, a division of Anne Holland Ventures Inc. All rights reserved. #TLE2015
Charles HuaRetailMeNotDirector of Optimization & AB Testingchua@rmn.com
© 2015, WhichTestWon, a division of Anne Holland Ventures Inc. All rights reserved. #TLE2015
Real-Life Test: Which is True About Charles?
I have 2 kids:(older kid’s birthday
month/day/year) x 2 =
(younger kid’s birthdaymonth/day/year)
Version A
At one point, I considered a career as an accountant
Version B
How good is your gut?
top related