Anatomy of a Feature Release How integrating User Testing, Data Driven Development & A/B Testing into your development lifecycle means you never have to give up on a new feature. (well, almost never)
Anatomy of a Feature Release
Nov 29, 2014
From ProductCampLA - March 2014. By integrating User Testing, Data Driven Development and A/B Testing into your development lifecycle you can make every feature release a success (well, almost every one).
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Anatomy of a Feature Release
How integrating User Testing, Data Driven Development & A/B Testing into your development lifecycle means you never have to give up on a new feature.
(well, almost never)
Every feature counts
Feature A
Feature B
Feature C
Feature D
Feature E
Before
IDEA Build Release Roll Back
NowIDEA Design
User test
Prototype User test
BuildRelease @
5%Analyze
QAFix bugsKPIs
down
KPIsup
No bugs*
Analyze discrep-ancies
Optimize
Release @ 50%
A/BTest
ROLL OUT!
* Well, no relevant bugs
(slowly)
Increase to 50/50
Test winner vs
control
DesignIDEA Design
User test
Left vs Right Form Open vs Closed Tabs vs Arrow Colors & Style
=
PrototypePrototype User
test
Umm. Yeah. We skipped this step.
Big mistake.
Build & Release (Round 1)Build
Release @ 5%
Analyze
QA Fix bugs KPIsdown
Fix & Release (Round 2)Release @
5%Analyze
QA (again)
Fix bugs(again)
KPIsdown
Tweak & Release (Round 3)No
bugs*Deep Dive
AnalysisOptimize
Release @ 50%
A/B Test
Winner Winner
(against control)
Is it this?
Or this?
* Well, no relevant bugs
So we optimized…These (along with a few others) VS. This
Winner Winner!
And a
12% lift!!
No one likes a Frankensite
RecapIDEA Design
User test
Prototype User test
BuildRelease @
5%Analyze
QAFix bugsKPIs
down
KPIsup
No bugs*
Analyze discrep-ancies
Optimize
Release @ 50%
A/BTest
ROLL OUT!
* Well, no relevant bugs
(slowly)
Increase to 50/50
Test winner vs
control
How to make data driven feature releases a reality
Set expectations• A data driven release can take a long time! A decent estimation for a major new
feature is ½ the time it took build.
Create a process• How will you test? When will decisions be made? When will fixes & tweaks go out?
Have the tools• Make sure your testing framework can support this. We use an internal system for
major splits and Optimizely for front end tweaks / optimizations.
Involve the team• Involve the whole team when analyzing results - sometimes the problem is obvious,
sometimes it is buried. You need a multitude of viewpoints to make this work.
Track EVERYTHING• Have tracking in place for all KPIs and relevant user interactions
Relax, you’ll make it happen
@zorstag pretty much everywhere Even LinkedIn: www.linkedin.com/in/zorstag/
Model Credits: Sundance & Cassie
Related Documents
