MARK HALVORSON • HEAD OF INTERACTIVE • ATLASSIAN SOFTWARE • @halv0112 A/B Testing How to break things and fail fast… without breaking things
Optimizely Experience Customer Story - Atlassian
Nov 01, 2014
This presentation was given by Mark Halvorson at the Optimizely Experience, Silicon Valley.
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
MARK HALVORSON • HEAD OF INTERACTIVE • ATLASSIAN SOFTWARE • @halv0112
A/B TestingHow to break things and fail fast…
without breaking things
We help great teams build better software, together. About Atlassian
• Products include: JIRA, Confluence, Bitbucket, HipChat…• Founded in 2002• Over 37,000 Customers• Over 700 employees globally• No sales people !
!
!
Think Big, Act Small, Fail Fast, Learn Rapidly.
" L E A N S O F T WA R E D E V E L O P M E N T ” B Y M A RY A N D T O M P O P P E N D I E C K ( 2 0 0 3 )
”“
Central “Growth Hacking” team to support: Atlassian A/B Testing is cross functional
• Email • Product Management • Customer Platform • Customer Advocates• Internal Systems
• Leads• Product Marketing• Support• Bitbucket• HipChat
SCHEDULE TO AVOID CONFLICTS
BUILD OUT GOOD CANDIDATES
TRIAGE BASED ON POTENTIAL IMPACT
COLLECT GREAT IDEAS
ANALYZE RESULTS
RUN EXPERIMENTS
Our Process
SCHEDULE TO AVOID CONFLICTS
BUILD OUT GOOD CANDIDATES
TRIAGE BASED ON POTENTIAL IMPACT
COLLECT GREAT IDEAS
ANALYZE RESULTS
RUN EXPERIMENTS
Our Process
Where do experiment ideas
come from?
Anywhere.
What are you are trying to improve?
Blimp Dashboard
JIRA – Ticket
Types of tests
• Pebbles• Change in copy or CTA presentation
• Rocks• New page (e.g. no Dev Tools family page)
• Boulders• Simplified order form
Examples
Reduce FrictionA B
Increase VisibilityA B
SimplifyA B
SCHEDULE TO AVOID CONFLICTS
BUILD OUT GOOD CANDIDATES
TRIAGE BASED ON POTENTIAL IMPACT
COLLECT GREAT IDEAS
ANALYZE RESULTS
RUN EXPERIMENTS
Our Process
What are you are trying to improve?
Blimp Dashboard
JIRA – Prioritized Backlog
JIRA – Ticket
JIRA – Ticket
Experiment Illuminati
SCHEDULE TO AVOID CONFLICTS
BUILD OUT GOOD CANDIDATES
TRIAGE BASED ON POTENTIAL IMPACT
COLLECT GREAT IDEAS
ANALYZE RESULTS
RUN EXPERIMENTS
Our Process
Optimizely – Build
Optimizely – Build
Prepare for results.
Confluence – Corresponding Test Page
Confluence – Corresponding Test Page
Confluence – Corresponding Test Page
Confluence – Corresponding Test Page
Confluence – Corresponding Test Page
Confluence – Corresponding Test Page
Confluence – Corresponding Test Page
Confluence – Corresponding Test Page
Confluence – Corresponding Test Page
Confluence – Corresponding Test Page
Confluence – Corresponding Test Page
Confluence – Corresponding Test Page
SCHEDULE TO AVOID CONFLICTS
BUILD OUT GOOD CANDIDATES
TRIAGE BASED ON POTENTIAL IMPACT
COLLECT GREAT IDEAS
ANALYZE RESULTS
RUN EXPERIMENTS
Our Process
JIRA – Prioritized Backlog
SCHEDULE TO AVOID CONFLICTS
BUILD OUT GOOD CANDIDATES
TRIAGE BASED ON POTENTIAL IMPACT
COLLECT GREAT IDEAS
ANALYZE RESULTS
RUN EXPERIMENTS
Our Process
Optimizely – Start Experiment
Optimizely – Project Code
Magnolia CMS – Include Optimizely Checkbox
Magnolia CMS – Include Optimizely Checkbox
Confluence – Corresponding Test Page
SCHEDULE TO AVOID CONFLICTS
BUILD OUT GOOD CANDIDATES
TRIAGE BASED ON POTENTIAL IMPACT
COLLECT GREAT IDEAS
ANALYZE RESULTS
RUN EXPERIMENTS
Our Process
Optimizely – Results
What are you are trying to improve?
Blimp Dashboard
Optimizely – Results
Confluence – Corresponding Test Page
Confluence – Results Archive
Our Test Philosophy
Our Test Philosophy
• Everything affecting conversion is tested
Our Test Philosophy
• Everything affecting conversion is tested
• Be aggressive
Our Test Philosophy
• Everything affecting conversion is tested
• Be aggressive
• Regular cadence, no traffic wasted
Our Test Philosophy
• Everything affecting conversion is tested
• Be aggressive
• Regular cadence, no traffic wasted
• Never run a test that you wouldn’t want to win
Key Take Aways
Key Take Aways
• Ideas are everywhere collect as many as you can, prioritize based on impact.
Key Take Aways
• Ideas are everywhere collect as many as you can, prioritize based on impact.
• Use ticket number everywhere so everyone is referring to the same test.
Key Take Aways
• Ideas are everywhere collect as many as you can, prioritize based on impact.
• Use ticket number everywhere so everyone is referring to the same test.
• Involve stakeholders and developers… and get approvals.
Key Take Aways
• Ideas are everywhere collect as many as you can, prioritize based on impact.
• Use ticket number everywhere so everyone is referring to the same test.
• Involve stakeholders and developers… and get approvals.
Thank you!
MARK HALVORSON • HEAD OF INTERACTIVE • ATLASSIAN SOFTWARE • @halv0112
Related Documents
