Experimental Product Development

i290 lean/agile product management unit 6: experimental product development

@jezhumble https://lapm.continuousdelivery.com/

humble@berkeley.edu

This work © 2015-2016 Jez Humble Licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.

identify experiments to test hypotheses

understand how to do outcome-based planning

describe hypothesis-driven development

understand why small batches are important

define A/B testing and the culture it enables

learning outcomes

EpicTheme

Story

impact mapping

Gojko Adzic, Impact Mapping

working backwards

http://www.allthingsdistributed.com/2006/11/working_backwards.html

@jezhumbleJeff Gothelf “Better product definition with Lean UX and Design” http://bit.ly/TylT6A

hypothesis-driven delivery

We believe that

[building this feature]

[for these people]

will achieve [this outcome].

We will know we are successful when we see [this signal from the market].

COST OF EXPERIMENTS

8

Production Software

SPEED

COST

new services

feasibility spike

service substitution

integration

Quantitative forecasting

real-time price experiment

Data sampling and modeling tests

Sketches & Paper Prototypes

Interactive Prototype

Software demo

Interviews & surveys

micro-niche

Wizard of Oz

VIABILITY (BUSINESS) | DESIRABILITY (CUSTOMER) | FEASIBILITY (TECH)

exercise

• choose a hypothesis from your assignment

• design an experiment to test your hypothesis

• what do you expect the results to be?

• what result will confirm your hypothesis?

• what result will disprove your hypothesis?

• how soon can we get the result?

“Etsy’s Product Development with Continuous Experimentation” Frank Harris and Nellwyn Thomas | http://bit.ly/19Z5izI

“Etsy’s Product Development with Continuous Experimentation” Frank Harris and Nellwyn Thomas | http://bit.ly/19Z5izI

“Etsy’s Product Development with Continuous Experimentation” Frank Harris and Nellwyn Thomas | http://bit.ly/19Z5izI

Jon Jenkins, “Velocity Culture, The Unmet Challenge in Ops” 2011 | http://bit.ly/1vJo1Ya

do less

“Evaluating well-designed and executed experiments that were designed to improve a key metric, only about 1/3 were successful at improving the key metric!”

“Online Experimentation at Microsoft” | Kohavi et al | http://stanford.io/130uW6X

“I think building this culture is the key to innovation. Creativity must flow from everywhere. Whether you are a summer intern or the CTO, any good idea must be able to seek an objective test, preferably a test that exposes the idea to real customers. Everyone must be able to experiment, learn, and iterate.”

http://glinden.blogspot.com/2006/04/early-amazon-shopping-cart.html

@jezhumble

WORK IN SMALL

BATCHES

WORK IN SMALL

BATCHES

WORK IN SMALL

BATCHES

WORK IN SMALL

BATCHES

WORK IN SMALL

BATCHES

WORK IN SMALL

BATCHES

WORK IN SMALL

BATCHES

WORK IN SMALL

BATCHES

WORK IN SMALL

BATCHES

higher quality

you can stop at any time with a working system

faster feedback (assuming people pay attention)

higher motivation

less rework

working in small batches

Don Reinertsen, Principles of Product Development Flow, ch5.

further reading

https://www.infoq.com/presentations/controlled-experiments

http://xp123.com/articles/resources-on-set-based-design/

Tom DeMarco & Tim Lister, Waltzing with Bears

Humble et al, Lean Enterprise ch 9

Gojko Adzic, Impact Mapping