In-Market Experiments: The Science of Placing Small Bets Fast

i

IN-MARKET EXPERIMENTS:THE SCIENCE OF PLACINGSMALL BETS FAST

January 2013

Meeting Logistics

› 35 minute presentation

› 25 minutes for Q&A

› Submit questions through Q&A widget at anytime—some answered during webinar and some post-webinar

› Slides and replay of event available after webinar

› Download our first webinar, Design Thinking for the Business Case: http://peerinsight.com/downloads/dt4bcwebinar.pdf

ABOUT TODAY’S WEBINAR

1

Tim Ogilvie is co-founder of Peer Insight. He is keen on helping large organizations create new experiences, services, and business models.

Tim is co-author of Designing for Growth but spends most of his time practicing in the field, not writing or teaching. Tim is passionate about entrepreneurship and placing small bets fast.

Jessica Dugan is a Senior Design Consultant at Peer Insight with a passion for people and their stories. Drawing on her background in design research and strategy, Jessica uses this passion to collaborate in the creation of innovative services that improve the quality of the life. She holds a Masters of Design and MBA from the Institute of Design in Chicago.

TODAY’S PRESENTERS

2

HOW DO LEAPS OF FAITH OCCUR?

Progressive: Rate Comparison “Ticker” (2001)

HP: Snapfish 1-hour print (2006)

The Hartford: FleetAhead (2009)

AARP: Life Reimagined (2011)

Will side-by-side quotes drive away customers we want to win? If we refer our

customer to a nearby retail photo finisher, do we lose her for life?

Will our fleet customers trade privacy for lower insurance premiums?

If we offer a different kind of service, can we expand our customer base?

3

Asked to describe design, Tim Brennan of Apple’s Creative Services drew the following picture:

DESIGN THINKING IS OFTEN TREATED AS A MYSTERY

? $

4

WE SEE DESIGN THINKING AS A PROBLEM-SOLVING PROCESS

What is? What if? What wows? What works?? $

5

OUR BUSINESS CASE METHOD FOLLOWS FIVE STEPS

*Adapted from Rita McGrath, “Discovery-driven Planning”

1. Formulate a novel value proposition

2. Define future success as an income statement*

3. Spell out the assumptions

› What would have to be true?

4. Test the make-or-break assumptions

› Thought experiments

› In-market experiments

5. Based on test results, refine the model, refine the value proposition, or re-assess the investment potential of this opportunity

6

1. Formulate a novel value proposition

2. Define future success as an income statement*

3. Spell out the assumptions

› What would have to be true?

4.

5. Based on test results, refine the model, refine the value proposition, or re-assess the investment potential of this opportunity

4. Test the make-or-break assumptions

› Thought experiments

› In-market experiments

OUR BUSINESS CASE METHOD FOLLOWS FIVE STEPS

*Adapted from Rita McGrath, “Discovery-driven Planning”

7

Thought Experiments

Hypothetical experiments that use logic and existing data.

› Envisioned experiences, require imagination

› Set in the future

› 2D and 3D

In-Market Experiments

Experiments conducted in the marketplace with actual partners and customers.

› Live experiences, meant to feel real (even if some elements are faked)

› Set in the present

› 4D, persist over time (even in brief)

THERE ARE TWO WAYS TO TEST ASSUMPTIONS

8

Progressive: Rate Comparison “Ticker” (2001)

HP: Snapfish 1-hour print (2006)

The Hartford: FleetAhead (2009)

AARP: Life Reimagined (2011)

REMEMBER THESE LEAPS OF FAITH?

Will side-by-side quotes drive away customers we want to win? If we refer our

customer to a nearby retail photo finisher, do we lose her for life?

Will our fleet customers trade privacy for lower insurance premiums?

If we offer a different kind of service, can we expand our customer base?

9

tantalizingpossibility

A B

PARALYSIS

what lies beneath

chasm of unintended consequences

potentialreward

POUNCE

A B

PARALYSIS POUNCE

ANOTHER INNOVATOR’S DILEMMA

10

SIX SINS OF IN-MARKET EXPERIMENTS

1. Hi-Fidelity Believing it has to look and feel real/polished

2. Quantifying Thinking it must yield quantitatively significant (and repeatable) results

3. Going Solo Using only your firm’s resources and IP to stand up the experiment—even if it takes longer

4. Business Model Next Focusing on the user experience and technical solution now and worrying about defensibility and scalability later

5. IP Protection Keeping all the ideas concealed until they are patent-protected

6. 100% Belief Committing to a specific vision of the solution and looking only for signs of success

11

tantalizingpossibility

A B

PARALYSIS

what lies beneath

chasm of unintended consequences

potentialreward

POUNCE

A B

PARALYSIS POUNCE

TWO POSSIBLE RESULTS – BOTH BAD

67%Unexplored opportunities

33%Large bets placed slowly (too-big-to-fail)

12

tantalizingpossibility

PROBE

what lies beneath

chasm of unintended consequences

potentialreward

B1 B2A B

PARALYSIS POUNCE

THE REMEDY? IN-MARKET EXPERIMENTS

13

SIN #1: HI-FIDELITY

Hi-Fidelity

Quantifying

Going Solo

Business Model Next

IP Protection

100% Belief

Low-Fidelity Fake-it-before-you-make-it

instead of try this

14

INSTEAD OF HI-FIDELITY TRY LOW-FIDELITY

Key Assumption

Customers who see a slightly lower price from a competitor will still favor Progressive.

PROGRESSIVE

484AM. FAMILY

562000

STATE FARM

525ALLSTATE

644000000 000

15

SIN #2: QUANTIFYING

Hi-Fidelity

Quantifying

Going Solo

Business Model Next

IP Protection

100% Belief

Build One (then 10, then 100)

instead of try this

16

INSTEAD OF QUANTIFYING TRY BUILD ONE

Key Assumption

Customers who try the kiosk for 2-hour service will return to Snapfish.

17

INSTEAD OF QUANTIFYING TRY BUILD ONE

Tool: Alpha-Beta Planning Table

Number  of  Installations

Number  of  Participants

Dollar  Investment

Build  Time

Test  Time

Fulfillment  Partners

Testing  What?

Is  it  Scalable?

Is  there  a  Hard  Stop?

What  is  the  Feature  Set?

Is  the  Technology  viable? No

none

Key  assumptions

No

Yes

Hypothesized  Features

0

<$500

5  days 30  days

3  weeks

Beta  Feature  Set  +

Same

...

...

IN-­‐MARKET  EXPERIMENTS

Yes  (back-­‐end  platform+year  1  biz  model)

Yes

Same  as  Beta,  plus

NoFull  Scale

No

Specific,  Limited  Features

Yes

No

Key  assumptions

Small  only

5  months

9  weeks

Branded,  if  possible

150  days  (+option  to  extend)

Yes

Initial  Feature  Set  

Full  Initial  Feature  Set

No

COMMERCIAL  LAUNCH

30X

30X

$5M+

DETAILS ALPHA

2  sites

$750k-­‐$1.5M

150-­‐200

THOUGHT  EXPERIMENTS

15-­‐20

<$500k

10  sites

BETA

0  sites

18

SIN #3: GOING SOLO

Hi-Fidelity

Quantifying

Going Solo

Business Model Next

IP Protection

100% Belief

Other People’s Money (and IP)

instead of try this

19

INSTEAD OF GOING SOLO TRY OTHER PEOPLE’S MONEY

Key Assumption

Fleet customers will incur the inconvenience of telematics to get a lower premium.

20

INSTEAD OF GOING SOLO TRY OTHER PEOPLE’S MONEY

Key Assumption

Customers will engage in peer-to-peer mentoring in an environment hosted by AARP and its partners.

21

SIN #4: BUSINESS MODEL NEXT

Hi-Fidelity

Quantifying

Going Solo

Business Model Next

IP Protection

100% Belief

Control Points Nowinstead of try this

22

UBIQUITOUS CONSTRAINED

Free, available to all Commodity

Many providers exist

Several providers exist

Possible to observe + copy

Branded + expensive to copy

Producible by only a few firms

Protected by contractual secrecy

Protected by patent

Protected by concealment

Brand

Device

Data transmission

Display/cues to drivers

Display/cues to manager

Historical data

Customized interventions

Re-pricing algorithms

Revenue sharing model

Architecture control

INSTEAD OF BUSINESS MODEL NEXT TRY CONTROL POINTS NOW

Tool: Control Point Grid

?

?

?

23

OTHER COMMON SINS

Hi-Fidelity

Quantifying

Going Solo

Business Model Next

IP Protection

100% Belief

IP Connection Services IP table; 3-ring template

instead of try this

Innovation Accounting Split tests Reverse income statement

instead of try this

24

RECAP: SIX SINS OF IN-MARKET EXPERIMENTS

1. Hi-Fidelity

2. Quantifying

3. Going Solo

4. Business Model Next

5. IP Protection

6. 100% Belief

25

BUT THEN THE WORLD LOOKS LIKE THIS

tantalizingpossibility

A B

PARALYSIS

what lies beneath

chasm of unintended consequences

potentialreward

POUNCE

A B

PARALYSIS POUNCE

26

INSTEAD: USE IN-MARKET EXPERIMENTS

the science of placing small bets fast

Hi-Fidelity

Quantifying

Going Solo

Business Model Next

IP Protection

100% Belief

instead of

Low-Fidelity

Build One

Other People’s Money

Control Points Now

IP Connection

Innovation Accounting

try this using these tools!

Key assumptions list

Alpha-Beta table

Control point grid

Services IP table 3-ring template

Split tests Reverse income statement

27

IN-MARKET EXPERIMENTS IN PERSPECTIVE

IN-MARKET EXPERIMENTS

TECHNICAL BUILD + QUANTITATIVE RESEARCH

WHAT HOW“building the right thing” “building things right”

TIM OGILVIE [email protected] 703 314 3123

JESSICA DUGAN [email protected] 312 532 8729

CHECK OUT OUR BLOGpeerinsight.com/musings

FOLLOW US ON TWITTER@peerinsight

Thank you!