Melbourne Business School - mba talk october 14 - croll - 40m - lean analytics

Lean Analytics

Melbourne Business School October 14, 2015

@acroll

Percent of businesses that fail

Year 1 Year 2 Year 3 Year 4 Year 5 Year 6 Year 7 Year 8 Year 9 Year 10

71%69%66%63%60%55%50%44%36%25%

http://www.statisticbrain.com/startup-failure-by-industry/

Still operating after 4 yearsFinance Insurance and Real Estate

Education and HealthAgriculture

ServicesWholesale

MiningManufacturing

ConstructionRetail

Transportation Communication and UtilitiesInformation 37%

45%

47%

47%

49%

51%

54%

55%

56%

56%

58%

http://www.statisticbrain.com/startup-failure-by-industry/

Don’t sell what you can make. Make what you can sell.

Kevin Costner is a lousy entrepreneur.

The core of Lean is iteration.

Unfortunately, we’re all liars.

Everyone’s idea is the best right?

People love this part!

(but that’s not always a good thing)

This is where things fall apart.

No data, no learning.

Most startups don’t know what they’ll be when they grow up.

Hotmailwas a database company

Flickrwas going to be an MMO

Twitter was a podcasting company

Autodesk made desktop automation

Paypalfirst built for Palmpilots

Freshbookswas invoicing for a web design firm

Wikipedia was to be written by experts only

Mitelwas a lawnmower company

Analytics can help.

Analytics is the measurement of movement towards your business

goals.

In a startup, the purpose of analytics is to iterate to product/market fit

before the money runs out.

We’re bad at data.

What are the chances the other is a boy?

BB BG

GB GG

2 of 3 (66%) are boys.

GB GG BG

A good metric is:

Understandable

If you’re busy explaining the data, you won’t be busy acting on it.

Comparative

Comparison is context.

A ratio or rate

The only way to measure change and roll up the tension between two metrics (MPH)

Behaviorchanging

What will you do differently based on the results you collect?

The simplest rule

badmetric.

If a metric won’t change how you behave, it’s a

h"p://www.flickr.com/photos/circasassy/7858155676/

Metrics help you know yourself.

Acquisition

Hybrid

Loyalty

70%of retailers

20%of retailers

10%of retailers

You are just like

Customers that buy >1x in 90d

Once

2-2.5per year

>2.5per year

Your customers will buy from you

Then you are in this mode

1-15%

15-30%

>30%

Low acquisition cost, high checkout

Increasing return rates, market share

Loyalty, selection, inventory size

Focus on

(Thanks to Kevin Hillstrom for this.)

MayAprMarFeb

Slicing and dicing data

Jan

0

5,000

Activ

e use

rs

Cohort: Comparison of similar groups along a timeline. (this is the April cohort)

A/B test: Changing one thing (i.e. color) and measuring the result (i.e. revenue.)

Multivariateanalysis Changing several things at once to see which correlates with a result.

☀☁☀☁

Segment: Cross-sectional

comparison of all people divided by

some attribute (age, gender, etc.)

☀

☁

Which of these two companies is doing better?

  January February March April May

Rev/customer $5.00 $4.50 $4.33 $4.25 $4.50Is this company growing or stagnating?

Cohort 1 2 3 4 5

January $5 $3 $2 $1 $0.5

February $6 $4 $2 $1

March $7 $6 $5

April   $8 $7

May       $9

How about this one?

Cohort 1 2 3 4 5

January $5 $3 $2 $1 $0.5

February $6 $4 $2 $1  

March $7 $6 $5    

April $8 $7      

May $9        

Averages $7 $5 $3 $1 $0.5

Look at the same data in cohorts

The Lean Analytics framework.

Eric’s three engines of growth

Virality

Make people invite friends.

How many they tell, how fast they

tell them.

Price

Spend money to get customers.

Customers are worth more than

they cost.

Stickiness

Keep people coming back.

Approach

Get customers faster than you

lose them.

Math that matters

Dave’s Pirate MetricsAARRR

AcquisitionHow do your users become aware of you?

SEO, SEM, widgets, email, PR, campaigns, blogs ...

ActivationDo drive-by visitors subscribe, use, etc?

Features, design, tone, compensation, affirmation ...

RetentionDoes a one-time user become engaged?

Notifications, alerts, reminders, emails, updates...

RevenueDo you make money from user activity?

Transactions, clicks, subscriptions, DLC, analytics...

ReferralDo users promote your product?

Email, widgets, campaigns, likes, RTs, affiliates...

Stage

EMPATHY I’ve found a real, poorly-met need that a reachable market faces.

STICKINESS I’ve figured out how to solve the problem in a way they will keep using and pay for.

VIRALITY I’ve found ways to get them to tell their friends, either intrinsically or through incentives.

REVENUE The users and features fuel growth organically and artificially.

SCALE I’ve found a sustainable, scalable business with the right margins in a healthy ecosystem.

GateTh

e fiv

e st

ages

Six business model archetypes.

E-commerce SaaS MediaMobileapp

User-gencontent

2-sidedmarket

The business you’re in

(Which means eye charts like these.)

Customer Acquisition Cost

paid direct search wom inherent virality

VISITOR

Freemium/trial offer

Enrollment

User

Disengaged User

Cancel

Freemium churn

Engaged User

Free user disengagement

Reactivate

Cancel

Trial abandonment rate

Invite Others

Paying Customer

Reactivationrate

Paid conversion

FORMER USERS

User Lifetime Value

Reactivate

FORMER CUSTOMERS

Customer Lifetime Value

Viral coefficientViral rate

Resolution

Support data

Account Cancelled Billing Info Exp.

Paid Churn Rate

Tiering

Capacity Limit

Upselling rate Upselling

Disengaged DissatisfiedTrial Over

Model + Stage = One Metric That Matters.

One Metric That Matters.

The business you’re in

E-Com SaaS Mobile 2-Sided Media UCG

Empathy

Stickiness

Virality

Revenue

ScaleThe

stag

e yo

u’re

at

Really? Just one?

Yes, one.

In a startup, focus is hard to achieve.

Having only one metric addresses this problem.

www.theeastsiderla.com

Metrics are like squeeze toys.

http://www.flickr.com/photos/connortarter/4791605202/

Empathy

Stickiness

Virality

Revenue

Scale

E-commerce SaaS MediaMobile

appUser-gencontent

2-sidedmarket

Interviews; qualitative results; quantitative scoring; surveys

Loyalty, conversion

CAC, shares, reactivation

Transaction, CLV

Affiliates, white-label

Engagement, churn

Inherent virality, CAC

Upselling, CAC, CLV

API, magic #, mktplace

Content, spam

Invites, sharing

Ads, donations

Analytics, user data

Inventory, listings

SEM, sharing

Transactions, commission

Other verticals

(Money from transactions)

Downloads, churn, virality

WoM, app ratings, CAC

CLV, ARPDAU

Spinoffs, publishers

(Money from active users)

Traffic, visits, returns

Content virality, SEM

CPE, affiliate %, eyeballs

Syndication, licenses

(Money from ad clicks)

Better: bit.ly/BigLeanTable

Drawing some lines in the sand.

A company loses a quarter of its customers every year.

Is this good or bad?

Not knowing what normal is makes you do stupid things.

Baseline: 5-7% growth a week

“A good growth rate during YC is 5-7% a week,” he says. “If you can hit 10% a week you're doing exceptionally well. If you can only manage 1%, it's a sign you haven't yet figured out what you're doing.” At revenue stage, measure growth in revenue. Before that, measure growth in active users.

Paul Graham, Y Combinator

• Are there enough people who really care enough to sustain a 5% growth rate?

• Don’t strive for a 5% growth at the expense of really understanding your customers and building a meaningful solution

• Once you’re a pre-revenue startup at or near product/market fit, you should have 5% growth of active users each week

• Once you’re generating revenues, they should grow at 5% a week

It’s oxygenYou need customers to keep learning It’s a substitute for solvency

Photo by Paul Miller on Flickr. https://w

ww

.flickr.com/photos/94674772@

N03/8788576498

Baseline: 10% visitor engagement/day

Fred Wilson’s social ratios

30% of users/month use web or mobile app

10% of users/day use web or mobile app

1% of users/day use it concurrently

Baseline: 2-5% monthly churn• The best SaaS get 1.5% - 3% a month. They have multiple Ph.D’s

on the job.• Get below a 5% monthly churn rate before you know you’ve got a

business that’s ready to grow (Mark MacLeod) and around 2% before you really step on the gas (David Skok)

• Last-ditch appeals and reactivation can have a big impact. Facebook’s “don’t leave” reduces attrition by 7%.

Baseline: Calculating customer lifetime

25%monthly churn

100/25=4The average

customer lasts 4 months

5%monthly churn

100/5=20The average

customer lasts 20 months

2%monthly churn

100/2=50The average

customer lasts 50 months

Baseline: CAC under 1/3 of CLV• CLV is wrong. CAC Is probably wrong, too.• Time kills all plans: It’ll take a long time to find

out whether your churn and revenue projections are right

• Cashflow: You’re basically “loaning” the customer money between acquisition and CLV.

• It keeps you honest: Limiting yourself to a CAC of only a third of your CLV will forces you to verify costs sooner.

Lifetime of 20 mo.$30/mo. per

customer$600 CLV

$200 CACNow segment those users!

1/3 spend

Who is worth more?

Today

A Lifetime:$200

Roberto Medri, Etsy

B Lifetime:$200

Visits

The Lean Analytics cycle

Draw a new linePivot orgive up

Try again

Success!

Did we move the needle?

Measure the results

Make changes in production

Design a test

Hypothesis

With data:find a

commonality

Without data: make a good

guess

Find a potential improvement

Draw a linePick a KPI

Do AirBnB hosts get more business if their property is professionally photographed?

Gut instinct (hypothesis)Professional photography helps AirBnB’s business

Candidate solution (MVP)20 field photographers posing as employees

Measure the resultsCompare photographed listings to a control group

Make a decision Launch photography as a new feature for all hosts

5,000 shoots per month by February 2012

Hang on a second.

Gut instinct (hypothesis)Professional photography helps AirBnB’s business

SRSLY?

Draw a new linePivot orgive up

Try again

Success!

Did we move the needle?

Measure the results

Make changes in production

Design a test

Hypothesis

With data:find a

commonality

Without data: make a good

guess

Find a potential improvement

Draw a linePick a KPI

“Gee, those houses that do well look really

nice.”

Maybe it’s the camera.

“Computer: What do all the

highly rented houses have in

common?”

Camera model.

With data:find a commonality

Without data: make a good guess

Landing page design A/B testing

Cohort analysis General analytics

URL shortening

Funnel analytics

Influencer Marketing

Publisher analytics

SaaS analytics

Gaming analytics

User interaction Customer satisfaction KPI dashboardsUser segmentation

User analytics Spying on users

When you’re a startup your goal is to find a sustainable,

repeatable business model.

When you’re a big company your goal is to perpetuate one.

In a startup, the purpose of analytics is to iterate to product/market fit

before the money runs out.

In a big company, analytics replaces opinion with fact.

Arbitron and radio data

Times a song in “heavy rotation” is played daily

2007 2012

266

Conclusions

“The most important figures that one needs for management are unknown or unknowable, but successful management must nevertheless take account of them.”

Lloyd S. Nelson

Pic by Twodolla on Flickr. http://www.flickr.com/photos/twodolla/3168857844

ARCHIMEDES HAD TAKEN

BATHS BEFORE.

Once, a leader convinced others in the absence of data.

Now, a leader knows what questions to ask.

Alistair Croll [email protected] @acroll

Ben Yoskovitz [email protected] @byosko