Actionable metrics

ACTIONABLE METRICSHOW TO MEASURE WHAT

MATTERS BEFORE PRODUCT/MARKET FIT

ASH MAURYA@ashmaurya

http://www.ashmaurya.com

What is an Actionable Metric?

Ties specific, repeatable actions to observed results

Vanity Metrics

Customer Discovery(Problem/Solution Fit)

Customer Validation(Product/Market Fit)

Customer Creation(Scale)

3 Stages of a Lean Startup

How You Measure Matters

3 Rules for Actionable Metrics

1. Measure the “Right” Macro2. Create Simple Reports3. Metrics are People too

3 Rules for Actionable Metrics

1. Measure the “Right” Macro2. Create Simple Reports3. Metrics are People too

Identify Key Metrics

Acquisition

Activation

Retention

Revenue

Referral

How do users find you?

Do users have a great first experience?

Do users come back?

How do you make money?

Do users tell others?

Identify Key Metrics

Acquisition

Activation

Retention

Revenue

Referral

How do users find you?

Do users have a great first experience?

Do users come back?

How do you make money?

Do users tell others?

Before Product/Market Fit

Map Metrics to Actions

Activation Actions

Signup Download Share Stuff Tell Friends

Retention Actions

1. Revisits2. Churn - cancellations, no activity3. Loyalty - days since last visit4. Key Activity - Shared 1 new album/movie per month

3 Rules for Actionable Metrics

1. Measure the “Right” Macro2. Create Simple Reports3. Metrics are People too

Funnel Analysis - The Good

Conversion Funnel for June

1. Simple2. Visual3. Maps well to Activation Flow

Signed-up 4200 (100%)

Downloaded 3400 (81%)

Did key activity 1880 (45%)

Purchased 375 (9%)

Funnel Analysis - The Bad

How do you:1. Track long lifecycle events

Conversion Funnel for June

Signed-up 4200 (100%)

Downloaded 3400 (81%)

Did key activity 1880 (45%)

Purchased 375 (9%)

Funnel Analysis - The Bad

How do you:1. Track long lifecycle events2. Handle split tests

Conversion Funnel for June

Signed-up 4200 (100%)

Downloaded 3400 (81%)

Did key activity 1880 (45%)

Purchased 375 (9%)

Funnel Analysis - The Bad

How do you:1. Track long lifecycle events2. Handle split tests3. Measure Retention

Conversion Funnel for June

Signed-up 4200 (100%)

Downloaded 3400 (81%)

Did key activity 1880 (45%)

Purchased 375 (9%)

Funnels Alone Are Not Enough.Say Hello to the Cohort.

What is Cohort Analysis

A cohort is a group of people who share a common characteristic over a period of time e.g. join date.

Conversion Funnel for June

Signed-up 4200 (100%)

Downloaded 3400 (81%)

Did key activity 1880 (45%)

Purchased 375 (9%)

Tracking Long Lifecycle Events

Tracking Long Lifecycle Events

Week 1 Cohort Week 2 Cohort Week 3 Cohort

Signed-up 1000 (100%)

Downloaded 800 (80%)

Did key activity 500 (50%)

Purchased 100 (10%)

Signed-up 900 (100%)

Downloaded 750 (83%)

Did key activity 380 (42%)

Purchased 95 (11%)

Signed-up 1100 (100%)

Downloaded 900 (81%)

Did key activity 650 (60%)

Purchased 180 (16%)

Week 4 Cohort

Signed-up 1200 (100%)

Downloaded 950 (80%)

Did key activity 350 (30%)

Purchased 0 (0 %)

Reporting Period: June 1 - June 30 2010

Tracking Split Tests

Control - Free Trial Hypothesis A - Freemium

Signed-up 1500 (100%)

Downloaded 1200 (80%)

Did key activity 450 (30%)

Purchased 75 (5%)

Signed-up 1000 (100%)

Downloaded 800 (80%)

Did key activity 400 (40%)

Purchased 100 (10%)

Reporting Period: June 1 - June 30 2010

Tracking Split Tests

SEO Cohort: “Photo Sharing” SEO Cohort: “Cloud Photo Backup”

Signed-up 1500 (100%)

Downloaded 1200 (80%)

Did key activity 750 (50%)

Purchased 225 (15%)

Signed-up 1000 (100%)

Downloaded 800 (80%)

Did key activity 400 (40%)

Purchased 100 (10%)

Reporting Period: June 1 - June 30 2010

Tracking Retention

Month 1 Month 2 Month 3 Month 4 Month 5 Month 6 Month 7

(Joined in)Week 1 100% 10% 9% 9% 7% 7% 7%

Week 2 100% 12% 10% 10% 8% 7% ?

Week 3 100% 16% 14% 13% 12% ?

Week 4 100% 17% 15% 14% ?

Week 5 100% 20% 19% ?

Week 6 100% 22% ?

… … … … … … … …

3 Rules for Actionable Metrics

1. Measure the “Right” Macro2. Create Simple Reports3. Metrics are People too

Validate Qualitatively, Verify Quantitatively

Who are these people?

Cohort - Free Trial

Signed-up 1000 (100%)

Downloaded 800 (80%)

Did key activity 400 (40%)

Purchased 100 (10%)

Reporting Period: June 1 - June 30 2010

Who are these people?

Cohort - Free Trial

Signed-up 1000 (100%)

Downloaded 800 (80%)

Did key activity 400 (40%)

Purchased 100 (10%)

Reporting Period: June 1 - June 30 2010

Email

[email protected]

[email protected]

[email protected]

[email protected]

…

[email protected]

Failed Downloads List

How do I create these reports?

3rd Party Tools versus Homegrown

3rd PartyFunnel

AnalysisRetention Cohorts

FunnelCohorts

Metrics to People

KISSMetrics Ad-hoc No No Not easy

MixPanel Static Yes No Not easy

Google Analytics

Static/Limited No No No

3rd Party Tools versus Homegrown

3rd PartyFunnel

AnalysisRetention Cohorts

FunnelCohorts

Metrics to People

KISSMetrics Ad-hoc No No Not easy

MixPanel Static Yes No Not easy

Google Analytics

Static/Limited No No No

Homegrown Comments

Index Cards Manual and time consuming process

SQL + Excel Complex queries and pivot tables

Events Database Separate database plus additional code

What’s Next?

Acquisition

Activation

Retention

Revenue

Referral

How do users find you?

Do users have a great first experience?

Do users come back?

How do you make money?

Do users tell others?

After Product/Market Fit