Measure or Die

Measure or Die

Neil McCarthyKen Pascual

Justifying featuresExecution decisionsDetermine success

HOW DATA HELPS US PRIORITIZEStep 1: Justifying Ideas

Previous signup flow experiment

Clues from the past

Brainstorming

Idea!

Does this align with our goals?

• Vision: Yammer is a primary communication tool that users use to get work done.

• Goal: Grow the number of users that use the product.

• Problem: We lose too many potential users in our signup flow.

How important is this?

big drop!

And HALF of our invites come from the signup flow!

Conclusion: This seems important.

Why justify?

HOW DATA HELPS US DETERMINE THE SUCCESS OF THE PROJECT

Step 2: Determining Success

What’s your hypothesis?

What we came up with…

If we move the email confirmation step to the end of the signup flow,then more users will make it through the signup flow and more users will be retained.

Secondary objective: Kill the Desktop app

We need another hypothesis

“If we remove the prompt to install the desktop app, then we will lose an acceptable amount of engagement”.

A/B Test Results: Killing the Desktop app prompt

A/B Test Results: Killing the Desktop app prompt

Interpretations

• Hypothesis disproved (in a good way).• Users find a way to post.

Results!

A/B Test Results: Moving the email activation step

A/B Test Results: Moving the email activation step

Interpretations

• Users saw the steps in a different order, and dropped off in similar percentages, BUT– Less users saw the prompt to “check email”

because it was at the end.– Thus, less users activated via email.– Thus, less users had the opportunity to engage.

Why we chose this example

How do you do this?

• Gotta log data!• Choose your own metrics• Data driven vs. data informed• Build your processes around data• How much PMs should know about statistics.

Q & A

Thanks for coming!

• Neil McCarthy, @the_neil• Ken Pascual, @vorped• Adam Zadikoff (original PM for this project),

@zadikafka

• Test results from the first signup flow test: https://medium.com/yammer-product/13eeb6435a73

• These slides on Slideshare coming soon!