Designing and Optimizing the DNA of a Killer App by Dan Olsen - Startonomics

Copyright © 2008 Olsen Solutions LLC

Designing & OptimizingDesigning & Optimizingthe DNA of a Killer Appthe DNA of a Killer AppDan Olsen, CEO & Founder, YourVersionDan Olsen, CEO & Founder, YourVersionStartonomics Startonomics ∙∙ San Francisco San Francisco ∙∙ Oct 2, 2008 Oct 2, 2008

Copyright Copyright ©© 2008 Olsen Solutions LLC2008 Olsen Solutions LLC

Building the DNA of a Killer App with Building the DNA of a Killer App with Metrics and OptimizationMetrics and Optimization

How to integrate metrics & optimizationHow to integrate metrics & optimization

Using metrics to optimize value creationUsing metrics to optimize value creationBusiness ValueBusiness Value

Customer ValueCustomer Value

UI DesignUI Design

Illustrate with real world examplesIllustrate with real world examples


Adding Metrics and Optimization to your Product Process

PlanPlan

DesignDesign

DevelopDevelop

BusinessObjectives

ProductObjectives

Prioritized Feature List

Scoping

Requirements & Design

Code Test Launch

Site Level

Feature Level

OptimizeOptimize Metrics & User Feedback


Optimization through Iteration:Optimization through Iteration:Continuous ImprovementContinuous Improvement

Measurethe metric

Analyzethe metric

Identify top opportunitiesto improve

Design & develop the enhancement

Launch theenhancement

Learning

Gaining knowledge:

• Market

• Customer

• Domain

• Usability


OptimizingOptimizingBusiness Value Creation Business Value Creation

with Metricswith Metrics


Approaching Business as an Approaching Business as an Optimization ExerciseOptimization Exercise

Given reality as it exists today,Given reality as it exists today,

optimize our business resultsoptimize our business results

subject to our resource constraints.subject to our resource constraints.

Profit = Profit = RevenueRevenue ‐‐ CostCost

Unique VisitorsUnique Visitors x x Ad Revenue per VisitorAd Revenue per Visitor

Impressions/VisitorImpressions/Visitor x Effective CPM / 1000x Effective CPM / 1000

Visits/Visitor x Pageviews/Visit x Impressions/PVVisits/Visitor x Pageviews/Visit x Impressions/PV

New VisitorsNew Visitors + Returning Visitors+ Returning Visitors

Invited VisitorsInvited Visitors + Uninvited Visitors+ Uninvited Visitors

# of Users Sending Invites x Invites Sent/User x Invite Clic# of Users Sending Invites x Invites Sent/User x Invite Clickk‐‐through Ratethrough Rate


Define the Equation of your BusinessDefine the Equation of your Business““Peeling the OnionPeeling the Onion””


How to Track Your MetricsHow to Track Your Metrics

Track each metric as daily time seriesTrack each metric as daily time series

Create ratios from primary metrics: X / YCreate ratios from primary metrics: X / YExample: How good is your registration page?Example: How good is your registration page?Okay:Okay: # of registered users per day# of registered users per dayBetter:Better: registration conversion rate =registration conversion rate =

# registered users / # uniques to reg page# registered users / # uniques to reg page

DateDateUnique Unique VisitorsVisitors

Page Page viewsviews

Ad Ad RevenueRevenue

New User New User SignSign‐‐upsups ……

4/24/084/24/08 10,10010,100 29,60029,600 2525 490490

4/27/084/27/08 10,50010,500 27,10027,100 2424 480480

……


Sample Signup Page Yield DataSample Signup Page Yield Data

Daily Signup Page Yield vs. TimeNew Registered Users divided by Unique Visitors to Signup Page

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

1/31 2/14 2/28 3/14 3/28 4/11 4/25 5/9 5/23 6/6 6/20 7/4 7/18 8/1 8/15 8/29 9/12 9/26 10/10

Dai

ly S

ignu

p Pa

ge Y

ield

Changedmessaging

Added questionsto signup page

Started requiringregistration


Using Metrics to Optimize the Using Metrics to Optimize the Equation of your BusinessEquation of your Business

What are the metrics for your business?What are the metrics for your business?Where is current value for each metric? Where is current value for each metric? How many resources to How many resources to ““movemove”” each metric?each metric?

DeveloperDeveloper‐‐hours, time, moneyhours, time, moneyWhich metrics have highest ROI opportunities?Which metrics have highest ROI opportunities?

Return

Return

InvestmentInvestment

Return

Return

InvestmentInvestmentRe

turn

Return

InvestmentInvestment

Metric AMetric AGood ROIGood ROI

Metric BMetric BBad ROIBad ROI

Metric CMetric CGreat ROIGreat ROI

• Which metric has highest ROI opportunity?


Case Study:Case Study:Optimizing Viral Loop MetricsOptimizing Viral Loop Metrics

ActiveUsers

Prospective Users

Invite Click

Succeed

Inviteclick-through rate

Conversion rate

Don’t Click

Fail

Invites per sender

% of users sending invites

• Multiplied together, these metrics determine your viral ratio

Users

% of users who are active

= 15% = 2.3

= 85%

Registration Process


Maximum Upside Potential of a MetricMaximum Upside Potential of a Metric

0

100%

Registration conversion rate

0

100%

% of users sending invitations

0

?

Avg # of invites sent per sender

2.3

85%

15%

Max possible improvement

0.15 / 0.85 = 18% 0.85 / 0.15 = 570% ? / 2.3 = ?%


Doubling Number of Invitations Sent per Sender by Doubling Number of Invitations Sent per Sender by Adding Address Book ImporterAdding Address Book Importer


OptimizingOptimizingCustomer Value Creation Customer Value Creation

with Metricswith Metrics


The Fuzzy Math of Customer ValueThe Fuzzy Math of Customer Value

You create customer value with a product thatYou create customer value with a product thatSatisfies customersSatisfies customers’’ needsneedsIs easy to useIs easy to useHas a good priceHas a good priceIs better than other alternativesIs better than other alternatives

Customer value = customer satisfactionCustomer value = customer satisfactionNot fuzzy: # of users and frequency of useNot fuzzy: # of users and frequency of useUnderstanding user needs and satisfactionUnderstanding user needs and satisfaction

Quantitative: site analytics, usage metrics, surveysQuantitative: site analytics, usage metrics, surveysQualitative: support emails, usability sessions (best)Qualitative: support emails, usability sessions (best)


What is Your Value Proposition?What is Your Value Proposition?

Which user benefits are most important?Which user benefits are most important?How well does your product deliver?How well does your product deliver?

Benefit 1

ImportanceImportanceto Userto User

Current UserCurrent UserSatisfactionSatisfaction

Benefit 2

User BenefitUser BenefitUpsideUpside

PotentialPotential

Benefit 4

Benefit 5

Benefit 6

High

High

Med

Low

Low

Low High – +

Doesn’t matter

Benefit 3 Lowopportunity

opportunity

?

?


Analyzing Product Ideas by ROIAnalyzing Product Ideas by ROI

Investment (developer-weeks)

Ret

urn

(Val

ue C

reat

ed)

Idea C

Idea B

Idea D

Idea A

Idea F

1

1

2 3 4

2

3

4?


Prioritizing Product Ideas by ROIPrioritizing Product Ideas by ROI

Investment (developer-weeks)

Ret

urn

(Val

ue C

reat

ed)

Idea B

Idea A

Idea C

1

1

2 3 4

2345678

5


The UI Design IcebergThe UI Design Iceberg

VisualDesign

InteractionDesign

InformationArchitecture

ConceptualDesign

Recommended reading: Jesse James Garrett’s“Elements of User Experience” chart, free at www.jjg.net

What most people seeand react to

What good PMs and Designers think about


UI questions are never yes/no! (not binary)UI questions are never yes/no! (not binary)Should ask: Should ask: ““What percentage of users What percentage of users ……??””UI changes impact your metricsUI changes impact your metrics

Impact can be positive, negative, small, largeImpact can be positive, negative, small, largeSeek highSeek high‐‐ROI UI changesROI UI changes

Approaching UI Design AnalyticallyApproaching UI Design Analytically

Typical UI design question:Typical UI design question:““When using web pages, do users scroll down?When using web pages, do users scroll down?””

‐‐ YesYes‐‐ NoNo


Put Key Conversion Actions Above The FoldPut Key Conversion Actions Above The Fold

Landing Page ALanding Page A Landing Page BLanding Page B

Key conversion action is above the fold

Key conversion action is below the fold

The Fold


The Fold IsnThe Fold Isn’’t Binary Eithert Binary Either

600 px

768 px

1024 px

Data courtesy of ClickTaleData courtesy of ClickTaleFree trial at www.clicktale.comFree trial at www.clicktale.com

% of U

sers The chrome steals

about 170 pixels


Case Study: Account Signup Process RedesignCase Study: Account Signup Process Redesign

100%

62.3%58.8%

50.9%

34.4% 32.7%

0%

20%

40%

60%

80%

100%

% of U

sers

% of U

sers

Sign in /Sign in /RegistrationRegistration

Account Account TypeType

Cash vs.Cash vs.MarginMargin

5 Partner5 PartnerPagesPages

3 Partner3 PartnerPagesPages

Biggest drop


OpenAccount

Sign in

Account Selection

Register

56%

44%

Forget Password

Registration Process

45% drop off(20% of total)

36% overall drop off for this step

70%(32% of Total)

17% drop off (10% of total)

20% drop off(6% of total)

30%(14% of Total)

80%(26% of Total)

55%(24% of Total)

64%of Total

Analysis of Sign In/Registration FlowAnalysis of Sign In/Registration Flow

Change Password

83%(46% of Total)

Abandonment Rate (7 Day Moving Average)

0%

10%

20%

30%

40%

50%

60%

70%

80%

10/7

/02

10/1

4/02

10/2

1/02

10/2

8/02

11/4

/02

11/1

1/02

11/1

8/02

11/2

5/02

12/2

/02

12/9

/02

12/1

6/02

12/2

3/02

12/3

0/02

1/6/

03

1/13

/03

1/20

/03

Aba

ndon

men

t Rat

e (7

Day

Mov

ing

Ave

rage

)

Steps 1-2


Redesigned User Flow Improved Redesigned User Flow Improved Registration Conversion RateRegistration Conversion Rate

37% improvement in conversion rate

ReleasedNew Design


Summary: Building the DNA of a Killer AppSummary: Building the DNA of a Killer App

Define what success meansDefine what success meansCustomer value propositionCustomer value proposition

Equation of your businessEquation of your business

Instrument your site and track key metricsInstrument your site and track key metrics

Identify opportunities and prioritize by ROIIdentify opportunities and prioritize by ROI

Launch, learn, and iterateLaunch, learn, and iterate

Copyright © 2008 Olsen Solutions LLC

[email protected]@yourversion.com

Questions?Questions?

www.yourversion.comwww.yourversion.com

http://www.yourversion.com/

