From Good to Great How the testing program is evolving at Dell Dell Online Analytics & Testing Team Oct. 6 th , 2010
From Good to GreatHow the testing program is evolving at Dell
Dell Online Analytics & Testing Team
Oct. 6th, 2010
2 Online Analytics, Testing & Targeting
Agenda
What’s the vision?(Ed Wu – Senior Manager, Global Testing)
What’ve learned?(Dave Nelson – Global Testing Manager)
Got process?(Carol Gohl – PMO)
Online Analytics, Testing & Targeting3
What’s the vision?
Online Analytics, Testing & Targeting4
Dell Online Today
Online: Dell.com + Premier + Global Portal + B2B + Channel Portal + eSupport +
Community
An online orderis placed
every2 seconds
34 supported languages
Googled 1M times per day
58Kchannelpartnersonline
170M visitsto community properties
190KPremierPages
725K B2Btransactions per year
13Bpage views
in FY’09
1Bvisits per year to
Dell.com
Online Analytics, Testing & Targeting5
Our journey started more than a decade ago…
Regional implementation
Limited expertise and talent bench
Regional KPI Standard KPI’s – global dashboard
Internal testing capability
Global implementation
Work closely with vendors & consultants
Full funnel testing
1:1 BT experience
Advanced segmentation
Early Adoption Standardizatio
nMaturity
VOC solution
1996-2007 2008-2009 2010-
Low
Global team across BU
Industry leading testing capability
Regional team Global team by BU
Grow in-house expertise
In-house clickstream tool
Online Analytics, Testing & Targeting6
Great technology
Strong back-office support
Committed to testing
Good people
Our Strengths
Online Analytics, Testing & Targeting7
“GOOD IS THE ENEMY OF GREAT”
Jim Collins
Online Analytics, Testing & Targeting8
To be thetechnologye-commerce leaderby offering the mostrelevant experienceand solutions todrive high customerloyalty
Dell.comOnline Analytics,
Testing & Targeting
To become an innovation engine through insightful analytics & the best-in-class testing capability to enable constant and rapid changes
Online Analytics, Testing & Targeting9
People
Double the size of the team
On-boarding process to provide support
and create connections
On-going training & development plan
If I were running a company today, I would have one priority above all others: to acquire as many of the best people as I could.
Jim Collins
Online Analytics, Testing & Targeting
Team
Global Analytics
Global Testing
eDell Programs
Reporting
PMO
10
Online Analytics, Testing & Targeting11
Accountability
Because they lack accountability, they fail to achieve credibility, and they have no authenticity.
Jim Collins
2x Revenue upside Y/Y
400-600 tests globally
Monetization model based on test results
with statistical significance
Complete transparency & alignment with
finance & business
# of Tests Rev Upside
FY10
FY10
FY11
FY11
Online Analytics, Testing & Targeting12
Got Process?
Online Analytics, Testing & Targeting13
Holy Grail of Web Analytics
Online Analytics, Testing & Targeting14
Entering Carol Gohl…
• 10 years experience in PMO’s– Information technology
deployments
– Sales force transformation
– Construction and infrastructure
– Retail sales
– Product launches
• But, knew nothing about web analytics!
Online Analytics, Testing & Targeting15
Key Findings in the First 30 Days
• Entrepreneurial vs organizational strategy
• Same people ….different roles!
• Multiple “unique” processes!
• Great testing ideas…capability challenges
• Investment in Analytics tools – but not process and workflow tools
Look…
Listen…
Learn…
Online Analytics, Testing & Targeting16
Planning cadencePipeline management
Standardized methodologies
PRIORITIZE the elephants and the cows….
“We’ve never done that”
“They” wont let you do that
“That’s too risky” “We can’t do that”
Online Analytics, Testing & Targeting17
Process Transformation Strategy
PMO
Management
Personnel
Define
Align
Execute
Define
Align
Execute
Arch
itecte
d
Colla
bora
tive
Faster deploymentNear term benefit
Not readily adoptedHard to sustain
Slower deploymentLasting benefit
More readily adoptedBecomes organic
Online Analytics, Testing & Targeting18
Big Picture First
Test Plan
Design
Code
QA
LaunchMonitor
Analyze
Brainstorm Recipes
Analytics Manager Lead
Test Manager Lead
Sign off required
Documentation required
Communicate
pipeline
roadmap
implement
Institutionalize learning
Online Analytics, Testing & Targeting19
…and then get into Weeds
L1 Activity Activity What/deliverable Frequency When Who How/Format Doc/tool needed
Strategic PipelineGather internal developments/benchmarks
Global guidance Quarterly w eek 5-6 Analytics Lead
Internal reports and performance review (BoB tracker, Global Test Plan, Omniture reports, X-segment reporting)
Strategic PipelineGet COE input from COE leads on trends and opportunties
Partial presentations from COE leads (Benchmarking, CE etc)
Quarterly w eek 5-6Global Analytics Planning Manager (Joel) w ith input COE leads
Pow er point presentation(Partial) presentation on latest COE findings
Strategic PipelineLock testing needs from eDell w ith eDell manager (Glenn)
Planning document eDell Quarterly w eek 5-6Global Analytics Planning Manager (Joel)
Pow erpoint presentationE-dell planning presentation (Glenn)
Strategic Pipeline Analyze Global Trend reports
(Partial) presentation(s) on internal trends, industry trends, effectiveness/opportunities and BP on key online funnel elements (ODG, Consideration, Conversion, CE). Includes status on Analytics & Test plan.
Quarterly w eek 5-6Global Analytics Planning Manager (Joel)
Pow erpoint presentation
Ammo dashboard, Omniture Dashboard, X segment reports, Industry reports, Opportunity calculator
Strategic PipelineGenerate and ID Global Focus areas for analysis
Presentation on summarized key strategic areas including global long list of prioritized test ideas using/synthysizing COE findings, eDell input and global trends reports
Quarterly Week 5-6Global Analytics Planning Manager (Joel)
Pow erpoint presentation to Analytics managers
Input previous process steps
Strategic Pipeline BoB updateUpdate Bob tracker/w iki w ith outcome of global focus areas
QuarterlyMonthly testing/analytics calibration meeting
Global Analytics Planning Manager (Joel) and Global Testing Planning Manager (Dave)
PPT/Wiki · PPT/Wiki
Strategic PipelineQuarterly Planning Schedule Comms
Week 6 and forward until week 1 E-Dell lock – weekly meeting
QuarterlyRegional Analytics Manager to Regional Stakeholders
Weekly meeting discussion /PP
Quarterly planning schedule review; Weekly meeting minutes
Schedule, progress timeline etc
Strategic PipelineProvide global guidance on Global focus areas
Review meeting w ith Regional Analytical managers using Presentation on Global Focus areas
Quarterly Week 7Global Analytics Planning Manager (Joel)
Pow erpoint presentation to Analytics managers
Pow er point and conf call
Strategic PipelineUpdated global guidance based on results trends, needs of the business
Monthly email and update in staff meeting Monthly M1/M2/M3Global Analytics Planning Manager (Joel)
Email providing pow erpoint Pow erpoint and conf call
Strategic Pipeline Assess regional business priorities Notes of regional business priorities Monthly Week 8 Regional Analytics managerSolicit regional priorities from stakeholders in meetings
Template/Agenda
Strategic Pipeline Review Global Guidance no documentation Quarterly Week 8 Regional Analytics managerCompare idea pipeline to tops dow n and regional business guidance
-
Strategic Pipeline
Determine and prioritize key improvement areas (pages) based on global guidance and regional analysis
Partial presentation based on analysis MonthlyWeek 8 for next QM1, w eek 13 for next QM2, Week 4 for next QM3
Regional Analytics managerList of strategic areas to optimize (4-8 areas)
Pow erpoint presentation
Strategic PipelineSubmit ideas for M1/2/3 for pipeline visibility
Goal, Test name, hypothesis, region, page MonthlyWeek 8 for next QM1, w eek 2 for next QM2, Week 6 for next QM5
Regional Analytics manager Sharepoint - new idea status ·
Strategic PipelineDraft pipeline of test ideas by key improvement area
Quarterly list of 20-30 test ideas to cover each improvement area and to be ready at M1 presentation. Including 5-10 top ideas for M1. At M2, M3 review the list is updated. Value H, M, L and $ and prioritized
QuarterlyWeek 8 for next QM1, w eek 2 for next QM2, Week 6 for next QM5
Regional Analytics managerPow erpoint format one slide per improvement area w ith test ideas for Q and for M1/2/3
Pow erpoint presentation
Strategic PipelineBrainstorm regional pipeline for Q and M1
Lock f irst 2 test ideas for M1. QuarterlyWeek 8 for M1 then per iterative frequency
Regional Analytics manager Pow erpoint presentation Pow erpoint
Strategic Pipeline
Value ideas H,M,L w ith $ and prioritize w ithin key improvement area for any new ideas from above brainstorming
Revised valued pipeline for 13 w eeks (3 months) - Recommendation for how many days to run the test;
QuarterlyWeek 11 for next QM1, w eek 2 for next QM2, Week 6 for next QM6
Regional Analytics manager
Strategic PipelineLock strategic opportunities and test ideas for the Quarter and M1/2/3 in regional review session
Presentation to stakeholders of strategic areas including the regional pipeline for total Q (at least 1 test idea per w eek - idea description (one liner)). Ensure that M1 presentation contains Quarterly view . Then update for M2/3 review s. Include Q value estimation is included.
Quarterly Week 11 for next QM1 Regional Analytics manager
Collaborate w ith regions to generate prioritized quarterly and monthly pipeline in regional brainstorming session
Pow erpoint presentation
Strategic PipelineSign off f irst tw o test plans for M1W1 and 2
Approved test plans Quarterly Week 11 for next QM2 Regional Analytics manager Present locked test plans Pow erpoint presentation
Regional Analytics Manager
Online Analytics, Testing & Targeting
Check Major Milestones
WK11 WK2 WK6 WK11WK13
Analy
tics
M
anager
Test
ing
Manager
• Strategic Areas (NQ)
• List of test ideas (NQ)
• Annualized upside (NQ)
M2 testing priorities– wireframes / feasibility
M3 testing priorities– wireframes / feasibility
• Strategic Areas (NQ)
• List of test ideas (NQ)
• Annualized upside (NQ)
Lock M1 schedule; First test in M1 ready to launch
WK4 WK8
Lock M2 schedule; First test in M2 ready to launch
Lock M3 schedule; First test in M3 ready to launch
Test Execution
Idea Generation
M1 testing priorities– wireframes / feasibility
M1 testing priorities– wireframes / feasibility
Weekly Optimization Meeting
Analytics Manager LeadTesting Manager Lead
20
Online Analytics, Testing & Targeting21
Establish a Feedback Mechanism
Overall Satisfaction
Improvement Areas
Online Analytics, Testing & Targeting22
Process Change is Iterative…just like testing!
“Don't let the perfect be the enemy of the good”
Voltaire
Online Analytics, Testing & Targeting23
Lesson Learned
Online Analytics, Testing & Targeting
Focusing on Strategic Areas
24
Key Pages
Cart and Checkout
Targeting
Online Analytics, Testing & Targeting
Understanding Customer Intent
25
Specific product offers for customers desiring an efficient shopping
experience
Improved site navigation for
customers want to take more time to find
the right product
Online Analytics, Testing & Targeting26
Targeting Relevant ContentConsumer Customer
s
Small & Medium Business Customer
Online Analytics, Testing & Targeting27
US Cons
US Cons
Checkout Testing - July
US SMB
UK Cons
US Cons
US Cons
DE Cons
JP Cons
AU Cons
CN Cons
BR SMB
CN SMB
JP SMB
AU SMB
US SMB
MX Cons
US Cons
MX SMB
BR Cons
Checkout Testing - August
US ConsCN
Cons
AU Cons
US Cons
US Cons
MX SMB
CA SMB
FR Cons
CA Cons
AU SMB
Checkout Testing - Sept
Testing Globally…an Iterative Process
Online Analytics, Testing & Targeting
Understand Why Something Works (or Doesn’t)
28
Simplified Cart Test – We changed too
much
vs.
Online Analytics, Testing & Targeting29
Wrap it Up
Online Analytics, Testing & Targeting30
Key Takeaways
• People first – invest more!
• Hold yourself accountable – be brave!
• Process is a necessity, not an option – start now!
• Scalability is key to success – leverage it!
• Failure is just as valuable as success – communicate it!
Online Analytics, Testing & Targeting31
Embracing a New Culture:testing as a “religion”
Testing baked into dell.com projects from beginning
Tests are run rigorously with clearly defined
process
Results captured and shared widely, good
or bad
Online Analytics, Testing & Targeting32
“It took Einstein ten years of groping through the fog to get the theory of special relativity, and he was a bright guy.”
Jim Collins
Online Analytics, Testing & Targeting
Thank You