Persoonlijke overtuiging Wat betekent CRO naar EBG: IPV alleen ideeën op roadmap valideren Experimenten moet ‘innovaties’ drijven en het exploreren dmv experimenten bepaalt strategische richting ROAR model
Persoonlijke overtuiging Wat betekent CRO naar EBG:
IPV alleen ideeën op roadmap valideren Experimenten moet ‘innovaties’ drijven en het exploreren dmv
experimenten bepaalt strategische richting ROAR model
Wie is de tweede in golf?
Wie is de tweede in de Formule 1?
The winner takes it all
The winner takes it all
“De nummer 1 wordt 52% vaker gekozen dan de nummer 2”
“89% van de consumenten ruilt een aanbieder in voor een ander wanneer de service tegenvalt”
Hoi!
Benjamin Groenewoud
Product Owner CRO @ Travix
Eerder: SNS Bank, ClickValue, Mirabeau
Hoi!
Benjamin Groenewoud
Product Owner CRO @ Travix
Eerder: SNS Bank, ClickValue, Mirabeau
Conversion Rate Optimization
Evidence Based Growth
CRO @ Travix Lessons Learned
Emerce Conversion April 13, 2017
Benjamin Groenewoud Product Owner CRO @ Travix
Onze merken
About
35 countries
550+ employees
3.5M+ passengers a year
50+ planes filled every day
Travix is a global online travel agency (OTA) operating from offices in the
Netherlands, Germany, India, Singapore, Hong Kong, the UK and the USA.
With our 5 brands and an extensive portfolio of 35 websites we are market leader
in several countries and growing rapidly on all continents.
Our footprint
Our market
Wide range of airlines Price Transparency Deep range of ancillary services Website Usability & Scalability High Quality Customer Service
OTA META AIRLINE TRAVEL AGENT
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓ ✓
SERVICES
1. CRO at Travix so far
2. The future of CRO: EBG
3. Q & A
How to guide you through our lesson learned
CRO at Travix so far
Lesson #1 CRO is not a person
August 2015 Our redesign
More and better testing
Our former process Lineair from step to step
1. Conduct research
2. Hypothesize 3. Create treatment
4. Run experiment
5. Analyze results
6. Run follow-up experiment
Me, myself and I
Begging for some CRO fellows A-Team Approach
Multidisciplinary CRO Team
Is there value? Proof value + collect learnings Experiment regret
Earn Earn more / try to learn Earn, learn, & turn (innovate business)
CRO Specialist Specialist team / in-house + outsourced Embedded in product teams / in-house
Just try something Structured earn and learn driven team Structured earn and learn driven company
Optimizely NH site / Google Optimize Automated server site testing framework
Preaching CRO Share learnings frequently across Travix Workshops and knowledge sessions
Steering
Earn vs. Learn
Capacity
Process
Technology
Knowledge sharing
MacGyver Approach
One man show - jack of all trades
What I wanted
A multi-disciplinary team to do:
● More analysis
● More experiments
● Better experiments
● Evidence Everything
“So you want to pull in an entire team for CRO?
Don’t think that’s gonna happen...”
Campaigning...
Begging...
Waiting...
Praying...
Gaze placement
1. Use mouse- or eye-tracking tools to find what
content draws attention (fovea’s focus)
2. Analyze the path of the fovea (the ‘order’)
3. Test which chunks of content are the most
persuasive
4. Place that content ‘within the gaze-path’, and in
the right order
✓ “We only notice what’s in our gaze”
Or this 99 Persuasion Techniques (1 - 34)
Needs & Motivation > Autonomy > Self-efficacy > Response-efficacy > Need for Relatedness > Need for meaning > Outcome expectancies > Social rewarding > Fudge factor > Present focus bias > Ambiguity aversion > Reward substitution
Attention & Perception > Visual cuing > Gaze cuing > Gaze distraction > Gaze placement > Attentional bias > Focussing effect > Von Restorff effect > Unnoticable differences > Confirmation bias > Visual fluency
Memory & Learning > Flashbulbing > Availability heuristic > Self-generation effect > Choice-supportive bias > Recognition over recall > Affective forecasting > Chunking > Operant conditioning > Procedural fluency > Peak-end rule > Direct Priming > Associative Priming > Cue dependent forgetting
Or this 99 Persuasion Techniques (35 - 58)
Self control > Implementation Intention > Personal writing > Trigger words > Audaciousness > Headline Formula > Engaging Images
Persuasive marketing > Equivalence Framing > Emphasis Framing > Mental Accounting > Bait-and-Switch > Defaults > Front-Loading > Loss Aversion > Storytelling > Serial Positioning > Anchoring & Adjustments
> Hyperbolic Discounting > Immediacy Effect > Price Partitioning > Reference Pricing > Free Effect > Payment Consolidation > Bundled Pricing > Limited Time
Or this 99 Persuasion Techniques (59 - 90)
Social & external influences > Comparison targeting > Positional targeting > Time targeting > Domestic Country Bias > Country of Origin Bias > Weather Effects > Groupthink > Normative social influence > Self-fulfilling prophecy > Injunctive norms > Descriptive norms
> Reference Group > Informational Influence > Social Facilitation > Mimicry > Vicarious Modeling > Upward Comparison > Downward Comparison
System 2 > Low Balling > Sparking Trigger > The law of distraction > Facilitating Trigger > The law of least effort > Reminders > Ego Depletion > Awareness Alerts > Base Rate Neglect > One-choice paradox > Choice paradox > Post-Decision Rationalization > Hindsight Bias > Focusing Effect
Or this 99 Persuasion Techniques (91 - 99)
System 1 > Foot in the door > Decoy effect > Fear appeals > Ambiguity aversion > Self-generation effect > Endowment effect > Affect heuristic > Rhyme as a reason > HALO effect
‘System 2’
‘System 1’
Can system 1 be tricked?
Can system 1 be tricked?
A psychologist?!
Lesson #2 The proof is in the pudding
Showing examples makes more sense Hotjar click and scroll maps
10% reaches this line
A
Random experiments “Let’s try Trustpilot on our homepage”
Random experiments “Let’s try Trustpilot on our homepage”
B
WINNER
It required quite some patience
It required quite some patience
But finally it took off...
From McGyver to… A-Team Approach
Multidisciplinary CRO Team
Is there value? Proof value + collect learnings Experiment regret
Earn Earn more / try to learn Earn, learn, & turn (innovate business)
CRO Specialist Specialist team / in-house + outsourced Embedded in product teams / in-house
Just try something Structured earn and learn driven team Structured earn and learn driven company
Optimizely NH site / Google Optimize Automated server site testing framework
Preaching CRO Share learnings frequently across Travix Workshops and knowledge sessions
Steering
Earn vs. Learn
Capacity
Process
Technology
Knowledge sharing
MacGyver Approach
One man show - jack of all trades
From McGyver to… a real team! A-Team Approach
Multidisciplinary CRO Team
Is there value? Proof value + collect learnings Experiment regret
Earn Earn more / try to learn Earn, learn, & turn (innovate business)
CRO Specialist Specialist team / in-house + outsourced Embedded in product teams / in-house
Just try something Structured earn and learn driven team Structured earn and learn driven company
Optimizely NH site / Google Optimize Automated server site testing framework
Preaching CRO Share learnings frequently across Travix Workshops and knowledge sessions
Steering
Earn vs. Learn
Capacity
Process
Technology
Knowledge sharing
MacGyver Approach
One man show - jack of all trades
From 1 chair
Dreaming of my own party
To 9 chairs
Joost Fromberg Customer Behaviour
Specialist
Daan Bouwmeester Web Analyst
Nicolas Biglie Visual Designer
Gerard Stroomer UX Designer
Benjamin Groenewoud Product Owner
Pavel Fokin Front-end Developer
Aliaksandr Zanouski Front-end Developer
Jorrin Geurts UX Conversion Designer
Tom van den Berg Web Analyst
Please meet the CRO Party “Keep calm and crunch some numbers”
Run experiments to improve conversion / learn user behaviour
Support Product Owners in validating new features
The CRO Team works on two streams
Squeeze optimalisatie vs. innovatie Projecten worden bepaald van bovenaf (top-down)
Bij hogere maturity kan dit cyclus worden waarbij directors op basis van exploratie dmv experimenten de strategie fine-tunen (EBG)
Lesson #3 CRO should be about intelligence
Directe waarde vs. Lange termijn waarde Resultaat vs. Ethiek
Voorbeeld BIG
EBG: both value and intelligence
Adding Direct Value Short-term goal
Customer Intelligence Long-term goal
It’s all about the money, money, money
It’s all about the money, money, money No that’s not funny
- Placeholder -
Belang van mobile Groei mobile voor Travix brands
Krijgt minder aandacht vanwege minder ‘directe’ biz value
In the end EBG is a religion You have believe it’s the way to go
ACT CRO as a transaction machine
Mapping van thema’s op fase van journey Voorstudie + >30 experimenten leveren cruciale inzichten over wat klanten drijft
Homepage
Search Results
Checkout
A
Random experiments “Let’s try Trustpilot on our homepage”
Random experiments “Let’s try Trustpilot on our homepage”
B
WINNER
Our former process Lineair from step to step
1. Conduct research
2. Hypothesize 3. Create treatment
4. Run experiment
5. Analyze results
6. Run follow-up experiment
ACT CRO as a transaction machine
AC & ACT From transactional to customer intelligence
Starting point September 2017: Pre-study
Starting point September 2017: Pre-study
1. Scientific & market research
2. Qualitative research
● Customer Service data
● Online customer feedback
● Usability tests
1. Quantitative research
● Google Analytics
● Heatmaps / session recordings
● Previous experiments
Conclusion Booking a flight feels too difficult and too ‘tricky’
Price dispersion
of the market
Complex
products
Constant state of
‘high perceived risk’
“Industry is playing
tricky”
Klanten zijn extreem prijsgedreven
● Long time on page
● High exit rate
● Lot’s of users clicking
back to homepage to
refine their search
F FIND
Findings Search Results Google Analytics
Conclusions for search results
1. Users look quite ‘system 2’ (rational)
2. There is a lot of information
and there are a lot of tools
(filters, tabs, etc.)
1. This costs a lot of mental energy
(which system 2 doesn’t have)
A ANALYZE
Analyze the data From data to insights
When putting more focus on ‘Change your flight’
→ users will not be distracted by clicking back to the homepage
→ which will lead to more use of the Change button and a higher CR.
A ANALYZE
Analyze how to optimize Coming up with the hypothesis
A ANALYZE
Analyze how to optimize Coming up with the hypothesis
A
C CREATE
Create Create a design which challenges the hypothesis
B
C CREATE
Create Create a design which challenges the hypothesis
B
WINNER for XL screens
C CREATE
Create Create a design which challenges the hypothesis
INCONCLUSIVE
AC & ACT From transactional to customer intelligence
FACT & ACT From transactional to customer intelligence to transformational
1.Growth
1.Customer intelligence
1.A better decision making process
This will result in
The future of CRO: EBG Transformation
Conversion Rate Optimization
Evidence Based Growth
From A-Team to... A-Team Approach
Multidisciplinary CRO Team
Is there value? Proof value + collect learnings Experiment regret
Earn Earn more / try to learn Earn, learn, & turn (innovate business)
CRO Specialist Specialist team / in-house + outsourced Embedded in product teams / in-house
Just try something Structured earn and learn driven team Structured earn and learn driven company
Optimizely NH site / Google Optimize Automated server site testing framework
Preaching CRO Share learnings frequently across Travix Workshops and knowledge sessions
Steering
Earn vs. Learn
Capacity
Process
Technology
Knowledge sharing
MacGyver Approach
One man show - jack of all trades
From A-Team to... Army! A-Team Approach
Multidisciplinary CRO Team
Army Approach
Evidence Based Everything
Is there value? Proof value + collect learnings Experiment regret
Earn Earn more / try to learn Earn, learn, & turn (innovate business)
CRO Specialist Specialist team / in-house + outsourced Embedded in product teams / in-house
Just try something Structured earn and learn driven team Structured earn and learn driven company
Optimizely NH site / Google Optimize Automated server site testing framework
Preaching CRO Share learnings frequently across Travix Workshops and knowledge sessions
Steering
Earn vs. Learn
Capacity
Process
Technology
Knowledge sharing
MacGyver Approach
One man show - jack of all trades
Strategy
Board Members Business Owners
Product Development binnen Travix
Landing Pages & Mkt. Automation
Product Owner Front-end Devs Back-end Devs
Search & Shopping
Product Owner Front-end Devs Back-end Devs
Checkout & X-sell
Product Owner Front-end Devs Back-end Devs
My Account & OPS features
Product Owner Front-end Devs Back-end Devs
Design Team
Product Owner / UX Designer UX Designer
Visual Designer
CRO Team
Psycholoog Webanalisten (2)
Front-end Devs (3)
Product Owner UX Designer
Visual Designer
THINK
BUILD
MEASURE &
OPTIMIZE
DESIGN
MVP
MVP
MVP
Strategy
Board Members Business Owners
De toekomst
Landing Pages & Mkt. Automation
Product Owner Front-end Devs Back-end Devs
Search & Shopping
Product Owner Front-end Devs Back-end Devs
Checkout & X-sell
Product Owner Front-end Devs Back-end Devs
My Account & OPS features
Product Owner Front-end Devs Back-end Devs
Design Team
Product Owner / UX Designer UX Designer
Visual Designer
Customer Intelligence Team
Psycholoog Webanalisten
Product Owner UX Designer
Visual Designer
FACTS &
ANALYZE
BUILD
MEASURE &
OPTIMIZE
CREATE &
TEST
MVP
MVP
MVP
Product Development binnen Travix
Landing Pages & Mkt. Automation ● Product
Owner ● Front-end
Devs ● Back-end
Devs
Search & Shopping ● Product
Owner ● Front-end
Devs ● Back-end
Devs
Checkout & X-sell ● Product
Owner ● Front-end
Devs ● Back-end
Devs
My Account & OPS features ● Product
Owner ● Front-end
Devs ● Back-end
Devs
This feature will
deliver us €1
extra per
booking
A warning about
correct name details
will lower the
number of calls
We need to
accelerate
personalisation
MVP
MVP
MVP
MVP
MVP
MVP
MVP
MVP
MVP
MVP
MVP
MVP
Product Development binnen Travix Landing Pages & Mkt. Automation ● Product Owner ● Front-end Devs ● Back-end Devs
Search & Shopping ● Product Owner ● Front-end Devs ● Back-end Devs
Checkout & X-sell ● Product Owner ● Front-end Devs ● Back-end Devs
My Account & OPS features ● Product Owner ● Front-end Devs ● Back-end Devs
Design Team ● Product Owner / Sr. UX Designer ● UX Designer ● Visual Designer
CRO Team ● Product Owner ● UX Designer ● Visual Designer
● Psycholoog ● Webanalisten (2) ● Front-end Devs (3)
Innovation (roadmap)
Product Development binnen Travix Landing Pages & Mkt. Automation ● Product Owner ● Front-end Devs ● Back-end Devs
Search & Shopping ● Product Owner ● Front-end Devs ● Back-end Devs
Checkout & X-sell ● Product Owner ● Front-end Devs ● Back-end Devs
My Account & OPS features ● Product Owner ● Front-end Devs ● Back-end Devs
Design Team ● Product Owner / Sr. UX Designer ● UX Designer ● Visual Designer
CRO Team ● Product Owner ● UX Designer ● Visual Designer
● Psycholoog ● Webanalisten (2) ● Front-end Devs (3)
Innovation (roadmap)
Enablers
Product Development binnen Travix Landing Pages & Mkt. Automation ● Product Owner ● Front-end Devs ● Back-end Devs
Search & Shopping ● Product Owner ● Front-end Devs ● Back-end Devs
Checkout & X-sell ● Product Owner ● Front-end Devs ● Back-end Devs
My Account & OPS features ● Product Owner ● Front-end Devs ● Back-end Devs
Design Team ● Product Owner / Sr. UX Designer ● UX Designer ● Visual Designer
CRO Team ● Product Owner ● UX Designer ● Visual Designer
● Psycholoog ● Webanalisten (2) ● Front-end Devs (3)
Innovation (roadmap)
Optimization Enablers
Product Development binnen Travix Landing Pages & Mkt. Automation ● Product Owner ● Webanalist ● Designer ● Front-end Devs ● Back-end Devs
Search & Shopping ● Product Owner ● Webanalist ● Designer ● Front-end Devs ● Back-end Devs
Checkout & X-sell ● Product Owner ● Webanalist ● Designer ● Front-end Devs ● Back-end Devs
My Account & OPS features ● Product Owner ● Webanalist ● Designer ● Front-end Devs ● Back-end Devs
Customer Intelligence Team
Product Development binnen Travix Landing Pages & Mkt. Automation ● Product Owner ● Webanalist ● Designer ● Front-end Devs ● Back-end Devs
Search & Shopping ● Product Owner ● Webanalist ● Designer ● Front-end Devs ● Back-end Devs
Checkout & X-sell ● Product Owner ● Webanalist ● Designer ● Front-end Devs ● Back-end Devs
My Account & OPS features ● Product Owner ● Webanalist ● Designer ● Front-end Devs ● Back-end Devs
Customer Intelligence Team
Innovation From RAT to MVP
Optimization From minimal to awesome
Intelligence From insights to strategy
Product Development binnen Travix Landing Pages & Mkt. Automation ● Product Owner ● Webanalist ● Designer ● Front-end Devs ● Back-end Devs
Search & Shopping ● Product Owner ● Webanalist ● Designer ● Front-end Devs ● Back-end Devs
Checkout & X-sell ● Product Owner ● Webanalist ● Designer ● Front-end Devs ● Back-end Devs
My Account & OPS features ● Product Owner ● Webanalist ● Designer ● Front-end Devs ● Back-end Devs
Customer Intelligence Team
Innovation From RAT to MVP
Optimization From minimal to awesome
Intelligence From insights to strategy
● Hoe blijven we competitief? ● Creëren we hiermee een effect? ● Hoe bouwen we een succesvolle pilot?
● Hoe ziet klantgedrag eruit per segment? ● Waar liggen verbeterkansen? ● Hoe maken we dit onderdeel van een 7 een 9,5?
● Wat drijft onze klanten? ● Wat houdt ze tegen om bij ons te boeken? ● Welke tools hebben de teams nodig om hier
achter te komen?
Product Development binnen Travix Landing Pages & Mkt. Automation ● Product Owner ● Webanalist ● Designer ● Front-end Devs ● Back-end Devs
Search & Shopping ● Product Owner ● Webanalist ● Designer ● Front-end Devs ● Back-end Devs
Checkout & X-sell ● Product Owner ● Webanalist ● Designer ● Front-end Devs ● Back-end Devs
My Account & OPS features ● Product Owner ● Webanalist ● Designer ● Front-end Devs ● Back-end Devs
Customer Intelligence Team
"Throw out your roadmap, and trust in the
process and your people. Go where the
tests take you. That may end up in a
different place than you envisioned but it
is where your customers are taking you."
Stuart Frisby - Director of Design,
Booking.com
Product Development binnen Travix Landing Pages & Mkt. Automation ● Product Owner ● Webanalist ● Designer ● Front-end Devs ● Back-end Devs
Search & Shopping ● Product Owner ● Webanalist ● Designer ● Front-end Devs ● Back-end Devs
Checkout & X-sell ● Product Owner ● Webanalist ● Designer ● Front-end Devs ● Back-end Devs
My Account & OPS features ● Product Owner ● Webanalist ● Designer ● Front-end Devs ● Back-end Devs
Customer Intelligence Team
From CRO to EBG From validation to exploring the strategy
CRO at Travix (status quo)
● ‘The business’ defines a roadmap based on
ideas and/or (some) data
● The ideas will be transformed into an MVP
● The MVP will be validated by the CRO Team
EBG at Travix (future)
● There is no roadmap
● Strategy is driven ‘bottom up’ solely by analysis
and experiments
● The outcomes are driving new hypothesis to
validate
● Hypothesis are tested fast and cheap (RAT)
before investing in an MVP
Lesson #4 Don’t sit on an island
Don’t keep sitting in the ivory tower Try to involve all colleagues in your data-driven process
Moving from party to army requires:
● Belief in the way of working
● The right skills in each team
● Standardized process
● One backlog
● One way of calculating prio and value
● The right KPIs
Don’t keep sitting in the ivory tower Try to involve all colleagues in your data-driven process
Moving from party to army requires:
● Belief in the way of working
● The right skills in each team
● Standardized process
● One backlog
● One way of calculating prio and value
● The right KPIs
Lesson #5 Speed over Quality
Increasing velocity will be our main focus
>1200 A/B tests per year Always 800 tests live 7.000 A/B tests per year
99% chance you’re in a test 502 A/B tests per year 100.000 A/B tests per year
Development is purely focussed on quality But the last 20% takes 80% of the time
The MVP is dead From MVP to RAT (Riskiest Assumption Test)
Every step still takes a lot of time Automation is key to make EBG scalable
Optimize testing bandwidth How can you run as many experiments as possible?
Bandit algorithm
When a message seems to work
better, the system will send more
traffic to the winning variant. Or a
message that performs worse will get
almost no traffic anymore. This
makes we’re earning more money
during the experiment.
How they work
WINNER
Automated experiment framework
Automatically...
● Creating experiments when deploying a new feature (server-side)
● Automatically calculate how long experiment needs to run
● Statistical checks / monitoring
● Automated analysis
● Rolling out winner to all visitors automatically
● Use unused bandwidth to run e.g. bandit tests
Pre-analysis Clusteranalyse: welke segmenten zijn het belangrijkst? Welke segmenten presteren boven- en benedengemiddeld? Creation Remains creative... Deployment
Create optimal test planning automatically (using Potential and Impact) Creating experiments when deploying a new feature (server-side) Integrated bandit system
No A/B-tests? → Launch bandit experiments automatically (Smart Notifications) → ‘RAT system for exploring assumptions / make money
Automatically calculate how long experiment needs to run Statistical checks / monitoring Post analysis
Automated analysis Rolling out winner to all visitors automatically
Optimization using ad-hoc testing
Level 1 Level 2
Level 3 Level 4
Level 5
Continuous optimization with clear goals & plan
Evidence Based Growth Team is driving our
roadmaps
Evidence Based Growth is the
way of working across Travix
Evidence Based Growth is in the DNA of ProdDev
Now
From Product Teams to
‘Product Result Teams’ CRO Specialist
Big data specialists, data
mining, customer journey
analysts
Dedicated and multi-
disciplinary CRO Team
Level 3
Knowing what segment is
driven by which determinants
AI experts
Level 1
CRO Basics
Level 4
Business Intelligence
Data Mining
Level 2
Advanced knowledge of user
behaviour and persuasion
Level 5
Artificial Intelligence
& Machine Learning
Experiment on profiles,
steer on NPS and CES
(3-6 experiments / week)
Testing random ideas
(0,5 experiments / week)
Algorithm testing, cross-
channel / -device
(8-10 experiments / week)
Testing strategies
based on data
(2 experiments / week)
Steer on CLV, evidence-
based everything
(15+ experiments / week)
Optimized process with
high velocity, involving
more colleagues
No process defined / ad-hoc
Super-optimized process,
used by all teams with same
workflow and definitions
Clear process with closed
feedback-loop
Front-end + Flight content
Focus on balance quality /
quantity
Only front-end experiments
Injecting javascript
Company roadmap is based
and validated on data-driven
analysis and learnings
Own server-side
testing platform
Only front-end experiments
Focus on max. quality
Predictive analytics tools,
recommendation engines,
etc.
People
Knowledge
Testing strategy and frequency
Process
Technology
Manager Product
Development VP Marketing CCO + CTO
All board members +
directors Entire company
To achieve in 2017
Sponsor
1. Involve all product teams in EBG
2. Start using RAT before MVP
3. Prioritise quantity over quality
4. Set experiment KPI’s (e.g. regret, reporting)
5. Start building a server-side experimentation platform
From CRO to EBG Key focus areas 2017 / 2018
Afsluiter - Grappige
notificatie o.i.d.
The End Thanks for your attention!
Any questions? Email me at [email protected]
Or find me on LinkedIn:
https://www.linkedin.com/in/benjamingroenewoud