Our testing culture & our testing programMartijn Scheybeler / Leads Growth, SEO & Analytics / [email protected] / @MartijnSch
• How we created a culture of testing?• How were / are we able to scale so fast?• How it influences our (company / teams) culture?• Summary: How can YOU scale your testing program?
What do I like to talk about…
Digital Data Tips Tuesday - #ddtt - @MartijnSch
• Why we do what we do?• Team structure & dynamics• Creating experiment ideas + planning• Creating & coding experiments• Analysing experiments• What’s next?
… and it goes something like this
Digital Data Tips Tuesday - #ddtt - @MartijnSch
Why we do what we do?
Digital Data Tips Tuesday - #ddtt - @MartijnSch
It’s all about:
Digital Data Tips Tuesday - #ddtt - @MartijnSch
VELOCITY!
Digital Data Tips Tuesday - #ddtt - @MartijnSch
or in other words:
Digital Data Tips Tuesday - #ddtt - @MartijnSch
FAIL OFTEN, FAIL FAST
Digital Data Tips Tuesday - #ddtt - @MartijnSch
Team Structure & Dynamics
Digital Data Tips Tuesday - #ddtt - @MartijnSch
• Design, Dev, QA, Project Management, Legal, CEO?YOU’RE DOING IT WRONG!
• Can you design, code & analyse? You don’t need a team.• Cross-functional or not?
Team structure
Digital Data Tips Tuesday - #ddtt - @MartijnSch
Team structure
Digital Data Tips Tuesday - #ddtt - @MartijnSch
• 1 Web Analyst: Designing, Coding & Analysis• 2 Interns: Coding, Analysis• Me: Strategy, Coding, Analysis• Support: 1 designer + 1 front-end developer.
Not cross functional (at the moment)!
Team structure
Digital Data Tips Tuesday - #ddtt - @MartijnSch
What does our culture looks like?
• Better ask for forgiveness, than permission.• F*CK IT, WE’LL DO IT LIVE!• It’s not rocket science!• We’ll do a much as possible inhouse + open.
TNW ‘Core Values’
Digital Data Tips Tuesday - #ddtt - @MartijnSch
• “This is how you access our live servers, need GTM as well?”• “Legal, who?”• “We still have 5 design reviews laying around, DAMMIT!?”• “10 new tests this week! YOLO!”
Team dynamics
Digital Data Tips Tuesday - #ddtt - @MartijnSch
“You don’t have decades of experience I see. This is not how it works in the real world.”
Digital Data Tips Tuesday - #ddtt - @MartijnSch
- Random Enterprise Employee
Take away: make sure you have buy in!
Boris (CEO) “I had a very good idea this weekend! Can we make experiments more aggressive.”
Experiment Ideas + Planning
Digital Data Tips Tuesday - #ddtt - @MartijnSch
“ We don’t want you to run this test! ”
Digital Data Tips Tuesday - #ddtt - @MartijnSch
• Ok, we do have a planning• “It doesn’t need to be rocket science”• We don’t have a huge backlog
Ideas + Planning, Planwhat?
Digital Data Tips Tuesday - #ddtt - @MartijnSch
• Designer: “I created a new design, we should test it!”• Developer: “I looked at your A/B testing script: refactored it”• Analytics: “We have a backlog but we could use new ideas”• Me: “Can we do 20 tests a week?”
Creating experiments
Digital Data Tips Tuesday - #ddtt - @MartijnSch
Deploying in 3..2..1! Flexibility v.s. Scale v.s. €0 Costs
• Designed by our CRO team• Working with our design team to test very big things• Responsible: no way to hide behind bad code• Responsible: communicate everything to the company
Creating experiments
Digital Data Tips Tuesday - #ddtt - @MartijnSch
</Coding Experiments>
Digital Data Tips Tuesday - #ddtt - @MartijnSch
• Google Tag Manager• A/B Testing & Multivariate Testing: http://tnw.to/testingJS • Custom JavaScript / jQuery for changes• Supported by the dataLayer for targeting / segmenting• We could do better: back-end testing!
Coding experiments
Digital Data Tips Tuesday - #ddtt - @MartijnSch
Analysing Experiments
Digital Data Tips Tuesday - #ddtt - @MartijnSch
Automated reporting via TNW Analytics: -20%
Ideas, Backlog, Documentation, Analysis
Digital Data Tips Tuesday - #ddtt - @MartijnSch
• Send variant through Custom Dimensions• Track everything with event tracking• ‘Tracking plans’ so we also know what we track
Google Analytics
Digital Data Tips Tuesday - #ddtt - @MartijnSch
• Google Analytics Reporting API• Custom Dimensions / Event Tracking / Goals• Automated reporting• Ultimate pleasure and fun + no annoyed team members
Analysing experiments
Digital Data Tips Tuesday - #ddtt - @MartijnSch
TNW Analytics: Adding experiments + ideasAutomated reporting via TNW Analytics: -20%
• Grow our team: 20 people?• Cross functional, get developers on board, a mistake?• Automate & Innovate even more• Drink more …?
What’s next?
Digital Data Tips Tuesday - #ddtt - @MartijnSch
Create / Embrace a culture of testing: don’t create silos
Make the whole team + organisation come up with ideas
Learn to code so you can create experiments yourself / faster
Want to scale? Make sure you have the resources to automate
Take aways
Digital Data Tips Tuesday - #ddtt - @MartijnSch
Martijn ScheybelerLead Growth, SEO & Analytics - TNW
Email: [email protected]: @MartijnSch
Thank you!Slides: http://slideshare.net/MartijnScheijbeler
Code: http://github.com/martijnsch/cro
Digital Data Tips Tuesday - #ddtt - @MartijnSch
Digital Data Tips Tuesday - #ddtt - @MartijnSch