Project Title 00.00.2015 Proprietary & Confidential 1 Spotify From a Small Club to a Giant Stadium Scaling Engineering at Spoti Wouter de Bie - Director of Engineering wouter@spoti.com
Project Title 00.00.2015 Proprietary & Confidential 1Spotify
From a Small Club to a Giant StadiumScaling Engineering at Spotify Wouter de Bie - Director of Engineering [email protected]
2Project Title 00.00.2015 Proprietary & ConfidentialSpotify
How did we go from this..
3Project Title 00.00.2015 Proprietary & ConfidentialSpotify
.. to this?
00.00.2015 4Spotify
What is our scale?
00.00.2015 5Spotify
Spotify was founded in 2006. First launched in Sweden in 2008
Spotify went from 1 to 60+ countries in 9 years
Spotify went from 0 to 150 Million users in 9 years
00.00.2015 6Spotify
Currently we’re at about 3500 employees, of which 1700 are in R&D
Spotify consists of 400+ micro services
We have 60+ PB of data under management
00.00.2015 7Spotify
Scaling Spotify • The organization • Our technology platform
I’ll be talking about the two parts that we scaled over the years: our org and tech platform.
00.00.2015 8Spotify
Tech organization
We started with an oldskool tech org, as seen in many organizations.
9
Project Title 00.00.2015 Proprietary & ConfidentialSpotify
Backend Frontend Data QA
Different team doing different things. The Backend Team does backend, the frontend team does frontend, etc
10
Project Title 00.00.2015 Proprietary & ConfidentialSpotify
Backend Frontend Data QA
But all these teams are dependent on each other. And as the org grows, those dependencies become bigger and bigger. Also, these teams don’t move in lock-step and have different priorities and eventually there is the chance of a dead-lock since they are all waiting for each other. Since we need to stay ahead in the game, we need to move fast.
00.00.2015 11Spotify
The Matrix organization
Because of this we designed the Matrix Org
00.00.2015 12Spotify
Autonomy is the word…
It’s all about autonomy. We believe that through autonomy (aka cutting down on dependencies), we achieve speed.
13
Project Title 00.00.2015 Proprietary & ConfidentialSpotify
Backend Frontend Data QA
14
Project Title 00.00.2015 Proprietary & ConfidentialSpotify
Backend Frontend Data QA
Instead of a team per competence, we create a team that can take on a task or mission from A to Z with all the competence needed.
15
Project Title 00.00.2015 Proprietary & ConfidentialSpotify
Backend Frontend Data QA
Squad
We call this a squad
16
Project Title 00.00.2015 Proprietary & ConfidentialSpotify
Backend Frontend Data QA
Squad
Agile Coach
Product Manager
And we add a product manager and an Agile Coach
17Project Title 00.00.2015 Proprietary & ConfidentialSpotify
Who’s the Boss?
Problem: who’s the boss of the squad? PM? AC?
18
Project Title 00.00.2015 Proprietary & ConfidentialSpotify
Backend Frontend Data QA
Chapter
We organize hierarchically in Chapters
19
Project Title 00.00.2015 Proprietary & ConfidentialSpotify
Backend Frontend Data QA
Chapter
Chapter Lead
And a Chapter Lead is a first line engineering manager
20Project Title 00.00.2015 Proprietary & ConfidentialSpotify
VP Engineering
And those chapter leads reported to the VP of Engineering, who reported to the CTO
21Project Title 00.00.2015 Proprietary & ConfidentialSpotify
“What if we have multiple $competence chapters?”
22Project Title 00.00.2015 Proprietary & ConfidentialSpotify
Say that we have a bunch of chapters
23Project Title 00.00.2015 Proprietary & ConfidentialSpotify
And some of them have the same competence (e.g. backend)
24Project Title 00.00.2015 Proprietary & ConfidentialSpotify
Guild
We create a Guild where engineers with that competence come together
00.00.2015 25Spotify
Tribes
26Project Title 00.00.2015 Proprietary & ConfidentialSpotify
VP Engineering
Initially our org looked like this. All CLs reporting to the VP of Eng
27Project Title 00.00.2015 Proprietary & ConfidentialSpotify
VP Engineering
But this group grew too big, so we had to do something
28Project Title 00.00.2015 Proprietary & ConfidentialSpotify
We split up different chapters and let them report to multiple VPs.
29Project Title 00.00.2015 Proprietary & ConfidentialSpotify
Tribe Tribe
And the sub organization we call a tribe.
30Project Title 00.00.2015 Proprietary & ConfidentialSpotify
“What if we have multiple $competence chapters in different
tribes?”
00.00.2015 31Spotify
Still one guild for one competence. Just a little bigger..
Project Title 00.00.2015 Proprietary & ConfidentialSpotify
CTO Head of Product
At the time we did this, our product org was separate and had PMs “embedded” in squads. Product reported to a Head of Product
00.00.2015 33Spotify
Missions
Project Title 00.00.2015 Proprietary & ConfidentialSpotify
CTO
After a while, the amount of tribes became to big for the CTO, so we had to come up with something else.
Project Title 00.00.2015 Proprietary & ConfidentialSpotify
Head of R&D
We split the tribes into groups
Project Title 00.00.2015 Proprietary & ConfidentialSpotify
Head of R&D
Mission
And called it a mission
Project Title 00.00.2015 Proprietary & ConfidentialSpotify
Creator User Engagement
Core Experience Revenue Growth
Opportunities
Technology Platform
Data
Design / Product Insights
Now we have 8 mission in total and we make a distinction between vertical an horizontal missions. Here are the vertical missions
Project Title 00.00.2015 Proprietary & ConfidentialSpotify
Creator User Engagement
Core Experience Revenue Growth
Opportunities
Technology Platform
Data
Design / Product Insights
And the horizontal missions that support the vertical ones
Project Title 00.00.2015 Proprietary & ConfidentialSpotify
Creator User Engagement
Core Experience Revenue Growth
Opportunities
Technology Platform
Data
Design / Product Insights
00.00.2015 40Spotify
A Typical Mision • One or more Engineering Tribes • Product organization • People Operations • (sometimes other mission specific sub-orgs)
Project Title 00.00.2015 Proprietary & ConfidentialSpotify
VP Data
Engineering Operations Research Solutions Design Product
My mission
42Project Title 00.00.2015 Proprietary & ConfidentialSpotify
“What if we have multiple $competence chapters in different tribes
in different missions?”
00.00.2015 43Spotify
Still one guild for one competence. Just a little bigger..
00.00.2015 44Spotify
Organizational Learnings
Some learnings that we have had over the years.
00.00.2015 45Spotify
Organizational Learnings
No Org Model is Perfect Spotify is optimized for speed
through autonomy, Other things don’t come easy and
have to be mitigated for
00.00.2015 46Spotify
Organizational Learnings
Our org model seems complex
On-boarding is sometimes a little hard because of terms
and internal roles used
00.00.2015 47Spotify
Organizational Learnings
Dependencies move up the stack
First there were team dependencies, then tribe dependencies and now Mission dependencies
00.00.2015 48Spotify
Scaling Technology
The only stereotypical stock photo image. Apart from the org there is tech
00.00.2015 49Spotify
Micro Services vs
Monolith And have a standard protocol
What does this accomplish?
50Project Title 00.00.2015 Proprietary & ConfidentialSpotify
Autonomy
00.00.2015 51Spotify
Ops in squads a.k.a. DevOps
What does this accomplish?
52Project Title 00.00.2015 Proprietary & ConfidentialSpotify
Autonomy
00.00.2015 53Spotify
Transition to the Cloud Google Cloud Platform in our case
What does this accomplish?
54Project Title 00.00.2015 Proprietary & ConfidentialSpotify
Autonomy … and also because other companies are better at running a data center
00.00.2015 55Spotify
Trying to prevent multi-tenant solutions … think about a 3500 node cluster with 1500 users
What does this accomplish?
56Project Title 00.00.2015 Proprietary & ConfidentialSpotify
Autonomy
00.00.2015 57Spotify
Trying to prevent multi-tenant solutions … think about a 3500 node cluster with 1500 users
00.00.2015 58Spotify
The downside No organizational model is perfect.
• Autonomy requires a lot of coordination.
• Some things need centralized management like data governance or quality
Obviously there is a downside too..
59Project Title 00.00.2015 Proprietary & ConfidentialSpotify
That’s it! Questions?