Modelling Microservices at Spotify - Petter Mahlen

Modelling Microservices

Petter Måhlén, Spotify [email protected]

May 2016

About meInfrastructure at Spotify since 2013

• Service Discovery (Nameless) • Service Framework (http://spotify.github.io/

apollo/) • System-Z • And some more

About this talk• Why model microservices? • Our solution: System-Z • Design • Learnings and Impact

=> Ideas about running microservices at scale

Why model microservices?

Z axis: ShardingY axis: Splitting

X axis: Cloning

http://artofscalability.com/



X axis: Cloning


~14k servers

~1600 things



X axis: Cloning


plus ~100 teamswriting code

~14k servers

~1600 things

Spotify Organisation

https://labs.spotify.com/2014/03/27/spotify-engineering-culture-part-1/

Problems to Solve

Discovering and understanding: • What things are “out there” • Deployments and configurations • The system as a whole, how do things fit together? • How to get more information: ownership • What’s broken and how to fix it.

What came before

Emil

What came before

Emil

ServiceDB

What came before

Emil

ServiceDB

System-Z

Enter System-Z

Some terminology

• Component - a thing • microservice • data store • data pipeline • client component

• System - some components

These terms, like most things Z, are intentionally vague

As anyone

Overviews of ‘everything’, e.g. • Container versions used

• 30, by 240 components • Various kinds of metrics

• 212 experimental components • 14 squads don’t indicate a slack channel

As anyone




As anyone




As anyone




As a component user

For some component: • Figure out who owns it • Understand its API • See its state • Find documentation

As a component user

As a component user

As a component user

As a component user

As a component user

As an owner

• Understand deployment status/configuration • Provision servers • Managing deployments • Understand dependencies

As an owner


As an owner


As an owner


As an owner


As an owner


As an owner


As an owner


As an owner


As an owner


Dependency Maps

And...• Create new components • Configure routing between data centres • Check activity (logins, system updates, etc) • Set up service monitoring dashboards • Check historical build information • Configure data processing pipeline monitoring • Store miscellaneous component metadata • …

Core data model

• Many many-to-many relations • Features add specific data • Discovery names as

indirection

user2

user2

login

CALLERS

user2

user2

login

user3

attributes

create

CALLERS

user2

user2

login

user2-legacy

user3

attributes

create

CALLERS

user2

user2

login

user2-legacy

user2-

user3

attributes

create

CALLERS

user2

user2

login

user2-legacy

user2-

user3

attributes

create

CALLERS

user2

user2

login

user2-legacy

user2-

user3

attributes

create

CALLERS

user2

user2

login

user2-legacy

user2-

user3

attributes

createcreate

attributes

CALLERSuser3

user2

user2

login

user2-legacy

user2-

user3

attributes

createcreate

attributes

CALLERSuser3

user2

user2

login

user2-legacy

user2-

user3

attributes

createcreate

attributes

CALLERSuser3

user2

login user2-

user3

attributes

createcreate

attributes

CALLERSuser3

user2

login user2-

user3

attributes

createcreate

attributes

CALLERSuser3

user3

attributes

createcreate

attributes

CALLERSuser3

Frontend

Backend

UI Platform

Base Capacity Deployment

Cortana Helios

Sysmodel

RDCSquab BargeServerDB

Git repos

YAML files

• Loose/multiple schemas • Files live with code

Dirty Data• Organisational change => ownership confusion • Infrastructure evolution => runtime confusion • Owners don’t benefit from metadata quality

Usage

What users say• The administration panel of the entire Spotify

backend. • Cthulhu • The IO product to rule them all! • Hydra-headed • Kickin rad. • A source of truth to discover services and libraries

at Spotify • A service without a definition of done.

Impact

• Teams integrating features, making them easier to find

• Teams talking about features => more consistent • System-Z mentioned as ‘great’ in 2016 ‘What

sucks’ • Swiss Army Knife (good?)

Conclusions• Microservices => many small things, big picture is

hard • Metadata about microservices helps understand

the system • Our metadata is dirty; this is probably unavoidable • Combining many tools => better collaboration and

consistency • The metadata is useful in different ways for

different users

Questions?

Modelling Microservices at Spotify - Petter Mahlen

Technology