Deployment Nirvana

Deployment NirvanaDeployment NirvanaDeployment NirvanaAdrian Pike @adrianpike

The Web is changing.

Monolithic Apps Services

The Web is changing.Scaling hardware is “easy”! 😊

The Web is changing.Scaling people is really really hard.

The Web is changing.

Complex app, simple deployments

The Web is changing.Simple apps, complex deployments

Hi, I'm Adrian

Hi, I'm Adrian

Hi, I'm Adrian. 🙌

• Stride

• Moz

• Welltok

That guy.

In the beginning, we wanted to build a software.

In the beginning, we wanted to build a software.

Moz Local

• Primarily about transforming API data.

• Parallel IO —> node.js!

• Large pile of independent services.

Why SoA

• Team scaling vs. performance scaling.

• Minimize cognitive load.

• Composability!

Why not SoA• Complex Interdependencies.

• Reliability.

• Debugging.

• Toolchains.

• Integration testing.

• Deployment.

A decision was made.

The descent into madness.🌀

The descent into madness.🌀

• Circus.

• Collateral flapping.

• Deploys result in downtime.

• Service-level versioning.

Problem 0: Circus

Today’s hard lesson - vet your tools.

Problem 1: Collateral Flap

• We sucked at node.

• Domains weren’t a thing.

• Crashes happened a lot.

Problem 1: Collateral Flap

• Multiple instances of a given backend.

• Service registry and an HTTP proxy.

We “solved”

it!

Problem 2: Deployment Downtime

• Well now that we’ve got a port registry…

Problem 2: Deployment Downtime

• Fire up a new backend that registers itself.

• Cleanly reap old backends, they finish their tea, crumpets, and requests and go away.

We “solved”

it!

DMV + Seaport + Viaduct• Pile of home-rolled, loosely coupled services.

DMV + Seaport + Viaduct is the greatest thing.

• Shields crashes!

• Zero-downtime deploys!

• Lots of control!

DMV + Seaport + Viaduct is the worst thing ever.

• Client tooling is immature.

• Development is basicallytorture.

• Inter-service versioning is unsolved.

Service versioning• “Solved” problem.

• URL Versioning

• Query Params

• Timestamps

• Inter-service dependencies are real, and require our attention.

• Is there a better way?

Service versioning redux

• git commit -m “replaced cats with dogs, life’s better now”

• b8d825ddd9eb63336f1da4ec46b1a47eecab7427

• That looks suspiciously like a version.

Routing layer versioning• We’ve got a HTTP proxy already.

• Let’s extend it to route to git SHA’s as versions.

• We route domains to default SHAs.

production.foo.com 5e3… Service!

staging.foo.com 61a…A

different service!

Routing Layer Versioning• Advantages:

• As an engineer, I don’t have to build versioning.

• We can test multiple releases simultaneously.

• Disadvantages:

• Deployed services have to be SHA reflective.

• We have to run multiple instances of a given service.

• Inter-service versioning.

Inter-service versioning

• Service A is dependent on Service B.

• Think a business logic service (A) and a data layer service (B).

• They might need to change in lockstep.

A

B

Inter-service versioning• We build a “Release version” to encapsulate all

our services at a given revision.

A

B

V1

A

B’

V2

Inter-service Communication• When A communicates with B, it’s got to specify a

version.

• Three ways;

• I have to think about it. (Use API versioning or some such)

• Lock at deploy time.

• Pass a header.

Client Tooling

• Developer environment

• Just use Foreman.

• Assets and such.

• Just 12factorize.

Lessons learned

• Inter-service versioning is unsolved.

• Rolling deployments are required.

• Shielding engineers from complexity.

• Caveat: Without magic.

Let’s deploy better. 🌅

• Focus on engineer happiness.

• Better tooling.

• Minimize moving parts.

• 4 daemons + 2 persistence layers == wat

• Make it pluggable.

One more thing…☝️

http://adrianpike.github.io/harbor