Microservices Practitioner Summit Jan '15 - Don't Build a Distributed Monolith - Ben Christensen, Facebook

Ben Christensen@benjchristensen

Microservices Summit – Jan 2016Avoid Distributed Monoliths

Don't Couple SystemsWith Binary Dependencies

Shared Libraries&

Network Clients

Shared Librariesthat are required

Shared Librariesoften called "the platform"

Shared Libraries(and the transitive variety)

Network Clientsof the "official" variety

What does binary coupling look like?

Common Examples ...

RoutingDiscoveryLoggingTracing

Fault Injection

GuavaRxJavaLog4j

Apache CommonsSpringetc ...

SpringStrutsNetty

TomcatApache HttpClient

Not long until 100s of libraries

are required to exist in a given system

Not long until 100s of libraries

are required to exist in a given system

This is a "distributed monolith"

Have you ever seen it take

months to upgrade a library across your

company?

Have you ever seen it take

months to upgrade a library across your

company?

This is a "distributed monolith"

Will it take a ~year to use a new language for

a new service?

Will it take a ~year to use a new language for

a new service?

This is a "distributed monolith"

These Symptoms == Lost BenefitsLost Benefits

Lost Benefits

Polyglot

Lost Benefits

Polyglot

Can Java, .Net, Node.js, Go, Rust, C++, etc co-exist in your system? idiomatically?

Lost Benefits

Organizational and Technical Decoupling

Lost Benefits

Organizational and Technical Decoupling

Can an individual team adopt a new language or platform without convincing a central authority?

Lost Benefits

Organizational and Technical Decoupling

Can individual teams choose a different concurrency model than the "core platform"?

Lost Benefits

Temporal Decoupling

Lost Benefits

Temporal Decoupling

Can an individual team upgrade their networking stack?

Lost Benefits

Temporal Decoupling

Can they upgrade to the newest version of Guava?

But isn't shared code good?

But isn't shared code good?

Not Always

Not necessarily the right principle to prioritize across system boundaries.

But isn't shared code good?

"First, you lose true technology heterogeneity. The library typically has to be in the same language, or at the very

least run on the same platform."

"Building Microservices" – Sam Newman

"Second, the ease with which you can scale parts of your system independently

from each other is curtailed."

"Building Microservices" – Sam Newman

"...you cannot deploy a new library without redeploying the entire process, so your ability to deploy changes in isolation

is reduced."

"Building Microservices" – Sam Newman

"And perhaps the kicker is that you lack the obvious seams around which to erect architectural safety measures to ensure

system resiliency."

"Building Microservices" – Sam Newman

But DRY!?!?

"Building Microservices" – Sam NewmanPage 59, "DRY and the Perils of Code Reuse in a Microservice World"

"This approach, however, can be deceptively dangerous in a microservice

architecture."

"Building Microservices" – Sam NewmanPage 59, "DRY and the Perils of Code Reuse in a Microservice World"

"One of the things we want to avoid at all costs is overly coupling a microservice and consumers such that any small change to

the microservice itself can cause unnecessary changes to the consumer."

"Building Microservices" – Sam NewmanPage 59, "DRY and the Perils of Code Reuse in a Microservice World"

"If your use of shared code ever leaks outside your service boundary, you have introduced a potential form of coupling."

"Building Microservices" – Sam NewmanPage 59, "DRY and the Perils of Code Reuse in a Microservice World"

"The evils of too much coupling between services are far worse than the problems

caused by code duplication."

Page 59"DRY and the Perils of Code Reuse

in a Microservice World"

Just go read it ...

Observed Outcomes

Client library becomes ONLY official way to access the service.

Service logic drifts into the client.

Nearly impossible to adopt new architectures, languages, etc.

Far Reaching Effects

Consuming team is at the mercy of the service-owner.

Brittle "Black Market" Clients

Projection of technical decisions, architecture, and resource utilization

on all service consumers.

Operational complexity is spread to all consumers.

Team consuming 10 services now potentially has arbitrary code from 10

teams to operate and debug.

So what is the alternative?

Contracts & Protocols!

Like programming languages use interfaces and APIs.

Services should hide all implementation details.

Network Protocol & Data Contract

Consume with any language and any technology!Iterate and change over time!

No dependency on service implementation!

Just like the Internet!

But, but, but!!!

What about ... ?

What about ... ?

standardized logging, fault injection,distributed tracing, discovery, routing,

bulkheading, etc, etc, etc

Legitimate Needs for Standardization

Standardization Does Not Need Binary Coupling

Standardization via Protocols & Contracts

Enabled via independent common libraries that consumers can choose to use

... or reimplement to suit their needs.

Standardization via Protocols & Contracts

Standardization via Protocols & Contracts

Example

Public AWS APIswith

various available clients

Standardization via Auditing

Standardization via Auditing

An "integration test" for new services.

Tracing? Logging? Fault injection? Routing?

Doesn't this make it harder to start a new service?

It could. But it doesn't need to.

Common "Tech Stacks"do not break this pursuit of decoupling

Key is that existence of protocols and contracts

allow new stacks to be built.

Anything achieved by any libraryshould be replaceable by coding against

protocols and contracts.

Litmus test ...

Can a group of engineers wanting to use the new hotness

build a new stack without convincing the rest of the company?

and without resorting to sidecars and proxies?

What might this look like?

Shared Libraries&

Network Clients

Transitive Dependencies

Shade Internal Dependencies

Transitive Dependencies

or copy/paste the needed method!

Transitive Dependencies

If part of public API ...

Transitive Dependencies

If part of public API ...

it can't ever have a breaking change.

Transitive Dependencies

(so no libraries that bump their major version every 6-12 months)

Transitive Dependencies

If part of public API ...

it can't ever have a breaking change.

OkHttp & RxJava as examples

http://jakewharton.com/java-interoperability-policy-for-major-version-updates/https://publicobject.com/2015/12/12/com-squareup-okhttp3/

https://github.com/ReactiveX/RxJava/issues/3170 https://github.com/ReactiveX/RxJava/issues/3173

Transitive Dependencies

Shared Libraries&

Network Clients

/pets: get: description: Returns all pets from the system that the user has access to produces: - application/json responses: '200': description: A list of pets. schema: type: array items: $ref: '#/definitions/pet'

Swagger / OpenAPI

Swagger / OpenAPI

5: optional TweetType tweetType = TweetType.TWEET; 16: optional string language = "english";}

typedef list<Tweet> TweetList

struct TweetSearchResult { 1: TweetList tweets;}

exception TwitterUnavailable { 1: string message;}

const i32 MAX_RESULTS = 100;

service Twitter { void ping(), bool postTweet(1:Tweet tweet) throws (1:TwitterUnavailable unavailable), TweetSearchResult searchTweets(1:string query); oneway void zip()}

message Person { required string name = 1; required int32 id = 2; optional string email = 3;

enum PhoneType { MOBILE = 0; HOME = 1; WORK = 2; }

message PhoneNumber { required string number = 1; optional PhoneType type = 2 [default = HOME]; }

repeated PhoneNumber phone = 4;}

message AddressBook { repeated Person person = 1;}

@version("0.1.0")package hello {

@doc("A value class for Request data for the hello service.") class Request { String text; }

@doc("A value class for Response data from the hello service.") class Response { @doc("A greeting from the hello service.") String result; }

@doc("The hello service.") interface Hello extends Service {

@doc("Respond to a hello request.") @delegate(self.rpc, {"timeout": 3000}) Response hello(Request request);

}

@doc("A client adapter for the hello service.") class HelloClient extends Client, Hello {}

@doc("A server adapter for the hello service.") class HelloServer extends Server<Hello> {}

}

Quark by DataWire

/pets: get: description: Returns all pets from the system that the user has access to produces: - application/json responses: '200': description: A list of pets. schema: type: array items: $ref: '#/definitions/pet'

Single | Multi | N | Infinite

Beyond Request/Response

Caching Tiers

Default Fallback Values

Flow Control & Health

So why do we fail at this so often?

Ease.

Short-term feels more productive.

Service-owners have "first-mover" advantage.

Delayed cost of decoupling is high. And it's very hard.

The solutions have limited tax on the short-term.

So let's look beyond the short-term ease.

Avoid Binary Couplingby using

Contracts, Protocols & Automated Tooling