Move fast and consumer driven contract test things

Making SoundCloud’s µ-services safe(r) to deploy

Move Fast and Consumer-Driven-Contract-Test Things

XP Days Ukraine 2017

2.5 years @ SoundCloud Monetization Team Backend Engineer

@alonpeer

About SoundCloud

a cloud full of sounds

135M tracks, 175M listeners, 12 hours/minute

300+ services / 50+ teams

Agenda

● Testing, monolith style

● My first steps @ SoundCloud

● Introducing contract tests

● Pactifying our services

● Caveats

● QA Q&A

Testing, monolith style

● One service, one client

● API changes are easy

● One client team to sync with for migrations

● API deprecation is easy

Testing, monolith styleThe good ol’ days

● Different requirements per client

=> Code complexity increases

● Harder to deploy without breaking at least one client

Testing, monolith styleMo clients, mo problems

● More manual QA

expensive, slow, prone to human error,

doesn’t scale

● More end-to-end tests

Maintenance nightmare, flaky, slow,

creates bottlenecks

Testing, monolith styleMo clients, mo problems

My first steps @ SoundCloud


Whoa, micro-services

My first steps @ SoundCloudTesting strategy

E2E

Unit

Integration

My first steps @ SoundCloudDigging into post-mortems

© ofsmallthings

https://www.flickr.com/photos/ofsmallthings/8055153041


“A lack of trust in the acceptance tests caused us to largely ignore the warnings they generated.”

“We couldn’t figure out the impact of the broken acceptance tests, and assumed the only problem was the tests themselves, rather than the underlying code.”

“The commit went through as there weren't any tests for the serialisation from the client to the backend.”

Digging into post-mortems

© ofsmallthings

https://www.flickr.com/photos/ofsmallthings/8055153041

Introducing contract tests

Introducing contract testsMy daily routine

© LockeSteerpike

https://www.reddit.com/r/reactiongifs/comments/34fmis/mrw_im_enjoying_my_new_combedback_hairstyle_and/

Introducing contract testsMy daily routine

© rickandmorty.com

http://rickandmorty.com/

Unit tests are not enoughIntroducing contract tests

“#cake” >> { result = call(“cake”) assert(result == “lie”)}

GET “/cake”: return “lie”


GET “/cake”: return “truth”

“#cake” >> { result = call(“cake”) assert(result == “lie”)}


import JsonLibFoo

GET “/cake”: return toJson(current_state)

// { “current_state” : “lie” }

import JsonLibFoo

“#cake” >> { result = fromJson(call(“cake”)) assert(result == “lie”)}


import JsonLibBar


// { “currentState” : “lie” }

import JsonLibBar

“#cake” >> { result = fromJson(call(“cake”)) assert(result == “lie”)}

Introducing contract testsHow this works?

Consumer Provider

End-to-End Test


Consumer Provider

Unit Test Unit Test


Consumer Provider

Unit Test Unit Test


➢ Requesting “/cake”.

➢ Expecting a JSON response

with a key “current_state”.

➢ Asserting deserialization

of the response succeeds.

import JsonLibFoo


// { “current_state” : “lie” }

Consumer

Provider

Pactifying our serviceshttp://pact.io

http://pact.io

What’s Pact?Pactifying our services

● A family of frameworks designed for consumer driven

contract testing.

● Supports JSON over HTTP.

● Impl. for Java, Scala, Go, Ruby, .Net, Swift, JS etc.

● Tests run locally, no external dependencies.


Step 1: Define consumer expectations

© pact.io

http://pact.io


Step 2: Verify expectations on provider

© pact.io

http://pact.io

Consumer: client code (LikesClient)






Consumer: client unit test




Consumer: client contract unit test








Consumer: my-consumer-likes.json







Pactifying our servicesProvider side verification

● Collect pact files from consumers and verify them all.

● Keep your provider environment isolated.

○ Dockerized databases.

○ Test doubles for upstream services.

Pactifying our servicesProvider states


LikesApp

LikesDB


Set state:“User 1000 has liked 2 tracks”

LikesApp

LikesDB


INSERTlikes


LikesApp

LikesDB


INSERTlikes


HTTP Request: GET /likes/1000

LikesApp

LikesDB



HTTP Response: 200 OK

INSERTlikes


LikesApp

LikesDB


LikesApp

LikesDB

HTTP Response: 200 OK

INSERTlikes



HTTP Response: 404 Not Found

Set state:“User 2000 has liked no tracks”


DELETElikes

Pactifying our servicesPact broker

● Share pact files between projects.

● API for pact frameworks to fetch all pacts by provider.

● Support for versioning and tagging.

● Useful UI and visualization of the network.


© pact broker

https://github.com/bethesque/pact_broker


© pact broker



© pact broker


Pactifying our servicesOur CI pipeline

Push new change

Generate consumer contracts

DeployUpload

pacts & tag with ‘prod’

Consumer pipeline

Pactifying our servicesOur CI pipeline

Push new change Deploy

Upload pacts & tag with ‘prod’

Consumer pipeline

Push new change

Verify all ‘prod’-tagged

pactsDeploy

Provider pipeline

Generate consumer contracts

● Communication is key.

Caveats


● New moving parts.

Caveats



● Learning curve per pact framework.

Caveats




● Frameworks are WIP.

Caveats





● Provider side setup is time consuming.

Caveats






● Automating consumer-triggered provider verification.

Caveats






● Automating consumer-triggered provider verification.

● Adoption is essential.

Caveats

Recap

Recap

Write contract tests

Cons

umer

s

Recap

Write contract tests Generate pact files

Cons

umer

s

Recap


Publish to broker

Cons

umer

s

Recap


Publish to brokerVerify pacts

Cons

umer

s

Prov

iders

Recap


Publish to brokerVerify pacts

Cons

umer

s

Prov

iders

QA Q&A

@alonpeer

Move fast and consumer driven contract test things

Software