Scala, docker and testing, oh my! mario camou

Daniel Brown & Mario Camou

Scala, Dockerand Testing

oh my!

[Road]

Who are we?Daniel Brown

Software engineer at eBay

Tinkerer and hacker of electronic devices

Resident mad scientist

@_dlpb / https://github.com/dlpb

Mario Camou

Software engineer at eBay

3D printing enthusiast and Doctor Who fan

“Do what I do. Hold tight and pretend it’s a plan!”

—The Doctor, Season 7, Christmas Special

@thedoc / https://github.com/mcamou/

Agenda● Introduction to Docker● Why would you use Docker in tests?● How do you integrate Docker into a Scala project?● Lessons learned

Intro to Docker

[Docker]

Virtual Machines● eBay’s default way of working● Easy to provision (through a UI)● In the “cloud”● Choice of OS’s

Virtual Machines● Slow to provision internally● Both API and UI● Machines can, and do, disappear● Internal connection latency

Virtual Machines● Time consuming to set up● Not possible easily to automate setup● e.g. databases, web servers,

message queues (even with chef)

Enter DockerDocker • /ˈdɒkər/

Noun

An open platform for developers and sysadmins to build, ship and run applications“

[Docker]

Enter DockerDependencies

● Create images of dependencies

[Docker]

Enter DockerDependencies

● Create images of dependencies● Doesn't solve all setup issues● But only needs to be done once

[Docker]

Enter DockerDependencies

● Create images of dependencies● Doesn't solve all setup issues● But only needs to be done once● Less storage overhead than VM● Databases and webservers = GBs of

data

[Docker]

Enter DockerDependencies

WIN!

[Docker]

Welcome toTheReal World

[Bird]

Background: Our Service

[MFS]

Background: Our Service

[MFS]

Background: Our Goal

Decouple our tests from other services

[MFS]

Dependency Nightmares

Untested

Unreliable

Slow

Fire

Isolation is Key● Decompose the monolith● Identify services we are dependent upon● Stub the contract● Run your tests

[MFS]

Decompose the Monolith

[Stonehenge]

The MonolithA common way of working

● Everything is “in the library”● Antipatterns● Black Magic

Identify Services we are Dependent UponThis can be the tricky step when everything is “in the framework”

● May require the use of debuggers, network sniffers etc● Investment is worth it in the long run

Identify Services we are Dependent UponMove configuration of services out to easily controllable (and versionable) config

● Implement, or update, a client to use the new config● Decoupling production from tests

Stub the ContractHere you need to know

● The API mappings● Expected Responses

Stub the ContractGolden Rule: Keep it Simple!

● Have one response per stub● Keep them versioned● Put them in containers

Run your TestsNow that we know our dependencies, contracts, and have stubs, we can write tests

● These are focussed only on testing that our system behaves as expected when downstream services offer varying responses

● We are NOT testing the downstream services

Run your Tests● Create a number of different stubs in different docker images● Spin up the ones you need for your specific test● When you are done with the test, tear them down and start again

So How do you go about it?So, how do we go about it?

[Hands]

Dockerizing Scala

[Cat]

Normal Docker flow● Create a Dockerfile● Build the Docker image● Push it to the registry● Pull the image from every server

Normal Docker flow● No static checking of the Dockerfile● Artifact build is separate from image build

○ Do you have the right artifacts?○ Did you run the tests before building?○ Are the latest bits in the image?

Integrating Docker with sbt

[Construction Kit]

Integrating Docker with sbtsbt-docker

An sbt plugin that:

● Creates a Dockerfile● Creates an image based on that file● Pushes the image to a registry

https://github.com/marcuslonnberg/sbt-docker

Setting the Image NameimageNames in docker := Seq( ImageName( namespace = Some("myOrg"), repository = name.value, tag = Some(s"v${version.value}") ), ImageName( namespace = Some("myOrg"), repository = name.value, tag = Some("latest") ))

Some Useful valsval artifact = (assemblyOutputPath in assembly).valueval baseDir = "/srv"val preInstall = Seq( "/usr/bin/apt-get update", s"/usr/sbin/useradd -r -s /bin/false -d $baseDir myUser").mkString(" && ")

Configuring the Dockerfiledockerfile in docker := { new Dockerfile { from("java:openjdk-8-jre") user("myUser") entryPoint("bin/run.sh") runRaw(preInstall) copy(new File("bin/run-docker.sh"), s"$baseDir/run.sh") copy(new File("local/etc"), "$baseDir/etc") copy(artifact, s"$baseDir/app.jar") runRaw("chown -R myUser $baseDir && chmod 0544 $baseDir/run.sh") }}

Integrating with sbt-assemblyEnsure assembly always runs before Docker

docker <<= (docker dependsOn assembly)

Creating and Publishing the Imagesbt> dockersbt> dockerPush

Caveats and Recommendations● Create a fat JAR: sbt-assembly or sbt-native-packager● In non-Linux platforms, start up docker-machine and set up the

environment variables before starting sbt:

$ docker-machine start theMachine$ eval $(docker-machine env theMachine)

https://velvia.github.io/Docker-Scala-Sbt/

[Explosion]

Integration Testing with DockerCreate Docker image(s) containing stubbed external resources

● Tests always run with clean data● Resource startup is standardized● Does not require multiple VMs or network calls

Integration Testing with DockerBefore your tests:

● Start up the stubbed resource containers● Wait for the containers to start

After your tests:

● Stop the containers

Can we automate all of this?

Container Orchestration during TestsOrchestration platforms

● Docker Compose● Kubernetes ● …

Container Orchestration during TestsOrchestration platforms

● Docker Compose● Kubernetes ● …

Orchestrate inside the test code

Using the Docker Java APIval config = DockerClientConfig.createDefaultConfigBuilder() .withServerAddress("tcp://192.168.99.100:2376") .withDockerCertPath("/path/to/certificates") .buildval docker = DockerClientBuilder.getInstance(config).buildval callback = new PullImageResultCallbackdocker.pullImageCmd("mongo:2.6.12").exec(callback)callback.awaitSuccessval container = docker.createContainerCmd("mongo:2.6.12") .withCmd("mongod", "--nojournal", "--smallfiles", "--syncdelay", "0") .execdocker.startContainerCmd(container.getId).execdocker.stopContainerCmd(container.getId).execval exitcode = docker.waitContainerCmd(container.getId).exec

Scala-native Solutionsreactive-docker and tugboat

● Use the Docker REST API directly -> versioning problems● Unmaintained for > 1 year (not updated to Docker 1.2 API)● No TLS support

Using reactive-dockerimplicit val docker = Docker("192.168.99.100", 2375)val timeout = 30.secondsval name = "mongodb-test"val cmd = Seq("mongod", "--nojournal", "--smallfiles", "--syncdelay", "0")val cfg = ContainerConfiguration(Some("mongo:2.6.12"), Some(cmd))val (containerId, _) = Await.result( docker.containerCreate("mongo:2.6.5", cfg, Some(name)), timeout)val started = Await.result(docker.containerStart(containerId), timeout)val stopped = Await.ready(docker.containerStop(containerId), timeout)

https://github.com/almoehi/reactive-docker

Caveats and Recommendations● Use beforeAll to ensure containers start up before tests● Use afterAll to ensure containers stop after tests

Caveats:

● Multiple container start/stops can make tests run much slower● Need to check when resource (not just container) is up● Single start/stop means testOnly/testQuick will start up all

resources● Ctrl+C will not stop the stubbed resource containers

Introducing docker-it-scala● Uses the (official) docker-java library● Starts up resource containers in parallel

○ Only when they are needed○ Once when the tests start○ Waits for service (not just container) startup

● Automatically shuts down all started stub containers● Configured via code or via Typesafe Config

https://github.com/whisklabs/docker-it-scala

[Docker-It-Scala]

Defining a Resource Container● Resources are declared as traits and mixed into tests● Sample implementations available for Cassandra, ElasticSearch,

Kafka, MongoDB, Neo4j, PostgreSQL, Zookeeper (in the docker-testkit-samples package)

[Docker-It-Scala]

Defining Resource Container (Neo4j)trait DockerNeo4jService extends DockerKit { val neo4jContainer = DockerContainer("whisk/neo4j:2.1.8") .withPorts(7474 -> None) .withReadyChecker( DockerReadyChecker.HttpResponseCode(7474, "/db/data/") .within(100.millis) .looped(20, 1250.millis) ))

abstract override def dockerContainers: List[DockerContainer] = neo4jContainer :: super.dockerContainers}

[Docker-It-Scala]

Defining Resource Container (PostgreSQL)docker { postgres { image-name = "postgres:9.4.4" environmental-variables = ["POSTGRES_USER=nph", "POSTGRES_PASSWORD=suitup"] ready-checker { log-line = "database system is ready to accept connections" } port-maps { Default-postgres-port.internal = 5432 } }}

[Docker-It-Scala]

Defining Resource Container (PostgreSQL)trait DockerPostgresService extends DockerKitConfig {

val postgresContainer = configureDockerContainer("docker.postgres")

abstract override def dockerContainers: List[DockerContainer] =

postgresContainer :: super.dockerContainers

}

[Docker-It-Scala]

Writing Your Testsclass MyMongoSpec extends FunSpec with DockerMongodbService { // Test assumes the MongoDB container is running}

class MyPostgresSpec extends FunSpec with DockerNeo4jService { // Test assumes the Neo4j container is running}

class MyAllSpec extends FunSpec with DockerMongodbService with DockerNeo4jService with DockerPostgresService{ // Test assumes all 3 containers are running}

https://github.com/whisklabs/docker-it-scala[Docker-It-Scala]

What Did We Achieve?

[Space needle]

The Effect● We cut our end to end testing time

120 minutes -> 10 minutes

[Clock]

The Effect● We cut our end to end testing time

120 minutes -> 10 minutes

● Confidence

[Clock]

Going Further● Performance Measurements● Business Decisions

Performance measurements (from Stubbed Tests)Create mocks that can simulate (or approximate) real-world conditions

● E.g.○ Drop every third request○ Delay 5 seconds before responding○ Respond instantly for every request

Performance measurements (from Stubbed Tests)Use these new stubs to gather data about how your system performs

● When it is stressed;● When downstream services are stressed

Performance measurements (from Stubbed Tests)Use these new stubs to gather data about how your system performs

● When it is stressed;● When downstream services are stressed

Gather the metrics!

Performance measurements (from Stubbed Tests)

[MFS]

Going FurtherWhat did we decide from the graph?

Going FurtherWhat did we decide from the graph?

● Technical limitations for our product

Going FurtherWhat did we decide from the graph?

● Technical limitations for our product● Business policies for the product

SummaryIsolation is key to gathering meaningful test data and keeping testing strategies sane

Docker can ease the pain of managing stub dependencies during integration testing

Meaningful tests can tell you a lot about the behaviour of your system and therefore influence both architecture and UX

Q & A

[Audience]

Acknowledgements & Notes[Docker] Docker: www.docker.comDocker and the Docker logo are trademarks or registered trademarks of Docker, Inc. in the United States and/or other countries. Docker, Inc. and other parties may also have trademark rights in other terms used herein.

[Fire], Fire image, Creative Commons CC0 License https://www.pexels.com/photo/fire-orange-emergency-burning-1749/

[Clock], Clock Image, Creative Commonshttps://www.flickr.com/photos/75680924@N08/6830220892/in/photostream/

[Space needle] Space needle, Creative Commons Zerohttps://unsplash.com/photos/-48aJfQpFCE

[Explosion] Vehicle and building on fire, Creative Commons Zerohttps://unsplash.com/photos/28v9cq7ytNU

[Stonehenge] Stonehenge, Public Domain

[Cat] Firsalar the fluffy cat loves to sit in boxes, CC-BY 2.0

[Construction Kit] Free Universal Construction Kit by F.A.T. Lab and Sy-Lab http://fffff.at/free-universal-construction-kit/

[MFS] While MFS was released to production for a short time, it was not released for external customer use.

[Audience] Audience, Creative Commons Zerohttps://unsplash.com/photos/bBQ9lhB-wpY

[Bird] Bird Stare, Creative Commons Zerohttps://unsplash.com/photos/ig9lRTGT0h8

[Road] Yellow Brick road to Lost Hatch, CC-BY-NC 2.0 https://www.flickr.com/photos/wvs/352414272

[Hands] Mud Hands, CC-BY-NC 2.0 https://www.flickr.com/photos/migueltejadaflores/13783685515

[Docker-it-scala] Docker IT Scala, Whisk Labs, MIT https://github.com/whisklabs/docker-it-scala