OReilly SACON 2016 "A Practical Guide for Continuous Delivery with Containers"

(A Practical Guide to)Continuous Delivery with Containers

Daniel Bryant @danielbryantuk

01/05/2023 @danielbryantuk

Setting the scene…• Continuous delivery is a large topic

• Focusing on the process and tooling• Rather than each explicit step

• My O’Reilly mini-book will provide step-by-step instructions

• Assuming basic knowledge of Docker


Today…• Continuous Delivery (CD)

• The impact of containers on CD

• Creating a container pipeline

• Migrations: Architectural guidance

• Lessons learned the hard way


@danielbryantuk• Chief Scientist at OpenCredo, CTO at SpectoLabs

• Agile, architecture, CI/CD, DevOps

• Java, Go, JS, microservices, cloud, containers

• Leading change through the application of technology and teams

• London Java Community Associate• InfoQ Editor, DZone MVB, O’Reilly…• Conference regular: Devoxx, JavaOne, QCon…


Continuous Delivery


Continuous Delivery

• Produce valuable and robust software in short cycles

• Optimising for feedback and learning

• Not (necessarily) Continuous Deployment


Creation of a build pipeline is mandatory for continuous delivery



The Impact of Containers on CD


Containers: Expectations versus reality

“DevOps”


Container technology• OS-level virtualisation• cgroups, namespaces, rootfs

• Technology to package and execute software

• The container image becomes the source of truth

• Mechanical sympathy is vital


We’ll focus on Docker today• Docker images are built via a Dockerfile

• docker build –t danielbryantuk/test:1.4

• Publish images• docker push danielbryantuk/test:1.4

• Download images• docker pull danielbryantuk/test:1.4

• Run an image as a container• docker run –p 80:80 danielbryantuk/test:1.4




Quick interuption: Microservices…• Containers and microservices are

complementary

• Not covering details for deploying microservices today

• But if you are interested:• Consumer-based contracts • Service virtualisation• Synthetic transactions and semantic

monitoring https://specto.io/blog/recipe-for-designing-building-testing-microservices.html

https://specto.io/blog/recipe-for-designing-building-testing-microservices.html

https://specto.io/blog/recipe-for-designing-building-testing-microservices.html





Creating a Pipeline for Containers



Make your dev environment like production• Develop locally or copy/code in container

• Ensure language runtime/SDK is synced

• Must build/test containers locally• Perform (at least) happy path tests before

pushing code• All tests should be runnable locally


What to put in the Dockerfile

• OS choice• Exposed to OS (often implictly?)• Choose lightweight OS if possible e.g. Alpine, Debian

Jessie

• Configuration

• Build artifacts

• Exposing ports

• Java• JDK vs JRE• Oracle vs OpenJDK

• Golang• Statically compiled binary

• Python• Virtualenv


Please talk to the sysadmin people:Their operational knowledge is invaluable


Different dev and test containers?• Test container• Full OS (e.g. Ubuntu)• JDK • Test tools• Test data

• Easy to see configuration drift

• Interesting ONTEST proposal by Alexi Ledenev

http://blog.terranillius.com/post/docker_testing/





Building images with Jenkins• Standard Jenkins Java

• Gradle or Maven• SonarQube for code quality

• (Optionally) push to artifact repo• Nexus and Artifactory support Java artifacts

and Docker images

• Build Docker Image• Cloudbees Docker Build and Publish Plugin

https://wiki.jenkins-ci.org/display/JENKINS/CloudBees+Docker+Build+and+Publish+plugin

https://wiki.jenkins-ci.org/display/JENKINS/CloudBees+Docker+Build+and+Publish+plugin



Storing in an image registry (DockerHub)



A little context…


Introducing Docker Compose


Testing: Jenkins Pipeline as Code


Testing: Jenkins Pipeline as Code



Jenkins ‘BlueOcean’ Beta



Docker Compose & Jenkins Pipelines


Testing: Functional• Automate all the things!• Deploy to realistic environments

• API-driven functional• REST-assured

• UI-driven functional• Selenium• Serenity BDD• Geb


Testing: NFRs• Execution (runtime)• Security• Observability

• Evolvability (static)• Testabillity• Maintainability• Scalability• Extensibility


Testing: NFRs• Security testing • Findsecbugs• OWASP Dependency check• Bdd-security (OWASP ZAP) / Arachni • Gauntlt / Serverspec• Docker Bench for Security / AQUA

• Performance and Load testing • Gatling / Jmeter• Flood.io


Special mention: Container security testing



Special mention: Fault tolerance testing


Fault tolerance

techblog.netflix.com/2016/10/netflix-chaos-monkey-upgraded.html github.com/tomakehurst/saboteur

http://techblog.netflix.com/2016/10/netflix-chaos-monkey-upgraded.html

http://techblog.netflix.com/2016/10/netflix-chaos-monkey-upgraded.html

https://github.com/tomakehurst/saboteur


Hoverfly• Lightweight Service virtualisation • Open source (Apache 2.0)• Go-based / single binary • Written by @Spectolabs

• Flexible API simulation• HTTP / HTTPS• Highly performant


• Middleware• Remove PII• Rate limit• Add headers

• Middleware• Fault injection• Chaos monkey



Deploy• Test environments should represent

production (as much as possible)

• Fan-in infrastructure pipelines with applications as soon as possible

• Ask yourself: Do you really want to create a container platform?

01/05/2023 @danielbryantuk | @spoole167 49

Don’t underestimate the value of PaaS…


Post-deployment

01/05/2023 @danielbryantuk | @oakinger

When bad things happen, people are at the center


Monitoring is vital with continuous delivery• Host monitoring

• Container monitoring

• Application monitoring

https://github.com /Kentik/docker-monitor

https://github.com/Kentik/docker-monitor

https://github.com/Kentik/docker-monitor


Migrations: Architectural Guidance


Containerise the monolith?• For

• We know the monolith well

• Allows homogenization of the pipeline and deployment platform

• Can be a demonstrable win for tech and the business

• Against

• Can be difficult (100+ line scripts)

• Often not designed for operation within containers, nor cloud native

• Putting lipstick on a pig?


Key lessons learned• Conduct an architectural review

• Architecture for Developers, by Simon Brown• Architecture Interview, by Susan Fowler

• Look for data ingress/egress• File system access

• Support resource constraints/transience• Optimise for quick startup and shutdown • Evaluate approach to concurrency• Store configuration (secrets) remotely

https://leanpub.com/software-architecture-for-developers

http://www.susanjfowler.com/blog/2016/10/7/the-architecture-interview


Containers and cloud: Design for failure• Distributed Computing Principles• Jeff Hodges ‘Distributed Systems’ (bit.ly/1FeaVtt) • Scalable Web Architecture (bit.ly/1tt703O)• ‘For young bloods’ (bit.ly/1pKVepz)

• Design patterns• Timeouts / retries• Bulkheads / circuit-breakers

http://bit.ly/1FeaVtt



http://bit.ly/1tt703O



http://bit.ly/1pKVepz


New design patterns

bit.ly/2efe0TP

http://bit.ly/2efe0TP


Using containers does not obviate the need for good architecture


https://speakerdeck.com/caseywest/containercon-north-america-cloud-anti-patterns




Lessons Learned the Hard Way


Miscellaneous (but vital)• Beware of the ‘latest’ Docker tag• Properly version your containers

• Metadata is vital• Labels can be valuable• h/t MicroBadger

• www.notonthehighstreet.com case study and learnings• http://bit.ly/1PMlpIL

http://bit.ly/1PMlpIL

http://bit.ly/1PMlpIL

01/05/2023 @danielbryantuk | @spoole167 62

Mechanical sympathy: Docker and Java• Set container memory appropriately • JVM requirements = Heap size (Xmx) + Metaspace + JVM overhead• Account for native thread requirements e.g. thread stack size (Xss)• Default fork/join thread pool sizes (based from host CPU count)• Watch out for ulimits

• Entropy • Host entropy can soon be exhausted by crypto operations• –Djava.security.egd=file:/dev/urandom• Be aware of security ramifications


Summary


In summary• Continuous delivery is vitally important in modern architectures/ops

• Container images must be the (single) source of truth within pipeline

• Mechanical sympathy is important (assert properties in the pipeline)• We’re now bundling more into our artifact (e.g. an OS)• Not all developers are operationally aware

• The tooling is now becoming stable/mature• We need to re-apply old CD practices with new technologies/tooling


Bedtime reading


Thanks for listening

• Any questions?

• Feel free to contact me• @danielbryantuk• [email protected]