JAXLondon 2017 "Continuous Delivery with Containers and Java"

ContinuousDeliverywithContainers:TheGood,theBad,andtheUgly

DanielBryant@danielbryantuk

Containers:Expectationsversusreality

10/10/2017 @danielbryantuk

“DevOps”

Settingthescene…

• Continuousdeliveryisalargetopic• Nobusinessfocustoday(valuestreametc)• PaaSandServerless aresuperinteresting…• ButI’massumingyou’reall-inoncontainers

• Focusingtodayontheprocessandtooling• Nolivecodingtoday• Mini-bookcontainsmoredetails(thanksnginx!)

bit.ly/2jWDSF7

TL;DR– ContainersandCD

• Containerimagebecomesthebuildpipeline‘singlebinary’

• Addingmetadatatocontainersimagesisvital,butchallenging

• Mustvalidatecontainerconstraints(NFRs)• Cultivatecontainer‘mechanicalsympathy’

@danielbryantuk

• IndependentTechnicalConsultant,CTOatSpectoLabs• Architecture,DevOps,Java,microservices,cloud,containers

• ContinuousDelivery(CI/CD)advocate

• Leadingchangethroughtechnologyandteams

ContinuousDelivery

• Producevaluableandrobustsoftwareinshortcycles

• Optimising forfeedbackandlearning

• Not (necessarily)ContinuousDeployment

Creationofabuildpipelineismandatoryforcontinuousdelivery

TheImpactofcontainersonCD

Containertechnology(andCD)

• OS-levelvirtualisation• cgroups,namespaces,rootfs

• Packageandexecutesoftware

• Containerimage==‘singlebinary’

Creatingapipelineforcontainers

Makeyourdevenvironmentlikeproduction

• Developlocallyorcopy/codeincontainer

• Mustbuild/testcontainerslocally• Perform(atleast)happypathtests

QuickAside:Running*entire*systemlocally

https://news.ycombinator.com/item?id=13960107https://opencredo.com/working-locally-with-microservices/https://www.datawire.io/telepresence/ |https://hoverfly.io/

Makeyourdevenvironmentlikeproduction

• Developlocallyorcopy/codeincontainer

• Mustbuild/testcontainerslocally• Perform(atleast)happypathtests

• Useidenticalbaseimagesfromproduction• Withsameconfiguration

Lessonlearned:Dockerfile contentissuper important

• OSchoice

• Configuration

• Buildartifacts

• Exposingports

• Java• JDKvsJREandOraclevsOpenJDK?

• Golang• Staticallycompiledbinaryinscratch?

• Python• Virtualenv?

Pleasetalktothesysadminpeople:Theiroperationalknowledgeisinvaluable

Differenttestandprodcontainers?

• Create“test”versionofcontainer• FullOS(e.g.Ubuntu)• Testtoolsanddata

• Easytoseeapp/configurationdrift

• Usetestsidecarcontainersinstead

• ONTESTproposalbyAlexiLedenev

http://blog.terranillius.com/post/docker_testing/

Dockermulti-stagebuilds

http://blog.alexellis.io/mutli-stage-docker-builds/https://github.com/moby/moby/pull/31257https://github.com/moby/moby/pull/32063

Javaspecificstuff…

github.com/oracle/docker-images/tree/master/OracleJava jdk.java.net/9/ea

Hotoffthepress:Modularity

• Createminimalruntimeimages

• “jlink deliversaself-containeddistributionofyourapplicationandtheJVM,readytobeshipped.”

• Benefits:• Reducedfootprint• Performance• Security

BuildingimageswithJenkins

• Myreportcoversthis

• Buildasusual…

• BuildDockerImage• CloudbeesDockerBuildandPublishPlugin

• Pushimagetoregistry

Storinginanimageregistry(DockerHub)

Metadata– Bewareof“latest”DockerTag

• Bewareofthe‘latest’Dockertag

• “Latest”simplymeans• thelastbuild/tagthatranwithoutaspecifictag/versionspecified

• Ignore“latest”tag• Versionyourtags,everytime• danielbryantuk/test:2.4.1

Lessonlearned:Metadataisvaluable

• Applicationmetadata• Version/GITSHA

• Buildmetadata• Builddate• Imagename• Vendor

• Qualitymetadata• QAcontrol,signedbinaries,ephemeralsupport• Securityprofiles(AppArmor),Securityauditedetc

Metadata- AddingLabelsatbuildtime

• DockerLabels

• Addkey/valuedatatoimage

Metadata- AddingLabelsatbuildtime

• Microscaling Systems’Makefile

• LabellingautomatedbuildsonDockerHub (h/tRossFairbanks)• Createfile‘/hooks/build’

• label-schema.org• microbadger.com

Metadata- AddingLabelsatruntime

$ docker run -d --labeluk.co.danielbryant.lbname=frontdoor nginx

• Can’docker commit’,butcreatesnewimage

• Notpossibletoupdaterunningcontainer

• DockerProposal:Updatelabels #21721

LizRice(andAqua)totherescue!

github.com/aquasecurity/manifesto

Externalregistrywithmetadatasupport

Componenttesting

Testing:JenkinsPipeline(ascode)

Testingindividualcontainers

Integrationtesting

IntroducingDockerCompose

DockerCompose&JenkinsPipeline

EphemeralKubernetesClusters

• Kubernaut (WIP)

• Managesapoolofclusters

• ”Claim”afreshcluster

• UseHelmtoinstalldependencies

TestingNFRsinthebuildpipeline

• PerformanceandLoadtesting• Gatling/jmeter• Flood.io

• Securitytesting• Findsecbugs /OWASPDependencycheck• Bdd-security(OWASPZAP)/Arachni• Gauntlt /Serverspec• DockerBenchforSecurity/CoreOSClair

DelayingNFRstothe‘LastResponsibleMoment’

• Newsflash!• Sometimesthelastresponsiblemomentisup-front!

• Containers/microservices don’tmakethiseasier• Sometimesmoredifficult…

Mechanicalsympathy:DockerandJava

• WatchforJVMcgroup/taskset awareness• getAvailableProcessors()mayincorrectlyreportthenumberofcpus inDocker(JDK-8140793)• Runtime.availableProcessors()ignoresLinuxtaskset command(JDK-6515172)• Default fork/jointhreadpoolsizes(andothers)isbasedfromhostCPUcount

• Setcontainermemoryappropriately• JVMrequirements=Heapsize(Xmx)+Metaspace +JVMoverhead• Accountfornativethreadrequirementse.g.threadstacksize(Xss)

• Entropy• Hostentropycansoonbeexhaustedbycryptooperations

10/10/2017 @danielbryantuk 46

Deployment

skillsmatter.com/skillscasts/10668-looking-forward-to-daniel-bryant-talk

docs.aws.amazon.com/elasticbeanstalk/latest/dg/using-features.deploy-existing-version.html

Observabilityiscoretocontinuousdelivery

www.infoq.com/articles/monitoring-containers-at-scale

Containersarenotasilverbullet

Movingtocontainers:Goingall-in?

ShouldIbuildmyowncontainerplatform?

Probablynot(UnlessyouareGoogle,AWSorIBM)

Whateveryoudecide…pushitthroughapipelineASAP!

Usingcontainersdoesnotobviatetheneedforgoodarchitecturalpractices

https://speakerdeck.com/caseywest/containercon-north-america-cloud-anti-patterns

Summary

Insummary

• Continuousdeliveryisvitallyimportantinmodernarchitectures/ops

• Containerimagesmustbethe(single)sourceoftruthwithinpipeline• Andmetadataaddedasappropriate…

• Mechanicalsympathyisimportant(assertpropertiesinthepipeline)• Notalldevelopersareoperationallyaware

• Thetoolingisnowbecomingstable/mature• Weneedtore-applyexistingCDpracticeswithnewtechnologies/tooling

Bedtimereading

Thanksforlistening

• Anyquestions?

• Feelfreetocontactme• @danielbryantuk• daniel.bryant@tai-dev.co.uk

bit.ly/2jWDSF7

Comingsoon!

Bonusslides(forextracontext)

Containerise anexisting(monolithic)app?

• For

• Weknowthemonolithwell

• Allowshomogenizationofthepipelineanddeploymentplatform

• Canbeademonstrablewinfortechandthebusiness

• Against

• Canbedifficult(100+linescripts)

• Oftennotdesignedforoperationwithincontainers,norcloudnative

• Puttinglipstickonapig?

Keylessonslearned

• Conductanarchitecturalreview• ArchitectureforDevelopers,bySimonBrown• ArchitectureInterview,bySusanFowler

• Lookfordataingress/egress• Filesystemaccess

• Supportresourceconstraints/transience• Optimise forquickstartupandshutdown• Evaluateapproachtoconcurrency• Storeconfiguration(secrets)remotely

Newdesignpatterns

bit.ly/2efe0TP

Microservices…

Containersandmicroservices arecomplementary

Testinganddeploymentchange

https://specto.io/blog/recipe-for-designing-building-testing-microservices.html

JAXLondon 2017 "Continuous Delivery with Containers and Java"

Technology

Continuous Delivered Containers

The road to continuous deployment (V4:Jax) ·...

CONTINUOUS DELIVERY WITH DOCKER CONTAINERS AND … ·...

Devoxx 2017 "Continuous Delivery with Containers: The Good,....

1 Container Types Container Types Sequence Containers...

Using Containers for Continuous Integration and Continuous.....

Using containers and Continuous Packaging to Build native...

A DevOps State of Mind: Continuous Security with DevSecOps.....

London CD: "Continuous Delivery with Containers: Lessons...

Continuous Delivery of Containers with Drone & Kontena

NOMENCLATURE INDIVIDUAL EQUIPMENT AND · PDF...

Containers #101 Meetup: Containers & OpenStack

Continuous Delivery with Containers: The Good ... - JAX...

Continuous Compression Moulding Multilayer · SACMI...

Continuous Assurance: Continuous Integration meets ... ·.....

CI CD met containers · 2018-06-28 · CI CD met containers...