Continuous Delivery Distilled
Post on 27-Aug-2014
548 Views
Preview:
DESCRIPTION
Transcript
Con$nuous Delivery Dis$lled
Ma# Callanan @mcallana
Why Con/nuous Delivery?
• “The most important problem that we face as so?ware professionals is this: – If somebody thinks of a good idea, how do we deliver it to users as quickly as possible?” – CD Book
• “Our highest priority is: – to sa/sfy the customer through early and con$nuous delivery of valuable so?ware.” – Agile Manifesto
Why Con/nuous Delivery?
• Cycle /me should be measured in hours – not months – How long does a single change take to release?
• Quality should be built-‐in to the process – Not an a?erthought of manual inspec/on
• Feedback should be close as possible to point of failure – Late feedback is expensive
What is Con/nuous Delivery?
• A set of prac/ces and principles – aimed at, building, tes/ng, and releasing so?ware faster and more frequently
Principles of So>ware Delivery
Create a Repeatable, Reliable Process for Releasing So?ware
Automate Almost Everything
Version Control Everything
If It Hurts, Do It More O?en – “Bring the Pain Forward”
Build Quality In
“Done” Means Released
Everybody Is Responsible for the Delivery Process
Con/nuous Improvement
What is Con/nuous Delivery?
• A way of reducing cycle /me – How long does a single change take to release? – I.e. how long it takes a simple code change to get to produc/on?
What is Con/nuous Delivery?
• A way of joining development, deployment, tes/ng & release ac/vi/es – coordina/ng them to make the process as efficient and reliable as possible
– reducing feedback delays by automa/ng manual processes
Develop Deploy Test Release
Central Concept: Deployment Pipeline
• Make process visible • Enable automated tes/ng – Ensure quality is built into process
• Enable automated deploys to any environment • Improve feedback cycles
Commit stage
Acceptance stage
Exploratory tes$ng UAT Capacity
Tes$ng Produc$on
What is not Con/nuous Delivery: An/-‐pa#erns
• Deploying so?ware manually • Deploying to prod-‐like env only a?er dev is complete • Manual configura/on management of produc/on
Deployment Pipeline
Agile / Lean / DevOps / Culture
Configura$on Management Tes$ng Deployment
Automa$on Disciplined
Development
Risk Management
Con$nuous Integra$on
Building Blocks
Deployment Pipeline
Agile / Lean / DevOps / Culture
Configura$on Management Tes$ng Deployment
Automa$on Disciplined
Development
Risk Management
Con$nuous Integra$on
Building Blocks
Requirements • Analyst & Customer
Design • Architect
Development • Developer
QA • Tester
Release • Opera/ons
Months / Years!
Waterfall
Agile
Agile -‐> Con/nuous Delivery
Agile Culture
Analyst/Customer
Testers Devs
Acceptance Criteria
Automa/on
DevOps Culture
Analyst/Customer
Testers
Ops
Devs
Acceptance Criteria
Automa/on
Silos
• Ul/mately, we succeed or fail as a team – not as individuals
• Symptoms of Silos – Dev throws work over wall to Test throws over wall to Ops
– End up spending as much /me blaming each other as fixing defects that arise from siloed approach
h#p://www.gesilos.com.au/images/pellet/3%20x%2026%20Tonne%20Pellet%20Silos.jpg
Repairing Silos
• How to break down silos – Get everybody involved together from start of project – Ensure they can communicate on a frequent, regular basis – Increase visibility and self-‐serviceablilty
Lean
• Example Value Stream Map
Require-‐ ments
Develop-‐ment Tes$ng Staging
Capacity Tes$ng
Release
Value-‐ Added Time
Elapsed Non-‐Value Added
Time
Business Customer
Batch Size vs Risk
h#p://www.slideshare.net/jallspaw/ops-‐metametrics-‐the-‐currency-‐you-‐pay-‐for-‐change
• Reducing Batch Size Reduces Risk
Batch Size Science
High Release Cost = High Batch Size
Batch Size
Cost
Release Cost
Total Cost
Low Release Cost = Low Batch Size
Batch Size
Cost
Release Cost
Ac/ng on Feedback • The delivery team must receive feedback
and then act on it • Involve everybody in feedback process
– Work in cross-‐func$onal teams or meet o?en – Retrospec$ve: discuss how to improve
delivery process next itera/on
• Broadcast the informa/on – Big, visible dashboards ensures feedback gets into someone’s head
• Feedback must be acted upon – Requires discipline and planning – Stop and decide on a course of ac$on – Only once this is done should the team carry on with their work
Deployment Pipeline
Agile / Lean / DevOps / Culture
Configura$on Management Tes$ng Deployment
Automa$on Disciplined
Development
Risk Management
Building Blocks
Con$nuous Integra$on
Con/nuous Integra/on Developer • Commit to Version Control
Build Server • Compile
Tes/ng • Unit Tests • Component Tests
Repor/ng • Test Results • Coverage • Sta/c Analysis
Disciplined Development
Configura$on Management Tes$ng Deployment
Automa$on
Deployment Pipeline
Deployment Pipeline
Configura$on Management Tes$ng Deployment
Automa$on Disciplined
Development
Risk Management
Con$nuous Integra$on
Building Blocks
Agile / Lean / DevOps / Culture
Deployment Pipeline
Commit stage
Acceptance stage
Exploratory tes$ng UAT Capacity
Tes$ng Produc$on
Increasing confidence in build’s produc/on readiness
Environments become more produc/on-‐like
Faster feedback
• Tension: – Environmental Confidence vs Feedback Speed
Ar/fact Repository (e.g. Yum)
Version Control (e.g. Git)
Deployment Pipeline Source Code
Commit stage
Compile Sta/c analysis Unit Tests Coverage
Build binary ar/facts
Acceptance stage
Configure Env Deploy
Smoke Tests Acceptance Tests
ar6facts ar6facts
UAT
Deploy Compliance Tests
Smoke Tests
Capacity Stage
Deploy Compliance Tests
Smoke Tests Load Tests
Produc$on
Deploy Compliance Tests
Smoke Tests
ar6facts
CM code/data, Orchestra6on, Puppet, Chef, Fabric, Ansible,
etc
Testers Self-‐service deploys
Ops Push-‐bu4on releases
Env & App Config
Java, Ruby, Groovy, Scala, config, Maven,
Gradle, etc
Deployment Pipeline • Every commit is considered a release candidate • Not every release candidate is released • Candidates must first pass: – Compile, Unit Test, Sta/c Analysis, Create/Upload RPMs – Deploy RPM, Start server – Smoke Tests – Acceptance Tests – Performance Tests – UAT / Manual Exploratory – Business decision to go-‐live – Approval from gatekeepers
Deployment Pipeline
h#p://con/nuousdelivery.com/2010/02/con/nuous-‐delivery/
Deployment Pipeline -‐ Scripts • Deployment pipelines are powered by scripts • Script Ideals – Obey the Unix Philosophy – do one thing well – Environment-‐agnos/c – Sensible exit code
• Script Architecture – Build Scripts
• Convert build output into deployment input – Deploy Scripts
• Environment-‐independent instruc/ons to deploy so?ware remotely
• Can run on central command-‐and-‐control server – Test Scripts
• Repeatable instruc/ons to invoke automated tes/ng
Configura$on Management Tes$ng Deployment
Automa$on
Deployment Pipeline
Deployment Pipeline
Configura$on Management Tes$ng Deployment
Automa$on Disciplined
Development
Risk Management
Con$nuous Integra$on
Building Blocks
Disciplined Development
Agile / Lean / DevOps / Culture
Disciplines Incremental development
Feature switches
Mainline development • Can’t even do CI with mul/ple branches
Small releases
Keep code releasable
Keep automated tests green • TDD: Red before Green
Disciplined Development
Configura$on Management
Deployment Automa$on
Disciplined Development
Deployment Pipeline
Deployment Pipeline
Configura$on Management Tes$ng Deployment
Automa$on Disciplined
Development
Risk Management
Con$nuous Integra$on
Building Blocks
Tes$ng
Agile / Lean / DevOps / Culture
Tes/ng
h#p://lisacrispin.com/2011/11/08/using-‐the-‐agile-‐tes/ng-‐quadrants/
Tes/ng
You can not save money if you are more worried about money, than you are about quality.
W. Edwards Deming:
Costs ↓ Focus on Quality ↑
Costs ↑ + Quality ↓ Focus on Costs ↓
Tes/ng Triangle
Unit
Manual
Automated
Other
Exploratory
System
Acceptance
Unit
Tes/ng • “Eliminate the need for mass inspec/ons, as the way of life to
achieve quality, by building quality into the product in the first place.”
• “Quality doesn’t come from inspec/on, but from improvement of the process. Improve the process so that defects aren’t produced in the first place. This eliminates the need for inspec/on on a mass basis.”
• “Rou/ne inspec/on is the same as planning for defects, acknowledging that the process isn’t correct, or that the specifica/ons made no sense in the first place.”
• “Inspec/on is too late as well as ineffec$ve and costly.”
h#p://www.signsculpt.com.au/wp-‐content/uploads/2013/03/focus-‐on-‐quality.jpg h#p://leanandkanban.wordpress.com/2011/07/15/demings-‐14-‐points/
Prod-‐Like Environments • Every diff with Prod is a risk • Need to take calculated risks – Trade-‐offs for efficiency – Need to understand the cost
• E.g. – DBs – Configura/on Management – Load Balancers – Deployment Process
• Spot the Difference = waste
h#p://www.nairaland.com/a#achments/1500785_spot_jpge3cbd8220a9ec91ea49adz6c79aeb2
Configura$on Management Tes$ng Disciplined
Development
Deployment Pipeline
Deployment Pipeline
Configura$on Management Tes$ng Deployment
Automa$on Disciplined
Development
Risk Management
Con$nuous Integra$on
Building Blocks
Deployment Automa$on
Agile / Lean / DevOps / Culture
Deployment Automa/on
• Releases should be low-‐ceremony, stress-‐free events – We’ve already prac/ced deployment 100s of /mes using the exact same process in prod-‐like environments
– Don’t have to remember complex manual steps or rely on wri#en instruc/ons
– Only tes/ng required is to verify the environment • (not func/onality – already tested with every commit) • Tes/ng is automated
– Releases take minutes (not hours)
An/pa#ern: Deploying So?ware Manually
• An/pa#ern Signs: – Extensive, detailed release documenta$on – Reliance on manual tes$ng to confirm app is correct – Explaining why deployment is going wrong on release day – Frequent correc$ons to release process during release – Environments that differ in their configura$on – Releases that take more than a few minutes to perform – Releases that are unpredictable in their outcome
Deployment Automa/on • Deployments should tend towards being fully automated
– Human: 1) pick version & env, 2) press “deploy” bu#on • Automated deployment scripts = up-‐to-‐date doco
– Encourages collabora$on – everything is explicit in script • Don’t depend on the deployment expert
– Boring, repe$$ve manual tasks requiring significant exper$se == most certain way of ensuring human error
• Automated deployment process is cheap and easy to test – Deployment smoke tests
• Automated processes are fully auditable • Automated process must be used by everybody
– Should be the only way in which the so?ware is ever deployed
Con/nuous Deployment? • Idea: Take pipeline and make final “deploy to prod” step automa/c • Tests
– Automated tests have to be fantas/c -‐ automated UT/CT/FAT/NFAT covering en/re app – Have to write all tests first, so only when story is complete will check-‐ins pass ATs
• Intui/ve objec/on: too risky – But, more frequent releases ! lower risk in any par/cular release – Amount of change between releases goes down – Releasing every change ! risk is limited to just risk inherent in one change – Con/nuous deployment is a great way to reduce the risk of release – Forces you to do the right thing
• Pre-‐requisites: – You can’t do it without automa/ng your en/re build, deploy, test, & release process – You can’t do it without a comprehensive, reliable set of automated tests – You can’t do it without wri/ng system tests that run against a prod-‐like env
Con/nuous Deployment? • If you can’t actually release every set of changes that passes all tests, aim
to create process that lets you do so • Pipelines support this process by:
– Crea/ng repeatable, reliable, automated system for ge{ng changes into prod ASAP – Crea/ng highest quality so?ware using highest quality process -‐ massively reducing risks
• Con/nuous deployment takes this approach to its logical conclusion • It should be taken seriously, because it represents a paradigm shi> in the
way so?ware is delivered • Even if you have good reasons for not releasing every change you make—
and there are less such reasons than you might think—you should behave as if you were going to do so
Database Migra/on
DB v10
App v200 compa/ble with DB v10 & v11
App v210 compa/ble with
DB v11
App v220 compa/ble with DB v11 & v12
Time
App v230 compa/ble with
DB v12 …
DB v11
DB v12
… …
• Decouple database changes from applica/on deployment
Tes$ng Deployment Automa$on
Disciplined Development
Deployment Pipeline
Deployment Pipeline
Configura$on Management Tes$ng Deployment
Automa$on Disciplined
Development
Risk Management
Con$nuous Integra$on
Building Blocks
Configura$on Management
Agile / Lean / DevOps / Culture
Configura/on Management
• Provisioning & management should be automated
• Keep everything you need to create/maintain infra under version control – E.g. Kickstart, Puppet, DNS zone files, DHCP, SMTP, firewall, scripts
– Inputs to deployment pipeline (same as source code)
Deployment Toolchain
CI -‐> Repo -‐> Orchestra/on -‐> CM • Examples – TeamCity -‐> Yum -‐> Fabric -‐> Puppet/Hiera – Bamboo -‐> Yum -‐> Rundeck -‐> Chef – Jenkins -‐> Nexus -‐> Ansible
DevOps Toolchain
h#p://www.slideshare.net/AnthonyShortland/dto-‐chefconf2012
Example DevOps Toolchain
h#p://www.slideshare.net/AnthonyShortland/dto-‐chefconf2012
Deployment Pipeline
Configura$on Management Tes$ng Deployment
Automa$on Disciplined
Development
Con$nuous Integra$on
Building Blocks
Risk Management
Agile / Lean / DevOps / Culture
Risk Management
• Repeatability = confidence
h#p://www.slideshare.net/jallspaw/ops-‐metametrics-‐the-‐currency-‐you-‐pay-‐for-‐change
Risk Management • What about PCI?
– Lock down who is able to access “privileged” envs – Change mgmt process for making changes to privileged envs – Approvals from mgmt before deployments can be performed – Protect against poten/al malicious interven/ons – Audit all deployments
• Deployment pipeline makes it possible to enforce these strategies while enabling efficient delivery process – Automa$on over Documenta/on – Enforce Traceability
Risk Management • Process for managing approvals
– Automated CR mgmt system – Adding access control to deployment pipeline is a trivial exercise
• How does CAB decide whether change should be executed? – If risks outweigh benefits, change should not be made
• Principles: – Keep metrics on the system & make visible:
• How long, how many wai$ng, propor$on denied? • MTBF/MTTR • Cycle $me
– Regular retrospec$ves & improve system based on feedback
CD Maturity Model
Figure 15.1 Maturity model
CD Maturity Model
h#ps://flic.kr/p/afDTK9
Below this line, we’re slipping backwards
Above this line, we’re climbing the ladder of maturity
In One Slide
Quality Cycle Time
Beyond the Book
• How to manage interconnected pipelines? – Independent Releases: release one service at a /me
– Ensure all releases are backwards compa/ble with current consumers
– Discipline Required • Short cycle /me & small batch size • Feature Switching • System tes/ng/monitoring
– E.g. synthe/c transac/ons
Beyond the Book
• Consumer Driven Contracts – Allows independent development and release of dependent services
h#p://mar/nfowler.com/ar/cles/consumerDrivenContracts.html
C
B B
A
A C
Code
Tests
B dev/tester
Maintains
Calls
As ‘C’ changes, its tests for consumers ‘A’ and ‘B’ are executed
top related