Principles and Practices in Continuous Deployment at Etsy
Post on 23-Aug-2014
10190 Views
Preview:
DESCRIPTION
Transcript
Principles and Practices in Continuous Deployment
Mike Brittain
Engineering Director, Etsy
@mikebrittain mikebrittain.com/talks
“Continuous Deployment”Process by which our team deploys software changes to production services over 30 times per day.
Where we started
Principles for our Engineering team
Continuous Deployment
Business case
Five years ago…2-3 weeks of code changes
Release and rollback plans
Traffic and infrastructure management (Ops)
6-14 hours
Five years ago…“Deployment Army”
Stressful, especially when things go wrong
Long days and late nights
Scheduled downtime
pro·duc·tion [pruh-duhk-shuhn] (n)
1. This complex system of application code, distributed services, servers, networking gear, etc., upon which we’re going to try to carefully apply a complicated set of changes and hope that nothing goes wrong. Cross your fingers… here goes.
Software for large-scale web sites has been traditionally written by one group of people, then released and operated by a different group.
These two groups have very different levels of visibility into how the software works.
Stagnation
“…frequent and prolonged outages.”2010 CAPACITY PLAN
First, Principles.
Innovate or die
Innovate or die
Resolve scaling hurdles
Innovate or die
Resolve scaling hurdles
Mean-time-to-recovery
“Quality is not just testing pre-release. It also includes our adaptability and response time.”
- Jeff Sussna at ALM Forum, 2014
Innovate or die
Resolve scaling hurdles
Mean-time-to-recovery
Innovate or die
Resolve scaling hurdles
Mean-time-to-recovery
Healthy and talented engineering team
Autonomy, Mastery, Purpose.
“Drive: The surprising truth about what motivates us.” ~Dan Pink, at RSA http://youtu.be/u6XAPnuFjJc
Innovate or die
Resolve scaling hurdles
Mean-time-to-recovery
Healthy and talented engineering team
Stop stressing about releases
First, Principles.
http://timothyfitz.com/2009/02/10/continuous-deployment-at-imvu-doing-the-impossible-fifty-times-a-day/
In a software release process Fail Fast means releasing undeployed code as fast as possible, instead of waiting for a weekly release to break.
http://youtu.be/LdOe18KhtT4
SECRET WEAPON: Hired as VP, Tech-Ops at Etsy
Continuous Deployment
Continuous Delivery~ vs ~
Frequent check-ins directly to mainline.
Continuous Deployment Continuous Delivery✓ ✓
Continuous Integration and Automated tests.
Continuous Deployment Continuous Delivery✓ ✓
Keep the build green. We’re always ready to release.
Continuous Deployment Continuous Delivery✓ ✓
“One button” deploys.
Continuous Deployment Continuous Delivery✓ ✓
Business dictates when a build is deployed.
Continuous Deployment Continuous Delivery✓
Every passing build is deployed to production.
Continuous Deployment Continuous Delivery✓
All enhancements are gated by Config Flags. (“Branch in code”)
Continuous Deployment Continuous Delivery✓ ?
Most of the builds we deploy are “dark” changes.
CSS rules and properties
Copy in templates (e.g. typos)
New, un-referenced code (e.g. classes, funcs, templates)
Code paths behind disabled config flags
etc…
Dev Team Version Control Build & Unit Tests
Automated Acceptance Tests
User Acceptance Tests Release
Check inTrigger
Feedback
Source: http://en.wikipedia.org/wiki/Continuous_delivery
Continuous Delivery release pipeline
Dev Team Version Control Build & Unit Tests
Automated Acceptance Tests
User Acceptance Tests Release
Check inTrigger
Feedback
Check inTrigger
Feedback Trigger
Feedback
Source: http://en.wikipedia.org/wiki/Continuous_delivery
Continuous Delivery release pipeline
Dev Team Version Control Build & Unit Tests
Automated Acceptance Tests
User Acceptance Tests Release
Check inTrigger
Feedback
Check inTrigger
Feedback Trigger
Feedback
Check inTrigger
Feedback Trigger
Feedback Approval
ApprovalFeedback
Source: http://en.wikipedia.org/wiki/Continuous_delivery
Continuous Delivery release pipeline
Dev Team Version Control Build & Unit Tests
Automated Acceptance Tests
User Acceptance Tests Release
Check inTrigger
Feedback
Check inTrigger
Feedback Trigger
Feedback
Check inTrigger
Feedback Trigger
Feedback Approval
ApprovalFeedback
Continuous Delivery release pipeline
Dev / Integration Staging Production
Dev Team Version Control Build & Unit Tests
Automated Acceptance Tests
User Acceptance Tests Release
Check inTrigger
Feedback
Check inTrigger
Feedback Trigger
Feedback
Check inTrigger
Feedback Trigger
Feedback Approval
ApprovalFeedback
Continuous Delivery release pipeline
Dev / Integration Staging Production
Assumptions:
Staging is a perfect reflection of Production, with respect to hardware, configurations, data, overall load, capacity, etc.
Deploy process is infallible.
“What do you mean, ‘it’s not working in production?’ I TESTED IT BEFORE WE RELEASED!”
Dev Team Version Control Build & Unit Tests
Automated Acceptance Tests
User Acceptance Tests Release
Check inTrigger
TriggerApproval
Continuous Delivery release pipeline
Dev / Integration Staging Production
Check inTrigger
TriggerApproval
Check inTrigger
TriggerApproval
Check inTrigger
TriggerApproval
Check inTrigger
TriggerApproval
Check inTrigger
TriggerApproval
Dev Team Version Control Build & Unit Tests
Automated Acceptance Tests
User Acceptance Tests Release
Check inTrigger
TriggerApproval
Continuous Delivery release pipeline
Dev / Integration Staging Production
Check inTrigger
TriggerApproval
Check inTrigger
TriggerApproval
Check inTrigger
TriggerApproval
Check inTrigger
TriggerApproval
Approval
Check inTrigger
TriggerApproval
Feedback
Dev Team Version Control Build & Unit Tests
Automated Acceptance Tests
User Acceptance Tests Release
Check inTrigger
TriggerApproval
Continuous Delivery release pipeline
Check inTrigger
TriggerApproval
Check inTrigger
TriggerApproval
Check inTrigger
TriggerApproval
Check inTrigger
TriggerApproval
Approval
Check inTrigger
TriggerApproval
Feedback
"Because you’re integrating so frequently, there is significantly less back-tracking to discover where things went wrong , so you can spend more time building features.” !
—ThoughtWorks !
!http://www.thoughtworks.com/continuous-integration
Dev / Integration Staging Production
Dev Team Version Control Build & Unit Tests
Automated Acceptance Tests
User Acceptance Tests Release
Check inTrigger
TriggerApproval
Continuous Delivery release pipeline
Check inTrigger
TriggerApproval
Check inTrigger
TriggerApproval
Check inTrigger
TriggerApproval
Check inTrigger
TriggerApproval
Approval
Check inTrigger
TriggerApproval
Feedback
Dev / Integration Staging Production
Where’s the bug? !
In one of the numerous check-ins? Missing unit tests? Missing automated UA tests? Missing manual UA tests?
Dev Team Version Control Build & Unit Tests
Automated Acceptance Tests
User Acceptance Tests Release
Check inTrigger
TriggerApproval
Continuous Delivery release pipeline
Check inTrigger
TriggerApproval
Check inTrigger
TriggerApproval
Check inTrigger
TriggerApproval
Check inTrigger
TriggerApproval
Approval
Check inTrigger
TriggerApproval
Feedback
Dev / Integration Staging Production
Where’s the bug? !
In one of the numerous check-ins? Missing unit tests? Missing automated UA tests? Missing manual UA tests? !Data out of sync?
Server configurations out of sync?
Capacity vs. current load?
Deployment script?
Dev Team Version Control Build & Unit Tests
Automated Acceptance Tests
User Acceptance Tests Release
Check inTrigger
TriggerApproval
Continuous Delivery release pipeline
Check inTrigger
TriggerApproval
Check inTrigger
TriggerApproval
Check inTrigger
TriggerApproval
Check inTrigger
TriggerApproval
Approval
Check inTrigger
TriggerApproval
Feedback
Dev / Integration Staging Production
How will we know when something is wrong in production? !
How long will it take to resolve the issue?
Check inTrigger
We aim to reduce fundamental surprise in every release.
Furthermore, we optimize for detecting and recovering from failures quickly.
Pre-production validationCode deployed to de-pooled application (web) servers touching prod services and databases.
Smoke tests
Integration tests
Functional tests
User-Acceptance (ad hoc)
Production validationExactly the same server configs, services and data as pre-prod, but this is where we introduce application code to live traffic.
Production validationExactly the same server configs, services and data as pre-prod, but this is where we introduce application code to live traffic.
Smoke tests (esp. over public hostnames)
User-Acceptance testing behind config flags
Gratuitous monitoring
Customer support and forums
Single release
Many releases
50K LOC/month
Few opportunities for failureWide surface area (50,000 LOC) High MTTR !
All of the bugs we’ve written
More opportunities for failure Narrow surface area (< 100 LOC)
Low MTTR !
A fraction of the bugs we’vewritten per release
Imagine that we’ll write
Monitoring
Monitoring
PHP Warnings Bug Reports and Help Requests
Deploy logs
Post-Mortems
Check inTrigger
Feedback Trigger
Feedback Approval
Approval
Smoke Tests User Acceptance!Tests ReleaseDev Team Version Control Build & Unit
TestsAutomated
Acceptance TestsUser Acceptance
Tests Deploy (Prod) Monitoring and Automated Alerts
Continuous Deployment release pipeline
Feedback
Dev Pre-Production (“Princess”)
Production
Check inTrigger
Feedback Trigger
Feedback Approval
ApprovalFeedback
Smoke Tests User Acceptance!Tests ReleaseDev Team Version Control Build & Unit
TestsAutomated
Acceptance TestsUser Acceptance
Tests Deploy (Prod) Monitoring and Automated Alerts
Continuous Deployment release pipeline
CI
Dev Pre-Production (“Princess”)
Production
Check inTrigger
Feedback Trigger
Feedback Approval
ApprovalFeedback
Smoke Tests User Acceptance!Tests ReleaseDev Team Version Control Build & Unit
TestsAutomated
Acceptance TestsUser Acceptance
Tests Deploy (Prod) Monitoring and Automated Alerts
Continuous Deployment release pipeline
CI
ApprovalApproval
FeedbackFeedback
Feedback
Dev Pre-Production (“Princess”)
Production
Check inTrigger
Feedback Trigger
Feedback Approval
ApprovalFeedback
Smoke Tests User Acceptance!Tests ReleaseDev Team Version Control Build & Unit
TestsAutomated
Acceptance TestsUser Acceptance
Tests Deploy (Prod) Monitoring and Automated Alerts
Continuous Deployment release pipeline
CI
ApprovalApproval
FeedbackFeedback
Feedback ApprovalFeedback Approval
Feedback
“Allow buttons properly to inherit color from their parent node.”
Five years ago…2-3 weeks of code changes
Release and rollback plans
Traffic and infrastructure management (Ops)
6-14 hours
Five years ago…“Deployment Army”
Stressful, especially when things go wrong
Long days and late nights
Scheduled downtime
Why do we do this?
Innovate or die.
Resolve scaling hurdles.
Mean-time-to-recovery.
Healthy and talented engineering team.
Stop stressing about releases.
Innovate or die.
Resolve scaling hurdles.
Mean-time-to-recovery.
Healthy and talented engineering team.
Stop stressing about releases.
Admin-launch and whitelist
Ramp-up public traffic
mainline
header_redesign
search_filter_custom_orders
checkout_blue_button
listing_css_refactor
www.etsy.com
beta01.etsy.com
beta02.etsy.com
beta03.etsy.com
www.etsy.com
beta01.etsy.com
beta02.etsy.com
beta03.etsy.com
date":"30\/Mar\/2014:12:49:48","locale_currency_code":"USD","pref_language":"en-
US","region":"US","detected_currency_code":"USD","detected_language":"en-
US","detected_region":"US","accept-languages":"en-US","cdn-
provider":"","isMobileDevice":"0","isMobileSupported":"0","isMobileRequestIgnoreCookie":"0"
,"isTabletSupported":"0","isTouch":"0","isEtsyApp":"0","isPreviewRequest":"0","isChromeInst
antRequest":"0","isMozPrefetchRequest":"0","listing_ids":
[104073511,130604774,159651433,155451607,160523743,124025232,95186610,82967340,114692884,11
4767467,117266897,157579748],"scheduled_modules_content_ids":
[10808052776,10256029946],"primary_event":"1",".event_source":"web",".event_logger":"fronte
nd","php_ab_test_names":"translation_profiler.profiling;translation_profiler.logging;transl
ation_profiler.backend_event_logging;footer_redesign_20131201;international.languages.el;in
ternational.languages.ja;international.languages.no;international.languages.pl;internationa
l.languages.ro;international.languages.tr;simplified_locale_experience;full_site_ssl;admin_
toolbar;enabled_locale_subdirectories;affiliates.publishing.user_publishers;buyer_invites_r
ecipients;home_improvement;home_improvement.new_homepage;authoritative_items;refactored_foo
ter;conversations.rejuvination;contextual_homepage_recs.global;css_from_www;shrinkray.css;c
srf_nonce_refactor.allow_colon;csrf_nonce_refactor.reverse_order;csrf_nonce_refactor.no_encAnalytics connected to config names
Catapult
Observed impact
Time series data for duration of the experiment
Observed impact
Time series data for duration of the experiment
FrankProduct Manager
“I want to find out whether buyers will favor a single price for the product that includes shipping.”
https://www.etsy.com/shop/lucra
Eligibility requirements:
- Must be first page of visit
- Buyer & seller in same region
- etc…
Time: < 8 hours Staff: One !
Design, config flag (disabled), eligibility code in controller, template code, CSS, code review, automated tests, deployed code, config flag enabled.
We do not bundle the item price and shipping cost together today. !
https://www.etsy.com/shop/lucra
Ambitious Product Goal
Ambitious Product Goal
Monolithic Building and measuring many things at once.
Ambitious Product Goal
Monolithic Building and measuring many things at once.
Iterative One thing at a time, our design goal is always in sight.
Time: < 8 hours Staff: One !
Design, config flag (disabled), eligibility code in controller, template code, CSS, code review, automated tests, deployed code, config flag enabled.
Deployed Deployed
Is this for me?
http://timothyfitz.com/2009/02/10/continuous-deployment-at-imvu-doing-the-impossible-fifty-times-a-day/
“Maybe this is just viable for a single developer … your site will be down. A lot.”
etsystatus.com
@mikebrittain
Very end of 2009 Today
DEPLOYMENTS PER DAYAPP CODE CONFIG FILES
$1.35 Billion Goods sold in 2013 60+ Million Unique visitors per month !
175+ Committers, everyone deploys
http://www.etsy.com/blog/news/2013/etsy-statistics-december-2012-weather-report/Items by anjaysdesigns, betwixxt, OneStarLeatherGoods, mediumcontrol, TheDesignPallet
Thank you.
Mike Brittain
Engineering Director, Etsy
@mikebrittain mikebrittain.com/talks
top related