Continuous Deployment at Etsy: A Tale of Two Approaches

Continuous Deployment at

March 9, 2013

Ross [email protected]@beamrider9

:A Tale of Two Approaches

1

A quick primer on

2

is:The global marketplace we make together.

3

is:The premier destination for handmade goods,

vintage items, and craft supplies.

simplertimestoys lacklusterco norwesterseaglass

4

quick facts:

• 22+ million members

• 800,000+ active shops

• 18+ million items currently for sale

• 20 cents to list item, 3.5% transaction fee

• 400+ employees (majority in Brooklyn)

(as of March 2013)

5

$0

$200

$400

$600

$800

$1,000

2005 2006 2007 2008 2009 2010 2011 2012

Gross Merchandise Sales ($MM)

Since opening its doors in June 2005, Etsy has grown virtually non-stop.

6

A nice problem to have:

“Our site is so successful, how can we

move fast enough to keep up with demand?”

7

CONTINUOUSDEPLOYMENT

8

:The Early Years

(2005 - 2008)

9

: The Early Years

1. Spend significant time writing code

10

: The Early Years

2. Painful source control merge

1. Weeks writing code

11

: The Early Years

3. Hand off to someone else to deploy

1. Weeks writing code2. Painful merge

12

: The Early Years

4. Deploy, site goes down

1. Weeks writing code2. Painful merge3. Hand off to deployers

13

: The Early Years

5. Roll back deploy

1. Weeks writing code2. Painful merge3. Hand off to deployers

4. Deploy, site down

14

: The Early Years

6. Spend hours (days?) fixing bugs

1. Weeks writing code2. Painful merge3. Hand off to deployers

4. Deploy, site down5. Roll back deploy

15

: The Early Years

7. Go back to step 2

1. Weeks writing code2. Painful merge3. Hand off to deployers

4. Deploy, site down5. Roll back deploy6. Fix bugs

16

: The Early Years

WATERFALL!

17

: The Early YearsPros:

Early Etsy engineers used this release cycle to bootstrap the marketplace from nothing.

Forever grateful.

18

: The Early Years

• Large changesets• Infrequent deploys• Weak confidence in deploy success• Significant time spent deploying• Low ability to experiment/iterate/react• Developer stress/unhappiness

Cons:

19

: The Early YearsBy late 2008, Etsy is still a startup, but has

the deploy process of a much bulkier company.

Popularity is on the verge of outpacing capacity.

20

:Today

21

: Today

1. Small changesets, deployed frequently

22

: Today

2. Engineers deploy the site

1. Small changesets

23

: TodayAnd not just engineers, but also:

• Designers• Product Folks• Upper Management• Board Members• Dogs

24

: Today

3. Deploys are fast and near-effortless

1. Small changesets2. Engineers deploy

25

: Today

4. Most changes behind config flags(safer deploys)

1. Small changesets2. Engineers deploy3. Deploys are fast

26

: Today

5. Graphs/metrics to assess deploy

1. Small changesets2. Engineers deploy3. Deploys are fast

4. Changes behind flags

27

: Today

6. If issues, fix immediately & roll forward

1. Small changesets2. Engineers deploy3. Deploys are fast

4. Changes behind flags5. Copious graphs/metrics

28

This isn’t license to break stuff, quickly.

Engineer-driven QA and solid unit testing are integral parts of the process.

: Today

29

: Today

7. Repeat 25+ times per day, every day

1. Small changesets2. Engineers deploy3. Deploys are fast

4. Changes behind flags5. Copious graphs/metrics6. Fix fast & roll forward

30

Then:

1. Small changesets2. Engineers deploy3. Deploys are fast4. Changes behind flags5. Copious graphs/metrics6. Fix fast & roll forward

1. Weeks writing code2. Painful merge3. Hand off to deployers4. Deploy, site down5. Roll back deploy6. Fix bugs, go to step 2

Now:

31

Etsy Deploy Stats: 2012

• Deployed to production 6,419 times

• On average, 535/month, 25/day

• Additional 3,851 config-only deploys

• 196 different people deployed to prod

• Nov/Dec 2012: deployed 752 times

32

Why does it work?

33

• N = # of deploys• P = probability of site degradation• S = average severity of degradation• T = time to detect/resolve

= N * P * S * TExpectedDowntime

Continuous Deployment Math

34

• N = 1• P = 0.5• S = 0.7• T = 100

Before:

E.D. = 35

• N = 250 ↑↑↑↑• P = 0.1 ↓• S = 0.05 ↓↓• T = 5 ↓↓↓

Now:

E.D. = 6.25

N = # of deploys P = prob. of degradation

S = avg. severity of degradation T = time to detect/resolve

(all numbers completely arbitrary)

Continuous Deployment Math

35

Etsy circa 2013 (400+ employees)acts, in some ways, more like a startup than

Etsy circa 2008 (40+ employees).

Big Takeaway

36

Continuous Deployment makes possible: “Continuous Experimentation”

http://etsy.me/continuous-experimentation

37

Continuous Experimentation

1. Small changes2. Run experiment (A/B test)3. Analyze data4. Re-examine assumptions

Repeat continuously in pursuit of larger goals.

38

Heard since 2010: “Neat experiment, but this will never scale.”

As of 2013, Etsy has 100+ engineers -still going strong.

39

Some Etsy Customizations

Deploying is a first-class feature. Inability to deploy is a P1 incident (same as site down).

40

Some Etsy Customizations

• Internal admin site• API • Big data• Search• Blog• Deployinator itself

We continuously deploy not just the main Etsy website, but as much as possible:

41

Some Etsy Customizations

• Database schema changes• PCI-DSS environment (credit cards)

In the rare case we can’t continuously deploy, we create alternative tools:

We do continuously deploy as much of our payment processing as is safe & legal (98%).

42

Some Etsy Customizations

Keeping deploys fast is paramount and worth the investment in manpower & hardware.

43

Some Etsy CustomizationsContinuous deployment is all about moving

forward, sometimes at the expense of the past.

Our solution: engineering-wide bug rotation, one day a month, every engineer participates.

44

Continuous Deployment is a fantastic recruitment tool for attracting engineers who like to move fast and get stuff done.

Fun Fact:

45

Learn more:http://codeascraft.etsy.com/

Join the fun:http://www.etsy.com/careers

Etsy open source (Deployinator, StatsD)http://etsy.github.com/

46