Fast and Good: Alternate Approaches to Quality at Etsy - STPCon fall 2011

Noah Sussman STPCon Fall 2011, Dallas TX

#STPCon

Alternative Approaches to Quality at Etsyfast and good

Our mission is to empower people to change the way the global economy works. We see a world in which very-very small businesses have much-much more sway in shaping the economy, local living economies are thriving everywhere, and people value authorship and provenance as much as price and convenience. We are bringing heart to commerce and making the world more fair, more sustainable, and more fun.

September 2011: 489 Web deployments

110 people committed to trunk 4,519 changesets in all 9.2 changesets average per production deployment

September 2011: 489 Web deployments

80 people deployed to production 20 or fewer committers in each deployment 5.7 committers average per deployment

10 million registered members25 million unique visitors per month800,000 active sellers150 different countries10 million listed items$314 million in sales in 2010

In 2010, MTTR was 6.4 minutes for severity one outages

Six change-related outages in 2010Out of thousands of production deployments

“All existing revision control systems were built by people who build installed software.”

~Paul Hammond

All canonical software life cycles were designed by people who build installed software

“In the online world… there is one consumer of the software: you. The users are consumers of the service built atop the software.”

~ Theo Schlossnagle

Optimize for recovery rather than focusing on preventing failure.

Failure is inevitable.

Default to open access

No blame

Use your own judgement

“an airport without an air traffic controller”Chad Dickerson

faster deploys = smaller changesets = fewer change-related outages

Self-serve deployment

bleeding edge code,production data and services

princess

Shared development environment

Single dev databaseEveryone commits to trunk

low-ceremony process

A test is a design document first

Many small deployments, many small tests

Beyond developer testing1. Test planning2. GUI-driven regression testing3. Performance testing4. Security / Penetration testing5. Nagios

Make it easy to write and run tests

Make it painless to use fixtures and mocks

Most bugs, most of the time, are easily nailed given even an incomplete but suggestive characterization of their error conditions at source-code level.

~ESR, “How Many Eyeballs Tame Complexity,” 1997

Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it.

~ Brian Kernighan

Test to address technical debt

Keep the feedback loop short

Graph or it didn’t happenAt Etsy we monitor over 40,000 real-time metrics

Business Metrics

Application Metrics

System Metrics

We hire our users

Tests are one of the tools we use to detect and prevent failure

There will always be bugs in production

Unexpected failures tend to be catastrophic

Anomaly detection is hard

Just ship

How do you deploy large features?

Releasing a feature is decoupled from deploying code

if ($cfg["new_search"]) {// new hotness$resp = search_solr();

} else {// old busted$resp = search_grep();

}

$cfg = array( 'checkout' => true, 'homepage' => true, 'profiles' => true, 'new_search' => false,);

Dark launchesAdmin-only rolloutsA/B campaignsetsy.com/prototypesWe are always testing

A culture of responsibility

A community of artisans

You are a participant observer

etsy.com/careers

Noah Sussmannoah@etsy.com@noahsussman

photo creditspenguins: paulmcdee http://www.flickr.com/photos/paulmcdee/2806715569aerial photograph of snowy road: K. Leidorf http://leidorf.blogspot.com/search/label/20D03855death in the mirror: Richard Avedon for the New YorkerWomen at work on C-47 Douglas cargo transport http://www.flickr.com/photos/library_of_congress/2179137415/in/photostream/Woman aircraft worker http://www.flickr.com/photos/library_of_congress/2179930812/cool bike with wooden handlebars: Bertelli http://www.bertellibici.com/product.php?works_id=112clover keys: Brian Cerveny http://www.flickr.com/photos/briancerveny/48889960/in/photostream/Etsy office monitor wall and Etsy bike rack: Mashable's Cubicle Spy http://mashable.com/2010/10/13/etsy-office-picshorse fly: Lukjonis http://www.flickr.com/photos/38628972@N05/5884111163/in/photostream/Gandalf vs balrog: the brothers HildebrandtJust Ship t-shirt: Kellan http://www.etsy.com/shop/kellanem/soldbarn raising: dogpose http://www.flickr.com/photos/8371074@N05/2865798475/model train coupling: David E. Smith http://www.flickr.com/photos/david_e_smith/3566697122/sizes/l/in/photostream/

If I used your image and didn't credit you, it's because I couldn't figure out who you were -- email me and I will gladly credit you. If you object to my using an image to which you own the rights, please contact me and I will remove that image from the deck.