Development, Deployment and Collaboration at Etsy
Jan 15, 2015
At Etsy about 150 engineers deploy a single monolithic application more than 60 times a day. This process of deploying small changesets continuously enables us to build up and release robust features and detect and fix bugs extremely fast. All while serving over a billion page views per month. Developing and deploying at such a high velocity however only works because product developers and designers, infrastructure and operations engineers and the security team work closely together. We have an extremely open culture of sharing (inside and outside the company) and make sure we run into as few surprises as possible by bringing everybody on the same page about changes.
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Development, Deployment and Collaboration at Etsy
Daniel Schauenberg
@mrtazz
@mrtazz
@mrtazz
@mrtazzItem by TheBackPackShoppe
http://www.flickr.com/photos/brianglanz/1095706242
avg 50 deploys/day
avg n > m deploys/day
How comfortable are you deploying
a change right now?
@mrtazzhttp://www.flickr.com/photos/renaissancechambara/2349811492
small change
Config Flags
Item by RocajoStudio
“If this is your first day at Etsy, you deploy the site”
Developer VMs
@mrtazz
Developer VMs
• KVM
• Every engineer has one
• Fully Chef’d with the Etsy Stack
• Different sizes and Chef roles
Continuous Integration
@mrtazz
Continuous Integration
• Run set of tests before each deploy
• Full QA suite
• Princess/Production smoker tests
• Try (yup, there is one)
@mrtazz
The Bobs
• LXC virtualized hosts
• 14/physical hosts
• Spread over 3 SSDs
• Most of them attached to try
Item by decomodwalls
Deployinator
@mrtazz
Deployinator
• 2 Buttons, no ambiguity
• Overview of current state of deploy
• Links to Logwatcher and Dashboards
• Easy to add stacks for new tools to deploy
http://www.flickr.com/photos/jbgeronimi/6363087361
Monitoring
@mrtazz
Monitoring
• Devs do their feature monitoring
• Everybody can access all the graphs
• Dashboard All The Things!
• Stream All The Logs!
On Call
If you are writing code, you are
on-call
@mrtazz
On-Call Schedules• ops on-call
• dev on-call
• payments on-call
• support on-call
@mrtazz
Dev On-Call
• Scheduled for 6 months
• On-call roughly every 4 weeks for 1 week
• L1 and L2 escalations
• L1 if it’s your first time
Incident Response
@mrtazz
Incident Response
• “This graph looks funny”
• “Hey I just got paged for elevated error rate after deploys”
• “Supergrep is going crazy!!”
#warroom
@mrtazz
#warroom
• only outage related conversations
• coordinate investigation, communication, countermeasures and monitoring
• good place to lurk for new engineers
Post Mortems
blameless
Everybody’s invited
Learning Opportunity
Summary
@mrtazz
Summary
• These are things that work for *us*
• Culture is an on-going effort
• Share everything
• Encourage learning/teaching
@mrtazz
Summary
• Lunch ’n learns
• DC visits
• On-call for a day
• Bootcamps/Senior rotations
Questions?
Development, Deployment and Collaboration at EtsyDaniel Schauenberg
Related Documents
