Envato Dev Ops - Alt.Net Melbourne

Dev Ops, Ruby, Rails, and Envato

A team of 8 deploy heaps of times a day to a Ruby on Rails app that has scaled up to around 20 million

requests a week without an ops team.

lolwut?

Thursday, 30 June 2011

John Barton

@johnbarton

[email protected]


mailto:[email protected]

mailto:[email protected]

Envato?

http://envato.com


http://envato.com

http://envato.com

Stock Marketplaces&

Tutorial Network


Marketplaces


Like iStockPhoto but for other creative niches

oreBay for digital goods


Tuts+ Network


Big Blog Network for education in creative fields


The Marketplace


August 2006


One Marketplace


FlashDen


Rails 0.13bNo UsersNo Traffic


June 2011


9 Marketplaces


ActiveDen (nee FlashDen)AudioJungle

ThemeForestVideoHive

GraphicRiverCodeCanyon

3D OceanTuts+ Marketplace

PhotoDune


Rails 2.3.11830,462 Users

22 Million Requests Weekly


One Codebase and One Production Environment


What’s interesting about our deployments?


deploy frequently, high scale, no ops team


What do you need to do the same?


Some technical infrastructure, some

business/cultural


Cultural Bits


The “Dev Ops” Culture


“Culture of respect & trust, good attitude toward

failure...”

Ted Dzubia


“How about "culture of stop fucking up"?”

Ted Dzubiahttp://teddziuba.com/2011/03/devops-scam.html


http://teddziuba.com/2011/03/devops-scam.html

http://teddziuba.com/2011/03/devops-scam.html

A problem of competing priorities


Our Solution:Don’t have Ops


Outsource commodity platform bits:

virtualisation/cloud, have rackspace take care of db/

mailserver


Ensure the dev team has the skills to take care of the

rest


Take advantage of an individual's talents, but don’t rely upon them

ie. don’t accidentally create an ops guy


OK to have specialists... for special work.

Keeping the app running isn’t “special”


Within the Business


Preserve The “Golden Age”



3 Developers30 minute feedback cycledeploy, discuss on forums,

deploy again


Keep doing everything good about the the startup days

for as long as we can


Avoid Corporate Antipatterns


We’ve all been doing dev long enough to see this

stuff screwed up over and over again


Decision 3:Don’t do all that stuff


Ultimate Responsibility


"The fault, dear Brutus, is not in our

QA or Ops,But in ourselves."


Test Driven Developmentvs.

QA Team


Test Driven Infrastructurevs.

Ops Team


Both as a team and as individuals we own our work from when we are

asked to do it...


... until is is demonstrably error free and performant

in production


Everyone is in the (paid) on call roster


Everyone takes a turn at Level 2 Customer Support


Want those jobs to be easier?

Stop fucking up.


Process


LEAN / TPS Principles

...without the process



You cannot write code any faster than you can deploy

it to production


Long running projects?


A. B. C.

Always Be Cmerging

(the c is silent)


I would not want to try this with Subversion, or Visual Source Safe... and I never had enough money to try “Team Foundation Server”


Use GIT

http://git-scm.com/


http://git-scm.com/

http://git-scm.com/

Dark LaunchFeature FlagsPrivate Beta


User Community


We do trip up running this fast


But through years of openness with our users

via our forums and owning up to our mistakes


... we’ve ended up with a (relatively) sympathetic

community


Time Zones both help and hurt


Traffic peaks during US day means that if things go

wrong we’re usually asleep


But it makes it very easy to deploy during our business

hours


Open Source


No point you solving the same problem as the guy

down the street


No point the guy down the street solving the same

problem you did last week


Technology


Commodity Hosting is a prerequisite


Notice I’m not saying “Cloud Computing”




You can't trust “the cloud” but

you can trust “the cloud to be “the cloud”.


Make a conservative platform choice so we don’t have to sweat the

detailsie. Ubuntu 10.4 LTS


I guess you guys have to choose like Windows

Ultimate 2009 Edition?


Seriously though, Ruby guys love shiny new things, and we’re spoiled for choice

_even_ when it harms us.


So we make an effort to keep the stack as Vanilla as

possible


Virtualised servers in our own sandbox.

Cloud Flexibility - Cloud Shit-ness = WIN


Automate Configuration Management


Babushka

http://babushka.me/


http://babushka.me/

http://babushka.me/

Test drive your architecture...


...or borrow “deps” built by other people


Lets look at some code


Babushka is designed to be run manually on the

remote machine


Use Dollhousehttps://github.com/geelen/

dollhouse

To drive babushka remotely


https://github.com/geelen/dollhouse




moar code


A lot of people I take seriously, take Chef

seriously.

http://wiki.opscode.com/


http://wiki.opscode.com/display/chef

http://wiki.opscode.com/display/chef

Automate Deployment


Capistrano

https://github.com/capistrano/capistrano






Codes?


The un-sung hero


rake db:migrate


I’m so used to having it I can’t even remember how I

ever deployed database changes in the past


Performance & Scaling


Not as big a deal as everyone thinks


.NET runtime is faster... but it’s really not that big a deal


Shared-nothing load balanced app servers + out of request queue workers

not rocket surgery


Measure, deploy, measure again and then tweak or

rollback

New Relic FTW


There is no code faster than no code.


Why I’m happier than I was as a .NET programmer


Mainly because ofSystem.Threading.Timer


No Viewstate

(Yes I heard you’ve got MVC now instead)


A culture of people who fix things instead of waiting for

them to be fixed


People who care about expressiveness of code and

finding the right abstractions


Question?Thursday, 30 June 2011

Envato Dev Ops - Alt.Net Melbourne

Technology

ops teamthursday

ops guythursday

dev ops culturethursday

rails app

dev team hasthe skills

qa teamthursday

team foundation serverthursday

git http