Rails Hosting and the Woes (Engine Yard)

by Jamie van DykeEngine Yard Developer (UK)

Hosting and the Woes

Jamie van Dyke

My name is Jamie van Dyke, and in case you can see me from the back, I look ...

like this fella here. I used to be the ...

support

...Application Support Manager for Europe, but now I’ve transitioned over to ...

develop cool stuff

the Automation Development team over in ...

England

...the UK. ( So we do 24 hour development as well as support now ;-)

...just in case you forgot about us, we’re this little island here...we don’t have as many nukes as you...but we’ve got Gordon Ramsay who’s even scarier! Enough about me, here’s...

Tom Mornini

...Tom Mornini, who looks like...

...this. Study the face well and feel free to grab him at any time during the conference to talk shop, or just question about anything. Tom is our...

The Chief

...CTO, Tom has experience in...

jack

every technology at Engine Yard...and can put many to shame...

of all trades

...in most of these areas. Moving on, we have...

Taylor Weibley

...Taylor Weibley. Taylor looks...

...like this. Taylor is our...

king

...Application Support Director. He’s the man responsible for keeping everyone happy, so if you’re a customer and you’re not (is that even possible?), he’s the man to speak to. He has tonnes of skills in many...

Rails ++

...areas, of course Rails is one of them, but we consider him to be...

...one of our Supermen in the App Support Team. Finally, we have...

Edward Muller

...Edward Muller. Edward looks like...

...this, and he’s one of those incredibly clever people you meet that...

fluent in binary

...is able to talk to a computer instead of typing on the keyboard. Ed was and to some degree still is our...

cluster ninja

...Cluster Engineer. After Jayson Vantuyl conceived and built ey00, our first cluster, Edward went on to build all the others and make improvements along the way. He’s now my ...

automation manager

... Manager, in the automation department. We basically improve the lives of the Engine Yard staff, by automating the processes they do every day. For example, deploys can take a few hours manually, but with a sprinkling of automation we can pull that down to a few minutes.

</intro>

Enough about us, let’s talk about some of the problems that our customers have bumped into and how to solve them.

Problems

Through hundreds of customers and hours of debugging, we’ve found a recurring set of problems that crop up with Rails development, and we’d like to make you aware of them if you aren’t already. No surprise, top of the list...

active_record

...is Active Record. Too many times...

...I’ve seen a find(:all) that goes crazy and chews up memory like there’s unlimited banks of it. You all know the solution to this, in views you use pagination and in processing loops you use batches. The main cause of this going unnoticed before hitting production, is that...

development

...on a development machine you have a small set...

20 records

...of records. Your tests run against them and everything works fine. Your machine has 2Gb of RAM and you don’t sit watching the memory usage as you test. However...

production

...in production your site might be really popular and you’re running...

millions of records

...millions of users and assets for each of those. Running a find(:all) on that would probably cause a problem. :-/ So paginate, and make sure you have...

indexes

...properly indexed your tables. At Engine Yard we use a team of DBA’s and the MySQL Performance Blog Experts to help our customers get the best speed out of their databases.

plugins

There are a handful of technologies and plugins that cause problems for a lot of our customers too. For example...

ferret

...the popular ferret indexing engine along with numerous different plugins that provide interfaces to it, is constantly corrupting indexes. Initially you’d think that it’s multiple processes accessing the same index...but we always use ferret server, so it’s not that. Our solution is simple and effective...

sphinx

...switch over to sphinx, which has a lot less problems (if any) and does the same job. We’ve seen ferret indexes at 5Gb that are reduced to 1Gb when moved to sphinx. In conjunction with Evan Weaver’s ultrasphinx plugin, this is the ideal solution.

image science

image_science is great but there are a couple of gotchas for when you shift to using it in production, with monit...Image Science requires environment variables (INLINE_DIR or HOME) set up to work, monit clears these variables...so either use a wrapper script or set them in your app manually.

hodel 3000

...hodel compliant logging allows you to use a specific log format that reporting tools can handle. However, out of the few plugins that are out there for it, they neglect to inform you that you *need* to set logging to :info, otherwise your logs will (default to debug), be absolutely huge, and they don’t need to be if you want your reporting tools to work.

...ever wonder what kind of traffic you can expect from different media coverages? okay, let’s have a look at a few...

can haz trafficz?

...ever wonder what kind of traffic you can expect from different media coverages? okay, let’s have a look at a few...

Digg

so you get featured on digg’s homepage, how much traffic? Well, you’re looking...

10,000+ visitors

...at tens of thousands of visitors...but they’re all just checking out what’s new and featured, because...

signups --

...you’ll rarely get many signups to your service. It’s all about seeing what’s new. Which is exactly what we can say...

TechCrunch

...about TechCrunch. The only difference being the level of traffic, which means more likely...

1000+ visitors

...thousands of visitors, you might hit tens of thousands but it depends on what your site is all about. A Ruby specific site...

Ruby Inside/Flow

...like Ruby Inside or Ruby Flow, will send you between...

500-1000 visitors

...five hundred and a thousand visitors a day, and you’ll get that for a few days to a week, you’ll also get virtually...

signups --

...no signups from that. Which if you’re a ruby/rails specific site, will be higher than if you’re not. It’s all whether your site is relevant to the market with these smaller sites, when I say smaller I mean not on the scale of ...

The Today Show

...the Today Show, which is a great traffic provider, if you’re hosted with a provider who can scale you quickly and provide you with a team of Rails experts that will help you through it, by both scaling up your service, as well as providing quick advice on how to increase the efficiency of your pages. Someone like us ;-) You’re looking at...

100,000+ visitors

...a huge number of visitors which is dependent on your site content, of course. One of our customers called Catalog Choice got featured on the Today show and the traffic spike was enormous, as well as obtaining...

10,000 signups

...them ten thousand signups in the first hour alone. So this is definitely one of the media spots you want to get featured on. Another, not surprisingly, high traffic provider ...

Fox NewsBusiness Show

...that could do you well, is Fox News Business Show. From just showing a site url on the tv, you can expect...

2000+ connections

...a whole chunk of visitors. 2000-3000 simultaneous visitors come rushing your way. Pray to the Gods that you’re ready to scale up for this one. ;-)

$10,000,000

The ten million dollar question is always, how many users can one of our slices handle. Well, how long is a piece of string?

can you code?

Of course it’s all dependent on what your code quality is like. We can say that an average Rails application running over 3 mongrels, can trivially handle millions of unique visitors a month.

queries per page

So, take a look at how many queries are running per page...

cache cache cache

... cache wherever possible...

file io

...keep file io down to a minimum...

separate services

...and if you can, move your services on to different servers. Like assets in one place, application in another, database in another, etc.

goodies

Let’s wrap up the talk with a mention of some open source goodies we have that you can download...then we’ll move on to the Q and A section.

eycap(gems.engineyard.com)

This is a gem we developed that extends capistrano, it allows logging of deploys, has tons of tasks for sphinx / ferret / memcached / etc. If you use nginx, or monit, or sphinx or....whatever, then this gem will be handy for you, download it using ...

gem source -a http://gems.engineyard.comgem install eycap --no-rdoc --no-ri

(in deploy.rb) require ‘eycap’

...these commands, with sudo if you’re on a mac...and you get lots of capistrano goodies...like

...these.

ask us anything

We filled the rest of the presentation with a q & a section, which went extremely well. Thank you to everyone who came and participated.