NPW2009 - my.opera.com scalability v2.0

Nordic Perl Workshop ~ Oslo 2009cosimo streppone <[email protected]>

my.opera.com scalability v2

2000 2001 2002 2003 2004 2005 2006 2007 2008 2009

1 10 50257 205

430

887

1.640

2.500Users (k)

2000 2001 2002 2003 2004 2005 2006 2007 2008 2009

1 10 50 257 205 430887

1.6402.500

Users (k)ServersDyn req/s

espenao> proxys all over. MOC is dying?hkmoen> did MOC just die?...fred joined #myoperafred> lots of proxys...fred left #myopera...cosimo> WTF? ok, looking into it

static avatars(or “put your http servers at work”)

new storage subsystem

pools, serversfault tolerance, redundancy

webdav, http, ftp,scp, mogilefs?, ...

# Create resource object for avatarmy $res = MyOpera::Storage::Resource::Avatar->new( owner => '{userid}', content => '{binary data}',);

# Main storage subsystem handlemy $storage = MyOpera::Storage->new();

# Upload on pools of servers all at once my $ok = $storage->upload($res);

user uploads use case

resources(user uploads, binary blobs, ...)

pools, servers

URLshttp://static.myopera.com/pool1/avatars/a4/754/a1b2c3d4e5f6.../<userid>_o.pnghttp://static.myopera.com/pool1/avatars/a4/754/a1b2c3d4e5f6.../<userid>_t.jpghttp://static.myopera.com/pool1/avatars/a4/754/a1b2c3d4e5f6.../<userid>_m.jpghttp://static.myopera.com/pool1/avatars/a4/754/a1b2c3d4e5f6.../<userid>_l.jpg

package HTTP::DAV;

...

## Retrieves info about a DAV url## depth=0 avoids descending into dirs#sub propfind { my ($self, $url, $depth) = @_; $depth ||= 1;

# ...

}

crappy CGI-style URLs

new cool static URLs

http://my.opera.com/<user>/avatar.pl

http://static.myopera.com//avatars/pool1//dL/rLA/vdUxYn14kfe342QEM//<user-id>_s.jpg

/* results */

saved ~500k backend req/day

browser cache used!

shared-nothing architecture

(“zero tolerance for sharing”)

nfs serverdisk cacheusers store

. . . . .

lvs +lw httpd

mod_perlbackends

dogpile effect(or the cache “storms”)

nfs-based file cache/var/cache/myopera//var/cache/myopera/feeds//var/cache/myopera/blogs/...

/var/cache/myopera/user//var/cache/myopera/user/1/var/cache/myopera/user/10/var/cache/myopera/user/11/var/cache/myopera/user/2...(another 2 million entries).../var/cache/myopera/user/1999999

(on ext3)

soft counters

use MyOpera::Counter;

my $counter = MyOpera::Counter->new({ key => “blog.access-$entry”, sync => sub { my ($value) = @_; # Real table update code ... }});

$counter->inc();

dml statistics

[DML] time=1237308152, user=, url=/tinh_yeu_cua_anh_b88/blog/index.dml/tag/...,name=XWA::User, variable=active, type=module, elapsed=0.068473, host=my.opera.com[DML] time=1237308152, user=, url=/community/,name=XWA::User, variable=, type=module, elapsed=0.015935, host=my.opera.com[DML] ...

top time-intensive modules

XWA::XMLProd::User::Sidebar 2024.919s (27.2%, 0.28 s/call)XWA::User 1778.445s (23.9%, 0.09 s/call)XWA::User::Journal 1121.224s (15.1%, 0.24 s/call)XWA::User::Album 321.522s ( 4.3%, 0.17 s/call)XWA::User::Journal::Search 223.477s ( 3.0%, 20.32 s/call)XWA::User::Comments 188.011s ( 2.5%, 0.05 s/call)XWA::Skins 180.486s ( 2.4%, 0.49 s/call)XWA::User::JournalArchive 159.525s ( 2.1%, 4.43 s/call)XWA::User::Posts 146.644s ( 2.0%, 0.45 s/call)XWA::User::Picture 141.324s ( 1.9%, 0.10 s/call)XWA::Albums 93.740s ( 1.3%, 2.04 s/call)XWA::Journals 92.390s ( 1.2%, 2.37 s/call)

most called modules

XWA::User 18784 calls 28.9%XWA::XMLProd::User::Sidebar 7186 calls 11.1%XWA::User::Spotlight 6439 calls 9.9%XWA::Util::OneTimeKey 5576 calls 8.6%XWA::User::Journal 4681 calls 7.2%XWA::OIVWrapper 3974 calls 6.1%XWA::User::Comments 3890 calls 6.0%XWA::User::Album 1881 calls 2.9%XWA::User::Ratings 1500 calls 2.3%XWA::User::Picture 1485 calls 2.3%XWA::XMLProd::MemberCount 611 calls 0.9%XWA::XMLProd::Queue 589 calls 0.9%XWA::Spotlight 569 calls 0.9%MyOpera::DML::FastFacts 567 calls 0.9%XWA::XMLProd::Variables 562 calls 0.9%

slowest modules

XWA::User::Journal::Search 20.32s (11 calls)XWA::User::JournalArchive 4.43s (36 calls)XWA::User::GeneratePassword 3.76s ( 3 calls)XWA::XMLProd::Link::Bookmarks 3.45s (12 calls)XWA::Fans 3.37s (10 calls)XWA::User::ForumLatestFriends 3.36s (11 calls)XWA::User::ForumNew 2.42s ( 3 calls)XWA::Journals 2.37s (39 calls)XWA::Albums 2.04s (46 calls)XWA::XMLProd::UserSearch 1.87s ( 2 calls)XWA::Jahoo 1.83s (46 calls)XWA::Forum 1.64s (49 calls)XWA::XMLProd::Link::Trash 0.86s ( 2 calls)XWA::XMLProd::LocationSearch 0.72s (43 calls)XWA::User::ForumSearch 0.67s (78 calls)

?questions?

# iptables -L

Chain INPUT (policy ACCEPT)target prot opt source destinationACCEPT all -- x.x.x.x/x anywhere limit: avg 20/sec burst 5ACCEPT all -- y.y.y.y/x anywhere limit: avg 20/sec burst 5DROP all -- x.x.x.x/x anywhereDROP all -- y.y.y.y/y anywhere

low level TCP/IP “throttling”

full-page caching

- limiting “barrier”

- url hotlist

mod_cband

Web site monitoring,in case the live demo fails

:-)