Games for the Masses - Wie DevOps die Entwicklung von Architektur verändert (Webinale)

GAMES FOR THE MASSESWie DevOps die Entwicklung von Architektur verändert

Jesper Richter-‐Reichhelm, @jrireiMonday, June 4, 2012

DEV -‐ OPS

Monday, June 4, 2012

DEV -‐ OPS

Some say: Admins learn to be ‘Agile’

Scrum and Kanban


DEV -‐ OPS


Scrum and Kanban

Others say: Admins learn to program their setup

Chef, Puppet


DEV -‐ OPS


Scrum and Kanban

Others say: Admins learn to program their setup

Chef, Puppet

I say: This is all nice, but not enough

Developers need to learn a lot about opera?on


Classic roles in IT

Op

ArchDev


Classic roles in IT

Op

ArchDev


Classic roles in IT

Op

ArchDev


Classic roles in IT

Op

ArchDev


Classic roles in IT

Op

ArchDev


Classic roles in IT

Op

ArchDev


Classic roles in IT

Op

ArchDev


Classic roles in IT

Op

ArchDev


Classic roles in IT

Op

ArchDev


Classic roles in IT

Op

ArchDev

???



Our games all look the same

BackendFlash client



Game Session

Asynch. CommunicaPon

Flash client


Backend


State Changes

ValidaPon

Persistence


But the scale is interesPng

14 billion requests / month







>100,000 DB operaPons / second




>100,000 DB operaPons / second

>50,000 DB updates / second


Classic roles in IT

Op

ArchDev

???


Roles in Wooga’s IT


Arch



ArchArchDev



ArchArchDevArchDevOp







Wooga has dedicated game teams


Architecture EvoluPon at Wooga

The Start

The Next Step

Best of Two Worlds

Company Values


First game: Try to be flexible

Arch



Arch

• Ruby on Rails



Arch

• Ruby on Rails• Amazon EC2



Arch

• Ruby on Rails• Amazon EC2• MySQL


EvoluPon I: Use Ruby (on Rails)

Oct 2009


Basic setup using sharding

app app app app app app app app app

lb

MySQL

MySQL

slave slave


250K daily users

!"

#!!$!!!"

%$!!!$!!!"

%$#!!$!!!"

&$!!!$!!!"

'()*%!" +,-*%!" ./0*%!" +12*%%" '()*%%" +,-*%%" ./0*%%"

Life was good


250K daily users

!"

#!!$!!!"

%$!!!$!!!"

%$#!!$!!!"

&$!!!$!!!"

'()*%!" +,-*%!" ./0*%!" +12*%%" '()*%%" +,-*%%" ./0*%%"

Life was good NO MORE


Welcome to 6 weeks of pain!



Heavy opPmizaPons were necessary




Numerous small fixes regarding DB config




Numerous small fixes regarding DB configMore shards




Numerous small fixes regarding DB configMore shardsEven more shards




Numerous small fixes regarding DB configMore shardsEven more shardsSpliOng the model to get more shards


Early sharding hell: 8+8 DBs

app app app app appapp app

app app app app app app app app app

appapp

lb

MySQL

MySQL

MySQL

MySQL

MySQL

MySQL

MySQL

MySQL

slave slave slave slaveslave slave slave slave


At 500K daily users we were at a dead end

!"

#!!$!!!"

%$!!!$!!!"

%$#!!$!!!"

&$!!!$!!!"

'()*%!" +,-*%!" ./0*%!" +12*%%" '()*%%" +,-*%%" ./0*%%"



Arch




Dev

Arch




Dev

Arch


OpMonday, June 4, 2012

Pick the right tool for the job


Pick the right tool for the job


EvoluPon II: Use Redis as main DB

Oct 2009

Jan 2010


First game: Redis to the mix

Arch



First game: Redis to the mix

Arch

• Ruby on Rails• Amazon EC2• MySQL• Redis


Hybrid approach

60 GB data

50% writes

256 GB data

10% writeshUp://www.flickr.com/photos/erix/245657047/


http://www.flickr.com/photos/erix/245657047/

http://www.flickr.com/photos/erix/245657047/

One team saved the other one

!"

#!!$!!!"

%$!!!$!!!"

%$#!!$!!!"

&$!!!$!!!"

'()*%!" +,-*%!" ./0*%!" +12*%%" '()*%%" +,-*%%" ./0*%%"


One team saved the other one

!"

#!!$!!!"

%$!!!$!!!"

%$#!!$!!!"

&$!!!$!!!"

'()*%!" +,-*%!" ./0*%!" +12*%%" '()*%%" +,-*%%" ./0*%%"


Now more than 2M users / day

!"

#!!$!!!"

%$!!!$!!!"

%$#!!$!!!"

&$!!!$!!!"

'()*%!" +,-*%!" ./0*%!" +12*%%" '()*%%" +,-*%%" ./0*%%"


Now more than 2M users / day

!"

#!!$!!!"

%$!!!$!!!"

%$#!!$!!!"

&$!!!$!!!"

'()*%!" +,-*%!" ./0*%!" +12*%%" '()*%%" +,-*%%" ./0*%%"

AWS outagein Ireland


10 single-‐points-‐of-‐failure -‐ no fun at all!

app app app app app app app app app app app appapp



lb lb

redis redis redis redis redisMySQL

MySQL

MySQL

MySQL

MySQL

slave slave slave slave slaveslave slave slave slave slave


First game: Add Redis to the mix

Arch




Dev

Arch




Dev

Arch


OpMonday, June 4, 2012


The Start: Ruby

The Next Step

Best of Two Worlds

Company Values


Stateless servers and DBs

Server Database



Server Database



Server Database



Server Database



Server Database



Server Database


Find the flaw


Find the flaw

“Stateless applicaPon servers guarantee one thing:


Find the flaw

“Stateless applicaPon servers guarantee one thing:

The data is neverwhere you need it!”

Paolo Negri, Developer @ Wooga


Strong session pamern



User starts playing



User starts playing

Many transformaPons on the same data set



User starts playing


User stops playing



User starts playing


User stops playing


Stateful servers and DBs

Server Database



Server Database



Server Database



Server Database

One Game Session



Server Database

One Game Session


Next game: Don’t use a DB at all

Arch



Arch

• Erlang OTP



Arch

• Erlang OTP• Amazon EC2



Arch

• Erlang OTP• Amazon EC2• Amazon S3


EvoluPon III: Erlang and stateful servers

Oct 2009

Jan 2010

Oct 2010


Stateful servers are not so hard

session



sessionsessionsessionsession



Server




Server


S3



Server


S3



Server


S3



Server


S3



Server


S3



Server


S3



Server


Server


Server


S3


With stateful server the DB is less used

0

7,500

15,000

22,500

30,000

database operations / sec

Ruby Stateless Erlang Stateful


With stateful server the DB is less used

0

7,500

15,000

22,500

30,000

database operations / sec

Ruby Stateless Erlang Stateful

700



Arch




Arch


DevMonday, June 4, 2012


Arch


Dev OpMonday, June 4, 2012


The Start: Ruby

The Next Step: Erlang

Best of Two Worlds

Company Values


Last game: The ideal architecture

Arch



Arch

• Erlang OTP (state)



Arch

• Erlang OTP (state)• Ruby (logic)



Arch

• Erlang OTP (state)• Ruby (logic)• Amazon EC2



Arch

• Erlang OTP (state)• Ruby (logic)• Amazon EC2• Amazon S3



Arch


OpDevMonday, June 4, 2012

EvoluPon IV: The best out of two worlds

Oct 2009

Jan 2010

Oct 2010

Aug 2011


The basic setup looks exactly like before

Server


Server


Server


S3


Bringing 2 worlds together

Server

session

session

...

session

URL + state

response + new state



Server

session

session

...

session

sender URL + state




Server

session

session

...

session

Worker

Worker

Worker

Worker

Worker

sender URL + state




Server

session

session

...

session

receiver

Worker

Worker

Worker

Worker

Worker

sender URL + state




Server

session

session

...

session

receiver

Worker

Worker

Worker

Worker

Worker

sender URL + state




Server

session

session

...

session

receiver

Worker

Worker

Worker

Worker

Worker

sender URL + state



Performance impact for large payloads

1 80

1000

2000

3000

4000

5000

6000

# of workers

requ

ests / s

200 kB payload



1 80

1000

2000

3000

4000

5000

6000

# of workers

requ

ests / s

PureErlang

RubyZMQ

200 kB payload



1 80

1000

2000

3000

4000

5000

6000

# of workers

requ

ests / s

PureErlang

RubyZMQ

400 rps

200 kB payload


NOT CPU bound


NOT CPU bound

25%


NOT CPU bound

25%

Fixed bandwith limit @ 300 MB/s



Last game: The ideal architecture?

Arch




Arch

DevMonday, June 4, 2012



Arch



Arch

• Erlang OTP (state)• Erlang OTP (logic) !!!• Amazon EC2• Amazon S3



Arch




Arch


Dev OpDevMonday, June 4, 2012


Arch


Dev OpDevMonday, June 4, 2012

So nothing new?


So nothing new?

We could not combine Ruby and Erlang, too bad.

At least not for this game.


So nothing new?



Erlang is great for inter process messaging...

User sessions are represented by Erlang processes.


So nothing new?





... and Server Side Events just work.

The server can push updates to the client.


So nothing new?





... and Server Side Events just work.

The server can push updates to the client.

Now, what could you do with this?


The next step in social gaming















The Start: Ruby

The Next Step: Erlang

Best of Two Worlds?

Company Values


Each new game brought us innovaPon

Oct 2009

Jan 2010

Oct 2010

Aug 2011


Small teams

overbig teams

We’ve learned to value


Collaboration

overcompetition



Generalists

overspecialists



Effort reduction

overcost reduction



Innovation

overrisk mitigation



A good value system


Small teams over

CollaboraPon over

Generalists over

Effort reducPon over

InnovaPon over

Big teams

Compe??on

Specialists

Cost reduc?on

Risk mi?ga?on


It works!


It works!

Be fast, be bold!


QuesPons?

Jesper Richter-‐Reichhelm@jrirei

slideshare.net/woogawooga.com/jobs



Games for the Masses - Wie DevOps die Entwicklung von Architektur verändert (Webinale)

Technology

inter process

250k daily

user starts

numerous small

user stops

db congmore

strong session

server side