ARCHITECTURE EVOLUTION AT WOOGA How does your company learn new things? Jesper RichterReichhelm, @jrirei
Apr 14, 2017
ARCHITECTURE EVOLUTIONAT WOOGA
How does your company learn new things?
Jesper Richter-‐Reichhelm, @jrirei
Our games all look the same
Flash client Backend
But the scale is interesFng
14 billion requests / month
But the scale is interesFng
14 billion requests / month
But the scale is interesFng
14 billion requests / month
>100,000 DB operaFons / second
But the scale is interesFng
14 billion requests / month
>100,000 DB operaFons / second
>50,000 DB updates / second
Architecture EvoluFon at Wooga
The Start
The Next Step
Best of Two Worlds
Oct 2009: Monster World based on Ruby
Good code quality
Easy to understandEasy to testEasy to refactor
Oct 2009: Monster World based on Ruby
Good code quality
Easy to understandEasy to testEasy to refactor
Oct 2009: Monster World based on Ruby
Good code quality
Easy to understandEasy to testEasy to refactor
A basic setup using sharding worked fine
app app app
lb
MySQL
MySQL
slave slave
A basic setup using sharding worked fine
app app appapp app
lb
MySQL
MySQL
slave slave
A basic setup using sharding worked fine
app app appapp appapp app app app
lb
MySQL
MySQL
slave slave
250K daily users
!"
#!!$!!!"
%$!!!$!!!"
%$#!!$!!!"
&$!!!$!!!"
'()*%!" +,-*%!" ./0*%!" +12*%%" '()*%%" +,-*%%" ./0*%%"
Life was good
250K daily users
!"
#!!$!!!"
%$!!!$!!!"
%$#!!$!!!"
&$!!!$!!!"
'()*%!" +,-*%!" ./0*%!" +12*%%" '()*%%" +,-*%%" ./0*%%"
Life was good NO MORE
Early sharding hell: 8 master and 8 slaves
app app app app appapp app
app app app app app app app app app
appapp
lb
MySQL
MySQL
slaveslave
Early sharding hell: 8 master and 8 slaves
app app app app appapp app
app app app app app app app app app
appapp
lb
MySQL
MySQL
slave
MySQL
MySQL
slaveslave slave
Early sharding hell: 8 master and 8 slaves
app app app app appapp app
app app app app app app app app app
appapp
lb
MySQL
MySQL
slave
MySQL
MySQL
MySQL
MySQL
slave slaveslave slave
MySQL
MySQL
slaveslave slave
At 500K daily users we were at a dead end
!"
#!!$!!!"
%$!!!$!!!"
%$#!!$!!!"
&$!!!$!!!"
'()*%!" +,-*%!" ./0*%!" +12*%%" '()*%%" +,-*%%" ./0*%%"
OUCH!
http://www.flickr.com/photos/billue_the_bear/
Big and staFc data in MySQL, rest goes to Redis
MySQL
Big and staFc data in MySQL, rest goes to Redis
MySQL Redis
Big and staFc data in MySQL, rest goes to Redis
256 GB data
10% writes
MySQL Redis
Big and staFc data in MySQL, rest goes to Redis
60 GB data
50% writes
256 GB data
10% writes
MySQL Redis
Redis saved the day
!"
#!!$!!!"
%$!!!$!!!"
%$#!!$!!!"
&$!!!$!!!"
'()*%!" +,-*%!" ./0*%!" +12*%%" '()*%%" +,-*%%" ./0*%%"
Redis saved the day
!"
#!!$!!!"
%$!!!$!!!"
%$#!!$!!!"
&$!!!$!!!"
'()*%!" +,-*%!" ./0*%!" +12*%%" '()*%%" +,-*%%" ./0*%%"
We now have more than 2 million users / day
!"
#!!$!!!"
%$!!!$!!!"
%$#!!$!!!"
&$!!!$!!!"
'()*%!" +,-*%!" ./0*%!" +12*%%" '()*%%" +,-*%%" ./0*%%"
We now have more than 2 million users / day
!"
#!!$!!!"
%$!!!$!!!"
%$#!!$!!!"
&$!!!$!!!"
'()*%!" +,-*%!" ./0*%!" +12*%%" '()*%%" +,-*%%" ./0*%%"
AWS servicedisrupFonin Ireland
10 single-‐points-‐of-‐failure -‐ no fun at all!
app app app app app app app app app app app appapp
app app app app app app app app app app app appapp
app app app app app app app app app app app appapp
lb lb
redis redisMySQL
MySQL
slave slaveslave slave
10 single-‐points-‐of-‐failure -‐ no fun at all!
app app app app app app app app app app app appapp
app app app app app app app app app app app appapp
app app app app app app app app app app app appapp
lb lb
redis redisMySQL
MySQL
slave slave
MySQL
slave
MySQL
MySQL
slave slave slave slave
10 single-‐points-‐of-‐failure -‐ no fun at all!
app app app app app app app app app app app appapp
app app app app app app app app app app app appapp
app app app app app app app app app app app appapp
lb lb
redis redisMySQL
MySQL
slave slave
redis redis
slave slave
redis
slave
MySQL
slave
MySQL
MySQL
slave slave slave slave
OUCH!
http://www.flickr.com/photos/wolfsavard/
Architecture EvoluFon at Wooga
The Start: Ruby + AutomaFon
The Next Step
Best of Two Worlds
Stateless servers and DBs
Server Database
Stateless servers and DBs
Server Database
Stateless servers and DBs
Server Database
Stateless servers and DBs
Server Database
Stateless servers and DBs
Server Database
Stateless servers and DBs
Server Database
“Stateless applica,on servers guarantee one thing:
“Stateless applica,on servers guarantee one thing:
The data is never
where you need it!”Paolo Negri, Developer @ Wooga
Stateful servers and DBs
Server Database
Stateful servers and DBs
Server Database
Stateful servers and DBs
Server Database
Stateful servers and DBs
Server Database
One Game Session
Stateful servers and DBs
Server Database
One Game Session
Oct 2010: Magic Land based on stateful server
If DBs are the problem
Don’t use themStore state in serverNeed to be robust
Oct 2010: Magic Land based on stateful server
If DBs are the problem
Don’t use themStore state in serverNeed to be robust
Oct 2010: Magic Land based on stateful server
If DBs are the problem
Don’t use themStore state in serverNeed to be robust
Stateful servers are not as hard as you think
session
Stateful servers are not as hard as you think
sessionsessionsessionsession
Stateful servers are not as hard as you think
Server
sessionsessionsessionsession
Stateful servers are not as hard as you think
Server
sessionsessionsessionsession
S3
Stateful servers are not as hard as you think
Server
sessionsessionsessionsession
S3
Stateful servers are not as hard as you think
Server
sessionsessionsessionsession
S3
Stateful servers are not as hard as you think
Server
sessionsessionsessionsession
S3
Stateful servers are not as hard as you think
Server
sessionsessionsessionsession
S3
Stateful servers are not as hard as you think
Server
sessionsessionsessionsession
S3
Stateful servers are not as hard as you think
Server
sessionsessionsessionsession
Server
sessionsessionsessionsession
Server
sessionsessionsessionsession
S3
With stateful server the DB is less used
0
7,500
15,000
22,500
30,000
database operations / sec
Ruby Stateless Erlang Stateful
With stateful server the DB is less used
0
7,500
15,000
22,500
30,000
database operations / sec
Ruby Stateless Erlang Stateful
700
Deploying with a stateful server
In order to bring up a new version
Deploying with a stateful server
In order to bring up a new version
Just deploy it
Hot code replacement is great!
There are even more advantages
Faster than Ruby (5,000 rps / node)-‐ CPU bound
There are even more advantages
Faster than Ruby (5,000 rps / node)-‐ CPU bound
Very few SPOFs-‐ ... and those are easy to recover
There are even more advantages
Faster than Ruby (5,000 rps / node)-‐ CPU bound
Very few SPOFs-‐ ... and those are easy to recover
TransacFonal logic-‐ Invariants instead of explicit error handling
NICE!
http://www.flickr.com/photos/aigle_dore/
Architecture EvoluFon at Wooga
The Start: Ruby + AutomaFon
The Next Step: Erlang + S3
Company Values
We’ve learned to value
A good value system
Small teams over Big teams
We’ve learned to value
A good value system
Small teams over Big teams
We’ve learned to value
CollaboraFon over Compe??on
A good value system
Small teams over Big teams
We’ve learned to value
Generalists over Specialists
CollaboraFon over Compe??on
A good value system
Small teams over Big teams
We’ve learned to value
Generalists over Specialists
Effort reducFon over Cost reduc?on
CollaboraFon over Compe??on
A good value system
Small teams over Big teams
We’ve learned to value
Generalists over Specialists
InnovaFon over Risk mi?ga?on
Effort reducFon over Cost reduc?on
CollaboraFon over Compe??on
A good value system
It works!
It works!
Be fast, be bold!
QuesFons?
Jesper Richter-‐Reichhelm@jrirei
slideshare.net/woogawooga.com/jobs