Games for the Masses - Wie DevOps die Entwicklung von Architektur verändert (Webinale)
Post on 19-Oct-2014
1875 Views
Preview:
DESCRIPTION
Transcript
GAMES FOR THE MASSESWie DevOps die Entwicklung von Architektur verändert
Jesper Richter-‐Reichhelm, @jrireiMonday, June 4, 2012
DEV -‐ OPS
Monday, June 4, 2012
DEV -‐ OPS
Some say: Admins learn to be ‘Agile’
Scrum and Kanban
Monday, June 4, 2012
DEV -‐ OPS
Some say: Admins learn to be ‘Agile’
Scrum and Kanban
Others say: Admins learn to program their setup
Chef, Puppet
Monday, June 4, 2012
DEV -‐ OPS
Some say: Admins learn to be ‘Agile’
Scrum and Kanban
Others say: Admins learn to program their setup
Chef, Puppet
I say: This is all nice, but not enough
Developers need to learn a lot about opera?on
Monday, June 4, 2012
Classic roles in IT
Op
ArchDev
Monday, June 4, 2012
Classic roles in IT
Op
ArchDev
Monday, June 4, 2012
Classic roles in IT
Op
ArchDev
Monday, June 4, 2012
Classic roles in IT
Op
ArchDev
Monday, June 4, 2012
Classic roles in IT
Op
ArchDev
Monday, June 4, 2012
Classic roles in IT
Op
ArchDev
Monday, June 4, 2012
Classic roles in IT
Op
ArchDev
Monday, June 4, 2012
Classic roles in IT
Op
ArchDev
Monday, June 4, 2012
Classic roles in IT
Op
ArchDev
Monday, June 4, 2012
Classic roles in IT
Op
ArchDev
???
Monday, June 4, 2012
Monday, June 4, 2012
Our games all look the same
BackendFlash client
Monday, June 4, 2012
Our games all look the same
Game Session
Asynch. CommunicaPon
Flash client
Monday, June 4, 2012
Backend
Our games all look the same
State Changes
ValidaPon
Persistence
Monday, June 4, 2012
But the scale is interesPng
14 billion requests / month
Monday, June 4, 2012
But the scale is interesPng
14 billion requests / month
Monday, June 4, 2012
But the scale is interesPng
14 billion requests / month
>100,000 DB operaPons / second
Monday, June 4, 2012
But the scale is interesPng
14 billion requests / month
>100,000 DB operaPons / second
>50,000 DB updates / second
Monday, June 4, 2012
Classic roles in IT
Op
ArchDev
???
Monday, June 4, 2012
Roles in Wooga’s IT
Monday, June 4, 2012
Arch
Roles in Wooga’s IT
Monday, June 4, 2012
ArchArchDev
Roles in Wooga’s IT
Monday, June 4, 2012
ArchArchDevArchDevOp
Roles in Wooga’s IT
Monday, June 4, 2012
ArchArchDevArchDevOp
Roles in Wooga’s IT
ArchArchDevArchDevOp
Monday, June 4, 2012
Wooga has dedicated game teams
Monday, June 4, 2012
Architecture EvoluPon at Wooga
The Start
The Next Step
Best of Two Worlds
Company Values
Monday, June 4, 2012
First game: Try to be flexible
Arch
Monday, June 4, 2012
First game: Try to be flexible
Arch
• Ruby on Rails
Monday, June 4, 2012
First game: Try to be flexible
Arch
• Ruby on Rails• Amazon EC2
Monday, June 4, 2012
First game: Try to be flexible
Arch
• Ruby on Rails• Amazon EC2• MySQL
Monday, June 4, 2012
EvoluPon I: Use Ruby (on Rails)
Oct 2009
Monday, June 4, 2012
Basic setup using sharding
app app app app app app app app app
lb
MySQL
MySQL
slave slave
Monday, June 4, 2012
250K daily users
!"
#!!$!!!"
%$!!!$!!!"
%$#!!$!!!"
&$!!!$!!!"
'()*%!" +,-*%!" ./0*%!" +12*%%" '()*%%" +,-*%%" ./0*%%"
Life was good
Monday, June 4, 2012
250K daily users
!"
#!!$!!!"
%$!!!$!!!"
%$#!!$!!!"
&$!!!$!!!"
'()*%!" +,-*%!" ./0*%!" +12*%%" '()*%%" +,-*%%" ./0*%%"
Life was good NO MORE
Monday, June 4, 2012
Welcome to 6 weeks of pain!
Monday, June 4, 2012
Welcome to 6 weeks of pain!
Heavy opPmizaPons were necessary
Monday, June 4, 2012
Welcome to 6 weeks of pain!
Heavy opPmizaPons were necessary
Numerous small fixes regarding DB config
Monday, June 4, 2012
Welcome to 6 weeks of pain!
Heavy opPmizaPons were necessary
Numerous small fixes regarding DB configMore shards
Monday, June 4, 2012
Welcome to 6 weeks of pain!
Heavy opPmizaPons were necessary
Numerous small fixes regarding DB configMore shardsEven more shards
Monday, June 4, 2012
Welcome to 6 weeks of pain!
Heavy opPmizaPons were necessary
Numerous small fixes regarding DB configMore shardsEven more shardsSpliOng the model to get more shards
Monday, June 4, 2012
Early sharding hell: 8+8 DBs
app app app app appapp app
app app app app app app app app app
appapp
lb
MySQL
MySQL
MySQL
MySQL
MySQL
MySQL
MySQL
MySQL
slave slave slave slaveslave slave slave slave
Monday, June 4, 2012
At 500K daily users we were at a dead end
!"
#!!$!!!"
%$!!!$!!!"
%$#!!$!!!"
&$!!!$!!!"
'()*%!" +,-*%!" ./0*%!" +12*%%" '()*%%" +,-*%%" ./0*%%"
Monday, June 4, 2012
First game: Try to be flexible
Arch
• Ruby on Rails• Amazon EC2• MySQL
Monday, June 4, 2012
First game: Try to be flexible
Dev
Arch
• Ruby on Rails• Amazon EC2• MySQL
Monday, June 4, 2012
First game: Try to be flexible
Dev
Arch
• Ruby on Rails• Amazon EC2• MySQL
OpMonday, June 4, 2012
Pick the right tool for the job
Monday, June 4, 2012
Pick the right tool for the job
Monday, June 4, 2012
EvoluPon II: Use Redis as main DB
Oct 2009
Jan 2010
Monday, June 4, 2012
First game: Redis to the mix
Arch
• Ruby on Rails• Amazon EC2• MySQL
Monday, June 4, 2012
First game: Redis to the mix
Arch
• Ruby on Rails• Amazon EC2• MySQL• Redis
Monday, June 4, 2012
Hybrid approach
60 GB data
50% writes
256 GB data
10% writeshUp://www.flickr.com/photos/erix/245657047/
Monday, June 4, 2012
One team saved the other one
!"
#!!$!!!"
%$!!!$!!!"
%$#!!$!!!"
&$!!!$!!!"
'()*%!" +,-*%!" ./0*%!" +12*%%" '()*%%" +,-*%%" ./0*%%"
Monday, June 4, 2012
One team saved the other one
!"
#!!$!!!"
%$!!!$!!!"
%$#!!$!!!"
&$!!!$!!!"
'()*%!" +,-*%!" ./0*%!" +12*%%" '()*%%" +,-*%%" ./0*%%"
Monday, June 4, 2012
Now more than 2M users / day
!"
#!!$!!!"
%$!!!$!!!"
%$#!!$!!!"
&$!!!$!!!"
'()*%!" +,-*%!" ./0*%!" +12*%%" '()*%%" +,-*%%" ./0*%%"
Monday, June 4, 2012
Now more than 2M users / day
!"
#!!$!!!"
%$!!!$!!!"
%$#!!$!!!"
&$!!!$!!!"
'()*%!" +,-*%!" ./0*%!" +12*%%" '()*%%" +,-*%%" ./0*%%"
AWS outagein Ireland
Monday, June 4, 2012
10 single-‐points-‐of-‐failure -‐ no fun at all!
app app app app app app app app app app app appapp
app app app app app app app app app app app appapp
app app app app app app app app app app app appapp
lb lb
redis redis redis redis redisMySQL
MySQL
MySQL
MySQL
MySQL
slave slave slave slave slaveslave slave slave slave slave
Monday, June 4, 2012
First game: Add Redis to the mix
Arch
• Ruby on Rails• Amazon EC2• MySQL• Redis
Monday, June 4, 2012
First game: Add Redis to the mix
Dev
Arch
• Ruby on Rails• Amazon EC2• MySQL• Redis
Monday, June 4, 2012
First game: Add Redis to the mix
Dev
Arch
• Ruby on Rails• Amazon EC2• MySQL• Redis
OpMonday, June 4, 2012
Architecture EvoluPon at Wooga
The Start: Ruby
The Next Step
Best of Two Worlds
Company Values
Monday, June 4, 2012
Stateless servers and DBs
Server Database
Monday, June 4, 2012
Stateless servers and DBs
Server Database
Monday, June 4, 2012
Stateless servers and DBs
Server Database
Monday, June 4, 2012
Stateless servers and DBs
Server Database
Monday, June 4, 2012
Stateless servers and DBs
Server Database
Monday, June 4, 2012
Stateless servers and DBs
Server Database
Monday, June 4, 2012
Find the flaw
Monday, June 4, 2012
Find the flaw
“Stateless applicaPon servers guarantee one thing:
Monday, June 4, 2012
Find the flaw
“Stateless applicaPon servers guarantee one thing:
The data is neverwhere you need it!”
Paolo Negri, Developer @ Wooga
Monday, June 4, 2012
Strong session pamern
Monday, June 4, 2012
Strong session pamern
User starts playing
Monday, June 4, 2012
Strong session pamern
User starts playing
Many transformaPons on the same data set
Monday, June 4, 2012
Strong session pamern
User starts playing
Many transformaPons on the same data set
User stops playing
Monday, June 4, 2012
Strong session pamern
User starts playing
Many transformaPons on the same data set
User stops playing
Monday, June 4, 2012
Stateful servers and DBs
Server Database
Monday, June 4, 2012
Stateful servers and DBs
Server Database
Monday, June 4, 2012
Stateful servers and DBs
Server Database
Monday, June 4, 2012
Stateful servers and DBs
Server Database
One Game Session
Monday, June 4, 2012
Stateful servers and DBs
Server Database
One Game Session
Monday, June 4, 2012
Next game: Don’t use a DB at all
Arch
Monday, June 4, 2012
Next game: Don’t use a DB at all
Arch
• Erlang OTP
Monday, June 4, 2012
Next game: Don’t use a DB at all
Arch
• Erlang OTP• Amazon EC2
Monday, June 4, 2012
Next game: Don’t use a DB at all
Arch
• Erlang OTP• Amazon EC2• Amazon S3
Monday, June 4, 2012
EvoluPon III: Erlang and stateful servers
Oct 2009
Jan 2010
Oct 2010
Monday, June 4, 2012
Stateful servers are not so hard
session
Monday, June 4, 2012
Stateful servers are not so hard
sessionsessionsessionsession
Monday, June 4, 2012
Stateful servers are not so hard
Server
sessionsessionsessionsession
Monday, June 4, 2012
Stateful servers are not so hard
Server
sessionsessionsessionsession
S3
Monday, June 4, 2012
Stateful servers are not so hard
Server
sessionsessionsessionsession
S3
Monday, June 4, 2012
Stateful servers are not so hard
Server
sessionsessionsessionsession
S3
Monday, June 4, 2012
Stateful servers are not so hard
Server
sessionsessionsessionsession
S3
Monday, June 4, 2012
Stateful servers are not so hard
Server
sessionsessionsessionsession
S3
Monday, June 4, 2012
Stateful servers are not so hard
Server
sessionsessionsessionsession
S3
Monday, June 4, 2012
Stateful servers are not so hard
Server
sessionsessionsessionsession
Server
sessionsessionsessionsession
Server
sessionsessionsessionsession
S3
Monday, June 4, 2012
With stateful server the DB is less used
0
7,500
15,000
22,500
30,000
database operations / sec
Ruby Stateless Erlang Stateful
Monday, June 4, 2012
With stateful server the DB is less used
0
7,500
15,000
22,500
30,000
database operations / sec
Ruby Stateless Erlang Stateful
700
Monday, June 4, 2012
Next game: Don’t use a DB at all
Arch
• Erlang OTP• Amazon EC2• Amazon S3
Monday, June 4, 2012
Next game: Don’t use a DB at all
Arch
• Erlang OTP• Amazon EC2• Amazon S3
DevMonday, June 4, 2012
Next game: Don’t use a DB at all
Arch
• Erlang OTP• Amazon EC2• Amazon S3
Dev OpMonday, June 4, 2012
Architecture EvoluPon at Wooga
The Start: Ruby
The Next Step: Erlang
Best of Two Worlds
Company Values
Monday, June 4, 2012
Last game: The ideal architecture
Arch
Monday, June 4, 2012
Last game: The ideal architecture
Arch
• Erlang OTP (state)
Monday, June 4, 2012
Last game: The ideal architecture
Arch
• Erlang OTP (state)• Ruby (logic)
Monday, June 4, 2012
Last game: The ideal architecture
Arch
• Erlang OTP (state)• Ruby (logic)• Amazon EC2
Monday, June 4, 2012
Last game: The ideal architecture
Arch
• Erlang OTP (state)• Ruby (logic)• Amazon EC2• Amazon S3
Monday, June 4, 2012
Last game: The ideal architecture
Arch
• Erlang OTP (state)• Ruby (logic)• Amazon EC2• Amazon S3
OpDevMonday, June 4, 2012
EvoluPon IV: The best out of two worlds
Oct 2009
Jan 2010
Oct 2010
Aug 2011
Monday, June 4, 2012
The basic setup looks exactly like before
Server
sessionsessionsessionsession
Server
sessionsessionsessionsession
Server
sessionsessionsessionsession
S3
Monday, June 4, 2012
Bringing 2 worlds together
Server
session
session
...
session
URL + state
response + new state
Monday, June 4, 2012
Bringing 2 worlds together
Server
session
session
...
session
sender URL + state
response + new state
Monday, June 4, 2012
Bringing 2 worlds together
Server
session
session
...
session
Worker
Worker
Worker
Worker
Worker
sender URL + state
response + new state
Monday, June 4, 2012
Bringing 2 worlds together
Server
session
session
...
session
receiver
Worker
Worker
Worker
Worker
Worker
sender URL + state
response + new state
Monday, June 4, 2012
Bringing 2 worlds together
Server
session
session
...
session
receiver
Worker
Worker
Worker
Worker
Worker
sender URL + state
response + new state
Monday, June 4, 2012
Bringing 2 worlds together
Server
session
session
...
session
receiver
Worker
Worker
Worker
Worker
Worker
sender URL + state
response + new state
Monday, June 4, 2012
Performance impact for large payloads
1 80
1000
2000
3000
4000
5000
6000
# of workers
requ
ests / s
200 kB payload
Monday, June 4, 2012
Performance impact for large payloads
1 80
1000
2000
3000
4000
5000
6000
# of workers
requ
ests / s
PureErlang
RubyZMQ
200 kB payload
Monday, June 4, 2012
Performance impact for large payloads
1 80
1000
2000
3000
4000
5000
6000
# of workers
requ
ests / s
PureErlang
RubyZMQ
400 rps
200 kB payload
Monday, June 4, 2012
NOT CPU bound
Monday, June 4, 2012
NOT CPU bound
25%
Monday, June 4, 2012
NOT CPU bound
25%
Fixed bandwith limit @ 300 MB/s
Monday, June 4, 2012
• Erlang OTP (state)• Ruby (logic)• Amazon EC2• Amazon S3
Last game: The ideal architecture?
Arch
Monday, June 4, 2012
• Erlang OTP (state)• Ruby (logic)• Amazon EC2• Amazon S3
Last game: The ideal architecture?
Arch
DevMonday, June 4, 2012
• Erlang OTP (state)• Ruby (logic)• Amazon EC2• Amazon S3
Last game: The ideal architecture?
Arch
Dev OpMonday, June 4, 2012
Last game: The ideal architecture?
Arch
• Erlang OTP (state)• Erlang OTP (logic) !!!• Amazon EC2• Amazon S3
Dev OpMonday, June 4, 2012
Last game: The ideal architecture?
Arch
• Erlang OTP (state)• Erlang OTP (logic) !!!• Amazon EC2• Amazon S3
Dev OpMonday, June 4, 2012
Last game: The ideal architecture?
Arch
• Erlang OTP (state)• Erlang OTP (logic) !!!• Amazon EC2• Amazon S3
Dev OpDevMonday, June 4, 2012
Last game: The ideal architecture?
Arch
• Erlang OTP (state)• Erlang OTP (logic) !!!• Amazon EC2• Amazon S3
Dev OpDevMonday, June 4, 2012
So nothing new?
Monday, June 4, 2012
So nothing new?
We could not combine Ruby and Erlang, too bad.
At least not for this game.
Monday, June 4, 2012
So nothing new?
We could not combine Ruby and Erlang, too bad.
At least not for this game.
Erlang is great for inter process messaging...
User sessions are represented by Erlang processes.
Monday, June 4, 2012
So nothing new?
We could not combine Ruby and Erlang, too bad.
At least not for this game.
Erlang is great for inter process messaging...
User sessions are represented by Erlang processes.
... and Server Side Events just work.
The server can push updates to the client.
Monday, June 4, 2012
So nothing new?
We could not combine Ruby and Erlang, too bad.
At least not for this game.
Erlang is great for inter process messaging...
User sessions are represented by Erlang processes.
... and Server Side Events just work.
The server can push updates to the client.
Now, what could you do with this?
Monday, June 4, 2012
The next step in social gaming
Monday, June 4, 2012
The next step in social gaming
Monday, June 4, 2012
The next step in social gaming
Monday, June 4, 2012
The next step in social gaming
Monday, June 4, 2012
The next step in social gaming
Monday, June 4, 2012
The next step in social gaming
Monday, June 4, 2012
The next step in social gaming
Monday, June 4, 2012
Architecture EvoluPon at Wooga
The Start: Ruby
The Next Step: Erlang
Best of Two Worlds?
Company Values
Monday, June 4, 2012
Each new game brought us innovaPon
Oct 2009
Jan 2010
Oct 2010
Aug 2011
Monday, June 4, 2012
Small teams
overbig teams
We’ve learned to value
Monday, June 4, 2012
Collaboration
overcompetition
We’ve learned to value
Monday, June 4, 2012
Generalists
overspecialists
We’ve learned to value
Monday, June 4, 2012
Effort reduction
overcost reduction
We’ve learned to value
Monday, June 4, 2012
Innovation
overrisk mitigation
We’ve learned to value
Monday, June 4, 2012
A good value system
We’ve learned to value
Small teams over
CollaboraPon over
Generalists over
Effort reducPon over
InnovaPon over
Big teams
Compe??on
Specialists
Cost reduc?on
Risk mi?ga?on
Monday, June 4, 2012
It works!
Monday, June 4, 2012
It works!
Be fast, be bold!
Monday, June 4, 2012
QuesPons?
Jesper Richter-‐Reichhelm@jrirei
slideshare.net/woogawooga.com/jobs
Monday, June 4, 2012
Monday, June 4, 2012
top related