AWS Summit Berlin 2013 - "Angrybirds fly in the cloud" - Scaling and Market needs for the Angrybirds Shop

Freitag, 3. Mai 13

Angrybirds fly in the cloud

Freitag, 3. Mai 13

Angrybirds fly in the cloud

Presented byDaniel Pötzinger

AWS Summit 2013

Freitag, 3. Mai 13

Freitag, 3. Mai 13

danpoetz

Freitag, 3. Mai 13

Freitag, 3. Mai 13

Freitag, 3. Mai 13

Freitag, 3. Mai 13

152,2 Million €

68,5 Million € Consumer Products

Freitag, 3. Mai 13

Freitag, 3. Mai 13

Freitag, 3. Mai 13

PC games

Plush toys, shirts, ...

Freitag, 3. Mai 13

> 24.000.000 likes > 500.000 followers

Social MediaImpact

Freitag, 3. Mai 13

Irregular Traffic Behaviour

Freitag, 3. Mai 13

Challenges

Freitag, 3. Mai 13

Challenges

Frequent development and releasing of new

features

Freitag, 3. Mai 13

Challenges

Frequent development and releasing of new

features

Scalable and stable Operation

Freitag, 3. Mai 13

Freitag, 3. Mai 13

The solution?

Freitag, 3. Mai 13

Freitag, 3. Mai 13

Continuous Deployment

CloudDevOps

Freitag, 3. Mai 13

Continuous Deployment

CloudDevOps

Freitag, 3. Mai 13

Continuous Deployment

CloudDevOps

Freitag, 3. Mai 13

Why Amazon Web Services ?

Freitag, 3. Mai 13

✓Scales quickly on demand✓Pay only what for what

you use

Freitag, 3. Mai 13

8 regions with 2-4 availability zonesA lot of „edge servers“ for content delivery

Freitag, 3. Mai 13

Think cleanly about your operation infrastructureFreitag, 3. Mai 13

Setting up a new environment should always be faster than fixing an existing one

Freitag, 3. Mai 13

What you should do

Freitag, 3. Mai 13

What you should do

Automate the setup of new servers

Freitag, 3. Mai 13

What you should do

Automate the setup of new servers

Automate failover

Freitag, 3. Mai 13

What you should do

Automate the setup of new servers

Automate failover

Automate scaling

Freitag, 3. Mai 13

What you should do

Automate the setup of new servers

Automate failover

Automate scaling

Don‘t do work that others did better

Freitag, 3. Mai 13

Powerfull AWS Services for Angrybirds

Freitag, 3. Mai 13

Powerfull AWS Services for Angrybirds

EC2

Freitag, 3. Mai 13

Powerfull AWS Services for Angrybirds

S3EC2

Freitag, 3. Mai 13

Powerfull AWS Services for Angrybirds

S3EC2 RDS

Freitag, 3. Mai 13

Powerfull AWS Services for Angrybirds

CloudFrontS3EC2 RDS

Freitag, 3. Mai 13

Powerfull AWS Services for Angrybirds

CloudFront ElastiCacheS3EC2 RDS

Freitag, 3. Mai 13

Powerfull AWS Services for Angrybirds

CloudFront ElastiCacheS3EC2 RDS ELB

Freitag, 3. Mai 13

Powerfull AWS Services for Angrybirds

CloudFront ElastiCacheS3EC2 RDS ELB R53

Freitag, 3. Mai 13

Powerfull AWS Services for Angrybirds

Amazon Web Services & Amazon Console

CloudFront ElastiCacheS3EC2 RDS ELB R53

Freitag, 3. Mai 13

Powerfull AWS Services for Angrybirds

Rightscale Services

Amazon Web Services & Amazon Console

CloudFront ElastiCacheS3EC2 RDS ELB R53

Freitag, 3. Mai 13

Freitag, 3. Mai 13

Freitag, 3. Mai 13

Cloud Architecture

Freitag, 3. Mai 13

Freitag, 3. Mai 13

Varnish Array

Freitag, 3. Mai 13

Varnish ArrayEC2 EC2

Freitag, 3. Mai 13

Varnish Array

Frontend ArrayEC2 EC2 EC2

EC2 EC2

Freitag, 3. Mai 13

Varnish Array

Frontend Array Backend ArrayEC2 EC2EC2 EC2 EC2

EC2 EC2

Freitag, 3. Mai 13

Varnish Array

Frontend Array Backend Array

RDS

EC2 EC2EC2 EC2 EC2

EC2 EC2

Freitag, 3. Mai 13

RDS

Varnish Array

Frontend Array Backend Array

RDS

EC2 EC2EC2 EC2 EC2

EC2 EC2

Freitag, 3. Mai 13

RDS

Varnish Array

Frontend Array Backend Array

RDS

S3Assets Storage

EC2 EC2EC2 EC2 EC2

EC2 EC2

Freitag, 3. Mai 13

RDS

Varnish Array

Frontend Array Backend Array

RDS

CloudFrontCDN for Assets

S3Assets Storage

EC2 EC2EC2 EC2 EC2

EC2 EC2

Freitag, 3. Mai 13

RDS

Varnish Array

Frontend Array Backend Array Worker Array

RDS

CloudFrontCDN for Assets

S3Assets Storage

EC2 EC2 EC2EC2 EC2 EC2

EC2 EC2

Freitag, 3. Mai 13

RDS

Varnish Array

Frontend Array Backend Array Worker Array

ELB

RDS

CloudFrontCDN for Assets

S3Assets Storage

EC2 EC2 EC2EC2 EC2 EC2

EC2 EC2

Freitag, 3. Mai 13

RDS

„Deployment Definition“ + Autoscaling

Varnish Array

Frontend Array Backend Array Worker Array

ELB

RDS

CloudFrontCDN for Assets

S3Assets Storage

EC2 EC2 EC2EC2 EC2 EC2

EC2 EC2

Freitag, 3. Mai 13

RDS

„Deployment Definition“ + Autoscaling

Varnish Array

Frontend Array Backend Array Worker Array

ELB

RDS

CloudFrontCDN for Assets

S3Assets Storage

EC2 EC2 EC2EC2 EC2 EC2

EC2 EC2Monitoring

Freitag, 3. Mai 13

RDS

„Deployment Definition“ + Autoscaling

Varnish Array

Frontend Array Backend Array Worker Array

ELB

RDS

CloudFrontCDN for Assets

S3Assets Storage

EC2 EC2 EC2EC2 EC2 EC2

EC2 EC2MonitoringAPI

Freitag, 3. Mai 13

RDS

„Deployment Definition“ + Autoscaling

Varnish Array

Frontend Array Backend Array Worker Array

CloudFrontCDN for Skin

ELB

RDS

CloudFrontCDN for Assets

S3Assets Storage

EC2 EC2 EC2EC2 EC2 EC2

EC2 EC2MonitoringAPI

Freitag, 3. Mai 13

RDS

„Deployment Definition“ + Autoscaling

Varnish Array

Frontend Array Backend Array Worker Array

CloudFrontCDN for Skin

ELBR53DNS

RDS

CloudFrontCDN for Assets

S3Assets Storage

EC2 EC2 EC2EC2 EC2 EC2

EC2 EC2MonitoringAPI

Freitag, 3. Mai 13

RDS

„Deployment Definition“ + Autoscaling

Varnish Array

Frontend Array Backend Array Worker Array

CloudFrontCDN for Skin

ELBR53DNS

RDSElastiCache

Cache Backend

CloudFrontCDN for Assets

S3Assets Storage

EC2 EC2 EC2EC2 EC2 EC2

EC2 EC2MonitoringAPI

Freitag, 3. Mai 13

RDS

„Deployment Definition“ + Autoscaling

Varnish Array

Frontend Array Backend Array Worker Array

CloudFrontCDN for Skin

ELBR53DNS

RDSS3

DeploymentElastiCache

Cache Backend

CloudFrontCDN for Assets

S3Assets Storage

EC2 EC2 EC2EC2 EC2 EC2

EC2 EC2MonitoringAPI

Freitag, 3. Mai 13

Scaling is about finding the bottleneck

Freitag, 3. Mai 13

CPU

Network

Storage (DB)Traffic

Possible BottlenecksOffloading, Optimisation, Caching, add more CPU (scale)...

Offloading, Optimisation, Scale Storage UP (H/V), Replace Storage...

Offloading (CDN, Proxy)Optimize Size of the SiteScale Network IS...

Freitag, 3. Mai 13

HTTP Caching

Traffic

Freitag, 3. Mai 13

HTTP Caching

Appl

icat

ion

Traffic

Freitag, 3. Mai 13

HTTP Caching

Appl

icat

ion

Offload expensive requests

Traffic

Freitag, 3. Mai 13

HTTP Caching

Appl

icat

ion

Offload expensive requests

Traffic

Brow

ser

Traffic

Freitag, 3. Mai 13

HTTP Caching

Appl

icat

ion

Offload expensive requests

Traffic

CDN

Brow

ser

Traffic

Freitag, 3. Mai 13

HTTP Caching

Appl

icat

ion

Varnish (Reverse Proxy) Traffic

Offload expensive requests

Traffic

CDN

Brow

ser

Traffic

Freitag, 3. Mai 13

HTTP Caching

Appl

icat

ion

Varnish (Reverse Proxy) Traffic

Offload expensive requests

Traffic

CDN

Brow

ser

TrafficPurging

Freitag, 3. Mai 13

10.000+Possible Pages / Seconde

from ELB & Varnish

Freitag, 3. Mai 13

Autoscaling

Traffic

Freitag, 3. Mai 13

Autoscaling

Traffic

Fron

tend

Arr

ay

EC2

EC2

EC2

Freitag, 3. Mai 13

Autoscaling

Traffic

Fron

tend

Arr

ay

EC2

EC2

EC2

Freitag, 3. Mai 13

Autoscaling

Traffic

Fron

tend

Arr

ay

EC2

EC2

EC2

EC2

EC2

EC2

Freitag, 3. Mai 13

Autoscaling

•Groups with policys•Min, Max•Votingrules for scaling up & down•Replace broken instances

Traffic

Fron

tend

Arr

ay

EC2

EC2

EC2

EC2

EC2

EC2

Freitag, 3. Mai 13

10.000possible Orders / Hour

Freitag, 3. Mai 13

Be prepared for changes

Freitag, 3. Mai 13

Freitag, 3. Mai 13

Freitag, 3. Mai 13

Continuous Deployment

Freitag, 3. Mai 13

Freitag, 3. Mai 13

Freitag, 3. Mai 13

Don‘t be scared of releasing to production

Freitag, 3. Mai 13

Late night bugfixing?Freitag, 3. Mai 13

That’s how a deployment should look like

Freitag, 3. Mai 13

And you?

Freitag, 3. Mai 13

And you?

• Do you use continuous integration?

Freitag, 3. Mai 13

And you?

• Do you use continuous integration?• Do you use automated acceptance tests?

Freitag, 3. Mai 13

And you?

• Do you use continuous integration?• Do you use automated acceptance tests?• Do you keep them green?

Freitag, 3. Mai 13

And you?

• Do you use continuous integration?• Do you use automated acceptance tests?• Do you keep them green?• Do you use continuous deployment?

Freitag, 3. Mai 13

Continuous Deployment

Freitag, 3. Mai 13

Continuous Deployment

We want on-demand deployment

Freitag, 3. Mai 13

Continuous Deployment

We want on-demand deploymentof tested features and fixes

Freitag, 3. Mai 13

Continuous Deployment

We want on-demand deploymentof tested features and fixesin a quick and reliable way

Freitag, 3. Mai 13

We reach this by

Freitag, 3. Mai 13

Automating everything

We reach this by

Freitag, 3. Mai 13

Automating everything

We reach this by

Build

Freitag, 3. Mai 13

Automating everything

We reach this by

BuildTesting

Freitag, 3. Mai 13

Automating everything

We reach this by

BuildTesting

Deployment

Freitag, 3. Mai 13

..and

Freitag, 3. Mai 13

doing it frequently

..and

Freitag, 3. Mai 13

doing it frequently

..and

flickr deploys 3 times a day...

Freitag, 3. Mai 13

Freitag, 3. Mai 13

Freitag, 3. Mai 13

The Deployment Pipeline

Freitag, 3. Mai 13

Freitag, 3. Mai 13

Freitag, 3. Mai 13

Commitand Build

Build Downstream and Test Strategy

SVN Backup Storage

Angry Birds Deployment Pipeline

Feedback to Developers

Freitag, 3. Mai 13

Commitand Build

Build Downstream and Test Strategy

SVN Backup Storage

Unit Tests

Angry Birds Deployment Pipeline

Feedback to Developers

Freitag, 3. Mai 13

Commitand Build

Build Downstream and Test Strategy

SVN Backup Storage

Unit Tests

Angry Birds Deployment Pipeline

Feedback to Developers

Freitag, 3. Mai 13

Commitand Build

Build Downstream and Test Strategy

SVN Backup Storage

Unit Tests Install on “latest”

Angry Birds Deployment Pipeline

Feedback to Developers

Freitag, 3. Mai 13

Commitand Build

Build Downstream and Test Strategy

SVN Backup Storage

Unit Tests Install on “latest”

Angry Birds Deployment Pipeline

Feedback to Developers

Freitag, 3. Mai 13

Selenium Acceptance Tests

Commitand Build

Build Downstream and Test Strategy

SVN Backup Storage

Unit Tests Install on “latest”

Angry Birds Deployment Pipeline

Feedback to Developers

Freitag, 3. Mai 13

Selenium Acceptance Tests

Commitand Build

Build Downstream and Test Strategy

SVN Backup Storage

Unit Tests Install on “latest”

Angry Birds Deployment Pipeline

Feedback to Developers

Freitag, 3. Mai 13

Selenium Acceptance Tests

Commitand Build

Build Downstream and Test Strategy

SVN Backup Storage

Unit Tests Install on “latest”

Performance Tests

Angry Birds Deployment Pipeline

Feedback to Developers

Freitag, 3. Mai 13

Selenium Acceptance Tests

Commitand Build

Build Downstream and Test Strategy

SVN Backup Storage

Unit Tests Install on “latest”

Performance Tests

Angry Birds Deployment Pipeline

Feedback to Developers

Freitag, 3. Mai 13

Angry Birds Deployment Pipeline

Install on Deployment

SystemCopy to S3

Cloud Deployment

Staging

Integration Tests

AOE + Rovio

Cloud deployment production

Freitag, 3. Mai 13

Varnish Array

AutoscalingFrontend Array

Backend Array

Worker Array

Cloud Deployment

ELB

Deployment „Version 88“

Start new release by✓ cloning old deployment✓ adjusting release number

RDSmySQL S3

V V V

WFE FE FE BE BE BE

R53DNS

Freitag, 3. Mai 13

Varnish Array

AutoscalingFrontend Array

Backend Array

Worker Array

Cloud Deployment

ELB

Deployment „Version 88“

RDSmySQL S3

V V V

WFE FE FE BE BE BE

R53DNS

Freitag, 3. Mai 13

Deployment „Version 89“

Varnish Array

AutoscalingFrontend Array

Backend Array

Worker Array

Cloud Deployment

ELB

Deployment „Version 88“

RDSmySQL S3

Varnish Array

AutoscalingFrontend Array

Backend Array

Worker Array

V V V

WFE FE FE BE BE BE

R53DNS

Freitag, 3. Mai 13

Deployment „Version 89“

Varnish Array

AutoscalingFrontend Array

Backend Array

Worker Array

Cloud Deployment

ELB

Deployment „Version 88“

RDSmySQL S3

Varnish Array

AutoscalingFrontend Array

Backend Array

Worker Array

ELB

W

V V V

WFE FE FE BE BE BE

R53DNS

Freitag, 3. Mai 13

Deployment „Version 89“

Varnish Array

AutoscalingFrontend Array

Backend Array

Worker Array

Cloud Deployment

ELB

Deployment „Version 88“

RDSmySQL S3

Varnish Array

AutoscalingFrontend Array

Backend Array

Worker Array

ELB

WFE FE FE BE BE BE

V V VV V V

WFE FE FE BE BE BE

R53DNS

Freitag, 3. Mai 13

Deployment „Version 89“

Varnish Array

AutoscalingFrontend Array

Backend Array

Worker Array

Cloud Deployment

ELB

Deployment „Version 88“

RDSmySQL S3

Varnish Array

AutoscalingFrontend Array

Backend Array

Worker Array

ELB

WFE FE FE BE BE BE

V V VV V V

WFE FE FE BE BE BE

R53DNS

Freitag, 3. Mai 13

0Seconds Downtime

Freitag, 3. Mai 13

Freitag, 3. Mai 13

Freitag, 3. Mai 13

The real life

Freitag, 3. Mai 13

Freitag, 3. Mai 13

Freitag, 3. Mai 13

Day 1 - Release 1

Freitag, 3. Mai 13

Day 2 - Release 2 - “Get ready for Space”

Freitag, 3. Mai 13

Day 3 - „People want it so badly“

Freitag, 3. Mai 13

New release deployed

Day 3 - „People want it so badly“

Freitag, 3. Mai 13

New release deployed

Space game launchedArray scales up

Day 3 - „People want it so badly“

Freitag, 3. Mai 13

New release deployed

Space game launchedArray scales up

USA wakes up

Day 3 - „People want it so badly“

Freitag, 3. Mai 13

Day 4 - Release 3 „Performance Improvement“

Freitag, 3. Mai 13

40% average CPU usage.

1-2 FE Servers are enough for normal operations

Day 4 - Release 3 „Performance Improvement“

Freitag, 3. Mai 13

Freitag, 3. Mai 13

Freitag, 3. Mai 13

Thanks!Follow us on Twitter:

AOE: @aoemediaDaniel P: @danpoetz

Flickr photo contributions in the slide:http://www.flickr.com/photos/the-wanderers-eye/4494147652/sizes/l/http://www.flickr.com/photos/51293088@N05/8195327886/sizes/l/http://www.flickr.com/photos/k1rsch/3818553037/sizes/l/

Freitag, 3. Mai 13