Deploying the 'League of Legends' Data Pipeline with Chef (ARC205) | AWS re:Invent 2013

DEPLOYING THE LEAGUE OF LEGENDS DATA PIPELINE WITH CHEF

ENGINEERING MANAGER| NOVEMBER 2013 TROTTER CASHION

2009 LAUNCH

TEAM ORIENTED ONLINE PC

GAME

100+ CHAMPS

LIVE PLAYERS VS. LIVE PLAYERS

Image source: herocomplex.la2mes.com/

WE WORK WITH BIG DATA

SEMI-GLOBAL DEPLOYMENT 0 DOWNTIME

RUNS IN CLOUD (AWS) + DATACENTER

70+ BILLION TESTED @

EVENTS PER DAY

7+ BILLION EVENTS PER DAY 10+ TABLES

100MM TO 1B ROWS/DAY

100+ TABLES

7+ PETABYTE GAME EVENT DATASET

AGENDA CHEF SERVER

CORE CONCEPTS

CHEF SOLO

OVERVIEW

GOLDEN IMAGES

BASED ON REAL WORLD USAGE Big Data Team from April 2012 to June 2013

Also incorporates subsequent work with Player Accounts team

OVERVIEW

OVERVIEW

WHAT DO WE WANT FROM AUTOMATED INFRASTRUCTURE?

OVERVIEW

WHAT DO WE WANT TO BE ABLE TO DO FOR PLAYERS?

TO PLAY LOL

‣  Minimize Unplanned Down2me

‣  Rolling Deploys ‣  Expand Capacity

NEW FEATURES

‣  Low Maintenance ‣  Low Barrier for New Services

WHAT PLAYERS WANT OVERVIEW

OVERVIEW

CORE: ALWAYS UP, EASY TO DEPLOY

AGENDA CHEF SERVER

CORE CONCEPTS

CHEF SOLO

OVERVIEW

GOLDEN IMAGES

UNDERLYING COMPONENTS CORE CONCEPTS


ELB

EC2


ELB

EC2


VPC VPC

ELB

EC2


VPC VPC

ELB

VGW

EC2


VPC VPC

ELB

DC

VGW

VPN

S3 DATA STORAGE

EC2


VPC VPC

ELB

VGW

DATA CENTER VS. CLOUD CORE CONCEPTS

DATA CENTER VS. CLOUD

Min. 2W & 3G

CORE CONCEPTS


DATA CENTER MODEL

Min. 2W & 3G

CORE CONCEPTS


DATA CENTER MODEL

W

Min. 2W & 3G

CORE CONCEPTS


DATA CENTER MODEL

W W

Min. 2W & 3G

CORE CONCEPTS


DATA CENTER MODEL

W W

G

Min. 2W & 3G

CORE CONCEPTS


DATA CENTER MODEL

W W

G G

Min. 2W & 3G

CORE CONCEPTS


DATA CENTER MODEL

W W

G G G

Min. 2W & 3G

CORE CONCEPTS


DATA CENTER MODEL

W W

G G G

Min. 2W & 3G

85%

CORE CONCEPTS


DATA CENTER MODEL

W W

G G G

Min. 2W & 3G

85%

CORE CONCEPTS


DATA CENTER MODEL

W W W

G G G

Min. 2W & 3G

85%

CORE CONCEPTS


DATA CENTER MODEL

W W W

G G G

MANUAL

manual scaling no Chef required

Min. 2W & 3G

85%

CORE CONCEPTS


DATA CENTER MODEL

W W W

G G G

MANUAL


AUTOSCALING CLOUD

Min. 2W & 3G

85%

CORE CONCEPTS


DATA CENTER MODEL

W W W

G G G

MANUAL


AUTOSCALING CLOUD

Min. 2W & 3G

85%

CORE CONCEPTS


DATA CENTER MODEL

W W W

G G G

MANUAL


AUTOSCALING CLOUD

G

Min. 2W & 3G

85%

CORE CONCEPTS

W W W W

G G G G


DATA CENTER MODEL

W W W

G G G

MANUAL


AUTOSCALING CLOUD

W W

G G G

W

G G

Min. 2W & 3G

85%

CORE CONCEPTS

W


DATA CENTER MODEL

W W W

G G G

MANUAL


AUTOSCALING CLOUD

W W W

G G G G

AUTO

automatic scaling automated provisioning needed

Min. 2W & 3G

85%

CORE CONCEPTS

W

G

userdata.sh

AMI

START

OPERATING SYSTEM

CORE CONCEPTS

userdata.sh

AMI

START

OPERATING SYSTEM

GOAL

Publish Hello World

•  Instantly

•  Automatically

CORE CONCEPTS

userdata.sh

AMI

START

OPERATING SYSTEM

GOAL

Publish Hello World

•  Instantly


1st BOOT

CORE CONCEPTS

userdata.sh

AMI

START

OPERATING SYSTEM

GOAL

Publish Hello World

•  Instantly


Userdata.sh 1st BOOT

CORE CONCEPTS

userdata.sh

GOAL

Publish Hello World

•  Instantly


AMI

START

OPERATING SYSTEM

#!/bin/bash

yum install apache

wget http://hw/hello_world.html…


CORE CONCEPTS

AMI

START

userdata.sh

OPERATING SYSTEM

yum install apache

wget s3://hw/hello_world.html… RE

PLAC

ED W

/

GOAL

Publish Hello World

•  Instantly


#!/bin/bash

yum install apache

wget s3://hw/hello_world.html…


wget http://hw/first-boot.json

chef-client

CORE CONCEPTS

IMMUTABLE SERVERS LIMITED TO 2 ACTIONS: 1)  Start & provision server 2)  Kill it

CORE CONCEPTS


WITH IMMUTABLE SERVERS, YOU CAN’T SSH INTO BOX & CHANGE THINGS

CORE CONCEPTS

WITH IMMUTABLE SERVERS, YOU PREVENT UNEXPECTED CHANGES TO SERVERS

IMMUTABLE SERVERS

W1 W1 W1

W1 W1

LIMITED TO 2 ACTIONS: 1)  Start & provision server 2)  Kill it


MUTABLE SERVER:

CHANGE EACH BOX TO NEW VERSION

CORE CONCEPTS


IMMUTABLE SERVERS

W2 W1 W1

W1 W1



MUTABLE SERVER:


CORE CONCEPTS


IMMUTABLE SERVERS

W2 W2 W1

W1 W1



MUTABLE SERVER:


CORE CONCEPTS


IMMUTABLE SERVERS

W2 W2 W2

W1 W1



MUTABLE SERVER:


CORE CONCEPTS


IMMUTABLE SERVERS

W2 W2 W2

W2 W1



MUTABLE SERVER:


CORE CONCEPTS


IMMUTABLE SERVERS

W2 W2 W2

W2 W2



MUTABLE SERVER:


CORE CONCEPTS




W1 W1 W1

W1 W1

IMMUTABLE SERVER:

FLIP ON BOXES WITH NEW VERSION, SHUT OFF EXISTING BOXES

CORE CONCEPTS


IMMUTABLE SERVERS

W2 W2 W2

W2 W2



IMMUTABLE SERVER:


W1 W1 W1

W1 W1

CORE CONCEPTS


IMMUTABLE SERVERS

W2 W2 W2

W2 W2



IMMUTABLE SERVER:


W1 W1 W1

W1 W1

CORE CONCEPTS



BENEFITS:

No snowflakes

Easy rollbacks

Cleaner deploys

CORE CONCEPTS

AGENDA CHEF SERVER

CORE CONCEPTS

CHEF SOLO

OVERVIEW

GOLDEN IMAGES

CHEF SOLO WHAT’S REQUIRED

•  All cookbooks in one tarball •  Somewhere to put the tarball •  Tell instances how to provision with the tarball



PACKAGING

•  Use Berkshelf •  `berkshelf package COOKBOOK_NAME` •  `tar czvf cookbooks-VERSION.tgz ./cookbooks`



PACKAGING


STORAGE OPTIONS

•  S3 •  `s3cmd` makes it easy to upload to S3

•  Internal asset server



PACKAGING


STORAGE OPTIONS

•  S3 •  `s3cmd` makes it easy to upload to S3

•  Internal asset server

PROVISIONING

•  userdata.sh •  Get cookbooks off s3 and untar to /var/chef/

cookbooks •  Get first-boot.json off s3 -> /etc/chef/first-

boot.json •  Get solo.rb off s3 -> /etc/chef/solo.rb •  Run `chef-solo`

CHEF SOLO

WHERE IT WORKS

•  Good when your Chef run is fast •  Easy to set up •  No single point of failure

CHEF SOLO: PROs VS CONs

WHERE IT FAILS

•  When Chef runs occasionally fail •  Configuration changes •  Service discovery

TO PLAY LOL

ü  Minimize Unplanned Down2me

ü  Rolling Deploys ü  Expand Capacity

NEW FEATURES

q  Low Maintenance q  Low Barrier for New Services

WHAT PLAYERS WANT CHEF SOLO

RECOMMENDATION:

CHEF SOLO

•  Use chef-solo approach to get started

•  Stop when you have multiple applications

AGENDA CHEF SERVER

CORE CONCEPTS

CHEF SOLO

OVERVIEW

AMIs

WHAT’S REQUIRED

•  Chef Server •  Cookbooks uploaded to Chef Server •  Tell instances how to provision with Chef Server

CHEF SERVER

WHAT’S REQUIRED


RUNNING CHEF SERVER

•  Run your own •  Hosted Chef

CHEF SERVER

WHAT’S REQUIRED


GETTING COOKBOOKS TO CHEF SERVER

•  Use berkshelf •  `berks upload COOKBOOK_NAME`

RUNNING CHEF SERVER


CHEF SERVER

PROVISIONING

•  userdata.sh •  Fetch validation.pem •  Fetch first-boot.json •  Fetch client.rb •  Run `chef-client …`

GETTING COOKBOOKS TO CHEF SERVER

•  Use berkshelf •  `berks upload COOKBOOK_NAME`

WHAT’S REQUIRED


RUNNING CHEF SERVER


CHEF SERVER

WHERE IT WORKS

•  Updating feature flag configuration •  Warning: Can affect rolling

deploys!

CHEF SERVER: PROs VS CONs

WHERE IT FAILS

•  Single point of failure in Chef Server •  Long Chef runs cause problems •  Can cause problems in large

organizations where multiple teams depend on the same cookbook

CHEF SERVER

TO PLAY LOL

q  Minimize Unplanned Down2me

q  Rolling Deploys q  Expand Capacity

NEW FEATURES

ü  Low Maintenance q  Low Barrier for New Services

WHAT PLAYERS WANT CHEF SERVER

CHEF SERVER

RECOMMENDATION:

•  Use chef-server when managing feature flags

•  Not bad when you have multiple applications to deploy

•  Pro tips:

•  Use a shutdown script to remove instances from the chef server

•  Only use chef in daemon mode if you’re certain only feature flags will change

AGENDA CHEF SERVER

CORE CONCEPTS

CHEF SOLO

OVERVIEW

GOLDEN IMAGES

GOLDEN IMAGES

WHAT’S REQUIRED

•  Means to create Golden Image •  Configuration management

WHAT’S REQUIRED


Creating Image

•  We use Chef-Solo •  Build an RPM •  Cookbook installs RPMs

GOLDEN IMAGES

WHAT’S REQUIRED


GOLDEN IMAGES

Configuration Management

•  Archaius (on disk config) •  Chef

Creating Image

•  We use Chef-Solo •  Build an RPM •  Cookbook installs RPMS

WHERE IT WORKS

•  Rollbacks are perfect and instances always boot

•  Boot times are very fast

CHEF SOLO: PROs VS CONs

WHERE IT FAILS

•  Need another method for configuration management

•  Need another method for service discovery

•  Releasing patches requires entire new build

GOLDEN IMAGES

GOLDEN IMAGES

TO PLAY LOL

ü  Minimize Unplanned Down2me

ü  Rolling Deploys ü  Expand Capacity

NEW FEATURES

ü  Low Maintenance q  Low Barrier for New Services

WHAT PLAYERS WANT GOLDEN IMAGES

RECOMMENDATION:

•  Use when auto scaling is a must

•  Requires an up-front effort to get going

•  Make sure you can afford it

•  Pro tip:

•  Use the Netflix stack: Asgard, Aminator, and Archaius

GOLDEN IMAGES

c THANK YOU! QUESTIONS? ENGINEERING MANAGER| NOVEMBER 2013 TROTTER CASHION

Deploying the 'League of Legends' Data Pipeline with Chef (ARC205) | AWS re:Invent 2013

Technology

w 3g85

provision server

cloudcore conceptsmin

chef required85

box changesthings

serversw1w1w1mutable

serversw2w1w1mutable

serversw2w2w1mutable