Top Banner
How to Replace a Jet Engine of Your System In-Flight Aysylu Greenberg, Software Engineer, Google
46

MesosCon Asia Keynote: Replacing a Jet Engine Mid-flight

Jan 19, 2017

Download

Software

Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Page 1: MesosCon Asia Keynote: Replacing a Jet Engine Mid-flight

How to Replace a Jet Engine of Your System In-Flight

Aysylu Greenberg, Software Engineer, Google

Page 2: MesosCon Asia Keynote: Replacing a Jet Engine Mid-flight

AysyluGreenberg@aysylu22

Page 3: MesosCon Asia Keynote: Replacing a Jet Engine Mid-flight

BuildSystemStoryPhoto by imanka / CC BY

Page 4: MesosCon Asia Keynote: Replacing a Jet Engine Mid-flight

TowardsDistributedBuildSystem

Page 5: MesosCon Asia Keynote: Replacing a Jet Engine Mid-flight

TowardsDistributedBuildSystem

Page 6: MesosCon Asia Keynote: Replacing a Jet Engine Mid-flight

TowardsDistributedBuildSystem

Page 7: MesosCon Asia Keynote: Replacing a Jet Engine Mid-flight

TowardsDistributedBuildSystem

Page 8: MesosCon Asia Keynote: Replacing a Jet Engine Mid-flight

TowardsDistributedBuildSystem

Page 9: MesosCon Asia Keynote: Replacing a Jet Engine Mid-flight

TowardsDistributedBuildSystem

Page 10: MesosCon Asia Keynote: Replacing a Jet Engine Mid-flight

Continuous integration

system

BuildRabbit

Source System

Google engineers &

teams

Release infrastructure

Integration testing

infrastructure

Build artifact storage

Blaze

Page 11: MesosCon Asia Keynote: Replacing a Jet Engine Mid-flight

Scale•  Engineers:>30,000developersin40+offices

Page 12: MesosCon Asia Keynote: Replacing a Jet Engine Mid-flight

Scale•  Engineers:>30,000developersin40+offices

•  Commits:15Kbyhumans+30Kbyrobots/day

Page 13: MesosCon Asia Keynote: Replacing a Jet Engine Mid-flight

Scale•  Engineers:>30,000developersin40+offices

•  Commits:15Kbyhumans+30Kbyrobots/day

•  Sourcecode:2billionLOC

Page 14: MesosCon Asia Keynote: Replacing a Jet Engine Mid-flight

Scale•  Engineers:>30,000developersin40+offices

•  Commits:15Kbyhumans+30Kbyrobots/day

•  Sourcecode:2billionLOC•  Buildsandtests:5MperdaythroughBuildRabbit

Page 15: MesosCon Asia Keynote: Replacing a Jet Engine Mid-flight

Scale•  Engineers:>30,000developersin40+offices

•  Commits:15Kbyhumans+30Kbyrobots/day

•  Sourcecode:2billionLOC•  Buildsandtests:5MperdaythroughBuildRabbit

•  PetabytesofoutputarRfacts

Page 16: MesosCon Asia Keynote: Replacing a Jet Engine Mid-flight

ArchitectureUpgradehttps://thetravelintern.com/hangzhou-things-to-do/

Page 17: MesosCon Asia Keynote: Replacing a Jet Engine Mid-flight

ArchitectureUpgradePhoto by vermazeren / CC BY

Page 18: MesosCon Asia Keynote: Replacing a Jet Engine Mid-flight

Continuous integration

system

BuildRabbit

Source System

Google engineers &

teams

Release infrastructure

Integration testing

infrastructure

Build artifact storage

Blaze

Page 19: MesosCon Asia Keynote: Replacing a Jet Engine Mid-flight

HowtoReplaceaJetEngineofyourSystem

In-Flight

Page 20: MesosCon Asia Keynote: Replacing a Jet Engine Mid-flight

https://en.wikipedia.org/wiki/File:Helicopter_aerial_refueling.jpg

Page 21: MesosCon Asia Keynote: Replacing a Jet Engine Mid-flight
Page 22: MesosCon Asia Keynote: Replacing a Jet Engine Mid-flight

MigraRonwithZeroDownRmePhoto by moonjazz / CC BY SA

Page 23: MesosCon Asia Keynote: Replacing a Jet Engine Mid-flight

OldArchitecture:Client/Server

Client for User

BuildRabbit Scheduler

BuildRabbit Worker

Page 24: MesosCon Asia Keynote: Replacing a Jet Engine Mid-flight

NewArchitecture:BuildService

Client for

User

Persistent Queue

BuildRabbit Worker

Build Artifacts Build Progress

Info

event RPC stream

Page 25: MesosCon Asia Keynote: Replacing a Jet Engine Mid-flight

Persistent Queue

BuildRabbit Worker

Build Artifacts Build Progress

Info

event RPC stream

NewArchitecture:BuildService

Client for

User

Page 26: MesosCon Asia Keynote: Replacing a Jet Engine Mid-flight

Client for

User

BuildRabbit Scheduler

BuildRabbit Worker

Page 27: MesosCon Asia Keynote: Replacing a Jet Engine Mid-flight

Client for

User

BuildRabbit Scheduler

BuildRabbit Worker

Persistent Queue

BuildRabbit Worker

Build Progress Info Build Artifacts

Client for

User

Page 28: MesosCon Asia Keynote: Replacing a Jet Engine Mid-flight

Photo by goosmurf / CC BY

FOCUSONINTERMEDIATESTATE

Page 29: MesosCon Asia Keynote: Replacing a Jet Engine Mid-flight

Client for User

BuildRabbit Scheduler

BuildRabbit Worker

Persistent Queue

Page 30: MesosCon Asia Keynote: Replacing a Jet Engine Mid-flight

Photo by goosmurf / CC BY

FOCUSONINTERMEDIATESTATE

Page 31: MesosCon Asia Keynote: Replacing a Jet Engine Mid-flight

Client for

User

BuildRabbit Scheduler

BuildRabbit Worker

Persistent Queue

BuildRabbit Worker

Build Progress Info Build Artifacts

Client for

User

Page 32: MesosCon Asia Keynote: Replacing a Jet Engine Mid-flight

Photo by goosmurf / CC BY

FOCUSONINTERMEDIATESTATE

Page 33: MesosCon Asia Keynote: Replacing a Jet Engine Mid-flight

Client for

User

BuildRabbit Scheduler

BuildRabbit Worker

Persistent Queue

BuildRabbit Worker

Build Progress Info Build Artifacts

Client for

User

Page 34: MesosCon Asia Keynote: Replacing a Jet Engine Mid-flight

Photo by goosmurf / CC BY

FOCUSONINTERMEDIATESTATE

Page 35: MesosCon Asia Keynote: Replacing a Jet Engine Mid-flight

ROLLOUTINCREMENTALLY

Photo by firepile / CC BY

Page 36: MesosCon Asia Keynote: Replacing a Jet Engine Mid-flight
Page 37: MesosCon Asia Keynote: Replacing a Jet Engine Mid-flight

ROLLOUTINCREMENTALLY

Photo by firepile / CC BY

Page 38: MesosCon Asia Keynote: Replacing a Jet Engine Mid-flight

Client for

User

BuildRabbit Scheduler

BuildRabbit Worker

Persistent Queue

BuildRabbit Worker

Build Progress Info Build Artifacts

Client for

User

Page 39: MesosCon Asia Keynote: Replacing a Jet Engine Mid-flight

ROLLOUTINCREMENTALLY

Photo by firepile / CC BY

Page 40: MesosCon Asia Keynote: Replacing a Jet Engine Mid-flight
Page 41: MesosCon Asia Keynote: Replacing a Jet Engine Mid-flight

PRACTICEROLLOUT

Photo by greg_photos / CC BY SA

Page 42: MesosCon Asia Keynote: Replacing a Jet Engine Mid-flight

Client for User

BuildRabbit Scheduler

BuildRabbit Worker

Persistent Queue

Page 43: MesosCon Asia Keynote: Replacing a Jet Engine Mid-flight

PRACTICEROLLOUT

Photo by greg_photos / CC BY SA

Page 44: MesosCon Asia Keynote: Replacing a Jet Engine Mid-flight

•  Topreventsimplestmistakes

•  For organization resilience

Page 45: MesosCon Asia Keynote: Replacing a Jet Engine Mid-flight

PRACTICEROLLOUT

FOCUSONINTERMEDIATESTATE

ROLLOUTINCREMENTALLY

Page 46: MesosCon Asia Keynote: Replacing a Jet Engine Mid-flight

How to Replace a Jet Engine of Your System In-Flight

Aysylu Greenberg, Software Engineer, Google