Implementing Rolling Upgrades Containerized Environment powered by Mesos, Marathon, Docker and HAProxy
Implementing Rolling Upgrades using Mesos, Marathon, Docker, HAProxy
Apr 16, 2017
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Implementing Rolling Upgrades
Containerized Environment powered by Mesos, Marathon, Docker and HAProxy
1. Deploy NEW application (with a new App ID) with the minimum set of instances (eg: 1 in the beginning)
as needed to manage the traffic
2. Wait for the NEW instance(s) to go healthy
3. Add the NEW Instance(s) to the HAProxy Pool
4. Load gets balanced between OLD and NEW Pool
5. DISABLE one of the OLD instances in the HAProxy Pool
6. Wait for the HAProxy Connection to drain on the Disabled OLD Instance
7. After waiting period is over, Remove the Disabled OLD Instance by issuing Docker STOP on the
Container
8. Scale the NEW Instances and let them add to the HAProxy Pool
9. Repeat the steps 4 - 8 till all OLD instances are removed, and all the NEW instances are added to
HAProxy
10. Delete the OLD Application
ALGORITHM: Rolling Upgrades
OLD OLD OLD OLD OLD OLD
HAProxy
CURRENT STATUSOld Instances are running and load balanced by HAProxy
OLD OLD OLD OLD OLD OLD
HAProxy
L4
DNS
Customer
Service Port 1
CURRENT STATUSOld Instances are running and load balanced by HAProxy
NEWOLD OLD OLD OLD OLD OLD
HAProxy
Minimum required to serve just
enough traffic
NEW INSTANCE DEPLOYED1 Instance of the New Version of Application is deployed
NEWOLD OLD OLD OLD OLD OLD
HAProxy
NEW INSTANCE DEPLOYEDHealth Check Passed
NEWOLD OLD OLD OLD OLD OLD
HAProxy
NEW INSTANCE DEPLOYEDHealth Check Passed. Added to Load Balancer.One Old Instance is DISABLED from Load Balancer
OLD OLD OLD OLD OLD OLD
HAProxy
L4
DNS
Customer
Service Port 1
NEW
NEW INSTANCE DEPLOYEDAdded to Load Balancer. 16% new requests goes to NEW Instance. No New Request will go to Old Instance #1
disabled
NEWOLD OLD OLD OLD OLD OLD
HAProxy
ZDD
Control
Marathon
Mesos
Control
HOW IS ROLLING UPGRADE IMPLEMENTEDZDD requests HAProxy to DISABLE one of the OLD Instance
ZDD is a Python Script part of the Marathon-lb Project from Mesosphere
disabled
NEWOLD OLD OLD OLD OLD OLD
HAProxy
ZDD
Control
Marathon
Mesos
Control
HOW IS ROLLING UPGRADE IMPLEMENTEDZDD now waits for N seconds to drain any existing connections
N seconds
NEWOLD OLD OLD OLD OLD OLD
HAProxyMesos
STOP
ZDD
Control
Marathon
Control
HOW IS ROLLING UPGRADE IMPLEMENTEDAfter N seconds, ZDD requests Marathon to STOP one of the Old Instances
NEWOLD OLD OLD OLD OLD OLD
HAProxyMesos
STOP
M seconds
ZDD
Control
Marathon
Control
HOW IS ROLLING UPGRADE IMPLEMENTEDMesos invokes Docker STOP on the Old Container, and waits for M seconds
NEWOLD OLD OLD OLD OLD
HAProxyMesos
Mseconds
STOP
N seconds
OLD
NEW NEWOLD OLD OLD OLD OLD
HAProxyMesos
Mseconds
STOP
N seconds
NEW NEWOLD OLD OLD OLD
HAProxy
M seconds
Mesos
STOP
NEW
N seconds
NEW NEWOLD OLD OLD
HAProxy
M seconds
Mesos
STOP
NEWNEW
N seconds
NEW NEWOLD OLD
HAProxy
M seconds
Mesos
STOP
NEWNEWNEW
N seconds
NEW NEWOLD
HAProxy
M seconds
Mesos
STOP
NEWNEWNEWNEW
N seconds
NEW NEW
HAProxyMesos
NEWNEWNEWNEWNEW
NEW NEW
HAProxyMesos
NEWNEWNEWNEWNEW
ROLLING UPGRADE COMPLETE !
NEW NEW
HAProxy
NEWNEWNEWNEW
L4
DNS
Customer
HOW IS ROLLING UPGRADE IMPLEMENTEDNow all NEW Instances have been added to HAProxy Pool
APP APP APP APP APP APP
HAProxy
ZDD
Control
Marathon
Mesos
Control
DEV TEST STAGING PROD JENKINSPIPELINE
ROLLING UPGRADESHAPPEN ACROSSALL ENVIRONMENTZDD orchestratesthe deployment
Related Documents
