MONGODB, OPS MANAGER & DOCKER @ SNCF Christophe TRINCAL / David TSANG-HIN-SUN / Sylvain CHAMBON MONGODB EUROPE 2016 - NOV 15th - LONDON FEEDBACK ON HOW TO DEPLOY MONGODB REPLICASET IN LESS THAN 5 MINUTES ! (PROOF OF CONCEPT)
MONGODB, OPS MANAGER & DOCKER @ SNCF
Christophe TRINCAL / David TSANG-HIN-SUN / Sylvain CHAMBON MONGODB EUROPE 2016 - NOV 15th - LONDON
FEEDBACK ON HOW TO DEPLOY MONGODB REPLICASET IN LESS THAN 5 MINUTES ! (PROOF OF CONCEPT)
MONGODB EUROPE 2016 - NOV 15th - LONDON
Christophe TRINCAL / David TSANG-HIN-SUN / Sylvain CHAMBON
AGENDA
01. MONGODB , PRESENTATION OF THE CONTEXT @ SNCF : Needs & Goals 02. INGREDIENTS / PRE-REQUISITES 03. RECIPES & ORCHESTRATION : how to combine Docker & Ops Manager for mongodb replicaset ? 04. CONCLUSION: Next steps & Improvements
MONGODB EUROPE 2016 - NOV 15th - LONDON
Christophe TRINCAL / David TSANG-HIN-SUN / Sylvain CHAMBON
MONGODB EUROPE 2016 - NOV 15th - LONDON
Christophe TRINCAL / David TSANG-HIN-SUN / Sylvain CHAMBON
ABOUT SNCF BUSINESS PROFILE
MONGODB EUROPE 2016 - NOV 15th - LONDON
Transport Group offering B to B and B to C services in several fields :
Christophe TRINCAL / David TSANG-HIN-SUN / Sylvain CHAMBON
MONGODB EUROPE 2016 - NOV 15th - LONDON
Christophe TRINCAL / David TSANG-HIN-SUN / Sylvain CHAMBON
ABOUT SNCF IT @ SNCF
MONGODB EUROPE 2016 - NOV 15th - LONDON
“Production IT” is the operator serving the IT System Departments of the group.
DC #3
DC #1a DC #1b
DC #2
Physical servers
Smartphones
Workstations Fixed telephone line
Operated applications
Christophe TRINCAL / David TSANG-HIN-SUN / Sylvain CHAMBON
MONGODB EUROPE 2016 - NOV 15th - LONDON
Christophe TRINCAL / David TSANG-HIN-SUN / Sylvain CHAMBON
NEEDS WANTS
HIGH AVAILABILITY
DELIVERY TIME
INDUSTRIALIZE
OPERATIONS
SIMPLIFICATION
SHARED SERVICES
AFFORDABLE PRICE
GOALS
RESILIENT FASTER
CHEAPER
NEEDS AND GOALS SERVICE OFFER
MONGODB EUROPE 2016 - NOV 15th - LONDON
Christophe TRINCAL / David TSANG-HIN-SUN / Sylvain CHAMBON
TRANSLATING THE REQUIREMENTS
MONGODB EUROPE 2016 - NOV 15th - LONDON
Christophe TRINCAL / David TSANG-HIN-SUN / Sylvain CHAMBON
CHEAPER
FASTER
MORE RESILIENT
SHARED INFRASTRUCTURE
AUTOMATED DEPLOYMENT
BACKUP, ALERTING POLICY
MONGODB EUROPE 2016 - NOV 15th - LONDON
Christophe TRINCAL / David TSANG-HIN-SUN / Sylvain CHAMBON
SELECTING INGREDIENTS
SHARED INFRASTRUCTURE
HARDWARE CLUSTER DOCKER
3 large servers (256 GB/6 TB)
3 Data Center rooms
MONGODB EUROPE 2016 - NOV 15th - LONDON
Christophe TRINCAL / David TSANG-HIN-SUN / Sylvain CHAMBON
CHEAPER
Lightweight containers
Full autonomy
Very fast startup
FASTER AUTOMATION
MONGODB EUROPE 2016 - NOV 15th - LONDON
Christophe TRINCAL / David TSANG-HIN-SUN / Sylvain CHAMBON
DOCKER
MONGODB EUROPE 2016 - NOV 15th - LONDON
Type of container : Small/Medium/Xlarge
Server details
Christophe TRINCAL / David TSANG-HIN-SUN / Sylvain CHAMBON
MONGODB EUROPE 2016 - NOV 15th - LONDON
Christophe TRINCAL / David TSANG-HIN-SUN / Sylvain CHAMBON
SELECTING INGREDIENTS
FASTER RESILIENT AUTOMATION BACKUP, ALERTS
OPS MANAGER
MongoDB’s Management
Solution
MONGODB EUROPE 2016 - NOV 15th - LONDON
Christophe TRINCAL / David TSANG-HIN-SUN / Sylvain CHAMBON
OPS MANAGER
MANAGEMENT
• MONITORING • BACKUP • AUTOMATION
• FULLY API-DRIVEABLE
MONGODB EUROPE 2016 - NOV 15th - LONDON
Christophe TRINCAL / David TSANG-HIN-SUN / Sylvain CHAMBON
INGREDIENTS
MONGODB EUROPE 2016 - NOV 15th - LONDON
1. OPS MANAGER
FASTER AND MORE RELIABLE DEPLOYMENTS
3 nodes for HA 2 MongoDB Enterprise Replica Sets
OPS 1
OPS Manager
HTTP Service
Backup database
(secondaire)
Application database (primaire)
Backup Daemon
Head DB
Head DB
Backup database (primaire)
Application database
(secondaire)
OPS 2
OPS Manager
HTTP Service
Backup Daemon
Head DB
Head DB
Application database
(secondaire)
OPS 3
OPS Manager
HTTP Service
Backup Daemon
Head DB
Head DB
Backup database
(secondaire)
Loadbalancer VIP
Ops Manager
Christophe TRINCAL / David TSANG-HIN-SUN / Sylvain CHAMBON
MONGODB EUROPE 2016 - NOV 15th - LONDON
Christophe TRINCAL / David TSANG-HIN-SUN / Sylvain CHAMBON
INGREDIENTS
MONGODB EUROPE 2016 - NOV 15th - LONDON
2. HARDWARE CLUSTER
CHEAPER DEPLOYMENT OF
MONGODB Linux Ubuntu 16.04 64bits / 256 GB RAM Pool of 30 IPs pre-allocated per server (10Gb network interface) Local SSD (6TB) storage per server
And some basic orchestration : bash, python script ….
Christophe TRINCAL / David TSANG-HIN-SUN / Sylvain CHAMBON
MONGODB EUROPE 2016 - NOV 15th - LONDON
Christophe TRINCAL / David TSANG-HIN-SUN / Sylvain CHAMBON
INGREDIENTS
MONGODB EUROPE 2016 - NOV 15th - LONDON
3. DOCKER ENGINE (V1.11)
CHEAPER, FASTER DEPLOYMENTS Standardised MongoDB deployment: 4 containers
And some basic orchestration : bash, python script ….
Mongodb replicaset
Christophe TRINCAL / David TSANG-HIN-SUN / Sylvain CHAMBON
MONGODB EUROPE 2016 - NOV 15th - LONDON
Christophe TRINCAL / David TSANG-HIN-SUN / Sylvain CHAMBON
PUTTING IT ALL TOGETHER
MONGODB EUROPE 2016 - NOV 15th - LONDON
1 2
3
Christophe TRINCAL / David TSANG-HIN-SUN / Sylvain CHAMBON
MONGODB EUROPE 2016 - NOV 15th - LONDON
Christophe TRINCAL / David TSANG-HIN-SUN / Sylvain CHAMBON
Our guideline: simple, simple, simple = only 2 commands (maybe 3 !)
l create_replicaset.py-groupsncf-test-sizesmall–name
sncf1-nb3–filepasswdfile.csv-backupdefault-alertingdefault-envprod–version3.4-dryrun
l remove_replicaset.py-namereplicaset-name
RECIPE & ORCHESTRATION
MONGODB EUROPE 2016 - NOV 15th - LONDON
Inthefuture:upgrade_replicaset.py–namereplicaset-name–sizelarge–nbnb-
replicaset-member–envprod
Christophe TRINCAL / David TSANG-HIN-SUN / Sylvain CHAMBON
MONGODB EUROPE 2016 - NOV 15th - LONDON
Christophe TRINCAL / David TSANG-HIN-SUN / Sylvain CHAMBON
create_replicaset.py–help-groupgroup_name-sizecontainer’ssizeS/M/XL–namereplicaset-name-nbnodes(3/3a/5/5a)-filepasswordfile.csv-backuppolicy-alertingpolicy-envprod/preprod-version3.2.10-dryrun(reportingonly)-helpThishelpmessage
MONGODB EUROPE 2016 - NOV 15th - LONDON
Ops Manager API – Create Group
Docker – Create Image
Capacity Planning
RECIPE & ORCHESTRATION
Christophe TRINCAL / David TSANG-HIN-SUN / Sylvain CHAMBON
MONGODB EUROPE 2016 - NOV 15th - LONDON
Christophe TRINCAL / David TSANG-HIN-SUN / Sylvain CHAMBON MONGODB EUROPE 2016 - NOV 15th - LONDON
r=requests.post(host+"/api/public/v1.0/groups",auth=HTTPDigestAuth(user,key),headers=headers,data=json.dumps(payload))j=r.json()group_id=j["id"]agent_api_key=j["agentApiKey"]
RECIPE & ORCHESTRATION
Christophe TRINCAL / David TSANG-HIN-SUN / Sylvain CHAMBON
Ops Manager API – Create Group
Docker – Create Image
Capacity Planning
MONGODB EUROPE 2016 - NOV 15th - LONDON
Christophe TRINCAL / David TSANG-HIN-SUN / Sylvain CHAMBON MONGODB EUROPE 2016 - NOV 15th - LONDON
RECIPE & ORCHESTRATION
Christophe TRINCAL / David TSANG-HIN-SUN / Sylvain CHAMBON
Ops Manager API – Create Group
Docker – Create Image
Capacity Planning
Dockerfile:FROMubuntu-sncf:16.04MAINTAINERext.osmozium.david.tsang-hin-sun@sncf.frRUNapt-getupdateRUNapt-getinstall-ynet-toolsRUNapt-getinstall-yvim-tinyRUNapt-getinstall-yaptitudeRUNapt-getinstall-ylibsasl2-2RUNapt-getinstall-yssl-certca-certificatesopensslRUNapt-getinstall-ymunin-nodeADDmongodb-mms-automation-agent-manager_2.5.18.1647-1_amd64.deb/tmp/mongodb-mms-automation-agent-manager_2.5.18.1647-1_amd64.debRUNdpkg-i/tmp/mongodb-mms-automation-agent-manager_2.5.18.1647-1_amd64.debADDautomation-agent.config/etc/mongodb-mms/automation-agent.configRUNchownmongodb:mongodb/etc/mongodb-mms/automation-agent.configRUNchmod600/etc/mongodb-mms/automation-agent.configRUNmkdir/data&&chown-Rmongodb:mongodb/dataADDrunautomationagent.sh/runautomationagent.shENTRYPOINT/runautomationagent.sh&&bash
MONGODB EUROPE 2016 - NOV 15th - LONDON
Christophe TRINCAL / David TSANG-HIN-SUN / Sylvain CHAMBON MONGODB EUROPE 2016 - NOV 15th - LONDON
Start containers
Create and mount LVM Volumes
Reserve IP addresses
RECIPE & ORCHESTRATION
Christophe TRINCAL / David TSANG-HIN-SUN / Sylvain CHAMBON
MONGODB EUROPE 2016 - NOV 15th - LONDON
Christophe TRINCAL / David TSANG-HIN-SUN / Sylvain CHAMBON MONGODB EUROPE 2016 - NOV 15th - LONDON
Start containers
Create and mount LVM Volumes
Reserve IP addresses
RECIPE & ORCHESTRATION
Christophe TRINCAL / David TSANG-HIN-SUN / Sylvain CHAMBON
MONGODB EUROPE 2016 - NOV 15th - LONDON
Christophe TRINCAL / David TSANG-HIN-SUN / Sylvain CHAMBON MONGODB EUROPE 2016 - NOV 15th - LONDON
Start containers
Create and mount LVM Volumes
Reserve IP addresses
RECIPE & ORCHESTRATION
Christophe TRINCAL / David TSANG-HIN-SUN / Sylvain CHAMBON
dockerrun--name$replica-node-$n\--restart=always-d-hAAA.BBB.CCC.DDD-m2G\--cpu-shares1024--blkio-weight=300\-pAAA.BBB.CCC.DDD:27017:27017-ti\-v/data/replicaset1/AAA.BBB.CCC.DDD:/data:rw–v\/home/docker/vol/replicaset1/AAA.BBB.CCC.DDD:/var/lib/mongodb-mms-automation/:rwsncf/ubuntu-replicaset1:16.04/bin/bash
MONGODB EUROPE 2016 - NOV 15th - LONDON
Christophe TRINCAL / David TSANG-HIN-SUN / Sylvain CHAMBON MONGODB EUROPE 2016 - NOV 15th - LONDON
Ops Manager Backup API
Ops Manager Alerts API
create_replicaset.py–help-groupgroup_name-sizecontainer’ssizeS/M/XL–namereplicaset-name-nbnodes(3/3a/5/5a)-filepasswordfile.csv-backuppolicy-alertingpolicy-envprod/preprod-version3.2.10-dryrun(reportingonly)-helpThishelpmessage
"replicaSets":[{"_id":"testReplicaSet","members":[{"_id":0,"arbiterOnly":false,"hidden":false,"host":"testReplicaSet_0","priority":10,"slaveDelay":0,"votes":1},{"_id":1,"arbiterOnly":false,"hidden":false,"host":"testReplicaSet_1","priority":1,"slaveDelay":0,"votes":1},{"_id":2,"arbiterOnly":true,"hidden":false,"host":"testReplicaSet_2","priority":1,"slaveDelay":0,"votes":1}]}
Ops Manager Automation API
RECIPE & ORCHESTRATION
Christophe TRINCAL / David TSANG-HIN-SUN / Sylvain CHAMBON
MONGODB EUROPE 2016 - NOV 15th - LONDON
Christophe TRINCAL / David TSANG-HIN-SUN / Sylvain CHAMBON MONGODB EUROPE 2016 - NOV 15th - LONDON
Ops Manager Backup API
Ops Manager Alerts API
Ops Manager Automation API
{"clusterId":"CLUSTER-ID","dailySnapshotRetentionDays":7,"groupId":"BACKUP-ID","monthlySnapshotRetentionMonths":13,"pointInTimeWindowHours":24,"snapshotIntervalHours":6,"snapshotRetentionDays":2,"weeklySnapshotRetentionWeeks":4}
RECIPE & ORCHESTRATION
Christophe TRINCAL / David TSANG-HIN-SUN / Sylvain CHAMBON
MONGODB EUROPE 2016 - NOV 15th - LONDON
Christophe TRINCAL / David TSANG-HIN-SUN / Sylvain CHAMBON MONGODB EUROPE 2016 - NOV 15th - LONDON
Ops Manager Backup API
Ops Manager Alerts API
Ops Manager Automation API
{"eventTypeName":"MONITORING_AGENT_DOWN","groupId":"GROUP-ID","notifications":[{"delayMin":0,"emailEnabled":true,"intervalMin":60,"smsEnabled":false,"typeName":"GROUP"}],"typeName":"AGENT",}
RECIPE & ORCHESTRATION
Christophe TRINCAL / David TSANG-HIN-SUN / Sylvain CHAMBON
MONGODB EUROPE 2016 - NOV 15th - LONDON
Christophe TRINCAL / David TSANG-HIN-SUN / Sylvain CHAMBON MONGODB EUROPE 2016 - NOV 15th - LONDON
Notify user
RECIPE & ORCHESTRATION
Christophe TRINCAL / David TSANG-HIN-SUN / Sylvain CHAMBON
MONGODB EUROPE 2016 - NOV 15th - LONDON
Christophe TRINCAL / David TSANG-HIN-SUN / Sylvain CHAMBON MONGODB EUROPE 2016 - NOV 15th - LONDON
Scalability ? - Add more physical hosts Security? - limit use of Global Owner role in Ops Manager API Availability? - Use of 3 Distinct DATACENTERS Software upgrades? - Docker 1.12 swarm enabled ? Docker API ? - Graphical interface for true As A Service experience?
IMPROVEMENTS TO THE RECIPE
Christophe TRINCAL / David TSANG-HIN-SUN / Sylvain CHAMBON
MONGODB EUROPE 2016 - NOV 15th - LONDON
Christophe TRINCAL / David TSANG-HIN-SUN / Sylvain CHAMBON
CONCLUSION
MONGODB EUROPE 2016 - NOV 15th - LONDON
Christophe TRINCAL / David TSANG-HIN-SUN / Sylvain CHAMBON
CHEAPER
FASTER
MORE RESILIENT
Several applications hosted on
shared services
End to end industrialization
Streamline architectures
Availability rate
Recovery point objective
Less than one hour data loss
MONGODB EUROPE 2016 - NOV 15th - LONDON
Christophe TRINCAL / David TSANG-HIN-SUN / Sylvain CHAMBON
NEXT STEPS & IMPROVEMENTS
Integrate to a software factory
Integrate into the internal
cloud
Create a Disaster
Recovery Plan
MONGODB EUROPE 2016 - NOV 15th - LONDON
Christophe TRINCAL / David TSANG-HIN-SUN / Sylvain CHAMBON
Next steps ….
…. and improvements
THANKS MERCI
MONGODB EUROPE 2016 - NOV 15th - LONDON
Christophe TRINCAL / David TSANG-HIN-SUN / Sylvain CHAMBON