Continuous Delivery of Infrastructure The case of the growing startup
3/49
Summary
● Introduction● Why Continuous Delivery of Infrastructure● Automate Configuration● Hardware Industrialization● Automate OS Install● Hardware availability● Continuous delivery
9/49
Few numbers of BlaBlaCar today
2 Data Centers in Paris
● 242 servers
● 109 virtual servers (VMWare)
● 110 EC2 instances during working day
– 20 instances the rest of the time
11/49
We have a strong agility culture
● Push our main app on production 10x a day● Developers make the push themselves
● App can be pushed only after a Bamboo's merge● Bamboo merge only after thousands of tests
14/49
Physical Server
● 3 racks on a single DC
● 12 servers per rack
● More than 20 differents
hardware configurations
15/49
Cloud (Amazon EC2)
● Only 10 to 20 instances always running
● Used for elasticity
● 10 to 20 instances running for test (Bamboo)
16/49
Network
● Strong Team who designed a very strong network– Rémi (@shtouff)
● Not a problem for us to scale the network
19/49
Improve our Service Deployment
● Started to work on a front web server– Ease configuration deployment
– Ease a new installation
● Then worked on all servers– First just for configuration
– Then all service deployment
21/49
Haute Couture
● Before our automation process:– One server configuration by service
– Multiple services
23/49
Next Step - Virtualization
● Create a virtualization platform– 4 big servers
– A big disk array
25/49
Cloud: Why on premise in 2015 ?
● Data privacy concerns● Limited elasticity required● Cost efficiency● History & Culture
26/49
Now – Standard Servers
● Define a single 1U server– Dual-CPU (12 cores)
– 128 Go of RAM
– Disk configuration vary on 3 flavors:● Only 2 SSD mirrored (RAID) ● Fast disk (SSD) needed● Very big capacity (lot of disks)
28/49
Automate debian installation
● All our infra is debian based● PXE● ipmi
● preseed– (thx to @jbfavre)
29/49
Virtualization => Template
● Work well on EC2 (only option for instance install)
● Never used in VMWare– Same install as physical server debian based
(thanks to Chef)
30/49
Define our needs to choose a tool
● IP allocation● Bootstrap chef-client● PXE preseed debian● Can install either a physical, virtual or cloud server● Configure the BMC interface
– HP (iLo)
– Dell (DRAC)
– SuperMicro
31/49
● Because Simon (@s_ourea) chose it !– More likely to be maintained
– Big community
● Fits our needs
33/49
We need more
● Chef / Foreman, great tools to:– Install OS
– Install all services and configure them
● No easy solution when you need a specific hardware on never used server
34/49
Collins !
● Nice tool to manage our hardware inventory.● A lot of good reasons to choose it:
– Simon (@s_ourea) chose it
– Fit our needs
● Linked with
Foreman
36/49
Install servers into DC● Limited added value in the team racking servers● Time consuming● DCs can be far away
● Simon(@s_ourea) & Rémi(@shtouff) pushed a new idea– Buy servers only when we can use a full rack
– Write an installation documentation
– Choose a company to rack
37/49
Our procedure to have a new rack
● Buy a new rack into the DC
● Our network team installs 2 switchs in the rack.
● They add links to our core network equipments
● Hey! Let's ask to our provider to rack and setup all servers, following Simon's documentation
38/49
The typical life of a server
● Discovery, and go into Collins
● Another team takes it and installs it via Foreman
● Chef is called via Foreman
● Chef is always running
40/49
Chef and tests
● Yes, we test all of our
infrastructure's services !
● Thanks to Chef and all of it's community:
KitchenCI + vagrant + serverspec, chefspec
41/49
Chef - Cookbook life
● Each cookbook in it's own git repository
● First test and merge with Bamboo
● Push in production with a new version number
● All servers converge to the desired state
42/49
What do you test ?
● Everything! (one day perhaps)– Server install
– Bootstrapping
– Convergence
– Clustering /!\ (not automatic)
– Version upgrade ( one day !)
– Coffee quality
43/49
Continuous delivery, really ?
● New cookbook versions pushed 10x times / day
● Provision several new servers / VM / instances by day
● Buy a rack full of servers, and install them in less than a day
46/49
What we will continue to do
● Test our infra (with or without container)● Industrialize our processes● Spread knowledge to all other teams● Dream solutions, and finally build them
● Listen to Simon (@s_ourea)
48/49
What to keep in mind
● Involve overall company
● Support company growth
● Don't get attached to tools
● Keep a “small team” spirit