Top Banner
Continuous Delivery of Infrastructure The case of the growing startup
49

BlaBlaCar and infrastructure automation

Jul 16, 2015

Download

Technology

sinfomicien
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Page 1: BlaBlaCar and infrastructure automation

Continuous Delivery of Infrastructure

The case of the growing startup

Page 2: BlaBlaCar and infrastructure automation

Nicolas Blanc - BlaBlArchitect

SinfomicSinfomic (1999)

@thewhitegeek

(2001)

(2005)

(2008)

(2012)

Page 3: BlaBlaCar and infrastructure automation

3/49

Summary

● Introduction● Why Continuous Delivery of Infrastructure● Automate Configuration● Hardware Industrialization● Automate OS Install● Hardware availability● Continuous delivery

Page 4: BlaBlaCar and infrastructure automation

Introduction

Page 5: BlaBlaCar and infrastructure automation

80€ 20€20€

20€

@BlaBlaCar_FR

20€

Trusted Ridesharing

Page 6: BlaBlaCar and infrastructure automation

People Powered Travel

A travel search engine

A trusted community

@BlaBlaCar_FR

Page 7: BlaBlaCar and infrastructure automation

Fast growth

@BlaBlaCar_FR

Page 8: BlaBlaCar and infrastructure automation

8/49

Why Infrastructure Automation ?

Page 9: BlaBlaCar and infrastructure automation

9/49

Few numbers of BlaBlaCar today

2 Data Centers in Paris

● 242 servers

● 109 virtual servers (VMWare)

● 110 EC2 instances during working day

– 20 instances the rest of the time

Page 10: BlaBlaCar and infrastructure automation

Our Continuous Delivery at the top.

Jealous from the development team !

Page 11: BlaBlaCar and infrastructure automation

11/49

We have a strong agility culture

● Push our main app on production 10x a day● Developers make the push themselves

● App can be pushed only after a Bamboo's merge● Bamboo merge only after thousands of tests

Page 12: BlaBlaCar and infrastructure automation

12/49

tools to manage them all

Page 13: BlaBlaCar and infrastructure automation

Old Infrastructure Overview

Page 14: BlaBlaCar and infrastructure automation

14/49

Physical Server

● 3 racks on a single DC

● 12 servers per rack

● More than 20 differents

hardware configurations

Page 15: BlaBlaCar and infrastructure automation

15/49

Cloud (Amazon EC2)

● Only 10 to 20 instances always running

● Used for elasticity

● 10 to 20 instances running for test (Bamboo)

Page 16: BlaBlaCar and infrastructure automation

16/49

Network

● Strong Team who designed a very strong network– Rémi (@shtouff)

● Not a problem for us to scale the network

Page 17: BlaBlaCar and infrastructure automation

Automate the Configuration

Page 18: BlaBlaCar and infrastructure automation

18/49

Choosing the right tool

Page 19: BlaBlaCar and infrastructure automation

19/49

Improve our Service Deployment

● Started to work on a front web server– Ease configuration deployment

– Ease a new installation

● Then worked on all servers– First just for configuration

– Then all service deployment

Page 20: BlaBlaCar and infrastructure automation

Hardware Industrialization

Page 21: BlaBlaCar and infrastructure automation

21/49

Haute Couture

● Before our automation process:– One server configuration by service

– Multiple services

Page 22: BlaBlaCar and infrastructure automation

22/49

Haute Couture - Caveats

● Tailored server are complicated to design● Hard to upgrade

Page 23: BlaBlaCar and infrastructure automation

23/49

Next Step - Virtualization

● Create a virtualization platform– 4 big servers

– A big disk array

Page 24: BlaBlaCar and infrastructure automation

24/49

Virtualization - Caveats

● I/Os Performances compare to physical server● Network Performances

Page 25: BlaBlaCar and infrastructure automation

25/49

Cloud: Why on premise in 2015 ?

● Data privacy concerns● Limited elasticity required● Cost efficiency● History & Culture

Page 26: BlaBlaCar and infrastructure automation

26/49

Now – Standard Servers

● Define a single 1U server– Dual-CPU (12 cores)

– 128 Go of RAM

– Disk configuration vary on 3 flavors:● Only 2 SSD mirrored (RAID) ● Fast disk (SSD) needed● Very big capacity (lot of disks)

Page 27: BlaBlaCar and infrastructure automation

Automate OS Installation

Page 28: BlaBlaCar and infrastructure automation

28/49

Automate debian installation

● All our infra is debian based● PXE● ipmi

● preseed– (thx to @jbfavre)

Page 29: BlaBlaCar and infrastructure automation

29/49

Virtualization => Template

● Work well on EC2 (only option for instance install)

● Never used in VMWare– Same install as physical server debian based

(thanks to Chef)

Page 30: BlaBlaCar and infrastructure automation

30/49

Define our needs to choose a tool

● IP allocation● Bootstrap chef-client● PXE preseed debian● Can install either a physical, virtual or cloud server● Configure the BMC interface

– HP (iLo)

– Dell (DRAC)

– SuperMicro

Page 31: BlaBlaCar and infrastructure automation

31/49

● Because Simon (@s_ourea) chose it !– More likely to be maintained

– Big community

● Fits our needs

Page 32: BlaBlaCar and infrastructure automation

Hardware Availability

Page 33: BlaBlaCar and infrastructure automation

33/49

We need more

● Chef / Foreman, great tools to:– Install OS

– Install all services and configure them

● No easy solution when you need a specific hardware on never used server

Page 34: BlaBlaCar and infrastructure automation

34/49

Collins !

● Nice tool to manage our hardware inventory.● A lot of good reasons to choose it:

– Simon (@s_ourea) chose it

– Fit our needs

● Linked with

Foreman

Page 35: BlaBlaCar and infrastructure automation

Hey ? What about the initial setup of a physical server ?

Page 36: BlaBlaCar and infrastructure automation

36/49

Install servers into DC● Limited added value in the team racking servers● Time consuming● DCs can be far away

● Simon(@s_ourea) & Rémi(@shtouff) pushed a new idea– Buy servers only when we can use a full rack

– Write an installation documentation

– Choose a company to rack

Page 37: BlaBlaCar and infrastructure automation

37/49

Our procedure to have a new rack

● Buy a new rack into the DC

● Our network team installs 2 switchs in the rack.

● They add links to our core network equipments

● Hey! Let's ask to our provider to rack and setup all servers, following Simon's documentation

Page 38: BlaBlaCar and infrastructure automation

38/49

The typical life of a server

● Discovery, and go into Collins

● Another team takes it and installs it via Foreman

● Chef is called via Foreman

● Chef is always running

Page 39: BlaBlaCar and infrastructure automation

Tests, and Continuous delivery really ?

Page 40: BlaBlaCar and infrastructure automation

40/49

Chef and tests

● Yes, we test all of our

infrastructure's services !

● Thanks to Chef and all of it's community:

KitchenCI + vagrant + serverspec, chefspec

Page 41: BlaBlaCar and infrastructure automation

41/49

Chef - Cookbook life

● Each cookbook in it's own git repository

● First test and merge with Bamboo

● Push in production with a new version number

● All servers converge to the desired state

Page 42: BlaBlaCar and infrastructure automation

42/49

What do you test ?

● Everything! (one day perhaps)– Server install

– Bootstrapping

– Convergence

– Clustering /!\ (not automatic)

– Version upgrade ( one day !)

– Coffee quality

Page 43: BlaBlaCar and infrastructure automation

43/49

Continuous delivery, really ?

● New cookbook versions pushed 10x times / day

● Provision several new servers / VM / instances by day

● Buy a rack full of servers, and install them in less than a day

Page 44: BlaBlaCar and infrastructure automation

So cool ! But...What will you do in the future ?

Page 45: BlaBlaCar and infrastructure automation

45/49

Container

CoreOS

Page 46: BlaBlaCar and infrastructure automation

46/49

What we will continue to do

● Test our infra (with or without container)● Industrialize our processes● Spread knowledge to all other teams● Dream solutions, and finally build them

● Listen to Simon (@s_ourea)

Page 47: BlaBlaCar and infrastructure automation

Conclusion

Page 48: BlaBlaCar and infrastructure automation

48/49

What to keep in mind

● Involve overall company

● Support company growth

● Don't get attached to tools

● Keep a “small team” spirit

Page 49: BlaBlaCar and infrastructure automation

That's all folks!

Thanks you!

Nicolas Blanc(@thewhitegeek)