EuroPython 2014 Devops Risk Mitigation

DevOps Risk Mitigationwww.immobilienscout24.de

Berlin | 23.07.2014 | Schlomo SchapiroSystems Architect, Open Source Evangelist

License: http://creativecommons.org/licenses/by-nc-nd/3.0/

Test Driven Infrastructure

@schlomoschapiro

ExpensiveFix

CheapFix

Costs Of Finding Bugs In Production

PlanDesignBudget

Develop Test

RU

NDEV

PlanDesignBudget

Develop Test RUN OPSTIME

PlanDesignBudget

Develop Test

PlanDesignBudget

Develop Test RUN

Proxyconfig

Databaseborked

„Buy Now“button broken

MTA dropall mail

Load BalancerConfiguration

Everything costsonly 0 €

Login possiblewithout password

NFS notavailable

DB Replicationstopped

No adsshown

BrokenCSS / JS

Tomcat won'tstart/stop

Service usernot defined

sudoersinvalid

DevOps: Respect & Learning

DEV learn from OPS to think about:

● Resources (CPU, RAM, Disk)● Services (Start, Stop, Status)● Dependencies (Start DB before App)● Logfiles (Rotate, Remove)● Disk Space● Monitoring and Alarming● ...

OPS learn from DEV to think about:

● Incremental Improvement● Infrastructure as Code● Version Control System● Coding (OO, Functions, Libraries …)● Code Quality● Unit & Integration Tests● Test Automation● ...

Untested=

Broken

Unit TestsTest the smallest possible components in an artifical environment.

System TestsTest the entire application in a real(istic) environment together with other applications.

Part of build process

Syntax checks

Scripts

Config Files

Data Files

Unit tests for functions/libs

Run program with test data

Check result

Check program behaviour with wrong/broken test data

Also run on Developer desktop

Quick feedback (~ seconds)

Install on test server

Run tests from outside

HTTP calls

Send emails

Try to login

Run tests from inside

Remote Exec (rsh, ssh …)http://go.schapiro.org/rshpitfall

Service Start, Stop & Status

Modify server to create good & bad test scenarios

Reboot

Unit Tests System Tests

UnitTests

%prep%setup -q

%installinstall … %{buildroot}/…install … %{buildroot}/…

%files%defattr(-,root,root,-)/...

BuildRequires: sudo%buildset -evisudo -c -f sudoers %installinstall -m 0440 sudoers -D \ %{buildroot}/etc/suoders.d/%{name} %files%defattr(-,root,root,-)/etc/suoders.d/%{name}

BuildRequires: PyYAML, pylint%buildset -e# syntax checksbash -n my_script.sh# Should be valid python codepylint -E yum-repo-propagate# should be valid YAML filepython -c "↩import yaml ↩yaml.safe_load(open('config.yaml'))↩"...

SystemTests

http://impreza-gt-club.ch/V2.0/Tests/WRX08/Koch2.jpg

BuildAuto-

mationSourceCode(SVN)

Monitor

Changes

Build Server

Run BuildJob

● Check out source● Run Unit Tests● Create RPM●

Sourc

eCod

eDEVYUMRepo

UploadPROYUMRepo

PropagateRPM

Test Server

Deploy

and

Run

Test

Job

yum

Prod Server

Deploy

to PROD

yum

BuildAuto-

mationSourceCode(SVN)

Monitor

Changes

Build Server

Run BuildJob

● Check out source● Run Unit Tests● Create RPM●

Sou

rce

Cod

eDEVYUMRepo

UploadPROYUMRepo

PropagateRPM

Test Server

Deploy

and

Run

Test Job

yum

Prod Server

Deploy

to PROD

yum

SAN mount service

Test via rsh

Mock SAN deviceswith losetup

service start, stop mounts/umounts

Error handling

HTTP Proxy Configuration

X-Forwarded-For header spoofs source

Check result for ERR_ACCESS_DENIED

Run tests for all function groups!

Test Server

GET http://external.com/X-Forwarded-For: 10.11.12.01

502 Bad Gateway✔GET http://external.com/X-Forwarded-For: 10.34.56.01

403 Forbidden

ERR_ACCESS_DENIED by proxy server

✘✘

VM Provisioning & Kickstart Installation

Test via HTTP API

Create broken VMs and check error reports

Create valid VM and install Linux OS

Scrape VM screen via OCR

http://github.com/Immobilienscout24/lab-manager-light

34 35 36

2 3

53

87 88 89 90 91 92 93 94 95 96 97

TIME

Continous Live Deployment

Deploy every application when it is ready.Automate the delivery chain from source till production.

Low Risk – Lots of Fun

http://go.schapiro.org/slides

Kontakt:Immobilien Scout GmbHAndreasstraße 1010243 Berlin

Fon: +49 30 243 01-1229 Email: schlomo.schapiro@immobilienscout24.deURL: www.immobilienscout24.de

Thank you very much!Please contact me for further questions and discussions.