Test Driven Infrastructure www.immobilienscout24.de Berlin | 09.04.2014 | Schlomo Schapiro Systemarchitekt, Open Source Evangelist License: http://creativecommons.org/licenses/by-nc-nd/3.0/ DevOps Risk Mitigation @schlomoschapiro [email protected]
Aug 23, 2014
Test Driven Infrastructurewww.immobilienscout24.de
Berlin | 09.04.2014 | Schlomo SchapiroSystemarchitekt, Open Source Evangelist
License: http://creativecommons.org/licenses/by-nc-nd/3.0/
DevOps Risk Mitigation
Slide 2 | Test Driven Infrastructure | Schlomo Schapiro
www.ImmobilienScout24.de
>2 billion PI per month
2 data center with ~1600 VM
~2.5 million outgoing email/day
total of ~600 employees
~30 crossfunctional IT teams
~160 in IT
15 years in business
part of Deutsche Telekom
Slide 3 | Test Driven Infrastructure | Schlomo Schapiro
ExpensiveFix
CheapFix
Costs Of Finding Bugs In Production
Slide 4 | Test Driven Infrastructure | Schlomo Schapiro
PlanDesignBudget
Develop Test
RU
NDEV
PlanDesignBudget
Develop Test RUN OPSTIME
Slide 5 | Test Driven Infrastructure | Schlomo Schapiro
PlanDesignBudget
Develop Test
PlanDesignBudget
Develop Test RUN
Proxyconfig
Databaseborked
„Buy Now“button broken
MTA dropall mail
Load BalancerConfiguration
Everything costsonly 0 €
Login possiblewithout password
NFS notavailable
DB Replicationstopped
No adsshown
BrokenCSS / JS
Tomcat won'tstart/stop
Service usernot defined
sudoersinvalid
Slide 6 | Test Driven Infrastructure | Schlomo Schapiro
DevOps: Respect & Learning
DEV learn from OPS to think about:
● Resources (CPU, RAM, Disk)● Services (Start, Stop, Status)● Dependencies (Start DB before App)● Logfiles (Rotate, Remove)● Disk Space● Monitoring and Alarming● ...
OPS learn from DEV to think about:
● Incremental Improvement● Infrastructure as Code● Version Control System● Coding (OO, Functions, Libraries …)● Code Quality● Unit & Integration Tests● Test Automation● ...
Slide 7 | Test Driven Infrastructure | Schlomo Schapiro
Untested=
Broken
Slide 8 | Test Driven Infrastructure | Schlomo Schapiro
Unit TestsTest the smallest possible components in an artifical environment.
System TestsTest the entire application in a real(istic) environment together with other applications.
Slide 9 | Test Driven Infrastructure | Schlomo Schapiro
Part of build process
Syntax checks
Scripts
Config Files
Data Files
Unit tests for functions/libs
Run program with test data
Check result
Check program behaviour with wrong/broken test data
Also run on Developer desktop
Quick feedback (~ seconds)
Install on test server
Run tests from outside
HTTP calls
Send emails
Try to login
Run tests from inside
Remote Exec (rsh, ssh …)http://go.schapiro.org/rshpitfall
Service Start, Stop & Status
Modify server to create good & bad test scenarios
Reboot
Unit Tests System Tests
Slide 10 | Test Driven Infrastructure | Schlomo Schapiro
UnitTests
Slide 11 | Test Driven Infrastructure | Schlomo Schapiro
%prep%setup -q
%installinstall … %{buildroot}/…install … %{buildroot}/…
%files%defattr(-,root,root,-)/...
Slide 12 | Test Driven Infrastructure | Schlomo Schapiro
BuildRequires: sudo%buildset -evisudo -c -f sudoers %installinstall -m 0440 sudoers -D \ %{buildroot}/etc/suoders.d/%{name} %files%defattr(-,root,root,-)/etc/suoders.d/%{name}
Slide 13 | Test Driven Infrastructure | Schlomo Schapiro
BuildRequires: PyYAML, pylint%buildset -e# syntax checksbash -n my_script.sh# Should be valid python codepylint -E yum-repo-propagate# should be valid YAML filepython -c "↩import yaml ↩yaml.safe_load(open('config.yaml'))↩"...
Slide 14 | Test Driven Infrastructure | Schlomo Schapiro
BuildRequires: python-unittest2, python-teamcity-messages, ...
%buildset +x +e # for teamcity
exec 1>&2 # join stdout to stderr to synchronize between them for teamcity output
FAILED=0TEAMCITY_PROJECT_NAME=1 python unit_tests.py || let FAILED++bash -n oldhomeinfo.sh || let FAILED++
(( FAILED == 0 )) || exit 1
Slide 15 | Test Driven Infrastructure | Schlomo Schapiro
More Examples for Unit Tests
Patching nsswitch.conf and PAM files
Syntax checking HTTPD, DNS, DHCP ... configuration files
Checking SSH Server & Client configurationshttp://go.schapiro.org/sshconfigtest
...
Slide 16 | Test Driven Infrastructure | Schlomo Schapiro
SystemTests
http://impreza-gt-club.ch/V2.0/Tests/WRX08/Koch2.jpg
Slide 17 | Test Driven Infrastructure | Schlomo Schapiro
BuildAuto-
mationSourceCode(SVN)
Monitor
Changes
Build Server
Run BuildJob
● Check out source● Run Unit Tests● Create RPM●
Sourc
eCod
eDEVYUMRepo
UploadPROYUMRepo
PropagateRPM
Test Server
Deploy
and
Run
Test
Job
yum
Prod Server
Deploy
to PROD
yum
Slide 18 | Test Driven Infrastructure | Schlomo Schapiro
BuildAuto-
mationSourceCode(SVN)
Monitor
Changes
Build Server
Run BuildJob
● Check out source● Run Unit Tests● Create RPM●
Sou
rce
Cod
eDEVYUMRepo
UploadPROYUMRepo
PropagateRPM
Test Server
Deploy
and
Run
Test Job
yum
Prod Server
Deploy
to PROD
yum
Slide 19 | Test Driven Infrastructure | Schlomo Schapiro
SAN mount service
Test via rsh
Mock SAN deviceswith losetup
service start, stop mounts/umounts
Error handling
Slide 20 | Test Driven Infrastructure | Schlomo Schapiro
HTTP Proxy Configuration
X-Forwarded-For header spoofs source
Check result for ERR_ACCESS_DENIED
Run tests for all function groups!
Test Server
GET http://external.com/X-Forwarded-For: 10.11.12.01
502 Bad Gateway✔GET http://external.com/X-Forwarded-For: 10.34.56.01
403 Forbidden
ERR_ACCESS_DENIED by proxy server
✘✘
Slide 21 | Test Driven Infrastructure | Schlomo Schapiro
Subversion Server Configuration
Tests Config RPM Makerhttps://github.com/yadt/yadt-config-rpm-maker
Tests 2 servers: Master & Slave
Replication
Failure and Recovery
Backup and Restore
MasterSVN svnsync
Backup
SlaveSVN
Backup
Slide 22 | Test Driven Infrastructure | Schlomo Schapiro
PAM & NSS Configuration
PAM & nsswitch.conf patching
Mock setup via rsh
Mock AD groups and users with nss_db
Check service status
Test login via ssh
Thorben Wengert / pixelio.de
Slide 23 | Test Driven Infrastructure | Schlomo Schapiro
VM Provisioning & Kickstart Installation
Test via HTTP API
Create broken VMs and check error reports
Create valid VM and install Linux OS
Scrape VM screen via OCR
http://github.com/Immobilienscout24/lab-manager-light
Slide 24 | Test Driven Infrastructure | Schlomo Schapiro
34 35 36
2 3
53
87 88 89 90 91 92 93 94 95 96 97
TIME
Continous Live Deployment
Deploy every application when it is ready.Automate the delivery chain from source till production.
Slide 25 | Test Driven Infrastructure | Schlomo SchapiroLow Risk – Lots of Fun
http://go.schapiro.org/slides
Slide 26 | Test Driven Infrastructure | Schlomo Schapiro
Kontakt:Immobilien Scout GmbHAndreasstraße 1010243 Berlin
Fon: +49 30 243 01-1229 Email: [email protected]: www.immobilienscout24.de
Thank you very much!Please contact me for further questions and discussions.