OSDC 2013 - Configuration Management and Linux Packages

Post on 08-May-2015

1101 Views

Category:

Technology

3 Downloads

Preview:

Click to see full reader

DESCRIPTION

What is the connection between configuration management (CM) and Linux packages? Is there a connection? Why do the Linux packages get in the way of CM all the time? Why should I care about this topic? In the modern world everybody has some tooling for CM, be it one of chef/puppet/cfengine or something else. All CM tools basically serve the same purpose: Automate everything between a blank new machine and a running system that is ready for production. That includes installing some packages and changing configuration files. Not really surprising, Linux packages actually serve a very similar purpose, though with a completely different objective. Packages also install other packages and also bring some configuration files. That is the reason why we should talk about CM and packages: Two different tool sets doing the same job with different objectives. That is also the reason why CM was invented and why distro packages tend to get in the way of what you are trying to achieve with CM. Look at sysadvent.blogspot.de/2012/12/day-10-packages-doing-too-much.html for a good example of this conflict of interests. When faced with the challenge of finding a new deployment and configuration management solution, we decided to try something different and radically new: Use Linux packages for configuration management! In this talk I will explain the rationale behind that decision and the design choices that allowed us to do this. We believe that this is a way out of the CM-packages conflict because we actually use the same tooling for everything: OS deployment, software rollout and configuration rollout. In our world there cannot be a conflict between distro packages and configuration because we actively design our packages and configuration to work seamlessly together with the distro packages. If the time permits I will be happy to share a live demo of how we work with configuration packages. The result of our work is published under the GPL at yadt-project.org, our tooling to create config RPMs from configuration data snippets kept in SVN can be found in https://github.com/yadt/yadt-config-rpm-maker. Take this as an example, the important part is package-based configuration rollout. It doesn't matter how you actually create those packages, it is only important that they are dumb and contain no install-time scripts.

Transcript

Configuration Managementand Linux Packages

www.immobilienscout24.de

OSDC Nürnberg | 17.04.2013 | Schlomo Schapiro | @schlomoschapiroSystemarchitekt, Open Source Evangelist

License: http://creativecommons.org/licenses/by-nc-nd/3.0/

Slide 2 | Config Management & Linux Packages | @schlomoschapiro

www.ImmobilienScout24.de

>2 billion PI per month

2 data center with ~1400 VM

total of ~600 employees

~30 crossfunctional IT teams

~160 in IT

15 years in business

part of Deutsche Telekom

Slide 3 | Config Management & Linux Packages | @schlomoschapiro

Why am I standing here?

My Puppet and Chef to only half the job blog posting (2012-07)

Packages Doing Too Much? blog post on SysAdvent (2012-12)

Talking to lots of people at lots of conferences

One Tool To Rule Them All

Slide 4 | Config Management & Linux Packages | @schlomoschapiro

BusinessDecision

To goLive

All Humans are on the Same Side

BuildConfigDeployTestSystems-Management

Automation

Scale OutData Centers

through packages

Slide 5 | Config Management & Linux Packages | @schlomoschapiro

DevOps

Slide 6 | Config Management & Linux Packages | @schlomoschapiro

Never change a running system

Run the changing system

Continous Delivery

Deploy When Ready

You Build It – You Run It!

Fail Fast – Fail Early

Run With The Pack – The Pack Will Protect You

Slide 7 | Config Management & Linux Packages | @schlomoschapiro

PROInfrastructureBuild

ApplicationBuild

ConfigurationBuild

QAYUM

Repos

Interface

DEV

Slide 8 | Config Management & Linux Packages | @schlomoschapiro

“Any relevant file should be either

deployed via a package

or

completely managed by an

application that is thus deployed.”

Slide 9 | Config Management & Linux Packages | @schlomoschapiro

„Every package must be verifyable –

stay away from package scripts.“

“Reducing the config package

reduces the deployment risk.”

Slide 10 | Config Management & Linux Packages | @schlomoschapiro

ConfigSVN

post-commit → YUM repos

Infrastructureas

Code

Slide 11 | Config Management & Linux Packages | @schlomoschapiro

TSTWEB05Location & Environment

Function Group

Instance

Configuration over Convention

Slide 12 | Config Management & Linux Packages | @schlomoschapiro

Big Picture – Static Structure

ConfigSVN

YUMRepository

IS24 softwareAutomated RPM

creation

config├── host│ └── tstweb02│├── loc│ └── tst│ ├── VARIABLES│ │ ├── RPM_REQUIRES│ │ ├── RPM_PROVIDES│ │ ├── DB_HOST│ │ └── DB_USER│ └── etc│ └── is24│ ├── web.properties│ └── db.properties├── typ│ └── web│├── loctyp│ └── tstweb│ └── etc│ └── is24│ └── web.properties└── all ├── VARIABLES │ └── SYSLOG_HOST └── etc └── is24 └── system.properties

is24-config-tstweb01-1.0-$rev.rpm:/etc/is24/system.properties/etc/is24/db.properties/etc/is24/web.properties

[root@tstweb01 ~]# yum update

svn ci

db.host=@@@DB_HOST@@@db.user=@@@DB_USER@@@db.port=3306

Post-commit hook createsis24-config-$hostname RPM● svn export● patch VARIABLES● fill in metadata

svn co

loghost=log.domain.com

Slide 13 | Config Management & Linux Packages | @schlomoschapiro

Static Structure – Pros and Cons

Variables follow samestructure as config data

Only one tool (SVN) needed

GUI talks only to SVN

Versioning and changetracking for variables

Variables and config fileschange together (atomic)

easy rollback→

Is this simple overlaying structure sufficient?

Simplify the world so that it is good enough!

ConfigSVN

YUMRepository

IS24 softwareAutomated RPM

creation

config├── host│ └── tstweb02│├── loc│ └── tst│ ├── VARIABLES│ │ ├── RPM_REQUIRES│ │ ├── RPM_PROVIDES│ │ ├── DB_HOST│ │ └── DB_USER│ └── etc│ └── is24│ ├── web.properties│ └── db.properties├── typ│ └── web│├── loctyp│ └── tstweb│ └── etc│ └── is24│ └── web.properties└── all ├── VARIABLES │ └── SYSLOG_HOST └── etc └── is24 └── system.properties

is24-config-tstweb01-1.0-$rev.rpm:/etc/is24/system.properties/etc/is24/db.properties/etc/is24/web.properties

[root@tstweb01 ~]# yum update

svn ci

db.host=@@@DB_HOST@@@db.user=@@@DB_USER@@@db.port=3306

Post-commit hook createsis24-config-$hostname RPM● svn export● patch VARIABLES● fill in metadata

svn co

loghost=log.domain.com

KISS!

Slide 14 | Config Management & Linux Packages | @schlomoschapiro

Example: Apache HTTPD

Design Goals:

Use and extend upstream httpd RPM

Configure MPM and service user per application

IS24 standard configuration everywhere

httpd RPM

/etc/httpd/conf/httpd.conf

/etc/sysconfig/httpd

Slide 15 | Config Management & Linux Packages | @schlomoschapiro

Example: is24-httpd RPM

if ! echo '# IS24 HTTPD conf framework. Read IS24_README!# This file is managed by %{name}­%{version}.%{release}# Put your stuff in /etc/conf/*/is24*.conf files!ServerRoot "/etc/httpd"Include conf/basic/is24*.confInclude conf/main/is24*.confInclude conf/other/is24*.conf' >/etc/httpd/conf/httpd.conf ; then    logger ­p user.err ­s ­t %name ­­ "ERROR: …"fi

Requires: httpdRequires(Pre): httpd%post

Slide 16 | Config Management & Linux Packages | @schlomoschapiro

Example: is24-httpd RPM

if ! echo '# HTTPD options can be configured in# additional /etc/httpd/conf/env/*.sh files# This file is managed by %{name}­%{version}.%{release}# Please add extra options to the OPTIONS Bash Arrayshopt ­s nullglobHTTPD=/usr/sbin/httpd.workerfor f in /etc/httpd/conf/env/*.sh ; do    source $fdoneOPTIONS="${OPTIONS[*]}" # flatten array' >/etc/sysconfig/httpd ; then    logger ­p user.err ­s ­t %name ­­ "ERROR: ..."fi

...

%post (continued)

Slide 17 | Config Management & Linux Packages | @schlomoschapiro

/etc/profile.d/java.sh

Requires: is24-jdk-6

Example: How To Upgrade Java Packages

Requires: is24-jdk-7

is24-jdk-7 is24-jdk-6Req

java-1.6.0-sun-develjava-1.7.0-oracle-devel

Req Req

Slide 18 | Config Management & Linux Packages | @schlomoschapiro

Separation of Concerns

KeepIt

Simple&

Stupid

The Tool Is You!

UseWhat

IsAlreadyThere

Slide 19 | Config Management & Linux Packages | @schlomoschapiro

http://yadt-project.org and http://github.com/YADT

http://github.com/ImmobilienScout24

http://bit.ly/is24techjobs

Slide 20 | Config Management & Linux Packages | @schlomoschapiro

Kontakt:Immobilien Scout GmbHAndreasstraße 1010243 Berlin

Fon: +49 30 243 01-1229 Email: schlomo.schapiro@immobilienscout24.deURL: www.immobilienscout24.de

Thank you very much!Please contact me for further questions and discussions.

Slide 21 | Practical DevOps | @schlomoschapiro

Photo URLs

Rowboot: http://www.photoreview.com.au/features/profiles/work-hard-get-lucky.aspxHolding hands: http://www.thechefalliance.com/Top-Chef-ServicesRobots: http://www.t-tek.com/announcements/conventional-palletizer-or-robot

Everything else is from OpenClipArt.org or custom made.

top related