Mo Servers, Mo Problems Mo Servers, Mo Problems
Jan 27, 2015
Mo Servers, Mo ProblemsMo Servers, Mo Problems
Really, containers vs. VMs
What is a problem?
How using containers instead of VMs can help you increase uptime and decrease problems requiring human intervention and decision-making
We are living in the future
We will get our handsdirty
Chapter 1
In which you begin to believe me
when I tell you, “Mo’ Servers Mo
Problems”
Let’s bundle optimized hosting along with amazing workflow tools, team management, and runtime analytics, stick it on a VM, and charge $$$!
Bro!
We did it!300 Clients300 Virtual Machines300 Problems
● We can now support Freemium!● Each site has 3+ environments● Containers for PHP-FPM, Nginx, Mount processes, MySQL DB and Redis● To 300 30GB VMs, 100,000 LAMP stacks, ~750,000 containers (TODO count?)
Problems = Infrastructrue*Sites
PITA Coefficient (O)
PITA Coefficient (O)
ODrupal Developer ~= 0.27
OPage View ~= 2.5e-7
OContainer ~= 0.005
ODrupal User ~= 0.025
OVM/Server ~= 8.3
With two Containers on one VMRisk = ½Likelihood * 2xConsequences
With two, single-tenant VMsRisk = 2xLikelihood * ½Consequences
Risk = Likelihood * Consequence
Self-healing Problems
Problems Requiring Basic Manual Intervention
Problems Requiring Decisions
Problems Requiring Coding
Problems Requiring Hard Decisions
Easy Hard
Humans Decisions Compound
5 servers means 10 (network) problems6 servers means 15 (network) problems
1 more server bought you 5 problems
http://aphyr.com/posts/288-the-network-is-reliable
O(N2) Network Failure Paths
If you want fewer Problems
● Increase Mean Time Between Failure○ You could get more reliable things….where?○ You can get fewer things!
● Decrease Mean Time To Resolution○ You can speed-up detection, insight, resolution○ You can reduce reliance on human decisions
“Chief Chirpa Sucks”
[nick@endpoint9a71a1ef ]$
vs.
[nick@ChiefChirpa ~]$
Chapter 2
In which we use English to describe
WTF containers are, and why people
might want to use them.
This is what our marketers say we built
Resource-constrained, system-isolated, metered processes.
Containers are simply....
Time to container$: systemd-nspawn -D /srv/debian/ date
Spawning namespace container on /srv/debian.
Init process in the container running as PID 9159.
Tue Jun 3 17:32:14 UTC 2014
real 0m0.007suser 0m0.001s
real 0m0.007s
Even if you just run one server...
OS Upgrades SuckCloud VMs get ‘weird’Container migration FTW.
End of lifeis a
way of life!
OS upgrade dropsavg server life
Container Migration to MariaDB
One-click migration to convert thousands of MySQL containers to MariaDB
Chapter 3
In which we plumb the depths of the
/proc filesystem, in search of clues
about CGroups and namespaces
Containersare based on the
CGroups and Namespacesfunctionality on the Linux kernel
cgroups is merely a hierarchy ofprocesses All processes
Development processes
PHP-FPM Drush
Production processes
Drush Rsync
75% 25%
cgroups is merely a hierarchy ofprocesses All processes
Processes for people I don’t like
PHP-FPM Drush
Processes forpeople I like
Drush Rsync
2%98%
cgroups submodules aka Controllers
● memory: Memory controller● cpuset: CPU set controller● cpuacct: CPU accounting controller● cpu: CPU scheduler controller● devices: Devices controller● blkio: I/O controller for block devices● net_cls: Network Class controller● ...
Kernel Interaction: /proc, /sys/fs # Inspect ip forwarding setting
$: cat /proc/sys/net/ipv4/ip_forward
# Turn ip forwarding off/on
$: echo "0" > /proc/sys/net/ipv4/ip_forward
$: echo "1" > /proc/sys/net/ipv4/ip_forward
# Examine file descriptors used by nginx..
$: ls -l /proc/$NGINX_PID/fd/
lrwx------ 1 root Jun 3 13:48 0 -> /dev/null
lrwx------ 1 root Jun 3 13:48 10 -> socket:[64376]
l-wx------ 1 root Jun 3 13:48 2 -> /var/log/nginx-access.log
# Nuke logs
$: rm -rf /var/log/nginx-access.log
# Read log (even after you rm -rf’d it!)
$: tail /proc/$NGINX_PID/fd/2
62.211.78.166 - - [05/May/2014:10:00:54 +0000] "GET /vtiger.php
Kernel Interaction: /proc, /sys/fs
# Create a Control Group named “AA”
$: mkdir /sys/fs/cgroup/memory/AA
# New directory magically contains...
$: ls /sys/fs/cgroup/memory/AA
cgroup.clone_children
memory.kmem.usage_in_bytes memory.limit_in_bytes
cgroup.procs memory.max_usage_in_bytes … ...
Managing cgroups: manually
# Limit AA’s memory to 100 bytes
$: echo 100 > /sys/fs/cgroup/cpu/AA/memory.limit_in_bytes
Managing cgroups: manually
Creating cgroups: libcgroups# Create a Control Group named “AA”
$: cgcreate -g cpu:AA
# Set the ‘cpu.shares’ to 100 for “AA”
$: cgset -r cpu.shares=100 AA
# Run a python script in the “AA” control group
$: cgexec -g cpu:AA python test.py
# Limit teensy’s memory to 100 bytes
$: cgcreate -g memory:teensy
$: cgset -r memory.limit_in_bytes=100 teensy
# Associate current shell’s PID with “teensy”
$: echo $$ > /sys/fs/cgroup/memory/teensy/tasks
# Any command will exhaust memory
$: ls
Killed
memory.limit_in_bytes in action
cpu.shares in action
PID USER PR NI VIRT RES SHR S %CPU 9693 root 20 0 107908 624 532 R 60.08 9692 root 20 0 107908 624 532 R 6.307
cpu.shares = 100
cpu.shares = 10
# Run script within each cgroup
$: cgexec -g cpu:AA python test.py &
$: cgexec -g cpu:BB python test.py &
$: top
● Mount● IPC● PID● User● UTS● Network
Kernel Namespaces
“Before one can share, one must first unshare” - Share Bear
# Run a shell with isolated
# network namespace:
$: unshare --net /bin/bash
Chapter 4
In which we agree that nobody (here)
wants to care about /proc, /sys/fs,
and we investigate alternatives
Container Managers
https://github.com/containers/container-rfc
LXC
● The liblxc library● Several language bindings (python3, lua,
ruby and Go)● A set of standard tools to control the
containers● Container templates
Let Me Contain That For You (lmctfy)
● Created by Google● Open Source(ish)● Every process at Google runs within lmctfy● Supports nested containers
systemd-nspawn
● From systemd project “PID EINS!”● Will ship with all Fedora, RHEL, Ubuntu1
[1] It will ship even with you on boardhttps://speakerdeck.com/joemiller/systemd-for-sysadmins-what-to-expect-from-your-new-service-overlord
# Launch Vagrant
$: vagrant ssh
# Install a base debian tree
$: debootstrap unstable /srv/debian/
# Launch a debian container
$: systemd-nspawn -D /srv/debian/
systemd-nspawn
Container Inception
# Launch a read-only debian container
$: systemd-nspawn --read-only -D /srv/debian/
systemd-nspawn
Docker“In its early age, the dotCloud platform used plain LXC (Linux Containers)....The platform evolved, bearing less and less similarity with usual Linux Containers.”1
[1] http://blog.dotcloud.com/under-the-hood-linux-kernels-on-dotcloud-part
[2] https://prague2013.drupal.org/session/automate-drupal-deployments-linux-containers-docker-and-vagrant
Check out @ricardoamaro’s Drupalcon Prague session2
Containerizeralater SpectrumDocker nspawn lxc lmctfy
And once you get containers….
http://coreos.com/blog/cluster-level-container-orchestration/
● Servers solve and create problems● Containers yield agile portability● Containers = CGroups + namespaces● Use tools to manage containers● The future is now
Pantheon, a platform for the content web, running 10s of Ks of LAMP CMS installshttps://www.getpantheon.com/customers IMAGES
Thanks!Nick [email protected]/nstielau/containerz
Image CreditsContainers: https://flic.kr/p/4o3Ria
Clouds: https://flic.kr/p/hHRdBL
Back to the Future (Lego): https://flic.kr/p/fbThy5
Dirty Hands: https://flic.kr/p/8G3aM5
Risk: https://flic.kr/p/81nfaV
Pita Equation: http://www.codecogs.com/latex/eqneditor.php
Pita Evil Eyes: http://www.clipartbest.com/cliparts/7ia/4eL/7ia4eL9iA.png
Containers http://bighugelabs.com/onblack.php?id=6764705137&size=large
CGroups http://fbcg.com/small-groups/
Pengiun Container: http://2.bp.blogspot.com/-47sakFH6uSw/UXgrhNqYF8I/AAAAAAAAHzQ/0W8zFVgR--w/s1600/lxc.png
No Logo: http://static.tumblr.com/i4bgb5d/Uzblps3wo/no-logo-1.jpg
Book sprectrum: https://flic.kr/p/k5jmja
Bottles: https://flic.kr/p/nj8jMn
Mac: https://flic.kr/p/auKEX2
Corn: https://flic.kr/p/6NVL68