PHP Containers at Scale: 5K Containers per Server

PANTHEON.IO

By David Strauss, Pantheon

PANTHEON.IO

Websites are getting more numerous...

PANTHEON.IO

...and more personal

PANTHEON.IO

Dev Workflow Multiplier

The 1990s● Flat HTML files

● Bugs affect single pages

● Pages edited on desktop and uploaded to server

● No mobile device use

The 2000s● Dynamic sites on basic

stacks

● Dev and QA on desktop

● Bugs can affect entire site — but can be troubleshooted with WAMP/MAMP

● Limited mobile device use, may not need to test

Today● Advanced stacks

● Dev on cloud or desktop

● QA on cloud

● Bugs can affect entire site — requiring the full production stack to troubleshoot

● Social and other integrations can’t run from desktop

● Must load pages from mobile devices to test

PANTHEON.IO

PANTHEON.IO

Traditional hosting architecture

PANTHEON.IO

●

●

●

●

●

●

The Existing Choice for Small Websites

PANTHEON.IO

●

●

●

●

●

●

●

The Existing Choice for Medium Websites

PANTHEON.IO

●

●

●

●

●

●

●

○

○

○

The Existing Choice for Big Websites

PANTHEON.IO

Scaling Hosting Architecture

PANTHEON.IO

●○○○

●○○○○

●○○

About Me

PANTHEON.IO

PANTHEON.IO

●

●

●

●

●

●

●

●

○

A Choice for All Websites?

PANTHEON.IO

Mo Servers, Mo Problems

With Thanks to Nick Stielau’s…

PANTHEON.IO

The Goals of Computing

1. Making it Work2. Making it Efficient

○ Running the software○ Developer time

3. There is no #3

PANTHEON.IO

Data centers took 2% of US power in 2012.

“Power, Pollution and the Internet,”New York Times, 2012

PANTHEON.IO

We’re not using that efficiently.

“Host server CPU utilization in Amazon EC2 cloud,” Huan Liu's Blog, 2012

7.3% Average

PANTHEON.IO

I’d like to sell you a time-share.

PANTHEON.IO

A Brief History of Timesharing

● 1950s Batch processing

● 1970s Terminals and VMs on mainframes

● 1980s Client/server

● 1990s Thin GUI clients to servers

● 2000s Web clients connect to servers

● 2008s Web/mobile clients connect to cloud VMs

PANTHEON.IO

Why People LikeVirtual Machines

PANTHEON.IO

Great About VMs: Consolidation

PANTHEON.IO

“Skeuomorphs are stories of utility frozen in time. A new kind of affordance—a cultural affordance—that provides the context we need to understand the possibilities for action. They don’t work because they coddle or educate the user—digital wood grain shelves and page-flips didn’t teach people how to read ebooks—they work because they leverage a user’s past experience and apply that understanding to something new.”

John Payne, “Does Skeuomorphic Design Matter?”

Great About VMs: Familiarity

PANTHEON.IO

Great About VMs: Slicing

PANTHEON.IO

Great About VMs: Portable Unit

Migration, failover, high availability,consistent hypervisors, consistent images

PANTHEON.IO

Great About VMs: Automation

PANTHEON.IO

Great About VMs:Maturity and Efficiency

99% Efficient at Running the OS and

Application

PANTHEON.IO

Containers arethe next step.

PANTHEON.IOExactly! Why stop at virtualization?

PANTHEON.IO

Containers Revolutionized Shipping Costs

PANTHEON.IO

An Amended History: Containers

● 1986 AIX 6.1 with Workload Partitions● 2000 FreeBSD 4.0 with Jails● 2005 Solaris 10 with Zones● 2007 Google lands cgroups in the Linux kernel● 2010 systemd● 2013 Docker and CoreOS● 2014 LXC 1.0 and Kubernetes● 2015 The App Container Spec, public container clouds

PANTHEON.IO

Containersvs. Virtual Machines

Let’s Contrast

PANTHEON.IO

“Skeuomorphs are material metaphors instantiated through our technologies in artifacts. They provide us with familiar cues to an unfamiliar domain, sometimes lighting our paths, sometimes leading us astray.”

Nicholas Gessler, “Skeuomorphs and Cultural Algorithms”

Familiar Doesn’t Make It Good

PANTHEON.IO

Tiny Container Slices are Useful

Rackspace retired 256MB VMs because you couldn’t run an OS and a useful workload in that space. Containers only need the resources for an application.

PANTHEON.IO

Efficiency in a New Category

Trains and planes are efficient, but not compared to making travel unnecessary.

Containers don’t need to run an operating system.

PANTHEON.IO

Containers are Portable— and Lighter

Migration ofApplicationvs. Full OS

PANTHEON.IO

Containers offer faster automation

PANTHEON.IO

Time to container

PANTHEON.IO

Containers at PantheonIn the Real World

PANTHEON.IO

Density at Pantheon

30GB servers/ 150 containers= 205MB each

PANTHEON.IO

Container Provisioning

Mostly< 20 secondsfully configured

Some have been on

bare metal!

PANTHEON.IO

Density: featherweight design

●○○○○

●○○

●○○○

PANTHEON.IO

Density: scheduling● Run servers at 90% utilization● Rebalance containers all the time

with the “migration dragon”

● Scoring algorithm focuses on avoiding resource saturation (CPU, disk I/O, network I/O) but prefers highly available (HA) distributions of customer resources

● Average container age is 50 days, allowed SSD rollout in a couple months with no customer involvement

● Modern Linux kernel capabilities for fair contention

PANTHEON.IO

Scaling the stack in softwareLet’s look real-time

PANTHEON.IO

The Bones of Containers

PANTHEON.IO

Containersare based on the

CGroups and Namespacesfunctionality on the Linux kernel

PANTHEON.IO

cgroups is a hierarchy of processes

All processes

Development processes

PHP-FPM Drush

Production processes

Drush Rsync

75% 25%

PANTHEON.IO

All processes

Processes for people I don’t like

PHP-FPM Drush

Processes forpeople I like

Drush Rsync

2%98%

cgroups is a hierarchy of processes

PANTHEON.IO

cgroups submodules aka Controllers

● memory: Memory controller● cpuset: CPU set controller● cpuacct: CPU accounting controller● cpu: CPU scheduler controller● devices: Devices controller● blkio: I/O controller for block devices● net_cls: Network Class controller● ...

PANTHEON.IO

Kernel Interaction: /proc, /sys/fs

PANTHEON.IO

Kernel Interaction: /proc, /sys/fs

PANTHEON.IO

…

Managing cgroups: manually

PANTHEON.IO

Managing cgroups: manually

PANTHEON.IO

Creating cgroups: libcgroups

PANTHEON.IO

memory.limit_in_bytes in action

PANTHEON.IO

cpu.shares in action

PID USER PR NI VIRT RES SHR S %CPU 9693 root 20 0 107908 624 532 R 60.08 9692 root 20 0 107908 624 532 R 6.307

cp

u.s

ha

re

s =

100

cpu.shares = 10

PANTHEON.IO

● Mount● IPC● PID● User● UTS● Network

Kernel Namespaces

PANTHEON.IO

“Before one can share, one must first unshare” —Share Bear

PANTHEON.IO

Container Frameworks

PANTHEON.IO

LXC

● The liblxc library● Several language bindings (python3, lua, ruby and Go)● A set of standard tools to control the containers● Container templates

PANTHEON.IO

Let Me Contain That For You (lmctfy)

● Created by Google,but public project sort of deprecated for Docker

● Open Source(ish)● Every process at Google runs within lmctfy● Supports nested containers

PANTHEON.IO

● From systemd project● Foundation for CoreOS’s Rocket (rkt) runtime● Ships with current Arch, Fedora, Suse, RHEL, Debian,

Ubuntu, and CentOS● Closest to what Pantheon uses

systemd-nspawn

PANTHEON.IO

systemd-nspawn

PANTHEON.IO

Rocket

● Built on systemd-nspawn● Uses App Container Spec for images● Maintained by CoreOS project

PANTHEON.IO

Docker

“In its early age, the dotCloud platform used plain LXC (Linux Containers)....The platform evolved, bearing less and less similarity with usual Linux Containers.”1

[1] http://blog.dotcloud.com/under-the-hood-linux-kernels-on-dotcloud-part[2] https://prague2013.drupal.org/session/automate-drupal-deployments-linux-containers-docker-and-vagrant

PANTHEON.IO

Containerization Spectrum

Docker Rocket nspawn lxc lmctfy

PANTHEON.IO

And once you get containers….

http://coreos.com/blog/cluster-level-container-orchestration/

PANTHEON.IO

Thanks!

Questions?Here or @davidstrauss

?

PANTHEON.IO

Photo Attributions

● Containers● Virtualization Diagram● Sliced Pie● Train● Robots● Videoconferencing● Timesharing● Containers graph● Transportation efficiency graph