Ironfan: Principle-Driven Scalable Ops Tools

Ironfan: Principle-DrivenScalable Ops Tools

http://github.com/infochimps-labs/ironfan

Ironfan

• Open-source core (Apache) with enterprise features

• http://github.com/infochimps-labs/ironfan

• Based on Chef (http://www.opscode.com/chef)

Principles

• Progressive - match what you’d whiteboard

• Declaratory - show all (and only) what’s interesting

• Coherent - full state is versioned as a whole

• Decoupled - minimize shared knowledge

• Helpful - common things are simple

• Inevitable - declarations imply actions

Ironfan:your system diagram

made real

Declaratory:simple stories, simply told

• HBase Master• HBase RegionServers

Simple, right?

HBase Cluster

• one-ish:• HBase master• HBase secondary master• Hadoop Namenode• Hadoop 2NN• Hadoop Jobtracker• Zookeeper Quorum x3

• many:• HBase RegionServer• HBase Stargate• HBase Thrift• Hadoop Datanode• Hadoop Tasktracker

HBase Cluster

Powerful Black Boxes

Beautiful Glue

Decouple

• HBase

• ElasticSearch

• Cassandra

• TokyoTyrant

• SimpleDB

• MongoDB

• MySQL

• Redis

• sqlite

• whisper (graphite)

• file system

• S3

Data Stores in Production

• Emacs

• Keynote

• Preview

• Chrome

• ruby (pry)

• Skitch

• finder

• flickr.com

• google image search

• ssh

Programs Used for This Talk

A Database is just a Bucket

Service Discovery:game recognize game

gibbon-nnjt-0

gibbon-worker-0

gibbon-hadoop-namenodegibbon-hadoop-jobtracker

¿dónde está mi namenode?

gibbon-nnjt-0

ack ack ack

gibbon-nnjt-0

gibbon-worker-0

¿dónde está mi namenode?

gibbon-nnjt-0

¿dónde está mi jobtracker?

gibbon-nnjt-0

gibbon-hadoop-namenodegibbon-hadoop-jobtracker

ack ack ack

gibbon-nn-0

gibbon-worker-0

¿dónde está mi namenode?

gibbon-nn-0

¿dónde está mi jobtracker?

gibbon-jt-0

ack ack ack

gibbon-jt-0

gibbon-hadoop-namenode

gibbon-hadoop-jobtracker

declaratory + decoupled =Recapitulation

ex: DB Migrationsthe Ironfan Way

• HBase Master• HBase RegionServers

alice

alice

bob

alice

bob

bob

Aspects:Cross-cutting Concerns

Amenities

• Logs

• Ports

• Daemons

• Dashboards

• Libs / Executables

Amenities:It’s good to be king

Amenities

• Logs: rotation, archiving

• Ports: remote monitoring (on and off)

• Daemons: process monitoring

• Dashboards: Dashboard dashboard

• Libs / Executables: Decoupled imports

• Versions: Shims, Discovery

Inevitability:born this way

Ironfan:

Chef:

10 lines

10 lines, 5 flaws

can you spot all 5?

hard coded

fails on bsd

not DRY

SECURITY HOLEdaemon-writeable

conf dir

10 lines, 5 flaws

‘0755’ not 755

Ironfan

Inevitabilitymust do this:

to get a log dir

gibbon-hadoop-namenode

so when you announce,

AWESOMENESS IS INEVITABLE

Contract:word is bond yo

Resource Sharing:can’t we all just get along?

• Region Server (dominate)• TaskTracker + Children (wussy)• DataNode (moderate)

• DataNode (moderate)• TaskTracker + Children (dominate)

HBase

Hadoop

Rails Apps > 5k LOCconsidered Harmful

one big app many apps tax

Resource Sharing:can’t we all just get along?

Monolithic Software

means Meetings

Meetings

are Death

Decentralize. Decouple.

Fast

Resilient

Simple

It’s not about Performance

We’re Hiring

[email protected]

github.com/ infochimps-labs

...

• System Diagram made real• Decoupled Components connect• Components Announce their capabilities• Announcements enable Service Discovery• Components announce cross-cutting Aspects• Aspects enable zero-conf Amenities• Announcement as Contract• Contracts enable zero-conf specification testing• Specs + monitoring enable zero-conf integration

testing• Systems Bind to provisioned resources• Resource Sharing