Server Administration in Python with Fabric, Cuisine and Watchdog

ffunctioninc.

Fabric, Cuisine & Watchdog

Sébastien Pierre, ffunction inc.@Montréal Python, February 2011

www.ffctn.com

http://www.ffctn.com/

ffunctioninc.

How to use Python for

Server AdministrationThanks to

FabricCuisine*

& Watchdog**custom tools

ffunctioninc.

The way we useservers

has changed

ffunctioninc.

WEBSERVER

The era of dedicated servers

DATABASESERVER

EMAILSERVER

Hosted in your server room or in colocation

ffunctioninc.

WEBSERVER


DATABASESERVER

EMAILSERVER


Sysadmins typicallySSH and configure

the servers live

Sysadmins typicallySSH and configure

the servers live

ffunctioninc.

WEBSERVER


DATABASESERVER

EMAILSERVER


The servers areconservatively managed,

updates are risky

The servers areconservatively managed,

updates are risky

ffunctioninc.

SLICE 1

The era of slices/VPS

SLICE 10

Linode.com

SLICE 11SLICE 9SLICE 1SLICE 1SLICE 1SLICE 1SLICE 6

Amazon Ec2

We now have multiplesmall virtual servers

(slices/VPS)

We now have multiplesmall virtual servers

(slices/VPS)

ffunctioninc.

SLICE 1


SLICE 10

Linode.com


Amazon Ec2

Often located in differentdata-centers

Often located in differentdata-centers

ffunctioninc.

SLICE 1


SLICE 10

Linode.com


Amazon Ec2

...and sometimes withdifferent providers

...and sometimes withdifferent providers

ffunctioninc.

SLICE 1


SLICE 10

Linode.com


Amazon Ec2

DEDICATEDSERVER 1

DEDICATEDSERVER 2

IWeb.com

We even sometimesstill have physical,dedicated servers

We even sometimesstill have physical,dedicated servers

ffunctioninc.

The challenge

ORDERSERVER

ffunctioninc.

The challenge

ORDERSERVER

SETUPSERVER

ffunctioninc.

The challenge

ORDERSERVER

SETUPSERVER

Create users, groupsCustomize config filesInstall base packages

Create users, groupsCustomize config filesInstall base packages

ffunctioninc.

The challenge

ORDERSERVER

SETUPSERVER

DEPLOYAPPLICATION

ffunctioninc.

The challenge

ORDERSERVER

SETUPSERVER

DEPLOYAPPLICATION

Install app-specificpackages

deploy applicationstart services

Install app-specificpackages

deploy applicationstart services

ffunctioninc.

The challenge

ORDERSERVER

SETUPSERVER

DEPLOYAPPLICATION

MAKE THIS PROCESS AS FAST (AND SIMPLE)AS POSSIBLE

ffunctioninc.

The challenge

ffunctioninc.

The challenge

Quickly integrate yournew server in the

existing architecture

Quickly integrate yournew server in the

existing architecture

ffunctioninc.

The challenge ...and make sureit's running!

...and make sureit's running!

ffunctioninc.

Today's menu

FABRICInteract with your remote machinesas if they were local

ffunctioninc.

Today's menu

FABRIC

CUISINE

Interact with your remote machinesas if they were local

Takes care of users, group, packagesand configuration of your new machine

ffunctioninc.

Today's menu

FABRIC

CUISINE

WATCHDOG



Ensures that your servers and servicesare up and running

ffunctioninc.

Today's menu

FABRIC

CUISINE

WATCHDOG



Ensures that your servers and servicesare up and running

Made byMade by

ffunctioninc.

Part 1

Fabric - http://fabfile.org

application deployment & systems administration tasks

ffunctioninc.

Fabric is a Python library and command-line tool

for streamlining the use of SSHfor application deployment

or systems administration tasks.

ffunctioninc.

Fabric is a Python library and command-line tool

for streamlining the use of SSHfor application deployment

or systems administration tasks.

Wait... what doesthat mean ?

Wait... what doesthat mean ?

ffunctioninc.

Streamlining SSH

version = os.popen(“ssh myserver 'cat /proc/version'”).read()

version = run(“cat /proc/version”)

By hand:

Using Fabric:

ffunctioninc.

Streamlining SSH

version = os.popen(“ssh myserver 'cat /proc/version').read()

from fabric.api import *env.hosts = [“myserver”]version = run(“cat /proc/version”)

By hand:

Using Fabric:

ffunctioninc.

Streamlining SSH



By hand:

Using Fabric:

You can specify multiple hosts and runthe same commands

across them

You can specify multiple hosts and runthe same commands

across them

ffunctioninc.

Streamlining SSH



By hand:

Using Fabric:

Connections will belazily created and

pooled

Connections will belazily created and

pooled

ffunctioninc.

Streamlining SSH



By hand:

Using Fabric:

Failures ($STATUS) willbe handled just like in Make

Failures ($STATUS) willbe handled just like in Make

ffunctioninc.

Example: Installing packages

sudo(“aptitude install nginx”)

if run("dpkg -s %s | grep 'Status:' ; true" % package).find("installed") == -1:

sudo("aptitude install '%s'" % (package)

ffunctioninc.





It's easy to take actiondepending on the result

It's easy to take actiondepending on the result

ffunctioninc.





Note that we add trueso that the run() always

succeeds** there are other ways...

Note that we add trueso that the run() always

succeeds** there are other ways...

ffunctioninc.

Example: retrieving system status

disk_usage = run(“df -kP”)mem_usage = run(“cat /proc/meminfo”)cpu_usage = run(“cat /proc/stat”

print disk_usage, mem_usage, cpu_info

ffunctioninc.

Example: retrieving system status

disk_usage = run(“df -kP”)mem_usage = run(“cat /proc/meminfo”)cpu_usage = run(“cat /proc/stat”

print disk_usage, mem_usage, cpu_info

Very useful for gettinglive information from

many different servers

Very useful for gettinglive information from

many different servers

ffunctioninc.

Fabfile.py

from fabric.api import *from mysetup import *

env.host = [“server1.myapp.com”]

def setup(): install_packages(“...”) update_configuration() create_users() start_daemons()

$ fab setup

ffunctioninc.

Fabfile.py




$ fab setup

Just like Make, youwrite rules that do

something

Just like Make, youwrite rules that do

something

ffunctioninc.

Fabfile.py




$ fab setup

...and you can specifyon which servers the rules

will run

...and you can specifyon which servers the rules

will run

ffunctioninc.

Multiple hosts

@hosts(“db1.myapp”)def backup_db():

run(...)

env.hosts = [“db1.myapp.com”,“db2.myapp.com”,“db3.myapp.com”

]

ffunctioninc.

Roles

$ fab -R web setup

env.roledefs = { 'web': ['www1', 'www2', 'www3'], 'dns': ['ns1', 'ns2']}

ffunctioninc.

Roles

$ fab -R web setup

env.roledefs = { 'web': ['www1', 'www2', 'www3'], 'dns': ['ns1', 'ns2']}

Will run the setup ruleonly on hosts members

of the web role.

Will run the setup ruleonly on hosts members

of the web role.

ffunctioninc.

Some facts about Fabric

Fabric 1.0 just released!On March, 4th 2011

3 years of developmentFirst commit 1161 days ago (on March 10th, 2011)

Related ProjectsOpscode's Chef and Puppet

ffunctioninc.

What's good about Fabric?

Low-levelBasically an ssh() command that returns the result

Simple primitivesrun(), sudo(), get(), put(), local(), prompt(), reboot()

No magicNo DSL, no abstraction, just a remote command API

ffunctioninc.

What could be improved ?

Ease common admin tasksUser, group creation. Files, directory operations.

Abstract primitivesLike install package, so that it works with different OS

TemplatesTo make creating/updating configuration files easy

ffunctioninc.

Cuisine:Chef-like functionality for Fabric

ffunctioninc.

Part 2

Cuisine

ffunctioninc.

What is Opscode's Chef?

RecipesScripts/packages to install and configure services and applications

APIA DSL-like Ruby API to interact with the OS (create users, groups, install packages, etc)

ArchitectureClient-server or “solo” mode to push and deploy your new configurations

http://wiki.opscode.com/display/chef/Home

ffunctioninc.

What I liked about Chef

FlexibleYou can use the API or shell commands

StructuredHelped me have a clear decomposition of the services installed per machine

CommunityLots of recipes already available from http://cookbooks.opscode.com/

ffunctioninc.

What I didn't like

Too many files and directoriesCode is spread out, hard to get the big picture

Abstraction overloadAPI not very well documented, frequent fall backs to plain shell scripts within the recipe

No “smart” recipeRecipes are applied all the time, even when it's not necessary

ffunctioninc.

The question that kept coming...

Django recipe: 5 files, 2 directories

sudo aptitude install apache2 python django-python

What it does, in essence

ffunctioninc.

The question that kept coming...

Django recipe: 5 files, 2 directories

sudo aptitude install apache2 python django-python

What it does, in essence

Is this really necessaryfor what I want to do ?

Is this really necessaryfor what I want to do ?

ffunctioninc.

What I loved about Fabric

Bare metalssh() function, simple and elegant set of primitives

No magicNo abstraction, no model, no compilation

Two-way communicationEasy to change the rule's behaviour according to the output (ex: do not install something that's already installed)

ffunctioninc.

What I needed

Fabric

ffunctioninc.

What I needed

Fabric

File I/OFile I/O

ffunctioninc.

What I needed

Fabric

File I/OFile I/O User/GroupManagement

User/GroupManagement

ffunctioninc.

What I needed

Fabric

File I/OFile I/O PackageManagement

PackageManagement



ffunctioninc.

What I needed

Fabric

File I/OFile I/O PackageManagement

PackageManagement



Text processing & TemplatesText processing & Templates

ffunctioninc.

How I wanted it

Simple “flat” API[object]_[operation] where operation is something in “create”, “read”, “update”, “write”, “remove”, “ensure”, etc...

Driven by needOnly implement a feature if I have a real need for it

No magicEverything is implemented using sh-compatible commands

No unnecessary structureEverything fits in one file, no imposed file layout

ffunctioninc.

Cuisine: Example fabfile.py

from cuisine import *


def setup():package_ensure(“python”, “apache2”, “python-django”)user_ensure(“admin”, uid=2000)upstart_ensure(“django”)

$ fab setup

ffunctioninc.





$ fab setup

Fabric's core functionsare already imported

Fabric's core functionsare already imported

ffunctioninc.





$ fab setup Cuisine's APIcalls

Cuisine's APIcalls

ffunctioninc.

File I/O

ffunctioninc.

Cuisine : File I/O

● file_exists does remote file exists?● file_read reads remote file● file_write write data to remote file● file_append appends data to remote file● file_attribs chmod & chown● file_remove

ffunctioninc.

Cuisine : File I/O

● file_exists does remote file exists?● file_read reads remote file● file_write write data to remote file● file_append appends data to remote file● file_attribs chmod & chown● file_remove

Supports owner/groupand mode change

Supports owner/groupand mode change

ffunctioninc.

Cuisine : File I/O (directories)

● dir_exists does remote file exists?● dir_ensure ensures that a directory exists● dir_attribs chmod & chown● dir_remove

ffunctioninc.

Cuisine : File I/O +

● file_update(location, updater=lambda _:_)

package_ensure("mongodb-snapshot")def update_configuration( text ): res = [] for line in text.split("\n"): if line.strip().startswith("dbpath="): res.append("dbpath=/data/mongodb") elif line.strip().startswith("logpath="): res.append("logpath=/data/logs/mongodb.log") else: res.append(line) return "\n".join(res)file_update("/etc/mongodb.conf", update_configuration)

ffunctioninc.




This replaces the values forconfiguration entriesdbpath and logpath

This replaces the values forconfiguration entriesdbpath and logpath

ffunctioninc.




The remote file will only bechanged if the content

is different

The remote file will only bechanged if the content

is different

ffunctioninc.

User Management

ffunctioninc.

Cuisine: User Management

● user_exists does the user exists?● user_create create the user● user_ensure create the user if it doesn't exist

ffunctioninc.

Cuisine: Group Management

● group_exists does the group exists?● group_create create the group● group_ensure create the group if it doesn't exist● group_user_exists does the user belong to the group?● group_user_add adds the user to the group● group_user_ensure

ffunctioninc.

Package Management

ffunctioninc.

Cuisine: Package Management

● package_exists is the package available ?● package_installed is it installed ?● package_install install the package● package_ensure ... only if it's not installed● package_upgrade upgrades the/all package(s)

ffunctioninc.

Text & Templates

ffunctioninc.

Cuisine: Text transformation

text_ensure_line(text, lines)

file_update("/home/user/.profile", lambda _:text_ensure_line(_,

"PYTHONPATH=/opt/lib/python:${PYTHONPATH};""export PYTHONPATH"

))

ffunctioninc.


text_ensure_line(text, lines)

file_update("/home/user/.profile", lambda _:text_ensure_line(_,

"PYTHONPATH=/opt/lib/python:${PYTHONPATH};""export PYTHONPATH"

))

Ensures that the PYTHONPATHvariable is set and exported,

If not, these lines will beappended.

Ensures that the PYTHONPATHvariable is set and exported,

If not, these lines will beappended.

ffunctioninc.


text_replace_line(text, old, new, find=.., process=...)

configuration = local_read("server.conf")for key, value in variables.items():

configuration, replaced = text_replace_line(configuration,key + "=",key + "=" + repr(value),process=lambda text:text.split("=")[0].strip()

)

ffunctioninc.





)

Replaces lines that look likeVARIABLE=VALUE

with the actual values from thevariables dictionary.

Replaces lines that look likeVARIABLE=VALUE

with the actual values from thevariables dictionary.

ffunctioninc.





)

The process lambda transformsinput lines before comparing

them.

Here the lines are strippedof spaces and of their value.

The process lambda transformsinput lines before comparing

them.

Here the lines are strippedof spaces and of their value.

ffunctioninc.


text_strip_margin(text)

file_write(".profile", text_strip_margin("""|export PATH="$HOME/bin":$PATH|set -o vi"""

))

ffunctioninc.


text_strip_margin(text)

file_write(".profile", text_strip_margin("""|export PATH="$HOME/bin":$PATH|set -o vi"""

))

Everything after the | separatorwill be output as content.

It allows to easily embed texttemplates within functions.

Everything after the | separatorwill be output as content.

It allows to easily embed texttemplates within functions.

ffunctioninc.


text_template(text, variables)

text_template(text_strip_margin("""|cd ${DAEMON_PATH}|exec ${DAEMON_EXEC_PATH}"""

), dict(DAEMON_PATH="/opt/mongodb",DAEMON_EXEC_PATH="/opt/mongodb/mongod"

))

ffunctioninc.


text_template(text, variables)

text_template(text_strip_margin("""|cd ${DAEMON_PATH}|exec ${DAEMON_EXEC_PATH}"""

), dict(DAEMON_PATH="/opt/mongodb",DAEMON_EXEC_PATH="/opt/mongodb/mongod"

))

This is a simple wrapperaround Python (safe)

string.template() function

This is a simple wrapperaround Python (safe)

string.template() function

ffunctioninc.

Cuisine: Goodies

● ssh_keygen generates DSA keys

● ssh_authorize authorizes your key on the remote server

● mode_sudo run() always uses sudo

● upstart_ensure ensures the given daemon is running

& more!

ffunctioninc.

Cuisine Tips: Structuring your rules

BOOTSTRAP

ffunctioninc.


BOOTSTRAP

You just received your newVPS, and you want to set itup so that you have a basesystem that you can accesswithout typing a password

You just received your newVPS, and you want to set itup so that you have a basesystem that you can accesswithout typing a password

ffunctioninc.


BOOTSTRAP SETUP

ffunctioninc.


BOOTSTRAP SETUP

You install your users, groups,preferred packages andconfiguration. You alsoinstall you applications.

You install your users, groups,preferred packages andconfiguration. You alsoinstall you applications.

ffunctioninc.


BOOTSTRAP SETUP UPDATE

ffunctioninc.



You want to deploy the newversion of the application

you just built

You want to deploy the newversion of the application

you just built

ffunctioninc.



def bootstrap():# Secure SSH, create admin user# Authorize SSH public keys# Remove unwanted packages

ffunctioninc.



def setup():# Create directories (ex: /opt/data, /opt/services, etc)# Create user/groups (ex: apps, services, etc)# Install base tools (ex: screen, fail2ban, zsh, etc)# Edit configuration (ex: profile, inputrc, etc)# Install and run your application

ffunctioninc.



def update():# Download your application update# Freeze/stop the running application# Install the update# Reload/restart your application# Test that everything is OK

ffunctioninc.

Why use Cuisine ?

● Simple API for remote-server manipulationFiles, users, groups, packages

● Shell commands for specific tasks onlyAvoid problems with your shell commands by only using run() for very specific tasks

● Cuisine tasks are not stupid*_ensure() commands won't do anything if it's not necessary

ffunctioninc.

Limitations

● Limited to sh-shellsOperations will not work under csh

● Only written/tested for Ubuntu LinuxContributors could easily port commands

ffunctioninc.

Get started !

On Github:http://github.com/sebastien/cuisine

1 short Python fileDocumented API

http://github.com/sebastien/cuisine

ffunctioninc.

Part 3

Watchdog

Server and services monitoring

ffunctioninc.

The problem

ffunctioninc.

The problem

Low disk spaceLow disk space

ffunctioninc.

The problem

Archive filesRotate logs

Purge cache

Archive filesRotate logs

Purge cache

ffunctioninc.

The problem HTTP serverhas highlatency

HTTP serverhas highlatency

ffunctioninc.

The problemRestart HTTP

server

Restart HTTPserver

ffunctioninc.

The problem

System loadis too high

System loadis too high

ffunctioninc.

The problem

re-niceimportantprocesses

re-niceimportantprocesses

ffunctioninc.

We want to be notifiedwhen problems occur

ffunctioninc.

We want automatic actions to be taken whenever possible

ffunctioninc.

(Some of the) existing solutions

Monit, God, Supervisord, UpstartFocus on starting/restarting daemons and services

Munin, CactiFocus on visualization of RRDTool data

CollectdFocus on collecting and publishing data

ffunctioninc.

The ideal tool

Wide spectrumData collection, service monitoring, actions

Easy setup and deploymentNo complex installation or configuration

Flexible server architectureCan monitor local or remote processes

Customizable and extensibleFrom restarting deamons to monitoring whole servers

ffunctioninc.

Hello, Watchdog!

SERVICE

ffunctioninc.

Hello, Watchdog!

RULE

SERVICE

ffunctioninc.

Hello, Watchdog!

RULE

SERVICE

A service is acollection of

RULES

A service is acollection of

RULES

ffunctioninc.

Hello, Watchdog!

RULE

SERVICE

HTTP RequestCPU, Disk, Mem %Process statusI/O Bandwidth

ffunctioninc.

Hello, Watchdog!

RULE

SERVICE


Each rule retrievesdata and processes it.Rules can SUCCEED

or FAIL

Each rule retrievesdata and processes it.Rules can SUCCEED

or FAIL

ffunctioninc.

Hello, Watchdog!

RULE

ACTION

SERVICE


ffunctioninc.

Hello, Watchdog!

RULE

ACTION

SERVICE


LoggingXMPP, Email notificationsStart/stop process….

ffunctioninc.

Hello, Watchdog!

RULE

ACTION

SERVICE


LoggingXMPP, Email notificationsStart/stop process….

Actions are boundto rule, triggeredon rule SUCCESS

or FAILURE

Actions are boundto rule, triggeredon rule SUCCESS

or FAILURE

ffunctioninc.

Execution Model

MONITOR

ffunctioninc.

Execution Model

MONITORRULE

(frequency in ms)

SERVICE DEFINITION

ffunctioninc.

Execution Model

MONITORRULE

(frequency in ms)

SERVICE DEFINITION

Services are registeredin the monitor

Services are registeredin the monitor

ffunctioninc.

Execution Model

MONITORRULE

(frequency in ms)

SERVICE DEFINITION

Rules defined in theservice are executed

every N ms(frequency)







ffunctioninc.

Execution Model

MONITORRULE

(frequency in ms)

ACTION

ACTION

ACTION

SERVICE DEFINITION

SUCCESS FAILURE

ffunctioninc.

Execution Model

MONITORRULE

(frequency in ms)

ACTION

ACTION

ACTION

SERVICE DEFINITION

If the rule SUCCEEDSactions will be

sequentially executed

If the rule SUCCEEDSactions will be

sequentially executed

SUCCESS FAILURE

ffunctioninc.

Execution Model

MONITORRULE

(frequency in ms)

ACTION

ACTION

ACTION

SERVICE DEFINITION

If the rule FAILfailure actions will besequentially executed

If the rule FAILfailure actions will besequentially executed

SUCCESS FAILURE

ffunctioninc.

Monitoring a remote machine

#!/usr/bin/env pythonfrom watchdog import *Monitor(

Service(name = "google-search-latency",monitor = (

HTTP(GET="http://www.google.ca/search?q=watchdog",freq=Time.s(1),timeout=Time.ms(80),fail=[

Print("Google search query took more than 50ms")]

))

)).run()

ffunctioninc.






))

)).run()

A monitor is like the“main” for Watchdog.

It actively monitorsservices.

A monitor is like the“main” for Watchdog.

It actively monitorsservices.

ffunctioninc.






))

)).run()

Don't forget to callrun() on it

Don't forget to callrun() on it

ffunctioninc.






))

)).run()

The service monitorsthe rules

The service monitorsthe rules

ffunctioninc.






))

)).run()

The HTTP ruleallows to test

an URL

The HTTP ruleallows to test

an URL

And we display amessage in case

of failure

And we display amessage in case

of failure

ffunctioninc.






))

)).run()

If it there is a 4XX orit timeouts, the rulewill fail and displayan error message

If it there is a 4XX orit timeouts, the rulewill fail and displayan error message

ffunctioninc.


$ python example-service-monitoring.py

2011-02-27T22:33:18 watchdog --- #0 (runners=1,threads=2,duration=0.57s)2011-02-27T22:33:18 watchdog [!] Failure on HTTP(GET="www.google.ca:80/search?q=watchdog",timeout=0.08) : Socket error: timed outGoogle search query took more than 50ms2011-02-27T22:33:19 watchdog --- #1 (runners=1,threads=2,duration=0.73s)2011-02-27T22:33:20 watchdog --- #2 (runners=1,threads=2,duration=0.54s)2011-02-27T22:33:21 watchdog --- #3 (runners=1,threads=2,duration=0.69s)2011-02-27T22:33:22 watchdog --- #4 (runners=1,threads=2,duration=0.77s)2011-02-27T22:33:23 watchdog --- #5 (runners=1,threads=2,duration=0.70s)

ffunctioninc.

Sending Email Notification

send_email = Email("[email protected]","[Watchdog]Google Search Latency Error", "Latency was over 80ms", "smtp.gmail.com", "myusername", "mypassword"

)

[…]HTTP(

GET="http://www.google.ca/search?q=watchdog",freq=Time.s(1),timeout=Time.ms(80),fail=[

send_email]

)

ffunctioninc.



)

[…]HTTP(


send_email]

)

The Email rule will sendan email to

[email protected] triggered

The Email rule will sendan email to

[email protected] triggered

mailto:[email protected]


ffunctioninc.



)

[…]HTTP(


send_email]

)

This is how we bind theaction to the rule failure

This is how we bind theaction to the rule failure

ffunctioninc.

Sending Email+Jabber Notification

send_xmpp = XMPP("[email protected]","Watchdog: Google search latency over 80ms","[email protected]", "myspassword"

)

[…]HTTP(


send_email, send_xmpp]

)

ffunctioninc.

Monitoring incident: when something fails repeatedly during a given period of

time

ffunctioninc.

Monitoring incident: when something fails repeatedly during a given period of

time

You don't want to benotified all the time,only when it really

matters.

You don't want to benotified all the time,only when it really

matters.

ffunctioninc.

Detecting incidents


Incident(errors = 5,during = Time.s(10),actions = [send_email,send_xmpp]

)]

)

ffunctioninc.

Detecting incidents



)]

)

An incident is a “smart”action : it will only dosomething when the

condition is met

An incident is a “smart”action : it will only dosomething when the

condition is met

ffunctioninc.

Detecting incidents



)]

)

When at least 5 errors...When at least 5 errors...

ffunctioninc.

Detecting incidents



)]

)

...happen over a 10seconds period

...happen over a 10seconds period

ffunctioninc.

Detecting incidents



)]

)

The Incident action willtrigger the given actions

The Incident action willtrigger the given actions

ffunctioninc.

Example: Ensuring a service is running

from watchdog import *Monitor(

Service(name="myservice-ensure-up",monitor=(

HTTP(GET="http://localhost:8000/",freq=Time.ms(500),fail=[

Incident(errors=5,during=Time.s(5),actions=[

Restart("myservice-start.py")])])))).run()

ffunctioninc.







We test if we canGET http://localhost:8000

within 500ms

We test if we canGET http://localhost:8000

within 500ms

http://localhost:8000/

http://localhost:8000/

ffunctioninc.







If we can't reach it during5 seconds

If we can't reach it during5 seconds

ffunctioninc.







We kill and restartmyservice-start.py

We kill and restartmyservice-start.py

ffunctioninc.

Example: Monitoring system health

from watchdog import *Monitor (

Service(name = "system-health",monitor = (

SystemInfo(freq=Time.s(1),success = (

LogResult("myserver.system.mem", extract=lambda r,_:r["memoryUsage"]),LogResult("myserver.system.disk", extract=lambda

r,_:reduce(max,r["diskUsage"].values())),LogResult("myserver.system.cpu", extract=lambda r,_:r["cpuUsage"]),

)),Delta(

Bandwidth("eth0", freq=Time.s(1)),extract = lambda v:v["total"]["bytes"]/1000.0/1000.0,success = [LogResult("myserver.system.eth0.sent")]

),SystemHealth(

cpu=0.90, disk=0.90, mem=0.90,freq=Time.s(60),fail=[Log(path="watchdog-system-failures.log")]

),)

)).run()

ffunctioninc.

Monitoring system health






)),Delta(


),SystemHealth(


),)

)).run()

ffunctioninc.







)),Delta(


),SystemHealth(


),)

)).run()

SystemInfo will retrievesystem information andreturn it as a dictionary

SystemInfo will retrievesystem information andreturn it as a dictionary

ffunctioninc.





LogResult("myserver.system.mem=", extract=lambda r,_:r["memoryUsage"]),LogResult("myserver.system.disk=", extract=lambda

r,_:reduce(max,r["diskUsage"].values())),LogResult("myserver.system.cpu=", extract=lambda r,_:r["cpuUsage"]),

)),Delta(


),SystemHealth(


),)

)).run()

We log each result byextracting the given

value from the resultdictionary (memoryUsage,

diskUsage,cpuUsage)

We log each result byextracting the given

value from the resultdictionary (memoryUsage,

diskUsage,cpuUsage)

ffunctioninc.







)),Delta(


),SystemHealth(


),)

)).run()

Bandwidth collectsnetwork interface

live traffic information

Bandwidth collectsnetwork interface

live traffic information

ffunctioninc.







)),Delta(

Bandwidth("eth0", freq=Time.s(1)),extract = lambda _:_["total"]["bytes"]/1000.0/1000.0,success = [LogResult("myserver.system.eth0.sent")]

),SystemHealth(


),)

)).run()

But we don't want thetotal amount, we justwant the difference.Delta does just that.

But we don't want thetotal amount, we justwant the difference.Delta does just that.

ffunctioninc.







)),Delta(

Bandwidth("eth0", freq=Time.s(1)),extract = lambda _:_["total"]["bytes"]/1000.0/1000.0,success = [LogResult("myserver.system.eth0.sent=")]

),SystemHealth(


),)

)).run()

We print the resultas before

We print the resultas before

ffunctioninc.







)),Delta(


),SystemHealth(


),)

)).run()

SystemHealth willfail whenever the usage

is above the giventhresholds

SystemHealth willfail whenever the usage

is above the giventhresholds

ffunctioninc.







)),Delta(


),SystemHealth(


),)

)).run()

We'll log failuresin a log file

We'll log failuresin a log file

ffunctioninc.

Watchdog: Decentralized architecture

APPSERVER

STATIC FILESERVER

DB SERVERSERVER

ffunctioninc.


APPSERVER

W

STATIC FILESERVER

DB SERVERSERVER

Ensures the App isrunning

(pid & HTTP test)

Ensures the App isrunning

(pid & HTTP test)

ffunctioninc.


APPSERVER

W

STATIC FILESERVER

W

DB SERVERSERVER

Ensures the static fileserver is running

an has lowlatency

Ensures the static fileserver is running

an has lowlatency

ffunctioninc.


APPSERVER

W

STATIC FILESERVER

W

DB SERVERSERVER

W

Ensures the DB isrunning and that

queriesare not too slow.

Ensures the DB isrunning and that

queriesare not too slow.

ffunctioninc.

Watchdog: Centralized Architecture

APPSERVER

STATIC FILESERVER

DB SERVERSERVER

ffunctioninc.


APPSERVER

STATIC FILESERVER

DB SERVERSERVER

PLATFORMSERVER

ffunctioninc.


APPSERVER

STATIC FILESERVER

DB SERVERSERVER

PLATFORMSERVER

W

Does high-level (HTTP,SQL) queries on theservers and execute

actions remotelywhen problems

are detected

Does high-level (HTTP,SQL) queries on theservers and execute

actions remotelywhen problems

are detected

ffunctioninc.

Watchdog: Deploying on Ubuntu

UPSTART!UPSTART!

ffunctioninc.


# upstart - Watchdog Configuration File# =====================================# updated: 2011-02-28

description "Watchdog - service monitoring daemon"author "Sebastien Pierre <[email protected]>"

start on (net-device-up and local-filesystems)stop on runlevel [016]

respawn

script # NOTE: Change this to wherever the watchdog is installed WATCHDOG_HOME=/opt/services/watchdog cd $WATCHDOG_HOME # NOTE: Change this to wherever your custom watchdog script is installed python watchdog.pyend script

console output# EOF

ffunctioninc.


# upstart - Watchdog Configuration File# =====================================# updated: 2011-02-28

description "Watchdog - service monitoring daemon"author "Sebastien Pierre <[email protected]>"

start on (net-device-up and local-filesystems)stop on runlevel [016]

respawn

script # NOTE: Change this to wherever the watchdog is installed WATCHDOG_HOME=/opt/services/watchdog cd $WATCHDOG_HOME # NOTE: Change this to wherever your custom watchdog script is installed python watchdog.pyend script

console output# EOF

Save this file as/etc/init/watchdog.conf

Save this file as/etc/init/watchdog.conf

ffunctioninc.

Watchdog: Overview

Monitoring DSLDeclarative programming to define monitoring strategy

Wide spectrumFrom data collection to incident detection

FlexibleDoes not impose a specific architecture

ffunctioninc.

Watchdog: Use cases

Ensure service availabilityTest and stop/restart when problems

Collect system statisticsLog or send data through the network

Alert on system or service healthTake actions when the system stats is above threshold

ffunctioninc.

Watchdog: What's coming?

ZeroMQ channelsData streaming and inter-watchdog comm.

DocumentationOnly the basics, need more love!

Contributors?Codebase is small and clear, start hacking!

ffunctioninc.

Get started !

On Github:http://github.com/sebastien/watchdog

1 Python fileDocumented API

http://github.com/sebastien/watchdog

ffunctioninc.

Merci !

[email protected]/sebastien


Server Administration in Python with Fabric, Cuisine and Watchdog

Documents

runcat procversion ffunction

centers ffunction

challengeorderserver

comamazon ec2sliceslice

new machine ffunction

amazon ec2 sliceslice

server room

readusing fabric