Chef The InstallShield ® of the 21st Century InstallShield is a registered trademark of Flexera Software LLC Neither Circonus nor Chef or their related companies have or make any claim thereto. Monday, May 6, 13
May 11, 2015
ChefThe InstallShield®
of the 21st Century
InstallShield is a registered trademark of Flexera Software LLCNeither Circonus nor Chef or their related companies have or make any claim thereto.
Monday, May 6, 13
Hi, I’m @postwaitMy background is in computing systems engineering
hardware debugging
cut kernel code
network debugging
storage debugging
cut user-space code
operating system release managment
Monday, May 6, 13
Circonus is...
An API-accessible, self-service, state-of-the-art, scalable monitoring and telemetry analysis platform.
With Chef cookbooks available to make monitoring your architecture elegant and simple.
Run as a SaaS... sometimes.
Monday, May 6, 13
This talk is...
not really about the Circonus service.
Monday, May 6, 13
This talk is...
about the challenges involved withturn-key software installation.
Monday, May 6, 13
Installing software
Installing (most closed and some open) softwareis easy.
RPM, deb, IPS, etc.(they all suck, but they all work passably well)
<pkgcommand> install <pkgname>e.g. on OmniOS:
pkg install chef
Monday, May 6, 13
Configuring Software
Varies widely from product to product.
This sucks.
This will never change.
Period.
Monday, May 6, 13
Operating Software
Is actually not that hard.
Most software that survives consumers, runs.
Except when it doesn’t.
This is why we monitor and measure things.
Monday, May 6, 13
What if...
Your software was a complex distributed system?
Your software was the monitoring system itself?
Your software wasn’t your SaaS,but instead your customer’s SaaS on their IaaS.
Monday, May 6, 13
We didn’t start here...
Our software started as SaaS only.
We ran the one, true copy.
Turns out there is still a strong business model around selling enterprise software thatcompanies run on-premise.(in their own cloud.... whatever).
Monday, May 6, 13
Right tool for the job...
In SaaS, we take the “right tool for the job” seriously.
In shipped software, we historically have not.
integration costs are high
support costs are high
licensing challenges
Monday, May 6, 13
A brief look at Circonus
PostgreSQL/pg_amqp, RabbitMQ, redis, memcached, Apache/mod_perl, Node.js, ElasticSearch, OpenSSL (ca)
CEP system (Ernie), case management system (Bert), real-time OLAP system (Razalbath), websockets/etc. (Enzo), metric storage (Srollup & Snowth), API services, web portal, broker (noitd), metric transit (stratcond), long-tail storage services
Monday, May 6, 13
More than 20asynchronous components
Everything can go wrong...it’s just like a Internet infrastructure
Despair.
Monday, May 6, 13
How Chef helped
First: Chef did not save the day.
It did help us quite a bit.
It provided a framework forconverging on an expected state.
Monday, May 6, 13
Chef...
Solo.
Think about it.
It just make sense.
Monday, May 6, 13
1 databag to rule them all
One databag ‘site.json’ that describesthe global topology of all that is Circonus.
It exposes all tunables our clients can control.
A special role called ‘self-configure’/opt/circonus/bin/run-chef self-configure
builds a node file with appropriate roles
Monday, May 6, 13
Chef templating...
Services need to know about other services
site.json has what and where
Chef uses that to template outall the configs forall the services runningon the current node
Monday, May 6, 13
Chef’s upsides
Chef knows how to start services andhow to restart them if they are disabled/in maintenance(this is crucially important in distributed systems)
It means that all dependent services canself-recover simply through diligence.
Monday, May 6, 13
Chef’s upsides
As everything is “automated,” the system is far less tolerant of procedures that sometimes don’t work.
It has the effect ofautomating the QA aroundinstallation and maintenance tasks
Monday, May 6, 13
Chef’s downsides
It has been horribly impracticable to
perform good upgrades (due to packaging)
act as omnipotent state control (due to cost)
Monday, May 6, 13
Chef’s downsides
We support OmniOS and Linux
I want to do the same thing on both:
Apps should live in their own filesystem, logs in another, data in another, use a filesystem for everything. And, of course, use ZFS.
Instead of role: “ernie”
we have roles: “ernie,” “ernie-omnios”, “ernie-rhel”
Monday, May 6, 13
Summary
Chef sucks. Long live Chef.
The process of automating through Chef has:
improved the quality of our deployment process
made it possible to ship a software platform to clients
that runs all the same bits as our production SaaS
and stays up-to-date with our latest bits
Monday, May 6, 13
Thanks
Eric Sproul ~ Circonus Release Engineer(who implemented all of these things)
Monday, May 6, 13