Transcript
Sumo Logic Confidential
Comprehensive Monitoring
for Docker
Christian Beedgen, CTO & Co-Founder, Sumo Logic
San Francisco Docker Meetup, February 9, 2016
Sumo Logic Confidential
$ whoami• Co-Founder & CTO, Sumo Logic
Cloud-based Machine Data Analytics ServiceApplications, Operations, Security
• Chief Architect, ArcSightMajor SIEM player in the enterprise spaceLog Management for security and compliance
Sumo Logic Confidential
Agenda
• The Logging Story So Far• Comprehensive Monitoring• If I Had A Wish…
Sumo Logic Confidential
December 2014, New York City
Sumo Logic Confidential
December 2014, New York City
Sumo Logic Confidential
December 2014, New York City
Sumo Logic Confidential
December 2014, New York City
Sumo Logic Confidential
December 2014, New York City
Sumo Logic Confidential
December 2014, New York City
Sumo Logic Confidential
December 2014, New York City
Sumo Logic Confidential
December 2014, New York City
Sumo Logic Confidential
December 2014, New York City
Sumo Logic Confidential
December 2014, New York City
Sumo Logic Confidential
December 2014, New York City
Sumo Logic Confidential
December 2014, New York City
Sumo Logic Confidential
December 2014, New York City
Sumo Logic Confidential
December 2014, New York City
Sumo Logic Confidential
December 2014, New York Cityhttp://www.slideshare.net/
raychaser/6-million-ways-to-log-in-docker-nyc-docker-meetup-12172014
Sumo Logic Confidential
Season 2Where Are We In Early 2016?
Sumo Logic Confidential
Basics• Logging in Docker as per 12factor.net
Sumo Logic Confidential
Basics• Logging in Docker as per 12factor.net
Sumo Logic Confidential
Basics• Logging in Docker as per 12factor.net
• Also, one process per container, plz!
Sumo Logic Confidential
Pre-Docker 1.6
• Docker simply collects stdout and stderr from a container• Wrapped in a bit of JSON and stored on disk
Sumo Logic Confidential
Pre-Docker 1.6
• Docker simply collects stdout and stderr from a container• Wrapped in a bit of JSON and stored on disk
Sumo Logic Confidential
Pre-Docker 1.6
• Docker simply collects stdout and stderr from a container• Wrapped in a bit of JSON and stored on disk
Sumo Logic Confidential
Pre-Docker 1.6
• Docker simply collects stdout and stderr from a container• Wrapped in a bit of JSON and stored on disk
Sumo Logic Confidential
Pre-Docker 1.6
• Early hardcore crowd would just collect /var/lib/docker/containers/**
• And then of course there’s the UX: docker logs• docker logs is using a daemon API for getting the logs• This leads to logspout – attach to API, forward to Syslog• https://github.com/gliderlabs/logspout
Sumo Logic Confidential
Docker 1.6 Introduced Log Drivers• Hallelujah• Initially supports json-file, syslog, null• json-file – default, this is the old mechanism
– Continues to this day to be required for API access and docker logs– Exception: As of 1.8, the journald driver (introduced in 1.7) supports docker logs
• docker run -–log-driver syslog
– Sends to local Syslog, no more writing to disk• docker run –-log-driver null
– STFU, basically
Sumo Logic Confidential
Docker 1.7 Introduces --log-opt
• Now we can pass parameters to the log drivers!• docker run \
--log-driver syslog \
--log-opt syslog-address=(udp|tcp)://… \
--log-opt syslog-facility=(kern|daemon|user|local0|…) \
--log-opt syslog-tag=“myapp”
• Forward directly to local Syslog aggregator, or to a cloud-based logging service
• Docker 1.7 also added support to log to journald
Sumo Logic Confidential
Also in Docker 1.8 – Options For json-file
• json-file still the default, still required for docker logs and /logs API• Long standing problem – will eventually fill up your disk• Folks have been using logrotate hacks…• Now, json-file log driver can be configured:
• Basically, keep up to max-file files, roll current at max-size
Sumo Logic Confidential
Docker 1.9 Introduces Log Tags For The Syslog Driver• Many containers can share a single aggregator downstream the log driver• All this muxing creates a problem – which log from which container?• Basically, there is a loss of meta data• Log Tags enable to use of container meta data as part of each message• --log-opt tag="{{.ImageName}}/{{.Name}}/{{.ID}}"
• Oct 13 18:33:19 play docker/hello-world/foobar/5790672ab6a0[9103]:
Hello from Docker.
Sumo Logic Confidential
Docker 1.9 Introduces Log Tags For The Syslog Driver• Many containers can share a single aggregator downstream the log driver• All this muxing creates a problem – which log from which container?• Basically, there is a loss of meta data• Log Tags enable to use of container meta data as part of each message• --log-opt tag="{{.ImageName}}/{{.Name}}/{{.ID}}"
• Oct 13 18:33:19 play docker/hello-world/foobar/5790672ab6a0[9103]:
Hello from Docker.
Sumo Logic Confidential
Docker 1.9 Introduces Log Tags For The Syslog Driver• Many containers can share a single aggregator downstream the log driver• All this muxing creates a problem – which log from which container?• Basically, there is a loss of meta data• Log Tags enable to use of container meta data as part of each message• --log-opt tag="{{.ImageName}}/{{.Name}}/{{.ID}}"
• Oct 13 18:33:19 play docker/hello-world/foobar/5790672ab6a0[9103]:
Hello from Docker.
Sumo Logic Confidential
Docker 1.9 Introduces Log Tags For The Syslog Driver• Many containers can share a single aggregator downstream the log driver• All this muxing creates a problem – which log from which container?• Basically, there is a loss of meta data• Log Tags enable to use of container meta data as part of each message• --log-opt tag="{{.ImageName}}/{{.Name}}/{{.ID}}"
• Oct 13 18:33:19 play docker/hello-world/foobar/5790672ab6a0[9103]:
Hello from Docker.
Sumo Logic Confidential
Docker 1.10 Adds TCP+TLS Support for Syslog
• This is really nice, because now you can send to the cloud securely
• 1.10 actually has a bug that prevents this from actually working• But fear not, it will be fixed in 1.10.1 - Thanks @cavalera!
Sumo Logic Confidential
Docker 1.10 Adds TCP+TLS Support for Syslog
• This is really nice, because now you can send to the cloud securely
• 1.10 actually has a bug that prevents this from actually working• But fear not, it will be fixed in 1.10.1 - Thanks @cavalera!
Sumo Logic Confidential
Forgive the plug…We are about to release a Cloud Syslog endpoint!
Sumo Logic Confidential
Docker 1.8, 1.9, 1.10 - Even More Log Drivers
• Fluentd
• GELF
• AWS
• Splunk
Sumo Logic Confidential
Sumo Logic Confidential
What Else?
• Proposal for Google Cloud Cloud Logging driver– https://github.com/docker/docker/issues/18726
• Proposal for a Redis driver (rejected today!)– https://github.com/docker/docker/pull/20146
• Proposal for a TCP, UDP, Unix Domain Socket driver– https://github.com/docker/docker/pull/18001
Sumo Logic Confidential
Where Will It End?
• Logging drivers have been a very large step forward in the last 12 months
• As part of the engine, need to go through the engine commit protocol
• This good, because there’s a lot of review stability• But it is also suboptimal because it is not really modular• And it adds more and more dependencies on third party libraries
• Can this be decoupled?
Sumo Logic Confidential
Sumo Logic Confidential
Let’s get back to this a little
later…
Sumo Logic Confidential
https://www.sumologic.com/2015/06/16/comprehensive-monitoring-for-docker-more-than-just-logs
Sumo Logic Confidential
The World… Does Not Live On Logs A Alone
Sumo Logic Confidential
For Comprehensive Monitoring & Troubleshooting
1.Events2.Configurations3.Logs4.Statistics5.Host and daemon logs
Sumo Logic Confidential
Events
• Enumerate all running containers, then– Start listening to the event stream, then…
• For each running container, and each start event...– Start collecting (more on that in a minute)
• Also, send each event JSON as a log– Now we can analyze what the daemon is doing!– What containers are getting started, stopped etc. etc. etc.
Sumo Logic Confidential
Sumo Logic Confidential
Configurations
• For each running container, and each start event– Call inspect API
• Send result JSON as a log• Now we have all the configurations in the logs!
– And we can quickly search for them when we troubleshoot
Sumo Logic Confidential
Logs
• For each running container, and each start event– Call logs API to open a stream
• Send each log… as a log
Sumo Logic Confidential
Statistics
• For each running container, and each start event– Call stats API to open
a stream• Send each received
JSON as a log• Now we have monitoring!
Sumo Logic Confidential
Host & Daemon Logs
• Include collector into host images• Or run collector as a container
– … and -v /var/log etc.
Sumo Logic Confidential
Thanks To The API, We Can Really Do This
Sumo Logic Confidential
Thanks To The API, We Can Really Do This
Sumo Logic Confidential
Thanks To The API, We Can Really Do This
Sumo Logic Confidential
If I Had A Wish…
Sumo Logic Confidential
I Really, Really <3 The API
• Of course, there’s always limitations• What we found
– logs API requires jsonfile driver (or journald) which means it touches the disk– Race condition between start event and have logs or stats stream established– Scaling connections on both daemon and client side
• 100 containers on host (or Swarm!)• 100 * (1 event stream connection + (1 logs + 1 stats stream per container)• 201 connections!
– Different mechanisms between events, logs, stats
Sumo Logic Confidential
So What Is That Dream?
• One combined stream for events, logs, stats• Either as an API call to pull, perhaps with a filter
– Name, label, type, start time
• Or as a registration API call to indicate where to send to– Daemon firehoses to the endpoint, with strict guarantees for the endpoint
• I personally could deal with either approach• Can we expand #18604 ”Logging driver plugins” to allow for this?
Sumo Logic Confidential
Thank you!@raychaser
top related