Top Banner
Sumo Logic Confidential Comprehensive Monitoring for Docker Christian Beedgen, CTO & Co-Founder, Sumo Logic San Francisco Docker Meetup, February 9, 2016
59

Comprehensive Monitoring for Docker

Apr 16, 2017

Download

Engineering

Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Page 1: Comprehensive Monitoring for Docker

Sumo Logic Confidential

Comprehensive Monitoring

for Docker

Christian Beedgen, CTO & Co-Founder, Sumo Logic

San Francisco Docker Meetup, February 9, 2016

Page 2: Comprehensive Monitoring for Docker

Sumo Logic Confidential

$ whoami• Co-Founder & CTO, Sumo Logic

Cloud-based Machine Data Analytics ServiceApplications, Operations, Security

• Chief Architect, ArcSightMajor SIEM player in the enterprise spaceLog Management for security and compliance

Page 3: Comprehensive Monitoring for Docker

Sumo Logic Confidential

Agenda

• The Logging Story So Far• Comprehensive Monitoring• If I Had A Wish…

Page 4: Comprehensive Monitoring for Docker

Sumo Logic Confidential

December 2014, New York City

Page 5: Comprehensive Monitoring for Docker

Sumo Logic Confidential

December 2014, New York City

Page 6: Comprehensive Monitoring for Docker

Sumo Logic Confidential

December 2014, New York City

Page 7: Comprehensive Monitoring for Docker

Sumo Logic Confidential

December 2014, New York City

Page 8: Comprehensive Monitoring for Docker

Sumo Logic Confidential

December 2014, New York City

Page 9: Comprehensive Monitoring for Docker

Sumo Logic Confidential

December 2014, New York City

Page 10: Comprehensive Monitoring for Docker

Sumo Logic Confidential

December 2014, New York City

Page 11: Comprehensive Monitoring for Docker

Sumo Logic Confidential

December 2014, New York City

Page 12: Comprehensive Monitoring for Docker

Sumo Logic Confidential

December 2014, New York City

Page 13: Comprehensive Monitoring for Docker

Sumo Logic Confidential

December 2014, New York City

Page 14: Comprehensive Monitoring for Docker

Sumo Logic Confidential

December 2014, New York City

Page 15: Comprehensive Monitoring for Docker

Sumo Logic Confidential

December 2014, New York City

Page 16: Comprehensive Monitoring for Docker

Sumo Logic Confidential

December 2014, New York City

Page 17: Comprehensive Monitoring for Docker

Sumo Logic Confidential

December 2014, New York City

Page 18: Comprehensive Monitoring for Docker

Sumo Logic Confidential

December 2014, New York Cityhttp://www.slideshare.net/

raychaser/6-million-ways-to-log-in-docker-nyc-docker-meetup-12172014

Page 19: Comprehensive Monitoring for Docker

Sumo Logic Confidential

Season 2Where Are We In Early 2016?

Page 20: Comprehensive Monitoring for Docker

Sumo Logic Confidential

Basics• Logging in Docker as per 12factor.net

Page 21: Comprehensive Monitoring for Docker

Sumo Logic Confidential

Basics• Logging in Docker as per 12factor.net

Page 22: Comprehensive Monitoring for Docker

Sumo Logic Confidential

Basics• Logging in Docker as per 12factor.net

• Also, one process per container, plz!

Page 23: Comprehensive Monitoring for Docker

Sumo Logic Confidential

Pre-Docker 1.6

• Docker simply collects stdout and stderr from a container• Wrapped in a bit of JSON and stored on disk

Page 24: Comprehensive Monitoring for Docker

Sumo Logic Confidential

Pre-Docker 1.6

• Docker simply collects stdout and stderr from a container• Wrapped in a bit of JSON and stored on disk

Page 25: Comprehensive Monitoring for Docker

Sumo Logic Confidential

Pre-Docker 1.6

• Docker simply collects stdout and stderr from a container• Wrapped in a bit of JSON and stored on disk

Page 26: Comprehensive Monitoring for Docker

Sumo Logic Confidential

Pre-Docker 1.6

• Docker simply collects stdout and stderr from a container• Wrapped in a bit of JSON and stored on disk

Page 27: Comprehensive Monitoring for Docker

Sumo Logic Confidential

Pre-Docker 1.6

• Early hardcore crowd would just collect /var/lib/docker/containers/**

• And then of course there’s the UX: docker logs• docker logs is using a daemon API for getting the logs• This leads to logspout – attach to API, forward to Syslog• https://github.com/gliderlabs/logspout

Page 28: Comprehensive Monitoring for Docker

Sumo Logic Confidential

Docker 1.6 Introduced Log Drivers• Hallelujah• Initially supports json-file, syslog, null• json-file – default, this is the old mechanism

– Continues to this day to be required for API access and docker logs– Exception: As of 1.8, the journald driver (introduced in 1.7) supports docker logs

• docker run -–log-driver syslog

– Sends to local Syslog, no more writing to disk• docker run –-log-driver null

– STFU, basically

Page 29: Comprehensive Monitoring for Docker

Sumo Logic Confidential

Docker 1.7 Introduces --log-opt

• Now we can pass parameters to the log drivers!• docker run \

--log-driver syslog \

--log-opt syslog-address=(udp|tcp)://… \

--log-opt syslog-facility=(kern|daemon|user|local0|…) \

--log-opt syslog-tag=“myapp”

• Forward directly to local Syslog aggregator, or to a cloud-based logging service

• Docker 1.7 also added support to log to journald

Page 30: Comprehensive Monitoring for Docker

Sumo Logic Confidential

Also in Docker 1.8 – Options For json-file

• json-file still the default, still required for docker logs and /logs API• Long standing problem – will eventually fill up your disk• Folks have been using logrotate hacks…• Now, json-file log driver can be configured:

• Basically, keep up to max-file files, roll current at max-size

Page 31: Comprehensive Monitoring for Docker

Sumo Logic Confidential

Docker 1.9 Introduces Log Tags For The Syslog Driver• Many containers can share a single aggregator downstream the log driver• All this muxing creates a problem – which log from which container?• Basically, there is a loss of meta data• Log Tags enable to use of container meta data as part of each message• --log-opt tag="{{.ImageName}}/{{.Name}}/{{.ID}}"

• Oct 13 18:33:19 play docker/hello-world/foobar/5790672ab6a0[9103]:

Hello from Docker.

Page 32: Comprehensive Monitoring for Docker

Sumo Logic Confidential

Docker 1.9 Introduces Log Tags For The Syslog Driver• Many containers can share a single aggregator downstream the log driver• All this muxing creates a problem – which log from which container?• Basically, there is a loss of meta data• Log Tags enable to use of container meta data as part of each message• --log-opt tag="{{.ImageName}}/{{.Name}}/{{.ID}}"

• Oct 13 18:33:19 play docker/hello-world/foobar/5790672ab6a0[9103]:

Hello from Docker.

Page 33: Comprehensive Monitoring for Docker

Sumo Logic Confidential

Docker 1.9 Introduces Log Tags For The Syslog Driver• Many containers can share a single aggregator downstream the log driver• All this muxing creates a problem – which log from which container?• Basically, there is a loss of meta data• Log Tags enable to use of container meta data as part of each message• --log-opt tag="{{.ImageName}}/{{.Name}}/{{.ID}}"

• Oct 13 18:33:19 play docker/hello-world/foobar/5790672ab6a0[9103]:

Hello from Docker.

Page 34: Comprehensive Monitoring for Docker

Sumo Logic Confidential

Docker 1.9 Introduces Log Tags For The Syslog Driver• Many containers can share a single aggregator downstream the log driver• All this muxing creates a problem – which log from which container?• Basically, there is a loss of meta data• Log Tags enable to use of container meta data as part of each message• --log-opt tag="{{.ImageName}}/{{.Name}}/{{.ID}}"

• Oct 13 18:33:19 play docker/hello-world/foobar/5790672ab6a0[9103]:

Hello from Docker.

Page 35: Comprehensive Monitoring for Docker

Sumo Logic Confidential

Docker 1.10 Adds TCP+TLS Support for Syslog

• This is really nice, because now you can send to the cloud securely

• 1.10 actually has a bug that prevents this from actually working• But fear not, it will be fixed in 1.10.1 - Thanks @cavalera!

Page 36: Comprehensive Monitoring for Docker

Sumo Logic Confidential

Docker 1.10 Adds TCP+TLS Support for Syslog

• This is really nice, because now you can send to the cloud securely

• 1.10 actually has a bug that prevents this from actually working• But fear not, it will be fixed in 1.10.1 - Thanks @cavalera!

Page 37: Comprehensive Monitoring for Docker

Sumo Logic Confidential

Forgive the plug…We are about to release a Cloud Syslog endpoint!

Page 38: Comprehensive Monitoring for Docker

Sumo Logic Confidential

Docker 1.8, 1.9, 1.10 - Even More Log Drivers

• Fluentd

• GELF

• AWS

• Splunk

Page 39: Comprehensive Monitoring for Docker

Sumo Logic Confidential

Page 40: Comprehensive Monitoring for Docker

Sumo Logic Confidential

What Else?

• Proposal for Google Cloud Cloud Logging driver– https://github.com/docker/docker/issues/18726

• Proposal for a Redis driver (rejected today!)– https://github.com/docker/docker/pull/20146

• Proposal for a TCP, UDP, Unix Domain Socket driver– https://github.com/docker/docker/pull/18001

Page 41: Comprehensive Monitoring for Docker

Sumo Logic Confidential

Where Will It End?

• Logging drivers have been a very large step forward in the last 12 months

• As part of the engine, need to go through the engine commit protocol

• This good, because there’s a lot of review stability• But it is also suboptimal because it is not really modular• And it adds more and more dependencies on third party libraries

• Can this be decoupled?

Page 42: Comprehensive Monitoring for Docker

Sumo Logic Confidential

Page 43: Comprehensive Monitoring for Docker

Sumo Logic Confidential

Let’s get back to this a little

later…

Page 45: Comprehensive Monitoring for Docker

Sumo Logic Confidential

The World… Does Not Live On Logs A Alone

Page 46: Comprehensive Monitoring for Docker

Sumo Logic Confidential

For Comprehensive Monitoring & Troubleshooting

1.Events2.Configurations3.Logs4.Statistics5.Host and daemon logs

Page 47: Comprehensive Monitoring for Docker

Sumo Logic Confidential

Events

• Enumerate all running containers, then– Start listening to the event stream, then…

• For each running container, and each start event...– Start collecting (more on that in a minute)

• Also, send each event JSON as a log– Now we can analyze what the daemon is doing!– What containers are getting started, stopped etc. etc. etc.

Page 48: Comprehensive Monitoring for Docker

Sumo Logic Confidential

Page 49: Comprehensive Monitoring for Docker

Sumo Logic Confidential

Configurations

• For each running container, and each start event– Call inspect API

• Send result JSON as a log• Now we have all the configurations in the logs!

– And we can quickly search for them when we troubleshoot

Page 50: Comprehensive Monitoring for Docker

Sumo Logic Confidential

Logs

• For each running container, and each start event– Call logs API to open a stream

• Send each log… as a log

Page 51: Comprehensive Monitoring for Docker

Sumo Logic Confidential

Statistics

• For each running container, and each start event– Call stats API to open

a stream• Send each received

JSON as a log• Now we have monitoring!

Page 52: Comprehensive Monitoring for Docker

Sumo Logic Confidential

Host & Daemon Logs

• Include collector into host images• Or run collector as a container

– … and -v /var/log etc.

Page 53: Comprehensive Monitoring for Docker

Sumo Logic Confidential

Thanks To The API, We Can Really Do This

Page 54: Comprehensive Monitoring for Docker

Sumo Logic Confidential

Thanks To The API, We Can Really Do This

Page 55: Comprehensive Monitoring for Docker

Sumo Logic Confidential

Thanks To The API, We Can Really Do This

Page 56: Comprehensive Monitoring for Docker

Sumo Logic Confidential

If I Had A Wish…

Page 57: Comprehensive Monitoring for Docker

Sumo Logic Confidential

I Really, Really <3 The API

• Of course, there’s always limitations• What we found

– logs API requires jsonfile driver (or journald) which means it touches the disk– Race condition between start event and have logs or stats stream established– Scaling connections on both daemon and client side

• 100 containers on host (or Swarm!)• 100 * (1 event stream connection + (1 logs + 1 stats stream per container)• 201 connections!

– Different mechanisms between events, logs, stats

Page 58: Comprehensive Monitoring for Docker

Sumo Logic Confidential

So What Is That Dream?

• One combined stream for events, logs, stats• Either as an API call to pull, perhaps with a filter

– Name, label, type, start time

• Or as a registration API call to indicate where to send to– Daemon firehoses to the endpoint, with strict guarantees for the endpoint

• I personally could deal with either approach• Can we expand #18604 ”Logging driver plugins” to allow for this?

Page 59: Comprehensive Monitoring for Docker

Sumo Logic Confidential

Thank you!@raychaser