Logging & Metrics With Docker A Comprehensive Monitoring Solution Stefan Zier June 13 th , 2015
Aug 14, 2015
Logging & Metrics With DockerA Comprehensive Monitoring Solution
Stefan Zier
June 13th, 2015
whoami
Infrastructure, Backend Dev/Architect
Chief Architect, Sumo Logic, since 2010
Server & Infrastructure, ArcSight (HP), 2001-
2010
Mandatory Slide Showing Shipping Containers
Docker – What’s making debugging hard?
One more layer of abstraction
Container per app = File system
per process
File systems short lived, transient
Resource schedulers = no
container affinity to host
What Our Customers Are Telling Us
We have one process per container
We like to log to stdout
We have multiple processes per container
We run the Sumo Logic collector on the Docker host
We are looking into using Beanstalk with Docker
We are using Amazon ECS
Everyone here loves Docker
We are logging straight from the application
We are using /dev/log for Syslog
We want immutable infrastructure
GoalGet logs from our containerized applications to a centralized logging platform.
How do apps emit logs
Append to a file
Use syslog()
Use log4j, log4net, slf4, etc.
printf() to stdout
Getting logs out of the container - Files
Use VOLUME to mount a host directory
Collect files from the host
Collect files from another container sharing the VOLUME
Need to manage disk space, i.e. rotate logs
App (where supported)
Host
Yet another container with logrotate
docker run -v /tmp/clogs:/tmp/clogs -d --name="sumo-logic-collector" sumologic/collector:latest-file [Access ID] [Access key]
Getting logs out - Syslog
VOLUME /dev/log from host and use host
syslogd
Run a syslogd inside the container
Emit TCP/UDP
Write to a file using VOLUME
Emit syslog TCP/UDP directly from the app
docker run -d -p 514:514 -p 514:514/udp \ --name="sumo-logic-collector” \ sumologic/collector:latest-syslog [Access ID] [Access key]
Getting logs out – Logging frameworks
Sumo Logic blog on official collector imageshttp://www.sumologic.com/blog/company/an-official-docker-image-for-the-sumo-logic-collector
https://github.com/SumoLogic/sumologic-collector-docker
Rainer Gerhards on Rsyslog’s file input modulehttp://www.slideshare.net/rainergerhards1/using-wildcards-with-rsyslogs-file-monitor-imfile
OWASP Log Injectionhttps://www.owasp.org/index.php/Log_injection
Getting logs out – Logging frameworks
Directly to network destinations
HTTP/HTTPS
Also support files, stdout, etc.
Getting logs out – Logging frameworks
Various application stackshttp://help.papertrailapp.com/
Log4Jhttps://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/net/SyslogAppender.html
Apache Web Server http://httpd.apache.org/docs/trunk/mod/mod_syslog.html
https://raymii.org/s/snippets/Apache_access_and_error_log_to_syslog.html
Nginxhttp://nginx.org/en/docs/syslog.html
Postgreshttp://www.postgresql.org/docs/9.1/static/runtime-config-logging.html
Sumo Logic blog on official syslog collector imagehttp://www.sumologic.com/blog/company/an-official-docker-image-for-the-sumo-logic-collector
https://github.com/SumoLogic/sumologic-collector-docker
Getting logs out – stdout
Simply printf()
Logging framework to console
Symlink to /dev/stdout or /dev/stderr
Configure paths to /dev/stdout or /dev/stderr
RUN ln -sf /dev/stdout /var/log/nginx/access.logRUN ln -sf /dev/stderr /var/log/nginx/error.log
Docker Logging Drivers
What Docker provides
Captures stdout/stderr
Feeds it to logging drivers
docker logs command
Returns the entire log every time
Works with json-file driver only
Can tail logs
docker logs –tf –-tail 0 [ID]
Docker Logging Drivers
Configured on docker run
stdout and stderr dispatched to drivers
json-file (default pre 1.6)
syslog
journald
No stats, no events
json-file driver
Output unbounded, can fill up the host disk
Requires logrotate on the Docker host
https://github.com/docker/docker/issues/7333
Stats
Docker Stats
Per-container cgroups metrics (like docker
stats)
Memory
CPU
Block I/OCONTAINER CPU % MEM USAGE/LIMIT MEM % NET I/Ocollector 2.23% 232.6 MiB/2 GiB 11.36% 191.9 KiB/636.3 KiB
RequirementsHow would we want it to work?
What information do we want to collect?
Timestamp
Log message
Docker host info
Container ID
Image ID
Process ID
How should it work?
Use docker logging infrastructure
Minimal moving parts
Containerized - don’t touch the host
Complete – pick up all available data
Automatically discover new containers
Docker APIThe solution maybe?
Docker API
Docker daemon has a REST API
TCP or unix socket
Streaming APIs
Docker Events (container lifecycle updates)
Container Stats (CPU, memory used, …)
App Logs (container stdout/stderr)
Collecting via Docker API
Discover new containers via events
Start streaming their logs and stats
When they go away, stop
Do all of this via the API
Send all of it to centralized log management
Collecting via Docker API, continued
Single component to do it
Zero footprint on the host
Follows Docker standard way of logging
One more thing…
Introducing: Sumo Logic Docker Source
Sumo Logic Docker Source
Active development
Early access expected later this year
Demo Time
fin.Questions?
@stefanzier