Monitoring and Log Management for

Monitoring and Log Management for

Docker Swarm and Kubernetes

Stefan Thies Sematext Group, Inc.

Sematext & I

Logsene

SPM

logs

metrics

Docker Agent#nodejs

Agenda

•What is • Centralized Log Management + Performance Monitoring• Kubernetes / Swarm

•Container Logs

•Container Metrics

•Example: #Swarm3k Monitoring

• Summary

Centralized Log Management

LogagentLogagent

Centralized Monitoring

Expose Metrics

Collect Metrics

Ship Metrics

Store Metrics

Aggregate Metrics

Visualize Metrics• Correlation

with Logs

Anomaly

Detection

Alerting

Server +App / ContainerConfiguration

Monitoring Agents

Time SeriesDatabase

Dashboard Tools, Alerting Tools, ChatOps Tools

https://sematext.com/blog/2016/07/19/open-source-docker-monitoring-logging/

Orchestration

Container

POD

Node Node 1

POD 1

Namespace ns1

Kibana Elasticsearch

POD 2

Namespace ns2

Redis

Services (proxy)

Replication Controllers

DaemonSets

3

HorizontalPodAutoscaler

Kubernetes Dashboard / Heapster

•Current status

• Shows basic resource usagefor workloads (Pod)

• Simple logs view

•Heapster is required for autoscaling features

Orchestration

Container

Stacks

Nodes Node 1

ELK

(compose, app bundle)

Kibana 1 Elasticsearch 1

Redis

(service)

redis1

3

Node 2

ELK

Elasticsearch 2

Elasticsearch 3

Kubernetes != Swarm

•Common base is Docker• Docker Logs & Metrics• Docker API

Container Logs

Docker Logging DriversD

ock

erjson-file (default) Files

journald (CoreOS) System journal

Syslog

TCP

UDP

Fluentd TCP

$plunk TCP

Gelf

Centralized Log Management

Local Log Shipper

Docker logs

Containers (should) log to stdout/stderr !!!

docker logs container_id

docker logs container_name

Docker

API

Docker

client

Container logs

Fun with Docker logging drivers

$ docker run --log-driver=syslog --log-opt syslog-address=udp://$HOSTNAME:514 --log-opt tag=„{{.ImageName}}#{{.Name}}#{{.ID}}" -p 9003:80 –name nginx1 -d nginx

$ docker logs nginx 1

"logs" command is supported only for "json-file" and "journald" logging drivers (got: syslog)

Add Context!

More fun with TCP logging drivers!

docker run --log-driver=syslog --log-opt syslog-address=tcp://127.0.0.1:514 --log-opt tag="{{.ImageName}}#{{.Name}}#{{.ID}}" -p 9004:80 -d nginx

docker: Error response from daemon: Failed to initialize logging driver: dial tcp 127.0.0.1:514: getsockopt: connection refused.

Fix it – run syslog server first!

docker run -d -p 514:514 factorish/syslog -t tcp

docker run –logging-driver=syslog … nginx

curl localhost:9004

docker logs syslog

==> syslog listening on tcp

<30>Nov 17 18:23:43 nginx#nginx1#afebdfff0eed[1710]: 172.17.0.1 - - [17/Nov/2016:18:23:43 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.49.1" "-"

Is UDP better?

Alternatives?

DockerLog files

json-file orjournald

API

Agent

Remote Log Storage

Disk Buffer

Docker API provides

the most complete information!

Reliable networks and backend services?

Better buffer & retransmit in case of failure!

Attach metadata to logs/metrics or

route data to different servers

or indices?

“docker logs” works & logs are stored on local

disk!

Centralize search, analytics, alerts,

access permissionsParse logs

Automatic tagging of logs, metrics, events

• Automatic tagging of log / metrics with• Docker

• Container Name / ID• Image Name / ID• Labels / Environment• Hostname / IP

• Kubernetes• Namespace, Pod Name , UID

• Swarm• Swarm Service Name , ID, Compose Project, Container # (scale)

• Single collector for logs, metrics, events, metadata

• Base for correlation and visualisation

Container Metrics Collection

Collection

Metric collection via Docker API

Smart monitoring agent - all in one

Docker

API

Agent

Remote Storage

Disk Buffer

Docker API provides

Labels, Metrics, Logs, Events …

Reliable networks and backend services?

Better buffer & retransmit in case of failure!

Auto-tagging using container

labels.Discovery of

services Centralize logs, metrics, analytics, alerts, access

permissions

Metrics, Logs,

Events

Integrate application monitoring in the stack

-Custom images- add/remove

app with all req. options

- Start monitoring, reading config from etcd

AppConfig to expose

metrics

App MonitorConfigured for App

Container

Service Discoveryetcd

consul

Auto Discovery via Docker API and Labels?

App Containerconfig to expose

metrics

App MonitorDocker Monitor

run

disco

very

Docker

Automatic run

Key Container Metrics

Node Storage

•Good kids clean up their rooms. Good Docker ops clean up their disks by removing unused containers & images.

Number of containers per host

•Verify deployment strategies

CPU quota per container

Container memory and OOM counter

Docker Events

Swarm Task Status

Limit container resources for your apps!

• Set CPU quotas cpu-quota=6000• Limit Memory and configure App in container to the same limits! m 512mb•Disable Swap: memory-swap=0•To limit a Docker container from eating all your disk IO use

e.g. device-write-bps /dev/sda:1mb

Automatic Deployment of monitoring agents

• One command to run a service on each node joining the cluster

• Kubernetes: • DaemonSet creates a pod per

node kubectl create -f sematext-agent.yml

• Swarm: • Global Service docker service create –mode global ...

Swarm3k Monitoring

Swarm3k Requirements

•Monitoring • Host metrics • Container metrics• Docker Events• Task Monitoring

•Collect Container Logs: Task Errors only

•3000+ Nodes (actual: 4.7k)

•150.000 (actual: 60k)

•Duration 8 hours – 28 GB data collected

•Public/shared Dashboard for the community

Pre-flight test with 500 nodes

•60.000 containers deployed in less than 5 minutes!

Swarm3k in one picture

Limits in visualisation

Missing Labels to group hosts or

containers

Summary

• Setup of Monitoring & Logging is complex in dynamic environments

•Kubernetes != Swarm (yet). Common base: Docker Containers

• Smart Agents to collect, analyze, aggregate metrics, events and logs• Auto discovery of containers for data collection• Use metadata tag metrics & logs as base for correlation and visualization• Integrate monitoring in application stacks for app specific metrics • Auto Discovery of services and automatic configuration for application level

monitoring

Join US!Join US!

We are engineers!

We develop DevOps tools!

We are DevOps people!

We do fun stuff ;)

Join US!

join us

http://sematext.com/jobs

is hiring!

Thank you for listening! Get in touch!

Join US!Join US!

Stefan [email protected]@seti321

http://sematext.com@sematext

Join US!

join us

http://sematext.com/jobs

Come talk to us at the booth