Transcript
The Dark Art of Container Monitoring
Loris Degioanni
Me
Loris DegioanniCreator and CEO of sysdigPast: WinPcap, Wireshark
@lorisdegio
Containers Are Great…
• less overhead• faster deployments• reproducibility of environments• cost optimizations• Isolation• flexibility
…But Inspecting Containers Is Not Easy
Containers are:• Isolated• Self-Contained• Simple• Lightweight
…But Inspecting Containers Is Not Easy
Containers are:• Isolated• Self-Contained• Simple• Lightweight
Can monitoring and troubleshooting respect these properties?
•Containers•Overview of monitoring/troubleshooting options • command line• cAdvisor• Docker stats• sysdig
• Examples and demos
In This Talk
•Containers•Overview of monitoring/troubleshooting options • command line• cAdvisor• Docker stats• sysdig
• Examples and demos
In This Talk
•Containers•Overview of monitoring/troubleshooting options • command line• cAdvisor• Docker stats• sysdig
• Examples and demos
In This Talk
Hypervisor
Monitoring VMs
VM1 VM3VM2
Hypervisor
Monitoring VMs, Option 1
VM1 VM3VM2
Hypervisor-level instrumentation,Amazon CloudWatch
Hypervisor
Monitoring VMs, Option 2
VM1 VM3VM2
Monitoring Agent
OS
Monitoring Containers
Container1 Container3Container2
OS
Monitoring Containers, Option 1
Container1 Container3Container2
Monitoring Agent
OS
Monitoring Containers, Option 1
Container1 Container3Container2
Monitoring Agent
• Not scalable• Not composable• Adds dependencies/size• Kills the concept of one process per container
OS
Monitoring Containers, Option 2
Container1 Container3Container2
Container runtime – level monitoringKernel-level instrumentation
OS
Monitoring Containers, Option 3
Container1 MonitoringContainer
Container2
•Containers•Overview of monitoring/troubleshooting options • command line• cAdvisor• Docker stats• sysdig
• Examples and demos
In This Talk
•Containers•Overview of monitoring/troubleshooting options • command line• cAdvisor• Docker stats• sysdig
• Examples and demos
In This Talk
•Containers•Overview of monitoring/troubleshooting options • command line• cAdvisor• Docker stats• sysdig
• Examples and demos
In This Talk
In The Rest of this Talk
•Overview of Open Source monitoring/troubleshooting options • command line• cAdvisor• Docker stats• sysdig
• Examples and demos
In The Rest of this Talk
•Overview of Open Source monitoring/troubleshooting options • command line• cAdvisor• Docker stats• sysdig
• Examples and demos
Some Things We Want to Monitor
• Resource usage (CPU/Memory/Disk)• Network activity• File I/O activity• Errors/faults• Application activity/Logs• Topology
Resource Usage (CPU/Memory/Disk)
• ps/top/htop from the host• Pro: always there• Cons: Very limited container context (full cgroup string only), work only from
the main host
• docker ps / docker top / docker stats• Pro: always there, APIs• Cons: Docker only, limited interactivity
• cAdvisor• sysdig
Installing cAdvisorsudo docker run \--volume=/:/rootfs:ro \--volume=/var/run:/var/run:rw \--volume=/sys:/sys:ro \--volume=/var/lib/docker/:/var/lib/docker:ro \--publish=8080:8080 \--detach=true \--name=cadvisor \google/cadvisor:latest
cAdvisor Pros and Cons
Pros• easy to install• works from a container• user friendly (web UI)• API• Integrations (e.g. InfluxDB, Prometheus, Heapster)
Cons: • (very) Limited set of metrics• Containers are black boxes
Docker Stats APIThe most sophisticated Docker stats poller ever:echo -e "GET /containers/mysql/stats HTTP/1.1\r\n" | nc -U /var/run/docker.sock
Info:http://docs.docker.com/reference/api/docker_remote_api_v1.17/#get-container-stats-based-on-resource-usage
Docker Stats API Pros and Cons
Pros• Part of Docker• Richer than cAdvisor• REST API• Integrations
• docker-collectd-plugin github.com/cloudwatt/docker-collectd-plugin• commercial vendors
Cons: • Docker only• Containers are black boxes
sysdig
•Capture system events, filter them, run useful scripts• strace + tcpdump + lsof + htop + Lua•Open Source•Nice curses UI
Design Goals
•Production-ready• Simple • lightweight
•Rich data• Instant value•Natural workflow•Native support for containers
Design Goals
•Production-ready• Simple • lightweight
•Rich data• Instant value•Natural workflow•Native support for containers
Installing sysdig
docker run -i -t --name sysdig --privileged -v /var/run/docker.sock:/host/var/run/docker.sock -v /dev:/host/dev -v /proc:/host/proc:ro -v /boot:/host/boot:ro -v /lib/modules:/host/lib/modules:ro -v /usr:/host/usr:ro sysdig/sysdig
sysdig Architecture
Kernel
Container1
Docker
Container2
Docker
Container3
LXCAppApp
sysdig Architecture
Kernel
Container1
Docker
Container2
Docker
Container3
LXCAppApp
Instrumentation through kernel module
sysdig Architecture
Kernel
Container1
Docker
Container2
Docker
Container3
LXCAppApp
sysdig
Docker
Capture andanalysis
sysdig Architecture
Kernel
Container1
Docker
Container2
Docker
Container3
LXCAppApp
sysdig
Docker
(optionally) Save to a trace file
foo.scap
Viewing Network Activity
•cAdvisor•docker stats <container>• iftop/tcpdump/tshark in the container•sysdig
Viewing Disk Activity
•iotop/lsof in the container•sysdig
Viewing Application Activity
•Code/container instrumentation•Logs•sysdig
Thank You!
www.sysdig.orghttps://github.com/draios/sysdigwww.sysdig.com
top related