Monitoring, Logging and Tracing on Kubernetes

#Perform2017

with Heapster, Grafana, Fluentd, Elastic Stack, Jolokia and OpenTracingMartin Etmajer @metmajerDynatrace, Technology Lead

Monitoring, Logging and Tracing on Kubernetes

Wolfgang Amadeus Mozart1756 – 1791

The „Jay Z“ of classical music

Mike Grabner

Schwarzenegger

Schnitzel

#Perform20178

Monitoringwith Kubernetes Dashboard, Heapster, influxdb & Grafana

$ git clone https://github.com/kubernetes/dashboard.git$ kubectl create –f src/deploy/kubernetes-dashboard.yaml

Example: Deploy Kubernetes Dashboardlikely to be present in your kube-system already

We metrics!♥

collects resources usage, etc.

collects resources usage, etc.

exposedvia REST

Heapster

heapster

runs in a Pod

heapster

Heapster

source

Kubernetes

REST client

sinks

influxdb

GCL

kafkaelasticsearch

GCM

graphite

KubernetesDashboard

via built-in client

REST API

heapster

Example: Heapster (standalone)

source

Kubernetes

REST client

sinks

influxdb

GCL

kafkaelasticsearch

GCM

graphite

KubernetesDashboard

via built-in client

REST API

Example: Deploy Heapster (standalone)

$ git clone https://github.com/kubernetes/heapster.git$ kubectl create –f heapster/deploy/kube-config/standalone/

heapster

Example: Heapster (with influxdb and Grafana)

source

Kubernetes

REST client

sinks

influxdb grafana

KubernetesDashboard

via built-in client

REST API

query

Example: Deploy Heapster (with influxdb and Grafana)

$ git clone https://github.com/kubernetes/heapster.git$ kubectl create –f heapster/deploy/kube-config/influxdb/

#Perform201723

Loggingwith Fluentd and Elastic Stack

continuous stream of log data for each container

unified logging

Fluentd

runs oneach host

Example: Fluentd (for container logs to elasticsearch)

watch container logs

elasticsearch kibana

queryingestvia configuration

Example: Deploy Fluentd (for container logs to elasticsearch)

$ git clone https://github.com/kubernetes/kubernetes.git$ kubectl create –f kubernetes/cluster/addons/fluentd-elasticsearch/

#Perform201731

Monitoringwith Jolokia

remote access to JMX MBeans

remote access to JMX MBeans

exposed via REST

Java Management Extensions (JMX)to manage and monitor applications

Source: https://en.wikipedia.org/wiki/Java_Management_Extensions

DynamicMBean MXBeanStandard

MBeanInstrumentationLevel

instrument resources

MBeanServerAgent Level

agent for registered MBeans

Connectors Adaptors

RemoteManagementLevel

provides access to MBeanServerRMI, HTTP/SOAP HTML, SNMP

Example: Jolokia (get list of accessible MBeans)

$ curl http://localhost:8080/jolokia/list

includes memory, threading, gc, etc.

Example: Jolokia (read memory usage data)

$ curl http://localhost:8080/jolokia/read/java.lang:type=Memory

name of MBean

Example: Jolokia (read threading data)

$ curl http://localhost:8080/jolokia/read/java.lang:type=Threading

name of MBean

#Perform201738

Tracingwith OpenTracing

a formally specified standard API for reporting

with various API implementations

designgoals

Why distributed (end-to-end) tracing?

Source: https://research.google.com/pubs/pub36356.html

initiator

servicesremote calls

Why distributed (end-to-end) tracing?

Source: https://research.google.com/pubs/pub36356.html

initiator

servicesremote calls

What does the architecture look like?

Why distributed (end-to-end) tracing?

Source: https://research.google.com/pubs/pub36356.html

initiator

servicesremote calls

Where is most of the time spent?

Why distributed (end-to-end) tracing?

Source: https://research.google.com/pubs/pub36356.html

initiator

servicesremote calls

How are services invoked?

Why distributed (end-to-end) tracing?

Source: https://research.google.com/pubs/pub36356.html

initiator

servicesremote calls

Which services aren’t invoked?

#Perform201748

...and else?

Thank you.