promgen - prometheus managemnet tool / simpleclient_java hacks @ Prometheus casual

promgen - prometheus management tool

tokuhirom

Who are you?

Web Application Engineer

• Server side engineer

• Java

• Perl

• Server Admin

www1.example.com

node_exporter (httpd)

jmx_exporter (httpd)

Periodic GET request

Why Prometheus?

It’s good for lazy web programmers

Get data → Get stats

Async and lazy

Easy to integrate with grafana

But there’s no server list management tool

official method to manage servers, is writing YAML.

With consul or something

YAML file contains: exporter port number

alerting rules

Server admins need to write BIG YAML FILE.

Editing YAML via github pull-request

We want to manage it via web app.

We wrote web application to manage server list.

We are managing the server list in our deployment tool

promgen can integrate with our deployment tool

(Of course, it’s pluggable)

Note: We haven’t use consul or similar

tool(now).

Promgen

• Simple prometheus manager app

• exporter port management

• alert management

Screenshot

ProjectService

Project

Project

Host

Host

Host

Data structure

Managed by Deploy tool

Farm

Farm

Farm

Alerting

Alert Manager Hipchatpromgen-alerting

Web hook

Mail

MySQL

Architecture

• Ruby

• Sequel

• MySQL

• erubis

• sinatra

Why ruby?

I want to run it on prometheus host.

We shouldn’t use much memory

Part 2. Monitoring Java

processes via Prometheus

There are 3 patterns

1. Collect data from MBean in target JVM

App JVM (Runs jmx_exporter jetty) prom

HTTP

• Pros

• Easy to run

• Cons

• Hard to maintain YAML file

- pattern: '^com.zaxxer.hikari<type=Pool \((\w+)\)><>ThreadsAwaitingConnection'

name: hikaricp_pool_threads_awaiting_connection

labels:

source: "$1"

help: "HikariCP threads awaiting connection(pending threads) $1"

type: GAUGE

2. Collect data from JMX outside from target JVM

App JVM

jmx_exporter JVM prom

HTTPJMX

• Pros

• Easy to deploy

• Safety

• Cons

• Memory overhead

• Hard to maintain YAML file

3. Collect data by simpleclient_java

App JVM (Run simpleclient_java jetty) prom

HTTP

• Pros

• Clean

• lightweight

• no YAML required

• Cons

• Requires coding

simpleclient_java

simpleclient_dropwizard

Drop wizard metrics

MetricRegistryclient_java

simpleclient_hotspot

JVM core client_java

Default

simpleclient_log4j/logback

log4j/logback client_java

Error/Warn rate

simpleclient_springboot

Spring Boot Metrics client_java

NEW!

simpleclient_jetty

Jetty requests

status etc.

client_java

NEW!

simpleclient_hikaricp

HikariCP client_java

REJECTED!

Questions?