Chronix: Long Term Storage and Retrieval Technologyfor Anomaly Detection in Operational Data
Florian Lautenschlager,1 Michael Philippsen,2 Andreas Kumlehn,2 and Josef Adersberger11QAware GmbH, Munich, Germany 2University Erlangen-Nürnberg (FAU), Programming Systems Group, Erlangen
Chronix: Long Term Storage and Retrieval Technologyfor Anomaly Detection in Operational Data
Florian Lautenschlager,1 Michael Philippsen,2 Andreas Kumlehn,2 and Josef Adersberger11QAware GmbH, Munich, Germany 2University Erlangen-Nürnberg (FAU), Programming Systems Group, Erlangen
AbstractAnomalies in the runtime behavior of software systems, especiallyin distributed systems, are inevitable, expensive, and hard to lo-cate. To detect and correct such anomalies one has to automati-cally collect, store, and analyze the operational data of the run-time behavior, often represented as time series. There are efficientmeans both to collect and analyze the runtime behavior. Butgeneral-purpose time series databases do not focuson the specific needs of anomaly detection. Chronixis a domain specific time series database targeted atanomaly detection in operational data.
Detecting Anomalies in Running Software matters
•Resource consumption: anomalousmemory consumption, high CPU usage, . . .•Sporadic failure: blocking state,deadlock, dirty read, . . .•Security: port scanning activity,short frequent login attempts, . . .
↪→ Economic or reputation loss.
Anomaly Detection Tool Chain for Operational Data
Collection
Framework
Analysis
FrameworkTime Series Database
Collects operational data
from a running application
Asks the database for data
and analyzes the dataStores the time series data
General Purpose TSDB
• Brake shoe
• Resource hog
• Productivity obstacle
Chronix:
Domain specific TSDB
Domain specific sensors
and adaptors
Domain specific analysis
algorithms and tools
Application’s
Operational
Data
Types of operational data:
• Metrics: scalar values, e.g.,
rates, runtimes, total hits,
counters, …
• Events: single occurrences,
e.g., a user’s login, product
order, …
• Traces: sequences within a
software system, e.g., the
called methods, …
General-purpose TSDBs in Anomaly Detection
Requirements
Graphite
InfluxD
BOpenT
SDB
KairosDB
Prom
etheus
Genericdata model # G# # G# #
Analysissupport G# G# # G# G#
Lossless longterm storage # G#
No support for data types= Productivity obstacle
No support for analyses= Productivity obstacle + Brake shoe
High memory footprint= Performance hogHigh storage demands= Performance hogLoss of historical data= Brake shoe
What makes Chronix domain specific?
Option to pre-compute an extra representation of the data
Optional timestamp compression for almost-periodic time series
Records that meet the needs of the domain
Compression technique that suits the domain’s data
Underlying multi-dimensional storage
Domain specific query language with server-side evaluation
Domain specific commissioning of configuration parameters
How it works!
Example: Almost-periodic time series
Timestamp Value Metric Process Host
25.10.2016 00:00:01.546 218.34 ingester\time SmartHub QAMUC
25.10.2016 00:00:06.718 218.37 ingester\time SmartHub QAMUC
25.10.2016 00:00:11.891 218.49 ingester\time SmartHub QAMUC
25.10.2016 00:00:16.964 218.35 ingester\time SmartHub QAMUC
… … … … …
Optional Pre-compute Extras
Timestamp Value Metric Process Host SAX
25.10.2016 00:00:01.546 218.34 ingester\time SmartHub QAMUC A
25.10.2016 00:00:06.718 218.37 ingester\time SmartHub QAMUC B
25.10.2016 00:00:11.891 218.49 ingester\time SmartHub QAMUC C
25.10.2016 00:00:16.964 218.35 ingester\time SmartHub QAMUC B
… … … … … …
B
A
C
B
C
D
C
A
B
• Lossless storage that keeps all details as analyses may need them.• Programming interface to add extra domain specific "columns".• These "columns" speed up anomaly detection queries.
Optional Timestamp Compaction
Timestamp
25.10 … :01.546
25.10 … :06.718
25.10 … :11.891
25.10 … :16.964
…
Timestamp
25.10 … :01.546
5.172
5.173
5.073
…
Timestamp
25.10 … :01.546
5.172
0.001
0.1
…
Timestamp
25.10 … :01.546
5.172
-
-
…
Timestamp
25.10 … :01.546
5.172
…
space saved
space
saved
Drop diffs
below threshold
Calculate
deltas
Compute
diffs
between
them
If accumulated
drift > threshold store delta
•Date-Delta-Compaction for almost-periodic time series.• Functionally lossless as all relevant details are kept.
• Degree of inaccuracy is a configuration parameter of
Domain Specifc Records
Process Host SAX
SmartHub QAMUC A
SmartHub QAMUC B
SmartHub QAMUC C
SmartHub QAMUC B
… … …
1Record
metric: ingester\time
process: SmartHub
host: QAMUC
start: 25.10.2016 00:00:01.546
end: …
type: metric
data: Timestamp Value SAX
25.10.2016 00:00:01.546 218.34 A
5.172 218.37 B
- 218.49 C
- 218.35 B
1
BLOB
chunk
& convert
2
Timestamp Value Metric
25….:01.546 218.34 ingester\time
5.172 218.37 ingester\time
- 218.49 ingester\time
- 218.35 ingester\time
… … …
1
1 1 2
22
• Exploit repetitiveness and bundle "lines" into data chunks.• Programming interface for a specifc time series record encoding.
• Chunk size is a configuration parameter of
Domain Specific Compression
Record
metric: ingester\time
process: SmartHub
host: QAMUC
start: 25.10.2016 00:00:01.546
end: …
type: metric
data: 00105e0 e6b0 343b 9
07bc 0804 e7d508040
Record
metric: ingester\time
process: SmartHub
host: QAMUC
start: 25.10.2016 00:00:01.546
end: …
type: metric
data: Timestamp Value SAX
25.10.2016 00:00:01.546 218.34 A
5.172 218.37 B
- 218.49 C
- 218.35 B Compressed BLOB
serialize
& compress
• Lossless compression techniques minimizes the record size.• Domain data often has small increments, recurring patterns, etc.
• Choice of compression technqiue is a configuration parameter of
Multi-Dimensional Storage
Timestamp Value Metric Process Host
25.10.2016 00:00:01.546 218.34 ingester\time SmartHub QAMUC
25.10.2016 00:00:06.718 218.37 ingester\time SmartHub QAMUC
25.10.2016 00:00:11.891 218.49 ingester\time SmartHub QAMUC
25.10.2016 00:00:16.964 218.35 ingester\time SmartHub QAMUC
… … … … …
q=host:QAMUC AND metric:ingester*AND type:[metric OR trace] AND end:NOW-7MONTH
• Explorative: Users can use the attributes to find a record.•Correlating: Queries can use and combine all types.
Query Language & Server-Side eval.
Basic GraphiteInfluxDB
OpenTSDBKairosDB
PrometheusChronix
distinct × X × × × Xintegral X × × × × X
min/max/sum X X X X X Xbottom/top × X × × X X
first/last X X × X × X. . . . . . . . . . . . . . . . . . . . .
nnderivative X X × × × Xmovavg X X × × × X
divide/scale X X × X X X
High-levelsax [33] × × × × × X
fastdtw [38] × × × × × Xoutlier × × × × × X
trend × × × × × Xfrequency × × × × × X
grpsize × × × × × Xsplit × × × × × X
queryread
resultprocess
q=metric:ingester* & cf=outlier
• Basic functions & high-level built-in domain specific functions.• Plug-in interface to add functions for server-side evalution.
Domain Specific Commissioning
0102030405060708090
0 5 10 50 100 200 1000DDC threshold in ms
Rat
es in
% −
Ave
rage
in m
s
Inaccuracy RateAverage DeviationSpace Reduction
323436384042444648
32 64 128 256 512 1024Chunk size in kBytes
Tota
l acc
ess
time
in s
ec
gzipLZ4Snappy
•DDC-Threshold: 200 ms.•Compression & Chunk Size: gzip + 128 kByte.
Easily detectpattern!
Fast!
Best Values!
Select the
ideal
Compression!
Projects 1–3
Remove
Jitter!
EvaluationBenchmark ClientUbuntu 16.04.1 x64, 12 core, 32 GB Ram,
512 GB SSD
Java
Benchmark
Benchmark ServerUbuntu 16.04.1 x64, 12 core, 32 GB Ram,
512 GB SSD
Docker InfluxDB KairosDB
ChronixOpenTSDB
Queries
Time Series
HTTP
Data of 5 Industry Projects.Project 1 2 3 4 5 total
time series 1,080 8,567 4,538 500 24,055 38,740
pairs
(mio) metric 2.4 331.4 162.6 3.9 3,762.3 4,262.6
lsof 0.0 0.0 0.0 0.4 0.0 0.4strace 0.0 0.0 0.0 12.1 0.0 12.1
(a) Pairs and time series per project.
Project r 0.5 1 7 14 21 28 56 91 1801 – 3 q 15 30 30 10 5 3 1 2 0 96
q 2 11 15 8 12 5 1 2 2 584 & 5 b 1 6 5 7 2 4 4 1 2 32
h 2 6 10 8 6 6 3 2 0 43
(b) Time ranges r (days) and occurrences of queries (q) for raw data retrieval,and of queries with basic (b) and high-level (h) functions.
Memory footprint (in MBytes)
InfluxDB
OpenTSDB
KairosD
B
Chronix
Initially 33 2,726 8,763 446Import (max) 10,336 10,111 18,905 7,002Query (max) 8,269 9,712 11,230 4,792
Chronix has a 34%–69% smaller memory footprint.
Storage demands (in GBytes)
Project RawData
InfluxDB
OpenTSDB
KairosD
B
Chronix
4 1.2 0.2 0.2 0.3 0.15 107.0 10.7 16.9 26.5 8.6
total 108.2 10.9 17.1 26.8 8.7
Chronix saves 20%–68% of the storage space.
Data retrieval times (in s)
r q InfluxDB
OpenTSDB
KairosD
B
Chronix
0.5 2 4.3 2.8 4.4 0.91 11 5.2 5.6 6.6 5.37 15 34.1 17.4 26.8 7.0
14 8 36.2 14.2 25.5 4.021 12 76.5 29.8 55.0 6.028 5 7.9 3.9 5.6 0.556 1 35.4 12.4 24.1 1.291 2 47.5 15.5 33.8 1.1
180 2 96.7 36.7 66.6 1.1total 343.8 138.3 248.4 27.1
Chronix saves 80%–92% on data retrieval time.
Times for b- and h-queries (in s)
Basic (b) InfluxDB
OpenTSDB
KairosD
B
Chronix
4 Avg. 0.9 6.1 9.8 4.45 Max. 1.3 8.4 9.1 6.03 Min. 0.7 2.7 5.3 2.83 Dev. 6.7 16.7 21.1 2.35 Sum 0.7 6.0 12.0 2.04 Count 0.8 5.5 10.5 1.08 Perc. 10.2 25.8 34.5 8.6High-level (h)
12 Outlier 30.7 29.1 117.6 18.914 Trend 162.7 50.4 100.6 30.211 Freq. 47.3 23.9 45.7 16.33 GroupSize 218.9 2927.8 206.3 29.63 Split 123.1 2893.9 47.9 37.2
75 total 604.0 5996.3 620.4 159.3
Chronix saves 73%–97% on analysis times.
Typical sce
nario.
Impo
rtan
t!
Neede
dfo
r
exp
lora
tio
n!
Evaluatio
n:
Projects
4 & 5
ConclusionChronix exploits the characteristics of the domain in many ways andthus achieves better storage and query results. Chronix is open source. www.chronix.io
AcknowledgementsThis research was in part supported by the Bavarian Ministry of Economic Affairs and Media, Energy andTechnology as an IuK-grant for the project DfD – Design for Diagnosability.
