Alerting: more signal, less noise, less pain
Post on 17-Jul-2015
60 Views
Preview:
Transcript
Alert Signatures (example)
name | count ----------------------+------- Root disk space | 88 redis-queue | 71 Zombies | 50 Total Processes | 47 dispatcher | 37 pgsql backends | 35 cassandra JVM Heap | 32 SSH | 30
Naive: alert headers
Case 1: Top 5 = 25% in volumeAlert count by signature
%
01
23
45
67
Zoom on top 10
%
01
23
45
67
Sample size: 1123 alerts6 months
Case 2: Top 5 = 38% in volumeAlert count by signature
%
02
46
810
12
Zoom on top 10
%
02
46
810
12
Sample size:2324 alerts6 months
0 500 1000 1500 2000 2500 3000
020
4060
80
alert sample size
% in
vol
ume
of to
p 5
% in volume of top 5
Freq
uenc
y0 20 40 60 80
02
46
810
12
Solve the top 5 alertsand drop the volume by 20-80%Solve the top 5 alertsand drop the volume by 20-80%
Outlier (due to naive signature)Outlier (due to naive signature)
Top 5 over 103 alert streamsMin. 100 alerts per stream
Sunday
Monday
Tuesday
Wednesday
Thursday
Friday
Saturday
0
50
100
150
0
50
100
150
0
50
100
150
0
50
100
150
0
50
100
150
0
50
100
150
0
50
100
150
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23Hour of Day UTC
Aler
t cou
nt
dow
Sunday
Monday
Tuesday
Wednesday
Thursday
Friday
Saturday
Work daysWork daysZZZzzzZZZzzz
SundaySunday
SaturdaySaturday
EveningsEvenings
Steps
•Group alerts by signature
•Measure time elapsed between first and last occurrence & average/%-ile time elapsed between occurrences
•Graph
0
25
50
75
0 50 100 150 200Alert age in days
Aler
t cou
nt
Old and frequentOld and frequent
Age = days between first and last occurrenceAge = days between first and last occurrence
New and frequentNew and frequent
0
50
100
150
0 25 50 75Occurences per Alert
Aver
age
perio
d be
twee
n oc
curre
nces
Occur every 2-3 days on averageOccur every 2-3 days on average
Once in a blue moonOnce in a blue moon
Collect your alerts
• From PagerDuty (OpsGenie, Nagios, etc.)
• Import with Python (pygerduty)
• Store in PostgreSQL
Slides, Code & Data
https://github.com/alq666/velocity-ny-2013
top related