1 End-to-end Anomalous Event Detection in Production Networks Les Cottrell, Connie Logg, Felipe Haro, Mahesh Chhaparia (SLAC), Maxim Grigoriev (FNAL), Mark Sandford (Loughborough University) ESCC Meeting, Salt Lake City February, 2005 http://www.slac.stanford.edu/grp/scs/net/talk05/escc- feb05.ppt Partially funded by DOE/MICS for Internet End-to-end Performance Monitoring (IEPM)
19
Embed
End-to-end Anomalous Event Detection in Production Networks
End-to-end Anomalous Event Detection in Production Networks. Les Cottrell , Connie Logg, Felipe Haro, Mahesh Chhaparia (SLAC), Maxim Grigoriev (FNAL), Mark Sandford (Loughborough University) ESCC Meeting, Salt Lake City February, 2005 - PowerPoint PPT Presentation
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
1
End-to-end Anomalous Event Detection in Production Networks
Les Cottrell, Connie Logg, Felipe Haro, Mahesh Chhaparia (SLAC), Maxim Grigoriev (FNAL), Mark Sandford (Loughborough University)
– Three terms each with its own parameter () that take into account local smoothing, long term seasonal smoothing, and trends
12
H-W Implementation• Need regularly spaced data (else going back
one season is difficult, and gets out of sync):– Interpolate data: select bin size
• Average points in bin• If no points in first week bin then get data from future
weeks• For following weeks, missing data bins filled from
previous week
• Initial values for smoothing from NIST “Engineering Statistics Handbook”
• Choose parms by minimizing (1/N)Σ(Ft-yt)2
– Ft=forecast for time t as function of parameters, yt = observation at time t
13
H-W Implementation• Three implementations evaluated (two new)
– FNAL (Maxim Grigoriev)• Inspiration for evaluating this method
– Part of RRD (Brutlag)• Limited control over what it produces and how it works
– SLAC• Implemented NIST formulation, different
formulation/parameter values from Brutlag/FNAL, also added minimize sums of squares to get parms
14
Events• Can look at residuals (Ft – yt), or Χ2 • Could use K-S or plateau on: residuals, or on
the local smoothing (i.e. after removing long term seasonal effects)
15
Example• Local smoothing 99% weight for last 24 hours• Linear trend 50% last 24 hours• Seasonal 99% for last week• Within an 80 minute window, 80% points outside
deviation envelope ≡ event
Deviations
Observations
ForecastWeekend
Weekdays
16
Evaluation• Created a library of time series for 100 days
from June through Sep 2004 for 40 hosts• Analyzed using Plateau and saved all events
where trigger buffer filled (no filters on size of step)