Time series modeling for IDS alert management

Time Series Modeling for IDS Alert Management

Jouni Viinikka, Herve DebarFrance Telecom

BP 624314066 Caen Cedex, France

[email protected]

Ludovic Me, Renaud SeguierSupelec

BP 8112735511 Cesson Sevigne Cedex, France

[email protected]

ABSTRACTIntrusion detection systems create large amounts of alerts.Significant part of these alerts can be seen as backgroundnoise of an operational information system, and its quantitytypically overwhelms the user. In this paper we have threepoints to make. First, we present our findings regarding thecauses of this noise. Second, we provide some reasoning whyone would like to keep an eye on the noise despite the largenumber of alerts. Finally, one approach for monitoring thenoise with reasonable user load is proposed. The approachis based on modeling regularities in alert flows with classi-cal time series methods. We present experimentations andresults obtained using real world data.

Categories and Subject DescriptorsC.2.3 [Computer - Communication Networks]: Net-work Operations—Network monitoring ; C.2.0 [Computer -

Communication Networks]: General—Security and pro-

tection

General TermsSecurity, Experimentation

1. INTRODUCTIONIntrusion detection systems (IDS) create often excessive

amounts of alerts, a fact acknowledged widely in both sci-ence and industry [1, 7, 12]. In this section we have a lookat some of the causes of this alert overflow and position ourwork in the alert correlation domain.

In this paper by a sensor we mean a misuse- and network-based sensor. The diagnostic capabilities of current sensorsare modest [5], and a large majority of generated alerts canbe of little or no use for the operator of the system [7]. Thischaff is generated for diverse reasons, but can be roughlydivided into four classes. The last class is the most relevantwith respect to our work, and we will describe it in moredetail. 1) The behavior model of an anomaly-based sensor

Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies arenot made or distributed for profit or commercial advantage and that copiesbear this notice and the full citation on the first page. To copy otherwise, torepublish, to post on servers or to redistribute to lists, requires prior specificpermission and/or a fee.AsiaCCS’06 March 21–24, 2006, Taipei, TaiwanCopyright 2006 ACM 1-59593-272-0/06/0003 ...$5.00.

or the attack descriptions of a misuse-based sensor can beinaccurate, creating false positives. 2) The sensor may beunable to determine whether the seen attack is a real threatin the working environment of the sensor. 3) The sensor isoften mono eventual i.e. unable to associate related attacksteps, checking e.g. just one network packet at the time.4) The sensor is alerting on packets that a) are often causedby normal system functioning, but b) can also be premisesor consequences of attacks or problems. The difficulty isthat it is impossible to make the difference using the infor-mation carried by the packet. This implies that the flow

of these alerts contains both signs of of wanted, or at leastacceptable, traffic, and possibly signs of of unwanted traffic.Typically, these alerts are not associated with any vulnera-bilities, and consequently alert verification [9] using for ex-ample the knowledge of the configuration of the informationsystem is not possible.

Some of these issues could be, at least in theory and tosome extent, addressed by improving the sensors themselves.However, in practice the data capture and analysis capabil-ities of low level sensors are rather limited because of largeamount of data that needs to be monitored in (near) real-time. In addition, the visibility of one sensor can be toolimited for diagnosing large scale attacks or using knowl-edge about the operating environment at sensor level.

Alert correlation has been proposed as one solution tothe problem [6, 8, 14, 16]. Harshly simplifying, correlationis a higher level function taking multiple alerts and addi-tional information into account to 1) reduce the alert vol-ume, 2) improve the alert content, 3) and to track attacksspreading across multiple packets or consisting of multiplesteps. The alerts can be issued by from multiple, hetero-geneous sensors, and the additional information is, for in-stance, the configuration and vulnerability scan results ofthe monitored system.

Our work focuses on processing high alert volumes pro-duced by misuse- and network-based sensor Snort. Thealerts in question can mostly be associated with the causenumber four, the monitoring of common packets related tosystem functioning. In the rest of this paper, when speakingof alerts, we mean this particular type of alerts unless men-tioned otherwise. In our operating environment we cannoteliminate this type of alerts by deactivating the signatures,so we need a methodology to process them.

The proposed methodology builds on three basic ideas:1) instead of individual alerts we process alert flows, 2) thecomponent of a flow that can be attributed to the normalsystem behavior is modeled using autoregressive time series

model, and 3) only deviations from the normal flow behaviorare reported to the operator as meta-alerts.

The remaining of the paper is organized as follows. Wedefine the problem in section 2. In section 3 we presentrelated work and position our work with respect to it. Asour approach is driven by problems detected in the wild, wespend some space presenting the used alert corpus and ourobservations made from the data in section 4. These ob-servations have given the inspiration for this work, and weconsider them important. Section 5 explains the methodol-ogy we propose to process alerts generated by prolific signa-tures. This section describes also the used time series analy-sis methods. Practical results are discussed in section 6 andfinally we offer our conclusions in section 7.

2. PROBLEM STATEMENTIn this section we define the problem we aim to solve, and

justify its significance for the users of intrusion detectionsystems.

The main objective of our work is to allow the user tofocus on more relevant tasks by relieving him from the man-ual inspection of numerous benign alerts, and still providehim with the contextual information available only in theaggregates. It has been reported [7] that as much as 99% ofthe alerts produced by intrusion detection systems can befalse positives, alerts mistakenly triggered by benign pack-ets. First of all, we see that the definition of a false positiveis a bit problematic. We do think that as such, a large ma-jority of alerts is quite useless for the operator. However,we do not consider them all as false. Consider Snort signa-tures without associated vulnerability. For example, Snorthas signatures for ICMP Destination Unreachable messages,and when the sensor triggers the alert, it is not issued mis-takenly, even though likely to be rather useless as such forthe operator. We prefer to call these kinds of alerts as thebackground noise from an operational information system,not false alerts.

It is impossible to make the difference between normal ac-tivity and attacks/problems in the way they manifest them-selves in this type of alerts by looking at the individualalerts. However, we believe that the distinction can be madeby analyzing the aggregated alert flow, and especially its in-

tensity, the number of alerts in a time unit with respect to re-cent history. Returning to ICMP Destination Unreachable

messages, unusual changes in the alert intensity can indi-cate a problem, that is possibly security related. Thus weprefer to monitor alert aggregates instead of deactivatingthe prolific signatures or filtering on alert-by-alert basis. Asmentioned before, alert verification techniques are not a fea-sible solution for this type of alerts.

In addition, we have observed in our operating environ-ment rather significant regularities in alert flows, havingmost likely non-malicious origins. Hence not any changesis alert intensity, but only changes in and deviations fromthe regular behavior are interesting for the operator, andwould require further investigation. As these regularitiesaccount for a large majority of alerts, we aim to model theregularity in order to filter it out.

Ordinary network sensors, for example information pro-vided by netflow, 1 could be used to monitor some of thesepackets at the network level as has been done in e.g. [2].

1http://www.cisco.com/go/netflow/

However, we use Snort for the following reasons:

• Even though an ordinary network sensor could be usedto monitor for example ICMP messages, and differenttypes of messages could be separated with header in-formation, this is not enough for our purposes. Weneed more fine grained alert aggregation using pat-tern matching on packet payload. For example, ICMPEcho messages can contain information on the pur-ported origin of the ping packet in the payload, whichis of interest to us.

• By its packet aggregation, Netflow creates an abstrac-tion layer that is not present when we use individualalerts from Snort. It could be seen that we are per-forming ourselves a flow level abstraction that suitsour needs better than the one provided by Netflow.

• Last, but not least, Snort is the interface to the mon-itored network at our disposal. It is a fact we needto accept in our operating environment, and we try tomake the best use of available tools.

3. RELATED WORKIn this section we present related work, in intrusion de-

tection and alert correlation domains, and point out the dif-ferences between these approaches and our work.

3.1 Intrusion detectionIn [18,19] exponentially weighted moving average (EWMA)

control charts are used to monitor Solaris BSM event in-tensity to detect denial-of-service attacks. The monitoringmethod is similar to our previous work [17], but it is appliedat sensor level, and to host based data source.

Mahadik et al. [11] use EWMA based monitoring for Qual-ity of Service (QoS) parameters, like jitter and packet droprate, in a DiffServ network to detect attacks on QoS. Thusit can be seen to use network based data source. Our ap-proach uses different data source, namely alerts instead ofQoS parameters, and the processing method is different. Wedo apply a similar detection method for anomalies, but wepre-process the alert series and model the normal behaviorof the alert flow differently.

3.2 Alert correlationAmong the several proposed correlation methods and tech-

niques, we consider the following being related to our work.Qin and Lee [15] use Granger Causality Test to find un-known relationships in alert data. The time series data isformed as the number of occurrences of meta-alerts in a timeunit, and they search causal relationships between the se-ries. The meta-alerts are formed by aggregating alerts shar-ing other attributes, as defined in IDMEF, than the timestamp, therefore they use more fine grained aggregation cri-teria than we. We filter out the normal component in onespecific type of alert flow instead of trying to track attacksspreading across multiple steps (i.e./ alert series). In otherwords, the objectives and the used techniques are different.

Julisch [7,8] uses data mining to find root causes for falsealerts, and proposes to filter those alerts when it is not pos-sible to fix the root causes. The filters are based on alertattributes and work on alert-by-alert basis. Firstly, changesin the alert intensity created by the root cause can be in-teresting, and this information is lost, if such filters are in

Table 1: Five most prolific signatures in the first data

set

signature SID # of alerts

SNMP Request 1417 176 009Whatsup 482 72 427ICMP Dest Unr 485 57 420LOCAL-POLICY - 51 674Speedera 480 32 961sum 390 491

place. Secondly, for some alert flows, there is no finite set ofroot causes that could be easily filtered. Episode rules [13]could provide visibility over several alerts, but in [8] theywere reported as error prone and laborious approach.

4. ALERT CORPUSIn this section we describe the used alert corpus used in

the tests. We will also provide justification for the narrowscope of the approach, define different flow profiles, and an-alyze five high-volume alert flows in detail.

The data set consists of alerts generated by three Snortsensors deployed in an operational information system, onecloser to the Internet, and two in more protected areas. Thesensors store the alerts in a database on a remote server, andour processing component is situated on the server. As themonitored traffic was real, the absolute truth concerning thenature of observed traffic remains unknown. On the otherhand, alerts from a real system contain a wealthy amountof background noise, which is challenging for the correlationmethods and might be difficult to simulate. The data is thesame that was used in [17], and the main motivation for thiswork was the shortcomings of the model used in that paperwith certain types of alert flows. These shortcomings arediscussed in more detail in section 6.

The data was accumulated over 42 days, and containedover 500k alerts. The sensors had the default set of signa-tures activated, at that time approximately 2000 signaturesplus some additional custom signatures. 315 signatures hadtriggered alerts, and only five had generated 68% of the to-tal number of alerts. Table 1 shows these five signatureswith their Snort IDs (SID) and the number of alerts gen-erated by them. All these five react on packets, that canbe generated by normal system functioning. Therefore weconsider it worthwhile to focus on processing this type ofalerts. This reasoning may be specific to the informationsystem in question. However, the reported examples, suchas in [8] would suggest that this could apply also to a widerrange of information systems.

4.1 Profiling alert flowsWhile investigating collected alerts, this corpus and gen-

erally, we have identified four flow profiles for alerts gener-ated by prolific signatures. Profiles are defined according totheir regularity and our capability to explain seen behavior,normal or anomalous.

A Known, constant: The alert generation is almost con-stant. The flow has relatively few anomalies that wecan explain and attribute to 1) change in the interac-tion configuration 2) a problem.

B Known, periodic: The alert flow contains clearly vis-ible periodicity and possibly a constant componentwith a benign origin. The flow contains some anoma-lies, of which we can explain a majority.

C Unknown, periodic: The alert flow is less stable thanin class B, it has more anomalies visible and we do notknow how to explain them.

D Unknown, random-like: The flow seems to be more orless random, only very little or no structure is visiblewith plain eye. We have only limited explanations forthe origins of these alerts.

The regularity and explicability span two axes, and theplacement of classes with respect to them is depicted inFig. 1. None of these four classes falls into the second quad-rant, as those alerts are usually associated with a vulnera-bility, and are either true manifestations of attacks or falsepositives. Most correlation work focuses on processing thistype of alerts via techniques like alert fusion, verification,and finding the prerequisites and the consequences amongthem. In the first quadrant we have two different classes,A and B. The reason in having two separate classes is thethe different type of regularity, constant and periodic, ofthe classes A and B, respectively. Packets causing constantalert flows have typically machine related origins given theirclock-like behavior, whereas humane activity can create pe-riodic behavior. For example, if the network traffic trig-gering the alerts is created by actions of a large number ofpersons, natural rhythms with period of one day or week arelikely to exist in the alert flows.

4.2 Alert flow analysisNow let’s have a closer look at the five signatures and the

alert flows they generated. For each, we give a short de-scription, enumerate the identified interesting phenomena,and provide explanations for the phenomena when we havethem. Actual signature documentation can be found on theSnort web site2 with the Snort ID. Figure 2 shows the alertintensities as the function of time for the measurement pe-riod. Dotted lines show the division to estimation and vali-dation data, as will be explained in section 6.

� � � � � �� � � � � �� � � � � �� � � � � �� � � � � �� � � � � �

� � � � � �� � � � � �� � � � � �� � � � � �� � � � � �� � � � � �

nonexplicable

explicable 1

3 4

B

regularitylowregularity

high

D

C

A

2

Figure 1: The alert classes with respect to the two

axes, regularity and explicability

2http://www.snort.org

SNMP Request UDP reacts to Simple Network Man-agement Protocol (SNMP) request from external sources to-wards internal hosts.

As the source address is external, the request messages arelikely caused by a misconfiguration outside the operator’scontrol. The alert flow is extremely regular, as can be seenin Fig 2(a), with few peaks and valleys plus some smalleranomalies. In this case, there were few particular sourceaddresses responsible for the large bulk of alerts i.e. theroot causes were identified, and alerts generated by thosenodes could have been filtered out. However, in that casethe operator would have also missed the sharp change in thealert intensity marked with p1.

We identified five interesting phenomena in the flow. Thefirst, p1 is a huge peak, likely caused by a change in theinteraction configuration at the external source, since it alsointroduced an increase in the constant component of theflow. Then both p2 and p4 are smaller peaks, and p3, p5

are drops, probably due to connectivity problems, in theotherwise extremely constant alert flow.

There was also low intensity activity that is almost in-visible in the picture. This type of low-level activity wouldbe easily lost among the bulk of alerts if the alerts wereprocessed manually. In the case of completely shunning thesignature this activity would have been simply impossible todetect.

We place this alert flow in the class A because of its con-stant and explicable nature.

ICMP PING WhatsupGold Windows is triggered byICMP Echo messages with a payload claiming them to becreated by a performance measurement tool3. The sourceaddress is by definition external, so the root cause for thesealerts is likely to be out of operator’s sphere of influence.

In this flow, visible in Fig. 2(b), we see a constant compo-nent, and on top of which we have a periodic component cre-ating peaks on five workdays but no additional alerts duringthe week-ends. Alerts generated by these two componentsare regular and considered as legitimate, actually originat-ing from the usage of the performance measurement tool.The tool includes a server polling the the network nodes tofind out their availability, and thus the normal flow behavioris independent of the general network traffic volume in con-trast to the case speedera. This automated origin explainsalso the high regularity of the daily and weekly variationsand the existence of the constant component.

The interesting phenomena are the shifts in the constantlevel. Phenomena p1, p4, p6 and p7 are drops, and p2, p3,p5 and p8 are increases in the constant component intensity.As with the previous flow, these are changes in the alertgeneration rate, and filtering based solely on alert attributeswould prevent the operator from seeing these changes.

We place this flow in the class C because of its periodicnature, and since even though we know the origins of thenormal behavior, it is difficult to enumerate them for theanomalies.

ICMP Destination Unreachable Communication

Administratively Prohibited reacts to particular ICMPmessages normally generated when a network node, e.g. arouter, drops a packet for administrative reasons.

For this flow, shown in Fig. 2(c), there are several possi-ble causes for the alerts like backscatter from DDoS attacks,

3http://www.ipswitch.com/Products/WhatsUp/professional/

network outages or routing problems. Given the erratic pro-file, it is likely a combination of all these and more. In otherwords, we could not define the root causes. These messagesare like background radiation, almost any network is likelyto have them in the inbound traffic as hosts within try toconnect to invalid/non-accessible addresses. Consequently,filtering based on individual alerts would be difficult. Theflow profile does not contain as much structure as the oth-ers. When smoothing the intensity with EWMA some struc-ture came visible, indicating a rather weak week rhythm. Acorrelation between the network usage and incoming ICMP

Destination Unreachable seems quite logical, and wouldexplain the week rhythm.

We place this flow in the class C, as we cannot associatethe high variability nor the anomalies to a small set of rootcauses.

LOCAL-POLICY External connexion from HTTP

server is a custom signature with quite self-explicative name.The rationale for this signature is that as all the back-endservers are internal, a web server should not be on the SYNside of a TCP handshake with external hosts, unless for ex-ample infected by a worm. Nonetheless they did and up tothis day we have not been able to explain this behavior.

The alert intensity is depicted in Fig. 2(d). Even thoughthe profile consists of impulses, some periodicity can befound. There are peaks of thousands of alerts following aweekly rhythm, and smaller peaks following daily and evenshorter cycles. In addition there are anomalies: p1 is missingand p3 is a change in the low-level activity. The phenomenonp2 is an extremely high peak with respect to past observa-tions, and p4 and p5 are large peaks breaking the normalrhythm.

We place this flow in the class C as we lack explana-tions for both normal and anomalous behavior, and sincethe peaks have periodic nature.

ICMP PING speedera triggers on ICMP Echo mes-sages carrying a payload claiming the packet to originatefrom a server belonging to content distribution companySpeedera4 trying to determine the closest cache to the hostrequesting content.

The number of these messages is proportional to the ac-cesses to sites hosted by the company. Consequently theflow, shown in Fig. 2(e), has strong periodic component withpeaks during working hours and valleys during night timeand weekends. We consider the periodic component as nor-mal system behavior. Two anomalies were identified, p1 isa peak during high activity hours, and p2 is a peak duringlow activity period. These are likely to be anomalies causedby an increase in the legitimate traffic, related to somethinglike the release of Pirelli’s calendar on a server hosted bySpeedera.

We place the flow in the class B because it is highly peri-odic, and as we are able to explain both the normal behaviorand the anomalies.

Next some conclusions we have made from the analysis.In general, large amounts of alerts are not false positives inthe strict sense, but background noise generated by prolificsignatures reacting to the normal functioning of the system,as discussed earlier. Referring to the noise source classes ofsection 1, these alerts fall into category four. The interestingphenomena in these flows are often intermittent, and con-

4http://www.speedera.com

sist of 1) changes in the intensity generated by the normalsystem behavior or by so called root causes, and/or 2) ap-pearance of abnormal sources, i.e. something else than theroot causes, both representing typically intermittent activ-ity. We need flow level filtering since alert by alert filteringgenerally loses the case one and without any filtering thecase two risks to be lost in the noise.

We wish also to point out that four out of five signa-tures trigger alerts almost in a non-ending stream. Withoutfurther automation of alert processing, the operator wouldneed to be analyzing these alerts constantly. This would bea tedious task, requiring at least time and likely to strainanalysts’ mental resources.

Based on our observations we suppose that the regular-ity and the stationary structure present in these alert flowshave benign origins. By definition stationary structure hasexisted in the past and has remained unchanged. Thus itshould be normal also according to the underlying hypoth-esis of anomaly detection.

5. METHODOLOGY FOR ALERT FLOWMODELING AND FILTERING

In this section we propose a methodology to model andfilter alert flows. To present our observations in a moreexact manner, we will use some notation and concepts fromthe field of time series analysis, mostly according to [3, 4].It should be mentioned that the theory and methods onwhich we build are classical and simple, our intention is tomap them to alert processing and to our problem, and thento propose a methodology to apply these methods. As themethods are simple, they are likely not the most suitable forall alert flows. However, this is our first trial with techniquesfrom the time series analysis discipline, and we wanted tosee first what we can achieve with these simple methods.In addition, obtained results will serve as a yard stick tomeasure the improvements in future work. We begin withan overview of the proposed methodology, and then describeeach step in more detail.

5.1 OverviewAn alert flow is the stream of successive alerts meeting

the aggregation criteria. We aggregate alerts according tothe signature that generated the alert, and view the flowintensity measurements taken at fixed intervals as a timeseries St. The objective is to filter out from this flow or timeseries the components that correspond to normal activity ofthe information system.

A discrete-time time series model for a set of observations{xt}, is a representation of the joint distributions of a se-quence of random variables {Xt} of which {xt} is postulatedto be one realization. Each xt is recorded at time t, wherethe set T0 of observation times is a discrete set. Here weconsider only observations made with fixed time intervals.In practice, the specification of joint distributions is rarelyavailable, and unfeasible to estimate, only first- and secondorder moments of the joint distributions are used.

We use a model St for alert series

St = Xt + Et , (1)

where Xt represents the part of the alert flow caused by thenormal activity, and Et the remaining i.e. the abnormalpart of the alert flow. As mentioned before, as the result

of alert flow analysis, we consider regularity and stationaryphenomena in alert flows as part of normal system behav-ior. Whereas the intermittent, non-stationary phenomena inalert series we regard as caused by abnormal system behav-ior. This abnormal behavior can be attacks or more generalproblems. The distinction between alerts related to normaland abnormal activity is invisible at the alert level, but canbe seen at the flow level.

According to the classical decomposition model [3] a timeseries Xt can be considered to contain three, trend, periodic

and random, components

Xt = Tt + Pt + Rt , (2)

where Tt is a slowly changing trend, Pt is the periodic com-ponent with a known period d, and Rt is random station-ary component. Note that Rt is not necessarily completelyrandom in the common sense of the word, but can containstructure that does not fit into Tt nor into Pt. In the litera-ture Rt is also called the noise component, but to avoid thecollision with our term background noise, we prefer to userandom component for Rt.

For example, Tt would correspond to the slow changesin the alert intensity caused by the overall changes of themonitored traffic over the time. Pt would correspond to theperiodicity in alert flows created by periodic system activ-ity related to working hours and working days. As a morespecific example, ICMP Ping speedera has a strong periodiccomponent, see Fig. 2(e). Less trivial behavior that is stillpresent and invariant over a longer time would correspondto the structured part of Rt.

To let the operator focus on alerts warranting further in-vestigation, we filter out alerts conforming to the descriptionof normal system behavior i.e. the components of alert flowthat fit into the Xt of the equation (2). The abnormal com-ponent of an alert flow is the residual series Et = St − Xt,and only most significant phenomena of Et are signaled tothe operator as meta-alerts.

Figure 3 depicts the steps in this process. The transfor-mation from data to series S is in this case trivial, as we onlycount the number of alerts matching the aggregation criteriain a time unit, one hour. We acknowledge that the choice ofthe time unit affects especially the timeliness of the detec-tion and the visibility of certain phenomena in alert flows.Given the noise-like nature of monitored alerts, we see thatthe hourly measurements are sufficient. We also consideronly univariate series even though other transforms creat-ing a multivariate series from alert data to series could beenvisaged. In 5.2 and 5.3 we describe how to remove Tt andPt, respectively, from St to obtain first detrended series S′

t

and then S′′

t from which the periodic component is removed.Now the series S′′

t contains the random, Rt, and the abnor-mal, Et, components. The structure in Rt is captured in atime series model, described in 5.4, and Et is obtained as the

difference between model output St

′′

and the observationsS′′

t . Finally, the detection of most significant anomalies isdetailed in 5.5.

5.2 Removing the TrendFirst of all, we assume that the trend Tt possibly present

in alert flow is linear. The reasoning behind this is basedon the common sense and on our observations. If the trendis of higher degree, then there is a serious problem with thesensors, and the alert storage capacity is running out very

10

100

1000

Feb−08 Feb−15 Feb−22 Mar−01 Mar−08 Mar−15

Ale

rts p

er

hour

Date

p1

p3

p5

p2p4

(a) SNMP Request, Class A

1

10

100

1000

Feb−08 Feb−15 Feb−22 Mar−01 Mar−08 Mar−15

Ale

rts p

er

hour

Date

p6

p8

p1p2

p7

p3

p4 p5

(b) Whatsup, Class C

1

10

100

1000

Feb−08 Feb−15 Feb−22 Mar−01 Mar−08 Mar−15

Ale

rts p

er

hour

Date

(c) Dest. Unr., Class D

1

10

100

1000

10000

Feb−08 Feb−15 Feb−22 Mar−01 Mar−08 Mar−15

Ale

rts p

er

hour

Date

p3

p4 p5

p1

p2

(d) LOCAL-POLICY, Class C

1

10

100

1000

Feb−08 Feb−15 Feb−22 Mar−01 Mar−08 Mar−15

Ale

rts p

er

hour

Date

p1

p2

(e) Speedera, Class B

Figure 2: Hourly alert intensity for five most prolific flows aggregated by signature. Horizontal axis is the

time, and the vertical shows the number of alerts per hour in log-scale. The dashed line shows the division

to estimation and validation data used for experimentations. The arrows point out phenomena in which we

are interested in

alertsdetection

stationarystructureremoval

stationarymodel of

structure

s′′

1es

′

1s1

∇d

s′′

1

AR...

datatosignal

.

.

.

s1

sn

∇1

trend removaltrend removal

Figure 3: Diagram of the detection process.

0 5 10 15 20 25 30 35 40 45−800

−600

−400

−200

0

200

400

600

800

1000

Figure 4: SNMP Request UDP after detrending.

The constant component has been eliminated, but

the changes of constant level remain visible

soon. This cannot be the normal situation, and if happen-ing, we do not wish to eliminate such information from theseries.

There are several ways to remove the trend and the pe-riodic components from time series [4]. We have chosen touse lag-d differencing operator, ∇d, for two reasons. First,it does not require the trend to remain constant over time,and second, it does not require estimation of several param-eters. An additional plus is that it can be applied to removeboth the trend and the periodic components. It is definedas

∇dXt = Xt − Xt−d . (3)

In other words, the resulting series is the difference betweentwo observations of the original series, d time units apart.With d = 1 this is the analogy of the derivation operator forcontinuous functions.

When applying ∇1 to a linear trend Tt of form Tt = bt+c,we obtain ∇1Tt = Tt − Tt−1 = bt + c − (b (t − 1)) + c = b,the slope of the trend function. If viewed as the deriva-tion, this step leaves us with a series representing the rateof change in the alert flow. For example with SNMP Request

UDP, this step removes the constant component, visible inFig 2(a), and makes the level shifts, and thus the interest-

ing phenomena, more apparent. The transformed series, i.e.the one consisting of the values S′

t of Fig. 3, is showed inFig. 4.

We apply ∇1 to all series. This will cost us a loss ofinformation contained in the series, but for the anomalydetection the linear trend and the absolute value of the alertintensity are not necessary.

5.3 Removing the PeriodicityAs mentioned above, the ∇d operator can be used also

to remove the periodic component from time series. Theapplication of ∇d to the model Xt of the equation (2) with-out the trend component, which can be removed as shownabove, and where Pt has the period d, results to

∇dXt = Pt − Pt−d + Rt − Rt−d = Rt − Rt−d . (4)

After this operation, were left with a random component(Rt − Rt−d). The application of ∇d requires the knowledgeof the period d. We will develop this point further in sec-tion 6, for the moment let us assume that it is known.

We decide whether ∇d is applied to a series based on sam-ple autocorrelation function values of S′. It can be shown [3,p. 222] that for a series without any structure, about 95%of the sample autocorrelations should fall within the bounds±1.96/

√n, where n is the number of samples. If the sample

autocorrelation values for the lag d are outside these limits,we apply ∇d to the series.

5.4 Removing the Stationary StructureTo remove the remaining stationary structure Rt from the

series we build a model of this structure. We then use theseries from which we have removed trend and periodicity,S′′

t , as input to the model and then remove the model’s

output St

′′

from S′′

t . This leaves us the abnormal componentEt.

The structure in Rt is captured in an AR(p) time seriesmodel, defined as

Rt =

pX

k=1

φkRt−k + Zt , (5)

where {Zt} ∼ WN`

0, σ2´

i.e. white noise. In English thismeans that the value of Rt at the current instant t is ob-tained as a sum of two terms, the weighted sum of p previousobservations of Rt, and the noise term.

The model of equation (5) is a parametric model, and thuswe need to 1) choose the model degree p, and 2) estimate

parameters φk for k = 1, . . . , p before we can use the model.Generally, the choice of p is done by building different mod-els, and choosing the “best” according to some criterion.In our case this is a decent compromise between interestingphenomena versus the non-interesting signaled by the modelusing the estimation data.

The parameter estimation is done using an algorithm basedon least squares. In brief, this means that the chosen param-eters minimize the square of difference between observationsand the prediction done by the model.5

If the model of the equation (2) would describe the normalcomponent of the alert series St sufficiently, after the trendand the periodicity have been removed, the need for furthermodeling with AR models could be defined by testing thewhiteness of the remaining series for example with Ljung-Box test [10]. If S′′ resembles white noise, there is no morestructure to be modeled, and this step is of no use. However,as the real world alert data contains anomalies, and model ofthe equation (2) is not perfect, this approach is less feasible.For the moment we build the AR model for every series,which can cause some unnecessary computational overhead.

This step differs from the previous ones since we use train-ing data for the parameter estimation. Once the training hasbeen done and the parameters have been estimated, we canuse each new observation S′′

t as input to the model, and ob-tain Et as difference between the observation and the modeloutput St

′′

.

5.5 Anomaly DetectionAfter these steps, we have isolated the abnormal compo-

nent Et of the alert series. If the original series St doesnot contain any anomalies, and our model of normal be-havior Xt is exact, the abnormal component is white noise,Et ∼ WN

`

0, σ2´

.However, in reality usually neither is the case. Anoma-

lies in alert series and model insufficiencies, both at con-ceptual level and in the parameter estimation, mean thatEt is not white noise. To avoid signaling artifacts causedby model deficits and random variations, we pick out onlythe most significant changes in Et. An anomaly is signaledif the current value et differs more than n standard devia-tions from the average of the past values. The default valuefor n is three, but it can be adjusted to increase or reducethe number of anomalies signaled. The average of past val-ues and standard deviation are estimated using exponen-tially weighted moving averages. The detection method isdescribed with more details in [17].

The knowledge concerning the normal activity could beinteresting as such, as it can help the operator to betterunderstand the monitored system. However, this kind ofanalysis would need some level of understanding of the un-derlying theory from the operator. To keep things simplewe present him only the most significant phenomena of theabnormal part of the alert flow.

6. RESULTSIn this section we present the results obtained when we

applying this methodology to the alerts described in sec-tion 4. The first part of alert corpus, 406 observations, wasused as estimation data for AR model parameters, and the

5For details, see http://www.mathworks.com/access/helpdesk/help/toolbox/ident/arx.html

0 1 2 3 4 5 6 7 8 9−0.2

−0.1

0

0.1

0.2

0.3

0.4

0.5

Sam

ple

AC

F

Lag

Figure 5: The sample autocorrelation values for

Whatsup after trend removal

Table 2: Strongest periods found by algorithm

Flow Lags (hours)

SNMP - - - -Whatsup 168 24 23 15Dest Unr 144 72 48 24LOCAL-POLICY 168 167 24 24Speedera 24 12 10 2

latter part, 600 observations, was left for validation. Thetool was implemented using Matlab.

6.1 The Periodicity in Alert FlowsTo remove the periodic component Pt with a period d

using ∇d, we need to know d. Visual inspection of alertseries and the sample autocorrelation function (ACF) valuesas well as the intuition suggested periods close to one day orweek. Figure 5 shows the sample ACF values for Whatsup

up to a lag of nine days. The dashed lines show the 95%confidence interval for sample ACF values of white noise.One can see that there is rather strong positive correlationfor one and seven days shifts as the highest sample ACFvalues are at lags corresponding to one week and one day.

We also used an algorithm that removed the periodic com-ponent corresponding to the lag of the largest absolute valueof the sample ACF, and then worked down recursively to-wards shorter lags. Table 2 shows the first four lags usedby the algorithm for the periodicity removal. The first i.e.the strongest periodic component removed had the perioda multiple of 24 hours for all but SNMP Request udp, whichdoes not have any periodic component present as can beenseen in Fig 2(a). If the alert series showed significant au-tocorrelation, it had strong weekly and daily components.This observation confirmed the intuition we had had con-cerning the periodicity in the alert flows.

We chose to use ∇d only with d corresponding to a week,if at all, for the following reasons: 1) every application of∇d suppresses information from series, 2) causes the loss offirst d observations, since there is not enough historic datato apply the operator, 3) applying ∇d with d corresponding

to a day or a week removed the majority of the strong au-tocorrelations, and 4) the autocorrelations with the lag of168 hours (a week) were among the strongest, As explainedin section 5.3, the sample ACF value corresponding to a lagof one week is used to determine whether ∇d is applied to aflow.

6.2 Defining the model degreesWe used several model degrees p, namely 4, 10, 16, 26 with

the estimation data to find out the most suitable for eachflow. In addition to AR models, a range of ARMA(p, q)models were estimated. At least with the used model de-grees and estimation methods they did not give significantimprovement if any over the AR models but their estimationis more resource consuming.

The p was chosen such that we could detect as many aspossible of the interesting phenomena described in Sect 4.2,and have as few meta-alerts issued on the non-interestingflow behavior. According to anomalies signaled from esti-mation data, we chose the model degrees as follows: SNMP

AR(4), Whatsup AR(26), Dest Unr AR(26), LOCAL POLICY

AR(26), and speedera AR(26).

6.3 Detected anomaliesAfter the model degrees were fixed and parameters esti-

mated from the estimation data, we applied the completealert processing method to the validation data. Next we ex-amine in detail the anomalies signaled for each flow by thechosen models. The overall statistics can be seen in Table 3.For each flow the number of signaled anomalies is in columnAn. The detection of known and interesting phenomenapi is covered in columns K+ and K-, showing the signaledand missed phenomena, respectively. The pi were identifiedin section 4.2. Note that not all pi are in the validationdata and K- does not contribute into An. In addition, thetool signaled also new anomalies. These can be 1) new, in-teresting phenomena unidentified in the manual inspection,2) uninteresting phenomena that at the first glance couldseem somewhat significant, but actually are part of the nor-mal behavior, 3) artifacts created by the transforms we haveperformed in the processing chain, like the use of ∇week op-erator. The case one phenomena are useful to the operator,and the number of occurrences is recorded in column N+.The case two anomalies are rather harmless, and usuallyquite easily identified as such. However, they do waste op-erator’s time. The case three is the worst, as the tool signalsartificial anomalies where they do not exist. Depending onthe alert flow, their correct identification can be trivial orquite difficult. The number of occurrences for the two latteris recorded in column N-. For speedera we depict the N+and N- in Fig. 6 We used also the EWMA model of [17] onvalidation data for comparison.

SNMP All the four known phenomena in the validationdata were signaled, and all the additional anomalies are realanomalies, small vibrations on top of the constant alert flow.In that sense they are not harmful at all, we just consideredthe eleven others less interesting than the largest ones. Thenew phenomena signaled can be manifestations of SNMPtraffic injection, or harmless but intermittent behavior inthe otherwise extremely constant flow.

For this flow the EWMA approach provided the samemeta-alerts, which is not surprising given the constant na-ture of the alert flow.

Table 3: Signaled and missed phenomena

Flow An K+ K- N+ N-

SNMP 15 p2, p3, p4, p5 - 11 0Whatsup 11 p2, p3, p6, p8 p4, p5, p7 2 5Dest unr 12 - - 3 9Local policy 12 p2, p4, p5 p3 4 5Speedera 5 p1, p1 - 1 2Total 55 14 4 21 21

Whatsup The known phenomena p2, p3, p6, and p8 weredetected, but p4, p5, and p7 were missed by the tool. Allthe phenomena are shifts in constant level activity. Thecommon factors for K- are the smallness of the shift andthe fact that the shift is placed after other anomalies. Thereason for this will be discussed in more detail in section 6.4.

The two new interesting anomalies are actually the de-layed detection of missed p4 and p7 as the tool reacts to thechanged height of a weekday peak after the constant leveldecreases.

Of the five N- occurrences, two are extra anomalies orduplicates, signaled right after an interesting known or newphenomenon. The remaining three were issued due to changedintensity of the weekday peaks compared to the observationsof the previous week. We did not consider these interestingenough, and placed them into this category. As such, thesephenomena highlighted by the tool are not harmful, or caneven be useful for the operator.

Compared to the EWMA approach, the signaled anoma-lies are almost the same. However, the model of normal be-havior is more accurate, and even the missed known anoma-lies can be seen in the residual series Et even though thedetection method did not pick them out.

Destination Unreachable This flow did not contain anyknown, interesting phenomena, only a rather weak weeklyrhythm. The tool signaled in total twelve anomalies.

We placed three into the interesting category, as theypointed out overall intensity changes in both high (week-day) and low (weekend) activity periods with respect to thesituation week before. Of the nine N- anomalies, two wereclearly artifacts created by ∇week. They were relatively easyto identify as artificial phenomenon.

For the remaining seven we could not give an explanation.At first this might seem like a disappointing result. However,it is also interesting to note that the periodicity removal stepexplains quite many peaks of the original flow, which couldhave seemed suspicious by manual inspection. Even thoughif we only signal three interesting phenomena, we are ableto explain many more as part of the weekly rhythm.

The EWMA approach signaled over 50% more anomalies,mostly peaks part of the weekly rhythm. It was unable topoint out the level shifts we found out using the proposedmethodology.

LOCAL-POLICY The known, interesting phenomenap2, p4, and p5 were signaled, and p3 was missed. Given thatthe missed p3 is a change in high frequency peak intensitylevels from 1 to 2, among the peaks reaching up to 10000alerts, we do not consider this as a serious shortcoming.

The tool pointed out four new, interesting phenomena.These were additional or missing peaks breaking the weeklyrhythm, or changes in the intensity of periodic peaks. Look-ing the situation a posteriori, these seem quite evident, but

before more thorough analysis of the flow structure thesewere not so easy to pick out.

The five N- contain two artifacts generated by ∇week, twoanomalies signaled on extremely high (3000 and 8000 alerts)peaks that however are part of the weekly rhythm, and oneanomaly that could be caused by variations in the low-levelcomponents. These variations are interesting, but as wecannot be sure that the anomaly is really signaled because ofthese variations, it was assigned in N-. As the two artificialanomalies were easy to identify, the N- category for this flowcontains rather harmless meta-alerts.

Even though the alert flow is quite challenging to pro-cess, consisting mainly of huge alert impulses, we are ableto pick out the known interesting phenomena. In addition,the analysis performed with this methodology helped us tobetter understand the structure in the alert flow by signalingnew anomalies, and by not reacting to certain peaks.

The EWMA approach had difficulties particularly withthis flow. In practice it signaled every peak, and was unableto leave out the peaks following the weekly rhythm.

Speedera The flow is depicted with the signaled anoma-lies in Fig. 6. It contained only two known, interesting phe-nomena, p1 and p2, both of which were detected.

The N+ contained only one alert n2, pointing out a lower-than-normal intensity Friday.

In the N- two signaled anomalies were related to arti-facts created by ∇week. Both known anomalies are echoedin transformed series ( s′′1 of Fig. 3) one week later, p1 caus-ing n3, and p2 causing n4. Knowing the behavior of the∇week, these artifacts were easy to identify in this flow.

The phenomenon n1 in the beginning of the series is anartifact of two adjacent anomalies created by the two facts,1) an AR(p) model starts to provide good predictions start-ing only from (p + 1)th observation fed into the equation (5),and 2) the detection component needs to receive more datafor correct thresholding. This type of artifact is created bothwith this methodology and with the EWMA approach forevery flow. We can ignore these anomalies systematically,and thus have excluded them from the statistics.

We chose to visualize speedera’s results as they containgood examples of unwanted artifacts that can be created by∇week. The results also show how well we are able to filterout the alerts being part of the normal flow behavior.

The EWMA approach signals an anomaly only in the be-ginning of every Monday, when the periodic intensity in-crease takes place. As such it is unable to both cope withthe strong periodic component, and to detect any non dras-tic changes in the flow behavior.

6.4 DiscussionOverall, we can model regular and periodic behavior in

alert flows, and we detect sharp and impulse-like peaks andvalleys (all five flows) outside the day and the week rhythmsof the alert flow. On the other hand, normal behavior ofalert flows consisting only of impulse like peaks, e.g. LOCALPOLICY, is difficult to capture with proposed model. Wecan also detect, up to certain degree, shifts in the constantcomponents (SNMP and Whatsup) and the overall intensity(Dest. Unr.). However, the series detrending makes thedetection of these shifts more difficult, as only the transitionremains visible afterwards.

Preceding variations in the residual series (abnormal com-ponent) can mask the current anomaly because the detec-

0 5 10 15 20 250

100

200

Time (d)

Inte

nsity

0 5 10 15 20 250

0.5

1

n1 n3n2

p1

p2 n4

Figure 6: The detected anomalies for speedera. The

series is the original alert flow, corresponding to the

values s1 of Fig. 3 and signaled anomalies are indi-

cated as peaks

tion threshold is based on the standard deviation. Thereforesmall level shifts, like the ones in Whatsup, are in dangerto be lost when alert flow is not very regular or constant.However, the shifts are present in the residuals, and couldbe picked out with other means.

In other words, we could possibly catch more anomaliesby developing the detection component of the processingchain (Fig. 3) or just by customizing the smoothing factorand the alerting thresholds, However, the current approachworks sufficiently well yet being rather generic, and generictools are easier to deploy. For these reasons we regarded theextra step not worth the trouble.

Now coming back to our main objective: allowing the op-erator to focus on more relevant tasks by relieving him fromthe manual inspection of numerous benign alerts. As we areprocessing alert flows and the meta-alerts point to a time in-terval, a direct comparison of alert numbers before and afteris not the most suitable metric to use. If investigating thealerts manually, the operator is probably not monitoring thistype of alerts constantly, but more likely checks from timeto time if the situation seems normal or not. Let us assumefor the sake of the discussion that from time to time is oncein an hour. The operator would need to perform at least aminimal check every hour during which alerts are generated.Since in normal situation alerts occur all the time, also zeroactivity intervals could incite additional checks.

Table 4 shows the number of meta-alerts issued by the toolin the second column and the number of non-zero intervalsrequiring inspection without automated processing in thevalidation data for each flow in the third column. The fourthcolumn shows the time gain as the proportion of these onehour time slots freed from manual inspection when alertprocessing is automated. The proposed approach can relievethe operator from 90% or more of the status checks, andthe gained time can be spent on more relevant tasks. Inaddition, the analysis done by applying the methodologycan point out phenomena that would be difficult to see viamanual inspection, even if looking at the alerts at the flowlevel.

Table 4: The gain in time slots with the methodol-

ogy

Flow Automated Manual Gain

SNMP 15 564 0.97Whatsup 11 390 0.97Dest Unr 12 556 0.98LOCAL-POLICY 12 118 0.90Speedera 5 518 0.99

On the negative side, the methodology signals few artifi-cial phenomena, but they seem to be usually rather easy toidentify. The approach also misses some interesting phenom-ena, typically small changes close to larger disturbations.These costs need to be weighed against the gain. From theresults we can see, that the number of false positives is rel-atively small. However, we cannot provide comprehensivestatistics on the quality of detection as we lack analyzedand labeled real world alert sets.

The proposed methodology has been tailored for our needs,for the alert types and volumes we encounter. Part of thereported flow behaviors are likely to be specific to our en-vironment. This is not a general solution for all alert pro-cessing, but a component for a specific task in larger alertmanagement system. However, we believe that similar prob-lems exist also in other information systems, and that thisprocessing methodology can be useful also elsewhere. Itshould also be pointed out that like all anomaly detectionapproaches, the methodology does not provide diagnostic ofthe meta-alert’s cause, and it is not even intended to helpin root cause analysis.

7. CONCLUSIONSIn this paper we have presented an analysis of real world

alert data. The alerts consisted mainly of types that canbe considered as background noise of an operational infor-mation system. Among the noise there can be premisesor consequences of attacks or more general problems, andtherefore monitoring this kind of background noise can beinteresting with suitable methods, despite the high alert vol-umes. The analysis revealed significant structure in the alertflows, both visually and with basic mathematical tools.

Based on the observations, we proposed a methodologyto process these alerts in order to highlight the interestingphenomena contained in the alert flows. We assume thatthe stationary structure in these flows originates from thenormal behavior of the monitored system. We model thenormal behavior, and then signal deviations from the model.The process consists of two phases, the estimation and thedetection. In both we perform the necessary steps to buildan alert series, and to remove the trend and the periodic-ity from the series. Then, during the estimation phase, weestimate a time series model for the remaining structure inthe signal. During the detection phase, the model output iscompared to the observations to isolate the abnormal com-ponent of the alert flow for further analysis.

We presented results obtained with a tool using this method-ology. They indicate that we can free significant amount ofthe time spent on analyzing these alerts when compared tothe manual processing. As with all automated processingmethods, there is a risk of filtering interesting alerts. In ad-

dition the proposed processing has the possible side effectof creating artificial anomalies into the alert data. However,these risks seem to be relatively small compared to the gains.

As we use real world data for model estimation, we areincorporating also existing anomalies in addition to the nor-mal behavior into model parameters. The results show thatanomalies in estimation data did not have significant ad-verse effect on the detection capabilities. One should how-ever keep this fact in mind, and avoid fitting the model toowell to the data.

We have used one hour sampling interval in this work. Itcan seem long, but given the nature of processed alerts weconsider this reasonable. As discussed previously, this typeof alerts are unlikely to be treated in real-time in any case.One might need to decrease the sampling interval for moretimely detection, especially if trying to react to automatedthreats such as worms. Worm detection is not per se thegoal of this work. In addition changes in sampling intervalmay affect the observed flow behaviors, and make certaincharacteristics appear or disappear.

Currently we are performing this processing at the man-ager level of the intrusion detection framework. For certainalert flows, especially where no pattern matching is neededand the aggregation can be done using for example packetheader information, it could be possible to push this kind ofprocessing towards the sensors.

8. REFERENCES[1] S. Axelsson. The Base-Rate Fallacy and Its

Implications for the Difficulty of Intrusion Detection.In Proc. of the ACM CCS’99, Nov. 1999.

[2] P. Barford, J. Kline, D. Plonka, and A. Ron. A SignalAnalysis of Network Traffic Anomalies. In Proc of

ACM SIGCOMM Internet Measurement Workshop,Nov. 2002.

[3] P. J. Brockwell and R. A. Davis. Time series: theory

and methods. Springer Texts in Statistics, 1991.

[4] P. J. Brockwell and R. A. Davis. Introduction to time

series and forecasting. Springer Texts in Statistics,2002.

[5] H. Debar and B. Morin. Evaluation of the DiagnosticCapabilities of Commercial Intrusion DetectionSystems. In Proc. of the RAID’02. Springer–Verlag,2002.

[6] H. Debar and A. Wespi. Aggregation and Correlationof Intrusion-Detection Alerts. In Proc. of the

RAID’01. Springer–Verlag, 2001.

[7] K. Julisch. Mining Alarm Clusters to Improve AlarmHandling Efficiency. In Proc. of the ACSAC’01, Dec.2001.

[8] K. Julisch and M. Dacier. Mining Intrusion DetectionAlarms for Actionable Knowledge. In Proc. of the

SIGKDD’02, 2002.

[9] C. Kruegel and W. Robertson. Alert verification:Determining the success of intrusion attempts. InProc. of the DIMVA’04, Dortmund, Germany, July2004.

[10] G. M. Ljung and G. E. P. Box. On a Measure of Lackof Fit in Time Series Models. Biometrica,65(2):297–303, Aug. 1978.

[11] V. A. Mahadik, X. Wu, and D. S. Reeves. Detection ofDenial of QoS Attacks Based on χ2 Statistic and

EWMA Control Chart. URL:http://arqos.csc.ncsu.edu/papers.htm, Feb. 2002.

[12] S. Manganaris, M. Christensen, D. Zerkle, andK. Hermiz. A Data Mining Analysis of RTID Alarms.RAID’99, 1999.

[13] H. Mannila, H. Toivonen, and A. I. Virkamo.Discovering Frequent Episodes in Sequences. In Proc.

of the KDD’95, 1995.

[14] P. A. Porras, M. W. Fong, and A. Valdes. AMission-Impact-Based Approach to INFOSEC AlarmCorrelation. In Proc. of the RAID’02.Springer–Verlag, 2002.

[15] X. Qin and W. Lee. Statistical Causality Analysis ofINFOSEC Alert Data. In Proc. of the RAID’03.Springer–Verlag, 2003.

[16] A. Valdes and K. Skinner. Probabilistic AlertCorrelation. In Proc. of the RAID’01.Springer–Verlag, 2001.

[17] J. Viinikka and H. Debar. Monitoring IDSBackground Noise Using EWMA Control Charts andAlert Information. In Proc. of the RAID’04,Springer–Verlag, 2004.

[18] N. Ye, C. Borror, and Y. Chang. EWMA Techniquesfor Computer Intrusion Detection Through AnomalousChanges In Event Intensity. Quality and Reliability

Engineering International, 18:443–451, 2002.

[19] N. Ye, S. Vilbert, and Q. Chen. Computer IntrusionDetection Through EWMA for Autocorrelated andUncorrelated Data. IEEE Transactions on Reliability,52(1):75–82, Mar. 2003.