A Technical Evaluation and Critique of: Techniques and Tools for Analyzing Intrusion Alerts by Peng Ning, Yun Cui, Douglas S. Reeves, and Dingbang Xu Angela.

A Technical Evaluation and Critique of:

“Techniques and Tools for Analyzing Intrusion Alerts”

by Peng Ning, Yun Cui, Douglas S. Reeves, and

Dingbang Xu

A Technical Evaluation and Critique of:

“Techniques and Tools for Analyzing Intrusion Alerts”

by Peng Ning, Yun Cui, Douglas S. Reeves, and

Dingbang Xu

Angela OrebaughIT862

4/28/05

Angela OrebaughIT862

4/28/05

Formal Framework

Framework Highlights

• Correlates alerts on the basis of prerequisites and consequences of attacks

• Matches the consequences of some prior alerts with the prerequisites of some later ones

• Constructs attack scenarios

• Represented in Hyperalert Correlation Graph

• Uses nodes to represent alerts and edges to represent the relationships between the alerts.

Framework Notation• Prerequisite Predicate

• UDPVulnerableToBOF(VictimIP, VictimPort)

• Consequence Predicate

• {GainRootAccess (VictimIP), rhostsModified (VictimIP)}

• Logical combination of predicates for complex attacks

• UDPVulnerableToBOF(VictimIP, VictimPort) ^ (UDPAccessibleViaFirewall (VictimeIP, VictimPort)

Framework Notation (2)• HyperAlert Type T (fact, prerequisite,

consequence)

• SadmindBufferOverflow = ({VictimIP, VictimPort}, ExistHost (VictimIP) ^ VulnerableSadmind (VictimIP), {GainRootAccess (VictimIP)})

• HyperAlert Instance h

• hSadmindBOF = {(VictimIP = 152.1.19.5, VictimPort = 1235), (VictimIP = 152.1.19.7, VictimPort = 1235)}

• ExistHost (152.1.19.5) ^ VulnerableSadmind (152.1.19.5), ExistHost (152.1.19.7) ^ VulnerableSadmind (152.1.19.7)

• GainRootAccess (152.1.19.5), GainRootAccess (152.1.19.7)

Hyperalert Correlation

• In a sequence S of hyperalerts, a hyperalert h is...

• Correlated hyperalert

• if there exists another hyperalert h’ in S such that either h prepares for h’ or h’ prepares for h.

• Isolated hyperalert

• if no such h’ exists

Hyperalert Correlation Graph

Additional Utilities• Aggregation/Disaggregation

• All hyperalerts of type FTP-BOF combined• All hyperalerts that are DoS are combined

• Focused Analysis• SrcIP = 192.168.1.1 V DestIP = 192.168.20.1

• Clustering Analysis• (A1.SrcIP = A2.SrcIP) ^ (A1.DestIP = A2.DestIP)

• Frequency Analysis• Counting the number of raw alerts that share the same

destination IP address to fnd the most frequently hit target

• Link Analysis• How two IP addresses are realted to each other in a

collection of alerts

• Association Analysis• Many attacks are from source IP 152.14.51.14 to destination

IP 129.14.1.31 at destination port 80

TIAA(Toolkit for Intrusion Alert

Analysis)

TIAA Architecture

Most Recent TIAA Software

• http://discovery.csc.ncsu.edu/~pning/software/correlator

• Current version 0.4

• Tested on Windows 2000 and XP with MS SQL Server

• Newly added features

• Association Analysis (Extracting frequent coourrences of attribute values from a set of alerts)

• Attack Strategy Extraction (Extracting attack strategies from a correlation graph)

• Missed Attack Hypotheses (Hypothesizing possibly missed attacks)

TIAA Knowledge Base<hyper-alertType Name="SadmindOverflow">

<Fact FactName="DestIPAddress" FactType="varchar(15)"></Fact> <Fact FactName="DestPort" FactType="int"></Fact><Protocol ProtocolName="RPC"></Protocol> <Protocol ProtocolName="SADMIND"></Protocol> <Prerequisite>

<Predicate Name="ExistHost"> <Arg id="3" ArgName="DestIPAddress"></Arg>

</Predicate> <Predicate Name="VulnerableSadmind">

<Arg id="22" ArgName="DestIPAddress"></Arg> </Predicate>

</Prerequisite> <Consequence>

<Predicate Name="GainRootAccess"> <Arg id="18" ArgName="DestIPAddress"></Arg>

</Predicate> </Consequence>

</hyper-alertType>

Experiments

Experiments

• 2000 DARPA intrusion detection dataset

• Aimed at evaluating the effectiveness of the proposed alert correlation method in constructing attack scenarios and its ability to differentiate true and false alerts.

• DEFCON 8 CTF

• Intended to evaluate the usefulness of the analysis utilities in dealing with large collections of intrusion alerts.

DARPA Dataset• LLDOS 1.0 - Series of attacks in which an attacker

probes, breaks in, installs the components necessary to launch a DDoS attack, and launches a DDoS attack against an off-site server.

• LLDOS 2.0.2 - Similar sequence of attacks by a more sophisticated attacker.

• Each dataset contains network traffic collected from both the DMZ and the internal network.

• Testing used 4 sets of experiments, each with either the DMZ of the inside network traffic of one dataset.

DEFCON 8 CTF Dataset

• Capture the flag contest

• Attacks range from script kiddie to sophisticated attacker

• Largest graph had 2,940 nodes and 25,321 edges

• On average each graph had 21.75 nodes and 310.56 edges

DARPA Experiment Results

• TIAA revealed the structure and high-level strategy of the sequence of attacks

• Real Secure generated duplicate alerts for several attacks

• Correlated a few false alerts• ISS >93% false alert rate• TIAA reduced to 5% for LLDOS 1.0 and 23%-40% for LLDOS

2.0.2

• Correlated normal alerts that were not attacks

• Missed the Telnet portion of the attack

• LLDOS 2.0.2 results were unsatisfactory

DEFCON Experiment Results

• Probably some missed alerts - So many attacks occurring at once

• Alert aggregation reduced the largest graph to 77 nodes and 347 edges

• 7 clear stages of attacks

• Utilities helped discover several attack strategies

• Scanning attacks followed by attacks that may lead to execution of arbitrary code

• Not good for forensics

Related Work

First Class of Approaches• Staniford 2002

• Probability distribution for normal traffic to detect portscan attacks

• SPICE/SPADE

• Valdes and Skinner 2001• Mathematical framework for correlating alerts that match

closely but not perfectly• EMERALD

• Cuppens 2001• Alert clustering and merging via expert system approach• Also uses pre and post attack conditions based on LAMDA• MIRADOR project

• Julisch 2001• Alarm clustering to determine root causes• Alarm clustering and summarizing

Second Class of Approaches

• Eckmann 2002• State transition based attack scenarios• State Transition Analysis Technique Language (STATL)

• Cuppens and Ortalo 2000• Attack scenarios with pre and post conditions using

the LAMBDA attack description language

• Debar and Wespi 2001• Detects duplicates and consequences according to

explicit rules• Built on top of Tivoli Enterprise Console

Third Class of Approaches

• Templeton and Levitt 2000• Capabilities/concepts attack model describes unknown attacks

and predict attacker actions• JIGSAW attack specification language

• Cuppens and Miege 2002• CRIM module based on LAMBDA to cluster, merge, and

correlate alerts• MIRADOR project

• Morin 2002• M2D2 data model for correlation• Uses correlation function to detect false positives

• Porras 2002• Mission-impact-based approach• MCorrelator uses an internal topology map for correlation

• Peng Ning, et. al. 2003

Vulnerability Analysis Approach

• Ritchey and Ammann 2000• Modeling based approach based on host vulnerability, host

connectivity, current point of view of attacker, exploits that can change the state of the model

• Uses a state machine to encode the vulnerabilities

• Sheyner 2002• Automated technique for generating and analyzing attack

graphs• Based on intruder preconditions, network preconditions,

intruder effects, and network effects

• Jha 2002• Expands on Sheyner paper• Presents a formal and detailed explanation of the model• Presents an algorithm to compute the reliability for a

network

Summary

Major Contributions

• Hyperalert correlation graphs

• Partial satisfaction of attack prerequisites

• Uses possible consequences instead of actual consequences

• Analysis Utilities

• TIAA

Framework Critique

• Successfully revealed relationships between alerts and strategies behind the attacks

• Effectively reduces the number of alerts via aggregation

• Needs to address partial satisfaction of prerequisites more thoroughly

• It does not address security architecture• It doesn’t address network issues• IDS evasion• May not discover stealthy and intelligent attacks• Can still be evaded

TIAA Critique• Needs to provide more information on the

Knowledge base• Does the accuracy of the system all come down to the

robustness of the knowledge base?

• “the results produced by our correlation techniques are only as good as the hyperalert information provide by the user”

• Only supports IDMEF and ISS Real Secure alerts• http://www.sans.org/rr/whitepapers/detection/

1080.php

• Only works with a commercial database - MS SQL Server

• Not meant for the inexperienced use

Testing Critique

• Uses ISS Real Secure

• Are the DARPA and DEFCON CTF datasets the best methods of testing?• DARPA dataset has received lots of criticism

• DEFCON CTF is all attack traffic and not much else

• What is the best testing approach anyway?• Test network, live network, replay data taken from

other networks, DARPA or other pre-generated datasets

• Tested DARPA datasets separately

Architecture• Does not address a recommended security architecture for

optimization

• Sensor placement is critical to correlation

Firewall w/logs

Desktopw/HIDS and AV

Web

IDS

Mail

Management Station

DNS

Database

Router w/logs

IDS

IDS

Internet

Additional Research Opportunities

• Expanded use of hyperalert correlation graphs• Attacker profiling• Predictive analysis• IDS tuning• Input to incident response procedures

• Incorporate forensic analysis

• Integrate framework with complementary correlation methods for better performance

• Use TIAA as part of penetration testing team

• Automatic generation of knowledge base by learning algorithm

Discussion...

• Critiques of the framework?

• Critiques of the TIAA toolkit?

• Critiques of the testing?

• Additional research/expansion opportunities?