Data Science for Network Security - LVEE• Developed by Farsight Security for transporting network packets, particulary - DNS • Low-latency and compactness • Support binary and

Data Sciencefor

Network Security

Dmitry Orekhov

Collecting Data

Internet

Data Flow

AnalyticsTelemetry

Traffic

Alerts,Actions

Internet

BA

C

Sensors: Vantage

Host

Services

Network

OSI

Syslogs

Specific logs

Traffic

Sensors: Domain

• NetFlow is a traffic summarization standard developed by Cisco Systems and originally used for network services billing.

• The heart of NetFlow is the concept of a flow, which is an approximation of a TCP session.

• Plenty of Open Source implementations

Transport: NetFlow

Nmsg

DNSQr

DNS Payload

Transport: Nmsg

• Developed by Farsight Security for transporting network packets, particulary -DNS

• Low-latency and compactness

• Support binary and presentation forms; protobuf for protocols

• Open impelentation (GNU)

SDN Controller

OpenFlowPacket-In

Network

Трафик

Свич SDN

Transport: OpenFlow

Use Case

DNS Tunneling

?

DNS

Question

Answer

Question

Answer

DNS Message: Big Picture

DNS

Header

Question

Answer

QName

Resource Record

Resource Record

Name

Name

Base32

Here you are: Tunnel

DNS Packet DNS Packet DNS Packet

TCP Packet

...

TCP Packet

TCP Packet

TCP Packet

TCP Packet

TCP Packet

SSH Tunnel

Discovery methods

Payload analysis

• Size of request and response• Entropy of hostnames• Statistical Analysis• Uncommon Record Types

Traffic Analysis

• Volume of DNS traffic per IP address

• Volume of DNS traffic per domain• Number of hostnames per domain• Geographic location of DNS server• Domain history• Orphan DNS requests

Effective Intrusion Detection

DNS Message

Header

Question

Answer

QName

Resource Record

Resource Record

Name

Name

... needs deep packet inspection

... and

… low-latency

DNS Message

DNS Message

DNS Message

Stream StreamStream Stream

Solution: Lambda Architecture

Online modelStreaming Processing (Spark Streaming)

Batch processing (MapReduce)

Transformation

Transformation

TransformationStream

Stream

Alert

AlgorithmsIncremental algorithmsOutlier Detection• Median Absolute Deviation, MAD• Standard Deviation from average• Standard Deviation from Moving AverageПотоковая классификация• Incremental decision tree• Hoeffding Tree (VFDT)• Half-Spaсe Trees

Offline modelStreaming Processing (Spark Streaming)

Batch Processing (MapReduce)

FilterStream

Stream

Stream

Stre

am

Alert

Offline Model - Алгоритмы

Analyze entire data set at onceHypothesis tests• Simple outlier detection far a period• Statistical criteria• Kolmogorov-Smirnov testDecision TreesAuto Regressive (AR) Moving Average (MA)

Data

Storage(Hive

or Cassandra)

ArchitectureSpark Streaming

Parsing Enrichment

Raw

Online Algorithms Alert Rules

Alerts

Kafka

Alert

TopicSpark/MapReduce

Batch

Network Appliance

Stream

Traffic

Topic

?