Developed by Cisco Systems in 1996 The value of information in the cache was a secondary discovery –Initially designed as a switching path NetFlow is now.

• Developed by Cisco Systems in 1996

• The value of information in the cache was a secondary discovery– Initially designed as a switching path

• NetFlow is now the primary network accounting technology in the industry

• Answers questions regarding IP traffic: who, what, where, when, and how

• NetFlow version 9 an IETF standard

Netflow Overview

Traffic Analysis

• What we needs– application performance– application-based accounting – network security– Network behavior, application recognition

• ‘debug ip packet’ in router?• IP Sniffing in shared LAN (or using switch to do so)• Port Span in switch (how about port span in router?)• Circuit Sniffing• Netflow• What we prefer in backbone:

– Embeded– Fixed length partial packet export– Real-time filtered packet export

Addressing The Needs with Netflow

Netflow Possible Applications

• Network Monitoring• Network planning• Security Analysis• Application Monitoring• User Monitoring• Traffic Engineering• Peering Agreement• Usage-base Billing• Destination sensitive billing

What is a flow?

Defined by seven unique keys:

1. Source IP address

2. Destination IP address

3. Source port

4. Destination port

5. Layer 3 protocol

6. TOS byte (DSCP)

7. Input interface (ifIndex)

Exported DataA Flow is Unidirectional!

NetFlow Sequence

1. Create and update flows in NetFlow Cache2. Expiration

3. Aggregation?

4. Export Version

5. Transport Protocol

step1• Inactive timer expired (15 sec is default)• Active timer expired (30 min (1800 sec) is default)• NetFlow cache is full (oldest flows are expired)• RST or FIN TCP Flag

He

ad

er

ExportPacket

Payload(flows)

step2

step3

Protocol Pkts SrcPort DstPort Bytes/Pkt

11 11000 00A2 00A2 1528

SrcIf SrcIPadd DstIf DstIPadd Protocol TOS Flgs Pkts SrcPort SrcMsk SrcAS DstPort DstMsk DstAS NextHop Bytes/Pkt Active Idle

Fa1/0 173.100.21.2 Fa0/0 10.0.227.12 11 80 10 11000 00A2 /24 5 00A2 /24 15 10.0.23.2 1528 1800 4

e.g. Protocol-Port Aggregation Scheme becomes

step4

SrcIf SrcIPadd DstIf DstIPadd Protocol TOS Flgs Pkts SrcPort SrcMsk SrcAS DstPort DstMsk DstAS NextHop Bytes/Pkt Active Idle

Fa1/0 173.100.21.2 Fa0/0 10.0.227.12 11 80 10 11000 00A2 /24 5 00A2 /24 15 10.0.23.2 1528 1745 4

Fa1/0 173.100.3.2 Fa0/0 10.0.227.12 6 40 0 2491 15 /26 196 15 /24 15 10.0.23.2 740 41.5 1

Fa1/0 173.100.20.2 Fa0/0 10.0.227.12 11 80 10 10000 00A1 /24 180 00A1 /24 15 10.0.23.2 1428 1145.5 3

Fa1/0 173.100.6.2 Fa0/0 10.0.227.12 6 40 0 2210 19 /30 180 19 /24 15 10.0.23.2 1040 24.5 14

YesNo

Aggregated Flows – export Version 8 or 9Non-Aggregated Flows – export Version 5 or 9

step5

NetFlow Sequence (continued)

Netflow Processing Order

Pre-Processing

FeaturesAnd

Services

PostProcessing

Packet Sampling

Filtering

IPMulticastMPLSIPv6

Aggregation schemesNon-key fields lookupExport

Creating Export Packets

Core Network(IP, MPLS)

Enable NetFlow

Traffic

Collector (Solaris, HP-UX, or Linux)

UDP NetFlowExport Packe

ts

Application: Performance Billing Security

PE

Export Packets• Approximately 1500 bytes• Typically contain 20-50 flow re

cords• Sent more frequently if traffic

increases on NetFlow-enabled interfaces

NetFlow Principles

• Inbound traffic only (with some exceptions)• Unidirectional flow• Accounts for both transit traffic and traffic destined fo

r the router• Works with Cisco Express Forwarding (CEF) or fast s

witching • Almost supported on all interfaces and Cisco IOS

Software platforms• Provides the sub-interface information in the flow

records• 6500/7600 enables Netflow on all interfaces by default

SiSiSiSi

Comprehensive Platform Support

GSR 12000GSR 12000

Catalyst 4500

Catalyst 4500

7200/7500/7200/7500/

37003700

2500/

2600

2500/

2600

36003600

AS5300/5800

AS5300/5800

4500/47004500/4700

1400/1600/1700

1400/1600/1700

Catalyst 5000/6500/

7600

Catalyst 5000/6500/

7600

ESR10000ESR

10000

NetFlow Versions

Version 5 - Flow Format

• Source IP Address• Destination IP Address

• Packet Count• Byte Count

Usage

QoS

Timeof Day

Routing and

Peering

• Input ifIndex• Output ifIndex

• Type of Service• TCP Flags• Protocol

• Start sysUpTime• End sysUpTime

• Source TCP/UDP Port• Destination TCP/UDP Port

• Next Hop Address• Source AS Number• Dest. AS Number• Source Prefix Mask• Dest. Prefix Mask

From/to

Application

Blue – Key Field (7)Red - Lookup Field (5)Black- Value Field (6)

Netflow Configuration Commands

• ip flow-export version <version> [origin-as | peer-as | bgp-nexthop]– e.g. ip flow-export version 5

• ip flow-export destination <address> <port>– e.g. ip flow-export destination 10.0.0.1 65001

• ip flow export source <interface>– default is interface with best route to collector. Recommendation: configure

loopback interface.• ip flow-aggregation cache <name of aggregation scheme>

– select the aggregation cache• ip flow-cache timeout inactive <seconds>

– sets the seconds an inactive flow will remain in the cache before expiration. 15 seconds is default

• ip flow-cache timeout active <mintues>– sets the minutes an active flow will remain in the cache bvefore expiration.

30 minutes is default• ip flow-cache entries <number>

– sets the maximum number of flow entries in the cache. The default varies dependent on platform.

Netflow Show Commands

• show ip cache [verbose] flow– shows Netflow statistics

• show cache flow aggregation <name of aggregation scheme>– shows netflow statistics for the configured aggregation sch

eme

• show ip flow export– shows export statistics

• clear ip cache flow– clears netflow statistics

• clear ip flow stats– clears export statistics

Show ip cache flowIP packet size distribution (2175M total packets): 1-32 64 96 128 160 192 224 256 288 320 352 384 416 448 480 .001 .440 .139 .014 .008 .000 .000 .000 .000 .000 .000 .000 .011 .000 .000

512 544 576 1024 1536 2048 2560 3072 3584 4096 4608 .000 .000 .000 .002 .377 .000 .000 .000 .000 .000 .000

IP Flow Switching Cache, 4456704 bytes 550 active, 64986 inactive, 509378135 added 3145787062 ager polls, 0 flow alloc failures Active flows timeout in 30 minutes Inactive flows timeout in 15 seconds

Protocol Total Flows Packets Bytes Packets Active(Sec) Idle(Sec)-------- Flows /Sec /Flow /Pkt /Sec /Flow /FlowTCP-WWW 10431912 2.4 10 181 25.7 7.1 20.4TCP-SMTP 773843 0.1 6 98 1.1 8.3 16.7…….Total: 509377507 118.5 4 567 506.4 1.7 15.9

SrcIf SrcIPaddress DstIPaddress Pr SrcP DstP Pkts Te7/3 219.245.101.77 202.205.5.3 tcp 1444 1203 1 Te7/3 84.97.234.47 202.204.192.18 udp 7692 2881 1 Te7/3 222.81.87.163 202.205.3.203 tcp 1172

Show ip flow export

Router> sh ip flow exportFlow export v5 is enabled for main cache Exporting flows to 192.168.1.2 (2055) 192.168.2.3 (2054) Exporting using source interface Loopback0 Version 5 flow records, origin-as 998016649 flows exported in 33267252 udp datagrams 0 flows failed due to lack of export packet 0 export packets were sent up to process level 0 export packets were dropped due to no fib 0 export packets were dropped due to adjacency issues 0 export packets were dropped due to fragmentation failures 0 export packets were dropped due to encapsulation fixup failure

s 0 export packets were dropped enqueuing for the RP 0 export packets were dropped due to IPC rate limiting

Version 7

• Adds NetFlow switching support for: Cisco Catalyst 5000 Series Switches with an RSM Cisco Catalyst 5000 Series Switches with an MSFC

• Uses MultiLayer Switching (MLS) or CEF with Cisco Catalyst 6000 Series Switches with SUP2

• IP unicast only No multicast or IPX, even if MLS can do all three

• MLS cache is the equivalent of the NetFlow cache

Version 8

• Router-based aggregation• Enables router to summarize NetFlow data• Reduces NetFlow Export data volume• Decreases NetFlow Export bandwidth requirements• Currently 11 aggregation schemes

Five original schemes Six new schemes with the TOS byte field

• Several aggregations can be enabled simultaneously

Version 9

Fixed formats (versions 1, 5, 7, and 8) are not flexible and adaptable

Cisco needed to build a new version each time a customer wanted to export new fields

When new versions are created, partners need to reengineer to support the new export format

Solution: Build a flexible and extensible export format!

Netflow v9 Principles

• Version 9 is an export format• Still a push model• Sent the template regularly (configurable)• Independent of the underlying protocol, it is

ready for any reliable protocol (ie: TCP, SCTP)

• Advantage: we can add new technologies and data types quickly• E.g. MPLS, IPv6, BGP Next Hop, Multicast

Netflow V9 Template

• NetFlow Version 9 Export format is template based. Version 9 record format consists of a packet header followed by at least one or more template or data FlowSets. A template FlowSet (collection of one or more template) provides a description of the fields that will be present in future data FlowSets. Templates provide an extensible design to the record format, a feature that should allow future enhancements to NetFlow services without requiring concurrent changes to the basic flow-record format.

• template composed of type and length• flow records composed of template ID and value

• sent the template regularly (configurable), because of UDP

Netflow Version 9 Scenario

Netflow v9: Example for Template Definition

Netflow Version9 Export Packet

Netflow v9: Example for 1 Export Packet

NetFlow v9 Export Packet

Data FlowSetTemplate FlowSet Option

Template

FlowSet

HeaderFlowSet ID #1

Data FlowSetFlowSet ID #2

Template ID

(specific

Field types

and lengths)

(version,

# packets,

sequence #,

Source ID)

• Matching ID #s is the way to associate Template to the Data Records

• The Header follows the same format as prior NetFlow versions so Collectors will be backward compatible

• Each Data Record represents one flow

• If exported flows have the same fields then they can be contained in the same Template Record e.g. unicast traffic can be combined with multicast records

• If exported flows have different fields then they can’t be contained in the same Template Record e.g. BGP next-hop can’t be combined with MPLS Aware NetFlow records

Flows from

Interface A

Flows from

Interface B

To support technologies such as

MPLS or Multicast, this export format can

be leveraged to easily insert new fields

Option Data

FlowSetFlowSet ID

Option Data

Record

(Field values)

Option Data

Record

(Field values)

Template Record

Template ID #2

(specific Field types and lengths)

Template Record

Template ID #1

(specific Field types and lengths)

Data Record

(Field values)

Data Record

(Field values)

Data Record

(Field values)

NetFlow v9 Export

test(config)# ip flow-export version ? 1 5 9 test(config)# ip flow-export version 9 .

Configuring Version 9 export

test(config)# ip flow-aggregation cache as

test(config-flow-cache)# enabled

test(config-flow-cache)# export ?

destination Specify the Destination IP address

version configure aggregation cache export version

test(config-flow-cache)# export version ?

8 Version 8 export format

9 Version 9 export format

test(config-flow-cache)# export version 9

Configuring Version 9 export for an aggregation scheme

Export versions available for standard NetFlow flows

Export versions available for aggregated NetFlow flows

IETF: IP Flow information Export(IPFIX) Working Group

• IPFIX is an effort to:– Define the notion of a "standard IP flow"– Devise data encoding for IP flows– Consider the notion of IP flow information export based upo

n packet sampling– Identify and address any security privacy concerns affectin

g flow data– Specify the transport mapping for carrying IP flow informati

on(IETF approved congestion-aware transport protocol)

– Netflow version 9 has been selected as a basis for the IPFIX protocol

IETF: Packet Sampling WG(PSAMP)

• PSAMP agreed to use IPFIX(Netflow version9) for export

• PSAMP is an effort to:– specify a set of selection operations by which pac

kets are sampled– describe protocols by which information on sampl

ed packets is reported to applicatons

• http://www.ietf.org/html.charters/psamp-charter.html

• Note: Netflow is already using some sampling mechanisms

NetFlow Infrastructure

NetFlow Uses

• Attack Mitigation• User (IP)

monitoring• Application

monitoring



monitoring

• Billing• Chargeback• AS Peer

Monitoring


Monitoring

• Traffic Engineering

• Traffic Analysis


• Traffic Analysis

Ap

pli

cati

on

s • Attack Mitigation• User (IP)


monitoring



monitoring


Monitoring


Monitoring

Net

wo

rk L

ayer

AccessAccess DistributionDistribution DistributionDistribution AccessAccessCoreCore

Net

Flo

wF

eatu

res

• Aggregation Schemes (v8)

• “show ip cache flow” command

• Arbor Networks



• Arbor Networks

• NetFlow MPLS Egress Accounting

• BGP Next-hop (v9)

• Multicast NetFlow (v9)




• MPLS Aware NetFlow (v9)


• Sampled NetFlow

• MPLS Aware NetFlow (v9)


• Sampled NetFlow









• Arbor Networks



• Arbor Networks

Netflow Collector(NFC) 5.0

Netflow on the Network Analysis Module (NAM)

Netflow Partners

Billing

Flat-rate billing does not necessarily scaleCompetitive pricing models can be created with usage-based billing

Usage-based billing considerationsTime of dayWithin or outside of the network ApplicationDistance-basedQuality of Service (QoS) / Class of Service (CoS)Bandwidth usageTransit or peerData transferredTraffic class

Tracking Users

Who are my top N talkers, and what percentage of traffic do they represent?

How many users are on the network at a given time? When will upgrades affect the least number of users?

How long do users spend connected to the network? Where Internet sites do they use? What is a typical pattern of usage between sites? Are users staying within an acceptable usage

policy (AUP)? Alarm DOS attacks like smurf, fraggle, and SYN flood

Will watch for these attack, regardless of source / destination

Principle Netflow Benefits

Service ProviderService Provider EnterpriseEnterprise

• Internet access monitoring (protocol distribution, where traffic is going/coming)

• User Monitoring

• Application Monitoring

• Charge Back billing for departments

• Security Monitoring

• Internet access monitoring (protocol distribution, where traffic is going/coming)

• User Monitoring

• Application Monitoring

• Charge Back billing for departments


• Peering arrangements

• Network Planning


• Accounting and billing


• Peering arrangements

• Network Planning


• Accounting and billing


NetFlow – Charge Back Billing

R&DHR

Finance

Account per network (rather that per IP addresses)

Internet

Example: charge the department for the cost of the Internet link

NetFlow – Peering Agreement

Account per BGP AS, to Review Peering Agreements

ISP

UunetDigexErolsBBNAT&T

AMUC&WJHUPACBell Internet ServiceRCNOARnetSURAnetCompuserve

OLABSNETWebTVWEC

Public Routers 1, 2, 3 Month of September—Outbound Traffic

NetFlow – Peering Agreement

20%

32%

4%6%

8%

8%

10%

1% 1%1%

1%1%

1%

2%1%

1%1%

MPLS Aware NetFlow (v9)

IP Fields

Source and destination IP address

Input and output sub-interfaces

Transport layer protocol

Source and destination application port numbers

8 bit IP Type of Service (ToS)

TCP Flags (accumulation from all packets in the flow)

MPLS Fields

Up to three incoming MPLS labels with experimental (EXP) bits and end-of-stack (S) bit

Position of each of the three labels

Type of the top label

IP address associated with the top label

Traditional NetFlow Fields

Number of packets

Number of bytes (count either IP or MPLS header / payload)

Time-stamps of first and last packets in the flow

MPLS

Traditional NetFlow for IP to MPLS traffic

PEPE PP PEPE

Egress MPLS NetFlow Accounting• IP information only• Ideal for billing• Current availability: Cisco IOS Software Releases 12.0(10)ST and 12.1(5)T

MPLS Aware NetFlow (version 9)• Exports up to three MPLS labels, and IP packet information• Ideal for Traffic Engineering• Will be available in Cisco IOS Software Releases 12.0(24)S, 12.2S, and 12.3

Traffic Flow

IP

IP

Egress MPLS NetFlow Accountingfor MPLS to IP traffic

MPLS Aware NetFlow (version 9)

MPLS

Autonomous System

3600-4(config)# ip flow-export version 5 ? origin-as record origin AS peer-as record peer AS <cr>

3600-4(config)#

• Origin-ASSpecifies that export statistics include the origin autonomous system (AS) for the source and destination

• Peer-ASSpecifies that export statistics include the peer AS for the source and destination

Autonomous System

AS 101

Configuring Peer-AS•Source AS = AS 103•Destination AS = AS 105

NetFlow enabled

AS 103 AS 104

AS 105

AS 106

AS 102

Router(config)#ip flow-export version 5 peer-as

Autonomous System

AS 101

NetFlow enabled

AS 103 AS 104

AS 105

AS 106Configuring Origin-AS

• Source AS = AS 101• Destination AS = AS 106

AS 102

Router(config)#ip flow-export version 5 origin-as

BGP next-hop

• Supported only in version 9 export• For traffic engineering/analysis and possible

billing applications• Fields that are exported include all those

found in version 5 export• Will be supported in Cisco IOS Software

Releases 12.0(26)S, 12.2S, and 12.3

BGP next-hop

Netflow BGP next-hop

BGP next-hop Details

• Supported only in version 9 export• For traffic engineering/analysis (traffic matrix) and p

ossible billing applications. "What is the Next hop IP address of my BGP traffic?"

• exported fields include all version 5 fields, including IP next hop

• Adds 16 bytes to each Netflow flow record (goes from 64 bytes to 80 bytes), while CPU increase is negligible

• Edge to Edge traffic matrix for engineering/analysis and possible billing applications

• Supported in Cisco IOS Software releases 12.0(26)S, 12.2(18)S, and 12.3(1)

BGP next-hop

pamela(config)# ip flow-export version ? 1 5 9 pamela(config)# ip flow-export version 9 .

Configuring Version 9 export

pamela(config)# ip flow-export version 9 ?

bgp-nexthop record BGP NextHop

origin-as record origin AS

peer-as record peer AS

<cr>

pamela(config)# ip flow-export version 9 bgp-nexthop

Configuring Version 9 export with BGP next-hop

Multicast NetFlow

Three types of NetFlow implementations for Multicast traffic:

1. Traditional NetFlow

2. Multicast NetFlow Ingress

3. Multicast NetFlow Egress

Multicast – Traditional NetFlow

Eth 0

Eth 3Eth 1

Eth 2

Interface Ethernet 0

ip route-cache flow

ip flow-export version 9

ip flow-export destination 127.0.0.1 9995

127.0.0.1

NetFlowCollector

server

Traditional NetFlow configuration

10.0.0.2

(S, G) - (10.0.0.2, 224.10.10.100)

Flow Record Created in NetFlow Cache

• There is only one flow per NetFlow configured input interface• The 7 Key fields that define a unique flow are marked in red • Destination interface is marked as “Null”• Bytes and Packets are the incoming values

SrcIf SrcIPadd DstIf DstIPadd Protocol TOS Flgs SrcPort SrcMsk DstPort DstMsk NextHop Bytes Packets Active Idle

Eth 0 10.0.0.2 Null 224.10.10.100 11 80 10 00A2 /24 00A2 /24 23100 21 1745 4

Multicast NetFlow Ingress


ip multicast netflow ingress



Multicast NetFlow Ingress configuration

Flow Record Created in NetFlow Cache

• There is only one flow per NetFlow configured input interface• The 7 Key fields that define a unique flow are marked in red • Destination interface is marked as “Null”• Bytes and Packets are the outgoing values


Eth 0 10.0.0.2 Null 224.10.10.100 11 80 10 00A2 /24 00A2 /24 69300 63 1745 4

Eth 0

Eth 3Eth 1

Eth 2

127.0.0.1

NetFlowCollector

server

10.0.0.2

(S, G) - (10.0.0.2, 224.10.10.100)

Multicast NetFlow Egress


ip multicast netflow egress







Multicast NetFlow Egress configuration

Flow Records Created in NetFlow Cache

• There is one flow per Multicast NetFlow Egress configured output interface• One of the 7 Key fields that define a unique flow has changed from Source Interface to Destination Interface • Bytes and Packets are the outgoing values


Eth 0 10.0.0.2 Eth 1 224.10.10.100 11 80 10 00A2 /24 00A2 /24 23100 21 1745 4

Eth 0 10.0.0.2 Eth 2 224.10.10.100 11 80 10 00A2 /24 00A2 /24 23100 21 1745 4

Eth 0 10.0.0.2 Eth 3 224.10.10.100 11 80 10 00A2 /24 00A2 /24 23100 21 1745 4

Eth 0

Eth 3Eth 1

Eth 2

127.0.0.1

NetFlowCollector

server

10.0.0.2

(S, G) - (10.0.0.2, 224.10.10.100)

Multicast NetFlow – Summary

Supported via NetFlow version 9 export formatAvailability

Cisco IOS Software Releases 12.0(27)S, 12.2S, and 12.3 Not supported in 120000

Performance: Ingress vs. EgressMulticast NetFlow Ingress and traditional NetFlow will have similar performance numbers Multicast NetFlow Egress will have performance impact that is proportional to the number of interfaces on which it is enabled (include input interface)

Cisco Catalyst 6500/7600 Series SwitchesDo not currently support the tracking of multicast traffic via NetFlow due to current ASIC limitationWill have this support in a future Supervisor

How to Identify a Security Attack?

• Suddenly highly-increased overall traffic in the network

• Higher CPU and memory utilization of network devices

• Unexpectedly large amount of traffic generated by individual hosts

• Increased number of accounting records generated• Multiple accounting records with abnormal content,

like one packet per flow record (e.g. TCP SYN flood)• A changed mix of traffic applications, e.g. a sudden

increase of "unknown" applications• An increase of certain traffic types and messages,

e.g. TCP resets or ICMP messages• An increasing number of ACL violations

What Does a DOS Attack Look Like?

NetFlow – Mitigating Attacks

1. Cost Saver• “sh ip cache flow” command to find top volume flows • Identify source of attack• Write access-list to block• Monitor via “show ip cache flow” & “Null” entry in DestIf field

to show that it is blocked• Prefix-port aggregation can be configured, while “sh ip cache

flow aggregation prefix-port” is used

2. Most Effective• Arbor Networks leverages NetFlow to provide a quicker

response and more sophisticated solution

Security Analysis: Best Practices

Quality of Service Example

DiffServ fieldAKA

IP DSCP markings

Early Congestion Notification (ECN) bits

DS5 DS4 DS3 DS2 DS1 DS0 ECN ECN

128 64 32 16 8 4 2 1

Precedence bits

ToS bits

Quality of Service ExampleTOS byte

DS5 DS4 DS3 DS2 DS1 DS0 ECN ECN128 64 32 16 8 4 2 1

Precedence bits Decimal Precedence Function1 1 1 x x x x x 224 7 Network Control (link layer keepalives)1 1 0 x x x x x 192 6 Internetwork Control (Routing Protocols)1 0 1 x x x x x 160 5 CRITIC/ECP (Express Forwarding)1 0 0 x x x x x 128 4 Flash Override (Class 4)0 1 1 x x x x x 96 3 Flash (Class 3)0 1 0 x x x x x 64 2 Immediate (Class 2)0 0 1 x x x x x 32 1 Priority (Class 1)0 0 0 x x x x x 0 0 Routine (Best effort)

Delay, Throughput, and Reliability bitsDelay bit

x x x 0 x x x x 0 Delay - normalx x x 1 x x x x 16 Delay - low

Throughput bitx x x x 0 x x x 0 Throughput - normalx x x x 1 x x x 8 Throughtput - high

Reliability bitx x x x x 0 x x 0 Reliability - normalx x x x x 1 x x 4 Reliability - high

Early Congestion Notification (ECN) bitsECN-capable Transport (ECT) bit

Congestion Experienced (CE) bitx x x x x x 0 0 0 Not ECN-capablex x x x x x 0 1 1 Endpoints of transport protocol ECN-capablex x x x x x 1 0 2 Endpoints of transport protocol ECN-capablex x x x x x 1 1 3 Congestion experienced

Tracking TOS with NetFlow

7200-3-netflow# show ip cache verbose flowSrcIf SrcIPaddress DstIf DstIPaddress Pr TOS Flgs PktsPort Msk AS Port Msk AS NextHop B/Pk ActiveSR6/0 210.210.210.2 PO1/0 200.200.200.2 FF 00 10 21K0000 /0 0 0000 /0 0 0.0.0.0 1496 665.4SR6/0 210.210.210.2 PO1/0 200.200.200.2 06 C0 00 21K0000 /0 0 0000 /0 0 0.0.0.0 1496 666.0

7200-3-netflow# show ip cache verbose flow SrcIf SrcIPaddress DstIf DstIPaddress Pr TOS Flgs PktsPort Msk AS Port Msk AS NextHop B/Pk ActiveEt1/1 52.52.52.1 Fd4/0 42.42.42.1 01 55 10 37480000 /8 50 0000 /8 40 202.120.130.2 28 17.8Et1/2 52.52.52.1 Fd4/0 42.42.42.1 01 CC 10 35680000 /8 50 0000 /8 40 202.120.130.2 28 17.8Et1/2 10.1.3.2 Fd4/0 42.42.42.1 01 C0 10 11240000 /0 0 0000 /8 40 202.120.130.2 28 17.8

Hex Decimal Binary55 85 0101 0101 Precedence 2 - Immediate (Class 2), Delay - low, Reliability - high, Endpoints of transport protocol ECN-capableC0 192 1100 0000 Precedence 6 - Internetwork Control (Routing Protocols)CC 204 1100 1100 Precedence 6 - Internetwork Control (Routing Protocols), Throughput - high, Reliability - high

Sampled NetFlow

Deterministic Original typeCisco 12000 Series Internet RoutersCisco Catalyst 6500 Series Switches – Release 12.1(13)E

Random (recommended per statistical principles)Cisco IOS Software Releases 12.0(26)S, 12.2S, and 12.3 Cisco 2500, 2600, 3600, 7200, and 7500 Series RoutersCisco 12000 Series Internet Routers

Time-based Cisco Catalyst 6500 Series Switches – Release 12.1(13)E

Trajectory (Hash-based)in development

Sampling configuration

• GSR 12xxx (IOS Version: 12.0(31)S2:R1(config)# ip flow-sampling-mode packet-interval 256

R1(config-if)# ip route-cache flow sampled input

R1(config-if)# ip route-cache flow sampled output

bj2-bgw(config)#ip flow-sampling-mode packet-interval ?

<10-16382> Specify the packet interval at which to sample

• 7609: (12.2(18)SXD6)R1(config)# mls flow ip source

R1(config)# mls nde sender version 5

R1(config)# mls sampling time-based 64 // 64:1

R1(config-if)# ip route-cache flow

R1(config-if)# mls netflow sampling

Cisco Catalyst 6500 and 7600 Series Switches

• Export is centrally via the supervisor and MSFC, each line card has its own hardware NetFlow cache and forwarding table, i.e. distributed platform

Cisco 12000 Series Internet Routers – NetFlow

• Engine 0 – software support• Engine 1 – software support• Engine 2 – supported in ASICs, but lower priority

so beware if running many other features• Engine 3 – version 5 support in software, version

8 support in ASIC• Engine 4 – not supported• Engine 4+ – supported in ASICs

Cisco 12000 Series Internet Routers Sampled NetFlow

Engine Full NetFlow Sampled NetFlow

0

1

2

3

4

4+

Not supportedSupported

Scaling - Memory Utilization

Scaling - Sample TrafficDeterministic vs. Random Sampling

Sampled Netflow Details

• Deterministic– Cisco C6500/7600 Series switches(12.1(13)E)– Cisco 12000 series internet routers (12.0(11)S and

12.0(14)ST)

• Random (select packet to export per statistical principles)– Cisco IOS Software Releases 12.0(26)S, 12.2S(18), and

12.3(1)T– Cisco 800, 1700, 1800, 2600, 2800, 3600, 3700, 3800, 7200,

and 7500 series routers

• Time-based– Cisco C6500/7600 series Random and Time based sampling

12.1(13)E

Sampled Netflow CPU Reduction

Netflow Multiple Export Destinations

Performance Testing Conclusions

• NetFlow Data Export (single/dual)No significant impact

• NetFlow v5 versus v8: little or not impact• NetFlow Feature Acceleration:

>200 lines of ACLs and/or Policy Based-Routing (PBR)

• NetFlow versus Sampled NetFlow on the Cisco 12000 Series Internet Routers

23% versus 3% (65,000 flows, 1:100)

Number of Active Flows Additional CPU Utilization

10,000 <4%

45,000 <12%

65,000 <16%

• Additional CPU utilization

Performance TestingNetFlow Version 9

• Similar CPU and throughput numbers result from configuration of both NetFlow version 5 and 9

• No change in NetFlow performance after the addition of version 9

Cisco IOS Software Releases 12.0(24)S, 12.2S, and 12.3

• CPU is slightly higher immediately following initial boot up or configuration

Caused by sending Template Flowsets to Collector

Reducing Performance Impact

Reduce CPU and memory impact on the router, collector, or network:

• Aging timers (router)

• Sampled NetFlow (router)

• Enable NetFlow Feature Acceleration (router)

• Flow Masks (only Cat6000/7600)

• Enable on specific sub-interface (upcoming router feature)

• Aggregation schemes (v8 on router or on collector)

• Filters (router or collector)

• Data Compression (collector)

• Increase collection bucket sizes (collector)

• Collector and router can be placed on the same LAN segment (network)

Netflow Deployment: Rules of Thumb

Netflow Deployment: Considerations

Cisco Netflow MIB

Netflow MIB applications

• Netflow Configuration• Checking Netflow Configuration• Monitoring and security

– export statistics– protocol statistics– top flows information (top talkers)

Netflow Mib Overview

• Defined groups of objects1. cnfCacheInfo

• A group of objects related to cache information and configuration stored per cache configuration.

2. cnfExportInfo• A group of objects related to Export configuration and information.

4. cnfExportStatistics• Provides export statistics.

5. cnfProtocolStatistics• Provides a summary of NetFlow cache statistics per protocol and po

rt. 6. cnfExportTemplate

• Provides Template based Version 9 flow export information and statistic.

7. cnfTopFlows • Provides top Netflow flows.

Netflow MIB Monitoring

Egress Netflow Accounting

Netflow and IPv6

• Collects IPv6 flow records• Based on Netflow Version9• Support or both ingress and egress traffic• "Full NetFlow" i.e. non-sampled• Data export is still IPv4• Available in release 12.3(7)T

Netflow Summary

• Netflow is a mature Cisco IOS feature (in Cisco IOS since 1996)

• Netflow provides input for Accounting, Performance, Fault, Security, and Billing Applications

• Cisco has IETF and industry leadership• Netflow v9 eases the exporting of additional f

ields• A lot of new features have been added

SFlow

• sFlow® is an industry standard technology for monitoring high speed switched networks, Juniper’s devices support it.

• similar to netflow• NetStream from Huawei Company

• SFlow Packet: Packet header (eg MAC,IPv4,IPv6,IPX,AppleTalk,TCP,UDP, ICMP) Sample process parameters (rate, pool etc.) Input/output ports Priority (802.1p and TOS) VLAN (802.1Q) Source/destination prefix Next hop address Source AS, Source Peer AS Destination AS Path Communities, local preference User IDs (TACACS/RADIUS) for source/destination URL associated with source/destination Interface statistics (RFC 1573, RFC 2233, and RFC 2358)

Tools for Netflow

• Cisco NFC• Arbor Peakflow

• Flow tools• Ntop

– http://ww.ntop.org

• Etc.

Flow-tools

• Flow-tools is library and a collection of programs used to collect, send, process, and generate reports from NetFlow data.

• Can be used together on a single server or distributed to multiple servers for large deployments.

• The flow-tools library provides an API for development of custom applications for NetFlow export versions 1,5,6 and the 14 currently defined version 8 subversions.

• Version 9 is not supported now

Flow-tools utilities

• flow-capture - Collect, compress, store, and manage disk space for exported flows from a router.

• flow-cat - Concatenate flow files. Typically flow files will contain a small window of 5 or 15 minutes of exports. Flow-cat can be used to append files for generating reports that span longer time periods.

• flow-fanout - Replicate NetFlow datagrams to unicast or multicast destinations. Flow-fanout is used to facilitate multiple collectors attached to a single router.

• flow-report - Generate reports for NetFlow data sets. Reports include source/destination IP pairs, source/destination AS, and top talkers. Over 50 reports are currently supported.

• flow-tag - Tag flows based on IP address or AS #. Flow-tag is used to group flows by customer network. The tags can later be used with flow-fanout or flow-report to generate customer based traffic reports.

• flow-filter - Filter flows based on any of the export fields. Flow-filter is used in-line with other programs to generate reports based on flows matching filter expressions.

• flow-import - Import data from ASCII or cflowd format.• flow-export - Export data to ASCII or cflowd format.

Flow-tools utilities( Cont.)

• flow-send - Send data over the network using the NetFlow protocol.

• flow-receive - Receive exports using the NetFlow protocol without storing to disk like flow-capture.

• flow-gen - Generate test data.• flow-dscan - Simple tool for detecting some types of network

scanning and Denial of Service attacks.• flow-merge - Merge flow files in chronoligical order.• flow-xlate - Perform translations on some flow fields.• flow-expire - Expire flows using the same policy of flow-captur

e.• flow-header - Display meta information in flow file.• flow-split - Split flow files into smaller files based on size, time,

or tags.

Configuration in Cisco Router

R1(config)# ip flow-export source Loopback0

R1(config)# ip flow-export version 5 origin-as

R1(config)# ip flow-export destination 202.112.xx.xx 9800

R1(config-if)# ip route-cache flow

flow-capture

• Flow-tools most useful and important command• flow-capture -w /flows/dat -m 255.255.248.0 -E5G 0/10.0.0.1/9800

– Receive flows from the exporter at 10.0.0.1 port 9800. Maintain 5 Gigabytes of flow files in /flows/dat. Mask the source and destination IP addresses contained in the flow exports with 255.255.248.0.

• flow-capture -w /flows/dat 0/0/9800 -S5– Receive flows from any exporter on port 9800. Do not perfo

rm any flow file space management. Store the exports in /flows/dat. Emit a stat log message every 5 minutes.

Flow-cat

Flow-print

FreeBSD1# flow-print < ft-v01.2006-09-02.134114+0800srcIP dstIP prot sPort dPort octets pkts

202.204.79.253 202.204.239.227 6 4414 1433 48 1

202.204.79.253 202.204.239.229 6 4450 1433 96 2

202.204.79.253 202.204.239.240 6 4535 1433 48 1

202.204.79.253 202.204.239.228 6 4443 1433 48 1

202.204.79.253 202.204.239.233 6 4472 1433 96 2

202.204.79.253 202.204.239.231 6 4461 1433 48 1

Flow-stat

Flow-stat exam. 1

% flow-cat -p /flows/dat | flow-statIP packet size distribution: 1-32 64 96 128 160 192 224 256 288 320 352 384 416 448 480 .000 .906 .029 .004 .002 .009 .001 .001 .004 .027 .004 .000 .000 .000 .000

512 544 576 1024 1536 2048 2560 3072 3584 4096 4608 .000 .000 .001 .001 .012 .000 .000 .000 .000 .000 .000

Packets per flow distribution: 1 2 4 8 12 16 20 24 28 32 36 40 44 48 52 .812 .157 .010 .013 .006 .001 .000 .000 .000 .000 .000 .001 .000 .000 .000

60 100 200 300 400 500 600 700 800 900 >900 .000 .001 .000 .000 .000 .000 .000 .000 .000 .000 .000

Octets per flow distribution: 32 64 128 256 512 1280 2048 2816 3584 4352 5120 5888 6656 7424 8192 .000 .754 .183 .009 .012 .015 .014 .008 .004 .002 .000 .000 .000 .000 .000

8960 9728 10496 11264 12032 12800 13568 14336 15104 15872 >15872 .000 .000 .000 .000 .001 .000 .000 .000 .000 .000 .001

Flow time distribution: 10 50 100 200 500 1000 2000 3000 4000 5000 6000 7000 8000 9000 10000 .812 .000 .000 .000 .000 .000 .001 .131 .015 .001 .004 .004 .004 .002 .001

12000 14000 16000 18000 20000 22000 24000 26000 28000 30000 >30000 .000 .001 .001 .002 .001 .000 .002 .001 .001 .000 .017

formats

Flow-stat exam. 2

• flow-cat -p /flows/dat | flow-stat -f10 -S4– Provide a report on top source/destination

IP pairs sorted by octets

# Fields: Total# Symbols: Disabled# Sorting: Descending Field 4# Name: Source/Destination IP## src IPaddr dst IPaddr flows octets packets#202.204.192.1 10.20.0.12 1 3720 12 202.204.192.1 10.20.0.8 3 3128 11 202.204.192.1 10.20.0.9 2 3269 11 202.204.193.1 64.84.7.4 1 390 3 202.204.204.148 221.137.69.66 3 144 3 216.186.143.246 202.204.227.118 1 144 3 202.204.79.253 202.204.239.233 1 96 2

Flow-scan

Netflow in CERNET-POP Traffic Statistics

Netflow in CERNET-POP PPS Statistics

Netflow in CERNET-POP Average Packet Size Statistics

Netflow in CERNET-POP Protocol Statistics

Thank You!

• Most materials in this PPT is from network, thanks goes to the authors

• Any Questions?

Developed by Cisco Systems in 1996 The value of information in the cache was a secondary discovery –Initially designed as a switching path NetFlow is now.

Documents