Top Banner
Clean Pipes 2.0
39
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Page 1: Clean Pipes Whitepaper 1

 

 

 

 

 

    

Clean Pipes 2.0  

Page 2: Clean Pipes Whitepaper 1

 

 

Clean Pipe Solution 2.0  Executive Summary .................................................................................................................................3 

Best Current Practices ..........................................................................................................................5 Network Infrastructure BCPs ....................................................................................................................... 5 Host Based BCPs ................................................................................................................................................ 5 Dedicated DDoS BCPs ...................................................................................................................................... 6 

Cisco Clean Pipes Solution Overview ..................................................................................................6 Evolution of Cisco Clean Pipes Solution ....................................................................................................... 6 Protection Mechanism of Cisco Clean Pipes Solution ............................................................................... 8 

Cisco Clean Pipes 2.0 Components ................................................................................................... 10 Cisco NetFlow ................................................................................................................................................... 10 Arbor Peakflow SP .......................................................................................................................................... 11 Arbor Peakflow SP Threat Management System (TMS) ....................................................................... 13 

DDoS Protection Flow in Clean Pipes 2.0 ........................................................................................ 15 1.  Baseline and Thresholds ....................................................................................................................... 15 2.  Detection ................................................................................................................................................... 15 3.  Diversion ................................................................................................................................................... 15 4.  Scrubbing ................................................................................................................................................. 16 5.  Injection .................................................................................................................................................... 16 

Migration to Cisco Clean Pipes 2.0 ................................................................................................... 17 

Cisco Anomaly Guard and TMS Countermeasure Comparison ........................................... 18 

Clean Pipes 2.0 Deployment Considerations ............................................................................. 19 NetFlow considerations for Peakflow SP .............................................................................................. 20 Deployment of Peakflow SP PI .................................................................................................................. 21 Deployment of Peakflow SP Collectors .................................................................................................. 21 Deployment of Peakflow SP TMS .............................................................................................................. 23 Peakflow SP Communication Ports ......................................................................................................... 24 Data collection .............................................................................................................................................................. 24 Inter‐appliance communication (all appliances) ........................................................................................... 25 Peakflow SP PI and leader appliances ................................................................................................................ 25 

Scaling Clean Pipes 2.0 ................................................................................................................................. 25 

Best Practices in Cisco Clean Pipes 2.0 ........................................................................................ 26 

Conclusion ............................................................................................................................................. 28 

Appendix ................................................................................................................................................ 29 Appendix­A: Peakflow SP System­wide Enforced and Guideline Limits .................................... 29 Appendix­B: Peakflow SP PI 5500 Appliance Enforced and Guideline Limits ......................... 32 Appendix­C: Peakflow SP CP 5500­series Appliance Enforced and Guideline Limits ........... 33 Appendix­D: Peakflow SP TMS Appliance Limits ............................................................................... 35 Appendix E: Six­Phase Approach to Infrastructure Security ......................................................... 38 

Page 3: Clean Pipes Whitepaper 1

 

 

 

Executive Summary Distributed denial-of-service (DDoS) attacks target network infrastructures or computer services resources. The primary goal of DDoS attacks is to deny legitimate users access to a particular computer or network resources, which results in service degradation, loss of reputation, and irretrievable data loss. DDoS attacks are aimed at organizations of all sizes and types that have an online presence, including businesses, government agencies, academic institutions, and even individuals. Many enterprises are migrating to cloud computing models, making use of centralized data centers and virtualization, to reduce capital and operations expenses. The data centers that house these large virtualized data stores are particularly sensitive targets to DDoS attack as a single attack can produce considerable collateral damage beyond the direct victim. DDoS has evolved from random hacker exploits to organized criminal activities that often involve botnets, which are large groups of compromised host computers controlled by a central commander. The size, complexity, and sophistication of DDoS attacks are increasing at alarming rates. According to Arbor Networks’ Annual Worldwide Infrastructure Security Report (2009), Internet service providers have seen DDoS attacks as large as 49 Gbps. The numbers have increased more than a hundredfold since 2001, more than 100 percent since 2007, and 23 percent since 2008. The recent slowing in DDoS growth is likely a result of attacks reaching underlying Internet physical constraints and a migration to more effective types of denial-of-service attack. To address these inevitable and growing network threats, network operators are urged to employ the best current practices (BCP) for protecting networks. BCPs are proactive methods that have been adopted in the industry to prepare networks against threats. BCPs include network infrastructure best practices, host best practices, and deployment of dedicated DDoS detection and mitigation solutions. Cisco® Clean Pipes Solution is a purpose-built architecture for dedicated DDoS detection and mitigation. As opposed to traditional DDoS defense techniques, the Cisco Clean Pipes solution can accurately distinguish legitimate traffic from malicious traffic destined for a mission-critical host or application. It precisely blocks the attack traffic while allowing legitimate traffic to pass through. The Clean Pipes Solution allows service providers to deliver in-cloud, managed anti-DDoS services to their customers. It also provides enterprises with the ability to defeat DDoS attacks on their own premises with surgical DDoS attack detection and protection. Given the constant evolving nature of DDoS attacks, the Cisco Clean Pipes Solution has also rapidly evolved. It started with Cisco Anomaly Guard appliance, which had 1 Gbps mitigation capability and was followed by the Cisco Anomaly Guard Module, which provided up to 3 Gbps mitigation per module and could be clustered to offer 10+ Gbps protection.

Page 4: Clean Pipes Whitepaper 1

 

 

Cisco is partnering with Arbor Networks to continue to provide a comprehensive and tightly integrated anti-DDoS solution, which will evolve the Cisco Clean Pipes Solution to version 2.0. In Clean Pipes 2.0, Cisco and Arbor will closely collaborate in the integration of Cisco NetFlow technology and Arbor DDoS detection and mitigation technology to provide more advanced and higher performance anti-DDoS protection. Cisco is no longer developing the Cisco Anomaly Guard Module and the Cisco Anomaly Detector Module. Arbor Networks is known for its Peakflow® SP solution, which provides comprehensive DDoS detection, surgical mitigation, and reporting for service provider networks. The partnership between Arbor Networks and Cisco partnership is not new as Arbor’s Peakflow solutions have long made use of Cisco’s NetFlow technology. Also, Arbor’s Peakflow SP product was a supported option for attack detection in Clean Pipes Solution 1.0 and 1.5. For Clean Pipes 2.0, Arbor’s Peakflow SP product will be used for anomaly detection, while the Peakflow SP Threat Management System (TMS) product will be used for surgical mitigation of DDoS attacks. For customers using Clean Pipes solution 1.0 or 1.5 today, Clean Pipes 2.0 is the migration path to achieve higher scalability and new functionalities in the future. Cisco and Arbor will continue the joint effort of creating an ever more tightly integrated anti-DDoS solution.

Page 5: Clean Pipes Whitepaper 1

 

 

Best Current Practices  The industry has agreed on several best current practices (BCPs) that should be proactively deployed by network operators responsible for Internet-facing infrastructure and properties. These practices were established in accordance with the Cisco-Arbor developed six phases of infrastructure security.

Network Infrastructure BCPs  Network  infrastructure  BCPS  are  proactive  measures  that  are  implemented  directly  on Cisco router and switching infrastructure along with other network devices.  

• Interface ACLs should be employed at the relevant network edges (peering/transit, customer aggregation edge, and so on) to protect the network infrastructure itself.

• Service-specific ACLs should be used on data center routers to restrict traffic destined for Internet-facing servers to the ports and protocols associated with the services and applications on those servers. 

• Control and management plane protection mechanisms should be deployed in accordance with device, protocol, and vendor recommendations.

•  All network infrastructure devices should be accessible only through designated management hosts, and this access should be facilitated through a dedicated out-of-band management network.

• Flow telemetry using Cisco NetFlow should be enabled at all network edges and exported into a collection/analysis system such as Peakflow SP.

• Source-based remotely triggered blackholing (S/RTBH) is a powerful reaction technique that allows thousands of attacking source IPs to be rapidly blackholed on the basis of their source addresses. S/RTBH uses BGP as a control-plane mechanism to instantaneously signal edge devices to start dropping attack traffic.

• Reverse-proxy caching in front of Internet-facing web properties allows for scaling of capacity as well as a policy control point which enables filtering of Layer 7 application protocol traffic.

Host‐Based BCPs  Host‐based  BCPs  are measures  that  are  applied  directly  on  hosts  that  may  come  under attack and provide a degree of initial protection. They include:  

• Proactive patching of the host. • Server  hardening,  including  the  shutdown  of  any  unnecessary  services  and  host‐

based  ACLs,  restricting  access  to  the  server  to  only  specific  source  hosts  and  on specific ports. 

Page 6: Clean Pipes Whitepaper 1

 

 

• Out‐of‐band management access to the device. • Service  specific  configuration  hardening,  including  shutdown  of  unused  features 

and access mechanisms. • IP stack tuning. • Employing antivirus and antispam mechanisms. 

 

Dedicated DDoS BCPs  The  use  of  dedicated  DDoS  detection  and  Intelligent  DDoS  Mitigation  Systems  (IDMS) completes the net of a comprehensive DDoS protection infrastructure. DDoS detection and IDMS  devices  are  designed  to  detect  DDoS  events  as  they  occur,  provide  traceback  and analysis to operators, and intelligently mitigate attacks by dropping malicious traffic while preserving legitimate traffic. Cisco Clean Pipes is the industry BCP for dedicated DDoS and mitigation.  

Cisco Clean Pipes Solution Overview The Cisco Clean Pipes Solution enables service providers to provide DDoS protection services to their customers and simultaneously harden and protect their own networks. Enterprise customers can also deploy Cisco Clean Pipes Solution on their own premises to protect their network infrastructure and server resources from DDoS attacks.

Evolution of Cisco Clean Pipes Solution The essentials of Cisco Clean Pipes Solution are the DDoS attack detection and mitigation devices. Table 1 shows Anti-DDoS devices in Clean Pipes 1.x and 2.0. Table 1 Anti-DDoS Devices in the Cisco Clean Pipe Solution Detection Device Mitigation Device Clean Pipes 1.0 & 1.5 Cisco Anomaly Detection

Appliance; Cisco Anomaly Detection Module for Cisco Catalyst 6500/ Cisco 7600; Arbor Peakflow SP;

Cisco Anomaly Guard Appliance; Cisco Anomaly Guard Module for Cisco Catalyst® 6500/ Cisco 7600 Series

Clean Pipes 2.0 Arbor Peakflow SP Arbor Peakflow SP Threat Management System (TMS)

Page 7: Clean Pipes Whitepaper 1

 

 

In Clean Pipes 1.0 and 1.5, detection could be done by either Cisco Anomaly Detection Appliance/Anomaly Detection Module or Arbor Peakflow SP. Cisco Anomaly Guard Appliance/Anomaly Guard Module was the featured attack mitigation devices. Clean Pipes 2.0 features Arbor Peakflow SP for detection and Arbor Peakflow Threat Management System for mitigation. Cisco routing and switching devices will provide Arbor Peakflow SP with NetFlow telemetry information, which Peakflow SP uses to analyze and establish network traffic profile and detect traffic anomaly. Differences exist among Cisco and Arbor anti-DDoS devices in terms of how they function for DDoS attack detection and mitigation. Cisco Traffic Anomaly Detection vs. Arbor Peakflow SP Both Cisco Traffic Anomaly Detection Appliance and Cisco Traffic Anomaly Detection Modules are packet-based anomaly detectors. They monitor a mirrored copy of selected inbound traffic flowing toward destinations under protection, building detailed profiles of normal behavior of each protected devices. Any activities deviating from these profiles can be potential attacks. If it senses abnormal or anomalous behavior, the Cisco Traffic Anomaly Detection device dynamically configures a set of filters to record the events and trigger an alarm to the network staff. It can also signal Cisco Anomaly Guard devices to activate the protection and mitigation if configured to do so. Arbor Peakflow SP is a Netflow-based anomaly detector. It receives NetFlow telemetry from Cisco routers and switches in the network. It continually models network behavior based on the Netflow statistics, creating baselines of expected traffic rates. Any events deviating from the established baseline model are identified as an anomaly and trigger an alert that can lead to further actions, including informing the network staff and activating the DDoS protection function on the mitigation device. Cisco Traffic Anomaly Guard vs. Arbor Threat Management System (TMS) Cisco Traffic Anomaly Guard XT and Cisco Traffic Anomaly Guard Modules are designed as central intelligence devices that are capable of both detecting and mitigating attacks, once they have been activated. When the Cisco Traffic Anomaly Detector Module identifies a potential attack, it alerts the Cisco Anomaly Guard Module to begin dynamic diversion, which redirects traffic destined for the targeted resources, and only that traffic, for inspection and scrubbing. From the point of this diversion, Cisco Guard operates independently from other devices. It applies blocking techniques based on Cisco's multilayer verification process architecture, which delivers multiple interactive layers of defense to identify and block all types of attacks.

Page 8: Clean Pipes Whitepaper 1

 

 

Arbor’s Peakflow SP Threat Management System (TMS) provides centralized cleaning capacity in the network, providing the active packet level processing needed to thwart complex attacks. TMS maintains active communication with the Peakflow SP system for ongoing exchange of mitigation activities, health of the TMS scrubbing capacity, real-time data exchange for mitigation, and supplemental application visibility. The TMS features a set of mitigation countermeasures that isolate and block malicious traffic while passing desirable traffic. The countermeasures available currently include anti-spoofing, host authentication techniques, packet level threshold, application-specific threshold, protocol verification, baseline enforcement, idle discovery, blacklist/whitelist, and payload filtering techniques. Countermeasures are continuously added or updated as new threat vectors emerge. In addition, extensive real-time and post-mitigation reports are available on the Peakflow SP system, allowing operators to make more informed decisions on how to adapt defenses during and between attacks. The Peakflow SP system provides a single interface for command, control, and reporting for up to 50 TMS devices working in parallel. Together, the Peakflow SP system and the TMS devices provide a comprehensive threat management solution for the entire network.

Protection Mechanism of Cisco Clean Pipes Solution DDoS attacks are difficult network threats to defend against. They mimic valid requests, spoof source identification, and use armies of compromised “zombie” hosts to initiate attacks so that illegitimate packets are indistinguishable from legitimate packets. This makes detection more difficult and threat mitigation with business continuity more challenging. Network devices and traditional perimeter security technologies such as firewalls and intrusion detection systems do not by themselves provide adequate DDoS protection. To pick up where traditional DDoS defense techniques leave off, the Cisco Clean Pipes solution can accurately distinguish good traffic from bad traffic destined for a mission-critical host or application. It not only detects the presence of an attack, but also filters out only the bad traffic, allowing good traffic to pass through, enabling maximum business and service continuity. This solution offers three major functional elements that work toward protecting a network from DDoS attacks:

Detection

Identify and classify

attacks based on anomaly characteristics

Diversion/Injection

Divert “dirty” traffic to the

cleaning center to be “scrubbed” Inject clean traffic back to the

DDoS targeted host

Mitigation

Antispoofing, anomaly recognition, and packet

inspection Cleaning(scrubbing) of

“dirty” traffic

Page 9: Clean Pipes Whitepaper 1

 

 

• Detection: The fundamental premise of detecting attacks is to build a baseline of normal network traffic levels and then look for anomalies in traffic patterns compared with the baseline. A network traffic anomaly is an event or condition in the network characterized by a statistical abnormality compared to typical traffic patterns gleaned from previously collected profiles and baselines. Any differences in traffic patterns that are above a certain threshold trigger an alarm.

• Traffic diversion and injection: Traffic diversion is the mechanism used to instruct an

upstream router in the core network to divert traffic of compromised servers to the mitigation devices for scrubbing the “dirty” traffic. After scrubbing off anomaly packets, the cleaned traffic is injected back to the normal data path to reach the destination in the network. There are multiple mechanisms for traffic diversion and injection that are discussed in later sections.

• Mitigation: Mitigation in the Cisco Clean Pipes solution is the process in which attack

traffic is “scrubbed,” that is, checked via anti-spoofing, anomaly recognition, packet inspection, and cleaned to drop bad traffic and allow legitimate traffic to the same destination.

The figure below depicts the typical Cisco Clean Pipes Solution architecture.

In general, Cisco Clean Pipes Solution provide four service deployment models, based on the common Clean Pipes architecture, along with design guidelines tailored for DDoS protection for different parts of the SP infrastructure and customer networks:

• Managed Network DDoS Protection provides enterprises effective protection against DDoS attacks on their last-mile connections to service providers and internal infrastructures through the Cisco Clean Pipes service offered by service providers.

Page 10: Clean Pipes Whitepaper 1

 

 

• Managed Hosting DDoS Protection enables hosting providers to protect their web and

other hosting services from DDoS attacks.

• Peering Edge DDoS Protection enables service providers to prevent bandwidth saturation by DDoS attacks against their peering points.

• On-premise DDoS Protection enables enterprises to deploy anti-DDoS detection and

protection on their own premises with finer granularity for anomaly detection and protection.

Cisco Clean Pipes 2.0 Components

Cisco NetFlow NetFlow has become a standard for acquiring IP operational data for many customers. Applications for NetFlow data are constantly being invented, one of which is anti-DDoS protection. The highly scalable view of network traffic characteristics provided by NetFlow data makes NetFlow technology the most widely deployed DDoS identification technology for large-scale IP networks. At the same time, the granular flow information enables NetFlow-based DDoS detection devices, such as Arbor Peakflow SP, to provide surgical detection of traffic anomalies.  NetFlow classifies IP packets into flows and generates flow records that can be exported to a flow collector for further analysis. Each flow is defined by its unique seven-key characteristics:

• Ingress interface • IP protocol type • Type-of-service (ToS) byte • Source IP address • Destination IP address • Source port number • Destination port number

Led by Cisco core routing and switching platforms, Cisco NetFlow technology has been constantly enhanced and refreshed, with features such as:

• Supports multiple NetFlow formats • Industry’s premier platform to support v9 • Highly scalable NetFlow table • Flexible NetFlow which allows users to select which key or non-key fields to

define a flow • Time-based sampled NetFlow

Page 11: Clean Pipes Whitepaper 1

 

 

• Packet-based sampled NetFlow • NetFlow for both ingress and egress traffic • NetFlow for MPLS and Multicast traffic • NetFlow for bridged traffic (enables bump-in-the-wire deployments)

When the network is operating under a normal situation, NetFlow yields enough data to profile the network traffic and establish a baseline, which is used for traffic anomaly detection. In the event of DDoS attacks, the statistical NetFlow information shows deviations from the traffic baseline that may be the first sign of attacks. Further analysis of traffic pattern and behavior can be carried out with the detailed flow information. Once a traffic anomaly is identified, countermeasures can be initiated manually by the network operator or automatically by the anti-DDoS protection system. When NetFlow is used in anti-DDoS protection, it is usually deployed across the edge of a service provider or enterprise network to monitor inbound traffic on edge and peer interfaces, because these are the typical ingress points for most attacks. The router maintains a live NetFlow cache to track the current flows. IP flow information can be exported from the NetFlow cache to an external collector for further analysis. In Clean Pipes 2.0, Arbor Peakflow SP is the flow collector. Flow data from multiple collectors can be mapped to identify the network nodes under DDoS attack and also to determine the attack characteristics. Cisco continuously invests in high-performance NetFlow technology and collaborates with Arbor Networks to ensure that the Clean Pipe Solution 2.0 is enhanced with the newest NetFlow features. For more information about NetFlow, visit: http://www.cisco.com/en/US/tech/tk812/tsd_technology_support_protocol_home.html

Arbor Peakflow SP Arbor Networks Peakflow SP is a scalable platform that provides a comprehensive solution delivering powerful DDoS protection as well as traffic and routing analysis to service providers and their customers. Peakflow SP provides three leading solutions to the marketplace: managed security services enablement, infrastructure security, and traffic and routing visibility and analysis. The Peakflow SP solution scales with its multitier detection architecture of collectors:

• Tier 1: Peakflow SP Portal Intelligence (PI), provides a central point of command and control including event correlation and traceback. PI systems provide the leader and central command function for the deployment.

• Tier 2: Peakflow SP Collector Platform (CP), collects NetFlow statistics from multiple routers and acts as a correlation engine syncing data sets between all

Page 12: Clean Pipes Whitepaper 1

 

 

network collectors and the PI system. • Tier 3: Peakflow SP Flow Sensor (FS), acts as additional layer of NetFlow and data

collection designed to scale the Peakflow deployment to the largest worldwide networks.  

The figure below shows the Clean Pipes 2.0 Solution architecture.

For Clean Pipes v1.5, the Peakflow SP solution works in conjunction with the Cisco Guard for DDoS protection. Upon receiving an anomaly fingerprint for a zone from a Peakflow SP CP collector, the Peakflow SP PI controller establishes a SSH connection to activate the Cisco Guard, putting the zone under attack in protection mode. For the Clean Pipes 2.0 Solution, Peakflow SP offers a streamlined approach to DDoS attack detection, traceback, and mitigation. Peakflow SP CP systems first build baselines of normal behavior across the network, using flow data available from existing routers. In contrast to inline data collection methods, Peakflow SP collects Cisco NetFlow flow-based statistics from Cisco routers, which allows Peakflow SP to scale with the network. Alternatively, Peakflow SP TMS can use packet capture/SPAN ports on routers in which NetFlow is not available. Neither NetFlow nor packet capture imposes a performance or reliability impact upon the network; the data collection is nonintrusive. The Peakflow SP networkwide anomaly detection identifies attacks using the two most effective

Page 13: Clean Pipes Whitepaper 1

 

 

methods available: signature analysis and dynamic profiling. Arbor’s Active Threat Feed (ATF), a data feed of traffic signatures that pinpoint potential threats and concerns to network security, can be used to match traffic reported through NetFlow. Alerts based on ATF matches can be reliably detected through NetFlow analysis. Peakflow also actively detects anomalies through misuse identification and dynamic profile detection. NetFlow provides Peakflow SP the perspective to run signature analysis pervasively with a high level of accuracy while augmenting that networkwide visibility with targeted packet processing analysis of suspect traffic through Arbor’s Peakflow SP TMS.

Arbor Peakflow SP Threat Management System (TMS) Arbor Network’s Peakflow SP Threat Management System (TMS) provides surgical mitigation, service analysis and reporting. TMS provides scrubbing and application-specific visibility to the Peakflow SP system. TMS can be deployed in centralized scrubbing locations, regional service points of presence, and IDCs for infrastructure protection and clean pipes. The TMS can also be deployed as a dedicated solution for specific service protection and visibility. Arbor Networks TMS is a separate hardware platform for advanced, high-speed traffic scrubbing and analysis. The TMS product family contains systems that provide throughput from 1.5 Gbps up to 40 Gbps. The range of TMS models offered provides the correct performance package for each of the multiple use-cases of the Clean Pipes 2.0 Solution. Each system provides the same feature and functionality at different performance levels to meet the desired use case. Deployed in centralized locations for large scrubbing center architectures, the 4000 series TMS models feature a multi-slot chassis that can provide from 10 to 40 Gbps performance. The 5 Gbps TMS-3050 and 10 Gbps TMS-3110 are also available for smaller scrubbing center locations. Multiple deployments of TMS distributed throughout the network within the network can provide a distributed response to a coordinated DDoS event. Grouping multiple TMS systems into a single logical entity provides network operators the best solution to difficult trade-offs of backhauling attack traffic across the network or requiring each of the systems to be individually configured and managed. Grouping the TMS systems provides the ability to maintain a disturbed attack to a geographically limited set of locations and protects collateral damage of network assets. Using Peakflow SP for central command and control, up to 50 TMS systems can be grouped together to mitigate attacks up to 2 Tbps.

Page 14: Clean Pipes Whitepaper 1

 

 

The figure below shows the range of TMS models available in the Clean Pipes 2.0 Solution.

TMS can also be offered as a dedicated solution for specific service protection or customers of a Clean Pipes service. As a dedicated solution, TMS provides specific mitigation actions to specific customers. Taking advantage of the integration with Peakflow SP CP systems and the ability to model customer traffic through NetFlow analysis and saving these baselines in Managed Objects, each TMS can also employ a customer-specific template of mitigation countermeasures to ensure custom handling of the event with respect to sensitive traffic. The TMS interacts with distributed Peakflow SP CP systems in the network to both baseline data at Layer 7 as well as provide scrubbing statistics and forensic data from attack events. Advanced visibility within the attack event provides actionable data to the user while the mitigation is

Page 15: Clean Pipes Whitepaper 1

 

 

ongoing. This near-real-time interface provides superior management of DDoS events, providing the ability to drill into attack packets, correlate data common to the attack traffic streams, and measure the effects of countermeasure filters or REGEX expressions before they are employed into the configuration, helping ensure the smallest possible negative impact to good traffic during the event.

DDoS Protection Flow in Clean Pipes 2.0

1. Baseline and Thresholds Collection of NetFlow data  from various  router  locations  and  correlating  this data  into  a comprehensive  model  of  the  network  is  critical  to  developing  a  surgical  response  to threats. The Clean Pipes components provide this functionality natively in the solution and ensure  that  normal  network  traffic  variability  is  accounted  for  in  these  measurements. Further  developing  granular  models  of  network  assets,  customers,  services,  and infrastructure serves to provide accurate levels of granularity relative to the network scale for pinpoint detection. The Clean Pipes 2.0 solution provides the ability to accurately build thousands of models, each with relative baselines, thresholds, and traffic pattern reporting to scale to the very largest service offerings and global networks. 

2. Detection Data retrieved from NetFlow updates provided by Cisco router infrastructure is correlated to  the  baseline  and  threshold  data  held  in  the  Peakflow  SP  system.  Clean  Pipes  2.0 identifies threshold violations and provides actionable information to the operations teams as  rapidly  as  possible.  The  system  then  provides  options  to  the  operator  who  has  the options of auto‐mitigation, manual mitigation, use of layered mitigation techniques such as ACL or interface level filters on Cisco infrastructure, or blackholing traffic. If the attack is to be mitigated through the scrubbing technology such as TMS, then the operator initiates a diversion event.  

3. Diversion Diversion surgically redirects traffic into scrubbing locations using the BGP control plane, removes  traffic attacking  the network, and passes  it  to  locations of  the network with  the scale  and  the  scrubbing  systems  deployed  to  mitigate  the  attack.  Architecture  goals  of following best practices have greatly enhanced methods  for diversion and distribution of diverted traffic to scrubbing systems. Methods to accomplish this include BGP Anycast, BGP route‐maps,  and  BGP  community  use,  all  of  which  can  help  ensure  that  the  network  is resilient to the threat and maintaining a self‐defending network.  

Page 16: Clean Pipes Whitepaper 1

 

 

4. Scrubbing Scrubbing  involves  identification  of  malicious  and  legitimate  traffic  through  DPI  Packet analysis, heuristics, and validation methods called countermeasures. Each countermeasure can provide additional granular  identification of  traffic. Malicious  traffic  is  removed  from the traffic stream and legitimate traffic is placed back into the network. All actions taken by the TMS are reported in both the real‐time mitigation report and the after‐action reports for the event. 

5. Reinjection Post‐processed  legitimate traffic  is  then routed to  the ultimate destination of  the original traffic  flow.  As with  diversion,  architectures  exist  to  optimize  injection  paths  and  traffic delivery  to victims of  attacks. These methods  can  take advantage of network capabilities enabled by Cisco routers and enable the Clean Pipes provider to add additional dedicated, clean capacity into the destination network.    The following diagram maps the DDoS protection flow in Clean Pipes 2.0.

Page 17: Clean Pipes Whitepaper 1

 

 

Migration to Cisco Clean Pipes 2.0 As the Cisco Guard and Detector Modules approach the end of their lifecycle, the Arbor Peakflow SP TMS replaces the scrubbing technology in the Clean Pipes solution. The migration from the Cisco Guard to the TMS can be achieved through a mapping of protected resources configured in the Cisco Guards (Zones) to Peakflow SP (Managed Objects) as well as deployments of TMS in similar architectures as the Cisco Guard. Understanding the terminologies of Cisco Guard and Arbor PS/TMS and knowing how to map them will greatly increase knowledge transfer from deployments of Clean Pipes 1.0/1.5 with Cisco Guard appliance/module for successful migration to Clean Pipes 2.0 with Arbor TMS. The table below shows the mapping of key concepts between these two technologies. Cisco Guard  Arbor Peakflow SP TMS Common DefinitionZone  Managed Object  The basic model that builds baseline, detection,

mitigation, and reporting. Model definitions are flexible and can be combined with many layers of data to specifically match critical areas of interest on the network. These models are used to monitor customer, peer, service, or profiled relationships. 

-  Boundary  A demarcation point between administrative domains. Peakflow SP immediately builds a topological map of the monitored network using the network definition as the default global boundary. Boundaries are flexible and can be inherited, such as a global boundary, or specifically configured relative to the Managed Object monitored. The global boundary defines the point traffic enters or exits the monitored network. 

Baseline  Baseline  Collection of traffic behavior profiles building the expected traffic volume and anomaly detection thresholds. 

Protect mode / Zone protection enabled 

Mitigation   Enabled scrubbing configuration. This can include details about the destination, BGP prefix used to change traffic path, and active and passive filtering rules. 

Filter  Countermeasure  Rule describing an evaluation of traffic to be scrubbed by the solution  

Template  Mitigation Template Preset configuration information used to protect a destination from specific vectors or to use specific filters in protection. 

Page 18: Clean Pipes Whitepaper 1

 

 

Diversion  Off-ramp  BGP Prefix announcement of destination with a change in Next-hop attribute to ensure that inbound traffic will pass through the scrubbing solution. 

Re-injection  On-ramp  Returning the cleaned traffic from the scrubbing system to the network in a loop-free path to the protected destination. 

Cisco Anomaly Guard and TMS Countermeasure Comparison  Cisco Anomaly Guard and Arbor Peakflow SP TMS provide significant mitigation capabilities based on available countermeasures. These countermeasures are designed to deal with the current common DDoS attack types and also help protect a target from a zero-day attack. Both solutions provide strong protection from spoofed source attacks and resource consumption attacks at the application layer, session layer, or network layer. Both solutions provide defenses to brute-force flooding attacks such as packet-per-second floods, TCP connection floods, and UDP and ICMP floods, to name just a few examples of common attacks. TMS provides for advanced capabilities to filter web-enabled (HTTP-based) services through authentication, validation, request tracking, and limits as well as payload filtering. Each of these countermeasures, or any set of them, can be brought into service across a group of TMS systems through the configuration of a single mitigation. This capability sets the TMS apart from previous mitigation solutions for large, distributed network deployments.

Mitigation Countermeasure Function TMS Cisco Guard Whitelist / blacklist filtering √ √ Per source IP rate thresholds √ √

TCP SYN Authentication with reset to Host √ √

TCP SYN Authentication with refresh sent to host Target Release 5.5 Q3 2010 √

TCP SYN Authentication with HTTP Authentication √ √

TCP SYN Authentication with safe reset to host √ √

TCP SYN ACK Authentication √ √

TCP other flag authentication √ Basic/Default authentication of other protocols for client based on passed TCP authentication √

Strong Mode TCP Authentication using proxy * √ DNS Authentication through packet drop / retransmission √ DNS Authentication By Reflexive-Redirection √ DNS Request Type Limiting By Source /32 √

Page 19: Clean Pipes Whitepaper 1

 

 

DNS Cache Poison Defense * √

DNS Authentication by converting to TCP Target Release 5.5 Q3 2010 √

Strong Mode DNS Auth using TCP and TTL √ Regex based filtering √ √

DNS DPI REGEX Filtering √ ?

HTTP Header REGEX Filtering √ √

Protocol baseline enforcement √ √

Source /24 based baseline enforcement √

Connection metrics based baseline enforcement Target Release 5.5 Q3 2010 √

Rate limiting √ √

Malformed HTTP √ Malformed SIP √ Malformed DNS √ √

SIP Authentication √ SIP source request thresholding √ HTTP source IP rate thresholding √ HTTP source IP object get rate thresholding √ TCP Idle Timeout √ √ TCP multiple bad connection blacklisting √

* Requires the mitigation system to be in-line in both directions Despite the differences in specific filtering options available, each of the mitigation systems have demonstrated an ability  to effectively mitigate  the  types  of  threats  that are  seen on the Internet today. Cisco and Arbor performed joint testing on the mitigation capabilities of the TMS system and showed that the TMS was able to mitigate the same types of attacks that the Guard was capable of mitigating. It sometimes used similar methods as the Guard solution and sometimes used different, equally effective methods. 

Clean Pipes 2.0 Deployment Considerations The Clean Pipes 2.0 solution provides a wide variety of protection values.

• Managed DDoS Detection and Protection Services: Arbor Peakflow SP provides a solution that allows providers to offer a turnkey managed DDoS service. Managed objects monitor each component of the subscribed customer network and services.

• Managed Hosting DDoS Protection: Protection of critical services at the application layer with unique capabilities to further differentiate increasingly sophisticated attacks at the application layer. Managed services offerings can be offered granularly

Page 20: Clean Pipes Whitepaper 1

 

 

to customers, protecting applications or critical services, or protecting the entire customer site.

• Peering Point Bandwidth Protection: Detection and mitigation capabilities at the peering points can protect against collateral damage from transit of DDoS attacks.

• Network Service Protection: Arbor Networks TMS can provide service-specific application reporting or dedicate DDoS protection to critical network services.

NetFlow considerations for Peakflow SP NetFlow telemetry is an  intrinsic part of  the Peakflow SP solution. Peakflow SP leverages NetFlow data  to provide operators complete visibility  into network  traffic characteristics and  rates,  create  baselines  of  normal  traffic,  detect  deviations  from  these  baselines  that may be due to threats, characterize the threats, and trace them back to network borders. In short,  NetFlow  provides  the  basic  building  blocks  for  each  of  the  six  phases  of infrastructure security.  Pervasive monitoring of the network is a necessary component of the Clean Pipes solution, so NetFlow  telemetry  should  be  exported  from all  Cisco  routers where  critical  data may traverse including those in the network peering, core, distribution, and data center.   It is recommended that ingress NetFlow be enabled on all logical interfaces for each router being monitored. This provides the Peakflow solution with a full picture of what traffic  is going  through  the  router  regardless  of  direction.  Peakflow  has  the  ability  to  focus  on desired traffic within each NetFlow data stream, so it is not necessary to filter what traffic is sent to the Peakflow SP collectors.   Unsampled NetFlow provides accurate flow information on the network traffic that can be used by features, such as ATF and fingerprints, that rely on matching certain behaviors for more  effective  triggers.  However,  when  network  traffic  is  high,  unsampled  NetFlow  can limit the scalability of anomaly detection due to the excessive processing workload on the flow reporting devices and  the Peakflow appliance.  In  this case,  sampled NetFlow can be implemented. The appropriate sampling rate to apply depends on what type of router it is, how  much  traffic  is  going  through  the  router,  and  what  line  cards  are  available  in  the router. Sampling rates can go over 1000:1 in large hardware‐accelerated platforms (Cisco CRS‐1,  Cisco  Nexus®  7000).  In  general  the  more  traffic  going  through,  the  higher  the sampling rate to apply.   When exporting NetFlow from routers that are carrying IPv6 or MPLS traffic, it is necessary to utilize NetFlow v9 with IPv6 and MPLS explicitly enabled to get visibility into this traffic.  The following are specific considerations when enabling NetFlow on Cisco devices:  

• NetFlow versions 5, 7, and 9 are supported on Peakflow devices. 

Page 21: Clean Pipes Whitepaper 1

 

 

• NetFlow export  should  always be  set  to  a 1 minute  active  flow  timeout  to  ensure that real‐time analysis can be done on longer lived connections. 

• On Cisco Catalyst switches, full‐interface flow mask should be used to ensure that all NetFlow fields available will be populated. 

• Peakflow SP supports a single sampling rate per network device, so do not configure multiple sampling rates on a single router. 

Deployment of Peakflow SP PI  The Peakflow PI appliances provide users direct access to the data stored and distributed in the collector devices in the network. The PI functions as a leader to coordinate all data reports, alerts, and system health data from the individual collectors and then present that data in a unified view. The PI appliance provides for secure access to user, administrators, and customer service groups. Each PI deployed provides access to the deployment through the GUI or API. PI appliances support hot/hot active redundancy of up to 10 devices and together act as a unified access point into the deployment. Administrative, DDoS Alert, and API access is synced between PI systems to ensure that each system provides equal levels of access based on centralized access, accounting, and user privileges. For the hardware enforced scalability limits and the guidelines for scaling the Peakflow SP PI deployment, refer to Appendix B.

Deployment of Peakflow SP Collectors Peakflow SP collectors provide distributed data collection and detection in the network deployment. Detection for specific customers or services is done by processing and correlation of NetFlow information from any point that traffic passes through the network. The Peakflow system has the ability to scale to the network cloud for detection and reporting. By intelligently matching traffic to the object definition within the system, each configured managed object baseline is developed constantly and reported virtually. Placement of the collector is critical to optimize detection and reporting visibility for the protection and reporting desired. Pervasively monitoring peering capacity, external border connectivity, and long-haul capacity typically is valuable in increased transit optimization. Protecting the network from off-net or transiting attack traffic is often the first level of protection in successful Managed DDoS services as well as Peering point protection. Deployments of Collector platforms at the aggregation level or in the network core ensure that on-net traffic does not become a threat to customers or network assets from internal attack. This layer of detection can be critical in isolation of internal network attacks, customer-to-customer attacks, capacity issues, and critical network service monitoring.

Page 22: Clean Pipes Whitepaper 1

 

 

Setting network NetFlow settings for sampling rate, export timing, NetFlow export locations, and pervasive enablement all affect the detection and reporting of data on the network. Peakflow SP can provide very accurate detection and reporting through sampled NetFlow processing. Evaluating the correct sampling ratio relies on both the ability of the router platform and the software version it is running. Recent versions of Cisco IOS® Software and Cisco router infrastructure provide extremely accurate and high-performance platforms for NetFlow export to Peakflow SP Collectors. Peakflow SP Collectors also manage downstream systems such as TMS. TMS reports health and mitigation statistics as well as additional NetFlow data back into the Peakflow SP collector. All inter-device communications are carried out over SSL. Through this secure connection, the collector and TMS have a dedicated link to exchange detection data, baselines, thresholds, and mitigation configurations. This link is secure and can be maintained over geographically separate devices. The following diagram illustrates deployment concepts of Peakflow SP CP and TMS.

  For the hardware enforced scalability limits and the guidelines for scaling the Peakflow SP CP deployment, refer to Appendix C.

Page 23: Clean Pipes Whitepaper 1

 

 

Deployment of Peakflow SP TMS  Deployment of the mitigation component of the Clean Pipes solution can be optimized for the specific network architecture with consideration of the specific protection values required. The TMS can be deployed within the peering layer or adjacent to the network border to ensure that off-net attacks are mitigated directly at the network edge. The benefits are a direct solution to peering point threats and the ability to keep good and bad traffic separated. Scrubbing capacity can be deployed in central locations to provide consistent service to customers. Regional mitigation or scrubbing centers provide dedicated locations where diverted traffic can be scrubbed and returned to its ultimate destination. Providing regionally or geographically based mitigation capacity can help avoid excessive backhaul of traffic. Dedicated TMS deployments can provide advanced value for specific infrastructure, customers, or services. Dedicated systems placed in data centers adjacent to the resource can ensure that application reporting, service-specific reports, performance data, change alerts, and packet-level forensics can serve normal operations but also improve troubleshooting and application visibility. Placing TMS appliances in data centers also provides protection for critical resources from sources within the network that may not be protected by peering and central scrubbing locations. Broadband consumers, infected hosts within the network, and customer networks must be considered potential threats when building defenses. The following diagram depicts TMS network deployment locations.

Page 24: Clean Pipes Whitepaper 1

 

 

Diversion of traffic into the TMS is most often triggered by an IP traffic routing change such as a BGP announcement changing the attack target destination route to a next-hop of the mitigation systems available. BGP route maps, anycast route announcements, and community attribute settings can all be used to design a solution for diverting traffic into TMS systems. The TMS provides the ability to group devices into a single event to gain efficiency in central management and help in scaling mitigation capabilities. In some cases diversion can also be done using local techniques at Layer 2. Policy-based routing architectures, static ARP table entries, and VLAN mappings all can provide for layer 2 diversions into the TMS systems. Reinjection of traffic through the TMS must avoid the chosen diversion method to avoid routing loops. Reinjection is typically done through GRE encapsulation from the TMS to the provider edge or customer-premises device. Another well-known method is to configure a MPLS VRF instance to separate the forwarding data from the diversion segment and the reinjection segment. Last, VLANs can be used to separate the forwarding path if the destination is reachable through a Layer 2 domain. Successful reinjection of cleaned traffic into the network sends traffic from the normal data path to a clean virtual pipe anywhere within the network. For the hardware enforced scalability limits and the guidelines for scaling the Peakflow SP TMS deployment, refer to Appendix D.

Peakflow SP Communication Ports  For Peakflow SP to function properly, the appropriate communication ports must be allowed through the network devices and firewalls for all appliances:

• NTP • SNMP management (recommended) • Management console < > CP/FS/PI/TMS

Data collection 

The following ports must be allowed through the firewalls so that the Peakflow SP CP, FS, and TMS appliances can collect data:

Port Traffic flow

BGP 179 TCP appliance < > routers

UDP flows any (default) • router > CP • router > FS

SNMP • CP > router • FS > router

Page 25: Clean Pipes Whitepaper 1

 

 

Inter‐appliance communication (all appliances) 

The following ports must be allowed through the firewalls so that the CP, FS and TMS appliances can communicate with other appliances:

Port Traffic flow

ArborFlow (31373 UDP) • FS > CP appliance • TMS > SP appliance (if ArborFlow is enabled)

HTTPS 443 • CP browser > SP leader appliance

SSL 40000-40030 (configurable) • CP/FS/PI > CP/FS/PI

 

Peakflow SP PI and leader appliances 

The following ports must be allowed through the following firewalls for the PI and leader appliances:

Port Traffic flow

HTTPS 443 CP browser > SP leader appliance

Remote Arbor services • remote services HTTPS/443 • routeviews (off by default) • ATF • anonymous statistics • fingerprint sharing

Local services • DNS • NTP • AAA/TACACS

 

Scaling Clean Pipes 2.0 Peakflow SP provides the largest network scale available today. Using a many-to-one monitoring model made possible by using NetFlow, the system can scale to monitor over 2200 routers

Page 26: Clean Pipes Whitepaper 1

 

 

within the network. As more and more collectors are deployed for visibility, each collector increases the total number of routers monitored. As the number of collectors increases, another layer of the detection hierarchy can be added to further increase its ability to provide detection across very large networks of routers. The Flow Sensor provides collector-level functions at the aggregation level of the network. This extension of the collector platform further increases the ability of Peakflow SP to provide coverage of the external border of the network as well as the internal aggregation edge for detection. TMS provides a range of capacity in the mitigation portion of the solution ranging from 1.5 Gbps to 50 Gbps throughput. Collectors can manage up to 50 TMS systems within a deployment, providing up to 2 Tbps of mitigation capacity. Peakflow SP allows for up to 1000 native managed objects on a system for baseline, threshold, and reporting data. This number can be scaled to a total of 10,000 managed objects monitored on a single deployment with the addition of Business Intelligence appliances. This additional component adds to the number of managed objects in incremental blocks of 500 to keep pace with network and service growth. Appendixes A through D provide the scalability numbers and guidelines for the Peakflow SP system and each individual appliance. Best Practices in Cisco Clean Pipes 2.0 General deployments of Clean Pipes Solutions follow a set of known best practices to ensure the successful delivery of the service. Operational experience and problems have improved these practices over the course of the last decade.

• Operationalize the Six Phases of Security Best Practice. Jointly developed by Arbor Networks and Cisco, the six phases of security is a framework for continuous assessment, action, and improvement. Following this process of preparedness and improvement is critical to deploying any security solution, including a Clean Pipes solution.

• Maintain a separate and secure management network for the Clean Pipes solution. The communication between detection and mitigation components is most critical during an attack so a separate, isolated management network is required to maintain communication during adverse conditions.

• Out-of-band management to the devices in the Clean Pipes solution must be in place to

ensure that network operators have full access to the systems at all time and can perform necessary maintenance without significant outages.

• Build sufficient mitigation capacity. Minimum capacity requirements are typically equal

to the available bandwidth at the network border. Typically providers target enough

Page 27: Clean Pipes Whitepaper 1

 

 

capacity to protect critical infrastructure and match that capacity to ensure that peering capacity can be cleaned. This ensures that attacks that overwhelm the capacity of a given resource can still be effectively mitigated.

• Ensure comprehensive path detection for the network resources protected. All critical

paths of traffic to or from critical network resources should have a level of detection that can measure traffic threshold violations.

• Minimize traffic backhaul and maintain distribution of the attack traffic. Traffic backhaul

and aggregation of the attack traffic may tax the capability of parts of the network. Managing attack traffic in a distributed manner limits the possibility that it will overwhelm network resources.

• Manage attacks as near the source of the attack as possible. Tracing traffic back to its

source and limiting the network’s exposure to it limits collateral damage and impact of the event on the network.

• Build redundancy into the mitigation solution so that the network will remain protected

even if one or more mitigation devices become unreachable.

• Drop known threat traffic as quickly as possible. Bringing operational and historical knowledge of attack vectors and their sources can cut the amount of traffic that must be analyzed when protecting a network service such as DNS, VoIP, or web services. Running a coarse-grained filter before a more surgical filter can limit the amount of advanced scrubbing capacity required to mitigate attacks.

• Automate response and traffic redirection whenever possible. Automation increases the response time available for network events and attacks. Where possible, coarse filters, traffic diversion, and traffic scrubbing should be automated to ensure rapid response to threats.

• Maintain reports and data from the alert through the mitigation for analysis. Reporting

and comprehensive history of mitigation actions provide learning opportunities for future events.

Page 28: Clean Pipes Whitepaper 1

 

 

Conclusion Clean Pipes solutions have been rolled out throughout the world as an answer to established and evolving DDoS threats. These solutions have been a result of the close relationship between Arbor Networks and Cisco to develop as well as leverage each company’s technology to bring about a better combined solution. Clean Pipes 2.0 now focuses technology from each of the providing partners into their core competency and furthers the development of these technologies by focused expertise of each vendor. The partnership between Cisco and Arbor Networks continues to provide best-of-breed solutions to customer problems and evolve existing solutions to the next phase of capabilities.  

Page 29: Clean Pipes Whitepaper 1

 

 

 

Appendixes  

Appendix A: Peakflow SP System wide Enforced and Guideline Limits  The following table includes the enforced limits for Peakflow SP System:  

Type Limit

CP appliances 35 Important: You must assign a PI appliance as the leader if you have five or more CP, TMS, or BI appliances in your deployment. Up to 30 CP appliances are supported in a single deployment.

FS appliances 150 (maximum of 5 per CP appliance)

TMS appliances 50 (maximum of 5 per CP appliance) Note: Up to 3 stacked TMS 2700 appliances count as 1 appliance toward this limit.

PI appliances 10

Third-party firewalls 10 (Cisco Guard)

Monitored routers 2,250 (maximum of 5 per CP appliance or 15 per FS appliance)

Monitored interfaces 100,000

Managed objects 10,000 Note:  If you have more than 1,000 managed objects, then you must add Business Intelligence (BI) appliances for additional managed object storage.

Mitigation templates 1,000

Fingerprints (2.0) 100

Applications 500 Note: These are also bound by the managed object limit.

Page 30: Clean Pipes Whitepaper 1

 

 

The following table includes the guideline limits for Peakflow SP System:  

Type Limit

Concurrent logins per PI appliance on a 10 PI load

125 (requires multiple PI appliances)

Concurrent logins to the leader appliance

10

Configured users 500

SOAP queries per minute, per appliance

200

Simultaneously active DoS alerts 1,000

BGP traps 100

Active fingerprints 20

Managed objects with filters 20

Reports 500 completed or up to 20 GB of disk space

Report templates 500

Active mitigation actions 1,000 Note: This includes blackhole off-ramps, TMS mitigations, Cisco Guard off-ramps, and Flowspec mitigations.

CIDR group prefixes 50,000 Note: This limit applies to CIDR entries across all CIDR groups, including duplicates.

Unique CIDR blocks across all CIDR groups for all managed objects

2,000

Number of prefixes per CIDR group entry

100 Example: A CIDR group with three prefixes (10.0.5.0/24, 10.0.6.0/24, 10.0.8.0/24) can have more prefixes but cannot exceed 100 total.

Page 31: Clean Pipes Whitepaper 1

 

 

Auto-configuration rules 5,000

Archived alerts 100,000

Alert deletions per day 2,000

Multisite members 15

Services 50

CIDRs defined per service 100

 

Page 32: Clean Pipes Whitepaper 1

 

 

 

Appendix B: Peakflow SP PI 5500 Appliance Enforced and Guideline Limits 

The following table includes the enforced and guideline limits for the PI 5500 appliance:

Type Limit

Enforced • Non-leader PI supports all mitigation configuration except third-party • Leader PI supports all mitigation configuration, including third-party

Guidelines • Supports software upgrades from CP 5000 but may require new hardware

• 200 automated SOAP queries per minute • 250 managed services systems (systemwide) • 50 simultaneous users per PI appliance • 120 simultaneous users (systemwide) • 100 Web 2.0 API objects

 

Page 33: Clean Pipes Whitepaper 1

 

 

 

Appendix C: Peakflow SP CP 5500‐series Appliance Enforced and Guideline Limits 

The following table includes the enforced limits for the CP 5500 series of appliances:

Type Model Limit

Data sources (routers or appliances)

CP 5500-5 5 Note: Up to 3 stacked TMS 2700 appliances count as 1 appliance toward this limit.

CP 5500-2 2 Note: Up to 3 stacked TMS 2700 appliances count as 1 appliance toward this limit.

Interfaces seen All 20,000

Interfaces monitored All 10,000

Mitigation slots

CP 5500-5 2

CP 5500-2 1

OSPF area All 1

Simultaneous DoS alerts

All 300

Page 34: Clean Pipes Whitepaper 1

 

 

The following table includes the guideline limits for the CP 5500-series appliances:

Type Limit

Input: flows per second 50,000 (supports peak rates of up to 100,000 fps) Note: Flow throughput depends on how many managed objects match as in/out per flow. The current supported flow limit assumes that no more than five managed objects match per flow.

Input: ArborFlows per second from the TMS and FS appliance

50,000 (supports peak rates of up to 50,000 fps)

Forensic flows  Up to four days Note: This is constrained by the systemwide limit. 

Steady‐state BGP routes  2 million (300,000 per every router) 

Local managed objects per CP appliance 

500 Note: This limit is constrained by the system limit. 

Ongoing DoS alerts  300 

BGP flaps per second  1,000 

  

 

 

 

Page 35: Clean Pipes Whitepaper 1

 

 

Appendix D: Peakflow SP TMS Appliance Limits 

TMS 4100 appliance limits

The following table includes the application monitoring and mitigation limits for the TMS 4100:

Type Limit

Offramping or inline traffic 10 Gbps

Offramping or inline traffic 8 Mpps

Ongoing mitigations per appliance 50

Interfaces (physical, logics, sub-interface / VLAN) 1,000

TMS 4200 appliance limits

The following table includes the application monitoring and mitigation limits for the TMS 4200:

Type Limit

Offramping or inline traffic 20 Gbps

Offramping or inline traffic 16 Mpps

Ongoing mitigations per appliance 50

Interfaces (physical, logics, sub-interface / VLAN) 1,000

Page 36: Clean Pipes Whitepaper 1

 

 

TMS 4300 appliance limits

The following table includes the application monitoring and mitigation limits for the TMS 4300:

Type Limit

Offramping or inline traffic 30 Gbps

Offramping or inline traffic 24 Mpps

Ongoing mitigations per appliance 50

Interfaces (physical, logics, sub-interface / VLAN) 1,000

TMS 4400 appliance limits

The following table includes the application monitoring and mitigation limits for the TMS 4400:

Type Limit

Offramping or inline traffic 40 Gbps

Offramping or inline traffic 32 Mpps

Ongoing mitigations per appliance 50

Interfaces (physical, logics, sub-interface / VLAN) 1,000

Page 37: Clean Pipes Whitepaper 1

 

 

TMS 3100 and 3110 appliance limits

The following table includes the application monitoring and mitigation limits for the TMS 3100 and 3110 appliances:

Type Limit

Offramping or inline traffic 10 Gbps

Offramping or inline traffic 8 Mpps

Ongoing mitigations per appliance 50

Interfaces (physical, logics, sub-interface / VLAN) 1,000

TMS 3050 appliance limits

The following table includes the application monitoring and mitigation limits for the TMS 3050 appliance:

Type Limit

Offramping or inline traffic 5 Gbps

Offramping or inline traffic 3.5 Mpps

Ongoing mitigations per appliance 50

Interfaces (physical, logics, sub-interface / VLAN) 1,000

Page 38: Clean Pipes Whitepaper 1

 

 

Appendix E: Six‐Phase Approach to Infrastructure Security 

Cisco and Arbor advocate a six-phase framework for deploying security systems. The six phases are:

• Preparation  • Detection  • Classification  • Traceback  • Reaction • Post‐mortem  

 

Although the six-phase approach was designed primarily to counter DDoS attacks, this framework provides a good overall approach to securing service provider environments.

Preparation Phase 

Preparation is probably the most important of the six phases. This phase includes setting up both technical and nontechnical processes, tools, and organizational structure that constitute the security system. The tasks in the preparation phase include:

• Select, develop, install, and test the security tools and techniques you will use.  • Define and agree upon security policy and incident response procedures.  • Set  up  communications  channels with  service  provider  peers  and  customers,  and 

establish equipment vendor incident response teams.  

Identification Phase  

In the identification phase, you detect unusual activity or behavior and activate appropriate measures after an alert is raised. You can use many tools and data sources to identify these issues, including NetFlow information, SNMP information about the CPU, and interface

Page 39: Clean Pipes Whitepaper 1

 

 

utilization data. A customer's report that service is unavailable is often an early indicator of an attack.

Classification Phase  

After an attack has been detected, you'll need to collect comprehensive information about it, including the spoofed or non-spoofed source addresses, destination IP addresses, packet sizes, and Layer 4 information, such as protocol and port numbers. NetFlow telemetry can provide information about the attack based on the data elements tracked in each individual flow.

Traceback Phase  

Assuming you have identified the attack vector in the preceding phase, you now need to identify the ingress points in order to mitigate the attack efficiently. The traceback phase entails tracing the attack flows from the attacked sections of the network toward the network edges. You can take a hop-by-hop approach tracking the sources upstream from the victim toward network edges, or you can directly jump on the network ingress points to check them for the presence of attack flows. You can track flows through the network in various ways: through ACLs (with or without the log-input clause), by deploying NetFlow, or by using backscatter mechanisms.

Reaction Phase  

In this phase, you mitigate the attacking flows using the various mechanisms you identified as appropriate for your network during the preparation phase. These tools and techniques can include ACLs, remotely triggered source-based and destination-based blackholing, rate limiting, or traffic scrubbing. For a service provider, it is important that the techniques can be deployed quickly and on a large number of ingress points. A primary concern is deploying techniques that have minimal or no negative impact on benign traffic flows and on the performance of the network.

Post­Mortem Phase  

The post-mortem phase is critical. This is where you review the whole attack-handling process, analyze the experience, and look for ways to improve either organizational or technical aspects of the response. By incorporating post-mortem conclusions into a new preparation phase, you can begin to close what is often referred to as the security wheel to help ensure that the security of the service provider network remains at a high level. The security wheel illustrates that security is a cycle in which security measures are tested and improved and policies are updated so that they reflect changing security needs and drive security enhancement. Because Internet attacks are not a temporary phenomenon and they will only become more sophisticated, it is important to continually review and refine attack-handling tools and procedures.