Defending the IDC: Applied Security Solutions - die.netmirror.die.net/misc/defending-ddos/idcappliedpub.pdf · Defending the IDC: Applied Security Solutions ... !A data center is

Defending the IDC: Applied Security Solutions

Roland Dobbins <[email protected]> Solutions Architect +66-83-266-6344 BKK mobile +65-8396-3230 SIN mobile Arbor Public

Page 2 - Arbor Public

What Is an Internet Data Center (IDC)?

Source - www.wikipedia.org

An Internet Data Center (IDC) meets the above definition, with the additional characteristic of connectivity to the general Internet.

A data center is a facility used to house computer systems and associated components, such as telecommunications and storage systems. It generally includes redundant or backup power supplies, redundant data communications connections, environmental controls (air conditioning, fire suppression, etc.), and special security devices.


Why Do SPs Invest in an IDC?

  Locate critical public-facing business IT functions in the Internet Data Center (IDC) to assure business continuity!– Web presence & branding"– Marketing"– Online sales"– E-commerce/supply chain"– Internet email"– Customer support"

  Major revenue opportunity for managed services!!


Trends in the IDC Market

  IDC operators face the same ‘race to the bottom’ we see in transit SPs – commoditization of their core offerings.

  IDC operators are focusing (USA & EMEA leading the way, APAC seeing movement) on maximizing revenue from each floor-tile, from each RU of rack space, rather than just continuing to build out facilities.

  Energy costs, both for powering equipment and for cooling it, account for more than 50% of the opex of most IDCs, eclipsing payroll as the #1 generator of opex.

  Increasingly, IDC operators are beginning to look for architectural frameworks which allow them to roll out new services without doing one-offs every time.

  Virtualization of computing/storage/networking/services, coupled with automation of same, is an unstoppable force.


Types of IDC Operators

  Facilities owners – these operators build IDCs, then lease them to various IDC SPs who offer different types of services.

  Some facilities owners offer services (hosting, co-location, etc.) directly to end-customers.

  Increasingly, facilities owners are concentrating on ways to boot revenues – offering new services with minimal capex/opex per customer is a primary focus.

  There’s a further tier of sub-leasers, who buy up space/capacity and then lease it to operators who provide services to end-customers.

  Many of these facilities owners will provide the infrastructure for others to become ‘virtual IDC’ operators.

  They can then offer additional services such as IP transit, security services, et. al. to these virtual operators.


Hosting & Co-Location – Going Virtual

  Shared hosting is a baseline IDC service – everyone offers it, less & less money to be made, the ‘hockey-stick’ of upward growth has leveled off.

  Co-location is still growing in APAC, but at a much-reduced rate, as everyone figures out how to reduce/eliminate excessive capex & opex.

  ‘Virtual Private Servers’ (VPS) are the dominant trend – also called ‘virtual co-location’. IDC operators can pack hundreds or even thousands of customers into the same floor-tile/RU/KW required for a single physical co-located customer.

  Much for convenient for the end-customer – no hardware to buy/depreciate, don’t have to manage hardware outages, click to dynamically expand or reduce capacity as needed.

  This trend will eventually kill shared hosting and drastically reduce physical co-location.


ASPs, CDNs, Gaming Operators

  Application Service Providers (ASPs) are major customers for IDCs; co-location has been their preferred model, though this is moving to VPS and to cloud.

  Content Distribution Networks continue to be heavy physical co-location customers, due to specialized hardware/operational requirements.

  Gaming operators are also heavy physical co-location customers due to their special requirements, though this is beginning to move to the cloud with emerging PaaS offerings.

  Very few ASPs, CDNs, and/or gaming operators build and operate their own physical facilities. They tend to lease or sub-lease, they run with minimal operational personnel, and often out-task system/network operations to facilities owners or other third parties.


Cloud Computing – A Definition

A general-purpose, multi-tenant information processing model in which modular, pooled, and dynamically scalable computing, storage, networking, application, and service resources are represented to users and developers as a unitary system, logically abstracted from the underlying physical infrastructure and featuring a common set of APIs, client access mechanisms, and administrative/management functions.


DDoS Attacks – The #1 Security Threat to Cloud Computing!

•  Discussions of cloud security priorities tend to focus on confidentiality of data, privacy, separation of application logic in a multi-tenanted cloud infrastructure.

•  The cloud security elephant no one wants to discuss is DDoS - why?

•  DDoS is the #1 security threat to the cloud model – DDoS shuts the cloud down. No cloud availability = no revenue for cloud providers!

•  Most security researchers tend to ignore DDoS – why?

•  Cloud providers don’t use resiliency against DDoS as a selling point for their services – why?

•  The reality is that we’ve all been dependent upon ‘the cloud’ for years – search engines, Web mail, IM, social networking, weblogs, etc. – and it’s disruptive for ordinary users when these services are unavailable. What will it be like when even more core information infrastructure is dependent upon continuous Internet-wide availability? Netbooks, mobile apps/data?


The Importance of Security within the IDC

Total Visibility in all aspects

of the IDC.

Complete Control over all traffic in

the IDC.

Building a Secure Infrastructure for the Online Economy

Maximize Availability of all revenue-generating IDC services.


DDoS – the #1 Security Threat to the IDC!

  IDC operators make money when their clients are up and available and accessible – when availability suffers, SLAs take a hit, customers will go elsewhere.

  Many IDC operators have little to no understanding of security, don’t have robust architectures, have no opsec plan, no engagement with the operational community.

  Even with shared hosting and physical co-location, the IDC itself is by definition multi-tenant, so collateral damage is common.

  Almost all DDoS attacks are aimed directly or indirectly at taking down servers . . . which reside within IDCs.

  ~50% of all criminal DDoS attacks we see are criminals DDoSing one another; they’ll DDoS botnet C&C servers to hijack the botnets of rivals. Botnet C&C servers are, by and large, compromised legitimate servers located within IDCs.


The Secure IDC as a Competitive Advantage

  Security is a must for cloud migration to take place – it’s the #1 concern enterprise BDMs/TDMs have about cloud-based services.

  Laws/regulations increasingly require more stringent security capabilities (MAS Clean Pipes requirement is groundbreaking, will be replicated worldwide, over time).

  Complete network visibility enables total network control – most IDC operators have limited/no visibility into their networks, and thus don’t have positive control over their networks!

  Leading with security is a way to differentiate from the competition and to catch the attention of non-technical BDMs.


IDC End-Customers Require a Strong, Resilient IDC Infrastructure From their IDC SP


Without a DDoS Mitigation Solution, This is What Happens to End-Customers


And the End-Customers Will Go Elsewhere!


IDC SPs Must Have The Right Tools for the Job!


Speaking the Language of IDC Operators

  Peakflow SP is a scalable, multi-tenant detection/classification/traceback network visibility solution, which also provides valuable traffic engineering, peering/transit analysis, and troubleshooting capabilities.

  The TMS is a scalable, multi-tenant, virtualized Intelligent DDoS Mitigation System (IDMS).

  Arbor-based Clean Pipes is a ‘sticky’ service which encourages end-customer loyalty.

  Arbor’s solution protects tenants/customers from outside attack.

  Arbor’s solution protects tenants/customers from intra-IDC attacks.

  Arbor’s solution protects the IDC from tenants/customers.   Arbor’s solution alerts tenants/customers to compromised

servers.


Arbor’s Solutions Offer True OPSEC Value

  Arbor provides real-time network visibility and control capabilities based upon our flow analysis, S/RTBH, IDMS, mDLP, & quarantine capabilities.

  Peakflow SP provides IDC operators with all the same benefits as transit operators – traffic engineering, capacity planning, statistical anomaly-detection for inbound/outbound DDoS.

  TMS provides IDC operators with DDoS defense against inbound attacks – protect SP’s own infrastructure, offer as a managed service for customers.

  Peakflow/X provides mDLP & quarantine for SP’s own servers, hosted/co-located customers.


Firewalls and IDS/’IPS’ don’t help!

  It’s time to put the firewall and IDS/’IPS’ myth to rest!

Firewalls are policy-enforcement devices – they can’t help with DDoS, and in most cases, the policies applied to the firewalls have been devised with no visibility into network traffic, so the firewall rules bear little relation to what should actually be permitted and denied.

IDS/’IPS’ are by definition always behind the attackers – in order to have a signature for something, you must have seen it before.

IDS/’IPS’ have proven to be totally ineffective at dealing with application-layer compromises, which is how most hosts are botted and used for DDoS, spam, corporate espionage, identity theft, theft of intellectual property, etc.

Firewalls & IDS/’IPS’ output reams of syslog which lacks context, and which nobody analyzes. It is almost impossible to relate this syslog output to network behaviors.

End-customers subscribe to traditional managed security services based on firewalls and IDS/’IPS’, and still get compromised!

Firewall & IDS/’IPS’ deployments cause performance & usability problems, and don’t scale, shouldn’t be deployed in front of servers!


Arbor DDoS Solution: Diversion/Offramping

NetFlow to Arbor Peakflow SP

Managed Object 1: Web Servers

Managed Object 2: Name Servers Managed Object 3:

E-Commerce Application

Arbor TMS






1. Detect

Target


Arbor TMS





1. Detect

2. Activate: Auto/Manual

Target


Arbor TMS







1. Detect


3. Divert Only Target’s Traffic

BGP Announcement

Target


Arbor TMS






1. Detect


4. Identify and Filter the Malicious

BGP Announcement

Target


Arbor TMS 3. Divert Only Target’s Traffic

Traffic Destined to the Target






1. Detect


Legitimate Traffic to

Target

BGP Announcement

5. Forward the Legitimate Target










1. Detect


Legitimate Traffic to

Target

6. Non-Targeted

Traffic Flows Freely

BGP Announcement

5. Forward the Legitimate


Target





Deployment Considerations

  What types of attacks must be detected and mitigated?

  Device/feature placement   Routing – diversion and re-injection of traffic   Performance and scalability   High availability   Management   SLAs   Is the network already stable, secure, and healthy?

Does an operational security (OPSEC) team exist and have adequate tools? These are key prerequisites!


Elements of the IDC Topology

  Peering/transit edge – connectivity to ‘the Internet’.   Distribution gateway – often called the ‘IDC core’. In many

cases, this is collapsed into the peering/transit edge.   Aggregation layer – a layer-3 hierarchical element which

separates and allows multiple layer-2 access domains to co-exist within the IDC without scaling issues.

  Services layer – a combined layer-2/-3 services ‘sidecar’ infrastructure which delivers common services such as load-balancing, caching, policy enforcement, et. al. to multiple servers/customers within the IDC. The services-layer infrastructure is typically homed into each instance of the aggregation layer.

  Access layer – the layer-2 network which provides the basic connectivity for servers within the IDC. This is where the servers live.


Clean Pipes DDoS Protection in the Core – Mitigate Before DDoS traffic enters the IDC!

Network DDoS Protection

IDC DDoS Protection

Peering Point DDoS Protection

Infrastructure DDoS Protection

NetFlow

NetFlow NetFlow

NetFlow


IDC Core/Dist.

Blades Rack End-of-Row

Gigabit Ethernet

10 Gigabit Ethernet

4Gb/8Gb Fibre Channel

Rack

IDC Access

10GbE Agg., IDC Services

Storage End-of-Row

FC Access

IP+BGP+MPLS Distribution Edge

Internet

10GbE and 4Gb FC Server Access

Blades

10GbE Unified Fabric Access

10 Gigabit FCoE/DCE

SAN A/B Storage Core

Gigabit Ethernet Access

10GbE Core/ Distribution

Aggregation, IDC Services

IDC Aggregation

Example - Modern IDC Infrastructure


IDC Flow Telemetry Considerations – Peakflow SP

  Flow telemetry is generally an edge technology – this holds true in the IDC, as well.

  The peering/transit edge, distribution gateway, and, in some applications, the access edge are all potential sources of flow telemetry.

  Cisco 6500, 7600, 4500, 3700, 2900, Nexus 5000, & various blade switches do not provide operationally useful flow telemetry.

  VMWare softswitch provides operationally useful flow telemetry.   Cisco CRS-1, 12000 (w/E3/E5 LCs), ASR 1000, 7200, ISRs (1x00, 2x00,

3x00 routers), Nexus 7000, Nexus 1000V (and soon the ASR 9000) provide good flow telemetry.

  HP ProCurve switches support S/Flow, w/some caveats.   All flow telemetry should be exported across the DCN, which is where

CPs/FSes/BIs and/or Peakflow/X controllers/collectors should be sited.


IDC Prerequisites for Defending Against DDoS

  Anti-spoofing via uRPF, DHCP Snooping/IP Source Guard   iACLs   CoPP   Loop guard/root guard/BPDU guard/spantree portfast.   pVLANs/VACLs   PACLs   GTSM   MD5 keying for IGP, iBGP, eBGP   DCN/OOB management network   Operationally useful flow telemetry from the infrastructure!


Where to Mitigate DDoS Attacks in the IDC?

  If the IDC is part of a transit SP network, mitigation in the core, northbound of the IDC, is preferred. In other words, outside the IDC.

  For pure IDC operators, the peering/transit edge, the distribution gateway, the services layer, or the aggregation layer are preferable.

  Building a separate distribution environment for the cleaning center is preferable to hanging TMSes from the peering/transit edge or distribution gateway, more control and less complexity.

  The services layer is a natural topological location for mitigation, assuming it provides layer-3 connectivity & onramping can be accomplished in the context of the routing topology. Utilize existing services infrastructure, if possible.

  ‘Flat’ IDCs with only a distribution gateway and directly-homed layer-2 access networks are problematic – we generally require at least one more southbound hop for onramping. It may be technically possible to home directly into the access layer via layer-2, but it is very complex, very situationally-specific, very risky, and doesn’t allow much in the way of oversubscription. Should generally be avoided!

  In-line doesn’t scale, introduces operational complexity, drastically reduces oversubscription capability, doesn’t scale – avoid it!

  S/RTBH & flow-spec can be used at all edges for inbound and outbound DDoS – best tool to stop compromised customer hosts from attacking one another, which is a growing problem!


Peers/Transit

Peering/Transit Cleaning Center


Peers/Transit

Core/Distribution Cleaning Center


Peers/Transit

Collapsed Distribution/Peering/Transit Cleaning Center


Peers/Transit

Collapsed Distribution/Peering/Transit Cleaning Center – Direct Attachment


Peers/Transit

Services Layer Cleaning Center – If Layer-2/-3 Logical Topology Permits


‘Flat’ Layer-2 – Problematic!

Peers/Transit


Scaling a Cleaning Center: Clustering Topology with ECMP/CEF Load-leveling

Load-Leveling Router – up to

16 TMSes, 640gb/sec w/N7K

Arbor TMS IDMSes

TMS Mitigation

Cluster

Target host

Attack

TMS

TMS

TMS

TMS

TMS

TMS

Multiple Cleaning Centers via Anycast


Protecting the IDC DNS Infrastructure

  Like other SPs, IDC operators should have a well-designed, functionally-separated, ‘bulkheaded’ DNS infrastructure.

  Public-facing authoritative DNS servers (‘external resolvers’) must be protected from externally-sourced DDoS.

  Public-facing internal resolvers servicing recursive lookups from within the IDC must be protected from externally-sourced DDoS, and, by extension, indirect internally-sourced DDoS.

  Internal-facing caching-only resolvers must be protected from deliberate or inadvertent DDoS originating from within the IDC.


Protecting the IDC DNS Infrastructure


IDC ‘Clean Pipes’ Outsourcing Considerations

  Multi-homing presents an intractable problem for IDCs seeking to outsource Clean Pipes to their upstreams/peers – and reducing capacity/links when under DDoS is counterproductive.

  Coordinating with one or more external entities for timely mitigation can be extremely problematic. If multiple providers are involved, they may have radically different capabilities, SLAs, etc.

  Outsourcing can potentially work if an IDC is single-homed, but the above caveats still apply.

  The best way for IDC operators to outsource Clean Pipes is if they’re leasing or sub-leasing from an IDC operator who provides the service as an overlay to transit services to the IDC tenant operators.

  For most IDC operators, an organic Clean Pipes capability is by far the better choice – control, responsiveness, revenue.


The ATLAS Global Sensor Network

1.  ATLAS sensors are deployed in global Internet darknet space to discover and

classify attack activity.

2.  This information is sent to an ATLAS central repository where it is combined

with Arbor Peakflow, third-party, and vulnerability data.

3.  ASERT analyzes combined data and converts into actionable intelligence which is posted on the ATLAS public

portal.


Arbor’s Active Threat Feed (ATF)


DLP/mDLP in the IDC with Peakflow/X

  SP and customer hosted/co-located servers are compromised more often than operators would like to admit.

  Due to the critical functionality of these servers, coupled with their typically high-bandwidth connectivity, this presents multiple dangers to the server owners and to the Internet at large.

  Just as Peakflow/X can be used for DLP/mDLP on the enterprise LAN, it can be used for the same purpose in the SP IDC.

  Peakflow/X can optionally quarantine compromised hosts via switchport shutdown or move to quarantine VLAN.

  Communications flows within the IDC (should be) much less chaotic than within the enterprise, surprisingly amenable to relational A/D.

  Even ‘broken’ 6500/7600 NetFlow can work for Peakflow/X, provided it’s selectively enabled on relevant interfaces and doesn’t overflow the mls table. Can use our probes via taps, too.

  Bring ATLAS to bear within the IDC!


PeakflowX – A Real-Time Anti-Botnet Solution for the IDC Arbor Peakflow X detects and can quarantine botted/compromised physical & virtual servers within the IDC

Key Features: – Behavioral Anomaly Detection – Active Threat Level Analysis System (ATLAS) – Active Threat Feed (ATF)

Benefits:

–  Identify compromised/botted servers within the IDC – stop outbound DDoS, spam, serving malware for phishing campaigns!

–  Identify compromised servers within the IDC used as botnet C&C servers – DDoS targets!

– Protect customer/user credit card info, identity information, etc. – avoid civil & criminal liability!

– Quarantine compromised servers quickly – prevent further server compromises!


Peakflow/X in the IDC

 Detect/quarantine compromised/botted customer physical and/or virtual hosts.

 Monitor the DCN/OOB management network.

 Monitor the internal IT network.  Make use of flow telemetry from Nexus

1000V and/or VMWare softswitch to monitor inter-VM communications


IDC Flow Telemetry Considerations – Peakflow/X

  Most layer-2 switches today don’t provide flow telemetry for inter-switchport communications – in most cases, a layer-3 boundary must be crossed for a flow to be generated.

  This is starting to change – Cisco Nexus 7000, Cisco Nexus 1000V, & VMWare softswitch produce true Layer-2 flow telemetry

  Other vendors may, as well – HP ProCurve line supports S/Flow.   Layer-2 flow telemetry can be generated via copy/capture VACLs,

SPAN/RPSAN (be careful not to exceed fps budgets!) and/or taps.

  Most blade switches today don’t support flow telemetry.   Selectively enable flow telemetry within the IDC mesh only where

needed, reduce flow budget for Peakflow/X collectors.   Over time, more devices will support flow telemetry, including

layer-2.   Over time, FCoE and other specialized transports will receive flow

telemetry support.


Peakflow/X in the IDC – Telemetry Sources

Server Farms Access Aggregation IDC Core Edge

Peer/Upstream A

NetFlow

Peakflow/X Telemetry Sources

Arbor Peakflow/X

IDC Edge

IDC Edge Peer/Upstream B


Peakflow/X & VMWare in the IDC

Inter-VM Traffic

10 GbE

10 GbE DCE

Fibre Channel

SAN A

10 GbE FCoE/DCE

Agg. SAN B Services

Access Access

Nexus 1000v or VMWare Softswitch

Peakflow/X

NetFlow


Q&A

Thank You!

Roland Dobbins <[email protected]> Solutions Architect +66-83-266-6344 BKK mobile +65-8396-3230 SIN mobile


Defending the IDC: Applied Security Solutions - die.netmirror.die.net/misc/defending-ddos/idcappliedpub.pdf · Defending the IDC: Applied Security Solutions ... !A data center is

Documents