This webinar describes the motivation for using IP/MPLS technologies as a Data Center inter-connect technology. Data Center Interconnect is a service that Service Providers (SP) can offer to interconnect Enterprise Data Centers, interconnect SP Data Centers, or allow Enterprise to connect to SP-owned compute and storage resources.
Watch the replay at: https://cisco.webex.com/ciscosales/lsr.php?AT=pb&SP=EC&rID=67897237&rKey=f7260083ab1d593f
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Enhancing IP/MPLS based Carrier Services to address the Data Center Interconnect Market Patrick Warichet: Technical Marketing Engineer NOSTG Vipul Shah: Engineering Product Manager NOSTG
We are offering various Ethernet Services and Ethernet OAM learning opportunities at Cisco Live Orlando. Learn more and register at: http://www.ciscolive.com/us/attendees/education/
Technical Session Session Number Building Carrier Ethernet Services Using Cisco Ethernet Virtual Circuit Framework
BRKSPG-2204
Ethernet OAM: Technical Overview and Deployment Scenarios BRKNMS-2202
Redundancy Mechanisms for Carrier Ethernet and Layer 2 VPN Services
BRKSPG-2207
Towards Massively Scalable Ethernet: Technologies and Standards
• Data Center Interconnect Requirements • Data Center Interconnect Using Virtual Private LAN Service • Data Center Interconnect Using Ethernet -VPN • Summary
interconnection between main SP DCs ! Owned by SP DC team
! Addresses SP2SP only
! Very high bandwidth – packet / optical solution likely the most cost effective
• DCI Requires Technology Evolution in Data Center and SP NGN for: ! Multihoming
! Scale (MAC-addresses, Number of Service Instances
! Loadbalancing
! Optimal Forwarding
! Multicast optimization
! Multitenancy
• NGN Based DCI Interconnection models: ! Enterprise to Enterprise (E2E) ! Enterprise to Service Provider (E2SP) ! Service Provider to Service Provider
• Data Centre Interconnection (DCI) refers to interconnection of data centres primarily in support of two use cases:
• Workload mobility ! Moving workloads across Enterprise DCs ! Moving workloads from Ent to SP – “cloud bursting” ! Moving workloads from SP to SP – resilience and maintenance ! Needs mobility of the IP address – can be achieved at L2 and L3
• Business Continuance - Server clustering and related transaction and storage replication ! Often requires L2 extension – but increasingly support for L3
• Global server load balancing (GSLB) distributes client traffic to servers across remote data centers ! e.g. using DNS, L3, route health injection, HTTP
redirection
• GSLB commonly deployed with local Server Load Balancing (SLB) to Server Clusters
• Server clustering technologies and storage sync / replication often require L2 extension – but increasingly support L3 also ! Active/Standby ; Heartbeat based ! Potentially out of span of control SP
Examples Microsoft MSCS*, Veritas Cluster Server*, Solaris Sun Cluster Enterprise, VMware Cluster, Oracle RAC (Real Appl.Cluster), HP MC/ServiceGuard, HP NonStop, HP Open VMS/TruCluster, IBM HACMP, EMS/Legato Automated Availability Mgr
Data Center Interconnect: Layer 2 Extension Technology
• Virtual Private LAN Service (VPLS) ! The best available option in shipping code ! Does not meet some of the data center interconnect requirements for large SP
Multitenant Deployment options
• Ethernet-VPN (E-VPN) / Provider Backbone Bridging Ethernet VPN (PBB-EVPN) ! New technologies to meet all of the large SP multitenant data center interconnect
• Not optimal with multicast ! Enhancements are maturing (using Label Switched Multicast with VPLS instead of
ingress resplication)
• No active/active dual-homing per flow ! Per VLAN is possible
• Does not hide customer mac-addresses • PW scaling • Handoff scaling and Service Instance Scaling ! 4k services per physical interface ! 000’s of VSI’s (hardware limitations)
• MPLS Edge Switch (MES) is an E-VPN PE • ESI Used when CE is multi-homed to two MES • Allows remote MESes to load-balance to a multihomed CE • 10 octet value created by: ! Configuration ! Snooping of LACP (Multi-Homed Device) ! Snooping of STP BPDUs (Multi-Homed Network) ! (LLDP , although currently out of scope of the draft)
• ESI has a new associated MPLS Label extended BGP Community (used for split-horizon procedures in multi-homing)
Multi-purpose route for • Auto-discovery of sites, vlans, VPNs, and site/vlan • Load Balancing • Split Horizon filtering on an Ethernet segment
MAC Route • Advertisement of Client/Provider MACs
Inclusive Multicast Route • Auto-discovery of MESes belonging to a VPN for setting up multicast tree
Selective Multicast Route • Auto-discovery of MESes belonging to a given c-(S,G) of a VPN for setting up a multicast tree
Ethernet Segment Route • DF election on an Ethernet segment • MC-LAG state synch on an Ethernet segment • Loop prevention on an ES during transient state
Leaf Auto-Discovery (AD) Route • Multicast Tree Auto-discovery
It looks easy but not so fast ! • In the shown example, how do we
ensure that ! ARP broadcast packet doesn’t get loopback to the
originating Agg device (Agg-1) : Split Horizon for ESI
! Either PE3 or PE4 forward the broadcast frame to the far-end dual-homed device (Agg-2) Designated Forwarder Selection
! When PE3 wants to forward a packet with destination address MAC1, it needs to send it to both PE1 and PE2 even though it only learned MAC1 from PE1 Aliasing
• In order for PE3 to be able to perform load balancing among the flows destined to MAC1 on ESI-1, it needs to know that: a) ESI-1 sits behind both PE1 and PE2 b) MAC-1 is associated with ESI-1
• PE1 and PE2 use Ethernet AD route to advertise ESI-1
• PE1 uses MAC route to advertise MAC1 • All the remote PEs (PE3 and PE4) use these
two routes in combination to associate a) MAC1 to ESI-1 b) subsequently MAC-1 to [PE1 and PE2]
• PE/MESes perform Proxy ARP wthe destination is in the same subnet as the sender of the ARP request
• The ARP request is not forwarded to other PE/MESes • How does the PE/MES learn the IP address bound to the MAC address
when the MAC address is remote? ! A PE/MES advertises the local IP to MAC bindings in the BGP MAC routes
• How does a PE/MES learn the IP to MAC binding when the MAC address is local? ! Control or management plane between MES and CEs or data plane snooping
• Advertise local B-MAC addresses in BGP to all other PEs that have at least one VPN in common just like E-VPN
• Build a forwarding table from remote BGP advertisements just like E-VPN (e.g., association of B-MAC to MPLS labels)
• PEs perform PBB functionality just like PBB-VPLS C-MAC learning for traffic received from ACs and C-MAC/B-MAC association for traffic received from core
• DF Election with VLAN Carving ! Prevent duplicate delivery of flooded frames. ! Uses BGP Ethernet Segment Route. ! Performed per Segment rather than per (VLAN,
Segment). ! Non-DF ports are blocked for flooded traffic (multicast,
broadcast, unknown unicast). • Split Horizon for Ethernet Segment
! Prevent looping of traffic originated from a multi-homed segment.
! Performed based on B-MAC source address rather than ESI MPLS Label.
• Aliasing ! PEs connected to the same multi-homed Ethernet
Segment advertise the same B-MAC address. ! Remote PEs use these MAC Route advertisements for
aliasing load-balancing traffic destined to C-MACs reachable via a given B-MAC.
• Each PE advertises a MAC route per Ethernet Segment (carries B-MAC associated with Ethernet Segment). ! Both PEs advertise the same B-MAC for the same Ethernet Segment.
• Remote PE installs both next hops into FIB for associated B-MAC.
! Hashing used to load-balance traffic among next hops.
• Each PE advertises two MAC routes per Ethernet Segment: one with high Local Pref for active VLANs, and another with low Local Pref for standby VLANs.
• Remote PE installs the route with higher local pref into FIB for associated B-MAC.
• PE1 MAC Routes: ! Route 1: RD11, B-MAC1, RT3, Local Pref = High
! Route 2: RD12, B-MAC1, RT2, Local Pref = Low
• PE2 MAC Routes: ! Route 1: RD22, B-MAC2, RT3, Local Pref = Low
1. BGP MAC Advertisement Route Scalability ! Multiple orders of magnitude difference between C-MAC & B-MAC addresses
2. C-MAC Address Confinement ! With data plane C-MAC learning, C-MACs are never in RIB and are only present in FIB for active flows ! Whereas, with control plane C-MAC learning, C-MACs are always in RIB and maybe also in FIB
• C-MAC Mobility with MAC sub-netting ! Typically C-MACs are not managed and thus can not be sub-netted ! B-MACs on the other hand are always managed and can easily be sub-netted ! Even when C-MACs are sub-netted, VM mobility contradicts the effect of sub-
netting
• Avoid transient loop for known unicast when doing egress MAC lookup ! Since B-MAC SA is always transmitted with every frame, checking of every frame
against its source MAC SA for known unicast frame is already provided by PBB
• Actual requirements from a large SP: ! n DCs = 20 ! d cmacs per DC = 500k ! 2 PEs per DC ! c customers (with bridge domain per customer) = 8k ! l linecards per PE = 8 ! e edge switches per DC on average = 8
SP DCI Case Study
46
# Model Mac scale reqs per PE PW / LSP scale reqs per PE 1a VPLS 500k*20 = 10M 8k*(2*20-1) = 312k
1b VPLS + Cluster 500k*20 = 10M 8k*(20-1) = 152k
2 EVPN 500k*20 = 10M 2*20-1 = 39
3 PBB-VPLS (PE linecard is I component) 500k*20/8 = 1.25M 2*20-1 = 39
4 PBB-EVPN (PE linecard is I component) 500k*20/8 = 1.25M 2*20-1 = 39
• Data Center Interconnect is a new Service that can be offered by SP’s by enhancing their IP/MPLS Services ! L2 Ethernet Service ! L3 Service
• VPLS with enhancement can be leveraged to offer L2 Ethernet Service • E-VPN solves some of the VPLS Scaling issues ! Active, Active ! Better Multicast support ! Pseudo Wire Scaling ! Autoconfiguration
• PBB-EVPN solves all of the Scaling issues associated with multi-tenant, high-capacity DCI ! Service Instance Scaling ! MAC Address Scaling ! Interworking with next gen IntraDC Technologies
Which platform are you considering or currently have at the DCI edge? (select multiple choices, as applicable) A. ASR 9K B. Nexus 7K C. ASR 1K D. Catalyst 6K E. Another vendor equipment
Are you considering VPLS/PBB-EVPN/EVPN for your Data Center Interconnect technology? A. I would consider deployment in the next 12 months B. I would consider deployment in the next 12-24 months C. Undecided. Need further evaluation / understanding of the technology D. No plans to deploy / Not applicable to my environment E. I am considering another DCI technology at this time
Which features are most critical for your company when considering a DCI technology? (select multiple choices, as applicable) A. Route scaling associated with Multi-tenant networks B. Seamless interworking between intra-DC and DCI protocols C. Resiliency and load balancing D. Multicast E. Other
(V)LAN (Virtual) Local Area Network AD Auto-Discovery ARP Address Resolution Protocol AS Autonomous System BGP Border Gateway Protocol B-MAC Backbone MAC BPDU Bridge Protocol Data Unit CE Customer Edge C-MAC Customer MAC CWDM Coarse Wave Division Multiplexing DC Data Center DCI Data Center Interconnect DF Designated Forwarder DHD Dual Homed Device DHN Dual Homed Network DWDM Dense Wave Division Multiplexing ECMP Equal Cost Multi Path ESI Ethernet Segment ID EVI Ethernet VPN Instance E-VPN Ethernet VPN FAT Flow Aware Transport FC Fiber Channel FIB Forwarding Information Base ICCP Inter Chassis Control Protocol IGP Interior Gateway Protocol IP Internet Protocol L2-VPN Layer 2 VPN LACP Link Aggregation Control Protocol LDP Label Distribution Protocol LLDP Link Layer Discovery Protocol
LSM Label Switched Multicast MAC Media Access MC-APS Multi-Chassis Automatic Protection Switching MC-LAG Multi-Chassis Link Aggregation MES MPLS Edge Switch MHN Multi Homed Network MP2MP Multipoint to Multipoint MP2P Multipoint to Point MPLS Multi Protocol Label Switching MST Multiple Spanning Tree Protocol NGN Next Generation Network NLRI Network Layer Reachability Information NNI Network to Network Interface nV Network Virtualization P2MP Point to Multipoint P2P Point to Point PBB-EVPN Provider backbone Bridging Ethernet VPN PE Provider Edge PW Pseudo Wire RD Route Distinguisher RIB Routing Information Base RSTP Rapid Spanning Tree Protocol RTT Round Trip Time SAN Storage Area Network TRILL Transparent INterconnection of a Lot of Links UNI User to Netwok Interface VPLS Virtual Private LAN Service VPN Virtual Private Network VSI Virtual Switch Instance VSS Virtual Switch WAN Wide Area Network