Flexible Data Centre Fabric - FabricPath/TRILL, OTV, LISP and VXLAN Ron Fuller– CCIE #5851 (R&S/Storage) Technical Marketing Engineer, Nexus 7000 [email protected]
Flexible Data Centre Fabric -
FabricPath/TRILL, OTV, LISP
and VXLAN
Ron Fuller– CCIE #5851 (R&S/Storage)
Technical Marketing Engineer, Nexus 7000
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 3
Agenda
The Evolving Data Centre Fabric
FabricPath
VXLAN
LISP
LISP Host Mobility
OTV LAN Extension
Mobility with Extended Subnets
Nexus Fabric
1K Cisco Nexus x8
6
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 4
Goals of the Fabric Addressing Concurrent Workloads, Mobility and Latency
Architecture is evolving Rapidly – in the next 24 months
L2/L3 Boundary becomes less relevant
Clos Topologies dominate new implementations
HA models shift
Server Edge becomes more intelligent
DC Fabric becomes more scalable
Port Density
Adequate Buffer Capacity
Adequate Table Sizes
Low Latency Switching
Cut-through Switching
:
:
Priority Flow Control
Early Congestion Notification
FabricPath Multiple Trees
ECMP L2 & L3
Multi-tenancy
:
:
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 5
Goals of the Fabric Addressing High Availability and Fate Sharing
L3
L2
L3/L2
East-West traffic – Fate Sharing Domain
STP is the protocol of choice
1+1 redundancy – limited forwarding paths
East-West across L3 boundaries
OSPF/EIGRP are protocols of choice
N+1 redundancy – Broad forwarding Paths
North-South traffic
OSPF/EIGRP are protocols of choice
N+1 redundancy – Broad forwarding paths
Larger POD East-West Traffic – Fate Sharing Domain
N+1 redundancy
IS-IS is the protocol of choice
Broad forwarding paths
Broader Adjacency Support
Same number of physical boxes and links
Protocol behavior is L3-like
Multi-pathing over L2 and L3
More flexible L2 adjacency, better scale capacity
Better latency consistency within POD
L2
L3/L2
Cisco Confidential © 2010 Cisco and/or its affiliates. All rights reserved. 6
The traditional L2 vs. L3 debate has been based on a number of issues
Scalability
Availability
Requirements for the scalable design moving forward is a scalable, highly available switching fabric with the advantages of both L2 and L3
L2/L3
Goals of the FabricNot a L2 vs. L3 debate
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 7
Disadvantages of Layer 2
MAC address consumption
BPDU generation is CPU intensive with increasing number of VLANs
VLAN sprawl causes flooding and broadcasts to propagate even where they are not needed
Half of the links in the topology are blocking
Misconfigurations can cause Layer 2 loops which may make switches unmanageable
MAC Table
A
MAC Table
A
MAC Table
A
MAC Table
A
MAC Table
A
MAC Table
A
Layer 2
Domain
“Plug-and-Play” and Mobility vs. Availability and Scaling
Advantages of Layer 2
Practically “plug-n-play” – No user configuration is required to build forwarding database
It makes it simple to support teaming or L2 multicast for clusters
Easy to segment traffic with VLANs
Very fast movement of end station addresses (ability to update MAC address tables after a vMotion-type event)
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 8
Availability and Scaling vs. Restricted Workload Flexibility
Layer 3 Routed Topologies alleviate the consumption of L2 tables via route summarization
Layer 3 Routed topologies provide for a degree of fault isolation and
“Routed Access” provides the logical extension of the design philosophy
“Scaling Up” of the Access Switch via such mechanism as the FEX provide a degree of workload mobility
“L2” domain extension of some form is required for most workload mobility requirements
L3
L2
Workload Domain for most Hypervisor and Clustering based solutions is restricted by the Traditional
Layer 2/3 boundary
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 9
Segment-ID: Scaling Logical Groupings of Connectivity
Web Server
App Server
Database Server
S1 S2
S4
S3
802.1Q VLAN ID
12-bits 802.1Q
VLAN ID 12-bits
802.1ad standardized
frame format
VLAN ID 12-bits
VLAN ID 12-bits
SegmentId 24-bits
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 10
Location Identity Separation
FabricPath /
TRILL
VXLAN OTV LISP
Location Switch-ID
(IS-IS)
IP address
(IP protocols)
IP address
(IP protocols)
IP address
(IP protocols)
Identity Client MAC
(Flooding)
Client MAC
(Flooding)
Client MAC
(IS-IS)
Client IP/MAC
(Mapping DB)
Multi-tenancy 24-bit Segment Identifier
• Location reachability determined by traditional routing mechanisms in the Fabric
• Identity is mapped to location addresses
• All these technologies leverage Location/Identity Mapping
L2/L3 Fabric
Identity
Location
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 11
Scale
FabricPath, LISP, VXLAN & OTV
Requirement Intra-DC Inter-DC
Layer 2 connectivity FabricPath/TRILL/VXLAN OTV/VPLS
IP Mobility LISP LISP
Secure Segmentation VXLAN / Segment-ID VPNs (LISP/MPLS)
DC-east DC-west
POD POD POD POD
IP Network
Fabric Path
(Intra-DC L2)
OTV/VPLS
(Inter-DC x-L3)
App
OS
App
OS
App
OS
App
OS
VXLAN/OTV
(Intra-DC x-L3)
LISP
IP mobility
Fabric Path
(Intra-DC L2)
App
OS
App
OS
VXLAN/OTV
(Intra-DC x-L3)
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 12
Agenda
The Evolving Data Centre Fabric
FabricPath
VXLAN
LISP
LISP Host Mobility
OTV LAN Extension
Mobility with Extended Subnets
Nexus Fabric
1K Cisco Nexus x8
6
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 13
“FabricPath brings Layer 3 routing benefits to
flexible Layer 2 bridged Ethernet networks”
Easy Configuration
Plug & Play
Provisioning Flexibility
Multi-pathing (ECMP)
Fast Convergence
Highly Scalable
Switching Routing
FabricPath
Cisco FabricPath NX-OS Innovation Enhancing L2 with L3
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 14
• Scalable routing protocol with
proven implementation for fast
convergence upon network changes
• Link-state protocol ensures optimal
path between any 2 nodes
• Built-in authentication mechanism
enhances network security and
stability
• Inherent support for ECMP and
multi-topology maximize link
utilization
IS-IS
• Prevent potential MAC table
overflow in large scale L2 domain
• Traditional source-learning only on
Edge port for locally connected MAC
addresses
• Learning is disabled on Core port to
reduce MAC table utilization
• Non-local source-MAC only learned
if destination-MAC is already learned
as local entry
Optimal MAC Learning
• Creates hierarchical layer 2
address scheme with additional MAC
header
• Source and destination Switch_ID
written into outer MAC header at
L2MP edge
• Forwarding inside L2MP core
network is based on destination
Switch_ID
• Embedded path selector (FTAG)
provides multi-pathing for even
broadcast and multicast
• Built-in protections (TTL and
multicast RPF) minimize impact of
transient network issues
MAC-in-MAC
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 15
New Control Plane
Plug-n-Play L2 IS-IS manages forwarding topology
• IS-IS assigns addresses to all FabricPath switches automatically
• Compute shortest, pair-wise paths
• Support equal-cost paths between any FabricPath switch pairs
L1
FabricPath
Routing Table
L2 L3 L4
FabricPath
Switch IF
S10 L1
S20 L2
S30 L3
S40 L4
S200 L1, L2, L3, L4
… …
S400 L1, L2, L3, L4
S100 S200 S300 S400
S10 S20 S30 S40
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 16
Classical Ethernet (CE)
S10 S20 S30 S40
S100 S200 S300
1/1
• The association MAC address/Switch ID is maintained at the edge
• Core fabric leverages an independent routing topology from the edge • Scales MAC learning • Scales Core topology state
S300: CE MAC
Address Table MAC IF
B 1/2
… …
MAC IF
B 1/2
A S100
1/2
S300: FabricPath
Routing Table
Switch IF
… …
S100 L1, L2, L3, L4
FabricPath (FP)
Switch ID space:
Routing decisions
are made based on
the FabricPath
routing table
MAC adress space:
Switching based on
MAC address tables
S100 S300 A B
A B
New Data Plane
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 17
• Edge switches maintain both MAC address table and Switch ID table
• Ingress switch uses MAC table to determine destination Switch ID
• Egress switch uses MAC table (optionally) to determine output switchport
Local MACs point
to switchports
Remote MACs point
to Switch IDs
S10 S20 S30 S40
S100 S101 S200 FabricPath
MAC A MAC C MAC D MAC B
FabricPath
MAC Table on S100
MAC IF/SID
A e1/1
B e1/2
C S101
D S200
New Control and Data Plane
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 18
• FabricPath IS-IS manages Switch ID (routing) table
• All FabricPath-enabled switches automatically assigned Switch ID (no user configuration required)
• Algorithm computes shortest (best) paths to each Switch ID based on link metrics
• Equal-cost paths supported between FabricPath switches
S10 S20 S30 S40
S100 S101 S200
FabricPath
FabricPath
Routing Table on S100 Switch IF
S10 L1
S20 L2
S30 L3
S40 L4
S101 L1, L2, L3, L4
… …
S200 L1, L2, L3, L4
One „best‟ path
to S10 (via L1)
Four equal-cost
paths to S101
L1 L2 L4 L3
New Control and Data Plane
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 19
• Edge switch only learn the MAC of remote hosts when there are two way communications between remote hosts and local hosts
• Unknown unicast flooding alone won‟t have all switches within VLAN learn the source MAC
• Intermediate switches don‟t learn the MAC
• Hardware based MAC learning
FabricPath
A B
s3 s8
MAC IF
A s1,e1/1
… …
B e1/2
MAC IF
… …
s5
MAC IF
A e1/1
… …
B s8, e1/2
e1/1 e1/2
Scaling – Conversational Learning
Cisco Confidential © 2010 Cisco and/or its affiliates. All rights reserved. 20
CE Edge Ports
FP Core Ports
Spine Switch
Leaf Switch
Interface connected to traditional network device
Sends/receives traffic in standard 802.3 Ethernet frame format
Participates in STP domain
Forwarding based on MAC table
Classical Ethernet (CE)
S10 S20 S30 S40
S100 S200 S300
1/1 1/2
FabricPath (FP)
A B
Interface connected to another FabricPath device
Sends/receives traffic with FabricPath header
Does not run spanning tree
Does not perform MAC learning!
Exchanges topology info through L2 ISIS adjacency
Forwarding based on „Switch ID Table‟
Cisco FabricPath Terminology
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 21
Configuration Simplicity
Automatically handled by IS-IS
V10 V10 V10 V20 V20 V30 V30 V30
FabricPath
V10 V20 V30
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 22
• Multidestination traffic constrained to loop-free trees touching all FabricPath switches
• Root switch assigned for each multidestination tree in FabricPath domain
• Loop-free tree built from each Root and assigned a network-wide identifier (Ftag)
• Support for multiple multidestination trees provides multipathing for multi-destination traffic
Two trees supported in NX-OS release 5.1
S10 S20 S30 S40
S100 S101 S200 FabricPath
Root for
Tree 1
S10
S100
S101
S200
S20
S30
S40
Logical
Tree 1
Root for
Tree 2
S40
S100
S101
S200
S10
S20
S30
Logical
Tree 2
Root Root
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 23
• Extending FabricPath to the edge switches without requiring a redesign of the VLAN topology
• Each FP switch can have up to 2 Topology ID‟s defined (Topology ID‟s does not have to be unique).
• Each Topology will have 2 Multi-Destination Trees defined
FabricPath Topologies FabricPath
Topology „1‟
FabricPath Topology „0‟
Multi-Topology Support
FabricPath Topology
„2‟
VLAN 20 (DC Wide)
Common across entire
Data Center
VLAN 20 – DC Wide
VLAN 30 – POD Local (and non-unique)
VLAN 10 – POD Local (and unique)
VLAN 20 – DC Wide
VLAN 30 – POD Local (and non-unique)
VLAN 40 – POD Local (and unique)
Cisco Confidential © 2010 Cisco and/or its affiliates. All rights reserved. 24
Cisco FabricPath
Frame
Classical Ethernet Frame
• Switch ID – Unique number identifying each FabricPath switch
• Sub-Switch ID – Identifies devices/hosts connected via VPC+
• LID – Local ID, identifies the destination or source interface
• Ftag (Forwarding tag) – Unique number identifying topology and/or distribution tree
• TTL – Decremented at each switch hop to prevent frames looping infinitely
DMAC SMAC 802.1Q Etype CRC Payload
DMAC SMAC 802.1Q Etype Payload CRC
(new)
FP
Tag
(32)
Outer
SA
(48)
Outer
DA
(48)
Endnode ID
(5:0)
Endnode ID
(7:6)
U/L
I/G
RS
VD
OO
O/D
L
Etype
0x8903
6 bits 1 1 2 bits 1 1 12 bits 8 bits 16 bits 10 bits 6 bits 16 bits
Switch ID Sub
Switch ID Ftag TTL LID
Original CE Frame 16 bytes
Mac-in-Mac Header
Cisco Confidential © 2010 Cisco and/or its affiliates. All rights reserved. 25
FabricPath
MAC Table on S100
MAC IF/SID MAC IF/SID
A e1/13 (local)
e2/29
S10 S20 S30 S40
Root for
Tree 1
Root for
Tree 2
S100 S200 S300
MAC A MAC B
Multidestination
Trees on Switch 100
Tree IF
1 po10
2 po10,po20,po30,po40
Broadcast →
DMAC→FF
SMAC→A
Payload
Multidestination
Trees on Switch 10
Tree IF
1 po100,po200,po300
2 po100
po10 po20
po40
po30
Ftag →
Ftag →
DMAC→FF
SMAC→A
Payload
DA→FF Ftag→1
SA→100.0.12
DMAC→FF
SMAC→A
Payload
po100
po300
po200
e1/13
po10
po20 po30 po40
1
3
2
4
6
DMAC→FF
SMAC→A
Payload
DA→FF Ftag→1
SA→100.0.12
FabricPath
MAC Table on S200
MAC IF/SID
Multidestination
Trees on Switch 300
Tree IF
1 po10,po20,po30,po40
2 po40
5
Putting it all together – Host A to Host B (1) Broadcast ARP Request
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 26
• S100:
S100# sh mac address-table dynamic
Legend:
* - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
age - seconds since last seen,+ - primary entry using vPC Peer-Link
VLAN MAC Address Type age Secure NTFY Ports/SWID.SSID.LID
---------+-----------------+--------+---------+------+----+------------------
* 10 0000.0000.000a dynamic 0 F F Eth1/13
S100#
• S10 (and S20, S30, S40, S200, S300):
S10# sh mac address-table dynamic
Legend:
* - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
age - seconds since last seen,+ - primary entry using vPC Peer-Link
VLAN MAC Address Type age Secure NTFY Ports/SWID.SSID.LID
---------+-----------------+--------+---------+------+----+------------------
MAC A learned as
local entry on e1/13
MAC A not learned
on other switches
Putting it all together – Host A to Host B (1) Broadcast ARP Request
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 27
e2/29
S10 S20 S30 S40
Root for
Tree 1
Root for
Tree 2
S200 S300
MAC A MAC B
po10 po20
po40
po30
po100
po300
po200
e1/13
po10
po20 po30 po40
Multidestination
Trees on Switch 100
Tree IF
1 po10
2 po10,po20,po30,po40
Ftag →
Multidestination
Trees on Switch 10
Tree IF
1 po100,po200,po300
2 po100
Ftag →
11
10
DMAC→A
SMAC→B
Payload
DA→MC1 Ftag→1
SA→300.0.64
DMAC→A
SMAC→B
Payload
7 Unknown →
DMAC→A
SMAC→B
Payload
DA→MC1 Ftag→1
SA→300.0.64
FabricPath MAC Table on S300
MAC IF/SID
B e2/29 (local)
8 MISS
Multidestination
Trees on Switch 300
Tree IF
1 po10,po20,po30,po40
2 po40
9 FabricPath
MAC Table on S100
MAC IF/SID
A e1/13 (local)
MAC IF/SID
A e1/13 (local)
B 300.0.64 (remote)
12 DMAC→A
SMAC→B
Payload
(2) Broadcast ARP Reply
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 28
• S100:
S100# sh mac address-table dynamic
Legend:
* - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
age - seconds since last seen,+ - primary entry using vPC Peer-Link
VLAN MAC Address Type age Secure NTFY Ports/SWID.SSID.LID
---------+-----------------+--------+---------+------+----+------------------
* 10 0000.0000.000a dynamic 90 F F Eth1/13
10 0000.0000.000b dynamic 60 F F 300.0.64
S100#
• S300:
S300# sh mac address-table dynamic
Legend:
* - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
age - seconds since last seen,+ - primary entry using vPC Peer-Link
VLAN MAC Address Type age Secure NTFY Ports/SWID.SSID.LID
---------+-----------------+--------+---------+------+----+------------------
• 10 0000.0000.000b dynamic 0 F F Eth2/29
S100 learns MAC B as
remote entry reached
through S300
MAC B learned as
local entry on e2/29
Putting it all together – Host A to Host B MAC Address Table after the first ARP frame
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 29
e2/29
S10 S20 S30 S40
S200 S300
MAC A MAC B
po10 po20
po40
po30
e1/13
po10
po20 po30 po40
S100
DMAC→B
SMAC→A
Payload
FabricPath
MAC Table on S100
DMAC→B
SMAC→A
Payload
13 MAC IF/SID
A e1/13 (local)
B 300.0.64 (remote) B →
14
S300 →
FabricPath Routing
Table on S100
Switch IF
S10 po10
S20 po20
S30 po30
S40 po40
S200 po10, po20,
po30, po40
S300 po10, po20,
po30, po40
DMAC→B
SMAC→A
Payload
DA→300.0.64 Ftag→1
SA→100.0.12
15
S300 →
FabricPath Routing
Table on S30
Switch IF
… …
S300 po300 16
S300 →
Hash
DMAC→B
SMAC→A
Payload
DA→300.0.64 Ftag→1
SA→100.0.12
FabricPath
MAC Table on S300
MAC IF/SID
B e2/29 (local)
MAC IF/SID
A S100.0.12 (remote)
B e2/29 (local)
18
FabricPath Routing
Table on S300
Switch IF
… …
S300 Use LID (64)
17
If DMAC is known, then
learn remote MAC
po300
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 30
S100# sh mac address-table dynamic
Legend:
* - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
age - seconds since last seen,+ - primary entry using vPC Peer-Link
VLAN MAC Address Type age Secure NTFY Ports/SWID.SSID.LID
---------+-----------------+--------+---------+------+----+------------------
* 10 0000.0000.000a dynamic 90 F F Eth1/13
10 0000.0000.000b dynamic 60 F F 300.0.64
S300# sh mac address-table dynamic
Legend:
* - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
age - seconds since last seen,+ - primary entry using vPC Peer-Link
VLAN MAC Address Type age Secure NTFY Ports/SWID.SSID.LID
---------+-----------------+--------+---------+------+----+------------------
10 0000.0000.000a dynamic 30 F F 100.0.12
• 10 0000.0000.000b dynamic 90 F F Eth2/29
S100 learns MAC A as
remote entry reached
through S100
Putting it all together – Host A to Host B Unicast forwarding
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 31
FabricPath
A C B
S100 S300 S200
S10 S20 S30 S40
po10 po20
po30 po40
S100# sh fabricpath route
FabricPath Unicast Route Table
'a/b/c' denotes ftag/switch-id/subswitch-id
'[x/y]' denotes [admin distance/metric]
ftag 0 is local ftag
subswitch-id 0 is default subswitch-id
FabricPath Unicast Route Table for Topology-Default
0/100/0, number of next-hops: 0
via ---- , [60/0], 0 day/s 04:43:51, local
1/10/0, number of next-hops: 1
via Po10, [115/20], 0 day/s 02:24:02, isis_fabricpath-default
1/20/0, number of next-hops: 1
via Po20, [115/20], 0 day/s 04:43:25, isis_fabricpath-default
1/30/0, number of next-hops: 1
via Po30, [115/20], 0 day/s 04:43:25, isis_fabricpath-default
1/40/0, number of next-hops: 1
via Po40, [115/20], 0 day/s 04:43:25, isis_fabricpath-default
1/200/0, number of next-hops: 4
via Po10, [115/40], 0 day/s 02:24:02, isis_fabricpath-default
via Po20, [115/40], 0 day/s 04:43:06, isis_fabricpath-default
via Po30, [115/40], 0 day/s 04:43:06, isis_fabricpath-default
via Po40, [115/40], 0 day/s 04:43:06, isis_fabricpath-default
1/300/0, number of next-hops: 4
via Po10, [115/40], 0 day/s 02:24:02, isis_fabricpath-default
via Po20, [115/40], 0 day/s 04:43:25, isis_fabricpath-default
via Po30, [115/40], 0 day/s 04:43:25, isis_fabricpath-default
via Po40, [115/40], 0 day/s 04:43:25, isis_fabricpath-default
Topology (ftag), Switch
ID, Sub-Switch ID
Administrative distance,
routing metric
Client protocol
Next-hop interface(s)
Route age
Putting it all together – Host A to Host B Unicast Forwarding
Cisco Confidential © 2010 Cisco and/or its affiliates. All rights reserved. 32
FabricPath
Classical Ethernet (STP)
✖ STP Domain 1
STP Domain 2
CE Edge Ports
FabricPath (no STP)
BPDU BPDU
STP Domain
FabricPath Design STP Interaction
FabricPath domain appears as single Spanning-Tree bridge
All FabricPath bridges share a common (static) bridge ID
Cisco reserved MAC c84c.75fa.6000
STP BPDUs are not carried through the FabricPath network
Configure all FabricPath edge switches using “spanning-tree vlan <x> root primary” (or manually configure bridge
priority lower than any STP bridge)
Each FabricPath edge switch must be the root for all connected STP domains
Strongly recommended to use the same bridge priority on all FabricPath edge switches
Cisco Confidential © 2010 Cisco and/or its affiliates. All rights reserved. 33
L3
Layer 3 Integration at the Leaf/Edge
Provides a “cleaner” spine design
Traffic distributed equally across spines (no hot spot)
Increased number of hops to reach gateway (latency)
L3
FabricPath
FabricPath
L3
Layer 3 Boundary at the Spine
Straightforward with two spine switches
Considerations with more than two spines:
HSRP: Traffic polarized to spines on a per VLAN basis (South-North)
GLBP to distribute servers to different default gateways
Anycast FHRP future solution
FabricPathL2/L3 Boundary Location
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 34
• Simplest migration from most existing designs
• The spine is also used for routing with M1/F1 in the same VDC
• Consideration – MAC Learning and Scaling
• Compared to classic ethernet designs you gain:
Ease of configuration
MAC address table increased scalability and more efficient learning
Traffic distribution on all uplinks
Possibility to offload the spine by providing direct communication paths between the edge layer devices
[…]
edge
L3
edge/spine
s M1+F1
Conversational Learning Conversational Learning
Sw
itch
-id b
ase
d
forw
ard
ing
+ M
AC
lea
rnin
g fo
r
rou
ted
tra
ffic
M1+F1
L3 Domain
FabricPath L2/L3 Boundary Location Classic Two Switch Spine
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 35
L3 Domain
spine
edge
L3 edge
M1/F1
FP port FP port
M1/F1
• By separating the L3 function from the spine, the F1 card in the spine performs pure switch-id forwarding
• The L3 edge will need both M1/F1 in order to connect with Fabricpath ports to the spine
• The M1/F1 L3 edge will need to perform learning for the remote mac addresses
• L3 edge and spine can be combined in the same chassis by means of VDCs
spine
Conversational
Learning
Conversational Learning
Sw
itch-id
ba
se
d
forw
ard
ing
FabricPath L2/L3 Boundary Location Leaf/Spine/Boundary Architecture
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 36
Nexus Edge, Core & Boundary Nodes
Large Scale Fabric 4K VLAN’s, 128K MAC Address,
512K Routes
slot 1 slot 2 slot 3 slot 4 slot 5 slot 6 slot 7 slot 8
blade1 blade2 blade3 blade4 blade5 blade6 blade7 blade8
slot 1 slot 2 slot 3 slot 4 slot 5 slot 6 slot 7 slot 8
blade1 blade2 blade3 blade4 blade5 blade6 blade7 blade8
slot 1 slot 2 slot 3 slot 4 slot 5 slot 6 slot 7 slot 8
blade1 blade2 blade3 blade4 blade5 blade6 blade7 blade8
slot 1 slot 2 slot 3 slot 4 slot 5 slot 6 slot 7 slot 8
blade1 blade2 blade3 blade4 blade5 blade6 blade7 blade8
slot 1 slot 2 slot 3 slot 4 slot 5 slot 6 slot 7 slot 8
blade1 blade2 blade3 blade4 blade5 blade6 blade7 blade8
slot 1 slot 2 slot 3 slot 4 slot 5 slot 6 slot 7 slot 8
blade1 blade2 blade3 blade4 blade5 blade6 blade7 blade8
slot 1 slot 2 slot 3 slot 4 slot 5 slot 6 slot 7 slot 8
blade1 blade2 blade3 blade4 blade5 blade6 blade7 blade8
slot 1 slot 2 slot 3 slot 4 slot 5 slot 6 slot 7 slot 8
blade1 blade2 blade3 blade4 blade5 blade6 blade7 blade8
slot 1 slot 2 slot 3 slot 4 slot 5 slot 6 slot 7 slot 8
blade1 blade2 blade3 blade4 blade5 blade6 blade7 blade8
slot 1 slot 2 slot 3 slot 4 slot 5 slot 6 slot 7 slot 8
blade1 blade2 blade3 blade4 blade5 blade6 blade7 blade8
slot 1 slot 2 slot 3 slot 4 slot 5 slot 6 slot 7 slot 8
blade1 blade2 blade3 blade4 blade5 blade6 blade7 blade8
slot 1 slot 2 slot 3 slot 4 slot 5 slot 6 slot 7 slot 8
blade1 blade2 blade3 blade4 blade5 blade6 blade7 blade8
slot 1 slot 2 slot 3 slot 4 slot 5 slot 6 slot 7 slot 8
blade1 blade2 blade3 blade4 blade5 blade6 blade7 blade8
slot 1 slot 2 slot 3 slot 4 slot 5 slot 6 slot 7 slot 8
blade1 blade2 blade3 blade4 blade5 blade6 blade7 blade8
slot 1 slot 2 slot 3 slot 4 slot 5 slot 6 slot 7 slot 8
blade1 blade2 blade3 blade4 blade5 blade6 blade7 blade8
slot 1 slot 2 slot 3 slot 4 slot 5 slot 6 slot 7 slot 8
blade1 blade2 blade3 blade4 blade5 blade6 blade7 blade8
slot 1 slot 2 slot 3 slot 4 slot 5 slot 6 slot 7 slot 8
blade1 blade2 blade3 blade4 blade5 blade6 blade7 blade8
slot 1 slot 2 slot 3 slot 4 slot 5 slot 6 slot 7 slot 8
blade1 blade2 blade3 blade4 blade5 blade6 blade7 blade8
slot 1 slot 2 slot 3 slot 4 slot 5 slot 6 slot 7 slot 8
blade1 blade2 blade3 blade4 blade5 blade6 blade7 blade8
slot 1 slot 2 slot 3 slot 4 slot 5 slot 6 slot 7 slot 8
blade1 blade2 blade3 blade4 blade5 blade6 blade7 blade8
slot 1 slot 2 slot 3 slot 4 slot 5 slot 6 slot 7 slot 8
blade1 blade2 blade3 blade4 blade5 blade6 blade7 blade8
slot 1 slot 2 slot 3 slot 4 slot 5 slot 6 slot 7 slot 8
blade1 blade2 blade3 blade4 blade5 blade6 blade7 blade8
slot 1 slot 2 slot 3 slot 4 slot 5 slot 6 slot 7 slot 8
blade1 blade2 blade3 blade4 blade5 blade6 blade7 blade8
slot 1 slot 2 slot 3 slot 4 slot 5 slot 6 slot 7 slot 8
blade1 blade2 blade3 blade4 blade5 blade6 blade7 blade8
slot 1 slot 2 slot 3 slot 4 slot 5 slot 6 slot 7 slot 8
blade1 blade2 blade3 blade4 blade5 blade6 blade7 blade8
slot 1 slot 2 slot 3 slot 4 slot 5 slot 6 slot 7 slot 8
blade1 blade2 blade3 blade4 blade5 blade6 blade7 blade8
slot 1 slot 2 slot 3 slot 4 slot 5 slot 6 slot 7 slot 8
blade1 blade2 blade3 blade4 blade5 blade6 blade7 blade8
slot 1 slot 2 slot 3 slot 4 slot 5 slot 6 slot 7 slot 8
blade1 blade2 blade3 blade4 blade5 blade6 blade7 blade8
slot 1 slot 2 slot 3 slot 4 slot 5 slot 6 slot 7 slot 8
blade1 blade2 blade3 blade4 blade5 blade6 blade7 blade8
slot 1 slot 2 slot 3 slot 4 slot 5 slot 6 slot 7 slot 8
blade1 blade2 blade3 blade4 blade5 blade6 blade7 blade8
slot 1 slot 2 slot 3 slot 4 slot 5 slot 6 slot 7 slot 8
blade1 blade2 blade3 blade4 blade5 blade6 blade7 blade8
slot 1 slot 2 slot 3 slot 4 slot 5 slot 6 slot 7 slot 8
blade1 blade2 blade3 blade4 blade5 blade6 blade7 blade8
slot 1 slot 2 slot 3 slot 4 slot 5 slot 6 slot 7 slot 8
blade1 blade2 blade3 blade4 blade5 blade6 blade7 blade8
slot 1 slot 2 slot 3 slot 4 slot 5 slot 6 slot 7 slot 8
blade1 blade2 blade3 blade4 blade5 blade6 blade7 blade8
slot 1 slot 2 slot 3 slot 4 slot 5 slot 6 slot 7 slot 8
blade1 blade2 blade3 blade4 blade5 blade6 blade7 blade8
slot 1 slot 2 slot 3 slot 4 slot 5 slot 6 slot 7 slot 8
blade1 blade2 blade3 blade4 blade5 blade6 blade7 blade8
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 37
TRILL Forwarding
NextHop Header
Ethernet Header
TRILL Header
Cisco Forwarding
Outer CDCE DA
Outer CDCE DA
Outer CDCE SA
Outer CDCE SA
ET = DTAG
ET = 802.1Q Inner VLAN
Payload…
Inner MAC DA
Inner MAC SA Inner MAC DA Inner MAC SA
Inner MAC SA Inner MAC SA
FTAG TTL
3
1 0
Outer MAC DA
Outer MAC DA
Outer MAC SA
Outer MAC SA
ET = 802.1Q Outer VLAN
ET = TRILL
Egress RB Ingress RB
V/R/M, HopCnt
Inner MAC DA
Inner MAC DA
Inner MAC SA
Inner MAC SA
ET = 802.1Q Inner VLAN
Payload...
3
1 0
Standards Based + Cisco Extensions
• Nexus 5500, F1, F2
and all future HW are
capable of IETF
standards TRILL
• Support for TRILL in
NX-OS is pending
completion of
extensions to the
baseline protocol
• Multi-topology, VRRP
interaction, …
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 38
FabricPath
POD vPC POD
L3 Core
vPC+ POD vPC+ POD
L2+L3
FabricPath
Core
Site
1
Site 2 Site 4
Site
3
FabricPath FabricPath
Fabric
Path
F
abric
Path
FabricPath FabricPath
Flexibility in the Fabric - Layer 2 Routing
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 39
Agenda
The Evolving Data Centre Fabric
FabricPath
VXLAN
LISP
LISP Host Mobility
OTV LAN Extension
Mobility with Extended Subnets
Nexus Fabric
1K Cisco Nexus x8
6
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 40
• Customer Requirement
Secure movement of vApps across cloud infrastructure
• Solution: VXLAN
Millions of dedicated LAN segments
Security at Scale
vApp mobility across data centers & clouds
• VXLAN is network friendly
Efficient load sharing of links (port channel)
Supports NAT; better security controls
VXLAN IETF Draft: http://datatracker.ietf.org/doc/draft-mahalingam-dutt-dcops-vxlan/
Web
VM
DB
VM
vApp1
App
VM
vApp2 Web
VM
DB
VM
App
VM
Tenant Network
(VLAN)
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 41
Ethernet in IP overlay network
Entire L2 frame encapsulated in UDP
50 bytes of overhead
Include 24 bit VXLAN Identifier
16 Million logical networks
VXLAN can cross Layer 3 (IPv4 currently)
Tunnel between VEMs
VMs do NOT see VXLAN ID
IP multicast used for L2 broadcast/multicast, unknown unicast
Technology submitted to IETF for standardization (Cisco, VMware, Citrix, Red Hat, Broadcom, Arista, and Others)
Outer
MAC
DA
Outer
MAC
SA
Outer
802.1Q
Outer
IP DA
Outer
IP SA
Outer
UDP
VXLAN
Header (8
bytes)
Inner
MAC
DA
InnerM
AC
SA
Optional
Inner
802.1Q
Original
Ethernet
Payload
CRC
VXLAN Encapsulation Original Ethernet Frame
Flags8
bits
Res.
8 bits Reserved
24 bits
VXLAN
Networker
Identifier (VIN)
24 bits
Cisco Confidential © 2010 Cisco and/or its affiliates. All rights reserved. 42
• The Nexus 1000V VEMs act as the VXLAN Tunnel Endpoints (VTEP)
• Nexus 1000V uses a VMKNIC to terminate VTEP traffic
• VM to VM traffic on different access switches is encapsulated in a VXLAN header + UDP + IP
• VTEPs use multicast to deliver unknown destination VM MAC addresses to all VTEPs participating in a given VXLANs
• VM MAC to VTEP IP address mappings are gleaned from encapsulated packets
Similar to Ethernet bridge flood and learn behavior
• Known destination VM MAC addresses are carried over point to point tunnels between VTEPs
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 43 43
VXLAN‟s IP Any Source Multicast Group (*,G)
acts as a bus for delivery to all relevant VTEPs
for a given VNI
(Carries unknown/broadcast/multicast frames)
VTEP
VTEP
VTEP
VTEP
Direct Unicast tunnels between VTEPs
(Carries known unicast frames)
Bridge
Domain
Switch
End
System
End
System
VTEP
Access
Switch
Bridge
Domain
Switch
End
System
End
System
VTEP
Access
Switch
IP Multicast
Enabled Underlying
Network
VTEP = VXLAN Tunnel End Point VNI = VXLAN Network Identifier
Cisco Confidential © 2010 Cisco and/or its affiliates. All rights reserved. 45
Web
VM
Web
VM
DB
VM
DB
VM
Join Multicast
Group 239.1.1.1
Join Multicast
Group 239.2.2.2
Join Multicast
Group 239.2.2.2 Join Multicast
Group 239.1.1.1
VTEP Use Of IGMP IGMP Used to Join Each VXLANs Assigned Multicast Group on Demand
Cisco Confidential © 2010 Cisco and/or its affiliates. All rights reserved. 46
VM 1 VM 2 VM 3
VXLAN
VMKNIC
1.1.1.1
VXLAN
VMKNIC
2.2.2.2
VXLAN
VMKNIC
3.3.3.3
MAC:
abc MAC:
xyz
Multicast Multicast Multicast
VEM 1 VEM 2 VEM 3
VXLAN Example Data Flow VM1 Communicating with VM2 in a VXLAN
Cisco Confidential © 2010 Cisco and/or its affiliates. All rights reserved. 47
VM 1 VM 2 VM 3
VXLAN
VMKNIC
1.1.1.1
VXLAN
VMKNIC
2.2.2.2
VXLAN
VMKNIC
3.3.3.3
MAC:
abc MAC:
xyz
VM Source MAC Remote Host
VXLAN IP
VM1:abc 1.1.1.1
Unicast
MAC Table: VEM 2
Layer 3
VXLAN Example Data Flow VM1 Communicating with VM2 in a VXLAN
Cisco Confidential © 2010 Cisco and/or its affiliates. All rights reserved. 48
VM 1 VM 2 VM 3
VXLAN
VMKNIC
1.1.1.1
VXLAN
VMKNIC
2.2.2.2
VXLAN
VMKNIC
3.3.3.3
MAC:
abc MAC:
xyz
VM Source MAC Remote Host
VXLAN IP
VM1:abc 1.1.1.1
MAC Table: VEM 2 VM Source MAC Remote Host
VXLAN IP
VM2:xyz 2.2.2.2
MAC Table: VEM 1
VEM 1 VEM 2 VEM 3
VXLAN Example Data Flow VM1 Communicating with VM2 in a VXLAN
Cisco Confidential © 2010 Cisco and/or its affiliates. All rights reserved. 49
VM 1 VM 2 VM 3
VXLAN
VMKNIC
1.1.1.1
VXLAN
VMKNIC
2.2.2.2
VXLAN
VMKNIC
3.3.3.3
MAC:
abc MAC:
xyz
Unicast
VM Source MAC Remote Host
VXLAN IP
VM2:xyz 2.2.2.2
MAC Table: VEM 1
VM Source MAC Remote Host
VXLAN IP
VM1:abc 1.1.1.1
MAC Table: VEM 2
VXLAN Example Data Flow VM1 Communicating with VM2 in a VXLAN
Cisco Confidential © 2010 Cisco and/or its affiliates. All rights reserved. 50
Web
VM
App
VM
App
VM
DB
VM
• Encapsulate with Blue VXLAN ID
• Multicast to Servers Registered for
239.1.1.1 Multicast Group
VEM Discards Since No VM with
Blue VXLAN ID
VM Broadcast Frames Sent to More Servers
But Broadcast Domain Respected Within VXLAN Segment
Multiple VXLANs Can Share One Multicast Group Blue & Red VXLANs Share The 239.1.1.1 Multicast Group
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 51
Agenda
The Evolving Data Centre Fabric
FabricPath
VXLAN
LISP
LISP Host Mobility
OTV LAN Extension
Mobility with Extended Subnets
Nexus Fabric
1K Cisco Nexus x8
6
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 52
Use-Cases
DCI route optimization/mobility
Workload Portability to Cloud
Secure Multi-tenancy across organizations
Rapid IPv6 Deployment
Route scaling
Single Network Architecture Delivers:
VM Mobility (topology independent addressing)
Security: VPNs/Multi-tenancy
Route Scalability (on demand routing)
IPv6 enablement,
Routing Policy simplification
Benefits
Services integrated in a single architecture
Services can be offered across organizational boundaries (multiple providers)
Very large scale
Open model to integrate with cloud orchestrators
Cisco Confidential © 2010 Cisco and/or its affiliates. All rights reserved. 53
IPv6 Transition Support
v6-over-v4, v6-over-v6
v4-over-v6, v4-over-v4
IPv4
Internet
IPv6
Internet
v6
v6 v4 v6
LISP
Router LISP
Router
v6 Services
LISP Use Cases Consolidated Architecture with Multiple Applications
Efficient Multi-Homing
IP Portability
Ingress Traffic Engineering without BGP
LISP
Routers
LISP
Site
Internet
Host-Mobility
Cloud / Layer 3 VM moves
Segmentation
West-DC East-DC
LISP Site
IP Network
Multi-Tenancy and VPNs
Reduced CapEx/OpEx
Large scale Segmentation
West-DC East-DC
LISP Site
IP Network
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 54
IP core
Device IPv4 or IPv6
Address Represents
Identity and Location
Today‟s IP Behavior Loc/ID “Overloaded” Semantic
10.1.0.1 When the Device Moves, It Gets a
New IPv4 or IPv6 Address for Its
New Identity and Location 20.2.0.9
Device IPv4 or IPv6
Address Represents
Identity Only.
When the Device Moves, Keeps
Its IPv4 or IPv6 Address.
It Has the Same Identity
LISP Behavior Loc/ID “Split”
IP core
1.1.1.1
2.2.2.2
Only the Location Changes
10.1.0.1
10.1.0.1
Location Identity Separation Protocol What Do We Mean by “Location” and “Identity”?
Its Location Is Here!
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 55
A LISP Packet Walk How Does LISP Operate?
Non-LISP site
East-DC
LISP Site
IP Network
ETR
EID-to-RLOC mapping
5.1.1.1
5.3.3.3
1.1.1.1
5.2.2.2
10.3.0.0/24 10.2.0.0/24
West-DC
PITR
5.4.4.4
10.1.0.0/24
Non-LISP site
ITR S
D
DNS Entry: D.abc.com A 10.2.0.1
1
10.1.0.1 -> 10.2.0.1
2
EID-prefix: 10.2.0.0/24
Locator-set:
2.1.1.1, priority: 1, weight: 50 (D1)
2.1.2.1, priority: 1, weight: 50 (D2)
Mapping
Entry
3
This Policy Controlled
by Destination Site
10.1.0.1 -> 10.2.0.1
1.1.1.1 -> 2.1.1.1
4
10.1.0.1 -> 10.2.0.1
5
2.1.1.1 2.1.2.1 3.1.1.1 3.1.2.1
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 56
A LISP Packet Walk How About Non-LISP Sites?
Non-LISP Site
East-DC
IP Network
ETR
EID-to-RLOC mapping
5.1.1.1
5.3.3.3
5.2.2.2
10.3.0.0/24 10.2.0.0/24
West-DC
PITR
4.4.4.4
Non-LISP Site
S
D
DNS Entry: D.abc.com A 10.2.0.1
1
192.3.0.1 -> 10.2.0.1
2
EID-Prefix: 10.2.0.0/24
Locator-Set:
2.1.1.1, priority: 1, weight: 50 (D1)
2.1.2.1, priority: 1, weight: 50 (D2)
Mapping
Entry
3
192.3.0.1 -> 10.2.0.1
4.4.4.4- > 2.1.2.1
4
192.3.0.1 -> 10.2.0.1
5
2.1.1.1 2.1.2.1 3.1.1.1 3.1.2.1
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 57
LISP Roles
• Tunnel Routers - xTRs
• Edge devices in charge of encap/decap
• Ingress/Egress Tunnel Routers (ITR/ETR)
• EID to RLOC Mapping DB
• Contains RLOC to EID mappings
• Distributed across multiple Map Servers (MS)
• MS may connect over an ALT network
• Proxy Tunnel Routers - PxTR
• Coexistence between LISP and non-LISP sites
• Ingress/Egress: PITR, PETR
Address Spaces
• EID = End-point Identifier
• Host IP or prefix
• RLOC = Routing Locator
• IP address of routers in the backbone
Prefix Next-hop w.x.y.1 e.f.g.h
x.y.w.2 e.f.g.h
z.q.r.5 e.f.g.h
z.q.r.5 e.f.g.h
Mapping
DB
ITR
ETR
Non-LISP
EID Space
EID Space
RLOC Space
EID RLOC a.a.a.0/24 w.x.y.1
b.b.b.0/24 x.y.w.2
c.c.c.0/24 z.q.r.5
d.d.0.0/16 z.q.r.5
EID RLOC a.a.a.0/24 w.x.y.1
b.b.b.0/24 x.y.w.2
c.c.c.0/24 z.q.r.5
d.d.0.0/16 z.q.r.5
EID RLOC a.a.a.0/24 w.x.y.1
b.b.b.0/24 x.y.w.2
c.c.c.0/24 z.q.r.5
d.d.0.0/16 z.q.r.5
ALT
PxTR
LISP Roles and Address Spaces What Are the Different Components Involved?
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 58
LISP Mapping Database The Basics – Registration and Resolution
West-DC East-DC
X Z
Y
Y
10.2.0.2
10.2.0.0 /16 10.3.0.0/16
Map Server / Resolver: 5.1.1.1
2.1.1.1 2.1.2.1 3.1.1.1 3.1.2.1
LISP Site
ITR
10.2.0.0/16 -> (2.1.1.1, 2.1.2.1) Database Mapping Entry (on ETR):
10.3.0.0/16 -> (3.1.1.1, 3.1.2.1) Database Mapping Entry (on ETR):
ETR ETR ETR ETR
Map-Reply 10.2.0.0/16 -> (2.1.1.1, 2.1.2.1)
10.2.0.0/16-> (2.1.1.1, 2.1.2.1) Mapping Cache Entry (on ITR):
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 59
West-DC East-DC
Non-LISP Sites
PITR LISP Site
IP Network
EID RLOC LISP Encap/Decap
ITR
Mapping DB
5.1.1.1
5.3.3.3
1.1.1.1
10.2.0.0/24
5.2.2.2
Basic LISP Configuration
ETR
2.1.1.1 2.1.2.1
Branch Routers
ip lisp itr-etr
ip lisp ITR map-resolver 5.3.3.3
DC Aggregation Routers
ip lisp itr-etr
ip lisp database-mapping 10.2.0.0/24 2.1.1.1 p1 w50
ip lisp database-mapping 10.2.0.0/24 2.1.2.1 p1 w50
ip lisp ETR map-server 5.1.1.1 key s3cr3t
ip lisp ETR map-server 5.2.2.2 key s3cr3t
Border Routers Between Backbones
ip lisp proxy-itr
ip lisp ITR map-resolver 5.3.3.3
Servers
ip lisp map-resolver
ip lisp map-server
lisp site west-DC
authentication-key 0 s3cr3t
eid-prefix 10.2.0.0/24
Usually Devices Will Be Configured as ITRs and ETRs
to Handle Traffic in Both Directions;
We Illustrate Only One Direction for Simplicity
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 60
Agenda
The Evolving Data Centre Fabric
FabricPath
VXLAN
LISP
LISP Host Mobility
OTV LAN Extension
Mobility with Extended Subnets
Nexus Fabric
1K Cisco Nexus x8
6
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 61
West-DC East-DC
Non-LISP Sites
PxTR LISP Site
IP Network
EID RLOC LISP Encap/Decap
xTR
Needs:
•Global IP-Mobility across subnets
•Optimized routing across extended subnet sites
LISP Solution:
•Automated move detection on xTRs
•Dynamically update EID-to-RLOC mappings
•Traffic Redirection on ITRs or PITRs
Benefits:
•Direct Path (no triangulation)
•Connections maintained across move
•No routing re-convergence
•No DNS updates required
•Transparent to the hosts
•Global Scalability (cloud bursting)
• IPv4/IPv6 Support
LAN Extensions
Mapping DB
LISP Host-Mobility
LISP-VM (xTR)
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 62
Routing for Extended Subnets
Active-Active Data Centers
Distributed Clusters
Moves With LAN Extension
Host-Mobility Scenarios
West-DC East-DC
Non-LISP Site
IP Network
Mapping DB
LISP-VM (xTR)
LAN Extension
LISP Site
xTR
Application Members Distributed (Broadcasts across sites)
IP Mobility Across Subnets
Disaster Recovery
Cloud Bursting
Moves Without LAN Extension
West-DC East-DC
LISP Site
Internet or Shared WAN
xTR
Mapping DB DR Location or Cloud Provider
DC
LISP-VM (xTR)
Application Members in One Location
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 63
• The new xTR checks the source of received traffic
• Configured dynamic-EIDs define which prefixes may roam
West-DC East-DC
LISP-VM (xTR)
X Z
Y
Y
Mapping DB
10.2.0.2
10.2.0.0 /16 10.3.0.0/16
5.1.1.1 5.2.2.2
lisp dynamic-eid roamer
database-mapping 10.2.0.0/24 <RLOC-C> p1 w50
database-mapping 10.2.0.0/24 <RLOC-D> p1 w50
map-server 5.1.1.1 key abcd
interface vlan 100
lisp mobility roamer
A B C D
Received a Packet …
… It’s from a “New” Host
… It’s in the Dynamic-EID Allowed
Range
…It’s a Move!
Register the /32 with LISP
LISP Host-Mobility - Move Detection Monitor the Source of Received Traffic
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 64
• When a host move is detected, updates are triggered:
The host-to-location mapping in the Database is updated to reflect the new location
The old ETR is notified of the move
ITRs are notified to update their Map-caches
• Ingress routers (ITRs or PITRs) now send traffic to the new location
West-DC East-DC
LISP-VM (xTR)
X Z
Y
Y
Mapping DB
10.2.0.2
10.2.0.0 /16 10.3.0.0 /16
A B C D
LISP Site xTR
10.2.0.0/16 – RLOC A, B
10.2.0.2/32 – RLOC C, D
LISP Host-Mobility - Traffic Redirection Update Location Mappings for the Host System Wide
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 65
• SVI (Interface VLAN x) and HSRP configured as usual (Consistent GWY-MAC configured across all dynamic subnets)
• The lisp mobility <dyn-eid-map> command enables proxy-arp functionality on the SVI
The LISP-VM router services first hop routing requests for both local and roaming subnets
• Hosts can move anywhere and always talk to a local gateway with the same MAC
West-DC East-DC
LISP-VM (xTR)
A B C D
HSRP
ARP
GWY-MAC
HSRP
ARP
GWY-MAC
HSRP Active
interface Ethernet2/4
ip address 10.1.0.6/24
lisp mobility roamer
ip proxy-arp
hsrp 101
mac-address 0000.0e1d.010c
ip 10.2.0.1
HSRP Active
interface vlan 100
ip address 10.2.0.5/24
lisp mobility roamer
ip proxy-arp
hsrp 101
mac-address 0000.0e1d.010c
ip 10.2.0.1
interface vlan 200
ip address 10.2.0.8/24
lisp mobility roamer
ip proxy-arp
hsrp 201
mac-address 0000.0e1d.010c
ip 10.3..0.1
interface vlan 100
ip address 10.3.0.7/24
lisp mobility roamer
ip proxy-arp
hsrp 201
mac-address 0000.0e1d.010c
ip 10.3.0.1
10.2.0.0 /24 10.3.0.0 /24
LISP Host-Mobility - First Hop Routing Across Different Subnets
10.2.0.2
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 66
Null0 host routes indicate the host is “away”
West-DC East-DC
X
Y
Y
Mapping DB
10.2.0.2
10.2.0.0 /16 10.3.0.0 /16
5.1.1.1 5.2.2.2
A B C D
Routing Table:
10.3.0.0/16 – Local
10.2.0.2/32 – Local
Routing Table:
10.3.0.0/16 – Local
10.2.0.2/32 – Local
Map-Notify
10.2.0.2/32 <C,D>
1
Routing Table:
10.2.0.0/16 – Local
10.2.0.2/32 – Null0
Routing Table:
10.2.0.0/16 – Local
10.2.0.2/32 – Null0
Map-Notify
10.2.0.2/32 <C,D>
Map-Register
10.2.0.2/32 <C,D>
10.2.0.0/16 – RLOC A, B
10.2.0.2/32 – RLOC C, D
3
7 5
9
2
4
6
8
10
Map-Notify
10.2.0.2/32 <C,D>
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 67
1. ITRs and PITRs with cached mappings continue to send traffic to the old locators
The old xTR knows the host has moved (Null0 route).
2. Old xTR sends Solicit Map Request (SMR) messages to any encapsulators sending traffic to the moved host
3. The ITR then initiates a new map request process
4. An updated map-reply is issued from the new location
5. The ITR Map Cache is updated
• Traffic is now re-directed
• SMRs are an important integrity measure to avoid unsolicited map responses and spoofing
West-DC East-DC
LISP-VM (xTR)
X Z Y
Y
Mapping DB
10.2.0.2
10.2.0.0 /16 10.3.0.0 /16
A B C D
LISP site
ITR
10.2.0.2/32 – RLOC C,D
Map Cache @ ITR
10.2.0.0/16 – RLOC A,B
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 68
West-DC East-DC
LISP-VM (xTR)
X Z Y
A B C D
ip lisp ITR-ETR
ip lisp database-mapping 10.2.0.0/16 <RLOC-A>
ip lisp database-mapping 10.2.0.0/16 <RLOC-B>
lisp dynamic-eid roamer
database-mapping 10.2.0.0/24 <RLOC-A>
database-mapping 10.2.0.0/24 <RLOC-B>
map-server 1.1.1.1 key abcd
map-notify-group 239.1.1.1
interface vlan 100
ip address 10.2.0.10 /16
lisp mobility roamer
ip proxy-arp
hsrp 101
mac-address 0000.0e1d.010c
ip 10.2.0.1 Mapping DB
ip lisp ITR-ETR
ip lisp database-mapping 10.3.0.0/16 <RLOC-C>
ip lisp database-mapping 10.3.0.0/16 <RLOC-D>
lisp dynamic-eid roamer
database-mapping 10.2.0.0/24 <RLOC-C>
database-mapping 10.2.0.0/24 <RLOC-D>
map-server 1.1.1.1 key abcd
map-notify-group 239.2.2.2
interface vlan 100
ip address 10.3.0.11 /16
lisp mobility roamer
ip proxy-arp
hsrp 201
mac-address 0000.0e1d.010c
ip 10.3.0.1
10.2.0.0 /16 10.3.0.0 /16
LISP Host-Mobility Configuration Across Subnets (No LAN Extensions)
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 69
Agenda
The Evolving Data Centre Fabric
FabricPath
VXLAN
LISP
LISP Host Mobility
OTV LAN Extension
Mobility with Extended Subnets
Nexus Fabric
1K Cisco Nexus x8
6
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 70
• Ethernet LAN Extension over any Network
Works over dark fiber, MPLS, or IP
Multi-data center scalability
• Simplified Configuration & Operation
Seamless overlay - No network re-design
Single touch site configuration
• High Resiliency
Failure domain isolation
Seamless Multi-homing
• Maximizes available bandwidth
Automated multi-pathing
Optimal multicast replication
Simplifying LAN Extensions
Many Physical Sites –
One Logical Data Center
Any Workload, Anytime, Anywhere
Unleashing the Full Potential of Compute Virtualization
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 71
Transport
Infrastructure
OTV OTV OTV OTV
MAC TABLE
VLAN MAC IF
100 MAC 1 Eth 2
100 MAC 2 Eth 1
100 MAC 3 IP B
100 MAC 4 IP B
MAC 1 MAC 3
IP A IP B MAC 1 MAC 3
MAC TABLE
VLAN MAC IF
100 MAC 1 IP A
100 MAC 2 IP A
100 MAC 3 Eth 3
100 MAC 4 Eth 4
Layer 2
Lookup
5 IP A IP B MAC 1 MAC 3 MAC 1 MAC 3 Layer 2
Lookup
1 Encap
2 Decap
4
MAC 1 MAC 3 West
Site MAC 1 MAC 3
East
Site
1. Layer 2 lookup on the destination MAC.
MAC 3 is reachable through IP B
2. The Edge Device encapsulates the frame
3. The transport delivers the packet to the
Edge Device on site East
4. The Edge Device on site East receives
and decapsulates the packet
5. Layer 2 lookup on the original frame.
MAC 3 is a local MAC
6. The frame is delivered to the destination
3
6
IP A IP B
OTV Data Plane Inter-Site Packet Flow
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 72
West
OTV
The OTV Control Plane
• OTV proactively advertises MAC reachability (control-plane learning)
• MAC addresses advertised in the background once OTV has been configured
• IS-IS is the OTV Control Protocol running between the Edge Devices
• No specific configuration is required
IP A IP B
IP C
East
South
MAC Addresses
Advertisements OTV
OTV
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 73
Agenda
The Evolving Data Centre Fabric
FabricPath
VXLAN
LISP
LISP Host Mobility
OTV LAN Extension
Mobility with Extended Subnets
Nexus Fabric
1K Cisco Nexus x8
6
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 74
• A subnet usually implies location
• Yet we use LAN extensions to stretch subnets across locations
Location semantics of subnets are lost
• Traditional routing relies on the location semantics of the subnet
Can‟t tell if a server is at the East or West location of the subnet
• More granular (host level) information is required
LISP provides host level location semantics
Ingress Routing Challenge in DCI Extending Subnets Creates a Routing Challenge
West-DC East-DC
IP Network
LAN Extension
LISP site
xTR
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 75
Host-Mobility and Multi-homing ETR updates – Extended Subnets Null0 host routes indicate the host is “away”
West-DC East-DC
X
Y
Y
Mapping DB
10.2.0.2
10.2.0.0 /16 10.2.0.0 /16
5.1.1.1 5.2.2.2
A B C D
Routing Table:
10.2.0.0/16 – Local
10.2.0.0/24 – Null0
10.2.0.2/32 – Local
Routing Table:
10.2.0.0/16 – Local
10.2.0.0/24 – Null0
10.2.0.2/32 – Local
Map-Notify
10.2.0.2/32 <C,D>
Routing Table:
10.2.0.0/16 – Local
10.2.0.0/24 – Null0
10.2.0.2/32 – Null0
Routing Table:
10.2.0.0/16 – Local
10.2.0.0/24 – Null0
10.2.0.2/32 – Null0
Map-Register
10.2.0.2/32 <C,D>
10.2.0.0/16 – RLOC A, B
10.2.0.2/32 – RLOC C, D
3
5
3
2
4
6
4
Map-Notify
10.2.0.2/32 <C,D>
OTV
4
1
10.2.0.0 /24 is the dyn-EID
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 76
Refreshing the map caches
1. ITRs and PITRs with cached mappings continue to send traffic to the old locators
1. The old xTR knows the host has moved (Null0 route).
2. Old xTR sends Solicit Map Request (SMR) messages to any encapsulators sending traffic to the moved host
3. The ITR then initiates a new map request process
4. An updated map-reply is issued from the new location
5. The ITR Map Cache is updated
• Traffic is now re-directed
• SMRs are an important integrity measure to avoid unsolicited map responses and spoofing
West-DC East-DC
LISP-VM (xTR)
X Z Y
Y
Mapping DB
10.2.0.2
10.2.0.0 /16 10.2.0.0 /16
A B C D
LISP site
ITR
10.2.0.2/32 – RLOC C,D
Map Cache @ ITR
10.2.0.0/16 – RLOC A,B
OTV
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 77
• Consistent GWY-IP and GWY-MAC configured across all sites
Consistent HSRP group number across sites consistent GWY-MAC
• Servers can move anywhere and always talk to a local gateway with the same IP/MAC
West-DC East-DC
LISP-VM (xTR)
A B C D
HSRP
ARP
GWY-MAC
HSRP
ARP
GWY-MAC
HSRP Active
HSRP Active
10.2.0.0 /24 10.2.0.0 /24
LISP Host-Mobility - First Hop Routing With Extended Subnets
LAN Ext.
interface Ethernet2/4
ip address 10.2.0.6/24
lisp mobility roamer
lisp extended-subnet-mode
hsrp 101
ip 10.2.0.1
interface vlan 100
ip address 10.2.0.5/24
lisp mobility roamer
lisp extended-subnet-mode
hsrp 101
ip 10.2.0.1
interface vlan 200
ip address 10.2.0.8/24
lisp mobility roamer
lisp extended-subnet-mode
hsrp 101
ip 10.2.0.1
interface vlan 100
ip address 10.2.0.7/24
lisp mobility roamer
lisp extended-subnet-mode
hsrp 101
ip 10.2.0.1
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 78
West-DC East-DC
LISP-VM (xTR)
X Z Y
10.2.0.0/16
1.1.1.1 2.2.2.2
A B C D
LAN Ext.
ip lisp ITR-ETR
ip lisp database-mapping 10.2.0.0/16 <RLOC-A>
ip lisp database-mapping 10.2.0.0/16 <RLOC-B>
ip lisp database-mapping 10.2.0.0/16 <RLOC-C>
ip lisp database-mapping 10.2.0.0/16 <RLOC-D>
lisp dynamic-eid roamer
database-mapping 10.2.0.0/24 <RLOC-A> …
database-mapping 10.2.0.0/24 <RLOC-B>
map-server 1.1.1.1 key abcd
map-notify-group 239.10.10.10
interface vlan 100
ip address 10.2.0.10 /16
lisp mobility roamer
lisp extended-subnet-mode
hsrp 101
ip 10.2.0.1
Mapping DB
ip lisp ITR-ETR
ip lisp database-mapping 10.2.0.0/16 <RLOC-A>
ip lisp database-mapping 10.2.0.0/16 <RLOC-B>
ip lisp database-mapping 10.2.0.0/16 <RLOC-C>
ip lisp database-mapping 10.2.0.0/16 <RLOC-D>
lisp dynamic-eid roamer
database-mapping 10.2.0.0/24 <RLOC-C>
database-mapping 10.2.0.0/24 <RLOC-D>
map-server 1.1.1.1 key abcd
map-notify-group 239.10.10.10
interface vlan 100
ip address 10.2.0.11 /16
lisp mobility roamer
lisp extended-subnet-mode
hsrp 101
ip 10.2.0.1
LISP VM-Mobility Configuration With Extended Subnets Use “Extended-Subnet-Mode”
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 79
• Clients (192.168.0.1 & 192.168.2.1 communicate with Server 10.2.0.2
• Client-server traffic is LISP encapsulated at the ITRs or PITRs
Client-to-server:
to ETRs C or D
Server-to-client:
to ETR (F) for LISP sites
to PETR (G) for non-LISP sites
• Server-Server off-subnet traffic across sites is also LISP encapsulated
All Off-Subnet/Off-Site Traffic Is LISP Encapsulated
West-DC East-DC
LISP-VM (xTR)
X Y
Y
Mapping DB
10.2.0.2
10.2.0.0 /16 10.3.0.0 /16
A B C D
LISP Site xTR
F
CLIENT 10.1.0.1
Non-LISP Sites
PxTR G
CLIENT 192.168.2.1
192.168.2.1 10.2.0.2
10.1.0.1 10.2.0.2
10.1.0.1 10.2.0.2
192.168.2.1 10.2.0.2
FC 10.1.0.1 10.2.0.2
GD 192.168.2.1 10.2.0.2
Off-Subnet Client-Server Traffic
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 80
• Live moves and cluster member dispersion
• Traffic between X & Y uses the LAN Extension
• Link-local-multicast handled by the LAN Extension
• Cold moves, no application dispersion
• X- Y traffic is sent to the LISP-VM router & LISP encapsulated
• Need LAN extensions for link-local multicast traffic
On Subnet Traffic Across L3 boundaries With LAN Extension Without LAN Extensions
West-DC
East-DC
LISP-VM (xTR)
Z
Y
Y
10.2.0.2
A
10.2.0.0/16
LAN Ext.
B C D
10.2.0.3 10.2.0.2
West-DC
East-DC
LISP-VM (xTR)
Z
Y
Y
10.2.0.2
A
10.2.0.3
X
Mapping DB
B C D
BC 10.2.0.3 10.2.0.2
10.2.0.0/16 10.3.0.0/16
10.2.0.3
X
On-Subnet Server-Server Traffic
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 81
Agenda
The Evolving Data Centre Fabric
FabricPath
VXLAN
LISP
LISP Host Mobility
OTV LAN Extension
Mobility with Extended Subnets
Nexus Fabric
1K Cisco Nexus x8
6
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 82
DC-east DC-west
• Enhance application availability by distributing Cluster members across PODs and across locations
• Distance limited by application latency budget and storage replication
• Two types of traffic specific to the cluster:
Non-routable heartbeats: FabricPath (Intra-DC) & OTV (Inter-DC) provide LAN connectivity
Front-end IP connectivity: LISP provides mobility for cluster virtual-IP failover
POD POD POD POD
IP Network
OS OS OS
App Cluster Distributed App (GeoCluster)
Fabric Path
(Intra-DC)
OTV
(Inter-DC)
LISP
IP mobility
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 83
Intra-DC Inter-DC
Virtual Machines VXLAN (x-L3), FabricPath (L2) OTV (x-L3)
Physical Machines FabricPath (L2), VXLAN GWY (future) OTV (x-L3)
DC-east DC-west
• VXLAN & FP provide elasticity within the DC within a L2 POD and across PODs
• OTV extends the LAN across DC sites without compromising network stability
• LISP integrates with SLBs and balances traffic across the SLBs (Future)
POD POD POD POD
IP Network
Fabric Path
(Intra-DC L2)
OTV
(Inter-DC x-L3)
App
OS
App
OS
App
OS
App
OS
VXLAN
(Intra-DC x-L3)
LISP
IP mobility
SLB
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 84
DC-east DC-west
• Reduce Disaster Recovery Bring-up times - Less Network Changes/Operations = Faster recovery times
• Preserve IP addressing with LISP host mobility
No reconfiguration of applications or network service policies
No routing re-convergence
Automatic routing re-localization (upon application bring-up at DR)
• VXLAN segments move along with the applications (vApps)
POD POD POD POD
IP Network
App
OS
App
OS
App
OS
VXLAN
(Intra-DC x-L3)
LISP
IP mobility
App
OS
App
OS
App
OS
VXLAN
(Intra-DC x-L3)
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 85
DC-east DC-west
• Move virtual Applications (vApps) to private cloud PODs
Move VMs and virtual Segments (VXLANs)
• LISP host mobility allows the vApp GWY to roam
Maintain GWY IP address and optimal reachability
• VXLAN segments move along with the applications (vApps)
Very large scale of virtual segments can move and extend across L3 boundaries
POD POD POD POD
IP Network
LISP
IP mobility
vxlan 1
V
M
V
M
vxlan 2
vxlan 3
GWY
V
M
web
app
db
VSG
IP1
vApp = Collection of
VMs and segments
with a GWY
vxlan 1
VM
VM
vxlan 2
vxlan 3
GWY
VM
web
app
db
vxlan 1
VM
VM
vxlan 2
vxlan 3
GWY
VM
web
app
db
vxlan 1
VM
VM
vxlan 2
vxlan 3
GWY
VM
web
app
db
vxlan 1
VM
VM
vxlan 2
vxlan 3
GWY
VM
web
app
db
vxlan 1
VM
VM
vxlan 2
vxlan 3
GWY
VM
web
app
db
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 86
Scale
Requirement Intra-DC Inter-DC
Layer 2 connectivity FabricPath/TRILL/VXLAN OTV/VPLS
IP Mobility LISP LISP
Secure Segmentation VXLAN / Segment-ID VPNs (LISP/MPLS)
DC-east DC-west
POD POD POD POD
IP Network
Fabric Path
(Intra-DC L2)
OTV/VPLS
(Inter-DC x-L3)
App
OS
App
OS
App
OS
App
OS
VXLAN/OTV
(Intra-DC x-L3)
LISP
IP mobility
Fabric Path
(Intra-DC L2)
App
OS
App
OS
VXLAN/OTV
(Intra-DC x-L3)
Complimentary CapabilitiesFabricPath, VXLAN, LISP