Fibre Channel over Ethernet and Nexus 5000 © 2008 Cisco Systems, Inc. All rights reserved. 1 [email protected] Višnja Milovanović [email protected] Systems Engineer
Fibre Channel over Ethernet and Nexus 5000
© 2008 Cisco Systems, Inc. All rights reserved. [email protected]
Višnja Milovanović
Systems Engineer
FCoE
© 2008 Cisco Systems, Inc. All rights reserved. 2
10GbE Drivers in the Datacenter
Multi-Core CPU architectures allowing bigger and multiple workloads on the same machine
Server virtualization driving the need for more bandwidth per
© 2008 Cisco Systems, Inc. All rights reserved. 3
Server virtualization driving the need for more bandwidth per server due to server consolidation
Growing need for network storage driving the demand for higher network bandwidth to the server
Multi-Core CPUs and Server Virtualization driving the demand for higher bandwidth network connections
Merging the requirements
� LAN/IP
Must be Ethernet
Losing some frames is good for TCP
� Storage
Must follow the Fibre Channel model
© 2008 Cisco Systems, Inc. All rights reserved. 4
Must follow the Fibre Channel model
Losing frames is not an option
� IPC
Don’t care of the underlying network, provided that
It is cheap
It is low latency
It supports APIs like OFED, RDS, MPI, etc. . .
Can Ethernet be lossless?
� Yes, with 802.3x Pause
For short distances equivalent to FC credits
� But… when customers turn it on, the results are confusing
Standard allows for asymmetric implementations
© 2008 Cisco Systems, Inc. All rights reserved. 5
Standard allows for asymmetric implementations
Vendor implementations are inconsistent
� Anyhow Pause is not enough
It applies to the whole link
� Cisco proposes Priority Flow Control
One pause per IEEE 802.1p priority code point
Public domain (no standard activity yet)
Embraced by many network gear vendors
What is Fibre Channel over Ethernet?
� From a Fibre Channel standpoint it’s
FC connectivity over a new type of cable called… an Ethernet cloud
� From an Ethernet standpoints it’s
Yet another ULP (Upper Layer Protocol) to be transported, but… a challenging one!
© 2008 Cisco Systems, Inc. All rights reserved. 6
but… a challenging one!
� And technically…
FCoE is an extension of Fibre Channelonto a Lossless Ethernet fabric
FCoE Enablers
� 10Gbps Ethernet
� Lossless Ethernet
Matches the lossless behavior guaranteed in FC by B2B credits
� Ethernet jumbo frames
© 2008 Cisco Systems, Inc. All rights reserved. 7
Eth
ern
et
He
ad
er
FC
oE
He
ad
er
FC
He
ad
er
FC Payload
CR
C
EO
F
FC
S
Same as a physical FC frame
Control information: version, ordered sets (SOF, EOF)
Normal ethernet frame, ethertype = FCoE
Max FC frame payload = 2112 bytes
Feature / Standard Benefit
Priority Flow Control (PFC)IEEE 802.1Qbb
Enable multiple traffic types to share a common Ethernet link without interfering with each other
Bandwidth Management IEEE 802.1Qaz
Enable consistent management of QoS at the network level by providing consistent scheduling
Evolving Ethernet
© 2008 Cisco Systems, Inc. All rights reserved. 8
Congestion Management IEEE 802.1Qau
End-to-end congestion management for L2 network (future)
Data Center Bridging Exchange Protocol (DCBX)
Management protocol for enhanced Ethernet capabilities
L2 Multipath forUnicast and Multicast
Increase bandwidth, multiple active paths. No spanning tree (future)
Enabling Differentiated Services in an Ethernet Fabric
Data Center Bridging Capability Exchange Protocol (DCBCXP)
Devices need to discover the edge of the enhanced Ethernet cloud:
Each edge switch needs to learn that it is connected to a legacy switch.
Servers need to learn whether or not they are connected to Enhanced Ethernet device.
DCBX discovery = DCE
CNA
CNA
DCBX discovery =
Classical Ethernet
DCBX discovery =
Classical EthernetStandard 10 GigE NIC
C6500
Fibre Channel= No
© 2008 Cisco Systems, Inc. All rights reserved. 9
Within the Enhanced Ethernet cloud, devices need to discover the capabilities of its peers.
DCBX utilizes the link-layer discovery protocol (LLDP) and handles local operational configuration for each feature
Fibre Channel= No
DCBX used
DCBX Overview
Auto-negotiation of capability and configuration
� Priority Flow Control capability and all associated CoS values enabled with PFC
� FCoE capability
� Allows one link peer to push config to other link peer
FCoE CoS value
� Logical Link Up/down signaling of Ethernet and FC
© 2008 Cisco Systems, Inc. All rights reserved. 10
� DCBX negotiation failures will result in vfc not coming up
� Per-priority-pause not enabled on CoS values with PFC config
� Link partners can choose supported features and willingness to accept configuration from peer.
� Is supported on in point to point configuration
http://download.intel.com/technology/eedc/dcb_cep_spec.pdf
http://www.ieee802.org/1/files/public/docs2008/
Priority Flow Control
Priority based Flow ControlPriority based Flow Control
© 2008 Cisco Systems, Inc. All rights reserved. 11
• Enables lossless behavior for each class of service
• PAUSE sent per priority when buffers limit exceeded
Priority-based Flow Control (PFC)
� PAUSE functionality per Ethernet priority
IEEE 802.1Q defines 8 priorities
Traffic classes are mapped to different priorities:
no traffic interference
Storage traffic may be paused while IP traffic is being forwarded
Or, vice versa
© 2008 Cisco Systems, Inc. All rights reserved. 12
Requires independent resources per priority (buffers)
� High level of industry support
Cisco distributed proposal
Standard Track in IEEE 802.1Qbb
Ethertype = IEEE 802.1Q Priority CFI VLAN ID
IEEE 802.1Q tag
16 3 1 12 bits
Priority based bandwidth management
Priority based Bandwidth Management
Priority based Bandwidth Management
© 2008 Cisco Systems, Inc. All rights reserved. 13
• Enables Intelligent sharing of bandwidth between traffic classes control of bandwidth
• 802.1Qaz Enhanced Transmission
Distinct Managementand Control Planes
Single Data Plan
MDSMDS
SAN BSAN AvPC
LAN
NexusNexus
L2MP: Port Channel with vPC
© 2008 Cisco Systems, Inc. All rights reserved. 14
10 GbE/FCoE/DataCenter Ethernet
Consolidated I/OServers to Access Layer
Data Center Ethernet / FCoE Links
Cisco Nexus 5000
10 GbE
Fibre Channel
Virtual Port-channels Preserves ExistingDual SANEnvironments
Distinct Managementand Control Planes
Single Data Plan
MDSMDS
SAN BSAN A
Preserves ExistingDual SAN
VPC
Virtual Port-channels
LAN
NexusNexus
L2MP: vPC at the access layer
© 2008 Cisco Systems, Inc. All rights reserved. 15
Consolidated I/OServers to Access Layer
Data Centre Ethernet / FCoE Links
Cisco Nexus 5000
Dual SANEnvironments
10 GbE/FCoE/DataCenter Ethernet
10 GbE
Fibre Channel
Virtual Port-channels
Active/Active
VPC
L2MP: Example of a Fat Tree
� All links are forwarding
� IS-IS computes the forwarding tables
© 2008 Cisco Systems, Inc. All rights reserved. 16
L2MPBackbone
Data Center Access ArchitectureVirtualized Access Switch
LAN
10GE - DCE
SAN A SAN B� Virtualized Access Layer
Architectural flexibility
1GE to 10GE transition
� Consolidated and Consistent Operations
� Virtualization Optimized
© 2008 Cisco Systems, Inc. All rights reserved. 17
1GE
10GE - DCE
Nexus 1000v
� Virtualization Optimized
VN-Link
� Network Fabric
vPC
Unified Fabric (FCoE)
Lossless Ethernet (DCE)
L2 Multipathing (DCE)Virtualized Access Switch
Unified I/O or I/O Consolidation
© 2008 Cisco Systems, Inc. All rights reserved. [email protected]
I/O Consolidation
FC HBA
FC HBA
NIC
FC Traffic
FC Traffic
Enet Traffic
FC HBA
FC HBA
NIC
FC HBA
FC HBA
NIC
Today:� Parallel LAN/SAN Infrastructure
� Inefficient use of Network Infrastructure
� 5+ connections per server – higher adapter and cabling costs
Unified I/O Use Case
© 2008 Cisco Systems, Inc. All rights reserved. 19
NIC
NIC
Enet Traffic
Enet Traffic
NIC
NIC
NIC
NIC
Adds downstream port costs; cap-ex and op-ex
Each connection adds additional points of failure in the fabric
Power and cooling
� Longer lead time for server provisioning
� Multiple fault domains – complex diagnostics
� Management complexity – firmware, driver-patching, versioning
Management
SAN BSAN ALAN
Today:
� Aggregation/Core switches
Unified I/O Use Case
FC HBA
FC HBA
NIC
© 2008 Cisco Systems, Inc. All rights reserved. 20
Ethernet
FC
� Access – Top of the Rack switches
� Servers
NIC
NIC
FCoE I/O Consolidation BenefitFewer HBA/NIC’s per Server
FCoE CNA
FC HBA
FC HBA
NIC
FC Traffic
FC Traffic
Enet Traffic
FCoE
&
© 2008 Cisco Systems, Inc. All rights reserved. 21
Customers purchase fewer NIC’s and HBA’s
FCoE CNANIC
NIC
Enet Traffic
Enet Traffic
Enet@
10GE
FCoE Converged Network Adaptor
Management
SAN BSAN ALAN
Today
Unified I/O Use Case
Unified I/O Phase 1Unified I/O Phase 1� Reduction of server adapters
� Fewer Cables
� Simplification of access layer & cabling
� Gateway free implementation - fits in installed base of existing LAN and SAN
� L2 Multipathing Access – Distribution
© 2008 Cisco Systems, Inc. All rights reserved. 22
FCoE
Ethernet
FC
FCoE Switch
� Lower TCO
� Investment Protection (LANs and SANs)
� Consistent Operational Model
� One set of ToR Switches
DCE Fabric w/ FCoE
Unified I/O Phase 2
� Elimination of parallel network infrastructure
� L2/L3 Multipathing end to end
� Faster infrastructure provisioning
Unified I/O Use Case
Management
StorageArrays
Unified I/O Phase 2
© 2008 Cisco Systems, Inc. All rights reserved. 23
provisioning
� Lower TCO
� Disk array access via DCE or Native FCFCoE
Switch
DCE and FCoE
FC
Unified I/O Phase 3
� Datacenter wide Unified Fabric for LAN and SAN
� L2/L3 Multipathing end to end
� Consistent network policies across datacenter
Management
SAN BSAN ALAN
Today:
Management
SAN BSAN ALAN
Unified I/O Phase 1 (Mid 2008)
Unified I/O Use Case
Management
DCE Fabric w/ FCoE
Unified I/O Phase 3
StorageArrays
© 2008 Cisco Systems, Inc. All rights reserved. 24
across datacenter
� Lower TCO
DCE and FCoE
Ethernet
FC
DCE and FCoE
Ethernet
FC
DCE and FCoE
Ethernet cabling
© 2008 Cisco Systems, Inc. All rights reserved. [email protected]
Evolution of Ethernet Physical MediaRole of Transport in Enabling these Technologies!
100Mb 1Gb 10Gb
UTP Cat 5 UTP Cat 5
SFP Fiber
10Mb
UTP Cat 3
Mid 1980’s Mid 1990’s Early 2000’s Late 2000’s
X2
SFP+ Cu (BER better than 10 )
SFP+ Fiber
-18
© 2008 Cisco Systems, Inc. All rights reserved. 26
CableTransceiver
Latency (link)Power
(each side)DistanceTechnology
Twinax ~0.1µµµµs~0.1W10mSFP+ CUCopper
MM 62.5µµµµmMM 50µµµµm
~01W82m300m
SFP+ SRshort reach
MM OM2MM OM3
~01W10m100m
SFP+ USRultra short reach
Cat6Cat6a/7Cat6a/7
2.5µµµµs2.5µµµµs1.5µµµµs
~8W~8W~4W
55m100m30m
10GBASE-T
Cat 6/6a/7
iSCSI SRPFCP FCP FCP FCP
SCSI Layer
Operating System / Applications
Encapsulation technologies
© 2008 Cisco Systems, Inc. All rights reserved. 27
Ethernet E. Eth
IP
TCP
iSCSI
IB
SRP
IP
TCP
FCIP
FCP
IP
TCP
iFCP
FCP
FCoE
FCP
FC
FCP
1, 2, 4, (8), 10 Gbps 1, 10 . . . Gbps 10, 20 Gbps
FCP
SCSI Layer
OS / Applications
Encapsulation technologies
� FCP layer is untouched
� Allows same management tools for Fibre Channel
� Allows same Fibre Channel drivers
� Allows same Multipathing software
© 2008 Cisco Systems, Inc. All rights reserved. 28
E. Ethernet
FCoE
FCP
1, 10 . . . Gbps
� Allows same Multipathing software
� Simplifies certifications with OSMs
� Evolution rather than Revolution
FCoE and iSCSI
� Extension of FC (10G)
� Same Lossless and Performance
� More Oriented towards 1GE
� Applicable for SMB
� New Management
iSCSIiSCSIFCoEFCoE
© 2008 Cisco Systems, Inc. All rights reserved. 29
and Performance Characteristics as FC
� Targeted for Enterprise
� Same Management Model as FC
� Minimal Operational Impact
� New Management Model
� Limited Native Target Support
� Requires Stateful Gateway
FCoE AdvantagesFCoE is managed like FC at initiator, target, and switch level
Same Operational Model
Same Operational Model
Easy to Understand
Easy to Understand
Completely based on the
FC model
Same host-to-switch and switch-to-
switch behavior of FCAligned with the Aligned with the
© 2008 Cisco Systems, Inc. All rights reserved. 30
Same Techniques ofTraffic ManagementSame Techniques ofTraffic Management
Same Managementand Security ModelsSame Managementand Security Models
switch behavior of FC
E.g., in order delivery or
FSPF load balancing
WWNs, FC-IDs, hard/soft
zoning, DNS, RSCN
Aligned with the FC-BB-4 Model,
Standardizedin FC-BB-5
Aligned with the FC-BB-4 Model,
Standardizedin FC-BB-5
FCoE architecture
© 2008 Cisco Systems, Inc. All rights reserved. [email protected]
Terminology
� ENode (FCoE Node): a Fibre Channel HBA and a FCM implemented within an Ethernet NIC
� FCM (FCoE Mapper): the function performing the encapsulation and de-capsulation of FC frames in Ethernet frames
© 2008 Cisco Systems, Inc. All rights reserved. 32
Ethernet frames
� FCF (FCoE Forwarder): a Fibre Channel switch implementing one or more FCMs to attach to an ethernet network
ENode Architecture
FC Node
FCM FCM…
© 2008 Cisco Systems, Inc. All rights reserved. 33
� One MAC address per FCoE Mapper
Enet
port
Enet
port…
Fibre Channel Forwarder Architecture
FCport
FCport
FCport
FC
FC Switch
FCM
© 2008 Cisco Systems, Inc. All rights reserved. 34
Eth
port
Eth
port
Eth
port
Eth
port
Eth
port
Eth
port
Eth
port
Eth
port
Ethernet BridgeFCport
� One Universal MAC address per FCMapper
FCoE MAC addresses
� VE_Ports and VF_Ports always use MAC addresses derived from the switch pool
� VN_Ports may use two types of MAC addresses:
SPMA (Server Provided MAC Addresses)
FPMA (Fabric Provided MAC Addresses)
© 2008 Cisco Systems, Inc. All rights reserved. 35
FPMA (Fabric Provided MAC Addresses)
� MAC Addresses are negotiated in FIP
� Initial deployment will use FPMA only
Initial Login Flow ladder
ENode FCoE Switch
VLANDiscovery
FIP:FCoEInitialization
VLANDiscovery
FCFDiscovery
FCFDiscovery
© 2008 Cisco Systems, Inc. All rights reserved. 36
FLOGI/FDISC FLOGI/FDISC Accept
FC CommandFC Commandresponses
Initialization Protocol
FCOEProtocol
Discovery Discovery
Some terminology
� Port types
VF_port / VN_port
VE_port
F_port trunking / channeling
� Device types
© 2008 Cisco Systems, Inc. All rights reserved. 37
Fibre Channel Forwarder (FCF)
FIP snooping bridge
� Switching modes
FC switching
NPV (NPIV gateway)
FIP snooping
FCoE Forwarding (VE_ports)
� FCoE frames have:
MAC addresses (hop-by-hop)
FC addresses (end-to-end)
© 2008 Cisco Systems, Inc. All rights reserved. 38
Ethernet linkFC Fabric
FC Domain 7 FC Domain 3MAC AFCID 7.1.1 FCID 1.1.1
MAC C
D_ID = FC-ID (1.1.1)S_ID = FC-ID (7.1.1)
FC Frame
D_ID = FC-ID (1.1.1)S_ID = FC-ID (7.1.1)
FC Frame
Ethernet link
FC Storage
FCoE Frame
D_ID = FC-ID (1.1.1)S_ID = FC-ID (7.1.1)
Dest. = MAC BSrce. = MAC A
D_ID = FC-ID (1.1.1)S_ID = FC-ID (7.1.1)
Dest. = MAC CSrce. = MAC B
FC link
FC Domain 1MAC B
VE_port VE_port VF_port VN_port
FCoE Forwarding (FIP snooping)
Ethernet linkFC Fabric
Ethernet link
FC Storage
FC link
VF_port VN_port
© 2008 Cisco Systems, Inc. All rights reserved. 39
FC Fabric
FC Domain 7 FC Domain 1MAC AFCID 7.1.1 FCID 1.1.1
MAC C
D_ID = FC-ID (1.1.1)S_ID = FC-ID (7.1.1)
FC Frame
D_ID = FC-ID (1.1.1)S_ID = FC-ID (7.1.1)
FC Frame
FCoE Frame
D_ID = FC-ID (1.1.1)S_ID = FC-ID (7.1.1)
Dest. = MAC CSrce. = MAC A
D_ID = FC-ID (1.1.1)S_ID = FC-ID (7.1.1)
Dest. = MAC CSrce. = MAC A
FC link
No domainMAC B
Nexus Switching
© 2008 Cisco Systems, Inc. All rights reserved. [email protected]
Data Center Access EvolutionVirtual Access
© 2008 Cisco Systems, Inc. All rights reserved. 41
DC Virtual
Access
Nexus 7010 10-Slot Chassis
� First chassis in Nexus 7000 product family
� Optimized for data center environments
� High density
256 10G interfaces per system
� High performance
1.2Tbps system bandwidth at initial release
© 2008 Cisco Systems, Inc. All rights reserved. 42
1.2Tbps system bandwidth at initial release
80Gbps per slot
60Mpps per slot
� Future proof
Initial fabric provides up to 4.1Tbps
Product family scaleable to 15+Tbps
40/100G and Unified Fabric ready
33.1-38”(84-96.5cm)
17.3” (43.9cm)
21 RU36.5”
(92.7cm)
Nexus 5000 FamilyMulti-Protocol Server Switch
Industry’s First I/O Consolidation Virtualization Fabric for Enterprise Data Center
Industry’s First I/O Consolidation Virtualization Fabric for Enterprise Data Center
SwitchFamily
56-Port L2 Switch
• 40 Ports 10GE fixed
• 2 Expansion Modules
28-Port L2 Switch
• 20 Ports 10GE fixed
• 1 Expansion Module
NX5010 NX5020
© 2008 Cisco Systems, Inc. All rights reserved. 43
OS
Cisco Fabric Manager and Cisco Data Center Network Manager
Cisco NX-OS
FC + Ethernet
• 4 Ports 10GE
• 4 Ports 1/2/4G FC
CNA
• 2 Port 10GE/FCoE
Mgmt
ExpansionModules Ethernet
• 6 Ports 10GE
Partners
Fibre Channel
• 8 Ports 1/2/4G FC
Server Adapter
• FCoE SW stack
All 10GE switch/module ports are FCoE/Data Center Ethernet capableAll 10GE switch/module ports are FCoE/Data Center Ethernet capable
Front and Rear Panels
All 10GE Ports Are FCoE Capable!Nx5020Nx5020
N+1 Redundant FansDual Redundant
© 2008 Cisco Systems, Inc. All rights reserved. 44
Expansion
Modules
Cables Connect in the Rear for Ease of Server WiringCables Connect in the Rear for Ease of Server Wiring
Power
EntryBase 10GE
10/100/1000
Out of Band ManagementConsole
N+1 Redundant FansDual Redundant
Power Supplies
Nexus 2000 Fabric ExtenderVirtual Chassis
Nexus 5000Virtualized chassis
+
Nexus 5000
=
© 2008 Cisco Systems, Inc. All rights reserved. 45
The Nexus 2000 Fabric Extender (FEX) acts as a remote linecardfor the Nexus 5000, retaining all centralized management and configurationon the Nexus 5000, transforming it into a Virtualized Chassis
+Nexus 2000 Fabric Extender
=
Data Center Access ArchitectureN5K/N2K Advantages – Flexible Cabling
� Cisco Nexus Fabric Extender (FEX) and Nexus 5000 provide a Flexible Access Solution
� Migration to ToR for 10GE servers or selective 1GE server racks if required (mix of ToR and EoR)
� Mixed cabling environment (optimized as required)
� Flexible support for Future Requirements
© 2008 Cisco Systems, Inc. All rights reserved. 46
Combination of EoR and ToR cabling
Nexus 5000/2000 Mixed ToR & EoR
. . .
Cisco Nexus 1000V Architecture
VMW ESX
Server 1
VMware vSwitch
VMW ESX
Server 2
VMware vSwitch
VMW ESX
Server 3
VMware vSwitch
VM #1
VM #4
VM #3
VM #2
VM #5
VM #8
VM #7
VM #6
VM #9
VM #12
VM #11
VM #10
VEM VEM VEMNexus 1000V
© 2008 Cisco Systems, Inc. All rights reserved. 47
Virtual Supervisor Module (VSM)
� Virtual or Physical appliance running Cisco OS (supports HA)
� Performs management, monitoring, & configuration
� Tight integration with VMware Virtual Center
Virtual Ethernet Module (VEM)� Enables advanced networking
capability on the hypervisor
� Hypervisor dependent code
� Provides each VM with dedicated “switch port”
� Collection of VEMs = 1 Distributed Switch
Virtual Center
VMW ESX VMW ESX VMW ESX
Nexus 1000V
VSM
Cisco VN-LinkIncrease Operational Efficiency
ServerServer
VM #5
VM #8
VM #7
VM #6
VM #4
VM #3
VM #2
VM #1
Policy-Based VM Connectivity
Non-DisruptiveOperational Model
Mobility of Network & Security Properties
Cisco VN-Link—Virtual Network Link
© 2008 Cisco Systems, Inc. All rights reserved. 48
VMW ESXVMW ESX
Cisco VN-Link
#5 #8#7#6#4#3#2#1
Network Benefits
� Unifies network mgmt and ops
� Improves operational security
� Enhances VM network features
� Ensures policy persistence
� Enables VM-level visibility
Virtual Center
Server Benefits
� Maintains existing VM mgmt
� Reduces deployment time
� Improves scalability
� Reduces operational workload
� Enables VM-level visibility
Thank you!
© 2008 Cisco Systems, Inc. All rights reserved. [email protected]
© 2008 Cisco Systems, Inc. All rights reserved. 50505050© 2003, Cisco Systems, Inc. All rights reserved.Presentation_ID