8/9/2019 FCoE - Fibre Channel Over Ethernet
1/65
Fibre Channel over Ethernet (FCoE)
John L Hufferd, Brocade
8/9/2019 FCoE - Fibre Channel Over Ethernet
2/65
Fibre Channel over Ethernet (FCoE)© 2008 Storage Networking Industry Association. All Rights Reserved. 2
SNIA Legal Notice
The material contained in this tutorial iscopyrighted by the SNIA.Member companies and individuals may use thismaterial in presentations and literature under thefollowing conditions:
Any slide or slides used must be reproduced withoutmodificationThe SNIA must be acknowledged as source of any
material used in the body of any document containingmaterial from these presentations.
This presentation is a project of the SNIA
Education Committee.
8/9/2019 FCoE - Fibre Channel Over Ethernet
3/65
8/9/2019 FCoE - Fibre Channel Over Ethernet
4/65
Fibre Channel over Ethernet (FCoE)© 2008 Storage Networking Industry Association. All Rights Reserved. 4
Agenda
Introduction
Goals & Requirements
Consolidation
Architecture
Discovery & Link Instantiation
Topologies
Scenarios
Summary
8/9/2019 FCoE - Fibre Channel Over Ethernet
5/65
Fibre Channel over Ethernet (FCoE)© 2008 Storage Networking Industry Association. All Rights Reserved. 5
Introduction
This presentation provides an overview of FibreChannel over Ethernet (FCoE)
One should think about FCoE as placing the FCprotocol on a new physical link
Ethernet links instead of physical FC linksBut it is still Fibre Channel
The protocol is being defined in the INCITS FibreChannel (T11) technical committee
Many details of the protocol still need to be definedbut the significant major issues have been resolved
Target T11 Standards completion is the 2H08
8/9/2019 FCoE - Fibre Channel Over Ethernet
6/65
Fibre Channel over Ethernet (FCoE)© 2008 Storage Networking Industry Association. All Rights Reserved. 6
The Origins of Data Center Ethernet
The technology has evolved continuously, showing a great ability toadapt to new technologies and increasing business requirements
Increasing Scalabil ity, Feature, Function
I n c r e a s
i n g
P e r f o r m a n c e
Integration of Carrier-grade features
Incremental Protocol Enhancements
Logical Partitioning
Evolution from shared media to dedicated media
1973
2008-2009 CEE
Introduction of Ethernet
10Mbps
100Mbps
1Gbps
10Gbps(Converged Enhanced Ethernet)
8/9/2019 FCoE - Fibre Channel Over Ethernet
7/65
Fibre Channel over Ethernet (FCoE)© 2008 Storage Networking Industry Association. All Rights Reserved. 7
Goals/Requirements
8/9/2019 FCoE - Fibre Channel Over Ethernet
8/65
Fibre Channel over Ethernet (FCoE)© 2008 Storage Networking Industry Association. All Rights Reserved. 8
FCoE Goals/Requirements (part 1)
FCoE Fabrics must be built with FCoE SwitchesSwitches with Ethernet ports that provide FCoE capabilities andservices – Called, in the standard, an FC Forwarder FCFSwitches that include the functions of traditional FC switches
Standard Ethernet switches may also exist in the fabric but switcheswith FCoE capabilities are required
FCoE fabrics must operate seamlessly with real FC Fabrics
FC services must operate identically on FCoE fabrics andFibre Channel fabrics
FCoE must support all Fibre Channel advanced features(e.g. virtual fabrics, IFR, security, etc.) transparently
FCoE is NOT a replacement for FCIP or iFCPFCIP & iFCP use TCP/IPFCIP/iFCP is for inter-switch links beyond the Data Center
8/9/2019 FCoE - Fibre Channel Over Ethernet
9/65
Fibre Channel over Ethernet (FCoE)© 2008 Storage Networking Industry Association. All Rights Reserved. 9
Integrated Ethernet & FCoE switch (FCF)with FC connectionsSome implementations may combine the features and capabilities of an EthernetSwitch with the features and capabilities of a FC switch
• Support Ethernet and IP standards for switching, pathing and routing
• Support FC standards for switching, pathing and routing
• Support current and enhanced Ethernet Standards
• Adapt between FCoE and FC
An FCoE Port (N_Port, F_Port or E_Port)
• Has the same function as in FC• But is layered on top of Ethernet
• Called VN_Port, VF_Port and VE_Port
Because many logical (virtual)Ports can share one physical port
Note:FCF Pathing and Forwarding utilizes the FSPF (Fabric Shortest Path First) protocol
Non-FCoE Ethernet traffic is relayed using conventional 802.1 defined mechanisms such as
STP (Spanning Tree Protocol) and MSTP (Multiple STP)See additional FSPF and STP explanation in Appendix
FCFCFCCEE Ethernet Ports(with IP & FCoE VF_Port &VE_Port capabilities)
8/9/2019 FCoE - Fibre Channel Over Ethernet
10/65
Fibre Channel over Ethernet (FCoE)© 2008 Storage Networking Industry Association. All Rights Reserved. 10
Connections to a Combo FCoE Switch
Applications
TCPUDPIP
SCSI
Fibre Channel
FCoE
Lossless Ethernet MAC (CEE)
• Fibre Channel is carried over lossless Ethernet as a L3 protocol
Combo Lossless Ethernet (CEE) Switch
with FCoE Switch (FCF) capabilities
(FCoE VN_Port)
Ethernet port with IP & FCoE VF_Portcapabilities
IP address 123.45.67.89
iSCSI
SCSI
8/9/2019 FCoE - Fibre Channel Over Ethernet
11/65
Fibre Channel over Ethernet (FCoE)© 2008 Storage Networking Industry Association. All Rights Reserved. 11
FCoE Goals/Requirements (part 2)
FCoE requires specific Ethernet extensions to be implementedLossless switches and fabrics (e.g., supporting IEEE 802.3 PAUSE)configurations are requiredJumboframe support is required (not a standard, but widely available)
Deployments of FCoE should utilize the advances in Ethernetcurrently being discussed in IEEE 802.1, specifically:
Priority-based Flow Control (PFC)
Enhanced Transmission Selection (ETS)DCB Capability eXchange Protocol (DCB CXP)
These 802.1 advances are important for Consolidated
Flows (Messaging, Clustering and Storage)This set of functions has been called Data Center Ethernet, or CEE –Converged Enhanced Ethernet (intended for a Data CenterEnvironment)
FCoE should require no changes to FC software
8/9/2019 FCoE - Fibre Channel Over Ethernet
12/65
Fibre Channel over Ethernet (FCoE)© 2008 Storage Networking Industry Association. All Rights Reserved. 12
Consolidation
8/9/2019 FCoE - Fibre Channel Over Ethernet
13/65
Fibre Channel over Ethernet (FCoE)© 2008 Storage Networking Industry Association. All Rights Reserved. 13
Dramatic reduction in adapter, switch ports and cabling4-6 cables to 2 cables per server
Seamless connection to the installed based of existing SANs
and LANsRequires high performance lightweight frame mappers vs.heavy weight gateways
FCoE has no need to terminate and re-initiate a SCSI connection
(e.g. iSCSI to FC)Effective sharing of high bandwidth links
High End 10GE Server & NIC/HBAConsolidation
Today
With CEEOS3DB Server OS2 App Ser ver OS1Web Server
Messaging
MPIRDMA
FCHBA
OS3DB Server
OS2 App Server
OS1Web Server
IB/EthernetCluster
Hyper Visor (vmWare, Xen, etc)
E-HBA(CEE)
Data Center CEE Network
EthernetTCP/IP
FC SAN
Data Center CEE Network
E-HBA(CEE)E-HBA(CEE)
•NIC
•TCP acceleration .•MPI , RDMA
over Ethernet•FCoE
OS3DB Server
OS2 App Ser ver
OS1Web Server
8/9/2019 FCoE - Fibre Channel Over Ethernet
14/65
Fibre Channel over Ethernet (FCoE)© 2008 Storage Networking Industry Association. All Rights Reserved. 14
Architecture
8/9/2019 FCoE - Fibre Channel Over Ethernet
15/65
Fibre Channel over Ethernet (FCoE)© 2008 Storage Networking Industry Association. All Rights Reserved. 15
FC Encapsulation Into Network Packets(2 FCoE Related Packet types)
IEEE 802.3Layers
FC Levels(Unchanged)
FCoE Mapping
FC-4FC-3
FC-2
FC-1
FC-0
FC-4FC-3
FC-2
MAC
PHY
Frame Check Sequence(CRC)
Protocol contr ol in formation: Version, SOF, EOF, etc.FC Imbedded Frames: Same as in Physical FC
Ethernet
Header FCSFCoE
Header FC Header SCSI Commands/Data
Ethertype“ FCoE”
Fibre Channel over Ethernet (Fibre Channel over Ethernet ( FCoEFCoE ) Packets) Packets
Protocol contro l information: Version, Op-codes, etc.Discovery and Logi n/Logout Parameters
EthernetHeader FCS
FIPHeader
Descriptors
Ethertype“ FIP”
FCoE Initialization Protocol (FCoE Initialization Protocol ( FIPFIP ) Packets) Packets
Ethernet Header provides things needed fo r thephysical network, including “ Ethertype ”
8/9/2019 FCoE - Fibre Channel Over Ethernet
16/65
Fibre Channel over Ethernet (FCoE)© 2008 Storage Networking Industry Association. All Rights Reserved. 16
FIP Protocol and FCoE Protocol
Discovery PhaseFCFs Discovery each other and form a FabricENodes and FCFs Discover
Potential VN_Port VF_Port pairing
Capabilities of Potential pairingLogin Phase
ENodes chose among discovered FCFs’ PortsCreates association between ENode Ports and FCF Ports
VN_Port VF_Port Logical FC LinkTwo allowed alternatives for the ENode MAC Addressing
Fabric Provided MAC Addresses (FPMA)Server Provided MAC Addresses (SPMA)Chosen by FCF (FPMA & SPMA Described latter)
Uses: FLOGI, FLOGI ACC, LOGO, …
Data Transfer PhasePLOGI/PRLI
All other FC protocol frames (ELS, FC4 ULPs. etc.)
FCoEInitializationProtocol ( FIP )
FCoE Protocol
8/9/2019 FCoE - Fibre Channel Over Ethernet
17/65
Fibre Channel over Ethernet (FCoE)© 2008 Storage Networking Industry Association. All Rights Reserved. 17
FIP Operation Format
Word 31-24 23-16 15-8 7-00 Destination MAC Address (6 Bytes)
1
2 Source MAC Address (6 Bytes)3 ET=FIP (16 bits) Ver (4b) Reserved (12 bits)
4 FIP Operation Code
5 Reserved FIP subcode Descriptor List Length
6 Flags FPSP
S F
PAD to minimum length or mini-Jumbo length…n
n+1 Ethernet FCS
Optional IEEE 802.1q4 ByteTag goes here
Descriptor listvariesIn size
Ethernet framesize
Is 64Bytes to 2220Bytes
FIP Operation Code Reserved FIP SubCode
Descriptor List Length Flags FP SP S F
Descriptor List
Solicited bit FCF bitCapability Bits (SPMA or FPMA)
See Appendix forDescriptor list items
8/9/2019 FCoE - Fibre Channel Over Ethernet
18/65
Fibre Channel over Ethernet (FCoE)© 2008 Storage Networking Industry Association. All Rights Reserved. 18
FC’s Encapsulation in Ethernet (FCoE)
Word 31-24 23-16 15-8 7-00 Destination MAC Address (6 Bytes)
1
2 Source MAC Address (6 Bytes)3 ET=FCoE (16 bits) Ver (4b) Reserved (12 bits )
4 Reserved
5 Reserved
6 Reserved SOF (8 bits)
7 Encapsulated FC FrameFC Frame = Minimum 28 Bytes (7 Words)
Maximum 2180 Bytes (545 Words)(including FC-CRC)
…
n
n+1 EOF (8 bit s) Reserved
n+2 Ethernet FCS
Optional IEEE 802.1q4 ByteTag goes here
This field varies
In size
Ethernet framesizeIs 64Bytes to 2220Bytes
8/9/2019 FCoE - Fibre Channel Over Ethernet
19/65
Fibre Channel over Ethernet (FCoE)© 2008 Storage Networking Industry Association. All Rights Reserved. 19
FCF Model
LinkEnd
Point(LEP)
8/9/2019 FCoE - Fibre Channel Over Ethernet
20/65
Fibre Channel over Ethernet (FCoE)© 2008 Storage Networking Industry Association. All Rights Reserved. 20
ENode (HBA) Model
Each ENode (HBA) may have multiple Physical Ethernet PortsEach Physical Port may have multiple Logical VN_Ports
LinkEnd
Point(LEP)
8/9/2019 FCoE - Fibre Channel Over Ethernet
21/65
Fibre Channel over Ethernet (FCoE)© 2008 Storage Networking Industry Association. All Rights Reserved. 21
Model of the ENODE with MultipleLogical FC interfaces
FCoEController
Lossless Ethernet MAC Ethernet_Port
FCEntityVN_Port
FCoEEntity
FCoE_LEP
FC-3 /FC-4s
MAC Addressof “ Burnt-in
MAC
FCEntityVN_Port
FCoEEntity
FCoE_LEP
FC-3 /FC-4s
. . .
• For each logical N_Port
(VN_Port) there is one FLOGIand perhaps 100’s of FDISC
• Each VN_Port is seen bythe Host as a separate(logical) FC connection
• The number of (logical) FCconnections is
implementation dependent
• Only one MAC Address is required for the FCoE Controller and the VN_Ports on asingle physical MAC (aka Server Provided MAC Address – SPMA)• FCF may chose to specify new MAC addresses for each VN_Port (aka FabricProvided MAC Address – FPMA)
In thismodel thisis where
FC-2functions
live
MAC Address ofFCoE_LEP(VN_Port)
May or m ay not bethe same as theFCoE cont roller
In this model thisis where the
Encapsulation/De-Encapsulation
functions live
Multiple FC NPIVinstances on a single
logical FC Host interface
M l i l L i l FC i
8/9/2019 FCoE - Fibre Channel Over Ethernet
22/65
Fibre Channel over Ethernet (FCoE)© 2008 Storage Networking Industry Association. All Rights Reserved. 22
The Logical FC Link is defined by a MAC Address pair • A VN_Port MAC Address
• A VF_Port MAC AddressFor a logical FC link the FCoE Frames are always sent to and received from a specific FCF’sMAC Address• Therefore, pathing to and from the FC driver is always defined by the MAC Address of the
partner FCF’s VF_Port
Multiple Logical FC connectionsvia a single Ethernet MAC
Examples of
single MACswithconnectionsto twodifferentFCFs Switch
8/9/2019 FCoE - Fibre Channel Over Ethernet
23/65
Fibre Channel over Ethernet (FCoE)© 2008 Storage Networking Industry Association. All Rights Reserved. 23
Functions of an FCoE Initiator ASIC
NIC Function F C oE
f un c
t i on
FCFunction
Host PCIe interface
ExternalPort
FCoE
ASIC
• Has a Normal NIC interface(A) to the Host
• Has one or more NormalFC interfaces (B,C) to theHost
• FCoE functions not seen bythe Host
• FCoE functions perform theEncapsulation and De-encapsulation
• The FCoE functionInstantiates a Logical FC
N_Port, called a VN_Port
LosslessEthernet
MAC
A B C
8/9/2019 FCoE - Fibre Channel Over Ethernet
24/65
Fibre Channel over Ethernet (FCoE)© 2008 Storage Networking Industry Association. All Rights Reserved. 24
HBA with Multiple Logical FC Interfaces
• MAY have one “burnt-inMAC address for both IPand FCoE/FIP packets
Or
• MAY have different “Burnt-in” MACs for IP andFCoE/FIP packets
• Used to separate HWbased FCoE from otherEthernet Traffic
• Most NICs come withseveral “Burnt-in” MAC
Addresses
• The FCoE controller will perform the FIP functions and will instantiate newVN_Ports as FCoE Link End Point (LEP)• With the same MAC address as the FCoE Controller (SPMA)• Or with a new MAC address specified by the FCF (FPMA)
F C oE
C on
t r ol l er
F C - 3
/ F C -4
s F
C
E n
t i t y
V N
_P
or
t F
C oE
E n
t i t y
F C oE
_L E P
Lossless Ethernet MACEthernet_Port
NIC
Etype=FCoE
or
FIP?
FCoE Chip
MAC Add ress of
“Burnt-inMAC
FCoE Function
F C - 3
/ F C -4
s
F C
E n
t i t y
V N
_P
or t
F C oE
E n
t i t y
F C oE
_L E P
MAC Add ress of
“Burnt-inMAC
A B C
8/9/2019 FCoE - Fibre Channel Over Ethernet
25/65
Fibre Channel over Ethernet (FCoE)© 2008 Storage Networking Industry Association. All Rights Reserved. 25
FIP (FCoE Initiation Protocol)
Discovery and Link Instantiation
8/9/2019 FCoE - Fibre Channel Over Ethernet
26/65
Fibre Channel over Ethernet (FCoE)© 2008 Storage Networking Industry Association. All Rights Reserved. 26
ENode to FCF Logical Links
FCfabric
H1
H2ClassicalEthernet
LAN
FCF B
LosslessEthernetSwitch
FCF A
VN_Ports,VN_Port_Names
FCoE_LEP
VN_Port
FCoE_LEP
VN_Port
FCoE_LEP
VN_Port
FCoE_LEP
VN_Port
FCoE_LEPs
VF_Port
FCoE_LEPs
VF_Port
VF_Port,VF_Port_Name
FCoE_LEPs
VF_Port
FCoE_LEPs
VF_Port
FCF-MAC(A)
FCF-MAC(B)
MAC(H1)
MAC(H2)
FCF-MAC(x): A MAC address of a Lossless Ethernet port of FCF xMAC(y): A MAC address of a Lossless Ethernet port of ENode y
Note: The FCFs mustdiscover each other andcreate a Fabric before theENode Discover FCFs
An ENode must Locate FCFs with a Discovery protocol and then establish a Logical FCLink with an FCF (VN_Port VF_Port) before an Normal FC frame flow
8/9/2019 FCoE - Fibre Channel Over Ethernet
27/65
8/9/2019 FCoE - Fibre Channel Over Ethernet
28/65
Fibre Channel over Ethernet (FCoE)© 2008 Storage Networking Industry Association. All Rights Reserved. 28
Subsequent Login Flow ladder (1 Phase)
End-Node FCF
FLOGI
Login Phase
Normal FCProcessing
FCoE InitializationProtocol
FCOE ProtocolSee Appendix for more details in Discovery Consideration and Ac tions
FC Command(Using the FCFselected MAC
Address as the SA)FC Command responses
(using the command’s SA as a DA)
FLOGI ACC with the FCF’schosen VN_Port MAC addressas a descriptor value
8/9/2019 FCoE - Fibre Channel Over Ethernet
29/65
Fibre Channel over Ethernet (FCoE)© 2008 Storage Networking Industry Association. All Rights Reserved. 29
Topologies
8/9/2019 FCoE - Fibre Channel Over Ethernet
30/65
Fibre Channel over Ethernet (FCoE)© 2008 Storage Networking Industry Association. All Rights Reserved. 30
FCoE Switch & FC Fabric
FCFCFCFCoE Switch
An FCoE Switch may connect to a normal FC switchVia the FC E-Port Ethernet
FC
Note: FCoE serversand storage wil lprobably use an FCoEHBA (or chip set)
8/9/2019 FCoE - Fibre Channel Over Ethernet
31/65
Fibre Channel over Ethernet (FCoE)© 2008 Storage Networking Industry Association. All Rights Reserved. 31
Multiple FCoE Switch (FCF) Topologies
EthernetFC
FCFCFCLossless
EthernetFabric
FCFCFC
Lossless EthernetFabric
FCoE Switc h
A Lossless Eth ernet Fabric can be made up of Combo FCoE SwitchesLossless Ethernet switches config ured into a Lossless Ethernet
Fabric can Front the FCoE Switch
FCFCFC
FCoE
Switch
FCoESwitch
FCoE
SwitchLossless Ethernet
FCoE Switches deployed at t he edges of the Lossl ess Ethernet FabricFCoE Switches connected via VE_Ports and Lossless Ethernet
FCFCFC
FCoESwitch
FCoESwitch
LosslessEthernet1
LosslessEthernet
2
LosslessEthernet
4
FCoESwitch
LosslessEthernet
3
A VE_Port in an FCF may connect to o ther VE_Port in another FCF And an FCF FC E_Port may connect to an FC swi tch E_Port
Status of Current Data Center
8/9/2019 FCoE - Fibre Channel Over Ethernet
32/65
Fibre Channel over Ethernet (FCoE)© 2008 Storage Networking Industry Association. All Rights Reserved. 32
Status of Current Data CenterNetworking
There can be 3 differentnetworks
System Area Network (SyAN)
Used for Clustering/Low LatencyStorage area Network (SAN)
Used to Access to StorageLAN/WAN External Networks
Used for General messaging
Used for Client-Server MessagingUsed for NAS
Often divided into at least 3management domains
Data Center Server (clustering)NetworkData Center Storage NetworkOutfacing (IP) Network
LAN/WANs – Messaging – NAS
Data Center
Remote Offices
System Area Network (SyAN)•Clustering Fabric•InfiniBand
•Myrinet•Ethernet•CTC
Data Center Server & Storage Network
Management Group
Outfacing (IP) NetworkManagement Group
LAN/WAN•Messaging•NAS
Storage Area Network- Fibre Channel
Note: with multiple Data Centers there may also
be interconnects with DWDM, FCIP/iFCP, etc.
8/9/2019 FCoE - Fibre Channel Over Ethernet
33/65
Fibre Channel over Ethernet (FCoE)© 2008 Storage Networking Industry Association. All Rights Reserved. 33
Current FabricsRemoteRemoteOfficesOffices
Clustering Network
There is a FC Storage Network, a Clustering Network & an “ Outfacing” Network
IP Network Management Group
FC
FC
FC
Ficon
Ficon StorageController
Mainframe
Data Center Server & Storage Network Management Group
File Storage Arrays (NAS)
iSCSIStorage
FCNetwork
Ficon
(LAN/WAN)(LAN/WAN)••MessagingMessaging••NASNAS
OutfacingIP Network
Local & RemoteLocal & RemoteBusiness CampusBusiness Campus
Focus: LowLatency & High
Bandwidth
Focus: Protection,Bandwidth/Congestion
Management
FC Link
EthernetLink
FiconLink
Clustering Network
Note: with multiple Data Centers there may also be interconnects with DWDM, FCIP/iFCP, etc.
Vie of the F t re Data Center
8/9/2019 FCoE - Fibre Channel Over Ethernet
34/65
Fibre Channel over Ethernet (FCoE)© 2008 Storage Networking Industry Association. All Rights Reserved. 34
View of the Future Data CenterNetworks
RemoteRemoteOfficesOffices
Data Center Server &Storage Network
Management Group
Outfacing (IP) NetworkManagement Group
Storage & SystemStorage & System Area Network Area Network••Logically Single FabricLogically Single Fabric
(with FCoE)(with FCoE)
Other IP NetsOther IP Nets(LAN/WAN)(LAN/WAN)••MessagingMessaging••NASNAS
TheThe Server/Storage NetworksServer/Storage Networkswill become a Consolidatedwill become a ConsolidatedFabricFabric
Managed by the Data CenterManaged by the Data CenterSystem and StorageSystem and StorageManagement GroupManagement Group
– – Includes Storage andIncludes Storage and
clustering provisioningclustering provisioning – – SAN and SyAN managedSAN and SyAN managedas a single fabricas a single fabric
Focus: High Bandwidth andFocus: High Bandwidth andLow LatencyLow Latency
The Outfacing (The Outfacing ( IP) ManagementIP) ManagementGroupGroup remains the sameremains the same
Focus: Protection, BandwidthFocus: Protection, Bandwidthand Congestion Managementand Congestion ManagementNote: with multiple Data Centers there may alsobe interconnects with DWDM, FCIP/iFCP, etc.
Phase in: FC & High Performance Lossless
8/9/2019 FCoE - Fibre Channel Over Ethernet
35/65
Fibre Channel over Ethernet (FCoE)© 2008 Storage Networking Industry Association. All Rights Reserved. 35
Phase in: FC & High Performance LosslessEthernet Unify into a Data Center Fabric
RemoteRemoteOfficesOffices
FC
FC
FCoE
FC
(LAN/WAN)(LAN/WAN)••MessagingMessaging••NASNAS
OutfacingIP Network
Ficon
iSCSIStorage
Ficon
Ficon StorageController
Mainframe
File Storage Arrays (NAS)
• FCoE permits intermixing of multiple Connection types/protocols• Clustering messaging, General Messaging, and Storage
• The DataCenter Fabric will “ Trunk” to the “ Outfacing” Network (including iSCSI sys)• Some Customers may want keep a mixed environment on-going
DataCenter Fabric
EthernetSW
EthernetSW
FC & EthernetSW Blades
FC Link
EthernetLink
FiconLink
BusinessCampus
with iSCSIconnections
Including iSCSIGateways
Note: with multiple Data Centersthere may also be interconnects
with DWDM, FCIP/iFCP, etc.
8/9/2019 FCoE - Fibre Channel Over Ethernet
36/65
Fibre Channel over Ethernet (FCoE)© 2008 Storage Networking Industry Association. All Rights Reserved. 36
Scenarios
8/9/2019 FCoE - Fibre Channel Over Ethernet
37/65
8/9/2019 FCoE - Fibre Channel Over Ethernet
38/65
Fibre Channel over Ethernet (FCoE)© 2008 Storage Networking Industry Association. All Rights Reserved. 38
Scenario 2: FCoE Right & Wrong
FCFCFC
LosslessEthernet
ClassicalEthernet
FCoESwitch
Internet
FCoESwitches
LosslessEthernet
FCoE Flows
Invalid FCoE Flows
8/9/2019 FCoE - Fibre Channel Over Ethernet
39/65
Fibre Channel over Ethernet (FCoE)© 2008 Storage Networking Industry Association. All Rights Reserved. 39
Summary
8/9/2019 FCoE - Fibre Channel Over Ethernet
40/65
Fibre Channel over Ethernet (FCoE)© 2008 Storage Networking Industry Association. All Rights Reserved. 40
Summary
FCoE is a simple, efficient mechanism forencapsulating Fibre Channel in Ethernet frames
FCoE is being standardized in INCITS FibreChannel (T11) technical committee
Target completion is 2H08
Maximum benefit of Fibre Channel is achieved:Evolutionary model of FC Switches and FC SANsEmphasis placed on capitalizing on the benefits ofConverged Enhanced Ethernet (CEE)
Being discussed in the IEEE 802.1 standards working group
8/9/2019 FCoE - Fibre Channel Over Ethernet
41/65
Thank You!
8/9/2019 FCoE - Fibre Channel Over Ethernet
42/65
Fibre Channel over Ethernet (FCoE)© 2008 Storage Networking Industry Association. All Rights Reserved. 42
Q&A / Feedback
Please send any questions or comments on thispresentation to SNIA: [email protected]
For additional information refer to
http://www.t11.org/fcoe
Many thanks to the following individualsfor their contributions to this tutorial.
SNIA Education Committee
Claudio DeSanti Howard Golds tein Walter DeyRobert Snively Suresh Vobbili setty Silvano GaiJoe Peliss ier John Hufferd
mailto:[email protected]://www.t11.org/fcoehttp://www.t11.org/fcoemailto:[email protected]
8/9/2019 FCoE - Fibre Channel Over Ethernet
43/65
Fibre Channel over Ethernet (FCoE)© 2008 Storage Networking Industry Association. All Rights Reserved. 43
Appendix
AdditionalRequirements
FCoE Relation to ISOLayersFlows
Additional TopologiesFSPF and STPFIP Considerationsand ActionsFIP DescriptorsPause vs. BB_Credit
8/9/2019 FCoE - Fibre Channel Over Ethernet
44/65
Fibre Channel over Ethernet (FCoE)© 2008 Storage Networking Industry Association. All Rights Reserved. 44
FCoE Goals/Requirements (part 3)
FCoE must be a direct mapping of Fibre Channel overan Ethernet networkFCoE must be layered on top of Ethernet
FSPF used to route FCoE packetsEthernet Spanning Tree (STP), MSTP, etc, is at a layer below
FCoE to allow an evolutionary approach towardsconsolidation of fabrics
The Fibre Channel N_Port, F_Port and E-Port constructs mustbe retained
With FCoE, ports may be connected with Logical Ethernet Links – May pass through Ethernet switches – Identified by pairs of end point MAC addresses
Physical Ethernet Links can replace physical FC LinksPhysical Ethernet Links can carry all Ethernet traffic, includingFCoE, but combined traffic needs the CEE capabilities
8/9/2019 FCoE - Fibre Channel Over Ethernet
45/65
Fibre Channel over Ethernet (FCoE)© 2008 Storage Networking Industry Association. All Rights Reserved. 45
FCoE Goals/Requirements (part 4)
“Combo FCoE Switches ” may be built that supportnormal Ethernet traffic, FCoE traffic , & FC trafficThe FCoE solutions should appear as a Fibre Channel toa Fibre Channel experienced customer FCoE should keep the Fibre Channel operationsindependent from Ethernet forwarding
Keeps management /Troubleshooting simpleCommon physical structures, different logical structures
Based on Ethertype (Ethertype = FCoE)
Storage Management should be unchanged
8/9/2019 FCoE - Fibre Channel Over Ethernet
46/65
Fibre Channel over Ethernet (FCoE)© 2008 Storage Networking Industry Association. All Rights Reserved. 46
FCoE Relation to ISO Layers
8/9/2019 FCoE - Fibre Channel Over Ethernet
47/65
Fibre Channel over Ethernet (FCoE)© 2008 Storage Networking Industry Association. All Rights Reserved. 47
Flows
L i l F b i T l
8/9/2019 FCoE - Fibre Channel Over Ethernet
48/65
Fibre Channel over Ethernet (FCoE)© 2008 Storage Networking Industry Association. All Rights Reserved. 48
Logical Fabric Topology
FCFCFC
FCoESwitch
FCoESwitch
FCoESwitch
LosslessEthernet
LosslessEthernet
A
H1
H2
H3H4 S2
EthernetDestination
& Source
EncapsulatedFC Frame
D_IDS_ID
FCoE-A MACFCoE-H2 MAC
FC_ID for S1FC_ID for H2 FC_ID for S1FC_ID for H2 FC_ID for S1FC_ID for H2 FC_ID for S1FC_ID for H2
FCoE-B MACFCoE-A MAC
FCoE-C MACFCoE-B MAC
FCoE-S1 MACFCoE-C MAC
EthernetFC
LogicalTransaction Path
An FCoE Switch receives FCoE frames addressed to its FC-MAC address andforwards them based on the D_ID of the encapsulated FC frame
An FCoE Switch rewrites the SA and DA of an FCoE frame
LosslessEthernet
LosslessEthernet
S1
Path #1 Path #2 Path #3 Path #4
#1
#2
#3
#4B
C
8/9/2019 FCoE - Fibre Channel Over Ethernet
49/65
8/9/2019 FCoE - Fibre Channel Over Ethernet
50/65
Fibre Channel over Ethernet (FCoE)© 2008 Storage Networking Industry Association. All Rights Reserved. 50
FC Host to FCoE Storage
FCFCFC
FCoESwitch
FCoESwitch
Lossless Ethernet
H1
H2
H3
S1
S3
FCoE-S2 MACEthernet
Destination& Source
FCoE-A MACEncapsulated
FC Frame
D_ID
S_ID
FC_ID for S2
FC_ID for H5
FC_ID for S2
FC_ID for H5
FC_ID for S2
FC_ID for H5
FCoE-A MAC
FCoE-C MAC
EthernetSwitch
EthernetSwitch
EthernetSwitch
EthernetSwitch
H5S2
EthernetSwitch
EthernetFC
LogicalTransaction Path
Path #3 Path #2
Path #1
#1
#2
#3
C A
8/9/2019 FCoE - Fibre Channel Over Ethernet
51/65
Fibre Channel over Ethernet (FCoE)© 2008 Storage Networking Industry Association. All Rights Reserved. 51
Additional Topologies
8/9/2019 FCoE - Fibre Channel Over Ethernet
52/65
Fibre Channel over Ethernet (FCoE)© 2008 Storage Networking Industry Association. All Rights Reserved. 52
Additional Topologies (1)
FCoESwitch (A)
FCoESwitch (B)
FCoESwitch (E)
FCoESwitch (F)
LosslessEthernetSwitch
LosslessEthernetSwitch
FCFCFC
Example of Topologies with Rack Mount servers
8/9/2019 FCoE - Fibre Channel Over Ethernet
53/65
Fibre Channel over Ethernet (FCoE)© 2008 Storage Networking Industry Association. All Rights Reserved. 53
Additional Topologies (2)
FCoESwitch (B)
FCoESwitch (E)
FCoESwitch (F)
LosslessEthernetSwitch
LosslessEthernetSwitch
FCFCFC
FCoESwitch (A)
Equivalent to Blade servers with N_Port_ID Virtualization (NPIV) SupportExample of Topologies with Blade Servers
8/9/2019 FCoE - Fibre Channel Over Ethernet
54/65
Fibre Channel over Ethernet (FCoE)© 2008 Storage Networking Industry Association. All Rights Reserved. 54
FSPF and STP
8/9/2019 FCoE - Fibre Channel Over Ethernet
55/65
Fibre Channel over Ethernet (FCoE)© 2008 Storage Networking Industry Association. All Rights Reserved. 55
FSPF & STP Concepts with FCoE (basic)
FSPF & STP Concepts with FCoE (interconnected)
8/9/2019 FCoE - Fibre Channel Over Ethernet
56/65
Fibre Channel over Ethernet (FCoE)© 2008 Storage Networking Industry Association. All Rights Reserved. 56
FSPF & STP Concepts with FCoE (interconnected)
8/9/2019 FCoE - Fibre Channel Over Ethernet
57/65
8/9/2019 FCoE - Fibre Channel Over Ethernet
58/65
Fibre Channel over Ethernet (FCoE)© 2008 Storage Networking Industry Association. All Rights Reserved. 58
FIP Considerations and Actions
d
8/9/2019 FCoE - Fibre Channel Over Ethernet
59/65
Fibre Channel over Ethernet (FCoE)© 2008 Storage Networking Industry Association. All Rights Reserved. 59
FCoE Discovery Considerations
The Discovery phase of the FCoE Initialization Protocol (FIP)uses two types of messages, Solicitations and AdvertisementsThe FIP Discovery Phase helps define the FCF Ports that areavailable for the Link instantiation PhaseThe ENodes discover the FCF ports that can become VF_Portsand FCFs discover other FCF ports that can become VE_Ports
ENodes Solici t (via Multicast) Advertisements from FCFs whilespecifying their capabilitiesIn response FCF’s Advertise their availability and capabilities backto the ENodesFCF’s Multicast their existent to other FCF
The FIP Discovery phase exchanges solicitation and/or Advertisements between (HBA and/or FCF) “ FCoE Controllers”
C i A i
8/9/2019 FCoE - Fibre Channel Over Ethernet
60/65
Fibre Channel over Ethernet (FCoE)© 2008 Storage Networking Industry Association. All Rights Reserved. 60
FCF Discovery Actions
FCF supporting VE_Ports:Discovers other VE_Port capable FCF-MACs, connectedto the same Lossless Ethernet segment, by:
Transmitting a multicast Solicitation to ‘All-FCFs’(with the FCF bit set to one)
Receives back Jumbo Unicast Advertisements from VE capableMACs
To verify the support of Ethernet Jumbo frames in the pathIn response to receipt of a Multicast to “All-FCFs” from anotherFCF
Has its own VE capable MACs sends their own Jumbo Unicast Advertisements
Instantiates VE_Port to VE_Port connections andExchanges FC ELP (Extended Link Protocol) and Fabricconfiguration (using Ethertype=FCoE) with the other
FCFs VE_Port capable MACs
EN d A i
8/9/2019 FCoE - Fibre Channel Over Ethernet
61/65
Fibre Channel over Ethernet (FCoE)© 2008 Storage Networking Industry Association. All Rights Reserved. 61
ENode ActionsWhen an ENode becomes operational:
The ENode discovers the VF_Port capable FCF-MACsconnected to the same Lossless Ethernet segment
Transmits a multicast Solicitation to ‘All-FCFs’ (with the FCF bit setto zero)Receives Jumbo unicast Advertisements from select compatibleVF_Port capable FCF-MACsStores the discovered FCF-MACs in an FCF list
When an ENode receives an Advertisement that anew FCF is available, it may send a unicastSolicitation to it and receive a Jumbo unicast
Advertisement in reply
To verify the support of Ethernet Jumbo frames in the pathMay then perform FLOGIs (with Ethertype-FIP) to avendor specific subset of the FCF-MACs in the FCFlist
8/9/2019 FCoE - Fibre Channel Over Ethernet
62/65
FIP D i (1)
8/9/2019 FCoE - Fibre Channel Over Ethernet
63/65
Fibre Channel over Ethernet (FCoE)© 2008 Storage Networking Industry Association. All Rights Reserved. 63
FIP Descriptors (1)
FIP D i (2)
8/9/2019 FCoE - Fibre Channel Over Ethernet
64/65
Fibre Channel over Ethernet (FCoE)© 2008 Storage Networking Industry Association. All Rights Reserved. 64
FIP Descriptors (2)
P BB C dit
8/9/2019 FCoE - Fibre Channel Over Ethernet
65/65
Pause vs. BB_Credit
Both mechanisms are used to avoid dropping framesWith different trade-offs
The Pause mechanism requires at least the (2 x RTT x
bandwidth) product on a link as buffer spaceBut allows Buffer handling in an arbitrary wayWell suited for networks with limited (bandwidth x delay) product(e.g. within the data center)
The Pause frame is handled by the MAC layer Similar to the R_RDY handling by the FC-1 level
The BB_Credit mechanism prevents loosing framesover any link
But links go under-utilized if link credits (& buffers) are < thatneeded for (RTT x BW)
Requires buffer handling in maximum frame size units