Top Banner

of 26

FCoE - Fibre Channel Over Ethernet

Jun 01, 2018

Download

Documents

Tuan Beo
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
  • 8/9/2019 FCoE - Fibre Channel Over Ethernet

    1/65

    Fibre Channel over Ethernet (FCoE)

    John L Hufferd, Brocade

  • 8/9/2019 FCoE - Fibre Channel Over Ethernet

    2/65

    Fibre Channel over Ethernet (FCoE)© 2008 Storage Networking Industry Association. All Rights Reserved. 2

    SNIA Legal Notice

    The material contained in this tutorial iscopyrighted by the SNIA.Member companies and individuals may use thismaterial in presentations and literature under thefollowing conditions:

    Any slide or slides used must be reproduced withoutmodificationThe SNIA must be acknowledged as source of any

    material used in the body of any document containingmaterial from these presentations.

    This presentation is a project of the SNIA

    Education Committee.

  • 8/9/2019 FCoE - Fibre Channel Over Ethernet

    3/65

  • 8/9/2019 FCoE - Fibre Channel Over Ethernet

    4/65

    Fibre Channel over Ethernet (FCoE)© 2008 Storage Networking Industry Association. All Rights Reserved. 4

    Agenda

    Introduction

    Goals & Requirements

    Consolidation

    Architecture

    Discovery & Link Instantiation

    Topologies

    Scenarios

    Summary

  • 8/9/2019 FCoE - Fibre Channel Over Ethernet

    5/65

    Fibre Channel over Ethernet (FCoE)© 2008 Storage Networking Industry Association. All Rights Reserved. 5

    Introduction

    This presentation provides an overview of FibreChannel over Ethernet (FCoE)

    One should think about FCoE as placing the FCprotocol on a new physical link

    Ethernet links instead of physical FC linksBut it is still Fibre Channel

    The protocol is being defined in the INCITS FibreChannel (T11) technical committee

    Many details of the protocol still need to be definedbut the significant major issues have been resolved

    Target T11 Standards completion is the 2H08

  • 8/9/2019 FCoE - Fibre Channel Over Ethernet

    6/65

    Fibre Channel over Ethernet (FCoE)© 2008 Storage Networking Industry Association. All Rights Reserved. 6

    The Origins of Data Center Ethernet

    The technology has evolved continuously, showing a great ability toadapt to new technologies and increasing business requirements

    Increasing Scalabil ity, Feature, Function

    I n c r e a s

    i n g

    P e r f o r m a n c e

    Integration of Carrier-grade features

    Incremental Protocol Enhancements

    Logical Partitioning

    Evolution from shared media to dedicated media

    1973

    2008-2009 CEE

    Introduction of Ethernet

    10Mbps

    100Mbps

    1Gbps

    10Gbps(Converged Enhanced Ethernet)

  • 8/9/2019 FCoE - Fibre Channel Over Ethernet

    7/65

    Fibre Channel over Ethernet (FCoE)© 2008 Storage Networking Industry Association. All Rights Reserved. 7

    Goals/Requirements

  • 8/9/2019 FCoE - Fibre Channel Over Ethernet

    8/65

    Fibre Channel over Ethernet (FCoE)© 2008 Storage Networking Industry Association. All Rights Reserved. 8

    FCoE Goals/Requirements (part 1)

    FCoE Fabrics must be built with FCoE SwitchesSwitches with Ethernet ports that provide FCoE capabilities andservices – Called, in the standard, an FC Forwarder FCFSwitches that include the functions of traditional FC switches

    Standard Ethernet switches may also exist in the fabric but switcheswith FCoE capabilities are required

    FCoE fabrics must operate seamlessly with real FC Fabrics

    FC services must operate identically on FCoE fabrics andFibre Channel fabrics

    FCoE must support all Fibre Channel advanced features(e.g. virtual fabrics, IFR, security, etc.) transparently

    FCoE is NOT a replacement for FCIP or iFCPFCIP & iFCP use TCP/IPFCIP/iFCP is for inter-switch links beyond the Data Center

  • 8/9/2019 FCoE - Fibre Channel Over Ethernet

    9/65

    Fibre Channel over Ethernet (FCoE)© 2008 Storage Networking Industry Association. All Rights Reserved. 9

    Integrated Ethernet & FCoE switch (FCF)with FC connectionsSome implementations may combine the features and capabilities of an EthernetSwitch with the features and capabilities of a FC switch

    • Support Ethernet and IP standards for switching, pathing and routing

    • Support FC standards for switching, pathing and routing

    • Support current and enhanced Ethernet Standards

    • Adapt between FCoE and FC

    An FCoE Port (N_Port, F_Port or E_Port)

    • Has the same function as in FC• But is layered on top of Ethernet

    • Called VN_Port, VF_Port and VE_Port

    Because many logical (virtual)Ports can share one physical port

    Note:FCF Pathing and Forwarding utilizes the FSPF (Fabric Shortest Path First) protocol

    Non-FCoE Ethernet traffic is relayed using conventional 802.1 defined mechanisms such as

    STP (Spanning Tree Protocol) and MSTP (Multiple STP)See additional FSPF and STP explanation in Appendix

    FCFCFCCEE Ethernet Ports(with IP & FCoE VF_Port &VE_Port capabilities)

  • 8/9/2019 FCoE - Fibre Channel Over Ethernet

    10/65

    Fibre Channel over Ethernet (FCoE)© 2008 Storage Networking Industry Association. All Rights Reserved. 10

    Connections to a Combo FCoE Switch

    Applications

    TCPUDPIP

    SCSI

    Fibre Channel

    FCoE

    Lossless Ethernet MAC (CEE)

    • Fibre Channel is carried over lossless Ethernet as a L3 protocol

    Combo Lossless Ethernet (CEE) Switch

    with FCoE Switch (FCF) capabilities

    (FCoE VN_Port)

    Ethernet port with IP & FCoE VF_Portcapabilities

    IP address 123.45.67.89

    iSCSI

    SCSI

  • 8/9/2019 FCoE - Fibre Channel Over Ethernet

    11/65

    Fibre Channel over Ethernet (FCoE)© 2008 Storage Networking Industry Association. All Rights Reserved. 11

    FCoE Goals/Requirements (part 2)

    FCoE requires specific Ethernet extensions to be implementedLossless switches and fabrics (e.g., supporting IEEE 802.3 PAUSE)configurations are requiredJumboframe support is required (not a standard, but widely available)

    Deployments of FCoE should utilize the advances in Ethernetcurrently being discussed in IEEE 802.1, specifically:

    Priority-based Flow Control (PFC)

    Enhanced Transmission Selection (ETS)DCB Capability eXchange Protocol (DCB CXP)

    These 802.1 advances are important for Consolidated

    Flows (Messaging, Clustering and Storage)This set of functions has been called Data Center Ethernet, or CEE –Converged Enhanced Ethernet (intended for a Data CenterEnvironment)

    FCoE should require no changes to FC software

  • 8/9/2019 FCoE - Fibre Channel Over Ethernet

    12/65

    Fibre Channel over Ethernet (FCoE)© 2008 Storage Networking Industry Association. All Rights Reserved. 12

    Consolidation

  • 8/9/2019 FCoE - Fibre Channel Over Ethernet

    13/65

    Fibre Channel over Ethernet (FCoE)© 2008 Storage Networking Industry Association. All Rights Reserved. 13

    Dramatic reduction in adapter, switch ports and cabling4-6 cables to 2 cables per server

    Seamless connection to the installed based of existing SANs

    and LANsRequires high performance lightweight frame mappers vs.heavy weight gateways

    FCoE has no need to terminate and re-initiate a SCSI connection

    (e.g. iSCSI to FC)Effective sharing of high bandwidth links

    High End 10GE Server & NIC/HBAConsolidation

    Today

    With CEEOS3DB Server OS2 App Ser ver OS1Web Server

    Messaging

    MPIRDMA

    FCHBA

    OS3DB Server

    OS2 App Server

    OS1Web Server

    IB/EthernetCluster

    Hyper Visor (vmWare, Xen, etc)

    E-HBA(CEE)

    Data Center CEE Network

    EthernetTCP/IP

    FC SAN

    Data Center CEE Network

    E-HBA(CEE)E-HBA(CEE)

    •NIC

    •TCP acceleration .•MPI , RDMA

    over Ethernet•FCoE

    OS3DB Server

    OS2 App Ser ver

    OS1Web Server

  • 8/9/2019 FCoE - Fibre Channel Over Ethernet

    14/65

    Fibre Channel over Ethernet (FCoE)© 2008 Storage Networking Industry Association. All Rights Reserved. 14

    Architecture

  • 8/9/2019 FCoE - Fibre Channel Over Ethernet

    15/65

    Fibre Channel over Ethernet (FCoE)© 2008 Storage Networking Industry Association. All Rights Reserved. 15

    FC Encapsulation Into Network Packets(2 FCoE Related Packet types)

    IEEE 802.3Layers

    FC Levels(Unchanged)

    FCoE Mapping

    FC-4FC-3

    FC-2

    FC-1

    FC-0

    FC-4FC-3

    FC-2

    MAC

    PHY

    Frame Check Sequence(CRC)

    Protocol contr ol in formation: Version, SOF, EOF, etc.FC Imbedded Frames: Same as in Physical FC

    Ethernet

    Header FCSFCoE

    Header FC Header SCSI Commands/Data

    Ethertype“ FCoE”

    Fibre Channel over Ethernet (Fibre Channel over Ethernet ( FCoEFCoE ) Packets) Packets

    Protocol contro l information: Version, Op-codes, etc.Discovery and Logi n/Logout Parameters

    EthernetHeader FCS

    FIPHeader

    Descriptors

    Ethertype“ FIP”

    FCoE Initialization Protocol (FCoE Initialization Protocol ( FIPFIP ) Packets) Packets

    Ethernet Header provides things needed fo r thephysical network, including “ Ethertype ”

  • 8/9/2019 FCoE - Fibre Channel Over Ethernet

    16/65

    Fibre Channel over Ethernet (FCoE)© 2008 Storage Networking Industry Association. All Rights Reserved. 16

    FIP Protocol and FCoE Protocol

    Discovery PhaseFCFs Discovery each other and form a FabricENodes and FCFs Discover

    Potential VN_Port VF_Port pairing

    Capabilities of Potential pairingLogin Phase

    ENodes chose among discovered FCFs’ PortsCreates association between ENode Ports and FCF Ports

    VN_Port VF_Port Logical FC LinkTwo allowed alternatives for the ENode MAC Addressing

    Fabric Provided MAC Addresses (FPMA)Server Provided MAC Addresses (SPMA)Chosen by FCF (FPMA & SPMA Described latter)

    Uses: FLOGI, FLOGI ACC, LOGO, …

    Data Transfer PhasePLOGI/PRLI

    All other FC protocol frames (ELS, FC4 ULPs. etc.)

    FCoEInitializationProtocol ( FIP )

    FCoE Protocol

  • 8/9/2019 FCoE - Fibre Channel Over Ethernet

    17/65

    Fibre Channel over Ethernet (FCoE)© 2008 Storage Networking Industry Association. All Rights Reserved. 17

    FIP Operation Format

    Word 31-24 23-16 15-8 7-00 Destination MAC Address (6 Bytes)

    1

    2 Source MAC Address (6 Bytes)3 ET=FIP (16 bits) Ver (4b) Reserved (12 bits)

    4 FIP Operation Code

    5 Reserved FIP subcode Descriptor List Length

    6 Flags FPSP

    S F

    PAD to minimum length or mini-Jumbo length…n

    n+1 Ethernet FCS

    Optional IEEE 802.1q4 ByteTag goes here

    Descriptor listvariesIn size

    Ethernet framesize

    Is 64Bytes to 2220Bytes

    FIP Operation Code Reserved FIP SubCode

    Descriptor List Length Flags FP SP S F

    Descriptor List

    Solicited bit FCF bitCapability Bits (SPMA or FPMA)

    See Appendix forDescriptor list items

  • 8/9/2019 FCoE - Fibre Channel Over Ethernet

    18/65

    Fibre Channel over Ethernet (FCoE)© 2008 Storage Networking Industry Association. All Rights Reserved. 18

    FC’s Encapsulation in Ethernet (FCoE)

    Word 31-24 23-16 15-8 7-00 Destination MAC Address (6 Bytes)

    1

    2 Source MAC Address (6 Bytes)3 ET=FCoE (16 bits) Ver (4b) Reserved (12 bits )

    4 Reserved

    5 Reserved

    6 Reserved SOF (8 bits)

    7 Encapsulated FC FrameFC Frame = Minimum 28 Bytes (7 Words)

    Maximum 2180 Bytes (545 Words)(including FC-CRC)

    n

    n+1 EOF (8 bit s) Reserved

    n+2 Ethernet FCS

    Optional IEEE 802.1q4 ByteTag goes here

    This field varies

    In size

    Ethernet framesizeIs 64Bytes to 2220Bytes

  • 8/9/2019 FCoE - Fibre Channel Over Ethernet

    19/65

    Fibre Channel over Ethernet (FCoE)© 2008 Storage Networking Industry Association. All Rights Reserved. 19

    FCF Model

    LinkEnd

    Point(LEP)

  • 8/9/2019 FCoE - Fibre Channel Over Ethernet

    20/65

    Fibre Channel over Ethernet (FCoE)© 2008 Storage Networking Industry Association. All Rights Reserved. 20

    ENode (HBA) Model

    Each ENode (HBA) may have multiple Physical Ethernet PortsEach Physical Port may have multiple Logical VN_Ports

    LinkEnd

    Point(LEP)

  • 8/9/2019 FCoE - Fibre Channel Over Ethernet

    21/65

    Fibre Channel over Ethernet (FCoE)© 2008 Storage Networking Industry Association. All Rights Reserved. 21

    Model of the ENODE with MultipleLogical FC interfaces

    FCoEController

    Lossless Ethernet MAC Ethernet_Port

    FCEntityVN_Port

    FCoEEntity

    FCoE_LEP

    FC-3 /FC-4s

    MAC Addressof “ Burnt-in

    MAC

    FCEntityVN_Port

    FCoEEntity

    FCoE_LEP

    FC-3 /FC-4s

    . . .

    • For each logical N_Port

    (VN_Port) there is one FLOGIand perhaps 100’s of FDISC

    • Each VN_Port is seen bythe Host as a separate(logical) FC connection

    • The number of (logical) FCconnections is

    implementation dependent

    • Only one MAC Address is required for the FCoE Controller and the VN_Ports on asingle physical MAC (aka Server Provided MAC Address – SPMA)• FCF may chose to specify new MAC addresses for each VN_Port (aka FabricProvided MAC Address – FPMA)

    In thismodel thisis where

    FC-2functions

    live

    MAC Address ofFCoE_LEP(VN_Port)

    May or m ay not bethe same as theFCoE cont roller

    In this model thisis where the

    Encapsulation/De-Encapsulation

    functions live

    Multiple FC NPIVinstances on a single

    logical FC Host interface

    M l i l L i l FC i

  • 8/9/2019 FCoE - Fibre Channel Over Ethernet

    22/65

    Fibre Channel over Ethernet (FCoE)© 2008 Storage Networking Industry Association. All Rights Reserved. 22

    The Logical FC Link is defined by a MAC Address pair • A VN_Port MAC Address

    • A VF_Port MAC AddressFor a logical FC link the FCoE Frames are always sent to and received from a specific FCF’sMAC Address• Therefore, pathing to and from the FC driver is always defined by the MAC Address of the

    partner FCF’s VF_Port

    Multiple Logical FC connectionsvia a single Ethernet MAC

    Examples of

    single MACswithconnectionsto twodifferentFCFs Switch

  • 8/9/2019 FCoE - Fibre Channel Over Ethernet

    23/65

    Fibre Channel over Ethernet (FCoE)© 2008 Storage Networking Industry Association. All Rights Reserved. 23

    Functions of an FCoE Initiator ASIC

    NIC Function F C oE

    f un c

    t i on

    FCFunction

    Host PCIe interface

    ExternalPort

    FCoE

    ASIC

    • Has a Normal NIC interface(A) to the Host

    • Has one or more NormalFC interfaces (B,C) to theHost

    • FCoE functions not seen bythe Host

    • FCoE functions perform theEncapsulation and De-encapsulation

    • The FCoE functionInstantiates a Logical FC

    N_Port, called a VN_Port

    LosslessEthernet

    MAC

    A B C

  • 8/9/2019 FCoE - Fibre Channel Over Ethernet

    24/65

    Fibre Channel over Ethernet (FCoE)© 2008 Storage Networking Industry Association. All Rights Reserved. 24

    HBA with Multiple Logical FC Interfaces

    • MAY have one “burnt-inMAC address for both IPand FCoE/FIP packets

    Or

    • MAY have different “Burnt-in” MACs for IP andFCoE/FIP packets

    • Used to separate HWbased FCoE from otherEthernet Traffic

    • Most NICs come withseveral “Burnt-in” MAC

    Addresses

    • The FCoE controller will perform the FIP functions and will instantiate newVN_Ports as FCoE Link End Point (LEP)• With the same MAC address as the FCoE Controller (SPMA)• Or with a new MAC address specified by the FCF (FPMA)

    F C oE

    C on

    t r ol l er

    F C - 3

    / F C -4

    s F

    C

    E n

    t i t y

    V N

    _P

    or

    t F

    C oE

    E n

    t i t y

    F C oE

    _L E P

    Lossless Ethernet MACEthernet_Port

    NIC

    Etype=FCoE

    or

    FIP?

    FCoE Chip

    MAC Add ress of

    “Burnt-inMAC

    FCoE Function

    F C - 3

    / F C -4

    s

    F C

    E n

    t i t y

    V N

    _P

    or t

    F C oE

    E n

    t i t y

    F C oE

    _L E P

    MAC Add ress of

    “Burnt-inMAC

    A B C

  • 8/9/2019 FCoE - Fibre Channel Over Ethernet

    25/65

    Fibre Channel over Ethernet (FCoE)© 2008 Storage Networking Industry Association. All Rights Reserved. 25

    FIP (FCoE Initiation Protocol)

    Discovery and Link Instantiation

  • 8/9/2019 FCoE - Fibre Channel Over Ethernet

    26/65

    Fibre Channel over Ethernet (FCoE)© 2008 Storage Networking Industry Association. All Rights Reserved. 26

    ENode to FCF Logical Links

    FCfabric

    H1

    H2ClassicalEthernet

    LAN

    FCF B

    LosslessEthernetSwitch

    FCF A

    VN_Ports,VN_Port_Names

    FCoE_LEP

    VN_Port

    FCoE_LEP

    VN_Port

    FCoE_LEP

    VN_Port

    FCoE_LEP

    VN_Port

    FCoE_LEPs

    VF_Port

    FCoE_LEPs

    VF_Port

    VF_Port,VF_Port_Name

    FCoE_LEPs

    VF_Port

    FCoE_LEPs

    VF_Port

    FCF-MAC(A)

    FCF-MAC(B)

    MAC(H1)

    MAC(H2)

    FCF-MAC(x): A MAC address of a Lossless Ethernet port of FCF xMAC(y): A MAC address of a Lossless Ethernet port of ENode y

    Note: The FCFs mustdiscover each other andcreate a Fabric before theENode Discover FCFs

    An ENode must Locate FCFs with a Discovery protocol and then establish a Logical FCLink with an FCF (VN_Port VF_Port) before an Normal FC frame flow

  • 8/9/2019 FCoE - Fibre Channel Over Ethernet

    27/65

  • 8/9/2019 FCoE - Fibre Channel Over Ethernet

    28/65

    Fibre Channel over Ethernet (FCoE)© 2008 Storage Networking Industry Association. All Rights Reserved. 28

    Subsequent Login Flow ladder (1 Phase)

    End-Node FCF

    FLOGI

    Login Phase

    Normal FCProcessing

    FCoE InitializationProtocol

    FCOE ProtocolSee Appendix for more details in Discovery Consideration and Ac tions

    FC Command(Using the FCFselected MAC

    Address as the SA)FC Command responses

    (using the command’s SA as a DA)

    FLOGI ACC with the FCF’schosen VN_Port MAC addressas a descriptor value

  • 8/9/2019 FCoE - Fibre Channel Over Ethernet

    29/65

    Fibre Channel over Ethernet (FCoE)© 2008 Storage Networking Industry Association. All Rights Reserved. 29

    Topologies

  • 8/9/2019 FCoE - Fibre Channel Over Ethernet

    30/65

    Fibre Channel over Ethernet (FCoE)© 2008 Storage Networking Industry Association. All Rights Reserved. 30

    FCoE Switch & FC Fabric

    FCFCFCFCoE Switch

    An FCoE Switch may connect to a normal FC switchVia the FC E-Port Ethernet

    FC

    Note: FCoE serversand storage wil lprobably use an FCoEHBA (or chip set)

  • 8/9/2019 FCoE - Fibre Channel Over Ethernet

    31/65

    Fibre Channel over Ethernet (FCoE)© 2008 Storage Networking Industry Association. All Rights Reserved. 31

    Multiple FCoE Switch (FCF) Topologies

    EthernetFC

    FCFCFCLossless

    EthernetFabric

    FCFCFC

    Lossless EthernetFabric

    FCoE Switc h

    A Lossless Eth ernet Fabric can be made up of Combo FCoE SwitchesLossless Ethernet switches config ured into a Lossless Ethernet

    Fabric can Front the FCoE Switch

    FCFCFC

    FCoE

    Switch

    FCoESwitch

    FCoE

    SwitchLossless Ethernet

    FCoE Switches deployed at t he edges of the Lossl ess Ethernet FabricFCoE Switches connected via VE_Ports and Lossless Ethernet

    FCFCFC

    FCoESwitch

    FCoESwitch

    LosslessEthernet1

    LosslessEthernet

    2

    LosslessEthernet

    4

    FCoESwitch

    LosslessEthernet

    3

    A VE_Port in an FCF may connect to o ther VE_Port in another FCF And an FCF FC E_Port may connect to an FC swi tch E_Port

    Status of Current Data Center

  • 8/9/2019 FCoE - Fibre Channel Over Ethernet

    32/65

    Fibre Channel over Ethernet (FCoE)© 2008 Storage Networking Industry Association. All Rights Reserved. 32

    Status of Current Data CenterNetworking

    There can be 3 differentnetworks

    System Area Network (SyAN)

    Used for Clustering/Low LatencyStorage area Network (SAN)

    Used to Access to StorageLAN/WAN External Networks

    Used for General messaging

    Used for Client-Server MessagingUsed for NAS

    Often divided into at least 3management domains

    Data Center Server (clustering)NetworkData Center Storage NetworkOutfacing (IP) Network

    LAN/WANs – Messaging – NAS

    Data Center

    Remote Offices

    System Area Network (SyAN)•Clustering Fabric•InfiniBand

    •Myrinet•Ethernet•CTC

    Data Center Server & Storage Network

    Management Group

    Outfacing (IP) NetworkManagement Group

    LAN/WAN•Messaging•NAS

    Storage Area Network- Fibre Channel

    Note: with multiple Data Centers there may also

    be interconnects with DWDM, FCIP/iFCP, etc.

  • 8/9/2019 FCoE - Fibre Channel Over Ethernet

    33/65

    Fibre Channel over Ethernet (FCoE)© 2008 Storage Networking Industry Association. All Rights Reserved. 33

    Current FabricsRemoteRemoteOfficesOffices

    Clustering Network

    There is a FC Storage Network, a Clustering Network & an “ Outfacing” Network

    IP Network Management Group

    FC

    FC

    FC

    Ficon

    Ficon StorageController

    Mainframe

    Data Center Server & Storage Network Management Group

    File Storage Arrays (NAS)

    iSCSIStorage

    FCNetwork

    Ficon

    (LAN/WAN)(LAN/WAN)••MessagingMessaging••NASNAS

    OutfacingIP Network

    Local & RemoteLocal & RemoteBusiness CampusBusiness Campus

    Focus: LowLatency & High

    Bandwidth

    Focus: Protection,Bandwidth/Congestion

    Management

    FC Link

    EthernetLink

    FiconLink

    Clustering Network

    Note: with multiple Data Centers there may also be interconnects with DWDM, FCIP/iFCP, etc.

    Vie of the F t re Data Center

  • 8/9/2019 FCoE - Fibre Channel Over Ethernet

    34/65

    Fibre Channel over Ethernet (FCoE)© 2008 Storage Networking Industry Association. All Rights Reserved. 34

    View of the Future Data CenterNetworks

    RemoteRemoteOfficesOffices

    Data Center Server &Storage Network

    Management Group

    Outfacing (IP) NetworkManagement Group

    Storage & SystemStorage & System Area Network Area Network••Logically Single FabricLogically Single Fabric

    (with FCoE)(with FCoE)

    Other IP NetsOther IP Nets(LAN/WAN)(LAN/WAN)••MessagingMessaging••NASNAS

    TheThe Server/Storage NetworksServer/Storage Networkswill become a Consolidatedwill become a ConsolidatedFabricFabric

    Managed by the Data CenterManaged by the Data CenterSystem and StorageSystem and StorageManagement GroupManagement Group

    – – Includes Storage andIncludes Storage and

    clustering provisioningclustering provisioning – – SAN and SyAN managedSAN and SyAN managedas a single fabricas a single fabric

    Focus: High Bandwidth andFocus: High Bandwidth andLow LatencyLow Latency

    The Outfacing (The Outfacing ( IP) ManagementIP) ManagementGroupGroup remains the sameremains the same

    Focus: Protection, BandwidthFocus: Protection, Bandwidthand Congestion Managementand Congestion ManagementNote: with multiple Data Centers there may alsobe interconnects with DWDM, FCIP/iFCP, etc.

    Phase in: FC & High Performance Lossless

  • 8/9/2019 FCoE - Fibre Channel Over Ethernet

    35/65

    Fibre Channel over Ethernet (FCoE)© 2008 Storage Networking Industry Association. All Rights Reserved. 35

    Phase in: FC & High Performance LosslessEthernet Unify into a Data Center Fabric

    RemoteRemoteOfficesOffices

    FC

    FC

    FCoE

    FC

    (LAN/WAN)(LAN/WAN)••MessagingMessaging••NASNAS

    OutfacingIP Network

    Ficon

    iSCSIStorage

    Ficon

    Ficon StorageController

    Mainframe

    File Storage Arrays (NAS)

    • FCoE permits intermixing of multiple Connection types/protocols• Clustering messaging, General Messaging, and Storage

    • The DataCenter Fabric will “ Trunk” to the “ Outfacing” Network (including iSCSI sys)• Some Customers may want keep a mixed environment on-going

    DataCenter Fabric

    EthernetSW

    EthernetSW

    FC & EthernetSW Blades

    FC Link

    EthernetLink

    FiconLink

    BusinessCampus

    with iSCSIconnections

    Including iSCSIGateways

    Note: with multiple Data Centersthere may also be interconnects

    with DWDM, FCIP/iFCP, etc.

  • 8/9/2019 FCoE - Fibre Channel Over Ethernet

    36/65

    Fibre Channel over Ethernet (FCoE)© 2008 Storage Networking Industry Association. All Rights Reserved. 36

    Scenarios

  • 8/9/2019 FCoE - Fibre Channel Over Ethernet

    37/65

  • 8/9/2019 FCoE - Fibre Channel Over Ethernet

    38/65

    Fibre Channel over Ethernet (FCoE)© 2008 Storage Networking Industry Association. All Rights Reserved. 38

    Scenario 2: FCoE Right & Wrong

    FCFCFC

    LosslessEthernet

    ClassicalEthernet

    FCoESwitch

    Internet

    FCoESwitches

    LosslessEthernet

    FCoE Flows

    Invalid FCoE Flows

  • 8/9/2019 FCoE - Fibre Channel Over Ethernet

    39/65

    Fibre Channel over Ethernet (FCoE)© 2008 Storage Networking Industry Association. All Rights Reserved. 39

    Summary

  • 8/9/2019 FCoE - Fibre Channel Over Ethernet

    40/65

    Fibre Channel over Ethernet (FCoE)© 2008 Storage Networking Industry Association. All Rights Reserved. 40

    Summary

    FCoE is a simple, efficient mechanism forencapsulating Fibre Channel in Ethernet frames

    FCoE is being standardized in INCITS FibreChannel (T11) technical committee

    Target completion is 2H08

    Maximum benefit of Fibre Channel is achieved:Evolutionary model of FC Switches and FC SANsEmphasis placed on capitalizing on the benefits ofConverged Enhanced Ethernet (CEE)

    Being discussed in the IEEE 802.1 standards working group

  • 8/9/2019 FCoE - Fibre Channel Over Ethernet

    41/65

    Thank You!

  • 8/9/2019 FCoE - Fibre Channel Over Ethernet

    42/65

    Fibre Channel over Ethernet (FCoE)© 2008 Storage Networking Industry Association. All Rights Reserved. 42

    Q&A / Feedback

    Please send any questions or comments on thispresentation to SNIA: [email protected]

    For additional information refer to

    http://www.t11.org/fcoe

    Many thanks to the following individualsfor their contributions to this tutorial.

    SNIA Education Committee

    Claudio DeSanti Howard Golds tein Walter DeyRobert Snively Suresh Vobbili setty Silvano GaiJoe Peliss ier John Hufferd

    mailto:[email protected]://www.t11.org/fcoehttp://www.t11.org/fcoemailto:[email protected]

  • 8/9/2019 FCoE - Fibre Channel Over Ethernet

    43/65

    Fibre Channel over Ethernet (FCoE)© 2008 Storage Networking Industry Association. All Rights Reserved. 43

    Appendix

    AdditionalRequirements

    FCoE Relation to ISOLayersFlows

    Additional TopologiesFSPF and STPFIP Considerationsand ActionsFIP DescriptorsPause vs. BB_Credit

  • 8/9/2019 FCoE - Fibre Channel Over Ethernet

    44/65

    Fibre Channel over Ethernet (FCoE)© 2008 Storage Networking Industry Association. All Rights Reserved. 44

    FCoE Goals/Requirements (part 3)

    FCoE must be a direct mapping of Fibre Channel overan Ethernet networkFCoE must be layered on top of Ethernet

    FSPF used to route FCoE packetsEthernet Spanning Tree (STP), MSTP, etc, is at a layer below

    FCoE to allow an evolutionary approach towardsconsolidation of fabrics

    The Fibre Channel N_Port, F_Port and E-Port constructs mustbe retained

    With FCoE, ports may be connected with Logical Ethernet Links – May pass through Ethernet switches – Identified by pairs of end point MAC addresses

    Physical Ethernet Links can replace physical FC LinksPhysical Ethernet Links can carry all Ethernet traffic, includingFCoE, but combined traffic needs the CEE capabilities

  • 8/9/2019 FCoE - Fibre Channel Over Ethernet

    45/65

    Fibre Channel over Ethernet (FCoE)© 2008 Storage Networking Industry Association. All Rights Reserved. 45

    FCoE Goals/Requirements (part 4)

    “Combo FCoE Switches ” may be built that supportnormal Ethernet traffic, FCoE traffic , & FC trafficThe FCoE solutions should appear as a Fibre Channel toa Fibre Channel experienced customer FCoE should keep the Fibre Channel operationsindependent from Ethernet forwarding

    Keeps management /Troubleshooting simpleCommon physical structures, different logical structures

    Based on Ethertype (Ethertype = FCoE)

    Storage Management should be unchanged

  • 8/9/2019 FCoE - Fibre Channel Over Ethernet

    46/65

    Fibre Channel over Ethernet (FCoE)© 2008 Storage Networking Industry Association. All Rights Reserved. 46

    FCoE Relation to ISO Layers

  • 8/9/2019 FCoE - Fibre Channel Over Ethernet

    47/65

    Fibre Channel over Ethernet (FCoE)© 2008 Storage Networking Industry Association. All Rights Reserved. 47

    Flows

    L i l F b i T l

  • 8/9/2019 FCoE - Fibre Channel Over Ethernet

    48/65

    Fibre Channel over Ethernet (FCoE)© 2008 Storage Networking Industry Association. All Rights Reserved. 48

    Logical Fabric Topology

    FCFCFC

    FCoESwitch

    FCoESwitch

    FCoESwitch

    LosslessEthernet

    LosslessEthernet

    A

    H1

    H2

    H3H4 S2

    EthernetDestination

    & Source

    EncapsulatedFC Frame

    D_IDS_ID

    FCoE-A MACFCoE-H2 MAC

    FC_ID for S1FC_ID for H2 FC_ID for S1FC_ID for H2 FC_ID for S1FC_ID for H2 FC_ID for S1FC_ID for H2

    FCoE-B MACFCoE-A MAC

    FCoE-C MACFCoE-B MAC

    FCoE-S1 MACFCoE-C MAC

    EthernetFC

    LogicalTransaction Path

    An FCoE Switch receives FCoE frames addressed to its FC-MAC address andforwards them based on the D_ID of the encapsulated FC frame

    An FCoE Switch rewrites the SA and DA of an FCoE frame

    LosslessEthernet

    LosslessEthernet

    S1

    Path #1 Path #2 Path #3 Path #4

    #1

    #2

    #3

    #4B

    C

  • 8/9/2019 FCoE - Fibre Channel Over Ethernet

    49/65

  • 8/9/2019 FCoE - Fibre Channel Over Ethernet

    50/65

    Fibre Channel over Ethernet (FCoE)© 2008 Storage Networking Industry Association. All Rights Reserved. 50

    FC Host to FCoE Storage

    FCFCFC

    FCoESwitch

    FCoESwitch

    Lossless Ethernet

    H1

    H2

    H3

    S1

    S3

    FCoE-S2 MACEthernet

    Destination& Source

    FCoE-A MACEncapsulated

    FC Frame

    D_ID

    S_ID

    FC_ID for S2

    FC_ID for H5

    FC_ID for S2

    FC_ID for H5

    FC_ID for S2

    FC_ID for H5

    FCoE-A MAC

    FCoE-C MAC

    EthernetSwitch

    EthernetSwitch

    EthernetSwitch

    EthernetSwitch

    H5S2

    EthernetSwitch

    EthernetFC

    LogicalTransaction Path

    Path #3 Path #2

    Path #1

    #1

    #2

    #3

    C A

  • 8/9/2019 FCoE - Fibre Channel Over Ethernet

    51/65

    Fibre Channel over Ethernet (FCoE)© 2008 Storage Networking Industry Association. All Rights Reserved. 51

    Additional Topologies

  • 8/9/2019 FCoE - Fibre Channel Over Ethernet

    52/65

    Fibre Channel over Ethernet (FCoE)© 2008 Storage Networking Industry Association. All Rights Reserved. 52

    Additional Topologies (1)

    FCoESwitch (A)

    FCoESwitch (B)

    FCoESwitch (E)

    FCoESwitch (F)

    LosslessEthernetSwitch

    LosslessEthernetSwitch

    FCFCFC

    Example of Topologies with Rack Mount servers

  • 8/9/2019 FCoE - Fibre Channel Over Ethernet

    53/65

    Fibre Channel over Ethernet (FCoE)© 2008 Storage Networking Industry Association. All Rights Reserved. 53

    Additional Topologies (2)

    FCoESwitch (B)

    FCoESwitch (E)

    FCoESwitch (F)

    LosslessEthernetSwitch

    LosslessEthernetSwitch

    FCFCFC

    FCoESwitch (A)

    Equivalent to Blade servers with N_Port_ID Virtualization (NPIV) SupportExample of Topologies with Blade Servers

  • 8/9/2019 FCoE - Fibre Channel Over Ethernet

    54/65

    Fibre Channel over Ethernet (FCoE)© 2008 Storage Networking Industry Association. All Rights Reserved. 54

    FSPF and STP

  • 8/9/2019 FCoE - Fibre Channel Over Ethernet

    55/65

    Fibre Channel over Ethernet (FCoE)© 2008 Storage Networking Industry Association. All Rights Reserved. 55

    FSPF & STP Concepts with FCoE (basic)

    FSPF & STP Concepts with FCoE (interconnected)

  • 8/9/2019 FCoE - Fibre Channel Over Ethernet

    56/65

    Fibre Channel over Ethernet (FCoE)© 2008 Storage Networking Industry Association. All Rights Reserved. 56

    FSPF & STP Concepts with FCoE (interconnected)

  • 8/9/2019 FCoE - Fibre Channel Over Ethernet

    57/65

  • 8/9/2019 FCoE - Fibre Channel Over Ethernet

    58/65

    Fibre Channel over Ethernet (FCoE)© 2008 Storage Networking Industry Association. All Rights Reserved. 58

    FIP Considerations and Actions

    d

  • 8/9/2019 FCoE - Fibre Channel Over Ethernet

    59/65

    Fibre Channel over Ethernet (FCoE)© 2008 Storage Networking Industry Association. All Rights Reserved. 59

    FCoE Discovery Considerations

    The Discovery phase of the FCoE Initialization Protocol (FIP)uses two types of messages, Solicitations and AdvertisementsThe FIP Discovery Phase helps define the FCF Ports that areavailable for the Link instantiation PhaseThe ENodes discover the FCF ports that can become VF_Portsand FCFs discover other FCF ports that can become VE_Ports

    ENodes Solici t (via Multicast) Advertisements from FCFs whilespecifying their capabilitiesIn response FCF’s Advertise their availability and capabilities backto the ENodesFCF’s Multicast their existent to other FCF

    The FIP Discovery phase exchanges solicitation and/or Advertisements between (HBA and/or FCF) “ FCoE Controllers”

    C i A i

  • 8/9/2019 FCoE - Fibre Channel Over Ethernet

    60/65

    Fibre Channel over Ethernet (FCoE)© 2008 Storage Networking Industry Association. All Rights Reserved. 60

    FCF Discovery Actions

    FCF supporting VE_Ports:Discovers other VE_Port capable FCF-MACs, connectedto the same Lossless Ethernet segment, by:

    Transmitting a multicast Solicitation to ‘All-FCFs’(with the FCF bit set to one)

    Receives back Jumbo Unicast Advertisements from VE capableMACs

    To verify the support of Ethernet Jumbo frames in the pathIn response to receipt of a Multicast to “All-FCFs” from anotherFCF

    Has its own VE capable MACs sends their own Jumbo Unicast Advertisements

    Instantiates VE_Port to VE_Port connections andExchanges FC ELP (Extended Link Protocol) and Fabricconfiguration (using Ethertype=FCoE) with the other

    FCFs VE_Port capable MACs

    EN d A i

  • 8/9/2019 FCoE - Fibre Channel Over Ethernet

    61/65

    Fibre Channel over Ethernet (FCoE)© 2008 Storage Networking Industry Association. All Rights Reserved. 61

    ENode ActionsWhen an ENode becomes operational:

    The ENode discovers the VF_Port capable FCF-MACsconnected to the same Lossless Ethernet segment

    Transmits a multicast Solicitation to ‘All-FCFs’ (with the FCF bit setto zero)Receives Jumbo unicast Advertisements from select compatibleVF_Port capable FCF-MACsStores the discovered FCF-MACs in an FCF list

    When an ENode receives an Advertisement that anew FCF is available, it may send a unicastSolicitation to it and receive a Jumbo unicast

    Advertisement in reply

    To verify the support of Ethernet Jumbo frames in the pathMay then perform FLOGIs (with Ethertype-FIP) to avendor specific subset of the FCF-MACs in the FCFlist

  • 8/9/2019 FCoE - Fibre Channel Over Ethernet

    62/65

    FIP D i (1)

  • 8/9/2019 FCoE - Fibre Channel Over Ethernet

    63/65

    Fibre Channel over Ethernet (FCoE)© 2008 Storage Networking Industry Association. All Rights Reserved. 63

    FIP Descriptors (1)

    FIP D i (2)

  • 8/9/2019 FCoE - Fibre Channel Over Ethernet

    64/65

    Fibre Channel over Ethernet (FCoE)© 2008 Storage Networking Industry Association. All Rights Reserved. 64

    FIP Descriptors (2)

    P BB C dit

  • 8/9/2019 FCoE - Fibre Channel Over Ethernet

    65/65

    Pause vs. BB_Credit

    Both mechanisms are used to avoid dropping framesWith different trade-offs

    The Pause mechanism requires at least the (2 x RTT x

    bandwidth) product on a link as buffer spaceBut allows Buffer handling in an arbitrary wayWell suited for networks with limited (bandwidth x delay) product(e.g. within the data center)

    The Pause frame is handled by the MAC layer Similar to the R_RDY handling by the FC-1 level

    The BB_Credit mechanism prevents loosing framesover any link

    But links go under-utilized if link credits (& buffers) are < thatneeded for (RTT x BW)

    Requires buffer handling in maximum frame size units