FCoE - Design Operations and Management 1

BRKSAN-2047

FCoE - Design, Operations and Management Best Practices

© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation_ID 2

Before We Get Started

Intermediate level session focused on Unified Data Centre Design using Fibre Channel over Ethernet

Prerequisites:basic understanding of Fibre Channel and Storage Designbasic understanding of Ethernet and LAN Designbasic understanding of the FCoE protocol and terminology

Other recommended sessions


AgendaWhy are we here?Background Information

DCB StandardFCoE Protocol InformationFCoE Building Blocks and Terminology

Design RequirementsClassical Ethernet + Classical Fibre Channel = ??

Single Hop DesignsMulti-Hop DesignsFCoE Deployment ConsiderationsQuestions


The Evolving Data Centre AccessThe Consolidated Nexus Edge Layer

slot 1slot 2slot 3slot 4slot 5slot 6slot 7slot 8

blade1blade2blade3blade4blade5blade6blade7blade8











Cor

e/A

ggre

gatio

n La

yer

Virt

ualiz

ed E

dge/

Acc

ess

Laye

r

The Access Layer is becoming more than just a port aggregatorEdge of the growing Layer 2 topology

Scaling of STP Edge PortsVirtual embedded switchesvPC and loop free designsLayer 2 Multi-Pathing (future)

Foundational element for Unified I/O ‘and’Unified Wire

DCB and Multi-Hop FCoE SupportEnhanced Multi-hop FCoE with E-NPV

Single Point for Access Management VN-Tag and Port Extension – Nexus 2000 (current) VSM and VN-Link (future)

FC



Understand the design requirements of a Unified Network

Be able to design single-hop Unified Networks available today which meet the demands of both SAN and LAN networks

Start the conversation between Network and Storage teams regarding consolidation and FCoE beyond the access layer

Understand the Operations and Management aspects of a Unified Network

Why are we here?Session Objectives







CEE (Converged Enhanced Ethernet) is an informal group of companies that submitted initial inputs to the DCB WGs.

Data Centre EvolutionIEEE DCB (Data Centre Bridging)

Feature / Standard Standards Status

Priority Flow Control IEEE 802.1Qbb (PFC)

PAR approved, Editor Claudio DeSanti (Cisco), draft 1.0 published

Bandwidth Management IEEE 802.1Qaz (ETS)

PAR approved, Editor Craig Carlson (Qlogic), draft 0.2 published

Data Center Bridging Exchange Protocol (DCBX)

This is part of:Bandwidth Management IEEE 802.1Qaz

** Nexus 5000 supports CEE-DCBX as well as previous generations (CIN-DCBX)


Packet

R_R

DY

Fibre ChannelFibre Channel

Transmit Queues Ethernet Link Receive Buffers

EightVirtualLanes

OneOne OneOne

TwoTwo TwoTwo

ThreeThree ThreeThree

FourFour FourFour

FiveFive FiveFive

SevenSeven SevenSeven

EightEight EightEight

SixSix SixSix

STOP PAUSE

B2B CreditsB2B Credits

Enables lossless Ethernet using PAUSE based on a COS as defined in 802.1pWhen link is congested, CoS assigned to “no-drop” will be PAUSEDOther traffic assigned to other CoS values will continue to transmit and rely on upper layer protocols for retransmissionNot only for FCoE traffic

Priority Flow ControlFCoE Flow Control Mechanism


Once feature fcoe is configured, 2 classes are made by default

Priority Flow ControlOperations Configuration – Switch Level

DCB Switch

DCB CNA Adapter

class-fcoe is configured to be no-drop with an MTU of 2158

Best Practice - use the default COS value of 3 for FCoE/no-drop trafficCan be changed through QOS class-map configuration


Checking the PFC settings on an interface

VL bmap = COS set for PFC

Priority Flow ControlVerifying Configurations

VL bmap Binary COS

1 00000001 02 00000010 14 00000100 28 00001000 316 00010000 432 00100000 564 01000000 6128 10000000 7

show interface priority-flow-control

Shows ports where PFC is configured, the COS value associated with PFC as well as the PAUSE packets received and sent on that port


Offered Traffic

t1 t2 t3

10 GE Link Realized Traffic Utilization

3G/s HPC Traffic3G/s

2G/s

3G/sStorage Traffic3G/s

3G/s

LAN Traffic4G/s

5G/s3G/s

t1 t2 t3

3G/s 3G/s

3G/s 3G/s 3G/s

2G/s

3G/s 4G/s 6G/s

Prevents a single traffic class of “hogging” all the bandwidth and starving other classesWhen a given load doesn’t fully utilize its allocated bandwidth, it is available to other classes Helps accommodate for classes of a “bursty” nature

Enhanced Transmission SelectionBandwidth Management


Enhanced Transmission SelectionBandwidth Management

Once feature fcoe is configured, 2 classes are made by defaultBy default, each class is given 50% of the available bandwidth

1Gig FC HBAs

1Gig Ethernet NICs

Traditional Server

A typical server has equal BW per traffic type

Best Practice : FCoE and Ethernet each receive 50%Can be changed through QoS settings when higher demands for certain traffic exist (i.e. HPC traffic, more Ethernet NICs)


Discovers DCB capabilities of peer devicesNegotiates Ethernet capability’s – PFC, ETS, CoS valuesSimplifies management of DCB nodes

Allows for configuration and distribution of parameters from one node to another

Responsible for Logical Link Up/Down signaling of Ethernet and Fibre ChannelUses Link Layer Discovery Protocol (LLDP) defined by 802.1AB to exchange and discover DCB capabilitiesDCBX negotiation failures result in:

per-priority-pause not enabled on CoS valuesvfc not coming up – when DCBX is being used in FCoE environment

Data Center Bridging eXchangeControl Protocol







FCoE Benefits

Mapping of FC frames over Ethernet

Enables FC to run on a lossless Data Center Ethernet network

Wire Server Once

Fewer cables and adapters and switches

Software Provisioning of I/O

Interoperates with existing SANs

No gateway—stateless

Standard – June 3, 2009FibreChannel

Ethernet

Fibre Channel over Ethernet


From a Fibre Channel standpoint it’sFC connectivity over a new type of cable called… Ethernet

From an Ethernet standpoints it’sYet another ULP (Upper Layer Protocol) to be transported

FC-0 Physical Interface

FC-1 Encoding

FC-2 Framing & Flow Control

FC-3 Generic Services

FC-4 ULP Mapping

Ethernet Media Access Control

Ethernet Physical Layer

FC-2 Framing & Flow Control

FC-3 Generic Services

FC-4 ULP Mapping

FCoE Logical End Point

Fiber Channel over Ethernet Protocol Mapping


Unified FabricFibre Channel over Ethernet (FCoE)

FCoE is Fibre Channel at the host and switch level

Same Operational Model Same Operational Model

Same Techniques ofTraffic ManagementSame Techniques ofTraffic Management

Same Managementand Security ModelsSame Managementand Security Models

Easy to UnderstandEasy to Understand

Completely based on the FC model

Same host-to-switch and switch-to-switch behavior of FC

E.g., in order delivery or FSPF load balancing

WWNs, FC-IDs, hard/soft zoning, DNS, RSCN

Aligned with the FC-BB-4 Model,

Standardizedin FC-BB-5

Aligned with the FC-BB-4 Model,

Standardizedin FC-BB-5


Both Protocols Have…• Two different Ethertypes• Two different frame formats• Both are defined in FC-BB-5

FCoE itself Is the data plane protocol

It is used to carry most of the FC frames and all the SCSI traffic

Uses Fabric Assigned MAC address (dynamic)

FCoE itself Is the data plane protocol

It is used to carry most of the FC frames and all the SCSI traffic

Uses Fabric Assigned MAC address (dynamic)

FIP (FCoE Initialization Protocol)It is the control plane protocol

It is used to discover the FC entities connected to an Ethernet cloud

It is also used to login to and logout from the FC fabric

FIP (FCoE Initialization Protocol)It is the control plane protocol

It is used to discover the FC entities connected to an Ethernet cloud

It is also used to login to and logout from the FC fabric

http://www.cisco.biz/en/US/prod/collateral/switches/ps9441/ps9670/white_paper_c11-560403.html http://www.cisco.biz/en/US/prod/collateral/switches/ps9441/ps9670/white_paper_c11-560403.html

Fiber Channel over EthernetData and Control plane


Protocol used by FCoE capable devices to discover other FCoE capable devices within the Ethernet Cloud

Enables FCoE adapters (CNAs) to discover FCoE switches (FCFs) on a VLAN (the FCoE VLAN)Establishes a virtual link with between the adapter and FCF or between two FCFs (VE_ports) – accomplished with a FLOGI

FIP frames use a different Ethertype from FCoE frames making FIP-Snooping by DCB capable Ethernet bridgesBuilding foundation for future multi-hop FCoE topologies

Multi-hop refers to FCoE extending beyond a single “hop” or “access” switchToday, “multi-hop” is achievable with a Nexus 4000 (FIP Snooping Bridge) connected to Nexus 5000 (FCF)

Fibre Channel over Ethernet ProtocolFCoE Initialization Protocol (FIP)


Step 1: FCoE VLAN DiscoveryFIP sends out a multicast to

ALL_FCF_MAC address looking for the FCoE VLAN

FIP frames use the native VLAN

Step 2: FCF DiscoveryFIP sends out a multicast to the

ALL_FCF_MAC address on theFCoE VLAN to find the FCFs answering for that FCoE VLAN

FCF’s responds back with their MAC address

Step 3: Fabric LoginFIP sends a FLOGI request to the

FCF_MAC found in Step 2Establishes a virtual link between

host and FCF

Enode Initiator

FCoE SwitchFCF

VLANDiscover

y

FLOGI/FDISC

FLOGI/FDISC Accept

FC Command

FC Command Responses

FCoEInitialization

Protocol (FIP)

FCoEProtocol

VLANDiscovery

FCFDiscovery

SolicitationFCF

DiscoveryAdvertisement

Fiber Channel over Ethernet ProtocolFCoE Initialization Protocol (FIP)

** FIP does not carry any Fibre Channel frames


The FCoE VLAN is manually configured on the Nexus 5000

The FCF-MAC address is configured on the Nexus 5000 by default once feature fcoe has been configured

This is the MAC address returned in step 2 of the FIP exchangeThis MAC is used by the host to login to the FCoE fabric


** FIP does not carry any Fibre Channel frames



Step 3 - login process: show flogi database and show fcoe database show the logins and associated FCiDs, xWWNs and FCoE MAC addresses







FCF : Fibre Channel Forwarder (Nexus 5000, Nexus 7000, MDS 9000)FPMA : A unique MAC address that is assigned by an FCF to a single EnodeEnode : a Fiber Channel node that is able to transmit FCoE frames using one or more ENode MACs.FCoE Pass-Through : a DCB device capable of passing FCoE frames to an FCF (i.e. FIP-Snooping)

FIP Snooping BridgeEthernet N-Port Virtualizer

Single hop FCoE : running FCoE between the host and the first hop access level switchMulti-hop FCoE : the extension of FCoE beyond a single hop into the Aggregation and Core layers of the Data Centre Network

FCF

E_Node

FCoE Building BlocksThe Acronyms Defined

© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 26Session_IDPresentation_ID

Enode MAC AddressFibre Channel over Ethernet Addressing Scheme

Enode MAC assigned for each FCIDEnode MAC composed of a FC-MAP and FCID

FC-MAP is the upper 24 bits of the Enode’s MACFCID is the lower 24 bits of the Enode’s MAC

FCoE forwarding decisions still made based on FSPF and the FCID within the Enode MAC

FC FabricFC Fabric

Domain ID

FC-MAP(0E-FC-xx)

FC-ID7.8.9

FC-MACAddress

FC-MAP(0E-FC-xx)

FC-ID10.00.01

Fibre Channel FCID Addressing


Fibre Channel Drivers

Ethernet Drivers

Operating System

PCIe

Ethernet

Fibre Channel

10GbE

10GbE

Link

Ethernet Driver bound to Ethernet NIC PCI address

FC Driver bound to FC

HBA PCI address

Replaces multiple adapters per server, consolidating both Ethernet and FC on a single interfaceAppears to the operation system as individual interfaces (NICs and HBAs)First Generation CNAs from support PFC and CIN-DCBXSecond Generation CNAs support PFC, CEE-DCBX as well as FIP

Single chip implementation

FCoE Building BlocksConverged Network Adapter


FCF (Fibre Channel Forwarder) is the Fibre Channel forwarding element inside an FCoE switch

Fibre Channel logins (FLOGIs) happens at the FCFConsumes a Domain ID

FCoE encap/decap happens within the FCFForwarding based on FC information

Ethport

Ethport

Ethport

Ethport

Ethport

Ethport

Ethport

Ethport

Ethernet Bridge

FCport

FCport

FCport

FCport

FCF

FCoE SwitchFC Domain ID : 15FC Domain ID : 15

FCoE Building BlocksFibre Channel Forwarder


VE_Port

VF_Port

VF_Port

VE_Port

VN_Port

VN_Port

Fibre Channel over Ethernet Switch

E_NPV SwitchVF_Port VNP_PortFCF

Switch

End Node

End Node

FCoE Switch : FCF

**Available NOW

**EagleHawk + Timeframe**EagleHawk Timeframe

FCoE Building BlocksFCoE Port Types


Unified I/O – using Ethernet as the transport medium in all network environments -- no long needing separate cabling options for LAN and SAN networks

Unified Wire – a single DCB Ethernet link actively carrying both LAN and Storage (FC/FCoE/NAS/iSCSI) traffic simultaneouslyUnified Dedicate Wire -- a single DCB Ethernet link capable of carrying all traffic types but actively dedicated to a single traffic type for traffic engineering purposes

Unified Fabric – An Ethernet Network made up of “Unified Wires”everywhere: all protocols – network and storage –transverse all links simultaneously

FCoE Building BlocksThe New Buzzzword…”Unified”


CNA

Unified Dedicated Wire

Unified Dedicated Wire

L2

L3

Core

Aggregation

Shared Access

Fabric ‘A’ Fabric ‘B’

Unified WireUnified Wire

Unified Wire to the access switchcost savings in the reduction of required equipment“cable once” for all servers to have access to both LAN and SAN networks

Unified Dedicated Wire from access to aggregation

separate links for SAN and LAN traffic - both links are same I/O (10GE)advanced Ethernet features can be applied to the LAN linksmaintains fabric isolation

FCoE Building BlocksUnfied Wire vs Unified Dedicated Wire


L2

L3

Core

Aggregation

Access

Virtual Port-Channel (VPC)

Ethernet an Storage traffic EVERYWHERE

Ethernet an Storage traffic EVERYWHERE

A single networkAll links carry all types of traffic simultaneously

all/any Storage and Network protocols

Possible reduction of equipment leading to cost savingsAbolition of Fabric A and Fabric B

Single SAN fabric with redundant fabric services

FCoE Building BlocksThe Unified Fabric - Definition


Unified Technology

L2

L3

Core

Aggregation

Access

Core

Edge

Fabric ‘A’ Fabric ‘B’

Ether-channel Multi-pathing


NIC /

CNA

CNA

Fibre Channel over Ethernet SAN

Fibre Channel over Ethernet SAN

Native Ethernet LANNative Ethernet LAN

LAN and SAN networks share the same Unified I/O building blocks: switches and cablingmaintains operations, management and troubleshooting



DCB StandardFCoE Protocol InformationFCoE Building Blocks




Ethernet is non-deterministic.Flow control is destination-basedRelies on TCP drop-retransmission / sliding window

Fibre-Channel is deterministic.Flow control is source-based (B2B credits)Services are fabric integrated (no loop concept)

The Design RequirementsEthernet vs Fibre Channel


Ethernet/IPGoal is to provide any-to-any connectivity

Unaware of packet loss – relies on ULPs for retransmission and windowingProvides the transport without worrying about the servicesEast-west vs north-south traffic ratios are undefinedServices provided by upper layers

Network design has been optimized for:Control protocol interaction (STP, OSPF, EIGRP, L2/L3 boundary, …)High Availability from a transport perspective by connecting nodes in mesh architecturesHigh Availability for Services is implemented seperately

??

?

?

????

?

?

??

Switch Switch

Switch

?

Client/Server Relationships are not pre-defined

? ?

?

Fabric topology and traffic flows are highly flexible

The Design RequirementsClassical Ethernet


Servers typically dual homed to two or more access switches

LAN switches have redundant connections to the next layer

Distribution and Core can be collapsed into a single box

L2/L3 boundary typically deployed in the aggregation layer

Spanning tree or advanced L2 technologies (vPC) used to prevent loops within the L2 boundaryL3 routes are summarized to the core

Services deployed in the L2/L3 boundary of the network (load-balancing, firewall, NAM, etc)

L2

L3

Core

Aggregation

Access



Outside Data Center “cloud”

STP

STP

The Design RequirementsLAN Design – Access/Aggregation/Core


Fibre Channel SANTransport and Services are on the same layer in the same devicesWell defined end device relationships (initiators and targets)Does not tolerate packet drop – requires lossless transportOnly north-south traffic, east-west traffic mostly irrelevant

Network designs optimized for Scale and Availability

High availability of network services provided through dual fabric architecture

SAN ‘A’ and SAN ‘B’ : physically separate and redundant fabrics

Strict change isolation - end to end driver certification

Client/Server Relationships are

pre-defined

I(c)

I(c)T(s)

T2

I5

I4I3I2I1

I0

T1T0

Switch Switch

Switch

DNS FSPF

ZoneRSCN DNS

FSPF Zone

RSCN

DNS

Zone

FSPF

RSCN

Fabric topology, services and traffic flows are structured

The Design RequirementsClassical Fibre Channel


“Edge-Core” Topology

Servers connect to the edge switches

Storage devices connect to one or more core switches

Core switches provide stroage services to one or more edge switches, thus serviceing more servers in the fabric

ISLs have to be designed so that overall fan-in ratio of servers to storage and overall end-to-end oversubscription are maintained

HA achieved in two physically separate, but identical, redundant SAN fabrics

FC

CoreCore CoreCore

The Design RequirementsSAN Design – Two Tier Topology


“Edge-Core-Edge” Topology

For environments where future growth of the network has the number of storage devices exceeding the number of ports available at the core switch

A set of edge switches dedicated to server connectivity and another set of dedicated for storage devices

Extra edge can also be “services edge”for advanced network services

Core is for transport only, rarely accommodates end nodes

HA achieved with dual fabrics

FC

CoreCore CoreCore

The Design RequirementsSAN Design – Three Tier Topology


??

?

?

????

?

?

??

Switch Switch

Switch

?

T2

I5

I4I3I2I1

I0

T1T0

Switch Switch

Switch

DNS FSPF

ZoneRSCN DNS

FSPF Zone

RSCNDNS

Zone

FSPFRSCN

Question Do we build a FC network on top of an Ethernet Cloud? Or and Ethernet Network on top of a Fibre Channel Fabric?

Unified Fabric design has to incorporate the super-set of requirements

Network -- Lossless ‘and’ Lossfull Topologies Transport – undefined (any-to-any) ‘and’defined (one-to-one)High Availability – redundant network topology (mesh/full mesh) ‘and’ physically separate redundant fabricsBandwidth – FC fan-in and oversubscription ratios ‘and’ Ethernet oversubscriptionSecurity – FC controls (zoning, port security, …) ‘and’ IP controls (CISF, ACL, …) Manageability and Visibility – Hop by hop visibility for FC ‘and’ the cloud for Ethernet

The Design RequirementsClassical Ethernet + Classical Fibre Channel == ??


Can’t we just fold down the dotted line??FC

Core

Core

Core

Core

L2

L3

Core

Aggregation

Access



Outside Data Center

“cloud”

STP

STP

The Design RequirementsClassical Ethernet + Classical Fibre Channel == ??

Fold

Her

eFo

ld H

ere


To expand the reach of Unified I/OBlade Server environments (e.g. Nexus 4000 / UCS)Core and backbone links and devices

SAN scalabilityBuild-up the edge, from 20% attach-rate up to 100%Allow LAN and SAN to scale independently

To introduce the support for native FCoE Storage arrays

Preserve SAN design best practicesOversubscription, Fan-in ratios, hop count practices honored

Preserve SAN and LAN management modelsDeterministic management of FC flows through all devices - No opaque LAN “clouds” transporting SAN traffic

The Design RequirementsFCoE Design Objectives





Single Hop DesignsMulti-Hop DesignsFCoE Deployment ConsidersationsQuestions


Host connected over unified wire to first hop access switch

Access switch (Nexus 5000) is the FCFFibre Channel ports on the access switch can be in NPV or Switch mode for native FC traffic

DCBX is used to negotiate the enhanced Ethernet capabilitiesFIP is use to negotiate the FCoE capabilities as well as the host login processFCoE runs from host to access switch FCF – native Ethernet and native FC break off at the access layer

FC

CNACNA

FC FabricFC Fabric

ENodeENode

Target

Target

Ethernet FabricEthernet Fabric

DCB capable Switchacting as an FCF

DCB capable Switchacting as an FCF

Unified WireUnified Wire

Single Hop DesignToday’s Solution


The first phase of the Unified Fabric evolution design focused on the fabric edge

Unified the LAN Access and the SAN Edge by using FCoE

Consolidated Adapters, Cabling and Switching at the first hop in the fabrics

The Unified Edge supports multiple LAN and SAN topology options

Virtualized Data Center LAN designs

Fibre Channel edge with direct attached initiators and targets

Fibre Channel edge-core and edge-core-edge designs

Fibre Channel NPV edge designs The Unified Edge

Fabric A Fabric BLAN Fabric

FC

FCoEFC

LAN Access/SAN

Edge

Single Hop DesignUnified Wire at the Access

Nexus 5000 FCF-A

Nexus 5000 FCF-B


Fibre Channel Drivers

Ethernet Drivers

Operating System

PCIe

Ethernet

Fibre Channel

10GbE

10GbE

Link

Converged Network Adapter (CNA) presents two PCI address to the Operating System (OS)

OS loads two unique sets of drivers and manages two unique application topologies

Server participates in both topologies seperately

Two stacks and thus two views of the same ‘unified wire’

SAN Multi-Pathing provides failover between two fabrics (SAN ‘A’ and SAN ‘B’)

NIC Teaming provides failover within the same fabric (VLAN)

Ethernet Driver bound to

Ethernet NIC PCI address

FC Driver bound to FC

HBA PCI address

Unified Wire shared by both

FC and IP topologies

Nexus Unified Edge supports both FC and IP

topologies

Nexus Edge participates in both distinct FC and IP Core

topologies

Nexus 5000 FCF-A

Nexus 5000 FCF-B

Single Hop DesignThe CNA Point of View


Fabric A

Direct Attach Topologies

CEE-DCBX

Generation 1 CNA

CIN-DCBX

Generation 2 CNA

Fabric BLAN Fabric

VN

VF Direct attach VN_Port to

VF_Port

In this first phase we were limited to direct attached CNAs at the access

Generation 1 CNA

Utilized Cisco, Intel, Nuova Data Center Bridging Exchange protocol (CIN-DCBX)

Only supports direct attachment of an ‘VN_Port’ to an ‘VF_Port’ over the ‘unified wire’

Generation 2 CNA

Utilizes Converged Enhanced Ethernet Data Center Bridging Exchange protocol (CEE-DCBX)

Utilizes FCoE Initialization Protocol (FIP) as defined by the T.11 FC-BB-5 specification

Supports both direct and multi-hop attachment

Single Hop DesignThe CNA Point of View

Nexus 5000 FCF-A

Nexus 5000 FCF-A


Fabric A

Direct Attach Topologies

Fabric BLAN Fabric

VN

VF Direct attach VN_Port to

VF_Port

Physical link is brought up (today requires 10GE)

DCBX negotiation – discovers DCB capable devices and negotiates lossless Ethernet capabilities/configs

FIP Process – discovery and negotiation of FCoE devices and characteristics

FCoE VLAN Discovery

FCF Discovery on the specific FCoE VLAN

Fabric Login - builds the logical wire from the end node to the FCF (VN_port to VF_port)

FCoE traffic flows from host to target; LAN traffic flows

Single Hop DesignAttaching an Initiator

Nexus 5000 FCF-A

Nexus 5000 FCF-B


VLAN 10,30

VLAN 10,20

Maintaining the two distinctedge/access topologies

Isolated SAN edge switches: SAN ‘A’ and SAN ‘B’

LAN Access switches connected to the same LAN fabric and carrying the same VLANs

Server participates in both topologies but may have different High Availability approaches

SAN Multi-Pathing provides failover between two fabrics

NIC Teaming provides failover to the same fabric (VLAN)

VSAN 2

VLAN 10 VSAN 3

Three concurrent topologies:LAN, SAN ‘A’ and SAN ‘B’


FC

FCoE

Single Hop DesignTwo Distinct Topologies

Nexus 5000 FCF-A

Nexus 5000 FCF-B


VLAN 10,30

VLAN 10,20

A VLAN is dedicated for every VSAN in the fabric

FIP is used to discover the FCoE VLAN and signal it to the hosts

Trunking not required on the host driver – all FCoE frames are tagged by the CNA

FCoE VLANs must not be configured on Ethernet links that are not designate for FCoE

Maintains isolated edge switches for SAN ‘A’ and ‘B’ and separate LAN switches for NIC 1 and NIC 2 (standard NIC teaming)

! VLAN 20 is dedicated for VSAN 2 FCoE traffic(config)# vlan 20 (config-vlan)# fcoe vsan 2

VSAN 2

STP Edge Trunk


Nexus 5000 FCF-A

Nexus 5000 FCF-B

VSAN 3

Single Hop DesignThe FCoE VLAN


VLAN 10,30

VLAN 10,20

In order to maintain the integrity of FC forwarding over FCoE, FCoE VLANs are treated differently than LAN VLANs

No flooding, MAC learning, broadcasts, etc.

The FCoE VLAN must not be configured as a native VLAN

FIP uses native VLAN

Separate FCoE VLANs must be used for FCoE in SAN-A and SAN-B

Unified Wires must be configured as trunk ports and STP edge ports

! VLAN 20 is dedicated for VSAN 2 FCoE traffic(config)# vlan 20 (config-vlan)# fcoe vsan 2

VSAN 2

STP Edge Trunk


Nexus 5000 FCF

Nexus 5000 FCF

VSAN 3

Single Hop DesignThe FCoE VLAN


VLAN 10,30

VLAN 10,20

FCoE Fabric ‘A’ will have a different VLAN topology than FCoE Fabric ‘B’ which are different from the LAN Fabric

PVRST+ allows unique topology per VLAN

MST requires that all switches in the same Region have the same mapping of VLANs to instances

MST does not require that all VLANs be defined in all switches

A separate instance must be used for FCoE VLANs

Recommended: three seperate instances – native Ethernet VLANs, SAN ‘A’ VLANs and SAN ‘B’ VLANs

spanning-tree mst configurationname FCoE-Fabricrevision 5instance 5 vlan 1-19,40-3967,4048-4093instance 10 vlan 20-29instance 15 vlan 30-39


VSAN 3VSAN 2

VLAN 10

Nexus 5000 FCF-A

Nexus 5000 FCF-B

Single Hop DesignThe FCoE VLAN and STP


Optimal layer 2 LAN design often leverages Multi-Chassis Etherchannel (MCEC)

Nexus utilizes Virtual Port Channel (vPC) to enable MCEC either between switches or to 802.3ad attached servers

MCEC provides network based load sharing and redundancy without introducing layer 2 loops in the topology

MCEC results in diverging LAN and SAN high availability topologies

FC maintains separate SAN ‘A’ and SAN ‘B’ topologiesLAN utilizes a single logical topology Direct Attach vPC Topology

MCEC

vPC Peers

vPC Peer Link


Nexus 5000 FCF-A

Nexus 5000 FCF-B

Single Hop DesignUnified Wires and MCEC


In vPC enabled topologies in order to ensure correct forwarding behavior for SAN traffic specific design and forwarding rules must be followed

With the NX-OS 4.1(3) releases a ‘vfc’ interface can only be associated with a vPC etherchannel with one (1)CNA port attached to each edge switchWhile the port-channel is the same on N5K-1 and N5K-2, the FCoE VLANs are differentvPC configuration works with Gen-2 FIP enabled CNAs ONLYFCoE VLANs are ‘not’ carried on thevPC peer-linkFCoE and FIP ethertypes are ‘not’forwarded over the vPC peer link

Direct Attach vPC Topology

VLAN 10,30

VLAN 10,20STP Edge Trunk

VLAN 10 ONLY HERE!


Nexus 5000 FCF-A

Nexus 5000 FCF-B

Single Hop DesignUnified Wires and MCEC


Nexus 5000 FCF-B

Nexus 5000 FCF-A

VLAN 10,30VLAN 10,20

VLAN 10,20,30

Dual CNA (FC initiator) connected via an Etherchannel to a single edge switch is unsupported

A ‘vfc’ interface can only be bound to a port channel with one local interface

Not consistent with Fibre Channel High Availability design requirements (No isolation of SAN ‘A’ and SAN ‘B’)

If SAN design evolves to a shared physical with only VSAN isolation for SAN ‘A’ and ‘B’ this ‘could’ change (currently this appears to be a big ‘if’)

ISLs between the Nexus 5000 access switches breaks SAN HA requirements

Single homed dual CNA Direct Attach

Topology


Single Hop DesignUnsupported Topologies

© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation_ID 57© 2009 Cisco Systems, Inc. All rights reserved.Presentation_ID © 2009 Cisco Systems, Inc. All rights reserved.Presentation_ID 5757

32 server facing 10Gig/FCoE ports

T11 standard based FIP/FCoE support on all ports

8 10Gig/FCoE uplink ports for connections to the Nexus 5000

Management and configuration handled by the Nexus 5000

Support for Converged Enhanced Ethernet including PFC

Part of the Cisco Nexus 2000 Fabric Extender family

FEX-2232Remote Line Card of

the Nexus 5000

Single Hop DesignIntroduction of 10Gig/FCoE Fabric Extender

Nexus 2232


Parent Switch: Acts as the combined Supervisor and Switching Fabric for the virtual switch

Fabric Links: Network Interface Forts (NIFs) Extends the Switching Fabric to the remote line card (Connects Nexus 5000 to Fabric Extender)

Host Interfaces (HIF)

Fabric connectivity between Nexus 5000 and Nexus 2000 (FEX) can leverage either pinning or port-channels

Nexus 5000

FEX100 FEX101

dc11-5020-1# show interface fex-fabric Fabric Fabric Fex FEX

Fex Port Port State Uplink Model Serial ---------------------------------------------------------------100 Eth1/17 Active 1 N2K-C2148T-1GE JAF1311AFLL100 Eth1/18 Active 2 N2K-C2148T-1GE JAF1311AFLL100 Eth1/19 Active 3 N2K-C2148T-1GE JAF1311AFLL100 Eth1/20 Active 4 N2K-C2148T-1GE JAF1311AFLL101 Eth1/21 Active 1 N2K-C2148T-1GE JAF1311AFMT101 Eth1/22 Active 2 N2K-C2148T-1GE JAF1311AFMT

Single Hop DesignIntroduction of 10Gig/FCoE Fabric Extender


Nexus 223210GE FEX

Nexus 223210GE FEX

SAN BSAN AFEX-2232 extends the reach of 10Gig Ethernet/FCoE to distributed line card (ToR)

• Support for up to 384 10Gig/FCoE attached hosts managed by a single Nexus 5000 (FCS number may vary)

• Nexus 5000 is the FCF or can be in a FCoE pass-through mode (when supported)

• In the presence of FCoE -- Nexus 2232 needs to be single homed to upstream Nexus 5000 (straight through N2K) to ensure SAN ‘A’and SAN ‘B’ isolation

Nexus 223210GE FEX

Nexus 223210GE FEX

Nexus 5000

Nexus 5000

Requires FIP enabled CNAs

TE

TE

Single Hop DesignExtending the Unified Access


Nexus 223210GE FEX

Nexus 223210GE FEX

SAN BSAN A

Fabric Links Option 1: Single

Homed Port Channel

Fabric Links Option 2: Static

Pinned

Server Option 1: FCoE on

individual links. Ethernet traffic is Active/Standby

• Server Ethernet driver connected to the FEX in NIC Teaming (AFT, TLB) or with vPC (802.3ad)

• FCoE runs over vPC member port with a single link from server to FEX

• FEX single homed to upstream Nexus 5000

• FEX fabric links can be connected to Nexus 5000 with individual links (static pinning) or a port channel

• oversubscribed 4:1

• Consistent with separate LAN Access and SAN Edge Topologies

Server Option 2: FCoE on a vPC

member PC with a single link

Nexus 223210GE FEX

Nexus 223210GE FEX


Single Hop DesignExtending the FCoE Edge – Nexus 2232

Nexus 5000 FCF-A Nexus 5000

FCF-B


Nexus 223210GE FEX

Nexus 223210GE FEX

SAN BSAN A

Fabric Links: vPC Port Channel

Nexus 2232 can not be configured in a dual homed configuration (vPC between two N5K) when configured to support FCoE attached servers

MCEC Port Channel will not keep SAN ‘A’ and San ‘B’ traffic isolated

Nexus 2000 not supported with dedicated FCoE and dedicated IP upstream fabric links

Nexus 2232 can only currently be connected to the Nexus 5000 when configured to support FCoE attached servers

Nexus 7000 will support Nexus 2000 in Ethernet only mode in CY2010 (support for FCoE on FEX targeted for CY2011 on next generation N7K line cards)

Nexus 223210GE FEX

Nexus 223210GE FEX

Nexus 5000Nexus 5000 Nexus 7000Nexus 7000

Single Hop DesignExtending the FCoE Edge – Nexus 2232







What is NPIV?N-Port ID Virtualization (NPIV) provides a means to assign multiple FCIDs to a single N_Port

allows multiple applications to share the same Fiber Channel adapter port

different pWWN allows access control, zoning, and port security to be implemented at the application level

usage applies to applications such as VMWare, MS Virtual Server and Citrix

Application Server FC NPIV Core Switch

Email

Web

File Services

Email I/ON_Port_ID 1

Web I/ON_Port_ID 2

File Services I/ON_Port_ID 3

F_Port

F_PortF_Port

N_PortN_Port


What is NPV?N-Port Virtualizer (NPV) utilizes NPIV functionality to allow a “switch” to act like aserver performing multiple logins through a single physical link

Physical servers connected to the NPV switch login to the upstream NPIV core switch

Physical uplink from NPV switch to FC NPIV core switch does actual “FLOGI”Subsequent logins are converted (proxy) to “FDISC” to login to upstream FC switch

No local switching is done on an FC switch in NPV mode

FC edge switch in NPV mode Does not take up a domain ID

Scalability will be dependent on FC “login” limitation (MDS is ~10K per fabric)

Nexus 5000, MDS 91xx, MDS blade switches, UCS Fabric Interconnect FC NPIV Core Switch

Eth1/1

Eth1/2

Eth1/3

Server1N_Port_ID 1

Server2N_Port_ID 2

Server3N_Port_ID 3

F_Port

N-Port

F-Port

F-PortNP-Port


FCFFCF

DCB + FIP Snooping

Bridge

DCB + FIP Snooping

Bridge

What design considerations do we have when extending FCoE beyond the Unified Edge?

High Availability for both LAN and SAN

Oversubscription for SAN and LAN

Ethernet Layer 2 and STP design

Where does Unified Wire make sense over Unified Dedicated Wire?

Unified Wire provides for sharing of a single link for both FC and Ethernet traffic


Multi - Hop DesignConsiderations for FCoE Multi-hop


FCFFCF

DCB + FIP Snooping

Bridge

DCB + FIP Snooping

Bridge

Growing FCoE fabrics is achived by connecting multiple FCoE capable devices together

An FCF contains a domain ID


Multi - Hop DesignThe Need for FCoE Pass Through


Multi-hop FCoE networks allow for FCoE traffic to extend past the access layer (first hop)

In Multi-hop FCoE the role of a transit Ethernet bridge needs to be evaluated

Avoid Domain ID exhaustionEase management

FIP Snooping is a minimum requirement suggested in FC-BB-5

Ethernet NPV (E-NPV) is a new capability intended to solve a number of design and management challenges

FCFFCF

FCFFCF

SAN BSAN A

DCB Capable Ethernet Switch




VN

VN

VF

VF

Multi - Hop DesignFCoE Pass-through options


What is FIP-Snooping?Efficient, automatic configuration of ACLs used to lock down the forwarding path accomplished by snooping FIP packets going from CNA to FCF

Why FIP-Snooping?Security - Protection from MAC Address spoofing of FCoE end devices (ENode)Fibre Channel links are Point-to-PointEthernet bridges can utilize ACLs to provide the equivalent path control (equivalent of point-to-point)

Support for FIP-Snooping?Nexus 4000 (Blade switch for IBM BC H)

Multi - Hop DesignFIP-Snooping


FCFFCF

FCF MAC 0E.FC.00.DD.EE.FF

FIP Capable Multi-Hop Topology

FIP Snooping

FIP Snooping

ENodeENode

Spoofed MAC 0E.FC.00.DD.EE.FF

ENode MAC 0E.FC.00.07.08.09

FIP Snooping – Nexus 4000Security (Protection from MAC Address spoofing of FCoE end devices “ENode”)Fibre Channel links are Point-to-PointEthernet bridges can utilize ACLs to provide the equivalent path control (equivalent of point-to-point)FIP protocol allows for efficient automatic configuration of the ACLs necessary to lock down the forwarding path (FIP Snooping)

Ethernet-NPV (E-NPV) – FutureIntelligent proxying FIP functions between a CNA and an FCFAdded control to FCF logins/mappings and load-balancing

SANMulti - Hop DesignExtending FCoE with FIP Snooping


On the control plane (FIP ethertype), an Ethernet NPV bridge improves over a "FIP snooping bridge" by intelligently proxying FIP functions between a CNA and an FCF

- takes control of how a live network will build FCoE connectivity- makes the connectivity very predictable, without the need for an FCF at the next hop from the CNA

On the data plane (FCoE ethertype), an “Ethernet NPV bridge” offers more ways to engineer traffic between CNA-facing ports and FCF-facing ports

An “Ethernet NPV bridge” knows nothing about Fibre Channel, and can’t parse packets with FCoE ethertype

Multi - Hop DesignEthernet NPV Bridge


Proxys FIP functions between a CNA and an FCFFCoE VLAN configuration and assignment

FCF Assignment

E-NPV load balance logins from the CNAs evenly across the available FCF uplink ports

E-NPV will take VSAN into account when mapping or ‘pinning’logins from a CNA to an FCF uplink

Operations and management process are in line with today’s SAN-Admin practicesSimilar to NPV in a native Fibre Channel network

**Name subject to change

Multi - Hop DesignEthernet NPV Bridge


FC

Target

TargetFABRIC A

E_NPV bridgeE_NPV bridge

FCFFCFDomain ID and FC-MAP come from the FCF

FCoE Pass – through deviceAll FCoE Switching is performed at the upstream FCFAddressing is pass out by the upstream FCF

more FCoE connectivity to hosts without

Running into the domain ID issueLess-expensiveConsistent management

E_NPV is the “FIP-Snooping Plus”

FLO

GI

FLO

GI

VN

E_NPV does not consume a domain ID

E_Node MAC Address

VF

VF

VNP

Multi - Hop DesignEthernet NPV - Enode Login Process


SAN BSAN ANexus 4000 is a Unified Fabric capable Blade Switch

DCB enabled

FIP Snooping Bridge

Dual Topology requirements for FCoE multi-hop

Servers IP connection to the Nexus 4000 is Active/Standby

MCEC is not currently supported from blade server to Nexus 4000

Options 1: Unified Dedicated Wires from Nexus 4000 to Nexus 5000

Options 2: Single Unified Wire Port Channel from Nexus 4000 to Nexus 5000

Option 2: Single Homed Unified

Wire

Mezzanine Converged Network

Adapter

Option 1:Unified

Dedicated Wire

PCIe

Ethernet

Fibre Channel

10GbE

10GbE

Link

Multi - Hop DesignExtending FCoE with FIP Snooping

Nexus 5000 FCF-A

Nexus 5000 FCF-B

Nexus 4000 FIP Snooping

Bridge-B

Nexus 4000 FIP Snooping

Bridge-A


DCB + FIP Snooping

Bridge

DCB + FIP Snooping

Bridge

Extending FCoE Fibre Channel fabrics beyond direct attach initiators can be achieved in two basic ways

Extend the Unified Edge (Stage 1)

Add DCB enabled Ethernet switches between the VN and VF ports (stretch the ‘link’between the VN_Port and the VF_Port)

Extend Unified Fabric capabilities into the SAN Core

Leverage FCoE wires between Fibre Channel of Ethernet switches (VE_Ports)

Fabric ALAN Fabric

Using FCoE for ISL

between FC Switches

Extending FCoE into a multi-hop Ethernet

‘Access’ Fabric

VN

VF

VE

VE

Fabric B

VE

VE

Multi - Hop DesignExtending FCoE with VE_Ports

Nexus 5000 FCF-A

Nexus 5000 FCF-A

MDS 9000FCF-A


E-NPVE-NPV

SAN BSAN A

E-NPVE-NPV

Two basic design options are possible when we deploy any FCoE multi-hop configuration

Option 1 – Unified Dedicated Wire

Allows MCEC for IP/Ethernet

Dedicated FCoE links for Storage

Option 2 – Unified Wire

Leverage Server side failover mechanisms for both SAN and LAN

Allows for Unified Wire beyond the Server to first device


Wire

Option 1: Dedicated Links and Topologies

Multi - Hop DesignExtending FCoE with Ethernet NPV

Nexus 5000 FCF-A

Nexus 5000 FCF-B


FCFFCF FIP and FcoE frames loadshared over

MCEC on a per flow basis

NO SAN ‘A’ and SAN ‘B’ isolation

FIP and FcoE frames loadshared over

MCEC on a per flow basis

NO SAN ‘A’ and SAN ‘B’ isolation

SAN BSAN ASAN and LAN high availability design requirements are not always identical

Optimal layer 2 LAN design may not meet FC high availability and operational design requirements

Features such as vPC & MCEC are not viable and not supported beyond the direct attached server

Server has two stacks and manages two topologies

Layer 2 network has a single topology

L2MP and TRILL provide options to change the design paradigm and come up with potential solutions

DCB Enabled

DCB Enabled

Multi - Hop DesignUnsupported Topologies







Where is it efficient to leverage ‘unified wire’, shared links for both SAN and LAN traffic?

At the edge of the fabric the volume of end nodes allows for a greater degree of sharing for LAN and SANIn the core we will not reduce the number of links and will either maintain separate FC or FCoE links to the SAN core and Ethernet links to the LAN core

LAN and SAN HA models are very different (and not fully compatible)

FC and FCoE are prone to HOLB in the network and therefore we are limited in the physical topologies we can build

e.g. 10 x 10G uplinks to LAN aggregation will require 10 x 10G links to a next hop SAN core (with targets attached) – No savings, actually spending more to achieve this direct uplinks to SAN core

Targets are attached to the SAN core (the LAN aggregation and SAN core have different topology functions)

Where is it more beneficial to deploy two cores – SAN and LAN over a “unified core” topology

FCoE Deployment ConsiderationsDedicated Aggregation/Core Devices


Migration to 10G FCoE in place of 4/8G FC links (Ethernet price per bit economics)

Edge switch running as FCF with VE_ports connecting to FCF on Core switch

Must be careful of Domain ID creeping

FSPF forwarding for FCoE traffic is end-to-end

Hosts will log into the FCF which they are attached to (access FCF)

Storage devices will log into the FCF at the core/storage edge

Maintains HA requirements from both LAN and SAN perspective

VE PortsVE Ports

SAN BSAN A

FCoE Deployment ConsiderationsMigration Strategy for FCoE

Nexus 5000 FCF-B

MDS 9000FCF-B


Migration to 10G FCoE in place of 4/8G FC links (Ethernet price per bit economics)

Edge switch running either as FCF in NPV mode or in E-NPV mode with FCF migrating to the SAN Core

Ethernet NPV (E-NPV) is a new construct intended to solve a number of system management problems

Using E_NPV alleviates domain ID issue

HA planning for the SAN side required

Does loosing a core switch mean the loss of a whole fabric?

FCFFCF

FCFFCF

E-NPVE-NPV

SAN BSAN A

FCoE Deployment ConsiderationsMigration Strategy for FCoE


SAN BSAN ADoes passing FCoE traffic through a

larger aggregation point make sense?

Multiple links required to support the HA models

1:1 ratio between access to aggregation and aggregation to SAN core is required

SAN is more vulnerable to HOLB so need to plan for appropriate capacity in any core ISL

When is a direct Edge to Core links for FCoE are more cost effective than adding another hop?

Smaller Edge device more likely to be able to use under-provisioned uplinks

1:1 Ratio of links required unless E-NPV FCoE uplink is over-provisioned

CORECongestion on

Agg-Core links will HOLB all

attached edge devices

FCoE Deployment ConsiderationsShared Aggregation/Core Devices


vv

Different requirements for LAN and SAN network designs

Factors that will influence this use case

Port density

Operational roles and change management

Storage device types

Potentially viable for smaller environments

Larger environments will need dedicated FCoE ‘SAN’ devices providing target ports

Use connections to a SAN

Use a “storage” edge of other FCoE/DCB capable devices

Direct Attach FCoE Targets

CORE

Multiple VDCsFCoE SANLAN AggLAN Core

FCoE Deployment ConsiderationsShared Aggregation/Core Devices

Nexus 5000 FCF-A

Nexus 5000 FCF-B


FCFFCFFCFFCF

SAN BSAN A

Topology will vary based on scale (single vs multiple tiers) Architecture as defined for product development has a dual core Question - where is the demarc between Unified Wire and Unified FabricAs the topology grows less Unified WireIn all practical terms the ‘edge’ is the unified point for LAN and SAN (not the core/agg)

In smaller topologies where core and edge merge then everything collapses but the essential design elements remain

Dedicated SAN and LAN Core


CORE

FCoE Deployment ConsiderationsDedicated Aggregation/Core Devices


MCEC results in diverging LAN and SAN high availability topologies

FC maintains separate SAN ‘A’ and SAN ‘B’ topologiesLAN utilizes a single logical topology

In vPC enabled topologies in order to ensure correct forwarding behavior for SAN traffic specific design and forwarding rules must be followedWhile the port-channel is the same on N5K-1 and N5K-2, the FCoE VLANs are differentvPC configuration works with Gen-2 FIP enabled CNAs ONLYFCoE VLANs are ‘not’ carried on the vPC peer-linkFCoE and FIP ethertypes are ‘not’forwarded over the vPC peer link

N5K2N5K2

SAN BSAN A

Direct Attach vPC Topology

N5K1N5K1

Virtualized Access Switch - FCoEFCoE - Unified Wires at the Edge

MCEC for IP Only – VLAN 10

vPC Peers

VLAN 10,30

VLAN 10,20

STP Edge Trunk

vPC Peer Link VLAN 10 ONLY

HERE!


Nexus 223210GE FEX

Nexus 223210GE FEX

Virtualized Access Switch - FCoEExtending FCoE – Nexus 2232

SAN BSAN A

Fabric Links Option 1: Single

Homed Port Channel

Fabric Links Option 2: Static

Pinned

Nexus 5000 as FCF or as E-NPV

device

Server Option 1: FCoE on

individual links. Ethernet traffic is Active/Standby

FEX-2232 extends the reach of 10Gig Ethernet/FCoE to distributed line card (ToR)

• Support for up to 384 10Gig/FCoE attached hosts managed by a single Nexus 5000

• Nexus 5000 is the FCF or can be in FIP Snooping + mode (when supported)

• Currently Nexus 2232 needs to be single homed to upstream Nexus 5000 (straight through N2K) to ensure SAN ‘A’ and SAN ‘B’ isolation

• Server Ethernet driver connected to the FEX in NIC Teaming (AFT, TLB) or with vPC (802.3ad)

Server Option 2: FCoE on a vPC

member PC with a single link

Nexus 223210GE FEX

Nexus 223210GE FEX

Nexus 5000Nexus 5000 Nexus 5000Nexus 5000



FCoE Multi-Tier Fabric Design Extending FCoE past the Unified Edge

DCB + FIP Snooping

Bridge

DCB + FIP Snooping

Bridge

Extending FCoE Fibre Channel fabrics beyond direct attach initiators can be achieved in two basic ways

Extend the Unified Edge

Add DCB enabled Ethernet switches between the VN and VF ports (stretch the ‘link’ between the VN_Port and the VF_Port)

Extend Unified Fabric capabilities into the SAN Core

Leverage FCoE wires between Fibre Channel switches (VE_Ports)

What design considerations do we have when extending FCoE beyond the edge?

High Availability

Oversubscription for SAN and LAN

Ethernet layer 2 and STP design

Fabric ALAN Fabric

Using FCoE for ISL

between FC Switches

Extending FCoE into a multi-hop Ethernet

‘Access’ Fabric

VN

VF

VE

VE

FCFFCF

Fabric B

FCFSwitch Mode

FCFSwitch Mode

Please see session “BRKSAN-2047 - Storage and the Unified Fabric” for more information on FCoE


FCFFCFFCFFCF

Unified Fabric and HA DesignExtending FCoE past the Edge – Current State

SAN BSAN ANexus 4000 is a Unified Fabric capable Blade Switch

DCB enabled

FIP Snooping Bridge

Dual Topology requirements for FCoE multi-hop

Servers IP connection to the Nexus 4000 is Active/Standby

vPC is not currently supported from blade server to Nexus 4000

Separate dedicated FCoE links run from Nexus 4000 to Nexus 5000

Single homed Port Channel supported for N4K to N5K FCoE uplinks


Wire

Mezzanine Converged Network

Adapter

Nexus 4000DCB Blade

Switch

Nexus 4000DCB Blade

SwitchNexus 4000DCB Blade

Switch

Nexus 4000DCB Blade

Switch

FIP SnoopingFIP SnoopingFIP SnoopingFIP Snooping

Option 1: Dedicated Links and Topologies

PCIe

Ethernet

Fibre Channel

10GbE

10GbE

Link


Servers, FCoE attached StorageServers, FCoE

attached Storage

Virtualized Access Switch - FCoELarger Fabric Multi-Hop Topologies

Multi-hop edge/core/edge topologyCore SAN switches supporting FCoE

N7K with DCB/FCoE line cardsMDS with FCoE line cards (Sup2A)

Edge FC switches supporting either N5K - E-NPV with FCoE uplinks to the FCoE enabled core (VNP to VF)N5K or N7K - FC Switch with FCoE ISL uplinks (VE to VE)

Scaling of the fabric (FLOGI, …) will most likely drive the selection of which mode to deploy

N7K or MDS FCoE enabled Fabric

Switches

FC Attached Storage

FC Attached Storage

ServersServers

VE

Edge FCFSwitch ModeEdge FCF

Switch Mode

VE

Edge Switch in E-NPV

Mode

Edge Switch in E-NPV

Mode

VF

VNP VE

VE

Please see session “BRKSAN-2047 - Storage and the Unified Fabric” for more information on FCoE

© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation_ID 89© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential14497_04_2008_c1

Evolution of the Data Centre ArchitectureVirtualized Access Layer

The Evolving Data Centre AccessWhere is the edge?

Phase 1 – Physical Virtualization (Nexus2000)

Decoupling Layer 1 and Layer 2vPC – Redundancy in the AccessDesign Considerations

Phase 2 – Hypervisor Virtualized Switching (Nexus 1000v)

Components of Nexus 1000vDesign Considerations

Phase 3 – Unifying the Fabric (Nexus & FCoE)Integrating the Unified Compute FabricThe Virtualized Access Layer


Evolution of the DC Access ArchitectureUCS 6100 – End Host Mode













UCS Fabric Interconnect supports two modes of operation

Switch ModeEnd Host Mode <- Recommended

In End Host Mode the Fabric Interconnects don’t function like regular LAN switches

They don’t forward frames based on destination MAC addresses They don’t run spanning-tree!They don’t learn MAC addresses from external LAN switchesForwarding is based on server-to-uplink pinningActs a true Layer 2 stub device and never reflects traffic back upstreamLoop-free topology without STP













Spanning Tree - Rapid

PVST+ or MST

Spanning Tree Edge Ports

Access/Edge Layer

Aggregation/Core

6100 - EHM


























FCFFCFFCFFCF

FCoE Multi-TierPotential Migration for UCS

SAN BSAN A

FCoE design will follow the same evolution Should 6100 become a fabric switch

Technically viable but does it add too much operational complexity (e.g. SW certification cycles, operational change management, …)

Migration from NPV to E-NPV mode Again key question is where is the demarcation between ‘Unified Wire’and Unified FabricNote: 6100 Fabric HA required due to lack of vPC uplinks when running Unified Wire from 6100 to next hop

Dedicated SAN and LAN Core


CORE

E-NPVE-NPV


























Switch ModeSwitch Mode

FCoE Multi-TierPotential Migration for UCS

SAN B

Option 1 –FC/FCoE targets

Smaller scaleWhat do we collapse to?What is the migration to FCoE targetsAgain key question is where is the demarc between ‘Unified Wire’and Unified FabricIf we direct attach targets do we need to support 6100 as a switch in a multi-switch fabric?Is Option 1 viable only for standalone implementations?

CORE

E-NPVE-NPV

FC Switch ModeFC Switch Mode

Option 2 – E-NPV





Single Hop DesignsMulti-Hop DesignsFCoE Deployment ConsiderationsFuture??Questions


E-NPVE-NPVFCFFCF

L2MP provides the potential to change the design of the Data Center FabricL2MP is based on an edge/spine topologyMultiple forwarding topologies can be supported

Edge/Spine architecturesUnique topologies for different forwarding groups (VLANs) are possible

High Availability design rules change, ECMP and ‘routed’ designTraffic capacity planning can change as well due to new load sharing capabilitiesMay still find design value in multiple cores and dedicated links

Multi Topology

L2MP Supporting SAN A, SAN

B and Scalable

LAN L2MP Edge

L2MP Spine

L2MP Edge

FCoE in the FutureWhat about L2MP?


FCFFCFFCFFCF

E-NPVE-NPV

SAN BSAN ADoes L2MP with multi-topology support change this?

Still have two oversubscription models and traffic patterns to analyze

Multi-topology L2MP fixes the MCEC problem by allowing per VLAN topologies (unique SAN ‘A’ & ‘B’ and LAN)

Traffic Capacity Planning becomes far more complex as measured load varies amongst ECMP links

Unified Dedicated Wires are still the recommendation: Provides better traffic isolation than VSANs/VLANs: different ports, different protocols!

E-NPVE-NPV

These 2 links are more prone to congestion


VLAN 10 ONLY HERE!

FCoE in the FutureL2MP -- Unified Wire vs Unified Dedicated Wire


NAS and FCoE Attached Storage

NAS and FCoE Attached Storage

FCoE Multi-Tier Fabric DesignHow will Future LAN capabilities change the HA design options

L2MP is based on an edge/spine/edge topologyLarger SAN topologies utilize an edge/core/edge designsHigh Availability design rules change, ECMP and ‘routed’ designWith L2MP multiple forwarding topologies can be supported

Traffic capacity planning can change as well due to new load sharing capabilitiesDedicated or shared links for Storage, IP, vMotion, backup, ….

Common edge/core/edge for both NAS and FC/FCoE Storage

Consistent low latency from any server to any storage (cut-thru)L2MP provides potential for very large capacity designs

Multi Topology L2MP Supporting SAN A, SAN

B and LAN

Edge

FC Attached Storage

FC Attached Storage

Edge

Servers Leveraging Bock and File Based

Storage

Servers Leveraging Bock and File Based

Storage

Core


Redefinition of the Fabric Services and associated design requirements

We discussed it in FC-BB-5, it was too much work and we delayed, it is a good idea (it may happen in FC-BB-6 - that is probably 2 years away)

Prior to that operational changes may allow a different design approach that will meet both HA requirements (VSAN/VLAN isolation, L2MP and LAN design evolution, changing SAN support matrix, …)

Initiators and targets are connected directly to the Ethernet cloud

Inside the cloud the data plane is pure Ethernet

Zoning is enforced at the edge of the cloud

EthernetCloud

FC services can run on any server

FCoE in the FutureWhat about FC-BB-6?


Why Nexus Edge LayerConsistent Architecture for Heterogeneous Environments

























Spanning Tree Edge PortsMulti-Hop FCoE

VMVMVM

VMVMVM

NEXUS 1000v VM

VMVM

VMVMVM

NEXUS 1000v

VMVMVM

VMVMVM

NEXUS 1000v

VMVMVM

VMVMVM

NEXUS 1000v

VMVMVM

VMVMVM

NEXUS 1000v

VMVMVM

VMVMVM

NEXUS 1000v

VMVMVM

VMVMVM

NEXUS 1000v

Cor

e/A

ggre

gatio

n La

yer

Virt

ualiz

ed E

dge/

Acc

ess

Laye

r

1G and 10GE Blade Servers Pass-Thru

HP/IBM/Dell

N4K - DCB Blade Switch

IBM/Dell

1G and 10GE Rack Mount

ServersUCS Compute

Pod10GE Blade

(HP) UCS Compute Pod

Cor

e

10G DCB FCoE

SAN ‘A’ SAN ‘B’

Uniform Network Fabric supporting

Heterogeneous Compute Environments

Unified Access Layer







Complete Your Online Session Evaluation

Give us your feedback and you could win fabulous prizes. Winners announced daily.

Receive 20 Cisco Preferred Access points for each session evaluation you complete.

Complete your session evaluation online now (open a browser through our wireless network to access our portal) or visit one of the Internet stations throughout the Convention Center.

Don’t forget to activate your Cisco Live and Networkers Virtual account for access to all session materials, communities, and on-demand and live activities throughout the year. Activate your account at any internet station or visit www.ciscolivevirtual.com.


Check the Recommended Reading brochure for suggested products available at the Cisco Store

Enter to Win a 12-Book Libraryof Your Choice from Cisco Press

Visit the Cisco Store in the World of Solutions, where you will be asked to enter this Session ID code


FCoE - Design Operations and Management 1

Documents