Top Banner
© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 1 DCB and FCoE Deep dive Jaromír Pilař ([email protected] ) Consulting Systems Engineer, CCIE 2910
59

FCoE Deep Dive

Apr 24, 2015

Download

Documents

rkritsada6805
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Page 1: FCoE Deep Dive

© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 1

DCB and FCoE Deep dive

Jaromír Pilař ([email protected])Consulting Systems Engineer, CCIE 2910

Page 2: FCoE Deep Dive

© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 2

What Is I/O Consolidation� IT organizations operate multiple parallel networks

IP and other LAN protocols over an Ethernet networkSAN over a Fibre Channel networkHPC/IPC over an InfiniBand network

� I/O consolidation supports all three types of traffic onto a single network

� Servers have a common interface adapter that supports all three types of traffic

IPC: Inter-Process Communication

Page 3: FCoE Deep Dive

© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 3

Consolidation - one of major trends in datacenterBut where is the main consolidation potential ?

� Majority of ports in fabric is in access layer regardless of fabric type => access layer has the highest potential for consolidation

� Different fabrics (network, SAN, HPC) have different requirements => do we have the technology which can serve them all at once?

� If we have it => is the technology mature enough and affordable to be massively deployed?

Page 4: FCoE Deep Dive

© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 4

ProcessorMemory

I/O Consolidation in the Network

LAN

Storag

e

IPCProcessorMemory

I/O Subsystem

LAN

Storag

eIPC

I/O I/O I/O

IPC: Inter-Process Communication

Page 5: FCoE Deep Dive

© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 5

FC TrafficFC HBA

I/O Consolidation in the Host� Fewer CNAs (Converged Network Adapters) instead of NICs, HBAs, and HCAs

� Limited number of interfaces for Blade Servers

All Traffic Goes over 10 GE

CNA

CNA

FC TrafficFC HBA

NIC Enet TrafficNIC Enet TrafficNIC Enet TrafficHCA IPC Traffic

IPC TrafficHCA

Page 6: FCoE Deep Dive

© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 6

Cabling and I/O Consolidation

Page 7: FCoE Deep Dive

© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 7

Merging the Requirements

LAN/IP

� Must be EthernetToo much investmentToo many applications that assume Ethernet

� Must follow the Fibre Channel model

� Losing frames is not an option

Storage IPC(Inter-Process

Communication)

� Doesn’t care of the underlying network, provided that:

It is cheapIt is low latencyIt supports APIs like OFED, RDS, MPI, sockets

Page 8: FCoE Deep Dive

© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 8

Why Consolidation Attempts Have Not Succeeded Yet?� Previous attempts

Fibre ChannelNever credible as data network infrastructure

InfiniBandNot Ethernet

iSCSINot Fibre Channel

� Before PCI-Express there was not enough I/O bandwidth in the servers

� It needs to be Ethernet, but…1 GE didn’t have enough bandwidth

Page 9: FCoE Deep Dive

© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 9

Multicore CPU Architectures Allowing Bigger and Multiple Workloads on the Same Machine

Server Virtualization Driving the Need for More Bandwidth per Server Due to Server Consolidation

Growing Need for Network Storage Driving the Demand for Higher Network Bandwidth to the Server

Drivers for 10GE to the Servers

Multicore CPUs and Server Virtualization Driving the Demand for Higher Bandwidth Network Connections

Page 10: FCoE Deep Dive

© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 10

Enabling Technologies

Page 11: FCoE Deep Dive

© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 11

Three Challenges + One

Page 12: FCoE Deep Dive

© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 12

Why Are Frames Lost?Collision

� No longer present in full duplex Ethernet

� Very rare in the data center

Transmission Error

� Most common cause

� Congestion is a switch issue, not a link issue

A full duplex IEEE 802.3 link does not lose frames

� It must be dealt with in the bridge/switch

By IEEE 802.1

Congestion

Page 13: FCoE Deep Dive

© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 13

Can Ethernet Be Lossless?� Yes, with Ethernet PAUSE Frame

PAUSESTOP

Ethernet Link

Switch A Switch B

Queue Full

� Defined in IEEE 802.3—Annex 31BThe PAUSE operation is used to inhibit transmission of data frames for a specified period of time

� Ethernet PAUSE transforms Ethernet into a lossless fabric

Page 14: FCoE Deep Dive

© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 14

A

How PAUSE WorksThreshold

PAUSEFrame

Stop SendingFrames for ThisInterval of Time

PAUSEFrame

Start SendingFrames Again

B

Page 15: FCoE Deep Dive

© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 15

Let’s Compare PAUSE with FC Buffer to Buffer Credit� Eight credits preagreed

R_RDYR_RDY

A B

Page 16: FCoE Deep Dive

© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 16

PAUSE Frame Format� A standard Ethernet frame, not tagged

� EtherType = 0x8808 means MAC Control Frame

� Opcode = 0x0101 means PAUSE� Pause_Time is the time the link needs to remain paused in Pause Quanta (512-bits time)

� There is a single Pause_Time for the whole link

CRC

Pad42 Bytes

01:80:C2:00:00:01

Source Station MAC

EtherType = 0x8808

PAUSE Frame

Pause_TimeOpcode = 0x0001

Page 17: FCoE Deep Dive

© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 17

Why Is PAUSE Not Widely Deployed?� Inconsistent implementations

Standard allows for asymmetric implementations Easy to fix

� PAUSE applies to the whole linksSingle mechanism for all traffic classes

� This may cause “traffic interference”e.g., Storage traffic paused due to a congestion on IP traffic

Page 18: FCoE Deep Dive

© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 18

Priority Flow Control (PFC)� a.k.a. PPP (Per Priority Pause)� PFC enables PAUSE functionality per Ethernet priority

IEEE 802.1Q defines eight prioritiesTraffic classes are mapped to different priorities:

No traffic interferenceIP traffic may be paused while storage traffic is being forwardedOr, vice versa

Requires independent resources per priority (buffers)� High level of industry support

Cisco distributed proposalStandard track in IEEE 802.1Qbb

EtherType = IEEE 802.1Q Priority CFI VLAN IDIEEE 802.1Q Tag

16 3 1 12 Bits

Page 19: FCoE Deep Dive

© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 19

Priority Flow Control in Action

EightPriorities

Switch A Switch B

Transmit QueuesEthernet Link

Receive Queues

OneTwoThreeFourFive

SevenEight

Six

OneTwoThreeFourFive

SevenEight

SixSTOP PAUSE

Page 20: FCoE Deep Dive

© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 20

PFC Frame Format� Similar to the PAUSE frame� Opcode = 0x0101 is used to distinguish PFC from PAUSE

� Class vector indicates for which priorities the frame carries valid Pause information

� There are eight Time fields, one per priority

Class Enable VectorTime (Class 0)

CRC

Pad28 Bytes

01:80:C2:00:00:01

Source Station MAC

EtherType = 0x8808Opcode = 0x0101

Time (Class 1)Time (Class 2)Time (Class 3)Time (Class 4)Time (Class 5)Time (Class 6)Time (class 7)

Priority Flow Control

Page 21: FCoE Deep Dive

© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 21

In Order to Build a Deployable I/O Consolidation Solution, the Following Additional Components Are Required:

Is Anything Else Required?

� Discovery protocol (DCBX)� Bandwidth manager

Page 22: FCoE Deep Dive

© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 22

DCBX� Hop-by-hop negotiation for:

Priority Flow Control (PFC)Bandwidth managementApplicationsLogical link-down

� Based on LLDP (Link Level Discovery Protocol)Added reliable transport

� Allows either full configuration or configuration checkingLink partners can choose supported features and willingness to accept configuration from peer

Page 23: FCoE Deep Dive

© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 23

Bandwidth Management� IEEE 802.1Q defines priorities, but not a simple, effective, and consistent scheduling mechanism

� Products typically implement some form of Deficit Weighted Round Robin (DWRR)

Configuration and interworking is problematic� Proposal for HW-efficient, two-level DWRR with strict priority support

Consistent behavior and configuration across network elements� Standard track in IEEE 802.1Qaz

Page 24: FCoE Deep Dive

© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 24

Priority Groups

Priorities Are Assigned to Individual

Traffic Classes

PriorityGroups

Priority Groups Are Then Scheduled

First Level of Scheduling Inside Each Group

Final Link Behavior

LAN

SAN

IPC

Page 25: FCoE Deep Dive

© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 25

Example of Link Bandwidth Allocation10 GE Link Realized Traffic Utilization

T1 T2 T3

LAN Traffic(40%)

Storage Traffic(30%)

(30%) HPC Traffic(30%)

(30%)

(30%)

(20%)

(50%)

(30%)

HPC Traffic—Priority Class “High”—20% Guaranteed BandwidthLAN Traffic—Priority Class “Medium”—50% Guaranteed BandwidthStorage Traffic—Priority Class “Medium-High”—30% Default Bandwidth

Offered Traffic

3 Gbs 4 Gbs 6 Gbs

3 Gbs 3 Gbs

3 Gbs 3 Gbs 3 Gbs

2 Gbs

T1 T2 T3

Page 26: FCoE Deep Dive

© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 26

FCoE: Fibre Channel over Ethernet

Page 27: FCoE Deep Dive

© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 27

What is Fibre Channel over Ethernet?� From a Fibre Channel standpoint it’s

FC connectivity over a new type of cable called… an Ethernet cloud

� From an Ethernet standpoints it’sYet another ULP (Upper Layer Protocol) to be transported, but… a challenging one!

� And technically…

FCoE is an extension of Fibre Channelonto a Lossless Ethernet fabric

Page 28: FCoE Deep Dive

© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 28

FCoE Enablers and EncapsulationEthe

rnet

Head

er

FCoE

Head

er

FCHe

ader FC Payload CRC

EOF

FCS

Same as a physical FC frame

Control information: version, ordered sets (SOF, EOF)

Normal ethernet frame, ethertype = FCoE

� 10Gbps Ethernet� Lossless Ethernet

Matches the lossless behavior guaranteed in FC by B2B credits� Ethernet jumbo frames

Max FC frame payload = 2112 bytesTotal max frame size = 2180 bytes

Page 29: FCoE Deep Dive

© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 29

FCoE Is Fibre ChannelFCoE Is Fibre Channel at the Host and Switch Level

Same Operational Model Same Techniques ofTraffic Management

Same Managementand Security Models

Easy to Understand

Completely Based on the FC ModelSame Host-to-Switch and Switch-to-Switch Behavior of FCe.g., in Order Delivery or FSPF Load BalancingWWNs, FC-IDs, Hard/Soft Zoning, DNS, RSCN

Page 30: FCoE Deep Dive

© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 30

The two protocols have:• Two different Ethertypes• Two different frame formats• Both are defined in FC-BB-5

Protocol Organization

FCoE itself � Is the data plane protocol� It is used to carry most of the

FC frames and all the SCSI traffic

� Uses Fabric Assigned MAC address (dynamic)

FCoE itself � Is the data plane protocol� It is used to carry most of the

FC frames and all the SCSI traffic

� Uses Fabric Assigned MAC address (dynamic)

FIP (FCoE Initialization Protocol)

� It is the control plane protocol� It is used to discover the FC

entities connected to an Ethernet cloud

� It is also used to login to and logout from the FC fabric

� Uses unique BIA on CNA for MAC

FIP (FCoE Initialization Protocol)

� It is the control plane protocol� It is used to discover the FC

entities connected to an Ethernet cloud

� It is also used to login to and logout from the FC fabric

� Uses unique BIA on CNA for MAC

http://www.cisco.biz/en/US/prod/collateral/switches/ps9441/ps9670/white_paper_c11-560403.html

Page 31: FCoE Deep Dive

© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 31

All Standards for FCoE Are Technically StableStatus of the Standards

PFC

ETS

DCBX

Inv Dev Appr Pub

Technically Stable

FC-BB-5

Inv Dev Appr Pub

Inv Dev Appr Pub

Inv Dev Appr Pub

Technically stable in October, 2008Completed in June 2009Published in May, 2010

Completed in July 2010, awaiting publication

Completed in July 2010 (completing Approval Phase 3)

Completed in July 2010 (completing Approval Phase 3)

DCB

Page 32: FCoE Deep Dive

© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 32

Myths and Misunderstandings

Page 33: FCoE Deep Dive

© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 33

Myth: “You Can’t Do End-to-End FCoE.”

StorageNewsletter.com., “Exclusive Interview with Darren Thomas, Head of Dell Storage.” June 29, 2010

The FC-BB-5 standard fully supports end-to-end FCoE

If someone says it does not, it means he did not read the standard

However, current implementations may be behind the standard and do not fully support it yet

Page 34: FCoE Deep Dive

© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 34

Myth: "FC-BB-6 Means the FCoE Standard Isn't Done Yet."

� Standards are like operating systems - they add features to previous versions

� Different versions (e.g., FC-BB-4, FC-BB-5, FC-BB-6) have different features

� FC-BB-5 fully defined the way to transport Fibre Channel over Ethernet

�FC-BB-6 is working on adding features and functionality

Page 35: FCoE Deep Dive

© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 35

QCN Operates At a Different Level Than FCoE

Myth: “You Need QCN (802.1Qau) for End-To-End FCoE.”

� QCN is a core-to-edge protocol to deal with persistent congestion situations in a Layer 2 network

H1

H2

H3

CongestionTraffic

QCN message

QCN messageDA: H3SA: H1

DA: H3SA: H2

DA: H1SA: H3

DA: H2SA: H3

When congestion is detectedthe core switch samples some frames, swaps their MAC addresses, and sends notifications backward

Page 36: FCoE Deep Dive

© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 36

FCoE Crosses Layer 2 DomainsTherefore, QCN is Useless for FCoE

DA: FCF-MAC(A)SA: FPMA(H2)

Encaps. FC frameD_ID = FC-ID(T2)S_ID = FC-ID(H2)

DA: FPMA(T2)SA: FCF-MAC(C)

Encaps. FC frameD_ID = FC-ID(T2)S_ID = FC-ID(H2)

DA: FCF-MAC(B)SA: FCF-MAC(A)

Encaps. FC frameD_ID = FC-ID(T2)S_ID = FC-ID(H2)

DA: FCF-MAC(C)SA: FCF-MAC(B)

Encaps. FC frameD_ID = FC-ID(T2)S_ID = FC-ID(H2)

H1

H2

T1

T2

FCF A FCF B FCF C

Page 37: FCoE Deep Dive

© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 37

What Does This Mean?

Source: FC-BB-5 rev 2.00, June 4, 2009

FCoE = Layer 3 From Ethernet Perspective

Layer 2 of the Fibre Channel model maps to Layer 3 of the Ethernet model

FC-0

Ethernet (OSI) ModelFC Model

FC-1FC-2PFC-2MFC-2VFC-3FC-4

Layer 1 - PHYLayer 2 - MAC

“Layer 3”FCoE Entity

FC-2VFC-3FC-4 FC Levels

(Unchanged)

IEEE 802.3 Layers

FC-2

QCN happens here

FCoE happens here (including Multihop)

Page 38: FCoE Deep Dive

© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 38

ResponseMyth: "You Need TRILL To Run FCoE"

TRILL defines an alternative way to Spanning Tree to forward Ethernet frames in an Ethernet network

Also supports multipathingHas nothing to do with congestion

Source: Mellor, Chris. ”DCB is Not Enough.” The Register August 3, 2010

Page 39: FCoE Deep Dive

© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 39

What Does This Mean?

Source: FC-BB-5 rev 2.00, June 4, 2009

FCoE = Layer 3 From Ethernet Perspective

� Layer 2 of the Fibre Channel model maps to Layer 3 of the Ethernet model

FC-0

Ethernet (OSI) ModelFC Model

FC-1FC-2PFC-2MFC-2VFC-3FC-4

Layer 1 - PHYLayer 2 - MAC

“Layer 3”FCoE Entity

FC-2VFC-3FC-4 FC Levels

(Unchanged)

IEEE 802.3 Layers

FC-2

TRILL happens here

FCoE happens here (including Multihop)

Page 40: FCoE Deep Dive

© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 40

Nexus 5000 and 2000 family products for Unified Fabric

Page 41: FCoE Deep Dive

© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 41

Cisco Nexus 5010/50201st generation of Nexus 5xxx familyIndustry’s First I/O Consolidation Virtualization Fabric for

Enterprise Data CenterIndustry’s First I/O Consolidation Virtualization Fabric for

Enterprise Data Center

Nexus 5000SwitchFamily Nexus 5020 - 56-Port L2 Switch

• 40 Ports 10GE/FCoE/DCE, fixed• 2 Expansion Modules

Nexus 5010 - 28-Port L2 Switch• 20 Ports 10GE/FCoE/DCE, fixed

• 1 Expansion Module

FC + Ethernet • 4 Ports 10GbE/FCoE/DCE

• 4 Ports 1/2/4G FC

Fibre Channel • 8 Ports 1/2/4G FC

ExpansionModules Ethernet

• 6 Ports 10GE/FCoE/DCE

OSCisco Fabric Manager and Cisco Data Center Network Manager

Cisco NX-OSMgmt

PartnersSW FCoE/DCE + 2x10GE2x10GE/DCE/FCoE 2x10GE

Fibre Channel • 6 Ports 2/4/8G FC

Page 42: FCoE Deep Dive

© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 42

Mgmt 0, Console, USB

Mgmt 0, Console, USB Redundant Fan ModulesRedundant Fan Modules Redundant 750W AC Power SuppliesRedundant 750W AC Power Supplies

• 32 Fixed SFP+ Ports• Line Rate• Hardware Capable of 1/10 Gigabit Ethernet• Traditional Ethernet or Fibre Channel over Ethernet• L3 capable (post FCS)• FabricPath and TRILL capable (post FCS)• 40 GE ready

• 32 Fixed SFP+ Ports• Line Rate• Hardware Capable of 1/10 Gigabit Ethernet• Traditional Ethernet or Fibre Channel over Ethernet• L3 capable (post FCS)• FabricPath and TRILL capable (post FCS)• 40 GE ready

Expansion Modules (GEM2)• 16p SFP+ Ethernet Ports• 8p Eth + 8p Native FCExpansion Modules (GEM2)• 16p SFP+ Ethernet Ports• 8p Eth + 8p Native FC

Cisco Nexus 55482nd generation of Nexus 5xxx family

Page 43: FCoE Deep Dive

© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 43

� 32x 1/10GE host interfaces; 8x 10GE on network interfaces� 10GE interfaces support FCoE� HW supports 1G but SW support in a post-FCS release

� Can mix-and-match with existing GE and next-gen GE FEX in network topologies� Host port-channel support� ACL classification� SPAN source/destination support� Only FIP enabled CNAs supported (Gen 2)

Nexus 2000 family extension10GE FCOE capable Fabric Extender - Nexus 2232

32 10GE/FCoE SFP+ Downlinks 8 10GE/FCoE SFP+ Uplinks

Page 44: FCoE Deep Dive

© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 44

Unified FabricInitial Deployments

• Servers and FCoE targets are directly connected to the Nexus 5000 over 10Gig FCoE

• Nexus 5000 operates as the FCF• Native Ethernet LAN network and Native Fibre Channel network break off at the Nexus 5000 access layer

Direct Attached Topology

Enhanced Ethernet and FCoEEthernet LANNative Fibre Channel

SAN ASAN A SAN BSAN B

FIP enabled CNAsFIP enabled CNAs

vPCvPC

FIP or Pre-FIP enabled CNAsFIP or Pre-FIP enabled CNAs

FCoE TargetsFCoE Targets

Ethernet/LANEthernet/LAN

Nexus 5000FCF

Nexus 5000FCF Nexus 5000

FCFNexus 5000

FCF

Page 45: FCoE Deep Dive

© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 45

Unified FabricMultihop FCoE Deployment with Nexus 4000I

• Blade servers connect to Nexus 4000 over 10Gig FCoE

� Nexus 4000 is a FIP-Snooping Bridge

• Nexus 4000 connects to Nexus 5000 over 10Gig FCoE � Nexus 5000 operates as the FCF

• Native Ethernet LAN network and Native Fibre Channel network break off at the Nexus 5000

Enhanced Ethernet and FCoEEthernet LANNative Fibre Channel

SAN ASAN A SAN BSAN B

Nexus 5000FCF

Nexus 5000FCF

FCoE TargetsFCoE Targets

Blade Chassis

Nexus 4000: FIP Snooping BridgeCNA mezzanine

cards

Nexus 4000: FIP Snooping BridgeCNA mezzanine

cards

Nexus 5000FCF

Nexus 5000FCF

Ethernet/LANEthernet/LAN

Page 46: FCoE Deep Dive

© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 46

• Servers connect to FEX 2232 over 10Gig FCoE� Server connections to the FEX can be Active/Standy or over a vPC

• FEX 2232 is single homed to upstream Nexus 5000� FEX 2232 can be connected with individual links or a port-channel

Enhanced Ethernet and FCoEEthernet LANNative Fibre Channel

Nexus 5000FCF

Nexus 5000FCF

Nexus 5000FCF

Nexus 5000FCF

vPCvPC

FEX-2232FEX-2232 FEX-2232FEX-2232

Ethernet/LAN CoreEthernet/LAN Core SAN ASAN A SAN BSAN B

FIP enabled CNAs

Unified FabricMultihop FCoE Deployment with Nexus 2232PP

Page 47: FCoE Deep Dive

© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 47

� With NX-OS 5.0(2)N2, VE_Ports are supported on/between the Nexus 5000 and Nexus 5500 Series Switches

� VE_Ports are run between switches acting as Fibre Channel Forwarders (FCFs)

� VE_Ports are bound to the underlying 10G infrastructure� VE_Ports can be bound to a single 10GE port

� VE_Ports can be bound to a port-channel interface consisting of multiple 10GE links

VN

VE

VF

VE

VF

VN

FCoEFC

All above switches are Nexus 5X00 Series acting as an FCF

All above switches are Nexus 5X00 Series acting as an FCF

VE

VE

Unified FabricMultihop FCoE Deployment using VE ports

Page 48: FCoE Deep Dive

© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 48

FCoEAdapters

Page 49: FCoE Deep Dive

© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 49

PCIe

Ethernet10G

bE

10GbE

Link

PCIe

Fibre ChannelEthernet

HBA

HBA

Link

Fibre Channel Drivers

Ethernet Drivers

Operating System

Fibre Channel Drivers

Ethernet Drivers

Operating System

PCIe

Fibre ChannelEthernet

10GbEE

10GbEE

Link

CNA: Converged Network AdapterLAN CNAHBA

Page 50: FCoE Deep Dive

© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 50

View from Operating System � Standard drivers� Same management

� Operating system independent interfaces

Page 51: FCoE Deep Dive

© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 51

Qlogic 8042 Gen1 CNA

� PCI Express Gen1 x8� Dual port 10GbE

Passive copperOptical SR

� Multi-chip solution� Full height, full length� Power = 27W� QLogic 4Gb FC controller and drivers� Intel Ethernet controller and drivers� Windows, Linux, & Vmware (ESX 3.5U4 & 4.0) support

Page 52: FCoE Deep Dive

© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 52

Qlogic 8100 Series Gen 2 CNA� PCI Express x8 slot� Single and dual port 10GbE

Active & passive copperOptical SR & LROperates with QLogic optics only

� Fully Integrated ASIC� Power ~7.4W (Dual Port with Optical SR)

No heat sink required� Low Profile Form Factor

Page 53: FCoE Deep Dive

© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 53

FCOE vs. FC performance test

Page 54: FCoE Deep Dive

© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 54

FCOE vs. FC test - results

Page 55: FCoE Deep Dive

© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 55

Open-FCoE Software

SCSI Layer

HBA Driver

Linux Kernel

HBA

HBA

HBA Mgmt Plane

File System layers

Fibre

SCSI Layer

FCoE Layer

Linux Kernel

FCoE

Net Device

FCoE Mgmt Plane

File System layers

Ethernet

Ethernet Driver

OpenFC Layer

Ethernet

Server Server

Page 56: FCoE Deep Dive

© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 56

More information

Page 57: FCoE Deep Dive

© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 57

More Information� Standards Sites

�http://ieee.org�http://t11.org/fcoe�http://fcoe.com

� Case Studies�http://www.cisco.com/en/US/products/ps9670/prod_case_studies_list.html

� Book�I/O Consolidation in the Data Center- By Silvano Gai & Claudio DeSanti

Page 58: FCoE Deep Dive

© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 58

Thank You

Page 59: FCoE Deep Dive

© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 59