The Journey Towards the Converged Data Center: Fibre Channel over Ethernet (FCoE) and iSCSI

1© Copyright 2009 EMC Corporation. All rights reserved.

The Journey Towards the Converged Data Center: Fibre Channel over Ethernet (FCoE) and iSCSI

Stuart Miniman, Technologist, Office of the CTOEMC CorporationSeptember 2, 2009


Series Host: Dr. Burt Kaliski

Dr. Kaliski joined EMC Corporation in 2006 as a result of its acquisition of RSA Security, where he was chief scientist and vice president of research, leading RSA Laboratories. Following the merger, he took on responsibility for developing a corporate research program for EMC. In this role, Dr. Kaliski reports directly to Jeffrey Nick, Senior Vice President and Chief Technology Officer at EMC.

Dr. Kaliski’s path to EMC began at the RSA startup that came out of MIT in the 1980s, where he was the company’s first full-time scientist and in 1991 helped launch RSA Laboratories. Dr. Kaliski coordinated the development of the Public-Key Cryptography Standards (PKCS) and served as chair of the IEEE P1363 working group, general chair of CRYPTO ’91, program chair of CRYPTO ’97 and CHES 2002, and a member of the advisory board for the Encyclopedia of Cryptography and Security. He was recently appointed as a guest professor at Wuhan University’s College of Computer Science.

Dr. Kaliski received his bachelor’s, master’s and Ph.D. degrees in computer science from MIT, where his research focused on cryptography.

Director, EMC Innovation Network, Office of the CTO

Dr. Burt Kaliski


Today’s Speaker: Stuart Miniman

Stuart is a technologist in EMC’s CTO Office focused on networking and virtualization technologies. In 9 years with EMC, he has helped develop solutions with storage networking technologies including Fibre Channel over Ethernet (FCoE), Fibre Channel, iSCSI and Distance Solutions (IP, SONET,WDM).

Stuart is also interested in innovation and social media. You can find Stuart’s blog at http://blogstu.wordpress.com and also find him on a variety of other social networking sites.

Prior to EMC, Stuart worked at Lucent Technologies (now Avaya) and American Power Conversion. He holds a BS Degree in Mechanical Engineering from Cornell University and an MBA from Bryant University.

Technologist, Office of the CTO

Stuart Miniman


Agenda

History of Network Convergence

iSCSI and FCoE

Deployments of FCoE and iSCSI


Converged I/O History

ServerNet – Tandem - 1994

Future I/O Compaq, IBM, HP

Next Generation I/O (NGIO)Intel, Microsoft, Sun

System I/O - 1999

Infiniband – 1999Cisco, IBM, Intel,

Sun, Mellanox, Voltaire

Ethernet – iSCSI - 2003

Ethernet – FCoE - 2007Cisco, Nuova, Brocade, IBM, HP,

EMC, Intel, Qlogic, EmulexBroadcom, Sun, Mellanox, HDS

iSer - 2006iSCSI RDMA


The Challenge of Change

Network convergence is not a new idea – limiting factors include:

Once these are overcome,

how many factors can a customer change at a time?

Drivers Management VirtualizationReplicationCabling

Reliability Security ManagementEcosystemBandwidth


Today’s solutions

1990 2000 20101980

Defined73

Standard83

Widespread93

Defined85

Standard94

Widespread03

iSCSIiSCSIDefined

00 02Widespread

08

Standard

Large scale, reliable, manageable solutions

Simple solutions built to leverage

existing LAN


Why iSCSI was developed?

Link

IPsec

IP

TCP

iSCSI

SCSI

Link

IPsec

IP

TCP

iSCSI

SCSI

Initiator Target

IP Network

Provides physical network capability (Layer 2 Ethernet, Cat 5, MAC, etc.)

Provides IP routing (Layer 3) capability so packets can find their way through the network

Reliable data transport and delivery (TCPWindows, ACKs, ordering, etc.)

Delivery of iSCSIProtocol Data Unit (PDU) for SCSI functionality (initiator, target, data read / write, etc.)


Rack Server Environment Today

Servers connect to LAN, NAS and iSCSI SAN with NICs

Servers connect to FC SAN with HBAs

Many environments today are still 1 Gigabit Ethernet

Multiple server adapters, multiple cables, power and cooling costs– Storage is a separate network

(including iSCSI)

Rack-mounted servers

EthernetFibre Channel

Ethernet LAN

1 Gigabit Ethernet

1 Gigabit EthernetNICs

Storage

Fibre Channel SAN

FibreChannelHBAs

1 Gigabit Ethernet

iSCSI SAN

Note: NAS will continue to be part of the solution. Everywhere that yousee Ethernet or 10Gb Ethernet in thispresentation, NAS can be considered

part of the unified storage solution


10Gb Ethernet allows for Converged Data Center

Maturation of 10 Gigabit Ethernet– 10 Gigabit Ethernet allows replacement of n x 1Gb with a much smaller

number (start with 2) of 10Gb Adapters– Many storage applications require > 1Gb bandwidth

10 Gigabit Ethernet simplifies server, network and storage infrastructure– Reduces the number of cables and server adapters– Lowers capital expenditures and administrative costs – Reduces server power and cooling costs– Blade servers and server virtualization drive consolidated bandwidth

10 Gigabit Ethernet is the answer!iSCSI and FCoE both leverage this inflection point

LAN

SANSingle Wire for Network and Storage10 GbE


10 Gigabit Ethernet Cabling

High power and cost today

Keep existing cabling layout (> 1 B ports) and patch panel infrastructure

Cat6 55m; Cat 6a 100m

Cat6 or Cat6a

Copper (10GBase-T) / RJ-45

Distance limited to rackCables vary

Passive = Very low power

5mTwinaxCopper / SFP+DA (direct attach)

Optical historically 1% of overall Ethernet ports

Provides extended distance for backbone or core

OM2 82m; OM3 300m

OM2 (orange) or OM3 (aqua)

Optical (multimode) / LC

–+DistanceCableType / Connector

*10GBase-CX is another copper option, not expected in most 10 Gb Ethernet


Time To Widespread Adoption

1990 2000 20101980

Defined73

Standard83

Widespread93

Defined85

Standard94

Widespread03

07 09 ??Defined

Standard

iSCSIiSCSIDefined

00 02Widespread

08

Standard

Standard

10 Gigabit Ethernet10 Gigabit Ethernet02 09

Widespread


Why a New Option for FC Customers?

FC has a large and well managed install base– Want a solution that is attractive for customers with FC expertise /

investment– Previous convergence options did not allow for incremental adoption

Requirement for a Data Center solution that can provide I/O consolidation

– 10 Gigabit Ethernet makes this option available

Leveraging Ethernet infrastructure and skill set has always beenattractive

FCoE allows an Ethernet-based SAN to be introduced into the FC-based Data Center

without breaking existing administrative tools and workflows


FCoE Extends FC on a Single Network

Network Driver

FC Driver

Converged Network Adapter

Server sees storage traffic as FC

FC network

FC storage

Ethernet Network

Converged Network Switch

EthernetFC

FCoE SW Stack

Standard 10G NIC

Lossless Ethernet Links2 options

SAN sees host as FC


Standards for Next Generation Data Center

Fibre Channel over Ethernet(FCoE) protocol

– Developed by International Committee for Information Technology Standards (INCITS) T11 Fibre Channel Interfaces Technical Committee

– Fibre Channel over Ethernet allows native Fibre Channel to travel unaltered over Ethernet

– FC-BB-5 standard ratified in June 2009

– FC-BB-6 in process to expand solution

Converged Enhanced Ethernet(CEE)

– Developed by Ethernet IEEEData Center Bridging Task Group

– Converged Enhanced Ethernet creates an Ethernet environment that drops frames as rarely as Fibre Channel

– Technology commonly referred to as Lossless Ethernet

– IEEE standards targeting ratification in 2009/2010

– Requirement for FCoE; Enhancement for iSCSI

Two emerging parallel industry standards seek to drive I/O consolidation in large data centers over time:

Companies working on the standard committeesKey participants: Brocade, Cisco, EMC, Emulex, HP, IBM, Intel, QLogic, Sun, others


Agenda


iSCSI and FCoEDeployments of FCoE and iSCSI


CRCEthernetHeader

iSCSI is SCSI functionality transported using TCP/IP for delivery and routing in a standard Ethernet/IP environment

iSCSI and FCoE Framing

TCP/IP and iSCSI require CPU processing

FCoE is FC frames encapsulated in Layer 2 Ethernet frames designed to utilize a Lossless Ethernet environment

– Large maximum size of FC requires Ethernet Jumbo Frames – No TCP, so Lossless environment required– No IP routing

Ethe

rnet

Hea

der

FCoE

Hea

der

FCH

eade

r

FC Payload CR

CEO

FFC

S

FCoE Frame

iSCSI Frame IP TCP iSCSI Data

FC Frame


FCoE Frame Formats

Destination MAC Address

Source MAC Address

IEEE 802.1Q Tag

ET = FCoE Ver Reserved

Reserved

Reserved SOF

Encapsulated FC Frame(Including FC-CRC)

EOF Reserved

FCS

Reserved

FCoE Frame FormatBit 0 Bit 31

Ethernet frames give a 1:1 encapsulation of FC frames– No segmenting FC frames across

multiple Ethernet frames– FCoE flow control is Ethernet based

BB Credit/R_RDY replaced by Pause/PFC mechanism

FC frames are large, require Jumbo frames– Max FC payload size is 2112 bytes– Max FCoE frame size is 2180 bytes

Also created an FCoE Initialization Protocol (FIP) for:– Discovery– Login – To determine if the MAC address is

server provided (SPMA) or fabric provided (FPMA)


Vswitch VMkernel storage stack

Storage Drivers and Virtualization

NIC NICFCHBA

FCHBA

vNIC vNICvSCSI vSCSI

LAN traffic FC traffic

CNA

CNA

LAN traffic FCoE follows FC path

Hypervisor

iSCSI traffic iSCSI traffic

*iSCSI initiator can also be in the VM


Storage Drivers and Virtualization

NIC NICFCHBA

FCHBA

vNIC vNICvSCSI vSCSI

Hypervisor

FCoE software in the guest would send traffic through the vSwitch to the vNIC

SW FCoE

SW FCoE

VMkernel storage stack

No FCoE access here currently

VswitchvSwitch is not

Lossless


Multipathing Mechanisms for iSCSI and FCoE

Ethernet trunking and NIC Teaming– Link layer (2), below TCP, transparent to iSCSI and FCoE

Multiple TCP connections– In a single iSCSI session (layer 5)– Same or different hardware (Ethernet) ports– Difficult when TCP and iSCSI are offloaded

Multiple iSCSI/FCoE sessions or paths– Multipathing software (e.g., PowerPath) above iSCSI or FCoE– Same or different hardware (e.g., Ethernet) ports

iSCSI also supports HTTP-style redirects– Target has been temporarily or permanently moved


Lossless Ethernet

Limit the environment only to the Data Center– FCoE is Layer 2 only

IEEE 802.1 Data Center Bridging (DCB) is the standards task group

Converged Enhanced Ethernet (CEE) is an industry consensus term which covers three link level features

– Priority Flow Control (PFC, IEEE 802.1Qbb)– Enhanced Transmission Selection (ETS, IEEE 802.1Qaz)– Data Center Bridging Exchange Notification (DCBX, currently part of IEEE

802.1Qaz, leverages 802.1AB (LLDP))

Data Center Ethernet is a Cisco term for CEE plus additional functionality including Congestion Notification (IEEE 802.1Qau)

Enhanced Ethernet provides the Lossless Infrastructure which will enable FCoE and augment iSCSI storage traffic


PAUSE and Priority Flow Control

PAUSE transforms Ethernet into a lossless fabricClassical 802.3x PAUSE is rarely implemented since it stops all trafficPriority Flow Control (PFC), formerly known as Per Priority PAUSE (PPP) or Class Based Flow Control

– PFC will be limited to Data Center

A new PAUSE function that can halt traffic according to priority tag while allowing traffic at other priority levels to continue

– Creates lossless virtual lanes

Per priority link level flow control– Only affect traffic that needs it– Ability to enable it per priority– Not simply 8 x 802.3x PAUSE

Switch A Switch B


Enhanced Transmission Selection and Data Center Bridging Exchange Protocol (DCBX)

Enhanced Transmission Selection (ETS) provides a common management framework for bandwidth management

Allows configuration of HPC & storage traffic to have appropriately higher priorityWhen a given load in a class does not fully utilize its allocated bandwidth, ETS allows other traffic classes to use the available bandwidthMaintain low latency treatment of certain traffic classes

Offered Traffic

t1 t2 t3

10 GE Link Realized Traffic Utilization

3G/s HPC Traffic3G/s

2G/s

3G/sStorage Traffic3G/s

3G/s

LAN Traffic4G/s

5G/s3G/s

t1 t2 t3

3G/s 3G/s

3G/s 3G/s 3G/s

2G/s

3G/s 4G/s 6G/s

Data Center Bridging Exchange Protocol (DCBX) is responsible for configuration of link parameters for DCB functionsDetermines which devices support Enhanced Ethernet functions


Beyond Link Level

Congestion notification– IEEE 802.1Qau targeted for April 2010

ratificationAllows a switch to notify attached ports to slow down transmission due to heavy traffic, in order to reduce the chances of packet drops or network deadlocksMoves the management of congestion back to the edge, which helps alleviate network-wide bottlenecks

Layer 2 multipathing– IETF TRILL - TRansparent Interconnection of

Lots of LinksUsed with the Spanning Tree Protocol to provide more efficient bridging and bandwidth aggregationFocuses on a bridging capability that will increase bandwidth by allowing and aggregating multiple network pathsStandards are stable; products are coming soon

Throttle

Switch

Transmit QueueSwitch

Receive Buffer

Throttle


40 and 100 Gigabit Ethernet

IEEE P802.3ba Task Force states that bandwidth requirements for computing and networking applications are growing at different rates, which necessitates two distinct data rates, 40 Gb/s and 100 Gb/s

IEEE target for standard completion of 40 GbE & 100 GbE is 2010

40 GbE products shipping today supporting existing fiber plant and plan is for 100 GbE to also support 10m copper, 100m MMF (use OM4 for extended reach) and SMF

Cost of 40 GbE or 100 GbE is currently 5 – 10 x 10 GbE– Adoption will become more economically attractive at 2.5x which will take a

couple of years


Agenda


iSCSI and FCoE

Deployments of FCoE and iSCSI


FCoE and iSCSI

FCoE

FC expertise / install base

FC management

Layer 2 Ethernet

Use FCIP for distance

T11 Standards complete

Ethernet

Leverage Ethernet/IP expertise

10 Gigabit Ethernet

Lossless Ethernet (standards in process)

iSCSI

No FC expertise needed

Supports distance connectivity (L3 IP routing)

Strong virtualization affinity

Standards since 2003

Choose based on scalability, management, and skill set


iSCSI Deployment

iSCSI grew to > 10% of SAN market revenue in 2008 *

Many deployments are small environments, which replace DAS

– Strong affinity in SMB/commercial markets

Seeing strong growth of Unified Storage– Supports iSCSI, FC, and NAS

iSCSI with 10 Gigabit Ethernet becoming available

Ethernet

iSCSI SAN

* According to IDC, 3/09


FCoE Server Phase (Today)

FC HBAs1 Gb NICs


Rack Mounted Servers

10 GbE CNAs

FC Attach

FCoE with direct attach of server to Converged Network Switch at top of rack or end of rowTightly controlled solutionServer 10 GE adapters may be CNA or NICStorage is still a separate network

Ethernet LAN

Storage

Fibre Channel SAN

EthernetFC


FCoE Network Phase (2009 / 2010)

Converged Network Switches move out of the rack from a tightly controlled environment into a unified network

Maintains existing LAN and SAN management

Overlapping domains may compel cultural adjustments


10 GbE CNAs


FC Attach

Ethernet Network (IP, FCoE) and CNS

Ethernet LAN

Storage

Fibre Channel SAN

EthernetFC


FCoE Storage Phase (2010+)

Single Ethernet network for IP and storage trafficEnd-to-End Ethernet with native FCoE (future)End-to-end solution still maintains upper layer FC at host and storage


10GbE CNAs


Storage

FC & FCoE SAN

FibreChannel& FCoE attach

Ethernet LAN

FCoEStorage

EthernetFC


PrivateCloud

Virtualized Data Center

CloudComputing

Next Generation Data Center

10 Gigabit Ethernet Fibre

Channel

virtualization

common infrastructure

common management

EMC is working with the standards communities and partners to deliver the same reliability and robustness in the next generation virtual data center that we deliver today

The Converged Data Center sets theoperational and capital efficiency foundationsfor the virtual data center and private clouds


Related References

FCoE in the EMC Topology Guide http://elabnavigator.emc.com

Industry site with consolidated information http://www.fcoe.com/

EMC FCoE Videos – search “FCoE” on YouTube

EMC FCoE whitepaperhttp://www.emc.com/collateral/hardware/white-papers/h5916-intro-to-fcoe-wp.pdf

T11 FCoE activity http://www.t11.org/fcoe

IEEE 802.1 Data Center Bridging task group pagehttp://www.ieee802.org/1/pages/dcbridges.html

EMC Bloggers covering these technologies:– Chad Sakac http://virtualgeek.typepad.com/– Chuck Hollis http://chucksblog.typepad.com/– David Graham http://flickerdown.com/– Stuart Miniman http://blogstu.wordpress.com/


Questions

To ask a question from the phone, press *1

To type a question via WebEx, click on the Q&A tab

Starting tomorrow, by registering for the recording, you can also download a copy of the presentation in PDF. Look for this recording at: http://www.emc.com/events/ondemand-events.esp

You can also email us at: [email protected]

Visit us today, at: www.emc.com


Upcoming events and discussion

Registration Info: http://info.emc.com/mk/get/AMA00012439_LP?reg_src=SA_JN

Visit http://labs.emc.com If you would like to continue today’s conversation, view the upcoming schedule, or access recordings and PDFs of previous lectures.

Harnessing Untapped I/O Potential on Commodity Hardware(Professor Peter Honeyman, scientific director, University of Michigan Center for Information Technology Integration)

October 7

Evolutionary Computation - Evolve, Don’t Invent (Dr. Conor Ryan, University of Limerick)

November 4

2009 Events

Stuart Miniman, Technologist, Office of the CTOEMC Corporation

http://blogstu.wordpress.com

The Journey Towards the Converged Data Center: Fibre Channel over Ethernet (FCoE) and iSCSI

Technology

emc innovation network

iscsi copyright

cto stuart

ethernet iscsi

kaliski reports

hds copyright

fibre channel

iscsi rdmaethernet fcoe