Converged Data Center: FCoE, iSCSI and the Future of Storage Networking

1 © Copyright 2011 EMC Corporation. All rights reserved.

Converged Data Center: FCoE, iSCSI and the Future of Storage Networking David L. Black, Distinguished Engineer Office of the CTO


Agenda • Network Convergence • Protocols & Standards • Solution Evolution • Conclusion and

Summary


10Gb Ethernet Converged Data Center •  Maturation of 10 Gigabit Ethernet

–  Replace multiple 1Gb adapters with fewer 10Gb adapters (start with 2) –  Single network simplifies mobility for virtualization/cloud deployments

•  10 Gigabit Ethernet simplifies infrastructure –  Reduces number of cables and server adapters –  Lowers capital expenditures and administrative costs –  Reduces server power and cooling costs –  Blade servers and server virtualization drive consolidated bandwidth

FCoE and iSCSI both leverage this inflection point

LAN

SAN Single Wire for Network and Storage 10 GbE


Conventional Rack Servers •  Servers connect to LAN, NAS and

iSCSI SAN with NICs •  Servers connect to FC SAN with

HBAs •  Many environments today are still

1 Gigabit Ethernet •  Multiple server adapters, multiple

cables, power and cooling costs –  Storage is a separate network (including

iSCSI)

Rack-mounted servers

Ethernet Fibre Channel

Ethernet LAN

1 Gigabit Ethernet

1 Gigabit Ethernet NICs

Storage

Fibre Channel SAN

Fibre Channel HBAs

1 Gigabit Ethernet

iSCSI SAN

Note: NAS will continue to be part of the solution. Everywhere that you see Ethernet or 10Gb Ethernet in this presentation, NAS can be considered part of the unified storage solution


FCoE: Why a New Option for FC Customers? • FC: large and well managed installed base

–  Leverage FC expertise / investment –  Other convergence options not incremental for existing FC

• Data Center solution for I/O consolidation • Leverage Ethernet infrastructure and skill set

FCoE allows an Ethernet-based SAN to be introduced into an FC-based Data Center

without breaking existing administrative tools and workflows


FCoE Extends FC on a Single Network

Network Driver

FC Driver

Converged Network Adapter

Server sees storage traffic as FC

FC network

FC storage

Ethernet Network

FCoE Switch

Lossless Ethernet SAN sees host as FC

Ethernet FC


Time to Widespread Adoption 1990 2000 2010 1980

Ethernet

Fibre Channel

Defined 73

Standard 83

Widespread 93

Defined 85

Standard 94

Widespread 03

iSCSI

Defined 00 04

Widespread 08

Standard

Standard

10 Gigabit Ethernet 02 09

Widespread

FCoE 07 09 ??

Defined Standard

Widespread


Further Developments: What’s Next?

16 GFC

40/100 Gb Ethernet

32 GFC

•  40 & 100 Gb Ethernet (IEEE) standards completed in June 2010 •  16GFC (T11) standard completed in January 2011, 32GFC is next (2012

target) –  FC throughput doubles, encoding change optimizes analog bandwidth

•  8GFC: 800 MB/sec max, 8.5 Ghz •  16GFC: 1600 MB/sec max, 14.025 Ghz


Agenda

• Network Convergence • Protocols & Standards • Solution Evolution • Conclusion and Summary


iSCSI Introduction •  Transport storage (SCSI) over standard Ethernet

–  Reliability through TCP

•  More flexible than FC due to IP routing

•  Good performance

•  iSCSI has thrived –  Especially where the server, storage and network

administrators are the same person

Link

IP

TCP

iSCSI

SCSI

IP Network


iSCSI Introduction (continued) •  Standardized in 2004: IETF RFC 3720

–  Stable: No major changes since 2004 –  iSCSI Corrections and Clarifications: IETF RFC 5048 (2007)

•  iSCSI Session: One Initiator and one Target –  Multiple TCP connections allowed in a session

•  Important iSCSI additions to SCSI –  Immediate and unsolicited data to avoid round trip –  Login phase for connection setup –  Explicit logout for clean teardown


iSCSI Read Example

Optimization: Good status can be included with last “Data in” PDU

Command Complete

Receive Data

SCSI Read Command

Initiator Target

Status

Data in PDU

Target Data in PDU

Data in PDU


iSCSI Write Example

Optimization: Immediate or unsolicited data avoids a round trip

Status

Data out PDU

Data out PDU

Data out PDU

Data out PDU

Initiator

R2T

Target

SCSI Write Command

Ready to Transmit (R2T)

Command Complete

Receive Data

Receive Data


CRC Ethernet Header

iSCSI and FCoE Framing •  iSCSI is SCSI functionality transported using TCP/IP for delivery and

routing in a standard Ethernet/IP environment

�  FCoE is FC frames encapsulated in Layer 2 Ethernet frames – No TCP, so Lossless Ethernet required – No IP routing

Ethe

rnet

H

eade

r

FCoE

H

eade

r

FC

Hea

der

FC Payload

CR

C

EOF

FCS

iSCSI Frame IP TCP iSCSI Data

FC Frame

FCoE Frame


FCoE Frame Format

•  1:1 encapsulation of FC frames –  No segmenting of FC frames across

multiple Ethernet frames –  FCoE flow control is Ethernet based

•  FCoE requires Mini Jumbo frames –  Max FC payload size: 2180 bytes –  Max FCoE frame size: 2240 bytes

•  FCoE Initialization Protocol (FIP) –  Discovery: VLAN and FCoE

switches –  FC login to discovered FCoE

switches

Destination MAC Address

Source MAC Address

IEEE 802.1Q Tag

ET = FCoE Ver Reserved

Reserved

Reserved SOF

Encapsulated FC Frame (Including FC-CRC)

EOF Reserved

FCS

Reserved

Bit 0 Bit 31


FCoE Initialization Protocol

•  Native Fiber Channel link: Fiber has exactly 2 endpoints (simple) –  Discovery: Who is at the other end?

•  Response to FC login contains answer –  Liveness: Is the other end still there?

•  Is the optical link lit and synchronized?

•  FCoE virtual link: Ethernet LAN or VLAN has more than 2 endpoints –  Discovery: Choice of endpoints (FCoE switches)

•  Where should the FC login be sent? –  Liveness: FCoE virtual link may span multiple Ethernet links

•  What if attached link is ok, but some other link is not?

•  FCoE configuration concern: Do mini jumbo frames (2.5k) work? •  FIP: FCoE Initialization Protocol

–  Discover other endpoint, create and initialize virtual link with FCoE switch –  Mini jumbo frame support: Large frame is part of discovery –  Periodic LKA (Link Keep Alive) messages after initialization

Ethernet is more than a cable


FCoE Switch Discovery Step 1: FIP Solicitation

FCoE/FC Switches

DCB Ethernet

FC SAN

�  Select FCoE VLAN first (pre-configured or use FIP) �  Solicitation is multicast: Server can discover multiple switches

�  Solicitation identifies Server (FC WWN for FCoE CNA)

–  CNA = Converged Network Adapter (FCoE analog of HBA) –  Switch may choose which servers to respond to (default: respond to all)

Solicitation

Server


FCoE Switch Discovery Step 2: FIP Advertisement

FCoE/FC Switches

DCB Ethernet

FC SAN

�  Advertisement identifies switch (FC WWN) –  Multiple switches may respond, advertisement includes priority –  Server chooses FCoE switch by priority (smallest number wins)

�  Advertisement padded to max FC frame size: Test mini jumbo frame support

Advertisement

Advertisement

Priority = 1

Priority = 25

Server


FIP Switch Discovery Step 3: FIP-based FC Login

FCoE/FC Switches

DCB Ethernet

FC SAN

�  FIP encapsulated FC Login –  Server sends FC Fabric Login (FLOGI) to selected switch –  Switch responds with FC FLOGI ACC (accept) that contains assigned FCID

�  Subsequent traffic: Standard FC frames (FCoE encapsulated)

Priority = 1

Priority = 25

FLOGI

FLOGI ACC

Server


FCoE and Ethernet Standards

Fiber Channel over Ethernet (FCoE) –  Developed by International

Committee for Information Technology Standards (INCITS) T11 Fiber Channel Interfaces Technical Committee

–  Fiber Channel over Ethernet allows Fiber Channel traffic over Ethernet

–  FC-BB-5 standard ratified June 2009

–  FC-BB-6 in process to expand solution

Data Center Bridging Ethernet (DCB) –  Developed by IEEE Data Center

Bridging (DCB) Task Group –  DCB Ethernet drops frames as

rarely as Fiber Channel –  Technology commonly referred to as

Lossless Ethernet –  IEEE standards: final approval

March 2011 –  DCB: Required for FCoE –  DCB: Enhancement for iSCSI

Two complementary standards efforts

Companies working on the standard committees Key participants: Brocade, Cisco, EMC, Emulex, HP, IBM, Intel, QLogic, Oracle(Sun), others


FC-BB-6 – Major features •  Direct connection of servers to storage

–  PT2PT [point to point]: Single cable –  VN2VN [VN_Port to VN_Port]: Dedicated Ethernet LAN or VLAN

•  Better support for FC fabric scaling –  Distribute logical FC fabric switch functionality –  Enables every DCB Ethernet switch to participate in FCoE

For more, see Erik Smith’s (EMC E-Lab) presentation: FCoE - Topologies, Protocol, and Limitations

Tues 5:00pm and Wed 4:15pm


Lossless Ethernet (DCB) •  IEEE 802.1 Data Center Bridging (DCB)

•  Link enhancements: standardized, initial products available 1.  Enhanced Transmission Selection (ETS) 2.  Priority Flow Control (PFC) 3.  Data Center Bridging Exchange Protocol (DCBX)

•  DCB-enabled LAN: Network portion that must be lossless –  Generally limited to data center distances per link –  Can use long-distance optics, but uncommon in practice

Enhanced Ethernet provides the Lossless Infrastructure that enables FCoE


Enhanced Transmission Selection DCB part 1: IEEE 802.1Qaz Management framework for link bandwidth

•  Priority configuration and bandwidth reservation –  E.g., HPC & storage traffic: higher priority & reserved bandwidth

•  Bandwidth utilization –  Unused higher priority

bandwidth available to other traffic

•  Low latency assured to higher priority traffic

Offered Traffic

t1 t2 t3

10 GE Link Realized Traffic Utilization

3G/s HPC Traffic 3G/s

2G/s

3G/s Storage Traffic 3G/s

3G/s

LAN Traffic 4G/s

5G/s 3G/s

t1 t2 t3

3G/s 3G/s

3G/s 3G/s 3G/s

2G/s

3G/s 4G/s 6G/s


Switch A Switch B

PAUSE and Priority Flow Control DCB part 2: IEEE 802.1Qbb & 802.3bd •  Classic PAUSE can produce lossless Ethernet behavior

–  Classic 802.3x PAUSE stops all traffic: Rarely implemented

•  New PAUSE: Priority Flow Control (PFC) –  Pause per priority level –  No effect on traffic at other priority levels –  Creates lossless virtual lanes

•  Per-priority link flow control –  Enabled/disabled by priority

•  Only affect traffic that needs it

–  More than 8-way 802.3x PAUSE


DCBX ─ Data Center Bridging Capability eXchange DCB part 3: IEEE 802.1Qaz (again)

•  Ethernet Link configuration (single link) –  Extends Link Layer Discovery Protocol (LLDP)

•  Reliably enables lossless behavior (DCB) –  e.g., exchange Ethernet priority values for FCoE and FIP

•  FCoE virtual links should not be instantiated without DCBX

FCoE/FC Switches

DCB Ethernet

FC SAN Server

DCBX


Ethernet Spanning Trees and FCoE •  Reminder: FCoE is Ethernet only, no IP routing

–  Ethernet (layer 2) is bridged, not routed •  Spanning Tree Protocol (STP): Prevents (deadly) forwarding loops

–  Elects a Root Switch, disables redundant paths to create a tree •  Causes problems in large Ethernet networks

–  No network multipathing –  Inefficient link utilization

SiSiSiSi

SiSi SiSi SiSiSiSi SiSi

Root Switchè


SiSiSiSi

SiSi SiSi SiSiSiSi SiSi

TRILL – Transparent Interconnection of Lots of Links •  Layer 2 routing among Ethernet switches

–  In contrast to IP routing at layer 3 –  IS-IS routing protocol for inter-switch Ethernet traffic –  Blocks Spanning Tree Protocol

•  TRILL encapsulates Ethernet frames –  Not used with end systems (NICs) –  NICs use link teaming/aggregation

All links active è


Ethernet Cabling Choices

Type / Connector Cable 1Gb 10Gb 40/100Gb

Copper (10GBase-T) / RJ-45

Cat6 or Cat6a

Majority of existing cabling (e.g., Cat 5e)

Some products on market, but not for FCoE yet

Cat 6 55m Cat 6a 100m

Not supported

Optical (multimode) / LC

OM2 (orange) OM3 (aqua)

OM4 (aqua)

Rare for Ethernet Typical for FC

Most backbone deployments are optical OM2 82m OM3 300m

Expect shift to optical w/ 40/100Gb OM3 100m

OM4 125m

Copper / SFP+DA (direct attach)

Twinax N/A Low power 5-10m distance (Rack solution)

Different short-distance option (QSFP)


Shared

Virtual Machines and Storage Resources Private Storage Resources

–  Accessed directly by VM •  Device driver in VM’s OS

–  Managed as part of VM •  Not visible to virtualization

management (e.g., vCenter) –  If disk is local: No vMotion

Shared Storage Resources –  Accessed by Hypervisor

•  Device driver in hypervisor –  Managed as part of

virtualization (e.g., vCenter)

Private


virtual switch Hypervisor driver

Storage Drivers and Server Virtualization

NIC NIC FC HBA

FC HBA

vNIC vNIC vSCSI vSCSI

LAN traffic FC traffic

Hypervisor

iSCSI traffic *iSCSI initiator can also be in the VM (Private Storage)


virtual switch Hypervisor driver

Storage Drivers and Server Virtualization

NIC NIC FC HBA

FC HBA


CNA

CNA

LAN traffic FCoE follows FC path

Hypervisor

iSCSI traffic

*iSCSI initiator can also be in the VM (Private Storage)


Software FCoE and Server Virtualization

NIC NIC FC HBA

FC HBA


Hypervisor

FCoE software in VMs would send traffic through the virtual switch to the NICs

SW FCoE

SW FCoE

Hypervisor driver

virtual switch

Virtual Switches in ESX/ESXi

(including Cisco Nexus 1000v) and Hyper-V are not

Lossless (not DCB) Not a problem for

iSCSI, NFS or CIFS in a Virtual

Machine


Storage Virtual Appliance (SVA): Sharing Private Storage

Storage Virtual Appliance –  Virtual Machine that provides

storage to hypervisors –  Direct physical storage access

•  e.g., RDM (Raw Device Mapping)

–  Exports storage to hypervisors •  ESX & ESXi: iSCSI (or NFS)

–  Avoid vMotion for SVAs

This SVA example: Simplified –  Availability: Multiple SVAs

mirror or RAID across servers –  Scale: SVAs provide shared

storage to vSphere server cluster

iSCSI


Agenda

• Network Convergence • Protocols & Standards • Solution Evolution • Conclusion and Summary


FCoE and iSCSI

FCoE

FC expertise / install base

FC management

Layer 2 Ethernet

Use FCIP for distance

Ethernet

Leverage Ethernet/IP expertise 10 Gigabit Ethernet

Lossless Ethernet

iSCSI

No FC expertise needed

Supports distance (Layer 3 IP routing)

Strong virtualization affinity


iSCSI Deployment •  10 Gb iSCSI solutions available

–  Traditional Ethernet •  TCP recovers from dropped packets

–  Lossless Ethernet (DCB)

•  iSCSI: natively routable (IP) –  Can use VLAN(s) to isolate traffic

•  iSCSI solutions: smaller scale than FC

–  Single FC director : larger than most iSCSI environments

Ethernet

iSCSI SAN


•  FCoE with Converged Network Switch at top of rack or end of row

•  Tightly controlled solution •  Server 10 GE adapters may be CNA or

NIC •  Storage still a separate network

FCoE Server Phase (1)

FC HBAs

1 Gb NICs

Converged Network Switch

Rack Mounted Servers

10 GbE CNAs

FC Attach

Ethernet LAN

Storage

Fiber Channel SAN

Ethernet FC


FC Attach

FCoE Network Phase (2) �  Converged Network Switches move out of

rack into unified network

� Maintains existing LAN and SAN management Overlapping admin domains may compel cultural adjustments



10 GbE CNAs

Ethernet LAN

Storage

Fiber Channel SAN

Ethernet FC

Ethernet Network (IP, FCoE)



FCoE Storage Phase (3) •  Single Ethernet network for IP and storage traffic •  End-to-End Ethernet with native FCoE •  FC/FCoE configured and managed as an FC SAN

–  Leverage FC management skills and procedures



10 GbE CNAs

Ethernet LAN

Storage

FC & FCoE SAN

Ethernet FC

Fiber Channel & FCoE

attach

FCoE Storage


Convergence at 10 Gigabit Ethernet •  Two paths to a Converged Network

–  iSCSI purely Ethernet –  FCoE enables mix of FC and Ethernet (or all

Ethernet) •  FC that you have today or buy tomorrow is compatible

•  Choose based on scalability, management, and skill set



10 GbE CNAs

Ethernet LAN

FC & FCoE SAN

Ethernet FC iSCSI/FCoE

Storage

Fiber Channel & FCoE

attach


EMC and Ethernet •  Best Practices

–  Google “FCoE Tech Book” (FCoE & Ethernet)

•  Services –  Design, Implementation,

Performance and Security offerings for networks

•  Products –  Ethernet equipment for creating

Converged Network Environments


Agenda

• Network Convergence • Protocols & Standards • Solution Evolution • Conclusion and

Summary


Summary •  Converged data center environments can be built using

10Gb Ethernet

•  Achieving a converged network requires consideration of technology, processes/best practices and organizational dynamics

•  10 Gigabit Ethernet solutions are maturing –  Active industry participation is creating standards that allow

solutions that can integrate into existing data centers –  Continued use of FC and adoption of FCoE can be flexible due to

shared management –  FCoE and iSCSI will follow Ethernet roadmap to 40 and 100

Gigabits/sec


Related Session and Resources •  FCoE - Topologies, Protocol, and Limitations

–  Tuesday 5:00p & Wednesday 4:15p •  Birds of a Feather: The Future of Storage Networking

–  Wednesday 8:30a

•  Cisco - Building Cloud-Ready Storage with Cisco and EMC –  Tuesday 10:00a

•  FCoE in the EMC Topology Guide –  http://elabnavigator.emc.com

•  EMC FCoE Videos: Search for “FCoE” on YouTube •  EMC FCoE Introduction whitepaper

–  http://www.emc.com/collateral/hardware/white-papers/h5916-intro-to-fcoe-wp.pdf

•  FCoE Blog by Erik Smith (E-Lab) –  http://www.brasstacksblog.typepad.com


Q&A



THANK YOU