Fc Basics

Fibre Channel Basic

Добринский Павел

Системный инженер

[email protected]

+7-985-922-61-33

Краткий обзор проду ктов Brocade19 сентября 2008 © 2008 Brocade Communications Systems, Inc.

All Rights Reser ved. 2

Fibre Channel – Combining SCSI with

Networking



Think of Fibre Channel As Super SCSI

� Run faster…

� Better cables…

� Connect more…

� Lower overhead…

� Go longer

distance…

� Powerful

management…

Fibre Channel is

like Super SCSI

Fibre Channel is

like Super SCSI

SUPER

SCSI FC FRAME

SCSI FC FRAME

SCSI FC FRAME

Fibre Channel Highway(a new low-level data mover)



Fibre Channel Technical Capabilities

� No emitted radio frequency signals (RFI), thus immune to induced

electromagnetic signals

� Serial data transmission at 1.0625, 2.125, and 4.25Gbit with a transfer speed

of 0.94 nanoseconds per bit (very low latency)

� Using 8-bit/10-bit encoding/decoding to translate 8-bit information to 10-bit

format for serial transmission (clock signal embedded)

� Up to 16 million node connectivity (Switched Fabric mode)

� Variable frame size with a maximum user payload of 2112 bytes. Supports

block transfers up to 128MB in size.

� Superior error correction rate (bit error rate = 1x10-12) 1 bit error each 16 min

at 1gbit

� Supports full duplex transmission(simultaneous send and receive)



Fibre Channel As an Open Standard� Fibre Channel development began in 1988; the NCITS T11: I/O interface

(X3.230-1994) standard was completed in 1994.

� Reference URL: http://www.t11.org

� Endorsed by many other standard parties and vendors

Industry Standards

Business Solutions

Product Interoperability

ANSI DMTF IETF

FC

IA

SNIA

Fib

re Allian

ce

Open, Multi-

Vendor SANs

FC-PH

FC-GS

FC-SW

FC-AL

FC-CT

FC-LS

FC-FG

FC-FLA

1. FC-SB (Single Byte Mapping Protocol)

2. FC-SB-2 (Single Byte Protocol Mapping 2)

3. FC-LE (Link Encapsulation)

4. FC-PH (Physical and Signaling)

5. FC-PH (Physical and Signaling) Amendment 1

6. FC-PH (Physical and Signaling) Amendment 2

7. FC-PH-2 (Physical and Signaling 2)

8. FC-PH-3 (Physical and Signaling 3)

9. FC-FG (Fabric Generi c Requirements)

10. FC-GS (Generi c Services)

11. FC-GS-3 (Generic Services 3)

12. FC-SW (Switch Fabric)

13. FC-SW-2 (Switch Fabric 2)

14. FC-AL (Arbitration Loop)

15. FC-AL-2 (Arbitration Loop 2)

16. FC-BB (Backbone)

17. FC-FP (Mapping to HIPPI-FC)

18. HIPPI-FC (FC-PH Encapsulation)



Fibre Channel HardwareFiber Optic Cables

Glass Core

Glass Cladding

Coating

Glass Core

Glass Cladding

Coating

Multimode Fiber Single-mode Fiber



Fibre Channel Basics

� Wavelengths are expressed in nanometers

� The speed of light in fibre is 2/3 speed of light in vacuum• Light travels at ~5nsec per meter in glass

� Multimode fibre carries numerous modes, or frequencies, and

carries short-wave laser light.

� Single-mode fibre has a smaller core that allows only one mode

of light and carries long-wave laser light.

� Signal loss is caused by dispersion and attenuation.

� Specific regions in the optical spectrum, windows, have low

optical attenuation. 850nm, 1310nm, 1550nm, and 1625nm.

� Optical power budgets, or link loss budgets, measured in decibels

(dBs), are used to manage optical signal loss.



Fibre Channel Hardware - Connectors

HSSDC2 Copper Connector

- Fits in SFP Media Slots

- Smaller than HSSDC

LC Optical Connector

- Standard on 2 Gb/s Switches

- Becoming More Popular

- Bonded or Unbonded

- Usually Purchased as Duplex

Cable Shown as Bonded Duplex

SC Optical Connector

- Standard on 1 Gb/s Switches

- Most Widely Used Connector

- Bonded or Unbonded

- Single or Duplex

Cable Shown as Bonded Duplex

HSSDC Copper Connector

- Smaller than DB-9

- Easier to insert/remove



Fibre Channel HardwareGBICs, SFPs, and HBA’s

Optical LC SFP

Optical SC GBIC

HSSDC2 SFP

HSSDC GBIC

Fibre HBA



Fibre Channel HardwareOptical Media Comparison Chart

40kmN/AN/AN/A10XFP

25kmN/AN/AN/A8SFP+

50, 80,100kmN/AN/AN/A4SFP

30kmN/AN/AN/A2SFP

LW

N/A300m82m33m10XFP

N/A150m50m21m8SFP+

N/A380m150m70m4SFP/SFP

+

N/A500m300m150m2SFP/SFP

+

N/A860m500m300m1SFP

SW

9um

50um/2000M

Hz

(OM3)

50um/500MHz (OM2)

62.5um/200MHz (OM1)

Maximum

DistanceMulti-Mode Media Maximum Distance

Speed,

Gbps

Form

Factor

Transceiver

Type



Brocade SilkWorm HardwareGigabit Interface Connector (GBIC) Options

� SFP LC (2/4/8 Gb/s Products)– Short Wavelength (SWL) Optic (780 nm – 850 nm laser, 50/62.5 µm Multimode)

– Long Wavelength (LWL) Optic (1310 nm laser, 9 µm Single-mode)

– Extended Long Wavelength (ELWL) Optic (1550 nm laser, 9 µm Single-Mode)

– Copper (HSSDC2 Female Connector – 2Gb/s Only)

– All Qualified SFPs are Intelligent

� SC GBICs (1 Gb/s Products)– Short Wavelength (SWL) Optic (780 nm – 850 nm laser, 50/62.5 µm Multimode)

– Long Wavelength (LWL) Optic (1310 nm laser, 9 µm Single-mode)

– Extended Long Wavelength (ELWL) Optic (1550 nm laser, 9 µm Single-Mode)

– Passive Copper

– Active Copper



Fibre ChannelFibre Channel Protocol Layers

Fast File

TransfersStreams IPI SCSI HIPPI SBCCS 802.2 IP FC - CTBB

Common Services

Framing Protocol / Flow Control / COS

Encode / Decode / PSM & LPSM

Audio/Video Channels Networks FC Services

133

Mbaud

266

Mbaud

531

Mbaud

1,063

Mbaud

2,125

Mbaud

10.63

Gbaud

FC - PH

Media: Optical – Laser, LED

Copper – Coax, Twisted Pair

FC-1

FC-2

FC-0

FC-3

FC-4

Common transport for

upper-layer protocols

4,250

Mbaud

8,500

Mbaud



Fibre Channel Layered Architecture

Fibre Channel

Physical

Data Link

Network

Transport

Session

Presentation

Application

OSI Reference Model

Data & Applications

FC-0: Physical Interface

FC-3: Common ServicesFC-2: Data DeliveryFC-1: Byte Encoding

FC-4: Upper-LayerProtocols Mapping

Hard

ware

With fewer processor interruptions, application can run faster.



Flexible Deployment Topologies

Point-to-Point

Only 2 Devices

(Direct Connect)

Switched Fabric

Up to 16 Million Devices

(Fibre Channel Switches)

Arbitrated Loop

Up to 126 Devices

(Fibre Channel Hubs)



Fibre ChannelArbitrated Loop

Fibre Channel Arbitrated Loop - the transmit of each port is connected to the receive of the next port

– Reduced cost path into FC “SCSI Replacement”

– May use FC Hub technology

– Easy for vendors to develop

– Difficult for customers to deploy

– Limited possible ports (126) plus the Loop Master (FL_Port)

– Lower overall throughput – 100/200MB maximum bandwidth

– Limited any to any connectivity - ports on the loop have to arbitrate for control of the loop in order to be able to communicate with a another port on the loop. While this communication is happening all other ports are waiting to get their turn.

– Primitive flow control causing QOS issues

FC-AL was developed to fill the gap between the limited capabilities of point-to-point and the relatively expensive switched fabric. Since only two devices can communicate at a time, the loop is a blocking topology, which means the bandwidth is shared between all devices on the loop.



Fibre ChannelSwitched Fabric

� Large connectivity on non-shared media, which allows concurrent communicating pairs

� Highest performance level

� High scalability

� Good fault isolation

� Embedded management and services



FC Protocol Levels

EXCHANGE

(Transaction)

EXCHANGE

(Transaction)EXCHANGE

(Transaction)

ULP

FC-4

1st Information Unit Information Unit Last Information Unit

1st Sequence Last SequenceSequence

FC-3 Frame 1 Frame 2 Frame n-1 Frame nFrame

FC-2SOF

Ordered Set

Data

Word

Data

Word

EOF

Ordered SetData

Word

Data

Word

FC-18B/10B

Character

8B/10B

Character 8B/10B

Character

8B/10B

Character

8B/10B

Character

8B/10B

Character

FC-0bit

k

bit

l

bit

m

bit

n

bit

o

bit

p

bit

q

bit

r

bit

s

bit

t

bit

u

bit

v

bit

w

bit

x

bit

y

bit

z

bit

a

bit

b

bit

c

bit

d

bit

e

bit

f

bit

g



SCSI-FCP Write Command

IU 3Status

IU 2Data

IU 1Command Sequence 1SI SRFrame

Initiator Node Target Node

Frame SISR

Sequence 4

Sequence 3

StatusChunk

CommandChunk

DataChunks

WriteRequest

Request

to Send ChunkFrames

SISR

SISRSequence 2Frame

Exchange



SCSI-FCP Write Command cont.

Initiator Target

write request [seq_cnt 0]S_ID, D_ID, LUN, LBA, Length

x_fer ready [s

eq_cnt0]

S_ID, …, corre

cted Length

*

status good [s

eq_cnt0]

data n [seq_cnt n-1]

data 1 [seq_cnt 0]

Exchange

ox_id

Sequence

seq_id

Have to be

last frame!!!

* Depends on free buffers (i.e. 10MB should be written but only 2MB free buffers avail. – 2MB is sent back and after 2MB the next x_fer ready is sent)



SCSI-FCP Read Command

Sequence 1

IU 3Status

IU 2Data

IU 1Command

SI SRFrame

Target Node

Frame SISR

Sequence 3

Frames

SISR

Sequence 2

StatusChunk

CommandChunk

DataChunks

Initiator Node

Exchange



FCP frame control

� Data Droop



FCP frame control (i.e. SCSI-FCP Write Command)

Initiator Switch Targetwrite req (BB -1)

r_rdy(BB+1

)

write req (BB -1)

r_rdy(BB+1

)

x_ferrdy (

BB-1)

r_rdy (BB +1)x_ferrdy (

BB-1)

r_rdy (BB +1)

data 1 (BB -1)

r_rdy(BB+1

)data 1(BB -1)

r_rdy(BB+1

)

data n (BB -1)

r_rdy(BB+1

)data n (BB -1)

r_rdy(BB+1

)

status good

(BB-1)

r_rdy (BB +1)status good

(BB-1)

r_rdy (BB +1)

sequence

sequence

sequence

sequence

If BB=0 (i.e. lost r_rdyframes) the link will be reseted by sending LinkCreditReset (LR) and LinkCreditResetResponse(LRR).



Fibre ChannelFrame Format

S

O

FHEADER

C

R

C

E

O

FPAYLOAD

4 24 4 4Up to 2112

FRAME

2148 Bytes

Payload6-nPayload

Parameters5

RX_IDOX_ID4

SEQ_CNTDF_CTLSEQ_ID3

F_CTLTYPE2

S_IDCS_CTL1

D_IDR_CTL0H

E

A

D

E

R

Bits 7-0Bits 15-8Bits 23-16Bits 31-24Word



Fibre ChannelClasses of Service – There are 6 classes of service

� Class 1

Dedicated connection port to port with ack. It takes all the bandwidth until one port does a port logoff

� Class 2

It is is connectionless with ack: one <-> many. No bandwidth is allocated or guaranteed. IP uses this class. > Supported by Brocade

� Class 3

It is connectionless with no ack: buffer to buffer flow control. Errors are handled at higher level. No bandwidth is allocated or guaranteed. FCP uses this Class. > Supported by Brocade

� Class 4

It is connection oriented with ack. Ues virtual circuites. Can allocate requested amountof bandwidth (used in video)

� Class 6

Multicast, also known as uni-directional dedicated connection with ack used sometimes in avionics

� Class F

Switch to switch. It is connectionless with ack. > Supported by Brocade



Node

N-port

Switch

F-port

FC-2 Advanced Fabric Flow ControlHow Buffer to Buffer Credits Work

� A Fibre channel link is a PAIR of paths

� A path from "this" transmitter to the "other" receiver and a path from the "other" transmitter to "this“ receiver

� The "buffer" resides on each receiver, and that receiver tells the linked transmitter how manyBB_Credits are available

� Sending a frame through the transmitter decrements the B2B Credit Counter

� Receiving an R-Rdy through the receiver increments the B2B Credit Counter

Fiber Cable

transmit receive

receive transmit

I have 16 buffer credits

I have 107 buffer credits

BufferBuffer--toto--Buffer Credits are not negotiated!Buffer Credits are not negotiated!

And each receiver on the fiber cable And each receiver on the fiber cable

can state a different value!can state a different value!

16Avail.

B2B

Credit Cnt

107

Avail.

B2B

Credit Cnt

BC

Pool

BCPool

2 - 60



BB Credits

BB Credits

BB Credits

BB Credits

BB Credit Flow – Always Between Units

ISL

Director Director

BB Credits

BB Credits

Server Storage



Buffer´s and distance

1 Gbps or100 MBps

2 Gbps or200 MBps

4 Gbps or400 MBps

10 Gbps or

1000 MBps

Typical Data Frame is 2112 Bytes in Size

1km 2 km 3 km 4 km0 km

BB Credit Information Is Exchanged During FLOGI

1 Buffer Credit (BC) 1 Buffer Credit (BC)

0.5 BB Credit/Km

1 BB Credit/Km

2 BB Credit/Km

6 BB Credit/Km

8 Gbps or

800 MBps4 BB Credit/Km



FCP frame control



Fibre ChannelBuffer Credits

S

O

FHEADER

C

R

C

E

O

FPAYLOAD

BUFFER CREDIT

Buffer credits determine the amount of data in transit at any time. A buffer credit allows a device to send a frame before an R_RDY (receiver ready) is received. As the data

requirements increase, so should the buffer credits. The physics of light dictate that a fiber has a latency of 5ns/meter, or 5usec/km. The round-trip latency over 10km would

be 100usec. The longer the distance, the more credits needed. When talking about speeds of 2Gbps, the credits must be doubled, since the time is halved.

HBAs need to be able to support high numbers of credits. Some HBAs storeI/O information, others relay it to the server. This can add to the throughputlatency. More expensive HBAs will have a context cache to hold such

information.



SAN Port InterfacesFC Node WWN Format



SAN Port InterfacesFC Port WWN Format



SAN Port InterfacesNode WWN and Port WWN



Fibre ChannelPort Types

� Device Ports

– N_Port - Node Port, a Fabric device directly attached

– NL_Port - Node Loop Port, a device attached to a loop

� Switch Ports

– U_Port - Universal Port, a port waiting to become some other port

type

– G_Port - Generic Port, a port waiting to be an E or F_Port

– F_Port - Fabric Port, a port to which an N_Port attaches

– FL_Port – Fabric Loop Port, a port to which a loop attaches

– E_Port – Expansion port used for inter-switch links



Fibre ChannelExpansion Port – E_Port

� Allows two switches to be connected to create a

multi-switch Fabric

� Connections are called ISLs (Inter Switch Links)

� Also used to create trunks (eight ISLs combined)

� Supports 16 virtual channels per ISL



Fibre ChannelBrocade Specialization – Virtual Channels

Eight Virtual Channels improve performance

– VC 0 - For Link Control Frames

– VC 1 - For Class 2 ACKs and Link Control Frames

– VC 2 - For Data

– VC 3 - For Data

– VC 4 - For Data

– VC 5 - For Data

– VC 6 - For Multicast traffic

– VC 7 - For Broadcast traffic

– VC for QoS

Switch E SwitchEOne ISL,

16 Virtual Channels

Condor2 increasesData VCs to 12

(total 32)



Fibre ChannelBrocade Specialization – Virtual Channels (Cont.)

Initiators

E_Port

E_Port

Targets

1 x ISL4 x Data VCs

� Each physical link is partitioned into eight virtual channels

� Each VC has independent flow control

� Bottlenecks on one VC do not impact other VCs

� An anti-starvation mechanism prevents blocking



SAN Port InterfacesFC Addressing

PORT ID: XX YY ZZ

where:

XX Domain is a value between 0x1 to 0xEF

YY is the port number (0-255)

ZZ is the AL_PA for a loop device

or 00 for a F_Port or any Value for NPIV



Fibre ChannelAddress Scheme

Private Loop

Address

Fabric Assigned

Address

Public Loop

Address

00 00 PP NN NN 00 LL LL PP

Where “PP” = the

local loop address

(AL_PA). The private address only

uses the last byte (8 bits) of the 24-bit

Fabric address.

Where “NN NN 00” =

the address of any

Fabric-attached device that has

logged into the Fabric. These

devices use all 24

bits of the address.

Where “LL LL” is

assigned by the

Fabric at login; and “PP” = the local

loop address (AL_PA). These

devices use all 24

bits of the address, and the last 8 bits is

the AL_PA.



Fibre ChannelAddressing Examples

Private

NL

FL

FC-AL

NL

00 00 CA

Private

00 00 08

NL

Public

09 0E 04

Switch

Domain#9

00 01 02 03

15

14

13

12

11 10 09 08

04

05

06

07

NL

Public

09 01 01

FL

N

Fabric

09 04 00

F

FabricN

09 0A 00

F

Brocade Fabric Services Overview



Key Fabric Services

� Distributed Name Server– Standards-based network address assignment– Dynamic scalability

Auto Discovery – Zero Administration

� Fabric Controller– SCR registrations, RSCN messages delivery

� FSPF Routing– Scalability and Cascading, Any port to any port

Redundant, No single points of failure

� Zoning– WWN and Port based

Software and Hardware Enforcement



Fibre ChannelRecognizing Well Known Fabric Addresses



Brocade Fabric Services OverviewDistributed Functional Model at FFFFFC

DistributedNameServer






Brocade Fabric Services OverviewPort and Node Attributes in Name Server database

� Port attributes

– Port Identifier (Native port address ID)

– Port Name (World Wide Name)

– Class of Service (2, 3)

– FC-4 Types (SCSI, IP)

– Port Type (N, NL)

– Symbolic Port Name (free-form information)

� Node attributes

– Node Name (World Wide Name)

– Fibre Channel IP Address

– Initial Process Associator

– Symbolic Node Name (free-form information)



Brocade Fabric Services OverviewDevice Communication Example



Brocade Fabric Services OverviewRegistered State Change Notifications (RSCN)

Functional View

Fabric Controller

Well-Known AddressFFFFFD

HBAs need to be “good citizens” to

work properly with the Name Server. The attributes of a good citizen are:

- It supports RSCNs

- It queries the Name Server for available ports

- It accesses only ports that are defined by the Name Server



Brocade Fabric Services OverviewRSCNs

The switch will deliver an RSCN to an Nx_Port only if the following happens:

� The Nx_Port has registered (using SCR) to receive RSCNs

� The Nx_Port is still logged into the Fabric

� The registration function matches the RCSN Type

� The domain and area fields of the affected Nx_Port are different from the destination Nx_Port

Sent to an end device at most once every 500 ms

Three formats of RSCNs:

• Single PID (same as existing behavior)

• Multiple PID (multiple PIDs are aggregated into one payload)

• Fabric RSCN (single fabric format RSCN sent)



Brocade Fabric Services OverviewState Changes and Notifications

� SCN – State Change Notification, used for internal state

change notifications only. This is the switch logging that the

port is online or is an Fx_Port.

� SCR – State Change Registration, issued (requested) by an

Nx_Port.

� RSCN – Registered State Change Notification, issued

(responded) by the Fabric or an N_Port.

Fibre Channel Routing



Fibre Channel RoutingFabric Terminology

Inter-switch Links (ISLs)

– E_Port-to-E_Port links

– Communicates using Class F service

– Will be segmented if link parameters are incompatible

Principal Switch

– Selected when the fabric initializes, before routing is established

– Manages the assignment of unique domain IDs

– Provides time synchronization to all other switches in the fabric

Principal ISL

– ISL used to communicate between the Principal Switch and otherswitches in the fabric



Fibre Channel RoutingPrincipal Switch Path

Domain 4

Domain 1

Domain 6Domain 3

Domain 5Domain 2

Switch 4 is Principal Switch (fabricprincipal Lowest WWN)

Upstream: noneDownstream: 4-3,4-6

Upstream: 3-4Downstream: 3-2,3-5

Upstream: 6-4Downstream: none

Upstream: 5-3Downstream: 5-1



Principal ISL

ISL



Fibre Channel RoutingWhat is a Path?

A Path is a chain of switches from source to

destination

Domain1

Domain2

Domain3

Domain4

Source011F00

JBODDestination

041600



Fibre Channel RoutingUsing a Path

Two valid paths from source host to destination storage

12

3

14

7

11 5

2

13



Fibre Channel RoutingWhat is a Route?

A route is a map to reach the next hop between an input

port and an output E_Port

12

3

14

7

11 5

2

13

13

4

Domain 1

Domain 2

Domain 3



Fibre Channel RoutingFSPF Algorithm

Routing Algorithm

Uses cost/weight and bandwidth

Can apply static routes if needed

Automatic Failover

- Fault detection 150ms

- Self heals in 500ms

- Alternate route live in 650ms



Fibre Channel RoutingPath and Route Selection Architecture

Paths:

Fully distributed - No single point of failure within the Fabric

Fast recovery in case of link or switch failure; identifies switches by domain ID

Traffic load sharing over equivalent paths

Link State Protocol — “Fabric Shortest Path First” (FSPF)

Finds the shortest path to each domain, then programs the hardware routing tables

Routes:

Dynamically (all SilkWorm® switches)

– Round robin

Statically (for the SilkWorm 2000 series and above)

– Administrator can configure the route

– Automatically re-routes upon ISL going away and static routing will again take effect upon ISL return



Fibre Channel RoutingRouting policy

� The routing policy is responsible for selecting a route based onone of two user-selected routing policies

• Port-based Routing

• Exchange-based Routing

� 2 Gbit/sec ASIC routing is handled by the Fabric Shortest Path First (FSPF) protocol and uses only the port-based routing policy

� 4 and 8 Gbit/sec ASICs use the FSPF protocol and either port-based routing or exchange-based routing (default)

� Each switch has its own routing policy

� Different policies can exist in the same fabric



Fibre Channel RoutingPort-Based Routing Policy

� FSPF calculates the cost of each link and determines the lowest cost path

• Within each switch, the input port from the source is assigned to an output port toward the destination (a route)

• Routes are allocated via round-robin assignment

• Chosen routes are used until one of the devices in the fabric goes offline or the fabric changes

• Changes in fabric, when Dynamic Load Sharing is enabled, cause FSPF to recalculate the routes and may reassign the output port to better distribute devices across equal cost routes

� Dynamic Load Sharing (DLS) and In-Order Delivery (IOD) options

� 2 Gbit/sec ASICs support only the port-based routing policy

� 4 and 8 Gbit/sec ASICs support port-based routing and exchange-based routing



Fibre Channel RoutingPort Based Routing

Round robin allocates a route from shortest equivalent paths based on link cost

FSPF default link costs� 1000 at 1 Gbit/sec

� 500 at 2 / 4 / 8 Gbit/sec



Fibre Channel RoutingPort Based Routing (cont.)

Source ID and destination domain are used to allocate routes

Devices are round-robin allocated to available equal cost routes

It is possible to have congestion, if too many high I/O requiring devices are allocated to a single route



Fibre Channel RoutingExchange-based Routing

� Frames in an exchange are identified by common

characteristics called frame parameters

� Exchange-based routing policy exchange frame

parameters include: (DID/SID/OXID)• The default policy on 4 / 8 Gbit/sec switches

• Available on 4 / 8 Gbit/sec switches only

� Cannot create static routes

� DLS always is enabled and it cannot be switched

off



Fibre Channel RoutingExchange-based Routing (cont.)

The exchange-based routing policy uses an internal hash to allocate aparticular SID, DID, OXID exchange to an ISL

Exchanges are allocated proportional to the bandwidth available on each of the routes

All sequences of frames within a SID, DID, and OXID exchange (the same hash) will traverse an assigned ISL



Fibre Channel RoutingDisplay Routing Information Overview

Paths and link connections can be verified using switchshow,fcping, and pathinfo

The FSPF protocol determines routing on a local basis

� Fabric topology information is known at every switch –displayed with urouteshow, topologyshow & other routehelp commands

� No global, edge-to-edge routing table is maintained

For a listing of route-related commands use routehelp



Fibre Channel RoutingDynamic Load Sharing

dlsSet

� This command allows load sharing to take place when a

fabric change occurs.

dlsReset

� This command prevents load sharing from taking place

when a fabric change occurs.

dlsShow

� This command displays the state of the Dynamic Load

Sharing option.



Fibre Channel RoutingRoute Selection (dlsSet Example)

00 01 02 03

15

14

13

12

11 10 09 08

04

05

06

07

00 01 02 03

15

14

13

12

11 10 09 08

04

05

06

07

Host 1

(On-Line 1st)

Host 2

(On-Line 2nd)

Host 3

(On-Line 3rd)



Fibre Channel RoutingRoute Selection (dlsSet Example Cont.)

00 01 02 03

15

14

13

12

11 10 09 08

04

05

06

07

00 01 02 03

15

14

13

12

11 10 09 08

04

05

06

07

Host 1

Host 2

(Off-Line)

Host 3

With DLS not set connections are not re-routed



Fibre Channel RoutingRoute Selection (dlsSet Example Cont.)

00 01 02 03

15

14

13

12

11 10 09 08

04

05

06

07

00 01 02 03

15

14

13

12

11 10 09 08

04

05

06

07

Host 1

Host 2(Off-Line)

Host 3

With DLS set connections are re-routed



Fibre Channel RoutingIn Order Delivery

iodSet

� This command enforces in-order delivery of frames during

fabric topology changes.

iodReset

� This command allows out-of-order delivery of frames during

fabric topology changes.

iodShow

� This command indicates if IOD is turned on or not.



Fibre Channel RoutingIn Order Delivery (iodSet Example)

00 01 02 03

15

14

13

12

11 10 09 08

04

05

06

07

00 01 02 03

15

14

13

12

11 10 09 08

04

05

06

07

Host 1

Host 2

Host 3

1

Frames

In

Queue



Fibre Channel RoutingIn Order Delivery (iodSet Example Cont.)

00 01 02 03

15

14

13

12

11 10 09 08

04

05

06

07

00 01 02 03

15

14

13

12

11 10 09 08

04

05

06

07

Host 1

Host 2

(Off-Line)

Host 3

Frames

InQueue

1



Fibre Channel RoutingIn Order Delivery (iodSet Example Cont.)

00 01 02 03

15

14

13

12

11 10 09 08

04

05

06

07

00 01 02 03

15

14

13

12

11 10 09 08

04

05

06

07

Host 1

Host 2

(Off-Line)

Host 3

1

Frames

In

Queue

2

With “iodReset”, frame 2

arrives before frame 1



Fibre Channel RoutingSetting a Link Cost Metric

linkCost <port #>,<value>

sw2:admin> linkCost 3,250

The default metrics are:

1000 at 1 Gbps

500 at 2/4/8 Gbps



Fibre Channel RoutingRoute Selection Default Behavior

00 01 02 03

15

14

13

12

11 10 09 08

04

05

06

07

00 01 02 03

15

14

13

12

11 10 09 08

04

05

06

07

Host 1

(On-Line 1st)

Host 2

(On-Line 2nd)

Host 3

(On-Line 3rd)

00 01 02 03

15

14

13

12

11 10 09 08

04

05

06

07500

50

0

500500

linkCost Default = “500”Domain 1 has 3 routes to Domain 3

� 2 routes have total metric = 500

� 1 route has total metric = 1000

(never used)

Only routes with least total metric will be

kept in routing tables

Domain 2

Domain 1

Domain 3



Fibre Channel RoutingRoute Selection Using Cost

00 01 02 03

15

14

13

12

11 10 09 08

04

05

06

07

00 01 02 03

15

14

13

12

11 10 09 08

04

05

06

07

Host 1

(On-Line 1st)

Host 2

(On-Line 2nd)

Host 3

(On-Line 3rd)

00 01 02 03

15

14

13

12

11 10 09 08

04

05

06

07250

25

0

500500

Command invoked from Domain 1:

sw1:admin>linkCost 10,250

Command invoked from Domain 2:

sw2:admin>linkCost 7,250

Domain 2

Domain 1

Domain 3

linkCost is not bi-directional!



Fibre Channel RoutingRoute Selection Using Cost

00 01 02 03

15

14

13

12

11 10 09 08

04

05

06

07

00 01 02 03

15

14

13

12

11 10 09 08

04

05

06

07

Host 1

(On-Line 1st)

Host 2

(On-Line 2nd)

Host 3

(On-Line 3rd)

00 01 02 03

15

14

13

12

11 10 09 08

04

05

06

07250

25

0

500500

No linkCost commands issued from Domain 3 to Domain 2 or Domain 2 to Domain 1

Domain 2 Domain 3500

50

0

Domain 1



Switch Latency is NOT an issue

~4-13,000

Micro-seconds

~1000 microseconds

2 microseconds

2 microseconds

DiskSubsystem

Latency

Switch HopLatency

StorageCache hit

StorageCache miss

TotalLatency

SAN Design



Key SAN Design Decisions

Fabric size and number of fabrics

Selection of switching elements- Number of ports

- Bandwidth

- RAS (Reliability/Availability/Serviceability)

Fabric architectural model- Flat

- Core/Edge

- Inter-Fabric Routed

Geographical considerations- Storage over MAN/WAN

SAN Management and Diagnostic



Fabric Zoning

– The server in the red zone sees one loop of disks

– The server in the blue zone sees two storage arrays

– The server in the green zone sees one loop and one array

– No server sees loop 2



Basic Design Principles

Zoning Enforcement

� Session Enforcement

- Name Server restricts PLOGIs

� Hardware Enforcement

- Available through ASIC hardware logic checking

- Denies illegal access from “bad citizens”

- More secure then session

� Enforcement based on how members in a zone are defined



Basic Design Principles

Zoning Rules– Devices to Storage: one to

many

(one HBA port -> to all storage

ports)

– Storage to Storage: one to one

– Use Aliases for WWNs or Ports

– Do not mix WWN and Port

zoning

(if possible)



Brocade (Port Based) Trunking

– Provides trunked ISL Bandwith up to 64 Gbit/sec (Condor2)

FOS: Dynamic Path Selection (DPS) – EOS: Open Trunking

(OT)

– Balances loads across trunk groups at the exchange level

– Preserves in-order delivery within an exchange

Basic Design Principles (cont’d)ISL Optimization

Condor-ASIC



Optical Budget

Optical Budget is affected by:

– Fiber attenuation (Loss = 0.25db/km for 1550nm)

– Splices

– Patch Panels/Connectors (add 0.5db per connector)

– Optical components (filters, amplifiers, etc)

– Bends in fiber

– Contamination (dirt/oil on connectors)

Basic Optical Budget = Output Power – Input Sensitivity

Pout = -15 dBm R = -22 dBm

Budget = 7 dB (28km/1550nm)

This is a parameter used in the design of optical transmission networks. It is the difference between

output power level of the source and the receiver sensitivity. It is generally measured in dB (Decibel).

Fc Basics

Documents