1 Fibre Channel – Chapter 9. 635.412.71 Spring 05 Class #5: Token Ring LANs & Fibre Channel 2 Fibre Channel Introduction n Originally developed for mainframe.

1

Fibre Channel – Chapter 9

635.412.71 Spring 05Class #5: Token Ring LANs & Fibre

Channel 2

Fibre ChannelIntroduction

Originally developed for mainframe & supercomputing environments to connect together high speed clusters & storage

Development began in 1988 under the auspices of the ANSI T11 committee (device level interfaces) and culminated in the approval of the ANSI standard in 1994

Besides its use as a very high bandwidth I/O channel technology, there is increasing interest in Fibre Channel as a LAN technology because of its high speed and unique combination of channel & network oriented properties:

– Data-type qualifiers for routing data into specific interface buffers– Link-level constructs designed to support individual I/O operations– Support for existing I/O interface specifications (SCSI, HIPPI, etc.)– Full multiplexing capabilities– Peer-to-peer connectivity between any two ports in a FC network– Ability to internetwork with other LAN, WAN, & I/O technologies


Channel 3

Fibre ChannelIntroduction

Comparsion of Fibre Channel with Gigabit Ethernet and ATM [Table 9.1] Fibre Channel Gigabit Ethernet ATM

Applications Storage, Network, Video, & CPU Clusters

Network Network, Video, Multimedia

Topologies Point-to-Point, loop/hub, switched

Point-to-Point, hub, switched

Switched

Data Rate 3.2-Gbps 1-Gbps 2.4-Gbps

Guaranteed Delivery

Yes No No

Congestion data loss

Class 3 only Yes Yes

Frame Size Variable: 0-2,148 bytes Variable: 0-1518 bytes Fixed: 53 bytes

Flow Control Credit-based Rate-based Rate-based

Physical Media Twisted Pair, Coax, and Fiber

UTP, Coax, and Fiber UTP and Fiber


Channel 4

Fibre ChannelArchitecture

Designed to provide a common, efficient, high-speed transport to a wide variety of devices through a single port type

Requirements outlined by the Fibre Channel Association:– Full-duplex links over a fiber pair (one transmit/one receive)– Bi-directional performance up to 3.2-Gbps on a single link– Support over distances up to 10 kilometers– Small connectors for high density applications– High-capacity utilization with distance insensitivity– Greater connectivity than existing multi-drop channels– Broad availability at reasonable cost– Support for multiple cost/performance levels, from PCs to clusters– Ability to carry multiple protocols and command sets

The best way to meet such demanding requirements was to develop a transport mechanism based on simple point-to-point links & a switching network


Channel 5

Fibre ChannelTerminology

Fibre Channel, having a different heritage than other LAN/WAN technologies, has different terminology [Table 9.2]

– Dedicated Connection: A circuit guaranteed and retained by the fabric for two specified N_Ports

– Exchange: The basic mechanism that transfers information, consisting of one or more related non-concurrent sequences in one or both directions

– Fabric: The entity that interconnects various N_Ports attached to it and handle the routing of frames

– Intermix: A mode of service that reserves the full FC capacity for a dedicated (Class 1) connection but allows the transport of additional connectionless data if space is available

– Node: A collection of one or more N_Ports– Operation: A set of one or more, possibly concurrent, exchanges

that is associated with a logical construct above the FC-2 layer


Channel 6

Fibre ChannelTerminology (continued)

Fibre Channel, having a different heritage than other LAN/WAN technologies, has different terminology [Table 9.2]

– Dedicated Connection: A circuit guaranteed and retained by the fabric for two

– Originator: The logical function associated with an N_Ports that initiates an exchange

– Port: The hardware entity within a node that performs data communications over a FC link

– Responder: The logical function in a N_Port responsible for supporting an exchange initiated by an originator

– Sequence: A set of one or more data frames with a common sequence ID transmitted unidirectionally from one N_Port to another N_Port, with a corresponding response, if applicable, transmitted in response to each data frame


Channel 7


Fibre Channel Elements– The key elements of a FC network are the end devices called

nodes and the collection of switching elements called the fabric– Communication between nodes across a FC network consists of

transmission of frames across the point-to-point links & fabric– Each node has one or more N_Ports for connection to the fabric– Nodes connect to F_Ports on the fabric via bi-directional point-to-

point links Fabrics can be a single switch or a general collection of switching

elements Frames may be buffered within the fabric, making it possible for

nodes to connect to the fabric at different data rates The fabric is a switched architecture, not a shared access medium, so

no MAC issues are encountered and no MAC sublayer is necessary– The FC network scales easily in terms of ports, data rate, and

distance covered and through its layered protocol architecture interworks with existing LAN and I/O protocols


Channel 8


Basic Fibre Channel Architectural Diagram


Channel 9

Fibre ChannelExample Architecture

RAID array RAID array RAID arrayRAID array

Com3

Ethernet Switch

GeneralLAN

iSCSI EnabledServer

APP Server #3w/ FC HBA



Fibre ChannelSwitch w/ iSCSI bridge


Channel 10

Fibre ChannelProtocol Specifications

Fibre Channel Protocol Architecture– The Fibre Channel standard reference model is organized into

five levels [Figure 9.3 and Table 9.3] These are not ‘levels’ in the strict sense of the OSI model but are

instead functional groupings of services and/or definitions The standard does not dictate actual implementations,

relationships between the levels, or the specific interfaces between levels

– Levels FC-0, FC-1, and FC-2 are defined together in a standard called the Fibre Channel Physical and Signaling Interface (FC-PH)

– No final standard has been issued for FC-3– A number of standards have been developed at FC-4 specifying

how Fibre Channel interfaces to existing LAN and I/O technologies


Channel 11


Fibre Channel Protocol Architecture (continued)


Channel 12


Fibre Channel Protocol Architecture (continued)– Details on the FC-0 level

A variety of physical media and data rates are allowed:– Data rates: 100-Mbps to 3.2-Gbps– Media: fiber optic, coaxial cable, and STP– Distance: 50 m to 10 km depending on data rate and

media– The FC-1 level uses a 8B/10B encoding scheme in which 8

bits of data from the FC-2 level are encoded into a 10 bit binary symbol


Channel 13


Fibre Channel Protocol Architecture (continued)– The FC-2 level is responsible for the transmission of data

between N_Ports, which requires the following: Addressing of N_Ports Permissible topologies of the fabric Classes of service Segmentation and reassembly of frames as well as higher level

grouping of frames (sequences and exchanges) Sequencing, flow control, and error control

– The FC-3 level provides a common set of services across multiple N_Ports

Striping: the process of using multiple ports to transmit a single data unit in parallel

Hunt groups: allows a connection to be established to any available N_Port in the group

Multicast (and broadcast)


Channel 14


Fibre Channel Protocol Architecture (continued)– The FC-4 level defines how other protocols interoperate with

Fibre Channel (specifically FC-PH) SCSI – a common device interface standard for computer

peripherals HIPPI – a high speed I/O channel used in mainframe and

supercomputing environments IEEE 802 – how IEEE 802 MAC frames map to Fibre Channel

frames ATM IP – how to map packets into Fibre Channel frames (RFC 2625)


Channel 15

Fibre ChannelPhysical Media and Topologies

A major strength of Fibre Channel is that it provides a range of options for the physical medium, the data rate on that medium, and the topology of the network

Transmission Media– A special shorthand nomenclature has been developed for

FC media – it basically consists of the following: Speed-Medium-Transmitter-Distance FC-0 options are listed in Figure 9.4

– Allowable Media Types Fiber Optic: both SM and both 50m and 62.5m MM Coaxial Cable: three types of 75 ohm cable are specified, a thick

RG-6/U, a thinner RG-59/U, and a miniature coax cable 0.1 inches in diameter

Shielded Twisted Pair: two types of 150 ohm cables are specified for use over short distances at data rates up to 200-Mbps: EIA-568 Type 1 STP: (two shielded twisted pair) or EIA-568 Type 2 STP (four pair STP)


Channel 16


Topologies– The most general FC topology is the fabric (switched) topology– Four basic topologies [Figure 9.5] are available in Fibre Channel:

point-to-point, fabric, arbitrated loop (no hub), and arbitrated loop with hub

Point-to-point connects two end nodes with no switches or routing

The fabric topology can contain an arbitrary number of switches, some connecting to nodes and others that just provide transport between other switches

– The fabric topology allows for easy scalability– In the fabric topology the overhead on nodes is minimized; they

are only responsible for managing the point-to-point link to their local switch

– Each port requires a unique address to allow frames to be delivered to the proper destination


Channel 17


Topologies (continued)– The arbitrated loop topology allows up to 126 nodes to be

connected in a simple, low-cost loop The ports on the loop are a special kind called NL_Ports

because they must perform special functions associated with loop management

Operation is roughly equivalent to other token ring protocols

There is a token acquisition protocol controlling loop access

– The fabric & loop topologies can be connected as long as one node can act as both an arbitrated loop & a fabric node that participates in routing decisions on the fabric

– The topology of a given FC network is discovered automatically as part of network initialization


Channel 18


Fibre Channel Topologies (continued)


Channel 19

Fibre ChannelFraming & Classes of Service

Framing Protocol– The FC-2 layer defines the rules for the transfer of frames

between nodes, comparable to the OSI data link layer– FC-2 specifies the types of frames, procedures for the

exchange of frames, frame formats, flow control, and classes of service

– Classes of Service FC-2 defines multiple classes of service; these classes are

determined by the way communication is established between two ports and their flow control and error control capabilities

Five classes of service are currently defined:– Class 1: Acknowledged Connection-oriented service– Class 2: Acknowledged Connectionless service– Class 3: Unacknowledged Connectionless service– Class 4: Fractional Bandwidth Connection-oriented service– Class 6: Unidirectional Connection service


Channel 20


FC-2 Classes of Service– Class 1 Service (Acknowledged Connection-oriented service)

Provides a dedicated path through the fabric which behaves to the end nodes like a point-to-point link

Also provides a guaranteed data rate with sequenced delivery of frames

The end node requests the setup of a Class 1 service connection using a special start-of-frame delimiter (SOFc1)

Class 1 service is advantageous for long constant bandwidth transfers of data (e.g. - streaming backups over a network)


Channel 21


FC-2 Classes of Service (continued)– Class 2 Service (Acknowledged Connectionless service)

Provides an acknowledged data transmission service without the overhead of setting up a connection through the fabric

Acknowledgements frames are returned by the receiving port, if a delivery cannot be made due to congestion a busy frame is returned

This is not the case with frames that cannot be delivered due to frame errors

Sequenced delivery is not guaranteed; frames can take different paths through the fabric if possible

Multiplexing of frames from different sources and/or destinations is allowed

Class 2 service is good for Storage Area Networks (SANs)


Channel 22


FC-2 Classes of Service (continued)– Class 3 Service (Unacknowledged Connectionless service)

Provides a basic datagram service with no connection setup No guaranteed nor acknowledged delivery Good for short bursts of data or delivery of multicast/broadcast data

– Class 4 Service (Fractional Bandwidth Connection-oriented service)

Provides a service similar to Class 1 but also provides Quality of Service (QoS) guarantees and reservations

Allows the specification of guaranteed bandwidth & bounded latency QoS parameters established separately for each direction Good for time-critical & real-time applications like videoconferencing

– Class 6 Service (Unidirectional Connection service)

Provides the reliable unicast delivery found in Class 1 but also supports reliable multicast and preemption

Good for video streaming and broadcasting


Channel 23

Fibre ChannelFrames, sequences, and exchanges

There is much more to the FC-2 layer than frames & classes of service; it defines a set of functional building blocks for higher layer services

– Also defines a number of protocols used to implement services at a port

– Typical protocols are creating or terminating a connection, transferring data, etc.

– Protocols consist of an exchange of information between N_Ports, which in turn consists of sequences, and sequences a composed of a related set of frames


Channel 24

Fibre ChannelFrames, sequences, and exchanges

Class of Service

N_Port Service

Fabric LoginProtocol

Exchange

Data Transfe rProtocol

Exchange

N_Port LogoutProtocol

Exchange

Sequence Sequence SequenceSequence

Fram e Fram e Fram e Fram e Fram e Fram e


Channel 25

Fibre ChannelFrames, sequences, and exchanges (continued)

There are two general types of frames: data and control– The three types of data frames are used to transfer higher

level information between N_Ports FC-4 Device Data: used to transfer higher-layer data units from

protocols specified in FC-4 standards (IP, SCSI, etc.) FC-4 Video Data: used to transmit streamed video between buffers

without an intermediate storage Link Data: used to support higher level control information between

N_Ports

– There are currently three types of link control frames defined:

Link Continue: functions as an acknowledgement in Fibre Channel sliding-window based data transfer

Link Response: used as a negative acknowledgement in FC sliding-window based data transfer

Link Command: A reset command used to reinitialize the sliding-window based transfer mechanism


Channel 26


Sequences– With Fibre Channel a maximum frame size is imposed at the

FC-2 layer but is transparent to higher layers– Higher layers set down chunks of data to FC-2, which may

need to break them up into a sequence of frames– The sequence of data frames needed to carry a single

higher-layer chunk of data may also be accompanied by one or more link control frames for acknowledgement

– FC-2 provides the segmentation and reassembly that supports the transmission of sequences as well as error control

– Errors in a frame that belongs to a sequence causes the retransmission of that whole sequence (and any others transmitted after it – go back N ARQ)


Channel 27


Exchanges– Exchanges are mechanisms for organizing multiple

sequences into a higher-level construct to allow easier interfacing to applications

– Examples of exchanges are SCSI disk operations like a read or write

– Can involve either a unidirectional or bi-directional transfer of sequences

– Within a given exchange, only a single sequence can be active (though sequences from different exchanges can be simultaneously active)


Channel 28


Protocols– An exchange is tied to a protocol that provides a specific

service for higher levels– Some common protocols that may be used by any higher

application: Fabric Login: executed upon initialization of an N_Port,

requires the exchange of the N_Port address, classes of service supported, and flow-control parameters

N_Port Login: the exchange of service parameters between a pair of N_Ports before data exchange (buffer space, service classes supported, etc.)

N_Port Logout: the termination of a connection between a pair of N_Ports


Channel 29


Flow Control– Fibre Channel provides a sophisticated set of flow control

mechanisms at two ‘levels’: end-to-end and buffer-to-buffer– The key to the FC flow control mechanisms is the concept of

credit: credit is negotiated at login and denotes the number of unacknowledged frames allowed at any time

– End-to-End Flow Control This type of flow control paces the flow of frames between

N_Ports Requires acknowledgements to operate, so end-to-end flow

control can be used only with Class 1 and Class 2 services

Two levels of credit are in use with Class 1 & 2 services -- end-to-end and node-to-switch

Two levels of credit are in use with Class 1 & 2 services -- end-to-end and node-to-switch

Class 4 may have the same flow control as Classes 1 and 2; can’t find a good answer because most current equipment only supports class 2 & 3

Class 4 may have the same flow control as Classes 1 and 2; can’t find a good answer because most current equipment only supports class 2 & 3


Channel 30

Fibre ChannelFlow Control (continued)

End-to-End Flow Control– Three types of acknowledgements are possible in a Class 1 or

Class 2 service ACK_1: acknowledges one data frame & decrements the

credit count by 1 ACK_N: acknowledges N data frames & decrements the credit

count by N ACK_0: acknowledges a whole sequence, decrementing the

credit count by the number of frames in the sequence


Channel 31


End-to-End Flow Control (continued)– Acknowledgement types cannot be mixed; if ACK_1 is initially

used for a Class 1 connection than it must be used for the duration of the connection

– Busy and Reject control frames are used for flow control The F_BSY frame indicates the fabric is busy and cannot

deliver a frame The P_BSY frame indicates the destination port is busy and

cannot accept a frame; the sender will try a predefined number of times to retransmit the frame

With the Reject (F_RJT and P_RJT) frames, delivery of the data frame is being denied (for some reason other than congestion)

When a frame belonging to a sequence is rejected the whole sequence must be retransmitted


Channel 32


Buffer-to-buffer Flow Control This is flow control across a pair of ports connected by a

point-to-point link, assuring that buffers are available in the ports at either end of the link

This mechanism is also applicable to all classes of service (including Class 3 datagram service)

A single type of control signal, the R_RDY frame, is used for buffer-to-buffer flow control

– As a data frame is transmitted across the link, the sender increments its credit count for the link

– At the receiving port the data frame is buffered as received– As soon as the data frame is switched to another port’s

buffer on the switch, the receiving port sends back the R_RDY frame to the sending port

– When the sending port receives the R_RDY frame it decrements the credit count, opening its send window by one frame


Channel 33


Frame Format [Figure 9.10]– The Fibre Channel Frame contains five general fields:

Start Delimiter Frame Header Data Cyclic Redundancy Check (CRC) End Delimiter

Sta rtDe lim ete r(4 bytes)

Fram e Header(24 bytes)

Da ta Fie ld(Variable : 0-2112 bytes)

EndDe lim ete r(4 bytes)

CRC(4 bytes)

Optiona l Headers


Channel 34


Frame Format - Start of Frame Delimiter– The start of Frame Delimiter includes a four byte set of non-

data symbols denoting the start of a frame and allowing synchronization

– The SOF delimiter comes in several varieties, each of which will specify the frame’s type and class of service

– Examples are SOF Class 1 connection (SOFc1), SOF normal (for data frames), and SOF fabric (for control frames in the fabric)


Channel 35


Frame Format - FC- 2 Frame Header– Contains the control data required at this level; consists of the

following fields: Routing control: contains two subfields, one that denotes the

type of frame (device data, link control, etc.) and the type of data within the frame

Destination Identifier: destination N_Port or F_Port– FC uses two levels of addressing: a globally unique identifier

(world wide port/node names) & a lower level port identifier• World wide/port name is used by higher layers and for

network management• Port identifier is the 3-byte that is used for frame routing

that consists of three parts: domain, area, and port– The hierarchical addressing structure facilitates routing and

management of the fabric– A mechanism for mapping between the two addresses is

necessary


Channel 36


Frame Format - FC- 2 Frame Header– Contains the control data required at this level; consists of the

following fields: Source Identifier: source N_Port or F_Port Type: if the routing control field specifies an FC-4 frame, then

this field specifies the payload protocol (SCSI, IP, etc.)– This field and the Route control field allow the destination

N_Port to deliver the data to the correct higher layer ‘user’ Frame control: contains control information relating to frame

content– Is frame a retransmission? Is frame part of a sequence?

Sequence ID: unique identifier for a sequence used for all frames belonging to it

Data Field control: specifies which, if any, of four optional headers are present


Channel 37


Frame Format - Frame Header (continued)– Contains the control data required at this level; consists of the

following fields: Sequence count: A unique number assigned sequentially to

each frame in a sequence (for flow control and proper reassembly of frames within a sequence)

Originator Exchange Identifier: a unique identifier assigned to the higher layer initiator of an exchange

Responder Exchange Identifier: a unique identifier assigned to the higher layer destination of an exchange

Parameter: used in different ways for link control and data frames

– Link control frames carry information specific to the control function in this field

– Data frames may carry an address meaningful to the upper layer protocol


Channel 38


Frame Format - Data Field– Contains user data in a multiple of four bytes chunks up to a

maximum of 2112 bytes– Can also include one or more optional headers whose

presence is denoted in the Data Field control field: Expiration Security optional header: can carry an expiration date

for the frame and well as other security data over and above the FC-PH standard

Optional Network Header: may be used by a bridge or gateway node interfacing to an external network to allow tunneling (includes 8 bit source and destination network addresses)

Optional Association Header: may help specify an upper layer process (or group of processes) associated with an exchange

Optional Device Header: if used the format is specified by the upper layer protocol used with the frame


Channel 39


Frame Format - CRC & End Delimeter– CRC field: the error detection algorithm is the same 32 bit

CRC used with FDDI and IEEE 802– End of Frame Delimiter

A four byte field denoting the end of the frame The EOF field may be modified by a switch in the fabric if it

finds an error in the frame or some other condition that invalidates the frame

There are three different EOF delimiters for valid frames: – EOFt denotes the end of a valid sequence– EOFdt is used with Class 1 service to indicate that the frame

is the last frame on the logical connection (i.e. – the connection is being terminated)

– EOFn is used to denote successful transmission of frames not covered by the first two


Channel 40

Fibre ChannelExamples of Equipment

Fibre Channel Equipment Manufacturers– High-end (“Director-Class”) Switches

Brocade Silkworm 2400 (http://www.brocade.com/products/directors/silkworm_24000/index.jsp)

McData Intrepid 6140 (http://www.mcdata.com/products/hardware/director/6140.html)

– Low-end (“Edge”) Switches EMC DS-16B3 (

http://www.emc.com/pdf/products/connectrix/connectrix_DS_16B2.pdf) Cisco MDS 9120 (

http://www.cisco.com/en/US/products/ps5993/index.html)

– Host-Bus Adapters (HBA) HP Storageworks FCA-2408 2Gbps PCI-X (

http://h18006.www1.hp.com/products/storageworks/fca2408/index.html) Qlogic QLA2200L 1Gbps PCI (

http://www.qlogic.com/support/product_resources.asp?id=118)


Channel 41

IEEE 802.3 Family of LAN ProtocolsHomework & Reading

Homework #4 - Due in four weeks (3/22)– The idea of using Ethernet as a service provider technology is

very attractive, but it lacks much of the functionality needed in that environment. Research a technology called Resilient Packet Ring (RPR) and write 1-1.5 pages on what its goals are and what functionality it provides.

– Fibre Channel continues to evolve as a networking technology: research and write 1-1.5 pages on two different enhancements are currently being developed (e.g. – higher speeds, new higher layer mappings, etc.)

– Redo OPNet Lab #1 using a 16-Mbps Token Ring instead of ethernet; answer all questions except #4.

Reading– This week’s material: Stallings chapters 8 and 9 – Next week: SONET, ATM, & ATM LANs (chapter 11)

1 Fibre Channel – Chapter 9. 635.412.71 Spring 05 Class #5: Token Ring LANs & Fibre Channel 2 Fibre Channel Introduction n Originally developed for mainframe.

Documents

fibre channel association

fibre channel chapter

routing data

io technologiesclass

dedicated class

different heritage

transport mechanism

fc capacity