Using buffer mechanisms in FIBRE CHANNEL technology

PAVEL BANKOV

Implementing buffer mechanisms in FIBRE CHANNEL technology

• Define what is Fibre Channel• Fibre Channel Layers

• Fibre Channel Class of Service (C.O.S.)

• Fibre Channel Flow Control

• Fibre Channel Error Detection and Recovery

• Fibre Channel Protocol Constructs• Conclusion

Objectives

What is FIBRE CHANNEL• A high-speed transmission technology used as a peripheral channel or

network backbone.

• Up to 16 Gb/s, full-duplex, serial, data communication technology; variable-length data frames (maximum 2 KB), 32-bit CRC on frames

• It supports several common transport protocols like Internet Protocol (IP) and SCSI.

• It operates over copper (coaxial/twisted pair) and fiber optic cables (MM/SM) at distances of up to 10 Kilometers

• Combines best attributes of a channel and a network

• High bandwidth, High data integrity, Highly reliable, Destination paced (Buffer credits), Scalable, High availability, Shared media, Circuit/packet, Multiple protocol support, Transport flexibility, 8b/10b

What is FIBRE CHANNEL

What is FIBRE CHANNEL• offers point-to-point, switched, and loop interfaces – FC-P2P, FC-AL,

FC-SW

• Standards for Fibre Channel are specified by the Fibre Channel Physical and Signalling standard, and the ANSI X3.230-1994, which is also ISO 14165-1

• Fibre Channel Protocol (FCP) is a transport protocol (similar to TCP used in IP networks) that predominantly transports SCSI commands over Fibre Channel networks

• Criticism - lack of compatibility, primarily because manufacturers sometimes interpret specifications differently and vary their implementations

• Port ASIC (FC HBAs, switches, or storage arrays)

What is FIBRE CHANNEL

• Fibre Channel does not follow the OSI model layering. It is a layered architecture with 5 layers: FC-0, FC-1, FC-2, FC-3, and FC-4. FC-PH is made up of FC-0, FC-1, and FC-2

Fibre Channel Layers

• FC-0 defines the physical interface characteristics (signaling rates, cables, connectors, distance capabilities, etc.)

• FC-1 defines the transmission protocol, including serial encoding and decoding rules, special characters, and error control

• FC-2, the signaling protocol, is the transport mechanism of Fibre Channel. It defines the framing rules of the data to be transferred between ports, the different mechanisms for controlling service classes, and the means of managing the sequence of a data transfer - frames, sequences, exchanges, login session. Fibre Channel zoning/fabric, RSCN


• FC-3 defines the common services provided by FC-PH to the ULPs. These services, which include striping, hunt groups, and multicast, are not yet implemented. Encryption/RAID

• FC-4 is a protocol-mapping layer, in which application protocols, such as SCSI, IP or FICON, are encapsulated into a PDU for delivery to FC2. LUN Masking


• Fibre Channel defines different delivery options for frame transmission and these are known as a particular Class of Service

• Class 1, 2, and 3 are the different services supported by Fibre Channel and are basically defined with regards to connection, in-order delivery, confirmation of delivery, and which type of flow control is used

• These classes of service are independent of the topology used

Fibre Channel C.O.S.

• Class 1 is a connection-oriented service that provides a dedicated connection between two ports allowing them to use the maximum available bandwidth. Class 1 provides confirmation of delivery and notification of non-delivery

• Class 2 is a connectionless class of service that provides confirmation of delivery and non-delivery of frames. No connection is established between the source and destination ports and it is up to the fabric to deliver the frames, possibly over different paths to the recipient. Frames will be transmitted in order but depending on the fabric may arrive at the recipient out of order


• Class 3 is again a connectionless class of service but with no confirmation of delivery or non-delivery of frames. No connection is established and, like Class 2 frames, Class 3 frames may arrive out of order. The biggest difference is that the delivery of frames is unacknowledged in that the destination port does not send any link control frame (ACK) on receipt of valid data frames. The flow control used in Class 3 is Buffer-to-Buffer

• The fact that there is no confirmation of delivery of frames in Class 3 Fibre Channel means error recovery must be handled at the Upper Layer Protocol. The confirmation is provided by the protocol being transported (i.e. SCSI)


• Class of Service summary


• When a port wants to send frames to another port the frames are sent from a buffer at the sending port and received into a buffer at the receiving end

• If there is no receive buffer space then that port cannot accept any more frames and this leads to busy responses and the possibility of frame loss

• Fibre Channel uses two different flow control mechanisms to pace the rate at which the sender is allowed to transmit frames

• End-to-End and Buffer-to-Buffer - both of these methods of flow control use a credit-based agreement between nodes to regulate the flow of frames between them. The credit value is the number of frames a receiving port has allowed a sending port to transmit to it

Fibre Channel Flow Control

• End-to-End credit (EE_Credit) is the maximum number of data frames a source port can send to a destination port without receiving an acknowledgement frame (ACK). This credit is granted during N_Port login and is replenished with the return of an ACK response frame


• BB_Credit (buffer-to-buffer credit) mechanism for hardware-based flow control. This means that a port has the ability to pace the frame flow into its processing buffers. This mechanism eliminates the need of switching hardware to discard frames due to high congestion

• It relies on the use of credits that are exchanged at login time (FLOGI for Nx_Ports, ELP for E_Ports). In the case of an Initiator or Target (Nx_Port), the port will grant some number of credits (typically between 8 and 64) to the switch. It does this by specifying the number of credits to be granted to the switch in the FLOGI frame


• Login and credit initialization process


• Example - 400 MB/s link with maximum of 200 BB_Credit


• In Fibre Channel, error detection occurs at a number of different levels

• At the FC-0 level errors associated with signal quality

• At the FC-1 level where it checks for invalid transmission characters

• At the FC-2 level frames are checked for various conditions such as CRC error detected, RECEIVE BUFFERS state

• FC-2 Time-outs – 4 different timers used

Fibre Channel Error Detection and Recovery

• R_T_TOV (Receiver-transmitter time-out) - used to time events at the link level (Loss of synchronization), generally controlled in hardware for all link configurations, default value in FC Standard is 100ms

• The E_D_TOV (Error Detect time-out) - time-out value is the timer for transmission of consecutive data frames and responses at the sequence level. Basically this is a short value and indicates how long a sequence can take to complete. It is a time-out value for communicating between two N_Ports that is negotiated at login time. Typical value for this is 2 seconds


• R_A_TOV (Resource Allocation time-out) - time-out value for how long to hold resources associated with a failed operation (needed to free shared resources for reuse). Value to determine how long a port needs to keep responding to a link service request before an error is detected (typical 10 seconds)

• CR_TOV (Connection Request Time-out) - determines how long the fabric can hold a class-1 request in the queue during connection establishment


• The next values are provided in microwatts (mW) or dBm and can be extremely useful when debugging why a link will not come up (dBM = 10 * LOG mW)


• Brocade FOS platforms: using the command sfpshow [portnumber]


• The fundamental protocol structures in the Fibre Channel are called constructs

Fibre Channel Protocol Constructs

• Fibre Channel Frame Structure

• 0–2048 bytes Typical MTU

Fibre Channel Protocol Constructs

• From the materials provided, it seems that there is no correlation between FC buffers (flow control mechanism) and LUN size/number ratio

Conclusion

Thank you!

Using buffer mechanisms in FIBRE CHANNEL technology

Documents