Storage area network and System area network (SAN) –What are they? –Network requirements –Hardware/software issues –References: Ulf Troppens, Rainer Erkens,

• Storage area network and System area network (SAN)– What are they?

– Network requirements

– Hardware/software issues

– References: • Ulf Troppens, Rainer Erkens, a nd Wolfgang Muller, “Storage

Networks Explained - basic and application of Fibre Channel SAN, NAS, iSCSI and Infiniband”, John Wiley & Sons, 2004.

• W. J. Dally and B. Towles, “Principles and Practices of Interconnection Networks”, Morgan Kaufmann, 2004.

• Ajay V. Bhatt, “Creating a Third Generation I/O Interconnect,” available at http://www.express-lane.org

• Storage area network (SAN):– Server-centric IT architecture: storage devices

exist only with servers

• Storage-centric IT architecture: SCSI cables are replaced by a network (storage is now independent of servers).

• Storage area network (SAN) requirement:– Serial transmission for high speed and long

distance– Low transmission errors– Low delay of transmitted data

• Needs to make it feel like using a local disk

• Low delay is a relative term:– The disk subsystem has around 1ms – 10ms latency itself.

– The communication protocol should not use CPU.

• Current Storage area network (SAN) technology (IBM):– Fibre Channel– TCP/IP + Gigabit Ethernet (iSCSI)– InfiniBand

• System area network: a network with a high bandwidth and a low lantency that serves as a connection between computers in a distributed computer system.

• Why system area network:– Historically, the system area network comes with a

particular parallel machine (supercomputer, e.g. Cray T3D, Cray T3E, SGI origin 2000, IBM SP, Thinking machine CM5, Intel Polygon)

• The network is very expensive due to low volume• CPU is two generations behind

– A more cost effective way to build these system is to decouple the processor technology from the networking technology.

– To form cheaper clusters of workstations with the off-the-shelf system area network technology (compared to traditional supercomputers).

– current system area networks:• Myrinet, Quadrics, Infiniband

• System area network requirement:– Low latency and high bandwidth at the

application level.• Not just at the hardware level

• Not just at the system level

– Implicitation:• Hardware, network interface, software messaging

layer should work together to achieve the goal.

– Infiniband is designed as both storage area network and system area network.

• Hardware issues:– High speed links:

• Infiniband: 2.5Gbps = 250MBps, 10Gbps=1GBps, 30 Gbps = 1GBps

• Fibre channel: 100MBps, 200MBps, 400MBps, 1GBps.

• Myrinet: up to 9.6Gbps

• As a reference PCI bus: 100MBps– NIC may need to attach to the memory bridge

• A typical PC:

• A workstation connected to a system area network:

• When the number of end points is large, multiple switches will be needed.

• Topology

• Switching

• Routing

• Topology– Static arrangement of channels and nodes in an

interconnection network– Trade-off between cost and performance

• Cost: the number and complexity of chips, density and length of the interconnections, etc.

• Performance:– Bandwidth and latency: also depend on other factors other

than topology

– Topology performance metrics: Bisection bandwidth, diameter, nodal degree, channel load

• A cut of a network is the set of channels that partitions the set of all nodes into two disjoint sets.

• A bisection of a network is a cut that partitions the network nodes in roughly half.

• The bisection bandwidth of a network is the minimum bandwidth over all bisections of the network.

• The diameter of a network is the largest minimal hop count over all pairs of nodes.

• Under a particular traffic pattern, the channel that carries the largest fraction of traffic determines the maximum channel load of the topology.

• Example topologies:– Regular or irregular– Regular topologies are mostly derived from two

main families: butterflies (k-ary n-flies) or tori (k-ary n-cubes)

• Switching: how a packet pass a switch– Message/packet/flit

• Traditional scheme: store-and-forward– Time = H (S + P/B)

• Cut-through switch:– Forward to the next link after the header flit is

received. Stop only when the next hop buffer is not available.

– Time = H S + P/B, when S << P/B, the time does not depend on the number of hops!!!

• Wormhole routing:– Cut-through switches still allocate buffer to

packets. May require a large amount of buffers– Wormhole routing only allocates buffer for one

flit for each packet.– Latency is the same as cut-through switching.– When the packet is block, the whole flit “train”

is block, occupying links.• Solution: add more virtual channels.

• The deadlock problem in wormhole routing:– Need deadlock free routing scheme to select the

right path

• Cut-through switch and wormhole switch are widely used in system are networks– Routing in such systems is an issue!!– Shortest path routing may result in deadlock.– Deadlock free routing:

• Cut-through switch and wormhole switch are widely used in system are networks– Routing in such systems is an issue!!– Shortest path routing may result in deadlock.– Deadlock free routing:

• Basic idea: fix the priority of channels and using the channels with increasing priority.

• Example: up/down routing

• Up/down routing:– Select a node as the root

– Build a spanning tree from the root

– Nodes are partitioned into layers based on the position in the spanning tree

– The channel from a lower layer node to a higher layer node is the up link, the channel from a higher layer node to a lower layer node is a down link, channels between nodes in the same layer are marked as up or down link based on the node number

– In the valid route: an up channel cannot follow an down channel.

– These exists at least one valid path between each pair of nodes.

• Problems with deadlock free routing:– Load balancing is a problem, traffic are not

evenly distributed– Non-adaptive version of the deadlock free

routing scheme is also a problem• How to map the routes in order to get good

performance (metrics: maximum channel load?)

• More on the problem to be discussed later.

• Hardware/software codesign and software API issues:– What functionality should be implemented in

the hardware.• E.g. adaptive routing may imply out of order

packets

– Chien’04 paper gives good answers to some of these questions.