Storage Cheap or Fast, Pick One. Storage Great--you can do a lot of computation. But this often generates a lot of data. Where are you going to put it?

Storage

Cheap or Fast, Pick One

Storage

Great--you can do a lot of computation. But this often generates a lot of data. Where are you going to put it?

In the demo clusters we run, the front end just has a little more disk than the compute nodes. This is used for the NFS-mounted home directories. But single drives do not have very good performance and are vulnerable to single-point failure

RAID

The next step up is RAID (Redundant Array of Inexpensive|Independent Disks)

This uses a pool of disks to save data. Rather than spend billions building special high-capacity disks, greater capacity is achieved by simply putting PC disks into RAIDs.

Typically you need a RAID controller on the host

There are several types/levels of RAID

RAID 0

RAID 0 writes blocks to multiple disks without redundancyBecause the data is being written to multiple disks the

controller can work in parallel on both read and write, improving performance

If any error occurs data can be lostDon’t use on mission critical data; only for performanceIdeally you have one drive per controller

RAID 1

This is mirroring. The same data is written to two disks. If either disk fails a complete copy of the data is available at the other disk

Uses 2X the storage space, can get better performance because the OS can pick the disk with the least seek or rotational latency

RAID-5

RAID5 uses “parity” or redundant information. If a block fails, enough parity information is available to recover the data

The parity information is spread across all the disks

High read rate, medium write rateA disk failure requires a rebuild as the parity

information is used to re-create the data lost

RAID-10

RAID-10 is striping plus mirroring, so you get good performance plus a fully mirrored data, at the expense of 2X disk

StorageRAID-5 is a reasonable choice most of the time.There are many commodity vendors of RAID arraysSCSI RAID arrays are expensive, the disks are

expensive, and the disks have low capacity, but the RAID arrays have good performance

ATA raid arrays have excellent price (1/3-1/2 that of SCSI drives) and capacity, somewhat lower performance

Apple ATA RAID: 7 TB, $11.5KPromise Vtrak 15110: $4K plus 15 400GB SATA

disks at $300 = 6 TB for $8,500

Storage in Clusters

OK, so new you’ve got a RAID array. Now what?A pretty typical approach is to attach the RAID

array to a host and then have that host serve NFS.

Sometimes called “Network Attached Storage” or NAS

Rather than being attached to the front end this may be a node of its own in the clusterC1

FEC2

S1RAID

NAS

You can easily set up a Windows or open source NAS box and put it on the network. In the linux world this is just a linux box running Samba for Windows directory shares

You can also buy pre-configured “appliances” for disk space. These are often boxes that run a stripped-down Linux variant and have a bunch of disks stuffed into them, along with a web-based interface.

NAS Appliances

Sun StorEdge 5210Dell PowerVault 745NSnap Server 4200 Linksys, Iomega NAS (SOHO, combined with wireless & print server)NetApp (enterprise class)Emerging class: a NAS front-end/gateway to a

SAN

Storage Area Networks

SANs are high performance, and you pay for it. A SAN has storage devices on a special network

that carries only I/O traffic. The storage devices are connected to a set of servers. All the servers share all the storage devices on the SAN.

In effect each server is attached to two networks: one for communication with other hosts, the other reserved for communicating with storage devices

Storage Area Network

SANs

SANs let you add storage to a pool that can be shared by all servers connected to the SAN

This can be a hard problem to solve due to parallelism. The original SANs sometimes had a single server attached to a single disk device to prevent simultaneous access

More recent SANs use a distributed filesystem to avoid concurrency problems

Single Machine Filesystems

AttachedDisk

Server

This is a fairly simple problem to solve because thereis only one entity making requests of the disk-- “give me 200 blocks”. The server can keep track of blockallocations and file names and make that informationpermanent by writing it to disk

DFS

SANDisk

Server 1

Server 2

FC Switch

S1 asks for 500 disk blocks to hold a file named “Foo”.At the same time S2 asks for 200 blocks to hold a file named “Foo”Which wins? How can we ensure that none of the blocks for S1are also given to S2?Effectively we need to serialize the requests to the SAN disk, and we need something to keep track of file names and block allocations that can be queried by all the servers

MetaData

A solution is to use metadata. This is “data about data”. In effect it keeps track of things like filenames and block allocations, just like a single machine filesystem does

The metadata has to be available to all the servers who want to participate in the SAN, so it is typically written on the SAN itself

Meta

SANDisk

SANDisk

FCSwitch

Server 1

Server 2

Metadata Comms

As an added twist, most implementations send the metadata over a separate physical network from the fibrechannel--usually gigabit ethernet. So every device in the SAN is usually connected by two networks

SanDisk

Server 1 Server 2 SanDisk

MetaData

FC Switch

Gig E Switch

SAN Disks

As you can see, the SAN “disks” are getting pretty complex. In reality the “disks” are usually RAID enclosures

The RAID enclosure has a FC port, an ethernet port, and a RAID controller for its disks

The SAN is really building on top of RAID building blocks

SAN

This arrangement has many advantages:

• Servers and storage devices can be spread farther apart

• All the storage devices go into a single, shared pool

• Very high performance; can do serverless backups

• Storage device availability not tied to the uptime of a host

• Can tie applications to specific storage types (ATA RAID for high volume, SCSI RAID for high traffic transactional databases)

OTOH, SANs probably won’t go commodity any time soon, since they have limited applicability to the desktop. Which means that they will remain expensive and complex

iSCSI

The most common SAN network is fibre channel (FC). The protocol used is often SCSI.

You can send SCSI over other network protocols; an emerging option is iSCSI, which puts SCSI in IP over gigabit ethernet. This is slower but exploits the existing IP infrastructure and enables WAN SANs.

Why recreate another, separate network for data traffic using a different hardware standard that requires new training? Why not just use well-understood IP networks that your people are already trained on?

The drawback is that iSCSI has somewhat higher latency, which is an issue for disk access

iSCSI

iSCSI simply sends SCSI commands encapsulated inside TCP/IP, just as FC SANs send SCSI commands encapsulated inside FC frames

This can enable wide-area SANs (if you have the bandwidth and are willing to live with the latency) since IP can be routed; a SAN distributed across an entire state or country is possible

Putting the SCSI commands inside TCP/IP can add a couple layers to the software stack, which unavoidably increases latency; Gbit Ethernet also has less bandwidth than FC

Implementations seem a bit immature for the enterprise; might ony make a splash with 10 gbit ethernet

Another option is ATA over Ethernet (AOE)

Fibre Channel Costs

Uses copper or optical fibre at 2+ Gbits/secDell/EMC AX100, 3 TB SATA disks, 1 8 port fibre

channel switch, 1 FC card = $16KFC cards approx. $500 each, 8 port FC switch

aprox. $2,500Filers with hundreds of TB are available if you’ve

got the checkbookOften SANs use SCSI disk arrays to maximize

performanceOptical FC has a range up to a few KM, so this

can be spread across a campus

SAN and Clusters

FE

C4

C1

C2

C3

FCStorage

FCStorage

FCSwitch

Very high performance--the compute nodes participate in theSAN and can share in the benefits of the high speed storagenetwork

InterconnectNetwork

Fiber Channel Switching Fabric

FCJukebox

SAN and Clusters

If you an afford this you can get very high performance; you might have four network interfaces (Gigabit ethernet, Infiniband, Gigabit ethernet for metadata, and FC) on each node

May work well in situations with heavy database access, very heavy image processing

Biological research, etc.

SAN Example

Apple sells XSAN, their OEM’d storage area network software; we are setting this up here

FCSwitch

Beatnik

Bongo

Raid 1

Raid2EthernetSwitch

XSAN

Each RAID enclosure of 14 disks may be divided up into multiple Logical Unit Numbers (LUNs). One LUN corresponds to one set of RAID disks--a single RAID box may have more than one collection of RAID sets, which may be of different types.

One of the RAID boxes has three LUNs: One set of seven disks in a RAID-5 configuration, one set of five disks in a RAID-5 configuration, and one set of two disks in a RAID-1 configuration

XSAN

XSAN builds “Volumes” (a virtual disk) out of “storage pools”. A storage pool consists of one or more LUNs.

XSAN

A storage pool is simply a collection of LUNs, and a volume consists of one or more storage pools

XSAN

The LUNs are themselves used as RAID elements; XSAN treats each of them as disks and does a RAID-0 across them. This means that LUNs in the same storage pool should be of the same size

FC Switch Config

The FC switch requires some configuration; this is done via a web-based interface. It’s not quite as easy as plug-and-go. “Initiators” are computers, and “targets” are RAID enclosures

Non-Apple

Non-Apple computers can participate in the SAN; there are Linux implementations of the XSAN software available from third parties

Each computer participating in the SAN must have the XSAN software; edu price is about $500 per copy (more for Linux)

Summary

Buy some disk