Implementing SAN in Linux [Read-Only]ilta.personifycloud.com/webfiles/.../1971/ImplementingSANandNASinL… · Implementing SAN & NAS with Linux by Mark Manoukian & Roy Koh. Housekeeping

Implementing SAN & NAS with Linuxby Mark Manoukian & Roy Koh

HousekeepingHousekeeping

• EvaluationsEvaluations– OSS2– Be HonestBe Honest– Comments

• Questions? Raise your hand wait for the • Questions? Raise your hand, wait for the Microphone.Ch k h l f h• Check the on-line version of this presentation for updates.

• Nerd• Nerd• GeekGeek• 2nd time ILTA attendee• 1st time speaker

“B b i di“Babysitters direct to you in y

20 minutes!”20 minutes!

STORAGE!STORAGE!

• 1956 - IBM Model 350 the first-ever hard disk 1956 - IBM Model 350, the first-ever hard disk drive.

• Ab t 4 4 M B t !• About 4.4 MegaBytes!

The Good News?The Good News?

• Storage is getting cheaper!Storage is getting cheaper!

Year Cost per MegaByte1956 $10,000.001980 $192.311984 $87.861989 $12.001994 $0 951994 $0.951999 $0.022004 $0 001152004 $0.00115

Cheap, Cheap, Cheap!Cheap, Cheap, Cheap!

Year Cost per GigaBytep g y2010 $0.16

Why are you here?Why are you here?• Learn to build your own SAN or NAS (DIY)• SAVE $$$• Know your systems better – hardware, softwareKnow your systems better hardware, software• Supplement your PRIVATE CLOUD

S 2 B kBl b il Sept 2009 BackBlaze built a 67 TB storage unit for $7,867http://bit ly/s0Emghttp://bit.ly/s0Emg

SAN & NAS in LinuxSAN & NAS in Linux

• SAN - AoE: Target & Initiator configAN f• SAN - iSCSI: Target & Initiator config

• NAS – Samba/CIFS• Openfiler• DRBD – Block level replicationDRBD Block level replication

What is SAN?What is SAN?

• Stands for Storage Area NetworkStands for Storage Area Network• Wikipedia: an architecture to attach remote

computer storage devices to servers in such a computer storage devices to servers in such a way that the devices appear as locally attached to the operating system

• Block-level operations• Protocols: Fibre Channel, iSCSI, ATA over Protocols: Fibre Channel, iSCSI, ATA over

Ethernet (AoE)• SAN is not NASSAN is not NAS

What is NAS?What is NAS?

• Stands for Network Attached StorageStands for Network Attached Storage.• Wikipedia: A NAS unit is a computer connected

to a network that onl pro ides file based to a network that only provides file-based data storage services to other devices on the networknetwork.

• Protocols used: NFS, SMB/CIFS, AFP.

What is Block Level?What is Block Level?

• a block is a sequence of bytes or bits having a block is a sequence of bytes or bits, having a nominal length (a block size).

• Wikipedia• Wikipedia:Blocking is used to facilitate the handling of the data stream by the computer program data-stream by the computer program receiving the data. Blocked data are normally read a whole block at a timeread a whole block at a time.

SAN Bl k L l•SAN = Block Level

•NAS = File Level

Questions?Questions?

SAN: AoE on LinuxSAN: AoE on Linux

• network protocol developed by the Brantley network protocol developed by the Brantley Coile Company

• light weight, does not rely on network layers light weight, does not rely on network layers (TCP or IP)

• designed for simple, high-performance access designed for simple, high performance access of SATA storage devices over Ethernet networks.

• like FCoE not like iSCSI• AoE exists on Linux as a kernel moduleAoE exists on Linux as a kernel module

AoE: Why use it?AoE: Why use it?

• Ethernet: is getting fasterEthernet: is getting faster• and cheaper!• and Ethernet can be multiported (switches).• Enterprise ATA (or SATA) disks are also getting

cheaper. • ATA over Ethernet (AoE) — a SAN at a ( )

fraction of the cost of Fibre Channel.

AoE on LinuxAoE on Linux

AoE Target setup on Linux SANAoE Target setup on Linux SAN

• Hardware RAID – make sure you Hardware RAID make sure you have Linux drivers

• Or you can use Software RAIDOr you can use Software RAID• Check AoE is enabled in kernel –

modprobe aoemodprobe aoe• AoE Target setup:

a. Install aoetools (sourceforge)a. Install aoetools (sourceforge)b. export block devices

vblade 0 1 eth0 /dev/sdb

AoE Target setup on Linux SANAoE Target setup on Linux SAN

vblade 0 1 eth0 /dev/sdbvblade 0 1 eth0 /dev/sdbWhat does this mean?

vblade = commandvblade = command1st number 0 = shelf2nd b 1 l t2nd number1 = sloteth0 = ethernet device at address 0/dev/sdb = name of exported block device

AoE Target setup on Linux SANAoE Target setup on Linux SAN

IMPORTANT!IMPORTANT!

fIf you are planning to have many AoE Target, make sure there are

f #uniqueness to your shelf and slot #s!

vblade 0 1 eth0 /dev/sdb

AoE Initiator (Linux client)AoE Initiator (Linux client)

On Linux server: On Linux server:

1. Make sure AoE exists in kernel d l # d bas a module # modprobe aoe

2. # aoe-stat3. # ls –l /dev/etherd4. # mkfs.ext3 /dev/etherd/e0.0/ / /5. # mount /dev/etherd/e0.0 /mnt

AoE Initiator (Windows client)AoE Initiator (Windows client)

On Windows server:On Windows server:

1. WinAoE: http://bit.ly/8UmigK2. RocketDivision StarPort iSCSI

and AoE Initiator: // /9http://bit.ly/9xUDEt

Screenshots of Starport InitiatorScreenshots of Starport Initiator

SAN: iSCSI on LinuxSAN: iSCSI on Linux

• iSCSI = Internet Small Computer Systems iSCSI Internet Small Computer Systems Interface

• IP based storage networking standard for • IP-based storage networking standard for linking data storage facilities

C / 860 3260• Uses TCP/IP Ports 860 and 3260

iSCSI on LinuxiSCSI on Linux

iSCSI Target setup on Linux SANiSCSI Target setup on Linux SAN

• Make sure hardware RAID is supported ppby Linux or use software RAID

• Install and configure iSCSI Enterprise T h //bi l /54GETarget http://bit.ly/54GEne

• Or in Debian# apt-get install iscsitarget# apt get install iscsitarget

• Edit config files:# nano /etc/default/iscsitarget

/ /# nano /etc/ietd.conf# nano /etc/initiators.allow

• Start the service• Start the service…

iSCSI Initiator (Linux client)iSCSI Initiator (Linux client)

On Linux:On Linux:

1. Install open-iscsihttp //www open iscsi org/http://www.open-iscsi.org/or apt-get install open-iscsi in DebianDebian

2. Edit config file# / / / d f# nano /etc/iscsi/iscsid.conf

iSCSI Initiator (Windows client)iSCSI Initiator (Windows client)

On Windows:On Windows:

1. Windows 7 & Windows 2008 is b ilt inbuilt-in

2. Windows XP & Windows 2003:MS SCS fDownload MS iSCSI software

initiator v2.08h //b l /2 AChttp://bit.ly/2F7AC1

3. Or use Starport’s Initiator

SAN Best PracticesSAN Best Practices

• Design and Plan your networkDesign and Plan your network• Network isolation (security)

Logical network isolation VLAN- Logical network isolation – VLAN- Physical network isolation

( )• Replication (Block level replication)

QUESTIONSQUESTIONS

• When setting up your Linux SANWhen setting up your Linux SAN

• [email protected]

NAS on LinuxNAS on Linux

• Centralized “file-based” storageCentralized file based storage• Wikipedia- “ NAS unit is a computer connected to

a network that only provides file-based data a network that only provides file-based data storage services to other devices on the network.”

• “NAS uses file based protocols such as NFS • NAS uses file-based protocols such as NFS (popular on UNIX systems), SMB/CIFS (used with MS Windows systems), or AFP (used with Apple MS Windows systems), or AFP (used with Apple Macintosh computers). NAS units rarely limit clients to a single protocol.”g p

Simply said:Simply said:

• NAS is file sharingNAS is file sharing• NAS can be a file share on a server

NA• NAS can also be a dedicated hardware device• NAS can also be an embedded device

The easy path…The easy path…

How to get things workingHow to get things working.

Linux Distro: Openfiler NSALinux Distro: Openfiler NSA

• Base on rpath - 32/64 bit http://bit ly/ySPPZBase on rpath - 32/64 bit. http://bit.ly/ySPPZ• Web-based GUI

M N N• Multiple NICs (NIC bonding)• Protocol rich:

1. Block level: iSCSI, Fibre Channel2. File level: NFS, SMB/CIFS, Http/DAV,

FTP, rsync• Does not support AoE out of the box BUT…pp

Openfiler NSA (SAN)Openfiler NSA (SAN)

1 iSCSI is built-in-GUI-web-configure-goodness1. iSCSI is built-in-GUI-web-configure-goodness.2. AOE is not built-in.

aoetools needs compiling and it worksaoetools needs compiling and it works.a. before compiling, install gcc

# conary update gcc# conary update gccb. download aoetools from Sourceforge

http //bit ly/cj2lawhttp://bit.ly/cj2law# make && make install

Openfiler NSA (SAN screenshots)Openfiler NSA (SAN screenshots)

Openfiler NSA (NAS)Openfiler NSA (NAS)

• Industry-standard protocol suite – SMB NFS Industry-standard protocol suite SMB, NFS, FTP, WebDAV

• Share management• Share management• Quota / resource allocation – group, user,

guest• Accounts management – PAM, NIS, LDAP, Active

Directory

Openfiler NSA (NAS screenshots)Openfiler NSA (NAS screenshots)

Other DistrosOther Distros

• FreeNAS (FreeBSD) – SAN and NASFreeNAS (FreeBSD) SAN and NASSMB/CIFS,FTP, NFS, TFTP, AFP, RSYNC, Unison, iSCSI (initiator and target)iSCSI (initiator and target)

• Other Linux distros –a Debian like distros Ub nt etc SAN and a. Debian-like distros, Ubuntu, etc. – SAN and

NAS = command lineb Other lin distrosb. Other linux distros

REPLICATE!REPLICATE!

Why Replicate Data?Why Replicate Data?

• Fail safe (backup)Fail safe (backup)• HA (high availability)

W li i f• Wikipedia: “Replication is the process of sharing information so as to ensure consistency between redundant resources, such as software or hardware components, to

l b l f l limprove reliability, fault-tolerance, or accessibility.”

Software ReplicationSoftware Replication

• SAN Block Level Replication – DRBD (Distributed SAN Block Level Replication DRBD (Distributed Replicated Block Device)

On to DRBD and beyond…On to DRBD and beyond…

62 AttorneysColumbus

62 Attorneys

4/8 TB4/8 TB

Offices

ClevelandMarion2 Attorneys4/8 TB

Marion1 Attorney

AoE DRBD

Solutions: Pieces and Parts

OpenFiler

iSCSIiSCSI

Di t ib t d

DRBD

DistributedReplicatedDRBD= pBlockDeviceDevice

Replicated Block DeviceReplicated Block Device

• RAID 1RAID 1• Between servers, instead of hard drives.

What’s it mean?What s it mean?

• Raid 1Raid 1• Across Servers, Instead of Drives

Runs on LinuxRuns on Linux

• RedHat\CentOSRedHat\CentOS• Suse• Debian• Ubuntu

I Use UbuntuI Use Ubuntu

• Create identically sized unmounted partitionsCreate identically, sized unmounted partitions.– One partition for DRBD metadata.

One partition for user data– One partition for user data.

• Give servers static IPs.• Make sure time on servers is in sync.

Implementing DRBDImplementing DRBD

• All of this is command line…All of this is command line…• In.stall DRBD from repository.• Configure drbd conf fileConfigure drbd.conf file.

– ComplementaryPrimary or Secondary– Primary or Secondary

• Load DRBD: modprobe DRBD• I t ll d fi H tb t f it• Install and configure Heartbeat from repository.• Set virtual IP address.

S DRBD• Start DRBD.

What does it look like?What does it look like?

Filesystem Size Used Avail Use% Mounted ony/dev/sda5 4.6G 430M 4.0G 10% /tmpfs 126M 0 126M 0% /dev/shm/dev/sda1 89M 11M 74M 13% /boot/dev/sda1 89M 11M 74M 13% /boot/dev/drbd0 24G 33M 23G 1% /data

File SystemsFile Systems

• NFSNFS• CIFS\SMB