IBM DS4000 Presentation

© 2008 IBM Corporation

Linux and Storage News

Alexander Warmuth

SOS05

New Enterprise Forum: Information Infrastructure - IBM System Storage and Storage Networking Technical Symposium

© 2008 IBM Corporation2 w3.escc.mainz.de.ibm.com Aug 15, 2008

European Storage Competence CentreThe ESCC provides end to end support for IBM storage solutions

Pre-sales support including:

–Technical Q&A

–Solution Design

–Proof of Concepts

–Customer briefings and workshops

Implementation and post-sales support including:

–Implementation Services

–Product Field Engineering

–Customer Care

Intranet Site : w3.escc.mainz.de.ibm.com


© 2008 IBM Corporation3 Aug 15, 2008

Objectives

Introduce the latest storage related Linux technologies

Explain how they work

What is ready for use?

What is supported?



Agenda

DS4000 and Linux native multipathing

SAN Boot from multipath devices

N-Port ID virtualization

Other Linux and storage news

Closing



DS4000 and Linux Native Multipathing





Closing



BlockDevices

BlockDevices

Device Mapper Multipathing

HardwareHBA

0HBA

1

Storage Server

A B

SAN

SCSI

Device Mapper

Built on top of existing block devices

Device scan with read

– Capacity

– Partitions

– File systems

– Meta data



BlockDevices

BlockDevices

Device Mapper Multipathing Magic

HardwareHBA

0HBA

1

Storage Server

A B

SAN

SCSI

Device Mapper

multipath

multipathd

/dev/disk/by-id/...

/by-path/...

/by-uuid/...



DS4000 modes of operation

AVT on

– full LUN access on both controllers

– LUN automatically transferred to accessing controller

– Not for cluster solutions with shared access to volumes

AVT off, RDAC Mode

– SCSI Inquiry, Test Unit Ready, Mode Sense / Select, Reserve / Release, Read Capacity possible

– No Read / Write access through alternate controller



kernel: Vendor: IBM Model: 1814 FAStT Rev: 0916

kernel: Type: Direct-Access ANSI SCSI revision: 05

kernel: SCSI device sdf: 104857600 512-byte hdwr sectors (53687 MB)

kernel: sdf: Write Protect is off

kernel: sdf: Mode Sense: 77 00 10 08

kernel: SCSI device sdf: drive cache: write back w/ FUA





kernel: sdf: unknown partition table

kernel: sd 4:0:0:1: Attached scsi disk sdf

kernel: sd 4:0:0:1: Attached scsi generic sg13 type 0

kernel: device-mapper: multipath rdac: using RDAC command with timeout 15000

BlockDevices

BlockDevices

Device Scan - Owning Controller

HardwareHBA

0HBA

1

Storage Server

A B

SAN

SCSI

Device Mapper











kernel: sdf: unknown partition table

kernel: sd 4:0:0:1: Attached scsi disk sdf


kernel: device-mapper: multipath rdac: using RDAC command with timeout 15000





kernel: SCSI device sde: 104857600 512-byte hdwr sectors (53687 MB)

kernel: sde: Write Protect is off

kernel: sde: Mode Sense: 77 00 10 08

kernel: SCSI device sde: drive cache: write back w/ FUA





kernel: sde:end_request: I/O error, dev sde, sector 0

kernel: Buffer I/O error on device sde, logical block 0

kernel: end_request: I/O error, dev sde, sector 0




kernel: unable to read partition table

kernel: sd 3:0:0:1: Attached scsi disk sde


BlockDevices

BlockDevices

RDAC Mode – Alternate Controller

HardwareHBA

0HBA

1

Storage Server

A B

SAN

SCSI

Device Mapper






















BlockDevices

BlockDevices

AVT Mode – Alternate Controller

HardwareHBA

0HBA

1

Storage Server

A B

SAN

SCSI

Device Mapper

Automatic volume transfer

– Access possible

– No error messages

Not suitable for cluster environments



mppUpper

BlockDevices

BlockDevices

RDAC Multipathing Driver

HardwareHBA

0HBA

1

Storage Server

A B

SAN

SCSI

mppVhba RDAC upper layer

– Inhibits mapping of block devices

– No device scan

RDAC virtual HBA

– New block devices

– Regular integration

– Device Mapper on top possible



BlockDevices

BlockDevices

DM-MP Development

HardwareHBA

0HBA

1

Storage Server

A B

SAN

SCSI

Device Mapper

Hardware Handler in SCSI layer

Inhibit device scan

Multipathing based on UUID

No read access neccessaryRDAC RDAC



BlockDevices

BlockDevices

DS4000 DM-MP Configuration

HardwareHBA

0HBA

1

Storage Server

A B

SAN

SCSI

Device Mapper

device {

vendor "IBM"

product "1814"

path_grouping_policy "group_by_prio"

path_checker "rdac"

hardware_handler "1 rdac"

prio "rdac�

}

defaults {

failback immediate

no_path_retry "queue"

}

Hardware table

Change defaults



Controlling DM-MPx3650-lab-10:~ # multipathd -k

multipathd> help

multipath-tools v0.4.8 (08/02, 2007)

CLI commands reference:

list|show paths

list|show maps|multipaths

list|show maps|multipaths status

list|show maps|multipaths stats

list|show maps|multipaths topology

list|show topology

list|show map|multipath $map topology

list|show config

list|show blacklist

list|show devices

add path $path

remove|del path $path

add map|multipath $map

remove|del map|multipath $map

switch|switchgroup map|multipath $map group $group

reconfigure

suspend map|multipath $map

resume map|multipath $map

reinstate path $path

fail path $path

multipathd>



BlockDevices

BlockDevices

DS4000 Examples

HardwareHBA

0HBA

1

Storage Server

A B

SAN

SCSI

Device Mapper

multipathd> show multipaths

name sysfs uuid

mpatha dm-0 3600a0b800011f9b200005e2947d762c0

mpathb dm-1 3600a0b800011fa780000340847e8a1f3

multipathd> show maps status

name failback queueing paths dm-st

mpatha immediate on 2 active

mpathb immediate on 2 active

multipathd> show maps stats

name path_faults switch_grp map_loads total_q_time q_timeouts

mpatha 1 3 0 0 0

mpathb 0 0 2 0 0

multipathd> show paths

hcil dev dev_t pri dm_st chk_st next_check

4:0:0:0 sdd 8:48 3 [active][ready] XXXX...... 8/20

3:0:0:0 sdc 8:32 0 [active][ghost] X......... 3/20

4:0:0:1 sdf 8:80 3 [active][ready] XXXXXXXX.. 16/20

3:0:0:1 sde 8:64 0 [active][ghost] XXXXXXXXX. 19/20

multipathd> show maps topology

mpatha (3600a0b800011f9b200005e2947d762c0) dm-0 IBM,1814 FAStT

[size=50G ][features=1 queue_if_no_path][hwhandler=1 rdac ]

\_ round-robin 0 [prio=3][active]

\_ 4:0:0:0 sdd 8:48 [active][ready]

\_ round-robin 0 [prio=0][enabled]

\_ 3:0:0:0 sdc 8:32 [active][ghost]

mpathb (3600a0b800011fa780000340847e8a1f3) dm-1 IBM,1814 FAStT

[size=50G ][features=0 ][hwhandler=1 rdac ]


\_ 4:0:0:1 sdf 8:80 [active][ready]


\_ 3:0:0:1 sde 8:64 [active][ghost]



DM-MP und SCSI Reservation

No SCSI Reservation support

Resource fencing in clusters

– Reservation on block devices

– Cluster file systems

– No shared access, data replication



DM-MP and DS4000 Summary

Supported by Linux vendors

– SLES 10 SP1

– RH-EL 5 U1

Not supported by IBM

RDAC functionality implemented

– Fail over / fail back functionality equivalent

– No SCSI reservation support

Error messages during startup and reconfiguration



SAN Boot from Multipath Devices





Closing



BlockDevices

SAN Boot Single Path

HardwareHBA

0HBA

1

Storage Server

A B

SAN

SCSI

No difference to local SCSI boot

No redundancy

Initial RAM Disk



BlockDevices

BlockDevices

SAN Boot Multipath

HardwareHBA

0HBA

1

Storage Server

A B

SAN

SCSI

Device Mapper

Root on multipath

– DM-MP in Initial RAM Disk implemented

– SLES: install on single path, turn on DM-MP afterwards

– RH-EL: turn on DM-MP for installation

Boot from multipath

– Some boot loaders not yet DM-MP aware

– BIOS problems still exist

Initial RAM Disk



Device scans lead to SCSI errors

– Long installation time

– Long boot time

– Boot loader configuration problems

Easier with AVT on

BlockDevices

BlockDevices

SAN Boot, DM-MP and DS4000

HardwareHBA

0HBA

1

Storage Server

A B

SAN

SCSI

Device Mapper






















SAN Boot and DM-MP Summary

Root on dm-multipath is easy

Boot from dm-multipath: boot loader concerns

Boot from SAN and DS4000

– Problematic with DM-MP

– Easy with RDAC

All other IBM storage systems ok



N-Port ID Virtualization





Closing


© 2008 IBM Corporation Aug 15, 2008

What is NPIV?

HBA “multiplication”

– Virtual adapters presented to the operating system

– Virtual N-ports presented to Fabric

• Dedicated WWPN• Appears just as physical port

HBA and switch must be NPIV capable

Used in virtualized systems

– VMWare

– Linux on System z and VM

– XEN integration in implementation



XEN Attachment Choices

XEN Hypervisor

Dom0

DomU1LVol

Physical volume

Logical Volume

Image file

DomU3

DomU2

File

Remember last year?



XEN Hypervisor

Switch

XEN: Storage Management TodayDom0 DomU1 DomU2

Storage Server

N-Port

N-Port

N-Port

Zoning and LUN Mapping to Hardware

Virtual drivers for disk assignment to images

N-Port



XEN Hypervisor

Switch

Virtual Storage in XEN - DisadvantagesDom0 DomU1 DomU2

Storage Server

N-Port

N-Port

N-Port

Management

Security

Performance Impact

Image migration

N-Port



XEN Hypervisor

Switch

XEN and Virtual N-PortsDom0 DomU1 DomU2

Storage Server

N-Port

N-Port

N-Port

Provided by HBA

– managed by Dom0

– Dedicated WWPN

– Appears just as physical port

Assigned to VM

Volume Mapping to V-PortV-Port V-Port

V-Port V-PortN-Port



XEN Hypervisor

Switch

Virtual Machine MigrationDom0 DomU1 DomU2

Storage Server

N-Port

N-Port

N-Port V-Port V-Port

V-Port V-PortN-Port

XEN Hypervisor

Dom0 DomU1'

V-PortN-Port

N-Port V-Port



Setup and Configure NPIV

Create virtual HBA

Similar commands to remove and configure V-Ports

V-Port management integrated into virtualization

# cd /sys/class/scsi_host/host2

# echo "2100001b32013aff:2000001b32013aff" > vport_create

WWPN WWNN



Virtual Ports in the SAN



NPIV Enabled SAN Example



Virtual Ports Seen by Storage System



Other Linux and Storage News





Closing



HBA Information

Find WWPN and WWNN

HBA Driver Info

# cat /sys/class/fc_host/host3/port_name

0x10000000c954ceba

# cat /sys/class/fc_host/host3/node_name

0x20000000c954ceba

# modinfo lpfc

filename: /lib/modules/2.6.16.60-0.9-smp/.../lpfc/lpfc.ko

version: 0:8.2.0.21

author: Emulex Corporation - [email protected]

description: Emulex LightPulse Fibre Channel SCSI driver 8.2.0.21

license: GPL



Dynamic Volume Resizing

Linux can't use additional capacity on-the-fly

– Minimum interruption: unmount file system

– Reboot usually not necessary

LVM: proven method to add capacity online

– Add new disk

– Expand volume group

– Expand logical volume

– Resize file system

+ Cap

Linux host

HBA 0

LUN 0

HBA 1

LUN 1Remember last year?



Agenda





Closing



Summary







Sources of Information

Linux and IBM Storage Community

– Thomas Gerisch, [email protected]

– Wiki: https://w3.webahead.ibm.com/w3ki/display/linuxstorage/Home

Enterprise Linux Services, Germany

– Michael Weisbach,[email protected]

– Wiki: https://els.chemnitz.de.ibm.com/wiki/index.php/Main_Page

Multipath Usage Guidehttp://sources.redhat.com/lvm2/wiki/MultipathUsageGuide

Multipath storage with Xen and DS4800http://www.ibm.com/developerworks/linux/library/l-multipath-xen/index.html

mailto:[email protected]

https://w3.webahead.ibm.com/w3ki/display/linuxstorage/Home

http://sources.redhat.com/lvm2/wiki/MultipathUsageGuide

http://www.ibm.com/developerworks/linux/library/l-multipath-xen/index.html



Questions & Discussion

More Questions?

Contact: [email protected]



Trademarks and DisclaimersThe following are trademarks of the International Business Machines Corporation in the United States and/or other countries. For a complete list of IBM Trademarks, see www.ibm.com/legal/copytrade.shtml: ADD TRADEMARKS FROM DOCUMENT THAT APPLY

The following are trademarks or registered trademarks of other companies:

Java and all Java based trademarks and logos are trademarks of Sun Microsystems, Inc., in the United States and other countries or both

Microsoft, Windows,Windows NT and the Windows logo are registered trademarks of Microsoft Corporation in the United States, other countries, or both.

Intel, Intel logo, Intel Inside, Intel Inside logo, Intel Centrino, Intel Centrino logo, Celeron, Intel Xeon, Intel SpeedStep, Itanium, and Pentium are trademarks or registered trademarks

of Intel Corporation or its subsidiaries in the United States and other countries.

UNIX is a registered trademark of The Open Group in the United States and other countries or both.

Linux is a trademark of Linus Torvalds in the United States, other countries, or both.

Other company, product, or service names may be trademarks or service marks of others.

NOTES:

Any performance data contained in this document was determined in a controlled environment. Actual results may vary significantly and are dependent on many factors including system hardware configuration and software design and configuration. Some measurements quoted in this document may have been made on development-level systems. There is no guarantee these measurements will be the same on generally-available systems. Users of this document should verify the applicable data for their specific environment.

IBM hardware products are manufactured from new parts, or new and serviceable used parts. Regardless, our warranty terms apply.

Information is provided “AS IS” without warranty of any kind.

All customer examples cited or described in this presentation are presented as illustrations of the manner in which some customers have used IBM products and the results they may have achieved. Actual environmental costs and performance characteristics will vary depending on individual customer configurations and conditions.

This publication was produced in the United States. IBM may not offer the products, services or features discussed in this document in other countries, and the information may be subject to change without notice. Consult your local IBM business contact for information on the product or services available in your area.

http://www.ibm.com/legal/copytrade.shtml



Trademarks and Disclaimers (continued)NOTES:

All statements regarding IBM's future direction and intent are subject to change or withdrawal without notice, and represent goals and objectives only.

Information about non-IBM products is obtained from the manufacturers of those products or their published announcements. IBM has not tested those products and cannot confirm the performance, compatibility, or any other claims related to non-IBM products. Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products.

Prices are suggested US list prices and are subject to change without notice. Starting price may not include a hard drive, operating system or other features. Contact your IBM representative or Business Partner for the most current pricing in your geography.

Any proposed use of claims in this presentation outside of the United States must be reviewed by local IBM country counsel prior to such use.

The information could include technical inaccuracies or typographical errors. Changes are periodically made to the information herein; these changes will be incorporated in new editions of the publication. IBM may make improvements and/or changes in the product(s) and/or the program(s) described in this publication at any time without notice.

Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites. The

materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk.

IBM makes no representation or warranty regarding third-party products or services including those designated as ServerProven, ClusterProven or BladeCenter Interoperability Program products. Support for these third-party (non-IBM) products is provided by non-IBM Manufacturers.

IBM may have patents or pending patent applications covering subject matter in this document. The furnishing of this document does not give you any license to these patents. Send license inquires, in writing, to IBM Director of Licensing, IBM Corporation, New Castle Drive, Armonk, NY 10504-1785 USA.

IBM DS4000 Presentation

Documents