-
1
IBM Software Group
August 2003 © 2003 IBM Corporation
Considerations for using TSM in a SAN
Steve Strutt, Tivoli Software, IBM [email protected]
Sept 2002Revision 5
IBM Software Group | Tivoli software
Considerations for using TSM in a SANl © 2003 IBM
Corporation2
AgendaSAN Exploitation - LAN-Free backupPerformance
characteristics
RequirementsLAN, Hardware, Software, Device, dependencies
SAN considerationsDevice fail-overHBA considerationsSAN design
considerationsDevice addressing considerations
Going LiveTesting, Diagnosing ProblemsHints and Tips
Question and Answer
-
2
IBM Software Group | Tivoli software
Considerations for using TSM in a SANl © 2003 IBM
Corporation3
LAN-free BackupAdvantages:- client data can be local or
SAN-attached- transparent to application/database- takes backup
traffic off the LAN- reduces CP cycles on backup server (no I/O)-
faster speed (usually)- only one backup server needs
administration
Disadvantages:- still requires CP cycles on client for backup
I/O- careful scheduling to avoid tape drive contention
(or exploit disk pooling)
TAPE
FC device
DISK
LAN
D A TAD A TA
DATA
Direct to tape
Disk pool staging
IBM Software Group | Tivoli software
Considerations for using TSM in a SANl © 2003 IBM
Corporation4
Performance characteristicsLAN-Free is not necessarily
faster
Only network eliminated as bottleneck
Could be other bottlenecksTape drives, Disk subsystemData
types
good performance for large files and databasessmall files,
performance limited by file system and TSM
architectureLAN-Free to tape
Potentially better performance for large files, as bottleneck
becomes file system or tape device
Small files cause tape drives to stop-start more and drives drop
out of streaming mode.
LAN-Free to diskIdeal for small files, no stop-start
overhead
-
3
IBM Software Group | Tivoli software
Considerations for using TSM in a SANl © 2003 IBM
Corporation5
Customer Performance FiguresLarge UK High Street Retailer
TDP for SQL Server on ESS to 3584 LTO Backup 61.4GB hour
(17MB/s) to single driveRestore 44.2GB hour (12.3MB/s) from single
drive
NT Filesystem on ESS disk to 3584 LTO - small filesBackup 10GB
hour to LTO Restore 6.2GB hour from LTO Could be slower than LAN if
tape drives do more stop/start operations
Large UK BankTDP for Exchange to 3583 LTO
Backup 52GB/h (14.4M B/s) to single driveRestore 51GB/h
(14.2MB/s) from single drive
IBM Software Group | Tivoli software
Considerations for using TSM in a SANl © 2003 IBM
Corporation6
Agenda - Requirements
Hardware – LAN
– Library support for LAN-Free
– SAN device supportSoftware
– Evolving TSM support for LAN-Free
– TSM code dependencies
-
4
IBM Software Group | Tivoli software
Considerations for using TSM in a SANl © 2003 IBM
Corporation7
Requirements - LAN
LAN-Free still requires LAN for meta dataFor Large files and
databases
Minimal usage Small files
Maybe the same if not more meta data on LAN than data on SAN if
files are very small.
LAN performance and loading still important
IBM Software Group | Tivoli software
Considerations for using TSM in a SANl © 2003 IBM
Corporation8
SAN Device support
Initially hardware configurations were certified by Tivolimany
different combinations, not all could be tested
Common configurations now testedThe Tivoli view is that hardware
is transparent to TSM
If the hardware vendors have validated the devices’ use in a SAN
and TSM supportsthe device, then it is supported in a SAN
configuration with Tivoli Storage Manager
-
5
IBM Software Group | Tivoli software
Considerations for using TSM in a SANl © 2003 IBM
Corporation9
Library support for LAN-FreeNative TSM LAN-Free support for:
– 3494 Libraries
– SCSI Libraries– Controlled via SCSI control path
– SCSI connect– Fibre Channel Connect
NO Native TSM LAN-Free support for– STK and ADIC AML
libraries
– Require Gresham EDT on every TSM server and Storage Agent. –
STK - ACSLS, Library Station (s390)– ADIC – DAS for AML/2,
AML/J
IBM Software Group | Tivoli software
Considerations for using TSM in a SANl © 2003 IBM
Corporation10
TSM evolving support for LAN-FreeLAN-Free supports
BA Client, file level
BA Client, Volume level/Image BackupPrior to 5.1.5, the tape was
rewound between each volume
No support for Backup Set restoreNT/W2K
Supported from 4.1.0, NT/W2K server and NT/W2K TDPs
4.2.1 for Backup Archive ClientSun Solaris and AIX
Supported from 4.2.0, TDPs and Backup Archive ClientHP-UX
Supported from 5.1.0, Full TSM device driver supportLinux (x86,
zLinux and zOS)
Supported from 5.2.0
-
6
IBM Software Group | Tivoli software
Considerations for using TSM in a SANl © 2003 IBM
Corporation11
TSM code dependencies
TSM 4.2 and 5.1– TSM Server and Storage Agent code MUST be at
the same PTF and
patch level
– Restricts ability to roll out new codeTSM 5.2
– TSM Server and Storage Agent code only dependant at version
and release level
– Independent of PTF level
– Easier to deploy and install maintenance
IBM Software Group | Tivoli software
Considerations for using TSM in a SANl © 2003 IBM
Corporation12
Agenda - SAN considerationsHigh Availability considerations
– SANs offer high availability
– Tape high availability options
– Fail over and tapeHBA and tape setupHBA SharingSAN design
considerationsAttaching tape drives to SANsSAN Operational
considerationsDevice addressing considerations
– Managing device addressing– Persistent Naming– Static Device
Naming
– Mapping device names to devices
-
7
IBM Software Group | Tivoli software
Considerations for using TSM in a SANl © 2003 IBM
Corporation13
SAN’s offer high availabilityWorks well for disk
– Arrays designed to have two access paths
– Device drivers exploit multiple paths– Failover of paths
– Clustering software fails over applications and disk
storage
– Effective TSM support for fail-over– DB and Log Mirrors, DB
page shadowing– support for MSCS and HACMP
Does not work well for tape– Tape drives usually single
connection
– SCSI Controlled tape libraries usually have a single control
point
– Drivers only exploit single path
– Failover issues
Clustering
Redundancy Multiple paths
IBM Software Group | Tivoli software
Considerations for using TSM in a SANl © 2003 IBM
Corporation14
Tape high availability options
3590 Dual SCSI/Fibre ports– AIX ATAPE driver can failover to
second pathIf the primary path, say rmt0 is not available, the
driver will use other paths/device
names via an alternative HBA/device port to access the same
device. This is described in the 3590 manual, ‘IBM TotalStorage
Tape Installation and User
Guide’. ftp://ftp.software.ibm.com/storage/devdrvr/Doc/
358x LTO libaries– New option to have two library controller
LUNs on different paths
– Use two different HBA’s/Switches– Only supported with AIX
ATAPE driver– Still single path to drives
– Divide drives between two HBA’s and switches– Still access
library, reduced number of drives
Good news for AIX
-
8
IBM Software Group | Tivoli software
Considerations for using TSM in a SANl © 2003 IBM
Corporation15
Fail-over and TapeSCSI Reserve/Release used to serialize access
to drives
– Maintains integrity if device definitions are wrongOutstanding
SCSI Reserves remain if server using drives go downDrives only
released if:
Failed server restarted
Drive powered off and on.
SCSI Reset issued
Potential Impact on other users
SCSI Attached Tape drives can be reset using SCSI Reset
Fibre attached tape drives do not honor SCSI Resets
IBM Software Group | Tivoli software
Considerations for using TSM in a SANl © 2003 IBM
Corporation16
Supported SCSI Fail-over configurationsTSM 4.2+ with MSCS on
W2K
– Devices registered in wizard, SCSI reset on fail-overTSM 5.1.5
with HACMP on AIX
– Devices defined in ‘startserver’ script, issues verifydevice
to reset device
SharedDisk
Shared Tape
Shared SCSIBus
TSM Serverin Cluster
TSM Clients
IP network
-
9
IBM Software Group | Tivoli software
Considerations for using TSM in a SANl © 2003 IBM
Corporation17
Supported Fibre Failover configurations
TSM Serverin HACMP cluster
TSM 5.2.0 with HACMP on AIX– Devices defined in ‘startserver’
script, issues verifyfcdevice to reset device
IBM Software Group | Tivoli software
Considerations for using TSM in a SANl © 2003 IBM
Corporation18
Fail-over and LAN-Free and Tape Library SharingOutstanding SCSI
Reserves remain if storage agents using drives go downDrives become
unavailable to other LAN-Free/Tape Sharing users
Only released if:Failed server restartedDrive powered off and
on.
Potential drive mapping issuesFC Reset issued
Potential Impact on other users
Potentially need more tape drives
Some horrible messages if TSM server tries to dismount tapes
belonging to failed storage agents.
-
10
IBM Software Group | Tivoli software
Considerations for using TSM in a SANl © 2003 IBM
Corporation19
Failure scenario
ANR8925W Drive DRIVE0 in library ATLP1000 has not been confirmed
for use by server UKSAN1_SA for over 1200 seconds.Drive will be
reclaimed for use by others.ANR8336I Verifying label of DLT volume
00157D in drive DRIVE0 (MT6.1.0.1).ANR8311E An I/O error occurred
while accessing drive DRIVE0 (MT6.1.0.1) for SETMODE operation,
errno = 1.ANR8355E I/O error reading label for volume 00157D in
drive DRIVE0 (MT6.1.0.1).ANR8311E An I/O error occurred while
accessing drive DRIVE0 (MT6.1.0.1) for OFFL operation, errno =
1.ANR8469E Dismount of DLT volume 00157D from drive DRIVE0
(MT6.1.0.1) in library ATLP1000 failed.ANR9999D mmsscsi.c(1647):
ThreadId Volume may still be in the drive DRIVE0
(MT6.1.0.1).ANR8446I Manual intervention required for library
ATLP1000.
Server running Storage Agent goes down (hardware failureFibre
loss), when using a tape device
TSM makes drive ‘unavailable’
IBM Software Group | Tivoli software
Considerations for using TSM in a SANl © 2003 IBM
Corporation20
Supporting LAN-Free fail-overNo TSM support for fail-over of
Storage AgentSupporting Storage Agent Fail-over
Configure two storage agents, one on each server
Different Storage Agent NamesNo dependency between TSM clients
and Storage Agent
BA client and TDP failover, Storage Agent does not
BA client and TDP automatically use new SA on fail-over
server
No device issues if LAN-Free client machine fails-over when not
using tape devices
Devices lost if LAN-Free client machine fails-over when using
tape drives
Require more tape drives
Procedures to recover lost drives
-
11
IBM Software Group | Tivoli software
Considerations for using TSM in a SANl © 2003 IBM
Corporation21
HBA and tape Setup considerationsCheck configuration
recommendations of tape hardware vendor
– IBM recommendations
NT/W2KMax Scatter-Gather must be set to 65 or greater
Unable to write to new tapes on Storage AgentW2K creates tapes
which cannot be readTSM will check for this in 4.2.1.11 and higher
levels.
Issues message, unable to use drive. W2K and LTO use 5.0.2.4 or
higher level of Ultrium LTO driver.
IBM Software Group | Tivoli software
Considerations for using TSM in a SANl © 2003 IBM
Corporation22
HBA SharingSharing of disk and tape on same HBA not always
supported by hardware vendors
Disk OK
Access to tape drives lost under high workload conditionsdrives
timeout, go offline
IBM supports disk and 3590 on AIX with 6227/8 adapter under
moderate workloads
IBM-SSG do not recommend sharing disk and tape in other
configurations.i.e. LTO some evidence that it is OK in low workload
environments, such as
previous NT/SQL server LAN-Free environment.
Possible solution, with new generation of adapters being
certified to support dual use without restrictions
-
12
IBM Software Group | Tivoli software
Considerations for using TSM in a SANl © 2003 IBM
Corporation23
SAN design
Most SANs designed for disk accessdata flow is optimized for
hosts disks
stovepipe design, separate SAN ‘islands’
Tape backup requires flow across SANMay need additional
Inter-Switch Links
More ports because of HBA Sharing issues
DiskArray
TAPE
IBM Software Group | Tivoli software
Considerations for using TSM in a SANl © 2003 IBM
Corporation24
Attaching tape drives to SANs
Most tape drives are Arbitrated Loop devicesNote some directors
do not support direct attachment of FC-AL devices
Brocade switches support Public and Private Loop devices in
fabric
InRange OK
McData directors do not support Arbitrated Loop Devicesrequire
use of Sphereon 4300 or similar departmental switch.Or SCSI
attached via Fibre-to-SCSI Gateway/Router
TAPETAPE
ED5000
4300 etc
TAPE
-
13
IBM Software Group | Tivoli software
Considerations for using TSM in a SANl © 2003 IBM
Corporation25
SAN Operational considerationsUsing fibre channel arbitrated
loops (Hubs)
During boot up sequence LIP will interrupt tape operations for
shared devices.
Rebooting servers can cause tape failures
When devices and servers are on the same loop, devices should
not be rebooted while tape is in use by a TSM server or Storage
Agent
Power up sequenceSAN, tape devices, then TSM Server and Storage
Agents
Scheduling to avoid drive contentionStorage Agents must wait if
no drives available.
Use MountWait parameter to avoid backup failuresManaging device
access
Device Class, Mount Limit
Device PATHs
IBM Software Group | Tivoli software
Considerations for using TSM in a SANl © 2003 IBM
Corporation26
Device Addressing ConsiderationsAll device addresses are defined
centrally on TSM server for LAN-Free
Library, Device and Path statements.Each host knows devices by
different device names
Changes in device names or SCSI addresses can cause
failuresRequirement to manage device addressing
All definitions must point to same physical deviceQuestion: How
to map all device names to the same physical device?
Device Namelb1.0.1.3//./tape0//./tape1/dev/rmt0/dev/rmt1
/dev/rmt/0st/dev/rmt/1st
TSM DefinitionLibrary Lib1 lb1.0.1.3Drive Drive0 //./tape0Drive
Drive1 //./tape1Path Drive0 /dev/rmt0Path Drive1 /dev/rmt1
Path Drive0 /dev/rmt/0stPath Drive1 /dev/rmt/1st
TAPE
TA PE
W2KTSM Server
AIX Storage Agent
Solaris Storage Agent
F C d ev ic eWWN1
WWN2
-
14
IBM Software Group | Tivoli software
Considerations for using TSM in a SANl © 2003 IBM
Corporation27
TapeDrives
SANGateway/Router
OS DeviceName to
TSM DeviceName
SCSI IDto
OS DeviceName
DeviceWWN
to SCSI ID
Device WWN
SCSI ID to LUN
ID1
ID2
ID3
SCSI Bus
WWN
WWN
Host
TSMOS
DeviceDriver
HBA
Managing Device Addressing in SANs - 1Host Device Names and SCSI
addresses can change
devices added or removeddevices failing
A number of address mappings occur.WWN to SCSI addressSCSI
address to device addressGateway/Router - device SCSI address to
LUN
Described in Redpiece: Managing device addressing of SAN
attached tape for use with Tivoli Storage Manager, REDP-0150-00
IBM Software Group | Tivoli software
Considerations for using TSM in a SANl © 2003 IBM
Corporation28
Managing Device Addressing in SANs - 2
Changes are more likely to happen if there are large numbers of
devices and hosts in SAN.
– New devices being added, or removedSolution
– Use HBA Persistent Naming– Fixes SCSI address to device
WWN
– Static device name mapping– Device Names remain unchanged–
Fixed device name to SCSI address mapping
– TSM 5.2– Automatic device tracking
-
15
IBM Software Group | Tivoli software
Considerations for using TSM in a SANl © 2003 IBM
Corporation29
HBA persistent naming supportHBA maintains a fixed WWN to SCSI
address relationshipSupport matrix
Platform Emulex Qlogic
AIX Not Applicable (use 6227/8 adapter)
Not Applicable (use 6227/8 adapter)
Windows NT/W2K
YES Yes (from 8.1.3 with
SANblade Manager) Solaris YES YES
IBM Software Group | Tivoli software
Considerations for using TSM in a SANl © 2003 IBM
Corporation30
Persistent Naming with Emulex on Windows
-
16
IBM Software Group | Tivoli software
Considerations for using TSM in a SANl © 2003 IBM
Corporation31
Qlogic HBA on WindowsQlogic from 8.1.3 onwards with SANBlade
Manager/SANSurfer
Download management utility from Qlogic Website
IBM Software Group | Tivoli software
Considerations for using TSM in a SANl © 2003 IBM
Corporation32
Static device naming with Microsoft WindowsTSM device driver
uses a naming convention which does not change as devices are added
or removed
mtx.y.z.nLTO device driver uses default Windows device
naming
\\.\tape0Can change if new devices added or removedWith W2K can
change in-flight if devices added or removed from SANRecommendation
to use mtx.y.z.n,
Use information from TSM Device Information screen to determine
mtx.y.z.n device addresses
In TSM 5.1, the mtx.y.z.n name is shown.
-
17
IBM Software Group | Tivoli software
Considerations for using TSM in a SANl © 2003 IBM
Corporation33
Static device naming on AIX and Solaris
AIX– Device drivers automatically track devices using serial
numbers
– This mapping is staticSolaris
– Static device naming convention
– Uses symbolic link to map device name to SCSI address
– Requires HBA Persistent Naming to be configured – HBA
configuration file
ls –l /dev/rmt/*lrwxrwxrwx 1 root other 45 Jan 3 14:22
/dev/rmt/0mt ->
../../devices/pci@1f,0/pci@1/scsi@2/mt@5,1:mt
IBM Software Group | Tivoli software
Considerations for using TSM in a SANl © 2003 IBM
Corporation34
TSM 5.2 Automatic device trackingTSM tracks devices by serial
number
– Specify S/N when defining devices– Automatic discovery of S/N
when device defined
At start of each operation TSM server and SA will check the
device is the one it expects it to be:
– Windows– Initiates a search for the device and changes mapping
to point to
new device and then continues operation.– UNIX
– Issues message and fails operation on that device
Avoids need for persistent binding on Windows– Qlogic complex to
set up. – Extra mgmt utilities. – Persistent binding options not
tested by hardware vendors!
-
18
IBM Software Group | Tivoli software
Considerations for using TSM in a SANl © 2003 IBM
Corporation35
Mapping device names to devices -1
In a SAN all definitions for a shared device must point to the
same physical device.
– Each host usually has a different device name for the device
depending on the hardware and configuration.
– Only common information available on each platform is the
deviceWWN and serial number.
Device Namelb1.0.1.3//./tape0//./tape1/dev/rmt0/dev/rmt1
/dev/rmt/0st/dev/rmt/1st
TSM DefinitionLibrary Lib1 lb1.0.1.3Drive Drive0 //./tape0Drive
Drive1 //./tape1Path Drive0 /dev/rmt0Path Drive1 /dev/rmt1
Path Drive0 /dev/rmt/0stPath Drive1 /dev/rmt/1st
TA PE
TA PE
W2KTSM Server
AIX Storage Agent
Solaris Storage Agent
F C d ev ic eWWN1
WWN2
IBM Software Group | Tivoli software
Considerations for using TSM in a SANl © 2003 IBM
Corporation36
Mapping device names to devices -2
1. Define drives on TSM server and determine relationship
between TSM device name, host device name and device WWN/Serial
No.
Also determine the Element Number of the device in the
library
2. For each (Storage Agent) host determine WWN/Serial # and
device name for each tape device.
3. Use WWN’s/Serial # to relate device names on each host to TSM
device name.
4. Define drives on TSM server using PATH statements
-
19
IBM Software Group | Tivoli software
Considerations for using TSM in a SANl © 2003 IBM
Corporation37
Mapping device names to devices-3TSM Device Name Drive0 Drive1
Drive…
WWN/Serial No.
Library Element Number
TSM Server
Storage Agent1
Storage Agent2
Storage Agent3
Host Device Names
……
IBM Software Group | Tivoli software
Considerations for using TSM in a SANl © 2003 IBM
Corporation38
Mapping device names to devices – Windows 1
TSM V5.1 – TSM Utilities, Device InformationShows a devices WWN
and Serial Number as well as device address (mtx.y.z.n)
Also allows central discovery of Storage Agent device
mappings
Requires SNIA HBA SAN Mgmt API to be installed
-
20
IBM Software Group | Tivoli software
Considerations for using TSM in a SANl © 2003 IBM
Corporation39
Mapping device names to devices – Windows 2 TSM Server, Device
Information display with Qlogic SNIA SAN Mgmt API InstalledDevice
information can also be displayed from this screen for Storage
Agents in the same Windows Domain
IBM Software Group | Tivoli software
Considerations for using TSM in a SANl © 2003 IBM
Corporation40
Mapping device names to devices -5
AIX lsattr –El mtx/rmtx shows devices WWNlscfg –vl mtx/rmtx
shows devices Serial Number
SolarisRelate device name to WWN using SCSI and LUN
addresses
ls -l shows device name and SCSI/LUN mapping
dmesg output shows SCSI Target address to WWN mapping
ls –l /dev/rmt/*lrwxrwxrwx 1 root other 45 Jan 3 14:22
/dev/rmt/0mt ->
../../devices/pci@1f,0/pci@1/scsi@2/mt@5,1:mt
dmesg
(/var/adm/messages)......qla2200-hba0-SCSI-target-id-5-fibre-channel-name="100000e00201d0d7";
-
21
IBM Software Group | Tivoli software
Considerations for using TSM in a SANl © 2003 IBM
Corporation41
Agenda - Going Live
Testing– testing
Diagnosing Problems– Storage agent messages
– TSM device utilities
IBM Software Group | Tivoli software
Considerations for using TSM in a SANl © 2003 IBM
Corporation42
Testing
Test after every hardware change– Changes can introduce errors
which can cause failure to recover data.
Proving data integrity– TSM 5.1 includes end to end CRC
checking
– use only during testing
-
22
IBM Software Group | Tivoli software
Considerations for using TSM in a SANl © 2003 IBM
Corporation43
Testing
Check tape hardware works reliably with TSM server in LAN
configuration firstCheck TDP’s and B/A Client work on LAN firstTest
each drive with each Storage Agent to check they are properly
defined and accessible.
Use BA ClientANR8779E (Session: 7, Origin: UKSAN4_SA) Unable
to
open drive /dev/mt1, error number=2.
Invalid device specifiederror number=16, Device Busy (SCSI
Reserved to another
system)
IBM Software Group | Tivoli software
Considerations for using TSM in a SANl © 2003 IBM
Corporation44
Diagnosing problems
Storage Agent can be run in foreground session, to see all
messages.
– All Storage Agent messages should be logged centrally in the
server Activity Log
– Can issue commands from TSM server consolestorage_agent1:
QUERY SESSION
mttest and lbtest utilities– Provided in utilities or devices
directory
– test operation of tape devices or library operations.– Shows
device serial numbers
-
23
IBM Software Group | Tivoli software
Considerations for using TSM in a SANl © 2003 IBM
Corporation45
Question and Answer