1 Tsm Admin Best Practices

© 2011 IBM Corporation

TSM Administration Revisited

Kathy Mitton, Tivoli Storage Manager Server Manager

TSM Symposium, September 2011

© 2011 IBM Corporation2

Abstract

This session will explore TSM server operations, daily maintenance, and best practices to

optimize your TSM server. The speaker will also discuss the administrative and reporting

capabilities for the server along with examples and rationale for managing and scheduling

server maintenance tasks.


Agenda

• Revisiting TSM practices• Lifecycle• Best Practices• Workflow• Scripts and Sequencing• Schedules• Operational Limits• Monitoring• References


Revisit TSM Practices Periodically To Keep Pace With Change

� TSM environments are impacted by

– data growth

– changing business needs

– people/personnel changes

� Many TSM installations propagate new TSM instances based on

– when the administrator first learned the product

– when administrator last had to develop a solution for the challenges faced by the

organization at that time

� Frequently newly deployed servers will duplicate existing servers in order to

– simplify administration, the more uniform things are the easier it is to administer

– leverage the time/effort spent in figuring out an earlier server deployment and

configuration

� Revisiting your TSM administrative setup may allow you to

– further streamline your operations

– identify and exploit newer TSM functionality

� Goals for TSM practices

– simplify administration

– reduce potential for errors


TSM And Your Business Are Dynamic

Storage Hierarchy

DB2

TSM ServerDatabase

Client Workloads change over time:• More clients• More data• Different types of clients• Network/infrastructure changes

Storage hierarchy changes based on:• Capacity requirements• Changes to H/W and vendors• Performance and cost needs of an organization

TSM Server and Database Change based on:• Growth• Changes to H/W and vendors

Take a Look at Your TSM Environment: • Are you using and exploiting the best functions and features TSM offers?

• Has your environment changed such that TSM is not being used optimally?


Have you looked at reporting lately?

An aggregated view of reporting and monitoring for the entire TSM environment

Reporting and monitoring introduced in

V6.1.

Release to release improvements in:

• Install

• Configuration

• Deployment

• More Reports


Have you looked at Administration Center lately?

With TSM 6.2, the administration center can

be used to orchestrate the “push” of updates to windows clients


Utilizing Best Practice Work Flow Will Minimize Problems

� “Best practice” concepts for Tivoli Storage Manager are based on:

– Field Experiences• Observations from customer implementations• Feedback through problem reports, market requirements, business partner

discussions, and other communication channels

– Development insight based on the design and implementation of the algorithms, processes, and code

� Generally, the topics discussed are applicable to both V5.x and V6.x TSM servers

– There may be command syntax shown that is specific to a newer release

– Some topics or discussion points may be applicable only to newer release• For example, the importance of BACKUP VOLHIST is much more significant for V6.x.


Overview of Server Workflow – Daily Cycle

Client Workload

Server Workload

Time

Data Ingest(Backup, Archive, HSM)

Server Maintenance Activities


TSM Wheel of LifeOverview

SAN

Whether viewed as a “sine wave” cycle or the “Wheel of Life”, some view of the cyclic nature of TSM operations and the daily support for these operations is helpful


Observing System Resource Relative to Workflow Cycle Will Help Provide Guidance For Changes

� The peaks during workload are limited by total available resource on the machine (CPU, Memory, I/O throughput, etc)

– The client workloads are usually done using schedules• Most often, the main data ingest is through a “nightly” backup window which may be

one or more schedules initiating the backup of various groups of clients

– The server actions are the back-end maintenance actions necessary to• protect the client data by performing backup storage pools• position the data appropriately in the hierarchy based on policies, storage

management, and the data flow through the server• perform the other server operations to keep the database, storage hierarchy, and

system healthy and ready for the next set of actions

� Client operations may happen (and often do) throughout the day

– For example, archive operation for DB logs can occur as needed as opposed to limited only to the nightly “ingest” window

– Resources such as mount points need to be considered for these “always possible” operations


Identify Peak Workload and Task Overlap For Workflow Improvement

� During the client workload phase, the server resources (storage, CPU, memory, and I/O bandwidth) should be devoted to supporting the workload – At the peak of the client workload, the majority of the server resources should be in

support of the client workload– At 6.x, we’ve tested over 500 concurrent client sessions and have seen that server

performance can degrade between 500 and 700 concurrent sessions– The number of concurrent sessions any single server might achieve will be highly

dependent on server resource

� During the server workload phase, the server resources are being dedicated to managing the recently received data from the client workload– These resources are necessary for the storage, policy management, and maintenance of

the server– Optimal server size is based on whether all operations can complete in 24hour period

� When workloads overlap or are not given sufficient resource impacts may occur:– Less CPU and memory available to support a given operation– Performance degradation– Insufficient space– Data placement may be sub-optimal– Operations may fail


Server Workflow Goals and Priorities

Storage Hierarchy

DB2

TSM ServerDatabase

Protect the client data:• Storage pool backup• Copy active• Database backup

Protect the server:• Data movement activities (reclamation, migration)• Expiration• Identify processing for deduplication enabled environments.

Disaster Recovery and Availability:• Onsite recovery through DB restore or clustering (where available).• Offsite recovery through DB restore + copy storage pools.• Other offsite recovery techniques


Illustration and Best Practice Sequencing of Server Workflows

Protecting the Client Data:

• STORAGE POOL BACKUP

• COPY ACTIVEDATA

• DATABASE BACKUP

• BACKUP VOLHIST/DEVCLASS

Protecting the Server:

• EXPIRE INVENTORY

• RECLAMATION

• MIGRATION

Prepare and Execute for Disaster

Recovery:

• DELETE VOLHIST

• MOVE DRMEDIA

• PREPARE

Identify

Time

Table Reorganization


Other Workflow Observations

� Flows and sequencing are not absolute– There may be reasons to sequence things differently– Proof is in providing capabilities specific to your environment

� Database backup for V6 changes the paradigms compared to V5– TYPE=FULL being used predominantly

• This causes the pruning of the ARCHIVE log space• Necessary for proper care and feeding of server health• Helps manage or mitigate amount of storage needed for archive logs• In practice, TYPE=INCR may backup close to the same amount and take almost the same

amount of time as TYPE=FULL– Take extra precautions to protect the volume history file for restore purposes.

• Make multiple copies of the file• Store to many different locations• Critical for RESTORE processing for the server• Unlike V5, V6 volume history can’t be built by hand and without volume history TSM server is

NOT restorable

� Deduplication exhibits different characteristics than traditional server processes:– IDENTIFY can be configured as ALWAYS on

• We do see environments that use it as an always on process• Reclamation workloads increase based on having to reclaim based on policy deletion (expiration)

and also deduplicate chunk elimination

� V6 Reorganization will require some isolated cycles


Employing Disruptive Technologies May Change Workflows

� Disruptive technologies are being used to bring new capabilities to TSM environments

� TSM processes and activities may be eliminated or require re-sequencing as result of disruptive technology

� External events or capabilities may affect what needs to be done within TSM

� One example of disruptive technology is offsite disaster recovery/availability:– Disk subsystem

• Replication of TSM db/log/homeDir to target site• Replication of FILE or other DISK storage pools to target site• Critical consideration: consistency groups

– VTL• Replication of storage pool to target site• Critical consideration: how to reconcile against available server database at target site

– TSM V6 HADR


Disruptive Technologies Example

DB DB

“Replication” of server database (DB, log) using either:• Device level replication with consistency groups.• V6.2 server using database HADR.

“Replication” of storage pool(s) using:• Disk device level replication with consistency groups.• VTL to VTL system replication.

Server A(Primary)

Server A(Disaster Recovery)


Validate Your Scripts Implement Best-Practice Workflow

� We still see scripts not exploiting PARALLEL and SERIAL sequencing enhancements even though these script enhancements have been around for many years

– Many scripts may not be running optimally because they aren’t exploiting the hardware– Or they may not be sequenced optimally based on the total workload that needs to be accomplished

� This workflow can be implemented via:– Scheduled administrative actions

• Schedule of TYPE=ADMIN– Scripts

• Important structure/sequencing commands:

PARALLEL

» Starts individual threads for each action within a parallel execution block

» Synchronizes progress by awaiting results from each parallel execution task

SERIAL

» Indicates commands should be run serially, one after the other on the server thread

» Used to re-converge or synchronize from a PARALLEL set of operations to a SERIAL (single threaded) set of operations

To Exploit PARALLEL and SERIAL script constructs commands should:

» Use “WAIT=YES” where available

» Also consider using “DURATION=NN” where necessary to better manage time


Illustration of PARALLEL and SERIAL

PARALLEL

SERIAL

Single Command (thread)

Until PARALLEL encountered.

5 Commands run in parallel.

Re-converge to single when

SERIAL keyword encountered.


Example Script

PARALLEL

BACKUP STGPOOL X WAIT=YES

BACKUP STGPOOL Y WAIT=YES

BACKUP STGPOOL Z WAIT=YES

SERIAL

PARALLEL

MIGRATE STGPOOL X HIGHMIG=nn LOWMIG=mm RECLAIM=NO WAIT=YES

MIGRATE STGPOOL Y HIGHMIG=nn LOWMIG=mm RECLAIM=NO WAIT=YES

MIGRATE STGPOOL Z HIGHMIG=nn LOWMIG=mm RECLAIM=NO WAIT=YES

EXPIRE INVENTORY DURATION=qq RESOURCE=nn WAIT=YES

SERIAL

PARALLEL

RECLAIM STGPOOL X THRESHOLD=nn DURATION=qq WAIT=YES

RECLAIM STGPOOL Y THRESHOLD=nn DURATION=qq WAIT=YES

RECLAIM STGPOOL Z THRESHOLD=nn DURATION=qq WAIT=YES

SERIAL

BACKUP DB TYPE=FULL WAIT=YES

BACKUP VOLHIST FILENAMES=/path1/volhist,/path2/volhist,/path3/volhist

BACKUP DEVCONFIG FILENAMES=/path1/dc,/path2/dc,/path3/dc


Script Illustrated

Parallel

Serial

Parallel

Serial

Parallel

Storage Pool Backups (x3)

Migration (x3) and Expiration

Serial

Reclamation (x3)

BACKUP DB, BACKUP VOLHIST,

BACKUP DEVCONFIG


Use Simple Visualization to Validate Your Schedule Optimization

� Using visualization techniques to depict schedules and their relationships one to the other

can be useful

� Helpful to:

– Determine what is running when

– What are the overlaps between schedules?

– When schedules are running, what resources are needed and what load or constraints

does this put on the server?


Administrative Schedules with Overlaps

Administrative Schedule Overlap and Sequencing (Current)

0 0.5 1 1.5 2 2.5 3 3.5

0:00

2:00

4:00

6:00

8:00

10:00

12:00

14:00

16:00

18:00

20:00

22:00

DB Backup

Backup VolHist

Expiration

Reclaim_Copy

Reclaim_Tape

StgPoolBk_Start

Migr_Start

An example of using a spreadsheet and schedule “window” information

to visualize schedule sequencing and overlaps.

Overlap 1 Overlap 2


Administrative Schedules Reconfigured to Reduce Overlap

Administrative Scheduling Overlap and Sequencing (Proposed)

0 0.5 1 1.5 2 2.5

0:00

2:00

4:00

6:00

8:00

10:00

12:00

14:00

16:00

18:00

20:00

22:00

Backup Stgpool

Migration

Expiration

Reclamation

DB Backup

Prepare

Proposed adjustments:

• Eliminate most overlap

• Only remaining overlap is expiration and migration which

generally contend for different resources

Tim

e o

f D

ay


Improve Reliability By Staying Within TSM Operational Limits

� The server operational limit is the point at which

– One or more critical server health and maintenance operations can not be contained within the time available to do it

– Available system resources (CPU, RAM, etc) are exhausted at or prior to peak load being satisfied and supported

� Many different indicators exist that may signal a server is at its operational limits. Depending upon how/why the limit is reached, symptoms may be:

– Degraded performance

– Failed operations

– Or no overt sign and not known until disaster recovery of server or clients is needed


TSM Operational Limits: Database Backup

� Backing up the server database is critical to the health and maintenance of the server

– Required in order to prune space from ARCHIVE log directories

– Needed in order to restore the database and recover from local device failure for database or active log

� TSM 6.1 tests database size to 1 TB, for V6.2 it is tested to 2 TB

– Operational limits due to DB size may be encountered PRIOR to these upper end tested value

– Will vary by customer based on workload, infrastructure, and organizational requirements (RTO)

� DB backup duration should periodically be evaluated to determine if limit is reached and another server is needed – Time needed to backup the database exceeds time available

– Time to restore database is longer than recovery time objectives (RTO) for a disaster recovery (real or simulated)


TSM Operational Limits: Saturation

� TSM is limited to the hardware and resources available to it

� Operational limit may be reached when:

– Server overruns/saturates available CPU on system at peak workload or less then peak workload

– Server overruns available RAM on system and drives high pagefile use

– I/O bandwidth is saturated:• DB or active log performance degraded because I/O can’t keep up• Storage pool actions performance degraded because I/O can’t keep up

– Saturation or overrun of CPU, RAM, I/O bandwidth achieved at or prior to achieving peak workload

– For example, in the lab we’ve demonstrated that more then 1500 concurrent client sessions to the SERVER pushed it to saturation with available memory and CPU such that performance degraded significantly


Process “Rule of Thumb”

� Many server processes can be run explicitly or implicitly in parallel (concurrently).– Example of explicit: Scheduling migration and expiration to run at the same time– Example of implicit: Expiration (V6) provides the “RESOURCE” parameter which indicates the

number of “threads” that will be used to perform the process

� As a general “rule of thumb”, do not specify more concurrent processes than there are CPU cores on the machines

– For example, a 16-way box running expiration and deduplication identify could be configured to run with:

• Expiration RESOURCE= set to a value in the range of 4 to 8• 4-6 Identify processes running

– In this case, if 8 and 6 were used, that would provide 8 expiration threads and 6 identify processes • These 14 tasks would roughly align with 14 of the available 16 CPU cores on the box• This would leave 2 CPU cores for support of other operations and tasks

� This is not an absolute and will vary by processor architecture and system workload. This is a “starting point” for server configuration and how to avoid overrunning the available resources

– Tied to a monitoring strategy where I/O, CPU, and memory are being watched, this will assist with managing an appropriate workload and sequencing on a server


TSM Operational Limits: Possibilities

� Many operational limits are resource or time based

� Taking steps to improve infrastructure may result in faster operations and may mitigate or remedy the operational limit

� For example, if the operational limit is database backup:– Using a faster device for the db backup may eliminate the limit– Improving I/O subsystem and bandwidth for DB and logs may address the issue

� In cases where it is not possible or practical to resolve via improved or changed infrastructure, this may represent a “cap” to the existing server and need to implement and balance workload to another TSM server


Monitoring: What’s Available…

� TSM Provides

– Messages

• Logged to local server activity log

• Event routing to many different targets

– Summary Information

– Accounting Records

– V6 Reporting and Monitoring

– Administration Center

� Third party vendor tools are also available:

– Reporting Tools

– Monitoring Capabilities and Tools


TSM Best Practice Monitoring May Involve More than Simply TSM

� TSM is a large, multi-threaded software application. It exploits or has dependencies on:

– CPU – the application and database perform many calculation/instruction intensive operations

– I/O to Disk:• This relates to the database, active log, and archive log• Bottlenecks such as not enough parallel I/O capability or insufficient bandwidth (small • channels) can affect server performance, scalability, and throughput

– I/O to Storage Hierarchy:• This can be disk (TSM device classes of type DISK or FILE) and sequential media

(Real tape and VTL’s)• Often controllers or other virtualized appliances used for storage devices. (SVC, VTL,

etc)• Devices may be locally attached (SCSI) or fiber attached (SAN)

– Network:• TSM is a client/server application with it’s client operations almost entirely network

driven


Consider Monitoring Items Internal and External to TSM

� External to TSM

– TSM is application software.

– Relies upon:

• Operating system

• Drivers

• Devices

• Firmware

� Internal to TSM

– Client Operations

– Server Processes

– Other

• Server tasks such as memory management, scheduling, etc.


TSM Topology: External to TSM

AIX

(TSM Server)

NIC

HBA

SCSI

Direct Attach Devices (Disk, Tape

LAN/WAN

SAN

Server/Host


Monitoring the Server/Host and Local Devices

� Example: System is AIX

– Most host hardware (NIC, HBA, SCSI, Planar, etc) logs to the system

– “errpt –a” to see device reported errors

� Monitor system resources:

– Tools like topas or others can be used to periodically look at the system and assess health• Topas can be configured to run periodically (AIX 6.1)• Inittab entry: /usr/bin/topasrec -L -s 300 -R 1 -r 6 -o /etc/perf/daily/ -ypersistent=1• “-r 6” indicates how many to keep around• Raw data viewed/formatted using topasout command

– Stats can be collected over time to see historical trends

– Check for filesystems running low on space• In particular /, /opt,

– Monitor paging space• Growth of page file use without corresponding increases in load or activity may indicate a resource

leak and can affect performance or ultimately lead to a server termination



AIX

(TSM Server)

NIC

HBA

SCSI


LAN/WAN

SAN

Server/Host


Monitoring the LAN/WAN

� Usually monitored during exception or periods of degradation

� Network teams/owner typically have monitoring tools in place to:

– Identify and alert to outages

– Identify and alert to degradation

� From TSM perspective:

– Symptoms would be failed client operations due to communication issues. (socket error,

send error, receive error)

– Not usually evaluated or investigated unless issues are occurring



AIX

(TSM Server)

NIC

HBA

SCSI


LAN/WAN

SAN

Server/Host


Monitoring for a SAN and SAN attached devices

� Errors not centrally logged.

– Usually logged to specific device that is generating the event

– Often times individual error logs need to be accessed and evaluated

� Errors can be anywhere in the “chain” from the HBA to the device

– Fiber Channel Switch, router gateway– Library– Drive– Disk Array– SAN controller (SVC or equivalent)

� Virtualization can hide/mask errors

– VTL, SAN Controller, etc are systems unto themselves running:• Embedded host, OS, drivers, devices, etc.

– Evaluation of health may require vendor involvement as the relationship between logs, devices, and errors or symptoms may not be surfaced to end-user


TSM Monitoring: Client Operations

� Client Operations:

– TSM clients (backup/archive, TDP, API, etc) are the core end-user representing the data being protected by TSM

– Operations may be:• Scheduled• Manually initiated• Automatically initiated (such as archiving log files from DB2/UDB client)

– Monitor for:• Failed sessions or schedules• Schedule issues such as missed or failed schedule events• Unusual or unexpected session termination• Clients log ANExxxx messages to the server

If the client encounters a local error (error on the client system) while an operation is in-flight, many of these will be reported and logged to the server

� Client operations record summary information (SELECT * FROM SUMMARY) as well as logging messages to the server activity log


TSM Monitoring: Server Processes

� Server Processes:

– Server maintenance tasks such as Expiration, Storage Pool Backup, Migration, Reclamation and such are done as server processes

– Many processes can be WAIT=YES (synchronous) or WAIT=NO (asynchronous)

– Server processes ALL issue process started and ended messages • End messages report statistics as applicable• End messages report SUCCESS or FAILURE of an operation

– Monitor for:• Failed processes

Cancelled processes will report as failed, there should be other messages logged to activity log indicating the cancellation

If processes are not succeeding, evaluate:» Was this an appropriate time/reason for process to be run?» If an “insufficient resource” issue, can additional resources be made available? Or can the

process be initiated at a different time when resources are available?

� Server processes record summary information (SELECT * FROM SUMMARY) as well as logging messages to the server activity log


TSM Monitoring: Analysis Using Message Tokens

� Activity log messages are tagged with session and process tokens.

– Session token example:ANR1234E xxxxx (SESSION: 985)

– Process token example: ANR2345E xxxx (PROCESS: 38291)

– Session and process token example:ANR3456E xxxx (SESSION: 765 PROCESS: 998512)

� Use:

– If a process or session encounters an error, query the activity log using the session or process number to see all the messages for that action

– Other messages before and after the failure may be found easily in this fashion.• For example: QUERY ACTLOG SEARCH=‘(SESSION: 985)’ to search for all the messages for

session 985

• More details about the events leading up to or the actual error itself may be seen by doing this


Conclusion

� Server Workflow Priorities:

– Protect Client Data

– Maintain the Server

– Protect the Server

� Priorities then provide sequencing of actions which can

– Be orchestrated via scheduled (type=admin) scripts

– Scripts structured using PARALLEL and SERIAL semantics to sequence actions and

manage resources while satisfying the workflow priority actions

� Operational limits

– Have been defined

– Steps to identify and possible actions have been discussed

� Monitoring considerations have been discussed for:

– Server topology

– And the server itself


A few useful links…

� V6 Deployment best practices:

http://www-01.ibm.com/support/docview.wss?uid=swg21421060

� Database Reorganization:


� Memory requirements for V6:


� TSM configured for HADR:

http://www.ibm.com/developerworks/wikis/display/tivolistoragemanager/Electronic+v

aulting+using+deduplicated+remote+copy+storage+pools

1 Tsm Admin Best Practices

Documents

highmig nn lowmig mm reclaim

threshold nn duration qq wait

direct attach devices

tsm operational limits

server activity log

reclaim stgpool

backup volhist

administration center