Hosted by Making Reliable and Restorable Backups Presented by: W. Curtis Preston President The Storage Group, Inc.
Hosted by
Making Reliable and Restorable Backups
Presented by:W. Curtis PrestonPresidentThe Storage Group, Inc.
Hosted by
Making good on your investment
Many SANs are built in order to simplify
backup, yet often fail for lack of good
design, processes and procedures.
There are several common mistakes that
people make when building a backup system
Avoiding these mistakes and taking proper
action, can create a backup system that is
reliable and restorable
Hosted by
What will we cover?
Common Backup Configuration
Mistakes
How to Avoid Them
• Sizing your backup system
• Configuration examples for NetBackup
• Configuration examples for NetWorker
Hosted by
Common Backup Configuration Mistakes
Hosted by
Where do these lessons come from?
Audits of real backup and recovery
systems
Lessons learned from real horror
stories
Many, many sleepless nights
Hosted by
Too little power
Not enough tape drives
Tape drives that aren’t
fast enough
Not enough slots in the
tape library
Not enough bandwidth
to the server
Hosted by
Too much powerStreaming tape drives must be
streamed
If you don’t, you will wear out
your tape drives and decrease
aggregate performance
Must match the speed of the pipe
to the speed of the tape
You can actually increase your
throughput by using fewer tape
drives
Hosted by
Not using multiplexing Defined: Sending multiple
backup jobs to the same drive simultaneously
Again, drives must be streamed
Multiplexing will impact restore performance, but not as much as you might think
Multiplexing can actually help your restore just as it can help your backups
Using multiplexing can greatly increase the utilization of your backup hardware
Tape drive
Backup Job
Backup Job
Backup Job
Backup Job
Hosted by
Not using multistreaming
Defined: Sending
multiple simultaneous
backup jobs from a
single client
Large systems cannot
be backed up serially
Multistreaming creates
a different job for each
filesystem
Backup Job
Backup Job
Backup Job
Backup Job
Server
Hosted by
Using include lists
Most major backup
software supports file
system discovery
Still, many administrators
use manually created
include lists
Any perceived value is
significantly outweighed
by the risk it creates
Hosted by
Too many full backups
If you are using a commercial backup and
recovery product with automated media
management and multiple levels, weekly full
backups are a waste of tape, time, and money
Monthly full backups, weekly cumulative
incrementals (1), and daily incrementals (9)
work just as well and use ¼ as much tape
Depending on the level of incremental activity,
quarterly backups can work just as well.
Hosted by
Not standardizingCreating custom configurations for each
client is easier, but much riskier
Creating a standard backup client
configuration can significantly decrease risk
Create a standard exclude list, etc. and push
it out to each client
Hosted by
Not even noticing!
Backups go ignored so
often. It’s like they’re
the bill collector nobody
wants to talk to
Backup reporting
products can really help
automate easy reporting
Don’t ignore backups.
They will bite you.
Hosted by
It’s just backups, right?
“I’m an experienced, seasoned systems administrator. This is just backups. How hard can they be?”
The data being backed up has become very complex, and the complexity of backup systems have matched that complexity with functionality – that also happens to be complex
Hosted by
Not thinking about diskTape is not as cheap as you
thought
Let’s examine a 4 TB library
20 slots, 2 drives $17K
20 tapes, $70 apiece $14K
Robotic license $10K
Total $41K
(does not include labor costs)
That’s about $10/GB
LIBRARY STORAGE MODULE
9710
StorageTek
EXIT
ENTER MENU
*
Hosted by
Disk is cheaper than you thoughtATA-based storage arrays as low as $5/GB
(disk only, needs filesystem)
Special function arrays• Quantum DX-30 looks and behaves like a Quantum P1000.
Can be used as target for “tape-based” backups (3 usable TB, $55K list, or $18/GB)
• NetApp R100 looks like other NetApp filer. Target for SnapVault and disk-based backups, source for SnapMirror (9+ usable TB, $175K list, or $18/GB)
ATA disks not suited for heavy, random access, but perfect for large block I/O (e.g. backups!)
Hosted by
You can do neat things with disk Incremental backups are
one of the greatest backup
performance challenges
Use as a target for all
incremental backups. (Full,
too, if you can afford it)
For off-site storage,
duplicate all disk-based
backups to tape
Leave disk-based backups
on disk
Hosted by
Now that I know…
Building a reliable and restorable backup system
Hosted by
Sizing the backup system
Hosted by
Server Size/Power
I/O performance more important than CPU
power
CPU, memory, I/O expandability paramount
Avoid overbuying by testing prospective
server under load
If you use Suns, you’ve got snoop and truss
Hosted by
Catalog/database Size Determine number of files (n)
Determine number of days in cycle (d)
(A cycle is a full backup and its associated incremental backups.)
Determine daily incremental size (i = n * .02)
Determine number of cycles on-line (c)
150-250 bytes per file, per backup
Use a 1.5 multiplier for growth and error
Index Size = (n + (i*d)) * c * 250 * 1.5
Hosted by
Library Size - drives
Network Backup• Buy twice as many backup drives as
your network will support
• Use only as many drives as the network
will support (You will get more with
less.)
• Use the other half of the drives for
duplicating
Hosted by
Library Size - drives
Local Backup• Most large servers have enough I/O bandwidth
to back themselves up within a reasonable time if you’re using NetBackup
• Usually a simple matter of mathematics: 8 hr window, 8 TBs = 1 TB/hr = 277 MB/s 30 10 Mb/s drives, 15 20 MB/s drives
• Must have sufficient bandwidth to tape drives• Filesystem vs. raw recoveries
Allow drives and time for duplicating!
Hosted by
Library Size - slots (all tape environment)
Should hold all onsite tapes
On-site tapes automatically expire and get
reused
Only offsite tapes require phys. mgmt.
Should monitor library via a script to
ensure that each pool has enough free
tapes before you go home
Watch for those downed drive messages
Hosted by
Library Size - slots (disk/tape environment) Do incremental backups to disk
Library only needs to hold on-site full tapes and
the latest set of copies.
On-site tapes and disk-based backups
automatically expire and get reused
Only offsite tapes require phys. mgmt.
Should monitor library and disk via a script to
ensure that each pool has enough free tapes
before you go home
Watch for those downed drive messages
Hosted by
Local or Remote Backup?
Throughput (in 8hrs), if you “own the wire:”• 10 Mb = 20 GB, 100 Mb = 200 GB
• GbE = 500 GB – 1 TB (Also must “own the box.”)
Greater than 500 GB should be “local”• Lan-free backups allow you to share a large tape
library by performing “local” backups to a “remote, shared” device
• More than one 500+ GBserver, buy a SAN!
• Only one 500+ GB server, plan for a SAN!
(NetBackup= SSO, NetWorker=DDS)
Hosted by
Multistreaming - NetBackupDefined: Starting multiple simultaneous backup jobs from a
single client
• Maximum jobs per client > 1
• Check “Allow multiple data streams”
• ALL_LOCAL_DRIVES, or multiple entries in file list
• Maximum jobs per policy > 1 or unchecked
• Need storage unit with more than one drive, or one drive
with multiplexing enabled
• Can change max jobs per client using the Server
Properties -> Clients tab (4.5)
• By default, will not exceed one job per filesystem, but can
bypass this if you make your own file list
Hosted by
Multistreaming (Parallelism) - NetWorker
Use “All” saveset or multiple entries in the saveset list
Set the parallelism setting for server and, if necessary, the storage node
Set client parallelism value in client attributes
Must have multiple drives available, or one drive with target sessions set higher than one
Will not exceed number of disks or logical volumes on the client (see maximum-sessions in manual)
Hosted by
Multiplexing – NetWorker
Set target sessions
per device, allocating
how many sessions
may be sent to that
device.
Global setting for all
backups that go to
that device
Hosted by
Multiplexing - NetBackup
• Max multiplexing per drive in storage unit
configuration > 1
• Media multiplexing in schedule > 1 Use higher multiplexing for incremental
backups if going to tape (6-8) Use lower multiplexing for local backups (2) No need to multiplex disk storage units
• Multiple policies can multiplex to the same
drive,
but multiple media servers cannot
Hosted by
Using Include lists -- notNetBackup –
ALL_LOCAL_DRIVES in file
list
NetWorker – All in
saveset field
Automatically excludes
NFS/CIFS drives
Does not include
dynamically mounted
drives not in /etc/*fstab
Hosted by
What about database clients?
Use scripts that parse lists of databases:
• /var/opt/oracle/oratab for Oracle
• MS-SQL list in registry
• Master database in Sybase
Some backup products support “All” for
databases
Remember to write standardize script with
parameters to backup databases.
Hosted by
Incremental backups - NetBackupCreate staggered monthly full backups using
calendar-based scheduling
Create staggered weekly cumulative incrementals using CBS
Create daily incremental backups using frequency based backups
(Check Allow after run day.)
Delete window from previous day for CBS
Hosted by
Incremental backups - NetWorker
Do not use the Default schedule!
Create 28 schedules with a monthly full, weekly level 1, and daily incremental, name them after the full day
Do not specify a schedule for the Group
Assign the 28 schedules evenly across all clients based on size
Hosted by
Standardization – NetWorker
Use All saveset
entry
To exclude files,
use standard
directives for all
clients
Hosted by
Standardization - NetBackupUse ALL_LOCAL_DRIVES
Non-Windows clients -
Use standard exclude list
and push out from
master using bpgp
Windows clients – Use
standard exclude list and
push out from master
using bpgetconfig –M and
bpsetconfig –h
Hosted by
Backup Reporting - NetBackupWatch activity and device
monitors
bperror
bpdbjobs -report
bpdbjobs –report –all_columns
/usr/openv/netbackup/logs
/usr/openv/logs
/usr/openv/volmgr/logs
Hosted by
Backup Reporting – NetWorker
Watch nwadmin screens
mminfo
nsrinfo
mmlocate
nsrmm
/nsr/logs
Hosted by
Disk-to-disk Backup - NetWorker
If using regular disk, use file type device
Disk backup extra cost with options
If using virtual tape library, treat it like a
tape library
Use cloning to duplicate disk-based
backups to tape and send them off-site
Hosted by
Disk-to-disk Backup - NetBackup
If using regular disk, use disk-based storage unit
(No extra cost for disk storage units!)
If using virtual tape library, treat it like a tape library
Use vault to duplicate disk-based backups to tape and send them off-site
Hosted by
What about my SAN and NAS?
Hosted by
SAN: LAN-free, Client-free, and Server-free backupNAS: NDMP filer to self, filer to filer, filer to server, & server to filer
Eth
ern
et
Data General
BackupServer
IBM
Backup Client
IBM
Backup Client
FC
FC
Router
Library
SCSISCSI
Disk Array
NAS Server
FCFC
FC Switchor Hub
FC
NAS Server
NAS Server
LibraryFC Switch
or Hub
FC
SCSI
Library
FC
NASSAN SAN
Virtual tape
Hosted by
LAN-free backups How does this work?
• SCSI Reserve/Release
• Third-party queuing
system
Levels of drive sharing
Restores
IBM
IBM
Disk
Disk
IBM
Disk
TapeLibrary
SCSI/FCRouter
SCSI
SCSI
SCSI
SCSI
FC
FC Switch
FC
FC
FC
Hosted by
How client-free backups work
Transaction Logs
TapeLibrary
DataSrvr
BackUpSrvr
B
primarydisk set
backupmirror
1
3
2a
TapeLibrary
BackUpSrvr
A2c
2b
LAN
TapeLibrary
DataSrvr
BackUpSrvr
primarydisk set
backupmirror
Transaction Logs
2
1Tape
Library
BackUpSrvr
A
LAN
Backup transaction logs to disk
Establish backup mirror
TapeLibrary
DataSrvr
BackUpSrvr
primarydisk set
backupmirror
4a 4b
Transaction Logs
1
2
3Tape
Library
BackUpSrvr
A
LAN
Split backup mirror and back it up
Hosted by
How client-free recoveries work
Restore backup Restore backup mirror from tape from tape
Restore primary mirror Restore primary mirror from backup mirror backup mirror
Replay transaction logs from disk
TapeLibrary
DataSrvr
BackUpSrvr
primarydisk set
backupmirror
Transaction Logs
1b
2c
TapeLibrary
BackUpSrvr
A
LAN
2b
2a
1a
TapeLibrary
DataSrvr
BackUpSrvr
Transaction Logs
1
B A
4
3
25Tape
Library
BackUpSrvr
A
LAN
TapeLibrary
DataSrvr
BackUpSrvr
primarydisk set
backupmirror
Transaction Logs
TapeLibrary
BackUpSrvr
A
LAN
Hosted by
Server-free backups
Server directs client to take a copy-on-write snapshot
Client and server record block and file associations
Server sends XCOPY request to SAN
TapeLibrary
DataSrvr
SANw/xcopysupport
primarydisk set
backupmirror
orsnapshot
3
Transaction Logs
1
2
TapeLibrary
BackUpSrvr
A
LAN
Virtual DiskProvided byDisk Array
Block D
Block EBlock F
FileB
Block A
Block B
Block C
FileA
Hosted by
Server-less Restores
Changing block locations
Image levelrestores
File levelrestores
TapeLibrary
DataSrvr
SANw/xcopysupport
primarydisk set
backupmirror
orsnapshot
2
Transaction Logs
1
TapeLibrary
BackUpSrvr
A
LAN
3
4
Virtual DiskProvided byDisk Array
Block D
Block EBlock F
FileB
Block A
Block B
Block C
FileA
Tape
Snapshotor
Mirror
Block D
Block EBlock F
Block A
Block B
Block C
Virtual DiskProvided byDisk Array
Block D
Block E Block F
FileB (deleted)
Block A
Block B
Block C
FileA
Snapshotor
Mirror
Block A
Block B
Block C
Block D
Block EBlock F
Tape
Backup Restore
Hosted by
NDMP Configurations
Filer to
self
Filer to
filer
Filer to
server
Server to
filer
LAN
Filer Filer Filer
BackupServer
Tape library Tape libraryTape library
OtherServer
Server to Filer
Filerto
Self
Filer to Filer
NDMP tapelibrary
Filerto
libraryFiler toServer
Hosted by
Using NDMP
Level of functionality depends on the
DMA you choose• Robotic Support
• Filer to Library Support
• Filer to Server Support
• Direct access restore support
Hosted by
Resources
Hosted by
Resources
Directories of products to help you
make a better backup system
http://www.storagemountain.com
Send questions to: