Disk Mirror and Recovery Tool, System Administration Guide
DOCPROPERTY "Information" \* MERGEFORMAT
Ericsson InternalSYSTEM ADM. GUIDE6 (128)
Prepared (also subject responsible if other)No.
REENIND1/1543-APR 901 0261 Uen
ApprovedCheckedDateRevReference
EEM/GO [Niklas Nordlund] DOCPROPERTY "Checked" \* MERGEFORMAT
2013-01-21AE DOCPROPERTY "Reference" \* MERGEFORMAT
Disk Mirror and Recovery Tool, System Administration GuideSparc
and x86
Copyright Ericsson AB 2011 - All Rights Reserved
Disclaimer
The contents of this document are subject to revision without
notice due to continued progress in methodology, design, and
manufacturing.
Ericsson shall have no liability for any error or damages of any
kind resulting from the use of this document.
Contents31Introduction
62Characteristics
83Terminology
124Installation
165Document Structure
186Mirror Definition
237Procedure Manual
318Tape Stations
379Internal Structure
4310NAS handling
5911New Dump Features
6212Synchronize Mirror
6413Detach Mirror (sub-menu)
7014Dump & Restore Filesystems (sub-menu)
10115Change boot-device
10316Volume Manipulations (sub-menu)
10717Root Disk Manipulations (sub-menu)
11018HW Changes (sub-menu)
11519HA Cluster Menu (sub-menu)
11920System Status (sub-menu)
12321System Check (sub-menu)
12522Miscellaneous
12823Glossary
12824References
1 Introduction
With the introduction of Blades, x86 and Unified storage (SAN
and NAS) in OSS-RC O11.2, DMR now supports two architectures. This
document is also meant for both Sparc and x86 architectures.
Currently most NAS related issues are contained in chapter 10 NAS
handling. X86 issues are detailed where they pertain. New backup
features are described in chapter 11.DMR, Disk Mirror &
Recovery Tool, is a product for managing the storage of root and
volume data, via Solaris Volume Manager and Veritas Volume Manager,
including functions like Dump&Restore, Detaching and Synching
Mirrors, and maintenance. This chapter explains the main usage of
DMR for each type of Ericsson installation.
1.1 Installation and Upgrade
Here are examples of operations performed with DMR in some
common OSS-RC procedures.
1.1.1 Small Upgrade
Dump system to tape so if the upgrade fails, the old system can
be restored
Detach Mirrors before small upgrade so that a roll back is
possible
Shrink/Expand volumes and swap space if required
1.1.2 New Hardware and Spare Wheel Large Upgrade
Dump system to tape so if the upgrade fails, the old system can
be restored
Dump live system to tape so it can be restored on a sparewheel,
after upgrading on sparewheel, the sparewheel system is dumped to
tape and restored to the offline mirror of live server
Mirrors are detached before large upgrade so that a roll back is
possible
Shrink/Expand volumes and swap space if required
Boot Other - is used when a rollback is needed on a live server
and the detached side has a different Solaris version installed on
it
1.1.3 Split domain Large Upgrade
Dump system to tape so if the upgrade fails, the old system can
be restored
Split System - is used when a mirror needs to be moved to
another host
Recover Versant - Versant databases need to be recovered after a
split domain upgrade. They are frozen before a split of the
mirrors
Remake Root Disk - Before the new root disk can boot, it will
need a new device tree.
1.2 Mirroring
Examples of DMR operation used in normal OSS-RC procedures.
1.2.1 Synchronize Mirrors
Used to synchronize two mirrors. I.e. to increase the redundancy
of the active volumes.
1.2.2 Detach Mirror
Used to detach a mirror so it is possible to rollback. I.e. to
freeze a copy of all data.Uses the same format as Dump to Tape
does.
1.2.3 Boot from Detached Mirror
Used to Rollback to the frozen copy.Only compatible with Detach
Mirror.
1.2.4 Split System
Another way of freezing a copy of all data. Used e.g. when a
mirror needs to be moved to another hostUses the same format as
Restore on a Mirror.
1.2.5 Boot Other
Used to activate another mirror. That mirror can be a frozen
copy of the same system, or another system created via a restore
from tape.Only compatible with Split System and Restore on a
Mirror.
1.3 Backup and Restore
These operations are also used by the End User, even though OMBS
is the officially recommended tool for Backup of OSS-RC.
1.3.1 Dump to tape
Data on all volumes can be saved to tape.
1.3.2 Restore on a Mirror
A system saved on a dump tape can be restored on one of the
mirrors, while another mirror is active.
2 Characteristics
DMR is a tool to manage the disks as mirrors, which is not a
clear definition in the system. It is a tool based on Veritas
Volume Manager (VxVM) and Solaris Volume Manager (SVM).
SVM is used for root, VxVM for local data and NAS API for
Network Attached Storage.
What DMR requires is Solaris, VxVM, and SVM. DMR also has
knowledge about:
Sybase and Versant
Some OSS-RC directories
The HA solution NAS APIApart from that, its fully generic, no
dependencies to any specific OSS.
The file /ericsson/config/system.ini is never used. It is used
to configure the OSS-RC system in general. But the disk definition
section is only used during Jumpstart. I.e. even though it defines
the disk mirrors, DMR does not consult it.
However the file /ericsson/config/storage.ini is used via the
NAS API. And /ericsson/config/cluster.ini is updated sometimes,
which is the base for creating main.cf.
2.1 Limitations
DMR has few limitations, but they can also be regarded as
recommended default behavior, i.e. the way the OSS has decided to
use the server.
Only SVM is supported on root.
Only VxVM is supported on data.
The root disk must have, as OSS-RC, certain meta mirrors. (d10,
d20, d30, [d70])
Supports Symantec/Veritas Storage Foundation until 5.1.
Only the blade with ossdg imported will handle the NAS.
2.2 Prerequisites
It is assumed that the user of this document is familiar with
UNIX and has knowledge of VxVM Volume Manager.
3 Terminology
Here are DMR, VXVM and SVM terms shortly explained.
3.1 DMR Terms
3.1.1 Mirror
In DMR the term Mirror is used. There is no master and mirror
side. All are equal copies of data. On volume level a copy is
called a plex, but on system level there is no term, nor
definition; what is a Mirror side?. This is what DMR does,
maintaining the complete mirror definition, which is not a concept
in VxVM, nor SVM, and certainly not for them together.
3.1.2 Detached Mirror
The term Detach is used when a synchronized plex is put offline
under a new and temporary volume (eg export_m2), inside the same
disk group (eg ossdg).
A mirror can be Detached either by Detach Mirror, or when doing
Dump and letting the frozen mirror stay Detached also after the
dump has terminated.
3.1.3 Split System
This is another way of freezing a whole Mirror side, but the
format is very different from Detached.
The active Disk Groups are split in two halves, creating new
disk groups (eg ossdg_m2), having the same volumes (eg export)
inside.
This format can be achieved by Split System or doing Restore a
Dump on a Mirror. Split System requires the license DGSJ included
in FlashSnap or the Enterprise licenses.
Since SVM is now used on root, the format for the root disk is
very similar in the two types of freezing a mirror.
3.1.4 Activating a Frozen Mirror
Depending on the format of the Mirrors, i.e. how the frozen
mirror was originally created, the correct procedure must be
chosen:
Boot Detached Mirror
Boot Other (for Split/Restore)
3.2 VxVM terms
Veritas Volume Manager is used to mirror the data disks.
3.2.1 Disk Group
A Disk Group is one or more disks grouped together. When a DG is
imported, all disks in that group are imported as well. The
information about a DG is stored redundant on all disks in the
group, in their respective private region.
The main objects in a DG are volumes, which store data on the
disks.
3.2.2 Volume
A Volume can be compared to a partition. Its a block of data,
with limited space. That space can be used as a raw device (swap)
or as a block device (file system).
The main objects in a volume are plexes, which contain the
actual data.
3.2.3 Plex
A Plex is normally a full copy of the volume. If the volume has
several plexes, its mirrored.
A plex consists of subdisks.
3.2.4 Subdisk
A subdisk is a fraction of a disk. The plexes point to subdisks,
which contain actual data.
3.2.5 Dirty Region Log
DRL is a Data Change Map (DCM) containing changed blocks in a
volume. A block is updated on one mirror only to start out with,
and if the power to the system fails, the system can use this map
to synchronize the mirrors rapidly during boot, copying only the
changed blocks.
DRL doesnt offer higher data redundancy, only faster re-sync
during boot. They do not make sense if the volume is not
mirrored.
This DCM map is stored in a plex, under the volume, next to the
data plexes.
3.2.6 Disk Access Name
The DA name is the physical name of the disk. The Solaris name,
which is found under /dev/dsk or /dev/vx/dmp.
The name has slice 2 (s2) at the end, unless the disk has an EFI
label, which is without the slice.
The name without the slice at the end is called Device tag, i.e.
what you find in format.
If Enclosure Based Names are enabled, the DA names are changed
to something like Disk_0 or SUN35100_1.
3.2.7 Disk Media Name
The DM name is an arbitrary name, a name that contains some kind
of information to the reader.
In OSS-RC and DMR the DM names follows a naming convention. For
each Mirror, these names are used:
1 -, disk2, disk3,...
-> here called: diskn2 -, disk2mirr, disk3mirr,... -> here
called: disknmirr3 -, disk2mirr3, disk3mirr3,... -> here called:
disknmirr3There is no Disk Media name for root.
3.2.8 Examples
Here is a list of some VxVM objects with name and example.
AcronymObject
ExampledgDiskgroup
ossdg vVolume
export plPlex
export-01sdSubdisk
disk2-01daDisk Access namec0t0d0s2dmDisk Media Namedisk2
3.3 SVM Terms
Solaris Volume Manager is used to mirror the root disk.
3.3.1 Meta Mirror
A Meta Mirror is the top meta device used to access the data. It
can be compared to a volume in VxVM.
The naming convention in OSS-RC for the meta mirror is a d
followed by a number and terminated with a 0 (zero).
On the root disk there are several slices to mirror:
Meta devSliceUsaged100rootd201swapd303dump device-5meta
DBd707alternative root
Device path: /dev/md/dsk/d10
3.3.2 Sub Mirror
The Sub Mirrors are meta devices below the meta mirror. Each sub
mirror contains a full copy of the meta mirror data. It can be
compared to a plex in VxVM.
The naming convention is to change the zero to the number of the
mirror is belongs to.
E.g. meta mirror d10 has sub mirrors d11 and d12.
Check status with:
# metastat c
Sub Mirrors cannot change name, so the Mirror definition has to
be changed if for example d12 is defined for Mirror 1.
3.3.3 Meta Device State Database
The SVM configuration is stored in several small databases,
spread over all disks used by SVM.
In OSS-RC there are 3 replicas per disk. All three are stored in
slice 5.
Check status of all replicas with:
# metadb [-i]4 Installation
4.1 Package installation
The DMR installation will only extract DMR on /ericsson:
# cd /var/tmp# gzcat 19089-CXC1724636_.tar.gz | tar xvf -# gzcat
19089-CXC1724636_.tar.gz | tar xvf -
# ist_run -d ERICdmr-.pkg pkgonly
If ist_run is not available do:
# pkginfo | grep dmr
# pkgrm ERICdmr
# pkgadd -d /ERICdmr.pkg all
# pkginfo -l ERICdmr
# cd /ericsson/dmr/bin
# ./dmtool
After it has been extracted, it should be executed once, to
define the existing disks as separate mirrors.
The pkgrm step above is optional. Only if you want to maintain
the original package name installed, without any .2 postfix.
4.2 Directory Structure
/ericsson/dmr/bin/dmtool (main tool)
.../bin/mtx {.sparc|.i386}
.../bin/run_rc.sh
.../bin/test_blk.sh
.../bin/setup_ssh.sh.../etc/DMR_Alarms.txt
.../etc/MTX_README
.../etc/README
.../etc/mtx-1.3.12.tar.gz
.../etc/dm_help.txt
.../etc/tape.def
.../etc/boot_aliases.template
.../etc/cxc.env_template.../etc/dmr1.xml
.../etc/dmr2.xml
.../etc/dmr3.xml
.../etc/dm_define.../etc/cxc.env
.../etc/disk_serial_no
.../etc/boot_aliases
.../etc/dm_hotspare
.../etc/dump_set
.../etc/remote_host
.../etc/remote_user
.../etc/remote_port
.../etc/remote_fs
.../etc/tape_host
.../etc/last_copy_dir
.../log/dm_screen_.log
.../log/dm_cmd_.log
.../log/dm_cmd.log
.../log/dm_screen.log.../log/dm_activity.log
.../log/dump_history.log
.../log/dm_dump__.log
.../log/freeze_versant_.log.../log/recover_versant.log.../log/recovery/*
.../log/vxpriv/*
.../log/cXtYdZs2/*
.../log/cp.status
The last files (in italic) are created during execution of
dmr
4.2.1 Setup PW free SSH login
For details about how to setup PW free SSH login, see chapter
11.5.4.3 CLI usage
DMR can be called from other scripts or from the command line,
without passing through all menus interactively.
4.3.1 Calling menu options
You can execute the menu items of DMR by adding it as a
parameter to dmtool. It can be any option from the menu, like sy,
c, m, b. Or the actual name of the procedure.
E.g.:
# dmtool sySynchronize mirrors
# dmtool bChange boot-device (also in cdrom mode)
# dmtool d dDump Filesystems to Tape
# dmtool r rRestore a Dump on a Mirror
# dmtool m 1Add new VX Licenses
# dmtool boot_otherBoot Other
See each sub chapter for details on possible parameters in CLI
mode.
The error code should be returned by dmtool as any normal
call.
4.3.2 Calling other non-menu procedures
Any procedure in DMR can be called, but some requires variable
to be set, so its not suitable. But those who dont, can are readily
accessible in the same was as explained above.
Check each procedure in dmtool for additional params.
E.g.:
# dmtool s remake_root c0t0d0s0 samehw
# dmtool s split_system 1 data4.3.3 FlagsSome additional flags
can be added to change the behavior of DMR. They should be located
before any procedure.
-sSilent. No questions. If a Question has a default value, then
its taken. If a question doesnt, the script exists with an
error.
-nNo info. To reduce information printed, like help text. -xTo
enable Stepping. DMR will stop at each major command and prompt the
user for directions. Enter e'x'ecute, 'e'dit, 's'kip, 'n'o-step,
'q'uit or 'h'elp$DMR_STEP_CMD=y can also be set in
cxc.env.$DMR_STEP_CMD_START= for which cmd to start
stepping$DMR_STEP_CMD_ONLY= for which cmd to stop for. -mTo avoid
the check at startup if the mirrors have been defined.(dm_define
file) To enable access to internal procedures early, before mirrors
have been defined.
-dTo force DMR not to enter CDROM-mode, to make sure the menus
are available as standard. 4.4 Hints
4.4.1 Command Log File
To monitor commands used by dmtool:
# tail -f /ericsson/dmr/log/dm_cmd.log
4.4.2 Shortcut
After the first run of dmtool, you will have a link from root to
the bin directory
# cd /dmr
# ./dmtool
5 Document Structure
The following chapters describe each DMR function in the order
in which they appear in the menu structure of the tool.
There is a separate chapter for the menu item Mirror Definition
as it is essential that this procedure is followed correctly.
There is also a chapter like a Procedure Manual, which describes
the steps how to perform some common procedures. (procedures that
require the use of more than one DMR menu option)
Most chapters follow the same format: first a brief Description
of what the option does, then the sub-chapter Silent/CLI examples
which shows possible in-parameters, and some have the sub-chapter
Technical Detail, which contains a more technical description.
5.1 Main Menu
# /ericsson/dmr/bin/dmtool
|=----------------------------------------------------------------------=||
DISK MIRROR & RECOVERY TOOL
||=----------------------------------------------------------------------=||
|| sy Synchronize Mirror || de Detach Mirror ... || d|r Dump &
Restore ... || b Change boot-device || v Volume Manipulations ...
|| ro Root Disk Manipulations ... || h HW Changes ... || ha HA
Cluster Menu ... || || s System Status ... || c System Check ... ||
m Miscellaneous ... ||
||=----------------------------------------------------------------------=||
COMMAND GUIDE Ericsson /// || Character(s) for option || Procedure
name || help Help text || q Quit ver: R1B01 2007-10-09
||=----------------------------------------------------------------------=|
5.2 Menu in CDROM mode
When the server is booted from a DVD, VxVM is not available and
most DMR menu items are not applicable. What is possible to do is
presented in a small menu like this:
|=----------------------------------------------------------------------=||
DISK MIRROR & RECOVERY TOOL - CDROM MODE -
||=----------------------------------------------------------------------=||
|| r Restore root || m Re-Make root disk || b Change boot-device ||
d Rebuild /devices || p Prepare for SVM || s Remove SVM on root ||
||=----------------------------------------------------------------------=||
COMMAND GUIDE Ericsson /// || Character for option || h Help || q
Quit ver: R1B01 2007-10-09
||=----------------------------------------------------------------------=|
A description of the menu items above can be found in different
chapters in this document.
Note: The term CDROM is here used for any kind of Mini-root,
booted from DVD or from the NW. Mini-root is installed/booted into
memory, and does not require any disk. Its a limited single user
environment.
6 Mirror Definition
6.1 General
At start-up the first time, the disks have to be organized into
desired mirrors. If the mirror definition cannot be found, the tool
will ask the user to do the definition.
Under sub-menu HW Changes, you find Redefine Mirrors, which can
be used to re-define the mirrors, if the first time went wrong.
Some data disks can be defined as hotspare disks. This is not
the default configuration, we use hotrelocate instead. Disks to be
used for hotspare can be placed in the file dm_hotspare, see below
in Technical Detail.
6.2 Technical Detail
There are no (altering) VxVM commands used when defining the
mirrors.
The mirror definition is saved in /ericsson/dmr/etc/dm_define
and /ericsson/dmr/etc/dm_hotspare.
The format of dm_define:Line 1; type of definition, i.e. the
word disk (bus is not used anymore)Line 2; mirror 1, all disks as a
list (the first disk must be the rootdisk)Line 3; mirror 2, as
above
Example: (c1t0 and c4t3 are root disks)disksc1t0d0s2 c1t1d0s2
c1t2d0s2c4t3d0s2 c2t1d0s2 c2t2d0s2
Hotrelocate is the default option and uses the following
procedure: Space is searched for on all disks when there is a disk
crash. The subdisks on a crashed disk, are then spread out over the
rest of the disks in the diskgroup, which might be inconvenient,
but on the other hand, all disk are used as columns/spindles for
striping the volumes (performance gain). If hotspare is preferred,
make sure the file /etc/init.d/vxvm-recover has commented out the
vxrelocd, and uncommented the vxsparecheck daemon.
6.3 Examples6.3.1 Automatic definition (dmtool suggests)
When dmtool is started for the first time it asks the user to
define the Mirrors. Below is an example where the suggestion from
the tool is exactly how the user wants it, and can just accept
it.
----------------------------< Define mirrors
>----------------------------
NO MIRROR DEFINITION FOUND!
PLEASE PAY ATTENTION! THE DEFINTION WILL BE USED FROM NOW
ON.
NOTE: There is no Master or Main side. All sides are Mirror
sides.
Having 2 sets of disks (copies of all data), define 2
Mirrors!
The definition will be saved in /ericsson/dmr/etc/dm_define,
where it can be checked (and modified).
How many mirrors should be defined (1-n) [2]:
===> Trying to make a suggestion
-> Checking disks
----------------------< Print Mirror Configuration
>----------------------
Intro:
All disks for each mirror will be displayed, with both disk
access name
and disk media name. The first disk in each mirror is always
considered
root disk by DMR.
Mirror 1 Mirror 2
------------ ------------ ------------ ------------
d11 c1t0d0s2 d12 c1t1d0s2
disk2 c1t2d0s2 disk2mirr c1t4d0s2
disk3 c1t3d0s2 disk3mirr c1t5d0s2
Is this a good mirror definition (y/n)? [y]
===> Mirror Status
Mirror 1: diskn RUNNING d10 c1t0d0s2 c1t2d0s2...
Mirror 2: disknmirr RUNNING d10 c1t1d0s2 c1t4d0s2...
Diskgroup/Volume Mirror: 1 2
---------------- ------- -------- --------
md/d10 #m #m
ossdg/export #m #m
ossdg/swap # #
ossdg/sybdev #m #m
6.3.2 Known Faults
6.3.2.1 HP68812 (HP71350) Sparc 12.2.10 - Incorrect disk
printout in dmtoolFault in DMR R3L01, fixed in R3M01.Problem: After
an Initial Install, when there is no mirror definition, DMR
presents a suggestion, having disks without s2.
Workaround: Answer 'n' on the suggestion, and continue the
definition. 6.3.3 Manual definition
Here is an example of a manual session, i.e. when the automatic
suggestion is not correct.
The Serial No displayed for each disk is useful in e.g. a
cluster, to make sure the right disk is specified. The Disk Access
name (c1t2d0s2) is not the same on different hosts. And the Disk
Media name (disk2) is not visible if its imported on another host.
But the Serial No is always the same.
===> Define manually
You will now define 2 Mirror(s).
Each Mirror has 1 root disk, and 0-n data disks.
Entering several disks, use a list or a range of disks.
E.g. '1,5,7' or '2 4 5' or '3 5-8' (='3 5 6 7 8').
Continue (y/n)? [y]
Current disks found in the system:
Bootddg: nodgDEVICE TYPE DISK GROUP STATUS
c1t0d0s2 auto:none - - online invalid
c1t1d0s2 auto:none - - online invalid
c1t2d0s2 auto:cdsdisk disk2 ossdg online
c1t3d0s2 auto:cdsdisk disk3 ossdg online
c1t4d0s2 auto:cdsdisk disk2mirr ossdg online
c1t5d0s2 auto:cdsdisk disk3mirr ossdg online
===> Defining Mirror 1 [diskn]
Enter ROOT disk for Mirror 1 [rootdisk]
# Disk Serial no Md Misc
- -------- ----------- --- ----
1 c1t0d0s2 0334B1SANJ d11
2 c1t1d0s2 0401B6R3BB d12
3 c1t2d0s2 0334B1S54H disk2/ossdg
4 c1t3d0s2 0334B1SB3D disk3/ossdg
5 c1t4d0s2 0334B1R4VW disk2mirr/ossdg
6 c1t5d0s2 0334B1SAGS disk3mirr/ossdg
Enter selection (q=quit): 1 Enter all DATA disks for Mirror 1
[diskn]
# Disk Serial no Dm/Dg Misc
- -------- ----------- ----- ----
1 c1t1d0s2 0401B6R3BB d12
2 c1t2d0s2 0334B1S54H disk2/ossdg
3 c1t3d0s2 0334B1SB3D disk3/ossdg
4 c1t4d0s2 0334B1R4VW disk2mirr/ossdg
5 c1t5d0s2 0334B1SAGS disk3mirr/ossdg
6 none
Enter selections (q=quit): 2,3===> Defining Mirror 2
[disknmirr]
Enter ROOT disk for Mirror 2 [rootmirr]
# Disk Serial no Dm/Dg Misc - -------- ----------- ----- ---- 1
c1t1d0s2 0401B6R3BB d12
2 c1t4d0s2 0334B1R4VW disk2mirr/ossdg
3 c1t5d0s2 0334B1SAGS disk3mirr/ossdg
Enter selection (q=quit): 1
Enter all DATA disks for Mirror 2 [disknmirr]
# Disk Serial no Dm/Dg Misc - -------- ----------- ----- ---- 1
c1t4d0s2 0334B1R4VW disk2mirr/ossdg
2 c1t5d0s2 0334B1SAGS disk3mirr/ossdg
3 none
Enter selections (q=quit): 1,2----------------------< Print
Mirror Configuration >----------------------
Intro:
All disks for each mirror will be displayed, with both disk
access name
and disk media name. The first disk in each mirror is always
considered
root disk by DMR.
Mirror 1 Mirror 2
------------ ------------ ------------ ------------
d10 c1t0d0s2 d12 c1t1d0s2
disk2 c1t2d0s2 disk2mirr c1t4d0s2
disk3 c1t3d0s2 disk3mirr c1t5d0s2
Is this correct (y/n)? [y]
6.3.4 Defining Mirrors in Cluster
6.3.4.1 Define the mirrors first on the Admin1 (Master).
Its recommended to have all data disks ordered, i.e. that the DA
names are starting with low numbers and increasing. E.g. c3t0d0s2
for disk2 down to c3t0d9s2 for disk11. Disks are listed in an
ordered way by DMR when prompting for selection. Its then easy to
use ranges like 2-11 when selecting.
For Mirror 2, the disks have to be selected so they work in
pair. For example, disk2 on M1 and disk2mirr on M2 must both be
used for e.g. ossdg, and disk11 and disk11mirr for e.g. sybasedg.
But they can be mixed. Like disk3 for sybasedg and disk4 for ossdg.
Its only on pair-level they have to be used for the same disk
group.
6.3.4.2 Then define on the Admin2 (Mate).
After a Make Cluster/Mate root disk, and when starting DMR for
the first time, it will prompt the user to define the Mirrors. The
root disks are different, but can have the same name as on the
Master, and the data disks are the same, but have different DA
names.
To support the user in defining the mirrors correctly, DMR will
use the master dm_define(.last) and the master
disk_serial_no(.last) file, to suggest a similar Mirror Definition,
with other DA names but each data disk in the same position. Only
the root disks will need to be defined manually.
If those files are not available, define manually. Its important
that each data disk is in the same position in dm_define, on both
hosts, regardless of DA name. Disk Media names will be same on both
hosts.
7 Procedure Manual
Here are a few procedures that involve the use of a sequence of
DMR options. For a detailed description of each option, see
relevant chapters.
Guide: Text in bold is an option in dmtool, e.g.;
Synchronize Mirror = The first option in the main menu
Boot Detached Mirror (Detach Mirror)= Option under a
sub-menu
# and 3# is a root prompt in multi user mode,
ok> = prompt in prom level
c# = prompt in cdrom mode
1# = prompt in single user mode
7.1 First Time usage
The first DMR is used on the server it requires specific
actions.
DMR Installation, see Installation chapter 3.
Define the mirrors, i.e. to tell DMR what disks pertain to what
mirror side. See Mirror Definition chapter 5.
Ensure the system is valid and conforms to standard. Use Check
Menu to verify. See System Check (sub-menu) chapter 19 for more
details.
7.2 Making a System Restore
There are two different ways to 'System Restore' depending on
the state of the system:
If the system has 2 mirrors and Solaris and Veritas running, use
option Restore a Dump on a Mirror (Dump & Restore). In this
way, the system will continue to run in multi user mode during the
whole restore. And even if downtime is not important, an advantage
is that the whole tape will be restored at once. (less
interaction)
If the system only has one mirror, or the disks are empty, use
DMR option Restore root to restore root in CDROM mode, and Restore
Data Volumes in Single user mode (Dump & Restore) to restore
the rest in single user mode.
Note: Restore Data Volumes in Single user mode can also be used
to restore a single volume, e.g. if home has crashed during normal
operation, and a full restore is not wanted. Note: If just some
files are missing, use Restore File instead.
7.3 Making a Rollback Mirror
It is often required to have a fall rollback procedure using the
disk mirrors, there are two different methods, depending on the
requirements.
If no major changes to the system are planned, and no
(temporary) DGSJ (FlashSnap/Enterprise) license is available, use
Detach Mirror.
If the Veritas SW is going to be updated, or major changes in
the Veritas object structures (Split DG, new volumes aso), and a
(temporary) DGSJ (FlashSnap/Enterprise) license is available, use
Split System.
7.4 Install a Correction Package, with success
Before you start, make sure that the system is mirrored well
using the system status menu to check all volumes and plexes are
ok. (there will be an automatic check also)
Detach Mirror (Detach Mirror), to freeze one disk Mirror.
Install Correction Package. Installation was successful this
time.
Synchronize Mirror, to get the system in a mirrored state
again.
7.5 Install a Correction Package, with rollback
Same as above, but if the installation fails,
Boot from Detached Mirror (Detach Mirror).
Synchronize Mirror, to get the system in a mirrored state
again.
Now start over again.
7.6 Restore a Dump tape, with minor problems
Restore a Dump on a Mirror (Dump & Restore), and answer yes
on scratch (from beginning).
Then an unexpected incident occurs during the restore, for
example, a power failure.
Restore a Dump on a Mirror, but now chose not to start from
beginning, and select only the missing volumes, starting from the
one that crashed.
7.7 Root disk Crash while Detached
If the running root disk crashes while one mirror is Detached,
there is no system to let DMR clean and prepare the (good) frozen
root disk. The frozen root is not bootable after a Detach
Mirror.
Note: After a Split System, the root disk is bootable. If booted
directly, it will not use any meta device, but rather the plain
root slice 0. That can be fixed by doing Init SVM on running
Root.
Background: The option Detach Mirror will put the frozen root in
a detach mode. i.e. the sub mirror d12 (if mirror 2 was detached),
is disconnected from the meta mirror d10. But the boot files in
/etc are still pointing to d10. I.e. it wont boot. The file system
on the frozen root needs to be altered in some way.
To recover from the crash and boot from the saved side, use the
following procedures:
7.7.1 Remake rollback-root disk in CDROM mode
Here the crashed root disk is c1t0d0, and the rollback root disk
is c2t0d0, i.e. the one that was detached/frozen:
Boot cdrom mode:
ok> boot cdrom -s
c# fsck /dev/rdsk/c2t0d0s0 (sometimes necessary)c# mount
/dev/dsk/c2t0d0s0 /mnt Where c2t0d0s0 is the detached root
disk.
c# cd /mnt/ericsson/dmr/bin
c# ./dmtool and select "Remake Root Disk"This will remove SVM
from the root disk, and prepare for boot amongst other things, see
Remake Root Disk for more details.
Then reboot: c# reboot -- -rs
7.7.2 Rename volumes
Use the procedure rename_detached_side in DMR, found under the
Misc sub-menu:
s# /dmr/dmtool m 5 (rename_detached_side)
This step should rename all data volumes so the detached side
will have the correct names, and the old side has a postfix, e.g.
export_m1.
Continue to Multi-User-Mode ('exit' or 'CTRL + D')S# exit 7.7.3
Synchronize Mirrors
Start DMR and make sure that the system is running on the old
frozen mirror, for both root and data;# /dmr/dmtoolSync Mirrors if
all seems ok, and start all over again:# /dmr/dmtool sy
7.8 Edit the Dump files before Restore
NOTE: the following option should only be used by experienced
personnel
If the user wishes to change the diskgroup names or
re-distribute the volumes on the dump, it is possible to do this by
editing the dump config file. To be able to edit the Dump config
files on the first block of the tape, before the restore starts,
dmtool has to be stopped in the middle:
# setenv FIXRDT 1# dmtool
And choose a Restore option. The script will then stop after
extracting the first block. In another shell tool, do this:
# cd /tmp/recovery/etc
# vi sysinfo (for example)Then press in the original cmdtool, to
continue the restore.
7.9 To merge two Diskgroups
E.g. to merge syb1dg and syb2dg into sybasedg.
Take a Dump of the system. Then use dmtool to restore it
again:
# setenv FIXRDT 1
# dmtool
And choose e.g. Restore a Dump on one Mirror. The script will
then stop after extracting the first block.
In another cmdtool, do this:
# cd /tmp/recovery/etc
Replace all occurrences of syb1dg and syb2dg with sybasedg:
# vi sysinfo
Create a new vx-.cfg file:
# cat vxsyb1dg.cfg vxsyb2dg.cfg > vxsybasedg.cfg
Then press in the original cmdtool, to continue the restore.
Answer y on scratch/clear disks.
After the restore is done, edit the new vfstab, to correctly
reflect the new diskgroups.
# vi /mnt/etc/vfstab
7.10 To Modify Disks in the System
Note: When below instructed to Clear Mirror, which is dmtool h
c, you can apply Clear Disks instead (dmtool h d) and only select
the data disks on that Mirror. In this way root redundancy is not
affected. (root disks can take quite some time to sync)7.10.1
Increase LUN size in Storage (X86)Note: This chapter is not part of
any OSS-RC procedure. This is not the official procedure to
increate LUN size.1. Clear Mirror (HW Changes); Select one Mirror
side.After this, the data LUNs on that Mirror are not in use. 2.
Increase the size of the cleared LUNs, from the storage.
(instructions to do this are not provided by this document)3.
Create new partition, new label and update Solaris and VxVM.A) For
SMI disks do: (smaller than 1TB)
# fdisk -B c1t500601603DE0208Bd8p0# format -e
c1t500601603DE0208Bd8 format> l Specify Label type[0]: 0 Ready
to label disk, continue? Y format> p format> p format> q#
prtvtoc /dev/rdsk/c1t500601603DE0208Bd8s2# vxdisk scandisksB) For
EFI disks do: (normally bigger than 1TB)
# fdisk -E c1t500601603DE0208Bd9p0# format -e
c1t500601603DE0208Bd9 format> l Specify Label type[1]: 1 Ready
to label disk, continue? Y format> p format> p format> q#
prtvtoc /dev/rdsk/c1t500601603DE0208Bd9# vxdisk scandisks
4. Synchronize Mirror to sync the new mirror.
Repeat for the other Mirror when sync has finished.7.10.2 New
disks to be Added to current disks
Add the new storage (For Sparc, go to prom level first).
Rescan HW (HW Changes); To enable the new disks.Clear Mirror (HW
Changes); Select one Mirror side.Redefine Mirror (HW Changes);
Redefine the cleared mirror.Synchronize Mirror to sync the new
mirror.
Repeat for the other Mirror when sync has finished.
7.10.3 New disks to Replace current disks (same controller)
Clear Mirror (HW Changes); Select one Mirror side.
Sparc: Go to prom level and replace storage on the cleared side,
and boot:ok> boot -rX86: Add the new LUNs to the Storage Group.
(no instruction provided)Rescan HW (HW Changes); To enable the new
disks.Redefine Mirror (HW Changes); Redefine the cleared mirror if
needed.Synchronize Mirror to sync the new mirror.
Repeat for the other Mirror when sync has finished.
7.10.4 New disks to Replace current disks (other controller)
Add the new storage (For Sparc, go to prom level).
Rescan HW (HW Changes); To enable the new disks.Clear Mirror (HW
Changes); Select one Mirror side.Redefine Mirror (HW Changes);
Redefine the cleared mirror.Synchronize Mirror to sync the new
mirror.
Repeat for the other Mirror when sync has finished.
7.11 Dump & Restore using dd, over 2 tapes
For a HA Replication system, data volumes can be dumped with dd
instead of vxdump/vxrestore. This is used to create an exact copy
of the data, which will later be synchronized over the LAN/WAN.
This method is only meant to be used in a HA Replication system,
when there is a need to synchronize the mate. Another way is to
bring one Mirror disk side to the mate. The normal backup routines
should be done with vxdump.
root file system will always be dumped with ufsdump. Other file
system, not Replicated, should not be used with dd.
The problem is that dd does not handle end of tape, it does not
wait for the next tape. So if the dump stretches over more than one
tape, several dumps have to be taken.
There are two ways, either take the dump, and if it gets full,
take another one, or prepare two setting files in advance, where
the file systems are distributed.
7.11.1 With no preparations
Take the dump with root and all volumes that are replicated. The
first tape will get full, where e.g. volume5 does not fit. Exit the
tool and insert a new tape and start a new dump. Now specify only
the last volumes, starting on volume5. Make sure the Mirrors are
synced afterwards.
When doing the restore on the mate, restore the first tape, and
specify only the first volumes, not including volume5. When done,
insert the next tape, and restore all volumes on that tape
(including volume5), but this time, answer n on Do you want to run
from scratch/Clear disks first.
These restores can be done using any of the 2 main methods;
Restore a Dump on a Mirror or Restore root in CDROM mode/Restore
Data Volumes in Single user mode. Doing these restores, answer n on
Use default options (n=custom).
7.11.2 Preparing dump setting files
Do one dd dump, to see which volume that fails, e.g. volume5,
and then Synchronize the Mirrors, before continuing.
Start a new dump session; (for guidelines on other questions,
see User Guide)
- Do not use previous settings.- Specify only the first volumes,
not including volume5.- Answer y on Save settings until next time".
(will be saved in ../etc/dump_set)- Answer n on OK to start".
(There is no need to really do the dump at this point)
Save the file ../etc/dump_set as e.g. ../etc/dd_setup_1.
Then start another session;
- Do not use previous settings.- Specify only the last volumes,
including volume5.- Answer y on Save settings until next time".-
Answer n on OK to start".
Save the file as e.g. ../etc/dd_setup_2. Now you have 2
predefined dump setting files. If you dont want to have e.g. which
Mirror or which tape station hardcoded, you can remove those lines
from the setting file, and you will be prompted instead. The most
important line is volumes=....
Make sure the Mirrors are synced afterwards.
When actually doing the dd-dump, specify like this:
# dmtool dumpfs file=../etc/dd_setup_1
# dmtool dumpfs file=../etc/dd_setup_2
Do the restore as normal, just make sure on the second tape, to
answer n on Do you want to run from scratch/Clear disks first.
8 Tape Stations
This chapter describes what kind of tape stations DMR supports,
and its limitations.
8.1 Supported Tape Stations
Practically all tape stations available are supported:
Normal tape stations
Autoloaders and Tape Archives
Remote tape stations (also in cdrom mode)
Local File System
Remove File System
In more detail:
Any normal tape station connected to the server, or on a remote
host. Like a LTO, DLT, DAT or Exabyte.
Autoloaders, like L280/L9/C4/SL48, connected to the same host,
will be managed fully by DMR. It is very important that the
OPERATING MODE is set correctly. It should be set to RANDOM, and
can be set manually in the small LCD screen in the front panel.
Autoloaders connected to a remote host, will not be managed
fully, but will work as a normal tape station, i.e. you will have
to load tapes manually. If the OPERATING MODE is set to STACKER, it
will automatically load the next tape when End Of Tape is
encountered during a dump.
Tape Archives, connected to the same host, will be managed
fully.
Tape Archives, connected to a remote host, will be treated as a
normal tape station.
If the remote tape is on an OMBS server, the drive has to be set
to DOWN, see chapter 8.4 for more details.
To access a tape station remotely, in CDROM mode, the LAN
interface has to be configured. DMR gives some support in this
task, see Managing Tapes and From dump tape, REMOTE tape
station.
The term Autoloader is sometimes used in DMR for both
Autoloaders, and Tape Archives.
Before using an Autoloader, make sure the Operating Mode is set
correctly, and then run option Setup MTX in the Dump & Restore
Filesystems sub-menu. [dmtool d m]
8.2 Limitations
An Autoloader with 2 drives is supported, but only 1 drive will
be used for the robot. The second drive can be used as a standalone
drive, without robot arm.
The load Next tape to drive option in Managing Autoloader, will
only load from the next slot, and not the next tape, i.e. the tapes
must be set consecutively.
8.3 Managing Tapes8.3.1 The Select and Check Tape Menu
This sub-menu can be found under Dump & restore. Its a
sub-procedure, normally called from another procedure. But can be
used separately to see what tapes exist, locally or remote.
Note: it is not necessary to run this option separately, before
doing a Dump or Restore.
[s] Search for local tapesPresents a list of local tape
stations. The first found LTO tape, will be the default choice. The
selection made by the user, will only be used if this procedure is
called from another procedure within DMR.
[d] Scan for new tape DriveIf a new tape drive has been
connected, it can be made available. (devfsadm is used to update
the device tree)
[m] Manage AutoloaderTo give the user control of the Autoloader,
to see what tape exists, in what slots, and to load the proper tape
into the drive.
[f] Setup FS tapeEnables dump/restore to/from a Local File
System, instead of a Tape.
[e] Explicitly specify driveIf a scan for each tape device is
not desired, or the tape device is not to be found under
/dev/rmt.
[c] Check for dumps on all tapesFor all tape stations, and all
tapes in each slot, the first block will be read, to see if it is a
dump tape. It will take a while, but the user receives a complete
inventory of the Autoloader. For DMR dumps, e.g. hostname and date
is displayed.
[r] Remote driveSpecify a remote host (or IP) and a user name
for login. A list of available tape stations is presented, for
selection. A remote Autoloader will be treated as a normal tape
station.
The protocol to access a remote drive has changed. Now PW-free
SSH access to the remote host is necessary. The script
/ericsson/dmr/bin/setup_ssh.sh can be used to setup SSH keys
between two hosts. See ch 11.5.
[t] Remote FS via ssh-tunnelSpecify a remote host (or IP) and a
user name for login. Enter the directory where the dump files
should be stored. One directory is used per dump, so enter e.g.
/base-dir/dump1.
The user should have SSH access to the remote host, without need
to give any password. The script setup_ssh.sh can help in setting
up SSH keys. See ch 11.5.
8.3.2 Installing MTX for Autoloaders
To be able to control Autoloaders, the freeware MTX is required.
Its not necessary to install MTX, but the system needs to be
prepared, so MTX can access the autoloader directly, via a SCSI
interface.
It can be done explicitly be using option Setup MTX, or
indirectly the first time the Autoloader is accessed by [m]
above.
The module sgen will be configured and loaded.
8.3.2.1 No Autoloader appears after configuring?
Check what target and LUN the Autoloader has in the system, i.e.
the robot arm SCSI device. Check then the file
/kernel/drv/sgen.conf. You might need to add more lines at the
bottom of the file, containing the Autoloader target/LUN. And then
recreate:
# ls -l /dev/scsi/changerprobably nothing# vi
/kernel/drv/sgen.conf
# update_drv f sgen
# devfsadm -C
# ls -l /dev/scsi/changerNow the autoloader should appear8.3.3
Controlling the Autoloader from the command prompt
To facilitate some common operations of an Autoloader, they have
been made available directly, as in-parameters to dmtool.
More operations are available via dmtool d s.
8.3.3.1 Load a Tape from a Slot
To load a tape into the drive, from a specific slot, simply
type:
# dmtool load
Where is optional.
This can be useful when running other tools towards the tape
station, tools that cannot control the Autoloader. E.g. program in
cron to load tape from slot 2, five minutes before a database dump
is taken. (rdt_dbdump)
55 2 * * * /ericsson/dmr/bin/dmtool load /dev/rmt/1 2
If only one Autoloader in the system, no need to specify
tape:
55 2 * * * /ericsson/dmr/bin/dmtool load 2
8.3.3.2 Unload the Drive
To unload a tape from the drive, and return it to its original
slot:
# dmtool unload
Where is optional. E.g.:
# dmtool unload
8.4 Drive DOWN on OMBS
OMBS 11.2 (NB 7) interrupts the DMR remote tape activity.
According to NB Admin Guide:
"The NetBackup avrd daemon periodically tries to rewind and read
data from media in the drives that are UP and are not currently
assigned in NetBackup.
To ensure reliable operation, do not use UNIX tape and drive
commands on the drives that are UP and controlled by ltid.
Users can use the NetBackup tpreq and tpunmount commands and the
drive_mount_notify and drive_unmount_notify scripts on those
drives.
To use UNIX commands on a drive, the drive must be down and ltid
must not be running." The remote drive must be set to DOWN in NB.
Its recommended to do this manually in the NB GUI.DMR will check if
it is an OMBS server and notify the user. DMR will also offer the
user to set the drive DOWN automatically, using the commands
below.
Note: These commands might affect the NB auto checks on other
drives.8.4.1 Set drive DOWN
Make sure 'ltid' is running, for 'vmoprcmd' to work:#
ltidanother ltid daemon is already running
Check for drive and status: # vmoprcmd -dp dsDrv DrivePath
Status Label Ready 0 /dev/rmt/0cbn UP Yes Yes 1 /dev/rmt/1cbn UP -
No
Set desired drive to DOWN:# vmoprcmd -down 0
Wait until drive has reached status DOWN:# vmoprcmd -dp ds |grep
/dev/rmt/0|nawk '{print $3}'DOWN
Then stop 'ltid' so it won't interfere with DMR:# stopltid8.4.2
Set drive UP
Make sure 'ltid' is running, for 'vmoprcmd' to work:# ltid
Check for drive and status: # vmoprcmd -dp dsDrv DrivePath
Status Label Ready 0 /dev/rmt/0cbn DOWN Yes Yes 1 /dev/rmt/1cbn UP
- No
Set desired drive to UP:# vmoprcmd -up 0
Let 'ltid' be running.8.4.3 Configuring auto-DOWNThe below entry
in cxc.env can be tuned.
#-----------------------------------------------------------------------
# A remote tape drive on a OMBS server must be put DOWN when
used by DMR.
# Otherwise NetBackup will interfer with DMR. Especially version
7.
# Setting DOWN should be done manually, e.g. via the GUI.
# DMR will warn the user and offer to actually put DOWN
automatically.
# If the variables below are set (y/n), DMR will use them as
default values.
# And use them, without asking, if no TERM.
#OMBS_DOWN=y
#OMBS_UP=y
However, there is no need to tune. If the variables below are
set (y/n), DMR will use them as default values when prompting the
user. If there is no terminal, the tuned values will be used. If no
terminal and no variables are set, then y is default for both DOWN
and UP.8.4.4 Known Faults
8.4.4.1 HP65163 (HP54389) 12.2 : OSSRC 12.2 dumped system cannot
be restored using R3K02 version of dmtool.
Fault in DMR R3L01, fixed in R3M01Problem: DMR fails to start
ltid on some NB 6.5 servers.
Workaround: If it fails the first time, answer 'n'o the second
time:
Set drive DOWN now (y/n)? [y] n Control the drive state
manually, as described in this chapter 8.4 above.
8.4.4.2 HP68944 DMR O11.3 - Problem bringing up/down OMBS tape
drivesFault in DMR R3L01 and earlier, fixed in R3M01Problem: To set
the remote NB drive UP or DOWN, DMR first starts ltid. Sometimes
that daemon hangs the tool.
Workaround: Answer 'n'o on the question to take OMBS drive
DOWN:
Set drive DOWN now (y/n)? [y] n
Control the drive state manually, as described in this chapter
8.4 above.
9 Internal Structure9.1 The environment file
The file /ericsson/dmr/etc/cxc.env can be used for tuning the
behaviour of DMR. The file is sourced by dmtool upon startup. The
variables set there, can also be set in the shell, manually,
depending on if a temporary or permanent behaviour is desired.
#-----------------------------------------------------------------------
# FMR=Fast Mirror Resynchronization
# If FastResync license is valid, DMR enables the fmr, on each
volume,
# before a split, and resets the fmr flag when synced back.
# This is done for safety reason; If the FastResync license
expires,
# the split/sync will fail...
# If FMR_OFF is set to n, DMR will leave the fmr flag on.
# This if another tool, than DMR, is operating in the
system.
#FMR_OFF=n
#-----------------------------------------------------------------------
# Normally DMR sets all defined root disks at boot devices.
# If one root disk fails, the next will be tried.
# If MULTI_BOOT is set to n, DMR will only set one root
disk.
#MULTI_BOOT=N
#-----------------------------------------------------------------------
# Normally DMR uses striping for data volumes. (unless only one
disk)
# If STRIPE is set to n, DMR will use concat.
# This can be useful in some SAN configurations, where several
LUNs
# are assigned to one data mirror. These LUNs should not be
striped.
#STRIPE=n
#-----------------------------------------------------------------------
# DMR distributes the DRL logs by default in a round-robin
fashion.
# If STRIPE=n, then the selection of DRL disk is handed over to
VxVM.
# If the above default behaviour doesn't fit your system,
uncomment
# this line, and set to 'n' or 'y'.
#DRL_DISTRIBUTE=n
#-----------------------------------------------------------------------
# When doing restore. If set, DMR stops after extracting the
dump
# files from the first block, and prompts to continue.
# In another shell, the dump config files (/tmp/recovery) can
be
# altered.
#RDTFIX=y
#FIXRDT=y
#-----------------------------------------------------------------------
# When Mirroring volumes. If set, DMR enables the user to
specify
# on what disks each volume should reside.
# Note: This will not split the disk group, only prepare for
it.
# Doing the actual DG split requires a veritas license.
(DGSJ)
# Note: A DG can be split without a special license, using
dump/restore,
# see procedure manual in UG.
#SPLITDG=y
#-----------------------------------------------------------------------
# In DMR, hotspare is disabled, unless enabled here.
#HOTSPARE=y
#-----------------------------------------------------------------------
# To enable the user to insert a reason for
detaching/splitting.
# Log in /ericsson/dmr/log/operator.log
#OPERATOR_REASON=y
#-----------------------------------------------------------------------
# If the SAME tape station is connected to BOTH servers
physically,
# for a Group Dump to work properly, set this variable.
# "Slave" session will stop and wait for Master, and prompt user
when to continue.
#SHARE_TAPE=y
#-----------------------------------------------------------------------
# List of Korn Shell scripts to be sourced
#DMR_PLUGIN_LIST=""
#-----------------------------------------------------------------------
# To turn off the use of EFI labels (EFI used for disks >1
TB)
#EFI=n
#-----------------------------------------------------------------------
# If SCT is activated, these volumes can be eliminated from
being dumped.
# The variable is a list of dg/vol items.
# Reg Exp can be used to specify a collection of volumes.
SKIP_FS="ossdg/rootckpt_*"
#-----------------------------------------------------------------------
# fssnap is used when dumping root in a single server
system.
# If fssnap is not desired or does not work, disable it
here.
#FSSNAP=n
#-----------------------------------------------------------------------
# The backing store file used for the root ufs snapshot is
normally located
# on /ossrc/dbdumps/root or /ossrc/upgrade/root.
# If another location is desired, specify that here.
#FSSNAP_BS=/ossrc/ericsson/root
#-----------------------------------------------------------------------
# Backup Monitoring: How many days of no successful system
Dump,
# before sending warnings to SM and FM.
MONITOR_DUMP_DAYS_THRESHOLD=7
#-----------------------------------------------------------------------
# Specify volumes that can be excluded and Dumps still counted
as complete.
#EXCLUDED_MONITOR_VOLUMES="ossdg/dbdumps ossdg/upgrade"
#-----------------------------------------------------------------------
# Backup Monitor to run every day
# Don't uncomment this line in cxc.env
# This line will be inserted into cron by dmtool
# You can adjust the time here
#0 8 * * * /ericsson/dmr/bin/dmtool -s sys_dump_monitor
#-----------------------------------------------------------------------
# To skip dumping vxpriv for each disk at every startup.
# When set, to force dumping vxpriv: dmtool dump_vxpriv_conf
noskip
#DUMP_VXPRIV=n
#-----------------------------------------------------------------------
# To hard code a value for the NAS snapshot/rollback
feature.
# It's used by space optimized snapshots only.
# They are used by DMR (Dump: All FS) and SCT (FS 'home').
# The cache should contain all changes for all snapshots.
# It will be created in the secondary pool.
#NAS_CACHE=5g
#-----------------------------------------------------------------------
# Used when testing to login with SSH, with help of Expect.
# Tune here if it doesn't fit your remote server prompt.
#EXPECT_PROMPT="> $|#[:]* $|[$] $"
#-----------------------------------------------------------------------
# Dumping using tmp-VxFS snapshots require a cache.
# Default is creating with 20% of data size.
#VXSNAP_CACHE_SIZE=0.2
#-----------------------------------------------------------------------
# A remote tape drive on a OMBS server must be put DOWN when
used by DMR.
# Otherwise NetBackup will interfer with DMR. Especially version
7.
# Setting DOWN should be done manually, e.g. via the GUI.
# DMR will warn the user and offer to actually put DOWN
automatically.
# If the variables below are set (y/n), DMR will use them as
default values.
# And use them, without asking, if no TERM.
#OMBS_DOWN=y
#OMBS_UP=y
#-----------------------------------------------------------------------
# When DMR is creating the list of disks, it's not calling
'format' for
# each disk, to make sure it's labelled properly.
# If that is desired, set this var to =y.
SCAN_DISKS=n
#-----------------------------------------------------------------------
# To reduce the number of disks, FC disks are not listed as
candidates
# for root. If FC disks are desired for root, set this var
=y.
FC_ROOT=n9.2 RC scripts and SMF
To perform various operations DMR needs to have scripts executed
during boot. Some scripts are permanent but most are transient.
Before Solaris 10, all scripts were found under the /etc/rc*d
directories.
In Solaris 10, SMF is used. The actual script is still located
under /etc/rc*d, but with a lower case s (s87dmtool).
Those scripts are not executed by the Legacy run, but rather by
DMR services.
The dmr1, dmr2 and dmr3 are three SMF FMRI resources, which
execute the created DMR scripts at the right moment.
Manifests are found here: /var/svc/manifest/dmr/dmr1.xmlLogs:
/var/svc/log/dmr-dmr1:default.log
9.3 DMR Alarms
DMR will send events and errors to SM. Errors are forwarded to
FM as (stateless) alarms.
These alerts are mainly sent when dumping or while monitoring
backups performed.
9.3.1 Installation
The file DMR_Alarms.txt will be installed automatically during
start of dmtool. (to
/etc/opt/ericsson/alarmfiles/DMR_Alarms.txt)
But it will not take effect until SM is (re-)started. If alarms
are desired before any reboot of the system, do:
# smtool -coldrestart SelfManagement -reason=other
-reasontext="dmr"
The default in OSS-RC is that the SM to FM alarm command is
defined. To check this, do:
# smtool -config SelfManagement | grep alarmCommandalarmCommand
/etc/opt/ericsson/nms_umts_ossrc_efmtrapsender/bin/snmpts.shA cron
job is also inserted during startup of dmtool. Its used to monitor
performed backups, and warn if the last successful dump is older
than a customized number of days.
All backups log the result in
/ericsson/dmr/log/dump_history.log.
9.3.2 Configuration
In the cxc.env file, three entries can be tuned regarding Backup
Monitoring.
9.3.2.1 MONITOR_DUMP_DAYS_THRESHOLD
The default is 7 days. This is used by the DMR cron job when
monitoring the backups (via dump_history.log). I.e. if the last
successful dump was done more than 7 days ago, an error/alarm is
sent.
9.3.2.2 EXCLUDED_MONITOR_VOLUMES
This variable contains a list of volumes, for example
"ossdg/dbdumps ossdg/upgrade". When considering if a dump is
complete or not, these volumes will not count. An incomplete dump
will not be regarded as a successful dump, and by so, the cron job
will warn when the threshold is passed.
9.3.2.3 Cron job
The entry in cxc.env can be altered to change the time when the
monitor should be launched. (Dont uncomment this line in
cxc.env)
#0 8 * * * /ericsson/dmr/bin/dmtool -s sys_dump_monitor
If the cron entry has already been inserted by DMR, edit the
crontab directly:
# crontab e
Either altering the timing or remove the line, and then start
dmtool, which would insert the line from cxc.env.
9.3.3 Interpreting the alarms
Several fields in the alarm/event text have null values. Those
can be ignored. Its basically the field Additional Information that
contains all information.9.3.3.1 BACKUP_FAILUREThe Backup failed
for some reason. Some details will be included in Additional
Information. Reasons can be problems with tape or volumes or the
backup was interrupted.9.3.3.2 QUIESCE_SYBASE_FAILURESomething went
wrong when doing a quiesce of Sybase. It can be either while
freezing databases or when releasing them. Or even before starting.
If the DBs were frozen but cant be released, then its critical. The
whole system is more or less frozen. Any database transaction is
put on hold.9.3.3.3 BACKUP_NOT_PERFORMEDThis event is generated by
the Backup Monitoring feature. Its a DMR cron-job that verifies
that a complete dump has been taken during the last X days. The
number of days can be tuned in cxc.env MONITOR_DUMP_DAYS_THRESHOLD
(default 7).
Note: If no Backup has been done, ever, there is no date to
calculate number of days since last dump, so the value -1 is
presented. It means that a DMR Backup should be taken.
10 NAS handling
This chapter describes how DMR, in various use cases, utilizes
the NAS.
10.1 Introduction
A Network Attached Storage (NAS) device is mandatory in OSS-RC
O11.2 x86/blade configuration. Home and PM files are moved from
local (VXVM) storage to an NFS server. The chosen default NFS
server is Symantec File Store (SFS).
Upon startup of DMR, the storage info has been updated for
NAS:
This is a HA Cluster system
OSSRC_O11_2_Shipment_11.2.8 AOM_901070 R1H
-> Dumping vxpriv config
............
Diskgroup/Volume Mirror: 1 2
---------------- ------- -------- --------
md/d10 #m #m
ossdg/3pp #m #m
ossdg/JUMP # #
ossdg/ericsson #m #m
ossdg/export #m #m
ossdg/swap # #
ossdg/upgrade #m #m
ossdg/versant #m #m
NAS FS Defined Created Mounted
------------ ------- ------- -------
eba_ebsg y # m
eba_ebss y # m
eba_ebsw y # m
eba_rede y # m
eba_rtt y # m
home y # m
segment1 y # m
sgwcg y # m
NAS Sys-Id:xb1140a Pool-Pri:xb1140_pri Pool-Sec:xb1140_sec
===> Mirror Status
Mirror 1: diskn RUNNING d10 c2t0d0s2
c0t5006016C44604B47d6s2...
Mirror 2: disknmirr RUNNING d10 c2t1d0s2
c0t5006016C44604B47d8s2...
10.2 Storage Manager
Installation scripts and tools like DMR interface with the NAS
via Storage Manager (STM). STM consists of an API and a Plug-In.
The API provides a stable interface towards the NAS head. The
Plug-In translates the API commands to device specific
commands.
To replace the NAS head with another vendor or model, only the
Plug-In needs to be replaced. API and installation scripts/tools
dont need to be changed.
The NAS API can also be used manually to interact with the NAS,
for example for maintenance purpose.
10.3 Symantec File Store
SFS consists of a two node cluster, running SLES Linux and a
full suite of Veritas products, like VCS, CFS and CVM.
The STM NAS SFS Plug-In logs in via PW free SSH calls, to
execute the specific SFS commands. User master is used for SFS
command sessions. There is also a support user that can be used to
gain access to the Linux platform, where also the traditional
Veritas CLI commands are available.
At the backend of SFS a SAN resides, typically a low range EMC
Clariion, where the actual file systems are stored.
10.4 Serving Multiple Systems
Several OSS-RC and ENIQ systems can utilize the same NAS head.
To be able to distinguish between file system home from system-1
and system-2, each file system has a System Identifier pre-pended
to it.
oss1/home, oss2/home,
10.5 Snapshots
10.5.1 Postfix name
Snapshots have to be unique and, and in the SFS case, a unique
name for each file system. To accomplish this, a snap-fix is
post-pended to the FS name. Some examples:
oss1/home/snap1, oss1/home/snap2 oss1/eba_rede/snap1,
oss1/eba_rede/snap2 oss2/home/dump, oss2/home/last
oss2/eba_rede/dump, oss2/eba_rede/lastWhere oss1/*/snap1 are the
same snapshot. I.e. all FS in oss1.
10.5.2 Types
Snapshots can be of two types, Space-Optimized (optim) and Full
Sized (full).
Space-Optimized snapshots are relatively quick to create, and
dont occupy much space, in the beginning. But they grow with time,
for each write to the main FS. They also add an additional I/O load
to the storage, so they are not suitable for I/O intense file
systems like PM FS.And if the FS has a lot of I/O, the snapshots
will become big.
Full Sized snapshots occupies the same size as the main FS, like
a mirror, and by so, takes time to create (synchronize).On the
other hand, they dont add any additional I/O to the storage.
10.5.3 Implementations
These snapshots can be seen in a system:
dump Created by DMR and used during Backup. They are
Space-Optimized, and are removed directly after Backup has
finished.
_m1 and _m2 Created by DMR and used during Detach. They are
Space-Optimized, and only created for file systems with
SNAP_TYPE=optim in storage.ini. Today, only home has optim set.The
postfix name is the same as used for the local Mirrors. One is
created during Detach and the other during Boot Detached.
sysidB/fs This type doesnt have any postfix. The Sys-Id has
changed instead. (e.g. oss1a > oss1b)Created by DMR Restore on a
Mirror and possibly during Split System. They are Full Sized and
only created if the user has asked for it. The default is to not
snapshot the NAS during Split System, which is the behavior used by
OSS-RC SW Upgrades.
sct01 to sct99 Created by SCT. They are Space-Optimized.
Currently only created for file system home.
last Created by SCT and used during Rollback. Its
Space-Optimized. Only created for home, and only one at a time.
10.6 Pools
Each system has two Pools. A Pool is an enclosure or container
of disks, similar to a Disk Pack containing a Mirror.
One of the Pools is regarded as the Primary pool, where all the
main file systems are stored. In the Secondary pool, snapshots are
created.
10.7 Storage INI file
The use of the NAS for each system is saved in an INI
configuration file:
/ericsson/config/storage.ini
There is a general section:
[Storage_NAS_GENERAL]SYS_ID=oss1a
POOL_PRI=oss1_a
POOL_SEC=oss1_b
And one section listing all file systems:
[Storage_NAS_SEGMENT1]
FS_NAME=segment1
SIZE_CALC=...
FS_SIZE=4800m
NFS_HOST=nas1
MOUNT_PATH=/var/opt/ericsson/pms/segment1
NFS_SHARE_OPTIONS="rw,no_root_squash"
SNAP_TYPE=full
EXPORT_TO_UAS=false
[Storage_NAS_HOME]
FS_NAME=home
SIZE_CALC=...
FS_SIZE=11800m
NFS_HOST=nas2
MOUNT_PATH=/home
NFS_SHARE_OPTIONS="rw,no_root_squash"
SNAP_TYPE=optim
EXPORT_TO_UAS=true
The NAS API reads this file to get default values when the user
(caller) specifies - (dash) as parameter.
10.8 Dump File Systems
Note: This chapter is an addendum to the feature explained in
chapter 14.1.
Note: This chapter is not about backing up the SFS
configuration.
A Dump is started as normal, and if in a cluster, another DMR
session has to be started when asked to (if Group Dump specified).
The blade having ossdg imported, will handle the NAS.
DMR creates an optim-snap, called dump, for each files system in
the NAS.
oss1/home/dump
oss1/sgwcg/dump
oss1/segment1/dump
...
It also creates a share for each snapshot, so they can be
mounted on the OSS (currently this takes quite some time). They are
then mounted under:
/tmp/dmrmnt/home
/tmp/dmrmnt/sgwcg
...
These snapshots are then removed when the Backup has
finished.
10.9 Restore on a Mirror
Note: This chapter is an addendum to the feature explained in
chapter 14.2.
Note: Related explanations are also found in chapter 14.2.7.
Before Restoring data, DMR creates a new set of main file
systems in the Secondary pool. It uses another Sys-Id (prefix) for
them to be unique.
10.9.1 New Sys-Id
The new Sys-Id is created according to the following schema:
A current Sys-id of oss1 or oss1a becomes oss1b for restore.
A current Sys-id of oss1b becomes oss1a for restore.
The parameter SYS_ID= is changed in the storage.ini file on the
new root, after its restored.
Note: If Sys-Id is changed, remember to add all previous
clients, see chapter 10.17.
10.9.2 Swapping Pool names
The parameters POOL_PRI= and POOL_SEC= will have swapped values
in the new storage.ini on the root belonging to the restored
side.
10.9.3 Method A: Restore on two blades
A Restore on a Mirror can be done in two different ways.
Depending a bit on if there are two Dumps, one for each blade, and
if there are two tape drives available.
A Restore on a Mirror is done when the old system is still
running Live.
Note: Its assumed here that Oss is running on Admin1 and Sybase
on Admin2. If in your case Oss is running on Admin2, just swap Adm1
and Adm2 in the below text.
10.9.3.1 Clear Disks before Restore
If disks are not cleared first, i.e. removed from the active
Diskgroup before the Restore, you must restore each Diskgroup on
the blade where its imported.
I.e. either restore on the correct blade (ossdg on Oss-blade and
sybasedg on Sybase-blade), or clear the disks first. To clear them
do either of these two methods:
Clear a Mirror on each blade having a Diskgroup imported:#
dmtool h cSelect a mirror. This will also clear the root disk,
which also will be restored.
If both Oss and Sybase are on the same blade, and Restore will
be done on the other blade, you could skip clearing the root disk:#
dmtool h dSelect all data disks on one Mirror. This will not clear
the root disk.
10.9.3.2 Restore on both blades
If Oss and Sybase Service Groups are running on separate blades
(Adm1 and Adm2), and the Dump consists of two separate Dumps, and
there are two tape drives available, so the Restores can be
executed in parallel, then consider this option:
Start a Restore on a Mirror on each blade, Admin1 and Admin 2.
They wont be Synced in execution, but they can run in parallel.
Reboot both blades onto to the new root disks.
The system will come up using Sys-Id-B and the old Secondary
Pool as its Primary Pool.
The benefit with this procedure is that the Restore is done in
parallel, and no downtime of Sybase Switch-Over.
10.9.3.3 Reboot both blades
DMR will reboot both blades at the same time. The DMR session on
the Mate/Admin2/Sybase will just exit, and the DMR session on
Master/Admin1/Oss will launch hatool p b to reboot the cluster.
The script hatool will make sure that VCS is started at the same
time when coming up, to avoid that one node is coming up before the
other node has gone down.
DMR will also wait for each other when going up, to rename disk
groups, before starting VCS.
10.9.4 Method B: Restore on one blade
This procedure is used if none of the criteria above is
fulfilled or this method is simply more suitable.
Basically; All activities are taking place on one blade (here
Adm1) and the other blade (here Adm2) is down.
Procedure:
Switch-Over Sybase from Adm2 to Adm1 (if not already there).#
hatool o s
NB: This creates some downtime. And then take down Adm2:2# init
0
Start a Restore on a Mirror on Adm1, and select a mirror.This
can be one complete restore or two sequential restores.1# dmtool r
r Reboot Adm1, which now will come up using Sys-Id-B and the old
Secondary Pool as its Primary Pool. NOTE: See 10.16.
Later on, create a new Adm2 (root disk) to complete the
cluster.Use the add_cluster_node script to do that.
10.9.5 Rollback
A Rollback can be done, up until the point the disks are
synchronized.
10.9.5.1 Method A: Two blades
Used if the old system is load shared on both Adm1 and Adm2;
then launch a Boot Other on both blades.
Adm1&2# dmtool de bo
See ch 10.9.3.3 for more details on rebooting both blades.
10.9.5.2 Method B: One blade
Make sure Adm2 is down:
2# init 0Boot up the old root disk on Adm1:
1# dmtool de bo
Or just change boot-device and boot up.
See ch 10.16 how to boot a single blade.
Both methods above should bring up the old system. You can now
either start over with a Restore or Synchronize.
10.9.6 Synchronize
If the current system is fine and no Rollback is foreseeable
then the disks can be synced to enter the default redundant
state.
Start a synchronization of the local storage: (as normal)
Adm1&2# dmtool sy
10.9.7 Cleanup Secondary Pool
The Secondary pool, i.e. the former Primary, contains a full set
of all file systems, having the old Sys-Id. They should be deleted,
before creating any snapshots.
Note: This has already been done if a synchronization was made
(with answer yes on cleaning NAS).
# dmtool nas_sysid_clean_all OR:
# nascli delete_fs -
10.9.8 Keep New Sys-Id
After the Restore, the system is using a different Sys-Id (e.g.
oss1b) and the system considers the second pool to be Primary and
the first pool to be Secondary. This is a perfectly fine state, and
can be left as is.
Note: If the Secondary Pool is shared with other systems, your
system must change back to the primary pool and Sys-Id.
Note: If Sys-Id is changed, remember to add all previous
clients, see chapter 10.17.
10.9.8.1 Update LDAP
If the new Sys-Id is to be kept, update LDAP so UAS can mount
the correct file system. See chapter 10.14.1 for more details.Skip
the next chapter if the Sys-Id shall be kept.
10.9.9 Change Back Sys-Id
After the Restore, the system is using a different Sys-Id (e.g.
oss1b) and the system considers the second pool to be Primary and
the first pool to be Secondary. This is a perfectly fine state, and
can be left as is.
However, if this is not desired, and the old state is preferred
and downtime is acceptable, then use this method to change
back.
10.9.9.1 Use NAS API to create new file systems
Create Full Sized snapshots in the Secondary pool using the
alternating Sys-Id. (here oss1a is the alternating Sys-Id, and the
current one is oss1b)
Adm1# nascli create_snapshot - full - sysid/oss1a -
Note: From now on, all changes on NAS will be lost on the new
set of file systems, i.e. a delta data gap.
Note: If not done before, all old oss1a NAS file systems have to
be deleted first, see chapter 10.9.6 to clean up.
10.9.9.2 Split off snapshots
Split off the snapshots from their respective main FS to become
new main file systems:
Adm1# nascli split_snapshot - -
10.9.9.3 Create shares
Create shares for the new file systems:
Adm1# nascli create_share oss1a - -
10.9.9.4 Test Mount
Test to mount home file system:
Adm1# nascli get_share oss1a home/vx/oss1a-homeAdm1# mount F nfs
nas1:/vx/oss1a-home /mntAdm1# ls /mntAdm1# umount /mnt
10.9.9.5 Update storage.ini
You can edit /ericsson/config/storage.ini manually. Just update
SYS_ID=, POOL_PRI= and POOL_SEC=. But its much easier to do this
via DMR.
Run dmtool, check the result and scp over the file to Adm2:
Adm1# /dmr/dmtool nas_swap_sys_id /Adm1# diff
/ericsson/config/storage.ini \/ericsson/config/storage.ini.dmrAdm1#
scp /ericsson/config/storage.ini \ :/ericsson/config
10.9.9.6 Update cluster.ini
Run dmtool, check the result and scp over the file to Adm2:
Adm1# /dmr/dmtool nas_update_cluster_ini /Adm1# diff
/ericsson/config/cluster.ini \
/ericsson/config/cluster.ini.dmrAdm1# scp
/ericsson/config/cluster.ini \ :/ericsson/config
10.9.9.7 Stop All Services
WARNING: Downtime starts here
Stop Services to un-mount old mount points:
Adm1# hatool p a
10.9.9.8 Update main.cf
Run create_maincf, check the result and scp over the file to
Adm2:(/haconf is a link to /etc/VRTSvcs/conf/config)
Adm1# /ericsson/core/cluster/bin/create_maincf Adm1# diff
/haconf/main.cf /haconf/main.cf.oldAdm1# scp /haconf/main.cf
:/haconf
10.9.9.9 Start Cluster
Adm1# hatool t c
The system should now mount and run on the file systems located
in the Primary Pool.
10.9.9.10 Copy Delta Data
The delta data that was produced on the Secondary side, can be
copied over (manually) to the Primary side, if so desire.
Here is an example of file system home. Repeat for all file
systems
Adm1# nascli get_share oss1b home/vx/oss1b-homeAdm1# mount F nfs
nas1:/vx/oss1b-home /mntAdm1# Adm1# umount /mnt
10.9.9.11 Cleanup Secondary Pool
When all seems ok, clean out the file systems with the
alternating Sys-Id located in the Secondary pool. They should be
deleted, before creating any new snapshots.
# dmtool nas_sysid_clean_all OR:
# nascli delete_fs -
10.10 Restore Data Volumes
Note: This chapter is an addendum to the feature explained in
chapter 14.4.
This option is performed in Single User mode.
Any existing file systems in the NAS with the current Sys-Id
prefix will be deleted first. All file systems to be restored are
created in the NAS using the current Sys-Id. Also Shares are
created and file systems are mounted, to be able to write data.
Note: Remember to add all previous clients, see chapter
10.17.
10.11 Detach Mirror
Note: This chapter is an addendum to the feature explained in
chapter 13.1.
When detaching a Mirror, root and local data are handled as
normal.
In the NAS, a Space-Optimized snapshot is created, using the
same postfix as for the VX volumes; _m1 or _m2.
One is created during Detach and the other during Boot
Detached.
They are created for file systems with SNAP_TYPE=optim in
storage.ini. Today, only home has optim set.
10.12 Boot Detached Mirror
Note: This chapter is an addendum to the feature explained in
chapter 13.2.
When the new root disk is booted up, the NAS snapshot is rolled
back, i.e. the snapshot content is copied to the main file system.
Before that, a backup snap is created, using the alternating
postfix, _m1 or _m2, that can be used if Boot Detached Mirror is
repeated.
Note: The NAS snapshot is rolled back when booing up the new
side. It can take quite some time, before being able to login to
the system.
10.13 Split System
Note: This chapter is an addendum to the feature explained in
chapter 13.5.
Split System is for creating a full copy of the system on disk.
Its used for both general rollback option and for OSS-RC SW
upgrades.
When used for OSS-RC SW upgrades, the cluster is split, and the
Mirrors are split. But the NAS is not touched. The offline blade to
be used for upgrade, mounts the NAS file systems read-only.
When executing dmtool de sp (dmtool split_system) the user is
prompted whether to take full sized snapshots or not. Default is to
not take any snapshots, which is the option used by the official SW
upgrades.
If answering yes on that question, a full set of file systems in
the NAS is created. They are located in the Secondary pool, using
the alternating Sys-Id as prefix.
The file systems are created via full sized snapshots which are
then split off from their main file systems, to become new main
file systems.
It can take quiet some time to create the full sized snapshots.
If the snapshots already exist, they are only refreshed, which is
much faster if not too much data has changed since last
refresh/creation.
Note: If Sys-Id is changed, remember to add all previous
clients, see chapter 10.17.
10.14 Boot Other
After a Restore on a Mirror, or a Split System with full-sized
snapshots, the other mirror has a different Sys-Id then the active
one.
Note: The default behavior of Split System is to not take
full-sized snapshots. I.e. the Sys-Id will not change.
If the Sys-Id has been changed the mount points have to be
updated to reflect this change.
These files are affected when Sys-Id has changed: storage.ini,
cluster.ini, main.cf, auto_direct.10.14.1 Update LDAP
The file /etc/auto_direct is replicated in LDAP, so LDAP needs
to be updated if Sys-Id has changed. 10.14.1.1 DMR R3S01 or
newer
The file /etc/auto_direct has been updated by DMR, but
maintain_ldap.bsh has to be executed to propagate the changes to
LDAP.Wait until the new side has come up ok, and VX volumes are
mounted. Then run maintain_ldap.bsh by simply starting dmtool and
it will launch it for you. The LDAP admin PW is required.
10.14.1.2 DMR older than R3S01
The file /etc/auto_direct has not been updated by DMR.Perform
Boot Other and wait until the new side has come up. Check your
current Sys-Id:# grep SYS_ID
/ericsson/config/storage.iniSYS_ID=xb1140b# grep '/vx/'
/etc/auto_direct/var/opt/ericsson / mashost:/var/opt/ericsson
/sgw/outputfiles nas1:/vx/xb1140a-sgwcg /nms_umts_pms_seg/segment1
nas1:/vx/xb1140a-segment1 /eba_rtt/data nas1:/vx/xb1140a-eba_rtt
/eba_rede/data nas1:/vx/xb1140a-eba_rede /eba_ebsw/data
nas1:/vx/xb1140a-eba_ebsw /eba_ebss/data nas1:/vx/xb1140a-eba_ebss
/eba_ebsg/data nas1:/vx/xb1140a-eba_ebsg /ccpdm/pm_storage
nas1:/vx/xb1140a-pm_storage /nms_cosm/polled_data
nas1:/vx/xb1140a-nms_cosm/home nas1:/vx/xb1140a-homeThen edit
auto_direct manually:# cp /etc/auto_direct /etc/auto_direct.old# vi
/etc/auto_direct
Change Sys-Id in all places. (i.e. change from xb1140a to
xb1140b)Then launch this script:
# /opt/ericsson/sck/bin/maintain_ldap.bsh
This will propagate the new changes to LDAP.Then reboot/remount
any non-Admin NFS clients, e.g. UAS.10.14.2 Reboot UAS
After LDAP updates, reboot clients like UAS. ENIQ needs to have
a few files altered to mount from the new share path.10.14.3
Addition info
Note: This chapter is an addendum to the feature explained in
chapter 13.6.
See ch 10.9.3.3 for more details on rebooting both blades.
If Sys-Id is changed, and you have an older DMR than R3S01, then
add all previous clients according to chapter 10.17.
10.15 Synchronize
Note: This chapter is an addendum to the feature explained in
chapter 12.
The Synchronize option treats root and local data same as
before.
There is a question whether to remove file systems or snapshots
in the NAS using the alternative Sys-Id prefix, normally located in
the secondary pool.
Those FS/snaps are normally not needed after Synchronization.
But there can be scenarios where they are desired, so give it a
thought before answering yes.
10.16 Booting a single node in a two node cluster
When e.g. Adm1 is down, and booting up Adm2, the blade is not
coming up automatically. This is due to it is a two node cluster
and VCS expects both nodes to join before continuing. This is seen
in the /halog (VCS enging A):
VCS CRITICAL V-16-1-11306 Did not receive cluster membership,
manual intervention may be needed for seeding
To get it up, login to the console and seed manually:
# /sbin/gabconfig c -x
See ch 4.2.2 Start the System when One Node is Down in Ref
[4].
10.17 NFS Clients
The Admin blades are NFS (SFS) clients, and are taken care of by
DMR during DMR operations, for example if Sys-Id changes, the Admin
blades are added as NFS clients to the new file systems.Change of
Sys-Id can occur when Restoring on a Mirror or doing Split System
plus Boot Other. The need for adding more NFS Clients occurs also
when doing Restore from scratch.
Originally DMR did not handle all the other NFS clients, like
UAS and ENIQ. DMR is updated now (R3S01) to add those clients as
well. If you have an older DMR, perform chapter 10.17.1.
10.17.1 Add NFS Clients manually
If you have a DMR version older than R3S01, NFS clients other
than Admin blades are not handled, like UAS and ENIQ. It might be
necessary to add those manually using the NAS API interface.
To add an NFS client, do:
# cd /ericsson/storage/bin# ./nascli add_client -| -| -|[
...]For example adding an existing UAS as client for home:
# ./nascli ac 10.10.10.10 - homeWhere 10.10.10.10 is the IP on
the Storage VLAN. It can contain a netmask as well:
10.10.10.10/32.The first - dash means the current Sys-Id. If you
run this command before reboot, specify the new Sys-Id, e.g.
oss1b.
The second dash means it takes the NFS share options from
storage.ini, which has as default rw,no_root_squash.
If home is replaced with a dash, the client will be added to all
file systems created in the NAS, for that Sys-Id.11 New Dump
Features
11.1 Temporary VXFS snapshots instead of Detach Mirror
The VxFS file systems have the possibility to create volatile
snapshots, which can be mounted for backup. This feature is now the
default option. There is no need to split the Mirrors, and the
risky sync-back operation is avoided too.
No extra license is required. But the snapshots are not
persistent through reboots.
All changes to the file system during backup must be contained
in a cache. DMR creates temporary cache volumes which are removed
afterwards. The default size of the cache is 20% of the data. (not
volume size)
Use dmtool s s to find out how much data exists in each disk
group and how much free space is available on each mirror. The
total free space is the sum of all mirrors free