Using the AIX Logical Volume Manager to perform SAN ...public.dhe.ibm.com/software/dw/aix/au-aixstorage-pdf.pdfIBM SAN Storage Migration with AIX and dedicated I/O Several years ago
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Using the AIX Logical Volume Manager to performSAN storage migrationsMigrating AIX systems to new SAN storage subsystems withthe Logical Volume Manager
Skill Level: Introductory
Chris GibsonAIX SpecialistSouthern Cross Computer Systems
13 Jul 2010
This article provides examples of how to migrate AIX® systems from old to new SANstorage subsystems. It covers both dedicated and virtual I/O systems. We willexamine IBM and non-IBM storage migrations, using the AIX Logical VolumeManager (LVM).
Introduction
Connecting with ChrisChris is a popular author and blogger on AIX. Browse Chris's otherarticles and check out his blog and My developerWorks profile
If you work with AIX long enough, a time will come when you will need to migrate anexisting AIX system from an old SAN storage device to a shiny, new SAN storagesubsystem. The lease may be up on the old device, or it may simply need to bereplaced by newer technology. This storage device could be an IBM product or thatof another vendor. Either way, you will be faced with the decision of how best tomigrate your AIX system to the newer device. Of course, you will also want tominimize the impact to your running systems as a result of the migration.
In this article, I will share some examples of how I migrated AIX systems from old to
new SAN storage subsystems. I will cover both dedicated and virtual I/O (VIO)systems. To demonstrate the approach, the examples will include both IBM andnon-IBM storage.
Preparation and planning
Any migration of this type requires careful preparation and planning. I’m going toassume that you or your SAN storage administrator have already cabled, configuredand connected your new SAN storage device to your existing SAN. Also, I'll assumethat your AIX systems already have SAN connectivity to your existing SAN fabric.
Before migrating, verify support with your storage vendor. Review the vendor'sstorage support matrix. These matrices are usually listed on the storage vendor'swebsite (or are available from the vendor's representative). They will highlight whichsystems are supported with their storage device. For example, for IBM Enterpriseclass storage devices (such as the DS8300), they provide a support matrix whichcan be used as a reference during your planning. The DS8300 Interoperability Matrix(see the Resources section) identifies important support and compatibilityconsiderations with respect to various host systems and adapters. They cover awide spectrum of support checks such as supported operating systems, patchlevels, required Fibre Channel (FC) adapter firmware, supported SAN switchtypes/firmware, and much more.
One of the most important pieces of the puzzle is the required multi-path I/O devicedrivers and recommended FC adapter firmware (Microcode) levels. I have seen allsorts of problems when these components are not checked prior to integrating a newstorage device into an existing SAN and AIX environment. Most vendors providetools to help with the planning and verification, for example, IBM provides the FixLevel Recommendation Tools website, as well the IBM HBA support site (see theResources section). If a vendor does not provide online tools to assist in theplanning, then I recommend you ask them for help directly. After all, it is in theirinterests to help you make their product work in your environment!
Another important (and surprisingly sometimes overlooked) stage in the planningand preparation is the design phase. Take the time to design how your new SANstorage device is going to fit into your existing SAN. Ask questions that will help thedesign process, such as:
• Can/should this device connect to the existing SAN or is this a good timeto provision a new SAN Fabric?
• How will the AIX systems connect to the new storage device?
• How will the AIX operating system and data be migrated from the old tothe new disk?
If it helps (and it usually does), draw pictures to help demonstrate answers to thesequestions. It will also help others to visualize and understand what you are trying toachieve. Start with a diagram that encapsulates your current state, then another thatdescribes how the new device will fit, followed by the proposed migration process orprocesses. Finally, at the end, state how the environment will look once the olddevice is no longer needed and all the data has been migrated from it.
Will you be migrating from a dedicated I/O environment to a virtual one? If you are,then I recommend you review the latest IBM Redbook on migrating from physical tovirtual storage (see Resources). This publication guides you through the migrationprocess and offers several methods for migrating.
If you are migrating a dedicated I/O environment to another dedicated I/Oconfiguration, then consider the software requirements on each LPAR. For example,for IBM DS8300 storage you need to ensure that you have the appropriate SDDPCMMPIO device drivers (e.g. devices.sddpcm.53.rte) and the DS8300 Host Attachmentkit (e.g. devices.fcp.disk.ibm.mpio.rte) prior to the migration (or immediately after),along with the required FC adapter Microcode (firmware).
If you already have a virtual I/O environment (running a virtual I/O server, or VIOS,for disk traffic), then consider the requirements for the VIOS. If you are migratingfrom IBM to IBM storage, then it is likely that you will simply need to update theMPIO code, FC adapter firmware and supporting device drivers. However, if you aremigrating from Vendor A to Vendor B, you may need a different approach. Thedesign process should shake out these considerations beforehand.
For example, if you are migrating (VIOS presented disk) from IBM DS to NetAppstorage, then it is likely that you will need to consider provisioning new VIO serversfor the NetApp disk. Rather than mix two vendors' MPIO code on the same VIOS, itmay be simpler to manage if each storage type has its own VIOS. I recommend thisapproach and my examples will cover how I chose to deploy this type ofconfiguration.
Unfortunately, I won't be discussing N-Port ID Virtualization (NPIV) in this article.NPIV adds another dimension to storage virtualization on the Power platform. NPIVwith a VIOS, utilizes virtual FC devices to present disk (and tape) natively to theclient LPARs. See the resources section for more information.
I'm also assuming that you do not have an IBM SAN Volume Controller (SVC) inyour environment. If you do have an SVC and all your AIX systems are alreadybehind it, then I suggest you take advantage of this product's amazing capabilities. Itcan migrate storage transparently, from one storage device to another, without thehost system ever being aware of the move.
If you don’t have an SVC, and you are considering implementing one, I say proceedwithout delay! With an SVC in your environment, storage migrations (for the AIX
administrator) become a thing of the past. And that’s just one of the manyadvantages to using this wonderful device. See the Resources section for moreinformation on the IBM SVC.
Without an SVC, AIX storage migrations will most likely involve the use of the AIXLogical Volume Manager (LVM). This is what I will cover in the examples that follow.
IBM SAN Storage Migration with AIX and dedicated I/O
Several years ago I needed to migrate a large number of AIX hosts from an old IBMESS (F20) to a shiny, new IBM DS8300 storage subsystem. The AIX hosts were allusing dedicated FC adapters (HBAs). Each host had at least two FC adaptersconnected to our SAN. The following diagram shows the high-level view of the SANstorage and AIX LPAR connectivity to the existing SAN Fabric:
Figure 1. IBM ESS to DS8300 storage migration - Dedicated I/O - Current state
The MPIO code for every AIX system connected to this type of storage had to beupdated to the latest SDD device driver and FC adapter Microcode. For example,our design document stated the following:
Ensure that the following software is installed on all AIX systems, at these levels:
Fileset devices.pci.df1000f7.com:5.2.0.50 is applied on the system.
IY62116 Abstract: after EEH error, attached hdisks failed
Fileset devices.pci.df1000f7.com:5.2.0.50 is applied on the system.
ibm2105.rte
32.6.100.24 COMMITTED IBM 2105 Disk Device
devices.sdd.52.rte
1.6.0.2 COMMITTED IBM Subsystem Device Driver for AIX V52
devices.fcp.disk.ibm.rte
1.0.0.0 COMMITTED IBM FCP Disk Device
Ensure that the latest Microcode for Fibre Channel adapters has been applied.Use the lscfg command to determine the Microcode level of the FC adapter.
$ lscfg -vpl fcs0 | grep Z9
Updating all of our 100+ AIX systems, before migrating, was a sizeable task.However, once completed we were able to move to the new storage device withoutan issue.
The approach to migrating data from the old disk to the new disk was to employ theAIX Logical Volume Manager. There were two LVM utilities at the core of our datamigration strategy (mirrorvg and migratepv). Both commands can copy/move databetween disks while the system is running. Due to the very I/O intensive nature ofthese commands, they could impose a slight performance impact to I/O on thesystem. Therefore, it was determined that we would not perform a data migrationwhen the system was running peak (disk I/O) load. We would schedule these tasksduring relatively quiet periods.
The arrow (from hdisk0 to hdisk11) in Figure 2 represents the LVM mirroring (andmigratepv) process for data migration.
The mirrorvg command would be used to migrate the operating system (rootvg) fromthe old ESS to the DS8300. However, for the application/data volume groups, wechose to use the migratepv command. This would give us some level of control overhow much additional I/O activity we could unleash on the running system. Some ofthe migrating systems were in production and we did not want to flood the I/Osubsystem and cause unnecessary performance issues.
Obviously before we could start, our storage team had to first attach and configurethe new DS8300 into our existing SAN. Once this was completed, we worked withthe storage team to determine what type and how many LUNs we would require foreach of the AIX systems that were migrating to this new device.
Our planning also captured each LPAR's hostname, the existing AIX hdisk names,the existing SDD (vpath) configuration, the World Wide Port Name (WWPN) for eachFC adapter, the current FC adapter name (e.g. fcs0 and fcs1), the current LUNconfiguration and the proposed new LUN configuration.
With the storage carved up, we were able to assign the new LUNs to the existingAIX hosts and perform the migration. Figure 2 below shows the DS8300 isconnected to our SAN and a LUN from it has been allocated to an existing AIXLPAR. The LUN appears to AIX as a hdisk device (hdisk11). This disk has beenallocated to an existing volume group (rootvg).
Figure 2. IBM ESS to DS8300 storage migration - Dedicated I/O - LVM Mirror -Migration state
Prior to migrating, we performed a backup of the system (including a mksysb). Themigration could execute while the applications were running on the system.However, at some point after the migration, a reboot would be required. The systemboots from SAN disk. We will be migrating the system to a new SAN disk for theoperating system. To ensure that the boot disk has migrated successfully we mustmake sure that we can boot the system on the new disk. The downside is that if thisfails, we would need to restore the system from a mksysb backup.
In hindsight (5 years later!), I could have suggested we use alt_disk_install (on AIX5.2) instead of mirrorvg. The alt_disk_install command (now replaced by thealt_disk_copy command in AIX 5.3) can clone an existing rootvg onto another disk.Using this method would have provided a more efficient back out path. Fortunately,we never had to initiate a back out during any storage migration.
Before making any changes to the existing system, we documented the systemconfiguration. The current disk, LVM and SDD (vpath) configuration was alsocaptured. Historically, SDD configured vpath devices to provide MPIO to hdiskdevices. To keep my examples simple and more generic, I will refer to hdisk devicesrather than the vpath devices. You would typically work with vpath devices in a pureSDD environment. Storage vendors will present MPIO disk differently. Please keepthis is in mind when dealing with your storage devices and MPIO code.
When the new LUN had been assigned to the AIX host, the cfgmgr command wasexecuted to pickup the new DS8300 disk. I confirmed that the new disk wasdiscovered and the paths had been configured correctly.
; A disk type of 2107 confirms it is DS8300 storage.# lsdev –Cc disk | grep 2107# lspv# lsvpcfg
At this point, I could add the new DS8300 LUN (hdisk11) to rootvg with the extendvgcommand. Then, I used the mirrorvg command to create an exact copy of the dataon the new disk. Using AIX LVM commands, this process was straightforward.
The mirroring process can take some time. As a precautionary measure, I madesure that there was a new (secondary) dump logical volume (LV) on the new disk,that a new boot image was created, and the boot list contained both the old and thenew hdisks. If I needed to restart the system at this point in the migration, I could beassured that I could boot from either disk.
# bosboot -a -d /dev/hdisk11# bosboot -a -d /dev/hdisk0
# bootlist –m normal -o# bootlist -m normal hdisk0 hdisk11# ipl_varyon –i
Once the mirrors had synced (i.e. lsvg –l rootvg did not show any stale partitions), Iremoved the old hdisk from rootvg. First I had to unmirror rootvg (unmirrorvg) fromthe older disk. I also made sure that any active dump device on this disk wasremoved. I temporarily changed the primary dump device to /dev/sysdumpnull. ThenI removed the disk from the volume group and the AIX Object Data Manager (ODM),with the rmdev command.
# ps –ef | grep sync# lsvg –l rootvg | grep –i stale# unmirrovg rootvg hdisk0# sysdumpdev -Pp /dev/sysdumpnull# rmlv hd7; no output from the lspv command, means no LV data on disk.# lspv –l hdisk0# reducevg rootvg hdisk0# chpv –c hdisk0# rmdev –dl hdisk0
After making these changes, it was important that I re-create the boot image, checkthe boot list contained only the new hdisk and that the primary dump device was setcorrectly.
# bosboot -a -d /dev/hdisk11# sysdumpdev -Pp /dev/hd71# bootlist –m normal -o# bootlist -m normal hdisk11# ipl_varyon -i
With the operating system now residing on the new disk, I focused on migrating thedata volume groups. The storage team assigned new data LUNs to the host andprovided me with a list of the LUN ids. First I needed to identify the DS8300 diskswhich would be used for the data migration. Both the lspv and lsdev commands can
display information relating to hdisks on an AIX system.
; hdisk NOT ASSIGNED TO A VG# lspvhdisk12 none Nonehdisk13 none None; A disk type of 2107 is displayed for DS8300 disks on AIX.# lsdev –Cc disk | grep 2107
With the correct disks identified (hdisk12 and 13), I could add these disks (withextendvg) to the existing data volume group (datavg).
# extendvg datavg hdisk12 hdisk13
The data migration from the old disk to the new disk would be accomplished by theLVM command, migratepv. I had to select the source and destination disks for themigratepv operation. For example, the following commands would migrate data fromhdisk2 to hdisk12 and hdisk3 to hdisk13. At the end of the migration, both hdisk2and hdisk3 would be empty and all of their data would now reside on hdisk12 andhdisk13 respectively.
To confirm that all the data (logical volumes) for each of the old hdisks had beenmigrated, I ran the lspv command to list the contents of each disk. The command didnot return any output, confirming that the disks were indeed empty. It was now safefor me to remove the old disks from the volume group and remove them from theODM.
With the migration complete, I could now ask the storage team to reclaim the oldLUNs on the F20.
To ensure that the boot disk and boot list had been configured correctly, I wouldreboot the system to verify. This would also ensure that the newly migrated disks,volume groups, logical volumes and filesystems would continue to function asexpected after the migration. The desired end state for our dedicated I/O system hadbeen achieved. The old ESS storage could now be decommissioned and removedfrom the data centre.
Mixed vendor SAN storage migration with AIX and virtual I/O
In the next scenario, I’ll discuss how I performed a similar storage migration. Thedifference with this example is that I had to migrate from IBM to NetApp storageusing the virtual I/O server (VIOS).
Let’s review the current state (the state of the environment prior to migrating to thenew storage). Figure 4 below shows that there is an AIX LPAR, connected to a pairof VIOS. Both are connected to IBM DS8300 storage. The VIOS pair (hvio1 andhvio2) are serving the DS8300 LUNs as virtual SCSI (VSCSI) disk to the clientLPAR. The LPAR has a single volume group, rootvg for the operating system, on asingle disk. Other disks and volume groups also exist for application data but are notdepicted for simplicity.
The NetApp storage device has been attached to our existing SAN fabric. However,none of the VIOS or AIX systems are accessing it at this time.
Figure 4. DS8300 to NetApp storage migration - DS8300 VIOS - Current state
Our procedure document outlined how we would migrate an existing AIX LPAR,which currently uses hvio1 and hvio2 for all disk (DS8300) traffic. The LPAR and theVIOS reside on a Power6 570 (570-1). The new NetApp VIOS (hvio11 and 12) alsoreside on the same managed system and will be used for all virtual I/O traffic to/fromthe NetApp storage device. These details are captured during our planning process:
Client LPAR: lpar1
Managed system: 570-1
DS8300 VIOS pair (source): hvio1 and hvio2
NetApp VIOS pair (target): hvio11 and hvio12
The current VIOS and LPAR FC adapter, DS8300 disk, virtual adapter and virtualdisk configuration is also captured and used in the planning for the new diskassignments. For example, the following extract from our planning spreadsheetshows each LPAR, the current DS8300 disks assigned, the current VIOS, theexisting vhost/vscsi relationship, the new NetApp LUNs required, the new VIOS andthe new vhost to vscsi mapping for each LPAR. For examples, refer to Figures 5, 6and 7.
Before we could migrate to the NetApp storage array, our Storage administrator firstconfigured the LUNs that we require and prepared to present them to the NetAppVIOS, hvio11 and hvio12. We have provided them with the following information (ata minimum) for the allocation to take place:
Of course, we backup our LPAR before we make any changes to it, just in case.This includes performing a mksysb and savevg backup, followed by a file levelbackup with our corporate backup tool.
Prior to starting the migration, we deployed two new VIOS (hvio11 and 12),specifically for use with NetApp. During our design phase, we determined that itwould be a good idea to deploy the latest version of VIO server, version 2.1, for thebuild of the NetApp VIOS. The NetApp MPIO Host Attachment software is installedon both NetApp VIOS. Both VIOS will SAN boot from NetApp storage. You can referto Figure 8.
Figure 8. DS8300 to NetApp storage migration - Introduction of NetAppstorage and VIOS
That is, vscsi2 and vscsi3, where vscsi2 is connected to hvio11 and vscsi3 isconnected hvio12. Again, using DLPAR, we assigned a new virtual SCSI serveradapter (vhost) to hvio11 and hvio12. I also ensure that we update the VIOS andLPAR partition profile (on the HMC) with these newly created virtual adapters.
We then mapped the LUN (as Virtual Target Device or VTD) to the client LPAR oneach VIOS and presented it to the LPAR. The disk (hdisk11) appears as a virtualSCSI disk on client. This new hdisk has been included in the existing root volumegroup (rootvg, also shown in Figure 8).
Once the LUN has been assigned to the NetApp VIOS, we perform our standardVIOS disk mapping to the client LPAR, that is, with mkvdev and lsmap to create andverify the disk has been assigned to the correct LPAR.
On the client LPAR, we ran cfgmgr to discover the new hdisk. We verify that the newdisk and paths are available. There should be two paths to the disk, one via vscsi2(hvio11) and one via vscsi3 (hvio12).
To migrate the data from the old to the new disk, the method is similar to theprevious example. Except this time we will use mirrorvg (instead of migratepv) for alldata migrations (including the data volume groups). These systems are eitherdevelopment or test systems. While we do not want to impact performance, some ofthese systems are relatively idle or have very low user numbers, so we can addadditional I/O activity without too much of a performance concern. Using mirrorvgalso simplifies the migration as we do not have to run a migratepv command againsteach of the source/target disks.
First, we add the new disk (hdisk11) to rootvg. Then we mirror the volume group withthe new hdisk. This mirroring process takes place in the background. You can referto Figure 9.
As a pre-cautionary measure, we verify that rootvg is now mirrored, include the newdisk in the boot list and create a new boot image for the mirrored volume group. Justas before, if I need to reboot the LPAR at this point, I can rest assured that I can
# bosboot -a -d /dev/hdisk11# bosboot –a –d /dev/hdisk0# bootlist -m normal hdisk0 hdisk11# bootlist –m normal –o# ipl_varyon -i
With rootvg mirrored successfully, we can now unmirror again, remove the old disk(hdisk0) from the volume group and remove it from the ODM. Again, we also need tocheck the boot list and recreate the boot image for this now non-mirrored volumegroup.
With our AIX OS now residing on the new hdisk (the NetApp LUN), we can nowmigrate the data volume groups. First, we identify the NetApp LUNs which will beused for the data volume group migration. The data LUNs had already beenassigned to the VIOS and mapped and configured on the client LPAR.
Again, we use extendvg to add the NetApp disks to the data volume group (datavg).
We mirror the volume group from the DS8300 disk (hdisk2) to the new NetApp disk(hdisk12). The mirroring process is set to run in the background with the –S flag. Toensure the volume group and logical volumes are synced, before proceeding, wecheck that there are no stale physical partitions (PPs) and that there are no LVMsync processes still running in the background.
# mirrorvg –S datavg hdisk12# ps –ef | grep lresync# lsvg –l datavgdatavg:LV NAME TYPE LPs PPs PVs LV STATE MOUNT POINTdatalv jfs2 96 192 2 open/syncd /dataloglv01 jfs2log 1 2 2 open/syncd N/A
With the mirroring process complete, we unmirror the volume group from theDS8300 storage (hdisk2).
# unmirrovg datavg hdisk2
To verify that there is no longer any data on the DS8300 hdisks, the lspv commandis run and should not return any output.
# lspv –l hdisk2
Now we can remove the DS8300 hdisks from the data volume group using reducevgand remove the disk from the ODM.
# reducevg datavg hdisk2# rmdev –dl hdisk2
As this is a virtual I/O environment, there are some additional steps we must executebefore we can hand back the DS8300 LUNs to the storage team.
First, we must remove the device mappings for the DS8300 disk from the DS8300VIOS (hvio1 and 2). We should also take note of the DS8300 LUN id (from pcmpathquery device) so we can provide the storage admin with a list of LUN ids that can bereclaimed. In the following example, we check the disk mappings for vhost2 andverify the backing device hdisk number. We then enter the OEM VIOS environmentto run the pcmpath utility and obtain the LUN ids associated with each hdisk. Next,we run the rmvdev command to remove any virtual target device (VTD) mappingassociated with the hdisk (e.g. vtscsi2). And finally we remove the hdisk from theODM on the VIOS. The old DS8300 LUNs can now be reclaimed by the storageteam.
The virtual SCSI server adapters (i.e. vhost2) are removed from each VIOS ($rmdev –dev vhost2). And using DLPAR, we remove these virtual adapters from theLPAR definition. The VIOS partition profile is also updated to reflect the removal ofthese virtual devices.
The original virtual SCSI adapters on the client LPAR, vscsi0 and vscsi1, are alsoremoved now using rmdev and DLPAR (the LPAR partition profile is also updated).
Again, at this point, you would reboot the LPAR once you are satisfied that all datamigrations have completed successfully. Verify correct system operation after thereboot (volume groups online, logical volumes open/syncd, filesystems mounted,etc.).
With the changes completed successfully we now document the systemconfiguration after the migration. At this point, we have reached our desired endstate. The AIX system is now using NetApp storage via our new VIOS. You can referto Figure 10.
The DS8300 (in this case) will remain in our environment. Likewise, the VIOS usedto serve DS8300 storage to client LPARs, will also remain. As it has beendetermined that the DS storage will be used for AIX systems requiring a high level ofperformance and availability. While all other non-critical AIX test systems will utilizethe NetApp device.
AIX storage migrations can be time consuming and tricky. Without the latest storagevirtualization technology, such as IBM's SVC, the task requires several phases,stages and tasks.
Fortunately, the AIX LVM is a very mature, stable and robust tool. It can greatlyempower us when faced with these challenges.
Using LVM, we can reduce the outage required for migrating the data. Having theability to move data between source and targets disks, while the system/applicationsare still running, is a huge benefit in an operational computing environment.
As always, I strongly recommend that you test your procedures in a non-productionenvironment before attempting a storage migration on a production system.
If you’ve found this article interesting, then there are other LVM commands whichare also worth researching, such as replacepv, redefinevg and recreatevg. Youmay also find the LVM hot spare policy of interest. Review the Resources sectionbelow to find out more.
Chris GibsonChris Gibson is an AIX systems specialist located in Melbourne,Australia. He is an IBM CATE, System p platform and AIX 5L, and aco-author of the IBM Redbooks publication, "NIM from A to Z in AIX5L."