Svc Mirror Final Copy

SVC Global Mirror

SVC GLOBAL MIRROR A practical review of important parameters and their impact on SVC.

Gonzalo Fuentes Mark Bruni

ATS Storage Center

Gaithersburg, MD

Page 1 of 19 Copyright IBM Corp. 2010 All rights reserved.

SVC Global Mirror

Overview Remote replication (or remote copying) is an important component in a well designed Storage solution. All large disk subsystems provide remote replication between Primary LUNs’ in one subsystem and Secondary LUNs in a similar subsystem. The SAN Volume Controller (SVC) provides the same kind of function between Primary VDisks in one SVC cluster and Secondary VDisks another. The SVC supports two kinds of replication between clusters: Metro Mirror - designed to support metropolitan distances (links between sites are at SAN

speeds) using synchronous copy services, ensuring that updates are committed to both the Primary and Secondary VDisks, (the two VDisks are synchronized) before sending confirmation of the completion to the server. For longer distances, this will introduce long latencies on write operations, potentially slowing down production.

Global Mirror - designed to support longer distances using asynchronous copy services, sends an acknowledgement to the server before the SVC receives a confirmation of write completion, from its remote SVC partner. This is generally done over WAN links with much less bandwidth than you seen in a SAN fabric.

There are many areas that require attention when planning to implement an SVC Mirror solution, and they are extensively documented in a couple of excellent REDBOOKS, which are: SVC Advanced Copy Services SG24-7574 SVC Best Practices and Perf. Guidelines SG24-7521 This paper will look at how WAN bandwidth affects Global Mirror processing, and parameters available to help manage the use of this bandwidth. This paper does not discuss how to “size” a Global Mirroring implementation. It does not discuss how to determine if you need a bigger SVC (more nodes) for Global Mirror, or how to determine what kind of bandwidth is needed between sites. Rather, it will show how traffic volumes and parameter settings affect the overall performance of a given implementation.

Global Mirror Processing There are two main aspects to mirrors: synching or re-synching, and “normal” running after synchronization.

Synching Up

When a global mirror is started between two VDisks, there is an initial period of copying from the Primary to the Secondary to get the two VDisks in synch. This is often referred to as Initial Synching. If a mirror is paused for some reason (say, due to a network outage), during this time changes to either the Primary or the Secondary are stored in bitmaps to keep track of what has changed. When the mirror is restarted, these bitmaps are used to choose which parts of the Primary need to read and copied to the Secondary. This is typically called a Re-synch. During an initial synch, the Primary is read sequentially, block by block, and the data is sent to the Secondary to be written there. When re-synching, a bit map is used to determine which parts of the VDisk have changed and need to be recopied.


SVC Global Mirror During a synch or re-synch, applications are free to access the Primary VDisk. Reads to the Primary are handled just like any other read. Writes to the Primary must be accounted for at the Secondary VDisk. When a Write comes in to a Primary VDisk, there will be some parts of the Primary that have already been copied to the Secondary, and other parts that are yet to be copied. If a Write operation is to a part of the Primary that has not been copied yet, then the Write is merely done to the Primary. If that part of the Primary has already been copied, then that Write must be replicated over to the Secondary. This is not considered part of Synchronization, but is actually “normal” mirror processing for those parts of the VDisk that are already synchronized. Care must be taken to manage this process.

After Synch

Once a mirror is fully synched, all Writes to the Primary are replicated to the Secondary. No other traffic (except a heartbeat and some handshaking) is necessary between the clusters. However, this Write traffic generated by applications can be significant.

Bandwidth Considerations for Global Mirror

To reliably run the Global Mirror, you must ensure that the bandwidth available between sites is not overrun except for short bursts. Without any control, synching operations might run at disk speed, which is typically much higher than the available bandwidth. Once mirrors are in synch, write traffic must not be so heavy as to overrun the link. What you normally see when a Global Mirror implementation overdrives the bandwidth between sites, is that processing slows down for a little while, and then the SVC pauses a mirror to try and reduce traffic. This may be enough. If not, the SVC will pause another mirror, and another, until the traffic is low enough for the link(s) to handle it. Finally, the links between sites may not be dedicated to mirror traffic. There may be other SAN traffic on the links, or there may be networking (IP) traffic between sites, all sharing the same bandwidth. In this case, the amount of bandwidth deemed to be available to mirroring must be decreased to reflect that bandwidth the other traffic is using.

Summary and Conclusions The Bandwidth parameter on SVC Partnership definitions does indeed limit synch traffic as expected. It is usually recommended to set the Bandwidth parameter to 30-50% of the pipe capacity between sites, at least initially. This parameter can be changed at any time and takes immediate effect, so best to “play it safe” at first. Mirrors that are stopped have negligible effect on local processing (the Primary side of the mirror). If Write traffic exceeds the WAN bandwidth available for a sufficiently long time, things will slow down, the SVC will detect this slowdown, and after 5 minutes (default) it will stop at least one mirror. Once enough mirrors have stopped to avoid overdriving the link, local processing picks back up and gradually gets back to “normal” levels.


SVC Global Mirror When planning for Global Mirror solution, traffic flows for the VDisks to be mirrored should be measured in some way, so that peaks can be found in the Write traffic. Enough WAN bandwidth needs to be provisioned to handle these peaks, or some mirrors will stop when they hit. (If you can anticipate the peak in write traffic—say a weekly database re-org—you may stop the mirrors on purpose, wait till the peak is over, and then restart the mirrors after the peak, but this requires careful planning and is beyond the scope of this paper.) When “protecting” WAN bandwidth from synch activity (via the Bandwidth parameter in the Partnership definition), remember that you must consider the need to simultaneously handle replicated Writes for all mirrors already synched, as well as any other traffic that may be on the link (such as other SAN traffic or normal (IP) network traffic). No matter how carefully you plan, there will be times when mirrors stop for one reason or another. A full network outage would pause all mirrors. If there are multiple links between sites, an outage on one link might reduce available bandwidth enough to cause some mirrors to suspend. Various human errors could cause mirrors to suspend as well. In all cases, if there have been changes to the PrimaryVDisk(s) during the time the mirror is paused – normal processing at the Primary Site would cause this – when the mirror is restarted, during the re-synch period the Secondary becomes Inconsistent. Changes are just “blasted” across based on the bitmap, and the Secondary is completely unusable until the re-synch finishes. Should there be a disaster at the Primary during this re-synch period, then the Primary is “gone” and the Secondary is unusable. For this reason it is highly recommended that Flashcopies (Point-In-Time copies) of the Secondary be taken, at a minimum before a mirror is restarted (or verify that you have a usable copy somewhere prior to restarting the mirror). Flashcopying a running Secondary is possible, or you can Flashcopy the Secondary while the mirror is paused (before restart), but in these instances, the Secondary (of an active Primary) is only “crash consistent,” sometimes referred to as a “power outage copy.” It is in a state that the Primary might have been in had it taken an outage at a particular time. Write I/O that is still in the host (application cache, file system cache) has not made it to the Primary and so of course is not at the Secondary either. Depending on the application, this “power outage copy” may be good enough. In other cases, it will not, and you will want to produce a “shutdown copy.” This may mean that you quiesce the appropriate application and unmount file systems, or it can be putting a database into “hot backup” or “backup mode” or something similar to that. (NOTE: If this is necessary, then these actions are already being taken to prepare for local backups at the Primary end. Whatever is done to prepare for backups at the Primary end, should be done prior to taking Flashcopies at the Secondary end. The sequence is: 1. Take the appropriate action (quiesce app, etc. or put database in particular state) 2. Manually pause the mirror – to flush all the leftover writes to the Secondary 3. One the mirror is stopped synchronized; bring the app back up to normal processing. All

changes to the Primary will merely be reflected in the local bitmap. Nothing more goes to the Secondary since the mirror is stopped.

4. Flashcopy the Secondary. This gives you a beautiful “shutdown copy” that you can use to test failover procedures, and protects you during re-synch.

5. Restart the mirror.


SVC Global Mirror Regardless of which approach you use, some sort of Flashcopy procedure at the Secondary is required to protect you from an outage at the Primary happening during a re-synch off the Secondary. In conclusion, an SVC Global Mirror solution can offer unique advantages, such as the ability to mirror between un-like storage devices, even from different vendors. But careful planning is required to make sure it is sized and configured correctly, to make sure that it is a solid component of any Disaster Recovery strategy. The information, tools and documentation (“Materials”) are being provided to IBM customers to assist them with customer installations. Such Materials are provided by IBM on an “as-is” basis. IBM makes no representations or warranties regarding these Materials and does not provide any guarantee or assurance that the use of such Materials will result in a successful customer installation.


SVC Global Mirror

Test Environment The following diagram shows the lab components that were used to simulate a Customer environment with two locations: Main site (LOCAL) and Recovery site (REMOTE): Two SVC Clusters (SVC5 and SVC6, each with 4 nodes) running 5.1.0.2 code One Windows host connected with two 2 Gb HBAs, running IOMETER to simulate the

server workload. One FCIP (1 Gb) link between the SANs, connected to an Empirix simulator, used to

constrain the link (e.g. latency and bandwidth) between the SVC Clusters, reflecting restrictions of the real world.

One local DS8300 with two 2 Gb Fibre Channel connections to SVC5 One remote DS8300 with two 2 Gb Fibre Channel connections to SVC6 Four SVC Mirror relationships in a single Consistency Group


SVC Global Mirror

TEST 1) SYNCHRONIZATION OF 4 VDISKS (25GB each) USING A BANDWIDTH PARAMETER OF 200 MB/s IN THE DEFINITION OF THE SVC PARTNERSHIP. No server-generated I/O was used in this first test, so the only traffic is replication traffic.

We had a link between the SVC5 (Local) and SVC6 (Remote) of 1 Gb (approx. 100 MB/s) Notice on the Cisco Fabric manager display, the FCIP link is totally saturated. (We are looking at the switch ports at the Remote Site; hence all the traffic is under the Receive column. The top row is the actual GigE port on the switch’s IP blade that the Empirix is attached to. The second row is the internal TE-port used by that IP blade.)

It took approximately 18 minutes to synchronize 4 Vdisks (25GB each). 102400 MB / 1098 Sec = 93.3MB/s, about as much as we can expect from a 1Gb Ethernet link.


SVC Global Mirror TEST 2) SYNCHRONIZATION OF 4 VDISKS (25GB each) USING A BANDWIDTH PARAMETER OF 60 MBs (OC12) IN THE Partnership Definition Once again, no server-based I/O was involved.

This time, the 1Gb link showed around 68% utilization. Synchronization took approximately 28 Minutes. 102,400 MB / 1658 Sec = 61.8 MB/s.


SVC Global Mirror

TEST 3) SYNCHRONIZATION OF 4 VDISKS (25GB each) USING A BANDWIDTH PARAMETER OF 15 MBs (OC3) IN THE PARTNERSHIP DEFINITION Again, no host-based traffic.

This time, the 1Gb link only showed 18% utilization. Synchronization took approximately 113 Minutes. 102400 MB / 6780 Sec = 15.1 MB/s.


SVC Global Mirror These 3 tests show the impact of the Bandwidth parameter of the SVC Mirror Partnership. This parameter should be coded taking into consideration the amount of Bandwidth the SVC Mirror may use while synchronizing. Remember that the “BANDWIDTH” parameter only affects Synchronization traffic. (Initial and other re-synchronizations). In this last test, once the data has been copied via synch, updates to that data (writes from the host) are unrestricted. NOTE: There is a maximum synch bandwidth per mirror that defaults to 25 MB/s. In our example we had 4 relationships for a max of 100 MBs, so even if we had used a higher capacity link between sites and increased the Bandwidth parameter to more than 100 MB/s, that restriction would still would have applied, and we would not have gotten more than 100 MB/s in synch traffic.. Starting with SVC 5.1, we have a CLI command to modify that 25MB/s limit:

svctask chcluster –relationshipbandwidthlimit 75 This example would increase each mirror relationship maximum synch rate to 75 MB/s. However this maximum is only reached if the aggregate is not constrained by the Bandwidth parameter in the Partnership definition.


SVC Global Mirror

TEST 4) Workload of 100% SEQ. WRITE using 32K blocks. Using the Empirix box in our lab, the link between SVC5 (Local) and SVC6 (Remote) was restricted to a speed of OC3. Then we started an IOMETER load with Write traffic much larger than the SVC Mirror OC3 bandwidth capacity to show the SVC’s reactions when there is not enough bandwidth to replicate the write traffic. At some point, the space holding the in-flight writes will fill up, and everything will slow down, even production traffic. The SVC will detect this slowdown and “watch” for the amount of time specified in the Link Tolerance parameter, which defaults to 300 seconds (five minutes). After that, it will pause the Consistency Group causing the most traffic (error code 1920) and re-assess. (We only have the one Consistency Group running, so that’s what will stop.) Below is a picture of the Link Tolerance parameter from the SVC GUI.

To establish a baseline, we deleted all the mirror definitions and ran the IOMETER with just normal local processing. Using one “Worker” per VDisk, we see about 227 MB/sec of Write activity to the 4 VDisks, with response time less than a millisecond.


SVC Global Mirror Doing the same thing with the mirrors defined and in a Consistent Stopped state, we got essentially the same results, showing that stopped mirrors do not affect processing at the Primary end.

Then we stopped IOMETER, started the mirrors, and waited until they synchronized .


SVC Global Mirror With the SVC Mirror synchronized, the same IOMETER workload was started at around 11:40AM, notice that the total Write MB/s (16.81MB/s) is much lower than when the mirror was not enabled, and the response time has increased considerably to 7.4ms:

The Cisco Fabric Manager shows about 19% utilization on the Gig E link, (restricted by the Empirix to run OC3), in other words, the OC3” Wan link” is full.


SVC Global Mirror The expected slowdown was very rapid; the error log shows a “1920” message about 10 minutes after the IOMETER traffic was started.

When the Mirror stopped, the host throughput got significantly better, we see the IOMETER throughput has increased significantly by 11 55AM (approximately 5 minutes after the mirror stopped.


SVC Global Mirror The Cisco stats show the host using both HBA ports at 61%:

At 12:00 it traffic had increased from 121 MBs to 156 MBs, still lower than the base line 224 MBs. This is probably due to time spent de-staging cache, as the SVC and the backend storage “catch up” to the heavy IOMETER workload. We did observe the traffic increasing steadily over time.


SVC Global Mirror TEST 5) Workload of 10% SEQ. WRITE and 90% SEQ. READ using 32K blocks. A mixed workload of mostly reads is more common than a 100% sequential writes load. First we started IOMETER with the mirrors stopped to get a base line. The 21 MB/sec of Write traffic is only slightly higher than the 15+ MB/sec of bandwidth available.


SVC Global Mirror Then we stopped the IOMETER, re-synched the mirrors, and then restarted IOMETER. The Write traffic is reduced, and the Write response time is higher than with the mirrors stopped.

After 2 minutes, response time increased to 3.4 ms, and the Write traffic has decreased.


SVC Global Mirror After 4 minutes, things have gotten even worse.

Since the IOMETER writes rate are only a bit higher than the “WAN” capacity it took longer to stop the mirror—around 14 minutes for the link to get oversaturated and the mirrors to stop.


SVC Global Mirror


Once the mirrors are stopped, the Write traffic increase (and response time decreases) as seen before.

These examples illustrate the effects of the Bandwidth parameter, and the behavior expected when the link(s) between sites are over-driven. While important, these are only a few of the issues that should be reviewed when planning of a Global Mirror implementation.

Svc Mirror Final Copy

Documents

copyright ibm corp

power outage copy

link tolerance parameter

svc global mirror

global mirror

response time

san traffic

wan bandwidth