Vvr Concepts

Understanding VERITAS Volume Replicator

Understanding VERITAS Volume Replicator By

Jim Senicka VERITAS Technical Product Management

VERITAS Software Corporation

February 2005

VERITAS ARCHITECT NETWORK

1


TABLE OF CONTENTS

Table of Contents ............................................................................................................... 3

Overview ............................................................................................................................. 4

Understanding the need for replication ............................................................................ 4

Replication Concepts.......................................................................................................... 5 Synchronous Replication.......................................................................................................... 5 Asynchronous Replication........................................................................................................ 6 Consistent VS. Current............................................................................................................. 6 Write Order Fidelity................................................................................................................. 6

Technical requirements of a replication solution............................................................. 7

VVR Overview and Components ....................................................................................... 7 Replicated Volume Groups ...................................................................................................... 7 Storage Replicator Log............................................................................................................. 8 Rlinks ......................................................................................................................................... 8

VVR technical details......................................................................................................... 8 Operational Modes and data flow ........................................................................................... 8

Synchronous.......................................................................................................................................... 8 Asynchronous ....................................................................................................................................... 9

Initializing Secondary Systems ................................................................................................ 9 Empty.................................................................................................................................................... 9 Over the wire (Autosync)...................................................................................................................... 9 Local mirroring ..................................................................................................................................... 9 Hot backup and Checkpoint.................................................................................................................. 9

Recovery after problems......................................................................................................... 10 Secondary/Network outage ................................................................................................................. 10 Secondary Failure ............................................................................................................................... 10 Primary Failure ................................................................................................................................... 10

VVR Role Changes.................................................................................................................. 10 Primary Migration............................................................................................................................... 10 Secondary Takeover............................................................................................................................ 10

Using the Secondary System .................................................................................................. 11 VVR in the customer environment .................................................................................. 11

Understanding Bandwidth Needs .......................................................................................... 11 Latency Considerations .......................................................................................................... 13 Effects of VVR on Application Performance........................................................................ 13 Using Asynchronous Replication to Decouple Application Latency .................................. 13

About the Author.............................................................................................................. 15


3


OVERVIEW VERITAS Volume Replicator can effectively and efficiently provide protection from disaster scenarios by replicating data to another location(s). VERITAS Volume Replicator allows organizations to replicate their data between any storage devices, over a standard IP connection, and across any distance for the ultimate in disaster recovery protection. This document provides current and potential VERITAS Storage Foundation users with a solid under-standing of the VERITAS Volume Replicator (VVR) product. It is intended to provide an architec-tural/technical overview, as well as a product capability understanding. UNDERSTANDING THE NEED FOR REPLICATION In today’s business environment, information processing system reliance continues to grow on an al-most daily basis. Information systems, which once aided a company in doing business, have become the business itself. As companies become more reliant on critical information systems, potential busi-ness disruption due to an information system loss increases. Over the past several years, numerous technologies have emerged to provide solid protection against individual component systems failure within data centers. State of the art products providing storage management, data protection (backup and recovery) and application clustering provide an excellent defense against component or system failures at the local level. Just as the level of protection available at the local level has grown, so has the reliance on the informa-tion systems being protected. Many companies now realize that local protection is no longer adequate protection. Loss of a complete data center or information processing facility would so greatly affect business capability that protection must be established at the data center level. This loss can stem from simple issues such as power or cooling failure, to natural disasters such as fire and flooding, to acts of terrorism or war. Many companies have implemented significant Disaster Recovery (DR) schemes to protect against the complete loss of a facility. Complete plans exist to recover voice and data capability in a remote loca-tion. One common issue among many DR plans is the information processing recovery plan. For many years companies have been taking regular data backups at the primary data center, then duplicating these tapes on a regular basis for shipment offsite to the DR facility. There is a significant problem with DR solutions based on offsite backup tape copies. This plan, con-sidered more than adequate a few years ago, does not consider the shear reliance on today’s informa-tion systems. Companies not only need information systems online, but also need accurate, up-to-the-minute information. Imagine a typical DR scheme built around offsite protection using duplicate tapes. At regular intervals, complete data backups are created at the local site. When the backups complete, copies of these tapes are made and sent to the DR site. This typically occurs, at best, every day and, worst case, up to a week between shipments. This means that following a primary data center disaster, data processing can resume at the DR facility with data a minimum of 24 hours out of date! Companies then face an issue where information systems are restored after some number of hours or days, de-pending on the time required to load all backup data, and, at that time, can begin processing based on data that was out of date when the disaster occurred. In many businesses, such as electronic com-merce, online banking and others, this is categorically unacceptable. To examine this problem closer, refer to the diagram below. Recovery Point Objective (RPO) repre-sents the “data currency” at the backup site. This point details the amount of data that is lost if a disas-ter occurs. At the center, we have the disaster. (This can be any type of disaster, including system loss due to power, flooding, cooling or any other problem). Recovery Time Objective (RTO) represents the time it takes to restore online services, using the remote site data.


4


The two sides of the disaster represent a unique set of problems. RPO represents lost data. All record of any business transaction during this time is lost. Imagine the case of selling a car in which the lost data identifies customers that have not paid for vehicles they have already received. Good for the cus-tomer, bad for the business. At the same time, this could be a customer paying for a vehicle, but their payment record is lost. This is bad for the customer and future business. RTO is a separate problem because it represents the time required to get back in business. RTO represents opportunity revenue, as well as a potential major hit in customer confidence and brand acceptance.

Synchronous Replication / Mirroring

Periodic Replication Asynchronous

Replication

Tape Backup

Tape Restore

ClusteringManual Migration



Asynchronous ReplicationAsynchronous ReplicationAsynchronous Replication

Tape Backup Tape Backup

Tape Restore Tape Restore

ClusteringManual Migration Manual Migration

RReeccoovveerryy PPooiinntt RReeccoovveerryy TTiimmee

Mins Days Hrs Secs Wks Days MinsHrs Wks Secs

d a t a l o s s d o w n t i m e

Ideally, we need to shrink the RPO and RTO windows as much as possible within budgetary con-straints. In a perfect world, RPO and RTO are zero. Using replication, we can maintain a current data copy at a remote site that is not hours or days behind, but minutes, seconds or, best of all, no time be-hind. Using replication can effectively remove the RPO portion of the problem. Using VVR, and VERITAS Cluster Server together to create a wide area failover solution, we can reduce RTO to the absolute minimum to start up the application at the remote side. This is no longer a case of Disaster Recovery, but actually Disaster Tolerance. Disaster Tolerance is effectively extending High Availability all the way out to the data center level and creating a Wide Area High Availability scheme. REPLICATION CONCEPTS Replication is a technology designed to maintain a duplicate data set on a completely independent storage system, possibly at a different geographical location. In most cases, this duplicate data set is updated automatically as the master, or primary, data set is updated. This differs from a tape backup and restore method, as replication is typically completely automatic and far less labor intensive. Repli-cation is typically run in one of two modes.

• Synchronous replication ensures updates are posted at the primary and all configured secondaries prior to returning the write response to the application.

• Asynchronous replication immediately returns write completions to the calling applica-tion and queues the write data for later transmission to the secondaries.

Different technologies from various vendors address issues such as persistently queuing write data for asynchronous writes and maintaining the ordering of those writes in different ways. The following sec-tion explains various replication concepts in more detail. SYNCHRONOUS REPLICATION Synchronous replication ensures that a write update has been posted to the secondary node(s) and the primary before the write operation completes at the application level. This way, in the event of a disas-ter at the primary location, data recovered from any surviving secondary server is completely up-to-date


5


because all servers share the exact same data state. Synchronous replication produces full data cur-rency, but may impact application performance in high latency or limited bandwidth situations. Syn-chronous replication is most effective in application environments with low update rates, but can be deployed in write-intensive environments where high bandwidth, low latency network connections aravailable.

e

SYNCHRONOUS REPLICATION pdates are written at the primary, and queued

ADuring asynchronous replication, application u for for-

rit-

rk

synchronous replication with adequate available bandwidth can help remove latency from a replica-

ONSISTENT VS. CURRENT s all the latest changes made at the primary processing site.

• Synchronous mode guarantees that the data on the secondary is up-to-date, at the

• arantee data currency, but does provide maximum .

ata is consistent if the system or application using it can be successfully restarted to a known, useable

eally we want the secondary to be an exact duplicate of the primary. If this is not possible, we need

RITE ORDER FIDELITY covery scenario, data written at the secondary must faithfully track

warding to each secondary host as network bandwidth allows. When the writing application experi-ences temporary surges in update rate, this queue may grow. Unlike synchronous replication, the wing application does not suffer from the response time degradation caused by each update incurring the cost of a network round trip. During periods when the update rate is less than the available netwobandwidth, this queue drains faster than it grows, allowing the secondary data state to catch up rapidly with that of the primary. Ation solution, but still provide near real-time updates. Application write operations are immediately ac-knowledged and updated data is sent to the remote site at nearly the same time. During an actual pri-mary site outage, only those transactions “on the wire” are lost. CData that is current or up-to-date containFor example, if you are replicating a database, the most recently committed transaction is available onthe secondary. Whether the data on the secondary must always be current, or not, is a business deci-sion. It is a consequence of choosing between synchronous and asynchronous replication.

cost of application performance Asynchronous mode does not guapplication availability and the ability to utilize more cost effective telecommunications

Dstate. For example, if the data being replicated is used by a database, the data is consistent if the data-base can be started and recovered to a useable state with zero data corruption. If the data contains a file system, the data is consistent if the file system check utility can be run and recover with no file sys-tem corruption. Idthe secondary to be an exact duplicate, minus some known amount of data updates. For example, in a database environment, one or more of the most recently committed transactions on the primary might be missing, but the secondary is in the exact state the primary was in prior to those transactions being applied. This means data must be applied to the secondary in exactly the same order it is applied at theprimary. Data blocks cannot be applied out of order, nor can partial updates be allowed. In order to pro-vide a consistent data set at a secondary, the replication solution must provide write order fidelity, ex-plained below.

WTo use the secondary in a disaster rethe primary. It can be behind in time, but must be a consistent image of the primary at some point in the past. This is called write order fidelity. Without write order fidelity, there is no guarantee a secondary will have consistent recoverable data. In a database environment, both the log and data spaces of a data-base management system both update in a fixed sequence. The log and data space are usually in dif-ferent volumes, and the data itself can span several additional volumes. A well designed replication solution needs to safeguard write order fidelity consistently. This may be accomplished by logically


6


grouping data volumes so the update order in the group is preserved within and among all secondarvolume copies.

y

ardware level replication solutions typically lack the ability to maintain write order fidelity when running

-

the user wishes to remove latency penalties imposed by synchronous replication, they lose recover-

, t

• rovide an up-to-the-second copy, or as near as possible, without

• support a secondary site(s) in

e

• tion must provide a “consistent” primary image at the secondary, regard-

VR is a logical extension of VERITAS Volume Manager, found in VERITAS Storage Foundation. It d

n -

VR is architecture-independent, which means it can replicate data between any storage platforms p-

REPLICATED VOLUME GROUPS

Hin asynchronous mode. This is because they lack a persistent queue of writes that have not yet been sent to the secondary. If a replication solution does not log each write in the order it is received, it is impossible to maintain write order fidelity in asynchronous mode. Because of this, VVR necessarily logs each write in the order it is received. Other approaches that do not do this cannot genuinely provide asynchronous support, as they only track changed writes. Ifability on the remote site (rendering the DR copy essentially useless). TECHNICAL REQUIREMENTS OF A REPLICATION SOLUTION

• Architecturally, a complete solution must provide an exact copy of all data at the primary siteincluding database files as well as any other necessary binary and control files. This data musbe accurate and recoverable. The replication solution must punduly effecting primary application operations. This means the replication solution must not inject unacceptable latency into primary application I/O traffic. The replication solution must be capable of being configured toan acceptable location. This means different businesses, operating in different areas have dif-ferent geographical requirements for disaster protection. The replication solution must operate at distances of hundreds or thousands of miles without adding undue cost or complexity. This means the replication solution MUST provide asynchronous support, over a long distance, without additional high cost items such as communication converters or additional disk spacfor “staging” data. The replication soluless of choosing Synchronous or Asynchronous replication.

VVR OVERVIEW AND COMPONENTS Vallows VERITAS Volume Manager volumes on one system to be exactly replicated to identically-sizevolumes on another system over an IP network. For example, an Oracle database may use several different volumes for various tablespaces, redo logs, archived redo logs, indexes and other storage. Each component is typically stored in a VxVM volume, or multiple volumes. The database may also use data stored in VERITAS VxFS file systems, created in these volumes for better manageability. VVR can provide an exact duplicate of these volumes on another system, at another site. The data ithe secondary volumes is always kept consistent, meaning the application can use the data as it is written. This requires the secondary copy is a faithful copy in terms of what volumes are written, in what order. This is very important in a DR mode, as the copy must be capable of being brought up as soonas possible. Vfrom any manufacturer. For example, expensive enterprise storage arrays using RAID 1+0 can be relicated to lower cost arrays at the remote site using more economical RAID 5. The only requirement is matching VxVM volume sizes on each side. Replication occurs over a standard IP link, removing the need for expensive proprietary network hardware.


7


VVR extends the concept of a Disk Group and provides the concept of a Replicated Volume Group ven VxVM Disk Group, configured for replication to one

).

among the volume updates. Write ordering is strictly

(RVG). An RVG is a volume subset within a gior more secondary systems (up to 32 secondary systems can be configured to replicate one primaryVolumes that are associated with an RVG and contain application data are called data volumes. Data volumes are replicated Volume Manager volumes and are distinct from Storage Replicator Log (SRL) volumes, which are described later. The RVG data volumes are under the control of an application, such as a Database Management Sys-em, that requires write-order fidelity t maintained

ll data writes destined for volumes configured for replication are first persistently-queued in a log plements the SRL at the primary side to store all changes for

n RLINK is a VVR Replication Link to a secondary RVG. Each RLINK on a Primary RVG represents ication link from the Primary RVG to a corresponding Secondary RVG. An RLINK on a

NICAL DETAILS PERATIONAL MODES AND DATA FLOW

are then sent, in parallel, out all config-ive an acknowledgement from their secon-

within an RVG during replication to ensure that each remote volume is always consistent, both inter-nally and with all other volumes of the group. Each RVG can have a maximum of 1023 data volumes. An RVG can be a primary or a secondary. A primary RVG receives write data from applications on the ost it is running on and forwards that data to configured secondary RVGs over rlinks (described be-h

low). A secondary RVG receives data from its configured primary and writes it to the proper volumes. A secondary RVG does not imply a secondary system. A system can be a primary for some RVGs and secondary for others. This allows one site to be used as a hot standby for another, and visa versa.

STORAGE REPLICATOR LOG Acalled the Storage Replicator Log. VVR imtransmission to the secondary(s). The SRL is a VxVM volume configured as part of an RVG. The SRL gives VVR the ability to associate writes to specific volumes within the replicated configuration in a spe-cific order, maintaining write order fidelity at the secondary. All writes sent to the VxVM volume layer, whether from an application such as a database writing directly to storage, or an application accessing storage via a file system are faithfully replicated in application write-order to the secondary.

RLINKS Athe communSecondary RVG represents the communication link from the Secondary RVG to the corresponding Primary RVG.

VVR TECHOSynchronous In Synchronous mode, all data writes first post to the SRL, and

s rlinks. When all Synchronous rlinks receured synchronoudaries, the application is acknowledged. The secondary acknowledges receipt as soon as the full writetransaction (as sent by the application on the primary) is received into VVR kernel memory space. Thisremoves actual secondary data volume writing from the application latency. The primary tracks these writes in its SRL until a second acknowledgement is received from the secondary, signalling that the data has been written to physical storage. Both acknowledgements have an associated timeout. If a packet is not acknowledged, VVR resends. Note: It is incorrect to assert VVR waits for the secondary data to be written to disk. In reality, VVR does not wait for data to be written at the secondary, only received. This improves application per-

m-

formance. But it tracks all acknowledged, uncommitted transactions and can replay any necessary transactions if the secondary crashes prior to actually writing the data. On the primary side, data is written as two independent writes to the SRL and actual data volumes. The actual data volume is written in the background asynchronously to maximize application write perforance. Once again, about it is incorrect to regard VVR writing twice as a performance issue. VVR does write twice, but only once where it affects application latency. The SRL write is a very fast write to a


8


sequential accessed log. The data volume is a normal write, done in a lazy fashion, when it affects per-formance the least. Should the primary crash at any point, SRL data is fully recoverable. Asynchronous Application writes in asynchronous mode are first placed in the SRL, then immediately acknowledged

cation. Data is then sent as-soon-as-possible to the secondary RVGs, based on

on solution must first begin with a known dupli- initialize secondary systems.

est method starts with completely empty systems on each side. This can be done if VVR is hile initially constructing a data center, prior to production. For VVR to use an empty data set,

ver-the-wire initialization essentially uses VVR to move all data from the primary to secondary over simple process, however, with larger data sets it can take a long time,

ocal mirroring is an option for very large data sets. In this method, the data storage array for the sec-ly placed at the primary site and attached as a VxVM plex. Once the mirror is com-

heckpoint initialization is a truly unique VVR feature. This allows huge data set synchronization using replication initiation. This capability is unique to VVR and is not possi-

a checkpoint initialization commences, a check-start m-

ck-

e then transported to the secondary site and loaded (remember the old saying “never un-erestimate the bandwidth of a station wagon full of tapes”). When the tape load is complete, the sec-

sistent versus up-to-date. Once the data is played between the checkpoints, the secondary is considered consistent, and is available as a re-

n

to the calling appliavailable bandwidth and the number of SRL outstanding writes. VVR’s asynchronous replication there-fore comprises the ideal choice for long-haul data mobility needs. INITIALIZING SECONDARY SYSTEMS In order to begin changed block replication, a replicaticate data set at each site. VVR offers several ways to

Empty The simplinstalled wboth sides must be identically empty. This can be achieved by creating volumes with the “Initialize Zero” setting from VMSA, or by copying /dev/zero into the volumes at each side. Over the wire (Autosync) Othe rlink. Overall this is a very especially if the primary is active while attempting to initialize the secondary. Local mirroring Londary site is initialplete, the plex is split off and the array is shipped to the secondary site. This mode allows large amounts of data to be initialized at SAN speeds, as well as allowing subsequent data written during theshipping period to be spooled to the primary SRL. Hot backup and Checkpoint Ctape technology and immediatelyble on hardware-based replication solutions. Checkpoint initialization is essentially a primary side hot backup, with the SRL providing the map of what was changed during the backup. When pointer is placed in the SRL. A full block-level backup is then taken of the primary volumes. When coplete, a check-end pointer is placed into the SRL. The data written between the check-start and cheend pointers represents data that was modified while the backup was taking place. This constitutes a hot backup. The tapes ardondary site is connected to the primary site with a “checkpoint attach” of the rlink. The primary then for-wards any data that had been written during the backup (that data between the check-start and check-end). Once this data is written to the secondary, the secondary is an exact duplicate of the primary, at the time the backup completed. At this point the secondary is consistent, and simply out of date. The SRL is then replayed to bring the secondary up to date. This example again surfaces the difference between conrecovery site. Once the SRL is replayed from the check-end to the end of the SRL, the secondary is theup-to-date.


9


RECOVERY AFTER PROBLEMS

VR is very robust in terms of tolerating network connectivity outages to the secondary. The SRL pro-le maintaining a consistent secondary.

ry/Network outage secondary system outage or outage of the network to the secondary is identical as far as VVR is

no longer available, the primary simply spools to the SRL. When

-0

he secondary storage, resulting in a data loss on ere are several methods to recover from a secondary failure. The first is to re-

the primary romotion of a secondary to a primary, affecting a disaster recovery takeover. This is

S ole changes are actions to swap the primary role for a replicated volume group to a system that was

n be due to an outage disabling or destroying the primary, where a

ary to secondary systems is a controlled shift of primary responsibility for an RVG. e existing primary SRL if necessary, and then control is handed to the existing

a

more “violent” event, in that the secondary is promoted to a pri-ng demotion of the original primary. When a takeover is accomplished, the

Vvides the key to recovering from outages, whi

SecondaAconcerned. When the secondary is the secondary is repaired or network problems are resolved, the SRL is drained to the secondary. The ability of VVR to tolerate extended outages is completely controlled by the size of the SRL. The ability to rapidly recover is governed by the size of the rlink pipe. For example, if the customer has a 3day outage and this results in 30 Gigabytes of data written, the SRL must be able to accommodate 3Gigabytes, plus additional data while the SRL drains. The ability to rapidly move the 30 Gigabytes to the secondary is set by the rlink network capacity. Secondary Failure

secondary failure is better defined as a failure of tAthe secondary side. Thbuild the secondary storage and re-initialize using one of the methods discussed above. The second method is to take regular backups of the secondary environment using a VVR feature called “Secon-dary Checkpoints”. Secondary Checkpoints allow a pointer to be placed in the primary SRL to desig-nate a location where a backup was last taken on the secondary. Assuming the primary has a large enough SRL and secondary backups are routinely taken, a failure of the secondary is repairable by reloading the last backup and rolling the SRL forward from the last secondary checkpoint. Primary Failure

failure of the primary can be broken into several possible problems. A complete failure ofAsite is handled by pexactly the scenario VVR is built for. For primary outages, such as server failure, or server panic, the customer has the choice to wait for the primary to recover, or shift operations to a secondary. For situa-tions involving actual data loss at the primary, the customer can shift operations to a secondary, or re-store data on the primary. VVR ROLE CHANGERpreviously a secondary. This casecondary is used to continue operations, or simply a role reversal to allow a secondary site to take over operations. Primary Migration

migration of primAData is flushed from thsecondary. The original primary is demoted to a secondary, and the original secondary is promoted toprimary. This very simple operation is accomplished with a single vradmin command initiated by the operator or VERITAS Cluster Server and allows rapid shift of replication primary between sites. There is zero chance for data loss, as all data outstanding at the primary site is sent to the secondary prior to allowing the migration to take place. Secondary Takeover

secondary takeover is a somewhat Amary without a correspondisecondary is brought up in read-write mode in the exact state it was in at time of takeover. Any data written by the primary in ASYNC mode and not sent to the secondary is not available. After a secon-


10


dary takeover, the original primary must be synchronized to be an exact duplicate of the new primaryData present on the old primary that was not sent to the new primary is lost. When a secondary comeup in a takeover mode, it maps all writes to all data volumes in Data Change Maps (DCM). The use of DCM allows the original primary to be synched with the new primary with a minimum amount of data transfer.

. s

SING THE SECONDARY SYSTEM se the replicated volumes at the secondary site?”

f an ongoing replication process. Most people under-

changes are

ld g

read to disk

r-

y

, a point-in-time copy must be made to allow a static view f the storage. VERITAS offers the capability to create full mirror break-offs which can be used on the

VR IN THE CUSTOMER ENVIRONMENT

answer is “enough”. Adequate ach secondary site in any given time period.

e

r the SRL eventually overflows. si-

t. This may be

UOne of the more common VVR questions is “Can I uThe answer is no, not while volumes are a part ostand that the volumes at the secondary site cannot be written to, as data corruption results. What is not understood is why the volumes may not be used in a “Read-Only” mode. VVR is forwarding all writes from the primary site to the secondary. At the secondary site, these writes are being written di-rectly to the VxVM volumes by VVR, without interaction with any other software layer. For example, at the primary side, a VxFS file system may be sitting on top of the volume in an RVG. Writes occurring to this volume happen through VxFS, so the file system is “aware” thatoccurring on the underlying volume. At the secondary site, VVR is changing the volumes directly, based on writes received from the primary. If someone mounts a file system in read-only mode, it woumount acceptably, but would be unusable. The file system would not see changes to the underlyindata in the volumes and would present inconsistent data to the requesting application. Imagine an application requesting data about a specific customer. Part of the data may be in memoryfrom a previous read request, so the file system uses the cached copy. It then performs a for the remainder of the data. This data may be newer data written by VVR to the volumes. In a data-base environment, the database itself caches blocks in memory. If a request were processed that in-volved cached blocks and newly read blocks, an inconsistent response to a query results. The overallissue is one of cache consistency. Applications (including VxFS) cache data in memory to improve peformance. VVR is replicating below the applications at the volume level and has no way to invalidate the application cache when it writes new data to the volumes. For this reason, VVR volumes on a sec-ondary system may not be used in any way while replication is active. This limitation is identical on anhardware based replication solution as well. In order to use the data in the secondary RVGosecondary host or any other SAN attached host at the secondary site. Customers can also choose to use “Space Optimized Snapshots” (SOS). A Space-Optimized Snapshot uses Copy On Write capabil-ity to keep a point in time view on a small cache volume. For more info on Point In Time solutions for use with VVR data, please see your VERITAS sales team.

VUNDERSTANDING BANDWIDTH NEEDS A very common question is “How much bandwidth do I need?” Thebandwidth must be available to move all write traffic to eFor example, if 10 Gigabytes of data are written in a 24 hour period, then enough bandwidth must bprovided to move 10 Gigabytes of data in 24 hours. The SRL can be used to spool data during time periods when write traffic exceeds replication band-width, but this data must drain at some future point, oIn the following chart, we see a case where update rate heavily exceeds rlink pipe size during the buness day. This results in the SRL filling over the course of the day and draining at nighacceptable, knowing that an outage during peak hours will result in a fairly substantial data loss.


11


SRL fill example 1

0

20

40

60

80

100

1:00

3:00

5:00

7:00

9:0011

:0013

:0015

:0017

:0019

:0021

:0023

:00

Time of day

SRL

Dep

th in

Gig

abyt

es

Data Written inGigabyte/HourRLINK Pipesize

SRL fill depth

In the next example, we see a case where data update rates average more than available rlink band-width, resulting in an SRL that continues to fill and eventually overflows.

T1 example, with data rates averaging more than 1 Gig/hour

012345

6:00

8:00

10:00

12:00

14:00

16:00

18:00

20:00

Time of day

SRL

Dep

th in

G

igab

ytes


SRL fill depth

In the final example, we have a case where rlink bandwidth exceeds write rate for all but a small portion of the day. This results in a very small window where the secondary is actually behind any significant amount.


12


Dual T1 example (1.4 Gigabyte/hour), with data rates averaging more than 1 Gig/hour

00.5

11.5

22.5

6:00

8:00

10:00

12:00

14:00

16:00

18:00

20:00

Time of day

SRL,

Pip

e an

d W

rites

in

Gig

abyt

es


SRL fill depth

LATENCY CONSIDERATIONS Probably more important than bandwidth is latency. In the VVR context, this refers to the amount of time to send a packet to the secondary and receive acknowledgement, discounting VVR overhead. In a synchronous environment, each application write must spool to the SRL, be sent to the secondary and acknowledged by the secondary before the write complete is sent to the calling application. As distances between sites increases, so does latency. Distance is not the real issue, but rather infra-structure between sites that induces latency. As a common example, consider using replication from New York to New Jersey. A direct path from Manhattan to Jersey City is no more than 5 miles, well within range of single mode fiber. If one pur-chases SONET services from a major provider, actual distance traveled could be much greater, as it follows whatever infrastructure the provider uses. (Traditional Multi-Megabit service has focused on available bandwidth, and not latency. So a New York to New Jersey could easily route through avail-able fiber through Boston). In the same setup, satellite services travel 52,000 miles! This example shows that the primary/secondary geographic distance is not the governing factor, but rather the intermediate infrastructure. Actual data traveling at near speed of light over fiber has very low latency. But intermediate switches, repeaters, routers, etc. can add significant latency. EFFECTS OF VVR ON APPLICATION PERFORMANCE VVR typically has very little effect on application performance (on the order of 3-5%) and in many cases will actually result in a net performance increase for an application. All data written for an application is written to a sequential log, with very little disk head movement. The actual write performance for a given application may actually show an increase, as the log method used by the SRL may be more efficient that writing to the actual data locations in the underlying volumes. As we have seen, there is no significant performance hit from VVR writing twice as evidenced by real world installations exhibiting little to no performance degradation. A second write does occur, but it has no effect on the application since it is performed by the operating system in the background. USING ASYNCHRONOUS REPLICATION TO DECOUPLE APPLICATION LATENCY


13


One of the most compelling real-world VVR features is its ability to maintain full consistency at secon-dary sites while operating in asynchronous mode. Maintaining write order fidelity in asynchronous mode allows VVR to truly exploit potential asynchronous replication performance benefits. By providing a high bandwidth connection, enterprises can completely remove the replication latency penalty and still main-tain near up-to-the-second data at the remote site. At the primary site, applications are acknowledged as soon as data is placed in the SRL. Hence, application processing continues normally. The data is then sent out almost immediately over the rlink to the secondary site(s). With adequate bandwidth, the SRL does not fill, so the actual data outstanding between primary and secondary is realistically what-ever data is currently on the wire. This means enterprises can have near up-to-the-second replication, at arbitrary distances, without application penalties.


14


About the Author Jim Senicka has been with VERITAS Software for over 5 years and is presently the Director of Techni-cal Product Management for VERITAS clustering products on UNIX platforms. His primary role is future product feature definition and product interoperability. He has also been a VERITAS Software Enter-prise Consulting Services enterprise architect and high availability practice lead. In this prior role, Jim provided vision and direction for creating and managing programs targeted at increasing VERITAS client availability and operational efficiency. Before joining VERITAS Software, Jim was a high availabil-ity product expert at Auspex Systems and Silicon Graphics working with Auspex ServerGuard and SGI Failsafe. In his spare time, among other passions such as flying radio controlled airplanes, Jim can usually be found racing motorcycles on closed circuit road courses with his track day organization, the Northeast Sport Bike Association. Below is a picture of Jim pulling away from yet another hapless competitor.

© 2005, Jim Senicka, All rights reserved.

Used with permission of Jim Senicka.


15


VERITAS Software Corporation Corporate Headquarters 350 Ellis Street Mountain View, CA 94043 650-527-8000 or 866-837-4827

For additional information about VERITAS Software, its products, VERITAS Architect Network, or the location of an office near you, please call our corporate headquarters or visit our Web site at www.veritas.com.

© 2005 VERITAS Software Corporation. All rights reserved. VERITAS, the VERITAS Logo, Volume Replicator ,

VERITAS Storage Foundation, and FlashSnap are trademarks or registered trademarks of VERITAS Software Corpora-

tion or its affiliates in the U.S. and other countries. Other names may be trademarks of their respective owners.


17

http://www.veritas.com/

Vvr Concepts

Documents

veritas architect

veritas cluster

veritas volume

data rates

veritas storage

decouple application

storage replicator

database management