How We Leveraged EMC CLARiiON , Celerra , and Centera ... · EMC Technologies Utilized and/or Discussed: CLARiiON SAN – CX700, CX3-40, Mirrorview /A . Celerra NAS – NS502G, CIFS,

How We Leveraged EMC CLARiiON®, Celerra®, and Centera® Platforms to Consolidate Three Company’s

Data Systems into a Unified Storage Environment

EMC Proven Professional™ Knowledge Sharing 2008

Christopher Johnson UNIX Administrator / Storage Architect

Spheris Operations, LLC [email protected]

2008 EMC Proven Professional Knowledge Sharing 1

mailto:[email protected]

Table of Contents

Abstract ............................................................................................................................... 3 Spheris Profile..................................................................................................................... 3 Migration by Virtualization ................................................................................................ 5 The Corporate Goal – From Many-to-One ......................................................................... 7 When Windows Clients and UNIX Apps Overlap ............................................................. 7 Virtual Data Movers – Planning for the Unknown............................................................. 8 Windows Home Directories, ACLs, and Orphaned Files................................................... 8 Using the Celerra for Application Integration .................................................................... 9 Transitioning from Direct Attached Storage to CLARiiON Back-end Storage for Business Critical Systems ................................................................................................. 10 The Benefits of CLARiiON Scalability............................................................................ 11 Unstructured Data Retention Defines the Need for Data Archiving ................................ 12 Replication—the Logical Path to Disaster Recovery ....................................................... 14 Why EMC Technologies Were the Right Choice for Centralizing our Enterprise Storage Environment...................................................................................................................... 15 About the Author .............................................................................................................. 16

Disclaimer: The views, processes or methodologies published in this compilation are those of the authors. They do not necessarily reflect EMC Corporation’s views, processes, or methodologies.


Abstract In today’s global markets, IT administrators are challenged to consolidate data systems

of newly acquired or consolidated companies into one unified environment. This article

describes how we used EMC CLARiiON®, Celerra®, and Centera® platforms to

consolidate the disparate data systems of three companies after a series of acquisitions.

I will describe the challenges we faced as we migrated both Windows and UNIX files

servers from multiple domains onto our Celerra NAS system. Also, with so much

unstructured data within our environment requiring long-term online access, I will discuss

how Centera provided us with a great option to meet the company’s needs and service

level agreements (SLAs). With an extensive VMware virtual environment, we use our

CLARiiON disk array for its backend storage foundation. Last, I will describe how we

used our CLARiiON disk arrays to migrate our various database servers’ storage away

from localized disk systems.

EMC Technologies Utilized and/or Discussed: CLARiiON SAN – CX700, CX3-40, Mirrorview /A

Celerra NAS – NS502G, CIFS, NFS, Celerra Replicator

Centera CAS – Parity Protection, CFA, CUA, Centera Replication

Spheris Profile

Spheris is a leading, global outsource provider of clinical documentation technology and

services. We support more than 500 health systems, hospitals, and group practices

throughout the United States. Founded by doctors, Spheris solutions address the needs

of practitioners, health information directors, IT directors, and administrators. Intimate

understanding of how doctors and healthcare facilities work enables Spheris to combine

advanced technology with expert teams to deliver comprehensive solutions.

Spheris' worldwide headquarters is located in Franklin, TN, with operations centers in

Minot, ND; Sterling, VA.; Bangalore, India; Coimbatore, India; and Delhi, India.


As a technology-based company, Spheris’ IT Infrastructure provides the core foundation

needed to stay ahead of the ever changing technology landscape. The primary data

center is located within the Franklin, TN corporate headquarters, with a secondary data

center, for disaster recovery, in Sterling, VA. Spheris’ enterprise environment is truly a

“heterogeneous environment”, using products from multiple vendors such as Cisco, HP,

Dell, McData, and EMC for networking and systems needs.

The result of a corporate merger between the companies formerly known as Total eMed

and EDiX, Spheris merged the two corporate data and application platforms into a

singular, centralized data resource. With dual data centers, and corporate operations in

both Florida and Tennessee, the first task was to consolidate and eliminate overlapping

business areas. In the midst of the Total eMed/EDiX consolidation, Spheris acquired

Healthscribe, Inc. of Sterling, VA, adding another piece to the convergence puzzle.

Spheris’ acquisition of Healthscribe, Inc., provided a third data center and corporate

operations location. Following an active hurricane season in 2004, Spheris made the

decision to consolidate the FL and TN data centers with plans to make the VA data

center a standby disaster recovery site.


Migration by Virtualization Moving and consolidating a multi-million dollar center is a formidable task, particularly

when the data center houses the vital application data that provides 85% of the

company’s generated revenues. The FL data center housed between 200 - 300 physical

HP/Compaq servers and Cisco networking equipment. This included several database

and application servers, as well as the company’s primary messaging environment. With

a tight schedule, and very little room for error or business interruptions, Spheris took a

leap of faith with VMware technology.

Scaled Down Representation of Spheris’ Pre-migration WAN

We virtualized over a third of the physical servers by leveraging our CLARiiON CX700

storage array and six HP Proliant DL585 servers running VMware ESX Server. The FL

and TN data center consolidation was completed in just over half of the planned

migration time. In addition to decreasing migration time, we also reduced the physical

server footprint in the data center by nearly a two-to-one ratio.


We used a process of dual servers in a primary to secondary role since our

consolidation required relocating systems from FL to TN with minimal downtime. The

primary server was the physical server in the FL data center, while the secondary server

was a virtual server within our VMware Virtual Infrastructure in TN. Once we decided on

the best date and time for system migration, the primary server in FL was taken offline

and we renamed the secondary server, in TN, with the original primary server name.

Once the server rename was complete, we updated all of the name resolution records in

the domain name service (DNS) and Windows Internet Name Service (WINS) to allow

for proper routing to the new primary server in TN.

Scaled Down Representation of Spheris’ TN DC Post-migration


The Corporate Goal – From Many-to-One The next major initiative was to collapse and retire all of the various file servers from the

combined FL and TN data centers into a singular storage space. We decided to utilize

the Celerra NS502G network-attached-storage (NAS) device to further reduce our

physical server footprint within the data center.

Scaled Down Representation of Spheris’ File Server Environment Pre-migration

When Windows Clients and UNIX Apps Overlap We faced a significant challenge in our data centralization efforts since both Windows

clients and UNIX applications had to have access to files and folders within the same file

systems. We implemented our Celerra for use within a multi-protocol environment since

Windows and UNIX use two different methods for data security. Common Internet File

System (CIFS) for Windows and Network File System (NFS) for UNIX were the protocols

used. The Celerra was ideal since it allows access rights to its shared file systems to use

each environment’s native access protocols. In short, it allowed file access by Windows

clients through access control lists (ACLs) and UNIX applications through their user

and/or group IDs (UID / GID).


Virtual Data Movers – Planning for the Unknown Two long-term issues had to be considered before implementing the Celerra NAS for file

server consolidations: true data segregation and disaster recovery implementation.

Although the underlying file systems and CIFS servers provided logical separation of

data to users, they still used the same root file system of the physical data movers.

Virtual data movers (VDMs) were the only way to truly isolate sensitive data from our

general network users. By encapsulating each VDM into its own isolated file systems,

users are only able to access data on the CIFS servers that are within each respective

VDM. The ability to replicate the VDM, its configurations, and all of its file system data

through replication to another Celerra, either local or remote, are added benefits of the

VDM.

Windows Home Directories, ACLs, and Orphaned Files Once we began our file server consolidations to the Celerra, we were immediately able

to use many of its built-in features, such as snapshots and multi-path access to a single

directory. However, we found a few issues that had to be worked around using custom

scripts.

The first issue was dealing with the migration of Windows home directories from multiple

domains. By default, a user’s home directory is not accessible to any other user,

including administrators. We had to have each user run a custom home directory

migration script to insure that all of their personal files were moved. Although Celerra

has its own built-in home directory migration tools, we chose this route because we had

multiple domains with some overlapping user accounts.

The next issue was two-fold and only became apparent after complete server migrations

had taken place. The Celerra did not properly migrate the file data and/or its security

identifiers when we migrated files from Windows servers to a Celerra CIFS share that

were owned by an irresolvable user account. What would cause this? Files and

directories are created by either users or applications; although theoretically applications

are users, in this context we mean human users. As employees came and went, any

files that a particular user created were owned solely by that user. When the user’s

domain account was deleted it would leave any files that the user owned in an orphaned


state. This caused the orphaned files migrate without any data and/or invalid security

identifiers. Once we identified the orphaned data, we had to administratively take

ownership and re-migrate the data to the Celerra. Now that this issue has been identified

and documented, EMC has created a set of pre-migration tools that can be used to

check for and identify orphaned files on servers before their data is migrated. In the end,

we successfully migrated more than twenty physical file servers onto Celerra CIFS

servers.

Using the Celerra for Application Integration Spheris is a company born of multiple companies combined into one, our focus then

turned to consolidation of our different transcription platforms. We were able to place

files within a common directory for access by different applications using different

Uniform Naming Convention (UNC) paths, thanks to the Celerra’s ability to share

directories with multiple CIFS servers. Although our applications consolidation is still

being implemented over a period of phases, without the Celerra this process would have

been much more difficult to accomplish.


Scaled Down Representation of Spheris’ File Server Environment Post-migration

Transitioning from Direct Attached Storage to CLARiiON Back-end Storage for Business Critical Systems Centralizing our corporate file-sharing server data was a major step toward our goal of

enterprise-wide data unification. However, the core systems of almost all enterprise

environments, including Spheris, rely upon their back-end database and messaging

systems. Our business critical (BC) environment consists of both Oracle and MS SQL

database servers, in addition to MS Exchange Messaging servers.

We presented CLARiiON LUNs to each BC server to handle their long-term storage

needs to make this transition unobtrusive to our production environment. We transitioned

our BC servers away from locally attached disk sub-systems onto CLARiiON storage

LUNs through the coordinated efforts of our database administrators and infrastructure

teams.

Scaled Down Representation of Spheris’ Isolated Business Critical Environment Pre-migration


The Benefits of CLARiiON Scalability The CLARiiON SAN provided Spheris with several scalability options that were either

unavailable or cost-prohibitive to attain with direct-attached disks systems.

CLARiiON back-end storage enabled us to utilize the benefits of MS Cluster services for

some our MS SQL database servers. This technology is a great benefit from both a

redundancy and systems expense standpoint. Through database clustering we can add

multiple nodes to the cluster and keep the database online during a physical node

outage. From a cost perspective, with CLARiiON back-end storage there is only one

storage expense for the entire cluster. Also, we can easily add LUNs to the cluster’s

storage resources as the database’s storage needs increase.

CLARiiON back-end storage also allowed us to take advantage of the CLARiiON’s built-

in redundancy features. Through snapshots and clones we are able to provide online

disk recovery options, based on a particular point in time. More important are the

different replication options that are simply not available with local-attached disk sub-

systems. With options such as Mirrorview™, SAN Copy™, and RecoverPoint we can

provide both internal replication within our TN data center as well as wide-area

replication to our remote data center in VA.


Scaled Down Representation of Spheris’ Centralized Business Critical Environment Post-migration

Unstructured Data Retention Defines the Need for Data Archiving A major misconception about centralized data architecture is that it offers unlimited

storage space. Couple that with the regulations and legalities of data governance

imposed by SOX, HIPPA, and client agreements that require data retention for longer

periods of time and you can see how a long-term archiving system is the only logical

solution.

We met with several vendors who offered various archiving technologies to address our

long-term archiving needs. After much debate and research, we selected the EMC

Centera archiving solution. The Centera provided the most flexible options for archiving

data through its various integration tools with our other EMC products.

Due to accelerated time frames and overextended development resources we had to

find a way to implement our Centera into our environment without the need for


developmental resources. We expediently integrated the Centera into our enterprise

thanks to the Centera Universal Archival (CUA) Gateway and Celerra File Archiver

(CFA). With a few custom scripts and the CUA, we were able to archive specific data

files from network shared directories to the Centera prior to implementing our CFA. We

implemented the CFA to migrate files directly from Celerra file systems to the Centera

with the Celerra NAS serving as our network file sharing system of the future. Long-term,

we plan to use either multiple CFA units or DiskXtender® for NAS to migrate all

unstructured data to our Centera.

The data protection scheme that best suited our overall archival needs was another

consideration during our Centera implementation. Our initial cluster consisted of two

four-node cubes that provided a choice of data mirroring or parity protection. Both

options had their advantages and disadvantages such as faster rebuild times in a

mirrored system versus more available space in a parity system. We chose the parity

protection scheme for our environment because of the vast amount of data we had to

archive to the Centera during the initial and subsequent phases of data archiving.

Scaled Down Representation of Spheris’ Archival Topology


Replication—the Logical Path to Disaster Recovery Exploring the various options for enterprise disaster recovery caused us to consider the

different replication technologies available through our EMC back-end systems. Each

system provided solid replication options to synchronize data between our TN and VA

data centers. Both Celerra and Centera provide IP-based replication, with the ability to

define what data to replicate. The CLARiiON offers replication options to fit replication

tasks for both local and/or wide area networks.

We used VDMs and Celerra replication to set up and implement NAS disaster recovery

in phases, replicating each specific VDM as its data is determined to be critical enough

to warrant replication. Within the Centera, archived data is written to individual virtual

pools which use IP-based replication in the same manner that Celerra VDMs allow for

specific data sets to be migrated.

We are in the planning stages of moving over to true block-level replication via our

CLARiiON storage arrays, although we currently use database log shipping in our initial

phase of database replication. We are determining the best solution for our current and

long-term environment in these preliminary steps. While there are other solutions that

offer pseudo block-level replication, we plan on leveraging our EMC investment by

implementing a solution through either Mirrorview /A, SAN Copy, RecoverPoint, and/or

Replication Manager.

We have already brought our WAN-based storage infrastructure online in preparation for

our long-term disaster recovery implementation. FCIP 1620 Fiber-to-IP bridge/routers

are installed and configured in both our TN and VA data centers. These units connect to

McData fiber switches, a Connectrix EDM Director in TN and two DSM24 workgroup

switches in VA, and provide the backbone of our enterprise storage fabric.


Scaled Down Representation of Spheris’ Storage Replication Topology

Why EMC Technologies Were the Right Choice for Centralizing our Enterprise Storage Environment There are many companies with new and emerging technologies vying for a place in the

fast growing world of enterprise storage. Some of these technologies are good, others

not so good, and some that may be great but are yet untested in an enterprise

production environment.

With EMC storage technologies we knew we were not only purchasing tried-and-true

enterprise-class products, we knew we were gaining a long-term partner for our

company. We recognized the value of a mature company with the resources to research

and bring to market products that are ready to go from the moment that they are shipped

to client sites.


Couple that with a full-fledged implementation team to guide each project through the

various phases and you have a formula for success, regardless of the size of the project

in scope. These qualities cannot be easily matched by other vendors regardless of their

size. Additionally, having access to 24-hour technical support is invaluable, with a

company such as ours.

In the end, Spheris did not need another vendor to sell us the next hot thing to hit the IT

world. We needed and found a true “follow the sun” partner in EMC… and that is why

EMC Was the Right Choice for Spheris!

About the Author In my current role as a UNIX administrator, I serve as Spheris’ primary architect and

administrator of our enterprise storage environment. I have been with Spheris for over

10 years, beginning as an employee with EDiX back in July 1997. During my career with

Spheris, I have served in multiple roles including help desk, client technical support, field

engineering, network administration, and systems administration. I have watched the

company grow from the early start-up phase into becoming one of the market-leading

medical information outsourcing companies. Currently holding certifications in Microsoft,

Cisco, and EMC technologies, I have not only a wealth of Spheris-related knowledge,

but also a great foundation and understanding of the overall enterprise network and

computing environments.


How We Leveraged EMC CLARiiON , Celerra , and Centera ... · EMC Technologies Utilized and/or Discussed: CLARiiON SAN – CX700, CX3-40, Mirrorview /A . Celerra NAS – NS502G, CIFS,

Documents