Advance SAN Services

8/10/2019 Advance SAN Services

1/109

2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSAN-3707 1

Advanced SAN Services

BRKSAN-3707

Mike Dunn

July 14, 2011

Network Consultant


2/109


What Are the Top Storage Initiatives?

0% 10% 20% 30% 40% 50%

Backup Redesign

Data Migration

Thin Provisioning

Virtualization Adoption

Reporting

Consolidation

Archiving

Tiered Storage Build Out

Technology Refresh

Deduplication

Source : TIPs Storage Research 2009

0% 10% 20% 30% 40% 50%

Improving Forecasting

Improving Performance

Data Migration

Disaster Recovery

Archiving

Virtualization Adoption

Technology Refresh


Backup Redesign

Consolidation

Source : TIPs Storage Research 2010

Top Five Solution Initiatives: Deduplication

Technology Refresh


Archiving

Consolidation


3/109


Agenda

SAN Consolidation with Virtualization

Inter-VSAN Routing (IVR)

N-Port Virtualizer (NPV) / NPIV

FlexAttach

Tiered Storage and Backup Design

Data Mobility Manager (DMM)

Storage Media Encryption (SME)

SANTap

Fibre Channel over Ethernet (FCoE)


4/109


SAN Consolidation withVirtualization


5/109


Isolated fabrics hosted on different SAN

switches

Application isolation is important

Virtual SANs (VSAN) are virtual fabrics

providing logical separation

Consolidated and manageable logical SANs

Traffic Isolation

Inter-VSAN Routing (IVR) route traffic

between VSANs to achieve cross-fabric

connectivityConsolidated SANs still isolated

Only relevant fabric events are propagated

Enables resource sharing

The Result of SAN Consolidation May RequireResource Sharing

Physical

SAN

Physical

SAN

PhysicalSAN

PhysicalIslands

VirtualFabrics

VSANVSAN VSAN

VSANVSAN

VSAN

RoutedVirtualFabric


6/109


IVR Is Needed When:

Certain devices (hosts andtapes) are in different VSANsfor isolation reason and theyneed to communicate at thesame time

No other devices can or needto communicate

IVR Configuration Includes:

VSAN Topology

Zones

Inter-VSAN Routing (IVR)Switch Virtualization

Backup VSAN

(V200)

A

pwwnT

Domain 60 Domain 50

Domain 20

Domain 10

pwwnH

B

IVR Enabled

Switches

B FabricA Fabric

IVR Provides Isolation andResource Sharing

Feature IVR Enable


7/109 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSAN-3707 7

Overlay data replication fabricson common physical fabric

No need for separate pair of switches

for each replication connectionUse one VSAN per replicationconnection

Share the common resourcesamong different replicationVSANs

Share common SAN Extensioncircuits amongst multiple virtualfabrics

IVR Use CaseCommon Resource Sharing

Tape Media Server

HRSAN

FinanceSAN

Common Physical Fabric

EngineeringSAN

TAPE

SAN

Tape Media Server

TapeMediaServer



Manually define the fabricrouting scope

List of VSANs to be used for IVR oneach switch

Contains one entry for each

IVR-enabled switch Same set of entries on all the

switches

List contains only VSANs includedin topology

Needs to be activatedTopology map is calculated which isused for routing

IVR VSAN TopologyDefines Routing Topology Scope

Backup VSAN

(VSAN 200)

OLTP VSAN(VSAN 100)

pwwnT

Domain 60

Domain 10

pwwnH

swwn1 vsan-range 100,200

swwn1

A Fabric Shown, Repeat for B Fabric

Email VSAN(VSAN 300)

DatabaseVSAN

(VSAN 500)



All the IVR-enabled switchesshould have same numberof entries

All the switches should

contain topology entries forother IVR-enabled entries

IVR VSAN TopologyAll IVR-Enabled Switches Need VSAN Topology

Backup VSAN(VSAN 200)

OLTP VSAN(VSAN 100)

pwwnT

pwwnH


swwn2swwn1

swwn1 vsan-range 100,200swwn2 vsan-range 100,200



Establishes communicationacross VSANs

Similar to regular zones

Extends zoning across VSANboundary

Needs to be activated onceIVR adds zones to relevant VSANs

All the IVR enabled switchesneed to have same IVRzones

One IVR Zoneset per topology

IVR Zoneset and ZonesDefines IVR Zoning


OLTP VSAN(VSAN 100)

pwwnT

Domain 60

Domain 10

pwwnH

IVR_zone1member pwwnHmember pwwnT

IVR_zone1member pwwnHmember pwwnT


ivr_zoneset1ivr_zone1member pwwnH vsan 100member pwwnT vsan 200



Virtual Switch (domain)

Representation of a switch (domain)in another VSAN

Virtual Device

Representation of a device another

VSAN

Device Advertisement

Process of instantiating thedomain/device in another VSAN

IVR Virtual Switch (Domain) and DeviceRepresenting Native Switch/Device


OLTP VSAN(VSAN 100)

pwwnT60.1.2

Domain 10

pwwnH10.1.2Domain 10

pwwnH

10.1.2

Domain 60



Switches arevirtualized withoriginal domain ID

Devices are

virtualized withoriginal FCID

IVR routes betweenVSANs

Domain IDs have tobe unique across allthe VSANs

Inter-VSAN Routing (IVR)Step by Step


OLTP VSAN(VSAN 100)

pwwnT60.1.2

60.1.2

pwwnH

10.3.4

Domain 60

Domain 60

10.3.4

Domain 10

Domain 10

NoFCID

Translation

VSAN Topology


IVR zonesivr_zone1pwwnh vsan 100

pwwnt vsan 200



IVR Network Address Translation (NAT)No Unique Domain Requirement

Note same domain ID inboth VSANs

With IVR NAT, eachVSAN is represented in

another VSAN using adomain ID

Switches are virtualizedusing domain assignedto its native VSAN

No unique domainrestriction since theFCIDs are translated

Backup VSAN

(VSAN 200)

OLTP VSAN(VSAN 100)

pwwnT10.1.2

pwwnH10.3.1

Domain 10

{10.3.1, 30.3.2}->

{70.1.2, 10.1.2}Domain 1010.1.2, 70.1.2}

->{30.3.2, 10.3.1}

VSAN 200 Is Represented asDomain 30 in VSAN 100

VSAN 100 Is Represented asDomain 70 in VSAN 200 70.1.2

Domain70

30.3.2

Domain 30



The Virtual Domain ID andFCID can change

When Devices/switches go down/up

Some HP-UX and AIX serversneed static FCIDs

Persistent Virtual Domain

Same Virtual domain can be assignedto a virtual switch

Persistent Virtual FCID

Same FCID is assigned to the virtualdevice

Domain has to be persistent

IVR NAT Domain and FCID PersistencyEnsure Domains/FCIDs Are Persistent

Backup VSAN

(VSAN 200)

OLTP VSAN(VSAN 100)

pwwnT50.1.2

30.3.2

pwwnH10.3.1

Domain 10

70.1.2

Domain 70

Domain 10

native-vsan 100 domain 70pwwn 11:22:33:44:55:66:77:88 fcid 70.1.2

Domain 30

native-vsan 200 domain 30pwwn 33:44:55:66:77:88:99:1 fcid 30.3.2

Domain 30



Enable auto topology on oneswitch

No manual configuration of VSAN list isneeded

Requires Cisco Fabric Services (CFS)distribution enabled on all switches

Relevant local VSANs on a IVRenabled switch are used

Only member VSAN and transit VSANsare used

IVR VSAN Automatic TopologyEliminates Manual Configuration Topology

Backup VSAN

(VSAN 200)

OLTP VSAN(VSAN 100)

pwwnT60.1.2

Domain 60

Domain 10

pwwnH10.1.2

Email VSAN(VSAN 300) Database VSAN

(VSAN 500)

swwn1 vsan-range 100,200swwn2 vsan-range 100,200

ivr vsan-topology auto



Consolidate many physicalfabrics into VSANs throughSAN extensions

Events are propagated acrossgeographically separated

fabricsThe edge VSAN events will becarried over the WAN/MAN

IVR Across Remote Data CentersEdge VSAN Extends Through the WAN Connection

OLTPVSAN(VSAN 100)

pwwnT

pwwnH

swwn1

Remote BackupVSAN

(VSAN 200)

swwn2

swwn1 vsan-range 100,200 IVR Enabled Switch

SONET/SDHDWDMCWDM

IP (Metro Eth)



The VSAN on the transitpath between membersVSANs

IVR enabled on Transit VSANborder switches

Only relevant and requiredevents are carried over transitVSAN

Isolates Edge VSANs from anytransit events in transit VSAN

Network

IVR Transit VSANSeparate VSAN for the WAN/MAN Connection

OLTP VSAN(VSAN 100)

pwwnT

pwwnH

swwn1


Transit VSAN(VSAN 300)

swwn2



wn1 vsan-range 100,300wn2 vsan-range 200,300

SONET/SDHDWDMCWDM

IP (Metro Eth)



Used for Consolidation of Fabric and Sharing of resources

Can Isolate Fabrics when Traversing Data Centers

Recommend use of Transit VSANs over WANs

IVR NAT allows duplication of Domain IDs

All switches in IVR Domain must be configured

Use Auto Topology for simpler topologies

Use Manual Configuration for complex networks

Persistent DomainID and FCIDs are available for HP-UX and AIX

deployments

IVR Summary



SAN Design Overview



A BBA

A B

Core Edge Is Typical SAN DesignOption

Storage on High Density Directors on

the CoreServers at the Edge:

End / Middle of Row with Directors

Top of Rack with Fabric Switches

Blade Chassis with Blade Switches

Total Number of Devices: 1,728

A B

Typical SAN Design



BAA

A

B

B

Edge - Core Edge Option

Storage on Directors SeparateAttached to Core

Core Directors Provide Routing andServices

Servers at the Edge:

End / Middle of Row with Directors

Top of Rack with Fabric Switches

Blade Chassis with Blade Switches

Total Number of Devices: 4,240

Alternate SAN Design

A BA B

A B


22/109


Scalability

Each fabric/blade Switch uses a singleDomain ID

Theoretical maximum Domain IDs is 239per VSAN

Supported number of domains is typicallysmaller ~ 40 (depends on storagevendor)

Manageability

More FC domains / switches to manage

Shared management of blade switchesbetween storage and serveradministrators

SAN Explosion with Fabric and BladeSwitches


23/109


SAN BSAN A

Top-of-Rack Design with Fabric Switches

14 Racks

32 DualAttachedServers perRack

A

B

Rack Mounted Servers

Top of Rack Design

MDS 91x4

MDS 9148

Number of Dual Attached Servers

Number of FC Switches per Fabric

448

29


24/109


SAN Virtualization with N-PortVirtualizer (NPV)


25/109


T11 standard Assigning multiple FC IDs to a single N_Port

Uses FDISC to get additional FCIDs

Shares the physical port but separate logins

NPIV (N_Port Identifier Virtualization)Multiple Logins on a Physical Port

FC

HBA

N_Port Controller

Login1 FCID=1.1.1

Login2 FCID=1.2.1

Login3 FCID=1.2.3

MDS 9000

F_Port

3 Logins

3 FCIDs

FLOGI

FDISC

FDISC

Server


26/109


NPV is a switch modeSwitch acts like a NPIV (N-Port ID Virtualization) host(conceptually an HBA aggregator)

NPV switch uplink is no longer an ISL (NP-port)

NPV switch does not use a Domain ID

Changing from switching mode to NPV mode is disruptive

Simplified Management

Fewer FC domains/switches in the fabric

NPV enabled switch is now managed like a NPIV enabledhost

Enables management by server administrators

N-Port Virtualizer (NPV)Switch Mode Aggregating Multiple HBA Logins

Blade/

FabricSwitch

Core Switch

fc1/2

fc1/3

fc1/1

F

NPV

E

E F

NP

Domain ID 10

Domain Id 20

ISL NPIVNP Link

NPV


27/109


Comparison Between NPIV and NPV

NPIV (N-Port ID Virtualization) Used by HBA and FC switches

Enables multiple logins on asingle interface

Allows SAN to control andmonitor virtual machines (VMs)

Used for VMWare, MS VirtualServer and Linux Xenapplications

NPV (N-Port Virtualizer) Used by FC (MDS 9124, 9134

and 9148), FCOE switches(Nexus 5K), blade switches andCisco UCS Fabric InterConnects(UCS6100)

Aggregate multiplephysical/logical logins to the coreswitch

Addresses the explosion ofnumber of FC switches

Used for server consolidation

applications


28/109


SAN BSAN A

Top-of-Rack Design with Fabric SwitchesUsing NPV

14 Racks

32 DualAttachedServers perRack

A

B

Rack Mounted Servers

Top of Rack Design

MDS 91x4

MDS 9148

Number of Dual Attached Servers

Number of FC Switches per Fabric

448

1


29/109


Uniform balancing of server loads onNP links

Server loads are not tied to any uplink

Benefit

Optimal uplink bandwidth utilization

NPV Auto Load BalancingAutomatic Balancing of Server Loads on NP Links

Blade1

Blade4

Blade Server Chassis

Blade2

SAN

BalancedLoad on NP

Links

Blade3

1

32

4


30/109


External link failure brings downserver connectivity

Applications are disrupted

Uplink Failure Means Traffic Disruption andDowntime to Servers

Blade1

NPV

Blade4


Blade2

SAN

Blade3

X

X X


31/109


F-Port PortChannelsBundle multiple ports in to 1logical link

Similar to ISL portchannels in FCand EtherChannels in Ethernet

Link failures do not affect theserver connectivity

Benefits

High-Availability - no disruption ifcable, port, or line cards fail

Optimal bandwidth utilization &higher aggregate bandwidth withload balancing

No application disruption

F-Port Port ChannelEnhance NPV Uplink Resiliency

StorageBlade System

Blade 1

Blade 2

Blade N

F-Port PortChannel

F-PortN-Port

Core Director

SAN

interface port-channel 1channel mode activeno shut

interface fc1/1channel-group 1

interface fc1/2channel-group 1

X


32/109


F-Port TrunkingUplinks carry multiple VSANs

Benefits

Extend VSAN benefits to Bladeservers

Separate management domains

Traffic Isolation and ability to hostdifferentiated services on blades

F-Port TrunkingVSAN Consolidation on NP Uplinks

StorageBlade System

Blade N

Core Director

VSAN 1

VSAN 2

VSAN 3

F-Port Trunkingon

F-Port Channel

F-PortN-Port

SAN

NPV

Interface fc1/1trunk mode ontrunk allowed-vsan 1-3

Interface port-channel 1trunk mode ontrunk allowed-vsan 1-3

Blade 2

Blade 1


33/109


Deploying NPV Using NPV WizardEnable NPV onBlade Switches

Enable NPIVon SAN Core

Pair NPV Switchesand SAN Core


34/109


Deploying NPV Using NPV Wizard

Setup VSAN

Apply theConfiguration

SetupConnections


35/109


Two levels of NPIV usage

From server to first level switch (NPV)

From NPV to the core SAN

Virtual servers connected to the NPV

devicesServers Supporting NPIV

VMware ESX using RDM mode

Nested NPIVConnecting NPIV Capable Hosts to NPV

NP

F

P2NP

F

P1

NPV Edge Switch

NPV-Core Switch

F F

P3 = vP1 P4 = vP5

vP2

vP3

vP4

vP6

vP7

vP8

NPIV

NPIV


36/109


NPV Scalability

Logins Number of Logins

Logins per Port 126 Gen 1/2

250 Gen 3Logins per Line Card 400 Gen1/2

800 Gen 3

Logins per Switch 2,000

Logins per Physical Fabric 10,000

Blade1

Blade4

Server Chassis

Blade2

SAN

Blade3

1

32

4

SwitchingMode

NPV Mode

Logins per Port 42 114

Logins per Port-Group 168 114

Logins per MDS 9124 1,008 684

Logins per MDS 9134 1,680 1,140

Logins per MDS 9124e 1,008 684

Logins per IBM Blade Switch 840 570

Login per MDS 9148 2,016 1,368

Reference


37/109


Multiple FCIDs to a single port NPV is a switch mode, switch acts like an NPIV aggregator

Solves the domain ID explosion

Simplifies fabric management

F-Port Channel provides failover and load-balancing

Wizard setup for simple configuration

NPV Summary


38/109


Server Virtualization Adoption

Challenges for SANAdministrators


39/109


Virtual Machines (VMs)Do not need pWWNs (no SAN identity)

Have no explicit fabric login

Meaning Virtual Machines without SAN identity

SAN administrator cannot control themSAN administrator cannot monitor them

SAN administrator cannot ensure service levels to them

Server Virtualization and SAN

NPIV All SAN Ad i i t t t


40/109


NPIV gives Virtual Servers SAN identity

Designed for virtual server environments Linux on zSeries, VMware

Allows SAN control of VMsZoning and LUN Masking at VM level

Multiple applications on the same port can use different IDsBetter utilization of the server connectivity

Monitor the VMs using Flowstats

NPIV Allows SAN Administrators toControl and Monitor VMs

LUN1(pwwnD1)

LUN2 (pwwnD2)

LUN3(pwwnD3)

Control andMonitor VMsin the SAN

FC

HBA

N_Port Controller

vpwwn1 FCID=1.1.1

vpwwn2 FCID=1.1.2

vpwwn2 FCID=1.1.3

MDS 9000

F_Port

Virtual Servers

EmailWebPrint

Zone_Emailvpwwn1pwwnD1

Zone_Webvpwwn1pwwnD1

Zone_Printvpwwn1

pwwnD1


41/109


Server Virtualization withFlexAttach


42/109


Server/HBA failures need SANand array configuration change

Configurations use port WWN (pWWN)

Needs co-ordination betweenserver and SAN administrators

Replacement of failed server

Generally needs a change-window

Server and SAN Administrator CoordinationLeads to Inefficient Operations

Blade1

BladeN


Storage

Failed

Blade .

SAN ZoningChange SAN

ArrayConfiguration(LUN Masking)Change

F Port

F Port

NP Port

HBA/ServerFailure

Zone myZonemember pwwn1member pwwn2member pwwnD

pwwn1LUN 0LUN1

pwwn2LUN0LUN2

pwwn1 pwwn2


43/109


How FlexAttach Works?Each F-port is assigned a virtual WWN

Blade server assumes virtual WWN of theport it is connected to

Benefits

Flexibility for Server Mobility - Adds, Movesand Changes

No SAN re-configuration required

Eliminate need for SAN and server team tocoordinate changes

Works only in NPV mode

Cisco FlexAttachFlexibility for Server Mobility

Blade1

BladeN


Storage

Blade2

.

No SAN ZoningChange SAN

No ArrayConfigurationChange

F Port

F Port

NP Port

Flex Attachon

NPV

Zone myZonemember vpwwn1member vpwwn2member pwwnD

vpwwn1LUN 0LUN1

vpwwn2LUN0LUN2

vpwwn1 vpwwn2

pwwn1 pwwn2

VirtualpWWNs

pwwn3


44/109


1. Interface fc1/1 isFlexAttach enabled andassigned a port wwnvpwwn1

2. Server S1 does FLOGIto interface fc1/1

3. pwwn1 FLOGI isrewritten to use vpwwn1FLOGI

4. vpwwn1 FLOGI isconverted to FDISC andregistered with FC NameServer

FlexAttachRewrites Real pWWN to Virtual Port WWN

Port WWN of S1 = pwwn1fc1/1 vpwwn1

Server S1

NPVFlexAttach

N

F

pwwn Rewrite Rules

Core Switch (MDS or 3rd Party Switch with NPIV Support)

Port WWN of S1= vpwwn1

Server S1 Is Known byvpwwn1 in the SAN

NP

F

pwwn1


45/109


No change needed inSAN or on blades

Replace the failed serveronto the same port

No explicit SAN-Server Admininteractions needed

Zero Touch ReplacementSame Port Replacement

NPV

BladeN

Blade Server

Storage

.

SAN Core

Blade1

NPV

BladeN

Blade Server

.Blade2

Blade2

Blade1X

New

Blade

vpwwn1

Zone myZonemember vpwwn1

member vpwwn2

member pwwnD


46/109


Move to a spareserver

Move the virtualpWWN to the newport

No physical

replacement

No explicit SAN-Server Admininteractionsneeded

Replace to a Spare ServerNo Need for Physical Replacement

Blade1

NPV

BladeN

Blade Server

Storage

.

SAN Core

Blade1

NPV

Spare

Blade

Blade Server

.Blade2

Blade2X

fc1/1 fc5/10vpwwn1

vpwwn1

Zone myZonemember vpwwn1

member vpwwn2

member pwwnD

Benefit

Flexibility for server mobilityacross different Bladechassis/Racks

Highly Scalable solution

SAN Pre Provisioning Independent of Servers


47/109


Pre-provision SANfor orderedserversNo need for serversto arrive

Use the FlexAttachgenerated virtual

pWWNsUse planned changecontrol for SANchange

No explicit SAN-Server Admininteractions

needed

SAN Pre-Provisioning Independent of ServersNo Change When the Servers Are Ready to Be Deployed

NPV

BladeN

Blade Server

Storage

.

SAN Core

Blade1

NPV

BladeN

Blade Server

.

New

Blade

Blade2

New

Blade

vp1 vp2

Configure SAN for

vp1 and vp2

Zone myZonemember vp1member vp2member pwwnD


48/109


Virtualize HBA (WWNs) to a switchportAssigns a virtual WWN to a switchport

Eliminates server and storage admin coordination for changes

Allows flexibility for server moves

Eases replacement of servers and HBAs

Pre-provision SAN ports and storage in advance of servers

FlexAttach Summary


49/109


Agenda

SAN Consolidation with VirtualizationInter-VSAN Routing (IVR)


FlexAttach




SANTap



50/109


Tiered Storage / Backup

Service Level Requirements for Application


51/109


Classify and manage dataaccording to application need

Classification Criteria

Access Times/Performance

Application Availability

Recovery Time and Point

Cost of Storage

Optimize the resourceutilization

Service Level Requirements for ApplicationData Vary

Source : EMC / HDS


52/109


Tier2

Tier3

Tier1

Data Classification to Create Storage Tiers

MissionCriticalData

Minutes toHours

Availability

Hours toDay

Availability

Onsite CDP

OffSite CDP

Onsite

Onsite Tape/VTL

OffSiteTape/VTL

Moving the data to the

different tiers using

Data Mobility Manager(DMM)

Network based mirroringusing

SANTap based solution

Encrypt the data for

compliance using

Storage MediaEncryption (SME)


53/109


Implementing Tiered Storage


54/109


Facilitates moving of data froman Existing Storage Pool to aNew Storage Pool via the SAN

Why is it done?

To Upgrade, Consolidate orReplace existing storage

How often?

Typically every 3 years uponlease expiry for a singleStorage Array

Ongoing activity at the IT

Department level consideringthe number of Storage Arrayspresent

Data Migration

Application Servers

SAN Fabric

ExistingStorage NewStorage

Oracle Clearcase Exchange


55/109


Data Migration Techniques Available

Server/Software Based Storage Array Based Appliance Based

ProsNo additional h/w

No re-wiringConsThroughput limited by host bandwidthLarge CPU cycles consumedLonger Migration timeClustered environments not supported

ProsOnline data migrationNo host software or agents.

ConsVendor lock-inLicense

ProsNo host software requiredMore scalable

ConsVirtualizes the source disk(PWWNs change)LUN mapping/masking handlingRe-configuration/Reboot of allhosts accessing this target.

Servers

SAN Fabric

ExistingStorage

NewStorage

Servers

SAN Fabric

ExistingStorage

NewStorage

Servers

SAN Fabric

ExistingStorage

NewStorage


56/109


SAN-based Data Migration

Advantages

SAN moves the data

No server software required

No Virtualization layer in the SAN

Scalable

Referred to as

Cisco MDS Data Mobility Manager [DMM]

Ciscos Data Migration

Application Servers

SAN Fabric

ExistingStorage

NewStorage

Oracle Clearcase Exchange


57/109


Cisco MDS Data Mobility Manager (DMM)

Existing

Storage

SOLARIS1-SRVR

Fabric A Fabric B

New

Storage

SAN based Data Migration

Switch: MDS 9509/9513/9216/9222iLine card : 32 Port Storage Services Module (SSM) or

MSM 18+4

Online/Offline Data Migration

Transparent Insertion/Removal via MDSFC Redirects

Sync Data Migration

Heterogeneous Array Migration

Dual Fabric

Unequal LUN Migration

Rate Adjusted Migration

Server/Storage based migration

Delayed Cut-Over

4.1 TB/Hour data movement rate

GUI for configuration/status

MSM MSM


58/109


Introduce Cisco MDS Data Mobility Engine(SSM / SN) into fabric

No re-configuration/re-wiring of the existing SAN

Enable via Configuration GUI

Server I/O continues transparently to the

Existing DisksUses Ciscos FC Redirect Feature

No SAN-based Volume Management

Data Mobility Engine moves data in thebackground

Heterogeneous Storage Arrays Disable via Configuration GUI

Cisco MDS DMM Online Mode

ExistingStorage

SOLARIS1-SRVR

Fabric

NewStorage

MSM


59/109


A target centric re-direct based transport is a

low level infrastructure used for transportationonlyRe-writing only the FC SID & DID.

Seamless integration of one or moreintelligent services in a fabric for a specificHost & Disk (I_T) pair.

No re-wiring or re-configuring existing Hosts& Disks.

No Splitting of fabrics into multiple VSANs.

Operate in a heterogeneous switchenvironment

Disk must be attached to a FC-Redirect aware

MDS, Host & MSM can be located anywherein the fabric.

Inter-switch communication is done over theexisting CFS infrastructure.

What Is FC-Redirect?

Old Array New Array

Application Server

MSM

DMMPrograms FC-

Redirect to

Send Traffic

Destined to

Old Array to

MSM

FC-Redirect

Traps and

Sends the

Packets to MSM

MSM Sends

Packets to Both

Old and New

Array


60/109


How DMM Works


61/109


The Server has one path to the existingstorage through each Fabric

In a Dual Fabric topology, a DMM Jobrequires 2 DMM modules one in each Fabric

The DMM module in each Fabric must support the VSAN ofinterest in that fabric

The 2 DMM modules in a Dual-Fabric DMMJob establish a IP connection between them

In each fabric, the server I/Os are redirectedthrough the DMM module in that fabric

DMM performs the data copy from theExisting to the New storage using one of the

fabrics. For a Dual Fabric Multi VSAN topology, the

DMM module in each fabric will handlemultiple VSANs

In Each Fabric, all paths from the Server tothe Existing storage will be redirected througha single DMM module

DMM Method 1

ExistingStorage

SOLARIS1-SRVR

SAN A

NewStorage

SAN B

MSMMSM

IP


62/109


Cisco MDS DMM AlgorithmServer

ExistingStorage

LUN

NewStorage

LUN

While (Regions Left) {

Select a Region;

Copy Region;

}

Migrated

Being Migrated

To Be Migrated

Dealing with Server IOs

Writes to MigratedArea Are Mirrored

Writes to BeingMigrated AreaAreQueued Temporarily

(Until Region HasBeen Migrated)

Writes to To BeMigrated AreaAreWritten to Existing

Storage Only

Server Reads Are

Read from Existing

Storage Only


63/109


Supports Dual Fabric Topology

One MSM in each Fabric

MSM performs

Modified Region Log (MRL) updatefor each WRITE I/O

Passes the WRITE I/O to theExisting Storage

Aggregation of MRL bitmap

Data movement for Migration

Server HBA Port, Existing/NewStorage Port needs to be in thesame VSAN (per Fabric)

Eliminates the mirror latency forServer WRITEs

DMM Method 2 - Async

Existing

Storage

SOLARIS1-SRVR

SAN A SAN B

New

Storage

MRL MRL

MRL BitmapMSM MSM

Cisco Data Mobility Manager


64/109


Mark All Regions in MRL Dirty

While (MRL Regions Left) {

Select a Region;

Copy Region;

Clear MRL Region

}

Server

ExistingStorageLUN

NewStorageLUN

Dealing with Server IOs

Writes Are Written to

Existing Storage Only

MRL Entry Is Updated

for Each Write Issued

Server Reads Are

Read from ExistingStorage Only

Modified Region Log [MRL]

Multiple Passes of

MRL Done Until All

Regions Are Clear

For Cut-Over Last

MRL Pass Done with

the LUN in the Offline

Mode

Cisco Data Mobility ManagerAsync Mode


65/109


New Storage in the Remote Data

Center needs to be in the sameVSAN as Server/Existing Storage

Production SAN or VSAN need tospan across Local/Remote DataCenters

Deployed topologyProduction SAN/VSANs constrainedwithin a Data Center

Replication or Migration SAN/VSAN spanData Centers

DMM Method 2 for Data Center Migration

New

Storage

FCIPCloud

RemoteData Center

LocalData Center

Existing

Storage

SOLARIS1-SRVR

SAN A SAN BMRL MRL

MSM MSM


66/109


SAN A and SAN B - Production

Contains: Server/Existing Storage

Traffic: Server to Existing Storage

LUN

Dual Fabric within Data Center

New Storage not visible

Replication SAN

Contains Existing/New Storage

Traffic: via Replication SAN to New

Storage

FC, DWDM or FCIP links

Three SAN/VSAN Topology

New

Storage

Replication SAN

Remote Data Center

Local Data Center

ExistingStorage

SOLARIS1-SRVR

SAN A SAN B


67/109


3 MSMs per DMM Job

MSM 1 / 2 in the Production SAN

Keeps track of Server Write I/Os viaModified Region Log [aka MRL] bitmap

Sends the MRL bitmap over IP to MSM 3

MSM 3 in the Replication SANMerges the MRL bitmap from MSM 1/ 2

Performs data movement from Existing toNew Storage based on the merged MRLbitmap

Replication SAN

Connected via DWDM links within the sameMetro Area

Connected via FCIP links across continents

DMM Method 3

New

Storage

Replication SAN

Remote Data Center

Local Data Center

ExistingStorage

SOLARIS1-SRVR

SAN A SAN B

MSM 1 MSM 2

MSM 3

Merged

MRL


68/109


Deployment Guidelines Do not add the same initiator/target port pair into more than one migration

job simultaneously. When using multipath ports, the server must not send simultaneous I/O

write requests to the same LUN from both multipath ports. The first I/Orequest must be acknowledged as completed before initiating the secondI/O request.

DMM is not compatible with LUN zoning.

DMM is not compatible with inter-VSAN routing (IVR). The server andstorage ports must be included in the same VSAN.

DMM is not compatible with SAN device virtualization (SDV).

DMM does not support migration to a smaller destination LUN.


69/109


There Are Two Types of DMM Licenses: Permanent License: This license (also called End User license) is only availableto end users that will be deploying DMM for their own data migration needs. Thepermanent license may not be used by users that expect to use the MDS platform(with the SSM / SN card) to sell migration services to other users.

180-day License: This license is a time-based license that is available to service

provider users that expect to sell MDS platform-based migration services. Usersthat qualify for the permanent license may purchase the 180-day License if theyso choose to do so.

Cisco MDS DMM Licenses


70/109


Method 1

2 SAN topologyServer, Existing/New Storage connected to each SAN and are in thesame VSAN

Server WRITE I/Os mirrored to Existing/New Storage in both SANs

Data Movement performed in one of the SANs

Method 2 - Async

2 SAN topologyServer, Existing/New Storage connected to each SAN and are in thesame VSAN

MRL bitmap tracks Server WRITE I/Os in both SANs

Data Movement performed in one of the SANs

Method 3 Data Center Migration3 SAN topology : 2 Production SANs and 1 Replication/Migration SAN

Server and Existing Storage connected to the Production SAN

Existing/New Storage connected to Replication/Migration SAN

MRL bitmap tracks Server WRITE I/Os in the Production SAN

Data Movement performed in the Replication SAN

Built on Method 2

Cisco DMM Recap


71/109


Storage Media Encryption(SME)


72/109


Cisco SME - Secure, Integrated Solution

Virtual Tape

Library

Tape

Devices

Application

Server

Name: XYZSSN: 1234567890Amount: $123,456Status: Gold Key Management

Center (KMC)TCP/IP

Name: XYZSSN: 1234567890

Amount: $123,456Status: Gold

@!$%!%!%!%%^&*&^%$#&%$#$%*!^

@*%$*^^^^%$@*)%#*@(*$%%%%#@

Encrypt

Disk

Array

2H CY2011

Encrypts media for SAN attached tapes,virtual tape libraries and disk arrays

Uses IEEE AES-256 encryption

Disk XTS, Tape GCM

CC EAL-3 and FIPS 140-2 certified switch

Solution includes Cisco KMC forprovisioning and key management

Integration with RSA Key Manager

Handles traffic from any VSAN in fabric Compresses tape data equal or better

than tape drives

Offline data recovery tool decrypts tapewithout MDS 9000 using Linux server


73/109


Delivering Encryption as a SAN Service

Name: XYZSSN: 1234567890Amount: $123,456Status: Gold

Name: XYZSSN: 1234567890Amount: $123,456Status: Gold

1. Insert Cisco MSM-18/4 or SSN16 modules or MDS 9222i switches2. SME is a licensed feature

3. Enable Cisco SME and setup encryption service

4. Provision encryption for specific storage devices

MDS 9500

SeriesMDS9200

Series

Storage MediaEncryption Service

@!$%!%!%!%%^&*&^%$#&%$#$%*!^@*%$*^^^^%$@*)%#*@(*$%%%%#@

@!$%!%!%!%%^&*&^%$#&%$#$%*!^@*%$*^^^^%$@*)%#*@(*$%%%%#@


74/109


Cisco SME - Scalable, Highly Available

Integrates transparently in MDS fabrics

using FC-redirect

Allows rapid deployment

No SAN re-configuration or re-wiring

Provision as a simple, logical process of

selecting what to encrypt

Provision at the data center level and notat the module level

Modular, clustered solution offers highlyscalable and reliable performance

Up to 4 switches and 32 encryption units

Support dual fabric configurations

Automatically load balances

Redirects traffic if a failure occurs

Disk Arrays, Tape Drives and VTLs

Media Servers

MSM-18/4 MSM-18/4


75/109


Cisco SME Disk Data Flow

Disk Array

Host

MSM-18/4 MSM-18/4

Dual-Fabric cluster

Traffic encrypted on all paths

Operations

Data Preparation

Rekey

Modes

Offline


76/109


Wizard-Based Provisioning

Cisco SME

is ready !

Wizard 1 Creating a Cluster

Selects Encryption Modules

Defines Key Management Policies

Generates and Stores Master Key

Wizard 2 - Adding a Tape Group

Selects Media Servers

Specifics Devices to Encrypt Tape

Volumes On


77/109


SME Key Management

Cisco KMC provisions and transportskeys securely

No new software required, based on CiscoFabric Manager

Managed through web browser interface

Provides essential key managementfunctions:

Archiving, replicating, recovering, andpurging media keys

Logging Cisco SME transactions

Accommodates single and multiple siteenvironments

Integration with RSA Key Manager

Cisco Key

Management Center

Disk Arrays,

Tape Drives

and VTL

Application Servers

Fabric A Fabric B

RSA Key Manager

MSM-18/4 MSM-18/4


78/109


Master Key Protection

Complexity

LevelofSecu

rity

A file with all master keys

Master keys encrypted with a password

Regular backup and archive

Basic

Smart Cards with Recovery Shares for Each

Master KeyWhere M of N Recovery Officers

Are Required to Recover a Master Key

Advanced

Smart Cards with All Master Keys

No Recovery Shares

Standard

Smart Cards

Options:2 of 3

2 of 5

3 of 5


79/109


Tape Key

SME Tape Key Hierarchy

Master key resides in smartcardsQuorum (M out of N) of smartcardsrequired to recover a master key

Recovery shares accomplishsecret sharing

Keys reside in clear-text onlywithin crypto boundary onswitch module

Unique key per tape, or per tapevolume group

Media keys wrapped by masterkey before storage or transportto Cisco key managementcenter

Option to store tape keys ontape media

Cisco KeyManagement

Center Tape VolumeGroup Key

Tape Key

Master Key


80/109


Secure System Architecture

Hardware and software architecture designed to meet FIPS140-2level two certification requirements

Tamper-evident: attempts to tamper with system are immediately

visible

Strong, standard AES-256 modes of encryption

Smart cards available for master key protection

Critical security parameters and media keys never leavesystem unencrypted

Role-based access control (RBAC) secures management

Enforces SME-specific roles

AAA server support allows centralized user authentication and accounting

(auditing)


81/109


Roles and Identities

SME Storage Admin, KeyManagement

Per-VSAN role-based accesscontrol (RBAC) limitsmanagement scope

SME Storage Administrator isresponsible for managing tapedevices and volume groups

SME Key Management role isresponsible for keyimport/export, archiving, etc.

SME Recovery Officer

Responsible for any recoveryfunction requiringa master key

Quorum of recovery officersneeded to perform recoveryprocedures (default is two out offive)

Security operations (SecOp)

staff may assume this roleFully Integrated with MDS CLI and

GUI RBAC (TACACS+, RADIUS)

SME Design Guide White Paper:

http://www.cisco.com/en/US/prod/collateral/ps4159/ps6409/ps4358/design_guide_c07-464433.html


82/109


SME Disk


83/109


SME Disk Overview

EMC1454-ES

SOLARIS1-SRVR

Fabric A Fabric B

SME NodeModule 8

SME NodeModule 4

Available NX-OS 5.2(1), 2nd Half 2011

SME Node

18+4 MSM/9222i

Encryption of Data flowing betweenServers and Storage

Dual Fabric Topology

Encryption performed on all the Fabricpaths

Supports SME Clusters

Supports SME Key Management

Encryption Encryption

SME NodeModule 2

SME NodeModule 9

SME Cluster

Clear Text I/O

Dual Fabric Data Center SAN Topology

Encrypted I/O


84/109


Fabric A

SME Disk Configuration Model Disk

Disk1,Disk 2,Disk3,Disk4

Multiple accessible paths [I,T,L]

Disk1:[HA, TA], [HB, TB]

Disk2:[HC, TA], [HD, TB]

Disk3:[HC, TA], [HD, TB]

Disk4:[HA, TA], [HB, TB]

A crypto disk has the following components:

Diskgroup name:adminassigned

Disk name: admin assigned

Zero or more paths

State: CLEAR, CRYPTO etc

Zero or one active key in the KMC

Zero or more archived keys in the KMC

Disk Group

A administrative label used to group a collection ofcrypto disks

Recorded as part of the crypto disk name in KMC

Storage Array

SRVR1 SRVR2

HAHB HC

HD

TA TBSRVR1 LUN MapServer Ports->HA,HB

LUN 0 : Disk 1LUN 1 : Disk 4

SRVR2 LUN MapServer Ports->HC,HD

LUN 0 : Disk 2LUN 1 : Disk 3

Fabric B

Disk Group

Disk 1

Disk 2

Disk 3

Disk 4

SME Node SME Node


85/109


SME Disk Key Hierarchy

A Two-Tier Hierarchy Is Used by CiscoSME Disk

LUN key encrypts data on the disk

These keys are unique for each LUN

Stored in the KMC, encrypted by theMaster Key

Master key encrypts LUN keys

Generated when a cryptographic clusteris created

There is a unique master key for eachcluster

LUN

LUN Key

Master Key

Smart Card

SME Key Management White Paper:

http://www.cisco.com/en/US/prod/collateral/ps4159/ps6409/ps4358/white_paper_c11-462423_ps6028_Products_White_Paper.html


86/109


Shared Infrastructure Between SME-T and SME-D

Cluster infrastructure remains unchanged and the existing cluster configuration

commands can be used as is for creating an SME cluster for disks

The crypto node configuration remains unchanged and the existing sme interface

config commands can be used as is for SME Disks

The KMC configuration remains unchanged. However there are backend changes due

to differences in usage of KMC for tapes vs disks

The discovery of IT nexus pair for SME cluster remains unchanged. However there are

backend changes due to differences in handling of the discovery of tapes vs disks

In short the existing SME configuration guide titled Cisco MDS 9000 Family Storage

Media Encryption Configuration Guide should suffice for the above components

As per design and implementation (pending testing) SME Tape and SME Disk should

be able to co-exist in the same SME cluster with the following underlying

understanding:

SME tape backup group and SME diskgroup share the same name space, i.e. a disk group cannot have the

same name as tape backup group name

An IT nexus will either have all tape devices or all disk devices


87/109


Cisco SME Review (1)

Architecture and FlexibilityIn addition to SME, SSN-16 can support multiple applications

Services Oriented SAN solution, not just a point product

SAN-level Provisioning and automatic load-balancing

Automatic assignment of flows to service engines, no static/manual configuration

required

High-availability of encryption engines

Integrated Clustering and HA

N+1 availability; in case of a failure, any available engines in the fabric picks up theload


88/109


Cisco SME Review (2)

Linear ScalabilityUp to 10 SSN16s in MDS 9513 and up to 40 encryption engines in a SAN. Adding anengine linearly increases capacity and throughput

Key Management

Integrated free key management solution as well as support for external enterprise

key manager Mix of SME Tape and SME Disk on the same SSN16

Save cost by eliminating the need for separate hardware


89/109


SANTap


90/109


SANTap

Appliance

Target

Initiator

SAN

Copy ofPrimary

I/O

= SAN Tap

Initiator Target I/O

Enables appliance-basedstorage applications withoutcompromising SAN integrity

About SAN Tap

MDS delivers a copy of primaryI/O to an appliance

Appliance provides the storageapplication

Examples of applications includeContinuous Data Protection (CDP),replication, etc.

Key customer benefitsPreserve integrity, availability, andperformance of primary I/O

No service disruption

Investment protection


91/109


Ease of Deployment

Insert Cisco MSM-18/4, MDS 9222i switches, or SSM Module

No rewiring required The hosts and targets do not have to be connected to MSM

No need to reconfigure hosts and targets

- The hosts continue to see the same WWNs for storage- The targets continue to see the same WWNs for host

SANTap is a licensed feature

MDS 9500

SeriesMDS9200Series

SANTap Service

ApplianceInitiator

Target


92/109


SANTap at Work

SANTap mirrors

write I/Os to RPA

Host VSAN

Target VSAN

WAN

ProductionLUN

LocalCDPCopy

LocalCDPJournal

LUN

SAN

ApplianceRecoverPoint

ApplianceRecoverPoint

SANTap out-of-

band fabricsplitting preserves:

I/O integrity

I/O availability

I/O performance

Remote SiteLocal Site


93/109


TargetVSAN

SANTap Configuration

Host pWWN =10:00:00:00:c9:a5:a6

HostVSAN

= SAN Tap

Host VSAN contains hostpWWN and Data Virtual Target(DVT) pWWN

DVT is real pWWN of target

port Target VSAN contains target

pWWN and Virtual Initiator (VI)

VI is real pWWN of host port

No need for devices to move toother switch/ports to work withSANTap

DVT pWWN =50:00:1f:e1:50:3b:09

Target pWWN =50:00:1f:e1:50:3b:09

Host VI pWWN =10:00:00:00:c9:a5:a6

ApplianceCopy ofPrimary

I/O

SANT d R P i D Fl


94/109


1

4

SANTap and RecoverPoint Data Flow

Host

Appliance Appliance

Array

Local Data Center

1

SANTap

2

2

3

3

4

LOCAL FLOW1. Write I/O is sent to MSM module2. Write I/O is then forward to both local Storage Array

and local Appliance3. Both local Storage Array and local Appliance

acknowledge Write I/O back to the MSM4. Once MSM receives both acknowledgements, then

sends acknowledgment to Application Server

REMOTE FLOW1. I/O is sent through the WAN to remote Appliance2. I/O is then sent to replication LUN(s) through the MSM3. I/O is then acknowledged back to the Remote

Appliance4. Remote Appliance then sends acknowledgement

back to Primary Data Center Appliance through theWAN

SANTap

Array

WAN

2 3

SANTap

Remote Data Center

Host

1

2

SANT S


95/109


Appliance-based storage application

MDS deliver a copy of I/O to the appliance

Enables Continuous Data Protection and Recovery

Copy of I/O is not in primary data path

No SAN re-wiring or reconfiguration required to implement

SANTap Summary

A d


96/109


SAN Consolidation with Virtualization

Inter-VSAN Routing (IVR)


FlexAttach




SANTap


Agenda


97/109


FCoE

FC E C lid ti Hi h


98/109


FCoE: Consolidation Highway

I/O Consolidation

Consolidates separate LAN, SAN, and server cluster network environments into aunified fabric.

Multi core CPU architectures driving increased network bandwidthdemands

Virtual Machines driving increased I/O connections and bandwidth Fibre Channel Prevalent Storage Solution

Same operational model as today

Incremental Implementation

Start at the Edge

Leverage FC tools investment and management applications

Low latency 10GE affordability (even optics)

S C ti it T d


99/109


Server Connectivity: Today

SAN A SAN B10GEBackbone

10GE

4/8 Gbps FC

S C ti it U ifi d ith FC E


100/109


Server Connectivity: Unified with FCoE

SAN A SAN B10GEBackbone

10GE

4/8 Gbps FC

Nexus

10GE FCoE

Session Summary


101/109


Session Summary

SAN Consolidation with VirtualizationInter-VSAN Routing (IVR)


FlexAttach




SANTap



102/109


Q&A

Other Sessions


103/109


BRKSAN-1121: SAN Core Edge Design Best Practices

BRKSAN-2047: FCOE Design, Operation, and Management BestPractices

BRKSAN-3123: Storage Cloud Concept and Design

BRKSAN-2704: SAN Extension Design and Operation

BRKDCT-1044: FCoE for the IP Network Engineer

Other Sessions

Additional Information


104/109


Cisco Storage Networking

http://www.cisco.com/go/storagenetworking

Cisco Data Center Networking

http://www.cisco.com/go/datacenter

Storage Network Industry Association (SNIA)

http://www.snia.org

Internet Engineering Task ForceIP Storage

http://www.ietf.org/html.charters/ips-charter.html

ANSI T11Fibre Channelhttp://www.t11.org/index.htm

Additional Information

Recommended Reading
http://www.cisco.com/go/storagenetworkinghttp://www.cisco.com/go/datacenterhttp://www.snia.org/http://www.ietf.org/html.charters/ips-charter.htmlhttp://www.t11.org/index.htmhttp://www.t11.org/index.htmhttp://www.ietf.org/html.charters/ips-charter.htmlhttp://www.ietf.org/html.charters/ips-charter.htmlhttp://www.ietf.org/html.charters/ips-charter.htmlhttp://www.snia.org/http://www.cisco.com/go/datacenterhttp://www.cisco.com/go/storagenetworking


105/109


Continue your Cisco Live learningexperience with further reading fromCisco Press

Check the Recommended Readingflyer for suggested books

Recommended Reading

Available Onsite at the Cisco Company Store

Complete Your OnlineSession Evaluation


106/109


Session Evaluation

Receive 25 Cisco Preferred Access points for each sessionevaluation you complete.

Give us your feedback and you could win fabulous prizes.Points are calculated on a daily basis. Winners will be notifiedby email after July 22nd.

Complete your session evaluation online now (open a browserthrough our wireless network to access our portal) or visit oneof the Internet stations throughout the Convention Center.

Dont forget to activate your Cisco Live and Networkers Virtual

account for access to all session materials, communities, andon-demand and live activities throughout the year. Activateyour account at any internet station or visitwww.ciscolivevirtual.com.
http://www.ciscolivevirtual.com/http://www.ciscolivevirtual.com/


107/109


Visit the Cisco Store forRelated Titles

http://theciscostores.com
http://theciscostore.com/http://theciscostore.com/


108/109



109/109

Thank you.

Advance SAN Services

Documents