Design and Solutions SAN Extension Paresh Gupta, Technical Marketing Engineer, Cisco Mark Allen, Manager Technical Marketing, Cisco January 2017
Design and SolutionsSAN ExtensionParesh Gupta, Technical Marketing Engineer, Cisco
Mark Allen, Manager Technical Marketing, Cisco
January 2017
2© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Resilience
Safety
Restoration
Business Continuity
Industry wide Ecosystems
Infrastructure
Planning
Competitive EdgeRedundancy16 member PortChannel
Disaster Recovery
Incident Management
Cloud Storage Protection
Customer SatisfactionStability
Restoration
Crisis Management
Investment Protection
RiskWhy
SAN Extension
3© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Hope for the best, plan for the worst
- Lee Child
4© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Recovery Point and Recovery Time Objective
Time
DisasterStrikes
Recovery PointLast Point Where
Data in Usable State
Recovery TimeSystem Recovered
and Operational
How far back? How long to recover?
Shorter RPO/RTO�̶ Higher $$$�̶ Replication�̶ Hot standby systems
Longer RPO/RTO�̶Lower $$$�̶Tape backup/restore�̶Cold standby systems
5© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
SANExtension
Minimize RPOData should be in sync before & after disaster
Minimize RTORecovery should be quick
Investment ProtectionInvestment for more than a decade
Choice of ProtocolFC or FCoE or FCIP
Acceptable LatencyLatency within acceptable limit ofReplication or Backup application
DistanceHow far is the recovery site?
$$
SecurityLink Encryption for security of data in motion
CostDesign within the budget
High AvailabilityIncreased availability
Considerations for SAN Extension Design
7© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Cisco Multi-Protocol Product Portfolio
12+ Years of Proven NX-OS Operating System Cisco Prime Data Center Network Manager (DCNM)
Cisco MDS9700
48x16G Line-Rate FC
LAN/SAN SAN COMPUTE
Cisco UCS C-Series
Rack Servers
Cisco UCS B-SeriesBlade Servers
Cisco UCS 6300 Series FI
Cisco UCS 6200 UP
Cisco Nexus 9000
Cisco Nexus 7000
Cisco Nexus 5600
Cisco Nexus 5500
CiscoNexus 3000
CiscoNexus 2000
24 x40GFCoE
Cisco MDS9250i
Cisco MDS 9148S
48x10G Line-Rate FCoE
Cisco MDS9396S
Nexus 5672UP-16G
16G FC: Nexus 2348UPQ
16G FC, 40G FCIP
Consistent and SimplifiedFeatures, Management, and Programmability
8© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Cisco MDS 9000 Switch Family
9RU
MDS 9710 MDS 97064 module slotsUp to 192 ports
14RU
26RU
MDS 97188 module slots
Up to 384 ports
FCIP SAN Extension
16 module slotsUp to 768 ports
MDS 9148S MDS 9396S MDS 9250i
48 x 16G FC
48 x 10Gbps FCoE
24 x 40 Gbps FCoE
24 x 16G FC, 8 x 1/10 GE & 2 x 40 GE
SAN Directors Director Modules
Fabric Switches
9© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Typical SAN Design
• Dual fabric design : 2 Fibre Channel connections from Server to Storage• Multipath software provides high availability• Separate Access and Replication fabrics• Dual fabrics maintained over SAN extension
Replication FabricReplication FabricAccess Fabric
“B” Fabric
“A” FabricDC
InterconnectNetwork
SiteA SiteB
MDS
MDS
MDS
MDS
MDS
MDS
10© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Introducing Virtual SAN (VSAN)
• Dual fabrics (E.g., yellow VSAN and red VSAN) over distance• Inter Switch Link (ISL) carry multiple VSANs (known as trunking)• Each VSAN maintains it’s own fabric services
• FSPF: ‘Fabric Shortest Path first’ for route calculation• Name server, zoning database, etc.
Replication VSANReplication VSANAccess VSAN
“B” Fabric
“A” FabricDC
InterconnectNetwork
SiteA SiteB
MDS
MDS
MDS
MDS
VSANs – Increased redundancy, scalability and reduced cost
11© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
High Availability (HA) replication design
• Client based protection by• PortChannel• Storage arrays• Rerouting by FSPF
• Network based Protection by • Optical protection schemes
Replication VSANReplication VSANAccess VSAN
“B” Fabric
“A” Fabric
SiteA SiteB
MDS
MDS
MDS
MDS
12© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Link HA via PortChannel
• Multi-protocol support : FC, FCoE or FCIP• Up to 16 members in a port channel• Increased Resilience and availability
• Single logical link• No FSPF re-calculations when members go down
• Route member links over diverse geographic paths
Replication VSANReplication VSANAccess VSAN
“B” Fabric
“A” Fabric
SiteA SiteB
MDS
MDS
MDS
MDS
13© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Extending optical FC SAN : B2B credit requirement
Frame Size 1 Gbps 2 Gbps 4 Gbps 8 Gbps 10 Gbps 16 Gbps
512 Bytes 2 BB/km 4 BB/km 8 BB/km 16 BB/km 24 BB/km 32 BB/km
1024 Bytes 1 BB/km 2 BB/km 4 BB/km 8 BB/km 12 BB/km 16 BB/km
2112 Bytes 0.5 BB/km 1 BB/km 2 BB/km 4 BB/km 6 BB/km 8 BB/km
B2B credit requirement increases with
Distance Speed Frame size
B2B
cre
dit
requ
irem
ent
14© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
SAN Extension over Dark Fiber
Replication VSANReplication VSANAccess VSAN
“B” Fabric
“A” Fabric
SiteA SiteB
• 1/2/4/8/10/16 Gbps FC• Distance : limited due to optics and fiber cable
• SW or LW (10 KM) or ER (40KM) optics. OM1, OM2, OM3 or OM4 cables
• Client protection only : PortChannel / Storage arrays / Rerouting by FSPF• Loss of path reduces bandwidth of only one fabric by 50%
• Cost: Low
15© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
SAN Extension over CWDM Network
• Colored CWDM SFPs (8G FC) used in FC switches (no transponder required)• Distance : limited due to optics and fiber cable and dB loss in MUX (max 40 KM)• Client protection only : PortChannel / Storage arrays / Rerouting by FSPF
• Loss of path reduces bandwidth of both “A” and “B” fabrics by 50%• No topology change (no FSPF recalculation)
• Cost: Fair
Replication VSANReplication VSANAccess VSAN
“B” Fabric
“A” Fabric
SiteA SiteB
MUX
MUX
MUX
MUX
MDS
MDS
MDS
MDS
16© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Dense Wavelength Division Multiplexing (DWDM)• Up to 32 channels per fiber• Longer Distance than CWDM : Use of Erbium-Doped Fiber Amplifiers (EDFA)• Multi Protocol Capability for data center to data center connectivity
• 1, 2, 4, 8, 10 or 16 Gbps FC, FICON, GigE, 10GigE, ESCON, IBM GDPS
• Client Protection : PortChannel / Storage arrays / Rerouting by FSPF • As well as Network Protection : Splitter / Line card
Optical Splitter Protection
Protected Lambda
Optical Splitter
Working Lambda
MDS MDS
Linecard or Y-Cable Protection
Y-cable
MDS MDS
Single transponder required Dual transponders required, more expensive
Protects against fiber breaks Protects against fiber breaks and Line card failure
17© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
SAN Extension over DWDM Network – Option 1
• Optical network sharing by both fabrics • Client protection via PortChannel – Recommended
• Single fiber cut will not affect fabric• Loss of path reduces bandwidth of both “A” and “B” fabrics by 50%
• Cost : High
Replication VSANReplication VSANAccess VSAN
“B” Fabric
“A” Fabric
SiteA SiteB
DWDM Ring
MDS
MDS
MDS
MDS
18© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
SAN Extension over DWDM Network – Option 2
• Dedicated optical network per fabric • Client protection via PortChannel – Recommended
• Single fiber cut will not affect fabric• Loss of path reduces bandwidth of only one fabric by 50%
• Cost : High+
Replication VSANReplication VSANAccess VSAN
“B” Fabric
“A” Fabric
SiteA SiteB
DWDM Ring
DWDM Ring
MDS
MDS
MDS
MDS
19© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
SAN Extension Technology Options
Limited by Optics (Power Budget)Dark Fiber1/2/4/8/10/16G FC, 10GE FCoE
CWDM1/2/4/8G FC, 10GE FCoE
DWDM1/2/4/8G FC, 10GE FCoE
SONET/SDH1/2/4G FC
Data Center Campus Metro Regional National
Increasing Distance
Sync
Sync
Sync
Limited by Optics (Power Budget)
Limited by B2B_CreditsOpt
ical
Async
Global
Sync
ProtectionClient Network
Cost
Cost
Cost
Cost
20© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Fibre Channel over Internet Protocol (FCIP)
SiteB FC SANSiteA FC SAN
IP Network
FCIP TunnelMDS MDS
Single FSPF routing domain
• IETF standard for Linking Fibre Channel SANs over IP (RFCs 3821 & 3643)• Point-to-point tunnel between FCIP link end-points
21© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
FCIP Frame Details
• Segmentation and reassembly at default MTU of 1500 B (performance hit)• Recommendation: End to end IP MTU of 2300 bytes• All Cisco FCIP products support jumbo frames
+ +
FCIPHeader
EthernetHeader
IPHeader
TCPHeader
TCPOpts FC Frame
Ethernet
CRC3214 20 20 12 28 4
94
EISLHdr
SO
F
4 8
2172
VSAN Routing for TE port
optHdr0-16
RTTM is constantly measured for Round Trip Time
2270=
4
22© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
SAN Extension over FCIP
• Client protection via PortChannel – Recommended• Portchannel individual FCIP links to separate Ethernet switches/routers• Each WAN link carries two FCIP tunnels• Global reach : Reliable delivery by TCP, No B2B credit requirement on FCIP link
IP network
Replication VSANReplication VSANAccess VSAN
“B” Fabric
“A” Fabric
SiteA SiteB
MDS
MDS
MDS
MDS
23© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
SAN Extension Technology Options
Limited by Optics (Power Budget)Dark Fiber1/2/4/8/10/16G FC, 10GE FCoE
CWDM1/2/4/8G FC, 10GE FCoE
DWDM1/2/4/8G FC, 10GE FCoE
SONET/SDH1/2/4G FC
Data Center Campus Metro Regional National
Increasing Distance
Sync
Sync
Sync
Limited by Optics (Power Budget)
Limited by B2B_CreditsOpt
ical
Async
Global
Sync
ProtectionClient Network
Cost
Cost
Cost
Cost
23
Async (WAN)MDS9000 FCIPGE, 10GE IP
Sync (Metro Eth)Cost
24© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Native FCoE SAN ExtensionFCDCB/FCoEEthernetFCoE
Replication VSANReplication VSANAccess VSAN
“B” Fabric
“A” Fabric
SiteA SiteB
Nexus
Nexus
Nexus
• FCoE SAN Extension uses same design principles as FC or FCIP• Separate VLANs/VSANs for Host and replication traffic• Multiple geographical diverse paths• Client (PortChannel) or Network Protection (DWDM or SONET/SDH)
• Distance depends on underlying media and Buffer (instead of B2B credits)• Typical FCoE SAN Extension rely on FC or FCIP to transport data over distance
25© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Native FCoE SAN ExtensionFCDCB/FCoEEthernetFCoE
• FCoE Attached disk replication uses FC or FCIP transport network• SAN Extension design criteria based on FC or FCIP interconnect network
Replication VSANReplication VSANAccess VSAN
“B” Fabric
“A” Fabric
SiteA SiteBNexus
MDS 9250i
MDS 9700
Nexus
MDS 9250i
MDS 9700
27© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Link Layer Security
IPNetworkDWDM
Name: XYZSSN: 1234567890Amount: $123,456Status: Gold
@!$%!%!%!%%^&*&^%$#&%$#$%*!^@*%$*^^^^%$@*)%#*@(*$%%%%#@
FC TrustSec IPSec
Name: XYZSSN: 1234567890Amount: $123,456Status: Gold
Name: XYZSSN: 1234567890Amount: $123,456Status: Gold
@!$%!%!%!%%^&*&^%$#&%$#$%*!^@*%$*^^^^%$@*)%#*@(*$%%%%#@
Primary DC
Secondary DC
Backup DC
• Hardware supported, no additional latency• DH-CHAP used for peer authentication• Encryption: AES 128 bit key
Fibre Channel TrustSec• Hardware support, no additional latency• Encryption: AES (128 or 256 bit key), DES
(56 bit), 3DES (168 bit)
IPSec
28© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Application I/O Acceleration• Distance impacts performance of disk replication and tape backups• Latency due to distance is compounded by multiple round trips per command
28
I/O Accelerator (IOA)
disk and tape over FC or FCIP
Write Accelerationdisk over FCIP (FCIP-WA)
Tape Acceleration tape over FCIP (FCIP-TA)
Solution
MDS 9250i24/10 SAN Extension Module (SEM)
29© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
FCIP Data Compression• Compression increases link data capacity or reduce consumed bandwidth• Two compression modes available
• Auto – Optimizes Compression based on bandwidth and data rate (Recommended)• Mode2 – Deflate based compression algorithm
• Data Compressibility is data stream dependent• All nulls or ones → high compression (>30:1) • Random data (e.g., encrypted) → low compression (~1:1)• “Typical” rate is 4:1 (MDS 9250i and 24/10 SEM), but may vary considerably
• Application throughput is the most important factor
Supported in HW on MDS 9250i and 24/10 SEM without any additional latency
31© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
SAN Extension Best PracticesDo not leave FCIP configuration to default values
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 320
5
10
15
20
25
30
35
40
45
Exponential “Slow Start”(increase 2x pkts per RTT)
LossLoss
# Round Trip Times (RTT)
Packets S
ent per Round Trip Low Throughput During
This Period
Linear “Congestion Avoidance”
(MDS +2/cwnd per ack)(TCP +1/cwnd per ack)
Congestion Window Halved on Packet Loss; Retransmission
Signals Congestion…Slow Start Threshold Adjusted
Traditional TCPMDS TCP implementation
32© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
SAN Extension Best Practices
• In built Shaper sends at a rate consumable by the downstream path• Immediately sends at “minimum-bandwidth” rate (avoids early stages of traditional
slow start)• Ramps up to “maximum-bandwidth” rate (using usual slow start and congestion
avoidance methods)
Configure TCP max and min bandwidth
Dedicated link Shared link with other FCIP tunnel
Shared link with non-storage traffic
max= path bandwidth (BW) max = allocated for this tunnel max = (link BW – other traffic BW)
min = 95% of max min = 80-95% of max min = 80-95% of max
switch(config-profile)# tcp max-bandwidth-mbps 1000 min-available-bandwidth-mbps 900 round-trip-time-ms 10
33© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
SAN Extension Best Practices
• TCP window size is directly proportional to RTT• Set correct RTT to fully utilize WAN pipe• Do not leave to default (1 ms)• Use ‘ping’
• OR ‘ips measure-rtt’ (preferred) to determine RTT
Configure correct value for Round Trip Time (RTT)
MDS9000# ping 192.168.20.2PING 192.168.20.2 (192.168.20.2) 56(84) bytes of data.64 bytes from 192.168.20.2: icmp_seq=1 ttl=254 time=0.740 ms64 bytes from 192.168.20.2: icmp_seq=2 ttl=254 time=0.621 ms64 bytes from 192.168.20.2: icmp_seq=3 ttl=254 time=17.8 ms
MDS9000# ips measure-rtt 192.168.20.2 interface ipStorage 1/2Round trip time is 111 micro seconds (0.11 milli seconds)
34© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
SAN Extension Best Practices
• Set proper IP DSCP values to prioritize FCIP traffic
Apply proper QoS policies when sharing link between storage and other traffic
35© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
FCIP Capacity Planning
FCIP LinkIP MAN / WAN
SiteA SiteB
MDS MDS
• Multiple parameters must be tuned to keep the WAN pipe full• TCP Parameters (Window size, max BW, Round trip time, SACK…) • Outstanding I/Os• Transfer size
• Standard traffic generating tools (like IOmeter) can be used• Requires test hosts and target
36© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
SAN Extension Tuner (SET)
• Lightweight tool (Only SCSI Read and Writes) integrated with NxOS on MDS• Allows you to configure an unused iSCSI interface as a FC Initiator and Target• Generates custom traffic and reports parameters
• I/O per second, Throughput, Round Trip Time, Compression ratio
• Configured by CLI or GUI (Cisco DCNM aka Fabric Manager)
FCIP LinkIP MAN / WAN
SiteA SiteBIPStorage1/1
IPStorage1/2 IPStorage1/2
IPStorage1/1
MDS MDS
37© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Top 3 pitfalls to avoid1. Do not over-complicate the design2. Understand the QoS policy. Storage traffic must be subjected to tcp-max-bw, not
the typical traffic policing3. Understand when to use Inter-VSAN Routing
38© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
What Cisco has done really wellInvestment Protection
48 x 16G FC
48 x 10Gbps FCoE
24 x 40 Gbps FCoE
24 x 16G FC, 8 x 1/10 GE & 2 x 40 GE
• All current (and future) modules can be used in any slot without any restrictions on MDS 9700
• Full FCIP backward interoperability is maintained
• 24/10 SEM module can be connected to MDS 9250i, MDS 9222i and SSN-16 module (for MDS 9500)
• Protects your investment for more than a decade
• Not mandatory to upgrade remote locations just because you upgraded the primary location
MDS 9250i
16 port GigE Storage Services Node (SSN-16)
MDS 9222i
FCIP
41© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Advanced FeaturesExtended Credits FCIP WACompression
Encryption
FCIP TAIOA QoSSET
FCIP TuningTCP max and min bandwidth
Round Trip Time (RTT)
SAN Extension for Business Continuity and Disaster Recovery