Achieving Ent SAN Perf 48000 Dir WP 01

8/3/2019 Achieving Ent SAN Perf 48000 Dir WP 01

1/18

STORAGE AREA

NETWORK

Achieving Enterprise SAN

Performance with the

Brocade 48000 Director

SAN WHITE PAPER

A best-in-class architecture enables the widest range

of efciency, performance, and exibility advantages.


2/18

The Brocade 48000 Director is the industrys highest-performing

platform for supporting enterprise-class Storage Area Network

(SAN) operations. With its intelligent fth-generation ASICs and new

hardware and software capabilities, the Brocade 48000 provides

a reliable foundation for fully connected multiprotocol SAN fabrics,

FICON solutions, and Meta SANs capable of supporting thousands of

servers and storage devices.

This paper describes how IT organizations can leverage the benets

of this SAN director to maximize performance, exibility, and data

availability in mission-critical environments. In addition to summarizingthe architectural advantages of the Brocade 48000, this paper explains

how the various blades used in the platform can help optimize

performance to address specic requirements.

For more information about SAN design or additional Brocade solutions,

such as the Brocade Multiprotocol Router, visit the Brocade Bookshelf

at www.brocade.com/products/sanadmin_bookshelf.


3/18

OVERVIEW

In May 2005, Brocade introduced the Brocade 48000 Director (see Figure 1), a third-generation

SAN director and the rst in the industry to provide 4 Gbit/sec Fibre Channel capabilities.

Since that time, the Brocade 48000 has become a key component in thousands of data

centers around the world.

Compared to competitive offerings introduced in 2006, the Brocade 48000 is the industrys

fastest and most advanced SAN director, providing numerous advantages:

The platform scales from as few as 16 to as many as 384 4 Gbit/sec ports in a

single domain.

The central memory architecture used in Brocade Application Specic Integrated Circuits

(ASICs) is never subject to Head of Line Blocking (HoLB).

The product design enables simultaneous uncongested operation on all ports as long as

simple best practices are followed.

The platform can provide over 1.5 Tbit/sec (3 Tbit/sec full duplex) of usable switching

capacity in a chassis designed to support even higher port speeds in the future.

In addition to providing the highest levels of performance, the Brocade 48000 features a

modular high-availability architecture that supports ve-nines environments. Moreover, the

platforms industry-leading power and cooling efciency help reduce ownership costs while

maximizing rack density. The Brocade 48000 uses just 2.9 watts per port in its largest

conguration (.75 watts per gigabit).

This is twice as efcient as its predecessor, and up to six times more efcient than

competitive products. This efciency not only reduces data center electric bil lsit reduces

cooling requirements and minimizes or eliminates the need for data center infrastructure

upgrades, such as new PDUs, power circuits, and larger HVAC units. In addition, the highly

integrated architecture uses fewer components per board, which improves key reliability

metrics such as Mean Time Between Failure (MTBF).

Figure 1. The Brocade

48000 Director in a

384-port conguration.

How Is Fibre Chnnel

Bnwith Mesure?

Fibre Channel is a full-duplex network

technology, meaning that transmission can

occur in both directions simultaneously.

However, much like a highway speed limit

sign, the name of the rated standard

(for example, 4 Gbit/sec) refers only to

the bandwidth going in one direction. While

a 4 Gbit/sec link could be considered

8 Gbit/sec full duplex, this is an uncommon

usage and potentially confusing.

When considering the aggregate switching

bandwidth of a SAN director, it is best to

use the same point of reference as the

Fibre Channel specication. Any bandwidth

measurement doubled to reect full-duplex

capabilities should always be explicitly labeledas such. Full-duplex transmission speeds

are included in this paper only to provide a

point of comparison to other vendors that

double aggregate measurements.


4/18

The Brocade 48000 is also highly exible, supporting Fibre Channel, FICON, FCIP with IPSEC,

and iSCSI today, and additional protocols in the future. IT organizations can easily mix various

Fibre Channel blade options to build an architecture that has the optimal price/performance

ratio to meet the requirements of specic SAN environments. As of late 2006, the Brocade

48000 supports the following blades:

Control processor CPU plus 256 Gbit/sec (512 Gbit/sec full duplex) backplane

switching module

16-port 4 Gbit/sec Fibre Channel blade (FC4-16 or 16-port blade)



16-port 4 Gbit/sec Fibre Channel routing blade with two FCIP ports (FR4-18i or router

blade) with FICON support

8-port 4 Gbit/sec Fibre Channel blade with eight iSCSI ports (FC4-16IP or iSCSI blade)

Even though it provides all of these enterprise-class capabilities, the Brocade 48000 has

plug-and-play setup characteristics, and IT organizations can follow a few simple guidelines

to maximize its performance and availability. This paper describes the directors internal

architecture and how to utilize the director and its blades to address particular business

requirements.

BROCadE 48000 PLaTFORM aSIC FEaTURES

There are many different ways to build a director: the possibilities include shared memory

architectures, crossbars, or bus designs. High-speed switches for both Ethernet and Fibre

Channel use shared memory designs for the highest performance, and commodity Ethernet

switches often use crossbars to lower development costs. Whatever the method, large modular

switches need some kind of internal connectivity between discrete components (blades,

modules, or linecards) over a midplane or backplane.

The Brocade 48000 features an internal Channeled Central Memory Architecture (CCMA)

fabric of Fibre Channel ASICs capable of switching at 256 Gbit/sec (512 Gbit/sec full duplex) per

chip. Each Brocade Condor ASIC has thirty-two 4 Gbit/sec ports that can be combined into

virtual interfaces of any size, up to the full capacity of the chip. The Brocade shared memory

architecture leverages the same protocol as the front-end ports, enabling back-end ports to

avoid latency due to protocol conversion overhead.

When a frame enters the ASIC, the destination address is read from the header, which enables

routing decisions to be made even before the whole frame has been received. This allows the

ASICs to perform cut-through routing. In other words, a frame can begin transmission out of

the correct destination port on the ASIC even before the initiating device has nished

transmitting it. Only Brocade offers a SAN architecture that can make these types of switching

decisions at the port level, thereby enabling local switching and the ability to deliver 1.5 TB of

bandwidth in the system. Local latency is 0.8 s and blade-to-blade latency is 2.4 s, the

fastest latency in the industry. As a result, the Brocade 48000 has the lowest delay and

highest performance of any Fibre Channel product in the industry.


5/18

Because port-blade Condor ASICs can act as independent switching engines, the Brocade

48000 can leverage localized switching within a port group in addition to switching over the

backplane. On the 16- and 32-port blades, local switching is performed within 16-port groups

and, on the 48-port blade, local switching is

performed within 24-port groups. Unlike

competitive offerings, frames being switched

within port groups do not need to traverse

the backplane. This enables every port on

high-density blades to communicate at full

4 Gbit/sec speed with port-to-port latency

of just 800 ns, 25 times better than the

next-fastest SAN director on the market.

The Brocade 48000 also has 1024 bufferto-buffer credits within each Condor ASIC to support

longer-distance congurations. Similarly, hardware-enforced zoning resources provide more

exible hardware-enforced zone sets as well as increased security between the connected

devices in a shared network. The Condor ASIC also enhances Brocade Inter-Switch Link (ISL)

Trunking features with 32 Gbit/sec frame-level trunks (up to eight 4 Gbit/sec links in a trunk)

and Dynamic Path Selection (DPS) for exchange-level and device-level balancing between

trunk groups. Up to eight trunks can be balanced for 256 Gbit/sec (512 Gbit/sec full duplex).

(A Fibre Channel exchange is generally equivalent to a SCSI operation.) Furthermore, Brocade

has signicantly improved frame-level trunking: trunks are now masterless. If any trunk

member drops, the trunk will not have to re-build. The trunk bandwidth will drop

proportionally but it will remain active.

BROCadE 48000 PLaTFORM aRCHITECTURE

In the Brocade 48000, each port blade has Condor ASICs that expose a certain number of ports

for connectivity and a certain number of ports to the control processors via the backplane.

The director uses an ASIC layout analogous to a fat-tree core/edge topology. The fat-tree

layout is symmetrical: all ports have equal access to all other ports. The director can switch

frames locally if the destination port is on the same ASIC as the source. This is an important

feature for high-density environments, because it allows over-subscribed blades to achieve

full uncongested line rate performance. No other director offers local switching: trafc must

traverse the crossbar ASIC even if traveling to a neighboring porta trait that ultimately

degrades performance.

The exible Brocade 48000 architecture utilizes a wide variety of blades for increasing port

density or introducing multiprotocol capabilities. IT organizations can easily mix the various

blades in the Brocade 48000 to address unique business requirements and ensure an optimal

price/performance ratio. The following blades are available (with more planned):

16-port Fibre Channel blade



Fibre Channel routing and FCIP blade

Fibre Channel and iSCSI blade

Unlike competitive

offerings, frames being

switched within port

groups do not need to

traverse the backplane.


6/18

16-port Fibre Chnnel Ble

On the 16-port blade, all ports have 64 Gbit/sec (128 Gbit/sec full duplex) of possible

external input, and the same internal bandwidth available. In other words, the blade has a

1:1 subscription ratio. It is useful for extremely high-performance servers, supercomputing

environments, high-performance shared storage subsystems, and SANs with unpredictable

trafc patterns.

The 16-port blade is highly integrated with just one active switching component (the ASIC)

and associated support componentsa design that results in lower power and cooling

requirements as well as a higher MTBF.

Figure 2 shows a functional block diagram and photograph of the 16-port blade, illustrating

the efciency of the design.

Figure 2.

16-port blade design.

32 Gbit/sec pipe

32 Gbit/sec pipe

ASIC

64 Gbit/sec to

Control Processor/Core

16 4 Gbit/sec ports

1:1 Subscription Ratio

at 4 Gbit/sec


7/18


8/18


The 32-port blade is designed with a 16:8 subscription ratio at 4 Gbit/sec for non-local trafc

and a 1:1 ratio at 2 Gbit/sec for any trafc pattern. If some or all of the attached servers and

storage devices run at 2 Gbit/sec, or if I/O proles are bursty, the 32-port blade typically

provides the same performance as the 16-port blade.

Figure 4 shows a functional block diagram and photograph of the 32-port blade.

Figure 5shows how the blade positions in the director are connected to each other using

32-port blades in a 256-port conguration.

Figure 4.


Figure 5.

Overview of a 256-port

conguration.

32 Gbit/sec Pipe

32 Gbit/sec Pipe

ASIC

16 4 Gbit/sec

Local Switching Group

16:8 Over-subscription

16 4 Gbit/sec



64 Gbit/sec to

Control Processor

Power and

Control Path

ASIC

ASIC


9/18

When connecting a large number of devices that need sustained 4 Gbit/sec line rates, IT

organizations can use locality to avoid congestion. The blade is divided into two 16-port

groups for local switching. The physically lower 16 ports (ports 0 to 7 and ports 16 to 23)

form one group and the upper ports (ports 8 to 15 and ports 24 to 31) form the other group.

Figure 6 illustrates the internal connectivity between 32-port blades and the control processors.

There are two ASICs on each port blade, and each ASIC has a group of 16 outward-facing

ports. For each group, there are two internal 8 Gbit/sec connections to each of the two

control processors, for a total of 32 Gbit/sec (64 Gbit/sec full duplex) in backplane switching

capacity. Trafc is balanced across the paths, such that the four 8 Gbit/sec connections form

this virtual 32 Gbit/sec backplane pipe. Any combination of the 16 outward-facing ports in a

group can use up to the full backplane bandwidth without congestion. This workload balancing

and the resulting optimized performance represent the automatic behavior of the architecture

and require no administration.

If more than 32 Gbit/sec of total throughput is needed for each 16-port group, high-priority

connections can be localized within the groupensuring that up to 16 devices or ISLs have

ample bandwidth to connect to devices on other blades. Such connections do not use the

backplane bandwidth. Likewise, localized trafc does not count against the subscription ratio

and cannot be impacted by trafc from other devices. Regardless of the number of devices

communicating over the backplane, locally switched devices are guaranteed 4 Gbit/sec

bandwidth. This Brocade-unique technology for local switching helps preserve bandwidth to

reduce the possibility of congestion in higher-density congurations.

Figure 6.

32-port blade internal

connectivity.16 4 Gbit/sec

Port Blade 1

CP 0

(slot 5)

16 4 Gbit/sec

Each line is a 16 Gbit/sec frame-balanced pipe

(32 Gbit/sec full-duplex)

32 Gbit/sec pipe


32 Gbit/sec pipe


CP 1

(slot 6)

Condor

ASIC

Condor

ASIC

Core Core


10/18


At 24:8, the 48-port blade has a higher backplane over-subscription ratio but also has larger

port groups to take advantage of locality. The backplane connectivity of this blade is identical

to the 32-port blade. The only difference is that, rather than just 16 ports per ASIC, the 48-port

blade exposes 24 outward-facing ports (96 Gbit/sec or 192 Gbit/sec full duplex of local

switching per ASIC).

This blade is especially useful for high-density SAN deployments where:

Large numbers of servers need to be connected to the director

Some or all hosts are running below line rate much of the time

Potential localization of most trafc ows is achievable

Figure 7 shows a functional block diagram and photograph of the 48-port blade.

Figure 7.


32 Gbit/sec Pipe

32 Gbit/sec Pipe

ASIC

ASIC

24 4 Gbit/secLocal Switching Group


24 4 Gbit/sec



Power and

Control Path

64 Gbit/sec to

Control Processor


11/18

Fibre Chnnel Routing n FCIP Ble

The Brocade FR4-18i routing blade consists of sixteen 4 Gbit/sec Fibre Channel ports with

enhanced routing processors and two Gigabit Ethernet ports for FCIP. The FCIP ports support

up to eight virtual tunnels, and up to 32 virtual tunnels are supported in a Brocade 48000

with two blades. In addition, the Brocade FR4-18i routing blade supports line-rate

performance, fast write, compression, encryption, tape pipelining, and FICON. The locality

groups are ports 0 to 7 and ports 8 to 15. Figure 8 shows a functional block diagram and

photograph of this blade.

Figure 8.

Fibre Channel routing and

FCIP blade design.

64 Gbit/sec to

Control Processor

32 Gbit/sec pipe

32 Gbit/sec pipe

Encryption

Compression

Power and

Control Path

8 4 Gbit/sec

local switching group

2 Gbit Ethernet ports

8 4 Gbit/sec

local switching group

Fibre Channel Switching

FCIP and Routing

ASIC

ASIC


12/18

Fibre Chnnel n iSCSI Ble

The Fibre Channel and iSCSI blade consists of eight 4 Gbit/sec Fibre Channel ports and eight

iSCSI-over-Gigabit Ethernet ports. All ports switch locally within the 8-port group. The iSCSI

ports act as a gateway to translate iSCSI hosts onto the Fibre Channel fabric. Because each

port supports up to 64 iSCSI initiators, one blade can support up to 512 servers. Populated

with four blades, a single Brocade 48000 can fan-in 2048 servers. Figure 9 shows a

functional block diagram and photograph of this blade.

64 Gbit/sec to

Control Processor

32 Gbit/sec pipe

32 Gbit/sec pipe

ASIC

Power and

Control Path

8 4 Gbit/sec

Fibre Channel ports

8 Gbit Ethernet ports

Fibre Channel Switching

iSCSI and Ethernet Block

Figure 9.

iSCSI blade design.


13/18

THE BENEFITS OF a CORE/EdGE NETWORK dESIGN

The core/edge network topology has emerged as the design of choice for large-scale, highly

available, high-performance SANs constructed with multiple switches of any size, from

any vendor.

The Brocade 48000 uses an internal architecture analogous to a core/edge fat-tree

topology, which is widely recognized as being the highest-performance arrangement of

switches. The Brocade 48000 is not actuallya fat-tree network of discrete switches, but

thinking of it in this way provides a useful visualization.

While IT organizations could build a network of 32-port switches with similar performancecharacteristics to the Brocade 48000 (128-port version), it would require twelve Brocade

4100 switches arranged in a fat-tree fashion. This network would require more complex

cabling, management of twelve discrete switching elements, non-trivial power and cooling,

three times the number of SFPs to support ISLs, and installation of a non-trivial cable plant.

As a result, this would make the switch-based design difcult to cost-justify. In contrast, the

Brocade 48000 delivers the same high level of performance without the associated

disadvantages, bringing fat-tree performance to IT organizations that could not previously

cost-justify the investment or overhead.

It is important to understand that the internal

ASIC connections within a Brocade 48000 are

not E_Ports connecting a network of switches.

The shared memory architecture enables theentire director to be a single domain and a

single hop in a Fibre Channel network.

Likewise, ASICs within a Brocade 48000 do

not connect via E_Ports. When a port blade is

removed, a fabric reconguration is not sent

across the network, thereby simplifying

operations.

However, unlike an actual fat-tree network, the Brocade 48000:

Is easier to deploy and manage than the analogous network of switches

Simplies the cable plant by eliminating the ISLs and SFP media

Is far more scalable, because it does not consist of a large number ofindependent domains

Is less expensive in terms of both initial and ongoing costs

Has far fewer active components and therefore much higher reliability

Does not run switch-to-switch protocols (E_Port) between blades

Provides multiprotocol support within a single chassis

Is capable of achieving greater performance due to internal routing optimizations

The shared memory

architecture enables the

entire director to be a

single domain and a single

hop in a Fibre Channel

network.


14/18

PERFORMaNCE IMPaCT OF CONTROL PROCESSOR FaILURE MOdES

Any types of failureswhether a control processor or core ASICon the Brocade 48000 are

extremely rare. According to reliability statistics from Brocade OEM Partners, Brocade 48000

control processors have a calculated Mean Time Between Replacement (MTBR) rate of

337,000 hours (more than 38 years) based on real-world eld performance. However, even

in the rare occurrence of a failure, the Brocade 48000 is designed for fast and easy control

processor replacement. This section describes potential failure scenarios and how the

Brocade 48000 is designed to minimize the impact on performance and provide the

highest level of system availability.

The Brocade 48000 has two control processor blades, each of which contains a CPU and a

group of ASICs that provide the core switching capacity between port groups. The control

processor functions are active-passive (hot-standby) redundant while the switching functions

are active-active. The control processor that has the active processor is known as the active

control processor blade, but both active and standby control processors have active core ASIC

elements. In some scenarios, such as failure handling, it is necessary to move routes from

one control processor to another. This section describes those scenarios and their impact on

data trafc and applications.

The ASICs and CPU blocks are separated in both hardware and software except for a common

DC power source. Figure 10 shows a functional block diagram and photograph of the control

processor blade, illustrating the efciency of the design and the separation between the

ASICs and CPU blocks.

Figure 10.

Control processor

blade design.

ASIC

ASIC

Control Path to Blades

Control Processor Power

Switching Power

Modem Management Port

Serial Management Port

Ethernet Management Port

Control Processor Block

Switching Block

256 Gbit/sec

(512 Gbit/sec full duplex)

to Backplane Blades


15/18

Filure in Control Processor Ble

If the processor section of the active control processor blade fails, this affects only the

management plane: the core ASICs are functionally separate and continue switching frames

without interruption. It is possible that a control processor block can fail completely while the

core ASICs continue to operate without degradation, or vice versa.

A control processor failure has no effect on the data plane: the standby control processor

would automatically take over and the switch would continue to operate without dropping any

data frames. Only during the short duration of the service procedure when the control processor

is physically replaced would there be a temporary degradation of available bandwidth.

In most real-world cases, even during this short service procedure, application performance

would not be degraded. For example, this would not affect locally switched ows, and if the

trafc that needs to traverse the control processors is less than the capacity of the maximum

system-wide bandwidth, no congestion would occur. Given the very high MTBF of the blade

and the fact that the outage can and should be scheduled during a time favorable to

operations, this characteristic would not have a noticeable effect in real-world SANs.

aSIC Filure or Ble Removl

If either control processor blade has a core ASIC element failure, or if the blade is removed,

the director no longer has access to one set of core ASICs. All conversations being

performed by these ASICs would be moved to the other blade.

What happens in a particular network depends on many factors. For example, the possibilityof OOD depends on the fabric-wide In-Order Delivery (IOD) ag: if the ag is set, no OOD would

occur. If it is not set, the application impact of OOD would depend on the HBA, target, SCSI

layer, le system, and application characteristics. Generally, this ag is set during installation

by the OEM or reseller responsible for supporting the SAN fabric, and is optimized for the

application environment. Most known currently shipping applications can withstand these

OOD behaviors.


16/18

Core Element Filure

The potential impact of a core element failure to overall system performance is straightforward.

If half of the core elements go ofine due to a hardware failure, half of the aggregate

backplane switching capacity would be ofine until the condition is corrected. A Brocade

48000 with just one core element can still provide 256 Gbit/sec (512 Gbit/sec full duplex)

of backplane switching bandwidth, or 32 Gbit/sec (64 Gbit/sec full duplex) to every director

slot. Note that in best-case scenarios, one competitive SAN director has only 48 Gbit/sec

(96 Gbit/sec full duplex) of bandwidth per slot.

Data ows would not necessarily become congested in the Brocade 48000 with one core

element failure. The worst case is that data ows might become congested, but this requires

that the director already be running at or near 100 percent of capacity on a sustained basis.

On systems with the most typical I/O patterns, the aggregate usage of the available backplane

bandwidth typically would not even be at 50 percent. In such environments there would be no

impact, even if the problem persisted for an extended period of time.

However, very few environments have all ports running at 4 Gbit/secwith a 100 percent load

on all data ows all the timeand use no local switching for any data ows. Even in a case

where the failure of a control processor with an MTBF of 337,000 hours occurs, performance

degradation would last only until repairs are completed. Such repairs or replacement could

be completed in as little as ve minutes.


17/18

SUMMaRY

With an aggregate chassis bandwidth (backplane plus local switching) nearly three times that

of competitive products, the Brocade 48000 is congestion-free in real-world cases that reect

the vast majority of SAN-based applications. Although congestion might occur in unique

situations, it would be infrequent, low-level, and unlikely to impact application performance.

Only in the worst contrived cases would congestion be noticeable at the application level. Even

in those cases, congestion could be eliminated with very little effort by using local switching.

The Brocade 48000 is designed to meet the most demanding performance requirements of

a director-class SAN solution. As demonstrated by Brocade testing, the Brocade 48000:

Delivers 4 Gbit/sec line-rate performance in full-duplex operation

on all ports simultaneously

Does not suffer from Head of Line Blocking (HoLB)

Supports local switching for the highest-performance applications

Is designed for maximum performance with real-world SAN trafc patterns

Supports multiprotocol blades and applications

Is designed to support future speeds and protocols

For more information about the Brocade 48000, visit www.brocade.com.

For more information about SAN design or other Brocade solutions, such as

the Brocade Multiprotocol Router, visit the Brocade Bookshelf at

www.brocade.com/products/sanadmin_bookshelf.


18/18

SAN WHITE PAPER

2007 Brocade Communications Systems, Inc. All Rights Reserved. 01/07 GA-WP-879-01

Brocade, the Brocade B-weave logo, Fabric OS, File Lifecycle Manager, MyView, Secure Fabric OS, SilkWorm, and

StorageX are registered trademarks and the Brocade B-wing symbol and Tapestry are trademarks of Brocade

Communications Systems, Inc., in the United States and/or in other countries. FICON is a registered trademark of IBM

Corporation in the U.S. and other countries. All other brands, products, or service names are or may be trademarks or

service marks of, and are used to identify, products or services of their respective owners.

Notice: This document is for informational purposes only and does not set forth any warranty, expressed or implied,

concerning any equipment, equipment feature, or service offered or to be offered by Brocade. Brocade reserves the

right to make changes to this document at any time, without notice, and assumes no responsibility for its use. This

informational document describes features that may not be currently available. Contact a Brocade sales ofce for

information on feature and product availability. Export of technical data contained in this document may require an

Corporte Hequrters

San Jose, CA USA

T: (408) 333-8000

[email protected]

Europen Hequrters

Geneva, Switzerland

T: +41 22 799 56 40

[email protected]

Asia Pacifc Headquarters

Singapore

T: +65-6538-4700

[email protected]

Achieving Ent SAN Perf 48000 Dir WP 01

Documents