Brocade Evaluator Group Fc vs Fcoe Rt

Test Validation

2014 Evaluator Group, Inc. All rights reserved.

Comparing Enterprise Storage Networking Options FC vs. FCoE Lab Validation

Author: Russ Fellows

February 18, 2014

Enabling you to make the best technology decisions

Test Validation Fibre Channel vs. FCoE Performance

Russ Fellows

p. 1

of 26

2014 Evaluator Group, Inc. All rights reserved. Reproduction of this publication in any form without prior written permission is prohibited.

Table of Contents

Executive Summary ...................................................................................................... 2 Findings Summary ..................................................................................................................... 3

Evaluation Overview ..................................................................................................... 4 Comparing FC to FCoE ............................................................................................................... 4

FC Environment ............................................................................................................................... 6 FCoE Environment ........................................................................................................................... 6 Storage Equipment .......................................................................................................................... 6

Validation Objectives .................................................................................................... 6 Scope ......................................................................................................................................... 7 Configuration Choices ............................................................................................................... 7 Test Approach ........................................................................................................................... 7

Application Software ....................................................................................................................... 8 FC vs. FCoE Configuration ................................................................................................................ 8

Test Results .................................................................................................................. 9 Ease of Use ................................................................................................................................ 9

Initial Setup ...................................................................................................................................... 9 Power and Cooling .................................................................................................................. 10 Cabling and Management ....................................................................................................... 11 Application Workload Test ...................................................................................................... 12 Storage Network Saturation Test ............................................................................................ 13

Saturation Test Results .................................................................................................................. 13 High Availability ........................................................................................................... 17


Russ Fellows

p. 2

of 26


Evaluation Summary .................................................................................................... 18 Performance ............................................................................................................................ 18 Ease of Use .............................................................................................................................. 18 High Availability ....................................................................................................................... 19

Trunking ......................................................................................................................................... 19 Final Observations ................................................................................................................... 19

Appendix A Configuration Overview ......................................................................... 20 FC Environment ............................................................................................................................. 20 FCoE Environment ......................................................................................................................... 20 FCoE Notes: .................................................................................................................................... 20

Appendix B - HP Brocade FC Environment .................................................................... 21 Appendix C - FCoE Environment ................................................................................... 22 Appendix D - Application Workloads ............................................................................ 23 Appendix E Answers to Questions on the Testing ...................................................... 24

Enabling you to make the best technology decisions


Russ Fellows

p. 3

of 26


Executive Summary Fibre Channel technology remains the dominant choice for storage connectivity in a majority of enterprises. This is due to its proven performance and manageability along with reliability and high availability features. Fibre Channel over Ethernet (FCoE) is a recent alternative that enables the Fibre Channel (FC) protocol to run over a new generation of Ethernet, known as DCB Ethernet. As a result, DCB Ethernet can support converged SAN and LAN traffic.

The availability of converged server and storage networking has led some IT organizations to consider FCoE storage connectivity when upgrading their existing IT environment, or for new application deployments. Proponents of FCoE state their solutions are less complex and more cost effective due primarily to a reduction in cabling along with the use of lower cost host adapters. Fibre Channel proponents claim better performance, higher reliability and easier administration at an equivalent cost.

Storage performance is the critical factor for many workloads including OLTP and many virtualized applications. In order to improve these applications performance, it is essential to reduce the latency or delay for individual I/O requests. As such, solid-state storage has become an increasingly popular method of improving application performance, but requires an equally high performance storage network.

Recently, Evaluator Group performed testing to evaluate Fibre Channel connectivity between blade servers and solid-state storage, compared to using FCoE connectivity; focusing on performance, ease of use and availability. The testing was done to provide facts to help IT and storage administrators understand the impact of storage connectivity on applications as they adopt solid-state storage, particularly in virtual server environments.

Evaluator Group comments: Evaluator Group believes that Fibre Channel connectivity is required in order to achieve the full benefits that solid-state storage is able to provide.

Findings Summary The tested configuration showed the following results:

FC provided lower response times as workloads surpassed 80% SAN utilization o FC response times were one-half to one-tenth of FCoE response times (2X to 10X faster)

FC provided higher performance with fewer connections than FCoE o Measured FC response times were lower, using 50% fewer connections than FCoE o Lower variation in FC results provided more predictable response times

FC used 20% to 30% less CPU than FCoE o CPU utilization was lower using FC than FCoE as shown on page 16


Russ Fellows

p. 4

of 26


Evaluation Overview Brocade commissioned Evaluator Group to compare two competing enterprise storage networking options, assessing the ease of use, performance and high availability features of these two alternative technologies. All testing occurred at Evaluator Group labs using a combination of Evaluator Group and Brocade equipment to perform the testing.

Test results indicate that as network utilization rates increase, Fibre Channel delivers consistent performance, while providing better performance during high bandwidth utilization. This report details the testing process, equipment and other findings.

The measured results were significant, despite the theoretical equivalency of the two configurations.

Evaluator Group comments: Evaluator Group found that a single 16 Gbps FC connection outperformed two, 10 Gbps FCoE connections as measured by application latency. Additionally, the Fibre Channel environment had superior high availability, reduced cabling and better management. Overall, 16 Gbps FC provides significant availability, performance and ease of use advantages compared to FCoE technology.

Testing occurred in November 2013, focusing on performance, ease of use and enterprise high availability features. The workloads utilized were designed to recreate actual enterprise applications rather than using synthetic workload generation tools in order to create an accurate assessment of the configurations used in enterprise environments. Evaluation criteria included:

Performance: o Performance of solutions as application workloads scale o Relative performance of the two solutions

Ease of Use o Configuration o Cabling and other required tasks

High Availability (HA) o Integrated support for MS Windows applications o MS Windows MMC interface

Comparing FC to FCoE Fibre Channel over Ethernet (FCoE) has moved past the evaluation stage into production in some instances, particularly between servers and top of rack switches. However, questions still remain regarding how the alternative storage networking options perform under load using the latest generation of blade servers and solid-state storage systems. Additional considerations include relative ease of use, configuration and high availability features.


Russ Fellows

p. 5

of 26


This evaluation highlights the differences found when running identical workloads on two configurations, shown in Figure 1, using the following products:

1. HP c7000 BladeSystem with a Brocade embedded 16 Gbps SAN switch 2. Cisco 5108 UCS blade system with 2208 Fabric Extender and a 6248 Fabric Interconnect 3. Solid-state storage system with 16 Gbps FC connectivity 4. Brocade 6510, Gen 5, 16 Gbps FC switch

Test configuration shown below in Figure 1 is the logical design of the test configuration:

Figure 1: Test Configuration

vs.

FCoE(2 x 10 GbE)

FCoE

Cisco UCS 6248 UPFabric InterconnectFC (2 x 8 Gb)

Gen 5 FC(2 x 16 Gb)

Gen 5 FC6510

Gen 5 FC6510

16 GbSolid-stateArray

Gen 5 FC(2 x 16 Gb)

16 GbSolid-stateArray

FC

Cisco UCS 5108 Chassis 2208 XP FEX Virtual HBA (VIC) 2 X B200 M3 Blades

HP BladeSystem c7000 Chassis Embedded 16 Gb FC Switch Qlogic 2600 HBA 2x BL460c BladesGen 5 FC

(1 x 16 Gb)


Russ Fellows

p. 6

of 26


FC Environment HP BladeSystemwith Brocade FC SAN Switch

HP c7000 blade enclosure: o 2 x BL460c Gen8 blade servers, each with dual, quad core CPU and 128 GB RAM o 1 x embedded Ethernet switch for LAN connectivity (4 x 1 GbE + 4 x 10 GbE) o 1 x HP 16 Gbps FC embedded switch for SAN connectivity (28 x 16 Gb)

HP BladeSystem, OA (Onboard Administrator) for single point management of blades and I/O connectivity

FCoE Environment Cisco Unified Compute Server with FCoE

Cisco 5108 blade chassis: o 2 x B200 M3 blade servers, each with dual, quad core CPU and 128 GB RAM o 2 x 2208 XP Fabric Extenders (with 1240 VIC - virtual interface connection to blades) o 1 x (non HA) 6248 UP Fabric Interconnect for unified SAN / LAN connectivity and FCoE to

FC bridge Cisco UCS Manager for unified management of server, unified fabric extenders along with LAN

and FCoE SAN host ports

Storage Equipment Common FC SAN Equipment

1 x External, 16 Gb FC attached solid-state storage 1 x (non HA) Brocade 6510 FC SAN switch for storage connectivity

o Gen 5 FC, supporting 16 Gb FC with additional ports on demand o Management options using web based tools, or advanced management with Brocade

Fabric Vision - Setup and deployment wizards with Dynamic Fabric Provisioning tool o High availability uses redundant, hot-pluggable components and non-disruptive upgrades

Validation Objectives Compare and contrast Fibre Channel (FC SAN) with FCoE solutions

o Configurations were equivalent when possible o Best practice configurations utilized o Focus on real-world scenarios and common deployments

Show performance of both alternatives Show management and ease of use for both environments Show power, space and cooling requirements for each solution


Russ Fellows

p. 7

of 26


The testing was both quantitative and qualitative in nature. The quantitative testing measured specific values for performance and time to accomplish major tasks. Qualitative testing was performed where value and judgment were considerations, such as comparing relative usability, ease of use and other comparative information.

This report highlights similarities and differences in test results. Evaluator Group commentary provides context and a narrative assessment of the results as experienced by Evaluator Group personnel.

Scope Performance

o Application latency o Application throughput o Server CPU utilization o Maximum number of VMs supportable

Complexity of solution, measured by: o Amount of equipment o Amount of time to configure o Number and general complexity of steps

Operational aspects of solution measured by: o Estimated energy consumption of solutions o Estimated RU requirements of solutions

Configuration Choices 1. The configuration for both FC and FCoE was non-redundant. Typical production environments

would use redundant components. This was chosen to minimize equipment requirements. 2. The test configuration utilized FC attached storage, rather than FCoE. This is due to the fact that

FC attached storage is the predominant connection technology today. Also, there are very few production deployments of FCoE storage, and end-to-end FCoE is immature.

Test Approach The tests were designed to assess two common enterprise configurations using equivalent blade systems, storage networking and high performance solid-state storage. The two configurations were chosen to be as close to identical as possible with the primary difference being the protocol used for storage networking.


Russ Fellows

p. 8

of 26


A demanding application workload set was chosen that would fully utilize the server and storage components of the solutions. The IOmark suite1 of workload tools was used to generate application storage workloads that recreate actual storage workloads without the CPU or memory component. These workloads were run on blade systems from two leading enterprise vendors using either FCoE or FC-only storage networking with applications using 16 Gbps FC flash storage.

Evaluator Group comments: Both test environments utilized the same solid-state, FC storage system. The testing utilized enterprise FC attached storage, since a majority of enterprises choose FC storage as their preferred storage device. Fibre Channel attached storage continues to be the most predominate connectivity method.

As a result, it was possible to isolate the storage networking protocols including targets and initiators as the primary factor for any differences in performance or other test results. Performance differences were a primary area of interest along with operational aspects of the SAN alternatives. Operational comparisons also included management efficiency, power and cooling.

Application Software

The performance limits were found running multiple applications instances on a storage infrastructure with FC and FCoE equipment. By adding successively more application workloads and instances, the maximum number of applications supported was determined for both infrastructures.

Testing was performed by generating workloads that recreated application loads identical to the original application. Three sets of application workload types were generated including read-only video rendering, VDI desktop applications and two industry standard databases, DVD store database and Exchange mail server. The applications included:

1. NL Video render (3 instances, uncompressed 1080p stream at 250 MB/s each) 2. VDI Desktop applications (560 instances each with MS Office, Acrobat, IE and other apps) 3. IOmark VM Top 2 applications (126 instance each of DVD data store and Exchange)

FC vs. FCoE Configuration

The configuration details for both environments are provided in Appendix A. The primary difference between the configurations was storage network connectivity.

1 Description of the IOmark suite of storage application benchmarks can be found at - www.iomark.org


Russ Fellows

p. 9

of 26


The FC environment utilized 16 Gbps, Gen 5 equipment including HBA, switches and FC attached solid-state storage

In contrast, the FCoE environment utilized an FCoE initiator, an FCoE to FC bridge along with Gen 5, 16 Gbps FC switch connected to the same FC attached solid-state storage

Test Results The test findings are derived from the testing outlined, with results detailed in the remainder of this report.

Ease of Use The comparison for ease of use was measured using several different methods, including:

Initial configuration of the blade environment Cabling and other connectivity considerations Ease of making storage connectivity changes

Initial Setup

Initial configuration is a contributing factor for many IT personnel. Although these operations are typically performed only during initial deployment, the experience does provide one measure of ease of use.

An initial installation was performed covering the following steps:

Install equipment into rack and connect power Connect LAN for initial configuration Perform initial power-on, and initialize system Configure system for initial use Connect SAN to system and storage Install VMware vSphere hypervisor onto physical server Install driver for HBAs where required (Qlogic FC QLA 2600 required driver install) Scan network adapters to recognize pre-configured storage LUNs

Evaluator Group comments: The UCS environment was significantly different from other servers and blades. The Cisco UCS Manager is a network centric concept with Cisco blade servers managed as one component of the UCS. This approach was less intuitive during installation than other enterprise systems Evaluator Group has tested.


Russ Fellows

p. 10

of 26


FC Environment The setup of the FC environment required approximately 3 hours of time. All setup occurred without requiring any special assistance. The one issue encountered was the need to change the management IP address for the embedded switch, which was quickly resolved.

FCoE Environment The setup of the FCoE environment required approximately 8 hours of time. Several issues were encountered while configuring the UCS equipment. Additional assistance was requested from a VAR with certified UCS engineers. After several attempts and configuration changes using the UCS setup wizard options, the setup was completed.

Power and Cooling

FC Environment The FC environment had the following power and cooling characteristics:2

HP c7000, w/ 2, BL460c blades (including embedded Ethernet and FC Switches) o 1031 Watts, 3518 BTU / hr. (measured)

Brocade 6510 FC Switch o 110 Watts, 375 BTU / hr. (published)

Total of 1141 Watts, 3893 BTU / hr.

FCoE Environment The FCoE environment had the following power and cooling characteristics:3

5108 Chassis w/ 2, B200 M3 blades (including Cisco 2208 FEX) (@ 50% load) o 874 Watts, 2979BTU /hr. (Cisco power calculator - see footnote below)

6248 UP Fabric Interconnect o 750 Watts, 2561 BTU / hr. (published)

Brocade 6510 FC Switch o 110 Watts, 375 BTU / hr. (published)

Total of 1734 Watts, 5937 BTU / hr.

2 HP and Brocade equipment based on measured and published specifications 3 Cisco UCS power based on Cisco published specs, and UCS power calculator v. 3.02


Russ Fellows

p. 11

of 26


Evaluator Group comments: The Cisco UCS equipment required 50% more power and cooling than did the HP blade with FC equipment. This is an advantage for the FC environment, as power and cooling are significant considerations. Additionally, typical Cisco FCoE environments using Cisco MDS switches would require additional power and cooling, further increasing the FC power advantage.

Cabling and Management Cabling and connectivity is another consideration, particularly for environments considering FCoE. Reduced cabling is often cited as an advantage of FCoE over FC.

The FCoE environment required more cables than did the FC environment when including both the LAN and SAN connections. Additional bandwidth and HA connectivity was possible, but not utilized. Both the FC and FCoE environments could be configured with dual, redundant storage and server connectivity. The reduced cabling advantage for FC would remain when moving from a single network to redundant networks.

Evaluator Group comments: When including both LAN and SAN connectivity, the FC environment required fewer connections for the same bandwidth. In comparing only the storage connectivity, the FC configuration required 50% fewer cables than did FCoE. This highlights and confirms the inaccuracy of the FCoE claim of fewer cables and connections.

FC Environment The FC environment had 4 total connections as follows:

1 x 10 Gbps Ethernet LAN connection to management network 1 x 16 Gbps FC SAN connection from HP c7000 blade to 6510 switch 2 x 16 Gbps FC SAN connections from Brocade 6510 FC switch to solid-state storage

FCoE Environment The FCoE environment had 9 total connections as follows:

4 x 10 Gbps converged Ethernet connections from UCS 2208 FEX to 6248UP interconnect 1 x 10 Gbps Ethernet connection from 6248 to management network 2 x 8 Gbps connections from 6248UP to Brocade 6510 FC switch 2 x 16 Gbps FC SAN connections from Brocade 6510 FC switch to solid-state storage

Note: For this test, a single 16 Gbps FC connection from the BladeSystem to the Brocade FC switch was used, compared to 2 x 8 Gbps FC connections from the 6248 interconnect to the FC switch. This was done to provide equivalent nominal performance and was not representative of a production configuration. This setup could be thought of as one half of an HA configuration.


Russ Fellows

p. 12

of 26


As shown, the FC environment delivered higher performance using fewer cables with reduced management complexity. Additionally, the FC environment used less power, cooling and space.

Evaluator Group Comments: It is important to note that the FC environment did not use an HA setup in order to provide the FCoE environment with equivalent nominal performance. Also, a typical Cisco UCS FCoE environment would utilize a Cisco MDS FC switch rather than a Brocade FC switch. Due to the higher latency of MDS switches, actual FCoE latencies would be higher, or worse than the results found here. 4

Application Workload Test As detailed previously, hundreds of applications workloads were configured to run simultaneously against multiple solid-state FC attached LUNs. Each workload was run for a 1 hour time period with successive instances of the same workload started after a delay of up to 15 minutes. The measurement period for the complete set of workloads was a 30-minute interval during which all workloads were running.

Specifically the following applications were tested:

Workload Total Instances

Non Linear Video Editing (at 250 MB/s) 3

IOmark-VM : DVD Store DB 126

IOmark-VM : Exchange Mail 126

IOmark-VDI : Heavy 560

Table 1: Workload Specifications

These applications are both representative of common applications within enterprise environments, but also represent a workload that requires more than 80% of a 16 Gbps FC link for both read and write operations. The tested workloads vary over time, similar to how enterprise workloads behave.

4 Note: Previously published data for MDS switches showed latencies that are higher than published Brocade 16 Gb FC latencies. Cisco has subsequently stopped publishing latencies for MDS products.


Russ Fellows

p. 13

of 26


Storage Network Saturation Test Comparing a single 16 Gbps FC connection to two 10 Gbps FCoE links was one of the performance tests. In this test, the FCoE environment had twice as many connections, and had a theoretical performance level higher than the 16 Gbps FC environment.

In addition to running application workloads, a network saturation test was run using various block sizes for both read and write operations. The maximum data rate of 1600 MBps for a single 16 Gbps FC link was used as the threshold of 100% maximum network workload. Workloads consisting of 10% (160 MB/s) to 90% of maximum (1440 MB/s) were generated with response times shown for each workload.

Saturation Test Results

The results for a 50% read, 50% write workload at 32 KB are shown below in Figure 3 for select workload levels. As depicted in Figure 3 below, the response times for FCoE began to increase significantly above those for FC beyond 80% of maximum saturation levels.

Evaluator Group comments: Comparing a single 16 Gbps FC connection to two, 10 Gbps FCoE connections, we found the Fibre Channel environment performed increasingly better as workloads approached the maximum data rates for the SAN link. Specifically, at 83% SAN saturation rates, the FC environment was 2X faster as measured by application response times. As workloads increased above 90%, latencies were measured to be 10X faster (or 1/10th of the latency) for the FC environment compared to FCoE.

Figure 3: Response time of FC vs. FCoE (Note: lower is better)

0.00#2.00#4.00#6.00#8.00#

10.00#12.00#14.00#

160$ 800$ 960$ 1120$ 1280$ 1440$

ms$$$Respo

nse$Time$

MB/s$@$32$KB$Blocksize$

FC$vs.$FCoE$A$Response$Times$

FC#

FCoE#


Russ Fellows

p. 14

of 26


Common Link Saturation Examples Link saturation can occur in real-world usage. This would typically occur when multiple applications experience I/O peaks at the same time. While the probability of such an event may seem low, it is in fact quite common. This is because many applications are inter-related. That is, many applications impact other applications, and a spike in one can drive spikes in other applications. For example, a web application server can drive usage in a business application server, which in turn can drive workloads on a back-end database server.

Additionally, applications exhibit peaks, or spikes in activity and do not generate constant I/O patterns. This can be clearly shown in the testing by observing measured I/O, as shown in Appendix D on page 22.

Response Time As shown above, the average response times for transactions were nearly identical for FC and FCoE up until network load reached 70% of maximum. Above this level, the FCoE response times began increasing faster than the FC environments response times. Above 80% network load, the rates diverged even further with FCoE experiencing over 8X higher latencies.

Predictability Variance and standard deviation are statistical methods for measuring predictability. The greater the variance, the greater the range is for the results. The variance increased as the response times increased. Overall the FC environment provided more consistent I/O response times leading to more predictable application performance.


Russ Fellows

p. 15

of 26


As shown below in Figure 4, the variation in response times - as measured by standard deviation - is significantly greater for the FCoE environment.

Figure 4: Response Variance FC vs. FCoE (Note: lower is better)

An analysis of the standard deviation of response time results also showed differences. The FCoE environment had more variation than did the FC environment at all network loads. The variance continued to increase as the workloads utilization increased. At 50% load, the difference was approximately 2X greater for FCoE, and at 90% load the difference in variance grew to over 10X as shown above in Figure 4.

Evaluator Group comments: A higher variation in response time for FCoE leads to decreased predictability of response times. The FC environment provided more deterministic results, which in turn provides more predicable quality of service. These factors all help IT users deliver higher service levels, or SLAs by using FC connectivity.

1.51 13.21

1.87 10.32

FC

FCoE

Average and Std. Dev in milli seconds (ms)

FC vs. FCoE - Response Variance Avg. StdDev Avg. Resp.


Russ Fellows

p. 16

of 26


CPU Utilization

CPU use was also measured during testing. The FCoE environment showed higher CPU usage, particularly as the application workloads increased. Higher CPU utilization for the same I/O rate shows that it would take more CPU cycles to run the equivalent workload using FCoE rather than FC. The CPU utilization is shown below in Figure 5.

Figure 5: CPU Utilization, FC vs. FCoE (Note: lower is better)

0.00 10.00 20.00 30.00 40.00 50.00 60.00

10% 50% 60% 70% 80% 90% % CPU

Utilization

Percent of 1600 MB/s Bandwidth @ 32KB Blocksize

FC vs. FCoE - CPU Utilization

FC FCoE


Russ Fellows

p. 17

of 26


High Availability Reliability, availability and serviceability are prime considerations in nearly all enterprise environments. Each of these aspects has different considerations and impacts organizations differently. The reliability of individual network links in an FC and enterprise Ethernet (aka DCB Ethernet) are similar. However, in order to maintain DCB Ethernet availability, storage traffic must be maintained separately from LAN traffic. By separating traffic and connections, the benefits of using converged Ethernet are lost.

One method for achieving high availability (HA) is through the use of redundant networks or fabric connectivity. Additional HA may be achieved through the use of trunking or port bonding. These are generic terms that may be applied to different networking technologies, including Ethernet and Fibre Channel. Ethernet operates at Layer 2 of the OSI stack, while Fibre Channel operates across several layers, although primarily Layer 2.

Ethernet port bonding, also known as link aggregation control protocol (LACP), may be established on switches and network interface equipment settings. However, LACP is not automated and requires specific ports and settings. With Cisco UCS, FC trunks known as Fibre Channel Port Channels may be created. However, as with Ethernet, these require specific settings on each switch within the switches and fabric with ports specifically denoted as belonging to a group.

Fibre Channel refers to the aggregation of multiple ports as trunking when applied to links between switches. This term refers specifically to the ability for multiple links to be automatically used to both increase bandwidth, and provide redundant connections, thus increasing the availability of the connections.

Evaluator Group comments: The FC environment was able to automatically create a trunk group between the blade enclosure and the Brocade FC switch, while the FCoE environment did not. In this respect, the tested Fibre Channel configuration had configuration advantages that provided HA and performance benefits compared to FCoE. These automated capabilities are increasingly important at scale.

Specifically, trunking is enabled across all Brocade switches by default, enabling trunk groups to be formed dynamically, with additional ports added to the trunk group automatically. In comparison, Cisco switches require configuration of port channels individually and those assignments remain static.

The ability to automatically detect additional connections and dynamically create groups of ports that added bandwidth, load balancing and high availability was something only observed with the Fibre Channel configuration tested.


Russ Fellows

p. 18

of 26


Evaluation Summary This evaluation provided an opportunity to test two competing configurations head to head, comparing a solution using FCoE connectivity compared to using FC technology. These results did not uncover a problem or failure of FCoE, rather it shows that for heavily utilized storage environments with flash-based, solid-state storage; the FC environment simply functioned better.

The tested configuration, using 16 Gbps FC vs. 10 Gbps FCoE showed the following results:

FC provided 50% faster response times as workloads surpassed 80% of SAN utilization FC provided higher performance with 50% fewer connections than FCoE FC provided higher reliability, due to automatic trunking when using 2 or more connections FC used between 20% - 30% less CPU than FCoE, FC used 50% less power and cooling than FCoE

As a result of these findings, Evaluator Group believes that Fibre Channel connectivity is the best option for enterprises using high performance, solid-state storage systems.

Performance Evaluator Group tested two leading enterprise configurations from multiple perspectives. The comparison was made using application based tests, with both configured as nearly identical as possible with the primary difference being the storage networking connectivity. The ease of use along with the high availability and increased performance of the Fibre Channel configuration was evident when compared to the FCoE environment.

Ease of Use There are multiple aspects to ease of use including:

Cabling (in general fewer connections are simpler to configure and manage) Little or no protocol bridging (bridging FCoE to FC adds complexity) Easier configuration of the environment

The FCoE environment required more cables than did the FC environment including both the LAN and SAN connections. By eliminating an extra connection from the UCS 6248 Fabric Extenders to an FC switch, one connection was eliminated. The increased bandwidth of the 16 Gbps FC environment also consolidated multiple 8 Gbps FC connections from the 6248 to an FC switch, further reducing cabling and configuration. With large configurations, the degree of savings become significant due to reduced cabling of 16 Gb FC compared to 10 Gb FCoE.


Russ Fellows

p. 19

of 26


High Availability Trunking

The FC environment was able to automatically create a trunk group between FC switches, while the FCoE environment did not. In this respect, the tested Fibre Channel environment had significant ease of use and configuration advantages for HA and performance compared to the FCoE environment. As configurations grow in size, Fibre Channels automated capabilities are increasingly advantageous.

With trunking enabled across Brocade switches by default, trunk groups are formed dynamically. This provides automatic improvement in HA and performance by dynamically utilizing all connections. In comparison, the Cisco FCoE environment requires static configuration of port channels for trunking.

Moreover, we found FC to provide significant advantages by reducing complexity while improving link reliability and bandwidth.

Final Observations Fibre Channel technology remains the dominant choice for storage connectivity in the majority of large enterprises. This is due to its proven performance, manageability and high availability features. FCoE is a recent alternative that enables the FC protocol to run over Ethernet connectivity. As a result, a single enterprise Ethernet connection is able to support converged SAN and LAN traffic.

The promise of a converged server and storage networking has resulted in some IT organizations considering the use of FCoE between their servers and Top-of-Rack switches. This evaluation commonly occurs when upgrading existing IT environments, or when deploying new applications.

The testing performed shows that as network utilization rates increase, Fibre Channel delivers consistent performance, requiring less server CPU cycles, while providing better performance during high bandwidth utilization.

As a result, Evaluator Group believes that Fibre Channel connectivity is required in order to achieve the full benefits that solid-state storage is able to provide.


Russ Fellows

p. 20

of 26


Appendix A Configuration Overview FC Environment

The Fibre Channel environment consisted of the following:

Server o HP c7000 BladeSystem o 2 x BL460c blades, w/ 2x E2600 CPUs and 128 GBPS RAM o QLogic embedded mezzanine 2600 FC HBA o 1 x Brocade embedded Gen 5, 16 Gbps FC switch

SAN o 1 x Brocade, 6510, Gen 5 - 16 Gbps FC Switch

Storage o 2 x Single controller solid-state storage, each with 1 @ 16 Gbps FC connection

FCoE Environment

The configured FCoE environment was chosen to be nearly identical, substituting Cisco UCS blade system for the HP blade system, and using the Cisco 6248 fabric interconnect to perform FCoE to FC bridging. The other change was using the included Cisco VIC FCoE device in place of a Qlogic FC HBA.

Server o Cisco UCS 5108 Blade System o 2 @ B200 M3 blades, w/ 2x E2600 CPUs and 128 GBPS RAM o 2 @ Cisco 2208 Fabric Extender (FEX) with VIC (virtual interconnect) o 1 @ Cisco 6248UP Fabric Interconnect (FCoE to FC bridging) 750W,

SAN o 1 - Brocade, 6510 Gen 5 - 16 Gbps FC Switch (110W, 375 BTU/hr.)

Storage o 2 x Single controller solid-state storage, each with 1 @ 16 Gbps FC connection

FCoE Notes:

Evaluator Group Comments: An additional consideration is that a typical FCoE environment with UCS would utilize a Cisco MDS FC switch, rather than a Brocade FC switch. Brocade publishes values of 1 microsecond for 6510 latencies, while the Cisco MDS has previously published values of 5 microseconds. Thus, actual latencies would likely be higher for an all Cisco deployment using FCoE between UCS and 6248, bridged to a FC connected Cisco MDS FC switch, than the tested environment.


Russ Fellows

p. 21

of 26


Appendix B - HP Brocade FC Environment The chosen FC environment was constructed using the following components:

HP c7000 blade enclosure HP BL460c blades (2 each)

o 2 @ E5 2609 CPUs / blade o 128 GB RAM / blade

HP 16 Gbps FC embedded switch QLogic 2600 embedded FC mezzanine

Show below is a screenshot of the HP BladeSystem Onboard Manager.

Figure 6: HP Onboard Administrator Manager GUI (Source: HP)


Russ Fellows

p. 22

of 26


Appendix C - FCoE Environment The chosen FCoE environment was constructed using the following components:

Cisco UCS 5108 blade enclosure Cisco B200 M3 blades (2 each)

o 2 @ E5 2609 CPUs / blade o 128 Gbps RAM / blade

Cisco 2208 - Fabric extender (FEX) Cisco 6248UP Fabric interconnect

Show below is a screenshot example of the Cisco UCS manager.

Figure 7: Typical Cisco UCS Manager GUI (Source: Cisco website)


Russ Fellows

p. 23

of 26


Appendix D - Application Workloads The application workloads were generated using I/O traces of actual application workloads.

Shown below is a trace for each datastore on a single ESXi host (blade server). Each datastore represents a single LUN mapped from a solid-state storage system used as a test target device. As seen below, each workload varied over time, and was not a static workload as is typically found with synthetic workloads.

One important consideration for real world environments is the dynamic nature of workloads. As seen below, each workload varies over time. Additionally, some workloads are positively correlated with other workloads. That is, an increased I/O rate for one application may in turn generate an increased I/O rate for other applications.

The workload shown below is identical for both the FC and FCoE environment, since this graph depicts each workloads I/O rate, which is the same regardless of supporting storage networking technology.

Evaluator Group Comments: All solid-state storage was utilized for both tests, enabling higher transactional throughput that conventional storage. Each of the I/O traces shown below represents one or more applications. The total I/O required is the sum of each of the individual traces. As seen below, peaks can occur, creating potential bottlenecks for environments without sufficient bandwidth. For the FCoE environment, bottlenecks were experienced during the periods of high total I/O activity.

Figure 8: Trace of a physical ESXi host running workloads


Russ Fellows

p. 24

of 26


Appendix E Answers to Questions on the Testing Background

The following was published after the first release of the test.

As an independent analyst firm, Evaluator Group was engaged by Brocade to conduct a comparison of two environments, a Cisco UCS with FCoE server connections and an HP BladeSystem with Fibre Channel server connections, both connected to Fibre Channel storage. We did so with the caveat that it was an independent review. We constructed the testing based upon discussions with our IT clients (typically Fortune 1000 firms). The goal was to compare configurations that customers commonly evaluate from two leading blade server vendors to help them make more informed choices.

The objective of the test was to compare the impact of deploying the latest generation Flash storage on high performance enterprise applications in virtual server environments. A 16 Gbs Fibre Channel flash storage array was selected as the storage target, and HP c7000 and Cisco UCS were selected for the blade servers given their market share leadership. Based on the most common deployment models with shared storage for each solution, UCS servers used FCoE to connect to a Fibre Channel SAN while end-to-end Fibre Channel connectivity was used on the HP c7000, with both connecting to the same solid state Fibre Channel storage array. The objective was to understand how the two different architectures would impact the application benchmarks using the same storage.

To conduct the test, we performed all of the testing in our lab, independent of vendors. Support was provided via the vendors support lines, avoiding special assistance or inside information. While our hypothesis was that end-to-end Fibre Channel would outperform FCoE to Fibre Channel, Brocade understood that outcomes are not known ahead of time. In some cases, results do not show a vendor or products favorable. That risk is borne by the vendor funding the work.

In order to ensure that the testing was fair and representative, we used application workloads representative of common and comparable customer environments. We chose to use a Fibre Channel SAN storage solution, given that Fibre Channel is the leading interconnect for shared storage among enterprise customers. The configurations were carefully chosen to be as fair as possible; ensuring storage network limitations were equal for both configurations.

Q&A

1) What type of FCoE initiators were used for Cisco UCS?

Cisco UCS VIC 1240 CNAs configured with virtual HBAs were used for testing.

2) This test does not seem to be the best way to compare only an FCoE connection to a Fibre Channel connection.


Russ Fellows

p. 25

of 26


The goal of the test was to compare two typical configurations. Customers are deploying high speed solid-state systems in virtual environments. Currently, a majority of our enterprise customers prefer to use a Fibre Channel SAN for their high performance applications. When comparing alternatives, customers often consider a blade server like the HP BladeSystem, with Fibre Channel interfaces. Alternatively, they may also consider Cisco UCS blade systems with FCoE interfaces. In both cases, it is common to connect the blade servers to a SAN (in this case a Fibre Channel switch), as opposed to directly connecting to FCoE storage.

3) What version of UCS software was used?

We utilized UCS GUI manager version 2.0(4b). The UCS chassis and blades utilized version 2.0 software as well. Although a more recent version was available, there were no known capabilities in the latest version that would affect the results and our conclusions.

4) Why was a screenshot of a simulator used in the paper?

An example screenshot published in the report was chosen since the intention was to show a typical representative GUI screenshot of both environments. All configuration information actually utilized in the analysis is provided in the report. Versions of UCS software used in this analysis are noted above.

5) Evaluator Group seems to penalize the UCS for being more difficult to setup, but maybe they just understand the HP system better.

Our objective was to report on our experience configuring both environments. While Evaluator Group has many years of experience setting up enterprise servers; we did not have prior specific experience setting up either a HP c7000 BladeServer, or a Cisco UCS. As explained, all testing and setup was performed within our lab environment, without direct assistance from any of the vendors. When issues arose, we utilized typical support mechanisms, calling into the toll-free support lines of the vendor, without asking for special assistance. Our opinion is based on the difficulty and amount of time needed to configure each system. Our experience was that it took us longer to configure the Cisco UCS than the HP system, and we reported that observation.

6) Why not use multiple FCoE connections between the UCS and the 6248UP?

Each UCS blade had one dedicated 10 Gbs FCoE link to the 6248UP. This made for a total of two 8 Gb Fibre Channel links from the UCS into the Fibre Channel SAN. The reason that more links were not utilized was to maintain fairness in the comparison. The alternative HP blade configuration used one 16 Gb link that was shared between the two HP blades. Thus, both configurations had equal amounts of bandwidth at the point of contention.


Russ Fellows

p. 26

of 26


About Evaluator Group Evaluator Group Inc. is dedicated to helping IT professionals and vendors create and implement strategies that make the most of the value of their storage and digital information. Evaluator Group services deliver in-depth, unbiased analysis on storage architectures, infrastructures and management for IT professionals. Since 1997 Evaluator Group has provided services for thousands of end users and vendor professionals through product and market evaluations, competitive analysis and education. www.evaluatorgroup.com Follow us on Twitter @evaluator_group

Copyright 2014 Evaluator Group, Inc. All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopying and recording, or stored in a database or retrieval system for any purpose without the express written consent of Evaluator Group Inc. The information contained in this document is subject to change without notice. Evaluator Group assumes no responsibility for errors or omissions. Evaluator Group makes no expressed or implied warranties in this document relating to the use or operation of the products described herein. In no event shall Evaluator Group be liable for any indirect, special, inconsequential or incidental damages arising out of or associated with any aspect of this publication, even if advised of the possibility of such damages. The Evaluator Series is a trademark of Evaluator Group, Inc. All other trademarks are the property of their respective companies.

Table of ContentsExecutive SummaryEvaluation OverviewValidation ObjectivesTest ResultsHigh AvailabilityEvaluation SummaryAppendix A Configuration OverviewAppendix B - HP Brocade FC EnvironmentAppendix C - FCoE EnvironmentAppendix D - Application WorkloadsAppendix E Answers to Questions on the Testing

Brocade Evaluator Group Fc vs Fcoe Rt

Documents