At the Intel Xeon E5-2600 v3 inflection point, this technology brief looks at how the latest Cisco UCS and HP BladeSystem blade servers match-up to workloads of the future.
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Document # TECHBRIEF2013005 v19 November, 2014 Page 10 of 24
Oversubscription
Almost no Oversubscription with HP BladeSystem
Oversubscription occurs when the I/O capacity of the adapter ports connected to chassis switch ports
exceeds the capacity of the switch ports. The oversubscription ratio is the sum of the capacity of the adapter
ports, divided by the capacity of the chassis interconnect ports. Below you can see that if you actually
configured 80Gb of bandwidth per UCS blade as mentioned above, you would be building a blade server
network with 4:1 oversubscription. In contrast, a comparable configured HP BladeSystem would result in
1.1:1 oversubscription — almost a 100% improvement in oversubscription when compared to Cisco.
8 ports x 10Gb from
Mid-Plane x 2 IO
Modules = 160 Gb
8 ports x 10Gb x 2 IO
Modules = 160Gb
4 ports x 10Gb
from VICs and 4
ports x 10Gb from
expansion cards
(80Gb) x 8 Servers
= 640Gb
16 ports x 20Gb from
Mid-plane to 4 x HP
Virtual Connect
Modules = 1,280Gb
4 HP Virtual Connect
Modules. Each with 4
x 40Gb ports + 8 x
10Gb ports + 2 x20Gb
ISL ports = 1,120Gb
2 ports x 20Gb
from FLOM + 2
ports x 20Gb for
Mezz. Card x 16
Servers = 1,280Gb
HP BladeSystem: Oversubscription = 1.1:1
Cisco UCS: Oversubscription = 4:1
Document # TECHBRIEF2013005 v19 November, 2014 Page 11 of 24
What Oversubscription Means
Blade Server I/O Hits The Wall
If you configured 80Gb of bandwidth per blade on both a Cisco UCS and HP BladeSystem, the Cisco 5108
chassis interconnects are oversubscribed with the second server. In contrast, fifteen HP blade servers can be
configured before reaching the bandwidth limit of the HP BladeSystem c7000 Platinum chassis
interconnects.
Number of Blade Servers It Takes to Hit the Limit of Chassis Interconnect Bandwidth
1.12 Tb/s
Chassis
Interconnect
Bandwidth
160Gb/s
Chassis
Interconnect
Bandwidth
Two fully configured UCS blade servers hit the limits of the 5108 fabric extenders (FEX). It takes fifteen fully configured HP ProLiant
Gen 9 blade servers to hit the bandwidth limit of the HP FlexFabric Modules.
Document # TECHBRIEF2013005 v19 November, 2014 Page 12 of 24
RDMA over Ethernet (RoCE)
RDMA over Converged Ethernet (RoCE)
InfiniBand networks were invented to overcome the need to plow through the Ethernet protocol stack to
complete an I/O transaction. InfiniBand boosts performance by eliminating layers of the stack for Remote
Direct Memory Access (RDMA). The Ethernet industry responded by developing an enhanced version of
Ethernet called Converged Ethernet (CE), featuring Priority Flow Control which is necessary to support RDMA
over Converged Ethernet (RoCE). Blade systems with switches supporting CE, and with NICs supporting
RDMA, can deliver I/O with lower latency and less CPU usage than previous generations of CNAs.
HP ProLiant Gen9 blade servers incorporate 20Gb FlexibleLOM NICs which are RDMA NICs. Cisco has
introduced RDMA LOM and Mezz NICs called the VIC 1340 and VIC 1380, respectively.
I/O Without RDMA
I/O With RDMA
Document # TECHBRIEF2013005 v19 November, 2014 Page 13 of 24
RoCE Blade Environment
Networked Storage Killer Apps for RoCE
A killer app for RoCE is SMB 3.0 file servers where users accessing shared storage experience the response
time of local storage. File servers turbo-charged with RoCE are commercially available via two Windows
Server 2012 features called SMB Multi-Channel and SMB Direct. With SMB Multichannel, SMB 3.0
automatically detects the RDMA capability and creates multiple RDMA connections for a single session. This
allows SMB to use the high throughput, low latency and low CPU utilization offered by SMB Direct.
HP FlexFabric 20Gb adapters (RDMA NICs) are certified by Microsoft for use in the killer app described above. As of 11/14/14 the VIC 1340 is not certified by Microsoft for SMB Direct.
Three Hyper-V Clusters and One File Server Cluster Using RDMA
In this diagram a single HP BladeSystem with HP 6125XLG Ethernet Blade Switches required to support RoCE, is a high performance
environment for 3 app clusters and 1 file server cluster. Hyper-V automatically senses the presence of RDMA NICs, then use multi-channel
communications to evacuate VMs in seconds, and uses direct memory access for higher I/O to shared storage inside the blade server.
Document # TECHBRIEF2013005 v19 November, 2014 Page 16 of 24
Why it Matters
IT consolidation is hugely important because it represents less hardware and simplified management. The
utilization of storage media leaped when storage was configured in a SAN and could be shared by many
servers. The utilization of physical servers dramatically increased when multiple virtual servers could be
hosted on a single physical server. Similarly, network utilization increases when more network protocols can
run on a single cable, adapter or switch.
Consolidation Metrics
There are two metrics for I/O consolidation: the convergence of network protocols, and the consolidation of
cables into higher bandwidth links.
Network Convergence
Cable Consolidation
Comparing I/O Consolidation 2
Convergence of Network Protocols
2.0 2014: Cluster/SDN Convergence
1.0 2008: LAN/SAN Convergence
IP
iSCSI
Converged Ethernet
(FCoE, Priority Flow Control)
IP
Converged Ethernet
iSCSI
FCoE
RDMA
NVGRE
VXLAN
At the Xeon E5-2600 inflection point, specialized adapters will no longer be needed to support RDMA. The new class of
adapters will also support new tunneling protocols which are essential components of software defined data centers.
Document # TECHBRIEF2013005 v19 November, 2014 Page 17 of 24
Wanted: One Blade Server Network for LANs, SANs, Cluster Networks and SDN
A new best practice for data center managers is to converge traditional shared computing infrastructure with
their growing infrastructure for distributed apps. This is made possible by a new generation of network
adapters and switches with support for the RDMA, VXLAN and NVGRE protocols. Support for these protocols
enables blade servers to converge LANs, SANs, Cluster networks and software defined networks (SDN) in a
single environment. It also allows data center managers to use software defined data center tools.
The HP 20Gb FlexibleLOM adapters supports stateless hardware offload of TCP, iSCSI and FCoE protocols for
LAN/SAN convergence, as well as hardware offload of RDMA, VXLAN and NVGRE for efficient support of
cluster and tunnel traffic. The Cisco VIC1340 supports all of the same protocols, with hardware offload for all
of the above except iSCSI.
Network Convergence 2.0 a Perfect Fit for a Webscale Private Cloud
The added support for RDMA over Converged Ethernet, NVGRE and VXLAN allow one adapter port on a blade server to support
four network environments. Hardware offload allow the blade server to use precious CPU resources for applications, instead of for
network protocol processing.
Network Convergence
Shared Distributed SDN
Document # TECHBRIEF2013005 v19 November, 2014 Page 18 of 24
A Single 40Gb Link Eliminates Cables for 40 x 1Gb Links or 4 x 10Gb Links
Until recently, 40GbE was used mostly for inter-switch connectivity and in the core of the network. The availability of 40GbE ports on servers sitting on the edge of the network has presented the opportunity for IT pros to consolidate dozens of 1GbE links and handfuls of 10GbE links with a single cable. This is an area where the HP BladeSystem stands out.
The Cisco UCS architecture makes extensive use of teaming of 10Gb ports to build uplinks with higher
bandwidth. That means lots of cables. Even the 40Gb port on the UCS Mini must be split into four cables. In
contrast, the HP Virtual Connect Modules on the HP BladeSystem include four 40GbE ports, which in the
apple-to-apples comparison below reduced the number of cables needed from 24 to 2.
Configuring Redundant 40Gb Uplinks for 16 Blade Servers
This diagram shows an apples-to-apples comparison of a 16 blade servers configured with redundant connections between servers and
switches, and redundant uplinks. Many more cables are needed in the Cisco UCS configuration because the switches are external, and
because of the lack of 40Gb ports. Note the Cisco Mini has a 40Gb port but it can only be used in a 4 x 10GbE configuration.
Cable Consolidation
Cisco UCS (24 cables) HP (2 cables)
4 x 10Gb 1 x 40Gb
Document # TECHBRIEF2013005 v19 November, 2014 Page 19 of 24
Why it Matters
A new era of agility awaits IT organizations who implement cloud operating systems designed to manage
multiple software defined data centers. Years required for a generation of hardware change will be replaced
by months required to deploy a software update. A foundation for this capability is overlay networks with
tunneling of L2 traffic across data centers using L3 networks. Support for tunneling protocols is embedded in
a new class of network adapters making it easy for private cloud builders to integrate their servers into a
cloud platform.
Conversely, IT organizations want to continue using native Fibre Channel SANs and want the flexibility to
choose “if” and “when” they converge LANs and SANs on Ethernet.
I/O Flexibility Metrics
There are two capabilities which are expected to effect I/O flexibility in Webscale private clouds.
More efficient delivery of tunnel traffic with hardware offload of tunnel protocol processing
Support for native Fibre Channel
Comparing I/O Flexibility 3
Document # TECHBRIEF2013005 v19 November, 2014 Page 20 of 24
Live Migrations a Killer App for VXLAN and NVGRE
One of the most valuable functions of server virtualization is live migration. This function frees system
administrators from the time-consuming and complex process of moving workloads to optimize performance
or mitigate a hardware failure. However, moving VMs on different networks requires extensive network
reconfiguration. IT organizations using data center infrastructure dispersed in public, private or hybrid clouds
simply can’t configure all servers and VMs on one local network, and need a tunneling mechanism to extend
live migrations.
Virtual Extensible LAN (VXLAN) and Network Virtualization using Generic Routing Encapsulation (NVGRE ) are
protocols for deploying overlay (virtual) networks on top of a Layer 3 networks. VXLAN and NVGRE are used
to isolate apps and tenants in a cloud and migrate virtual machines across long distances.
While VXLAN and NVGRE allow live migrations across racks and data centers. RoCE accelerates live
migrations. In a Microsoft TechEd demo, migrating Windows Server 2012 to a like system takes just under 1
minute 26 seconds. Windows Server 2012R2 performed the same migration in just over 32 seconds. Then
using RoCE during the live migration process combined with SMB Direct, it took just under 11 seconds,
Document # TECHBRIEF2013005 v19 November, 2014 Page 23 of 24
HP ProLiant Gen9 Blade Server
Designed for Workloads of the Future
The HP ProLiant Gen9 Blade Server is designed for I/O flexibility with a choice of HP FlexFabric converged
networking or parallel Ethernet and Fibre Channel networks. The HP ProLiant Gen9 Blade Server is also fully
compliant with Windows Server 2012 Virtual Fibre Channel—an innovation that will play an important role in
the virtualization of Tier-1 workloads with Microsoft Hyper-V.
HP ProLiant Gen9 Blade Servers in a c7000 Enclosure
HP Virtual Connect FlexFabric
20/40 F8 module supports “FlatSAN”
direct connectivity to native Fibre Channel
3PAR storage at a lower cost than using
Fibre Channel switches
HP Virtual Connect FlexFabric 20/40 F8 module supports LAN, SAN, NAS, iSCSI and FCoE connectivity
Native Fibre Channel server adapter
Over 12 million ports shipped on this stack
Complete enterprise OS support
Ethernet LAN on Motherboard (LOM) or Mezz adapter
Dual 10/20GbE Ports
Supports LAN, NAS, iSCSI and FCoE connectivity
Supports RoCE for scale-out cluster connectivity.
Supports NVGRE and VXLAN for migrating VMs across the cloud.
718203-B21 HP LPe1605
16Gb Fibre Channel HBA
HP FlexFabric 20Gb 2-port 650FLB Adapter
Document # TECHBRIEF2013005 v19 November, 2014 Page 24 of 24
Resources
Summary Infrastructure of the past is functionally defined and purpose-built. Servers are servers, networking is networking and storage is storage. These purpose-built devices are deployed with little ability to change the function as needs change. In the future, infrastructure needs to be more transformative, taking the shape of business demands. Potential power and flexibility is locked inside the aging Cisco UCS 5108 chassis which severely limits the use of new high-bandwidth networks and any network other than Ethernet/FCoE. The new HP BladeSystem answers the call with: • A new level of convergence which will allow for resources to be allocated at a very granular level,
improving efficiencies and ensuring optimal performance as workload demands change. • Interfaces to the software-defined data center. HP ProLiant Gen9 blade servers possess the capability to
respond to intelligent orchestration of infrastructure resources in real-time, as applications and user needs change.
• A cloud-ready architecture ready to scale-out, agile, and always on. • Workload-optimized for traditional share-everything applications and new share-nothing applications.
Related Links
OCe14000 Test Report HP FlexFabric Adapters Provided by Emulex HP BladeSystem HP Virtual Connect Technology HP BladeSystem and Cisco UCS Comparison Cisco Fabric Extender Cisco UCS Virtual Interface Card 1340 Cisco UCS 6324 Fabric Interconnect Data Sheet Cisco UCS Ethernet Switching Modes IT Brand Pulse
About the Author
Joe Kimpler is a senior analyst responsible for IT Brand Pulse Labs. Joe’s team manages the delivery of technical services including hands-on testing, product reviews, total cost of ownership studies and product launch collateral. He has over 30 years of experience in information technology and has held senior engineering and marketing positions at Fujitsu, Rockwell Semiconductors, Quantum and QLogic. Joe holds an engineering degree from the University of Illinois and a MBA in marketing.