Lab Testing Summary Report - Miercom

Lab Testing Summary

Report

Key findings and conclusions: • The Cisco ASR 9922 reroutes around failed links quicker – up

to 97 percent quicker for some services – and with much less user data loss than does the Alcatel-Lucent 7750 SR-12e

• The Cisco ASR 9922 consistently delivers line-rate throughput and QoS traffic handling, and low, consistent latency, even while handling 50 percent more service-processing complexity

• The Alcatel-Lucent 7750 SR-12e exhibited per-port VLAN restrictions, disappointing handling of high-priority QoS traffic and the FP3 failed to support its 200Gbps claim

• The Juniper MX960 3D could not handle the high service scale configuration required for some tests. It exhibited wide latency variability and resiliency issues with replicated-packet handling, and failed to meet the vendor's own 10GE throughput claims with even light service processing

April 2014 Report 140305

Product Category:

Ethernet Services Edge Routers

Vendors, Products Tested:

7750 SR-12e

ASR 9922

MX960 3D

Only Cisco delivered the full load of traffic received on its 24-port 10GE line card, with heavy VLAN and HQoS processing. Alcatel-Lucent forwarded just 54 percent via its 20-port card, and Juniper just 52 percent via its 8-port card.

T his report summarizes the dramatic results of Miercom's months' long, hands-on testing of leading Provider Edge (PE) routers, which are collectively called Ethernet Services Edge Routers.

The products tested, from Alcatel-Lucent, Cisco and Juniper, are these vendors' current top-of-the-line access routers for service providers. This bold comparative study is among the first to apply such 'multi-

Source: Miercom, April 2014

Aggregate Throughput for Vendor's Highest Density 10GE Line Card Tested with Maximum VLANs per Port and HQoS applied,

with 99 Percent-Load, Bidirectional Traffic Streams

ASR 9922 ALU 7750 MX 960

Throughput (Gbps) 222.9 36.4 16.8

0

50

100

150

200

250

Thro

ughp

ut

ASR 9922

ALU 7750

MX 960Drop

ped

Pack

ets

Drop

ped

Pack

ets

Ingress

ASR 9922 ALU 7750 MX 960

Throughput (Gbps) 222.9 164 40.6

0

50

100

150

200

250

Thro

ughp

ut

ASR 9922

ALU 7750

MX 960

Dropped Packets

Dropped Packets

Egress

Copyright © 2014 Miercom Ethernet Services Edge Routers Page 2

dimensional' tests to high-end, modular and multislot routers. A diverse but consistent set of Ethernet-based services was configured on each of the devices under test (DUTs), up to scale, or the maximum extent the system would allow, and then a staggering array of test equipment delivered Gigabits/s of traffic load to each, while carefully measuring different aspects of the DUTs' performance. The different performance aspects of the routers examined included: • Recovery from a variety of failed-path

scenarios, in terms of simulated user data lost and elapsed time until resumption of normal service data flows.

• The maximum sustainable throughput of the vendors' highest density 10-Gig Ethernet Services line cards, first with minimal per-port processing and then with maximum VLANs and Hierarchical Quality of Service (HQoS) applied on all ports to assess the handling of HQoS services under load.

• Point-to-multipoint and replicated-packet handling, in terms of point-to-multipoint throughput, replicated-packet latency and protection of replicated packet streams in the event of a router card failure.

Full details of the testing recapped here can be found in a companion White Paper, Report DR140121, issued along with this summary.

Rerouting Around a Failed Path A battery of tests measured how quickly the device under test (DUT) could reroute, or reconverge, around an internal-network link failure – when one or more equal-cost links exist – for the different services running. Also measured was how much user data was lost as a result of such a link outage. The tests were done using bi-directional traffic; test traffic was

delivered at below a 50 percent rate, to prevent oversubscription after the link fails. The links that were failed assessed the device under test's (DUT's) ability to respond to: a single high-speed core link member failure, in an Ethernet Link Aggregation group (LAG) and then the failure of a whole Ethernet LAG.

The test data streams were bi-directional flows for each unicast service, and unidirectional outbound (PE-to-CE) for multicast services which, by packet addresses and type, exercised most of the different services running on the DUT.

Half of the test traffic was being carried over two 100GE links that together comprised an aggregated Ethernet bundle (the Link Aggregation Group, or LAG). To determine the percent of traffic impacted by the failure scenario we caused, we presumed that each DUT load-balanced equally between equal-cost multiple paths and link members within a bundle. As shown in the diagram, 25 percent of the test traffic flows were being carried over each 100-gigabit/s Ethernet (100GE) link for the single high-speed core link failure.

To simulate the failure of the whole Ethernet group, both links were simultaneously failed, either by physically popping the line card in the DUT that handled the bi-directional traffic over these two 100GE links, or by gracefully shutting down the links on the core router. Three Cisco CRS core routers were used in exactly the same manner for all the tests.

In this battery of tests, the Juniper MX960 3D router was not included. In attempting to configure the number of concurrent services and high-speed links that our test bed specified, we encountered software issues with the MX960: Error messages indicated insufficient resources. The Juniper router could be successfully configured for less complex configurations, but in order to present a comparable "apples to apples" test environment, we deemed that including the Juniper MX960 in these tests would be inappropriate. Juniper was afforded the opportunity to address the issue but did not offer a resolution to the configuration issue.

Frame Loss by Service, Single-Link Failure Given the same multi-service flows of simulated customers' bidirectional traffic, frame loss by service was typically less with the Cisco ASR 9922 than with the Alcatel-Lucent 7750. For redirecting traffic around a single 100GE backbone link failure, loss for VPLS service was 73 percent less with Cisco than Alcatel-Lucent. EoMPLS service loss was nearly 80 percent less with Cisco.


Table 1 shows the amount of data loss, incurred by service, as a result of a single core-link outage, with the Cisco ASR 9922 and the Alcatel-Lucent 7750, before traffic could be redirected around the failed link and the service resumed normal traffic flow. Given the same multi-service flows of simulated customers' bidirectional traffic, frame loss by service was typically less with the Cisco ASR 9922 than with the Alcatel-Lucent 7750. For redirecting traffic around a single 100GE backbone link failure, loss for VPLS service was 73 percent less with Cisco than Alcatel-Lucent. EoMPLS service loss was nearly 80 percent less with Cisco. Table 2 shows the duration of service outage – that is, the time, in milliseconds, from the single 100GE core-link failure, until the device under test (DUT) successfully redirected traffic and full traffic flow for that service resumed. With the same multi-service flows of simulated customers' bidirectional traffic, the Cisco ASR 9922 rerouted most services around a failed network-core link faster than the Alcatel-Lucent 7750, enabling services to resume full traffic rates quicker. Full traffic levels for VPLS service, for example, resumed up to 65 percent faster with Cisco, and Ethernet-over-MPLS service 80 percent faster, than Alcatel-Lucent. Also tested was the device under test's (DUT's) ability to recover from an entire failed Ethernet bundle. When all links in a bundle fail, the path between the two points is severed, including IP-path connectivity. This is a higher-layer and more severe failure, requiring IP alternate-path re-routing and re-convergence. The Ethernet bundle failure was tested "locally," by popping the card within the device under test (DUT) that handled both links of the bundle; and "remotely," where the bundle was administratively shut down at the CRS core router end. As shown in Table 3, the amount of data loss from such a bundle failure can be substantial. However, loss was much less, by service, with the Cisco ASR than with the Alcatel-Lucent 7750. Loss for VPLS and Layer 3 VPN services were more than 99 percent less with Cisco than with Alcatel-Lucent. Also significant were the differences in the duration of service outages resulting from the failure of an entire Ethernet bundle. In the case

Table 1: Loss by Service, from a Failed 100GE Link

Service Cisco ASR

9922 (No. of

Frames Lost)

Alcatel-Lucent 7750

(No. of Frames Lost)

Virtual Private LAN Service (VPLS), LDP-controlled

3,196 8,599

Virtual Private LAN Service (VPLS), BGP-controlled

3,209 8,778

Ethernet over Multiprotocol Label Switching (EoMPLS)

599 2,864

High-Speed Internet (HIS) 844 2,410 Layer 3 Virtual Private Network (L3VPN) 8,085 22,890

IPv6 VPN 816 2,223 Multicast VPN 180 80 Broadband Network Gateway 2,320 2,505

Table 2: Service Resumption Times after 100GE Link Failure

Service Cisco ASR

9922 (Outage time in milliseconds)

Alcatel-Lucent 7750

(Outage time in milliseconds)


4.4 15.6


5.7 16.0


5.5 30.0

High-Speed Internet (HIS) 6.5 15.5 Layer 3 Virtual Private Network (L3VPN) 15.5 15.5

IPv6 VPN 5.5 5.0 Multicast VPN 30.0 17.5 Broadband Network Gateway 56.4 60.9

Table 3: Loss by Service, from a Remote Ethernet Bundle Failure

Service Cisco ASR

9922 (No. of

Frames Lost)

Alcatel-Lucent 7750

(No. of Frames Lost)


153,042 4,481,307


153,003 5,109,156


25,318 670,859

High-Speed Internet (HIS) 92,982 1,323,069 Layer 3 Virtual Private Network (L3VPN) 464,575 25,088,092

IPv6 VPN 47,017 1,978,832 Multicast VPN 3,984 46,568 Broadband Network Gateway 185,516 671,460



A notable difference in frame loss was observed between the tested routers, resulting from a single remote-PE-path failure, for Layer 3 (IP-based) Virtual Private Network (L3VPN) service.

With a single remote-link failure, our testing found that VPN IP traffic through the Alcatel-Lucent 7750 suffered 1.67 million lost frames, compared to just 180,000 lost packets with the Cisco ASR 9922. In other words, a simulated customer's Layer 3 VPN incurred nine times more loss through the Alcatel-Lucent 7750 than through the Cisco ASR 9922.

The duration of service outage for the Layer 3 VPN service, before reconvergence, is shown below for the Cisco ASR 9922 (top) and the Alcatel-Lucent 7750 (bottom), in response to the multiple-remote-path-link failure.

of the local bundle outage, the Cisco ASR 9922 was able to reconverge VPLS service, allowing full-rate traffic to resume, 97 percent quicker than the Alcatel-Lucent 7750. High-Speed Internet (HSI) service resumed 96 percent quicker with the Cisco ASR and Multicast VPN service 90 percent faster, than with the Alcatel-Lucent 7750.

With the remote Ethernet bundle failure, the Cisco ASR 9922 allowed full-rate traffic to resume for most services in less than 200 milliseconds. By comparison, reconverging around the same remote Ethernet bundle outage typically took the Alcatel-Lucent 7750 from 20 to 500 times longer.

Multi-Path Reconvergence Another series of tests assessed the device under test's (DUT's) ability to promptly address the failure of multiple remote routes, what is termed multi-path reconvergence. The state of the IP routes is learned from remote Provider Edge routers (PEs) using Multi-protocol BGP. In these tests the routes across the service provider's network are MPLS-based Virtual Private Network (VPN) routes. This convergence scenario for IP VPNs is commonly referred to as “BGP PIC Edge.”

The topology for this testing was complex, and readers are referred to the full White Paper report for complete test details. Customer-edge routers were dual-homed to different PE routers. Each PE had a unique route distinguisher, and one of the two PEs within each dual-homed CE "customer" connection had a higher local preference value for traffic routing. VLANs (per IEEE 802.1q) were used to provide IP-path separation between the primary and backup paths to each CE, which the test equipment emulated.

Test traffic was delivered bi-directionally for all services. However, the traffic of interest in these tests was IPv4 VPN traffic from the customers on the device under test (DUT) to the emulated PE customers connected at the far end of the network.

We exercised two path-failure scenarios: When two remote PE paths fail – we failed the path to the higher-preference-VPN PE; and when multiple paths fail – we failed paths to multiple-VLAN PEs.

This testing was conducted with each DUT configured at the same scale as the Rerouting Around a Failed Path test. As the scale tested for each DUT was based on its number of ports, the Cisco ASR 9922 had to manage 50 percent more prefixes impacted by the outages than the Alcatel-Lucent 7750 SR-12e.

Over 15 times faster recovery. Reconvergence after a remote multi-path failure took 1,892 milliseconds via the Alcatel-Lucent 7750, nearly two full seconds, before Layer 3 VPN traffic levels resumed normally. The same outage time for the Cisco ASR 9922 was 124 milliseconds – over 15 times faster than the Alcatel-Lucent 7750.

Duration of Outage before Multi-path Reconvergence


Cisco = 124 ms

Alcatel-Lucent = 1,892 ms


Overall Performance Another set of tests sought to measure the vendors' claimed performance of these Ethernet Services Edge Routers. We focused on the throughput of each vendor's highest density 10-Gigabit Ethernet (10GE) Services Edge line card.

The vendors offer quite different 10GE line cards. The below table shows the cards we tested, the highest density 10GE cards shipping at the time of our testing, and the vendor claims as far as the number of ports per card concurrently supporting line-rate throughput. That is the number of ports we tested in this round.

Device Under Test: Line Card

No. of 10GE

Ports on Line Card

Claimed Line Rate,

No. of 10GE Ports

No. of 10GE Ports

Tested

Alcatel-Lucent 7750: IMM-2PAC-

FP3 20x10GE 20 20 20

Cisco ASR 9922: 24x10GE A9K-

24x10GE-SE 24 24 24

Juniper MX960: MPC2E-3D-EQ with 8x10Ge

8 6 6

Bi-directional test traffic streams were applied to each port. An IMIX traffic pattern – a blend of IP packet types and packet sizes, from minimal 66 to 1,518 bytes – was used. The test systems generated about 3.82 million packets per second, per 10GE interface, delivering the specified rate of 99 percent of each 10GE port's capacity.

Tests were run initially with minimal additional processing per port – just a single VLAN per port was defined. Afterwards, we added service- processing complexity: we configured 850 VLANS per port with a shaper and basic three class Hierarchical Quality of Service (HQoS) processing on each VLAN. Cisco and Juniper supported this; Alcatel-Lucent, we learned, supported only 714 IPv4-enabled VLANs per port.

Of the 99 percent of full 10GE capacity applied on each port, the three HQoS classes (assured forwarding, expedited forwarding and best effort) accounted for equal 33 percent volumes of the traffic capacity. Traffic of each HQoS class was split equally to and from every VLAN configured.

The aggregate throughputs achieved for the vendors' 10GE line cards are shown on the front page of this report. These throughputs reflect the most complex per-port service-processing environment: with VLANs at scale and with HQoS processing applied. "Ingress" reflects throughput in one direction, "egress" in the other relative to the line card under test.

We observed considerable loss and improper handling of HQoS classes by both Alcatel-Lucent and Juniper. Throughputs in the egress direction are shown in the figures below.

Alcatel-Lucent 7750, Egress Throughput

Cisco ASR 9922, Egress Throughput

Juniper XM960, Egress Throughput



With just one VLAN configured per port and no other exertive processing, we observed appreciable packet loss with both the Juniper MX960 3D and the Alcatel-Lucent 7750 SR-12e. Juniper dropped 7 percent of the traffic in one direction, and so failed to support line rate on six 10GE ports, as claimed in the vendor's datasheets.

The Alcatel-Lucent 7750 SR-12e dropped 26 percent of the traffic in one direction and 28 percent in the other on the line card under test.

No packets were dropped by the Cisco ASR-9922 in the same configuration and with the same 99-percent bi-directional load. Cisco System’s claim to be able to handle line rate, full duplex, on 24 x 10GE ports was therefore validated.

Under heavy load, with hundreds of VLANs configured per port and with Hierarchical QoS enabled and running, Alcatel-Lucent processed and sent just 30 percent of the high-priority traffic in one direction, along with 5 percent of the default Best Effort. In the other direction it delivered 94 percent of the Assured Forwarding and 80 percent of the Expedited Forwarding.

Under load, Juniper processed and forwarded about 40 percent of the priority traffic and about 13 percent of the low-priority Best Effort traffic. In the other direction Juniper managed 90 percent of the high priority traffic, and about 38 percent of Best Effort traffic.

Only Cisco managed to correctly process 100 percent of all three HQoS traffic classes, in both directions, and with no loss.

Packet Replication, Multipoint A final battery of tests focused on packet replication, the key component in point-to-multipoint applications. We sought to exercise and observe the DUTs' ability to effectively replicate and deliver a high volume of traffic to output ports. We checked, too, to see how resilient point-to-multipoint traffic handling was when disruptions, such a line-card failure, occur within the router.

Tests were devised to compare DUT performance with regard to: • Packet-replication-induced latency. • Throughput performance for replicating a

single 9.5-Gbps flow. The DUT's performance was tested with a single multicast stream.

• Packet-replication resilience. We intentionally failed a customer/receiver-side line card to observe what impact it would have on other customers/receivers, connected via other line cards, who also relied on packet replication from the DUT.

These tests were performed with scaled (maximum configurable) services all enabled and active, but with no unicast traffic flowing. The number of 10GE output ports closely monitored for this testing were:

• Alcatel-Lucent: 48 x 10GE ports • Cisco: 72 x 10GE ports • Juniper: 60 x 10GE ports

The figure below shows the latency differences of a multicast stream on the DUT's output ports.

Latency for replicated traffic should be low and consistent across all output ports. In Juniper's case, latency was two or three times as much on some ports. Alcatel-Lucent's latency exhibited smaller variation. Cisco's latency was consistently low.


Multicast Latency by Port

Cisco Alcatel Juniper


The Cisco ASR 9922 exhibited remarkable consistency, delivering all point-to-multipoint traffic to all output ports with the same, consistent, sub-50-microsecond latency. The Alcatel 7750 exhibited variable latency, depending on the output port. The most inconsistent and variable latency was experienced by replicated-packet traffic through the Juniper MX960 3D. Latency varied considerably depending on output port.

Juniper exhibited the most variation in latency, as the chart below shows. The latency of replicated-packet traffic through the Alcatel-Lucent 7750 varied much less. Cisco delivered consistent latency; multicast traffic through the Cisco ASR 9922 varied barely 2 microseconds from the average of 48 microseconds.

Cisco and Juniper were both able to replicate and deliver the full 9.5-Gbit/s multicast input stream onto all output ports. However, Alcatel-Lucent exhibited serious issues in processing replicated-packet traffic, replicating and delivering just 20 percent of the input stream, as the chart below shows.

Our last test, of point-to-multipoint resiliency – the ability of the DUT to protect replicated packet streams – revealed an issue with Juniper.

In this test we sent a unidirectional input traffic stream at 9.5 Gbps, a near-maximum rate for large-sized, 1,518-byte packets, to a 10-Gigabit/s Ethernet (10GE) port. The DUT then has to replicate and forward this stream to every other 10GE port on the router. All services were active and running. Test equipment measured how much replicated-packet traffic was delivered on each output port. Then we intentionally failed one of the DUT's customer/receiver-side line cards, to see what impact it would have on other customers/ receivers, connected via other line cards.

With Cisco and Alcatel-Lucent there was no impact whatever on the other, unaffected output ports. On the Juniper MX960 3D, however, every output port experienced loss, over 8,000 large-size packets, when the line card was failed, until the other, unaffected, replicated-packet streams resumed normal, flow. This loss on all Juniper ports is shown in the chart below.

The Bottom Line The testing of high-end Provider Edge (PE) access routers in this study revealed crucial performance differences, especially under heavy traffic loads while running multiple concurrent services.

These multi-dimensional tests, among the first to scientifically apply such loads to these systems, caution service providers in accepting vendors' isolated performance claims.

Minimum / Average / Maximum Latency


Multicast Throughput

Gbps

Cisco Alcatel Juniper Source: Miercom, April 2014

Multicast Resiliency - Juniper

Ports

Packet Loss



Product names or services mentioned in this report are registered trademarks of their respective owners. Miercom makes every effort to ensure that information contained within our reports is accurate and complete, but is not liable for any errors, inaccuracies or omissions. Miercom is not liable for damages arising out of or related to the information contained within this report. Consult with professional services such as Miercom Consulting for specific customer needs analysis.

About Miercom’s Product Testing Services

Report 140305 [email protected] www.miercom.com

Miercom has hundreds of product-comparison analyses published over the years in leading network trade periodicals including Network World, Business Communications Review, Tech Web - NoJitter, Communications News, xchange, Internet Telephony and other leading publications. Miercom’s reputation as the leading, independent product test center is unquestioned. Miercom’s private test services include competitive product analyses, as well as individual product evaluations. Miercom features comprehensive certification and test programs including: Certified Interoperable, Certified Reliable, Certified Secure and Certified Green. Products may also be evaluated under the NetWORKS As Advertised program, the industry’s most thorough and trusted assessment for product usability and performance.

Before printing, please consider electronic distribution

Cisco Systems, Inc. 170 West Tasman Drive

San Jose, CA 1-800-553-6387 www.cisco.com

Miercom Performance Verified The Cisco ASR 9922 has demonstrated in independent testing that it delivers wire-speed throughput and performance under conditions where competitors cannot.

Stringent tests also found that the Cisco ASR 9922 recovers from failed network links much faster, and with much less loss of user data, than competitors. We especially laud the Cisco router's impeccable latency and QoS-handling performance.

Testing proved that the Cisco ASR 9922 has earned the Miercom Performance Verified Certification.

Cisco ASR 9000 Router Model ASR 9922

mailto:[email protected]�

http://www.miercom.com/�

http://www.cisco.com/�

Lab Testing Summary Report - Miercom

Documents