Nss Labs Firewall Group Test q1 2011 v9

NETWORK FIREWALL 2011 COMPARATIVE TEST RESULTS

CHECK POINT POWER-1 11065 CISCO ASA 5585

FORTINET FORTIGATE 3950

JUNIPER SRX 5800 PALO ALTO NETWORKS PA-4020

SONICWALL NSA E8500

APRIL 2011 METHODOLOGY VERSION: 3.0

Licensed to: Purchaser (Single-User, INTERNAL USE ONLY) To receive a licensed copy or report misuse, please contact NSS Labs at: +1 (760) 270-9852 or [email protected].

Network Firewall Comparative Test Results ©2011 NSS Labs, Inc. All rights reserved.

©2011 NSS Labs, Inc. All rights reserved. No part of this publication may be reproduced, photocopied, stored on a retrieval system, or transmitted without the express written consent of the authors.

Please note that access to or use of this report is conditioned on the following:

1. The information in this report is subject to change by NSS Labs without notice. 2. The information in this report is believed by NSS Labs to be accurate and reliable at the time of

publication, but is not guaranteed. All use of and reliance on this report are at the reader’s sole risk. NSS Labs is not liable or responsible for any damages, losses, or expenses arising from any error or omission in this report.

3. NO WARRANTIES, EXPRESS OR IMPLIED ARE GIVEN BY THE NSS LABS. ALL IMPLIED WARRANTIES, INCLUDING IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, AND NON-INFRINGEMENT ARE DISCLAIMED AND EXCLUDED BY NSS LABS. IN NO EVENT SHALL NSS LABS BE LIABLE FOR ANY CONSEQUENTIAL, INCIDENTAL OR INDIRECT DAMAGES, OR FOR ANY LOSS OF PROFIT, REVENUE, DATA, COMPUTER PROGRAMS, OR OTHER ASSETS, EVEN IF ADVISED OF THE POSSIBILITY THEREOF.

4. This report does not constitute an endorsement, recommendation, or guarantee of any of the products (hardware or software) tested or the hardware and software used in testing the products. The testing does not guarantee that there are no errors or defects in the products or that the products will meet the reader’s expectations, requirements, needs, or specifications, or that they will operate without interruption.

5. This report does not imply any endorsement, sponsorship, affiliation, or verification by or with any organizations mentioned in this report.

6. All trademarks, service marks, and trade names used in this report are the trademarks, service marks, and trade names of their respective owners.

CONTACT INFORMATION NSS Labs, Inc. P.O. Box 130573 Carlsbad, CA 92013 USA +1 (760) 270-9852 [email protected] www.nsslabs.com


EXECUTIVE SUMMARY Corporate networks and data are under attack more than ever, and the threats continue to change as do vendor solutions. Firewalls will continue to play a key role in layered defenses. An essential part of layered security, firewalls must be stable, fast, and easy to deploy and maintain.

During Q1 2011, NSS Labs performed the industry’s most rigorous test of leading firewall solutions. This report has been produced for our enterprise subscribers, as part of NSS Labs’ independent testing information services. Leading vendors were invited to participate fully at no cost, and NSS Labs received no vendor funding.

The time required to install and configure each device was recorded for purposes of estimating the total cost of ownership (TCO) calculations. Effectiveness and performance results were obtained with identical policies across products in order to provide comparable results.

Key Findings

• Five of the six products allowed external attackers to bypass the firewall and become an internal “trusted” machine.

• Three of the six products tested crashed when subjected to our stability tests. These kind of crashes indicate the existence of a vulnerability which an attacker may be able to exploit in the field, given enough time. This lack of resiliency is alarming, especially considering all three were certified by ICSA Labs and/or Common Criteria certified.

• Performance claims in vendor datasheets are generally grossly overstated. Measuring performance based upon RFC-2544 (UDP) does not provide an accurate representation of how the firewall will perform in live real-world environments.

Recommendations

• If you have one of the firewalls that has issues with TCP split handshake, review NSS Labs’ remediation guidelines and contact your vendor.

• Before implementing remediations, consider performance impacts of turning on additional protections. Contact NSS Labs for assistance with determining performance requirements and capabilities of devices.

• If your firewall failed NSS Labs’ stability tests, encourage your firewall vendor to address stability issues ASAP to avoid exploitation. Consult NSS Labs subsequent firewall test report later in 2011 to ensure your vendor has remediated the issues.

• If your firewall is crashing, locking up, or displaying other unstable behavior, it may be the subject of an attack.

• If your organization is extremely risk averse or highly sensitive to down-time, consider migration to one of the more stable firewall platforms in our tests.


TABLE OF CONTENTS 1 Introduction ................................................................................. 1

1.1 The Need for Firewalls ...................................................................... 1 1.2 The Need for Testing ........................................................................ 1 1.3 About This Test Methodology and Report .......................................... 1 1.4 Tested Products ................................................................................ 1 1.5 About NSS Labs ................................................................................ 2

2 Security Effectiveness .................................................................. 3 2.1 Firewall Policy Enforcement .............................................................. 3

3 Performance ................................................................................ 8

4 Stability & Reliability .................................................................. 12

5 Total Cost of Ownership & Value ................................................ 15

6 Product Guidance ....................................................................... 17 6.1 Recommend .................................................................................... 18 6.2 Neutral ........................................................................................... 18 6.3 Caution ........................................................................................... 20

7 Test Methodology Elements Overview ........................................ 21

Appendix A: Special Thanks ............................................................ 23


TABLE OF FIGURES Figure 1: Rated Throughput (Mbps) ........................................................................................... 8 Figure 2: Maximum Concurrent Connections vs. HTTP Transactions & TCP Connections per Second .... 9 Figure 3: Maximum HTTP Connections per Second with Various Size Payloads ............................... 10 Figure 4: Maximum THroughput (MBPS) with Various Size Payloads ............................................. 10 Figure 5: UDP THroughput (MBPS) .......................................................................................... 11 Figure 3 - 3 year TCO ............................................................................................................ 16

Network Firewall Comparative Test Results 1 ©2011 NSS Labs, Inc. All rights reserved.

1 INTRODUCTION

1.1 THE NEED FOR FIREWALLS Firewall technology has been around for at least 25 years, and has undergone several stages of development; from early packet and circuit firewalls to application layer and dynamic packet firewalls. Across these stages, the goal has continued to be to provide a protective barrier between internal and external networks, while allowing for productive communications to pass from one side to the other.

As firewalls will be deployed at critical choke-points in the network, the stability and reliability of a Firewall is imperative. Therefore prime directive of any firewall is that it must be stable / reliable. And it must not degrade network performance or it will never be installed.

In order to establish a secure perimeter, a firewall must provide granular control based upon the source and destination IP Addresses and ports.

The following capabilities are considered essential as part of a firewall:

• Basic packet filtering • Stateful multi-layer inspection • NAT • Highly Stable • Ability to operate at layer 3

1.2 THE NEED FOR TESTING In this Firewall Group Test, NSS Labs’ objective was to answer the critical questions about product capabilities and limitations that enterprises could not answer for themselves without great effort and investment in time, equipment, and specialized expertise. In the process, we discovered some failures in the way firewalls have been traditionally tested. As a result, we found stability was more of an issue than we had anticipated – given that all of the problematic products recently passed through certification with another major lab and/or were Common Criteria certified. And all but one vendor failed to properly handle a type of spoofing called a TCP Split Handshake attack.

We believe the results indicate the need for more in-depth testing on a recurring basis. Considering that “what gets measured, gets managed, and (hopefully) improved,” we look forward to working further with end-users and vendors to continue enhancing corporate defenses.

1.3 ABOUT THIS TEST METHODOLOGY AND REPORT NSS Labs’ test reports are designed to address the challenges faced by IT professionals in selecting and managing security products. The scope of this report is focused on:

• Security effectiveness • Performance • Stability • Total Cost of Ownership (TCO)

1.4 TESTED PRODUCTS NSS Labs’ testing is known to be the most rigorous in the industry, providing readers with hard-hitting real-world research and analysis. It is understandable that some vendors are concerned about the


marketing impact of potentially poor performance. However, enterprise and government buyers are increasingly asking what is getting through, why, and what to do about it. Vendors who chose to participate should be commended for their commitments to transparency and improvement.

In order to garner the greatest participation, and allay any potential concerns of bias, we invited all leading vendors to submit products at no cost. We selected products for inclusion based on enterprise client requests. The following is a current list of the products that were tested, sorted alphabetically:

Check Point Power-1 11065 Cisco ASA 5585-40 Fortinet Fortigate 3950B

Juniper SRX 5800 Palo Alto Networks PA-4020 Sonicwall NSA E8500

All firewalls were generally available (GA) products. No Beta or otherwise unavailable products were included.

1.5 ABOUT NSS LABS NSS Labs, Inc. is the world’s leading independent information security research and testing organization. Its expert analyses provide information technology professionals with the unbiased data they need to select the right product for their organizations. Pioneering intrusion detection and prevention system testing with the publication of the first such test criteria in 2001, NSS Labs also evaluates firewall, unified threat management, anti-malware, encryption, web application firewall, and other technologies on a regular basis. The firm’s real-world test methodology is the only one to assess security products against live Internet threats. NSS Labs tests are considered the most aggressive in the industry and its recommendations are highly regarded by enterprises. Founded in 1991, the company has offices in Carlsbad, California and Austin, Texas.


2 SECURITY EFFECTIVENESS This section verifies that the Device Under Test (DUT) is capable of enforcing a specified security policy effectively. NSS Labs’ firewall analysis is conducted by incrementally building upon a baseline configuration (simple routing with no policy restrictions and no content inspection) to a complex real world multiple zone configuration supporting many addressing modes, policies, applications, and inspection engines.

At each level of complexity, test traffic is passed across the firewall to ensure that only specified traffic is allowed and the rest is denied, and that appropriate log entries are recorded.

The firewall must support stateful firewalling either by managing state tables to prevent “traffic leakage” or as a stateful proxy. The ability to manage firewall policy across multiple interfaces/zones is a required. At a minimum, the firewall must provide a “trusted” internal interface, an “untrusted” external/Internet interface, and (optionally) one or more DMZ interfaces. In addition, a dedicated management interface (virtual or otherwise) is preferred.

2.1 FIREWALL POLICY ENFORCEMENT Policies are rules that are configured on a firewall to permit or deny access from one network resource to another based on identifying criteria such as: source, destination, and service. A term typically used to define the demarcation point of a network where policy is applied is a demilitarized zone (DMZ). Policies are typically written to permit or deny network traffic from one or more of the following zones:

• Untrusted – This is typically an external network and is considered to be an unknown and non-secure. An example of an untrusted network would be the Internet.

• DMZ – This is a network that is being isolated by the firewall restricting network traffic to and from hosts contained within the isolated network.

• Trusted – This is typically an internal network; a network that is considered secure and protected.

The NSS Labs Firewall certification tests performance and the ability to enforce policy between the following:

• Trusted to Untrusted • Untrusted to DMZ

Trusted to DMZNote: Firewalls must provide at a minimum one DMZ interface in order to provide a DMZ or “transition point” between untrusted and trusted networks.


2.1.1 BASELINE POLICY Routed configuration with an “allow all” policy

Product Result CheckPoint Power-1 11065 PASS Cisco ASA 5585-40 PASS Fortinet 3950B PASS Juniper SRX 5800 PASS Palo Alto PA-4020 PASS Sonicwall NSA E8500 PASS

2.1.2 SIMPLE POLICIES Simple outbound and inbound policies allowing basic browsing and e-mail access for internal clients and no external access


2.1.3 COMPLEX POLICIES Complex outbound and inbound policies consisting of many rules, objects, and services.


2.1.4 STATIC NAT (NETWORK ADDRESS TRANSLATION) Inbound Network Address Translation (NAT) to DMZ using fixed IP address translation with one-to-one mapping.



2.1.5 DYNAMIC/HIDE NAT (NETWORK ADDRESS TRANSLATION) Outbound Network Address Translation (NAT) (from Internal to External) where all outbound traffic “hides” behind the IP Address of the External Interface of the Firewall utilizing a pool of high ports to manage multiple connections.


2.1.6 SYN FLOOD PROTECTION The basis of a SYN Flood attack is to not complete the 3-way handshake necessary to establish communication. Specifically the attacker (client machine A in fig. 6) refusing to send the ACK signal to the host server (B) after receiving the SYN/ACK from Host B. Such a connection is called a half open connection.

Instead of sending an ACK, attacker A sends another SYN signal to the victim server. The server again acknowledges it with a SYN/ACK and B again refuses to send the final ACK signal. By repeating this several times the attacker tries to overflow the data structure of the host server. The data structure is built in the memory of the host server with the purpose of keeping records of connections to be completed (or half open connections). Since the data structure is of a finite size, it is possible to overflow it by establishing a large number of open connections.

Once overflow occurs the host server will not be able to accept new connections thus resulting in a denial of service. There is however a time-out associated with each of the connections (approximately 3 minutes) after which the host server will automatically drop the half open connections and can start accepting new connections. If the attacker can request connections at a rate higher than the victim servers ability to expire the pending connections then it is possible to crash the server.


Thus the objective of SYN flooding is to disable one side of the TCP connection which will result in one or more of the following:

• The server is unable to accept new connections. • The server crashes or becomes inoperative. • Authorization between servers is impaired.

The firewall is expected to protect against SYN Floods.


2.1.7 IP ADDRESS SPOOFING This test attempts to confuse the firewall into allowing traffic to pass from one network segment to another. Each IP packet header contains the source and destination address of the packet. The source address is normally the address that the packet was sent from. By forging the header so it contains a different source address, an attacker can make it appear that the packet was sent by a different (trusted) machine. The machine that receives spoofed packets will send response back to the forged source address.

The firewall is expected to protect against IP Address spoofing.



2.1.8 TCP SPLIT HANDSHAKE SPOOF This test attempts to confuse the Firewall into allowing traffic to pass from one network segment to another. The TCP Split handshake blends features of both the three way handshake and the simultaneous-open connection. The result is a TCP Spoof that allows an attacker to bypass the firewall by having the attacker instruct the target to “initiate” the session back with the attacker. Popular TCP/IP networking stacks respect this handshaking method, including Microsoft, Apple, and Linux stacks, with no modification.1

TCP spoofing attacks have been around for years, and presumed “cured” by modern firewalls. What makes this particular attack different is that it targets the client.

The firewall is expected to protect against TCP Split Handshake spoofing.

Product Result CheckPoint Power-1 11065 PASS Cisco ASA 5585-40 FAIL Fortinet 3950B FAIL Juniper SRX 5800 FAIL Palo Alto PA-4020 FAIL Sonicwall NSA E8500 FAIL

This is a critical failure of five out of six leading firewalls. In order to help our enterprise customers remediate these issues, we contacted the vendors in early February 2011 and provided details of the issues. At the time of printing, April 11, 2011, Juniper, Palo Alto and Sonicwall had provided remediation steps. These are detailed in a separate document available to registered users at no cost at www.nsslabs.com.

Cisco and Fortinet do not currently have remediations available, though we are still actively working with Cisco2 and Fortinet states it is working on validating a fix for a future release. Check back with us or the vendor for more information.

1 The TCP Split Handshake: Practical Effects on Modern Network Equipment, Tod Alien Beardsley & Jin 2 Cisco Bug ID CSCtn29349


3 PERFORMANCE NSS Labs collected extensive performance metrics during this test, according to our established methodology. The volumes of data produced by these tests are designed to capture maximum capacities or “the edge of performance” that may be obtainable for a given metric. In addition, our real-world traffic mix testing methods enable us to more accurately estimate the performance users can expect in their environments. Due to space considerations and the number of different products, we have summarized some of the most important figures that a network administrator should consider when sizing a deployment.

3.1.1 RATED THROUGHPUT NSS Labs rates product performance based upon the average of three traffic types: 21KB HTTP response traffic, a mix of perimeter traffic common in enterprises, and a mix of internal “core” traffic common in enterprises. Details of these traffic mixes are available in the Firewall Test Methodology.

FIGURE 1: RATED THROUGHPUT (MBPS)

3.1.2 CONNECTION DYNAMICS Beyond overall throughput of the device, connection dynamics can play an important role in sizing a security device that will not unduly impede the performance of a system or an application. Maximum Connection and transaction rates help size a device more accurately than simply looking at throughput. By knowing the maximum connections per second, it possible to predict maximum throughput based upon the traffic mix in a given enterprise environment. For example, if the device maximum HTTP CPS is 2,000, and average traffic size is 44KB such that 2,500 CPS = 1Gbps, then the tested device will achieve a maximum of 800 Mbps ((2,000/2,500) x 1,000 Mbps)) = 800 Mbps. Following is a subset of figures from our performance tests.

The aim of these tests is to stress the detection engine and determine how the sensor copes with large numbers of TCP connections per second, application layer transactions per second, and concurrent open connections. All packets contain valid payload and address data and these tests provide an excellent representation of a live network at various connection/transaction rates.

42,000

12,033

5,207

4,567

4,763

1,527

- 5,000 10,000 15,000 20,000 25,000 30,000 35,000 40,000 45,000

Juniper SRX 5800

Cisco ASA 5585-40

Palo Alto PA-4020

CheckPoint Power-1 11065

Fortinet 3950B

Sonicwall E8500


Note that in all tests, the following critical “breaking points”—where the final measurements are taken—are used:

Excessive concurrent TCP connections - latency within the firewall is causing unacceptable increase in open connections on the server-side.

Excessive response time for HTTP transactions/SMTP sessions - latency within the firewall is causing excessive delays and increased response time to the client.

Unsuccessful HTTP transactions sessions – normally, there should be zero unsuccessful transactions. Once these appear, it is an indication that excessive latency within the firewall is causing connections to time out.

Maximum Capacity

Product

Theoretical Max. Concurrent TCP

Connections w/Data

Maximum TCP Connections Per

Second

Maximum HTTP Transactions Per

Second CheckPoint Power-1 11065 762,500 25,900 99,940

Cisco ASA 5585-40 1,980,000 281,950 474,570

Fortinet 3950B 1,447,500 31,200 99,440

Juniper SRX 5800 14,000,000 290,000 440,000

Palo Alto PA-4020 523,000 22,400 158,000

Sonicwall E8500 2,500,000 36,000 102,000

The following chart depicts the relationship between the maximum concurrent connections and the number of HTTP transactions per second that can be transmitted and received through the device.

FIGURE 2: MAXIMUM CONCURRENT CONNECTIONS VS. HTTP TRANSACTIONS & TCP CONNECTIONS PER SECOND

CheckPoint Power-1

Cisco ASA 5585-40

Fortinet 3950B

Juniper SRX 5800

Palo Alto PA-4020

Sonicwall E8500

-

2,000,000

4,000,000

6,000,000

8,000,000

10,000,000

12,000,000

14,000,000

16,000,000

- 50,000 100,000 150,000 200,000 250,000 300,000 350,000 400,000 450,000 500,000


3.1.3 MAXIMUM HTTP CAPACITY These tests aim to stress the HTTP detection engine in order to determine how the sensor copes with detecting and blocking exploits under network loads of varying average packet size and varying connections per second. By creating genuine session-based traffic with varying session lengths, the sensor is forced to track valid TCP sessions, thus ensuring a higher workload than for simple packet-based background traffic.

FIGURE 3: MAXIMUM HTTP CONNECTIONS PER SECOND WITH VARIOUS SIZE PAYLOADS

Each transaction consists of a single HTTP GET request and there are no transaction delays (i.e. the web server responds immediately to all requests). All packets contain valid payload (a mix of binary and ASCII objects) and address data. This test provides an excellent representation of a live network (albeit one biased towards HTTP traffic) at various network loads.

FIGURE 4: MAXIMUM THROUGHPUT (MBPS) WITH VARIOUS SIZE PAYLOADS

44KB 21KB 10KB 4.5KB 1.7KB Juniper SRX 5800 165,000 320,000 357,000 370,000 390,000

Cisco ASA 5585-40 24,130 43,350 69,720 72,330 75,000

Fortinet 3950B 21,720 23,450 24,900 25,500 26,100

Palo Alto PA-4020 18,400 18,600 21,000 25,500 33,000

CheckPoint Power-1 11065 13,000 17,500 18,900 20,940 21,810

Sonicwall E8500 2,400 4,400 6,400 8,400 8,600

- 50,000

100,000 150,000 200,000 250,000 300,000 350,000 400,000 450,000

HTT

P C

PS

44KB 21KB 10KB 4.5KB 1.7KB Juniper SRX 5800 66,000 64,000 35,700 18,500 9,750

Cisco ASA 5585-40 9,652 8,670 6,972 3,617 1,875

Fortinet 3950B 8,688 4,690 2,490 1,275 653

Palo Alto PA-4020 7,360 3,720 2,100 1,275 825

CheckPoint Power-1 11065 5,200 3,500 1,890 1,047 545

Sonicwall E8500 960 880 640 420 215

- 10,000 20,000 30,000 40,000 50,000 60,000 70,000

Mbp

s


3.1.4 RAW PACKET PROCESSING PERFORMANCE (UDP TRAFFIC) The aim of this test is purely to determine the raw packet processing capability of each in-line port pair of the device. It is not real world, and can be misleading. It is included here primarily for legacy purposes.

This traffic does not attempt to simulate any form of “real-world” network condition. No TCP sessions are created during this test, and there is very little for the detection engine to do in the way of protocol analysis (although each vendor will be required to write a signature to detect the test packets to ensure that they are being passed through the detection engine and not “fast-tracked” from the inbound to outbound port).

FIGURE 5: UDP THROUGHPUT (MBPS)

128 256 512 1024 1514 Juniper SRX 5800 13,000 39,000 76,000 140,000 160,000

Fortinet 3950B 8,300 12,050 18,110 20,000 20,000

Palo Alto PA-4020 7,300 8,000 8,000 8,000 8,000

Cisco ASA 5585-40 7,280 11,470 15,330 15,530 15,930

CheckPoint Power-1 11065 1,940 3,650 6,925 11,425 12,140

Sonicwall E8500 770 1,400 2,840 5,500 6,000

-

20,000

40,000

60,000

80,000

100,000

120,000

140,000

160,000

180,000

Mbp

s


4 STABILITY & RELIABILITY

Long term stability is particularly important for an in-line device, where failure can produce network outages. These tests verify the stability of the DUT along with its ability to maintain security effectiveness while under normal load and while passing malicious traffic. Products that are not able to sustain legitimate traffic (or crash) while under hostile attack will not pass.

The DUT is required to remain operational and stable throughout these tests, and to block 100 per cent of previously blocked traffic, raising an alert for each. If any non-allowed traffic passes successfully - caused by either the volume of traffic or the DUT failing to bypass for any reason - this will result in a FAIL.

4.1.1 BLOCKING UNDER EXTENDED ATTACK The DUT is exposed to a constant stream of security policy violations over an extended period of time. The device is configured to block and alert, and thus this test provides an indication the effectiveness of both the blocking and alert handling mechanisms.

A continuous stream of security policy violations mixed with legitimate traffic is transmitted through the device at a maximum of 100Mbps (max 50,000 packets per second, average packet sizes in the range of 120-350 bytes) for 8 hours with no additional background traffic. This is not intended as a stress test in terms of traffic load (covered in the previous section) - merely a reliability test in terms of consistency of blocking performance.

The device is expected to remain operational and stable throughout this test, and to block 100 per cent of recognizable violations, raising an alert for each. If any recognizable policy violations are passed - caused by either the volume of traffic or the sensor failing open for any reason - this will result in a FAIL.

Product Result

CheckPoint Power-1 11065 PASS

Cisco ASA 5585-40 PASS

Fortinet 3950B PASS

Juniper SRX 5800 PASS

Palo Alto PA-4020 PASS

Sonicwall NSA E8500 PASS

4.1.2 PASSING LEGITIMATE TRAFFIC UNDER EXTENDED ATTACK This test is identical to 4.1.1, where the external interface of the device is exposed to a constant stream of attacks over an extended period of time.

The device is expected to remain operational and stable throughout this test, and to pass most/all of the legitimate traffic. If an excessive amount of legitimate traffic is blocked throughout this test - caused by either the volume of traffic or the DUT failing for any reason - this will result in a FAIL.


Product Result



Fortinet 3950B PASS

Juniper SRX 5800 PASS


Sonicwall NSA E8500 PASS

4.1.3 PROTOCOL FUZZING & MUTATION This test stresses the protocol stacks of the DUT by exposing it to traffic from various protocol randomizer and mutation tools. Several of the tools in this category are based on the ISIC test suite and the BreakingPoint Stack Scrambler component.

Traffic load is a maximum of 350Mbps and 60,000 packets per second (average packet size is 690 bytes). Results are presented as a PASS/FAIL - the device is expected to remain operational and capable of detecting and blocking attacks throughout the test.

Product Result



Fortinet 3950B FAIL

Juniper SRX 5800 FAIL


Sonicwall NSA E8500 FAIL

It is important to note that when a service or device to crashes it is most often due to a software vulnerability. And while not all vulnerabilities can be exploited, most can. Therefore we urge vendors whose devices failed this test to fix their devices at the earliest opportunity.

• NSS Labs considers a product to have failed this test if it becomes unstable and “falls over” – crashing and not allowing any traffic to flow. In effect, the firewall becomes a doorstop and must be power-cycled to recover.

• NSS Labs considers a “severe fail” if upon failure, all traffic is allowed to to pass through the firewall, or if the firewall itself is breached upon failure.

In the case of Fortinet, upon failure the device rebooted and permitted an attacker unauthenticated “root” access via ssh. Fortinet was notified of this failure condition of build 279 and promptly instructed us to upgrade to build 303, after which, failure did not permit unauthenticated root access. NSS Labs recommends that Fortinet customers running FortiOS 4.0 MR2 Patch 1, build 279 should consider updating at the earliest opportunity.


If Fortinet had not issued the patch, the device would have automatically garnered a “caution” rating. However, since Fortinet responded promptly and corrected the issue, we are feel a “neutral” rating is still appropriate. Further, NSS Labs has determined the most responsible action is to issue a “severe fail” rating in order to notify Fortinet customers with devices deployed in the field who may be running the unpatched version.


5 TOTAL COST OF OWNERSHIP & VALUE Firewall implementations can be complex projects with several factors affecting the overall cost of deployment, maintenance and upkeep. All of these should be considered over the course of the useful life of the solution.

1. Fees paid to the vendor for the initial product and yearly maintenance 2. Labor costs for installation, upkeep and tuning

No two network security products deliver the same security effectiveness or throughput, making ‘apples to apples’ comparisons extremely difficult. In order to capture the relative value of devices on the market and facilitate such comparisons, NSS Labs has developed a unique metric to enable value-based comparisons:

Price per protected megabit/sec = Cost / (security effectiveness * throughput).

Developed in 2009 by NSS Labs, this metric is used extensively in the sections below to evaluate cost of security, throughput and 3-year TCO. The figures here are based on list prices provided by vendors. NSS Labs clients can gain access to our TCO spreadsheets in order to customize comparisons based on special-offer pricing.

5.1.1 LABOR PER PRODUCT (IN HOURS) Firewall solutions are among the most complex products in information security discipline. With the shortage of skilled and experienced practitioners, it is important to consider the required time and resources to properly install, maintain and manage the solution. Failure to do so could result in products not achieving their full security potential.

This table estimates the annual labor required to maintain each device. There are three main components to be considered:

1. Installation – the time required to take the device out of the box, configure it, put it into the network, apply updates and patches, initial tuning, and set up desired logging and reporting.

2. Upkeep – the time required to apply periodic updates and patches from vendors, including hardware, software, and protection (signature/filter/rules) updates.

This table estimates the annual labor required to maintain each device. Since vendors sent their very best engineers to tune, NSS Labs’ assumptions are based upon the time required by a highly experienced security engineer ($75 per hour fully loaded). This allowed us to hold the talent cost variable constant and measure only the difference in time required to tune.

Product Installation (Hrs) Upkeep / Year (Hrs)

CheckPoint Power-1 11065 8 25

Cisco ASA 5585-40 8 25

Fortinet 3950B 8 25

Juniper SRX 5800 16 50

Palo Alto PA-4020 8 25

Sonicwall NSA E8500 8 25


5.1.2 PURCHASE PRICE AND TOTAL COST OF OWNERSHIP TCO incorporates the labor costs for each product over three years as described above with the fees paid to the vendor for purchase and maintenance licensing. Calculations are made as follows:

Value Description of Calculation

1 Year TCO Initial Purchase Price + Maintenance + (Installation + Upkeep) * Labor rate ($/hr)

2 Year TCO 1 Year TCO + (Upkeep hours) * Labor rate ($/hr)

3 Year TCO 2 Year TCO + (Upkeep hours) * Labor rate ($/hr)

Each vendor provided pricing information. When possible, we selected the 24/7 maintenance and support option with 24-hour replacement as this is the option most organizations will select.

Product Purchase Maintenance / year 1 Year TCO 2 Year TCO 3 Year TCO

CheckPoint Power-1 11065 $60,000 $12,000 $74,475 $88,350 $102,225

Cisco ASA 5585-40 $139,995 $23,519 $165,989 $191,383 $216,777

Fortinet 3950B $130,495 $10,500 $143,470 $155,845 $168,220

Juniper SRX 5800 $1,236,320 $67,930 $1,309,200 $1,380,880 $1,452,560

Palo Alto PA-4020 $35,000 $4,200 $41,675 $47,750 $53,825

Sonicwall NSA E8500 $39,995 $4,275 $46,745 $52,894 $59,044

5.1.3 THREE-YEAR TOTAL COST OF OWNERSHIP PER PROTECTED MBPS The following table illustrates the relative cost per unit of work performed. This is helpful in understanding value (Mbps of protected traffic). The least expensive product will not be the best value if it does not block attacks nor provide sufficiently high throughput.

Product Throughput 3 Year TCO Price / Mbps-Protected

Palo Alto PA-4020 5,207 $53,825 $10

Cisco ASA 5585-40 12,033 $216,777 $18

CheckPoint Power-1 11065 4,567 $102,225 $22

Juniper SRX 5800 42,000 $1,452,560 $35

Fortinet 3950B 4,763 $168,220 $35

Sonicwall NSA E8500 1,527 $59,044 $39

FIGURE 6 - 3 YEAR TCO


6 PRODUCT GUIDANCE

NSS Labs issues summary product guidance based on evaluation criteria that is important to information security professionals. The evaluation criteria are weighted as follows:

1. Security effectiveness - The primary reason for buying a firewall is to separate internal trusted networks from external untrusted networks while allowing select controlled traffic to flow between trusted and untrusted.

2. Resistance to Evasion- Failure in any evasion class permits attackers to circumvent protection.

3. Stability - Long term stability is particularly important for an in-line device, where failure can produce network outages

4. Performance – Correctly sizing a firewall is essential 5. Value – Customers should seek low TCO and high effectiveness and performance rankings.

Product Guidance Summary

NSS Labs’ recommendations are based solely on empirical test data, validated over multiple iterations. Failure to resist all evasion attempts prevents a firewall from achieving Recommended status. Consult the detailed product guidance section for more information about each product’s rating.

Products are listed alphabetically within their guidance rating groups.

Rating Products

Recommend Check Point Power-1 11065

Neutral

Cisco ASA 5585-40 Fortinet Fortigate 3950B Juniper SRX 5800 Palo Alto Networks PA-4020 Sonicwall NSA E8500

Caution

Evasion Techniques: A firewall’s effectiveness is significantly handicapped if its policies can be circumvented using obfuscation or evasion techniques, and our product guidance is adjusted to reflect this. Only one product passed our evasion testing.

Security Effectiveness & Cost: Buyers should not only consider the initial purchase price of a given product, but also the total cost of ownership and relative value of the product.


6.1 RECOMMEND A Recommend rating from NSS Labs indicates that a product has performed well and deserves strong consideration. Only the top technical products earn a Recommend rating from NSS Labs—regardless of market share, company size, or brand recognition. Full evasion resistance required.

6.1.1 CHECKPOINT POWER-1 11065 CheckPoint is one of the best known names in the firewall space, having pioneered stateful inspection technology back in the 90’s. CheckPoint’s acquisition of Nokia’s Security Appliance group in April 2009 has enabled the company to produce a much more cohesive and simplified product. The new Power-1 11000 series platforms, combined with the newly released R75 is a robust and stable firewall.

Product Max UDP Throughput NSS Labs Rated Throughput Power-1 11065 12,140 Mbps 5,207 Mbps

CheckPoint Power-1 11065 was one of only three products that was able to withstand our stability test and remain functional. In addition, it was the only product that properly handled the TCP Split Handshake attack.

6.2 NEUTRAL A Neutral rating from NSS Labs indicates that a product has performed reasonably well and should continue to be used if it is the incumbent within an organization. Products that earn a Neutral rating from NSS Labs deserve consideration during the purchasing process.

6.2.1 CISCO ASA 5585-40 Cisco is one of the marketshare leaders in the enterprise firewall space. The ASA appliances are Cisco’s intended replacement for the popular PIX firewalls. NSS Labs rates the ASA at 12,033 Mbps.

Product Max UDP Throughput NSS Labs Rated Throughput ASA 5585-40 15,930 Mbps 12,033 Mbps

Cisco ASA was one of only three products that was able to withstand our stability test and remain functional. However, it failed to properly handle the TCP Split Handshake attack. This is a serious weakness which leaves customers unprotected and should be remedied at the earliest opportunity. See NSS Labs remediation guide for additional information.

6.2.2 FORTINET FORTIGATE 3950B Fortinet is a major player in the UTM space which for the past few years has been making a push into the Enterprise in the firewall market. NSS Labs rates the Fortigate 3950B at 4,763 Mbps.

Product Max UDP Throughput NSS Labs Rated Throughput FortiGate 3950B 20,000 Mbps 4,763 Mbps

The Fortigate 3950B failed our stability test; upon failure the device rebooted and permitted an attacker unauthenticated “root” access via ssh. Fortinet was notified of this failure condition of build 279 and promptly instructed us to upgrade to (build 303). Once more we subjected the (patched) Fortigate 3950B to our stability test. The device crashed hard – essentially locking up to the point


that it was unresponsive even via console – requiring a manual power cycle. However, this time it did not permit unauthenticated remote login.

NSS Labs recommends that Fortinet customers running FortiOS 4.0 MR2 Patch 1, Build 279 should consider updating at the earliest opportunity. Further, the Fortigate failed to properly handle the TCP Split Handshake attack. This is a serious weakness which leaves customers unprotected. According to a Fortinet representative, there is currently no fix, but one will be included in the next release which is scheduled for the third week in May. See NSS Labs remediation guide for additional information.

6.2.3 JUNIPER SRX 5800 Juniper is one of the marketshare leaders in the firewall space, having acquired NetScreen in 2004. The Juniper SRX platform is a relatively new multi-function gateway and is a sister to the stand-alone firewall products from Juniper.

Product Max UDP Throughput NSS Labs Rated Throughput SRX 5800 160,000 Mbps 42,000 Mbps

The SRX-5800 has been rated by Juniper as passing a whopping 140,000 Mbps of UDP traffic. We found that to be a slightly conservative number based upon NSS Labs rating of 160,000 Mbps using 1514 byte packets. However, overall we rated the SRX 5800 as a 42,000 Mbps firewall based on real-world traffic – still very impressive, yet roughly 25% of the 140,000 Mbps rating.

The SRX failed our stability test; it crashed denying all traffic requiring a reboot of the blade. However, we were able to do so without requiring a full manual power cycle of the entire chassis by logging into the management blade and rebooting the service blade. Further, the SRX 5800 failed to properly handle the TCP Split Handshake attack. This is a serious weakness which leaves customers unprotected and should be remedied at the earliest opportunity. See NSS Labs remediation guide for additional information.

6.2.4 PALO ALTO NETWORK PA-4020 Palo Alto Networks is a new entrant in the firewall category, focusing primarily on the Next-Gen firewall market. The PA-4020 has been rated by Palo Alto as a 2 Gbps firewall, a very conservative number based upon NSS Labs rating of 5,207 Mbps.

Product Max UDP Throughput NSS Labs Rated Throughput PA-4020 8,000 Mbps 5,207 Mbps

The PA-4030 was one of only three products that was able to withstand our stability test and remain functional. However, it failed to properly handle TCP Split Handshake attack. This is a serious weakness which leaves customers unprotected and should be remedied at the earliest opportunity. See NSS Labs remediation guide for additional information.

6.2.5 SONICWALL E8500 Sonicwall is a challenger in the enterprise firewall category, having entered the market at the low end. The NSA E8500 has been rated by Sonicwall as as a 2.2 Gbps firewall. However, NSS Labs rates the E8500 at 1,527 Mbps.

Product Max UDP Throughput NSS Labs Rated Throughput NSA E8500 6,000 Mbps 1,527 Mbps


While the NSA E8500 comes with 8 Gigabit Ethernet ports, the device does not come with a dedicated management port. Therefore one of the eight ports must be used for management – rendering the device with 7 usable ports (assuming out-of-band management).

The Sonicwall NSA E8500 failed our stability test; it crashed requiring a full manual power cycle to recover. Further, the NSA E8500 failed to properly handle the TCP Split Handshake attack. This is a serious weakness which leaves customers unprotected and should be remedied at the earliest opportunity. See NSS Labs remediation guide for additional information.

6.3 CAUTION A Caution rating from NSS Labs indicates that a product has performed poorly. Organizations using one of these products should review their security posture and other threat mitigation factors, including possible alternative configurations and replacement. Products that earn a Caution rating from NSS Labs should not be short-listed or renewed.

6.3.1 OTHER FIREWALL VENDORS Our testing continues to show wide variations in security effectiveness and product reliability. Thus, buyers should view untested products with a fair amount of caution until they can be evaluated.


7 TEST METHODOLOGY ELEMENTS OVERVIEW

The following table lists the individual tests NSS Labs performed on each of the products. Direct references are provided to NSS Labs Test IDs from Sections 3 through 6 of NSS Labs’ Firewall Test Methodology v3.0.

Test ID Description Result 3 Security Effectiveness 3.1 Firewall Policy Enforcement 3.1.1 Baseline Policy

3.1.2 Simple Policy 3.1.3 Complex Policy 3.1.4 Static NAT 3.1.5 Dynamic / Hide NAT 3.1.6 Syn Flood Protection 3.1.7 Address Spoofing Protection 3.1.8 Session Hijacking Protection 3.1.9 TCP Split Handshake 4 Performance

4.1 Raw Packet Processing Performance (UDP Traffic) 4.1.1 128 Byte Packets

4.1.2 256 Byte Packets 4.1.3 512 Byte Packets 4.1.4 1024 Byte Packets 4.1.5 1514 Byte Packets 4.2 Latency - UDP

4.2.1 128 Byte Packets 4.2.2 256 Byte Packets 4.2.3 512 Byte Packets 4.2.4 1024 Byte Packets 4.2.5 1514 Byte Packets 4.3 Maximum Capacity

4.3.1 Theoretical Max. Concurrent TCP Connections 4.3.2 Theoretical Max. Concurrent TCP Connections w/Data 4.3.3 Stateful Protection at Max Concurrent Connections 4.3.4 Maximum TCP Connections Per Second 4.3.5 Maximum HTTP Connections Per Second 4.3.6 Maximum HTTP Transactions Per Second 4.4 HTTP Capacity With No Transaction Delays

4.4.1 2.500 Connections Per Second – 44Kbyte Response 4.4.2 5,000 Connections Per Second – 21Kbyte Response 4.4.3 10,000 Connections Per Second – 10Kbyte Response 4.4.4 20,000 Connections Per Second – 4.5Kbyte Response 4.4.5 40,000 Connections Per Second – 1.7Kbyte Response 4.5 “Real World” Traffic

4.5.1 “Real World” Protocol Mix (Perimeter) 4.5.2 “Real World” Protocol Mix (Core) 5 Stability & Reliability

5.1 Blocking Under Extended Attack 5.2 Passing Legitimate Traffic Under Extended Attack 5.3 Protocol Fuzzing & Mutation 6 Total Cost of Ownership & Value 6.1 Ease of Use 6.1.1 Initial Setup (Hours)


Test ID Description Result 6.1.2 Time Required for Upkeep (Hours per Year)

6.2 Expected Costs 6.2.1 Initial Purchase

6.2.2 Ongoing Maintenance & Support (Annual) 6.2.3 Installation Labor Cost (@$75/hr) 6.2.4 Management Labor Cost (per Year @$75/hr) 6.3 Total Cost of Ownership

6.3.1 Year 1 6.3.2 Year 2 6.3.3 Year 3 6.3.4 3 Year Total Cost of Ownership


APPENDIX A: SPECIAL THANKS Special thanks go to our test infrastructure partners who provide much of the equipment, software, and support that make this testing possible:

Nss Labs Firewall Group Test q1 2011 v9

Documents

leaves customers unprotected

nss labs recommends

nss labs considers

half open connections

common criteria certified

nss labs rates

http transactions

contact nss labs