Network Equipment Testing White Paper

White Paper on

Best Practices in Network Equipment Testing

© 2007, Jopasana Software & Systems Ltd.

Authors Atul Nene

Aniket Kavathekar Graphics Gajanan Kamthe

Executive Summary So much for the right piece of information at right time! The information hungry systems of today are putting more and more stress on the network bandwidth which is already shared between the network members. Be it a wired network or a wireless. Be it the high end Blackberry you use for email, the jukebox you use to download your favorite online music or the network printer in use for the office, each and every member dents the bandwidth and affects the network performance. The viruses and worms entering the networks in increasingly innovative methods do no good to help the performance either. To maintain the network performance above the acceptable threshold, the network elements participating in the complex enterprise networks should be made more robust, and as far as possible, more intelligent. This is especially true for routers and switches and all such types of ‘connecting’ devices that form the network’s glue. The manufacturers of such devices can test the network devices only to an extent permissible given the constraints of limited in-house resources. Thus the ball finally lands in the court of a specialized network equipment testing service provider to ensure and verify the functional efficiency of network equipment. The role of network elements like routers and switches becomes eminent in maintaining the optimum bandwidth and securing the information being exchanged. It becomes essential for a smart tester to test the network element in various environments which are similar to the end user network. Not only this, the network element developers also need to be made aware of the ‘live scenario’ at customers premises, where the element needs to be ‘intelligent’ enough to facilitate normal working irrespective of the peculiarities caused by the network on case to case basis. This white paper discusses the best practices that can be deployed by network element manufacturers and testers to ensure impeccable quality and make the products ‘real-world' ready. Testing practices, as well as planning and documentation practices are addressed. Given that existing network elements will be upgraded and newer elements will be developed by manufactures on a continuous basis, these best practices are highly relevant. Jopasana Software & Systems Ltd. 17, Electronic Co-operative Industrial Estate, Pune-Satara Road, Pune 411 009, Maharashtra, India. Tel nos.: +91(20)2422 8112, +91(20)2422 7879 http://www.jopasana.com [email protected]

© 2007,Jopasana Software & Systems Ltd.

Table of Contents

1. Introduction .........................................................................................................................2 2. Inside an Enterprise Network ..............................................................................................3 3. Constraints ..........................................................................................................................4 4. Best Test Practices .............................................................................................................5 5. Best Planning and Documentation Practices ......................................................................6 6. Benefits of Best Practices ...................................................................................................9 7. Conclusion ........................................................................................................................10

1. Introduction Picture this. A ship testing service which tells a frigate manufacturer about the elasticity in ship size required when the ship is supposed to enter the Bay of Bengal, the Sir Creek, the Panama, the Suez and the Thames. That sort of testing will suggest improvements in design in turn so that Arabs, Asians, South Americans would be as ‘equitably happy’ as European and Australian customers. Any design is a waste if the product is not sensitive to customer’s usage preferences. This rule applies to every tangible product. From ship building to high-end products in networking and communications, a round nut definitely can’t fix a square hole!

Enterprise Networks ‘Light bulb testing’ mindset is inadequate in software testing Exhaustive software or hardware testing can bring in the ‘ease of use’ inputs to the product developers if the testing is not restrained by ‘light bulb testing mentality’1. Given the complexity of enterprise networks (see SIDEBAR: Enterprise Networks), if the test engineer with the same light bulb testing’ mindset is given a network element such as a router or a switch that runs specialized software, the product may work in his lab given the idealized or clinical conditions, but may not at customer’s premise. In the networking domain a product manufacturer has to see whether his product is ‘sensitive and tunable’ enough in each ‘network environment’.

SI

DEB

AR

Enterprise networks today are highly complex and are made up of every conceivable piece of hardware from Internet Telephony phones to handheld devices. They comprise computers running varied kinds of operating systems, enterprise software and end-user software. The network elements that interconnect these devices are themselves so complex that one can get very fuzzy about where the software takes over from the hardware. And not to forget, they are made up of a combination of wired and wireless technologies.

Network-Equipment-Testing-Whitepaper.Doc 2 of 10

1 Imagine you are purchasing a light bulb from an electrician and you want to test it. A ‘typical’ shop owner will remove the bulb from its packing, connect it to socket, show you the light and say ‘I have tested it and it works!’ This is what we call ‘light bulb testing mentality’ where everything from the socket width, hole-clearance, voltage limit and maximum allowed current is standardized and what a test engineer has to do is to show that the circuit gets completed through the bulb and it works.


2. Inside an Enterprise Network While a product developer may guarantee a performance assurance in test-labs, real computer networks are likely to spring unpleasant surprises. A neophyte router or a switch is most likely to get overwhelmed by the hubbub that goes on inside an enterprise network environment. The factors are too many for listing purposes and they keep piling up. Let’s tour around the significant ones. Harmless residents Take an example of an organization of two hundred employees for ease of consideration. Let’s say the company has established – as is the norm in any IT enabled organization striving to become a globally connected paperless office – an intranet facility which takes care of all the HR, Administration, Visa procedures and all the back-end procedures one may imagine to run an organization. A further drill-down brings in an array of email clients, file servers, print sharing systems and test machines.

Intelligent Network Elements To the next level of this come hundreds of folders across all the two hundred users containing thousands of files of variety of types like word documents, jpgs, bmps, pdfs, flash files and so on. Let’s not also forget that these individual users are likely to use different communication modes like Skype talk, Google talk, Internet Messengers (IMs) like Yahoo, MSN and GAIM. These users are also likely to be participating in business as well as social networks like Ryze, LinkedIN and Orkut to name a few, that will bring in more traffic for the switch-like intelligent network element (see SIDEBAR: Intelligent Network Elements) to handle. All of the network components and residents mentioned above have a different expectation from switch in terms of speed of data transfer and identification of address. The applications like intranet score the heaviest as they are used both for storage as well as data transfer at different times.

SID

EBA

R

This term is still emerging and hence a dictionary definition as such is not yet available. However the collective consciousness across the networking and telecom domains loosely identifies an Intelligent Network Element as a multifunction device with embedded software that is a part of the enterprise network and is capable of connecting to and interconnecting rest of the participants of the network, be they voice and data capable telephone boxes or sophisticated servers running internet businesses. The generic functions of a network gateway or a router or a switch both unmanaged and manageable are associated with these super intelligent devices and they can work with other devices using various protocols (TCP-IP and SS7), technologies (Gigabit Ethernet and Radio Frequency) and standards.



Network miscreants Networks would be incomplete if miscreants are missing. They are of all forms, shapes, sizes and nature. A traveling data packet as well as the whole file system is susceptible to all sorts of held-ups, frisking, adultery and destruction. The routers in these cases need to do a policing act after identifying the severity and level of each. A virus may drop into the network by breaking into one email account, enter the network of the user by hiding behind downloaded files and show the real colors once the time is right to do the damage. This can be an active virus or a dormant virus programmed to be activated with date. Similarly there can be Trojans, malware programs and spy ware. Then, there are several hues of worms, adware and BOTS entering the network camouflaged with the identifiable and ‘well-known’ signatures. A router has to be ‘well-informed’ about all sorts of security threats these miscreants pose for well functioning of the network. It then also needs to stop them from entering the network.

Timelessly Timely “Time is an illusion, lunch time doubly so”, writes Douglas Adams, in his hugely popular title, The Hitchhikers Guide to the Galaxy. Researchers struggling with predictive techniques for prevention of network intrusions and anomalies have found a new facet of the word “timeless”. And guess what, giving plausible credibility to the time-space continuum theory is the fact that network miscreants can appear not just anytime, but also anywhere in the system. Specialist network administrators who think that they have locked down and secured every single piece of network equipment just need to wait till a representative from a customer of their organization pays a visit and wants to plug in his laptop to the network!

3. Constraints Network Constraints Considering the network components and their character, each network is unique in terms needs of data transfer, vulnerability to security threat, alarm threshold limits, the maximum data rate that is apt for a specific kind of traffic and the volume of data being transferred. A router, while manning the traffic and disallowing unwanted packets, needs to be sensitive to these constraints and needs to learn on the job. For example, in networks designed to carry VOIP traffic on higher priority than other data traffic, the network elements should enforce the bandwidth limit assigned to other data traffic, so as to keep rest of the unutilized bandwidth available to possible incoming voice calls. Same is true for thin client environments like banks, where operational effectiveness is determined by the terminals having a guaranteed minimum bandwidth dedicated to them. Cost Constraints The network constraints mentioned above need to be known before attempting to fine tune network equipment so as to make it robust. The only way to know these network constraints are by introducing the newly designed router or switch to a real life computer network. In ideal scenario, the equipment producer will have to create each and every type of network environment to test the universality of his product. This is not a practical solution from the investment point of view. It is not financially viable to invest in elaborate network setups just for the sake of testing a network product. That’s where a testing service provider comes to the rescue. Such a provider adds value by offering its customizable test setup to multiple product vendors, thereby reducing costs for each vendor.



Tuning a network equipment vis-à-vis the environment A smart test engineer can complement a product designer by • Testing the equipment for the performance parameters • Setting up the security alarm threshold for that network • Contributing to the intelligence of router by updating ‘run time environment’ information This ‘intelligence’ primarily consists of the knowledge of assigning appropriate values to various configuration parameters, based on the kind of network.

Tunable Parameters While testing a switch or a router in a ‘close to live’ environment, the test engineer can certainly identify specific functional as well as security related parameters that, when tuned, affect the elements performance, in turn affecting the network traffic. These parameters can be generally called as tunable parameters for enhancing the router performance. The features and functionality that they affect are listed below. Hardware Security Management Anomaly Detection Interface Ports Maximum allowed throughput Encrypted throughput Number of secured users Additional ports

MAC spoofing and flooding ARP spoofing and poisoning VLAN hopping and double tagging DHCP address exhaustion Switch impersonation Spanning Tree attacks

Configuration Policy Monitoring Reporting Event correlation

Zero-day worm protection Traffic Anomalies Protocol Anomalies Behavior Anomalies

4. Best Test Practices We recommend a set of best test practices that, will help network equipment manufacturers produce robust products. The first two practices are self explanatory and are generally carried out by manufacturers as part of their fundamental equipment testing.

Test for expected functionality and performance Test for unexpected functionality and extreme conditions

These best practices sound benign, and elaborate test planning and execution mechanisms are used to cover this aspect of testing. Nonetheless, bugs are observed in networking products that are out in the market and hence paying attention to these two practices is called for. These practices are included here for completeness.

Test with all possible kinds of applications that could utilize the network This best practice is self explanatory too, though often, it is either missed in entirety or not paid enough attention to. Most of the times, what is missed is that a truly comprehensive list of applications is not identified for testing. We think that infrastructure costs and dearth of acutely skilled test engineers cause this.



The following practices – from our experience – contribute significantly to the product quality and make the network elements truly enterprise ready.

False Positives Identify volatile parameters and tune the equipment-under-test for every application. This best practice needs a much closer look. Lets take an example. In a normally loaded network, 100+ simultaneous connections from a desktop to an internet site would indicate a malicious virus or worm, but there may be a specialized application that accesses the internet simultaneously via so many connections for its legitimate business functionality. Let’s take an example of an email sent from a Gmail account which contains hundred images as attachment. As a user downloads all the images, it would send a flurry of images appearing one after another on your desktop. The specialized virus detection program embedded into the specialized controlling network element, will take this event as an indication of a malicious virus or a worm. Thus a legitimate use would be mistaken for a virus entry. For such an application, the equipment-under-test has to be tuned to recognize that applications’ network traffic pattern and not raise an alarm, else it would not be deemed as behaving correctly.

Test for false positives This best practice is the kind of challenge that gets the toughest test engineers going (see SIDEBAR: False Positives). A packet level understanding of an application’s network traffic alone will enable an engineer to ensure that anomalies detected by the equipment under test are not false positives.

SID

EBA

R

In programs used to filter spam, a false positive is a legitimate message mistakenly marked as spam. Messages that are determined to be spam may be rejected by a server or client-side spam filter program and returned to the sender as bounce e-mail. One problem with many filtering tools is that if they are configured stringently enough to be effective, there's a fairly high chance of getting false positives. The risk of accidentally blocking an important message has been enough to deter many companies from implementing any anti-spam measures at all. In a different context, a false positive, also known as a false detection or false alarm, occurs when an antivirus program detects a known virus string in an uninfected file. The file, while not infected with an actual virus, does contain a string of characters that matches a string from an actual virus.A false positive can also occur when a program performs an action, which appears to the antivirus program to be a virus-like activity.

5. Best Planning and Documentation Practices We recommend a few best practices for documentation and reporting of testing carried out under various network scenarios for the equipment under test. We present these in the form of tear-offs or snippets from some actual reports documented in the course of live projects in our test labs. Typically, for a particular piece of network equipment – which is invariably a combination of hardware and software components – that is being tested, a set of applications that will form the runtime environment is identified. For each such application a detailed test plan – popularly known as the Test Procedure Document – is prepared. This piece of documentation specifically targets that particular functionality of the application which will generate traffic on the network. After all, that is what is of interest in our case. A plan is also prepared which lists down the sequence and timeframes when the test plans will be written for all application and the projected time span for their execution. This plan is part of the project planning and execution function and is tracked regularly for the purpose of reporting status.



Prepare a Test Strategy Note for each application

TearOff 1 is from a test plan and notes the application being executed and the details of the network configuration during the test, as well as the configuration parameters of the equipment under test. It also lists down target functionality to be executed and has space for noting observations and details pointing to where the packet capture data and screenshot capture data – if any – are located.

Track the execution status of individual applications TearOff 2 is from a planning document and helps to track the status of execution of individual applications.



Keep special track of applications that cause anomalies in the equipment under test

TearOff 3 denotes a summarized track of the applications that caused anomalies to be seen in the equipment under test. It also allows to note whether execution of a particular application is suspended for the current test cycle. That is where the “Build Used” column will come in handy. That column notes the identifying tag of the hardware and software combination of the equipment under test. Our experience shows that testing efforts can be optimized by stopping execution of the application causing problems for this cycle, fixing the problem in the equipment under test, and testing the suspended application again only when a new build with the fix is available. Testing of the rest of the applications should of course continue on the earlier build to get complete coverage.



Track complete test cycles

per build TearOff 4 is a classical piece from an execution management document that records and tracks a complete test cycle. Weekly and cumulative totals for projected vs. real numbers give a good measure of the effectiveness of the planning process and the efficiency of the test execution team.

6. Benefits of Best Practices Meticulous execution of the said best practices, especially in partnership to an experienced provider of test services, will benefit product quality in the following manner. Product Quality Development Process & Costs • Sets the default performance benchmarks • Adds to the network elements intelligence

towards deep threat containment • Gives maximum coverage of the applications • Makes the equipment-under-test most ready

for live deployment • Enhances product tunability • Helps develop security policies for the end-user

organization

• Reduces costs by leveraging infrastructure and people skills of the partner

• Generates ideas for future features of the product

• Contributes towards a short and successful beta test program for the product



7. Conclusion In the networking communications domain, a product developer is always perplexed by the dynamism and diversity of potential markets. An extra effort in terms of these best test practices will certainly bring in the most sought after ‘user friendliness’. That’s where the best stand apart from the pool of very goods! If the best practices mentioned in this document become part of every networking product developers toolbox, it will contribute to excellent product quality and hopefully elicit ‘oohs’ and ‘aahs’ towards customer experience, and thereby enhance end-user comfort. Once that benchmark in quality is reached by industry-wide products, one can plan to participate in the next phase of product intelligence and quality which is ‘self learning and tuning’ by products of their runtime environment. You are welcome to send comments, suggestions and questions to : Atul Nene ([email protected]).

About Jopasana Jopasana is a product engineering and software solutions company operating mainly in the hi-tech "real-time" and embedded systems domain, helping customers with complete product realization services and application engineering services for end-user applications. Jopasana provides state-of-art solutions in the area of networking and telecommunication. The knowledge about various networking protocols and protocol stacks; Internet Engineering Task Force (IETF) RFCs, Industry standards, technologies and network topologies are key factors in executing projects in this area. Jopasana, in Sanskrit, means to "nurture". At Jopasana, since our inception, in 1990, we have always strived to nurture relationships with our customers, our employees, our business partners, stake holders, and the communities that we operate in.


Network Equipment Testing White Paper

Documents

network devices

network performance

network bandwidth

network printer

wired network

testing practices

role of network elements

athe network elements