Performanc e Testing Tool Evaluation for Web Performance Suite 3.599White Paper Page 1 of 12 Performance Testing Tool Evaluation for Web Performance Suite 3.599 B y Arkadyuti Roy (169655) Perform ance Testing C oE PDF created with pdfFactory trial version www.pdffactory.com
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Performance Testing Tool Evaluation for Web Performance Suite 3.599
White Paper Page 3 of 12
Table of Contents
Abstract ................................................................................................................2 1.0 Introduction ................................................................................................4 2.0 Cognizant Evaluation: Criteria used by Cognizant for Evaluation .........4
2.1 Key Factors considered ........................................................................................ 4 2.2 Other factors (to be considered) .......................... ........................... ...................... 5
3.0 Load and Stress Test Capabilities ............................................................5 4.0 Tool Reporting Capability ..........................................................................7 5.0
Performance Testing Tool Evaluation for Web Performance Suite 3.599
White Paper Page 4 of 12
1.0 Introduction
The Web Performance Suite™ is web testing software targeted at web sites thatuse standard technologies such as web forms, J2EE, ASP, .NET/C#, PHP, Ruby,ColdFusion, Java, etc. Its growing list of modules currently can tackle everythingfrom baseline performance analysis to a load test with tens of thousands ofsimulated users.
The evaluation of web performer began with the need to evaluate the tool as aload generator tool in place of the Web Load tool that is currently being used forthe Web Re-launch project. To evaluate the tool a Cognizant benchmark criteriais followed and certain functionalities of the tool (Web Performance Suite 3.5xxDemo Version) have been evaluated and compared with other standard tools,like LoadRunner 7.8 (the most popular Performance testing tool in the market),
Silk Performer 6.0 ( the most powerful performance testing tool in the market)and IBM Rational Robot 7. The LoadRunner, Silk Performer and Rational Robotare already evaluated tools.
The problem with the most commonly used Load Testing Tools (likeLoadRunner, Silk Performer or Rational) is that they are costly. GenerallyLicenses are given on basis of the number of VUsers to be used for the tests, orin a timeline basis. That’s why the search for a cheaper tool like web Performer isencouraged. The license of Web Performance Suite is cheaper and it’s for Lifetime. Scripting is simpler in case of web performer in comparison to the othertools. But the tool has got some limitations while modifying the test cases as
there is no customized scripting as we get in case of LR, Silk or Rational.
2.0 Cognizant Evaluation: Criteria used byCognizant for Evaluation
While evaluating any load testing tool, following evaluation criteria are used byCognizant. The team has reviewed the product based on these key factors:
2.1 Key Factors consideredo Load and Stress Test Capabilitieso Performance Monitor Test Featureso Tool Reporting Capability
In addition certain additional factors will also need to be considered to ensure acomprehensive review of the product and the vendor (for due diligence):
PDF created with pdfFactory trial version www.pdffactory.com
Process-based virtual usersimulation No Yes Yes Yes
Centralized load test controller Yes Yes Yes Yes
Allows for variable accessmethods and ability to mixaccess methods in a singlescenario: modem simulationor various line speedsimulation Yes Yes Yes Yes
Allows for throttle control fordynamic load generation No Yes No Yes
Allows for automatic service-level violation (boundaryvalue) checks No Yes Yes Yes
Allows for variable recordinglevels (network, Web, API,and so on) Yes Yes Yes Yes
Load Test Analysis Capabilities
Allows for transactionbreakdown/drilldowncapabilities for integrityverification at the per client,
per session, and per instancelevel for virtual users Yes Yes Yes Yes
Load Test Execution Capabilities
Adding or removing usersduring test run
No Yes No Yes
Snapshot on error Yes Yes Yes Yes
Distributing load on across Yes yes yes yes
PDF created with pdfFactory trial version www.pdffactory.com
Performance Testing Tool Evaluation for Web Performance Suite 3.599
White Paper Page 7 of 12
agents
Load Test Feedback
Allows for Web applicationserver integration:ColdFusion, NetDynamics,Dynamo, and so on Yes Yes Yes Yes
Can run tests on varioushardware and softwareconfigurations Yes Yes Yes Yes
Compatible with SSLrecording Yes Yes Yes YesCompatible with one or moreof the relevant technologies:Linux, UNIX, NT, XWindows,
Windows CE, Win3.1, Win95,Win98, in2000, WinME Yes Yes Yes Yes
CITRIX / Microsoft TerminalServices Compatibility Yes Yes Yes Yes
4.0 Tool Reporting Capability
FEATURE WEBPERFORME
R 3.5
LoadRunner
7.8
Rational7
SilkPerform
er6.0
Summary Level Reporting Yes Yes Yes Yes
Test Report Presentation ,Customization Of Reports
Yes Yes Yes Yes
Automatic HTML ReportGeneration Settings
Yes Yes Yes Yes
Web page/componentbreakdown with component
size Yes Yes Yes YesMerging graphs No Yes No YesZoom in/out in the graphs,along with granularitydefinition No Yes Yes YesFiltering graphs in differenttimestamps. No Yes No Yes
PDF created with pdfFactory trial version www.pdffactory.com