Web Performance, Load, and Stress Tests Mainframe and Open System Network Natural Conference Boston Dieter W. Storr Los Angeles Times May 2004 [email protected].

Web Performance, Load, and Stress Tests

Mainframe and Open System Network

Natural Conference Boston

Dieter W. Storr

Los Angeles Times

May [email protected]

May 2004 Dieter W. Storr -- [email protected]

2

Performance, Load, and Stress TestsContents

What are performance, load, and stress tests?

Why and when should you test?

What response time do you expect?

What parts play a role in the network?

Network basics

Logical and physical transactions


3

Performance, Load, and Stress TestsContents

What tools can you use to test these parts?

When should you start performance tests?

Web site performance, analysis and tuning

ChecklistLinks


4

What are Performance Tests?

Test each part of the system, such as web server, web applications, databases, etc.

Tests can run usually without requesting page images in order to concentrate the testing on the script and code itself

?

?

?

?


5

What are Load Tests?

Load tests are performed by testing the website using the best estimate of the traffic your website needs to support.

Define the maximum time it should take for a page to load -- from a usability and customer experience standpoint, not from a technical one.

?

?

?

?


6

What are Stress Tests?

To simulate “brute force” attacks that apply excessive load to system, such as web server, web applications, databases, etc.

To estimate the maximum load that your system can support

To learn the traffic thresholds of your system

How it will respond after exceeding its threshold

?

?

?

?


7

What are Stress Tests?

Ramp test A variation of stress testThe number of users is increased

over the life of the test

?

?

?

?


8

Why Performance Tests?

Determine that all functions do what they meant to do (function test)

Ensure that tasks and objectives are easily to be accomplished (usability test)

Guarantee a result from the website in an acceptable time (performance test)


9

Why Performance Tests?

Optimize each part of the system for increased web traffic

Ensure that the end-user’s mouse click is not met with silence


10

Why Load Tests?

To compare the maximum time it should take for a page to load with the effective used time.

Calculate the impact of exceeding that maximum time. Will you lose sales or prospective customers?

Calculate the anticipated load and load pattern for the website.

Determine to do some work on the application or webserver.


11

Why Stress Tests?

Determine the maximum load

Test excessive system load

Determine the traffic thresholds of your system

Experience the reaction of the system after exceeding its threshold

Ramp tests can help to determine what maximum load a server can handle while providing optimal access to web resources


12

When Performance Tests?

You cannot start early enough!Start before a single line of code is written

Network Load balancer Application Database Web servers

During development, all software engineers and QA team should have access to performance test tools


13

When Load Tests?

As soon as web pages are working

QA team should conduct the first load tests

Performance testing should be part of the regular testing routine each day for each build of the software


14

What Response Time Do You Expect?

Does your company set limits?KEEP in mind:Many websites today have a serious

business mission => to take moneyReason: Advertisement opportunities,

selling retail products, etc.Websites and applications need to be

up and running at all times


15

What Response Time Do You Expect?ResponseTime

User's View

< 0.1 s User feels that the system is reactinginstantaneously

< 1.0 s The user experience is notcompromised. Although the user isunhappy with the wait, they are stillfocused on the current web page

<10 s As wait times get close to 10s, studieshave shown that the likelihood of userdistraction increases greatly

> 10 s User is most likely distracted from thecurrent website and loses interest

Source: Jakob Nielsen, Designing Web Usability, New Riders Publishing, 2000


16

What Response Time Do You Expect?

95% of all web requests should be processed in less than 10 seconds!

BE AWARE:If performance slows even a little, fickle web users are likely to jump quickly to a competitor’s site


17

What Parts Play a Role in The Network?

ADABAS

NATURAL

CGI

Web Server

Web Browser

Basic Connections


18

What Parts Play a Role in The Network?

HubsSwitchesRoutersFirewallsLoad BalancerApplication ServersDatabase ServersWeb Servers


19

Network Basics

Hub / Switch / Bridge / Router / Firewall

In data communications, a hub is a place of convergence where data arrives from one or more directions and is forwarded out in one or more other directions. A hub usually includes a switch of some kind.

See USB 7-Port HubTo connect severalUSB devices


20

Network Basics


In telecommunications, a switch is a network device that selects a path or circuit for sending a unit of data to its next destination.

See 8-Port Network Switch


21

Network Basics


A switch may also include the function of the router, a device or program that can determine the route and specifically what adjacent network point the data should be sent to.

In general, a switch is a simpler and faster mechanism than a router, which requires knowledge about the network and how to determine the route


22

Network Basics


In telecommunication networks, a bridge is a product that connects a local area network (LAN) to another local area network that uses the same protocol (for example, Ethernet or token ring).


23

Network Basics


On the Internet, a router is a device or, in some cases, software in a computer, that determines the next network point to which a packet should be forwarded toward its destination.

See 4-Port Cable/DSL Router


24

Network Basics

Hub / Switch / Bridge / Router / FirewallA system designed to prevent unauthorized

access to or from a private network. Firewalls can be implemented in both hardware and software, or a combination of both.

PIX Firewall from CISCOPrivate Internet Exchange Firewall

Gauntlet Firewall from NAINetwork Associates Inc.


25

Network Basics

Bridge


26

Web Server

Firewall

Appl. Server

InternetInternet

Screen Scraper

Firewall

Com-pleteNATURAL ADABAS

CICSCOBOL VSAM

Firewall

SSL

OracleFirstLogic

DBs

Web Server

Appl. Server

Screen Scraper

CISCO Router

e.g. Apache

e.g. WebLogic BEA

e.g. Attachmate

Main-frame

Open

System


27

Logical and Physical Transactions?

ADABAS

NATURAL

Appl. Server

Web Server

Web Browser


28

Response Time / Click Time Tests

ADABAS

NATURAL

Appl. Server

Web Server

Web BrowserFunction:

Display my account


29

What Tools Can You Use to Test?

Web Server Stress ToolUser simulationLogging periodSelect URLsURL recorderData mergingBrowser simulationReviewing summary and detailed log resultsAnalyzing graphical results

Web Browser


30

CSS Stress Test 5 Users 10 Clicks25 Sep 2003, 11:52-11:53


31

CSS Stress Test 5 Users 10 Clicks14 Oct 2003, 14:05


32

09/25/2003 11:52:02 AM --- WEBSERVER STRESS TOOL

User #1 Click #1 started, used URL #1 (CSS-Prod-1) POST-DATA= "[email protected]&password=<snip>

User #1 Click #1: Request 1 HTTP State: Initialized (0 ms)

User #1 Click #1: Request 1 HTTP State: DNS Lookup begin (7 ms)

User #1 Click #1: Request 1 HTTP State: DNS Lookup done (17 ms)

User #1 Click #1: Request 1 HTTP State: Connected (504 ms)

User #1 Click #1: Request 1 HTTP State: Waiting for header (512 ms)

User #1 Click #1: Request 1 HTTP State: Receiving Header (5937 ms) (=Time to First Byte)

User #1 Click #1: Request 1 HTTP State: Waiting for body (5947 ms)

User #1 Click #1: Request 1 HTTP State: Done (5999 ms)

User #1 Click #1: CLICK-Request 1: Time=6008 ms, TFB=5937ms, Bytes=16038, HTTP-StatusCode=200 (OK)

User #1 Click #1: !Success! All requests of click done (6046 ms)


33


APAS/INSIGHTADABAS

DWSCWUSE: SUMMARIZE (LS=150) COUNT (HD='COMMANDS') PERCENT (HD='% OF TOTAL' PF=G10) SUM(TOTAL-IO) (HD='TOTAL IO') PCT(TOTAL-IO) (HD='% OF TOTAL' PF=G10) MEAN(TOTAL-IO) (HD='AVG IO') MAX(TOTAL-IO) (HD='MAX IO') MEAN(DURATION) (HD='AVG DUR') MAX(DURATION) (HD='MAX DUR') MEAN(ENQ-TIME) (HD='AVG ENQ') MAX(ENQ-TIME) (HD='MAX ENQ') BY USER-ID WHERE USER-ID(2) = 'CW' INSIGHT-LINES = 30 PRINT TO STRESS01;


34

USER ADABAS ID COMMANDS % OF TOTAL -------- ---------- ----------- CW0016 269 |====== CW0017 269 |====== CW0018 269 |====== CW0019 369 |========= CW0020 369 |========= CW0021 369 |========= CW0022 369 |========= CW0023 369 |========= CW0024 269 |====== CW0025 269 |====== ---------- ----------- ======== 3,190 |========>

AVG DUR SEC MAX DUR SEC------------ --------- 0.00033 0.006 0.00052 0.008 0.00036 0.005 0.00058 0.014 0.00041 0.011 0.00043 0.008 0.00045 0.008 0.00038 0.007 0.00034 0.006 0.00049 0.009 ------------ --------- 0.014

ADABAS on Enterprise ServerCSS Stress Test

5 Users 10 Clicks25 Sep 2003, 11:52-11:53


35


APAS/INSIGHT ADABAS

0090 DWSCWOBX: EXTRACT (LS=150) 0100 NAT-LOAD-COUNT (HD='OBJ-LOAD') 0110 DATE TIME 0120 NAT-PROGRAM CMD 0130 USER-ID 0140 GLOBAL-ID 0150 WHERE USER-ID(2) = 'CW' AND 0160 (NAT-PROGRAM = 'IWS00P00' OR 0170 NAT-PROGRAM = 'IWD10P10') 0180 INSIGHT-LINES = 200 0190 PRINT TO STRESS01 0200 ;


36

NATURAL PROGRAM OBJ-LOAD -------- -------- IWD10P10 50 -------- 50

ADABAS on Enterprise ServerCSS Stress Test

5 Users 10 Clicks25 Sep 2003, 11:52-11:53


37

Response Time / Click Time Tests

ADABAS

NATURAL

Appl. Server

Web Server

Web BrowserFunction:

Display my account

200ms - 20s

0.00033s - 0.014s


38


STRESS -- self-writtenADABAS

1 Start process with heavy CPU workload 2 Start process with many ADABAS Calls 3 Start process with heavy CPU workload and many ADABAS calls 4 Start process with many ADABAS calls and excessive terminal I/O's 5 Start asynchronous processes (ADABAS & CPU)


39


40


41


42


43


44


Com-pleteUCTRL NETSPYTMON/MVS

TP Monitor


45

UCTRL17:49:14 TID 6 COMTEST USER ZINDWS 03/16/04 --- COM-PLETE CONTROL FACILITY --- UCT1

Function FC Function FC Function FC-------------------- -- -------------------- -- -------------------- --ADABAS interface ... AI Program lookaside .. IP Task groups ........ PGADABAS statistics .. AS Program in-stor dir. OP Task list .......... PLBuffer activities .. BA Resident programs .. RP Thread groups ...... TGCommon storage ..... CS Find program ....... FP Thread subgroups ... SGDB2 thread status .. D2 Queue overview ..... QO Thread list ........ TLFBPM statistics .... FB Roll activities .... RA User activities .... UAMessage log ........ CM Server overview .... SO User status......... US Terminal overview .. TO VSAM statistics .... VS Terminal status .... TS Print statistics ... PR

Select function .... Operand ............

Operator command:

Enter-PF1---PF2---PF3---PF4---PF5---PF6---PF7---PF8---PF9---PF10--PF11--PF12--- End


46

NETSPY Time Host Avg-Resp User Avg-Resp Worst Host_Resp No. Session No. Input No. Output

13:30 0.1 0.1 10.9 14 3777 399613:45 0.0 0.0 0.9 4 749 78014:00 0.1 0.1 5.7 25 10342 1102914:30 0.1 0.1 4.1 2 2094 2114

Netspy Response Time

0.0

2.0

4.0

6.0

8.0

10.0

12.0

13:30 13:45 14:00 14:30

Test Time

Re

spo

nse

Tim

e i

n S

eco

nd

s

Host Avg-Resp

User Avg-Resp

Worst Host_Resp


47

TP Monitor Com-plete Consumption

CPU TOTAL CPU % Paging Rate IO Rate19.31 1.60 0.00 26.50

COMTEST EXECUTION DETAIL

0.00

5.00

10.00

15.00

20.00

25.00

30.00

CPU TOTAL CPU % Paging Rate IO Rate

CATEGORY

VA

LU

E

Series1


48

TMON

Alerts Average response time CPU usage I/O usage Number of ended transactions rate


49


Attachmate

Run Time ApplicationManagement & Control Services (MSC) --

monitoring pools and sessionsJava console running in foreground

Screen Scraper


50


Attachmate Run Time Application

Number of sessions to host (mainframe)

Maximum sessionsNumber of failuresQueue size

Screen Scraper


51


Attachmate ServersNT utilities Sampling at 10-second

intervals

Screen Scraper


52


Attachmate Servers

Analyze results:CPU utilizationMemory utilizationRun queuesSwap space utilizationDisk I/O - most busy which disk hit the hardest I/O wait > 10%Page file - swaps

Screen Scraper


53


Cisco 2900 SwitchManual testing/sampling

Analyze resultsLoad on network componentsPacket throughput

Content Server Switch


54


Cisco 4000 Router (10BaseT)Manual testing/sampling

Analyze resultsLoad on network components

http://www.scd.ucar.edu/nets/devices/routers/ios_analysis.html

Packet throughputRouter Configuration:

http://www.swcp.com/~jgentry/topo/cisco.htm

Router


55

Dedicated line broadband speedsand prices

T-1 - 1.544 megabits per second (24 DS0 lines)Ave. cost $400.-$650./mo.

T-3 - 43.232 megabits per second (28 T1s)Ave. cost $6,000.-$16,000./mo.

OC-3 - 155 megabits per second (100 T1s)Ave. cost $20,000.-$45,000./mo.

OC-12 - 622 megabits per second (4 OC3s) no price

OC-48 - 2.5 gigabits per seconds (4 OC12s) no price

OC-192 - 9.6 gigabits per second (4 OC48s) no priceSource: http://www.infobahn.com/research-information.htmprices updated: 16 March 2004


56


OracleOracle utilitiesApplication-specific log filesPerformance Monitoring and Tuning Tools

• TKProf• UTLBSTAT.SQL and UTLESTAT.SQL - Begin and end

stats monitoring

• Statspack• Oracle Enterprise Manager - Tuning Pack

RDBMS


57


OraclePerformance Monitoring and

Tuning Tools (cont”)• DBXRAY: A real-time graphical snapshot of the health of your

Oracle database- anytime, anywhere.• Patrol: Intelligent and non-intrusive for proactive database

monitoring, including Oracle Parallel Server and Oracle Symmetric Replication environments.

• SQL-Explorer for Oracle The most advanced SQL tuning tool available. Identify, model and tune problematic SQL

statements to boost application performance.

RDBMS


58


Oracle

Analyze results:Number of Oracle processesBuffer hit cacheBlock I/O rateNumber of cursors createdNumber of connectionsLog information regarding number of

registrations, starts, etc.

RDBMS


59


SSL Accelerator• Secure Socket Layer (protocol between

HTTP and TCP) -- appliance or card

Manual testing/sampling

Analyze results:Load on network componentsPacket throughput

Optimizing SSL processing for Web http://www.nwfusion.com/news/tech/2000/1023tech.html?nf

SSL


60


PIXManual testing/sampling

Analyze results:Load on network componentsPacket throughput

Firewall


61


Gauntlet (several)Manual testing/sampling

Analyze results:MemoryDisk I/OFull system log

Firewall


62


Sun OSSolaris utilitiesSampling at 10-second intervals

Web Server


63


Sun OS

Analyze results:CPU utilizationMemory utilizationRun queuesSwap space utilizationDisk I/O -- most busy which disk hit the hardest I/O wait > 10%Page file -- swaps

Web Server


64


Apache ApplicationLog Analyzer

Analyze results:Number of sessionsBytes per sessionSource addressDestination address

Web Server


65


Sun OSSolaris utilitiesSampling at 10-second

intervals

Application Server


66


Sun OS

Analyze results:CPU utilizationMemory utilizationRun queuesSwap space utilizationDisk I/O -- most busy which disk hit the

hardest I/O wait > 10%Page file -- swaps

Application Server


67


BEA WebLogic ApplicationWebLogic administrative consoleSpotlight

Application Server


68


BEA WebLogic Application

Analyze results:Number of active queuesNumber of connectionsNumber of active socketsTypes of accessTime of access

Application Server


69

Web Site PerformanceAnalysis and Tuning

In-House or Web Caching Near the network edge, e.g. proxy cache or

browser: static HTML, JavaScript, and image files Cache-control header “no-cache” “re-fetch”

“expiry date”

Compression HTML is compressed before being sent through

the network.


70

Web Site PerformanceAnalysis and Tuning

SSL Acceleration (box or card) Consider the number of transactions per second

(TPS). TPS will be more important on SSL Acceleration appliances than cards.

An SSL Acceleration hardware appliance (box) can deal with transactions for all your web servers and scale quickly.

Network Load Balancing Round Robin Lowest response time Least connections Proximity


71

Checklist -- Getting Started?Determine your environment

• Mainframe, Open System, Middleware

Evaluate Stress Test Software• Write your own or buy one

Understand the different components of a web site download

• DNS lookup• Redirect• First byte download• Base page download• Content download


72

Checklist -- Getting Started?What kind of content?

• Dynamic or static• .html or image

Number of serversWhere does traffic comes from

• Internet, Intranet, or Extranet


73

Create a Stress Test Plan

Test Organization• Overview• Participants

• Technical team• Coordination team• Application team

• Objectives• Infrastructure• Application

• Assumptions / pretest conditions• Communication plan


74


Environment / Test Components• Overview• Overall success criteria• Transactions to be tested

• Logical and physical• Application functions

• Data indentification• Script development• High-level system diagram• Components

• Responsibility• Metrics• Tool / method


75


Sequence of Events• Summary• Process• Schedule of events


76

Web Speed is no Magic!


77

Links to Interesting Websites

http://www.edwardtsai.com/web-site-performance.html

http://www.nwfusion.com/news/tech/2000/1023tech.html?nf

http://www.scd.ucar.edu/nets/devices/routers/ios_analysis.html

http://www.swcp.com/~jgentry/topo/cisco.htm


78

Thank You!

Web Performance, Load, and Stress Tests Mainframe and Open System Network Natural Conference Boston Dieter W. Storr Los Angeles Times May 2004 [email protected].

Documents

performance tests

web performance

load tests performance

performance test tools

excessive load

software slide

web site performance

load pattern