Stress Test Results for CA Service Catalog r11 Test Results for CA Service Catalog r11.2 ... Borland SilkPerformer was used to create and run scripts to exercise the functionality

SERVICE CATALOG R11.2 APPROVAL/FULFILLMENT STRESS TEST RESULTS

Stress Test Results for CA Service Catalog r11.2

AUGUST 2008 VERSION 1.0

SERVICE CATALOG R11.2 APPROVAL/FULFILLMENT STRESS TEST RESULTS 2

Table of Contents

Executive Summary 3

Test Environment 4

Server topology 4

Data pre-populated in the system 6

Service Catalog CA Workflow process definitions 7

Service Catalog settings 10

Service Catalog’s CA Workflow settings 10

Service Desk settings 11

Database settings 11

Test Scenario 12

Service Catalog Requester 12

Service Catalog Approver 12

Service Catalog Refresher 13

Service Desk Refresher 13

Test Results 14

Transaction statistics 14

Transaction rate over time 15

Throughput rate over time 16

Timer response times over time 17

Transaction response time details 19

Server resources used 24

Conclusion 29


Executive Summary CA Service Catalog is the application used by end users to initiate self-service requests. Organizations populate the catalog with the services offered by IT and other departments. An end user browses the catalog and adds selected catalog items to the user’s shopping cart. During the check-out process, the user submits the shopping cart for any approvals required by the organization. After any approvals, the user’s request enters the fulfillment phase. Depending on the catalog items requested, the steps to fulfill the requested items might be different. In some cases, depending on the fulfillment process desired, a CA Service Desk change order might be created.

For large organizations, depending on the number of end users and the types of catalog items being offered, there could be a high amount of request submission, approval and fulfillment activity per day. The CA Service Catalog and CA Service Desk systems need to be scalable in order to support highly active systems.

CA performed a series of stress tests on CA Service Catalog r11.2 and CA Service Desk r11.2 for a large service provider whose client was a large national defense organization. Although these tests were specifically designed to replicate the test scenarios defined by that client, the results clearly demonstrate that CA Service Catalog and CA Service Desk are scalable, enterprise-ready solutions for any client who needs to provide self-service and change management capabilities to the enterprise.

The ten-hour test exercised the repeatable execution of the self-service request life cycle from submission through fulfillment at a rate of five requests a minute or approximately 3000 requests/ten hours. This request life cycle activity was occurring at the same time as 500 concurrent users were refreshing a CA Service Catalog screen and 500 concurrent users were refreshing a CA Service Desk screen, each at a rate of one refresh every two minutes, resulting in approximately 300,000 refreshes in ten hours.

The resulting performance showed an average page time of 1.051 seconds for Requesters, 0.612 seconds for Approvers, 0.739 seconds for Service Catalog Refreshers and 0.099 seconds for Service Desk Refreshers. The success rate for the test was 99.7%.

The remainder of this document provides details of the test environment, scenarios and results.


Test Environment Several servers were used to provide a distributed environment. Borland SilkPerformer was used to create and run scripts to exercise the functionality being tested. The systems were pre-populated with substantial amounts of data.

Server topology All systems used in the test were set up at the Labs On Demand facility at CA’s corporate headquarters in Islandia, NY, with the exception of the SilkPerformer client. Multiple servers were used for the testing as described in the table below. All systems were on a LAN and clustering was not used.

Server Software OS Hardware

S1 – SilkPerformer simulating user

SilkPerformer client 7.5.0 Windows Server Enterprise Edition 2003 SP1 as a VMware guest

Intel Pentium 4 CPU 2.8 GHz, 1.5 GB RAM, 40 GB drive with at least 4 GB free space

S2 – Service Catalog

CA Service Catalog r11.2 SP1 FP1

Windows Server Enterprise Edition 2003 SP2

Intel Pentium 4 CPU Quad 3.2 GHz, 4 GB RAM, 140 GB drive with at least 4 GB free space

S3 – CA Service Catalog’s CA Workflow 1.1.5

CA Workflow r1.1.5 SP1 Build 1.1.92.0


Intel Pentium III CPU Dual 3.2 GHz, 4 GB RAM, 140 GB drive with at least 4 GB free space

S4 – CA Service Desk with CA Workflow 1.0

CA Service Desk r11.2 FP1 With CA Workflow r1.0 Build 1.0.019.014


Intel Pentium 4 CPU Dual 2.4 GHz, 4 GB RAM, 140 GB drive with at least 4 GB free space

S5 – MDB

SQL Server 2005 SP1 Windows Server Enterprise Edition 2003 SP2

Intel Pentium 4 CPU Dual 3.2 GHz, 8 GB RAM, 146 GB drive with at least 20 GB free space

S6 – CA EEM CA EEM 8.2, configured to use its own data store for a user

Windows Server Enterprise Edition

Intel Pentium III CPU Quad 2.3 GHz, 8 GB RAM, 140 GB


repository 2003 SP1 drive with at least 4 GB free space

FIGURE 1: SERVER TOPOLOGY DIAGRAM


Data pre-populated in the system The system was pre-populated with data as shown in the table below.

Item Quantity

Service Catalog services Each service contains one service option group with three single-selection services (rows) each with three service option elements (cells), one of which is a form with five input fields

5,000

Service Catalog requests Each request contains one service option from the services pre-populated

400,000

Service Catalog user accounts All accounts set to use Catalog End User role for Service Catalog and Employee role for Service Desk

60,000

Service Desk change orders 10,000


Service Catalog and Workflow process definitions CA Service Catalog’s CA Workflow was prepared with process definitions to manage the request life cycle. For the approval phase, a single level of approval was assigned to the Requester’s manager (the Approver). After approval other fulfillment process definitions were used to create a Service Desk change order for each requested item.

The approval and fulfillment processes were accomplished using several major process definitions with the characteristics shown in the tables below.

Request Life Cycle Phase

Process Activity Node Actor Type

Number of Nodes using Actor Type

Approval All processes (including for reporting failures)

Total activity nodes 91

USM_ManagerApproval (986 KB, exported)


Java Script 1

Java Object 16

Web Service 10

Process actor (normal use)

4

Process actor (for reporting failures only)

16

Global User List 0

USM_Approval (772 KB, exported)


Java Script 1

Java Object 12

Web Service 10


Process (normal use) 0

Process (for reporting failures only)

12

Global User List 1

Fulfillment All processes (including for reporting failures)


USM_HW_Fulfillment_USD (950 KB, exported)


Java Script 1

Java Object 16

Web Service 9



16

Global User List 0

USM_CreateITUSDChangeOrder (2,090 KB, exported)


Java Script 1

Java Object 11

Web Service 28



35

Global User List 0


Error Reporting

USM_PostAlertToUSM (for reporting failures only) (146 KB, exported)


Java Script 1

Java Object 3

Web Service 2



2

Global User List 0


CA Service Catalog settings The following CA Service Catalog configuration settings were adjusted to promote optimal performance.

Item Setting

Tomcat threads 750

Tomcat connection timeout 1000000

Java initial heap size 512 MB

Java maximum heap size 1025 MB

Max database connection pool size 200

CA Workflow settings The following CA Service Catalog’s CA Workflow configuration settings were adjusted to promote optimal performance.

Item Setting

Tomcat threads 500

Process engines 500

Use snapshots True

Database engine pool max active 500

Database engine pool max wait -1

Java initial heap size 512 MB

Java maximum heap size 1536 MB


Service Desk settings The following CA Service Desk configuration setting was adjusted to promote optimal performance.

Item Setting

Tomcat threads 250

Database settings No changes were made to the default settings.


Test Scenario The test scripts simulated the request create, submit, approve and fulfill phases of the CA Service Catalog request life cycle, where fulfillment includes creating a Service Desk change order. In addition, the test scripts simulated additional CA Service Catalog and CA Service Desk activity unrelated to the request life cycle.

The details of the scripts are given below. For all scripts, to simulate users’ think time there was a pause of two seconds for each page displayed and a pause of five seconds between user logout and subsequent login.

Service Catalog Requester In the Service Catalog Requester scenario, a user in the Catalog End User role logs into Service Catalog, selects one of ten services to add to his cart and submits his request for approval, then logs out of CA Service Catalog. This test simulates the catalog entry selection and request submission phases of the Service Catalog request life cycle. This script results in approximately 3,000 requests created over a ten-hour time period.

The steps in the script are as follows:

1. Ten Service Catalog Requesters log in to Service Catalog at a rate of five users per minute, overall.

2. Each of the 10 users selects one of 10 services from the catalog to add to her cart.

3. Because there is a form associated with the service selected, the user completes the five fields on the form and continues.

4. The shopping cart displays and the user submits her cart.

5. The user logs out.

When the user submits her cart, CA Service Catalog causes an associated CA Workflow process definition to start. The process instance determines the manager of the Service Catalog Requester and assigns an approval task to that Service Catalog Approver.

Service Catalog Approver The Service Catalog Approver scenario tests the condition where a user in the Catalog End User role logs into CA Service Catalog, refreshes his pending actions list until a request approval appears, approves the first request in the list and then logs out of CA Service Catalog. This test simulates the request approval phase of the CA Service Catalog request life cycle.

The steps in the script are as follows.

1. Ten Service Catalog Approvers log in to Service Catalog at a rate of five users per minute, overall.


2. The 10 users each refresh the Pending Actions page until an approval pending action appears in the list.

3. The approval page displays for the first pending action in the list.

4. The request is approved.

5. The user logs out.

When the user approves the request, CA Service Catalog causes an associated CA Workflow process definition to start. The process instance creates a CA Service Desk change order. The change order category used is not associated with a CA Service Desk CA Workflow process definition.

Service Catalog Refresher In the Service Catalog Refresher scenario, a user in the Catalog End User role is logged into Service Catalog and continually refreshes the User Profile page. This test simulates random CA Service Catalog activity while the primary request life cycle test is being conducted.


1. All 500 Service Catalog Refreshers are logged in to CA Service Catalog at the beginning of the test. The users remain logged in during the entire test period.

2. Each of the 500 Service Catalog Refreshers re-displays the User Profile page once every two minutes.

3. At the end of the test period, all 500 Service Catalog Refreshers are logged out from Service Catalog.

Service Desk Refresher The Service Desk Refresher scenario tests the condition where a user in the Employee role is logged into CA Service Desk and continually refreshes the landing page. This test simulates random CA Service Desk activity while the primary request life cycle test is being conducted.


1. All 500 Service Desk Refreshers are logged in to CA Service Desk at the beginning of the test. The users remain logged in during the entire test period.

2. Each of the 500 Service Desk Refreshers re-displays the landing page once every two minutes.

3. At the end of the test period, all 500 Service Desk Refreshers are logged out from Service Desk.


Test Results

Transaction statistics Transactions statistics for each of the three tests are shown below. The errors encountered in the failed transactions were due to HTTP connection resets primarily during the first few minutes of the test period while the one-time login of the Service Catalog Refreshers and Service Desk Refreshers was occurring.

OVERALL TRANSACTION SUCCESS

Transaction Name # Successful Transactions

# Failed Transactions

% Successful Transactions

Submit Request 2996 0 100.00%

Approve Request 2995 0 100.00%

Refresh Service Catalog Page 150,801 19 99.99%

Refresh Service Desk Page 149,894 904 99.40%


Transaction rate over time

The chart below show the number of transactions included in all scripts over time. The high values shown at the start and end of the test period are due to the one-time login and logout of the Service Catalog Refreshers and Service Desk Refreshers. Otherwise, the chart shows consistent transaction rates throughout the test. During the test, 309,647 transactions were performed at an average of 8.19 transactions/second.


Throughput rate over time The charts below show the throughput from all scripts over time for the test. The high values shown at the start and end of the test period are due to the one-time login and logout of the Service Catalog Refreshers and Service Desk Refreshers. Otherwise, the charts show consistent throughput rates throughout the test. During the test 5,941,503kB of data was sent from the silk server, with an average throughput of 157.10kB/sec. 1,535,034 HTTP requests arrived at the Web server at an average of 40.59 hits/second.


Timer response times over time Timers record the response time from the user’s perspective. SilkPerformer separates timers into Page Timers and Custom Timers. Page Timers are used for simple pages while Custom Timers are used for more complex pages. The Page Timer charts also indicate the server busy time vs. the page and document download (network transmission time), including a histogram of this data showing what portion of the total response time was due to server busy and network transmission time. This detailed data is not available for Custom Timers.

Most of the timers used in the test scripts are Page Timers. Custom Timers are used for these script user interaction steps:

RequesterNext – The Service Catalog Requester clicks Next to select a service from the catalog.

RequesterAddToCart – The Service Catalog Requester enters values on the fields on the displayed form and clicks Add To Cart and Checkout to check out.

RequesterSubmitCart – The Service Catalog Requester clicks Submit to submit his cart.

ApproverApprove – The Service Catalog Approver selects Approve then OK to approve the request.

ServiceDeskUserLogin – The Service Desk Refresher navigates to the Service Desk login page.

The charts below show Page Timer and Custom Timer response times included in all scripts over time. The high values shown at the start and end of the test period are due to the one-time login and logout of the Service Catalog Refreshers and Service Desk Refreshers. Otherwise, the charts show consistent response times throughout the test. The overall response time as measured by the page timers ranged from 0.06 to 45.22 seconds with an average of 0.78 seconds with a standard deviation of 2.19. The custom timers measured an average page time of 0.46 seconds, average document download time of 0.45 seconds and an average server busy time of 0.43 seconds.



Transaction response time details The chart below shows overall average page time for each script. Average Page Time indicates the average time before a page is returned to the user and includes all steps in the script.

AVERAGE PAGE TIME FOR EACH SCRIPT

The charts below show detailed response times for each step in each script. Average Page Time indicates the average time before a page is returned to the user for the step in the script.


AVERAGE PAGE TIME FOR EACH STEP IN THE SCRIPT BY TEST

SUBMIT REQUEST



APPROVE REQUEST



SERVICE CATALOG REFRESH



SERVICE DESK REFRESH


Server resources used CPU and memory usage were monitored using the Windows Performance Monitor on each server used in the test. A graph displaying the usage is shown below for each machine. The yellow line is used for the number of Available MB of memory at a scale of .01. Available MB is the amount of physical memory, in Megabytes, immediately available for allocation to a process or for system use. It is equal to the sum of memory assigned to the standby (cached), free and zero page lists.

The red line indicates percentage processor time, which is the percentage of elapsed time that the processor spends to execute a non-idle thread. The percentage processor time is calculated by measuring the duration the idle thread is active in the sample interval, and subtracting that time from the interval duration. This is a primary indicator of processor activity, and displays the average percentage of busy time observed during the sample interval. It is calculated by monitoring the time that the service is inactive and subtracting that value from 100%.

The pink line is used to designate network activity. Network bytes total/sec is the rate at which bytes are sent and received over each network adapter, including framing characters. Network Interface\\Bytes Received/sec is a sum of Network Interface\\Bytes Received/sec and Network Interface\\Bytes Sent/sec.

RESOURCE USAGE FOR CA SERVICE CATALOG VIEW SERVER


RESOURCE USAGE FOR CA WORKFLOW SERVER


RESOURCE USAGE FOR CA SERVICE DESK SERVER


RESOURCE USAGE FOR CA EEM SERVER


RESOURCE USAGE FOR DATABASE SERVER


Conclusion These stress tests results provide reliable evidence that CA Service Catalog and CA Service Desk can be used together to provide the enterprise with a self-service request capability that meets the high standards for reliability, availability and throughput that large enterprises require.

Copyright © 2008 CA. All rights reserved. All trademarks, trade names, service marks and logos referenced herein belong to their respective companies.

Stress Test Results for CA Service Catalog r11 Test Results for CA Service Catalog r11.2 ... Borland SilkPerformer was used to create and run scripts to exercise the functionality

Documents