Top Banner
IBM Rational Asset Manager Version 7.0.0.2 Capacity and scalability benchmarks IBM Corporation Development Gili Mendel, Senior Technical Staff Member Sheehan Anderson, Software Developer Rich Kulp Performance test John Reinstrom, Chief Performance Test Architect Amy Pitts, Performance Tester Skye Bischoff, Software Test Specialist Jennifer Chang, Staff Software Engineer IBM Innovation Center Joanne Bick Lee Bliss Marlon Machado Scott Martin Cindy O’Brien Alan White IBM Software Services for Rational Bryan Miller, Service Offering Development Lead Level: Intermediate March 2008
33
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Page 1: IBM Rational Asset Manager Version 7.0.0.2

IBM Rational Asset Manager Version 7.0.0.2 Capacity and scalability benchmarks

IBM Corporation

Development

Gili Mendel, Senior Technical Staff Member Sheehan Anderson, Software Developer

Rich Kulp

Performance test John Reinstrom, Chief Performance Test Architect

Amy Pitts, Performance Tester Skye Bischoff, Software Test Specialist

Jennifer Chang, Staff Software Engineer

IBM Innovation Center Joanne Bick

Lee Bliss Marlon Machado

Scott Martin Cindy O’Brien

Alan White

IBM Software Services for Rational Bryan Miller, Service Offering Development Lead

Level: Intermediate

March 2008

Page 2: IBM Rational Asset Manager Version 7.0.0.2

IBM Rational Asset Manager capacity and scalability benchmark, Page 2 of 33

Copyright © 2008, IBM® Corporation. All rights reserved. Published by IBM® developerWorks®.

Contents

Abstract ................................................................................................................. 3

Purpose and content................................................................................................. 3

Benchmark test goals ............................................................................................... 3

Test configuration goals ............................................................................................ 4

Benchmark environment ........................................................................................... 4

Rational Asset Manager repository .......................................................................... 4

Test environment ................................................................................................. 6

Benchmark results ................................................................................................. 15

Asset scalability.................................................................................................. 16

User scalability ................................................................................................... 17

Servers resource utilization levels ......................................................................... 18

Appendix: Test results ............................................................................................ 25

Test matrix ........................................................................................................ 25

Trademarks........................................................................................................... 32

Resources............................................................................................................. 32

About the authors .................................................................................................. 33

Page 3: IBM Rational Asset Manager Version 7.0.0.2

IBM Rational Asset Manager capacity and scalability benchmark, Page 3 of 33

Copyright © 2008, IBM® Corporation. All rights reserved. Published by IBM® developerWorks®.

Abstract

The aim of this article is to provide guidance in specifying appropriate deployment architecture for IBM® Rational® Asset Manager. The authors provide specific results and analysis of a performance benchmark test that was conducted for Rational Asset Manager Version 7.0.0.2. This report also helps you interpret the results of this benchmark test and explains how to use these results to extrapolate specific hardware requirements for an environment that was not specifically tested.

Purpose and content

The aim of this article is to guide readers in specifying appropriate deployment architecture for IBM Rational Asset Manager. The authors give specific results and an analysis of a performance benchmark test that was conducted for Rational Asset Manager Version 7.0.0.2. In addition, they provide information to help you interpret the results of this benchmark, as well as to be able to use these results to extrapolate specific hardware requirements for an environment that was not specifically tested. This article does not provide an overview of Java™ 2 Platform, Enterprise Edition (J2EE) scaling, nor does it cover tuning parameters to be used in a large-scale deployment. You can find an overview of Rational Asset Manager as a J2EE application in the Rational Asset Manager 7.0 Capacity Planning Guide 1. For a comprehensive overview of how to tune a large IBM® WebSphere® Application Server cluster for a Rational Asset Manager deployment, consult the Rational Asset Manager Tuning Guide 2. (See the Resources section at the end for links to both.)

Benchmark test goals

These were the goals of this benchmark test:

Determine the level of user loads that could be driven across small, medium, and large cluster configurations. These ranged from a single server node running a single WebSphere container instance up to a 6-server node configuration with 4 WebSphere Application Server instances per node (for a total of 24 instances).

Collect comparative data on the impact of vertical versus horizontal scaling on the capacity of Rational Asset Manager. (Horizontal scaling refers to a configuration that expands the cluster by adding server nodes, where as vertical scaling refers to a configuration that increases the number of WebSphere Application Server instances running on a given server node.)

Verify that Rational Asset Manager can continue to scale both horizontally and vertically with a large amount of assets.

Collect performance data metrics for a mixture of configuration permutations to allow extrapolation or modeling for generic capacity level needs.

Page 4: IBM Rational Asset Manager Version 7.0.0.2

IBM Rational Asset Manager capacity and scalability benchmark, Page 4 of 33

Copyright © 2008, IBM® Corporation. All rights reserved. Published by IBM® developerWorks®.

Test configuration goals

We intended to expose the Rational Asset Manager repository to varying levels of assets by using 9 different configurations that represented small, medium, and large deployments. Our objective was to maintain an average response time of less than one second during our test duration. We also limited the configuration variation to only horizontal and vertical scaling of the WebSphere Application Server instances. The hardware used for the application servers, database, Web server, caching proxy, and load balancer remained constant. Table 1 represents the test matrix that we used to collect data for a particular asset level uploaded to Rational Asset Manager.

Table 1. Test matrix used for collecting data for particular asset levels uploaded to Rational Asset Manager

Application server configuration 1 node 3 nodes 6 nodes

1 instance Configuration 1 Configuration 4 Configuration 7

2 instances Configuration 2 Configuration 5 Configuration 8

4 instances Configuration 3 Configuration 6 Configuration 9

For each configuration, we exposed the cluster to the maximum number of users that could be sustained for the duration of our tests. Note that the server configuration that we used could support, at most, 4 WebSphere Application Server instances on a single server before performance degraded (see the Rational Asset Manager repository section, which follows, for specific server configurations).

Benchmark environment

Rational Asset Manager repository

Hardware topology

The basis for our configuration was 6 IBM AIX application servers, each running multiple instance of WebSphere Application Server. Figure 1 depicts the logical server deployment architecture. In reality, we used logical partitions (LPARs) on an IBM System 5 Enterprise Server, as detailed in Table 2. The communication network was a dedicated 1 Gbps Ethernet network. The storage area network used for the external disks was based on an IBM DS48005 Storage Area Network (SAN). Each server had the OS and paging space on separate internal drives.

Page 5: IBM Rational Asset Manager Version 7.0.0.2

IBM Rational Asset Manager capacity and scalability benchmark, Page 5 of 33

Copyright © 2008, IBM® Corporation. All rights reserved. Published by IBM® developerWorks®.

Figure 1. Logical Rational Asset Manager server topology

Table 2. Hardware components

Server System/CPU Memory External disk (DS 4800 SAN)

Application server: WebSphere Application Server 6.1 Fix Pack 11 2GB Java™ Virtual Machine (JVM) heap per instance Rational Asset Manager 7.0.0.2

6 p595 LPARs 4x1.9GHz Power5

15GB Lucene index 24+P RAID5 64K segment

Web Server IHS 6.1 Fix Pack 11

2 p595 LPARs 2x1.9GHz Power5

8GB

Database server IBM® DB2® 9.1 Fix Pack 3a

1 p570 LPAR 8x2.2GHz Power5+

32GB Data 8+P RAID5 log 4+P RAID5 64K segment

Asset (file) server 1 p570 LPAR 4x2.2GHz Power5+

16GB Assets’ .ras files 2x6+P RAID5 64K segment

Cache proxy edge server 6.1 2 p570 LPARs 2x2.2GHz Power5+

8GB

Load balancer edge server 6.1 1 p550Q LPAR 3x1.5GHz Power5Q

12GB

Network deployment WebSphere Application Server 6.1, Fix Pack 11

1 p520Q LPAR 2x1.5GHz Power5Q

5GB

Page 6: IBM Rational Asset Manager Version 7.0.0.2

IBM Rational Asset Manager capacity and scalability benchmark, Page 6 of 33

Copyright © 2008, IBM® Corporation. All rights reserved. Published by IBM® developerWorks®.

Test environment

Hardware topology

Figure 2. Logical test server topology

Table 3. Hardware components

Server System/CPU Memory

Workbench Rational Performance Tester 7.0.1

2x3.6GHz Pentium 8GB

Load Driver Rational Agent Controller (RAC) Rational Performance Tester 7.0.1

2x3.6GHz Pentium 4GB

License Server 2x3.6GHz Pentium 8GB

User simulation

The use cases were designed to represent simple, realistic Rational Asset Manager tasks (see the Users' use case simulation makeup subsection that follows). Each user simulated through IBM Rational Performance Tester was assigned a use case scenario based on the following realistic workload:

50% keyword search 29% faceted search 15% download assets content 5% download assets 1% submit assets

Each Rational Performance Tester virtual user (VU) was assigned a non-administrator user ID and password pair that was read from a Rational Performance Tester data pool. In this way, a given simulated user was not logged in multiple times.

Page 7: IBM Rational Asset Manager Version 7.0.0.2

IBM Rational Asset Manager capacity and scalability benchmark, Page 7 of 33

Copyright © 2008, IBM® Corporation. All rights reserved. Published by IBM® developerWorks®.

The wait times between pages were coded as think time in Rational Performance Tester. The think times were set to 2 minutes between pages in order to produce a page throughput rate of approximately 30 pages per hour per user. The Home, Sign In, and Sign Out pages were, by contrast, given a think time of 1 second to more quickly have the VU signed back in to Rational Asset Manager and performing the use case. In each test scenario, the VU signs in to Rational Asset Manager by using a set of credentials from a Rational Performance Tester data pool of user ID and password pairs, and then enters a 3-iteration loop where they perform the work of their assigned use cases. After the 3 iterations of this loop, the VU signs out of Rational Asset Manager. Because each test is contained inside of a loop in the Rational Performance Tester schedule, the VU immediately re-runs the same test, signing back in to Rational Asset Manager with the same credentials.

Performance guidelines

Each performance test consisted of both a ramp-up period, during which new users were added in a staggered fashion while existing users executed their test cases. This was immediately followed by a full-load period, during which the full user load executed their use cases. During the ramp-up period, users were added at the rate of 1 per 0.5 second; thus, the full ramp-up period was equal to the number of users divided by 2 seconds. The full-load period always lasted 10 minutes, regardless of the number of users. The goal in choosing this approach was to discover the maximum loads for a given configuration over a short time. It should be noted that runs for a shorter period of time may be able to support higher numbers of users than reported in this benchmark; whereas, runs over a longer period may be able to support fewer users. Figure 3 helps to explain this concept as observed in our test environment.

Page 8: IBM Rational Asset Manager Version 7.0.0.2

IBM Rational Asset Manager capacity and scalability benchmark, Page 8 of 33

Copyright © 2008, IBM® Corporation. All rights reserved. Published by IBM® developerWorks®.

Figure 3. Peak capacity and duration runs

If a test run reported under our performance guidelines had an average response time of 250ms, it was seen that a full load test run could last more than 30 minutes, and the average response time still did not exceed 2 seconds. However, if a response time was reported to be 450ms, a full load run would only last 15 minutes before response times began to exceed 2 seconds. During the benchmark, all runs had an average response time of 325ms ± 125ms. When reading the benchmark results (see Table 5 in the Test matrix section in the Appendix), which displays response times for all runs, keep Figure 3 in mind. If the peak expected capacity is 3,000 users over a 10-minute period, it may be acceptable to choose a configuration that shows a 450ms response time for this level of users. However, if you expect a continuous load of 3,000 users over hours or days, it would be best to choose a configuration that shows a lower response time or a configuration that supports a higher peak user capacity.

Users' use case simulation makeup

The following subsections describe specific simulated user workload scenarios used in the benchmark test runs. Transitions are marked with a think time. Download Assets scenario: In the Download Assets scenario, the VU executes these actions in a 3-iteration loop after signing into Rational Asset Manager:

1. Navigate to the Search page.

2. Drawing a string from an Rational Performance Tester data pool of search strings, enter it as keyword to search for an asset containing a small file.

Page 9: IBM Rational Asset Manager Version 7.0.0.2

IBM Rational Asset Manager capacity and scalability benchmark, Page 9 of 33

Copyright © 2008, IBM® Corporation. All rights reserved. Published by IBM® developerWorks®.

3. Drawing a string from another Rational Performance Tester data pool of the Global Unique Identifiers (GUIDs), use it as a GUID to go to the Content tab of the General Details page of an asset containing one small file (4 to 25 bytes).

4. Download the file.

5. Sign out after you complete the loop. Figure 4 shows the test structure.

Figure 4. Download assets

Page 10: IBM Rational Asset Manager Version 7.0.0.2

IBM Rational Asset Manager capacity and scalability benchmark, Page 10 of 33

Copyright © 2008, IBM® Corporation. All rights reserved. Published by IBM® developerWorks®.

Download Artifacts scenario: In the Download Artifacts scenario, after signing in to Rational Asset Manager, the VU executes these actions in a 3-iteration loop:

1. Navigate to the Search page.

2. Drawing a string from the Rational Performance Tester data pool of search strings, enter a keyword for an asset containing a small file.

3. Drawing a string from the Rational Performance Tester data pool of GUIDs, use a GUID to go to the General Details page of an asset containing one small file (4 to 25 bytes).

4. Download the artifact by clicking on the artifact’s link on the asset’s Content page.

5. Sign out after you complete the loop. The test structure is pictured in Figure 5 on the next page.

Page 11: IBM Rational Asset Manager Version 7.0.0.2

IBM Rational Asset Manager capacity and scalability benchmark, Page 11 of 33

Copyright © 2008, IBM® Corporation. All rights reserved. Published by IBM® developerWorks®.

Figure 5. Download artifacts

Keyword Search scenario: In the Keyword Search scenario, after signing in to Rational Asset Manager, the VU executes these actions in a 3-iteration loop:

1. Navigate to the Search page.

2. Drawing a string from the Rational Performance Tester data pool of search strings, enter a keyword for an asset containing a small file.

3. Drawing a string from the Rational Performance Tester data pool of GUIDs, use a GUID to go to the General Details page of an asset.

4. Sign out after you complete the loop.

Page 12: IBM Rational Asset Manager Version 7.0.0.2

IBM Rational Asset Manager capacity and scalability benchmark, Page 12 of 33

Copyright © 2008, IBM® Corporation. All rights reserved. Published by IBM® developerWorks®.

The test structure is pictured in Figure 6. Figure 6. Keyword search

Faceted Search scenario: In the Faceted Search scenario, after signing in to Rational Asset Manager, the VU executes these actions in a 3-iteration loop:

1. Navigate to the Search page.

2. Search for assets of a given Rational Asset Manager Type. In our configuration, there are 15 types defined to Rational Asset Manager, and assets were uploaded equally distributed among the 15 types. In this way, the search will return a results set with a size that is 1/15 the total number of assets.

3. Drawing a string from the Rational Performance Tester data pool of GUIDs, use a GUID to go to the General Details page of an asset.

Page 13: IBM Rational Asset Manager Version 7.0.0.2

IBM Rational Asset Manager capacity and scalability benchmark, Page 13 of 33

Copyright © 2008, IBM® Corporation. All rights reserved. Published by IBM® developerWorks®.

4. Sign out after you complete the loop. The test structure is pictured in Figure 7.

Figure 7. Faceted search

Submit Assets scenario: In the Submit Assets scenario, after signing in to Rational Asset Manager, the VU executes these actions in a 3-iteration loop:

1. Navigate to the Submit an Asset page.

2. Define a new asset by entering a Name and Short Description, and then choosing a Type and Community for the asset. These are the minimal fields that must be completed to upload an asset to Rational Asset Manager.

Page 14: IBM Rational Asset Manager Version 7.0.0.2

IBM Rational Asset Manager capacity and scalability benchmark, Page 14 of 33

Copyright © 2008, IBM® Corporation. All rights reserved. Published by IBM® developerWorks®.

3. Using the Basic interface for asset upload, navigate to the location on the Rational Performance Tester machines that contain a small file to upload (382 bytes), and attach the file.

4. From the Confirm asset page, submit the asset by clicking the Submit button.

5. From the Success page, navigate to the General Details page of the newly submitted asset.

6. Sign out after you complete the loop. The test structure is pictured in Figure 8.

Figure 8. Submit assets

Page 15: IBM Rational Asset Manager Version 7.0.0.2

IBM Rational Asset Manager capacity and scalability benchmark, Page 15 of 33

Copyright © 2008, IBM® Corporation. All rights reserved. Published by IBM® developerWorks®.

Benchmark results

Figure 9 reflects the overall scaling characteristics of Rational Asset Manager. The graph was created by using test runs on a repository with 2 million assets. User capacity is the user level (up to 5,000 users in the 2 million-asset tests) normalized by the average response time. As the results show, Rational Asset Manager is able to scale as the number of both nodes and instances is increased. These tests maintained a sub second average response time. These results satisfy our goal of proving the scalability of Rational Asset Manager across various configurations and asset levels. (We have verified similar results by using a configuration with 3 million assets.) You can also see from Figure 9 that a cluster running multiple instances scales more rapidly as we increase the number of nodes. This is evident from the surface graph’s steeper diagonal slope as we increase nodes with multiple WebSphere Application Server instances. This is an important planning consideration. Multiple instances will typically comprise a smaller server cluster for a given capacity. Fewer servers mean less complexity and maintenance costs.

Figure 9. Overall Rational Asset Manager scalability (Z axis represents capacity for users)

Figure 10 better displays the difference between horizontal and vertical scaling. The X axis for the vertical scaling line (squares) represents instances per a single node; whereas, the X axis for the horizontal scaling line (diamonds) represents nodes each with a single instance. The square line in Figure 10 is comparable to the slope of the left-edge bars in Figure 9. The diamond line in Figure 10 is comparable to the slope of the front bars of Figure 9.

Page 16: IBM Rational Asset Manager Version 7.0.0.2

IBM Rational Asset Manager capacity and scalability benchmark, Page 16 of 33

Copyright © 2008, IBM® Corporation. All rights reserved. Published by IBM® developerWorks®.

As the results show, vertical scaling is initially logarithmic and could degrade performance with too many instances. Diminishing vertical scaling is predominantly limited by the amount of memory available. With 15GB of memory and a 2GB heap size per instance, we observed that going beyond 4 instances diminished overall performance. Horizontal scaling, on the other hand, continued to provide improved performance. We did not attempt to scale beyond 6 application servers in this benchmark, because the capacity that these servers provided was sufficient to verify our initial goals. This linear horizontal clustering validates that the architecture of Rational Asset Manager scales. This is mainly because each server provides index services only to the WebSphere Application Server instances for that node.

Figure 10. Comparison of horizontal and vertical scaling

Asset scalability The number of assets stored in a repository can affect performance, although not significantly. The reason for the performance degradation in our tests was that some test scenarios returned (processed) a larger amount of search results as the number of assets grew. Figure 11 depicts the increase in the average response time as the number of assets in the repository was increased.

Page 17: IBM Rational Asset Manager Version 7.0.0.2

IBM Rational Asset Manager capacity and scalability benchmark, Page 17 of 33

Copyright © 2008, IBM® Corporation. All rights reserved. Published by IBM® developerWorks®.

Figure 11. Asset scalability

User scalability It is also important to consider user scalability, while holding the number of assets and configuration constant. Bottlenecks begin to occur on the application server primarily due to excessive garbage collection. This phenomenon likely causes response times to grow exponentially as users are added. Because of the nature of Java™ garbage collection and the long pauses associated with this process, remember that the capacity of a particular configuration over a short duration may be significantly greater than the capacity of the same configuration over a very long duration. Therefore, it is important to consider both of these factors when deciding the level of hardware needed to support particular user loads. (See Figure 12 on the next page.)

Page 18: IBM Rational Asset Manager Version 7.0.0.2

IBM Rational Asset Manager capacity and scalability benchmark, Page 18 of 33

Copyright © 2008, IBM® Corporation. All rights reserved. Published by IBM® developerWorks®.

Figure 12. User scalability

Servers resource utilization levels Disk, memory, and CPU use were also monitored on servers during the test runs. Figures 13, 14, and 15, which follow, show this data for some of the more demanding runs. As the results show, the Web servers, caching proxies, and load balancer were able to handle these large loads without heavy resource usage. Therefore, this article focuses on the resource use of the application servers and the database server.

Page 19: IBM Rational Asset Manager Version 7.0.0.2

IBM Rational Asset Manager capacity and scalability benchmark, Page 19 of 33

Copyright © 2008, IBM® Corporation. All rights reserved. Published by IBM® developerWorks®.

Figure 13. CPU use

Figure 14. Memory use

Page 20: IBM Rational Asset Manager Version 7.0.0.2

IBM Rational Asset Manager capacity and scalability benchmark, Page 20 of 33

Copyright © 2008, IBM® Corporation. All rights reserved. Published by IBM® developerWorks®.

Figure 15. Disk use

As mentioned previously, the number of assets in a repository can have an effect on performance if it is assumed that users’ searches will return a greater number of results. You can see that the application server will use a higher percentage of its CPU capacity; whereas, the database server is not as heavily affected. This is because the indexes against which the searches are performed reside on each application server, not the database server. This helps to avoid the problem of the database becoming a bottleneck in horizontal scaling configurations. (See Figure 16 on the next page.)

Page 21: IBM Rational Asset Manager Version 7.0.0.2

IBM Rational Asset Manager capacity and scalability benchmark, Page 21 of 33

Copyright © 2008, IBM® Corporation. All rights reserved. Published by IBM® developerWorks®.

Figure 16. Asset scaling

Although performance can be improved through vertical clustering, it is important to remember that each additional instance will require more memory. Figure 17 shows that memory use will grow in a linear fashion as instances are added, and this will be the primary bottleneck. Therefore, when adding vertical instances, it is important to observe the total system memory use to ensure that no paging will occur. Additionally, adding more memory to an application server can allow utilization of untapped CPU capacity.

Page 22: IBM Rational Asset Manager Version 7.0.0.2

IBM Rational Asset Manager capacity and scalability benchmark, Page 22 of 33

Copyright © 2008, IBM® Corporation. All rights reserved. Published by IBM® developerWorks®.

Figure 17. Vertical cluster scaling

Figure 18 shows how the single database server scales as the number of instances in a cluster is increased. Bear in mind that, in this benchmark, the database server was very powerful, comprising 8 processors, 32GB of memory, and an 8-disk RAID 5 array on which the data resided. You can see that the scaling with 2,000,000 assets is fairly linear as the number of instances is increased, and that under 5% of the resources were utilized, even with 24 instances. Because there is only a single database server, it is important to use a powerful server if a cluster with many instances is planned. It is also important to have a large number of disks and to configure the data and logs to reside on separate disks.

Page 23: IBM Rational Asset Manager Version 7.0.0.2

IBM Rational Asset Manager capacity and scalability benchmark, Page 23 of 33

Copyright © 2008, IBM® Corporation. All rights reserved. Published by IBM® developerWorks®.

Figure 18. Database scaling

Finally, storage requirements are an important factor to consider when deciding on needed hardware. The three major storage points required for Rational Asset Manager are asset files, the index, and the database. In a clustered environment, there will be only one asset storage location and one database storage location. A single index storage location will exist on each application server. The size of the index is dependent on both the size of the assets and whether the content of the assets is mostly text that can be indexed or binary content that cannot be indexed. This is described by Table 4. The index size will generally grow in a logarithmic fashion.

Table 4. Index storage requirements

Small asset files Large asset files

Small amounts of text in asset files

Small Medium

Large amounts of text in asset files

Medium Large

The size of the database is dependent on the size of asset descriptions and other repository activity, such as forums, tagging, and registered users. Metrics are also recorded in the database for many user activities, such as searching and downloading. Therefore, even if no new assets are being added to the repository, the database will grow over time as user activity metrics are recorded. The size of the database will generally grow in a linear fashion. although it is dependent on the repository use.

Page 24: IBM Rational Asset Manager Version 7.0.0.2

IBM Rational Asset Manager capacity and scalability benchmark, Page 24 of 33

Copyright © 2008, IBM® Corporation. All rights reserved. Published by IBM® developerWorks®.

Although some large assets were included, the average size of an asset in the test environment repository was less than 1KB. If you plan to have larger assets, this should be accounted for when considering the storage requirements for the Persist and Index folders. The database size will not be significantly affected by the size of the assets. Figure 19 shows the storage requirements from the benchmark.

Figure 19. Storage requirements

Page 25: IBM Rational Asset Manager Version 7.0.0.2

IBM Rational Asset Manager capacity and scalability benchmark, Page 25 of 33

Copyright © 2008, IBM® Corporation. All rights reserved. Published by IBM® developerWorks®.

Appendix: Test results

The test results described in this section are not a replacement for testing Rational Asset Manager performance in your own environment. Performance and scalability depend on a large number of factors, including but not limited to: hardware setup and tuning, operating system tuning, software tuning, network traffic and latency, end user use models, and the number of assets stored in Rational Asset Manager. It is impossible to cover every single factor affecting performance that exists in the test environment. Therefore, these results are not guaranteed, and it is expected that you will achieve different results in your own environment. This data is intended simply to provide a point of reference from which to begin your own environment design. All tests follow the format described in the User simulation section of this document. Ramp-up time for each test is number of users / 2 seconds. Each test was then driven for an additional 10 minutes with the full user load. Thus these results should not be considered maximum or sustained values. Instead, they represent user loads that can be driven over a short period of time. The average response time for all tests was kept below 450 ms. See the Performance guidelines section for information concerning the impact on the average response time with runs of longer durations.

Test matrix Table 5. Benchmark results

Assets Users as defined in the User Simulation section

Nodes Instances per node

Response time average in milliseconds

Page hits total

Page/sec. maximum observed

Total run minutes

100,000 400 1 1 244 3,000 5 14

100,000 700 1 2 250 5,500 7 16

100,000 1,200 1 4 415 11,100 15 20

100,000 1,200 3 1 279 11,200 15 20

100,000 2,100 3 2 340 26,000 22 28

100,000 3,300 3 4 319 51,500 40 38

100,000 2,300 6 1 431 30,000 23 30

100,000 3,500 6 2 218 57,600 40 40

100,000 6,000 6 4 250 136,000 70 60

300,000 400 1 1 258 3,000 5 14

300,000 700 1 2 311 5,600 7 16

300,000 1,100 1 4 352 10,500 13 20

300,000 1,200 3 1 331 11,200 15 20

300,000 2,000 3 2 274 24,100 22 27

300,000 3,300 3 4 345 51,500 35 38

300,000 2,100 6 1 245 26,100 23 28

Page 26: IBM Rational Asset Manager Version 7.0.0.2

IBM Rational Asset Manager capacity and scalability benchmark, Page 26 of 33

Copyright © 2008, IBM® Corporation. All rights reserved. Published by IBM® developerWorks®.

Assets Users as defined in the User Simulation section

Nodes Instances per node

Response time average in milliseconds

Page hits total

Page/sec. maximum observed

Total run minutes

300,000 3,500 6 2 257 57,500 40 40

300,000 6,000 6 4 283 136,300 70 60

1,000,000 400 1 1 358 3,000 5 14

1,000,000 700 1 2 418 5,600 7.5 16

1,000,000 950 1 4 399 8,100 10 18

1,000,000 950 3 1 284 8,000 10 18

1,000,000 1,900 3 2 342 22,100 22 26

1,000,000 3,000 3 4 435 43,300 32 35

1,000,000 2,000 6 1 270 24,100 23 27

1,000,000 3,300 6 2 276 51,500 35 38

1,000,000 5,500 6 4 368 114,600 60 56

2,000,000 400 1 1 407 3,000 5 14

2,000,000 700 1 2 400 5,600 8 16

2,000,000 800 1 4 381 6,600 9 17

2,000,000 800 3 1 309 6,600 9 17

2,000,000 1,700 3 2 386 19,600 21 25

2,000,000 2,700 3 4 437 37,600 31 33

2,000,000 1,900 6 1 346 22,200 22 26 2,000,000 3,100 6 2 357 45,800 35 36

2,000,000 5,000 6 4 400 100,500 59 52

User scalability

The number of users that can be driven in a configuration is affected by the number of application server nodes and instances, as well as the number of assets within the repository. The number of assets has an impact in the tests because it is assumed that a repository with a large number of assets will result in users performing search queries that return a larger number of results. In this benchmark, 15% of the users performed searches that returned (number of assets in the repository / 15) results. Figure 20, Figure 21, Figure 22, and Figure 23 show the maximum number of users driven in each configuration with the different asset levels that were tested. Results are fairly comparable between 100,000 and 300,000 assets, with a slight drop off between 300,000 and 1,000,000 assets and with another between 1,000,000 assets and 2,000,000 assets.

Page 27: IBM Rational Asset Manager Version 7.0.0.2

IBM Rational Asset Manager capacity and scalability benchmark, Page 27 of 33

Copyright © 2008, IBM® Corporation. All rights reserved. Published by IBM® developerWorks®.

Figure 20. User scalability with 100,000 assets

Figure 21. User scalability with 300,000 assets

Page 28: IBM Rational Asset Manager Version 7.0.0.2

IBM Rational Asset Manager capacity and scalability benchmark, Page 28 of 33

Copyright © 2008, IBM® Corporation. All rights reserved. Published by IBM® developerWorks®.

Figure 22. User scalability with 1,000,000 assets

Figure 23. User scalability with 2,000,000 assets

Page 29: IBM Rational Asset Manager Version 7.0.0.2

IBM Rational Asset Manager capacity and scalability benchmark, Page 29 of 33

Copyright © 2008, IBM® Corporation. All rights reserved. Published by IBM® developerWorks®.

Throughput scalability

Because the usage model for a “user” can vary widely, it may be preferable to use another measure as a definition of load capacity. The maximum observed page hits per second in each of the 9 configurations and 4 different asset levels can be used to derive and apply a different model for a user to these benchmark results. Figures 24, 25, 26, and 27 show that throughput is similar to user capacity for each configuration and repositories with increasing numbers of assets.

Figure 24. Page throughput with 100,000 assets

Page 30: IBM Rational Asset Manager Version 7.0.0.2

IBM Rational Asset Manager capacity and scalability benchmark, Page 30 of 33

Copyright © 2008, IBM® Corporation. All rights reserved. Published by IBM® developerWorks®.

Figure 25. Page throughput with 300,000 assets

Figure 26. Page throughput with 1,000,000 assets

Page 31: IBM Rational Asset Manager Version 7.0.0.2

IBM Rational Asset Manager capacity and scalability benchmark, Page 31 of 33

Copyright © 2008, IBM® Corporation. All rights reserved. Published by IBM® developerWorks®.

Figure 27. Page throughput with 2,000,000 assets

Page 32: IBM Rational Asset Manager Version 7.0.0.2

IBM Rational Asset Manager capacity and scalability benchmark, Page 32 of 33

Copyright © 2008, IBM® Corporation. All rights reserved. Published by IBM® developerWorks®.

Trademarks

IBM, the IBM logo, and Rational are trademarks of IBM Corporation in the United States, other countries, or both. Java and all Java-based trademarks are trademarks of Sun Microsystems, Inc. in the United States, other countries, or both. Other company, product, or service names may be trademarks or service marks of others. Share this article

Digg this story

Post to del.icio.us

Slashdot it!

Resources

Learn

• Publications and products mentioned in this article:

o Rational Asset Manager 7.0 Capacity Planning Guide1

o Rational Asset Manager Tuning Guide

o IBM System Storage DS4800

o IBM System P Enterprise Servers

• Check out the following Rational Asset Manager resources:

o IBM Rational Asset Manager V7.0.0: Capacity planning and configuration guide

o Rational Asset Manager demos

o Maximize reuse of your software assets with Rational Asset Manager, Part 1

o Maximizing reuse of your software assets with Rational Asset Manager, Part 2

o Federated metadata management with IBM Rational and WebSphere software

• In the Architecture area on developerWorks, get the resources you need to advance your skills in the architecture arena.

• Visit the Rational software area on developerWorks for technical resources and best practices for Rational Software Delivery Platform products.

• Subscribe to the developerWorks Rational zone newsletter. Every other week, you'll receive updates on the latest technical resources and best practices for the Rational Software Delivery Platform.

• Subscribe to the Rational Edge newsletter for articles on the concepts behind effective software development.

• Subscribe to the IBM developerWorks newsletter, a weekly update on the best developerWorks tutorials, articles, downloads, community activities, webcasts and events.

• Browse the technology bookstore for books on these and other technical topics.

Get products and technologies

• Download trial versions of IBM Rational software.

• Download IBM product evaluation versions and get your hands on application development tools and middleware products from DB2®, Lotus®, Rational®, Tivoli®, and WebSphere®.

Discuss

• Rational Asset Manager forum: Ask questions about Rational Asset Manager.

• Check out developerWorks blogs and get involved in the developerWorks community.

Page 33: IBM Rational Asset Manager Version 7.0.0.2

IBM Rational Asset Manager capacity and scalability benchmark, Page 33 of 33

Copyright © 2008, IBM® Corporation. All rights reserved. Published by IBM® developerWorks®.

About the authors Dr. Mendel is the technical lead and manager for IBM Rational Asset Manager software. Sheehan Anderson is a Rational Asset Manager software developer. John Reinstrom is the Chief Performance Test Architect for Rational system and performance tests. Amy Pitts is a software engineer for Rational system and performance tests. Skye Bischoff is a Software Test Specialist on the IBM Rational Performance Tester SVT team, focused on improving customer usability for Rational Performance Tester, Rational SOA Tester, and other Rational products. He has performance-tested IBM Rational Asset Manager in various clustered environments to ensure that it performs quickly, scales, and is resilient. Before working at IBM, Skye worked as a Flash designer and developer at T8Design. He received his B.S. in Computer Science from the University of Northern Iowa. Jennifer Chang is a Software Performance Test Engineer on the IBM Rational Performance Tester SVT team, working on performance testing IBM Rational Asset Manager in clustered and standalone environments. She received both her M.S. in Computer Science and M.A. in Economics from Tufts University, in Medford, Massachusetts. Bryan Miller is a Service Offering Development Lead for the Rational Brand Services team.