SAP CC Tuning Essentials v1 1

7/23/2019 SAP CC Tuning Essentials v1 1

http://slidepdf.com/reader/full/sap-cc-tuning-essentials-v1-1 1/39

SAP Convergent Charging

Performance and Tuning Essentials

SAP Convergent Charging 2.0/3.0

Version 1.1

21/09/2012

Nick Milani

[email protected]

Internal Only – Do not re-distribute

mailto:[email protected]





Version History

Version Author Status Date (YYYY-MM-DD)Change

Description

1.0 Nick Milani Released 20/09/2012 Initial version

1.1 Nick Milani Released 21/09/2012 Minor corrections

References

Title Version Link

SAP CC 2.0 Officialdocumentation

Various http://help.sap.com/cc20

SAP CC 3.0 Officialdocumentation

Various http://help.sap.com/cc30

SAP CC Solution SpecialistWiki

N/A https://wiki.wdf.sap.corp/wiki/display/FSCC/

SAP CC – Oracle best practices N/A https://wiki.wdf.sap.corp/wiki/display/FSCC/Oracle+Configuration+Guidelines

SAP Note: SAP CC – JVMrecommendations

6https://websmp130.sap-

ag.de/sap(bD1lbiZjPTAwMQ==)/bc/bsp/spn/sapnotes/index2.htm?numm=1536317

ARTIX Wiki N/Ahttps://wiki.wdf.sap.corp/wiki/display/FSCC/ARTIX+for+Convergent+Charging+

Acceptance+and+Regression+Tests

http://help.sap.com/cc20




https://wiki.wdf.sap.corp/wiki/display/FSCC/


https://wiki.wdf.sap.corp/wiki/display/FSCC/Oracle+Configuration+Guidelines


https://websmp130.sap-ag.de/sap(bD1lbiZjPTAwMQ==)/bc/bsp/spn/sapnotes/index2.htm?numm=1536317




https://wiki.wdf.sap.corp/wiki/display/FSCC/ARTIX+for+Convergent+Charging+Acceptance+and+Regression+Tests














Page 3 of 39 Copyright/Trademar

Table of Contents

1 Introduction .......................... .......................... ......................... .......................... .......................... ........................... . 5

2 Installation and Configuration Best Practices ......................... ......................... .......................... ......................... ....... 62.1 Hardware Sizing ..................................................................................................................................................... 6

2.2 SAP CC Installation ................................................................................................................................................. 7

2.2.1 Operating System ............................................................................................................................................ 7

2.3 SAP CC price plan design and offer configuration .................................................................................................... 8

2.3.1 Counters .......................................................................................................................................................... 8

2.3.2 Subscription sizes ............................................................................................................................................ 8

3 Performance Tuning ........................... .......................... ......................... .......................... ......................... ...............10

3.1 Under to hood of the SAP CC rating process ..........................................................................................................10

3.2 Cache overview .....................................................................................................................................................12

3.3 Instance Parameters ..............................................................................................................................................12

3.3.1 Exporting parameters from SAP CC server ......................................................................................................12

3.3.1 Performance related parameters ....................................................................................................................13

3.3.2 Verifying Instance parameter settings .............................................................................................................18

3.4 JVM Options ..........................................................................................................................................................19

3.4.1 Java Memory Management Overview .............................................................................................................193.4.2 Jstart.config ....................................................................................................................................................20

3.5 Database Tuning....................................................................................................................................................23

3.5.1 Why tune the database? .................................................................................................................................23

3.5.2 SAP CC Database Tuning Best Practices ...........................................................................................................25

3.5.3 Database Disks ...............................................................................................................................................26

3.5.1 Network infrastructure and SAN .....................................................................................................................28

3.5.2 HDD vs. SSD ....................................................................................................................................................28

3.5.3 In Memory Databases .....................................................................................................................................29

4 Measuring Performance ......................... .......................... .......................... ......................... .......................... ..........30

4.1 SAP CC Client overview ..........................................................................................................................................30

4.2 Testing tools ..........................................................................................................................................................30

4.2.1 Load Test Tool ................................................................................................................................................30




4.2.2 ARTIX ..............................................................................................................................................................31

4.2.1 BART...............................................................................................................................................................32

4.2.1 IEC ..................................................................................................................................................................32

4.2.2 Third party tools .............................................................................................................................................33

5 Appendices ........................... .......................... ......................... .......................... .......................... ...........................34

5.1 Benchmarks referenced in this document..............................................................................................................34

5.2 Hardware used for Benchmark Ref #1 ...................................................................................................................35

5.3 Test cases executed for Benchmark Ref #1 ............................................................................................................35

5.4 Raw results of tests carried out for Benchmark Ref #1 ...........................................................................................36

5.5 Hardware used for Benchmark Ref #2 ...................................................................................................................37

5.6 Test cases executed for Benchmark Ref #2 ............................................................................................................38

5.7 Raw results of tests carried out for benchmark Ref #2 ...........................................................................................39

6 Index of figures and tables .......................... ......................... .......................... ......................... .......................... ......39

6.1 Figures ..................................................................................................................................................................39

6.2 Tables....................................................................................................................................................................39




1 Introduction

The purpose of this document to provide guidelines and best practices to SAP Technical Consultants to who wish to

understand the performance and tuning capabilities of SAP Convergent Charging (SAP CC). This document aims to

explain the principles behind the most important performance tuning topics and explain how each can be addressed.

For many SAP CC installations, in depth performance tuning will simply not required because most default configuration

options will be sufficient for obtaining the required performance.

This document covers information related to SAP CC Core Server 2.0 and 3.0:

Sizing and Installation best practices

SAP CC price plan design and offer configuration SAP CC application performance tuning

JVM performance tuning

Database performance tuning

Methods for measuring performance

This document does not cover:

C2C and O2C performance tuning (SAP CI, SAP CRM)

BART tuning




2 Installation and Configuration Best Practices

Selecting the correct hardware and installing SAP Convergent Charging with the optimal configuration is paramount to

gaining the most out of a platform. If the sizing and installation is done the right way it will often not be necessary to

investigate any other performance tuning topics covered in this document.

2.1 Hardware Sizing

The first thing that needs to be done is sizing the platform. Luckily some detailed documents exist to assist in this

process.

The official sizing guidelines can be located on line in the official SAP CC documentation website. Taking inputs such as

the number of customers, online vs. offline traffic, and the number of rating events per day, the sizing guidelines will

recommend:

Server specifications (incl. CPU, RAM)

Number of servers

What SAP CC instances should/should not be installed on the same machine

Database configuration

Database disk sizes

You may notice that the sizing guidelines stop at a certain point when the number of customers or throughput is too

high. When your requirements are outside the known guidelines it is recommend that you consult SAP support.

It must also be said that at the time of writing, these guidelines are quite conservative and were released before major

performance improvements were gained in the second half of 2011. The documents also do not directly address IOPsrequirements (discussed in 0).

Nevertheless, it is important to consult these documents carefully and arrive with a platform that is going to comfortably

suit the performance needs now (and presumably several years into the future if growth is expected).




2.2 SAP CC Installation

This is not an installation guide. The official installation guide is available online from SAP.

There is also a useful guide on how to quickly install SAP CC 2.0 here on the CC Solution Specialists wiki. Most of the

steps are the same for CC 3.0. Additional resources can also be found within this same wiki.

Before installing, determine the sizing requirements and follow the recommendations in the sizing guide. Pay particularattention to what CC instances should be separated and which ones can be put on the same machine. The reasons for

these recommendations are simple – some instances have more resource requirements than others.

Consider the following table of a high performance benchmark1 and the amount of CPU and RAM that each instance was

consuming during a typical test:

Rating Test Dispatcher Guider Rater

CPU 4-7% 4-7% 40-60%

RAM 4GB 23GB 28GBTable 1 - CPU and RAM consumption during a high performance rating test

Provisioning Test Updater

CPU 80%

RAM 4GBTable 2 - CPU and RAM consumption during a high performance provisioning test

You can clearly see that during rating the Raters consume the most CPU and RAM. This is not surprising as the Raters

cache all the customer subscriptions and perform the rating on each transaction. The Guider also has a cache of accesses

which is why it is assigned a lot of RAM as well. The other instances do not have large caches. Because of their CPU

requirements, the Raters are recommended to be placed on their own machine or with another process that consumes

little resources. Consequently, Dispatchers and Guiders are often paired together as they don’t consume as many

resources.

BulkLoader instances were not measured as part of the above benchmarks. However, because they must work closely

with the Raters and don’t consume a lot of CPU they are recommended to be paired on the same machine as the Raters

(1 Bulkloader is required for every Rater).

2.2.1 Operating System

SAP CC is certified to work with a number of different operating systems. Most high performance projects and

benchmarks have been conducted with servers running Unix/Linux variations such as AIX and Redhat (both officially

supported with SAP CC). SUSE Linux, although officially unsupported, has also been used with several successful

benchmarks.

Operating systems themselves can be tuned to work optimally the various components of a SAP CC system (Java,

Database etc.) but these details are not covered in this document.




2.3 SAP CC price plan design and offer configuration

2.3.1 Counters

By far the biggest performance impact on non-session based rating performance is the number of counter updates per

transaction.

Consider the results from the Benchmark completed in 20112

Figure 1 - Benchmark results: TPS vs. counter updates

You can see that with 0 counter updates per transaction, SAP CC achieved almost 800,000 TPS but dropped to around

325,000 with just one counter update! Ultimately this performance hit needs to be addressed (HANA anyone?) but in

the meantime counters should be used sparingly whenever possible.

NOTE: Prepaid rating scenarios will usually result in a monetary balance in the subscriber account being modified. This

will have at least the impact of a counter update.

2.3.2 Subscription sizesThe size of a subscription can also dramatically impact the time it takes to rate events. Before rating can occur, a rater

must build a rating context which involves looking through the subscription and picking out the right price plan to use

for the event.

The following diagram shows results of a benchmark in 20111 where the aim was to see the maximum TPS CC could

process for a single subscription. When rating a single subscription, CC cannot process EDRs asynchronously as a

subscription object is locked while an event is being rated against it.

T P S

Counter updates per transaction




Figure 2 - TPS for a single subscription

As illustrated, CC could only process 194 TPS for the complex subscription (which was approximately 75k in memory).

That is 30 times less than the simple 2k subscription!

Another point of note is that when rating many complex subscriptions the Rater’s CPU is utilized much more (because it

has to work harder building rating contexts for these larger objects). In a test similar to the above, when 1000 complex

subscriptions were rated at the same time, the CPU was peaking at 100% compared with 40% during standard tests.

Ultimately this caused the major bottleneck for the test which was not the case in any other rating performance tests.

Other configuration objects can affect the size of the objects in memory for rating; in particular translation and tier

table sizes. Where possible these tables should be small i.e. under 1000 rows to keep the size of the subscription

smaller.

To measure the size of a subscription, open it using the desktop tool and save it to file. Look at the properties of the

generated file to reveal the “Size” value which represents the size that the file will be in memory.




3 Performance Tuning

If the sizing guidelines have been followed, the system has been configured as per the available recommendations but

the performance is not satisfactory, there are still additional options available to squeeze more performance out of the

system.

The following sections explain how SAP CC works internally and the options for trying to exploit better performance

from the system.

Warning: Changing the default parameters can potentially make the system unstable. Any parameter changes must be

done in a controlled environment and thoroughly tested. Never change parameters on a live production system without

having tested the modifications first. You have been warned!

3.1 Under to hood of the SAP CC rating process

Before discussing how the system can be tuned it is important to understand how SAP CC works internally.

The following diagram shows the interactions between the SAP CC instances during the rating process for a CDR:

Figure 3 - SAP CC instance interaction during rating

1) Rating Client calls a SAP CC Dispatcher with Chargeable Item details.

2) Dispatcher (using an algorithm on the User ID and Service ID) contacts the Guider that is responsible for the partition

that the access is in.

3) Guider looks up details and sends partition ID and subscription ID corresponding to the access.

4) Dispatcher, (using it’s map of Raters to partitions) sends the Subscription ID and Chargeable Item to the Rater

responsible for the subscription

5) Rater loads subscription from cache, builds rating context and rates Chargeable Item. It then returns result todispatcher

6) Dispatcher returns result to client application.

Note: After step 4, the Bulk Loader instance is able to pick up rated transactions and load them into the billing system.

This process occurs asynchronously.




At this level things appear to be fairly simple. However, EDRs are not rated one-by-one. This would be far too slow and

you would never be able to reach 1,000’s of transactions per second. Instead multi-threading and message queues are

used to enable the system to process more EDRs at the same time. A single thread is likely to only consume a small

amount of CPU so adding more threads enables the system to consume more and make better use of the available

hardware.

The following diagram which is a more detailed look at the inner workings of SAP CC during the rating process:

Figure 4 - Multithreading rating process steps

NOTE: The number of threads (4) shown in this diagram are an example only and is not necessarily the default nor recommended

values.

Each SAP CC instance can be assigned a number of processing threads and several queue sizes to control the number of

EDRs it is processing at any given time. Finding the right balance between these parameters is crucial to obtaining

maximum performance. If the queue sizes and number of threads are too small, you are likely to obtain really good

latency but not enough overall throughput. Conversely, if they are too large your latency will be horrendous and the

throughput is likely to be less than if you had set the parameters optimally.




3.2 Cache overview

SAP CC has three different caches which must be sized according to the volume of data the system will process (for

example the number of subscriptions). This is the first stop when tuning the environment and in many cases will be

enough to get the required performance.

If these caches are not sized large enough, SAP CC will not be able to cache all the required objects in memory causing

lots of DB reads which will significantly reduce performance. Other caches such as the offer-level cache exist but they

are automatically sized.

The following table gives the names of the caches, the instances using them and the objects the caches handle. The

parameters that control these cache sizes and how the required cache sizes can be calculated are explained in

subsequent sections (along with the other performance related parameters).

Cache Instance(s) Objects in the cache

Subscription Cache Rater(s)

Updater(s)

Subscriptions

Subscriber accounts

Counters

Guiding Cache Guider(s) Accesses

Session Cache Rater(s) Rating sessions (if session-based

rating is being used)

Note that each rater is in charge of its own subscriptions and its own sessions. So the number of objects in the

subscription cache and in the session cache depends on the number of raters in your platform. On the contrary, all the

accessible charges must be cached for each guider and all the account information must be cached for each charger.

The amount of objects in the cache and the free space available can be monitored with Admin+ (see official CC

documentation).

3.3 Instance Parameters

Many parameters in SAP CC can be configured. Some of them can be used to tune the performance of the SAP CC

environment and are explained in the following tables. Most gains can be achieved by correctly setting the most

important ones ( ‘10’ in the table).

3.3.1 Exporting parameters from SAP CC server

To export parameters and change in a CC environment you can follow the following steps (example is on Linux OS):

Start a command line session in the SAP CC Dispatcher’s script directory.

Export the parameters to a file: ./config.sh configuration export -login=xxxxx -

password=xxxxx parameters.xml all

Change required parameters




Re-import the file: ./config.sh configuration import -login=xxxxx -password=xxxxx

parameters.xml

Restart the servers

For more information and command information please consult the official SAP CC guides. In particular, there is a

Configuration Parameter Reference guide (paramref) that explains all the SAP CC configuration parameters in detail.

3.3.1 Performance related parameters

The following tables display the performance related parameters present in every CC instance, the default values,

description (from the paramref documentation) and notes detailing how to arrive at the optimal value for the

parameter.

Table 3 - Rater instance parameters

Importance (1

lowest-10

highest)

Parameter Name Default Description Notes

10 for recurring

fee performance

5 for rating

performance

ACTIVATION_THREA

D_COUNT

1 This parameter allows the user to

set the number of threads per rater

dedicated to the recurring and one-

shot rate activation.

This parameter does not affect rating but recurring f ee activation.

Recommendation for best performance is 1-3 times the number o

CPU cores assigned to Raters.

Limiting the number of threads will ensure that recurring fee

activation does not interfere with normal rating process.

2 AUDIT_ACTIVATION true This parameter enables or disables

the audit function.

If deactivated, CC will not write to DB for each action.

Not recommended for production.

10 (Only required

when using

BART)

BATCH_SERVICE_TH

READ_COUNT

4 This parameter allows setting the

number of threads reserved for

performing the batch service into

each rater.

The Batch service is used by BART.

Increasing this will usually enable the system consume more CPU.

Best performance is usually gained by this value being 1 -2 times

the number of cores of the available cores on the machine.

Setting it too high could overload CPU.

10 (Only required

when using

BART)

BATCH_SERVICE_QU

EUE_SIZE

100 This parameter defines the

maximum number of messages

that can be queued in the rater for

the batch service.

Should be larger than your maximum throughput target.

Can be set very large as long as memory does not become an

issue.

10 (Only required

when using

BART)

BATCH_SERVICE_RE

QUEST_BATCH_SIZE

10 Sets the batch size per thread. Increasing this will allow the

processes to batch more when needed.

Will increase latency but can improve throughput.

This value should be smaller than Guider the guider batching size

(to ensure that the batching will occur)

10 (Only required

for session based

rating)

SESSION_MEMORY_

SIZE

25M This parameter defines the total

size of the memory dedicated to

the storage of rating sessions foreach rater.

Must be at least average number of s essions x average session

size. Use Admin+ to investigate free space in the cache.

10 (Only required

for session based

rating)

SESSION_MEMORY_

INSTANCES

Computed The session cache is divided into

smaller sub caches for reducing

contention in the cache. This

parameter sets the maximum

number of sub caches into the

session cache.




This parameter is defaulted to the

first prime number greater than 8

times the number of CPUs. In

general, there is no need to modify

it.

10 SQLHELPER_CONNE

CTION_COUNT

20 This parameter gives the number of

connections to open with thedatabase.

Its value must be correlated to the

total number of threads running in

each instance. If the number of

connections to the database is too

small compared to the total

number of threads running in the

instance, the threads are not able

to render the service because they

do not find any available

connections to perform their

database requests.

Must be the total of all the threads being used by the rater

10 SQLHELPER_DB_INS

TANCE_COUNT

1 This parameter sets the number of

instances in case of multi-instancedatabase architecture.

To be relevant, the value must be

superior to 2 and the number of

raters must be a multiple of the

SQLHELPER_DB_INSTANCE_COUNT

parameter.

For example, if the SQLHELPER_DB_INSTANCE_COUNT = 2 and if N

is the number of rating instances, the N/2 first instances will beconnected to the first instance in SQLHELPER_JDBC_URI and the

N/2 last rating instances to the second one.

6 SQLHELPER_ID_INTE

RVAL

10000 When an instance creates an

object, it asks the DB to indicate a

correct identifier for the newly

created object. Because this

operation is very frequent, the

database provides a pool of

identifiers to the instance in order

to limit DB SQL request and

increase performances.

The SQLHELPER_ID_INTERVAL

parameter represents the size of

the pool of identifiers.

All the identifiers belonging to the

pool are considered as consumed

by the database. So if the instance

is stopped, all the identifiers

already requested and not yet used

by the instance are lost.

Should be set to 10x your optimal throughput

7 STATEFUL_SERVICE_

QUEUE_SIZE



that can be queued in the instance

for the stateful services.


Can be set very large as long as memory does not become an

issue.

10 STATEFUL_SERVICE_

REQUEST_BATCH_SI

ZE

10 This parameter defines the batch

size of request processed by one

thread of the stateful services.

Sets the batch size per thread. Increasing this will allow the

processes to batch more when needed.

Will increase latency but can improve throughput.

This value should be smaller than Guider the guider batching size

(to ensure that the batching will occur)

10 STATEFUL_SERVICE_

THREAD_COUNT


set the number of threads reserved

for performing the stateful services

Very important for rating performance for all scenarios that don’t

use the BATCH API.




into each instance. Increasing this will usually enable the system consume more CPU.

Best performance is usually gained by this value being 1 -2 times

the number of cores of the available cores on the machine.

Setting it too high could overload CPU.

10 SUBSCRIPTION_CAC

HE_SIZE


size reserved for caching

subscription information into each

instance.

Syntax: This size is defined by a

positive integer (mandatory) and a

unit (optional). The available units

are: 'K', 'M', 'G' for Kilo, Mega, Giga.

If no unit is specified, the default

unit is "bytes".

Note: Do not put any space

between the size and its unit.

Needs to be big enough so that the raters can cache all the active

customer subscriptions/contracts in the system.

Subscription sizes can be measured by saving them to XML and

looking at the file size.

9 SUBSCRIPTION_CAC

HE_INSTANCES

computed The subscription cache is divided

into smaller sub caches for

reducing contention in the cache.

This parameter sets the maximum


subscription cache.





it.

‘Computed’ is not recommended for high performance.

For best performance each instance should be small (1-2mb). So,

to calculate the instances required, divide the subscription cache

size by 1-2.

Table 4 - Guider instance parameters

Importance (1

lowest-10

highest)


2 AUDIT_ACTIVATION See above

9 GUIDING_CACHE_INSTANCES

computed

The guiding cache is divided intosmaller sub caches for reducing

contention in the cache. This

parameter sets the maximum


guiding cache.




general, there is no need to modify.

‘Computed’ is not recommended for high performance.

For best performance each instance should be small (1-2mb). So, to

calculate the instances required, divide the subscription cache size

by 1-2.

10 GUIDING_CACHE_SIZE 25M This parameter defines the total

size reserved for caching guiding

information into each guider.






unit is "bytes".



Examples: 116856, 2400K, 5M, 1G.

Needs to be big enough so that the raters can cache all the active

customer accesses in the system. These objects are very small so

the Guider cache does not usually need to be as big as the Rater

cache.

10 SQLHELPER_CONNECT As above See above




ION_COUNT

10 SQLHELPER_DB_INSTA

NCE_COUNT

As above See above

10 SQLHELPER_ID_INTER

VAL

As above See above

7 STATEFUL_SERVICE_Q

UEUE_SIZE

100 See above

10 STATEFUL_SERVICE_R

EQUEST_BATCH_SIZE

10 See above Batch size should be 2-3 times the size in the Rater

10 STATEFUL_SERVICE_T

HREAD_COUNT

4 See above Best value is approximately half of the available cores on the Guider

machines

Table 5 - Dispatcher instance parameters

Importance (1

lowest-10

highest)



7 DISPATCHER_SERVER_

QUEUE_SIZE



that can be queued in the instance

for the dispatcher service.

Should be large enough to handle all the EDRs across all the raters

in the environment

10 (Onlyrequired when

using BART)

BATCH_SERVICE_QUEUE_SIZE

100 This parameter allows setting thenumber of threads reserved for

performing the batch service into

each rater.


Can be set very large as long as memory does not become an issue.

10 (Only

required when

using BART)

BATCH_SERVICE_REQ

UEST_BATCH_SIZE



that can be queued in the rater for

the batch service.

Batch size should be 2-3 times the size in the Guider

10 (Only

required when

using BART)

BATCH_SERVICE_THRE

AD_COUNT

4 Best value is approximately half of the available cores on the

machine

9 GUIDER_RESPONSE_H

ANDLER_BATCH_SIZE


size of response processed by one

thread of the guider response

services.

Maximum batch size of messages being handled by a single thread

at a time. Batches will not be filled if the system throughput is not

generating enough responses in queue

Should be the similar to the STATEFUL_SERVICE_

REQUEST_BATCH_SIZE value

Increasing the value will cause each thread to use CPU for longer

periods of time and increase latency.

If set too low, you will have more locking time on the queue as

threads keep picking up EDRs

Will often require some trial and error to find right value.

9 RATER_RESPONSE_HA

NDLER_BATCH_SIZE


size of response processed by one

thread of the rater response

services.

See GUIDER_RESPONSE_HANDLER_BATCH_SIZE

10 SQLHELPER_CONNECT

ION_COUNT

See above

10 SQLHELPER_ID_INTERVAL

See above

10 STATEFUL_SERVICE_Q

UEUE_SIZE

See above

10 STATEFUL_SERVICE_R

EQUEST_BATCH_SIZE


size of request processed by one

thread of the stateful services.

Batch size should be 2-3 times that of the Guider's

10 STATEFUL_SERVICE_T

HREAD_COUNT


set the number of threads reserved

for performing the stateful services

into each instance.

Best value is approximately half of the available cores on the

machine




Table 6 - Updater instance parameters

Importance (1

lowest-10

highest)


2 ACTIVATION_SCHEDU

LER_ENABLED

true This parameter enables or disables

the automatic activation ofrecurring rates and one-shot rates.

When the

ACTIVATION_SCHEDULER_ENABLE

D parameter is set to true, a

scheduler is activated into the

updater. This scheduler wakes up

periodically according to the value

of the

ACTIVATION_SCHEDULER_RECURR

ENCE parameter.

Please note that recurring and one-

shot rates are still triggered when

usage events are rated, even if the

ACTIVATION_SCHEDULER_ENABLED parameter is set to false.

Can be turned off for benchmarks. Usually left on in production.


10 (for

provisioning

performance

only)

HTTP_SERVER_THREA

D_COUNT

4 Increasing this will allow the process to consume more CPU

1-4 times the number of cores

10 SQLHELPER_CONNECT

ION_COUNT

See above

10 SUBSCRIPTION_CACHE

_INSTANCES

compute

d

The subscription cache is divided

into smaller sub caches for

reducing contention in the cache.

This parameter sets the maximum


subscription cache.





it.

Cache instances size should be at least large enough to hold the

biggest subscriptions but the number of caches should be numerous

enough to avoid contention

10 SUBSCRIPTION_CACHE

_SIZE


size reserved for caching

subscription information into each

instance.






unit is "bytes".



Examples: 116856, 2400K, 5M, 1G.

Should be large enough to handle all the subscriptions of the largest

business operations you plan to do (i.e. bulk modifications)

Subscriptions are not stored in the Updater for a l ong time

5 XML_SCHEMA_VALID

ATION

true This parameter enables or disables

the xml schema validation (default

is enabled).

Turn it off for better performance




3.3.2 Verifying Instance parameter settings

It is recommended that any changes in the parameters are tested in a methodical and repeatable fashion. Avoid

changing too many parameters between each test. Ensure that the environment and test cases are kept similar between

tests so that parameter changes can be correctly evaluated.




3.4 JVM Options

JVM options can be set in the jstart.config file of each CC instance. Modifying this file will be most useful for tuning

Rater performance but it can also be useful in tweaking other instances.

3.4.1 Java Memory Management Overview

As any Java application is running, objects are created, used and eventually destroyed. When objects are created, they

take up some memory and the memory remains allocated while there are references for the use of the object. When

there are no references for an object, it is assumed to be no longer needed and the memory occupied by the object can

be reclaimed.

There is no explicit need to destroy an object as java handles the de-allocation automatically. The technique that

accomplishes this is known as Garbage Collection (GC).

Programs that do not de-allocate memory can eventually crash when there is no memory left in the system to allocate.

These programs are said to have memory leaks.

Each time Java performs a Garbage Collection the Java Virtual Machine (JVM) will pause momentarily. Obviously you

want to avoid this happening too frequently otherwise it will have a negative impact on the application’s performance.

Figure 5 - Java application profiling showing regular garbage collections

The default size of the JVM is 256Mb which is sufficient for many applications. But for applications such as SAP CC where

intense processing occurs, 256Mb is often not nearly enough. To tune the system performance it is possible to allocate

more memory the JVM and also control the way garbage collection works.

SAP CC 2.0 and 3.0 use SAP JVM 6.0.




3.4.2 Jstart.config

SAP note 1536317 contains recommendations on modifications that can be made to SAP CC’s JVM settings. If you follow

those recommendations and still feel that there are issues with the JVM, more advanced settings can be added as

described in this section.

The location of the jstart.config files will be: <SAP CC installation path>\<System ID>\<Instance iD>\config.

After SAP CC 2.0 SP9 (including SAP CC 3.0 and beyond) these are located in the SYS folder of the installation. E.g.<SAP

CC installation path>/<System ID>/</SYS/profile/jstart.

A separate file exists for each CC instance.

A typical file belonging to a Rater instance will look like the following. The parameter that allows you to set the JVM

options is highlighted.

rater-1.Id = 1rater-1.Name = rater-1rater-1.Type = java

rater-1.rootPath = C:/usr/sap/CC2/CCR01/exe/CC_CORE_SERVERrater-1.isWorker = truerater-1.sfSupport = falserater-1.LoadBalanceRestricted = falserater-1.Debuggable = yesrater-1.DebugPort = 59771rater-1.DebugMode = norater-1.mainClass = com.highdeal.launcher.Launcherrater-1.exePath =rater-1.libPath = C:/usr/sap/CC2/CCR01/exe/CC_CORE_SERVER/jars/ntamd64;$[jvm/libPath]rater-1.classPath =C:/usr/sap/CC2/CCR01/exe/CC_CORE_SERVER/jars/common_message.jar;C:/usr/sap/CC2/CCR01/exe/CC_CORE_SERVER/jars/common_sql.jar;C:/usr/sap/CC2/CCR01/exe/CC_CORE_SERVER/jars/common_ut

il.jar;C:/usr/sap/CC2/CCR01/exe/CC_CORE_SERVER/jars/core_admin.jar;C:/usr/sap/CC2/CCR01/exe/CC_CORE_SERVER/jars/core_client.jar;C:/usr/sap/CC2/CCR01/exe/CC_CORE_SERVER/jars/core_server.jar;C:/usr/sap/CC2/CCR01/exe/CC_CORE_SERVER/jars/iaik_jce.jar;C:/usr/sap/CC2/CCR01/exe/CC_CORE_SERVER/jars/likey.jar;C:/usr/sap/CC2/CCR01/exe/CC_CORE_SERVER/jars/ojdbc14.jar;C:/usr/sap/CC2/CCR01/exe/CC_CORE_SERVER/jars/sapjco3.jar;C:/usr/sap/CC2/CCR01/exe/CC_CORE_SERVER/jars/sqljdbc4.jar;C:/usr/sap/CC2/CCR01/exe/jstartupapi.jar;C:/usr/sap/CC2/CCR01/exe/jstartupimpl.jarrater-1.javaParameters = -Dhighdeal.home="C:\cc" -Dboot.config=C:/usr/sap/CC2/CCR01/config/boot.config -Dshutdown.timeout=20rater-1.parameters = -instance.id rater#1rater-1.shutdownMethod = com.highdeal.launcher.Launcher.shutdownrater-1.shutdownTimeout = 20

Table 7 - Standard Rater jstart.config

Options can be added to this parameter to control the amount of memory Java will allocate to the instance and how the

Java garbage collection will work.

Here is an example of a Rater config tuned for higher performance in Benchmark Ref#11:






rater-1.javaParameters = -Dsapcc.home="/srv" -Dboot.config=/usr/sap/CC2/CCR04/config/boot.config -Dshutdown.timeout=0 -Dfile.encoding=UTF-8 -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:+CMSIncrementalPacing -XX:CMSInitiatingOccupancyFraction=10 -XX:CMSIncrementalDutyCycleMin=0 -XX:CMSIncrementalDutyCycle=10 -XX:+ScavengeBeforeFullGC-XX:+UseSplitVerifier -XX:MaxDirectMemorySize=2500M –Xmx2g –Xms1g -Xss256k -XX:+PrintGC-XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:D:/usr/sap/CC2/CCR04/work/GC-

rater1.log Table 8 - Example of modified java parameters

Parameters explained:

Importance

(1= lowest,

10=highest)

Parameter Notes

8 -XX:+UseConcMarkSweepGC Sets the garbage collector policy to the concurrent (low

pause time) garbage collector (also known as CMS)

8 -XX:+CMSIncrementalMode Enables the incremental mode. (works only with -

XX:+UseConcMarkSweepGC)

8 -XX:+CMSIncrementalPacing Enables automatic adjustment of the incremental modeduty cycle based on statistics collected while the JVM is

running

8 -XX:CMSInitiatingOccupancyFraction= The command line

flag CMSInitiatingOccupancyFraction can be used to set

the level at which the collection is started. Its default

value is approximately 68%

8 -XX:CMSIncrementalDutyCycleMin= The percentage (0-100) which is the lower bound on the

duty cycle when CMSIncrementalPacing is enabled

8 -XX:CMSIncrementalDutyCycle= The percentage (0-100) of time between minor

collections that the concurrent collector is allowed to

run. If CMSIncrementalPacing is enabled, then this is justthe initial value.

8 -XX:+ScavengeBeforeFullGC Do young generation GC prior to a full GC

-XX:+UseSplitVerifier Option to Use the new type checker with

StackMapTable attributes

10 -XX:MaxDirectMemorySize= Must be equal to or greater than Xmx + Sub cache size

(i.e.

10 -Xmx Maximum Java heap size

10 -Xms Initial Java heap size

5 -Xss Sets the stack size for each thread

Some people have reported that it is necessary to

change stack size settings at the OS level for Linux. A call

to ulimit may be necessary, and is usually done with a

command in /etc/profile:

1 -XX:+PrintGC Option to print out garbage collection logs

1 -XX:+PrintGCDetails Option to print out garbage collection logs

1 -XX:+PrintGCDateStamps Option to print out garbage collection logs

1 -Xloggc: File to store garage collection logsTable 9 - JVM parameter definitions




The most important parameters are XX:MaxDirectMemorySize, Xmx and Xms. If these parameters are set too low then

you will likely have lower than optimal performance and in some cases even instance crashes. If you allocate too much

memory than there is available, the server will run out and crash. That being said, allocating more memory to CC

instances is a simple exercise to undertake, and by carefully testing the system with different values, massive

performance gains can be achieved.

The garbage collection options can help to reduce ‘stop-the-world’ garbage collection events which completely pause

the system while the garbage collection takes place. Modifying these values is relevant simply because the less time the

JVM spends doing GC’s the more time it can be processing CDRs. Any changes to these options must be carefully tested.

Full details of the Java HotSpot VM Options are detailed here:

http://www.oracle.com/technetwork/java/javase/tech/vmoptions-jsp-140102.html

At the time of writing it is understood that subsequent versions of the Hotspot JVM will enable even more control over

garbage collection options potentially enabling even more performance improvements by tweaking these parameters.







3.5 Database Tuning

Of all the tuning topics, the Database tuning is by far the most difficult and time consuming. It is also often the most

important. Database tuning must be done by an experienced DBA. The following sections describe some points that

should be checked to optimize SAP CC performance but these recommendations are by no means exhaustive from an

Oracle tuning perspective.

SAP CC 2.0 and 3.0 work with Oracle and SQL Server database technologies. No in-memory database technologies are

currently supported.

When it comes to tuning a database, there are several key concepts that are useful to know:

3.5.1 Why tune the database?

A lot of SAP CC’s processing happens “in memory” and therefore doesn’t rely on having a quick DB at all. CC does this by

caching all the required customer data, subscriptions, price plans and other objects so that it does not need to be loaded

during the rating process.

However, database reads and writes are still required at critical points during critical stages as explained below:

3.5.1.1 DB I/O during rating

DB I/O’s can occur during the rating process differently based on the type of rating that is being done.

3.5.1.1.1

Non-session based rating

SAP CC will need to do a database update if there are any counters being updated in a price plan. This is because it is

currently the only way to ensure consistency for the counters (imagine a counter that is shared across multiple

customers).




Figure 6 - Sap CC transaction flow for non-session based transactions

As you can see from the above diagram, the DB write happens at the end of a rating event. If multiple counters needed

to be updated they would all be updated during this step (in one DB transaction). Therefore, it is important that the DB

can handle the load that SAP CC will apply when updating counters (more details in subsequent sections).

IOPs required during rating (approximate):

Counter updates per CDR

TPS 1 2 5

500 125 250 625

1000 250 500 750

2000 500 1000 2500

5000 1250 2500 6250

10000 2500 5000 12500

Table 10- DB IOPs required during rating for non-session based, non-prepaid scenarios

3.5.1.1.2 Session-based rating

Session based rating is handled differently in SAP CC 2.0 and SAP CC 3.0.

In 2.0 rating sessions are not persistent – that is, sessions are not stored in the database and are lost if there is an

instance crash during a session. In 3.0 they are persistent and can be recovered in the case of an instance crash. The

downside of the 3.0 method is that additional database writes are required when sessions are created or updated.

Accordingly, additional IOPs and other resources must be allowed for. At the time of writing, detailed benchmarks have

not been carried out for session based rating in SAP CC 3.0. However, it is safe to assume that session based rating is wil

be less performing in 3.0 than 2.0.




3.5.1.1.1

Pre-paid scenarios

Prepaid rating scenarios will usually result in a monetary balance in the subscriber account being modified. This will have

at least the impact of a counter update.

3.5.1.2

DB I/O during ProvisioningWhen customer data is created or updated writes will occur in the SAP CC database. Provisioning requests are handled

by the SAP CC Updater instance.

Because provisioning performance usually only a ‘one off ’ event before a system goes live it is generally not as critical to

have really high performance. Tuning the provisioning is also made more difficult due to the fact that SAP CC currently

only supports one active Updater instance.

Still, sometimes is it is important to have fast provisioning as possible and following the guidelines in section 3.3.1 can

significantly improve performance.

The subsequent sections that discuss database disk issues do not focus a whole lot on performance for provisioning.Depending on your requirements, you may need to think about applying the principles recommended for best charging

performance on the disks that contain customer master data as well.

3.5.1.3 DB I/O during Cache Warm-up

When a Rater or Guider goes down or comes online, the customer data is re-distributed amongst all the active instances

SAP CC does this by re-assigning the partitions that each instance is responsible for. When a partition is re-assigned, the

customer information inside it must be reloaded from the DB and put in to the instance’s cache.

Previously (up until SAP CC 2.0 SP10), this reload was done ‘on demand’ i.e. If an EDR needed to be rated for a customerwho’s information was not in memory, SAP CC would load it before rating. During benchmarks executed in 2011

1 it was

discovered that this process was simply too slow if a large number of customers were ‘cold’ (i.e. not in memory). In

some tests that were running at 140,000 TPS, the performance would drop to just 4000 TPS because the system was

spending so much time loading customers one by one and all the message queues in the system were filled up waiting

with cold customer EDRs. What’s worse, it was taking hours for the system to return to 100% throughput. This was

simply unacceptable.

A change was made so that customer data is now loaded in bulk as a background process automatically and options are

available to control the resources dedicated to this warm up process. So, as soon as a partition is re-assigned the data

inside it loaded.

As a result the impact of losing or bringing up Rater or Guider is now minimal even at very high throughput.

3.5.2 SAP CC Database Tuning Best Practices

Best practices for setting up an Oracle database are available on the Solution Specialist Wiki.




Currently, the best way to achieve high DB performance is using Oracle RAC. This is Oracle’s clustering technology which

enables multiple DB instances in the environment to share the load and provide high availability.

At time of writing there are no specific best practices for SQL Server.

3.5.3 Database Disks

Aside from having enough space to store CC data, a number of disks are needed to simply to support the IO

requirements of the database when the system is performing a high volume of transactions. If you are aiming to build a

system with very high performance it is likely that you will end up with more space than will ever be needed just

because of the number of disks that are required handle the IOPs.

Here is a table of common commercially available disk drives and their indicative performance capabilities:

Device Type IOPS Interface

7,200 rpm SATA drives HDD ~75-100 IOPS SATA 3 Gb/s

10,000 rpm SATA drives HDD ~125-150 IOPS SATA 3 Gb/s

10,000 rpm SAS drives HDD ~140 IOPS SAS

15,000 rpm SAS drives HDD ~175-210 IOPS SAS

Simple SLC SSD SSD 400 IOPS SATA 3 Gb/s

Table 11 - Indicative disk performance

For a highly tuned performance benchmark executed in 2011

1

, with SAP CC running at approximately 140,000 TPS (non-session based rating), around 50,000 total IOPs were observed on the database. That means that the system would have

required somewhere around 400 10,000 RPM SATA drives to support the required level IOPs. Unfortunately the system

was not sized with enough disks and ultimately the lack of disks was the main system bottleneck. This particular

benchmark also happened to be configured with some resource hungry Oracle parameters (i.e. FLASHBACK on) but even

without these, the IOPs requirement was still somewhere around 30,000-40,000 IOPS.

Unsurprisingly, almost all of the IOs during rating are DB writes for counter updates. Here are results from a test that

was generating around 50,000 IOPS:




Figure 7 - Readout for a performance test showing total I/O rate and split between reads and writes

Looking at the above data you can see that the number of reads were so small they barely chart at all.

When it comes to data throughput for the same test, another clear split is shown between reads and writes:

Figure 8 - Readout for a performance test showing total data throughput and split between reads and writes

As always, when sizing the DB allow a comfortable room for error as to not get trapped with too few drives to handle

the required IOPs. If Oracle flashback is a required parameter from the customer, significantly more DB resources are

likely to be required. For that scenario it is recommended to check the latest Oracle Flashback best practices.

The IOPs requirements of SAP CC are considered very high. The reason for this is the rating architecture whereby each

rated item results in an update to the database (for session based rating and/or if the rated item resulted in a counter

update). Some batching does occur within SAP CC and can be controlled via system parameters which can both help

improve throughput (at the cost of latency).

3.5.3.1 Disk Configuration

As outlined in the Oracle configuration guidelines linked in 3.5.2 it is important the way the database’s disks are

configured and assigned to tablespaces of the CC database. Please follow the guidelines as closely as possible.




3.5.3.1.1

Oracle ASM

Oracle provides a feature called ASM (Automatic Storage Management) which simplifies administration of Oracle related

files by allowing the administrator to reference disk groups rather than individual disks and files, which are managed by

ASM.

Some enterprises prefer to use this feature as opposed to manually allocating the disks to tablespaces. This feature has

many advantages and in many cases may be sufficient for the performance required for a particular environment.

However, the Disk Configuration recommendations in this document have been developed based onnot using this

feature. In any event, an experienced DBA should be able to quickly locate disk bottlenecks and where possible change

up the configuration to obtain better performance.

3.5.1 Network infrastructure and SAN

The network infrastructure can also have a significant impact on the maximum performance that can be achieved on a

platform. All hardware vendors have different solutions available and ultimately they will be able to provide the bestrecommendation based the SAP CC server and database configuration.

Some examples of best practices that were received from IBM during a benchmark in 2011:

- Fiber channel disks on and cached based enclosure.

- Limit the number of disk ranks behind each device adapter pair. A disk is faster if it is alone when it

communicates with its controller.

- Multiple LUNs per disk if infrastructure supports parallelism to assist with random IO activity

- Fiber connection to the server hosting the database engine

- Large SAN cache size (>=2GB)

If at all possible please seek the advice of a storage expert when defining the network infrastructure and SAN

requirements.

3.5.2 HDD vs. SSD

SSD drives can allow a much higher level of IOPs than a traditional HDDs as well as improved latency and data

throughput. However it is not necessary to use SSDs for the all the table spaces in the database (disk cost alone is a very

good reason NOT to take this approach).

The following diagram highlights which table spaces are recommended to be allocated to SSD drives for bestperformance for non-session based rating:




Figure 9 - Ideal performance scenario for batch rating – specific tablespaces on SSD

Unsurprisingly, the counter table and log files (including redo) table spaces are recommended to be allocated to SSDs.

Why? Because they absorb almost all of the IO’s during rating. Counter tables are updated every time a transaction is

processed that requires a counter update, and the redo logs are updated every time any other table in the database is

updated. So, if you want the best rating throughput and latency you need to be using the best hardware available to

manage it which is currently SSDs.

3.5.2.1

Session based rating SSD recommendation

For environments that will execute session rating, the tablespace that maintains session related information should also

be placed on SSDs for optimal performance.

Figure 10 - Tablespace recommended being on SSD for optimal session based rating performance

3.5.3 In Memory Databases

Currently SAP CC does not function on any in-memory database technologies. Given their rising popularity it seems

inevitable that the feature will be added to the software before too long. At this time a new rulebook will need to be

written on how to tune an in-memory database for use with SAP CC!




4 Measuring Performance

4.1 SAP CC Client overview

SAP CC is capable of very high levels of transactions but it is only one part of the end-to-end solution. It is very important

that the client application that will send CDRs SAP CC takes advantage of the system architecture so that it is able to

send and receive CDR information as fast as the hardware will allow.

Important points when building or configuring a SAP CC client:

Threads: If the client application only has 1 thread, it will not be able to send many EDRs at once. SAP CC’s

multithreaded architecture allows massive performance gains to be had when the client can send CDRs on

multiple threads simultaneously.

Client message queue sizes: Typically the client will need to have some type of queuing system to monitor CDRs

that have been sent to CC and are waiting for a response. It is recommended that the queue sizes in the clientapplication are aligned to those in set in the CC system.

Queue notify: The client application must need to know when its queue is becoming full so that it can stop

sending CDRs until responses a received from CC. Conversely, it must know when free slots become available in

its queue so that more CDRs can be sent. A typical queue notify value (the threshold when EDRs can start being

sent to CC again) is around 10% less than that of queue size.

Response handling: The client application needs to be configured to be able to load CDRs and handle responses

as quickly as possible while still being able to maintain data consistency in the event of a crash. This aspect of

the design/configuration is typically driven by the specific rating scenario.

SAP CC client objects: As well as multiple threads, it can be advantageous to have multiple SAP CC clients

running (i.e. StatefulServiceClient objects from the CC API). Between 1 and 4 is usually sufficient. Too many willcause unnecessary overheads and connections to CC.

4.2 Testing tools

4.2.1 Load Test Tool

An internal tool called Load Test tool (LTT) exists that can be used to benchmark the performance of SAP CC. The tool is

not supported in any way and can only be used by technical consultants with a good understand of Unix, scripting and of

course SAP CC.

The tool can be obtained by speaking to SAP CC development or Solution Specialists.

LTT has the following features:

• Generates rating requests at configurable volumes

• Generates EDRs containing configurable randomly generated information




• Spreads EDRs across configurable number of subscribers in system

• Can generate traffic for any number of service types simultaneously

• Can generate EDRs at different volumes for each service

• Uses the asynchronous API

• Receives responses from SAP CC and logs the results

• Logs information to file. Metrics include - TPS, latency (min, max, average, median, 90th

percentile, 95th

percentile). Information can be summarized as configurable intervals as well.

• Java based – portable to Unix, Windows + other OS’s

Figure 11 - Load Test Tool architecture

At the time of writing there is no guide for using LTT.

4.2.2 ARTIX

ARTIX (Acceptance and Regression Test Inspection eXecution) server is another internal testing tool developed by the

Solution Specialist team. It is used primarily to create and remove customer master data in SAP CC and create charging

requests to unit test price plans. But, because it is designed to be used with 3rd

party tool called SOAP UI, it can leverage

additional functions that SOAP UI has available for load testing.




Figure 12 - ARTIX architecture

The guide for obtaining, setting up and running ARTIX is on the Solution Specialist Wiki. At the time of writing ARTIX

works with SAP 2.0 and 3.0.

It is expected (although not proven) that the overheads of going through ARTIX and SOAP UI would result in less

performance than using the LTT.

4.2.1 BART

This document has not focused on BART at all – However, BART does have one advantage in that it provides a report at

the end of a batch acquisition session, detailed elapsed time and other metrics. It is an obvious option for measuring

performance in environments that are already using it. It must be noted that the performance of BART can never match

that of SAP CC and will ultimately be a bottleneck as most of BART’s processing is reading and writing from the database

It also has many synchronous processing constraints and is not scalable in the way that SAP CC is.

4.2.1 IEC

Like BART, IEC is an out of the box tool that can be used to inject CDRs into SAP CC. On its own it will not provide

performance metrics but could be used as part of a wider solution.




4.2.2 Third party tools

A quick search on Google will reveal many other load testing tools on the market. There is no reason that they couldn’t

be integrated with SAP CC providing they are flexible enough to do so.

However, there are currently no guides for using any other 3rd

party tools for load testing.




5 Appendices

5.1 Benchmarks referenced in this document

At various times throughout this document, references are made to benchmarks that were carried out in 2011. Thehardware used and tests and results of those benchmarks are detailed below.

Ref # Title Customers Test Injection HW SAP CC HW DB HW DB disks

1 Q2-3 customer

benchmarks

100M 2 x IBM blades 9 x IBM blades 32 cores on IBM

P780

64 x SSD

24 x HDD

2 Independent

benchmarks

200M As above 13 x IBM blades As above As above

For more information please check the SAP CC Solution Specialist wiki.




5.2 Hardware used for Benchmark Ref #1

SAP Benchmark Phase 2 Infrastructure

BCH Chassis n401

1 GbE admin VLAN

Internet

LocalTime

Server

10.7.18.254

SAP

Desktops

OpenVPN

connections

Default Gw = 10.7.18.254

10GbE data VLAN

Admin VLAN: 1GbE

IP range10.7.18.0/24

Default GW 10.7.18.254

DNS 129.25.160.4

129.35.161.251

data VLAN: 10GbE

IP range: 10.1.29.0/24

no Default GW

no DNS

HS22V blade

BCH 401: 2*Intel Xeon x5670 6Cores 2,93Ghz

BCH 438: 2*Intel Xeon x5675 6Cores 3,86Ghz

72GB (18*4GB) memory

2*50GB SSD

RHEL 5U5 64 bits

eth0 on admin, eth2 on data VLAN’s

Database server LPAR

8 dedicated cores, 64 GB RAM

Oracle 11.2.0.2 (not installed)

Aix 6.1 TL6 SP5

en0 on admin, en1 on data VLAN’s

8 Gbps FCIBM Systems Director

Active Energy Manager 10.7.18.200

Advanced Mgt Module10.7.18.125

BNT L2-3 GbESM 110.7.18.127

1 0 .7 .1 8 .2 -1 0 .1 .2 9 .2

1 0 .7 .1 8 . 3 -1 0 .1 .2 9 . 3

1 0 .7 .1 8 .4 -1 0 .1 .2 9 .4

1 0 .7 .1 8 . 5 -1 0 .1 .2 9 . 5

1 0 .7 .1 8 . 8 -1 0 .1 .2 9 . 8

1 0 .7 .1 8 . 9 -1 0 .1 .2 9 . 9

1 0 .7 .1 8 .1 0 -1 0 .1 .2 9 .1 0

BCH Chassis n438

Advanced Mgt Module10.7.18.225

BNT L2-3 GbESM 110.7.18.227

1 0 .7 .1 8 .2 4 -1 0 .1 .2 9 .2 4

1 0 .7 .1 8 .2 5 -1 0 .1 .2 9 .2 5

1 0 .7 .1 8 .2 6 -1 0 .1 .2 9 .2 6

1 0 .7 .1 8 .2 7 -1 0 .1 .2 9 .2 7

DS8800 equipped with 144 600GB 10 KRPM SAS,

64 300GB SSD, and 256 GB cache

SAN

2 2 2 2

BNT 10GbESM

10.7.18.129

BNT 10GbESM

10.7.18.229

Four 8 Gbps FC links

44

One 10GbE Link

Database Server LPAR

(8 cores, 64 GB RAM)

10.7.18.101 – 10.1.29.101



10.7.18.102 – 10.1.29.102



10.7.18.103 – 10.1.29.103



10.7.18.104 – 10.1.29.104

Dedicated 64 Ways 3,86 GHz 1024 GB p780

S ixteen 300 GB SSD disks R AID10 Eight 300 GB SSD RAID 5(*) not decided yet whether HDD or SDD’s or both are needed

Figure 13 - Benchmark Ref#1 Hardware

5.3 Test cases executed for Benchmark Ref #1




Figure 14 - Benchmark Ref#1 Test cases

5.4 Raw results of tests carried out for Benchmark Ref #1

Test

Case

Name

Subs Nodes Description Duration Avg TPS Avg

Latency

(ms)

CPU Rater CPU

Dispatcher

/Guider

CPU DB IOPS

DB1 50M 50% DB 50M Subs. 50% of DB nodes 2h 126562 116 ms 43% 4% 99% 44047

DB2 75M 50% 75M Subs. 50% of DB nodes 2h 126567 116 ms 43% 4% 99% 52483








LO1 Optimized CC configuration for

lowest latencency. 375m request

per hour

2h 104896 35 ms 33% 4% 50% 49818

LO2 Optimized CC configuration for

lowest latencency

2h 133037 72 ms 43% 5% 52% 44925




API1 Process API read calls + Rating at

same time

2h 140017 78 ms 46% 5% 58% 50446

API2 Process API update calls + Rating

at same time

2h 137745 106 ms 50 % 7 % 56 % 46615

API3 Combination of API1 and API2

with additional background tasks

+ Rating at same time

4h 140427 91 ms 50 % 7 % 54 % 49405

RR1 Resource Removal - App Server 2h 121153 120 ms 46 % 6 % 56 % 46178

RR2 Resource Removal - RAC Node 2h 135545 104 ms 46 % 6 % 56 % 57186

RR3 Resource Removal - RAC Instance 2h 124514 117 ms 42 % 7 % 45 % 47258

V1 25% 25% of CC rating processes

(limited threads)

2h 54242 268 ms 16% 4% 22% 20983


(limited threads)

2h 99562 146 ms 29% 4% 35% 36029


(limited threads)

2h 125009 117 ms 37% 4% 49% 49762

V4 100% 140000

125% 160000

150% 190000

200% 230000

H1 25% of CC nodes (1 rater) 2h 4235 3428 ms 23% 1% 24% 2225

H2 50% of CC nodes (2 rater) 2h 62810 230 ms 49% 2% 33% 23485

H3 75% of CC nodes (3 raters) 2h 111151 131 ms 47 % 6 % 44 % 48013

IR1 Send 375M requests per hour (-

25%)

2h 104894 33 ms 33% 4% 49% 48394

IR2 Send 625M requests per hour

(+25%)

2h 144861 102 ms 48% 4% 54% 49658

ENT1 Large enterprise subscriptions

only

2h 7479 636 ms 81 % 1 % 7 % 3218

Table 12 - Benchmark Ref#1 Test results

5.5 Hardware used for Benchmark Ref #2




Figure 15 - Hardware used for Benchmark Ref#2

5.6 Test cases executed for Benchmark Ref #2

Figure 16 - Test cases executed for Benchmark Ref#2



5.7 Raw results of tests carried out for benchmark Ref #2

See 5.6

6 Index of figures and tables

6.1 Figures

Figure 1 - Benchmark results: TPS vs. counter updates ..................................................................................................... 8

Figure 2 - TPS for a single subscription ............................................................................................................................. 9

Figure 3 - SAP CC instance interaction during rating ........................................................................................................10

Figure 4 - Multithreading rating process steps .................................................................................................................11

Figure 5 - Java application profiling showing regular garbage collections .........................................................................19

Figure 6 - Sap CC transaction flow for non-session based transactions ............................................................................24

Figure 7 - Readout for a performance test showing total I/O rate and split between reads and writes ............................27

Figure 8 - Readout for a performance test showing total data throughput and split between reads and writes ...............27

Figure 9 - Ideal performance scenario for batch rating – specific tablespaces on SSD ......................................................29

Figure 10 - Tablespace recommended being on SSD for optimal session based rating performance ................................29

Figure 11 - Load Test Tool architecture ...........................................................................................................................31

Figure 12 - ARTIX architecture .........................................................................................................................................32

Figure 13 - Benchmark Ref#1 Hardware ..........................................................................................................................35

Figure 14 - Benchmark Ref#1 Test cases ..........................................................................................................................36

Figure 15 - Hardware used for Benchmark Ref#2.............................................................................................................38

Figure 16 - Test cases executed for Benchmark Ref#2 .....................................................................................................38

6.2 Tables

Table 1 - CPU and RAM consumption during a high performance rating test .................................................................... 7

Table 2 - CPU and RAM consumption during a high performance provisioning test .......................................................... 7

Table 3 - Rater instance parameters ................................................................................................................................13

Table 4 - Guider instance parameters ..............................................................................................................................15

Table 5 - Dispatcher instance parameters........................................................................................................................16

Table 6 - Updater instance parameters ...........................................................................................................................17

Table 7 - Standard Rater jstart.config ..............................................................................................................................20

Table 8 - Example of modified java parameters ...............................................................................................................21

Table 9 - JVM parameter definitions................................................................................................................................21

Table 10- DB IOPs required during rating for non-session based, non-prepaid scenarios .................................................24

Table 11 - Indicative disk performance ............................................................................................................................26

Table 12 - Benchmark Ref#1 Test results .........................................................................................................................37

SAP CC Tuning Essentials v1 1

Documents