August 01, 2008 Performance Modeling John Meisenbacher, MasterCard Worldwide.

August 01, 2008

Performance ModelingJohn Meisenbacher, MasterCard Worldwide

August 01, 2008 2

If You Have a Transaction Processing System…

• I can create a simple accurate CPU performance model…

….that can help you make your system more efficient…

…in one week…

…with the data you already collect…

…and so can you !

August 01, 2008 3

What is a Performance Model?

It provides the quantitative information you need to:

– Manage the performance of your system; and

– Understand the impact of:

– software changes;

– hardware / configuration changes; and

– actual / projected transaction load changes.

August 01, 2008 4

The CPU Performance Model

• What is the best model?– The best model is only as complex as it needs

to be to provide information that is sufficiently accurate to help you make the right decision.

August 01, 2008 5

Performance Management Goals

• Support capacity planning

• Identify performance changes with changing hardware, software and transactions.

• Isolate performance issues within the system.

• An accurate production model should:

– Be easy to use / understand; and

– Provide low cost and timely information.

August 01, 2008 6

About Transaction Processing…

• There are basic features of most transaction systems in the real world.

• The rate of transactions per second (TPS), varies:– During the day, there are peak hours

– During the week, there are peak days

– During the year, there are peak months / seasons

• Transaction processing systems are typically sized for the peak hour of the peak day of the next peak season.

August 01, 2008 7

Daily Transaction Cycles

August 01, 2008 8

So, You’re Asking Yourself…

Q. Is my hardware (CPU) properly sized to support my peak transaction period in my next peak season?

A. Possible ways to answer this question… – Use a commercial capacity planning tool.

– Run a performance test in the lab.

– Use a CPU model validated in production.

August 01, 2008 9

Commercial Capacity Planning Tools

• Commercial capacity planning tools typically focus on:– system size,

– acute system issues, and

– hardware projections.

• Potential challenges: – These tools typically do not easily integrate with

transaction rates, and therefore cannot model transaction systems.

– CCP tools may be good for monitoring and alerts, but unless transaction loads are factored in, this approach cannot give an precise answer.

August 01, 2008 10

Daily CPU Usage Pattern

August 01, 2008 11

Build a Performance Test Environment

This approach requires:• Building and maintaining a hardware test

environment that is as close as possible to production.

• Loading production software and building a database that is as close as possible to production.

• Developing a way to send production-like transactions through the system.

• Maxing out the system to see when it breaks.• Hoping your test setup is close enough to

production to matter (or calibrating it with a production CPU model).

August 01, 2008 12

A CPU Model of the System

A CPU model of the transaction processing systems integrates three key workstreams:

1. Business – the transactions processed;

2. Capacity Planning – the hardware systems used to process the transactions; and

3. Software Engineering – the software systems and databases that process the transactions.

August 01, 2008 13

About Transaction Rates…

• Most online real world transactions rise and fall during the day, week, month, and year.

• The CPU method takes advantage of these cycles.

• Multiple data points are used for correlation analysis.

• The approach can be used in production, possibly eliminating the need for a performance test environment.

August 01, 2008 14

Transaction Processing and CPU Usage Sample

August 01, 2008 15

Data Reported in Time Increments

• Report data around the peak period.• CPU usage in some time increment,

generally between 10 and 20 minutes. (15 minutes is common)– Shorter time periods provide more data and

variation, but may not improve accuracy.– Longer time periods require less data and hide

variation-reducing accuracy and precision.

• Transactions are typically logged by time.• Multiple days can be easily merged into one

dataset.

August 01, 2008 16

Create the Model

• Display results graphically• The model from the previous graph is simply

defined:– CPU utilization = CPU per TPS * TPS + overhead– y = a*x + b

• “System capacity” is the TPS rate that drives the system to your “safe” limit, such as 80% CPU utilization.– TPS Limit = (“Safe” CPU limit – overhead ) / CPU

per TPS

• Most systems are linear, but also can be exponential or logarithmic.

August 01, 2008 17

Graph of Correlation Shapes

August 01, 2008 18

Correlate TPS and CPU

August 01, 2008 19

Transaction Processing Systems

• The software-intensive system typically includes processes related to transactions:– The actual transaction processing

– Data management for look-ups and logging

– Communication processing

– Format conversion

– Also may include other processes, such as system monitoring, that are independent of transaction rates.

August 01, 2008 20

CPU Divided by Type of Processing

August 01, 2008 21

Use the Model to Manage the System• Factor the results into your planning

– Use the data to help identify any efficiencies for the upcoming peak season

• The change, in this real-world case, was made to transaction replication in the database.

• The release was rolled out into product and a new model created.

August 01, 2008 22

System After Application Performance Modification

August 01, 2008 23

Manage Performance within a Release

• The precision of the model can be used to measure changes in the system.

• Data from the previous day can be collected and quickly analyzed to determine if changes to the system have improved or degraded performance.

August 01, 2008 24

Variation of System Capacity

Monitoring Scripts added

And removed

Other scripts optimized

OK after release OK during peak period

August 01, 2008 25

Work with System Experts

• Key to success: support the business through cooperation between capacity planning and software experts.

• Creating the model, generally fairly easy, is only the first step.– Work with developers or vendors to

successfully manage and adjust system performance.

• Support the model’s quantitative information with experience and insight

August 01, 2008 26

Trust Your Experience and Intuition

• Use the model along with your understanding of hardware and software systems to direct your analysis and decision.

• Collect more production data to evaluate and refine these decisions with the goals of:– saving money for your company, and – delivering quality results to your customers.

Thank you.