CS 360 Lecture 11. In most computer systems: The cost of people (development) is much greater than the cost of hardware Yet, performance is important.

SOFTWARE SYSTEM PERFORMANCE

CS 360

Lecture 11

PERFORMANCE OF COMPUTER SYSTEMSIn most computer systems:

The cost of people (development) is much greater than the cost of hardware

Yet, performance is importantA single bottleneck can slow down an entire systemFuture loads may be much greater than predicted

2

WHEN PERFORMANCE MATTERS Real time systems

when computation must be fast enough to support the service provided, e.g., fly-by-wire control systems have tight response time requirements.

Very large computations where elapsed time may be measured in days, e.g., calculation of weather

forecasts must be fast enough for the forecasts to be useful.

User interfaces where humans have high expectations, e.g., mouse tracking must appear

instantaneous.

Transaction processing where staff need to be productive and customers not annoyed by delays,

e.g., airline check-in. 3

HIGH-PERFORMANCE COMPUTINGHigh-performance computing:

Large data collections (e.g., Amazon) Huge numbers of users (e.g., Google) Large computations (e.g., weather forecasting)

Must balance cost of hardware against cost of so(ware development Some configurations are very difficult to program and debug Sometimes it is possible to isolate applications programmers from the system complexities

4

PERFORMANCE CHALLENGES FOR ALL SOFTWARE SYSTEMSTasks

Predict performance problems before a system is created. Design and build a system that is not vulnerable to performance problems.

Identify causes and fix problems after a system is implemented.

5

INTERACTIONS BETWEEN HARDWARE AND SOFTWAREExamples

In a distributed system, what messages pass between nodes? How many times must the system read from disk for a single transaction?

What buffering and caching is used? Are operations in parallel or sequential? Are other systems competing for a shared resource (e.g., a network or server farm)?

How does the operating system schedule tasks?

6

LOOK FOR BOTTLENECKSUsually, CPU performance is not the limiting factor.

Hardware bottlenecks Reading data from disk Shortage of memory (including paging) Moving data from memory to CPU Network capacity (bandwidth)

Inefficient software Algorithms that do not scale well Parallel and sequential processing

7

LOOK FOR BOTTLENECKSCPU performance is a limi1ng constraint in certain domains: large data analysis (e.g., searching) mathematical computation (e.g., engineering) compression and encryption multimedia (e.g., video) perception (e.g., image processing)

8

TIMESCALE OF DIFFERENT COMPONENTS

9

Actual performance may be considerably less than theoretical peak performance

LOOK FOR BOTTLENECKS: UTILIZATIONUtilization is the proportion of the capacity of a service that is used on average. Utilization =

avg. service time for a transaction/avg. arrival time of transactions

Peak loads and temporary increases in demand can be much greater than the average.

10

PREDICTING SYSTEM PERFORMANCEDirect measurement on subsystem (benchmark)

Disk I/O CPU Network

Simulation

All require detailed understanding of the interaction between software and hardware systems.

11

DIRECT MEASUREMENT ON SUBSYSTEMTools for examining performance of subsystems:

Ex: Sysbench

Sysbench is a modular, cross-platform, multi-threaded benchmark tool used to quickly gain information about system performance. Tests the following:

File I/O performance Job scheduler performance Memory allocation and transfer speed Database service performance 12

SIMULATIONSBuild a computer program that models the system as set of states and events

advance simulated time determine which events occurred update state and event list

repeat

Discrete time simulation: Time is advanced in fixed steps (e.g., 1 ms)

Next event simulation: Time is advanced to next event

13

MEASUREMENTS ON OS/ARCHITECTURESBenchmarks:

Run tools on standard problem sets, sample inputs, or simulated loads on the system

Collect data on resource availability, utilization, performance, etc.

Example: Disk I/O benchmark using MySQL to determine performance of different cloud stacks (OpenStack, Eucalyptus, Docker, etc.) Network, base OS, hardware should remain the same across all tests.

14

EXAMPLE: DISTRIBUTED SYSTEM

15Benchmark: Throughput vs. number of CPU cores utilized

FIXING BAD PERFORMANCE If a system performs badly, begin by identifying the cause: Instrumentation:

Add timers to the code. Often this will reveal that delays are centered in a specific part of the system.

Test loads: Run the system with varying loads (high transaction rates, large input files, many

users, etc.). This may reveal the characteristics of when the system runs badly.

Design and code reviews: Team review of system design, program design, and suspect sections of code. This

may reveal an algorithm that is running very slowly, e.g., a sort, locking procedure, etc.

Find the underlying cause and fix it or the problem will return! 16

PREDICTING PERFORMANCE CHANGE: MOORE’S LAWOriginal version:

The density of transistors in an integrated circuit will double every year. Gordon Moore, Intel, 1965

Current version: Cost/performance of silicon chips doubles every 18 months.

17

MOORE’S LAW: RULES OF THUMB

Silicon chips: cost/performance improves 30% / year in 12 years = 20x in 24 years = 500x

Storage media: cost/performance improves 40% / year in 12 years = 50x in 24 years = 3,000x

These assumptions are conservative. During some periods, the increases have been considerably faster.

Note: Recently, the rate of performance increase in individual components, such as CPUs, has slowed down, but the overall rate of increase has been maintained by placing many CPU cores on a single chip.

18

MOORE’S LAW EXAMPLE Will this be a typical laptop?

Surely there will be some fundamental changes in how these resources are packaged and used.

19

PARKINSON’S LAWOriginal:

Work expands to fill the time available. C. Northcote Parkinson

Software development version: Demand will expand to use all the hardware available. Low prices will create new demands. Your software will be used on equipment that you have not envisioned.

20

FALSE ASSUMPTIONS FROM THE PAST

Be careful about the assumptions that you make Here are some past assump1ons that caused problems: Unix file system will never exceed 2 GB. AppleTalk networks will never have more than 256 hosts (28 bits).

GPS software will not last more than 1024 weeks. Two bits are sufficient to represent a year (Y2K bug). Etc., etc., .....

21

SOFTWARE PERFORMANCE REQUIREMENTS (DOCUMENTATION)

Performance specifications should, at minimum, cover the following concepts:In detail, what is the performance test scope?

What subsystems, interfaces, components, etc. are in and out of scope of the specific performance test?

For the user interfaces, how many concurrent users are expected for each?Specify peak vs. normal usage

What does the target system (hardware) look like?Specify all hardware configurations

22

SOFTWARE PERFORMANCE REQUIREMENTS (DOCUMENTATION)

Performance specifications should, at minimum, cover the following concepts:What is the Application Workload Mix for each component?Ex: 20% login, 40% search, 30% item select, 10% checkout

What is the System Workload Mix?What processes are running?

What are the time requirements for any/all processes?Specify peak vs. normal

23

SOFTWARE PERFORMANCE REQUIREMENTS (DOCUMENTATION)

The performance requirements should include test cases to demonstrate performance of Individual components, Interfaces, Groups of components, And the system as whole

24

SOFTWARE PERFORMANCE REQUIREMENTS (DOCUMENTATION)

Tasks needed for performance documentation:1. Configure the testing environment2. Initial test run – to check correctness of testing script3. Execute tests (repeatedly) 4. Record and analyze the results (Pass/Fail)

Investigate corrective action if test fails5. Re-tune components/interfaces/system and repeat process

25

SOFTWARE PERFORMANCE REQUIREMENTS (DOCUMENTATION)

Tasks needed for performance documentation:6. Gathering performance requirements7. Decide whether to use internal or external resources to perform

tests.8. Developing a plan to test for performance9. Specify testing tools10.Specify test data11.Develop proof of concept test scripts for each

component/interface

26