An Introduction to Software Performance Engineering

© 2012 Collaborative Consulting

Enterprise Software Performance EngineeringBuilding systems that scream so your users don’t:A key component of risk management

Presented By: Walter Kuketz Hosted By: Frank Days

22012 Collaborative Consulting

Housekeeping

Presentation will last 45 minutes

Submit questions via the chat window

Slides will be made available


Purpose of today’s webinar

Review the current challenges of being a performance engineer

Overview of the five knowledge areas of Software performance engineering body of knowldge

Where to inject PE tasks and activities into the SDLC to build more performant applications

Overview of application performance management (end-to-end user experience)

Roles and career path overview

Case study – 10X event


Performance engineering challenges today

Highly distributed

apps

Distributed teams

Evolving & changing

application architectures

Changing usage

patterns

• Mobile devices• Third party services• SaaS products• Cloud

• Performance engineering and testing• Architecture and development• Business

• Complex and tangled system of systems• Distributed Infrastructure• Distributed production operations

• Driven by mobile anywhere• Flash traffic events


Performance engineering challenges

• Performance organization; skillsets, multi disciplined, communication, split team

• Training and career paths

EnterpriseUser

experienceBudget Team skills

• Constant cost and budget pressure

• Maintaining a meaningful performance lab (production-like)

• User populations are exploding

• Web site performance directly impacts revenue

• Release schedules, monthly, quarterly

• Project based


No clear standards in the market for PE

Performance Engineer

Performance tester

Capacity planner

Performance Architect

Test automation technician

Performance analyst

Performance Data engineerTroubleshooting

Benchmarks?

?

?

?

?

?

?


Goal of the SPE Body of Knowledge

Define the profession of performance engineeringDefine profession

Establish a body of knowledge for performance engineering

Establish Body of Knowldge

Establish a common language for roles, responsibilities, and practice areas using the form of Knowledge Areas

Establish common language

Define the competencies shared across the knowledge areas

Define competencies

Capture common techniques for sharing across the industry

Capture common techniques


The SPE Body of KnowledgeK

now

ledg

e ar

eas

Enterprise Software Performance Engineering

Planning, coordination, information sharing, and control

SDLC &architecture

(SA)

Application performance managemen

t(APM)

Problem detection

and resolution

(PDR)

Underlying competencies and roles for delivering

Techniques supporting the knowledge areas

Performance validation and testing

(PVT)

Capacity planning(CP)


The Enterprise PE team: There must be a practice leader

A shared service and clearing house for all performance, scalability and capacity planning information. Transforms performance information to

business knowledge and advantage.

Communicates across the teams to share information

For instance, the production workload and the performance test workload – are they in sync?

Make sure the architecture team knows how the new release, or newest build behave in production.

The business risks are clearly identified and communicated.

Can provide performance guidelines to the developers, must influence the unit testing process.

Operations PT

Architecture and production

Business direction

Development teams


Information sharing across the knowledge areas

Performance validation,

testing

Capacity planning

APMSDLC,

Architecture, NFR

Growth plansWorkload

NFR’s

ResultsObservations

NFR’sVolumetrics,

workload

Results,Workloadscalability

User ExpWorkloadUtilization

ForecastWorkload

Enterprise PE planning and monitoring

Triage

SPE impacts Architects,Developers, testing, and

operations


PE: Knowledge areas

PVTPerformance testing and

validation

CPCapacity planning

APMApplication performance

management

PDRProblem detection and

resolution

SASoftware development

lifecycle and architecture


PE: Software development lifecycle and architecture


validation

CPCapacity planning


management


resolution




PE: SA (software development lifecycle)

Define Design DevelopFunctional

testPerformance

testDeploy

Waterfall methodologies

Need to plan for performance testing

Build code to performance requirements

Define

Design

BuildTest

Deploy Agile methods

Can a system be designed to support 10 TPS and 1 second response time work @ 100 TPS and 1 second?


SDLC: Define and design

Define

1. Project risk scorecard - Key business transactions

identified

- Workflow constraints

- SLA’s and penalties

2. Workload model and usage scenarios

3. Business volumes with peak factors

4. Business performance requirements

Design

1. Technology Architecture validated to the performance Usage Scenarios. How will it achieve the key performance requirements? Define the scalability approach.

2. Scalability to support flash events

3. Environments: include any required changes to develop and test the performance usage scenarios.

Communication to your downstream processes and people

Prepare to deal with flash events

Design to avoid flash events(they are expensive. E.g. Work with business why does the outbound email

have to send email to all xM customers at the same time

Sub-second response time.

Flash marketing event.


SDLC: Develop and functional testing

Develop

• The functions are mapped to software components. The performance req’s are assigned to the components.

• Unit testing for early validation of the performance req’s for the software components. Response time measurements.

• Integration testing include micro-performance testing of key functions. Code review of the key performance functions.

• Enhance unit and integration test for early validation of key performance functions

Functional test

• Verify business functions and key performance functions.

• Verify the key performance workflows and functions.

• Define and execute micro-performance testing in the QA process.

• Produce report on for each key performance Usage Scenario with response time and throughput.

Communication to your downstream processes and people

Functions are tested. Performance needs to be analyzed.

Greatest impact on performance and scalability, but you must know the goals.


Performance budgets

PROXY

Datacenter

WEB

MQ/ESB

DCOM

CORBA

SERVER

APP SVR

LDAP

WebServices

Code runs everywhere.

Respond to the user in 2 seconds.

Do your developers know how much time they have for their piece of the transaction?




PE: Performance testing and validation

CPCapacity planning


management


resolution


validation


PE: PTV (performance testing and validation)

What is changing? Where does it need to be validated? What (all) is being measured?

The project: Why are we testing and what are the success criteria?

Prerequisites:

Business goals and non-functional requirements are defined

Workload and scenarios defined

Environment defined (where are we testing)

Application in a stable state

Configuration management and release management ready

Core performance team identified

Supporting team committed


• Deep understanding of the application being tested

• Deep understanding of the technical architecture of the application

• Understanding of the business area supported by the applications

Business and application area

knowledge

• Performance engineers involved in design sessions

• Understand the nature of the data required for testing

• Able to understand technical gaps to meet requirements not supported by the tool

Early involvement with the applications

team

• Well-understood workload model and characterization

• All components in the test are monitored and reported on

• A defined triage process for troubleshooting

• A process to manage the inevitable unplanned tests and test archive exists

Test execution

• Testing environment is a close approximation of production, including database size

• Differences between test and production are clearly known

• Release and configuration management in place and procedures enforced

Environment

Organization: Performance team value

You still need to explain to the rest of the organization what you do

Performance team value


Performance team reference model

Primary test, design, and execution team

Performance engineer


Results analyst

Performance engineering support skills

Application architect

Technology SME

Planning and project

management

Data engineering

Must determine the core team members vs. on-demand team


Enterprise performance team

Performance test execution team

PT problem resolution

board

PT technology PMO

Strategy and planning

(architecture)**

Infrastructure**

Performance engineering and

optimization

Application support

Application deployment

Vendor management

Vendors

Vendor IBM Oracle EMC Informatica Vendor Vendor


Model test case executionIn the end, the test has to be run.

Select test case

Prepare test harness online

or batch messages

Prepare test monitoring

components

Confirm environment

readiness

Initiate test case

Select test case

Normal test completion?

Problem analysis and triage team

Gather test results

Txn RT, CPU, DB

Analysis for specific change?

Evaluate impact of change

Create test report; publish to repository

Are changes

required for retest?

Categorize change – enter change to CM tracking tool

Implement configuration

change

Can you retest with

the change?

Rerun the test case

Move to next test scenario

Move to next test case

10

1

2

2

0

No

YesYes

No

YesYes

NoNo




PE: Capacity planning


validation


management


resolution

CPCapacity planning


PE: CP (capacity planning)

The right system at the right price!

Workloads drive system transactions, which drive resource utilization

Determine capacity

requirements

Analyze current

capacity

Plan for future

• Forecast workloads

• Plan for usage

• Track utilization of existing systems

• Track workload• Define system to

support workloads

• Test

• Define service levels


Capacity: Use of resources

PROXY

Datacenter

WEB

MQ/ESB

DCOM

CORBA

SERVER

APP SVR

LDAP

WebServices

The overall system has a capacity; each component has a capacity.


Workload model validation

Acceptable

Actual workload

SystemMeasured response times,

throughputs, utilizations, etc.

Model calibration

Synthetic workload

SystemMeasured response times,

throughputs, utilizations, etc.

Yes

No




PE: Application performance management


validation

CPCapacity planning


resolution


management


Five elements of APM (Gartner)

Monitoring the performance of complex distributed applications

End user experience measurement – browser/mobile device

Create a model of the run-time environment (discovery)

Profile the performance and behavior of user-defined transactions

Performance metrics from each of the applications/systems technical components (Webserver, App server, Database, etc.)

Application performance management database

1

2

3

4

5


End user experience monitoring


Mobile

Track key user experience metrics

Network latency

Browser rendering time

Total end-user response time

Ren

derin

g

Mobile Real User Monitoring


And see every hop of the transaction

Mobile

App serverWeb serverProxy server

DatabaseMainframe

Data warehouse gateway

Dynamic Transaction Path Detection

Ren

derin

g




PE: Problem detection and resolution


validation

CPCapacity planning


management


resolution


Performance issues are found in:• Preproduction environment• Production and operations environments

Key business transactions are not meeting their service level objectives:• End user experience is degrading (acute or chronic)• Overnight processing is taking longer• Stability and performance issues

A solution must be found quickly (under pressure)

Performance issues and root-cause analysis

This leverages several knowledge areas


When problems occur

Flash web site events (all at once)

New code released causes problems

Infrastructure upgrade

Consolidation of applications and servers

New workload model (batch and online)


Underlying competencies

Analytical thinking

problem solver

Application and tool knowledge

Workload model

development

Statistics

Test data management

Project management

Communication skills

Behavioral characteristics

Test execution process

Forecasting

Business Knowledge

Monitoring and tuning

Underlying competencies


Technical competencies

Technical competencies

Code profiling

tools (Static analysis,

diagnostics)

Database monitoring

tools

End to end performance

tools

Deep dive tools

(HP Diagnostics, CA Wily, etc, Jprobe, etc)

Test design and

execution tools

Comprehensive results collection

and reporting

Resource utilization system

level

Queuing and

modeling tools

Statistics and Distributions

Test data creation

Little’s law


PE Techniques

Brainstorming Data modeling

Creating a workload model Performance testing

Triage approach Static code analysis

System metrics collection Logging and instrumentation


PE roles: The team – many possible roles

Managing and developing the core team members

Technical Project manager

Performance Architect

Sr. Performance Engineer


Sr. Test automation technician


Performance data engineer

Sr. Capacity planner

Capacity planner

Modeler

1

2

3

4

5

6

7

8

9

10


Career path: Performance engineer foundation

Grow your breadth of capabilities

Grow your leadership and communication


Performance testing

Lead technical

role

Workload models

Communication

Production triage

Technical architecture

Additional tech

competency

Core technical

competency


• Send out email to all registered customers (500,000 currently), there is a link in the email.

• Run multiple TV ad’s encouraging people to register and then come to the event, results in 100,000 new registrations

• No new functionality required to support the event

• Potential 600,000 users of the web site

Planning a large event to introduce a new product

Case study: Preparing for a flash marketing event

• How does this translate into system workload and utilization?

• Do we need more servers?

• The current system was not really designed for such a flash event

The business is forecasting at least a 10X increase during the event, in terms of product sales


PE: Process to prepare for the event (10X)

Forecast the workload

Measure the current workload and

resource utilization

Plan and execute PT for forecast

workload

SAPVT

APM PDR

CP CP

Tune application based on results


CP: Workload model

Step Transaction – by product Percentage

1 Login 100

2 Search product 100

3 View product detail 100

4 Add to shopping cart 50

5 Confirm purchase method 50

6 Shipping information 50

7 Send confirmation email 50

CP

The CP team uses this information to forecast the new workload 10X.


Review current Production workload, and the under lying system utilization.

• Average workload

• Current peak workload

CP: Analyze current capacity

System utilization100%

Response time

Service level

CP


• Use the workload model from CP

• Enable APM during testing

• Work with the Application Architect

PVT: planning, executing, and analysis

Methodology is iterative and tactical

Review infrastructure & architecture

Define business activity profiles &

service levels

• Identify risk areas• Review configuration

settings, topology, & sizing• Define points of

measurement

• Types & numbers of users• Business activities &

frequencies

Iterate testing & tuning

• Test data generation• Create test scripts• User & transaction profiles• Infrastructure configuration

Design & build tests

PVT


More information:

www.collaborative.com

[email protected]

www.correlsense.com/demo

www.real-user-monitoring.com

Questions?

Thank you!

Get your free copy SharePath RUM today!

http://www.collaborative.com/

mailto:[email protected]



http://www.correlsense.com/demo



http://www.real-user-monitoring.com/



An Introduction to Software Performance Engineering

Technology

profession of performance

collaborative consulting

performance guidelines

performance architect

performance tester

performance engineer

performance data engineer

profession engineering