Next-Generation Performance Testing with Lifecycle Monitoring

T5

Performance Testing

5/8/2014 9:45:00 AM

Next-Generation Performance

Testing with Lifecycle

Monitoring

Presented by:

Scott Barber

SmartBear

Brought to you by:

340 Corporate Way, Suite 300, Orange Park, FL 32073

888-268-8770 ∙ 904-278-0524 ∙ sqeinfo@sqe.com ∙ www.sqe.com

Scott Barber

SmartBear

Chief performance evangelist for SmartBear Scott Barber is a respected leader in the advancement of software testing practices, an industry activist, and load testing celebrity of sorts. Scott authored several books―Performance Testing Guidance for Web Applications, Beautiful Testing, How to Reduce the Cost of Testing, and Web Load Testing for Dummies―and more than 100 articles and blog posts. Founder/president of PerfTestPlus, Scott co-founded the WOPR, served as director of the AST and CMG, and is a founding member of ISST. His industry writing, speaking, and activism focus on improving the effectiveness and business alignment of software development practices. Learn more about Scott Barber.

4/26/2014

1

Next-Generation Performance Testing

with Lifecycle Monitoring

By:

Scott BarberProduct Owner – Load Testing Tools

SmartBear Software, Inc.

Revised for:

Next-Generation Performance Testing

with Lifecycle Monitoring

4/26/2014

2

Product Owner – Load Testing ToolsSmartBear Software, Inc. scott.barber@smartbear.com

@sbarber

Co-Founder: Workshop On Performance and Reliabilitywww.performance-workshop.org

Co-Author:Author: Contributing Author:

Books: www.perftestplus.com/pubs

About me: about.me/scott.barber

System or application characteristics related to:

Speed: - responsiveness- user experience

Scalability: - capacity- load- volume

Stability: - consistency- reliability- stress

What is Performance?

4/26/2014

3

What is a Performant System?

A system optimized to balance:

- Cost

- Time to Market

- Capacity

…while remaining focused on:

Quality of Service to System Users

What is Performance Testing?Testing designed to:

Inform software system optimization by balancing:

- Cost

- Time to Market

- Capacity

while remaining focused on the quality of service to system users.

4/26/2014

4

Performance vs. Load Testing?

Conception to Headstone

Not

Cradle to Grave

The Performance LifeCycle is…

4/26/2014

5

© PerfTestPlus, Inc.

Prevent Poor Performance…

… don’t just react when it happens.

Who is Responsible?

4/26/2014

6

Where does it fit in the SDLC?

Delivering Performant Systems;Commonly, Y2K -> Today

SPE (Software Performance Engineering)

� Math-centric Architect/Design method

+ Profiling� Context-free code optimization

+ Load Testing� Demonstrate (too late) poor performance characteristics

+ Capacity Planning� Math-centric guessing to determine hardware needs

+ APM (Application Performance Management)

� Proactively react to production issues

=> Inefficient & Reactionary Delivery & Maintenance

4/26/2014

7

Delivering Performant Systems;Moving Forward

Minimize Need for Complex Mathematics� Few teams have mathematicians w/ advanced degrees

Disconnect Accuracy from Actual Production Data� Because we can’t depend on what we don’t have yet

Remove Socio-Political Barriers� Collaboration improves information sharing & efficiency

Keep/Adapt most Valuable Practices� Many good concepts have been poorly implemented

Distribute Valuable Practices Throughout LifeCycle� Many valuable practices have suffered from stove-piping

We need one model for the whole ‘Performance Journey’

© PerfTestPlus, Inc.

4/26/2014

8

“DevOps” Perf Team Individual

• Code Profiling

• Requirements Revision

• Design Validation

• Budgets

• Units & Components

• Monitoring

• Tuning

• Load/Stress Testing

• Environment Config

• Capacity Planning

• Concurrency Checks

• Situation Analysis

• Snapshots

• Comparisons

• Niche Concerns

• Mini-Field Studies

(UAT)

• “What-if…”

Key Points• Entire Lifecycle

• Entire Team Focused

• Individual, Role, & Team Accountability & Responsibility

• Continuous & Collaborative

© PerfTestPlus, Inc.

© PerfTestPlus, Inc.

Preventing Poor Performance with

a little work…

every day…

from every one.

4/26/2014

9

© PerfTestPlus, Inc.

Units StoriesTiersResourcesGoals

AssessCompareInvestigateAcceptAnswer

Dev & Prod TimesResourcesSizesFrequenciesDashboard!

ProactiveMicro & Macro Establish GoalsUpdate Targets

© PerfTestPlus, Inc.

4/26/2014

10

© PerfTestPlus, Inc.

An Ounce of Prevention…

4/26/2014

11

© PerfTestPlus, Inc.

© PerfTestPlus, Inc.

4/26/2014

12

Load Tests

Stress Tests

Endurance Tests

Infrastructure/Configuration Verification

Load Variability Analysis

Background Load for Acceptance Tests…

What is it?

Or any other deliberate approximation of

Production Conditions

© PerfTestPlus, Inc.

4/26/2014

13

RPT is:

Inspired by Rapid Software Testing

Consistent with Rapid Software Testing themes

Sanctioned by James Bach, Michael Bolton & the RST

instructors to as a specific implementation of the

Rapid Testing Methodology

For more information about RST, visit:

http://www.satisfice.com/info_rst.shtml

“What have we got?What do we want?

How do we get there…?”--Bob Barber (Scott’s dad)

… as quickly, simply, and

cheaply as possible?

Evolved from:

4/26/2014

14

…a common man’s way of expressing the problem solving approach that classical engineers employ.

• Given: “What have we got?”

• Find: “What do we want?”

• Solve: “How do we get there?”

An approach to respond to a specific

performance-related question after 4 or fewer

hours of team effort with 1 or more of:

A) The answer

B) A partial answer

• To determine the value of additional effort

• The level of effort to provide the answer

C) Better questions to address the underlying

concern

What is it?

4/26/2014

15

1. Receive Question• Clarify the question• Understand the driver(s) behind the question

2. Generate Test Coverage Outline (TCO) (~20 min)• Simplest path to (partial) answer(s)• Comprehensive path to (partial) answer(s)

3. Transform TCO into Rapid Strategy (~20 min)• Only tasks that fit in time box• Stick to tasks requiring available resources

4. Execute Strategy (~2.5 hrs)• Snapshots are your friends• Anecdotal is sufficient

5. Consolidate/Analyze Data (~30 min)• Identify patterns• Confirm patterns (time permitting)

6. Report Results (~20 min)• Answer(s) / Time or Effort to get answer(s)• Follow-on questions of interest

~4 hr

Conceptual Approach

© PerfTestPlus, Inc.

© PerfTestPlus, Inc.

4/26/2014

16

© PerfTestPlus, Inc.

The Bottom Line

© PerfTestPlus, Inc.

4/26/2014

17

Addendum

Questions?

4/26/2014

18

Contact Info

about.me/scott.barber

Product Owner – Load Testing Tools

SmartBear Softwear, Inc.

E-mail:

Scott.barber@smartbear.com

Blog:

scott-barber.blogspot.com

Web Site:

SmartBear.com

Twitter:

@sbarber