Changing rules 3_performance testing across lifecycle_slideshare

CLOUD TESTINGRewriting the Rules of Performance Testing

RULE 3: Performance Testing Across the Lifecycle

SOASTA Webinar Series

Performance Has Been a Lab Exercise

Dev & Test Lab Production EnvironmentDEPLOY

Performance Testing is often considered hard & non-repeatable due to real constraints:

• Linear, multi-step approaches• Non-correlated, offline analysis• Disconnected teams• Costly test hardware• Time-intensive test setup• Inability to hit realistic scale

Status Quo:small test window prior

to release

Time-consuming testing limits deployment options

Do We Spend Time Where It Counts?

TIME

TESTHW Setup FIXANALYZE RETEST REPEAT?

BUILD TEST

ANALYZE

FIX

Traditional Approaches

Eliminate constraints and TEST MORE, at SCALE, in LESS TIME

Many approaches are process and pre-test heavy

Agile approaches deliver results fast

TEST BUILDING

TEST

ANALYZE

FIX

HW

Setup

HW

Setup

TEST

ANALYZE

FIX

HW

Setup

TEST

ANALYZE

FIX

HW

Setup

TEST

ANALYZE

FIX

HW

Setup

Extra time!

Continuous Performance Testing is Progressive

Dev & Test Lab Production Environment

Start small, early to root

out code-level issues

Include performance focus in iterations

Hit expected scale in

staging and maintenance windows

Test in Production

Recheck regularly for regression

and capacity

Plan for growth

Keys to agile performance testing? SPEED – SCALE – COST

s

Memory leaks

Scale and Time

Sta

ge /

Team

Performance Bugs Know No BordersIssues span teams, time and scale

Release& Deploy

Network& Ops

CDN file placement

Load Balancer configuration

Network bandwidth

Network configuration

DNS routing

Inadequate server resources

Default configuration settings

Unbalanced web serversAuto-scaling failures

Latency between systems

Slow third-party plug-ins

Garbage collection

Slow pages

Conflict with other apps

Dev & Test

Test Lab Staging Production (100% +++)

Method-level tuning

Max sockets exceeded

Firewall max capacity

Memory leaks

Caching issues and benefits

Capacity limits

Code bugs

Database thread counts

Inefficient database queries

Search technology limits

Security bottlenecks

Global latency variance

Complete Performance TestingProactive and Predictive

Performance Coverage Status

Begin to track performance coverage across a range of

critical objectives!

Track performance regression (and progression) over time!

Let’s Look at the NEW RULES

Matt Heusser

Performance Testing: Early, Often, Always

[email protected]

• Three concrete techniques

• No out of pocket cost

• Learn something new

• Expand your options for performance testing

Goals Today

Assumptions• Software Under Test is a web-based system• Prefer: Running Continuous Integration• Prefer: With some browser-driving automated checks

What has been will be again,    what has been done will be done again;

   there is nothing new under the sun. - Ecclesiastes 1:9 (NIV)

No, this is not new

•XP Explained: 2000

•Agile Manifesto: 2001

•XP Installed: 2002

•Selenium Released: 2004

•Scrum Gets Popular: 2005

But there is a little new here

Ten years of improvement in iterative development and incremental functional tests, yet …

* Performance testing tends to remain at the end, driving up cost and causing

delays *

Approach #1: Test Concurrency

• Script takes 10-15 minutes to execute

• Test in groups of 4 to 14

• Test for 15-45 minutes

• Add randomization at the end

A Sample Script …

1. Log-In2. Search for Part ABC33. Add Part to your Cart4. Search/Add part ABC45. Add to your cart6. Change Qty of ABC3 to 37. Checkout8. Logout9. Log-in10. etc

#2: Monitor and Aggregate Functional Test Results

•Wrap logs with timing

•Run the automated tests

•Report your finding

#2: Monitor and Aggregate:An Implementation

#3: Run functional tests under load

•Hook up a load testing tool running in the background

•Run the automated tests “just” like before

•Performance Tests become load tests!

•Bonus: Mine Staging and Prod for load test data!

Putting it all together:

• We covered three ways to incrementally test system performance

• Some performance testing earlier is better than none

• Most obvious performance tests are concurrency issues and can be found by a small group quickly.

• So go find them early!

• Finding perf/load earlier means the programmer will recognize what he did to introduce it – easy fix!

Q&A

Contact SOASTA:www.soasta.com/cloudtest/[email protected] us:

twitter.com/cloudtest

facebook.com/cloudtest

Contact [email protected] Matt:

twitter.com/mheusser

Questions & Answers will post on CloudLink: http://cloudlink.soasta.com

Previous & Upcoming SOASTA Webinars:Stop Cheating & Start Running Realistic Tests (w/ Scott Barber)

Run More Tests & Find More Issues (w/ Neil Gunther)http://www.soasta.com/info-center/webinars/