Load testing with Visual Studio and Azure - Andrew Siemer

Performance testing with Visual Studio & Azure

How to test your application using the tools you already know!

Andrew Siemer | Clear Measure

[email protected]

@asiemer

Andrew Siemerhttp://about.me/andrewsiemer

ASP InsiderMS v-TSP (Azure)

Azure Advisor ProgramFather of 6. Jack of all trades, master of some.

We are hiring!!!

Introduction

Agenda

• What is performance testing?

• How can Visual Studio help?

• How can Azure make load testing easy?

What is performance

testing?

What is it?

• A type of testing intended to determine• Responsiveness

• Throughput

• Reliability

• Scalability

…under a given workload

Why would I care?

• Are you ready to go to production?

• Can you handle the expected load?

• Do you have edge case bugs?

• How much traffic causes your app to tip over?

• Do you need to optimize something?

• Have new features caused the app to support less traffic?

• Do you have enough hardware to support known load?

• What causes your application to fail spectacularly?

Three types of performance tests

• Performance testing

• Load testing

• Stress testing

Performance testing

• Speed

• Scalability

• Stability

• Achieves expected performance in general use case

• Can be run often

• Cheap to run with low volume

Load testing

• Expected performance remains steady under production load

• Assumes everything is normal

• Does the network handle this load

• Can the database handle it

• Is the application still meeting SLA and usable

Stress testing

• Pushing the application beyond expected limits

• Identifies ceilings in capacity

• Tests low memory, disk space limitations, or dead server

• Helps see how and when an application will fail

• What happens when the network gets clogged

Many other types of tests

• Capacity (system capacity meets business volume)

• Component (component meeting expectations)

• Endurance (is performance consistent over time)

• Investigation (performance trending over time)

• Smoke (build ready for perf testing)

• Spike (temporarily exceeds expected load)

• Unit (segment of code reasonably efficient)

• Validation (faster or slower)

Baselines

• A baseline tells you where the app is now

• Allows you to see change over time• Target to demonstrates improvements

• Baselines can be created for• System

• Component

• Application

Benchmarking

• The comparison of a current test with a baseline• Looking for changes in the results

• Or a comparison against industry standards• Are my commerce pages as fast as the industry expects

When to shift from testing to tuning?

• Once testing has found unacceptable results

• When results are acceptable, but servers are running extra hot

• When an SLA is breached that we know we need to fix• Especially when it impacts a large test surface

Test planning

Before we test

• Define acceptance criteria

• Write down key scenarios

• Create workload model

• Target load levels

• Determine metrics to capture

• Design tests

Acceptance criteria

• Write down your objectives• Response time – page load speed

• Throughput - number of concurrent users

• Resource utilization – processor, memory, disk I/O, network I/O

• Maximum user load

• Business metrics – number of orders to handle

User scenarios

• “Happy paths” through your application

• Touches many components in your system

• Commonly used paths through the application

• Highest business value paths through the application

User scenarios - examples

• Log in

• Browse catalog

• Add to cart

• Check out

• Register

• Search

• Faceted navigation

User scenarios – specific example

• For a given scenario – determine activities in the test

• The scenario “add to cart” may include these activities• View home page

• View dirt bike category page

• Choose grips

• Add a grip to the cart

• Go to the cart

• Validate that the item is in the cart

User scenarios – most common/intensive

• Be sure to include highest use scenarios

• And include most resource intensive scenarios

• Look at your existing applications log files for top use case

Define the workload

• Know and attempt to simulate existing traffic patterns

• Understand user delay, or “think” time• A load test should not be a denial of service attack!

• Plan scenarios around average session times• Not too short or too long

• Not every scenario can be a new user, returning user, or either• Plan around the reality of your application

Target load levels

• Load levels are applied to “workload”

• Understand business volume as it relates to your objectives• Common load, vs. a big marketing push, or black Friday

• Key scenarios

• Distribution of work

• Average session times

Define what metrics to capture - business

• Too many metrics can make the results hard to read

• Ask questions related to performance that have specific answers• How many orders are place per minute

• What is the response time of the cart page

• Identify what metrics to capture based on your questions

• Looks for lower level metrics that help answer your questions

• Reevaluate this often – applications change – so should the metrics

Define what metrics to capture - application

• Network – hardware, switches, routers, gateways, load balancers

• System – disk, processor, memory, network

• Platform – the host of your app: IIS, worker role, web role, VM

• Application – perf counters, instrumentation, file locks, db locks, queue

Design tests

• Using your thought so far design specific tests to meet your needs

• Don’t change the tests because it is hard to write it as designed

• Test expected data as well as unexpected data (form validation, bad credit card, etc.)

• Be sure to include think time

• Best tests data is collected from production data (where applicable)

• Think about spiders, batch processes, while real users are browsing

• Don’t over simplify your tests!

How can Visual Studio help?

Visual Studio 2013 Ultimate

• VS feature matrix: http://goo.gl/3VtY01

• Ultimate gets you web load and performance testing

• Test Pro doesn’t get you this functionality!

Web performance tests

The beginnings of our load testing!

The test project

Recording a test

Recording a test – uh oh!

• The first time you try to record, IE likely won’t work

• Basically, you need to• Disable Enhanced Protection Mode

• Enable Web Test Recorder

• Go here for full steps to fix this: http://goo.gl/LcFNSJ

Recording a test – uh oh!

Recording a test

Recording a test

Recording a test

Verbose recording…likely too much!

Let’s clean up the recording

Pruned tests

Recorded tests cleaned up

Run the tests locally

Run the tests locally

Modifying test run settings

Making dynamic tests

Way more useful!

Once you start seeing repetition

Consider investigating patterns in the tests

• Look for patterns in the requests

• Create a data source with variable data• Sku’s

• Makes

• Models

• Year

• Dynamically create a wide set of tests

Adding a data source

• Once tests need to cover more…

• Or you need to create representative load…

• Types of data source• Database – “select * from tablename”

• CSV

• XML

Create a shippable datasource

Add the data source to your project

Point at the file then magic

Choose the table to include

And then we have a data source

Update recording to use data source

Update recording to use data source

If the test only runs once…

View the dynamic results

Conditional & loop steps

• Context parameter exists (test context)

• Cookie exists

• Cookie value comparisons

• Last request outcome• Pass or fail

• Last response code • http response codes

• Number comparison

• String comparison

• Probability rules• Return true some times

• Loops

Plug ins

• Web test plug ins

• Web test request plugins

• Should be in a separate assembly

• Allows you to run code pre or post a test/request execution

• Microsoft.VisualStudio.TestTools.WebTesting• WebTestPlugin

• WebTestRequestPlugin

Load tests

Web tests to the max!!!

• Use the web test scenarios you create

• But put the workload, load, metrics, browser mix, and other points around them

Load test wizard

• Running the load test wizard makes life easy!

When matching transaction percentages

When matching percentage of users per test

When matching pace of user per test

When matching pace of user per test

Choose to tests to include

Set distribution for each test

Define network mix

Define browser mix

Capture computer metrics

Configure the run frequency

Ready to run!

Errors?

But waitthere’s more!

How can Azure let you go big?

Your laptop can’t tip over production!

• Local tests are great to capture big pain points

• Or to capture simple content missing errors

• Or to run as a smoke test

Test farm in Azure is magic!

• With Visual Studio Online• And Azure

• We can go big!

Open local test settings

Choose to run in VSO

Connect to your VSO account

Choose your team project

And run the test!

Ok, that rocks, HOW MUCH?

• Based on “Virtual Users Minutes” – VUM

• Azure calculator for load testing: • http://goo.gl/XBrkY1

• $.0004/VUM for 20,001-2M VUM

• $.0002/VUM for 2,000,001-10M VUM

• $.0001/VUM for usage above 10M VUM/MO

Questions?

Andrew Siemer - Clear Measure

[email protected]

(512) 387-1976

@asiemer