Tis The Season: Load Testing Tips and Checklist for Retail Seasonal Readiness

‘Tis the SeasonA Holiday 2014 Ecommerce Checklist

Preparing for best case retail success

Holiday

2013 was

BIG

2014 will be

BIGGER

mobile

matters

your site

needs to

PERFORM

40 % abandon a

website that takes

more than 3

seconds to load

A 1 second delay

in page response

can result in a 7%

reduction in

conversions

79% of shoppers who

are dissatisfied with

website performance

are less likely to buy

from the same site

again.

If an ecommerce site is making

$10,000 a day, a 1 second

delay could cost you

$250,000 in lost sales every

year

Start NowPrepare for the holiday rush

‘Tis the SeasonA Holiday 2014 Ecommerce Checklist

Preparing for best case retail success

Elizabeth Scott, Tenzing

Director of Technical Services

@BethxScott

Brad Johnson, SOASTA

VP Product Marketing

@bradjohnsonsv

Your Hosts

Infrastructure and Application

Monitoring

Load Testing

Marketing

Agenda

Poll Question

Infrastructure and Application

Assess Your Infrastructure

Current infrastructure

performance and capacity

baseline:

Server

Hardware

Storage

Network

Database

Application

Assess Your Infrastructure

Know how many users can be on

your site: Do a load test

Understand your vendor’s capacity

planning process

Develop an emergency capacity plan

Ensure your

environment is up to date

on latest patch levels

Develop an emergency

patch and security plan

Secure Your Environment

Ensure ports on your

servers are not open

unnecessarily

Limit access


Develop and test your

Denial of Service

response strategy

Run a vulnerability scan


Tune queries,

search results and

session limits

Leverage a CDN

Tune Your Application

Find free capacity:

Adjust your caching

Adjust ‘type ahead’ search

Remove/adjust view all options

Adjust other features which are resource

hogs

Tune Your Application

Monitoring

Application Performance

End User Experience

Traffic Levels

Measure, Analyze, Act

•Upstream Connections

Payment Gateway

Shipping and Fulfillment

Measure, Analyze, Act

Plan for the Inevitable

Assess your risk …

WHAT WILL GO

DOWN AND WHY?


•Create a plan

Website overload plan

Incident response plan

Security/DoS response plan


COMMUNICATE


Modern Load Testing

True Story

• Division of $6B Company

True Story


• Cyber Monday & last Free Shipping Day

True Story



• “We wouldn’t disrupt the plan”

True Story




• Crashed on Cyber Monday

True Story





• No answers on Thursday

True Story





• No answers on Thursday

• Production testing on Friday

reproduced the issue

True Story

Is there anything missing from the

plan?

Prerequisite:

Know Your

Real Users

Modern Load Testing

Stop guessing

about what to test

Most accurate models of real user

behavior

Meaningful scale & appropriate

concurrency

Control to start, stop & modify tests

without delays

Test on your schedule

Modern Load Testing

Test for spikes – using the cloud

Failover Scenarios

Success Disasters

End to End

Test what you don’t think needs to be

tested

Modern Load Testing

Measure,

Test and Act

in Real Time

Modern Load Testing

Don’t Wait until it’s too late

Compress processes for more test iterations

Traditional

Approach

TIME

Test development Infra. setup Fix Repeat?RetestTest Analyze

Modern Load Testing



/

Traditional

Approach

TIME


CloudTest

Modern Load Testing



/

Traditional

Approach

TIME


/ / / / Extra time!CloudTest

Modern Load Testing

Use real user data to

iterate dev & test plans

Run iterative load

tests via Jenkins

Daily (or more frequent) updates

keep teams quality focused

Embedded failure

details & bottleneck

data isolate issues

Manage to a

performance

baseline

Mobile? Run functional

automation on real devices

for functional & performance

validation

Spin up suitable, consistent

test environments in the

cloud or the lab

Jenkins runs any

unit tests as usual

Test Continuously – including performance!

5. Don’t Wait

s

Memory leaks

Test Scale

Sta

ge / T

eam

Dev

& Test

Release

& Deploy

Network

& OpsCDN file placement

Load Balancer configuration

Network bandwidth

Network configuration

DNS routing

Inadequate server resources

Default configuration settings

Unbalanced web serversAuto-scaling failures

Latency between systems

Slow third-party plug-ins

Garbage collection

Database thread counts

Inefficient database queriesSlow pages

Conflict with other apps

Test Lab Staging Production (100% +++)

Search technology limits

Method-level tuning

Max sockets exceeded

Firewall max capacity

Global latency variance

Security bottlenecks

Continuous tuning for full-scale…and well beyond

Start early, end in the real environment

6. Test in production and at Production scale

• Tests for flash sales with

unknown spikes

• mPulse analyzes real user

performance

• Continuous tests of mobile

and web traffic

“SOASTA was very easy to work with and could

accommodate our requirements in a four-week

timeframe.” Gavin Cliffe, OZSale CTO.

OZSale

NO

ISSUES

“SOASTA was very easy to work with and could

accommodate our requirements in a four-week

timeframe.” Gavin Cliffe, OZSale CTO.

OZSale

160%

more

visitors

$5M

AUD in

one sale

50%

mobile

Critical Factors for

Performance Testing Retail Sites

Test for spikes

Test what you think

doesn’t need to be

tested

Test expecting failure

Analyze & act in real

time

Don’t forget about the

user experience

Monitor everything

Don’t wait until just

before the holidays

Test in production

Marketing

Communicate your campaign plan

internally and with stakeholders

Establish a campaign change

process

Approval

Communication

Campaign Management

Establish a change

freeze on application

and infrastructure

Limit publishing and

catalogue changes

Campaign Management

Campaign Best Practices

Segment your list into

smaller groups

Trickle your email

Manage your media content

Email Best Practices

Don’t get too fancy

Develop test cases

for all of your discount

codes

Promotions and Discounts

Avoid using discount

structures that require a

database search

Don’t use new discount

structures


TEST, TEST, TEST!


Are you interested

in a FREE Retail

Readiness

Performance Test

Contact SOASTA:

www.soasta.com

[email protected]

866.344.8766

Follow us:

Twitter.com/cloudtest

Facebook.com/cloudtest

Retail Readiness Page:

soasta.com/retail-readiness/

Case studies:

soasta.com/customers/ecommerce/

Webinars future and past:

soasta.com/knowledge-center/webinars/

CloudTest Lite is FREE (forever)!

soasta.com/FREE

Resources from SOASTA

Thank you

http://www.soasta.com

http://www.soasta.com/retail-readiness/

http://www.soasta.com/customers/ecommerce/

http://www.soasta.com/knowledge-center/webinars/

http://www.soasta.com/FREE

www.SOASTA.com

more than managed hosting. commerce anywhere.

www.tenzing.com

Tis The Season: Load Testing Tips and Checklist for Retail Seasonal Readiness

Technology