AGILE PERFORMANCE TESTING IN THE REAL WORLD · 14 1. Requirements and KPIs 2. Setup Performance CI Environment 3. Performance Scripting Process 4. Define your SLAs 5. Results incorporated

3/26/2019

1

AGILE PERFORMANCE TESTING

IN THE REAL WORLD

Amit PatelTotal Performance Consulting

PERFORMANCE

1 2

3/26/2019

2

3

The HealthCare.gov fiasco is a clear

example of how businesses can seriously

undermine their business goals when launching a

new app or user platform. Brands must do

performance testing even when their app or

website has much smaller loads than

HealthCare.gov.

FastCompany 4 Lessons From Healthcare.gov Epic Failure

OTHERS…

4

3 4

3/26/2019

3

5

PERFORMANCE MATTERS

A 1-second delay in page

load time

Equals 7% loss in

conversions

Equals 11% Fewer page

views Equals 16% decrease in customer

satisfaction

IN DOLLAR TERMS,this means that if your site earns $100,000 a day, you could

lose $2.5 MILLIONin sales.

5 6

3/26/2019

4

NOT ALL ABOUT THE MONEY

7

Customer satisfaction

Proper infrastructure sizing

Brandreputation

Predicatable deployments Understand scalablity

with or w/o cloud

CDNperformance

Container scalablity

Autoscaling in the cloud

7

7 8

3/26/2019

5

2 10

More frequent releases

Performance still often

left until the end

Performance often not

included until major

releases

Automation

implemented as part

of the SDLC, some CI,

CD maybe not

Teams still not fully

connected

9 10

3/26/2019

6

CHALLENGES

11

ProjectsPerformance

Engineers

2

11 12

3/26/2019

7

3 14

• Cultural and Behavioral changes to the organization

• Start to think about performance early from

requirements to initial architecture decisions

• Include performance as stories and backlog items

(non-functional requirements)

• Integrate with different team members

• Testability

• Creating Performance Defects

NEW REQUIREMENTS

13 14

3/26/2019

8

15

MEETINGS

FEEDBACK

TEAMWORK

16

15 16

3/26/2019

9

17

AUTOMATIONAUTOMATION

AUTOMATION

• DevOps emphasizes automation

• Build performance into the

DevOps automation process

• More data analysis conducted

• Testing is automated

• Scripting challenges

• Service Virtualization

• Infrastructure sizing• Availability of back end system• Test earlier (service not fully

ready yet)

DEVOPS TOOL STACK

SCM TESTING CI OPS DEPLOYMENT MONITORING COLLABORATIONCONFIG

17 18

3/26/2019

10

19

LOAD TESTING TOOLS

TYPICAL REQUIREMENTS:

• Project and product requirements

• Cloud based load

• Internal vs external traffic or both

• Skill sets required

• Budget

THINGS TO ALSO CONSIDER:

• Integration with CI solutions (s)

• Monitoring integrations

• SLAs

• API testing / Service Level Testing

20

CI TOOLS

• Scheduling/Triggering performance tests

• SLA pass/fail (error rates, response times, throughput…)

• Allows data to be trended over time (Response Times, Errors, Throughput…)

19 20

3/26/2019

11

21

• Production feedback loop

• Monitoring QA and Staging environments

• Compare Pre-PROD data with PROD

• Automate collection of data

APM SOLUTIONS

22

PRODUCTION FEEDBACK LOOP

21 22

3/26/2019

12

DEEP ANALYSIS

7

PERFORMANCE TEST ANALYSIS

23 24

3/26/2019

13

25

QA STAGE PROD

APM DEPLOYMENT

DEPLOY APM TOOL IN ALL ENVIONRMENTS

COMPARE DATA ACROSS ENVIORNMENT

QA

AUTOMATING DATA COLLECTION

25 26

3/26/2019

14

1. Requirements and KPIs

2. Setup Performance CI Environment

3. Performance Scripting Process

4. Define your SLAs

5. Results incorporated into CI solution

NEXT STEPS

28

REQUIREMENTS & KPIs

➢ Understanding what is the goal and KPIs required for success

➢ Understanding cross team requirements

➢ Types of Tests

➢ Team responsibilities

➢ Cloud Testing

➢ Circuit Breaker

Sample KPIs

• Database query performance

• Traffic being sent Mb/s

• Right amount of logging

• Response times• Time to deploy new code• Rollback time

• Response sizes

• Autoscaling

27 28

3/26/2019

15

29

• End-User Experience tests (UI)

• Traditional load tests• Scaled down tests• Large scale tests• Memory leak tests

• API tests (Service level tests)

• Integration tests

• Autoscaling validation testing

TYPES OF TESTS

PERFORMANCE CI ENVIRONMENT EXAMPLE

30

29 30

3/26/2019

16

HIGH-LEVEL AUTOMATION WORKFLOW EXAMPLE

31

Developer

Tester

Production Environment

Continuous Integration Server

Automated Performance Test

Commit Changes

Committed Changes Trigger CI Build

Source Control

Build Kick Off Test

/

SLAs

32

• Transaction level response times

• Request response times

• API response times

• Use case / script level response times

• Total # of Errors / Error rate

• Throughput

• Response sizes

31 32

3/26/2019

17

33

RESULTS

9

33 34

3/26/2019

18

50%Handled only 50% of projected peak traffic

10Struggled to handle 10 concurrent users without

seeing integration errors

3Back to back months of integration call issues

o Integration Timeouts

o Unable to diagnose and resolve issues

KEY CHALLENGES

36

TEST

BankIntegrationDLL

PROD

ProductIntegrationDLL

PROD

TEST

CASE OF THE INTEGRATION CALLS

35 36

3/26/2019

19

37

TESTING INTEGRATION CALLS

37 38

3/26/2019

20

39

CI SERVICE LEVEL TESTS

CI SERVICE LEVEL TESTS – BUILD TRENDS

9

39 40

3/26/2019

21

41

UI LEVEL TESTS – TRANSACTION TRENDS

Print

Review Application Search

Approve Case

42

TAKE AWAYS

AUTOMATE, LEARN, UPDATE, ITERATE

▪ Collaboration

▪ Testability

▪ Setting up the right tests

▪ Comparing results over time

▪ Automate result collection

41 42

3/26/2019

22

OPS

DEV

PERF

SHARED GOALS, REPORTING, KPIs, TOOLS

Access to PROD data

Testing in PROD

COLLABORATION

1

3 Noti f icat ions ( i. e. Emails, Chat)

Incoporate performance into every team

2 Plan for large scale on- demand tests

ADDITIONAL THOUGHTS

43 44

3/26/2019

23

THANK YOU FOR ATTENDING THIS SESSION

Amit PatelTotal Performance Consulting

@aapatel [email protected]

@TotalPerform www.totalperform.com

QUESTIONS

45