Performance Testing Web 2.0 Applications—in an Agile World

W17 Performance Testing

5/1/2013 3:00:00 PM

Performance Testing Web 2.0

Applications - in an Agile World

Presented by:

Mohit Verma

Tufts Health Plan

Brought to you by:

340 Corporate Way, Suite 300, Orange Park, FL 32073

888-268-8770 ∙ 904-278-0524 ∙ [email protected] ∙ www.sqe.com

Mohit Verma

At Tufts Health Plan Mohit Verma is the lead software performance architect in charge of performance testing of enterprise applications. The environment at Tufts includes web, mobile, EDI, SOA, legacy, and proprietary applications. Mohit has been working with open source performance and functional test tools for the past fourteen years. He has implemented performance test solutions for numerous complex and high-end projects. Previously, Mohit was at John Hancock Financial Services as a performance engineer, leading the performance testing of enterprise applications across all business units. Mohit is a member of the IEEE, Computer Measurement Group, and the Workshop on Performance and Reliability.

4/16/2013

1

Performance Engineering a

Web 2.0 Application in an

Agile world:STAREAST 2013

Mohit Verma

Tufts Health Plan

Lead Performance Architect

Company Confidential 1


Agenda

Web 1.0 vs Web 2.0

Project Description

Technical Environment

Performance Test Strategy

Test Results

Application Improvements/Tuning

Agile vs Waterfall

Learnings/Review

4/16/2013

2

Web 1.0 vs Web 2.0

Web 1.0 – typical static and server driven content - user has little control for optimizing content to individual needs

Web 2.0 – User Driven contents – user has many

tools to drive content through AJAX/Javascript, DOJO, YUI Toolkits, Trinidad (MyFaces),

etc

Blogs, Wikis

SecondLife, Facebook,

Twitter (social networking)



Project Description

� Since we are a Managed Care Provider (HMO)-

our applications typically support Health Care

Providers (hospitals, etc), Employers and our

Members. This Project supported the Customer

Service Reps and had 5 Sprints

� Sprint 1 - Member & Group functionality – 50 users

� Sprint 2 - Provider access functionality – 50 users

� Sprint 3 - Benefit functionality & updates to Member

and Provider – 150 users

� Sprint 4 - Referrals/Claims – no increase in users

� Sprint 5 – Adhoc SRs – no increase in users

4/16/2013

3


Technical Environment

Web Application Servers: Weblogic, WebSphere, JBOSS, Aqualogic Infrastructure

Security: CA SiteMinder

Web Server: Apache

Middleware: Tibco BusinessWorks and EDIFECS

Siebel, Lawson, Actuate,Cognos

Midrange: HP Itanium

Hardware: Linux, HP Itanium and Windows environment

Databases: Oracle and Sql-Server

EDI interfaces HIPAA 5010

Performance Test Tools: Loadrunner, Custom Built Scripts (VB), Open Source


Performance Testing Strategy:Attend daily SCRUM meetings to identify Performance Sensitive transactions early

Pair with Lead Application Architect

Achieve a 5 sec response time 90% of the time for all App screens (SLA)

Proactively test the new functionality via Web Service calls (Member, Benefits and Claims) prior to design completion during the Sprints

Test the Member, Provider, Referral, Benefits and Claims screens prior to deployment (Sprint-Based)

Conduct Testing in production like environment /configuration if possible

Execute Stress and Endurance Tests for Web profile (ALL WEB APPS) to identify breaking point and confirm stability of the Web environment under load

Execute & Monitor sociability tests (Web + EDI)

4/16/2013

4

Tests Executed

POC Tests:

• Benefits Web Service (200 Concurrent Users)

• Claims Web Service (200 Concurrent Users)

• Member Web Service (200 Concurrent Users)

Sprint 1:

• 50 User Test for Member & Group functionality

Sprint 2:

• 110 User Test for Member & Group & Provider

Sprint 3:

• 250 User Test for Member, Group, Provider & Benefits + ALL WEB

Sprint 4:

• 250 User Test for Member, Group, Provider, Referrals, Benefits & Claims + ALL WEB

Sprint 5:

• 250 User Test for Member, Group, Provider, Referrals, Benefits & Claims + ALL WEB

Company Confidential

POC Test Results (1 of 2) – Benefits &

Claims Web Service

Claims Transactions in RED were candidates for tuning to meet the 3 sec SLA

for the Benefit and Claims web services

200 User Test - 5/10/11 - Benefit Detail Web Service

Transaction Name SLA

Status

Minimum Average Maxi

mum

90

Percent

Pass

CMT_BenDetails 3 secs 0.032 0.058 4.163 0.07 1,990

Claims Web Service POC Test Results, 05/26/11

200 user test May 26, 2011

Transaction Name SLA

Status

Minimum Average Maxi

mum

90

Percent

Pass

CMT_ClaimsByClaimsNumber 3 secs 0.097 0.233 1.407 0.348 464

CMT_ClaimsByLargeProvider 3 secs 0.282 4.37 9.594 7.114 46

CMT_ClaimsByMemberID 3 secs 0.112 4.593 28.11 9.354 977

CMT_ClaimsBysmallProvider 3 secs 0.088 0.403 3.097 0.796 661


4/16/2013

5

POC Test Results (2) – Claims

Web Service


CLAIMS WEB SERVICE: Sept 29th -Nov 21 Test Results, response times in seconds for 90 %tile, 200 Concurrent users

Claims Web Service Test - Sept 29- Oct 7, 2011

- Searching Independent claims by providers

Large Provider

15 days,Small

provider - 12

months - Old

Database

Server

Large Provider

15 days,Small

provider - 12

months – New

Database Server

Large Provider

15 days,Small

provider - 6

months – New

Database Server

Large Provider

15 days,Small

provider - 12

months – New

Database Server

with new claims

query returning

10 rows at a time

Large Provider

30 days,Small

provider - 12

months – New

Database Server

with new claims

query returning

10 rows at a time

Large Provider 30

days,Small

provider - 12

months – New

Database Server

with new claims

query returning

10 rows at a time

– TRINIDAD Table

Detailing Turned

off

Transaction Name 90 Percent 90 Percent 90 Percent 90 Percent 90 Percent 90 Percent

ClaimsByClaimsNumber 0.4 0.28 0.28 2.1 2.2 0.2

ClaimsByLargeProvider 11.6 7.4 4.5 4.2 13.8 5.3

ClaimsByMemberID 3.6 2.2 2.2 2.7 3.7 1.7

ClaimsBysmallProvider 15.3 6.29 3.17 3.3 9.2 2.9

Application - 250 Web User

Performance Test Results90 percent times are reflective of 90th %tile times in seconds, for the server-side component onlyNumber of Concurrent Users 250User 250User 250User 250User 250User 250User

Changes Made to Environment (CMT

Managed Servers)

4 nodes for

CMT , JVM 2

GB, Services

CMT JVM

modified -

permsize ,

No Benefits

4 nodes for

CMT , JVM 2

GB, CMT JVM

permsize

modified.

Diffe rent

UserIDs

4 nodes for

CMT , JVM 2

GB, CMT JVM

permsize

modified.

Diffe rent

Use rIDs,

Upped

memory for

T ibco Service

4 nodes for

CMT , JVM 2

GB, CMT

JVM

permsize

modified.

Diffe rent

Use rIDs,

Upped

memory for

T ibco

Service

4 nodes for

CMT , JVM 2

GB, CMT

JVM

permsize

modified.

Diffe rent

UserIDs,

Upped

memory for

Services,

T ibco

Threads

doubled.

4 nodes for

CMT , JVM 2

GB, CMT JVM

permsize

modified.

Diffe rent

UserIDs,

Upped

memory for

T ibco, T ibco

Threads

tripled for

T ibco

Services

May 29 - July 15, 2011 29-Jun 7-Jul 11-Jul 12-Jul 14-Jul 15-Jul

T ransaction Name 90 Percent 90 Percent 90 Percent 90 Percent 90 Pe rcent 90 Pe rcent

ClickOnCoveringProviders 4.434 9.877 5.362 5.489 3.315 3.082

ClickOnDependentVerification 5.34 4.599 4.176 5.385 4.916 3.39

ClickonGroupBenefits 7.527 4.824 4.32 4.134 4.027

ClickonGroupDeductibleOOP 3.399 2.821 3.117 2.767 2.226

ClickOnGroupHistory 4.035 6.53 5.547 5.797 7.513 6.613

ClickOnGroupNameSearchButton 4.83 6.82 5.26 6.696 5.809 4.573

ClickonGroupPharmacyBenefits 3.399 2.902 3.035 2.759 2.204

ClickOnMemberHistory 5.037 11.096 7.571 7.008 5.866 5.304

ClickonMemberBenefits 17.06 9.339 9.112 8.705 8.457

ClickonMemberDeductibleOOP 9.526 7.064 9.377 8.372 6.694

ClickonMemberPharmacyBenefits 2.447 5.674 5.413 6.324 7.981 4.496

ClickOnMembersOnPlan 7.916 3.673 2.581 2.881 3.088 2.346

ClickOnMemLastNameSearch 6.61 10.426 9.935 11.391 8.663 6.496

ClickOnMemSearchButton 5.108 12.738 7.302 6.899 6.481 5.647

ClickOnNPISearch 2.423 11.692 5.985 5.975 5.023 5.23

ClickOnPrivileging 6.449 2.043 1.816 2.55 1.504 1.26

ClickOnProviderIdsearch 5.921 15.197 7.846 7.181 4.886 4.823

ClickOnProviderNameSearch 9.457 11.483 6.847 7.397 6.423 5.981

ClickOnSubscriberSearchButton 2.814 17.511 7.931 8.955 7.223 7.724

GroupProfileByGroupIDSearch 3.703 6.894 5.172 5.372 5.174 4.322

MemberProfileBySSNSearch 4.022 8.299 6.027 5.636 6.462 4.902

SelectGroupIDFromGroupNameSearch 2.828 4.001 3.606 3.543 3.491 2.697

SelectMemberIDFromSubscriberSearch 4.748 5.225 8.939 3.684 2.784 3.578

SelectMemIDFromLastNameSearchResults 6.103 6.048 3.354 4.007 2.983 2.847

SelectProviderIdFromNameResults 2.403 2.613 2.032 2.128 1.628 1.84

SelectProviderIdFromNPIResults 2.752 3.29 2.431 2.734 2.214 1.884

4/16/2013

6

Server Side Troubleshooting


Client Side Troubleshooting

(1)

12

4/16/2013

7

Browser-Based

Troubleshooting


CPU Monitoring – Unix Servers

Tufts Health Plan 14

4/16/2013

8

Response Time Graph


Additional Resources/Tools

for Client and Server Tuning

Manual Testers/UAT to support Web 2.0 functionality

Use UFT/QTP to emulate real-user

Dynatrace Ajax Browser – Free Version

HttpWatch

Jconsole

Desktop Engineer


4/16/2013

9

Tufts Health Plan 17

Client-Side Tuning – Speed of the

Web

18

Client Side Tuning - Browser Test Results- 7/7/11

IE 7 vs IE 8 with AntiVirus OptionsBrowser

Version

Antivirus Status Member Members on

Plan

Member

History

Group

History

Dependent

Verification

Coordination

of Benefits

IE 7 Version 8.5 6.499242424 3.6 7.6 9.0 4.1 4.1

IE 7

Version 8.8 with

exceptions 6.7 3.8 5.8 7.5 3.5 3.5

Difference from IE7 & 8.5 0.211868687 0.136616162 -1.815656566 -1.490909091 -0.531818182 -0.519696970

IE 8 Version 8.5 7.1 3.3 8.3 10.5 4.0 4.0

Difference from IE7 & 8.5 0.613888889 -0.311868687 0.707575758 1.537373737 -0.026767677 -0.089393939

Difference from IE7 & 8.8 0.402020202 -0.448484848 2.523232323 3.028282828 0.505050505 0.430303030

IE 8

Version 8.8 with

exceptions 6.133333333 2.873333333 4.584444444 9.937777778 3.413333333 3.664444444

Difference from IE7 & 8.5 -0.365909091 -0.745606061 -3.053434343 0.980202020 -0.651818182 -0.388585859

Difference from IE7 & 8.8 -0.577777778 -0.882222222 -1.237777778 2.471111111 -0.120000000 0.131111111

IE 8 No Anti Virus Software 5.7 2.5 5.5 6.6 2.7 3.0

Difference from IE7 & 8.5 -0.832575758 -1.152272727 -2.104545455 -2.357575758 -1.398484848 -1.053030303

Difference from IE7 & 8.8 -1.044444444 -1.288888889 -0.288888889 -0.866666667 -0.866666667 -0.533333333


4/16/2013

10


Application Performance

Improvements - Summary

� Use HashTable instead of HashMap for ThreadSafe transactions

� Increasing threads and JVM Heap with many components

� Claim Search modifications – limiting result set size

� Limiting searches by date range

� Differentiating Small vs Large providers

� Limiting searches by row count

� AntiVirus Modifications

� Add URL Exceptions to user desktops

� Adding more virtual servers – Web and Application servers

� Trinidad (MyFaces) toolkit – Datatable Column Detailing turned off

� Fetching email asynchronously via JMS

� Move from IE 7 (Windows XP) to IE 8

� F5 Switch: Caching, Compression

Version HTTP 1.0

server

(broadband

connection)

HTTP 1.1

server

(broadband

connection)

Internet

Explorer�7 and

earlier

4 2

Internet

Explorer�8

6 6

Agile vs Waterfall


Agile Waterfall

Pros Cons Pros Cons

Early Testing - Defect

Identification

Expensive Traditional performance requirements may be

addressed too late

Design validation Design may change

rapidly with

Requirements rendering

rework

Management typically

understands it or is

more familiar to it

Bad design if not found earlier

could severely impact application

and cause rework and delays

Component-Based (Fine

Grained)

Needs more

app/business team

involvement

Well formed SDLC Application-Based (Coarse-Grained)

Business Buy-in, quicker time-

to-market

Unstable Code, could run

into functional issues

Stable Code using after

QA has gone through

one Test Cycle

Scalability may be dependent on

modifying underlying components

Better End Product Rescripting required

which could be very

expensive to the the

rapidly changing

environments (builds,

etc)

Scripts more stable.

Usually less rescripting

required

Scripts more stable. Usually less

rescripting required

Typically more scalable as

testing done at component

level

Adhoc Project

Management

Scripts more stable.

Usually less rescripting

required

More bureaucracy typically

involved

Decreased Project Risk Usually more test cycles

needed over the

development of the

App/Product

Test Cycles are

typically limited in

alignment with Project

Timelines

More Collaboration and

Information Sharing

Test Environment may

not be prod-like

Test Environment is

typically prod-like

4/16/2013

11

Learnings/Summary

Web 2.0 requires more manual/automated testing in concert with Performance tests

Agile Methodology provides performance benefits in exchange for additional cost

Identify Non-Functional requirements early in order to test for optimal design

Server Side Tuning is still vital

Client Side Tuning requires more developer involvement


References

� http://msdn.microsoft.com/en-

us/library/cc304129(v=vs.85).aspx

� http://developer.yahoo.com/performance/

� http://blog.dynatrace.com/

� High Performance JavaScript (Build

Faster Web Application Interfaces)

Nicholas C. Zakas


Performance Testing Web 2.0 Applications—in an Agile World

Technology

web environment

benefits web service

vs web

user test

web profile

web apps

web service transaction

web service calls member