TESTING AND OPTIMIZING WEB APPLICATION’S PERFORMANCE AQA CASE STUDY
TESTING AND OPTIMIZING WEB APPLICATION’S PERFORMANCE
AQA CASE STUDY
Intro to Load Testing2
Copyright ©2009 TEST4LOAD – Software Load Test Experts
What is Load Testing?
Load testing generally refers to the practice of modeling the expected usage of a software program by simulating multiple users accessing the program's services concurrently.
As such, this testing is most relevant for multi-user systems, often one built using a client/server model, such as web applications.
Copyright ©2009 TEST4LOAD – Software Load Test Experts 3
Why Load Test Your Application?
High cost of mission-critical application failure
Simulate real-life workload conditions
Find boundaries of your system
Locate potential problems before your customers do
Reduce development cycle and time-to-market
Predict and reduce infrastructure costs
Copyright ©2009 TEST4LOAD – Software Load Test Experts 4
Load Testing Goals
Performance Ensure your application’s performance under basic
workload (baseline) Stress Find break points and boundaries of your applicationReliability How stable is your application under extended period
of loadScalability Determine how your application scales for the
workload growth
Copyright ©2009 TEST4LOAD – Software Load Test Experts 5
Load Testing and Product Life Cycle
Copyright ©2009 TEST4LOAD – Software Load Test Experts 6
• Define performance requirementsRequirements
• Design your system based on performance requirementsDesign
• Load test separate componentsDevelopment
• Load test a whole systemTesting
• Monitor your system’s performanceProduction
Load Testing Workflow
Plan Script Test Analyze Improve
Copyright ©2009 TEST4LOAD – Software Load Test Experts 7
AQA Case Study8
Copyright ©2009 TEST4LOAD – Software Load Test Experts
Background – AQA
The Assessment and Qualifications Alliance (AQA) is the largest of the three English exam boards.
Awarding 49% of full course GCSEs and 42% of A-levels nationally.
Processing over 3.5 million exams each year.
Providing online services to teachers, exams officers, examiners and students.
Copyright ©2009 TEST4LOAD – Software Load Test Experts 9
SUT – ERA (Enhanced Results Analysis)
ERA is a free on-line tool that gives teachers in UK an instant analysis of exam results.
ERA’s key features:
Grades reports by centre and subject
Cross-centre and years results comparison
Individual students grades analysis
Question by question analysis
Copyright ©2009 TEST4LOAD – Software Load Test Experts 10
The Challenge
Database size over 100 million records
10 million active records
70,000 users
Copyright ©2009 TEST4LOAD – Software Load Test Experts 11
22/08/2007 - Requests per 10 minutes
0
500
1000
1500
2000
2500
3000
00:00
01:00
02:00
03:00
04:00
05:00
06:00
07:00
08:00
09:00
10:00
11:00
12:00
13:00
14:00
15:00
16:00
17:00
18:00
19:00
20:00
21:00
22:00
23:00
14/08/2007 - Requests per 10 minutes
0
500
1000
1500
2000
2500
3000
00:00
01:00
02:00
03:00
04:00
05:00
06:00
07:00
08:00
09:00
10:00
11:00
12:00
13:00
14:00
15:00
16:00
17:00
18:00
19:00
20:00
21:00
22:00
23:00
Workload of less than
10 pages/min at usual
day
Pick of 300 pages/min
at results day
12
Deployment Architecture
Copyright ©2009 TEST4LOAD – Software Load Test Experts
13
Test Architecture
Copyright ©2009 TEST4LOAD – Software Load Test Experts
Server Configuration
Oracle Web Cache Microsoft Windows Server 32bit Oracle Web Cache 10.2.2
Oracle AS RedHat Linux ES4 U6 Oracle Application Server 10.3.3 ChartFX 6.2
Oracle RDBMS RedHat Linux AS4 U7 Oracle RDBMS 10.2.0.3
Identifying Test Scenarios
Pick 5 most common scenarios in your system
Don’t add scenarios which operate similar back-end flows
Identify problematic back-end flows ahead and make sure scenarios cover them.
14
Copyright ©2009 TEST4LOAD – Software Load Test Experts
Identifying Potential Bottlenecks
Search Usually sensitive to various input values.
Simple or not implemented paging can cause to slow response time
Search Results If paging is not implemented, result page can be very large
Back to search results page from item details can execute search again
External and 3rd Party Components 3rd party components can cause to deadlocks or poor performance
OSSO server can perform slower than the rest of the system
Export to Excel Usually returns large result set. Can cause to slow performance.
Copyright ©2009 TEST4LOAD – Software Load Test Experts 15
Execution Plan
Test the system with “light” input parameters first
Get through the “obvious” bottlenecks, constantly increasing the workload
Run nightly tests every night to ensure reliability and stability of the system
At level of 500 users, apply “heavy” input parameters
Test system for randomly distributed workload
Copyright ©2009 TEST4LOAD – Software Load Test Experts 16
17
10 Users – Conn. Pool Starvation
Copyright ©2009 TEST4LOAD – Software Load Test Experts
0%
20%
40%
60%
80%
100%
Connection Pool
Connections
Measurement Avg./Sum Max.
Page Time 1.6 sec 34 sec
HTTP 200 3,348
HTTP 304 2,190
HTTP 302 410
HTTP 500 48
18
100 Users – Apache Shows Its Limit
Copyright ©2009 TEST4LOAD – Software Load Test Experts
Measurement Avg./Sum Max.
Load Size 100
Page Time 5.6 sec 67 sec
Pages per Second 2.1 7.5
HTTP 503 8
19
500 Users – Slow DB & OSSO
Copyright ©2009 TEST4LOAD – Software Load Test Experts
Measurement Avg/Sum Max.
Load Size 1000
Page Time 24 sec 2.5 min
Pages per Second 7 19
HTTP 500 368
HTTP 503 157
1000 Users – Search & Large Content
Copyright ©2009 TEST4LOAD – Software Load Test Experts 20
Measurement Average/Sum
Max.
Load Size 1000
Page Time 1 min 7.5 min
HTTP 500 27
HTTP 503 626
21
Concluding 1000 Users 8 hrs Test
Copyright ©2009 TEST4LOAD – Software Load Test Experts
Measurement Average/Sum
Max.
Load Size 1000
Page Time 0.717 sec 6.169 sec
Load Issues Pyramid
10 – Conn. Starvation
100 – Apache Limit
500 – Slow DB Queries
1000 – Code, Search, Cache
Copyright ©2009 TEST4LOAD – Software Load Test Experts 22
Achieved Results
Supported concurrent sessions/users:
From 10 to 1000 (x100)
Reliability
From 90% failure to ~100% reliable system
Response Time at 1000 VU
From 1 min to less than a second
Pages per Minute
From 13 PPM to 166 PPM
Copyright ©2009 TEST4LOAD – Software Load Test Experts 23
Tips for Effective Load Testing24
Copyright ©2009 TEST4LOAD – Software Load Test Experts
Optimizing Search Mechanism
Use existing solutions.
Use free text search only as the last option.
Limit user input to reasonable values.
Implement paging on both search and result levels.
Improve “Back to search results” mechanism.
Copyright ©2009 TEST4LOAD – Software Load Test Experts 25
GZip Compression Effectiveness
Pros: Reduce size of large text pages by 80%-90%
Improve response time by 50%
Decrease throughput
Cons: Increased CPU utilization by 35%
Tip: Apply GZip selectively to text-based static resources and
large dynamic pages
Do not apply GZip to every response!
Copyright ©2009 TEST4LOAD – Software Load Test Experts 26
Database vs Web Servers Costs
Additional database setup and maintenance costs are significantly higher than of web server.
Make DB work easier by making web server work harder.
Memory costs less than CPU or new hardware – rely more on memory, but keep in mind that everything has its limits.
Copyright ©2009 TEST4LOAD – Software Load Test Experts 27
Developers and Testers Tips
Get to know your user.
Evaluate his system usage behavior or try to predict the common flow.
Know the “costly” operations of your system. Try to build the flow that omits recurrent usage of the operations.
Do not be reluctant to cache data, still don’t overuse the cache. Remember that shared data should be stored once for all users.
Copyright ©2009 TEST4LOAD – Software Load Test Experts 28
Project Manager Tips
Make sure all involved parties are available at the moment of test execution.
Share test findings among all project members.
Log all changes to the test environment. You will need to reapply them later to the production environment.
Make sure the final results meet the performance requirements and business user expectations.
Copyright ©2009 TEST4LOAD – Software Load Test Experts 29
Team Synergy
Synergy and cooperation between the following teams is crucial for the success of the test:
Database Team
Application/Development Team
System Team
Performance Team or Specialist
Copyright ©2009 TEST4LOAD – Software Load Test Experts 30
Nachum (Nick) Dimer
Owner & Chief Consultant
IL: +972.54.4600515
UK: +44.783.8673770
PL: +48.519.717875
www.test4load.com
Contact Info
Questions?