175 Varick Street NY, NY 10013|1.855.445.2285|[email protected] @alongir Performance Testing in the Agile Age
May 09, 2015
175 Varick Street NY, NY 10013|1.855.445.2285|[email protected] @alongir
Performance Testing in the
Agile Age
175 Varick Street NY, NY 10013|1.855.445.2285|[email protected] @alongir
Alon Girmonsky About The Speaker
• Hands-On Developer for the Past 20 Years • Expertise
– Large-Scale Web and Mobile Applications – IP & Internet
• Founder & CEO of BlazeMeter – A Load Testing Platform as a Service – 100% Compatible with Apache JMeter
USERS
20,000
TESTS
500,000
175 Varick Street NY, NY 10013|1.855.445.2285|[email protected] @alongir
The Agile Enterprise
• All organizational divisions - not just R&D • It’s about survival in a highly competitive
business environment • Innovate in short Iterations – a MUST! • New DNA • New IT Infrastructure (a.k.a Devop)
175 Varick Street NY, NY 10013|1.855.445.2285|[email protected] @alongir
Agility Enablers
Open Source
Cloud Compu/ng
E.G E.G
175 Varick Street NY, NY 10013|1.855.445.2285|[email protected] @alongir
Performance Testing
Waterfall Agile
Front-‐end
Back-‐end
Func/onal
Load E.G API
E.G Java Script
In The Agile Age
2005 2010 2014
175 Varick Street NY, NY 10013|1.855.445.2285|[email protected] @alongir
Agile? What it Means?
"We'd like to squeeze in a load test for XXXX’s biggest event of the year, Fantasy Football season. Our best opportunity to execute this is tonight 3-7am EST. We'd like to generate a constant 40,000 hits/sec for about 1 hour. .."
Traditionally this would have taken weeks and cost $$$
An Example from a Large Media Company
175 Varick Street NY, NY 10013|1.855.445.2285|[email protected] @alongir
Agile? What it Means?
Traditionally this would have taken weeks and cost $$$
Network Bottleneck Identified
175 Varick Street NY, NY 10013|1.855.445.2285|[email protected] @alongir
For Both Mobile and Web
Massive Global Simulation of Users, Devices and Mobile Networks
Agile Performance Testing
175 Varick Street NY, NY 10013|1.855.445.2285|[email protected] @alongir
480 CPU Cores and
1.8 TB of Memory
What Beast Has .. ?
175 Varick Street NY, NY 10013|1.855.445.2285|[email protected] @alongir
480 CPU cores and 1.8 TB of Memory A Typical Test
EC2 Large 2 cores, 7.5GB
EC2 Large 2 cores, 7.5GB
EC2 Large 2 cores, 7.5GB
EC2 Large 2 cores, 7.5GB
EC2 Large 2 cores, 7.5GB
EC2 XL 4 cores, 15GB
X 200
X 20
175 Varick Street NY, NY 10013|1.855.445.2285|[email protected] @alongir
Mobile Application Simulation A Common Use-Case
Using Agile Performance Testing
• You can create such test in a matter of minutes • You can run dozens of times per day on demand • You can enjoy an interactive reporting that provides an
immediate intelligence
Actions Sign-up Sign-in Purchase Browse
175 Varick Street NY, NY 10013|1.855.445.2285|[email protected] @alongir
Comprehensive scripting at your fingertips
Any Script, Any Scenario
• HTTP / HTTPS • Login / Signup • Form POSTing / Credit card transactions • Web services • Parameters extraction • Mobile • Mongo, MSSQL, Oracle SQL, MySQL • Multiple user profiles
» Any script » Any scenario » Any protocol
175 Varick Street NY, NY 10013|1.855.445.2285|[email protected] @alongir
Agile Summary
1 to 300,000 Concurrency
Open-Source Compatible
Self-Served
TTS < 10m
Any Protocol
Any Scenario
Multi Locations
On-Demand
Comprehensive
175 Varick Street NY, NY 10013|1.855.445.2285|[email protected] @alongir
The Load Sensitivity Point What to look for?
175 Varick Street NY, NY 10013|1.855.445.2285|[email protected] @alongir
Load Performance Types
Perfect Time
Sensitive
Bad
175 Varick Street NY, NY 10013|1.855.445.2285|[email protected] @alongir
Load Performance Types
Perfect
The Perfect Load
Time
175 Varick Street NY, NY 10013|1.855.445.2285|[email protected] @alongir
Load Performance Types
Sensitive
Sensitive to Load
Time
175 Varick Street NY, NY 10013|1.855.445.2285|[email protected] @alongir
Load Performance Types
Bad
Bad Performance
Time
175 Varick Street NY, NY 10013|1.855.445.2285|[email protected] @alongir
Load Performance Types
Perfect
Here They Are Again
Time
Sensitive
Bad
175 Varick Street NY, NY 10013|1.855.445.2285|[email protected]
What Can Be Achieved By Using Agile Performance
Testing
Apache JMeter + AWS or Commercial Vendors
Use Case Examples
175 Varick Street NY, NY 10013|1.855.445.2285|[email protected] @alongir
What to Look Out For? GEO Sensitive Testing
• Realistic Vs Unrealistic Simulation • Single Location Vs Geo Distributed Load • Small Subset of Target Servers • Unrealistic Connectivity
175 Varick Street NY, NY 10013|1.855.445.2285|[email protected] @alongir
Testing AKAMAI CDN GEO Sensitive Testing
175 Varick Street NY, NY 10013|1.855.445.2285|[email protected] @alongir
Testing AKAMAI CDN (ALL) GEO Sensitive Testing
175 Varick Street NY, NY 10013|1.855.445.2285|[email protected] @alongir
Testing AKAMAI CDN (VA Only) GEO Sensitive Testing
175 Varick Street NY, NY 10013|1.855.445.2285|[email protected] @alongir
In Real-time Pinpoint Accuracy
175 Varick Street NY, NY 10013|1.855.445.2285|[email protected] @alongir
In Real Time Pinpoint Accuracy
50%-50%
60%-40%
175 Varick Street NY, NY 10013|1.855.445.2285|[email protected] @alongir
In Real Time Pinpoint Accuracy
100 hits/s
1,000 hits/s
175 Varick Street NY, NY 10013|1.855.445.2285|[email protected] @alongir
Benefits Distributed Load
• More realistic • More load • Evenly distributed • More IPs (For a more even load balancing) • More locations (For GEO sensitive systems)
175 Varick Street NY, NY 10013|1.855.445.2285|[email protected] @alongir
Conclusions Agile Performance Testing
• Cloud Computing + Open Source = Agile • Any Protocol, Any Scenario, Any Scale • Self-Served, On-Demand, TTS < 10m • For Dev, Devop, Performance Engineer and Ops • Test Driven Development • Better Quality + Faster = Competitive !
175 Varick Street NY, NY 10013|1.855.445.2285|[email protected] @alongir
Thank you for your time!
Jolt Awards 2013: The Best Testing Tools