Agile Open Source Performance Testing Frank Cohen, +01 (408) 871-0122, [email protected] January 4, 2011 The Agile Performance Test Methodology and Tools A Workshop For CIOs, CTOs, Senior Managers
Nov 11, 2014
Agile Open Source Performance Testing
Frank Cohen, +01 (408) 871-0122, [email protected]
January 4, 2011
The Agile Performance Test Methodology and Tools
A Workshop For CIOs, CTOs, Senior Managers
Open Source Test Automation
Slides on Twitter‣http://twitter.com/fcohen
‣Please retweet
‣Please follow me
‣#ost
2
Open Source Test Automation
About PushToTest‣Our Mission: Test, To Make A More Reliable World
‣Make It Easier To Move From Manual To Automated Testing
‣Support Advanced Testing, including Agile, Load, Integration
‣Open Source Testing (OST) For Everyone
3
Open Source Test Automation
Web 2.0 Success Driving Business‣Rich Internet Applications
• Fast Time-To-Market
• Complex Business Workflows
• Client-less Access
‣SOA Composite Applications
• Faster Time-To-Market
• Increase Revenue
• Cost Savings
‣Service Enabled Applications
• Deliver More Integration Faster
• Less Cost4
Open Source Test Automation
When OST Is Your Best Choice
5
‣Modern Agile Software Development Techniques
‣Highly-Usable, Highly-Functional Applications for Web Deployment
‣Complex Data, Sophisticated User Interaction, Application Business Logic (Workflow)
http://www.pushtotest.com/10-ways-to-build-and-test-rich-internet-applications-ria.html
Open Source Test Automation
Agile Repurposing‣Agile Delivers More Projects Faster
‣Requires Repurposing Between Teams
‣Treats Tests As Business Asset (Version Control, Sharing, Reuse)
6
DevTest
Biz
IT
Business Requirements to LaunchIntegration Test and SLA Compliance
Functional Tests Repurposed AsLoad and Performance Tests
Open Source Test Automation
‣Proprietary Test Tools Offer:
‣QA Labs for 10,000 Users = $35,000 Per Year
‣Does Not Include: Cost of Test Employees, Root Cause Analysis Consultants
‣Does Not Guarantee Success
Testing Costs (TCO Model)
7
Functional Test
Functional Test Platform $19,000
Functional Tests $8,000
Protocols: AJAX, SOAP $12,000
Support $5,000
$40,000
Load Test
Load Test Platform $20,000
Simulated User Licenses $120,000
Recoding The Test $35,000
Re-qualify The Test $12,000
$187,000
Open Source Test Automation
What We Have For You
8
‣ Implement effective tests
‣Deploy tests to desktop, grid, and cloud environments
‣A clear tutorial approach to the PushToTest methodology
‣How It All Fits Together
Open Source Test Automation
OST Architecture
9
Functional
Test
Production
Monitor
Load Test
Continuous Integration
Repository Bug TrackerTest
ManagementMonitoring
Test Authoring
SoapUI (Pro)
Sahi (Pro)
JUnit Class Framework
Eclipse Java IDE
Fitness
Languages: Java, Python
Wiki Collaboration
Application Under TestWeb, RIA, BPM, SOA
PTTMonitorCPU, Net, Memory
Test Injector
BundlesMySQL, HtmlUnit
ScriptRunnersSahi, Selenium, SoapUI, Java
Data Production Librariescsv files, rdbms queries
Test Operation Console
RepositoryMySQL RDBMS
Results AnalysisBusiness Intelligence Tool
DesignerRecord/Playback
Grid and CloudTest Deployment
Open Source Test Automation
PTTMonitor
Load Injectors
An OST Cloud
10
Application Under Test
Central location from which a tester, developer, IT manager, CI service runs and monitors tests
TestMaker
Console
Thousands of virtual users perform real-life scenarios on the Application Under Test
TestNode
Stores results of every transaction, step, and monitored resource
Results DB
400+ charts identify the root cause to performance bottlenecks, functional issues
Analysis
Open Source Test Automation
PTTMonitor
Load Injectors
An OST Cloud
11
Application Under Test
Central location from which a tester, developer, IT manager, CI service runs and monitors tests
TestMaker
Console
Thousands of virtual users perform real-life scenarios on the Application Under Test
TestNode
Stores results of every transaction, step, and monitored resource
Results DB
400+ charts identify the root cause to performance bottlenecks, functional issues
Analysis
Open Source Test Automation
Component Approach
12
Data Production
Log-In
Component
setUp
runTest
tearDown
ApplicationTestScenario
Test Use Case
Product Search
Order Product
Log-Out
RDBMS DPL
CSV DPL
Custom DPL
Load Test
100 Users 500 Users 1000 Users
Log-In
Component
Component
Development
Database
QA Regression
Database
Load Test
Database Confirm Order
Production
Database
Open Source Test Automation
Ready For Load Testing?‣Selenium Core and RC Require A Browser
‣Selenium IDE Record In Firefox Only, Coding Required To Be Useful, No Data Enablement
‣TestMaker Object Designer for recording in IE
‣Selenium Grid for Functional Testing Only
‣No Reports or Results Analysis
‣No Cloud or Grid Deployment
‣No CI or QC Integration
‣RIA Testing using Ajax, Flex, Flash13
Open Source Test Automation
Test Authoring and Scale
14
Approach Advantages and Disadvantages Scale
Protocol Level Testing1000+ VUs per TestNodeScript-level codingExtra code for JavaScript
1000+ VUs
HTTP Archive (HAR)1000+ VUs per TestNodeScript-less Test AuthoringAutomatic for JavaScript
1000+ VUs
SeleniumHtmlUnit Record/PlaybackOperates client-side logic 20-200 VUs
Browser Functional TestingSame Test ScriptVisual Test DebuggingSlow Test Operation
2-4 VUs
Open Source Test Automation
15
Repurposing
Open Source Test Automation
TestMaker Demonstration‣Fun And Easy Test Script Creation
‣Selenium and TestMaker
‣Repurposing A Single Test
‣Functional Test, Load and Performance Test, Production Monitor
‣Results Analysis
‣Chart Engine
16
Open Source Test Automation
Calibration Testing
17
Open Source Test Automation
Expensive For Time
18
Open Source Test Automation
Start In The Middle
19
Open Source Test Automation
Work Down or Up
20
Next let’s talk about
Open Source Test Automation
PushToTest Options
‣OnDemand
‣QuickStart
‣POC Option
21
Open Source Test Automation
Immediate Benefits‣Product Solutions
‣Better, Lower-Cost, More Efficient Test Tool
‣Business Service Monitoring at the Application Tier (SLA Compliance)
‣Methodology Solutions
‣ Improved Test Methodology for my Organization and Team
‣Services
‣Turnkey Load Testing
‣ Integrate TestMaker Into My Build Environment
‣Add A Protocol, Chart, Task to TestMaker To Meet My Needs22
Open Source Test Automation
Individuals and Organizations
‣ Functional Testing with TestMaker Object Designer
‣ Repurposing Selenium, soapUI Sahi and unit tests into data-driven functional tests, load and performance tests, and business service monitors
‣ Load Testing To 50 Virtual Users
‣ Data-driven tests using CSV files.
‣ 25 Charts and Reports
‣ Community Support and access to all documentation23
‣ Load Testing to Millions of Virtual Users
‣ Cloud and Grid Test Deployment
‣ Collaborative Test Object Repository
‣ Data-driven tests, RDBMS Data Production Libraries
‣ Collaborative Results Analysis Repository
‣ Flex/Flash 3.0, 4.0, AMF
‣ 1000+ Charts and Reports
‣ Support, QuickStart, Training, Consulting
For the individual tester needing a no-cost test tool to test Web and RIA for functionality, load and performance, and monitoring.
For organizations running large scale business-critical Web, RIA, SOA, BPM and needing world-class tools, methodology, training, and support.
Open Source Test Automation
Pricing‣Annual Subscription Model Based On:
‣Seats for TestMaker Console Installation
‣Seats for SOAP and REST Test Authoring
‣Virtual User Levels for Load Testing
‣Count of Business Service Monitor
‣Technical support: 1 business day or faster response to emails, phone calls, instant messages
24
Open Source Test Automation
25
‣As-Needed, On-Demand Basis
‣Test In Your Environment, Or Our Cloud Test Environment, Or Both
‣ Ideal For Consulting Organizations Where Test Needs Are Unknown
‣AMI Distribution
‣Pay Only For What You Use
‣Annual Cap On Total Costs
Open Source Test Automation
The QuickStart Option
26
‣Your First Test, Built and Operated by PushToTest Experts
‣ Ideal For Organizations With Little or No Load Testing Experience
‣PushToTest Experts Build Your First Test
‣PushToTest Experts Operate The Test, Provide Root Cause Analysis and Mitigation
‣PushToTest Trains Your Team To Build Their Own Tests
‣$10,000 USD
Open Source Test Automation
The POC Option‣PushToTest Experts Help Your TestMaker Evaluation
‣Ensures TestMaker Works In Your Application/System
‣PushToTest Issues Price Proposal
‣Applies Engineers To Build and Operate Your Test
‣PushToTest Trains Your Team To Use TestMaker
‣Success of POC Confirms Intention To Purchase
‣Cost: $027
Open Source Test Automation
PushToTest Global Services‣Put Our Experts To Work For Your Projects
‣Methodology, Solutions, Support, Enthusiasm
‣Offices: USA (Texas and California,) Bangalore, London
‣The Near Shore Option
‣150 Test Experts
‣Costa Rica: CST, English, Inexpensive, On Site in 3-5 Hours
‣http://www.pushtotest.com/services
28
“PushToTest Uniquely Built Load Tests For Our Customers, Including Pepsi, And Worked With Our
Engineers To Surface and Solve Performance Bottlenecks.” Alan Rodriguez, Director of Testing, Tribal DDB Dallas
Open Source Test Automation
Avoid Retreats‣Load Test Shows: 3,230 Success, 12 Failures
‣Avoid The Urge For Perfection
‣Even The Best Managers Sometimes Go Back To Functional Testing
‣Even Failed Transactions Generate Load
‣Sometimes the whole point is...
29
Open Source Test Automation
30
Open Source Test Automation
31
Alex: load test is running fineInga: Coolio!Alex: Uh ohAlex: Success: 3210, Failure: 15Alex: Success: 3210, Failure: 204Alex: Success: 3210, Failure: 483Inga: What’s up?Alex: The whole application failedInga: Yep, avail threads = 0Alex: Back to the drawing board
Open Source Test Automation
32
Fear NotWe get paid to break things.
When you break it you understand it.When you understand it you can fix it.
Open Source Test Automation
Slides on Twitter‣http://twitter.com/fcohen
‣Please retweet
‣Please follow me
‣#ost
33
Open Source Test Automation
Where To Go From Here‣Evaluate Selenium, soapUI, TestMaker
‣http://www.pushtotest.com/products-comparison
‣Watch A Screencast and Tutorial
‣http://www.pushtotest.com/screencastcentral
‣Attend An Open Source Test Workshop
‣http://workshop.pushtotest.com
‣Ask For A Proposal on Licenses, Training, Support, Consulting
‣Call +01 408 871 0122, [email protected]
34