© 2011 Forrester Research, Inc. Reproduction Prohibited 1
Dec 07, 2014
© 2011 Forrester Research, Inc. Reproduction Prohibited1
© 2011 Forrester Research, Inc. Reproduction Prohibited2 © 2009 Forrester Research, Inc. Reproduction Prohibited
Testing Tools - How Teams Make It Work
Margo Visitacion, Vice President
February 17, 2011
© 2011 Forrester Research, Inc. Reproduction Prohibited3
Don’t lose before you get into the game……
© 2011 Forrester Research, Inc. Reproduction Prohibited4
Don’t lose before you get into the game
Why team focused testing can make the difference
Planning performance testing today can help budget planning tomorrow
Developing your test game plan
Recommendations
Agenda
© 2011 Forrester Research, Inc. Reproduction Prohibited5
If you don’t plan your quality and build the right team, you lose before you get into the game…….
© 2011 Forrester Research, Inc. Reproduction Prohibited6
More business is transacted online….
© 2011 Forrester Research, Inc. Reproduction Prohibited7
But problems increase your risk and costs …
Performance:– It only takes a moment to lose customers
– 74% of surveyed customers shop online….
– …. Yet 54% abandon shopping carts, a continuing plague for online vendors vendors
Security– Online banking: the future is now
– 90.5 million Americans now bank online…..
– ….Yet less than half of customers feel secure about their institutions’ security
Scalability– 70% of auto policies renew in January
– The global Internet population is now 1.6B and expects to grow another 42% by 2014
Do you have the right team in place?
© 2011 Forrester Research, Inc. Reproduction Prohibited8
In 2010 more money was directed to toward growth and expansion – don’t let poor performance shift dollars away from innovation
Poor quality diverts focus
© 2011 Forrester Research, Inc. Reproduction Prohibited9
Don’t lose before you get into the game
Why team focused testing can make the difference
Planning performance testing today can help budget planning tomorrow
Developing your test game plan
Recommendations
Agenda
© 2011 Forrester Research, Inc. Reproduction Prohibited10
Siloed approach to testing won’t let you answer these questions
Functionality• Are we meeting customer’s expectations• Does the application work the way we want it?
Performance• Are our applications accessible?• Can our clients complete orders in a timely fashion?
Security• Is client data protected?• Are transactions secure?
Scalability• Does the application scale up to meet peak performance
requirements• How stable are services when performance peaks?
© 2011 Forrester Research, Inc. Reproduction Prohibited11
Companies are planning to spend more on securing their customer’s loyalty
© 2011 Forrester Research, Inc. Reproduction Prohibited12
How will you change your behaviors to improve your approach to testing??
How will you “Change your ways”?– Challenges in testing are growing
– Performance, Data privacy, compliance
– Testing as usual won’t work today
– Early testing is imperative
– Teams look different because testing approaches have to change
IT budgets look better, but must bring value– Companies want greater flexibility for their $$$
– Automation MUST bring value for the money invested
Testing is ripe for SaaS– 21% are piloting SaaS for IT lifecycle components
– 26% are planning to adopt within the next year
© 2011 Forrester Research, Inc. Reproduction Prohibited13
Taking a team approach
BusinessStakeholders
•Setting requirements•Validates design•Approves change
Business analysts
•Gather requirements•Refines requirements•Reviews change•Supports testing
Developers
•Leverages requirements for design and build
•Unit tests•Some integration tests•Writes test harnesses•Supports testing
Testers
•Validates requirements•Helps define quality criteria•Designs test plans, scripts•Executes tests•Regression test•Oversee UAT
© 2011 Forrester Research, Inc. Reproduction Prohibited14
Testing can begin from day one to prevent defects
Business determinesobjectives, sets
requirements
Design and developmentbased upon requirements
Set test planning &Automation strategy
Business stakeholders, BAs & test team set quality and performance requirements for functional and non functional
requirements
Development &testing continues
Continuous build and integrationContinuous testingmanual and automatedStatic and automated performance testing starts
End to end testingUAT
DeploymentPost
Mortem
© 2011 Forrester Research, Inc. Reproduction Prohibited15
Use automation to keep up with change
Capital investments are scrutinized
Infrastructure and environments are challenged
Ramp up time takes too long
Usability is a requirement
Quick time to value is a necessity
Performance testing is cyclical but non negotiable
Testing is an enabler
Environmentsans
constraints
© 2011 Forrester Research, Inc. Reproduction Prohibited16
Agenda
Don’t lose before you get into the game
Why team focused testing can make the difference
Planning performance testing today can help budget planning tomorrow
Developing your test game plan
Recommendations
© 2011 Forrester Research, Inc. Reproduction Prohibited17
Costs: A drill down
Cost of testing• Organizations - Labor costs• Skills - recruiting costs• Tools - capital costs• Environments – capital costs• Support – opex costs
Cost of NOT testing• Customer reported defects• Business liability• Increased maintenance costs• Increased support costs
Poor software quality costs over $60B per year
Finding, repairing defects = approximately 35% of project budget
© 2011 Forrester Research, Inc. Reproduction Prohibited18
Why? Investing in quality can reduce risk, cost and raise value
0
20
40
60
80
100
120
Requirements Code Operations
Cost to Repair
© 2011 Forrester Research, Inc. Reproduction Prohibited19
Plan Today For Tomorrow’s Budget
•Performance data•Defects•Customer SatisfactionGather
historical information
•Leverage historical data for peak planning•Understand your tools usage and
requirementsPlan for today and tomorrow
•Consider on demand to cover what you need when you need it
•Look for flexible options – try to avoid getting trapped into expensive, no win situations
Allocate Funds
© 2011 Forrester Research, Inc. Reproduction Prohibited20
Don’t lose before you get into the game
Why team focused testing can make the difference
Planning performance testing today can help budget planning tomorrow
Developing your test game plan
Recommendations
Agenda
© 2011 Forrester Research, Inc. Reproduction Prohibited21
To win the game, the business needs to be involved
User stories
“Situational realities”
Requirem
ents
Risks/opportunities
© 2011 Forrester Research, Inc. Reproduction Prohibited22
Bringing the team together needs a repeatable, measurable process
Set expectationsCollaborate
ReviewDiscuss
PlanTest
Review
ExecuteReview Refine
Measure
© 2011 Forrester Research, Inc. Reproduction Prohibited23
Don’t lose before you get into the game
Why team focused testing can make the difference
Planning performance testing today can help budget planning tomorrow
Developing your test game plan
Recommendations
Agenda
© 2011 Forrester Research, Inc. Reproduction Prohibited24
Recommendations
Don’t leave performance testing to the end. – Problems will grow as business options expand
Anticipate the unexpected– Life is what happens while you’re busy making other plans
Leverage automation and timing– Flexible tools to performance-test, “just in time”
© 2009 Forrester Research, Inc. Reproduction Prohibited
Thank you
Margo Visitacion+1 [email protected]
Testing Tools Landscape- Load Testing
Matt Poepsel, VP of Performance Strategies, Compuware Gomez
Customer Expectations Are Rising For Web & Mobile Applications
47% of consumers expect a Web page to load in 2 seconds or less
58% of mobile phone users expect websites to load almost as quickly or faster on their mobile phone than their PC
MajorISP
Local ISP
Mobile Carrier
Internet
Content DeliveryNetworks
3rd Party/Cloud Services
Browsers and devices UsersUsers
Storage
Web Servers
App Servers
DB Servers
Mainframe
Load Balancers
Mobile Components
Network
The Web Application Delivery Chain
The Challenge Of Delivering Quality Web Experiences
• Inconsistent geo performance• Bad performance under load• Blocking content delivery• Incorrect geo-targeted content
• Network peering problems
• Outages
• Configuration issues• Oversubscribed POP• Poor routing optimization• Low cache hit rate
• Network peering problems
• Bandwidth throttling
• Inconsistent connectivity• Configuration
errors• Application
design issues• Code defects• Insufficient
infrastructure
• Network resource shortage
• Faulty content transcoding
• SMS routing / latency issues
• Poorly performing JavaScript
• Inconsistent CSS rendering
• Browser/device incompatibility
• Page size too big
• Conflicting HTML tag support
• Too many objects
• Content not optimized for device
• Low cache hit rate
Systems management
tools: “OK”…user is NOT happy
“Outside-in” customer point of view
Test & monitor your site the way your customers use it:• What they do (key pages and transactions)• Where they do it (geographic locations)• How they do it (browsers and mobile devices)• When they do it (normal and peak usage)
Determine the impact on their behavior and your business
Applications are no longer what you build inside your firewall
Number of hosts accessed directly by the browser, per user transaction, averaged across 3,000 companies 10.5
Who Owns End-User Quality?
Line of Business Managers
Product Managers
Marketers
Interface Designers
Application Developers
Architects
Quality Assurance Teams
Infrastructure/Operations Teams
Help Desk StaffAvoid at all costs:
Individual Success, Collective Failure
Quality is a Collective Function.Everybody owns a piece of the puzzle.
How can we work together?
Create Shared Goals
Identify Shared Metrics
Use a Communications Plan
Solve Problems as a Team
Celebrate as a Team
Situational Assessment and Improvement
1. Provide this 5-minute assessment to each of the End-User Quality contributors for your application:
2. The results provide a baseline. Share the results across the team.
3. Identify areas of improvement and execute a plan to get better.
4. Re-administer the assessment after 90 days.
Shared Goals
Shared Metrics
Communications
Solving problems as a team
Celebrating as a team
On a scale of 1 (poor) to 10 (excellent), how well are we are doing in these areas:
Questions
Reduced downtime 45%
Reduced seven-step transaction time by 50%
Increased conversions 10%
Validated decision to consolidate three data centers
Achieved under 3 second response time and 99%+ availability
Reduced homepage load time from 11.3 seconds to 3.4 seconds
Improved page load times 23%
Saved 50%+ in staff and fees
Gomez Customers Enjoy Measurable Benefits
For more information visit Gomez.com or contact us at +1 781.778.2700