1© 2013 SOASTA. All rights reserved.
Adopting Cloud Testing for Continuous Delivery, with IDC and SOASTA
Webinar
Presents
How cloud computing has impacted the Quality as a Service marketplace
Joined by:
2© 2013 SOASTA. All rights reserved.
Market perspectives and applications of cloud testing
In This Webinar
TODAY’S SOASTA PRESENTERS
Melinda Ballou: Program Director, Application Life-Cycle Management, IDC
Brad Johnson: VP Product Marketing, SOASTAMike Ostenberg: Sr. Solutions Architect, SOASTA
Agenda:
• Poll question• Market Perspective and Recommendations (IDC)• Cloud Testing Drivers and Use-Cases (SOASTA)• Overview and Demonstration of a Leading Cloud Testing
Platform
Questions: Submit in chat box during event
Copyright 2013 IDC. Reproduction is forbidden unless authorized. All rights reserved.
Adopting Cloud Testing for Continuous Delivery: Evolving Quality to Address Complexity
Adopting Cloud Testing for Continuous Delivery: Evolving Quality to Address Complexity
Melinda BallouIDCProgram DirectorApplication Lifecycle Management & Executive Strategies
Apr 15, 2023
© 2013 IDC 5
Industry Highlights: Trends Driving ASQ & ALM Adoption & ServicesIndustry Highlights: Trends Driving ASQ & ALM Adoption & Services
Organizations re-invest, seeking to do more with fewer resources with financial and staffing constraints; leveraging efficient, agile approaches to restore and sustain high performing, business-critical software.
Complex sourcing/off-shoring plus use of open source need strong teaming, effective code management, testing, change and IT project portfolio management. Services driven environment (SaaS/cloud,Devops emergence)
Global economic competition and local compliance across geographies demand quality, change and portfolio management, adaptability and rigor
Flexible, agile development paradigm with services creation increasingly drive technology and business collaboration; agile emergence also disruptive
Increasing security issues (as driver) and virtualization/cloud (as enabling technology) for ALM adoption; ad hoc approaches unsustainable
End-user experience and business impact challenges of rich Internet, mobile, embedded, with social media collaboration/community opportunities
SaaS license delivery mechanisms bring new models, de-capitalization benefits, flexibility, adoption speed, limit complexity, cloud on-ramp
Apr 15, 2023
© 2013 IDC 6
“Quality Gap”: High Cost of Failure, Cloud Benefits to Address Challenges “Quality Gap”: High Cost of Failure, Cloud Benefits to Address Challenges
Poor Quality = Increased Business Risk
Lost Revenue
($$$$$)
Lost Customers
Lost Productivity
Increased Costs
Lower ProfitsDamaged Brand
Apr 15, 2023
© 2013 IDC 7
App Development Life-Cycle: Old View Rapidly Shifting App Development Life-Cycle: Old View Rapidly Shifting
Assumes: Unidirectional flow, limited # of handoffs
Result:Silos within project team; gulf with operationsLittle understanding how efforts impact overall result Testing takes place LATE in the cycle!
Define Design Develop Test
OPS
Issues
Apr 15, 2023
© 2013 IDC 8
Closing the Loop: Leverage Skills & Tools for Agile, Iterative ApproachClosing the Loop: Leverage Skills & Tools for Agile, Iterative Approach
DesignDefine
DevelopSupport
Deploy & Monitor
Test & Tune
• Shared goals,• Shared metrics,• Shared tests, tools and skills
• Shared virtual lab• Quick iterationscan be disruptive• Demand mgmnt• Cloud adaptability• Enabling agile infrastructure access
Apr 15, 2023
© 2013 IDC 9
Three Key Challenges for IT Three Key Challenges for IT
IT must deliver new applications that have greater business value and higher quality, while managing costs …in the face of these 3 key challenges:
Increasing criticality of applications to the business
Increasing complexity of software systems and applications (from web to mobile to embedded… encompassing social systems of engagement to feed systems of record, performance demand with Big Data Analytics for business optimization)
Increasingly distributed teams with multi-sourcing of applications (from onshore to offshore to open source)
Apr 15, 2023
© 2013 IDC 10
Barriers to Prior Testing Approaches – Cloud Testing DriversBarriers to Prior Testing Approaches – Cloud Testing Drivers
• Agile velocity demands immediate, frequent, iterative testing
• Lack of system resources constrains testing usage – expense limits ability to mirror production configurations (mobile issues)
• User tendency to hoard physical systems for testing creates bottleneck & expense – need for management & coordination
• Challenges to test system configuration & lengthy set-up times limit usage and delays testing
• Lack of change management for testing configurations creates pain points for consistent testing and re-testing
• Need for variable provisioning for complex software deployment (including mobile) increasingly drives cloud adoption for testing to cut costs and increase business agility
Apr 15, 2023
© 2013 IDC 11
Coordinating across the Life-CycleCoordinating across the Life-Cycle
• Coordinating requirements, testing, and operational performance is key across emerging technologies
• Slow response time costs prohibitive for core business areas
• Organizations should leverage quality automation through requirements, unit test, system integration, pre-deployment & application performance testing with emerging cloud platforms
• Evaluating test automation for synthetic transactions and real user measurement and monitoring can help teams react and manage user application experience
• As business requirements change, a cogent life-cycle approach enables adaptive testing and monitoring responses
• Look to ALM SaaS alternatives initially as an on-ramp to cloud where appropriate – strategize through to deployment
Apr 15, 2023
© 2013 IDC
Q. Your organization's adoption of cloud services?
Source: IDC CloudTrack Survey, Winter 2012 n=493
% of Respondents
Adoption Is Building by Cloud TypeAdoption Is Building by Cloud Type
Apr 15, 2023
© 2013 IDC
Source: IDC CloudTrack Survey, Winter 2012 n=493
Mean rating by respondents
Q. How concerned are you about cloud...? (1-5 scale; 5 = extremely concerned)
Performance One of Top Three Concerns: Testing On & Of Cloud Can Help
Apr 15, 2023
© 2013 IDC
Cloud ComputingPublic Cloud Plans: Cloud Testing EmergingCloud ComputingPublic Cloud Plans: Cloud Testing Emerging
Use cloud for developing applications
Use cloud for testing applications
Use cloud for PaaS
Use cloud for application integration
Use cloud compute or storage resources
Use cloud database software
Use cloud for management & monitoring
Use cloud applications
0% 10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
6.7%
9.0%
10.0%
9.7%
9.3%
9.3%
11.3%
12.0%
6.7%
7.3%
8.7%
9.3%
10.0%
11.0%
9.0%
12.0%
No plans to Use Evaluating Now Beginning to UsePlanned Use in 2011 Will Use in 2012/13 DKNo Cloud Familiarity
14
What plans does your company have regarding public cloud services? (multiple response)
IDC 2011 NA Cloud Survey, N=300IT Developer and IT Management Respondents
Apr 15, 2023
© 2013 IDC
Cloud ComputingPerspective on Cloud Computing CapabilitiesCloud ComputingPerspective on Cloud Computing Capabilities
Lower overall cost
Fast application development
Lower admin cost
Makes IT far less complex
Applications that always scale up/down
Avoidance of up front costs
Fast application deployment
Standardized AD&D
Pay only for what you use
Immediate access to technology
Access to latest functionality
0% 10%
20%
30%
40%
50%
60%
70%
80%
90%
100%Completely Agree Mostly Agree Agree Mostly Disagree
Completely Disagree DK No Cloud Familiarity
15
How much do you agree that cloud computing delivers the following capabilities? (multiple response)
IDC 2011 NA Cloud Survey, N=300IT Developer and IT Management Respondents
Apr 15, 2023
© 2013 IDC
Cloud ComputingExpected BenefitsCloud ComputingExpected Benefits
DK or NS
Other
No perceived benefits
Access to best of breed tools
Access to more dev tool functionality
Easier AD&D of SOA applications
Standardized tools
Tool access to test applications
Tool access to deploy applications
Access to integrated tool suite
Access to most up to date tools
Tool access to build applications
More cost effect AD&D
0% 5% 10% 15% 20% 25% 30% 35% 40% 45%
16
What benefits does your organization expect to gain by using public cloud services? (multiple response)
IDC 2011 NA Cloud Survey, Cloud Knowledgeable, N=277 IT Developer and IT Management Respondents
Apr 15, 2023
© 2013 IDC
020
40
Other
Test data and test bed preparations to resemble live scenarios under high volume
Reusability of test artifacts to address rapid change in application functionalities
Complexity of test scenarios to design and execute
Ability to optimize usage of different platform resources (e.g. CPU, memory, battery)
Designing to different form factors (e.g. smartphone versus tablet)
Managing to robust security architecture and protection of end user application data
Rapid pace of change in mobile OS / hardware landscape
Expertise / developing skill sets across multiple mobile OS platforms
Investing in and maintaining varied mobile infrastructures
1.2%
6.5%
7.1%
13.2%
16.6%
26.5%
26.5%
32.3%
34.8%
35.4%
TotalN=325
Top Challenges in Launching Mobile Applications Top Challenges in Launching Mobile Applications
17
Share of Responses (%)
Highlights and Key Messages
Question: What are the biggest challenges you face in launching mobile applications, whether or not you are developing applications internally?
Highlights
Leading challenges concern:
Level of investment and maintenance required to support rapidly evolving mobile IT infrastructure
Building up or accessing resource expertise in the design, development and testing of mobile apps across multiple mobile platforms
Key Messages
Enterprise business priorities to grow revenue and innovate while improving internal operational efficiencies will lead many organizations to seek help from 3rd parties in implementing mobile strategies
It will be a challenge to keep up with the rapid pace of change in mobile OSs while addressing mobile security mandates, designing to different form factors, providing a smooth, productive customer experience
Note: Multiple responses allowed
Apr 15, 2023
© 2013 IDC
- Develop near, mid, and long-term goals for transformation & cloud
• Understand what’s utilized or not, and what’s bought and developed at the line of business
• Base your plan on your organization’s business goals, governance/compliance profile, IT skillsets, and data center investments
• Develop a “visualization” plan for a hybrid cloud strategy that takes you from your current state to your end state, based on level of virtualization, focus on infrastructure vs. apps, etc
• Make sure your plan puts your priorities first – whether cost reduction, consolidation, expansion into new competencies like mobile, analytics, data management, etc.
- Identify easy payback/low downside projects for public cloud services
• Develop/test/deploy and other compute and storage-intensive tasks
• Non-differentiating front-office functions first, the rest can follow -- Land then expand
• Investigate service management and brokerage tools for oversight into performance, service quality, usage
- Go at your own pace
• A majority of companies will always have on-premise gear, and “hybrid” cloud deployments
• Due diligence: compute and storage IaaS clouds are inexpensive, but mistakes are costly
• Be prepared to be more of a services broker, and deal with different types of outside providers than you’re used to: get ready for change
Guidance for Cloud Testing AdoptionTake a Strategic View of Cloud SourcingGuidance for Cloud Testing AdoptionTake a Strategic View of Cloud Sourcing
Apr 15, 2023
© 2013 IDC 19
SummarySummary
• IDC saw 37.7% revenue growth for cloud testing and ASQ SaaS in 2012 with five year CAGR through 2017of 31.1 (to reach $1.01B); re-assess options as solutions expand
• ASQ and development teams must bridge the gap to operations to gain access to appropriate cloud infrastructure for quality; help leverage and coordinate mobile quality
• IDC recommends an encompassing approach to business and IT management that includes cloud and SaaS testing
IT organizations should actively lobby the business and operations for access to cloud testing in the short-term and for other ALM areas over time
Executive management must plan for incorporation of cloud as part of the overall IT portfolio to increase business adaptability and cut costs for multi modal software
Companies should view cloud testing and ASQ SaaS as part of a comprehensive strategy from dev and provisioning through to build and operational planning. This strategy should include mobile as a bridge from systems of engagement to systems of record & operations.
Apr 15, 2023
© 2013 IDC
Melinda BallouProgram Director,
Application Life-Cycle & Executive Strategies
1. Business/IT alignment 2. IT-PPM & PPM strategies 3. IT staffing strategies4. Business strategy 5. IT portfolio mgmt.
Contributing AnalystsContributing Analysts
18
Melinda BallouProgram Director, Application Life-Cycle Mgmt & Executive Strategies
22© 2013 SOASTA. All rights reserved.
Business Drivers of CloudConsumer driven (even internal apps)
o Time To Market
6 week delivery plan @ Google
http://www.thechromesource.com/understanding-chromes-six-week-release-cycle/
Rapid updates and adoption
23© 2013 SOASTA. All rights reserved.
Business Drivers of CloudConsumer driven (even internal apps)
o Time to market
o Global access and device options
24© 2013 SOASTA. All rights reserved.
Business Drivers of CloudConsumer driven (even internal apps)
o Time to market
o Global access and device options
o End user reach and quality expectations
25© 2013 SOASTA. All rights reserved.
Goals of Using the Cloud for Testing
Speed
Map to Business Drivers
Global access and device options
• 800,000 servers – 54 locations – 17 countries
Affordability
Time to market
• Test Fast
End user reach and quality expectations
• Frequent, iterative, realistic for high coverage
Scale
26© 2013 SOASTA. All rights reserved.
The Path to Continuous Testing with the Cloud
o Understand your requirements
o Identify the cloud advantages for you
o Automate the obvious and most critical
o Connect to a continuous process
o Align teams with actionable information
27© 2013 SOASTA. All rights reserved.
Modern Use-Cases Drive Success and EvolutionIconic Launches to Legendary Landings
http://www.soasta.com/customers/case-studies/
28© 2013 SOASTA. All rights reserved.
IDC MarketScape 2013
o “a leader in Cloud Testing and ASQ SaaS”
o “leading innovator in the cloud testing arena”
o “Overall, SOASTA is well positioned as a best of breed vendor in this arena.”
o “Differentiators for SOASTA are its strength in cloud-based testing and evolving mobile support and a free CloudTest Lite version”
Worldwide Cloud Testing and ASQ SaaS Vendor Assessment
SOURCE: IDC MarketScape: Worldwide Cloud Testing and ASQ SaaS 2012–2013 Vendor Analysis — Enabling Business Agility and Quality in the Cloud, by Melinda-Carol Ballou, February 2013, IDC #237597, http://www.idc.com/getdoc.jsp?containerId=237597
IDC MarketScape vendor analysis model is designed to provide an overview of the competitive fitness of ICT suppliers in a given market. The research methodology utilizes a rigorous scoring methodology based on both qualitative and quantitative criteria that results in a single graphical illustration of each vendor’s position within a given market. The Capabilities score measures vendor product, go-to-market and business execution in the short-term. The Strategy score measures alignment of vendor strategies with customer requirements in a 3-5-year timeframe. Vendor market share is represented by the size of the circles. Vendor year-over-year growth rate relative to the given market is indicated by a plus, neutral or minus next to the vendor name.
29© 2013 SOASTA. All rights reserved.
Mike Ostenberg
SOASTA’s Approach & Architecture to Cloud Testing
30© 2013 SOASTA. All rights reserved.
The SOASTA PlatformEmbedded Quality for the Mobile Lifecycle
TouchTestContinuous Automated
Testingof Mobile
Applications
CloudTestLoad &
Performance
mPulseReal User Monitoring
31© 2013 SOASTA. All rights reserved.
SUT/AUT
Web Server
Web Server
App Server
Web Server
Web Server
App Server
App Server
App Server
Load Balancer
Cache
Cache
Database
CloudTestArchitecture
32© 2013 SOASTA. All rights reserved.
SUT/AUT
Web Server
Web Server
App Server
Web Server
Web Server
App Server
App Server
App Server
Load Balancer
Cache
Cache
Database
Customers
CloudTestArchitecture
33© 2013 SOASTA. All rights reserved.
SUT/AUT
Web Server
Web Server
App Server
Web Server
Web Server
App Server
App Server
App Server
Load Balancer
Cache
Cache
Database
Customers
CloudTestArchitecture
34© 2013 SOASTA. All rights reserved.
SUT/AUT
Web Server
Web Server
App Server
Web Server
Web Server
App Server
App Server
App Server
Load Balancer
Cache
Cache
Database
CloudTestArchitecture
Analytics
CloudTest
Database
Main
“Conductor”“Conductor”
35© 2013 SOASTA. All rights reserved.
SUT/AUT
Web Server
Web Server
App Server
Web Server
Web Server
App Server
App Server
App Server
Load Balancer
Cache
Cache
Database
CloudTestArchitecture
Analytics
CloudTest
Database
Main
“Conductor”“Conductor”
Perf Tester 1
36© 2013 SOASTA. All rights reserved.
SUT/AUT
Web Server
Web Server
App Server
Web Server
Web Server
App Server
App Server
App Server
Load Balancer
Cache
Cache
Database
CloudTestArchitecture
Analytics
CloudTest
Database
Main
“Conductor”“Conductor”
Perf Tester 1
37© 2013 SOASTA. All rights reserved.
SUT/AUT
Web Server
Web Server
App Server
Web Server
Web Server
App Server
App Server
App Server
Load Balancer
Cache
Cache
Database
CloudTestArchitecture
Analytics
CloudTest
Database
Main
“Conductor”“Conductor”
Perf Tester 1
38© 2013 SOASTA. All rights reserved.
SUT/AUT
Web Server
Web Server
App Server
Web Server
Web Server
App Server
App Server
App Server
Load Balancer
Cache
Cache
Database
CloudTestArchitecture
Analytics
CloudTest
Database
Main
39© 2013 SOASTA. All rights reserved.
SUT/AUT
Web Server
Web Server
App Server
Web Server
Web Server
App Server
App Server
App Server
Load Balancer
Cache
Cache
Database
CloudTestArchitecture
Analytics
CloudTest
Database
Main
Perf Tester 2Perf Tester 2
40© 2013 SOASTA. All rights reserved.
SUT/AUT
Web Server
Web Server
App Server
Web Server
Web Server
App Server
App Server
App Server
Load Balancer
Cache
Cache
Database
CloudTestArchitecture
Analytics
CloudTest
Database
Main
Perf Tester 2
41© 2013 SOASTA. All rights reserved.
SUT/AUT
Web Server
Web Server
App Server
Web Server
Web Server
App Server
App Server
App Server
Load Balancer
Cache
Cache
Database
CloudTestArchitecture
Analytics
CloudTest
Database
Main
Perf Tester 2
AnalyticsLoad
GeneratorLoad GeneratorLoad
Generator
IBM
42© 2013 SOASTA. All rights reserved.
SUT/AUT
Web Server
Web Server
App Server
Web Server
Web Server
App Server
App Server
App Server
Load Balancer
Cache
Cache
Database
CloudTestArchitecture
Analytics
CloudTest
Database
Main
Perf Tester 2
AnalyticsLoad
GeneratorLoad GeneratorLoad
Generator
IBM
AnalyticsLoad
GeneratorLoad GeneratorLoad
Generator
RackSpace
43© 2013 SOASTA. All rights reserved.
SUT/AUT
Web Server
Web Server
App Server
Web Server
Web Server
App Server
App Server
App Server
Load Balancer
Cache
Cache
Database
CloudTestArchitecture
Analytics
CloudTest
Database
Main
Perf Tester 2
AnalyticsLoad
GeneratorLoad GeneratorLoad
Generator
IBM
AnalyticsLoad
GeneratorLoad GeneratorLoad
Generator
RackSpace
AnalyticsLoad
GeneratorLoad GeneratorLoad
Generator
AWS East
44© 2013 SOASTA. All rights reserved.
SUT/AUT
Web Server
Web Server
App Server
Web Server
Web Server
App Server
App Server
App Server
Load Balancer
Cache
Cache
Database
Perf Tester 2
AnalyticsLoad
GeneratorLoad GeneratorLoad
Generator
AWS East
AnalyticsLoad
GeneratorLoad GeneratorLoad
Generator
IBM
AnalyticsLoad
GeneratorLoad GeneratorLoad
Generator
RackSpace
Analytics
CloudTest
Database
Main
Massively Scalable Cost Effective Real World Testing
CloudTestArchitecture
45© 2013 SOASTA. All rights reserved.
SUT/AUT
Web Server
Web Server
App Server
Web Server
Web Server
App Server
App Server
App Server
Load Balancer
Cache
Cache
Database
Perf Tester 2
AnalyticsLoad
GeneratorLoad GeneratorLoad
Generator
AWS East
AnalyticsLoad
GeneratorLoad GeneratorLoad
Generator
IBM
AnalyticsLoad
GeneratorLoad GeneratorLoad
Generator
RackSpace
Analytics
CloudTest
Database
Main
CloudTestArchitecture
“Conductor”
46© 2013 SOASTA. All rights reserved.
SUT/AUT
Web Server
Web Server
App Server
Web Server
Web Server
App Server
App Server
App Server
Load Balancer
Cache
Cache
Database
Perf Tester 2
AnalyticsLoad
GeneratorLoad GeneratorLoad
Generator
AWS East
AnalyticsLoad
GeneratorLoad GeneratorLoad
Generator
IBM
AnalyticsLoad
GeneratorLoad GeneratorLoad
Generator
RackSpace
Analytics
CloudTest
Database
Main
CloudTestArchitecture
“Conductor”
47© 2013 SOASTA. All rights reserved.
SUT/AUT
Web Server
Web Server
App Server
Web Server
Web Server
App Server
App Server
App Server
Load Balancer
Cache
Cache
Database
Perf Tester 2
AnalyticsLoad
GeneratorLoad GeneratorLoad
Generator
AWS East
AnalyticsLoad
GeneratorLoad GeneratorLoad
Generator
IBM
AnalyticsLoad
GeneratorLoad GeneratorLoad
Generator
RackSpaceCPU, Disk, Memory
Analytics
CloudTest
Database
Main
CloudTestArchitecture
“Conductor”
48© 2013 SOASTA. All rights reserved.
SUT/AUT
Web Server
Web Server
App Server
Web Server
Web Server
App Server
App Server
App Server
Load Balancer
Cache
Cache
Database
Perf Tester 2
AnalyticsLoad
GeneratorLoad GeneratorLoad
Generator
AWS East
AnalyticsLoad
GeneratorLoad GeneratorLoad
Generator
IBM
AnalyticsLoad
GeneratorLoad GeneratorLoad
Generator
RackSpaceCPU, Disk, Memory
Analytics
CloudTest
Database
Main
Massively Scalable Cost Effective Real World Testing
CloudTestArchitecture
“Conductor”
49© 2013 SOASTA. All rights reserved.
CloudTest benefits of the Cloud:
o Near-immediate near-infinite load capability
o Enables distributed real-time analytics
o Real-world load distribution
o Real-world infrastructure
o Cost efficiency:
• No Cap Ex
• Minimal Op-Ex
50© 2013 SOASTA. All rights reserved.
SOASTA Products in Action:Cloud Test Demonstration
51© 2013 SOASTA. All rights reserved.
Thanks
Contact SOASTA:www.soasta.com/cloudtest/
Follow us:
twitter.com/cloudtest
facebook.com/cloudtest
Knowledge Center• White Papers• Webinar Recordings• Case Studies
RESOURCES
www.SOASTA.com
CloudLink Community• Support• Tutorials• Video