Adopting Cloud Testing for Continuous Delivery

1© 2013 SOASTA. All rights reserved.

Adopting Cloud Testing for Continuous Delivery, with IDC and SOASTA

Webinar

Presents

How cloud computing has impacted the Quality as a Service marketplace

Joined by:


Market perspectives and applications of cloud testing

In This Webinar

TODAY’S SOASTA PRESENTERS

Melinda Ballou: Program Director, Application Life-Cycle Management, IDC

Brad Johnson: VP Product Marketing, SOASTAMike Ostenberg: Sr. Solutions Architect, SOASTA

Agenda:

• Poll question• Market Perspective and Recommendations (IDC)• Cloud Testing Drivers and Use-Cases (SOASTA)• Overview and Demonstration of a Leading Cloud Testing

Platform

Questions: Submit in chat box during event


Melinda Ballou

Market Perspective & Recommendations

Copyright 2013 IDC. Reproduction is forbidden unless authorized. All rights reserved.

Adopting Cloud Testing for Continuous Delivery: Evolving Quality to Address Complexity

Adopting Cloud Testing for Continuous Delivery: Evolving Quality to Address Complexity

Melinda BallouIDCProgram DirectorApplication Lifecycle Management & Executive Strategies

Apr 15, 2023

© 2013 IDC 5

Industry Highlights: Trends Driving ASQ & ALM Adoption & ServicesIndustry Highlights: Trends Driving ASQ & ALM Adoption & Services

Organizations re-invest, seeking to do more with fewer resources with financial and staffing constraints; leveraging efficient, agile approaches to restore and sustain high performing, business-critical software.

Complex sourcing/off-shoring plus use of open source need strong teaming, effective code management, testing, change and IT project portfolio management. Services driven environment (SaaS/cloud,Devops emergence)

Global economic competition and local compliance across geographies demand quality, change and portfolio management, adaptability and rigor

Flexible, agile development paradigm with services creation increasingly drive technology and business collaboration; agile emergence also disruptive

Increasing security issues (as driver) and virtualization/cloud (as enabling technology) for ALM adoption; ad hoc approaches unsustainable

End-user experience and business impact challenges of rich Internet, mobile, embedded, with social media collaboration/community opportunities

SaaS license delivery mechanisms bring new models, de-capitalization benefits, flexibility, adoption speed, limit complexity, cloud on-ramp

Apr 15, 2023

© 2013 IDC 6

“Quality Gap”: High Cost of Failure, Cloud Benefits to Address Challenges “Quality Gap”: High Cost of Failure, Cloud Benefits to Address Challenges

Poor Quality = Increased Business Risk

Lost Revenue

($$$$$)

Lost Customers

Lost Productivity

Increased Costs

Lower ProfitsDamaged Brand

Apr 15, 2023

© 2013 IDC 7

App Development Life-Cycle: Old View Rapidly Shifting App Development Life-Cycle: Old View Rapidly Shifting

Assumes: Unidirectional flow, limited # of handoffs

Result:Silos within project team; gulf with operationsLittle understanding how efforts impact overall result Testing takes place LATE in the cycle!

Define Design Develop Test

OPS

Issues

Apr 15, 2023

© 2013 IDC 8

Closing the Loop: Leverage Skills & Tools for Agile, Iterative ApproachClosing the Loop: Leverage Skills & Tools for Agile, Iterative Approach

DesignDefine

DevelopSupport

Deploy & Monitor

Test & Tune

• Shared goals,• Shared metrics,• Shared tests, tools and skills

• Shared virtual lab• Quick iterationscan be disruptive• Demand mgmnt• Cloud adaptability• Enabling agile infrastructure access

mballou

market data for agile here please

Apr 15, 2023

© 2013 IDC 9

Three Key Challenges for IT Three Key Challenges for IT

IT must deliver new applications that have greater business value and higher quality, while managing costs …in the face of these 3 key challenges:

Increasing criticality of applications to the business

Increasing complexity of software systems and applications (from web to mobile to embedded… encompassing social systems of engagement to feed systems of record, performance demand with Big Data Analytics for business optimization)

Increasingly distributed teams with multi-sourcing of applications (from onshore to offshore to open source)

Apr 15, 2023

© 2013 IDC 10

Barriers to Prior Testing Approaches – Cloud Testing DriversBarriers to Prior Testing Approaches – Cloud Testing Drivers

• Agile velocity demands immediate, frequent, iterative testing

• Lack of system resources constrains testing usage – expense limits ability to mirror production configurations (mobile issues)

• User tendency to hoard physical systems for testing creates bottleneck & expense – need for management & coordination

• Challenges to test system configuration & lengthy set-up times limit usage and delays testing

• Lack of change management for testing configurations creates pain points for consistent testing and re-testing

• Need for variable provisioning for complex software deployment (including mobile) increasingly drives cloud adoption for testing to cut costs and increase business agility

Apr 15, 2023

© 2013 IDC 11

Coordinating across the Life-CycleCoordinating across the Life-Cycle

• Coordinating requirements, testing, and operational performance is key across emerging technologies

• Slow response time costs prohibitive for core business areas

• Organizations should leverage quality automation through requirements, unit test, system integration, pre-deployment & application performance testing with emerging cloud platforms

• Evaluating test automation for synthetic transactions and real user measurement and monitoring can help teams react and manage user application experience

• As business requirements change, a cogent life-cycle approach enables adaptive testing and monitoring responses

• Look to ALM SaaS alternatives initially as an on-ramp to cloud where appropriate – strategize through to deployment

Apr 15, 2023

© 2013 IDC

Q. Your organization's adoption of cloud services?

Source: IDC CloudTrack Survey, Winter 2012 n=493

% of Respondents

Adoption Is Building by Cloud TypeAdoption Is Building by Cloud Type

Apr 15, 2023

© 2013 IDC

Source: IDC CloudTrack Survey, Winter 2012 n=493

Mean rating by respondents

Q. How concerned are you about cloud...? (1-5 scale; 5 = extremely concerned)

Performance One of Top Three Concerns: Testing On & Of Cloud Can Help

Apr 15, 2023

© 2013 IDC

Cloud ComputingPublic Cloud Plans: Cloud Testing EmergingCloud ComputingPublic Cloud Plans: Cloud Testing Emerging

Use cloud for developing applications

Use cloud for testing applications

Use cloud for PaaS

Use cloud for application integration

Use cloud compute or storage resources

Use cloud database software

Use cloud for management & monitoring

Use cloud applications

0% 10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

6.7%

9.0%

10.0%

9.7%

9.3%

9.3%

11.3%

12.0%

6.7%

7.3%

8.7%

9.3%

10.0%

11.0%

9.0%

12.0%

No plans to Use Evaluating Now Beginning to UsePlanned Use in 2011 Will Use in 2012/13 DKNo Cloud Familiarity

14

What plans does your company have regarding public cloud services? (multiple response)

IDC 2011 NA Cloud Survey, N=300IT Developer and IT Management Respondents

Apr 15, 2023

© 2013 IDC

Cloud ComputingPerspective on Cloud Computing CapabilitiesCloud ComputingPerspective on Cloud Computing Capabilities

Lower overall cost

Fast application development

Lower admin cost

Makes IT far less complex

Applications that always scale up/down

Avoidance of up front costs

Fast application deployment

Standardized AD&D

Pay only for what you use

Immediate access to technology

Access to latest functionality

0% 10%

20%

30%

40%

50%

60%

70%

80%

90%

100%Completely Agree Mostly Agree Agree Mostly Disagree

Completely Disagree DK No Cloud Familiarity

15

How much do you agree that cloud computing delivers the following capabilities? (multiple response)

IDC 2011 NA Cloud Survey, N=300IT Developer and IT Management Respondents

Apr 15, 2023

© 2013 IDC

Cloud ComputingExpected BenefitsCloud ComputingExpected Benefits

DK or NS

Other

No perceived benefits

Access to best of breed tools

Access to more dev tool functionality

Easier AD&D of SOA applications

Standardized tools

Tool access to test applications

Tool access to deploy applications

Access to integrated tool suite

Access to most up to date tools

Tool access to build applications

More cost effect AD&D

0% 5% 10% 15% 20% 25% 30% 35% 40% 45%

16

What benefits does your organization expect to gain by using public cloud services? (multiple response)

IDC 2011 NA Cloud Survey, Cloud Knowledgeable, N=277 IT Developer and IT Management Respondents

Apr 15, 2023

© 2013 IDC

020

40

Other

Test data and test bed preparations to resemble live scenarios under high volume

Reusability of test artifacts to address rapid change in application functionalities

Complexity of test scenarios to design and execute

Ability to optimize usage of different platform resources (e.g. CPU, memory, battery)

Designing to different form factors (e.g. smartphone versus tablet)

Managing to robust security architecture and protection of end user application data

Rapid pace of change in mobile OS / hardware landscape

Expertise / developing skill sets across multiple mobile OS platforms

Investing in and maintaining varied mobile infrastructures

1.2%

6.5%

7.1%

13.2%

16.6%

26.5%

26.5%

32.3%

34.8%

35.4%

TotalN=325

Top Challenges in Launching Mobile Applications Top Challenges in Launching Mobile Applications

17

Share of Responses (%)

Highlights and Key Messages

Question: What are the biggest challenges you face in launching mobile applications, whether or not you are developing applications internally?

Highlights

Leading challenges concern:

Level of investment and maintenance required to support rapidly evolving mobile IT infrastructure

Building up or accessing resource expertise in the design, development and testing of mobile apps across multiple mobile platforms

Key Messages

Enterprise business priorities to grow revenue and innovate while improving internal operational efficiencies will lead many organizations to seek help from 3rd parties in implementing mobile strategies

It will be a challenge to keep up with the rapid pace of change in mobile OSs while addressing mobile security mandates, designing to different form factors, providing a smooth, productive customer experience

Note: Multiple responses allowed

Apr 15, 2023

© 2013 IDC

- Develop near, mid, and long-term goals for transformation & cloud

• Understand what’s utilized or not, and what’s bought and developed at the line of business

• Base your plan on your organization’s business goals, governance/compliance profile, IT skillsets, and data center investments

• Develop a “visualization” plan for a hybrid cloud strategy that takes you from your current state to your end state, based on level of virtualization, focus on infrastructure vs. apps, etc

• Make sure your plan puts your priorities first – whether cost reduction, consolidation, expansion into new competencies like mobile, analytics, data management, etc.

- Identify easy payback/low downside projects for public cloud services

• Develop/test/deploy and other compute and storage-intensive tasks

• Non-differentiating front-office functions first, the rest can follow -- Land then expand

• Investigate service management and brokerage tools for oversight into performance, service quality, usage

- Go at your own pace

• A majority of companies will always have on-premise gear, and “hybrid” cloud deployments

• Due diligence: compute and storage IaaS clouds are inexpensive, but mistakes are costly

• Be prepared to be more of a services broker, and deal with different types of outside providers than you’re used to: get ready for change

Guidance for Cloud Testing AdoptionTake a Strategic View of Cloud SourcingGuidance for Cloud Testing AdoptionTake a Strategic View of Cloud Sourcing

Apr 15, 2023

© 2013 IDC 19

SummarySummary

• IDC saw 37.7% revenue growth for cloud testing and ASQ SaaS in 2012 with five year CAGR through 2017of 31.1 (to reach $1.01B); re-assess options as solutions expand

• ASQ and development teams must bridge the gap to operations to gain access to appropriate cloud infrastructure for quality; help leverage and coordinate mobile quality

• IDC recommends an encompassing approach to business and IT management that includes cloud and SaaS testing

IT organizations should actively lobby the business and operations for access to cloud testing in the short-term and for other ALM areas over time

Executive management must plan for incorporation of cloud as part of the overall IT portfolio to increase business adaptability and cut costs for multi modal software

Companies should view cloud testing and ASQ SaaS as part of a comprehensive strategy from dev and provisioning through to build and operational planning. This strategy should include mobile as a bridge from systems of engagement to systems of record & operations.

Apr 15, 2023

© 2013 IDC

Melinda BallouProgram Director,

Application Life-Cycle & Executive Strategies

1. Business/IT alignment 2. IT-PPM & PPM strategies 3. IT staffing strategies4. Business strategy 5. IT portfolio mgmt.

Contributing AnalystsContributing Analysts

18

Melinda BallouProgram Director, Application Life-Cycle Mgmt & Executive Strategies

[email protected]

mailto:[email protected]


Brad Johnson@bradjohnsonsv

Cloud Testing Drivers & Use Cases


Business Drivers of CloudConsumer driven (even internal apps)

o Time To Market

6 week delivery plan @ Google

http://www.thechromesource.com/understanding-chromes-six-week-release-cycle/

Rapid updates and adoption







o Time to market

o Global access and device options



o Time to market

o Global access and device options

o End user reach and quality expectations

http://www.soasta.com/products/mpulse/


Goals of Using the Cloud for Testing

Speed

Map to Business Drivers

Global access and device options

• 800,000 servers – 54 locations – 17 countries

Affordability

Time to market

• Test Fast

End user reach and quality expectations

• Frequent, iterative, realistic for high coverage

Scale


The Path to Continuous Testing with the Cloud

o Understand your requirements

o Identify the cloud advantages for you

o Automate the obvious and most critical

o Connect to a continuous process

o Align teams with actionable information


Modern Use-Cases Drive Success and EvolutionIconic Launches to Legendary Landings

http://www.soasta.com/customers/case-studies/



http://www.soasta.com/customers/entertainment/nasa/

http://www.soasta.com/customers/case-studies/cloudtest-helps-sap-co-innovation-lab-demonstrate-scalability-and-reliability/

http://www.soasta.com/customers/case-studies/bonobos-improves-web-site-performance-to-gear-up-for-cyber-monday/


IDC MarketScape 2013

o “a leader in Cloud Testing and ASQ SaaS”

o “leading innovator in the cloud testing arena”

o “Overall, SOASTA is well positioned as a best of breed vendor in this arena.”

o “Differentiators for SOASTA are its strength in cloud-based testing and evolving mobile support and a free CloudTest Lite version”

Worldwide Cloud Testing and ASQ SaaS Vendor Assessment

SOURCE: IDC MarketScape: Worldwide Cloud Testing and ASQ SaaS 2012–2013 Vendor Analysis — Enabling Business Agility and Quality in the Cloud, by Melinda-Carol Ballou, February 2013, IDC #237597, http://www.idc.com/getdoc.jsp?containerId=237597

IDC MarketScape vendor analysis model is designed to provide an overview of the competitive fitness of ICT suppliers in a given market. The research methodology utilizes a rigorous scoring methodology based on both qualitative and quantitative criteria that results in a single graphical illustration of each vendor’s position within a given market. The Capabilities score measures vendor product, go-to-market and business execution in the short-term. The Strategy score measures alignment of vendor strategies with customer requirements in a 3-5-year timeframe. Vendor market share is represented by the size of the circles. Vendor year-over-year growth rate relative to the given market is indicated by a plus, neutral or minus next to the vendor name.

http://www.idc.com/getdoc.jsp?containerId=237597




Mike Ostenberg

SOASTA’s Approach & Architecture to Cloud Testing


The SOASTA PlatformEmbedded Quality for the Mobile Lifecycle

TouchTestContinuous Automated

Testingof Mobile

Applications

CloudTestLoad &

Performance

mPulseReal User Monitoring


SUT/AUT

Web Server

Web Server

App Server

Web Server

Web Server

App Server

App Server

App Server

Load Balancer

Cache

Cache

Database

CloudTestArchitecture

http://www.soasta.com/products/cloudtest/


SUT/AUT

Web Server

Web Server

App Server

Web Server

Web Server

App Server

App Server

App Server

Load Balancer

Cache

Cache

Database

Customers




SUT/AUT

Web Server

Web Server

App Server

Web Server

Web Server

App Server

App Server

App Server

Load Balancer

Cache

Cache

Database

Customers




SUT/AUT

Web Server

Web Server

App Server

Web Server

Web Server

App Server

App Server

App Server

Load Balancer

Cache

Cache

Database


Analytics

CloudTest

Database

Main

“Conductor”“Conductor”



SUT/AUT

Web Server

Web Server

App Server

Web Server

Web Server

App Server

App Server

App Server

Load Balancer

Cache

Cache

Database


Analytics

CloudTest

Database

Main


Perf Tester 1



SUT/AUT

Web Server

Web Server

App Server

Web Server

Web Server

App Server

App Server

App Server

Load Balancer

Cache

Cache

Database


Analytics

CloudTest

Database

Main


Perf Tester 1



SUT/AUT

Web Server

Web Server

App Server

Web Server

Web Server

App Server

App Server

App Server

Load Balancer

Cache

Cache

Database


Analytics

CloudTest

Database

Main


Perf Tester 1



SUT/AUT

Web Server

Web Server

App Server

Web Server

Web Server

App Server

App Server

App Server

Load Balancer

Cache

Cache

Database


Analytics

CloudTest

Database

Main



SUT/AUT

Web Server

Web Server

App Server

Web Server

Web Server

App Server

App Server

App Server

Load Balancer

Cache

Cache

Database


Analytics

CloudTest

Database

Main

Perf Tester 2Perf Tester 2



SUT/AUT

Web Server

Web Server

App Server

Web Server

Web Server

App Server

App Server

App Server

Load Balancer

Cache

Cache

Database


Analytics

CloudTest

Database

Main

Perf Tester 2



SUT/AUT

Web Server

Web Server

App Server

Web Server

Web Server

App Server

App Server

App Server

Load Balancer

Cache

Cache

Database


Analytics

CloudTest

Database

Main

Perf Tester 2

AnalyticsLoad

GeneratorLoad GeneratorLoad

Generator

IBM



SUT/AUT

Web Server

Web Server

App Server

Web Server

Web Server

App Server

App Server

App Server

Load Balancer

Cache

Cache

Database


Analytics

CloudTest

Database

Main

Perf Tester 2

AnalyticsLoad


Generator

IBM

AnalyticsLoad


Generator

RackSpace



SUT/AUT

Web Server

Web Server

App Server

Web Server

Web Server

App Server

App Server

App Server

Load Balancer

Cache

Cache

Database


Analytics

CloudTest

Database

Main

Perf Tester 2

AnalyticsLoad


Generator

IBM

AnalyticsLoad


Generator

RackSpace

AnalyticsLoad


Generator

AWS East



SUT/AUT

Web Server

Web Server

App Server

Web Server

Web Server

App Server

App Server

App Server

Load Balancer

Cache

Cache

Database

Perf Tester 2

AnalyticsLoad


Generator

AWS East

AnalyticsLoad


Generator

IBM

AnalyticsLoad


Generator

RackSpace

Analytics

CloudTest

Database

Main

Massively Scalable Cost Effective Real World Testing




SUT/AUT

Web Server

Web Server

App Server

Web Server

Web Server

App Server

App Server

App Server

Load Balancer

Cache

Cache

Database

Perf Tester 2

AnalyticsLoad


Generator

AWS East

AnalyticsLoad


Generator

IBM

AnalyticsLoad


Generator

RackSpace

Analytics

CloudTest

Database

Main


“Conductor”



SUT/AUT

Web Server

Web Server

App Server

Web Server

Web Server

App Server

App Server

App Server

Load Balancer

Cache

Cache

Database

Perf Tester 2

AnalyticsLoad


Generator

AWS East

AnalyticsLoad


Generator

IBM

AnalyticsLoad


Generator

RackSpace

Analytics

CloudTest

Database

Main


“Conductor”



SUT/AUT

Web Server

Web Server

App Server

Web Server

Web Server

App Server

App Server

App Server

Load Balancer

Cache

Cache

Database

Perf Tester 2

AnalyticsLoad


Generator

AWS East

AnalyticsLoad


Generator

IBM

AnalyticsLoad


Generator

RackSpaceCPU, Disk, Memory

Analytics

CloudTest

Database

Main


“Conductor”



SUT/AUT

Web Server

Web Server

App Server

Web Server

Web Server

App Server

App Server

App Server

Load Balancer

Cache

Cache

Database

Perf Tester 2

AnalyticsLoad


Generator

AWS East

AnalyticsLoad


Generator

IBM

AnalyticsLoad


Generator

RackSpaceCPU, Disk, Memory

Analytics

CloudTest

Database

Main

Massively Scalable Cost Effective Real World Testing


“Conductor”



CloudTest benefits of the Cloud:

o Near-immediate near-infinite load capability

o Enables distributed real-time analytics

o Real-world load distribution

o Real-world infrastructure

o Cost efficiency:

• No Cap Ex

• Minimal Op-Ex


http://www.soasta.com/product-features/real-time-analytics/



http://www.soasta.com/products/campaign-global-test-cloud/

http://www.soasta.com/products/campaign-global-test-cloud/


SOASTA Products in Action:Cloud Test Demonstration

http://www.soasta.com/cloudtest/demos/intro-cloudtest-part-1/


Thanks

Contact SOASTA:www.soasta.com/cloudtest/

[email protected]

Follow us:

twitter.com/cloudtest

facebook.com/cloudtest

Knowledge Center• White Papers• Webinar Recordings• Case Studies

RESOURCES

www.SOASTA.com

CloudLink Community• Support• Tutorials• Video

http://www.soasta.com/cloudtest/

http://www.soasta.com/cloudtest/




http://cloudlink.soasta.com/t5/Knowledge-Base/ct-p/knowledgebase

http://www.SOASTA.com/

http://cloudlink.soasta.com/

http://cloudlink.soasta.com/t5/Support/ct-p/Support

http://www.soasta.com/free

http://www.soasta.com/knowledge-center/whitepapers/idc-marketscape-worldwide-cloud-testing-and-asq-saas-2012-2013-vendor-analysis-enabling-business-agility-and-quality-in-the-cloud/?showform=true

Adopting Cloud Testing for Continuous Delivery

Technology

cloud benefits

cloud computing

soasta agenda

idc brad johnson

agile approaches

evolving quality

higher quality

quality gap