Saas Testing Techniques_ValueLabs

05/07/12 11:15 AM

Cloud TestingTo ensure a successful cloud computing strategy, you must be able to:

Manage performance and availability across the entire cloud service delivery chain

Monitor cloud application performance from the end-user perspective

Test your cloud applications prior to deployment

Monitor your cloud applications after they go into production

The performance and availability of cloud applications can have a dramatic impact on user adoption and

revenue. Monitoring and testing the performance of those applications requires uninterrupted visibility

across the entire application delivery chain i.e. from your data center, through the Internet and cloud

service providers to your end user’s own device and browser

Note: Traditional data center monitoring tools simply won’t work in the cloud. You need to monitor and

test your cloud applications from the only perspective that really matters: your end users.

Cloud Testing has four major objectives:

To assure the quality of cloud-based applications deployed in a cloud, including their functional

services, business processes, and system performance as well as scalability based on a set of

application-based system requirements in a cloud

To validate software as a service (SaaS) in a cloud environment, including software performance,

scalability, security and measurement based on certain economic scales and pre-defined SLAs.

To check the provided automatic cloud-based functional services, for example auto-provisioned

functions

To test cloud compatibility and inter-operation capability between SaaS and applications in a

cloud infrastructure, for example, checking the APIs of SaaS and their cloud connectivity to others

i.e. another SaaS with in the same/different cloud or end user interface .

Below Table shows the detailed tasks and comparative view among different parties:

Test Type Testing focuses Cloud/SaaS-Oriented

Testing inside a Cloud

Online Application-

Based

Testing on a Cloud

Cloud-Based Application

Testing over Clouds

Service

Function

Testing

GUI-based and API

based

service functions

Testing SaaS/Cloud

based

service functions

inside a cloud

Testing online-

based

application service

functions on a

cloud

Testing cloud-based

application service

functions over a cloud

infrastructure

Integration

Testing

SaaS interactions

and

Cloud connections

Vendor-specific

component and

service

integration inside a

private/public cloud

Integration

between online

clients and back-

end

servers on a cloud

- End-to-end application

integration over clouds

- Integration with legacy

systems over clouds

API and

Connectivity

Testing

API interfaces

and connectivity

protocols (HTTPS,

REST, SOAP, RMI)

SaaS/Cloud API &

connectivity testing in

a cloud

Testing user-

centered

service APIs and

connectivity on a

cloud

Testing application service

APIs and connectivity over

Clouds

Performance

& Scalability

Testing

Performance and

scalability based on

a

SLA

SaaS/Cloud

performance

and scalability testing

in a cloud based on

the

given SLA

User-oriented

application

performance and

scalability

testing on a cloud

End-to-end system-level

performance and

scalability

inside/on/over cloud based

on a given SLA

Security SaaS/Application SaaS/Cloud security User-oriented System-level end-to-end

Testing data,

processes,

functions, and

user privacy

features and user

privacy

in a cloud

security and

privacy on a cloud

security over clouds

Interoperability

&

Compatibility

Testing

Validate different

client interfaces

and

technologies and

diverse

compatibilities on

different platforms

and

browsers

Testing Cloud/

SaaS compatibility,

connectivity protocols

and UI/client

technologies inside a

cloud

Testing user-

centered

interoperability,

compatibility of

platforms/

OS/browsers, and

client

technologies on a

cloud

Testing application

compatibility, end-to-end

interoperability and

application connectivity to

legacy systems over

clouds

Regression

Testing

Changed &

impacted

SaaS/Cloud service

features and

related APIs/

connectivity

Cloud/SaaS-oriented

regression testing

inside

a cloud

User-centered re-

validation

on a cloud

End-to-end application

system regression over

clouds

1.3 Cloud Testing VS. Conventional Software Testing:

Internet-Based Software Testing

(i.e. Distributed/Web-Based System

Infrastructure)

Cloud-Based Software Testing

Primary

Testing

Objectives

- Assure the quality of system

functions and performance based on

the given specifications

- Check usability, compatibility,

interoperability.

- Assure the quality of functions and

performance of SaaS , Clouds,

and applications by leveraging a cloud

environment

- Assure the quality of cloud elasticity &

scalability based a SLA

Testing as a

service

- In-house internal software testing as

engineering tasks

Real-time on-demand testing service offered

by a third-party

- Online testing service based on a pre-

defined SLA

Testing and

Execution

Time

- Offline test execution in a test lab.

- Testing a product before its delivery

- On-demand test execution by third-parties;

- Online test execution in a public cloud;

- Offline test execution in a private cloud

Testing

Environment

- A pre-fixed and configured test

environment in a test lab. with

purchased hardware and/or software

- An open public test environment with diverse

computing resources

- A scalable private test environment in a test

lab.

Testing

Costs

- Required hardware costs and

software (license) costs

- Engineering costs in a test process

- Based on a pre-defined service-level-

agreement (SLA)

- TaaS and Cloud testing service costs (pay-

as-you-test)

- Engineering costs in SaaS/Cloud/application

vendors

Test

Simulation

- Simulated online user access

- Simulated online traffic data

- Virtual/online user access simulation

- Virtual/online traffic data simulation

Function

Validation

- Validating component functions and

system functions

as well as service features

- SaaS/Cloud service functions, end-to-end

application functions

- Leveraged functions with legacy systems

Integration

Testing

- Function-based integration

- Component-based integration

- SaaS-based integration in a cloud

- SaaS integration between clouds

- Architecture-based integration

- Interface/connection integration

- Application-oriented end-to-end integration

over clouds

- Enterprise-oriented application integration

between SaaS/Cloud

and with legacy systems

Security

Testing

Aim to the following targets:

Function-based security

features

User privacy

Client/server access security

Process access security

Data/message integrity

Aim to the following targets:

SaaS/Cloud security features,

including monitor and measurement

User privacy in diverse web clients

End-to-end application security over

clouds

SaaS/Cloud API and connectivity

security

Security testing with virtual/real-time

tests in a vendor’s cloud

Scalability

&

Performance

Testing

- Performed a fixed test environment

- Apply simulated user

access, ,messages, and test

data

- Online monitor and evaluation

- Performed in a scalable test environment

based on a SLA

- Apply both virtual and real-time online test

data

- Online monitor, validation, and measurement

Characteristics of SaaSSoftware as a Service (SaaS) is defined as software that is deployed over the internet… With SaaS, a

provider licenses an application to customers either as a service on demand, through a subscription, in a

“pay-as-you-go” model, or (increasingly) at no charge when there is opportunity to generate revenue

from streams other than the user, such as from advertisement or user list sales. SaaS applications are

designed for end-users, delivered over the web.

Following are the characteristics of SaaS:

Web access to commercial software

Software is managed from a central location

Software delivered in a “one to many” model

Users not required to handle software upgrades and patches

Application Programming Interfaces (APIs) allow integration between different pieces of software

SaaS Attributes:

Integration with External Applications: Simple Object Access Protocol (SOAP)-based Service

Oriented Architecture (SOA), Extract Transform Load (ETL) and On Line Analytical Processing (OLAP)

Application Programming Interfaces (APIs)

Manageability: Multi-tenant architecture to support clients from a single instance in order to reduce

the costs of infrastructure, hosting and management

Performance: Distributed data caching and code optimization tools for improving performance and

response time

Scalability: Meta-database and load balancing for scalability

Security: Multi-tiered, multi-layered, role-based security model. Typically improves due to centralization

of data, increased security-focused resources, etc., but raises concerns about loss of control over certain

sensitive data. Security is often as good as or better than traditional systems, in part because providers

are able to devote resources to solving security issues that many customers cannot afford. Providers

typically log accesses, but accessing the audit logs themselves can be difficult or impossible

Time-to-Market: Distributed Agile methodology and platform (GlobalLogic Velocity) to accelerate time-

to-market and provide shorter release cycles

Usability: AJAX-based APIs to provide interactive, professional-looking Graphical User Interfaces

(GUIs) supported by a dedicated team of usability experts

Compatibility: Portability experts to provide consistent support across a variety of browser platforms

Availability: 24/7 in-house support services to ensure uptime and continuous availability

Expertise on Open Source: Use of tools to reduce total cost of ownership

Reliability: Improves through the use of multiple redundant sites, which makes it suitable for business

continuity and disaster recovery. Nonetheless, most major cloud computing services have suffered

outages and IT and business managers are able to do little when they are affected.

Sustainability: Comes about through improved resource utilization, more efficient systems, and

carbon neutrality. Nonetheless, computers and associated infrastructure are major consumers of energy

Maintainability: Usually this includes System/Integration testing, Performance testing, and User

Acceptance testing cycles. The client must be confident the new version of the software works in their

environment AND with all of the interfacing applications.

The process is significantly streamlined with SaaS. The client is relieved of the burden of testing the new

software release in their environment, as the SaaS provider handles this for them.

Note: If your implementation of a vendor's SaaS application is integrated with one or more external

application (be they on-premise or SaaS), you must work closely with the vendor to ensure that no APIs

upon which your integrations depend are being deprecated as part of this release. If you are dependent

upon deprecated APIs, you must re-write your interfaces to the new API or your intra-SaaS application

business process will fail.

Interoperability: Cloud computing architectures are a heterogeneous blend of technologies and

platforms. The various software applications residing in the cloud do not exist in isolation. They must be

able to communicate and exchange information transparently, irrespective of the technologies used to

implement them. Thus, interoperability among the cloud SaaS is a relevant and significant issue in cloud

computing. Interoperability between SaaS is possible using Web standards and middleware (possibly

hosted in Cloud)

Adaptability: The entire way the software runs can be tailored for individual organizations

and to let any company define the hierarchies specific to them, and yet the overall software

works out of a single code base

Customizable: In Software-as-a-Service (SaaS) delivery model a vendor maintains a single

application instance, which is used by multiple tenants. However, due to changing business requirements

tenants expect customizations. Providing such customizations is trivial to retain tenants but a challenge to

the vendor due to multi-tenancy.

The table below lists all the Attributes of SaaS Application:

User experience

Usability

o Responsiveness

o Efficiency

o Performance

Data

State

State full

Stateless

Stability

o Personalizable

User interface

o Graphical

o Interactive

o Distributed

o Textural

o None

Interaction model

o Device

o SaaS

o Online

Application constraints

Database constraints

Persistence

Online/Offline

Structure

Unstructured

Indexed

Searchable

Transaction management

Security

Emergency hot fix or breach

management

Security procedures

Trust relationship with platform

Applications security model

Data flow

Malicious code

Access controls

Maintainability

Available skill sets

Language support (dev)

Application standards

Technology implementation

Application-code complexity and

volume

Configuration management

Operational management

Remote access

Identity

Cryptography

Auditing

Authentication/Authorization model

Flexible

Technology

Affordability

Resource cost

Development

Available skills

Software enhancements cost

Licensing

Postproduction hardware

Decommissioning

Initial hardware

Scalability

Replication

Caching

Pooling

Software load balancing

Scale out

Scale up

Hardware load balancing

Conformability

Auditable

Regulatory

Standards

Availability

Technology/Configuration/

Implementation to support availability

Uptime requirement

Portability Reliability

Cross-platform

Within platform

Configuration management

Startup and automatic recovery

System performance

Recovery procedures and methods

Load balancing

Fault tolerance

Distributability

Local

Geo-distributed

Interoperability

Communications and data usage

Integration impacts

Architecture compatibility

Ease integration (APIs)

Extensibility

Meta-model

Configurable

Reusability

Distributable and reusable

Modularity

Hierarchy

Code abstraction

How we test our SaaS QA Platform?

The cloud is defined by its service model, deployment model and usage: Mostly

cloud applications are based on SaaS. They are Software as a Service solutions that run completely on

cloud infrastructure and platforms. Hence testing of SaaS applications is completely different from testing

the traditional applications. They need to be tested on three levels: namely the infrastructure, the platform

and the application itself. The usage of standard services of applications also means a change for system

testing.

In principle, it’s not different from testing any other application; it requires merging different techniques

used in daily basis.

Cloud/SaaS-oriented testing: This type of testing activities usually is performed

inside a cloud by engineers of cloud/SaaS vendors. The primary objective is to assure the

quality of the provided service functions offered in a cloud (or a SaaS program). These

engineers must go through unit testing, integration, system function validation, regression

testing and cross platform (compatibility) testing, as well as performance and scalability

evaluation. Since clouds and SaaS usually provide certain service APIs and connectivity

interfaces to their customers, it is required task for engineers to validate these APIs and

connectivity in a cloud environment. In addition, testing cloud-based or SaaS-based security

services and functional features must be tested. Furthermore, performance testing and

scalability evaluation in a cloud is very important and critical to cloud/SaaS vendors because

this assures the quality of cloud elasticity to support SaaS and cloud services inside a cloud.

Testing Categories: Following are list of testing techniques that can be used to test SAAS

platform at different phases:

Test Category Testing techniques

Business Testing Manual/Automation functional Testing

Exploratory Testing

End to End business workflow testing

Manual/Automated regression testing

Data integration and data migration testing

Checklist validation

Security Testing Application Security Testing

Network Security testing

User Access and Roles testing

Data security integrity testing

Compliance testing

Identity Federation mechanism testing

Performance Testing Scalability testing

Volume Testing

Availability testing

Reliability testing

Load testing for single instance

Load testing in a instance loaded environment

Compatibility Testing Multi-browser and OS compatibility

Localization testing

Accessibility testing from remote locations

Internalization testing

Interface backward compatibility testing

Live Testing Disaster recovery testing

Statefull scenario testing

Live upgrade testing

Saas Attribute Testing Multi-tenancy isolation testing

Api Integration testing

Billing mechanism testing

Functional Testing Checklist:

For any application we make sure that the functionality works as expected. This is the standard functional

testing to validate if the app is doing what it is supposed to do.

Conduct rigorous Manual tests as per defined test plans, keeping the end user in mind

Conduct Exploratory tests based on existing or new test cases

Conduct Browser compatibility testing to check performance of the application on different web

browsers

Conduct Regression testing on every release, minor upgrade, an integration or data migration.

Automate Functional and Regression tests

Conduct tests in target environment – whether it is your data center or the Cloud.

Conduct reliability testing to find the total defects of the application and thus reduce the number of

failures, during real time deployment.

Multi-platform support/Compatibility Testing:We can use combination different browsers versions and operating systems to perform cross-platform testing.

We have to use browsers and operating systems majorly used by the end users to find issues that end user

may encounter with his/her browser/operating system combinations.

Load over different clouds:Application / system stability is a major factor as the user count is expected to be in multiples of hundreds.

SaaS based application needs to handle large amounts of users and we don’t have the luxury of re-booting or

going down once in a while. Conduct load testing under normal as well as peak load conditions in multiple

environments i.e. to determine the limits

Stress over different clouds:Due to the cloud characteristics, it is imperative to identify issues as system is tested to breaking points

maximum expected capacity or often beyond to 2x, 3x,nx expected usage. Pushing systems to maximum load

capacity and beyond i.e. Exceed Break points

Capacity Testing:Being hosted in a cloud environment it is prudent to determine maximum capacity for current or future

hardware, bandwidth or other needs or to validate that installed hardware and network will support expected

usage scenarios i.e. Plan for the Future

Conduct scalability tests to determine the capacity of the application to scale up or down as per

requirements

Availability Testing:Conduct availability testing for a planned period of time and 24/7

Volume Testing:Conduct volume testing for your data

Performance/Latency over different clouds:Measure response times and isolate issues related to specific steps or actions while system is subjected

to increasing load from different locations and multi user operations. Measuring response time variance

over load and time

Reliability/Soak Testing: Measuring performance degradation over longer periods at varying load levels i.e. Reliability over

time

Remote Access and Usage:We make sure that all users regardless if they come from the US, Holland, India, Argentina, or

Australia can work with the system with good response times.There are many emulators that can help

test this in your lab

Failover Testing/Disaster recovery & rollback procedures:Another testing task coming to us from the IT side of the house. Here we run 2 main scenarios:

o System down that needs to be brought up quickly (with the same machines or with new ones

that require installation and configuration)

o Rollbacks to the last known stable version, including data.

o Verify Redundancy

Application/Infrastructure security testing:The goal of this is to test the underlying infrastructure and security of the app:

Test the security of the SaaS application for typical web application security issues such as HTTP

header injection, Cross Site scripting (XSS), SQL Injection etc.

Test security of the network where SaaS application is being deployed

Test possible scenarios of security attacks/threats

Test the application with respect to access privileges with the corresponding job roles (especially in a

multi-tenant environment)

Test the security, integrity & accessibility of test data (especially in a multi-tenant environment )

Determine situations that could make the SaaS application vulnerable

Test compliance with Payment Card Industry Data Security Standard (PCI Compliance)

Maintain logs of security warnings, errors and requests from unreliable sources

Security Management: Dedicated or shared firewall, firewall management; VPNs; intrusion detection &

IDS management; systems software hardening; security audit / vulnerability scanning and notification.

Vulnerability AssessmentsWhat are your weaknesses? Our vulnerability scans are based on

a variety of compliance regulations such as:

Payment Card Industry (PCI) Data Security Standard

Health Insurance Portability and Accountability Act (HIPAA)

Sarbanes-Oxley (SOX)

Gramm-Leach-Bliley Act (GLBA)

Federal Information Security Management Act (FISMA)

Statement on Auditing Standards Number 70 (SAS70)

Managed Firewalls: Test with different types of firewall devices such as From Cisco to Checkpoint to Sonic

wall and implement the best practices

Patch Management: Each customer environment is different and we stay aware of how change will affect

it. We work with end users to establish a patching strategy that meets end user needs. Every patch is

analyzed. A risk assessment is made to be sure that your environment is not only safer by applying the patch

but won't be adversely affected.

Intrusion Detection Systems (IDS): We use both Network Intrusion Detection Systems (NIDS) and Host-

based Intrusion Detection Systems (HIDS) to ensure that the “bad guys” stay out. With systems powered by

Cisco, Checkpoint and OSSEC we perform log analysis, file integrity checking, policy monitoring, root kit

detection, real-time alerting and active response.

Security Policies:We know that security is a top priority. Whether it's user admin or system builds, our

documented procedures are built on years of experience and industry best practices for security and

compliance.

Incident Response:If you're learning the hard way at your own or a third-party location and experience an

attack, the vendor team can leap into action to control and repair the damage. We understand how to contain

the breach, develop an action plan to systematically verify integrity of your network and all your devices, then

recommend and help implement solutions to protect from future attacks.

GRC (Governance Risk Compliance) testing: Devise a unique comprehensive testing strategy for

compliance with standards like PCI and government regulations

Scalability over different clouds: This assures the quality of cloud elasticity to support SaaS and

cloud services inside a cloud.

Operational Testing:This area is intended for the operations team whose objective is to make sure the apps are working fine, and

take care of customer service & billing. Usually, there are tools that are built as part of the product which help

the operational team members to monitor, track and analyze for issues. The areas to look for:

Application, Services, App Server, Platforms (OS), Databases and Data Center Level

Logs/Alerts/Warnings/Errors for functionality and performance.

Billing and Customer Support Tools, especially for integration

Integration and API Testing:Success of SaaS apps lie in how well you have thought of scenarios where third party developers can build

their own apps using your APIs, and add value to your product. So testing all the APIs for functionality,

security, usability, performance and completeness of documentation is critical to make them successful:

SaaS based application interactions and cloud connections with different client interfaces, database

servers

SaaS based integration in a Cloud

SaaS integration between Clouds

Application oriented end to end integration over clouds

Enterprise oriented application integration between Saas/Cloud and with Legacy systems

Usability over different clouds: Test the Responsiveness, efficiency, Performance and

Personalizable

Cloud based Unit testing: Unit testing on different clouds

Cloud-based application integration: Application integration testing on different clouds

End-to-end system function testing: System testing on different Clouds

Cloud based System Integration testing: SaaS usually provide certain service APIs and

connectivity interfaces to their customers, it is required task for engineers to validate these APIs and

connectivity in a cloud environment.

Saas based application interactions and cloud connections with different API interfaces and

connectivity protocols(HTTPS, REST, SOAP and RMI).

Live updates and deployments: Here is something we think about in regular applications:

How do you deploy the system while it is still running, and if needed how do you minimize down-time for

the users? This is something that is developed and handled by our IT department, but since the delivery

and update of the product is part of the overall user experience we test it constantly.

Internationalization/I18N: Since our platform is used by people around the world we make sure

that we support the use International Characters.

Use Case Scenarios: Ethernet Fabric by validating QoS of high density 10/40/100Gb Ethernet top of rack, end of row

and WAN optimization switches

Storage Networks by ensuring QoS of Fibre Channel and Fiber Channel over Ethernet storage

networking devices and storage systems

Application Networking by testing QoS of Firewalls, IPS, WAN accelerators, Proxy Servers, SSL

VPNs, etc

Cloud Virtualization by benchmarking QoS of virtual switching and virtual appliances from blade

servers to any point in the Ethernet and Storage fabric

Exchange documents, such as purchase orders, with any business partner over the Web with

B2B integration technologies, while eliminating the costs of proprietary EDI solutions.

Automate any mission-critical process such as Order-to-Cash, giving you more visibility into your

business, whether you represent marketing, sales, IT or support.

Interoperability Between Local and Global ADC Functions:Cloud balancing is based on making routing decisions based on a combination of local and global variables.

This requires interoperability between local and global ADC functions. Standards-based APIs may eventually

emerge that will facilitate the cross-vendor exchange of cloud balancing variables. In the mean time, in those

situations in which multiple ADC vendors are involved, IT organizations will need to take advantage of the

APIs supported by each vendor in order to achieve an integrated set of variables to use to make routing

decisions. Another option that IT organizations have is to adopt a single vendor strategy for both local and

global ADC functions. The feasibility of implementing a single vendor strategy across the enterprise and one

or more IaaS providers is enhanced if the ADC is available in a virtual appliance form factor.

Focuses on different client interfaces and connecting to legacy systems

Synchronizing Data between Cloud Sites:In order for an application to be executed at the data center that is selected by the cloud balancing

system, the target server instance must have access to the relevant data. In some cases, the data can be

accessed from a single central repository. In other cases, the data needs to co-located with the

application. The co-location of data can be achieved by migrating the data to the appropriate data center,

a task that typically requires highly effective optimization techniques. In addition, if the data is replicated

for simultaneous use at multiple cloud locations, the data needs to be synchronized via active-active

storage replication, which is highly sensitive to WAN latency.

Challenges:Most organizations report impediments to SaaS testing like – short notice periods for QA notification, frequent

testing of live upgrades, short validation cycle times, impact on multiple subscriber organizations, privacy

violations, errors due to rapid addition of new features, time taken for data migration, concerns over data

security & integrity etc. cloud the obvious benefits of SaaS testing.

1. Handling Changes through Frequent Releases: Every time the Application is upgraded,

the users have to understand the impact of the change, validate it against the existing system &

ensure that the impact on the existing features of the application is minimal. Managing and executing

all these activities within a short time span (1-2 weeks) is challenging.  When SaaS upgrades involve

interface upgrading, compatibility and integration issues across old and new interfaces crop up for the

subscribers.  Live upgrades being simulated or tested on the SaaS application impedes the activity of

the existing users.

2. Security Testing: Maintaining data security, accessibility & integrity on a single SaaS application

across multiple tenants. To understand individual privacy requirements, privilege levels, behavioral

patterns and provide adequate privacy to the data can be a daunting task. Cloud computing security

challenges fall into three broad categories:

Data Protection: Securing your data both at rest and in transit

User Authentication: Limiting access to data and monitoring who accesses the data

Disaster and Data Breach Contingency Planning

3. Integration Challenges: When subscribers integrate their internal enterprise applications with

SaaS, inbound and outbound data integration validations from client networks to the SaaS providers

is needed. In such cases it is very difficult to conduct thorough validation simultaneously ensuring

100% data security and privacy

4. Data Migration Issues: Data migration across different SaaS applications or from other

applications to SaaS can be challenging in terms of time taken for understanding the requirements

and the exhaustive integration validation processes

5. Licensing: The SaaS app licensing may vary by functionality, usage (such as volume of

transactions or amount of specific data) or # of named/concurrent users.  All this needs to be tested

across every release.

6. Performance testing: Successfully modeling the most-used business transactions, application

usage and user mix may require greater diligence than an on-premise application.

Risks: Accountability and Data Risk

User Identity Federation

Regularity Compliance

Business continuity and Resiliency

User Privacy & Secondary Usage of Data

Service & Data Integration

Multi-tenancy & Physical Security

Incidence Analysis & Forensics

Infrastructure Security

Non-production Environment Exposure

1. Accountability: In traditional data center, the owning organization(End user) is accountable for

security at all layers i.e. Application/ Database/Computing/Network/Storage layers. You can outsource

hosted services but you cannot outsource accountability.

In a cloud, who is accountable for security at these layers?

Data can be stored anywhere at different geographical locations:

How sensitive is the data? (Informal blogs, public network sharing posts, public news, New group

messages, Health Records, Criminal Records, Credit History and Payroll)

Who owns the data?

Is data encrypted single Vs multiple keys

Data Mitigation:

Logical isolation of the data of multiple consumers

Provider fully destroys deleted data

Multiple encryption keys

2. User Identity Federation:

Security Risks

Managing Identities across multiple providers

Less control over user lifecycle (off-boarding)

User experience

Mitigations

Federated Identity

Auth for backend integrations

Tighter user provisioning controls

3. Regulatory Compliance:

Data that is perceived to be secure in one country may not be perceived secure in another

country/region. European Union (EU) has very strict privacy laws and hence data stored in US may

not comply with those EU laws (US Patriot Act allows federal agencies limitless powers to access any

corporate data etc)

Lack of transparency in the underlying implementations makes it difficult for data owners to

demonstrate compliance (SOX/HIPAA etc.)

Lack of consistent standards and requirements for global regulatory compliance –data governance

can no longer be viewed from a point-to-point data flow perspective but rather a multi-point to multi-

point.

Mitigations

Apply risk management framework, case-by-case basis

Define data protection requirements and SLAs

Provider / Consumer agreement to a pre-defined RACI model

4. Business Continuity and Resiliency:

Lack of know-how and capabilities need

Cloud provider may be acquired by a consumer’s competitor

Monetary losses due to an outage

Mitigations

Contract defines Recovery Time Objectives and monetary penalty for downtime

Cloud provider’s Business Continuity program certified to standard such as BS 25999

5. User Privacy & Secondary Usage of Data:

Users Providers

Privacy of my data:

Address, Email, (Personally Identifiable

Information)

Health, personal financial info

Keep Revenue Up/ Cost Down:

Push out the liabilities to user via Privacy

and Acceptable Use Policy

Build Additional Services on users behavior

Personal Details (email, IMs,….) (targeted advertisements) e.g. Google

Email, banner adv.

Do minimal to achieve compliance

Keep their social applications more open

(increased adoption)

User personal data mined or used (sold) without consent-Targeted Advertisements, third

parties

User Privacy data transferred across jurisdictional borders

No opt out features for user (user can not delete data)

Lack of individual control on ensuring appropriate usage, sharing and protection of their

personal information.

Law Obligation for providers

Key escrows to law agencies

Subpoena

Mitigations

Policy Enactment

o Privacy and Acceptable Usage

o Consent (Opt In / Opt Out)

o Policy on Secondary Usage

De-identification of personal Information

Encrypted storage

Terms of Service with providers

o Responsibility on compliance

o Geographical affinity

6. Service and Data Integration:

Data traverses through the internet between end users and cloud data centers.

How secure the integrations are?

Mitigations

Encryption keys single Vs multiple

Secured protocols

7. Multi-tenancy and Physical Security:

Security Risks

Inadequate Logical Separations

Co-mingled Tenant Data

Malicious or Ignorant Tenants

Cross-Tenant Attacks

Side channel Attacks

Scanning other tenants

DoS

Shared Service-single point of failures

Uncoordinated Change Controls and

Misconfigs

Performance Risks

WordpressOutage June 2010

100sof tenants (CNN,..) down in multi-

tenant environment.

Uncoordinated Change in database

Mitigations

Architecting for Multi-Tenancy

Data Encryption (per tenant key management)

Controlled and coordinated Change Management

Transparency/Audit-ability of Administrative Access

Regular Third Party Assessments

Virtual Private Cloud (VPC)

8. Incidence Analysis & Forensic Support:

Complex integration and dynamics in cloud computing present significant challenges to timely

diagnosis and resolution of incidents such as:

Malware detection and

Immediate intrusion response to mitigate the impact.

Implications to Traditional Forensics? (Seizing equipment and analysis on media/data recovered)

International differences in relevant regulations …

Mitigations

Comprehensive logging

Without compromising Performance

Dedicated Forensic VMImages

Infrastructure Security:Malicious parties are actively scanning the internet for Vulnerable Applications or Services such as:

Active Unused Ports

Default Passwords

Default Configurations

Data

Mitigations

Segregation of duties and role based administrative privileges

Third party audits and app vulnerability assessments

Tiered architecture with appropriate security controls between them

Hardening(Networks, OS, Apps)

9. Non-production Environment Exposure:

Non-Production Environments are for design, development, and test activities internally within an

organization:

Typical non-prod environment use generic authentication credentials

Security flaws

Data copied to non-prod from its production equivalent

High risk of an unauthorized user getting access to the non production environment

Mitigations

Use multi layers of authentication

Non-prod data is not identical to production

Don’t use cloud for developing a highly sensitive app in the cloud

Over Coming Challenges of Saas Testing:

Challenges Mitigation PlanTesting frequent SaaS upgrades – Short notice

period

(1-2 weeks) for a QA notification to validate the

application

The use of automation tools for building regression suites

brings in business value and helps quickly validate the

impact of upgrades

Business knowledge for effective testing of

configurable and non-configurable components

Gain comprehensive and competent knowledge on the

configurable and non-configurable components of SaaS

applications

Any non-configurable upgrade/change to the application

will need to be assessed thoroughly since this will have an

impact on all SaaS subscribers

Though the configurable upgrade/change would not

impact every client, it is advised to validate the impact of

these changes as well

Validating interface compatibility The backward compatibility of a SaaS interface needs to

be validated to ensure that the organizations do not have

to make any changes at their end, and can continue using

SaaS applications as before

Compliance with government regulations and

other standards

Devise a unique comprehensive testing strategy for

compliance with standards like PCI and government

regulations

Data security and privacy Validation of strong encryptions is needed to ensure data

security

Data security and privacy would need to be thoroughly

validated amongst multiple tenant scenarios to ensure

that there are no loop holes

Testing access controls, multi-privileges for securityPerform access control and multi-privilege tests with users

that have varied roles, different privileges and are

executing unique activities (simulating real life usage

scenarios)

Data integration - inbound & outbound Test data transfers between an organization’s network

and SaaS applications.

- Also, measure, compare and validate the performance of

data migrations between SaaS applications and an

organization’s network

Simulating live upgrade testing Live upgrade tests should be carried out in cloud based

pre-production environments

Use automation tools to simulate the scenario of multiple

concurrent users logged on to a current SaaS version.

Conduct live upgrades in cloud based environments

Use automation tools to validate the accuracy of the

upgrade

Optimization of testing that is common to the

impacted core and non-core areas of SaaS when

getting customized

Create a test strategy to test the core product of SaaS

Create a standard suite of automated test cases to

validate the core SaaS product

Create a map/grid of the core and the non-core areas of

the SaaS application that are most likely to be impacted

during customization

Run a regression suite selecting the tests associated with

the impacted areas

Data migration from the existing system to SaaS

application

Identify the different data sources in the existing system

that need to be migrated to the SaaS application.

Select tools that will help in the data migration and in the

post migration validation

Frequent releases of feature rich SaaS

applications increases the time taken for testing,

owing to the significant number of pages to be

Create an automated test library for SaaS applications

that help reduce the associated testing effort that comes

with each frequent release

covered

Rapid addition of new features to the core SaaS

product to meet new customer demands and to

stay competitive. However, every change is a

potential security bug/ performance issue

Formulate a comprehensive strategy for testing the SaaS

applications with test tools that cover functional,

performance and security requirements

Maintain a test repository of results, performance

benchmarks and access privilege grids, which would

facilitate faster validation

Execute comprehensive tests with automated tools that

cover the functional and nonfunctional requirements.

Conduct a continual impact analysis of requirements and

regularly update the test library to help minimize risks.

Implementation of Saas Testing:Now, let’s take a look at the SaaS testing process itself. SaaS testing begins with assessing the

functional and non-functional requirements for the SaaS application, including business, operational

and non-functional needs. Once this is done, the focus then moves into understanding the usage

pattern of the application. This particular set factors in the variations due to geographies, peak periods

and network latencies across regions.

A test plan would need to be developed to include all components of the SaaS application. The plan

would also have details on how these components would be tested and the resources needed to carry

out the same. Once the test plan is approved, the QA team would prepare test cases, test suites and

eventually get the test data ready. The QA environment is then validated for its preparedness for SaaS

Testing. After the assessments confirm the preparedness, test data is populated in the QA

environment through data migration from the existing system.

Then the test team focuses on the automated test suite generation for functional and non-functional

validations. This would be followed by test execution, reporting, publishing and finally culminate with

the issuance of the SaaS readiness certification. See figure 3 (The SaaS Testing Process) to get details

on all steps and processes required for ensuring a systematic and successful SaaS Testing.

Assess the functional & non-functional test requirements

Understand the usage patterns

Test strategy & plan

Prepare test case & suite

Prepare test environment

Populate test data

Generate automated test suite for functional & non-functional test requirements

Execute SaaS testing, report & publish

SaaS Certification

Benefits of Saas TestingThere are multiple benefits that SaaS testing delivers to organizations:

Reduces effort required and go-to-market time associated in procurement, upgrades, renewals,

contracts, maintenance and deployment

Lowers costs associated with test tools, test environments, maintenance and upgrades.

Helps focus on the SaaS application configuration rather than on provisioning for the

application and associated infrastructure requirements

Significantly reduces CAPEX associated with setting up of environment for SaaS application,

helping convert the same into OPEX

Reduces shelf ware risk of SaaS application and testing tools associated with the validation of

the application

Testing costs are reduced by almost one third as the need to test client server installations,

multi-platform backend support, multiple versions of upgrades and backward compatibility is

completely eliminated

Using SaaS testing tools are not system or machine dependent. For example, any local

machine connected to a cloud network can be used for performance testing of the SaaS

application This helps save effort and overhead expenses associated with the installation,

configuration and maintenance of additional machines for enabling SaaS testing tools

Conclusion:

SaaS testing focuses on ensuring high quality across the application, its cloud characteristics and SaaS

attributes. It also includes testing for security, privacy, accessibility and standards compliance as well.

A thorough understanding of the SaaS application, the customer specific implementation, components

that are configurable and non-configurable and how any change or upgrade would impact the

application is absolutely needed to ensure a successful SaaS application testing. The automated

validation of the functional and non-functional requirements of the SaaS application helps shorten the

release cycle of frequent SaaS application upgrades and releases. The data integration/ migration

pertaining to SaaS applications would also need thorough validation. The key to successful SaaS

testing is putting together the right test strategy, automating the tests for functional and non-

functional requirements and leveraging best practices that would help maximize the investments in

SaaS and in turn help the organization achieve the intended business outcome.