Automating End-to-End Business Scenario Testing

- 1 -

Automating End To End Scenarios

Presenters:Sandra Alequin (Senior Manager, Allstate Insurance)

Monika Mehrotra (Senior Manager, lnfosys Ltd)

- 2 -

Enterprise testing (ETest) strategy was established in pursuit of a strategic objective: to improve agency and customer experience with a goal of minimizing production problems and reducing the risk due to major application releases on critical business functions.

ETest strategy implementation included establishing new processes, tools, team and environment. ETest consists of 200+ applications with over 600+ backend services and equally high number of interacting databases.

33 key business functions, which are highly integrated, are tested end-to-end prior to being put in production under this strategy.

Enterprise Testing (ETest) Strategy at Allstate

- 3 -

Executive Summary

Java Applications

.NET Applications

Siebel Applications

Mobile Applications

Open Web Applications

Mainframe

Presentation Layer Middle-Tier Data Layer

Mainframe

Web Servers

Application Servers

LDAP

MQ

TIBCO

SOAP

REST

JMS

EJBs

JDBC

Store Procedure

FTP

Mobile

Web Browsers

Applications

Enterprise Testing Environment – An Overview

Dat

abas

eD

atab

ase

Dat

abas

e

- 4 -

Over 200+ application from a mix of legacy (AS400, Mainframes) and new technologies (.NET, Java)

600+ services on different technologies like SOAP, REST etc

Unresponsive Backend Services can cause failures in tests and hence need constant pinging to keep them alive

Time consuming Defect Root Cause Analysis due to large number of systems and integrations

Deactivation of Test IDs post Data Refresh activities

Complexities in Enterprise Testing Environment

- 5 -

Validation points introduced across different layers for deeper and wider test coverage

5

Executive Summary

Java Applications

.NET Applications

Siebel Applications

Mobile Applications

Open Web Applications

Mainframe

Dat

abas

e

Presentation Layer Middle-Tier Data Layer

Mainframe

Web Servers

Application Servers

LDAP

MQ

TIBCO

SOAP

REST

JMS

EJBs

JDBC

Store Procedure

FTP

Mobile

Web Browsers

Applications

Existing validation points Newly introduced validation points

Solution to address E-Test Environment challenges

Dat

abas

eD

atab

ase

- 6 -

Executive Summary

• Implementation of trouble shooting at service layer to reduce effort spent root cause analysis

• Reduction in time spent on debugging and enable faster fixes for defects

• Validation of middle tier components that otherwise are not tested as part of GUI testing scenarios

• Deeper and wider coverage introduced into architecture with middle tier validation

• Validated integration and interaction points ensuring quality in systems integration

• Data navigating from one end to another in network is tracked for quality and reliability

Smoother Integration Better Coverage Defect Isolationa b c

Users - Presentation layerHeadless clients

consuming services

Data Access Logic Data Utilities Data Sources

Presentation – Middle Tier Integration

Middle Tier – Data Layer Integration

Service Service Service Service Service Service Serviceb

a

a

c

c a

b

Addition of Middle Tier Validation in E-Test

- 7 -

Complexity, size and time to execute E-Test regression suite warranted the need to automate the regression suite

An assessment on then E-Test processes were performed and an Integrated (frontend and backend) automation solution was arrived at to target improvement in efficiency and effectiveness of E-Test regression suite execution by giving deeper and wider test coverage.

De-risked and cost effective 2-Phased approach was adopted to build out the integrated automation.

Phase 1 was performed for feasibility analysis and devise solution for automation requirements.

Phase 2 was implementing the solution and rollout

Necessity for Integrated Test Automation Solution

- 8 -

QTP Automation Framework

Service Validation Framework

Frontend Layer

UI Components

Presentation Layer Logic

Backend Service Layer

Service Orchestration

Services

Backend Data Layer

Data Access Components

Data utilities

Data Validation using CA LISA

E Test Environment

Leverage LISA for early validation at service/data layer

Validating UI layer on a more stable

Backend

Integrated Automation Solution

HP Quality Center

HP Quick Test Professional

CA LISA

CA LISA

Backend (Service and Data layer) validation will be automated with CA LISA and Frontend (UI layer) validation will be automated with HP QTP. The solution will be orchestrated using HP QC and provides flexibility to validate layers in integrated or silo mode.

- 9 -

Switch of test machine is required to continue execution

Stop for Batch CycleResume after Batch Cycle

Data needs to be captured and transferred between the test scripts/test sets

LISA Scripts integration with QTP and HPQC for backend scripts execution

Typical ETest End-to-End Scenario for Automation? – Questions to be answered and solutions to be proven out

Customer does inquiry and requests for a Quote

Mobile Search Web Search

Agent contacts customer and the quote is converted to a request for new business

New business is received and reviewed by underwritersHome Office Staff verifications Underwriters

Underwriter provides approved , New business documents generated and Customer access documents

Document Management System

Customer Access Endorsement confirmationPolicy EndorsementRegistration

Renewal Validations

Validate Renewals processing after payments

Policy Endorsements, Renewals and billing

Endorsement Documents

Renewal of Policies

Renewal Billing

Renewal Documents:

Endorsement processing Renewals Payments

Sample ETest Scenario

??

?

?

Exclusive Agent Applications

Independent Agent Applications Home Office Staff

- 10 -

Tests were run on specific workstations to mimic customer and user experience. So how to switch test machines to continue automated execution?

How to handle the execution of a single test case across multiple job batches?

How would the data required for execution across batch of same test case maintained throughout the execution of the case, which may span across days?

How to integrate HPQC (Test Management), HPQTP (UI Automation) and CA LISA (Service layer validation) tools to perform an integrated test run?

Automation Challenges in ETest End-to-End Scenarios

- 11 -

Test Automation challenges addressed

Data Extraction & Persistence

Multiple Machines Execution

Execution Inline with Daily Batch cycles

LISA Scripts integration with QTP and HPQC

1 2 3 4

Customer does inquiry and requests for a Quote

Mobile Search Web Search

Agent contacts customer and the quote is converted to a request for new business

New business is received and reviewed by underwritersHome Office Staff verifications Underwriters

Underwriter provides approved , New business documents generated and Customer access documents

Document Management System

Customer Access Endorsement confirmationPolicy EndorsementRegistration

Renewal Validations

Validate Renewals processing after payments

Policy Endorsements, Renewals and billing

Endorsement Documents

Renewal of Policies

Renewal Billing

Renewal Documents:

Endorsement processing Renewals Payments

Sample ETest Scenario

21

3

4

Exclusive Agent Applications

Independent Agent Applications Home Office Staff

- 12 -12

# 1-Data Extraction & Persistence Data Persistence in XML Files

Dat

a Pe

rsis

tenc

e a

cros

s m

achi

nes

Dat

a Pe

rsis

tenc

e

acro

ss b

atch

runs

Customer Inquiry, Quote & New Business

Customer Machine: Customer Inquiry and

Begins Property Quote: (Mobile and Web)

Agent Machine: New Business

Purchase

Under Writer Machine; New

Business

Agent Machine: Endorsement : Agent

Customer Machine: Customer: Login to

“Customer Account", Register

Batch 1: Customer Inquiry and Begins Property Quote: (Web)

Batch 2: New Business Purchase: Agent

Batch 3: New Business : Agent

Data is getting read by different machines from XML

Reference # is persisted in XML and used by script for Policy # generation

1

1Policy # is persisted used by script and later used other scripts2

2

1 2

•Data is persisted in Shared LAN drive which is accessible by all machines•Data is persisted in form of XMLs for each test case.•XPATH Queries are employed for faster data retrieval

- 13 -13

Test Execution Dependency- Across Batches

Customer Inquiry, Quote & New BusinessCustomer Agent Search :

(Mobile and Web) Customer creates Quote

Customer Inquiry, Quote & New Business

Manage Agency by Agent Purchases quote and email validation

Validation of New Business

Batch 2

• Automated Script generates Execution Control XML for each Test Case• Test Script execution status with other details are captured in XML structure • Execution sets Test Set_Status “Y” in case of Pass “N” for failure• Next Test Script reads XML data and determines if execution is required or not

Invoke Next Batch test case if

Status=“Y”

Stop Execution for test case if

Status=“N”STOP

Determine Execution is required/not from

XML

Batch 1

# 2 - Execution Inline with Daily Batch cyclesCross Test Set Dependency with XMLs

- 14 -14

1A

1B

1C

E Test Release 1

HP QC Test Lab Folders at Scenario level

HP QC Test Lab Folder at Release level

Batch1

Batch2

BatchN…

Batch1

Batch2

BatchN…

Batch1

Batch2

BatchN…

HP QC Test Sets at Batch run levels

Each Test Set contains specific test scripts to be run as part of E Test Case

HPQC identifies available machine in the host group and triggers execution on the available machine within the group. Upon execution completion, test results are reported back to quality center and execution is moved to next test script

ACT MBTIVW UWR

Claims

Machine Groups

HP QC Test Cases

Batch 1 for 1A contains scripts to be executed before batch cycle process runs

#3 - Multiple Machines ExecutionHosting in HPQC

- 15 -15

# 4 - CA LISA integration with HPQTP & HPQC

CA LISA

- 16 -16

HP Quality Center

Mainframe Web Java

Enterprise Automation Components(Reusable Functions, Keyword functions, Framework Driver Engine)

Platforms Under Test

Automation Framework

Remote Machines

Testers

Java Specific Components

Web Specific Components

Mainframe Specific Components

Framework to handle Application Diversity

- 17 -

Test Case Modeling

17

Application Workflow

Payment

Work flow Data

Data1

DataWork flow

Work flow

Data

Data2

Data3

Data1

Data2

Data3

Data1

Data2

Data3

Modularizing the application workflow by creating multiple reusable Business Process Components (BPCs)

Workflow created in excel using action keywords

Multiple data sets can be mapped to the same workflow to easily create different data oriented test cases

Customer Registration Agent Search

Diagrammatic representation of test case modeling

Test Case modeling with the framework

- 18 -

Test Case Modeling

18

ETest Implementation Approach

Assessment of ETest

Environment and Test

Scenarios

ETest Automation Feasibility Analysis

ETest SOA Validation

Implementation

ETest SOA validation

rollout

ETest UI Automation Feasibility Analysis

ETest Automation

Unique requirements

identifiedProof Of

Concept for ETest

Automation

Complete Automation

Implementation in ETest

Rollout of Automation Testing for

ETest

- 19 -

Executive Summary

Current State Post Implementation Benefits

Quality

• Test steps scoped based on available effort

• End to End Business scenario validated in base state

• Volume test steps execution was increased by 3 times

• End to End Business scenario implemented in multiple states

• 200% improvement across Tests and ROCs

• Improved coverage at backend layer• Increased and consistent validation• Multiple Regression Runs in current

window

Time• 3-week window

to execute E2E tests

• 1-week window to execute E2E tests

• 33% reduction in the execution window

• Creating window for more execution• Reduced turn around time on defect

resolution

Cost• High manual

effort to rerun failed scenarios

• Overall team is reduced by 40%

• 30% reduction in execution cost• Early validation• Reduced issues and effort during End

to End• Lessen manual effort with automation

ETest Automation - Post Implementation Analysis

- 20 -

Executive SummaryAdditional Benefits of Integrated Automation Solution

Predictability Validation at backend layer resulting in early defect detection Post implementation UI execution happens on a validated backend

layer

Portability Solution can be deployed on E-Test or other environments with minimal effort

Coverage Improve coverage by introducing more validations at all layers of validation

Modularity & Scalability

Deploy solution in packages to enable selective run Ability to run automation scripts for specific scenario category

and/or applications.

- 21 -

Questions & Answers

- 22 -

Thank you.