Where Value & Innovation co-exist Testing Automation CORE and TAM Road Map October , 2005 Raja Ramesh
Where Value & Innovation co-exist
Testing Automation
CORE and TAM Road MapOctober , 2005
Raja Ramesh
Where Value & Innovation co-exist
April 11, 2023 2
Agenda
Why do we need test automation? What can be test automated? Need for automation at SHPS Automation objectives Automation script architecture Automation MANAGER CORE automation plans TAM automation plans Challenges on hand Automation demo Discussion
Where Value & Innovation co-exist
April 11, 2023 3
Why do we need test automation?
Speed up testing to accelerate releases Allow testing to happen more frequently Reduce costs of testing by reducing manual labor Improve test coverage Ensure consistency Improve the reliability of testing Allow testing to be done by staff with less skill Define the testing process and reduce dependence on the few who know it Make testing more interesting Develop programming skills
Where Value & Innovation co-exist
April 11, 2023 4
What can be test automated?
Regression testing Tests that can run on every build are excellent candidates for automation
Configuration tests If testing required for different platforms, automation may help
Test bed setup Automating setup procedures will save time
Non-GUI testing It’s almost always easier to automate command line and API tests than it
is to automate GUIs
Build readiness QA time can be better utilized if unstable builds can be identified through
automation
Where Value & Innovation co-exist
April 11, 2023 5
Need for automation at SHPS
# Item CORE TAM
1 Total Applications 41 18 *
2 Test Cases 2006 2055
3 Hours for Manual Execution 171 Man Hours 740 Man Hours
4 Frequency of Regression Testing 3 to 4 time/release ** 1 time/release
5 Sanity testing (Man Hours) 2-3 3-4
6 Frequency of Sanity Testing Every Build Every Build
* TAM CSR is executed for 23 clients and TAM PPWEB executed for 18 client.** Few application are executed only once a release. These applications occupy 40-50 Man hours of CORE regression and 10-12 hours of TAM regression.
Where Value & Innovation co-exist
April 11, 2023 6
Need for automation SHPS cont…
Multiple regressions is not possible CORE takes 21 man-days for one iteration TAM takes 93 man-days for one iteration
Cost of labor Reasonable labor is being spent on manual testing
Limitation on aggressive schedules Short release cycles not possible Parallel releases not possible
No tool/mode for confirming build readiness Manual cursory testing 3-4 hours effort
Prone to skip test scenarios Due to repeated testing of test cases, it is possible for the tester to
overlook/assume correctness of some scenarios
Where Value & Innovation co-exist
April 11, 2023 7
Automation objectives
Reduce the regression testing time by at least 60-70% Provide quick sanity testing facility at onsite Improve the regression testing execution frequency Develop scripts with minimal maintenance time Provide facility for everyone to run automation scripts Provide facility to receive automation test results through email
Where Value & Innovation co-exist
Automation overview
Where Value & Innovation co-exist
April 11, 2023 9
Script architecture - Guidelines
External data source Generic code Flexibility Controllability Maintainability Operability Reporting ability
Where Value & Innovation co-exist
April 11, 2023 10
Script architecture – Baseline suite
Start
Driver Script
Get GUI FilePath / Exception
Data
Database
.INIFile
ApplicationScript
GenerateReport
Get Test Data
Database
Invoke theApplication
Navigate to theScreen
GUIVerify?
Functional TestCase execution
No
Yes
ChecksResults
Results
Each Test Case
Next
Database
Stop
GUI Verification
Where Value & Innovation co-exist
April 11, 2023 11
Script architecture – Participant WEB
Start
Load all Clients(Active)
STAGE 1: DB Module
Get Active Links forPartcipant Menu and
DirectorMenu
Each Client
SQADatabase
Insert Data
NextCORE DB
STAGE 2: GUI Module
Each Client
GenerateGUI Map file
SQADatabase
Path
Next
STAGE 3: Integrated Module
Each Client
Next
Load GUI Map
SQADatabase
Link Checker
Update
Generate ReportResult
(email, excel)
Stop
Where Value & Innovation co-exist
April 11, 2023 12
Script architecture – Cursory testing
Start
ExecuteDriver Script
For eachModul/App.
Launch theModule/App.
Get Test CasesExecute theParent TC
Pass?Execute rest of the Test Cases
Yes
Next
No
SQADatabase
Genearte Report
Report(Email, Excel) Stop
Where Value & Innovation co-exist
April 11, 2023 13
Automation MANAGER
What is Automation Manager A utility to ease the operation folks jobs Can be executed by staff with less testing skills No dependency on offshore SQA
ON-FLY configure Applications that needs to be tested Test cases that needs to be executed (single, multiple or a range) Test results communication (email, results in shared folder etc)
Where Value & Innovation co-exist
April 11, 2023 14
Automation MANAGER – process flow
Start
InitiateAutomation
Manager
Choose theApplication to
execute(Single / Multiple)
Choose the TestCases for the app.(Single / Multiple /
Range)
Enter emailaddress for
receiving testresults
SCOPE: User / Tester SCOPE: Automation Manager
LaunchPscExec(Remote)
Update
.INI file(Remote file)
SCOPE: WinRunner Server
TriggerPsExec
Batch file(Remote)
TriggerWinRunnerBatch file
Driver Script Path
LoadDriver Script
LoadApplication Script
RetreiveTest Data
SQADatabase
ExecuteTest Cases
Each Application
Result
Next
Generate Report
Report(email / Excel) Stop
Where Value & Innovation co-exist
April 11, 2023 15
Automation MANAGER - prototype
Continued…
Where Value & Innovation co-exist
April 11, 2023 16
Automation manager – prototype cont…
Where Value & Innovation co-exist
CORE Automation
Where Value & Innovation co-exist
April 11, 2023 18
Current automation status - CORE
# Application Total Cases Cases Automated
Hours for Manual Execution
Manual Hours with current Automation
1 CSR 173+30 48 10 7
2 HR Access 224+143 59 12 8
3 Case Setup * 119+126 103 22 NA
* For Case Setup, scripting is completed. Test data needs to be added.
1 Total regression test cases 2006*
2 Total test cases automated 107 **
3 Effort required to perform total regression manually (Man hours) 170
4 Effort required to perform regression with current automation (Man hours) 163
* Includes functional & end-to-end cases for all the 41 CORE applications.
** Does not include 103 case setup test cases for which, test data still needs to be added.
Where Value & Innovation co-exist
April 11, 2023 19
Automation scope - CORE
# Regression Testing Cursory Testing
1 CSR Case Setup
2 HR Access CSR
3 Case Setup HR Access
4 Participant Web * Participant Web
5 Load Passwords CallDoc
6 CallDoc Template Setup
7 Knowledge Base Database checks
8 STD Load
9 AddrPBSI
10 Load PBSI
11 LoadEmailAddress
12 Template Setup
13 Survey Setup
Where Value & Innovation co-exist
April 11, 2023 20
Proposed automation road map - CORE
VII
[TBD]
VI
[End of Nov 05]
V
[End of Nov 05]
IV
[TBD]
III
[End of Oct 05]
II
[Mid of Nov 05]
PHASE I
- Sanity Test Automation
- Save 3 hours per buildI
[ Mid of Oct 05]
PHASE II
- Case Setup & Template Setup
- Save 18 hours per build
PHASE III
- Participant WEB
- Save 07 hours per build
PHASE IV
- Client Specific
- Save XX hours per build
PHASE V
- End to End cases Automation
- Save 21 hours per build
PHASE VI
- CallDoc, KB, STD Load,
- Load Password, Address PBSI,
- Load Email Address, Load PBSI,
- Survey Setup, Template Setup
- Save 27 hours per build
PHASE VII
- Test Cases Version Control
- Test Data Automation
Where Value & Innovation co-exist
April 11, 2023 21
Road map in detail – CORE
* Often, multiple phases are executed in parallel. Team is already working on Phase2, Phase3 & Phase6
# Phase Applications Cases Included
Expected Hours
Start Date End Date
1 Phase1 Cursory tests 30-40 420 9/26/05 10/12/05
2 Phase2 Case Setup 298 192 10/13/05 11/15/05
3 Phase3 Participant Web 160 600 9/12/05 10/31/05
4 Phase4 Client Specific tests TBI TBI TBD TBD
5 Phase5 End-to-end testing 200 480 10/13/05 11/23/05
6 Phase6 CallDoc, KB, STD Load, Load Password, Address PBSI, Load Email Address, Load PBSI, Survey Setup, Template Setup
220 608 9/19/05 11/28/05
7 Phase7 Miscellaneous – version control & test data reconstruction
TBI TBI TBD TBD
Where Value & Innovation co-exist
April 11, 2023 22
SUMMARY: CORE automation
Total regression cases (all applications functional & end-to-end) 2006
Total time required if regression is done manually (Man Hours) 170
# Phase Total Cases Automated
Total Hours Required Automate
Manual efforts still required to complete regression (Man Hours)
1 Before Phase1
107 NA 163
2 Phase1 30-40 420 NA
3 Phase2 298 192 145
4 Phase3 160 600 138
5 Phase4 TBI TBI NA
6 Phase5 200 480 117
7 Phase6 220 608 90 *
8 Phase7 TBI TBI NA
* There are about 30 applications that are executed only once a release. These applications occupy 40-50 hours of total regression time.
Where Value & Innovation co-exist
TAM Automation
Where Value & Innovation co-exist
April 11, 2023 24
Current automation status - TAM
# Application Total Cases
Cases Automated
Cases still need to be Automated
Hours for Manual Execution
Manual Hours with current Automation
1 TAM CSR 537 395 54 342 123.2
2 Participant Web 362 342 10 188 31
4 TAM Case Setup 705 00 705 11 NA
* CSR cases are executed for 23 clients and case setup cases are executed for 18 clients
1 Total regression test cases 2055*
2 Total test cases automated 737
3 Effort required to perform total regression manually (Man hours) 740
4 Effort required to perform regression with current automation (Man hours) 364
* Includes functional & end-to-end cases for all the 41 CORE applications.
Where Value & Innovation co-exist
April 11, 2023 25
Automation scope – TAM
# Cursory Testing Regression Testing
1 Core Case Setup 11 STD Daily TAM CSR
2 TAM Case Setup 12 STD Daily Cigna TAM Case setup
3 TAM CSR 13 STD Daily CNA TAM Participant Web
4 TAM Batch 14 STD Daily LEA
5 Auto Close 15 Intel TAM Out
6 Calculations 16 Participant Web
7 Email Batch 17 HR Access
8 Letter Batch
9 Printed Letters
10 TAM Termination Batch
Where Value & Innovation co-exist
April 11, 2023 26
Proposed automation road map - TAM
V
IV
III
II
I
Current
Current Status
- TAM CSR – 70% case automated
- Participant WEB – 95% cases automated
- Saving 376 hours per build
PHASE I
- Sanity Test Automation
- Save 3-4 hours per build
PHASE II
- TAM Case Setup
- Save 13 hours per build
PHASE III
- End to End cases Automation
- Save 4 hours per build
PHASE IV
-Test Data Automation
-Save script maintenance time
PHASE V
- Test Cases Version Control
- Save 40-50 hours of test result review time per build
Where Value & Innovation co-exist
April 11, 2023 27
Road map in detail – TAM
# Phase Applications Cases Included
Expected Hours
Start Date End Date
1 Phase1 Cursory tests 40-50 250
2 Phase2 CSR, Case Setup & HR Access
769 416
3 Phase3 End-to-end testing 32 150-170
4 Phase4 Test data reconstruction All TBD
5 Phase5 Miscellaneous TBI TBD
* Often, multiple phases are executed in parallel.
Where Value & Innovation co-exist
April 11, 2023 28
SUMMARY: TAM automation
Total regression cases (all applications functional & end-to-end) 2055
Total time required if regression is done manually (Man Hours) 740
# Phase Total Cases Automated
Total Hours Required Automate
Manual efforts still required to complete regression (Man Hours)
1 Before Phase1
737 NA 364
2 Phase1 40-50 250 NA
3 Phase2 769 416 351
4 Phase3 32 150-170 347 *
5 Phase4 TBI TBD NA
6 Phase5 TBI TBD 300-310
* On average, 200 man hours are spent on testing letters & emails application, which can not be automated. Also, there are about 10 applications that are executed only once a while. These applications occupy 10-12 hours of total regression time.
Where Value & Innovation co-exist
April 11, 2023 29
Challenges on hand
Research Extensive research is required to implement Phase5 & Phase6 (CORE)
Automatic Test Data Writing queries to pick test data from database requires a lot of additional study on the
system and tool.
Support from other teams Phase4 (CORE) requires support from Business development and Tech OPS teams
Prioritization of application automation SQA team needs SHPS support to finalize the list of applications for sanity testing
WinRunner instances Require multiple instances of WinRunner so that multiple tests can be executed
simultaneously
QuickTest Pro tool availability QuickTest Pro, a new regression testing tool is required to automate Knowledge Base
Where Value & Innovation co-exist
Automation DEMO
Where Value & Innovation co-exist
Discussion
Where Value & Innovation co-exist
Thank you!