8/12/2019 Automation Process QAAC
1/23
Test Automation Success:
Choosing the Right People & Process
Kiran Pyneni, Automation Manager
Aetna, Inc.
8/12/2019 Automation Process QAAC
2/23
Agenda
Automation Lifecycle
Automation Team
Metrics Of Our Success
Questions
Automation Coverage, Efforts Automated, Utilization & Cost Avoidance
Company Background
Strategy Team
Delivery Team
Benefits of Teams Strategy
Benefits of Process Integration
Our Journey
Automation Objectives
8/12/2019 Automation Process QAAC
3/23
Aetna Background
Aetna is one of the nation's leading diversified health care benefits
companies, serving members with information and resources to help themmake better informed decisions about their health care.
Products and management Programs:
Aetna offers a broad range of insurance and employee benefits products.
The first national, full-service health insurer to offer a consumer-directed healthplan, Aetna continues to lead the way with its Aetna HealthFund2 line of products,including HSA, HRA and RRA options.
Aetna offers a wide array of programs and services that help control risingemployee benefits costs while striving to improve the quality of health care, such ascase management; disease and patient safety programs; integrated medical,dental, pharmaceutical, behavioral health and disability information.
Aetna provides members with access to convenient tools and easy-to-understandinformation that can help them make better-informed decisions about their healthand financial wellbeing.
3
8/12/2019 Automation Process QAAC
4/23
Automation Team
Improv es eff ic iencies and qual i ty at our o rganizat ion by automating
consis tent and repeatable manual tasks us ing industry s tandard
techno logies / tools / framewo rks.
Automation
TeamAssessments &
DesignsReports & Metrics
Standards &
Best Practices
Automation
Development &
Maintenance
8/12/2019 Automation Process QAAC
5/23
Automation Objectives
Automation
Objectives
Automating Right Tasks
Sustaining and Growing
Automation Suite
Utilization of existing
automation
8/12/2019 Automation Process QAAC
6/23
Our Journey
Benefits Easy identification of
automation opportunities
Higher collaboration betweenapplication & automation teams
Challenges Automation development has
become expensive
Usage of different tools &solutions
Standards & Best practices arenot shared across applications
Benefits Standardization of automation
tool set at enterprise level
Sharing of coding best practices,templates and standards
Cost of automation developmentis lowered
Challenges Slower automation coverage
progression
Lower automated scriptsutilization
Funding Challenges
Decentralized Approach Centralized Approach
8/12/2019 Automation Process QAAC
7/23
Our Journey Contd.
Challenges & Solutions
Challenges
Identification of automation opportunities
Prioritization of automation projects
based on ROI
Cost of automation
Implementation of optimized designs
Solution
Structuring automation as strategy &
delivery team
Challenges
Early completion of maintenancefor
automation scripts
Funding for automation projects
Collaboration between automation &business teams
Solution
Create automation life cycle process and
integrate with company's overall software
development life cycle
8/12/2019 Automation Process QAAC
8/23
Automation Team Structure
Automation Team
Automation
Strategy Team
Automation
Delivery Team
8/12/2019 Automation Process QAAC
9/23
Automation Strategy Team
Definition: To provide automation strategic direction and identify
potential new automation opportunities
Characteristics: Project management experience
Various automation frameworks/tools knowledge
Ability to manage multiple projects in parallel
Very strong written and verbal communication skills
Very strong leadership and analytical skills
Activities: Process & Best Practices
Assessment & Strategies
Design Solutions & Techniques
Estimations & Planning
ROI Cost/Benefit Analysis
Reports & Metrics
Tool Technical Support & Analysis
8/12/2019 Automation Process QAAC
10/23
Automation Delivery Team
Definition: To develop, deliver, and maintain automation solutions
Characteristics: Understanding of programming principles and best
practices
Knowledge of various automation tools
Knowledge of multiple programming languages
Strong communication skills
Activities: Scripts Development
Scripts Maintenance
Coding Standards
Scripts Planning & Execution
Day to Day interactions w/End User
8/12/2019 Automation Process QAAC
11/23
Benefits of Teams Strategy
Benefits
Easy identification of automation opportunities
Prioritization of automation projects based on ROI
Improved automation design
Optimized the cost of automation
Clearly defined roles & responsibilities
Better communication process
Optimized automation project planning
8/12/2019 Automation Process QAAC
12/23
Preliminary Estimations
Automation Test Strategy
Business Case - ROI
Selected automation tool
Detailed Estimations
Go, No Go
Automation Project Plan
Framework documentation
Re-usable components,
Portability, test data grid
design, etc.
Business Process Test
(BPT) components
Automation Scripts
Test Data
Tested automation
scripts
Automated test
execution results
Lessons Learned
Sign Off
G1 G2 G3
Automation Evaluation
Understand functionality of
the project / application
Understand testing
processes
Understand target
environments
Understand test data
details
Verify automation tool
compatibility
Identify Automation
opportunities
Phases
Ac
tivities
Deliverables
Automation Preparation
Identify the resourcesneeded.
Understand the
requirements with the helpapplication demos.
Create automation
framework / design
Identify timelines and
milestones
Training required for
automation tools
Tool installation if required
Access & Environment
Setup
Automation Generation
Create Business Process
Test (BPT) components
Parameterize test data in
BPT components
Review components for
quality and correctness
String the BPT
components together to
create a test script
Develop automation scripts
for BPT components
Identify test data to
conduct unit & acceptance
testing
Unit testing of
automation
Deliver automation to
application team
Execute automation by
application team
Validation of ROI
Retrospection
Sign off
Execution & Acceptance
Automation Lifecycle
8/12/2019 Automation Process QAAC
13/23
Team Collaboration
AutomationEvaluation
Automation
Preparation
Automation
Generation
Execution
& Acceptance
Automation
Strategy Team
Automation
Delivery Team
8/12/2019 Automation Process QAAC
14/23
Automation ROI Calculator
8/12/2019 Automation Process QAAC
15/23
Automation ROI Calculator
I t ti f A t ti Lif l &
8/12/2019 Automation Process QAAC
16/23
Integration of Automation Lifecycle &
Software Development Lifecycle
Automation
Evaluation Feasibility/Analysis
Automation
Preparation Design
AutomationGeneration Implementation
Execution &
Acceptance
Testing &
Maintenance
8/12/2019 Automation Process QAAC
17/23
Benefits of Process Integration
Benefits
Improved understanding on automation process across
the entire organization
Better collaboration between business and automationteams
Funding challenges are removed
Improved on-time delivery of automation scripts
Helped to promote the automation across entireorganization
Formal escalation process developed
Initialized the creation of automation reports & metrics
Automation Coverage
8/12/2019 Automation Process QAAC
18/23
Automation Coverage
Regression Testing
42%Q1-2009
45%Q2-2009
47%Q3-2009
52%Q4-2009
35%Year-2008
http://www.gnurf.net/v3/wp-content/uploads/2009/04/041-vintage-racer.pnghttp://www.gnurf.net/v3/wp-content/uploads/2009/04/041-vintage-racer.pnghttp://www.gnurf.net/v3/wp-content/uploads/2009/04/041-vintage-racer.pnghttp://www.gnurf.net/v3/wp-content/uploads/2009/04/041-vintage-racer.pnghttp://www.gnurf.net/v3/wp-content/uploads/2009/04/041-vintage-racer.png8/12/2019 Automation Process QAAC
19/23
Manual Testing Efforts Automated
7,047 hrs
8,570 hrs
10,200 hrs
11,750 hrs
13,803 hrs
Year As of 2008 As of 2009 2009 only efforts
Total Efforts Automated 7047 hrs 13803 hrs 6756 hrs
Year-2008 Q1-2009 Q2-2009 Q3-2009 Q4-2009
8/12/2019 Automation Process QAAC
20/23
Utilization & Cost Avoidance
Q1-2009 Q2-2009 Q3-2009 Q4-2009
Cost Avoidance
7,200 hrs
10,810 hrs
9,618 hrs 9,867 hrs
Q1-2009 Q2-2009 Q3-2009 Q4-2009
Utilization
45%
Year-2008
87%
150% 134% 130%
8/12/2019 Automation Process QAAC
21/23
Savings Vs. Cost Avoidance
Cost Savings
Efforts difference between manual and automation testing
Manual (Not automation) execution frequency
Produced by automation strategy team after assessment
Automation ROI Calculator
Cost Avoidance
Test coverage difference between manual & automation testing
Efforts difference between manual and automation testing
Automation execution frequency
Generated from Quality Center after release implementation
Helped to improve the credibility of automation metrics significantly
8/12/2019 Automation Process QAAC
22/23
Automation MetricsBest Practices
Automation is a way of Testing and process followed for manual testing
can be applied
Best Practices followed in Test Management ToolQuality Center
Define automation work flow Regression Testing identifier
Automation Assessment status
Automation Development status Store automation inventory
Manage automated test scripts execution
8/12/2019 Automation Process QAAC
23/23
Questions