Secrets of Successful Automation Projects Presented by Paul M. Grossman Software Quality Assurance Analyst QTP Automation Framework Designer Secrets of.

Secrets of Successful

Automation Projects

Presented by

Paul M. Grossman

Software Quality Assurance AnalystQTP Automation Framework Designer

Chicago Quality Assurance Assocation

Portfolio of Custom Software Software Development: .Net, Java, C#

eCommerce (Oracle ATG)

Mobile Applications: iOS, Android, Windows Mobile/WP7, Blackberry

Quality Assurance Testing LoadRunner, QTP, HP ALMS

Custom Tool Development and Frameworks

Performance Testing

Mobile Testing: iOS, Android, Windows Mobile, Tablets

Developed Applications in the following sectors Games & Entertainment

Healthcare

Social Networking

4tegroup.com

Paul M. Grossman

11 years automation experience with HP Tools HP Quality Center 9.0 Certified Product

Consultant HP Quick Test Professional 9.0 Certified Product

Specialist Beta Tester of QTP 9.5 – 11.5 Speaker at Mercury/HP Events Became Automation Engineer due to Perfect

Timing! Manager caught me practicing magic tricks while waiting for system to reboot.

Session Goals

Key success factors in test automation today

Demonstrating test automation ROI Effective framework design techniques Successful script design for Agile

environments

Automation Challenges

Lack of communication of changes/details to testing group

Lack of understanding of should be automated

Lack of variation from plan in test scripts

Insufficient standards

Insufficient Analysis & Design

“Silver bullet syndrome” (automated testing is the “silver bullet” that will end all testing problems on a project)

Lack of change control

Case Study

Automation Project Failure at a Medical Device Manufacturer

Year 1: QA Manager wants automated testing. Buys WinRunner, accepts offer at new company.

Year 2: New QA Manager wants automated testing. Hired Lead Developer (C++), given an Assistant (VBscript).

Goal for automation: 100% coverage.

Year 3: Lead Developer moves to new department. Assistant told automation is high-priority, spare-time project.

Year 4: QA Manager moves to new division Assistant retires.

Results: 0 Defects found. Negative Return On Investment.

Unsuspecting manual tester assigned to automation team. New Division Manager expresses interest in automation.

Manager promotes me to Team

Lead… of defunct Automation

Project.

Top 5 Project Pitfalls

1. Lack of research & planning2. Lack of apparent progress3. Lack of experienced resources4. Not a full time project5. No buy-in

100% automation goal resulted in development in all product areas

20% development in all areas with nothing to show

Project handed over to assistant with no product training

Misperception: FT not needed because automation by it’s definition is automatic, right?

Testers & developers fueled rumors of tool’s stability

Mitigation Strategies

1. Lack of research & planning

2. Lack of apparent progress

3. Lack of experienced resources

4. Not a full time project

5. No buy-in

10% Automation goal

POC Pilot Project

Training

Full-time commitment

Repeat defects manually on a non-automated system

Tips And TricksAvoiding "100% Automation"

Trap 90% Automation can be

approached but it is generally not cost effective

New releases will have more functionality with constantly decreasing code coverage

A percentage of the code could be unreachable code, such as uncalled functions

Pick one: Functionality or Browsers Do the math: You won't have time for everything!

20 - 40% Automation makes a significant impact on the testing

process.

Math: App Developers outnumber your

team?Do they have 6-18 Months head start?

Do they have more than one tool?

Case Study

The Pilot Project

Automate the Acceptance Test (“Smoke Tests”)— Short tests, but too wide of coverage

Commitment to six weeks of full-time development— Became four weeks, 10 days of manual testing

Used “Record & Playback” to create the entire test script— Overwhelming maintenance

Demonstration to follow at the end of the project— Management convinced they purchased the right tool— Asked for training on Winrunner

Request to get training approved!

– Lessons Learned

Case Study

Functional Decomposition Crafted functions from Demo Script to create “TimeSaver”

utilities

Wrote discrete state change functions: TRUE, Worked! FALSE, Uh-Oh!

Modular functions made maintenance manageable

Automated the common interfaces of two additional products

Proposed automating easy features first

– ShowStopper #1: Memory Leak. First 13-hours of endurance test.

– Developers stop blaming tool, now ask for ad-hoc scriptsRequest to attend Mercury World Conference Approved!

Estimated Return On Investment

“Best Bang For The Buck”: Project TimeSavers

Automate tedious manual tasks— Create 64 Annotations - 30 Seconds— Perform 128 End to End Procedures - 30 Minute Drive Space

Test— Register 128 New Patients - 30 Minute Boundary Test

These tests were small, little maintenance Gets testers on your side, reduces their “pain” Scripts were easy to change into reusable

functions

First Quantifiable Return On Investment: $40 (1 Hour)

Final estimated ROI for the company: $2 Million

Estimated Return On Investment

Manual Tester: $40 Avg. hourly

wage, including overtime and

benefits.

Estimated Return On Investment

ROI Formula: Estimate Equivalent Cost of Automation

How much would it cost to hire someone to do this work manually?

Estimated hourly wage (Plus OT & Benefits): $40 Per Hour Average

Determine hours of runtime- 10 hour runs, 5 nights a week 10 x 5 = 50 Hours per system- on 3 systems 50 x 3 = 150 Hours per week

----------------------- 150 x $40 = $6,000

Extrapolated to the end of the year X 52 Weeks -----------------------

$312,000Four years of run time: $1,248,000

Estimated Return On Investment

What would some of the worst defects found by automation have cost the company had they gotten to

the field? Case Study Examples

—ShowStopper #2: Transient Boot Up Failure – Replicated with automation and high end UPS

—ShowStopper #3: Data Transfer Problem– Duplicated issue within five days. – Reason: Related to Network Load limitation.

We pegged the ROI of the worst detected defects by estimating the cost of a single lost customer sale: $100,000

Potential revenue loss from 7 ShowStoppers: $700,000

Plus 1.2 Million not paid to a manual tester: 1.9 Million Est. ROI

Used Functional Test Tool to uncover Network Load

issue?

Estimated Return On Investment

That was then…this is nowWhat would some of the worst defects found by automation have cost your company had they

gotten to the field?

—ShowStopper #1: Credit cards fail to authenticate?– Store is closed world wide.

—ShowStopper #2: Search path leads to infinite loop?– Customers frustrated with site jump to competitor.

—ShowStopper #3: 12-second+ response time?– Customers abandon order– Is customer support being effected?

– Slows down automation code development and debugging

– Slows down development programmers– Slows down manual testers

What if it's Dev, QA or Staging Environment?

Let's take a break!

When we come back…

Tips And Tricks Dealing with Failure What is a Framework anyway?

What is a Framework? Framework Options

Capture/replay

Function-based

Data-driven

Keyword-driven

Exhaustive

Random

Lexical Analyzer

Hybrid

What is a Framework?

Driver Loads & Executes Multiple Tests Facilitates Result Reporting

Object Repository or Identification Identifies objects by Class and Property Values Uses Regular Expressions for dynamic values

Data Pre-Defined Generated

Recovery Mechanism Browser closes Object invalid Error generated

A framework is a set of techniques:

• Reduce Maintenance

• Abstract Complexity

• Extend Coverage

Tips And Tricks Initial Project Design

Return To Home State Functionality — Get back to the starting point from anywhere in the application, even if it is not

running.

Navigation and Population— Low level custom actions to Click buttons, Open menus, Select item from lists.

getValidObject (sClass, sName) — A single function that returns a valid object reference from many close

matches. — Object exists (height, width & Abs_x > 0), is enabled, above underlay, closest

to text. — Uses Descriptive Programming or xPath to identify object

Tokens— Stores reusable values, such as an Order Number— Can return intrinsic values like Today's Date.

Assume Failure, Prove Success— Use a global variable & Return Codes for overall success. This creates a single

exit point for script execution.

Modify, Recover, Repeat

Tips And TricksModular Design

Functions… but No Subs— Don’t waste time deciding which to use— Subroutines have no built in functionality for return codes— Sub FileExists (strFile, rc) ‘ Return Code Workaround

All Functions need Return Codes— FileExists = TRUE— IF FileExists(strFile) THEN ProcessFile (strFile)

Use Option Explicit in VBscript — Reduces errors due to variable misspellings

Three Level Keyword Framework— Components (Param1, Param2)

o Keywords (Param1, Param2)All other supporting functions

Enter_CC "Declined"

getValidObjectCLICK "Ok"

Test Scenario# 3

Purchase Order With Discount Code and Pay with Declined Credit Card

Tips And TricksGlobal Variables and

Dictionaries

Minimize global variables:—gloObject: Reference set by

GetValidObject

Use a Dictionary Object—Global—No Naming Convention—Stores Object References—Can be Persistent!—Case Sensitive Key Names

dicFrameworkKey Vaue

"AlreadyFailed"

False

"NextMonth" "June"

"ObjectClass" "WebButton"

"gloObject" [OK Button]

Tips And TricksDealing with Failure

dicFramework("AlreadyFailed") = FALSE

Function CLICK (sButtonName)

If dicFramework("AlreadyFailed") = TRUE Then

Exit Function

End If

blnFound = getValidObject "WebButton", sButtonName 'Sets gloObject

If blnFound = TRUE then

gloObject.Click

Else

dicFramework("AlreadyFailed") = TRUE

End If

End Function

That's an odd naming convention?"glo" = "Global Object

Tips And TricksDealing with Failure

dicFramework("AlreadyFailed") = FALSE

Function CLICK (sButtonName)

…

If blnFound = TRUE then

On Error Resume Next

gloObject.Click

reportErr ("Click")

On Error Goto 0

Else

dicFramework("AlreadyFailed") = TRUE

End If

…

End Function

Function reportErr (sFunc)

If err.Number <> 0 Then

Report err.Description & " in " & sFunc

End If

End Function

Thinking Outside The Box getValidObject (sClass, sText)

getValidObject ("Link", "Login") 'Sets gloObject

Three Attempts

— Descriptive Programming (Exact Match)– oBrowser.oPage.Link("innerhtml:=(.*)?Login(.*)?")– False? Add "Index:=0" - True if there multiple objects

— Search Object Collection (First Valid Match)– oDesc("class") = "Link" : oDesc("Name") = ".*Login.*"– Set oLinksCollecton =

oBrowser.oPage.ChildObjects(oDesc)

— Switch Class – oBrowser.oPage.WebButton("innertext:=Login")– oBrowser.oPage.WebElement("innertext:=Login")

Thinking Outside The Box getValidObject (sClass, sText)

getValidObject ("Link", "Login") 'Sets gloObject

Validation – Is this object Visible?– Width > 0– Height > 0– Abs_x > 0– Source_index: Appears over an underling

object

This is not the List object you are looking for—List contains the string we want to select

Agile environments

Version Control—Not vulnerable to local system crash—Code Change Comparison—Merge Changes—Revert Changes

Agile environments

Continuous Integration with Jenkins—Run scripts when changes are made or at Night—Report overall Pass / Fail—Replaces Driver—Customizable

We are real programmers:

We should use similar tools and

processes as the App Developers.

Secrets of Successful Automation Projects

Successful Projects Require— Modular Design, Demonstration, Low Maintenance— Plan for Common Pitfalls— Leverages Agile Processes

Automated tests— Keywords & KW-Based Components— Supporting functions— Dynamic Reusable data— Object identification— Recovery & Cleanup

Potential loss can be quantified as ROI Training Increases ROI Always think outside the box!

Resources

Training Classes HP Certification classes HP Discover LinkedIn Users Group Meetings

On the Web cqaa.org stickyminds.com sqaforums.com advancedqtp.com wilsonmar.com

Thank you!

Learn more at CQAA.org

[email protected]

Additional Thanks to Lee Barnes

Utopia Solutions

www.UtopiaSolutions.com