Stories from the Trenches Seven Productivity Improvements for Test Automation Frameworks Presented By Chris Petrov Leverage Point Corporation SCQAA Inland.

Stories from the Trenches

Seven Productivity Improvements for Test Automation Frameworks

Presented By

Chris PetrovLeverage Point Corporation

SCQAA Inland Empire Chapter May 8, 2008

Leverage Point Corporation417B W. Foothill Blvd. #164Glendora, CA 91741626-857-9446www.levpoint.com

Leverage Point Corporation

• Founded in 2006• Specialize In Software Quality Assurance

Consulting and Staffing• Outsourced Testing• Test Automation Framework Development• Testing Project Support

Leverage Point Corporation – Copyright 2007 www.levpoint.com

"We will provide the best people, both technically and in customer focus, to deliver specialized high-value and indispensable IT services. Each of our employees will maintain a singular focus on using all of their experience, skill and creativity to ensure the quality, performance and reliability of our customer’s critical business applications.”

Productivity Improvements

• Faster Learning • Faster Test Creation• Reduced Maintenance • Faster Analysis• Flow


Productivity Improvements• Give the Users a Familiar Test Creation

Environment• Supply List of Available Test Actions

• Lightweight Interface• Intuitive vs. Usable

• Static Test Case Validation• Build Compartments• Improve user experience


Project Profile

• Clients Background • 2 Independent Organizations• Similar Applications• Compressed Test Schedules • Limited Application Availability• Different Framework Users

– Testers with no automation experience and no application experience in one

– Business analysts with application experience, but no testing experience in

the other


Project GoalsCreate a Keyword Driven Framework • Fast and Easy Test Case Creation• Require Minimal Training for Users with no

Automation Experience• Tests Cases Creation Before the Application is

Available• Frequent Incremental Deliverables


Automation Challenges• Complexity• Maintenance• Extensibility• Robustness• Learning Curve• ROI


Best Framework• What are your goals?• What are your products?• What are your users?• What is the culture?• What lies in the Future?• Trade offs


Framework Design• Repository• Engine• Documentation• Front End


Keywords Selection• Keyword Selection Options– Many (200+) Keywords Based on Business

Scenarios– Few (20-) Low Level Keywords– Combined Approach• Low Level Keywords• High Level Action available as executable scripts• Script Areas (Libraries)


Keywords Selection• Selecting Limited Amounts of Actions Available to the User

– 11 Frequently Used Actions• Enter• Click• Select• Run Script• Validate

• Hide Control Knowledge– It looks like a button, but it is a link – Click vs. Click Button

• Action Overloading -Using the same action for different controls• Click Link, Button, Checkbox• Select Radio Button, Dropdown, Checkbox

• Hiding Complex Flows Behind Scripts– Identifying the common business processes– Getting the best of both worlds


Keywords Selection


GUI Map Organization• Create Maintainable and User Friendly GUI Map• Map Interface for Application that is not yet

Available• Make the GUI Map customizable for the intended

product• Building Object Hierarchy from Usability Point of

View– “Menu -> Screen -> Frame -> Field” Approach– “Menu -> Screen -> Field” Approach

• GUI Map that allows user friendly naming convention• Removes the confusion for the tester


GUI Map Organization


GUI Map Organization• Object Naming Conventions

– Name the object exactly as it is on the screen– Screens – how do we name the screens to be recognizable even by

users that don’t know the application– Objects with the Same Label – how do we differentiate

• Change the name of both objects, to avoid misidentification• Add the Frame name after the label• Add the Action name after the label

– Objects without label• Use the Help file descriptions• Use Purpose + Default Values

– Grids• Column Name: Grid Name


Give the Users a Familiar Test Creation Environment

• No Testers interaction with the Test Tool• Easy to use• Familiar To your users• Domain Experts could write automated test cases• Fast Test Creation• Minimal Effort• Copy and Paste


Supply List of Available Test Actions• Problems when typing names

– Spelling (Disbursments - Disbursements)– Misidentifying Controls ( Address 1 - Address)– Using Wrong Duplicate Controls ( Zip Address - Zip Information)– Abbreviations (Addr – Address)– Ambiguous Entries – (Name - Last Name or First Name)– Partial Entries – (Last – Last Name)– Single/Plural– Remembering the control Name

• Dropdowns– Based on Menu, have only the Screens available for this menu as a choice– Based on Screen, have only the Fields children of that menu available in the

dropdown– Update the dropdown values dynamically– Demo or Movie


Framework Front End Demo


Lightweight Interface• No wizards• Playing Music• Performance Measurements

– Initial Prototype with 1000 screens – 45 sec– Redesigned Prototype with 1000 screens – 2 sec

• Identify Time Required for Specific Actions– No delays between actions

• Screen Frame Field or Screen Field Measurements


Intuitive vs. Usable• Difference• Frequency of Use• Design that Reduces Test Case Creation Time

– Screens by Tab Order – because the application already is measured for usability

– Fields by Alphabetical order• Tab Order Changes• Duplicate Controls are next to each other• Easier to find the control that you barely remember

– Keywords by frequency used


Dropdown Usability Design• Performance Measurements

– Initial Prototype with 1000 screens – 45 sec– Redesigned Prototype with 1000 screens – 2 sec

• Identify Time Required for Specific Actions– No delays between actions

• Screen Frame Field or Screen Field Measurements• Design that Reduces Test Case Creation Time

– Screens by Tab Order – because the application already is measured for usability

– Fields by Alphabetical order• Tab Order Changes• Duplicate Controls are next to each other• Easier to find the control that you barely remember

– Keywords by frequency usedLeverage Point Corporation – Copyright 2007 www.levpoint.com

Static Test Case Validation • Static Test Case Validation• Correct many errors before the test case is executed

– If possible fix the problems before, not during the test• Identifying Frequent Errors

– During the Initial Design– Root Cause Analysis of Errors in Existing Tests

• Adding Validation for Frequent Errors• Visual Indication of the Defect with Correction Advice• Formatting the report with the corresponding Font styles and colors• Error Report with identification of the file, sheet and the row the

error occurs• Single feature that reduced the Automation Debug Time by 40%


Static Test Case Validation


Compartments • Hide Complex Flows• Isolate Volatile Areas• Isolate Frequently Used Actions• Specify the Essential• Keep default values


Improve user experience • Continuously improve productivity– Watch the users– Improve the experience– Actively Eliminate problems– Flow

• Build the needed tools


Productivity Tools • GUI Map Builder– Build Complete GUI Map with Single Click

• “Where Used” Maintenance Tool– Update Test Cases when the GUI Changes

• Traceability Matrix with Integrated Progress Reports and Trends– One-Click Reporting

• UAT Builder– Build “Bare Bones” Test Cases from a List of

RequirementsLeverage Point Corporation – Copyright 2007 www.levpoint.com

Productivity Tools


Productivity Tools


Framework Updates• Designed for Unobtrusive Updates• Server Location• Single Update File• Notification that New Version is Available


Data Driven Add-in • Benefits and Limitations of the Data Driven Tests– Fast Design– Great for Discreet logic when equivalence partitioning is

used– Require Advanced User to write and parameterize them

• Getting the Best of Both Worlds– Build the test using the keywords– Parameterize them on the fly with the build in variable

functionality

• Running Tests from the Design Sheet– Demo or Movie


Data Driven Add-in


GUI Unit Test Add-in • Testing GUI Screens from the Specification• Learning Curve• Integration with the Keyword Driven

Approach– Changing the Paradigm


GUI Unit Test Add-in


DB Query Engine Add-in • Challenges

– Data Control of the Test Artifacts– Testing Converted Data– No Application Database Refresh Available– Ever Changing Data– Complex Database Structure– Different logic based on record data

• User Friendly Implementation– Hide the Complex SQL Statements behind user friendly names– Change the where clause on the fly– Mark the record as used, so it is not locked by other test case


Accomplishments• Established Sustainable, Scalable Automation

Solutions• Reduced Test Creation Time by 50%• Reduced the Test Automation Debug Time by

40%• Learning Curve of 2 to 8 Hours


Stories from the Trenches Seven Productivity Improvements for Test Automation Frameworks Presented By Chris Petrov Leverage Point Corporation SCQAA Inland.

Documents

point corporationfounded

leverage point corporation417b

usability point of viewmenu

application experience

test automation frameworks

testing experience

business scenariosfew

business analysts