Planning Test Automation (Wisely). Introductions Thomas Messerschmidt Test Automation Specialist Now @ Paramount Pictures Avner Uzan Test Automation Analyst.

Planning Test Automation

(Wisely)

Introductions

Thomas Messerschmidt

Test Automation Specialist

Now @ Paramount Pictures

Avner Uzan

Test Automation Analyst

2Planning Test Automation

Disclaimer

The views expressed in this presentation are those of the presenters & do not necessarily reflect the views of their current or past employers; their subsidiaries or affiliated companies!

3Planning Test Automation

4

Planning Automation - Premise Set the hook Basic Rules Methodology:

Keyword-Driven Spreadsheet-Test Cases Framework Construction

Scope: Smoke, Functional, System, Performance Prioritizing Test Construction

Best Kept Secrets (Trackers & Analysis Masters) Test Automation Best Practices

Wisely

Planning Test Automation 4

(Does Test automation Find Bugs?)

5

Premise:

Create a testing framework that will use easy-to-read spreadsheets to enumerate the actual test cases

Create & maintain generic & application specific functions & subroutines to support the automation

Planning Test Automation 5

Create maintainable automated test suites

Does Test Automation Find Bugs?

Test Automation VS. Manual

176

16134

0%

20%

40%

60%

80%

100%

Resources Defects

Defects

Resources

Automation Manual0

20

40

60

80

100

120

140

160

116

76

134

Automation Vs Manual

Bugs

Testers

7

Basic Rules

Plan your automation process thoroughly Establish & follow naming conventions Establish & persist a directory structure * Build automation for platform

independence Solicit development’s cooperation to

develop for testability Expand your knowledge with other SMEs MEASURE OR DIE

Planning Test Automation 7

8

Methodology

Smoke Testing – Keyword-driven Spreadsheets

Functional Testing – Keyword-driven Spreadsheets

System Testing – Functional Decomposition

Performance Testing – Weighted Functionality

Planning Test Automation 8

Zambelich Method

9Planning Test Automation 9

10

Spreadsheet-Test Cases

Planning Test Automation 10

11

Generic Functions

Planning Test Automation 11

12

App Specific Functions

Planning Test Automation 12

13

Sample Keyword Action

Planning Test Automation 13

14

Parameter & Data File

Planning Test Automation 14

15

Smoke TestObjectives Primarily validate builds Open and close as many GUIs as possible

Some windows are acct data dependant Manually test GUIs that could not be

automated

Smoke Test Execution Metrics Covers 115/120 L1 GUIs - 105/110 L2 GUIs Execution time: 1:25 / L1, 1:10 L2 This was run 200+ times on the WR version This has runs 200+ times on the QTP version Currently demonstrates 100% GUI Pass rate

Planning Test Automation 15

16

GUI FunctionalGUI Functional Objectives Validates the functionality of Each

GUI Verify screen functionality as per spec, NOT business

processes Validates objects exist on UI per spec Verifies object operations, calculations & navigations Verifies data formats ($-Amts, SSN, Phone #, etc.) Verifies field/object data comparisons across screens Verifies all error processing (invalid values, pop-ups,

etc.)

GUI Functional Scope Targeted two languages Planning Test Automation 16

17

System Test

System Test Objectives Focus on core functionality – align w/ project team &/or customer Regression test regular release & special projects Future plans include converting our system test to BPT & auto.

ScopeExecution Metrics 200 scripts of which about 175 are active - take about 4-5 days 50 full suite runs, 10,000 tests run over 5 year period

Use Functional Decomposition

Planning Test Automation 17

18

Performance Test

Planning Test Automation

19

Prioritize Test Construction

Target low hanging fruit to get early ROIPrioritize & build common functions first

Create tests for most frequented GUIs

Planning Test Automation 19

20

Optimizing Automation

Optimize automation for maximum coverage

Increase number of executing machines The downside Brake automation into “byte-size” pieces If one function can do the work of two

Planning Test Automation 20

21

Other Considerations Number of test platform / environments Availability of test platform (hours/day) Comparability of test environment to prod. Environment refresh Test data availability / harvesting (method of) Test data accuracy Product or application stability

Planning Test Automation 21

22

Trackers (Measure or Die)

Planning Test Automation 22

23

Analysis Masters

Demo Analysis Masters

Planning Test Automation 23

24

Performance Analysis Master

Planning Test Automation 24

Automation Best Practices Setup directory structure & use it consistently Define & establish filing conventions Publish changes to subroutines & functions. Keep test assets under version control Choose automation tools carefully based on compatibility to your

development platform, resource knowledge, budget. Include resource, time & process allowances for test asset

maintenance. Keep an inventory list of functions & routines easily accessible. If possible cross-reference use of functions & subroutines in scripts to

more predicatively understand the impact changes to these assets might have.

Prioritize tests to be automated carefully to avoid marginalizing ROI

25Planning Test Automation

Get management buy-in & commitment early Set realistic expectations – automation is not just the push of a

button Identify a set of relevant metrics & measure regularly & consistently Include regular progress reviews to ensure that the automation team

is in fact focused on the business or customer needs Share the wealth – the more people use it the richer you’ll be!

Automation Best Practices2

Additional References:

TOTALLY DATA-DRIVEN AUTOMATED TESTING – Keith Zambelichhttp://automation.org.uk/downloads/documentation/white_papers-totally_data_driven_automated_testing.doc

Keys to Functional Test Automation Success, by Thomas Murphy / The Gartner Group - Thanks to Microfocus an HP Partner for the link http://www.gartner.com/technology/media-products/reprints/microfocus/vol4/article4/article4.html

LoadRunner Test Analysis Master – A. Uzan http://www.youtube.com/watch?v=m860IurgHM8

26Planning Test Automation

27

Automation Demo

Demo Automation

(time permitting)

Planning Test Automation 27

Contact us:

Avner Uzan: [email protected]

Thomas Messerschmidt: [email protected]

28Planning Test Automation

Parting Thought On Automation

Why Automate?

Automation, fundamentally, is the mechanization of a manual task. Love, education & human interaction should not be automated.

However, there is no evil in taking a manual process & allowing a machine to do that which would take a person longer, work a person harder or take the pleasure out of doing a task over & over. However, if by automating something, the existence of life, the values of morality, the sensibilities of one’s existence or the livelihood of those who would otherwise perform the task manually is usurped, than automation has defeated it’s very purpose.

Automation should be carefully conceived so that future generations are not burdened with undoing the damage of unwitting or well intended but misguided endeavors.

29Planning Test Automation