John Fodeh, Cognizant Technology Solutions Adventures in Test Automation – Breaking the Boundaries of Regression Testing www.eurostarconferences.com @esconfs #esconfs Insert speaker picture here, no more than 150x150 pixels
Dec 18, 2014
John Fodeh, Cognizant Technology Solutions
Adventures in Test Automation – Breaking the Boundaries of Regression Testing
www.eurostarconferences.com
@esconfs #esconfs
Insert speaker picture here, no more than 150x150 pixels
John Fodeh
Cognizant Technology Solutions
Adventures in Test Automation Breaking the Boundaries of Regression Testing
| ©2012, Cognizant
Outline
• Innovation and testing
• The promise of automation
• Going beyond regression testing
• Automated Test Monkeys
• Model-Based and Behaviour-Driven approaches
• The human factor in automation
3
| ©2012, Cognizant
The Promise of Test Automation
• Replacing repetitive and tedious manual tasks
• Ensuring the consistency and repeatability of tests
• Performing tests that are difficult to run manually
• Accelerating test execution
4 * Image source: Rosemont Engineering
| ©2012, Cognizant
Traditional Test Automation
5
Typically
Automation of regression tests
Purpose
Testing of a previously tested program following
modification to ensure that defects have not been
introduced or uncovered in unchanged areas of the
software, as a result of the changes made. It is performed
when the software or its environment is changed.
[ISTQB Glossary, v2.1]
| ©2012, Cognizant
Characteristics of Traditional Test Automation
• Static
• Simple
• Synchronized
• Vulnerable
6
| ©2012, Cognizant
Monkey Testing
7
Monkey Testing refers to the
process of randomly exercising a
software program by means of an
automated test tool
| ©2012, Cognizant
Test Monkey
8
#script file: Test4.asc CP_KEY FOCUS_UP; TRACKBALL 0, 64; KEY L; KEY 5, LONG; KEY 8;
#script file: Test4.asc CP_KEY FOCUS_UP; TRACKBALL 0, 64; KEY L; KEY 5, LONG; KEY 8; CP_KEY FOCUS_UP;
Random Test Tool
#script file: Test4.asc KEY L; KEY 5, LONG; CP_KEY FOCUS_UP; #KEY BOX_SIZE_UP KEY 8; TRACKBALL 0, 64; KEY SHIFT, R;
Action List
Log file
Application Under Test
#script file: Test4.asc KEY L; KEY 5, LONG; CP_KEY FOCUS_UP; #KEY BOX_SIZE_UP KEY 8; TRACKBALL 0, 64; KEY SHIFT, R;
Utilities
| ©2012, Cognizant
Basic Features
A test monkey should:
• Select randomly from input range
• Enter input to AUT through the user interface
• Detect “life signs” of AUT
• Have robust logging facility
• (Re-) start and initialize AUT
• Have limited application knowledge but general environment awareness
9
| ©2012, Cognizant
Using Test Monkeys
• Indicator of application reliability
• Incorporated into daily “smoke” test
• Establish confidence in application
• System test tool
• Evaluation of release candidates
• Previously failed scripts are run again as part of regression tests
• Focus tests on specific areas of interest
• Combinations of sequential and random scripts
• Random scripts with weight on specific actions
10
| ©2012, Cognizant
Metrics – Application Reliability Stability Trend for 4101
0
20000
40000
60000
80000
100000
120000
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21
Build no.
Me
an
nu
mb
er
of
ran
do
m o
pe
rati
on
s b
etw
ee
n
Fa
ilu
res
Random operations Start Criterion Release Stability Criterion
11
Entry
Criterion
Release
Criterion
| ©2012, Cognizant
Added Value (1/2)
• Early testing
• Cost effective
• Negative testing
• Long and complex test runs
• Stability indicator
12
| ©2012, Cognizant
Limitations
• Misses obvious defects
• Does not emulate real use situations
• Long tests runs can be difficult to debug
• For a reliable statistical basis, long and multiple test runs are needed
13
Monkey Testing is a supplement
to your manual and automated
testing - not a replacement!
| ©2012, Cognizant
Advanced Test Monkeys
• Wide application knowledge
• Application modeled in state table
• Illegal input to test error handling and recovery
• Emulate real and complex use scenarios
• Effective in finding defects
• Higher development and maintenance cost
14
| ©2012, Cognizant
Test Monkey
15
Random Test Tool
State table
Log file
Application Under Test
Utilities
Start
State
Action End
State
Script Start state Action Probability
Input range End state
Scanning Adjust frequency 45 3 20 Scanning
Adjust brightness 20 1 100 Scanning
Abort 5 Displaying (abort)
Edit preferences 10 Editing preferences
Increment depth 20 Scanning
| ©2012, Cognizant
Intelligent Randomness
• Probability Tables
• Focus on selected areas
• Weight on specific use scenarios
• Continuous expansion and improvement
• Possibility to exclude certain parts
16
Select Text
Enter Text
Delete
Copy
Cut
Task n
…
Action
25%
5%
10%
5%
5%
* source: Whitmill, Kelly
| ©2012, Cognizant
Model-Based Test Monkeys
17
Random Test Tool
Log file
Application Under Test
Utilities
Start
State
Action End
State
Script State table Model
| ©2012, Cognizant
Next Step - Behavior-Driven approach
• Based on Behavioral specifications
• User Stories defined using ubiquitous language
• Principles of Test Driven Development
• Collaborative approach for business analysts, Software development & testing
• Shared tools and process
18
| ©2012, Cognizant
People Issues
• Applying test automation involves people • Your team members are not interchangeable machine
parts • Take that into account when implementing test
automation
19
20
12
Jo
hn
Fo
de
h
| ©2012, Cognizant
Summary
• No “one size fits all” solution
• The tools, processes and skills are there
• Maturity is about analyzing the current practice and finding ways to do evolve
• Are you ready for your innovation journey?
20