Software Testing

Test AutomationSteffen Goerlitz

Barry LangeMitchell Meerman

Harry SchultzTrevor Spees

What is test automation?

• “In software testing, test automation is the use of special software to control the execution of tests and the comparison of actual outcomes with predicted outcomes.” – Wikipedia, 12/06/2015

• Test automation is used to automate tasks that happen often where a test process is in place, or a process that would be difficult to do manually.

THE AUTOMATED TEST LIFECYCLE METHODOLOGY (ATLM)

• structured methodology geared toward ensuring successful implementation of automated testing

Advantages of Automation

• Test faster• Testing a string table with a program would always be quicker than by

hand.• Test more• Faster testing allows for broader testing scope.• Machines can find defects a human technician either may not look for

or accurately be able to test.

Advantages of Automation

• Potential observations:• Fixable bug finding rate• Find rate spikes• Fix rate• Testing being made possible• Nightmares averted• Test cycle time

Advantages of Automation

• Potential observations (cont.):• Development and test cycle time• Management confidence• Technical support costs• Customer retention• Reputation• Delivered solutions

Disadvantages of Automation

• GUI test procedures may not survive some development styles• e.g. Agile, Extreme Programming, etc.

• Additional code must be maintained• Technicians may adapt to changes in test procedure rather than changes

in the tested program• Automated testing may miss things that a human tester would

be able to find

Disadvantages of Automation

• Very expensive• In some cases, automation is not cost-effective• Regression testing and reduced test time helps offset the cost

• Not everything can be automated• e.g. password input

Types of Automation

• GUI testing• API testing• Black box testing• Data-driven testing

What should be tested?

• Product installation• Test data creation• GUI interaction• Problem detection• Defect logging

Types of Automated Tests

• Code Auditing• Coverage Monitor• Functional Tests• Load Tests• Test Management

Code Auditing

• Automatic qualification testing• Ensures compliance of code to specified standards

Coverage Monitor

• Determines quantitative measure of code coverage when tests are ran.• Very valuable for white-box testing

Functional Tests

• Aimed at examining overall functionality of the product• Replaces manual black-box testing• Only focuses on externally visible behavior

Load Tests

• Testing the application under heavy loads and measures response.• Can simulate many aspects• Change of hardware• Change of scenario• Virtual events

Test Management

• Involves people carrying out tests and correcting the detected errors• This test monitors performance of every item on test case

files

Test Automation in Extreme Programming

• Test-driven development (TDD)• Write a failing test, then write code that will pass the test

• Regression testing• Test automation fits well into regression testing and is performed

frequently in XP

Test Automation in Extreme Programming

• From the report Agile Test Automation by James Bach:• One “toolsmith”, or dedicated programmer, should be appointed to

the testing team• Provides tool support for entire test project, not just test execution• Reports to requests made by testers• Solutions valued based on how much they help testers and test managers

perform their jobs• Focus on short term testing

Toolsmith’s Duties in Test Automation

• Respond rapidly to requests for assistance from testers• Seek out test productivity problems• Investigate possible solutions in concert with testers• Apply technology to improve the test process

Toolsmith’s Duties in Test Automation

• Advocate for specific testability features in products• Research available tools and learn how to use them• Gather tools that developers or testers produce• Review upcoming product plans to assess automation

possibilities

Risks to Consider with Agile Automation

• Testers must be familiar with testing or automation will not cause significant improvements in productivity• The number of test cases created or executed in a day can be

very misleading and can lead to the number of tests more important than the errors that the tests find.• Testers may not want to work with technicians. Cooperation is

paramount or the system will fail.

Risks to Consider with Agile Automation

• Technicians may deliver testing solutions that are too maintenance intensive when compared to the value they provide• The technicians won’t be the end-all for test innovation;

cooperation will help pool the knowledge to be able to create efficient and effective test solutions.

Conclusion

• Test automation lets you test more software faster• Really good for regression testing but must be careful how it’s

implemented• Only worth using in Agile if implemented in fewer than 40

man-hours• When used properly, it helps create better quality software

Sources

• Bach, J. (2003, December 1). Agile Test Automation. Retrieved December 6, 2015, from http://www.satisfice.com/articles/agileauto-paper.pdf• Haria, M. (2009). Automated Software Testing. Retrieved

December 6, 2015, from http://www.cs.nott.ac.uk/~pszcah/G53QAT/Reports09/mxh06u/QAT09Report-mxh06u.doc