Automated testers agile evangelist

AUTOMATED TESTERSEssential or a luxury?

[email protected]

www.continuumtesting.com@continuumit

Discussion

Automated Test Specialists

• Waterfall and test automation• Agile and test automation – continuous delivery• Why automation testers may be detrimental to your agile project• What type of tester should we have on an agile project?• Q&A

?

Automation

Yes

Waterfall and automation

• QARun, WinRunner, QTP etc.• Feature rich but significant cost (licenses,

support etc).• Cost associated in investing in the test

automation team. • Biggest cost was subsequent maintenance.• Flakey tests not strongly coupled with

functionality.• Frequently the team was made up of part

time auto testers.

• Testers tried to get early involvement in projects (we promoted the V model) but we were frequently stopped at the door.

• So when do we start test automation?• Testing seen as a service.• Frequently squeezed.

Yer names not down, yer not

coming in!

Are non-agile approaches compatible with test automation?

Welcome to the wonderful world of Agile

In an agile world testers are positively encouraged to be integral to the project.

Lisa Crispin:“Where testing is treated as a separate “phase,” programmers may not be used to working closely with testers. Learning to treat testing as an integral part of the development process, as important as coding, helps programmers appreciate the feedback testers contribute.”

“We know from experience that without excellent automated acceptance test coverage, one of three things happens: either a lot of time is spent trying to find and fix bugs at the end of the process when you thought you were done, or you spend a great deal of time and money on manual acceptance and regression testing, or you end up releasing poor quality software.” Jez Humble & David Farley

Spotlight is back on test automation…

Automated continuous

deployments

Continuous integration

Configuration management

Automation

CONTINUOUS

DELIVERY

Why Automated Testers may not be the answer to your project needs…

• Cost of resource• Favoured test tools• Increased chance of miscommunication and silos

• Reporting features as defects• Falling behind• Dependency on auto tester• Fail to find defects early

• Automated testers are not developers• Interface level testing• Failing tests• Refactoring• Objectivity and critical approach

Cost of resource

=

Automation should cost no more than it saves.

Jobserve search…developers

PHP

Java

.Net

663

1929

2598

Jobserve search…testers

Test

Selenium

Cucumber

Capybara

3197

164

51

1

Favoured test tools…

Silos and miscommunication

• Disrupts collaboration – creates disconnects

• Encourages handover culture of gateways and separate phases

• Both devs and testers may see test automation as a separate function

• Tests are not driving development

• Manual testers may not have a full knowledge of automation coverage.

Symptom of silo – features become defects

• Detecting spurious defects when actually the automated test has found a new feature.

• Old problem of working with automated regression pack.

Symptom of silos – falling behind• Automation testers can significantly find

themselves lagging behind the development or even project effort.

• Are you even testing the right build?• Sometimes the automated tester will find real

defects as s/he is writing the tests. But the dev team have moved on to another story/sprint.

• All to frequently the testing is not planned in the estimations or even when it is it invariably falls behind and rolls over to the next sprint.

• Other tasks disrupt automation effort – like helping with manual test effort.

Symptom of silos – falling behind“Automated acceptance tests have to be created, owned and maintained by the delivery team - this should be the default position of all projects. To maintain the focus of the developers on the behaviour of the application, it is important that the acceptance tests are not owned by a separate test team. Otherwise you will find that the test team rapidly become snowed under with tests to repair as well as work to implement new tests for the new requirements that the developers were implementing.”Humble and Farley

Symptom of silo - dependency• Dependency on tester - to run tests, write tests, fix tests etc –

then no one else understands them.

• Manual testers may not have skills or know the tests well enough or are simply reluctant to get involved.

• Developers don't see it as their job to write acceptance tests.

Symptom of silo - Fail to find defects early• We should detect bugs early• Without coding against the tests we are losing the dynamic process of

capturing defects early.• We lose benefits of BDD/ATDD and early defect detection.

Defects not captured early in the process, cause a lot of pain….and money!

Testers are not developers

Testers are not developers…?• Additional overhead – auto tester has

to have not just automation skills but development infrastructure skills to keep their build current.

• e.g. Developers update their local builds (to use .Net 5, or switch to a mongo db, make changes to Maven/POM, new MVC version etc) and the auto tester is not in the loop.

• Stay up to date with branching strategies.

• Otherwise automated tester starts finding spurious issues.

• Is it a communication problem? What team do they fall under - do they attend developer meetings and retrospectives, join dev emails aswell as tester groups?

Interface level testing

Mike Cohn’sAutomation pyramid

Interface level testing

Servicelayer

Presentationlayer

High maintenance

Deeper, more reliable

relationship

Shallow presentation layer….

Replace the presentation layer and service layer tests stay the same….

Fixing failing tests…• What happens when tests fail - who reacts to the tests and makes sure

they are working?• We've already established that the automated tester or team may be

busy trying to keep up with new stories, so how much time will they need to dedicate to failing tests or builds?

Tests are set up to fail and alert developers and the team. But frequently they backfire on the automated testers who are held responsible for fixing them. More time may be spent fixing tests, than detecting defects.

Refactoring

Without a shared knowledge of the tests we may lose the benefit of confident refactoring.

Ideally you want the tests to be run in a suite of build tasks.

Objectivity and critical approach…

• We all want the sprint to burn down its story points right?

• Lose critical approach. Not exclusive to auto testers but also embedded testers.

• Are automated testers more focused on writing good tests and making those tests pass or in actually testing.

• Some automated tests written to work around the system i.e. make it pass, rather than really test the functionality.

Solutions…?First and foremost developers should ‘implement’ and ‘pass’ the acceptance tests.Can automation be successful outside of BDD/ATDD?

“When automation is done along with implementation, developers have to design the system to make it testable. When automation is delegated to testers or consultants, developers don't take care to implement the system in a way that makes it easy to validate. This leads to more costly and more difficult automation. It also causes tests to slip into the next iteration, interrupting the flow when problems come back.”Gojko Adzic

• Can an automated tester help a developer write the step definitions? Yes, they can pair on the test - but so can two developers.

• The tester can help write the test? Yes - but you don't need to be an automated tester to do that.

So where does that leave the automated test specialist?

Agile tester…Lisa Crispin:"I’m often asked: If programmers are doing these tests, why do software teams need specialized testers? The answer lies in the particular skills that testers bring to the party."

Reprise for the 'part time' automated testers of yore?

Agile tester…wish list• Understanding of test automation – clarity of what is being

tested and importantly what is not being tested • Ability to extend acceptance criteria and write good feature

files – Gherkin skills• Strong character - insist on definition of done• Technical ability to extend automated scenarios, write auto

regression packs / deployment smoke tests, fix tests, pair• Exploratory test skills• Energetic, confident, and energetic again

Thanks for listening…

Over to you…

Q&AHow many automation testers are on your team?

How does it work for your project?

• Cost of resource• Favoured test tools• Increased chance of miscommunication and silos

• Reporting features as defects• Falling behind• Dependency on auto tester• Fail to find defects early

• Automated testers are not developers• Interface level testing• Failing tests• Refactoring• Objectivity and critical approach