How to scale your Test Automation

How to scale yourTest Automation

Klaus SalchnerVP Engineering

https://ca.linkedin.com/in/ksalchner

https://twitter.com/ksalchner

https://twitter.com/BDCareers





https://twitter.com/BDcareers

https://twitter.com/BDcareers

3The problem statement

What we know & and want to make happen• We know we need the right amount of testing to assure quality• We know we need to write enough (more) unit testing• We know we need test automation for repeatability & scale• We know we need to measure test coverage to know our coverage

What we are wondering about• How do I scale my test automation so it stays manageable as the product grows?• What is the right balance between unit testing, API testing and UI testing?• How much should I automate vs. manually test?• How much should software engineers test vs. test engineers?

Is there a framework which can guide me in these decisions• Test Pyramid by Martin Fowler• Chapter 7 – Testing – of the book “Building Micro-services” by Sam Newman

4Test PyramidBy Mike Cohn

5Often this is our reality...By Mike Cohn

This reflects your culture when it comes to Testingand Quality

6Unit TestsThis should be your foundation

Unit Tests (ownership with Software Engineer)Smallest logical unit or testable unit which gets tested / scrutinized for proper operation

http://martinfowler.com/bliki/UnitTest.html

Key characteristics- Small scope- Run fast- Run independently- Run very often- Run at every check-in




8Component TestsOften lumped together with unit tests

Component/Module Tests (ownership with Software Engineer)• Next logical step up - tests / scrutinizes the proper operation of a component / module (in isolation)• It validates the “contract” your component/module has with other components/modules• Mocking is often used to decouple the results from other components/modules

http://martinfowler.com/bliki/ComponentTest.html

Key characteristics- Relative small in scope- Run fast- Run independently- Run often- Run at every check-in



9Integration TestsYour test to make sure it all works together

Integration Tests (ownership with Test Engineer & Software Engineer)Tests / scrutinizes the proper operation of several components/modules combined (integrated)Mocking is still used if you have lots of components

http://martinfowler.com/tags/testing.html

Key characteristics- Keep scope contained- Run as fast as possible- Run independently- Run often- Run at every check-in



10API TestsTest how the rest of the world access your code

API Tests (ownership with Test Engineer & Software Engineer)Tests / scrutinizes the proper operation of your API (in’s and out’s) – often part of integration testing


Key characteristics- Keep scope contained- Run as fast as possible- Run independently- Run often- Run at every check-in



11Continuous Delivery PipelineLet’s use the same language...

How does this fit into my Continuous Delivery• Run these tests as part of every build / check-in• Measure your code coverage, e.g. dotCover from JetBrains

Build Unit Tests ComponentTests Deploy Integration

TestsAPI

TestsDeploy

Measure your code coverage

http://martinfowler.com/bliki/DeploymentPipeline.html

http://martinfowler.com/bliki/ContinuousDelivery.html





12ExerciseTurn to your neighbour and...

Exercise One (3 min)Discuss the shortcomings your team has with Unit Tests and Components Tests. Then discuss how to improve it

Exercise Two (3 min)Discuss the shortcomings your team has with Integration Tests and API Tests. Then discuss how to improve it


14UI TestsTest like your customers see it

UI Tests (ownership with Test Engineer)• Tests / scrutinizes how customers access your application functionality• Costly since UI changes are more frequent and therefore require rework of UI Tests• Proliferation of Devices and Browsers makes it hard to run all combinations – focus on the key ones• Write them using HTML tags/IDs so that style, text, and positioning changes have no impact


Key characteristics- Keep scope contained- Run as fast as possible- Run as independently as possible- Run sub-set as frequent as possible- Run full-set at least once a day



15Acceptance TestsYour sanity check before it hits your customers

Acceptance Tests (ownership with Test Engineer and Product Owner)• Your sanity check to make sure your key use cases work and make sense to you and your customers• Acceptance by Product Owner that you address the customer pain/need of the epic/story• Should be really limited and only manual testing, as it is hard to scale


Key characteristics- Focus on the key use cases- Perform after team completed work- Creates right accountabilities



16ExerciseTurn to your neighbour and...

Exercise Three (3 min)Discuss the shortcomings your team has with UI Tests and Acceptance Tests. Then discuss how to improve it

17SummaryBy Mike Cohn


19Summary

Your friends for success are…• Small and independent test which can run independently • Tests which run fast and independently• Tests which you can run over and over again• Tests which run all the time, at every check-in

Invest, invest, invest• Aggressively in Unit Tests and Component Test – this should be your biggest portion• Actively in Integration Tests, API Tests and mocking – still bigger then your UI testing• Measure your code coverage and understand what your risk / coverage is – don’t go blind• Do your sanity check before you release

Your big enablers are• “Done Definition” on the team level which includes testing fully completed• A robust “Continuous Delivery” pipeline which executes this seamlessly all the time• Iterate with small chunks which go out to production and get used• A very active feedback loop from internal and external users / customers

20Summary

Your team owns all these tests• Don’t have other teams or individuals outside of the team do any of these tests. It just creates artificial

boundaries and points which need coordination• You are only Done when all these Tests are done

21ExerciseLet’s learn together as a team

Exercise Four (3 min)Now let’s share some of our challenges and successes, and how you or someone else has dealt with it.

Q&A

How to scale your Test Automation

Technology