Lessons Learned When Automating

Lessons Learned When AutomatingTabara de Testare Webinar

Alan Richardson

@EvilTester

[email protected]

EvilTester.comSeleniumSimplified.com

JavaForTesters.comCompendiumDev.co.uk

Lessons Learned When AutomatingTabara de Testare Webinar

Alan Richardson

@EvilTester

[email protected]

EvilTester.comSeleniumSimplified.com

JavaForTesters.comCompendiumDev.co.uk

BlurbI've been asked some very challenging questions about lessons learned, and how decisions are made during the process of automating and performing technical testing. In this webinar I'm going to answer them based on my experience. We'll discus how we know 'what to automate' which means we have to split our analysis into 'detection' and 'testing'. We'll cover lessons learned from solving problems, and making mistakes, and steps we can take during the problem solving process e.g for intermittent failures, and possible tool bugs. We'll discuss abstraction levels and the different levels of the technology stack to automate: how to do it, and how we make the decisions. We'll discuss coding primarily the differences, and the overlap, between the needs for coding for testing and coding for production deployment. We'll also cover some WebDriver specific answers to some of these questions. I'm also going to describe books and techniques that have helped me over the years when trying to deal with these questions on production projects. Also we'll take additional and follow up questions.

Automating

● Words & Definitions● What to automate?● Problems encountered automating?● Levels to automate at?● Improve testability for automating?

Words & Definitions

● 'test' 'automate' used loosely?● Can you automate testing?● What words to use?

What to Automate?

● How to decide if we should automate something ?– Any Heuristics?

'Detection' or 'Testing'

● Detect for 'known' problems when they occur● Test for unknowns and improve process

Detection

● Is it part of the Agile Acceptance criteria?● Is re-appearance a concern? Bug/Problem● Is it an area of the system that lacks lower levels of

verification?● Is it a problem we never want to re-appear?● Is it a risk/problem that is hard to detect if it manifests?● Is it a risk/problem that is slow to detect if it manifests?● Is it intermittent behaviour that we are trying to track

down?

Detection

● Is it part of the Agile Acceptance criteria?● Is re-appearance a concern? Bug/Problem● Is it an area of the system that lacks lower levels of verification?

● Is it a problem we never want to re-appear?● Is it a risk/problem that is hard to detect if it manifests?

● Is it a risk/problem that is slow to detect if it manifests?

● Is it intermittent behaviour that we are trying to track down?

Process

CoverageFeedback

WasteEffective

Debug

Ambush

'Testing'

● Is variability in the scope of data?● Future value in path/data/ combo execution?● Am I prepared to do this manually next time?● How easy to automate this?● Is this hard/slow to do manually?● Predictable results checking?● Explored enough already?

'Testing'

● Is variability in the scope of data?● Future value in path/data/ combo

execution?● Am I prepared to do this manually

next time?● How easy to automate this?● Is this hard/slow to do manually?● Predictable results checking?● Explored enough already?

Variety

Value

Lazy

Time

Risk

Checkable

Information

Secrets of Automating

● Path– subpaths

● Data–Variant– invariant

● Assertion

Login

Enter Details

Create Entity

AmendDetails

Choose Option

Logged InError

Created

Amend Created

ErrorAmended

!Logged In

Log out

Problems Encountered Automating

● What problems encountered writing automated tests?

● How to resolve?● Problems change over time

Problems Encountered At Start

● Lack of tool familiarity● Tool Immaturity● Choice of tools, risk of commitment● Hard to know what are your problems and what

are tool problems● No Abstractions

Problem Diagnostic

● Isolate issue with a small @Test● Make issue repeatable● Debug mode● Step slowly

– If no problem then synchronisation problem

● View tool source code● Different version combinations (down, up)● Identify workarounds

Problems Encountered Now

● Decide on level of abstraction● Decide on tech stack level to target● Decide on tooling to use● Unit test or not Unit test my code● Synchronisation issues● Ease of System Automating● Mobile & New platforms

Levels to automate at

● How do you decide which level to automate at?● Would you combine levels?● Do you use abstractions?

– Page Objects? Data Models? Other Models?

How do you decide which level to automate at? GUI? API? Unit? etc.

● What is your model of the system?● Where do you trust/value feedback from?● Where can you automate fast to add value

quickly?● What are you prepared to maintain?● What environments do/will you have?

Would you combine levels?

● e.g. using GUI to create account, editing info and then verifying from the DB if data was stored properly?

Would you combine levels?

● Yes● Path Segment (subpath) preconditions

– Create at a level that you trust

● Automate at the level of the risk you want to detect● Assert at multiple levels based on the conditions you

want to check– Created – check in DB

– Reported Created – check in API/HTML

– Rendered Created Message – check on GUI

Would you combine levels?

● Yes● Helps build abstraction layers that are clean● Avoids frameworks● Builds libraries● Can re-use in different ways

Do you use abstractions?

● Page Objects?– Yes, an abstraction of the physical GUI– Not just Pages: Components, Navigation

● Data Models?– Yes, abstraction of persistence, messaging and logical– Random data generation– 'Default' data

● Other Models?– Yes, path and system models– Layered execution models

● API, GUI as API, Files & Persistence

Improve testability for automating

● Advice to improve testability?● Tools?

– Re-use abstraction layers (different level of systems modelled – API, DB, GUI, etc.)

– execute via @Test– Simple batch scripts

● Use abstractions for exploratory testing● Executability

– Tool hooks – GUI ids, APIs, no https, etc.

Coding

● How is coding different for testers than for programmers?– Any different coding Skills?

– Language usage?

Differences

● Subset of the language● Junit rather than container● Coding for efficiency● YAGNI vs IDKWTAGN● Multiple Usages vs Controlled Access● Paths and Libraries vs Applications● Frameworks vs Libraries● Coding for Change vs Requirements (Requisite Variety)

Similarities

● Advanced Books● Static Analysis Tools● Unit Testing● TDD● Naming and Coding Conventions● Test Execution Runners● Libraries● Debugging

Skills

● Same skills required● Levels of Experiences different● Developers better be the best at coding● Project can afford for Testers to be less

experienced coders, supported by developers

Estimation

● “How much time is needed to automate an application?”

● How do you estimate when you are just starting to automate?

Estimation

● I tend to avoid these questions, unless they are part of a sprint planning for estimating the automating of specific acceptance criteria

● But if I have to...

Estimation

● Same way estimate any development project● Split into chunks● Make unknowns, risks and assumptions clear● Gain experience with tools to identify capabilities● Experiments to improve estimates and derisk● Depends on skills and experience● Depends on levels of change● What % dedicated to automating vs testing?● Easier on 'Agile' stories

Tools

● Is there another option (except Selenium WebDriver) which you would recommend for UI automation?

Tools

● No● http://seleniumsimplified.com/2016/01/can-i-

use-selenium-webdriver-to-automate-a-windows-desktop-application/

● Different technologies require different tools

WebDriver

● Locator strategy tips?● Problems encountered?● Implicit & Explicit Waits?● How to structure project?● Frameworks?● Disadvantages?

Location Strategy Tips?

● Aim for an ID● Optimised hierarchy starting at an ID● Build less for speed of execution and more

accuracy across multiple pages● More arguments about managing in the code

Common WebDriver Problems

● Synchronisation– Add more than you think– Sync prior to action– SlowLoadableComponent– 'ware remote

● Abstraction Layers– Refactoring

● Bug workarounds– JavaScriptExecutor– Inject cookies from HTTP calls– Monkey patching Ruby

Implicit & Explicit Waits

● Never Implicit Waits● And if Explicit waits still result in timeout?

– Missing Synchronisation

– Environment Speed Variability

– Remote Grid?

– May have to increase timeout on 'big state' actions

How to structure project?

● Maven Structure● \test

– The @Test code

● \src– The abstractions

● Packages– Refactor as we go

Frameworks or additional tools?

● No, I avoid frameworks as much as I can● WebDriver doesn't seem hard enough● Model application domain as abstraction layers● Closest to framework – Cucumber, Junit

– Cucumber – DSL

– Junit – test runner

– Both delegate/use domain abstractions

Disadvantages of WebDriver?

● Not fully supported by browser vendors yet– Safari/Apple

– Microsoft (Edge isn't complete yet)

● Compared to what?– Do browser vendors support any other tool?

– Google (Chrome), Mozilla (Firefox)

Career

● “How do you arrive/What was the journey from a technical side to having conference talks and training people?”

Career

● Do you feel strongly enough to be the change?● Are you prepared to do the work?

Techniques

Techniques that have helped

● Decision making● Redefinition● Books

Decision Making

● Responsibility● How do I know if I'm making the right decision?● What if I make the wrong decision?

Use words to help you

● Avoid ambiguity● Own your definitions

Books

● 'Clean Code'– References: Dijkstra, Hoare– Reference Peers: Myers, Yourdon, de marco, Jackson

● Others– 'Growing Object-Oriented Software','Working with legacy

code', 'Implementation Patterns', 'Domain Driven Design','refactoring'

● Systems– Cybernetics, Herbert Simon, Stafford Beer, Deming, John

Diebold

Future of Testing

● “How do you see the future of testing?”

Future of Testing

● Testing will, and always has…– been, contextual

– been about feedback

– involved coding and technical levels

– Involved exploration

– been implemented badly in some environments

Future of Testing

● Testing will,– Require more technical knowledge

– Require more testing knowledge

– Be recognised as more skill == better testing

– Be implemented badly in some environments

Future of Testing

A more important question is● “What are you doing, to improve your testing?”