Are Your Continuous Tests Too Fragile for Agile?

Paraso& Proprietary and Confiden1al 1

11/13/14

Are Your Con*nuous Tests Too Fragile for Agile

Arthur Hicken Paraso& Evangelist

Paraso& Proprietary and Confiden1al 2 Paraso& Proprietary and Confiden1al 2

Agenda

§  Challenges of Con1nuous Agile §  Tes1ng defined §  Tes1ng is hard (no one wants to do it) §  Best prac1ces §  A modest proposal: Spring cleaning


The role of QA in Agile

§  QA is part of development, not an a&erthought

§  Review developer tes1ng efforts §  Addi1onal tes1ng of requirements §  Itera1ve during development §  Fast discovery and repor1ng of defects §  Inves1gate root cause of defects §  Influence development policy


Challenges of Agile

§  Need to be fast §  Need to be flexible §  Tes1ng can be overlooked if not baked in §  Architectural compromises from schedule


Challenges of Con1nuous

§  Tests must produce binary decision go/no-‐go §  Reuse unit test and func1onal tests from dev to QA

§  High level of automa1on §  Requires disciplined mature process §  Tes1ng must automa&cally answer

§  Is it stable § Will it do what it’s supposed to do


Importance of Tes1ng

§  How do you know when you're done? §  How do you know if a fix for a minor bug broke a major func1on of the system?

§  How can the system evolve into something more than is currently envisioned?

§  Tes1ng, both unit and func1onal, needs to be an integrated part of the development process.


Tes1ng delivers

A maintained suite of unit tests: §  Represents the most prac1cal design possible §  Provides the best form of documenta1on for classes

§  Determines when a class is "done" §  Gives a developer confidence in the code §  Is a basis for refactoring quickly


Why Tes1ng is O&en Ignored

§  Perceived as “burdensome” §  Too much effort §  Too much 1me §  We’re on a schedule – no 1me to test.

§  Why is there always 1me to do it over, but never to do it right?

§  Developer Ego §  “I don’t need to test my code – I don’t make mistakes.”

§  Tes1ng means I have to admit that I might have made a mistake.

§  I don’t like to test – it’s beneath me. §  Tes1ng isn’t fun – I just want to code! §  That’s for the QA dept.


Defini1ons

§  Unit §  Func1onal §  Regression


Unit Tes1ng

Unit tests that validates your method behaves a certain way given a specific parameters

§  Regressions tes1ng § Validate behavior of exis1ng code hasn’t changed §  Scheduled and run automa1cally

§  Test Driven Development § Validate behavior of new code to be wrijen § Created before developing, turned into regression tests a&er


Func1onal Tes1ng

§  Incarna1ons of func1onal tests §  Unit tests (Junit, CppUnit, Nunit, etc) §  Integra1on tests §  Scripted tests §  Manual tests

§  Ways to create func1onal tests §  Manually wrijen §  Machine generated from applica1on tracing §  Machine generated from capturing traffic §  Machine generated from recording clicks


Regression Tes1ng

§  Contributors to the regression test suite §  Func1onal unit tests that pass § Machine generated unit tests for current behavior §  Unit test wrijen by developers and testers §  Tests wrijen to validate bug fixes

§  Regression test failures §  Unwanted change in behavior – fix the code §  Inten1onal change in specifica1on – update test controls


What’s the Difference?

Unit Tests •  The code is doing things right • Wrijen from programmers perspec1ve •  Confirm code gives proper output based on specific input

Func1onal Tests •  The code is doing the right thing • Wrijen from users perspec1ve •  Confirm the system does what the user expects it to do

Regression Tests •  It s1ll works


Why do Unit Tes1ng

So&ware bugs are inserted during coding phase, but o&en aren’t found un1l func1onal tes1ng, or worse they are found by customers. The longer a bug stays in the system, the more expensive it is to fix it.

Unit tes1ng is a strategy that allows you to find out if your system is behaving

the way you expect early in the SDLC. Unit tes1ng allows for easy regression tes1ng, so you can find out quickly

whether you have introduced some unexpected behavior into the system.

Paraso& Proprietary and Confiden1al 15

The Cost of Defects


Best Prac1ces


What’s Important

§  Unit tests require a LOT of care §  Unit tes1ng is a con1nuous opera1on §  Test suite maintenance is a commitment §  Control data (asser1ons) in the code


Test incorpora1on

§  Take exis1ng unit tests and schedule them into build process

§  Run and fix tests as part of build process §  Ideal goal: 0 failures and 100% code coverage §  Realis1c goal: 10% test failures and 80% code coverage

§  If the unit test suite is noisy, don’t qualify the build based on asser1on failures


Unit Test Crea1on

§ Maintainability §  Proper asser1ons §  Nightly runs §  Tests should be:

§  Automa1c (no human input or review) §  Repeatable (no strange dependencies)


Legacy Test Suites

§  Leverage your exis1ng suite §  Clean up incrementally §  Set policy based on current results §  Tighten policy when in compliance


Maintaining and Crea1ng UT

§ Maintaining is more of a problem than crea1ng.

§  Don’t create a test if you can’t or don’t plan to maintain it.

§  Crea1ng as you go creates a bejer test suite §  Tests not run regularly become irrelevant


Write tests with the code

§  Create the test when you’re wri1ng the code §  Have a test for every task you complete §  Increases understanding §  Improve asser1ons


Connect Unit Tests

§  Associate tests with Code and Requirements §  Naming Conven1ons should have meaning §  Use code comment tags @task @pr @fr §  Use check-‐in comments @task @pr @fr §  Improve change-‐based tes1ng


Specific Genera1on

§  Don’t be lured by 100% coverage §  Auto asser1ons are not meaningful unless validated

§  Review the tests as you work on the code


Code Review Test

§  Code for unit tests should be reviewed §  Consider doing test-‐design review §  Ensures you’re tes1ng what you think you are §  Helps reduce noise §  Helps improve understanding §  Review the code and the test code together


Tes1ng Excep1on Handling

§  Frequently Skipped §  Prevents crashes §  Prevents death spiral §  Rule of thirds

§  1/3 basic func1onality §  1/3 advanced features / infrequent serngs §  1/3 error handling


All About Asser1ons

§  Verifies that the code behaves as intended §  True/false §  Null or not null §  Specific values §  Complex condi1ons


Why Assert?

§  Excuses: § My code already has console output §  I can check it visually/manually

§  Binary output (pass/fail) §  Part of overall test suite §  No human interven1on


Asser1on failures

Introduced coding error • Fix the new code

Func1onality changed • Update the asser1on

Improper asser1on • Comment it out


Asser1on Policy

§  Assign failures as tasks §  Author of the test or of the changed code §  Try to reach developer that has the highest chance to understand the change

§  Developer assesses failure: §  Bug (fix) §  Changed func1onality (update asser1on) §  Bad asser1on (improve or remove) -‐ don’t delete; instead, comment out


What to assert

§  Ease of crea1ng asser1ons leads to poor choices

§  Anything CAN be checked – but should it be? §  Asser1ons should be connected to what you’re trying to test

§  Asser1ons should be logically invariant §  Check values by range, not equivalence §  “What do I expect here logically?”


Asser1on Example: Java

public Integer getNumber() { Scanner input = new Scanner( System.in ); System.out.print( "Enter a number between 0 and 10: " ); int number = input.nextInt(); // assert that the absolute value is >= 0 assert ( number >= 0 && number <= 10 ) : "bad number: " + number; return number

}


How to recognize bad asser1ons

§  How o&en do you WANT to manually verify an asser1on?

§  Daily or near-‐daily checking indicates poor asser1on.

§  Troublesome possibili1es: §  Date §  User or machine name §  OS §  Filesystem names and paths


Missing Asser1ons

§  Error was found a&er code changed §  Test suite never caught the error §  Update unit-‐test with new asser1on § Write new unit tests as necessary


Cleaning It Up


The Challenge: Management’s Perspec1ve

§  Need tests to reduce risks §  Tes1ng takes 1me from

features/fixes §  Not enough 1me §  Not enough people §  Tests and reports lack context


The Challenge: Development’s Perspec1ve

§  Hundreds of failing or broken tests

§  Test requirements unclear §  Exis1ng tests are dubious §  Deadline pressure


Spring Cleaning Solu1on

§  Throw out the useless, old junk §  Iden1fy and eliminate “noisy” tests

§  Organize the remaining chores §  Priori1ze unit test tasks

§  Keep it clean §  Establish and follow a rou1ne test maintenance plan


Throw out the Junk: Iden1fying Noisy Tests

§  Tests that are fragile §  Pass/Fail/Incomplete results frequently change §  Frequent changes are required to keep up-‐to-‐date

§  Tests that are outdated §  More than “n” releases have occurred despite this test failure §  The test applies to legacy code that is frozen §  The test has not been executed for “x” period of 1me

§  Tests that are unclear §  It takes more than 5 minutes to understand why the test exists §  The failure message is incorrect/unhelpful/irrelevant §  There is no associated requirement or defect

§  Be merciless!


Organize Chores: Priori1ze Tasks with Policy

§  All tasks are created equal §  Some tasks are more equal than others §  Policies establish priori1za1on and assignment


Keep it Clean: Establish a Rou1ne Process

§  Management and Development need to co-‐define a Policy §  Clear tes1ng goals that 1e tests to business needs §  Clear delinea1on of responsibility

§  Test crea1on and dele1on §  Test failure resolu1on §  Naming conven1ons §  Peer review

§  Clear and frequent communica1on §  Understand the technical and business impact §  Enhance traceability to project requirements §  Ask ques1ons

§  Focus on safety/quality, not coverage §  Keep the number of tests minimal §  Run all tests on a regularly scheduled, frequent basis §  Clean as you go


Spring Cleaning Recap

§  Find and delete “noisy” tests §  Use policies to priori1ze, assign, and plan §  Co-‐design and follow a process to keep it clean


§  Web §  hjp://www.paraso&.com/jsp/resources

§  Blog §  hjp://alm.paraso&.com

§ Social § Facebook: hjps://www.facebook.com/paraso&corpora1on

§ Twijer: @Paraso& @MustRead4Dev @CodeCurmudgeon

§ LinkedIn: hjp://www.linkedin.com/company/paraso&

§ Google+: +Paraso& +ArthurHickenCodeCurmudgeon § Google+ Community: Development Tes1ng Because its Bejer

Are Your Continuous Tests Too Fragile for Agile?

Software