What Do the Asserts in a Unit Test Tell Us About Code Quality? (CSMR2013)

What Do the Asserts in a Unit Test Tell us About Code Quality?Mauricio AnicheGustavo OlivaMarco Gerosa

University of São Paulo (USP)

IME-USP

Unit Tests and Code Quality Great synergy between a testable class

and a well-designed class (Feathers, 2012) The write of unit tests can become

complex as the interactions between software components grow out of control (McGregor, 2001)

Agile practitioners state that unit tests are a way to validate and improve class design (Beck et al, 2001)

Unit Tests and Asserts Every unit test contains three parts

Set up the scenario Invoke the behavior under test Validates the expected output

Assert instructions assertEquals (expected, calculated); assertTrue(), assertFalse(), and so on

No limits for the number of asserts per test

A little piece of codeclass InvoiceTest { @Test public void shouldCalculateTaxes() { // (i) setting up the scenario Invoice inv = new Invoice(5000.0);

// (ii) invoking the behavior double tax = inv.calculateTaxes();

// (iii) validating the output assertEquals (5000 0.06 , tax ); ∗ } }

Why would… … a test contain more than one assert? Is it a smell of bad code/design?

Research Design We selected 22 projects

19 from ASF 3 from a Brazilian consultancy

Data extraction from all projects Code metrics

Statistical Test Qualitative Analysis

Hypotheses H1: Code tested by only one assert does not

present lower cyclomatic complexity than code tested by more than one assert

H2: Code tested by only one assert does not present fewer lines of code than code tested by more than one assert

H3: Code tested by only one assert does not present lower quantity of method invocations than code tested by more than one assert

Data Extraction Test code

Number of asserts per test Production method being tested

Production code Cyclomatic Complexity (McCabe, 1976) Number of method invocations (Li and

Henry, 1993) Lines of Code

Heuristic to Extract the Production Method

class InvoiceTest { @Test public void shouldCalculateTaxes() { // (i) setting up the scenario Invoice inv = new Invoice(5000.0);

// (ii) invoking the behavior double tax = inv.calculateTaxes();

// (iii) validating the output assertEquals (5000 0.06 , tax ); ∗ }}

class Invoice { public double calculateTaxes() { // something… } }

Selected Projects Pre-selected all 213 projects from

Apache Eliminated those because of:

Minimum test ratio (unit tests / production methods >= 0.2)

Success in production method detection algorithm (higher than 50% in all unit tests that contain more than 1 assert)

Selected Projects

Asserts Distribution in Selected Projects

Statistical Test Separation in two different groups

The same method can appear in both groups

No duplicates Wilcoxon Signed-Rank Test

Results of the Test

Why more than 1 assert? 130 tests randomly selected Qualitative analysis:

More than one assert for the same object (40.4%)

Different inputs to the same method (38.9%) List/Array (9.9%) Others (6.8%) Extra assert to check if object is not null

(3.8%)

“Asserted Objects” We coined the term “asserted objects”

It counts not the number of asserts in a unit test, but the number of different instances of objects that are being asserted

assertEquals(10, obj.getA());assertEquals(20, obj.getB());

Counts as 1 “asserted object”

Distribution of Asserted Objects

Results of the Test

Summary of the Tests

Findings Counting the number of asserts in a unit test

does not give valuable feedback about code quality But counting the number of asserted objects may

provide useful information However, the difference between both groups was

not “big” A possible explanation:

Methods that contain higher CC, lines of code, and method invocations contains many different paths, and developers prefer to write all of it in a single unit test, rather than splitting in many of them

Threats to Validity Many unit tests variations that are not

supported by our tool The production method detection

algorithm Before/After methods The projects’ filtering process Integration and system tests

Related Work Most similar to our work: Bruntink and

van Deursen (2006). Significant correlation between class

metrics (fan out, size of a class, response for a class) and test metrics (size of the class, number of asserts).

Conclusions The number of asserts does not tell us

anything about the production code quality But the number of asserted objects does. “More than one asserted object per test”

smell The need of replicating the study with a

larger amount of industry projects

Contact Information Mauricio Aniche

aniche@ime.usp.br / @mauricioaniche Gustavo Oliva

goliva@ime.usp.br / @golivax Marco Gerosa

gerosa@ime.usp.br

Software Engineering & Collaborative Systems Research Lab (LAPESSC)http://lapessc.ime.usp.br/