A WHITE PAPER on Augmented Unit Test Generation September 2014
A WHITE PAPER on Augmented Unit Test Generation
September 2014
© 2014, HCL Technologies. Reproduction Prohibited. This document is protected under Copyright by the Author, all rights reserved.
Abstract ............................................................................................. 3
What are unit tests?.......................................................................... 3
How does automation of unit test case generation help? ............... 4
Approach behind test case automation............................................ 4
Example ........................................................................................... 11
Key benefits of this approach ......................................................... 13
Conclusion ....................................................................................... 13
References ...................................................................................... 14
Author Information ......................................................................... 15
TABLE OF CONTENTS
© 2014, HCL Technologies. Reproduction Prohibited. This document is protected under Copyright by the Author, all rights reserved.
3
Abstract
Unit testing is done by developers who write tests while building
any software. It is a simple but effective idea that improves time-
to-market, flexibility and the quality of any software released in the
market.
Preferably, each developer should produce a set of tests to validate
each individual method and class as they are being written. It takes
time to identify all the scenarios, write tests, run test cases and
monitor the results. Also, writing unit tests by hand is expensive,
labor intensive, tedious, and still leaves the possibility that many of
the execution paths are not tested and remain incomplete. For
beginners there is also the problem of understanding unit tests and
how to write them.
This problem can be solved to a large extent by automation, which
can help developers by generating test cases and scaffolding code
that enables and supports developers to quickly write unit test
cases. But instituting an automated unit testing practice across a
software development team can also be technically challenging and
time consuming.
This paper elaborates a simple approach on how unit test case
generation can be automated, helping the developers in reducing
the amount of engineering hours needed, while at the same time
increasing both completeness of the test coverage and the
software quality.
What are unit tests?
Unit tests are test cases that ensure logic is tested, by invoking
methods with different combinations of input parameter values, to
drive the code through different execution paths, thereby
Writing unit tests by hand is laborious and expensive, and still leaves the possibility that many of the execution paths might remain untested.
Hence, if there is a mechanism to automate large parts of writing unit tests, this helps developers to concentrate more on code development, which is writing code and spending minimum efforts in testing the code, thereby ensuring that there is no compromise in the quality.
© 2014, HCL Technologies. Reproduction Prohibited. This document is protected under Copyright by the Author, all rights reserved.
4
indicating to developers that each piece of code is tested and
ensures code quality..
How does automation of unit test case generation help?
Writing unit tests by hand is laborious, expensive and still leaves
the possibility that many of the possible execution paths remain
untested. To guarantee that as many important execution paths
would be exercised by a test suite, it requires a detailed analysis of
how the input parameters drive the code in various directions. This
is very time consuming and difficult for anything but trivial
functions. While integrating Code Coverage tools with the
development environment has alleviated this problem to some
extent, it still requires the developers to do a lot of probing in order
to write unit tests.
If there is a mechanism to automate large parts of writing unit tests,
this will help developers to concentrate more on the crux of
development - which is writing code - and they would exert the
minimum effort in testing the code, thereby ensuring that there is
no compromise in quality.
Approach behind test case automation
Automating test cases for each method inside classes or on test
suites is simply not an easy task. There should be a methodology
that defines the way, starting from parsing the information, finding
the execution paths, and creating test cases according to the
conditions to be tested.
Broadly, the approach that will clearly help in complementing the
efforts, and in creating a more exhaustive set of unit tests with
much less manual effort can be described as follows:
© 2014, HCL Technologies. Reproduction Prohibited. This document is protected under Copyright by the Author, all rights reserved.
5
1. Determine the number of test cases to be generated
(Cyclomatic Complexity is a good place to start)
2. Find test data that can enable a test scenario to be fulfilled
for each test case
3. Generate test case along with test data
1. Determine the number of test cases to be generated
Consider a file containing “n” number of methods. The number of
test cases that are to be generated for each method depends upon
the cyclomatic complexity of that particular method.
Cyclomatic complexity overview
Cyclomatic complexity is a static software metric pioneered in the
1970s by Thomas McCabe. The cyclomatic complexity number
(CCN) is a measure of the number of paths there are in a method. It
serves as a rough measure of code complexity and as a count of the
minimum number of test cases that are required to achieve full
code-coverage of the method.
The equation for calculating the cyclomatic complexity number
comes from the theory of graphs where it refers to the number of
paths from any point in a topological space to any other point. It is
expressed as:
Where, E represents the number of edges on the graph, N the
number of nodes on the graph, and P the number of connected
components. In programming terms, E represents the code
executed as a result of a decision, N is the number of decision
CCN = E - N + P
© 2014, HCL Technologies. Reproduction Prohibited. This document is protected under Copyright by the Author, all rights reserved.
6
points (conditional statements) and P is the number of ways to exit
the program. As a software metric it can be expressed as:
Formula behind the generation of test cases
As the cyclomatic complexity of a method is fundamentally the
number of paths contained in that method, general rule of thumb
states that in order to ensure a high level of test coverage, the
number of test cases for a method should equal the method's
cyclomatic complexity.
2. Determine the test data for each test case
Once the number of test cases has been identified, one has to
identify what type of test case scenarios are to be handled inside
each test case method respectively.
The following steps are applicable while calculating test data:
Identify the code paths - which means it can be conditional,
branching or looping statements
For each path traversal in a program we can have test
cases (satisfying respective true and false conditions)
For a sample logic flow as shown below:
CCN = "The number of decision points" + 1
© 2014, HCL Technologies. Reproduction Prohibited. This document is protected under Copyright by the Author, all rights reserved.
7
The number of test cases to be generated should be 4, since CCN
=4
Then test scenarios covered under each test case are as follows:
1. Path 1: 1,2,3,4,5,7,8,10,11
2. Path 2: 1,2,3,4,6,10,11
3. Path 3: 1,2,3,4,5,7,8,10,11
4. Path 4: 1,2,3,4,5,7,9,10,3,4,6,10,11
Determine the
variables that are present in
the code paths whose value
determine the true or false of
the code path. Variables can
be inferred in some cases
where there is actually no
need to define a variable but
are return values of method
invocations.
Identify the type of variable (can be basic data type like
integer, floating point, double, byte, etc. or complex data
types that can be classes by themselves)
Identify the scope of the variables. Variables are typically of
various scopes:
© 2014, HCL Technologies. Reproduction Prohibited. This document is protected under Copyright by the Author, all rights reserved.
8
o Method Scope – Variables that act as input
variables to the method under test
o Class Scope/Global Scope – Variables that are set
elsewhere (either using setter methods or by direct
assignment, but populated outside the scope of
the method under test)
o Computed Scope – Inferred Variables that are a
result (return types) of method invocations.
How can initialization or mocking of test data values be
done?
The following are the factors on which initialization or mocking of
test data values is done for each test case, covering its test scenario
1. Type
Depending upon the type of the variable, these are the two
classifications
1. Basic data type
2. Complex data type (Object references of some classes)
Basic data type handling
If a particular variable is a basic data type, then test data for it will
be determined based on the condition and the operand given in
the conditional or branching or looping statements.
© 2014, HCL Technologies. Reproduction Prohibited. This document is protected under Copyright by the Author, all rights reserved.
9
For e.g., if the condition inside the loop is i<8, operand is “<”, then
for the generation of a true case scenario the value of “i” can be
initialized to 7 and can be “9” for a false case.
Non basic data type handling
If the variable present in the code path is non basic, i.e. reference
of some class, then these types of variables cannot be initialized as
straightforward as that of basic types. There comes a need for
object mocking for those reference classes and returning the values
based on the condition.
What is the need for mocking?
When there are objects or class references, there comes a need to
keep dependencies out of the unit test. Mocking helps in achieving
this.
For fetching any data from a database, one would normally need a
dependency to the database to do this. However, with object
mocking, one can simulate the interaction with the database with
a mock framework, so it might return a particular dataset which
looks like the one returned from the database and can then test
the code to ensure that it handles translating a dataset to a user
object, rather than using it to test that a connection to the
database exists.
When there are objects or class references there comes a need to keep dependencies out of the unit test. Mocking helps in achieving this.
To fetch any data from the database one would normally need a dependency on the database to do this.
However, with object mocking, one can simulate the interaction with the database with a mock framework, so it might return a particular dataset which looks like the one returned from the database.
© 2014, HCL Technologies. Reproduction Prohibited. This document is protected under Copyright by the Author, all rights reserved.
10
Many mocking frameworks exist for different languages. In the Java
world, mocking frameworks like EasyMock and jMock are available
and in the .NET world, frameworks like NMock, EasyMock.NET, and
Moq are available.
2. Scope
Depending upon the scope of the variable, the test data
determined for that particular test case will vary, such as:
Method Scope
The initialized or mocked values for method variables are passed to
the test method by passing as the method parameter.
Class Scope/Global Scope
The initialized or mocked values for these types of variables are
passed to the test method by calling its setter or assignment
statement.
Computed Scope
Mocking should be carried on for all the dependent classes before
the test method call happens.
3. Generation of test cases with test data
Once the data along with test cases has been determined, the next
step is to generate the test cases with their respective test data.
Broadly, the steps needed for generation are as follows:
1. Deciding on the testing framework that is to be used for
writing unit test cases
Depending upon the scope of the variable, the test data determined for that particular test case will vary
Method Scope Class Scope/Global
Scope Computed Scope
© 2014, HCL Technologies. Reproduction Prohibited. This document is protected under Copyright by the Author, all rights reserved.
11
2. Based on the chosen framework, following the naming
conventions for defining class names and method names
for the test cases
3. Configuring the initial set up that is needed for any test
case to run
4. With the information obtained about the number of test
cases to be generated for a file containing n number of
methods along with test data information, generate test
cases along with test data
5. In cases where the generation cannot be complete, provide
sufficient template code and test scaffolding (example
object initialization and method invocation, handling
failures, handling success, etc.) for developers to complete
test cases along with sufficient documentation on the
scenario to be tested.
The objective of generating test cases is not to completely
automate all test scenarios. It might be possible that certain test
cases cannot be generated for various reasons. The objective of
automation in such scenarios is to accelerate the time taken to
write the test cases by creating a test scaffold, which will provide
the developer with all the necessary code to write the test cases
but still allowing the developer to fill in details wherever required.
Example
If the above procedure is applied to a language like Java, the
process of augumented unit test case generation pans out as
below:
Frameworks:
Testing framework – Junit 4.x
Mocking framework – EasyMock
© 2014, HCL Technologies. Reproduction Prohibited. This document is protected under Copyright by the Author, all rights reserved.
12
Eclipse modelling framework - Eclipse JDT (for
code parsing and generating)
A sample Java class
© 2014, HCL Technologies. Reproduction Prohibited. This document is protected under Copyright by the Author, all rights reserved.
13
A sample of generated test case for the above Java class
Key benefits of this approach
Ensures methodical and reliable test case coverage Helps in
covering all code paths even in case of high complexity
Reduces the maximum time effort spent by developers in
identifying the code path and how conditions are to be
handled
Helps in delivering good quality software on starting
Helps developers who are not exposed to unit testing
Accelerates software development speed
Conclusion
This paper discussed the approach towards augumenting unit test
case generation in order to ensure that methods are tested
sufficiently as well as provide sufficient guidance to developers to
write unit test cases.
© 2014, HCL Technologies. Reproduction Prohibited. This document is protected under Copyright by the Author, all rights reserved.
14
By enabling and guiding developers in understanding scenarios to
test, generating scaffolding code to help complete testing scenarios,
and by generating complete test cases wherever possible, the
approach ensures code quality and a shorter time for development
across projects.
References
Improved Software Development by Unit Test Automation
Why Unit Testing?
Code Metrics: Cyclomatic Complexity and Unit Tests
Unit_testing
© 2014, HCL Technologies. Reproduction Prohibited. This document is protected under Copyright by the Author, all rights reserved.
15
Author Info
Saranya Duraisamy is working with the ERS Practice Team as a Senior Software Engineer. She has over 2 years of software experience in developing J2E based applications and code generation tools.
Rajesh Venkatesan is a Solutions Architect with the ERS Practice team and has over a decade of experience in the development of large scale distributed systems and multi tenant cloud based architectures for HCL customers, and has filed multiple patents in this space. He is also responsible for building HCL IPs that impact the way software is built.
Hello, I'm from HCL's Engineering and R&D Services. We enable technology led organizations to go to market with innovative products and solutions. We partner with our customers in building world class products and creating associated solution delivery ecosystems to help bring market leadership. We develop engineering products, solutions and platforms across Aerospace and Defense, Automotive, Consumer Electronics, Software, Online, Industrial Manufacturing, Medical Devices, Networking & Telecom, Office Automation, Semiconductor and Servers & Storage for our customers. For more details contact: [email protected] Follow us on twitter: http://twitter.com/hclers Our blog: http://www.hcltech.com/blogs/engineering-and-rd-services Visit our website: http://www.hcltech.com/engineering-rd-services