03 Unit Testing

8/12/2019 03 Unit Testing

1/12

Unit testingis a method by which individual units ofsource codecan test.

How do you perform unit tests?

Unit testing simply verifies that individual units of code (mostly functions) work as expected. Usually

you write the test cases yourself, but some can be automatically generated.

The output from a test can be as simple as a console output, to a "green light"in a GUI such

asNUnit,or a different language-specific framework.

Performing unit tests is designed to be simple, generally the tests are written in the form of functions

that will determine whether a returned value equals the value you were expecting when you wrote

the function (or the value you will expectwhen you eventuallywrite it - this is calledTest Driven

Developmentwhen you write the tests first).

When should you perform unit tests?

They should be done as often as possible. When you are performing tests as part of thedevelopment process, your code is automatically going to be designed better than if you just wrote

the functions and then moved on. Also, concepts such asDependency Injectionare going to evolve

naturally into your code.

The most obvious benefit is knowing down the road that when a change is made, no other individual

units of code were affected by it if they all pass the tests.

What Is Unit Testing?

Before we get started setting up our environment and writing any code, let's defineexactly what unit testing is, why it's worth doing, and how to get started in incorporating

it in our projects.

At a high-level, unit testing refers to the practice of testing certain functions and areas

or unitsof our code. This gives us the ability to verify that our functions work as

expected. That is to say that for any function and given a set of inputs, we can

determine if the function is returning the proper values and will gracefully handle failures

during the course of execution should invalid input be provided.

Ultimately, this helps us to identify failures in our algorithms and/or logic to help improve

the quality of the code that composes a certain function. As you begin to write more and

more tests, you end up creating a suite of tests that you can run at any time during

development to continually verify the quality of your work.

8/12/2019 03 Unit Testing

2/12

A second advantage to approaching development from a unit testing perspective is that

you'll likely be writing code that is easy to test. Since unit testing requires that your code

be easily testable, it means that your code must support this particular type of

evaluation. As such, you're more likely to have a higher number of smaller, more

focused functions that provide a single operation on a set of data rather than large

functions performing a number of different operations.

A third advantage for writing solid unit tests and well-tested code is that you can prevent

future changes from breaking functionality. Since you're testing your code as you

introduce your functionality, you're going to begin developing a suite of test cases that

can be run each time you work on your logic. When a failure happens, you know that

you have something to address.

Of course, this comes at the expense of investing time to write a suite of tests early in

development, but as the project grows you can simply run the tests that you've

developed to ensure that existing functionality isn't broken when new functionality is

introduced.

What's the difference between unit tests and integration

tests?A unit test is a test written by the programmer to verify that a relatively small piece of code is doing

what it is intended to do. They are narrow in scope, they should be easy to write and execute, and

their effectiveness depends on what the programmer considers to be useful. The tests are intended

for the use of the programmer, they are not directly useful to anybody else, though, if they do their

job, testers and users downstream should benefit from seeing less bugs.

An integration test is done to demonstrate that different pieces of the system work together.

Integration tests cover whole applications, and they require much more effort to put together. They

usually require resources like database instances and hardware to be allocated for them. The

integration tests do a more convincing job of demonstrating the system works (especially to non-

programmers) than a set of unit tests can, at least to the extent the integration test environmentresembles production.

Actually "integration test" gets used for a wide variety of things, from full-on system tests against an

environment made to resemble production to any test that uses a resource (like a database or

queue) that isn't mocked out.

8/12/2019 03 Unit Testing

3/12

What is integrated testing in software testingDefinition of integration testing

Integration testing is the testing process insoftware testingto verify that when two or more

modules are interact and produced result satisfies with its original functional requirement ornot. Integrated testingis fall in Black box testing. Integrated testingwill start after completion

of unit testing. Software testingengineer is performing integration testing.

Why integrated testing require?

Integration testingfinds thebugsthat occur when two or more models integrated. Main purpose

ofIntegration testingis to identifying the functional, requirement and performance levelbugs.

When modules not integrated, they perform as per requirement but when they integrated,

functional, requirement and performance related issues will occurs due to the integration.

There are three different types of integration testingapproach in software testing.

1. Big Bang

2. Top down

3. Bottom up

1. Big Bang

Big Bang Integration testingapproach used to find the bugs when all the developed modules are

interacted with each other and create a complete software system then its produced result satisfying

with original requirement.

2. Top down

In Top down integrated testingapproach, all Top level integrated modules are tested first and its

sub modules tested from top to down step by step.

3. Bottom up

In Bottom up integrated testingapproach, all bottom (Sub Modules) level integrated sub modules

are tested first and its main modules tested from bottom to up step by step.

8/12/2019 03 Unit Testing

4/12

What Is Functional Testing Explain It with Example?Functional testingmeans testing the application against business requirements. Functional testing is

executed using the functional specifications given by the client or by the design specifications according

to use cases given by the design team. Role of functional testing is tovalidatingthe behavior of

an application.

Functional testingis more important because it always verifies that your system is fixed for release. The

functional tests define your working system in a useful manner. In functional testing tester has to

validate the application to see that all specified requirements of the client whatever we have said in SRS

or BRS have been incorporated or not.

Functional testingis always concentrating on customer requirements and whereas the Non-Functional

testing is always concentrating on customer expectations.

Functional and Non Functional Test Cases

Functional test cases target business goals and non functional test cases target performance, resource

utilization, usability, compatibility etc. Functional testing is a part of system testing.

Example of functional testing is explained below

Considering example if you are functionally testing a word processing application, a partial list of checks

you would perform minimally includes creating, saving, editing, spell checking and printing documents.

Types of Functional Testing

Functional testingfalls in to two categories:

1. Positive functional testing: - This testing carry exercising the applications functions with valid input

and also verifying that the outputs are correct.

Example:-

Again continuing with the word processing example, a positive test for the printing function might be to

print a document containing both text as well as graphics to a printer that is online, filled with paper and

for which the correct drivers are installed.

2. Negativefunctional testing: - This testing involves exercising application functionality using a

combination of invalid inputs, some unexpected operating conditions and by some other out-of-

bounds scenarios.

Example:-

Again continuing with the word processing example, a negative test for the printing function might be to

disconnect the printer from the computers while a document is printing.

What probably should happen in these scenarios are a plain-English error message displayed, informing

the user what happened and instructing him/her on how to fix the problem.

8/12/2019 03 Unit Testing

5/12

Conclusion:

At last we conclude that in functional testingfunctionality of the module is tested and structure is not

considered. It is performed, based on user's perspective. These tests ensure that the system does what

users are expecting it to do.

This type of testing means testing the functionality example include input the proper data and checking

the output as per the requirement documents.

Why code coverage analysis?

Code coverage is used in software testing, as it measures the quality of your test procedures, by describing the

degree to which the source code of a program has been tested.

All embedded developers are used to debugging; the process used to understand the behavior of a program such

that an error can be corrected. Debugging however, requires that an error has been detected in the first place.

Programming errors that you do not know exist cannot be debugged or corrected.

Code coverage analysis is a powerful tool for finding more of the bugs that most likely do exist in your software

whether you are aware of them or not. As such, a code coverage analysis tool can help you find more programming

errors, which enables you to release a software product of better quality.

More technically, code coverage analysis finds areas in your program that is not covered by your test cases, enabling

you to create additional tests that cover otherwise untested parts of your program. It is thus important to understand

that code coverage helps you understand the quality of your test procedures, not the quality of the code itself.

There are many types of code coverage of different strengths, such as statement or block coverage, function and

function call coverage, branch coverage, Modified condition/Decision coverage (MC/DC), etc.

Read more on different types of code coverage analysis here.

Atollic TrueANALYZER automate these work tasks by performing advanced code coverage analysis of your

application as it runs in your target hardware.

Types of code coverage analysis

Code coverage analysis is used to measure the quality of software testing, usually using dynamic execution flow

analysis.

There are many different types ofcode coverageanalysis, some very basic and others that are very rigorous and

complicated to perform without advanced tool support. To explain how the different types of code coverage analysis

works, let's consider the trivial code example below:

8/12/2019 03 Unit Testing

6/12

As we can see, this code section contains a red code block that is always executed, a green code block that is

sometimes executed (dependent on the result of the if-statement), and a blue code block that is always executed.

This can be visualised as an execution flow graph.

The execution flow graph above shows that this trivial code section contains two different execution paths:

One execution path runs the red code block and then jumps directly to the blue code block and runs that too (the if -

statement evaluates to false).

The other execution path runs the red code block, jumps to the green code block and runs that, and finally, it jumps to

the blue code block and run that too (the if-statement evaluates to true).

Statement coverage and Block coverage

This very basic type ofcode coverageanalysis is sometimes included in embedded debuggers. Statement coverage

can only report whether a statement has been executed or not, which potentially leaves many types of code

constructs (in particular execution branches) untested. A variant of Statement coverage is Block coverage, that

provides the same measurement based on code blocks instead of statements. Since every code block have a known

set of statements, there is a one-to-one relationship between statement coverage and block coverage from a test

quality point of view.

8/12/2019 03 Unit Testing

7/12

Function coverage

This basic type ofcode coverageanalysis is sometimes included in standard debuggers too. Function coverage can

only report whether a function has been called or not, it does not say anything about what was executed inside it, or

how or why the function was called. And it does not measure how many of the function calls that are made.

Function call coverage

Function call coverage measures how many of the available function calls have actually been made. For example, a

code section might make many function calls. To reach 100% Function call coverage, every function call in the code

section must be executed at least one time. If a call to a function is made, it is implied that function coverage is

fulfilled for that function too (if it is called, it is executed too).

Branch coverage

Branch coverage is an advanced type of code coverage, that requires that all code blocks and all execution paths

have been taken. As such it builds on top of statement or block coverage, adding more advanced requirements.

8/12/2019 03 Unit Testing

8/12

To fulfill Branch coverage for the code example above, the code must be executed minimum two times; one time with

the if-statement evaluating to false, and another time with the if-statement evaluating to true. Only then have all the

code blocks and all the execution paths been tested.

Also note that Branch coverage only considers the overall branch decision in the if-statement (i.e. that both if-true and

if-false are tested). But it does not consider how the expression evaluated to true or false, i.e. it does not consider the

subexpressions forming the overall branch decision.

Modified condition/decision coverage

Modified condition/decision coverage (MC/DC)is a very advanced type ofcode coverageanalysis. It builds on top of

Branch coverage, and as such, it too requires that all code blocks and all execution paths has been tested.

But it adds the requirement that all sub-expressions in complex branch expressions must be considered too. MC/DC

requires that all sub-expressions have been shown to, independently of other sub-expressions, drive the overall

branch decision.

8/12/2019 03 Unit Testing

9/12

Effectively, this means that the code section above must be executed many times, such that all subexpressions have

been the tested to independently of the others, be the driving decision factor in the overall branch decision.

Safety-critical software is often required to be tested such that it fulfillsMC/DC-levelcode coverageanalysis

successfully. RTCA DO-178B for example, requiresMC/DC-level testing of airborne software of "Level A" criticality,

which is the most safety-critical part of airborne software, such as the flight control or avionics system.

But any software project benefit from rigorouscode coverageanalysis, such as:

Companies who want to avoid bad reputation on the market by releasing products of poor quality

Products that are very difficult or expensive to upgrade in the field

Products that are produced in very high volume

Safety-critical or semi-safety-critical products

Summary

Atollic TrueANALYZER is a very powerful tool for code coverage analysis, as it performs test quality measurements

using dynamic execution flow analysis of your application as it runs in your target board.

Atollic TrueANALYZER performs Block coverage, Function coverage, Function call coverage, Branch coverage as

well as Modified condition/Decision coverage (MC/DC). The tool us super-simple to use, as it only requires two

mouse clicks to perform a full MC/DC test in your target board!

What is test stopping criteria and how do you arrive at it ?

The criteria required to 'stop testing' is determined by many factors. When a test plan is created it is

critical to define the exit criteria. Following are some of the decisions we take to explicitly stop testing

during the testing phase.

8/12/2019 03 Unit Testing

10/12

Constraint of time and budget

Time and budget have a critical role in deciding when to stop testing. Testing will be stopped when

the budget is exhausted. Some projects are time bound and postponing the release date will have a

critical impact. Hence testing will also be stopped when the timeline is exhausted.

Execution of a predefined base set

In this case, there will be a predefined base set of test cases that can ensure that all the critical

areas have been tested. So testing will be stopped when these cases are executed successfully.

Testing till the number of failed test cases drops below a threshold/ All the critical

issues are fixed

Testing will be stopped when the failure rate drops below a certain threshold which is predefined.

Sometimes testing will be stopped when all the critical bugs are fixed. Cosmetic bugs will be fixed

during the coming releases.

For a testing team, this will be a challenging situation. So if you are a tester in any of the above

situations, you can act accordingly and decrease the volume of testing.

Regression Testing ?

The selective retesting of asoftwaresystem that has been modified to ensure that anybugshave been fixed and that

no other previously working functions have failed as a result of the reparations and that newly added features have

not created problems with previous versions of the software. Also referred to as verification testing, regression testing

is initiated after aprogrammerhas attempted to fix a recognized problem or has addedsource codeto a program that

may have inadvertently introduced errors. It is a quality control measure to ensure that the newly modified code still

complies with its specified requirements and that unmodified code has not been affected by the maintenance activity.

Regression testing is the process of testing changes to computer programs to make sure that the older

programming still works with the new changes. Regression testing is a normal part of the program

development process and, in larger companies, is done by code testing specialists. Test department

coders develop code test scenarios and exercises that will test new units of code after they have been

written. These test cases form what becomes thetest bucket. Before a new version of a software

product is released, the old test cases are run against the new version to make sure that all the old

capabilities still work. The reason they might not work is because changing or adding new code to a

program can easily introduce errors into code that is not intended to be changed.

Introduction

Whenever developers change or modify their software, even a small tweak can have unexpected

consequences. Testing existing software applications to make sure that a change or addition hasnt

8/12/2019 03 Unit Testing

11/12

broken any existing functionality is called regression testing. Its purpose is to catch bugs that may have

been accidentally introduced into a new build or release candidate, and to ensure that previously

eradicated bugs continue to stay dead. By re-running testing scenarios that were originally scripted when

known problems were first fixed, you can make sure that any new changes to an application havent

resulted in a regression, or caused components that formerly worked to fail. Such tests can be performed

manually on small projects, but in most cases repeating a suite of tests each time an update is made istoo time-consuming and complicated to consider, so automated testing is typically required.

Understanding Regression Tests

Some software development teams try to get by without performing regular regression tests, opting to test

essential functions just once to make sure they work and, if they check out, proceeding with the hopeful

assumption that those functions will still work unless theyre directly modified again. In a way, this makes

sense: its natural to want to simply make a change, test it, and move on. Performing functional tests or

highly specific unit tests to determine that a new software component works as it should has been called

non-regression testingby Doug Hoffman and others. But it can be relatively easy to find a specific

problem when youre looking for it; whats harder is catching all the ones you dont expect.

Again, its important for developers and testers to always bear in mind that even small, seemingly

insignificant alterations to an applications source code can ripple outward in surprising ways, breaking

functions that seem completely unrelated to the new modification. When you run regression tests, youre

checking to make sure that your modification not only behaves as you want it to, but that it also ha snt

inadvertently caused problems in functions that had otherwise worked correctly when previously tested.

Fortunately for the would-be regression tester, on any given project your regression test libraries can be

built from the existing test cases developed from day one. Functional tests, unit tests, integration tests,

and build verification testsanything that has successfully verified, throughout the development process,

that various components work as intendedcan all be incorporated into a regression testing suite, and

regression tests, per se, dont necessarily need to be written. Each time you modify your source code,

you can simply re-run the potentially relevant tests to ensure that they continue to pass. Naturally, over

the course of a complex development project, those test casesand the various functions and processes

that they attempt to checkcan number in the thousands, making the use of automated testing software

mandatory for full-scale regression tests.

To reduce the stress levels of programming teams everywhere, various automated testing programs that

specialize in regression tests now make it relatively easy, with a few clicks of a mouse, to establish sets of

testing parameters and to check new iterations of code against previous software baselines, or control

states, highlighting inconsistencies in testing logs and specifying exactly where an unexpected function

broke and why. You may not have the bandwidth or time to let your software re-runeverytest, checking

the entire application for potential errors, but youll dramatically exceed what youd be able to checkmanually. Indeed, when it comes to automated regression-testing tools, sometimes it almost seems too

easy.

And if youre not careful, that can be a problem.

8/12/2019 03 Unit Testing

12/12

The Limitations of Automation

As with most forms of automated testing, setting a regression-testing program on autopilot is not a

surefire solution, and some conscious oversight and input is generally still needed to ensure that your

tests catch all the bugs they should. When you have the exact same suite of tests running repeatedly,

night after night, the testing process itself can become static. Over time, developers may learn how to

pass a fixed library of tests, and then your standard array of regression tests can inadvertently end up not

testing much of anything at all.

In some ways, making sure that your software continues to adhere to requirements specifications as you

develop it is like clearing a path through a minefield. To proceed safely through enemy territory,

professional minesweepers dont need to clear the entire field; they just need to clear a single path, set up

defined boundaries, and guard that path. Once enough soldiers are there guarding itsoldiers who, in

this analogy, represent your regression testsits unlikely that anyones going to slip a new mine onto

it. However, this continually clear path says nothing of all the other mines that might still be out there, old

or new, just waiting for aimless civilians to step on them.

Likewise, if your regression testing becomes too automated and rote, the whole point of doing it can

backfire. You can end up guaranteeing a clear software-development trajectory for yourself and your dev

team while unwittingly ignoring vast swaths of the application, letting your end users stumble upon

undetected glitches at their own peril. Walking along a single path of least resistance is, of course, easier

than stopping to sweep the entire application after each new step, but its worth the effort to take your

regression testing all the way by frequently scanning a little further afield. And that often just means

complementing your automation with some good old-fashioned manual tests.

The Future of Regression TestingFor regression testing to be effective, it needs to be seen as one part of a comprehensive testing

methodology that is cost-effective and efficient while still incorporating enough varietysuch as well-

designed frontend UI automated tests alongside targeted unit testing, based on smart risk prioritization

to prevent any aspects of your software applications from going unchecked. These days,

manyAgilework environments employing workflow practices such as XP (Extreme Programming), RUP

(Rational Unified Process), or Scrum appreciate regression testing as an essential aspect of a dynamic,

iterative development and deployment schedule.

But no matter what software development and quality-assurance process your organization uses, if you

take the time to put in enough careful planning up front, crafting a clear and diverse testing strategy withautomated regression testing at its core, you can help prevent projects from going over budget, keep your

team on track, and, most importantly, prevent unexpected bugs from damaging your products and your

companys bottom line.