SOFTWARE TEST – fra ”øv!” til ”yes!” IDA Embedded 07.04.2015 Troels Fedder Jensen, ASE [email protected]
SOFTWARE TEST –
fra ”øv!” til ”yes!”
IDA Embedded 07.04.2015
Troels Fedder Jensen, ASE
-2-
Today’s menu
• Introduction
• What and why?
• Unit tests and frameworks
• Dependencies – design for test
• State and interaction-based tests
• Isolation frameworks
• Test quality, test design
• Continuous integration
• (TDD)
-3-
What is software test? Or what should it be?
A systematic and objective verification of software’s state and behavior, based on objective and predefined criteria.
-4-
Software test activities – the V model
System specification
System design
Component design
Integration testing
System testing
Accept testing Requirements
Implement component
Unit test
-5-
Why test software?
Increased product quality
(More) predictable development
Increased customer satisfaction
Ensure easy code refactoring/expansion
Lower overall cost
Drives ”nice” design
Reassurance
Measure of progress
-7-
Specification
Design
Implementation
Test
Tests provide feedback Feedback drives improvements
Why test software?
-8-
Lots of tests provide lots of feedback Lots of feedback drives lots of improvements
Specification
Design
Implementation
Test
Why test software?
-9-
Specification
Design
Implementation
Test
Specification
Design
Implementation
Test
Increased product quality
(More) predictable development
Increased customer satisfaction
Reassurement
Eases code refactoring/expansion
Lower overall cost Drives ”nice” design
Instant confirmation
Measure of progress
Unit tests &
Unit test frameworks
System specification
System design
Component design
Integration testing
System testing
Accept testing Requirements
Implement component
Unit test
-12-
Definition of a unit and unit test
• A unit test consists of a number of individual test cases for a ”suitably” small, coherent collection of software
• Test cases are usually organized in test fixtures
• Each test case asserts the correct function of a single aspect of the unit
• Test cases should be fast, automated, repeatable, permanent (and a whole bunch of other things)
-13-
Example: The Circle class
// An application class
class Circle
{
private double _radius;
public Coordinate Center { get; set; }
public Circle(double radius)
{
(_radius < 0.0) throw new ArgumentException();
_radius=radius;
}
public double getArea() { return _radius*_radius*PI; }
public double getCircumference() { return 2*PI*_radius; }
}
-14-
A hand-coded unit test of Circle
// Test fixture for the Circle class
public class CircleUnitTest
{
private Circle uut; // The Unit-Under-Test (UUT)
// Test setup
public void setupTest() { uut = new Circle(3.0); }
// Test cases
public bool testArea() { return (uut.getArea() == 3.0*3.0*PI); }
public bool testCircumference() { return (uut.getCircumference() == 2 *PI*3.0); }
...
// Test runner
public bool runTests()
{
setupTests();
bool testResult = true;
// Collect result from test cases
testResult &= testArea();
testResult &= testCircumference();
...
// Report test result
return testResult;
}
}
-15-
Requirements for a UT framework
googletest
2.1 ABAP 2.2 ActionScript / Adobe Flex 2.3 Ada 2.4 AppleScript 2.5 ASCET 2.6 ASP 2.7 BPEL 2.8 C 2.9 C# 2.10 C++ 2.11 Cg 2.12 CFML (ColdFusion) 2.13 Clojure 2.14 Cobol 2.15 Common Lisp 2.16 Curl 2.17 Delphi 2.18 Emacs Lisp 2.19 Erlang 2.20 Fortran 2.21 F# 2.22 Groovy 2.23 Genexus 2.24 Haskell 2.25 Haxe 2.26 HLSL 2.27 ITT IDL 2.28 Internet 2.29 Java 2.30 JavaScript 2.31 Lasso 2.32 LaTeX 2.33 LabVIEW 2.34 LISP 2.35 Logtalk 2.36 Lua 2.37 MATLAB 2.38 .NET programming languages 2.39 Objective-C 2.40 OCaml 2.41 Object Pascal (Free Pascal) 2.42 PegaRULES Process Commander 2.43 Perl 2.44 PHP 2.45 PowerBuilder 2.46 Progress 4GL 2.47 Prolog 2.48 Python 2.49 R programming language 2.50 Racket 2.51 REALbasic 2.52 Rebol 2.53 RPG 2.54 Ruby 2.55 SAS 2.56 Scala 2.57 Scilab 2.58 Scheme 2.59 Shell 2.60 Simulink 2.61 Smalltalk 2.62 SQL and Database Procedural Languages 2.62.1 SQL 2.62.2 MySQL 2.62.3 PL/SQL 2.62.4 IBM DB2 SQL-PL 2.62.5 PostgreSQL 2.62.6 Transact-SQL 2.63 Swift 2.64 SystemVerilog 2.65 TargetLink 2.66 Tcl 2.67 TinyOS/nesC 2.68 TypeScript 2.69 Visual FoxPro 2.70 Visual Basic (VB6.0) 2.71 Visual Lisp 2.72 XML 2.73 XSLT 2.74 Other
-16-
NUnit sample test cases
[TestFixture] public class CircleUnitTests { public Circle _uut; // Unit-under-test [SetUp] public void SetUp() { _uut = new Circle(3.0); } [Test] public void Circle_GetArea_CorrectAreaCalculated() { Assert.That(uut.GetArea(), Is.EqualTo(3.0 * 3.0 * PI); } [Test] public void Circle_GetCircumference_CorrectCircumferenceCalculated() { Assert.That(uut.GetCircumference(), Is.EqualTo(2.0 * 3.0 * PI); } [Test] public void Circle_CtorNegativeRadius_ArgumentExceptionThrown() { Assert.Throws<ArgumentException>(uut = new Circle(-1.0)); } }
Easy addition of new tests
Setup/teardown
Good assertion constructs
Test for exceptions
Test automation
-17-
Demo 1: Unit tests for Calculator Add()+Sub()
• Setting up projects for application and unit tests
• Unit test definitions – Test fixtures (setup, teardown)
– Test cases
• Running the unit tests, checking results – NUnit GUI
– IDE-integrated tool
-20-
Dependencies
• Objects seldom work alone
– Objects have relations dependencies DroneCtrl needs references to interact with Motor and
Gyro objects DroneCtrl depends on Motor and Gyro
class DroneCtrl { private: list<Motor*> motors; Gyro* gyro; public: DroneCtrl() { for(…) motors.Add(new Motor()); gyro = new Gyro(); } … }
DroneCtrl Motor4
Gyro
+ GetDriftDirection(): uint+ GetDriftSpeed(): uint
+ SetSpeed(s: double)+ GetSpeed(): double
-21-
Dependencies
• Options for testing DroneCtrl: – Wait for Gyro and Motor classes to be implemented?
– Pollute dependencies with #ifdef TEST? (don’t tell me you haven’t!)
– Or…make design a bit (much!) more flexible with a couple of seams
class DroneCtrl { private: list<Motor*> motors; Gyro* gyro; public: DroneCtrl() { for(…) motors.Add(new Motor()); gyro = new Gyro(); } … }
-22-
Dependencies - seams
DroneCtrl IMotor4
Motor
IGyro
Gyro
+ GetDriftDirection(): uint+ GetDriftSpeed(): uint
+ SetSpeed(s: double)+ GetSpeed(): double
For the price of two interfaces and a constructor injection, you get… + adherence to OCP + DIP + lower coupling + facilitation of code re-use + control of the dependencies!
-23-
Dependency injection
public class DroneCtrl { private IGyro _gyro; private List<IMotor> _motors; public DroneCtrl(IGyro gyro, List<IMotor> motors) { _gyro = gyro; _motors = motors; } }
// DroneCtrl declaration in test [SetUp] void droneCtrlUTSetup() { IGyro* gyro = new FakeGyro(); List<IMotor> _motors; for(…) motors.Add(new FakeMotor()); DroneCtrl uut(motors, gyro); }
// DroneCtrl declaration for real int main() { IGyro* gyro = new Gyro(); List<IMotor> _motors; for(…) motors.Add(new Motor()); DroneCtrl uut(motors, gyro); }
Dependency injection in constructor – DroneCtrl remains oblivious to concrete type of dependencies
DroneCtrl IMotor4
IGyro
Gyro
+ GetDriftDirection(): uint+ GetDriftSpeed(): uint
+ SetSpeed(s: double)+ GetSpeed(): double
MotorFakeGyro FakeMotor
-25-
State and interaction-based tests
• New functionality to our drone: Power-On Self-Test (POST)
DroneCtrl::DoPOST()
Will succeed iff gyro and all motors pass their POSTs
DroneCtrl IMotor4
IGyro
Gyro
+ DoPOST(): bool+ GetDriftDirection(): uint+ GetDriftSpeed(): uint
+ DoPOST(): bool+ SetSpeed(s: double)+ GetSpeed(): double
FakeGyro MotorFakeMotor
+ DoPOST()+ State {private set}
Idle
Error
[POST OK]
[else]
stm DroneCtrl
InitDoPOST/
-26-
State and interaction-based tests
• Some interesting test cases for DroneCtrl: 1. If the gyro and all motors pass POST, DroneCtrl passes POST
2. If the gyro or one of the motors fail their POST, DroneCtrl fails POST
• How to test this? 1. Create IGyro and IMotor fakes that pass (or fail) POST
2. Execute DroneCtrl.DoPOST()
3. Check DroneCtrl.State()
DroneCtrl IMotor4
IGyro
Gyro
+ DoPOST(): bool+ GetDriftDirection(): uint+ GetDriftSpeed(): uint
+ DoPOST(): bool+ SetSpeed(s: double)+ GetSpeed(): double
FakeGyro MotorFakeMotor
+ DoPOST()+ State {private set}
Idle
Error
[POST OK]
[else]
stm DroneCtrl
InitDoPOST/
-27-
State and interaction-based tests
• This is a state-based test
• State-based tests are characterized by the following: – We are acting on the Unit Under Test (UUT)
– We are asserting that the UUT obtained the desired state after the action
– The fake dependencies are there to provide return values (fakes == stubs)
DroneCtrl IMotor4
IGyro
Gyro
+ DoPOST(): bool+ GetDriftDirection(): uint+ GetDriftSpeed(): uint
+ DoPOST(): bool+ SetSpeed(s: double)+ GetSpeed(): double
FakeGyro MotorFakeMotor
+ DoPOST()+ State {private set}
Idle
Error
[POST OK]
[else]
stm DroneCtrl
InitDoPOST/
-28-
State and interaction-based tests
• Another interesting test case for DroneCtrl: – That DroneCtrl actually does query the gyro/motors for their POST
results
• How to test this? 1. Create IGyro and IMotor fakes that pass (or fail) POST, and record
that their DoPOST() was called!
2. Run DroneCtrl::DoPOST()
3. Assert on fakes that their DoPOST was called (fakes == mocks)
DroneCtrl IMotor4
IGyro
Gyro
+ DoPOST(): bool+ GetDriftDirection(): uint+ GetDriftSpeed(): uint
+ DoPOST(): bool+ SetSpeed(s: double)+ GetSpeed(): double
FakeGyro MotorFakeMotor
+ DoPOST()+ State {private set}
Idle
Error
[POST OK]
[else]
stm DroneCtrl
InitDoPOST/
-29-
Stubs vs. Mocks
• The big difference between state-based and interarction-based tests is in where you make the assertion
• Stubs / state-based test – UUT interacts with stub
– Test asserts that UUT itself assumed correct state
– Stub state-based test
• Mocks / interaction-based test – UUT interacts with mock
– Test asserts that UUT interacted correctly with dependency
– Mock interaction-based test
-30-
Demo 2: State and behaviour tests DroneCtrl::DoPOST()
• DroneCtrl (UUT) and gyro/motor interfaces
• Stubs for state-based tests – Succeeding and failing POST
– Assert on UUT
• Mocks for interactions-based tests – Recording call to DoPOST()
– Assert on mocks
-31-
Using an isolation framework
• Hand-coding fakes takes time
• We can save time by using an isolation framework which helps us “isolate” our unit under test. – C# example: NSubstitute
– C++ example: Google Mock
• The basic idea is the same as before: Create fakes from interface definitions only
-32-
Isolation framework – faking gyro and motors
using NSubstitute; using NUnit.Framework; namespace Drone.Test.Unit { [TestFixture] public class DroneCtrlUnitTest { [Setup] public void setup() { var _gyroSub = Substitute.For<IGyro>(); // Create IGyro fake var _motorSub1 = Substitute.For<IMotor>(); / Create IMotor fake ... } } }
Note: No fake class implementing the interface, no hard-coded return values, no nothin’!
-34-
Demo 4: State and behaviour tests DroneCtrl::Hover() with NSubstitute
• Checking for methods called with correct arguments: When Hover() is called, … – Gyro.GetDriftDirection() + Gyro.GetDriftSpeed() must be called
– On relevant motors, GetSpeed() must be called
– On relevant motors, SetSpeed() must be called with correct arguments
0⁰
Gyro.GetDriftDirection() -> 45 Gyro.GetDriftSpeed() -> 3
SetSpeed( GetSpeed() + 2*3);
SetSpeed( GetSpeed() – 2*3);
-35-
Other neat things in isolation frameworks
• As a stub, … – Throw exceptions
– Fail on n’th call to a method
– Return strings
– Raise events
– …
• As a mock, … – Check that the expected exceptions are thrown
– Check that events are raised
– Check that methods are called a specific number of times
– Check that methods are not called
– …
-37-
Test quality - coverage
• Are our tests good enough? When does our test suite contain the necessary and sufficient tests?
• One measure of test quality is code coverage – A quantitative measure of how well our tests cover the application
code
– i.e. measures the quality of our tests, not our application!
• Line coverage (aka statement coverage) most used – not necessarily best, but probably easiest.
int q=4; int* p = 0; if (condition) p = &q; *p = 117;
int array[MAX]; for(int i=0; i<MAX; i++) { array[i] = i++; // ar[i] = i+1…I think }
-38-
• Make the goal 100% coverage, nothing less. – If something cannot be covered, make an informed decision to exclude
it from coverage
• If coverage is 95%, you really know nothing of the remaining 5%
• If 95% is ok, then 90% is also fine … so is 85% … naahhh – let’s make it 80%.
• Oh sh**, the customer is here, 44% is fine!
Two practical experiences
-39-
Test design – boundary value analysis
• Does 100% coverage mean that we’ve tested enough? Or have we maybe tested too much?
• Identifying the “necessary and sufficient” set of test cases can be helped along using boundary value analysis
• Boundary value: A value in the input range for which an output is expected to change in value or validity
• Examples: sign, square root, …or the Drone!
-40-
Drone drift - BVA
1
2 3
4
1,4 + 2,3 -
1 ++ 3 --
1,2 + 3,4 -
2 ++ 4 --
1,4 - 2,3 +
1 -- 3 ++
1,2 - 3,4 +
2 -- 4 ++
Boundary: 22.5
Boundary: 67.5
Boundary: 112.5
Boundary: 157.5 Boundary: 202.5
Boundary: 247.5
Boundary: 292.5
Boundary: 337.5 Boundary: 0, 360
-41-
Test design – boundary value analysis
• Test with input on either side of each boundary value for expected output
-44-
Continuous Integration
A software development practice where members of a team integrate their work frequently, usually each person integrates at least daily – leading to multiple integrations per day. Each integration is verified by an automated build (including test) to detect integration errors as quickly as possible (…) Martin Fowler, 2006
-45-
The basic setup
Code repository
Build application + ”Green Zone” tests
Build application + ”Green Zone” tests
Fast integration: Build application + ”Green Zone” tests
Nightly: Build application + ”Green Zone” tests + Integration tests + Code inspection + Reporting + …
Build server
Workstation 1
Workstation 2
-46-
Benefits of CI
• No ”integration hell” at the ”end” of a project
• Never more than hours from working (deployable) build
• Rapid feedback
• Several graded builds (continuous, nightly, …)
• Frequent check-ins modular code
• Metrics for code quality never out of date
• Allows thorough (nightly, non-green zone) tests
• Allows tests to be run in a true usage environment
-47-
The setup at ASE
Build application + ”Green Zone” tests
Build application + ”Green Zone” tests
Jenkins server
Student workstation (on VPN)
Git server
Student workstation (on VPN)
-48-
Demo 6: Continuous Integration of Drone
• Drone software updated locally
• Updates committed and pushed
• See CI run automagically
-49-
Details of the Git/Jenkins setup
• Git clone via ssh (R/W) by developers
• Git repo exposed (RO) on web server (symlink)
• Git post-update script ”kicks” Jenkins
• Jenkins clones project via http (RO)
-50-
Take-homes
• Software testing is worthwhile – Money-saver (when after the break-even point)
– A reassuring activity for developers
• Software tests rock from Day 1 – Remove lots of bugs early during development, when it’s cheap
• Use the tools – Unit test frameworks, isolation frameworks, coverage tools, CI tools, …
• Automate, automate, automate
-52-
Used resources
• Unit test frameworks – C# http://www.nunit.org/
– C++ https://code.google.com/p/googletest/
• Isolation frameworks
– NSubstitute http://nsubstitute.github.io/
– Google Mock https://code.google.com/p/googlemock/
• Continuous integration – Git http://git-scm.com/
– TortoiseGit https://code.google.com/p/tortoisegit/
– Jenkins http://jenkins-ci.org/