Advanced Software Engineering: Software Testingweb.cs.du.edu/~snarayan/sada/teaching/COMP3705/lecture/l2/L2_Blac… · Advanced Software Engineering: Software Testing COMP 3705(Lecture

Advanced Software Engineering: Software TestingCOMP 3705(Lecture 2)

Sada Narayanappa

Anneliese Andrews (Chair DU)Thomas ThelinCarina Andersson

Adv. Software Engineering: Sada Narayanappa 2

News & Project

♦News♦ Updated course program♦ Reading instructions♦ Questions?

♦Project Option 1♦ IMPORTANT to read the project description thoroughly♦ Schedule, Deadlines, Activities♦ Requirements (7-10 papers), project areas♦ Report, template, presentation

♦Project Option 2: date!


Lecture

♦Some more testing fundamentals♦Chapter 4 (Lab 1)

♦Black-box testing techniques♦Chapter 12 (Lab 2)

♦Statistical testing♦Usage modelling♦Reliability


Terminology

♦ Unit testing: testing a procedure, function, or class.♦ Integration testing: testing connection between units and

components.♦ System testing: test entire system.♦ Acceptance testing: testing to decide whether to purchase

the software.


Terminology (2)

♦Alpha testing: system testing by a user group within the developing organization.♦Beta testing: system testing by select customers.♦Regression testing: retesting after a software modification.


Test Scaffolding

Allows us to test incomplete systems.♦Test drivers: test components.♦Stubs: test a system when some components it

uses are not yet implemented.Often a short, dummy program --- a method with an

empty body.


Test Oracles

♦ Determine whether a test run completed with or without errors.

♦ Often a person, who monitors output.♦ Not a reliable method.

♦ Automatic oracles check output using another program.♦ Requires some kind of executable specification.


Testing Strategies:Black Box Testing

♦Test data derived solely from specifications.Also called “functional testing”.

♦Statistical testing.Used for reliability measurement and prediction.


Testing Theory:Why Is Testing So Difficult?

♦Theory often tells us what we can’t do.♦Testing theory main result: perfect testing is impossible.


An Abstract View of Testing

♦Let program P be a function with an input domain D (i.e., set of all integers).

♦We seek test data T, which will include selected inputs of type D.♦T is a subset of D.♦T must be of finite size.Why?


We Need a Test Oracle

♦Assume the best possible oracle --- the specification S, which is function with input domain D.

♦On a single test input i, our program passes the test whenP(i) = S(i)Or if we think of a spec as a Boolean function that

compares the input to the output: S(i, P(i))


Requirement For Perfect Testing [Howden 76]

1. If all of our tests pass, then the program is correct. ∀x[x ∈ T ⇒ P(x) = S(x)]

⇒∀y[y ∈ D ⇒ P(y) = S(y)] • If for all tests t in test set T, P(t) = S(t), then we

are sure that the program will work correctly for all elements in D.

• If any tests fail we look for a bug.


Requirement For Perfect Testing

2. We can tell whether the program will eventually halt and give a result for any t in our test set T.∀x[x∈ T ⇒ “∃ a computable procedure for

determining if P halts on input x”]


But, Both Requirements Are Impossible to Satisfy.

♦1st requirement can be satisfied only if T= D.We test all elements of the input domain.

♦2nd requirement depends on a solution to the halting problem, which has no solution.We can demonstrate the problem with Requirement 1 [Howden 78].


Other Undecidable Testing Problems

♦Is a control path feasible? Can I find data to execute a program control path?

♦Is some specified code reachable by any input data?These questions cannot, in general, be answered.


Software Testing Limitations

♦There is no perfect software testing.♦Testing can show defects, but can never show correctness.We may never find all of the program errors during testing.


Why test techniques?♦Exhaustive testing (use of all possible inputs and conditions) is impractical

♦ must use a subset of all possible test cases♦ want must have high probability of detecting faults

♦Need processes that help us selecting test cases♦ Different people – equal probability to detect faults

♦Effective testing – detect more faults♦ Focus attention on specific types of fault♦ Know you’re testing the right thing

♦Efficient testing – detect faults with less effort♦ Avoid duplication♦ Systematic techniques are measurable


Dimensions of testing

♦Testing combines techniques that focus on♦Testers – who does the testing♦Coverage – what gets tested♦Potential problems – why you're testing (risks / quality)♦Activities – how you test♦Evaluation – how to tell whether the test passed or

failed♦All testing should involve all five dimensions

♦Testing standards (e.g. IEEE)


Black-box testing

Ie

Input test data

OeOutput test results

System

Inputs causinganomalousbehaviour

Outputs which revealthe presence ofdefects


userqueries

numericaldata

output format requests

responsesto prompts

command key input

mouse picks on menuPartitioning is basedon input conditions

Equivalence partitioning


Equivalence partitioning

Sy stem

Ou tputs

Invalid in pu ts Valid in pu ts

Input condition:♦is a range

♦ one valid and two invalid classes are defined

♦requires a specific value ♦ one valid and two invalid

classes are defined♦is a boolean

♦ one valid and one invalid class are defined


Test Cases

♦Which test cases have the best chance of successfully uncovering faults?

♦ as near to the mid-point of the partition as possible♦ the boundaries of the partition and

♦Mid-point of a partition typically represents the “typical values”♦Boundary values represent the atypical or unusual values♦Usually identify equivalence partitions based on specs and experience


♦Consider a system specification which states that a program will accept between 4 and 10 input values (inclusive), where the input values must be 5 digit integers greater than or equal to 10000♦What are the equivalence partitions?

Equivalence Partitioning Example


Example Equivalence Partitions

Between 10000 and 99999Less than 10000 More than 99999

999910000 50000

10000099999

Input values

Between 4 and 10Less than 4 More than 10

34 7

1110

Number of input values


userqueries

numericaldata

output format requests

responsesto prompts

command key input

mouse picks on menu

outputdomain

Boundary value analysis


Boundary value analysis

♦Range a..b ⇒ a, b, just above a, just below b♦Number of values: max, min, just below min, just above max♦Output bounds should be checked♦Boundaries of externally visible data structures shall be checked (e.g. arrays)

Component–1 0…99 100 -10 –9…499 500–1 0…99 100 -10 –9…499 500


Some other black-box techniques

♦Risk-based testing, random testing♦Stress testing, performance testing♦Cause-and-effect graphing♦State-transition testing


Black Box Testing:Random Testing

♦Generate tests randomly.♦“Poorest methodology of all” [Meyers].♦Promoted by others.♦Statistical testing:

♦Test inputs from an operational profile.♦Based on reliability theory.♦Adds discipline to random testing.


Black Box Testing:Cause-Effect Analysis

♦Rely on pre-conditions and post-conditions and dream up cases.♦Identify impossible combinations.♦Construct decision table between input and output conditions.

Each column corresponds to a test case.


Error guessing

♦Exploratory testing, happy testing, ...♦Always worth including♦Can detect some failures that systematic techniques miss♦Consider

♦ Past failures (fault models)♦ Intuition♦ Experience♦ Brain storming♦ ”What is the craziest thing we can do?”♦ Lists in literature


Black Box Testing:Error Guessing

♦“Some people have a knack for ‘smelling out’errors” [Meyers].♦Enumerate a list of possible errors or error prone situations.♦Write test cases based on the list.Depends upon having fault models, theories on the causes and effects of program faults.


Usability testing

Characteristics

♦ Accessibility

♦ Responsiveness

♦ Efficiency

♦ Comprehensibility

Environments

♦ Free form tasks

♦ Procedure scripts

♦ Paper screens

♦ Mock-ups

♦ Field trial


Specification-based testing

♦Formal method♦Test cases derived from a (formal) specification (requirements or design)

Specification Test case generation

Test execution

Model (state chart)


Model-based Testing

Specification

Top-levelDesign

DetailedDesign

Coding

Unit Test

Integration

VALIDATION

Test phase

Requirements

Model

Usage


Need For Test Models

♦ Testing is a search problem.♦ Search for specific input & state combinations that cause

failures.♦ These combinations are rare.

♦ Brute force cannot be effective.Brute force can actually lead to overconfidence.

♦ Must target & test specific combinations.♦ Targets based on fault models.♦ Testing is automated to insure repeatable coverage of targets:

“target coverage”.


Model-Based Coverage

♦ We cannot enumerate all state/input combinations for the implementation under test (IUT).

♦ We can enumerate these combinations for a model.♦ Models allow automated target testing.

“Automated testing replaces the tester’s slingshot with a machine gun.”

“The model paints the target & casts the bullets.”


Test Model Elements

♦Subject: the IUT.♦Perspective: focus on aspects likely to be buggy

based on a fault model.♦Representation: often graphs (one format is UML)

or checklists.♦Technique: method for developing a model and

generate tests to cover model targets.


The Role of Models in Testing

♦Model validation: does it model the right thing?♦Verification: implementation correct?

Informal: checklistFormal: proof

♦Consistency checking: is representation instance of meta model?

♦Meta-model: UML, graphs, etc + technique♦Instance model: representation constructed


Models Roles in Testing

♦Responsibility-based testing.Does behavior conform to the model representation?

♦Implementation-based testing.Does behavior conform to a model of the

implementation.♦Product validation.

Does behavior conform to a requirements model, for example, Use Case models?


Models That Support Testability

♦Represent all features to be tested.♦Limit detail to reduce testing costs, while preserving essential detail.♦Represents all state events so that we can generate them.♦Represents all state & state actions so that we can observe them.


Model Types

♦Combinational models:♦Model combinations of input conditions.♦Based on decision tables.

♦State Machines:♦Output depends upon current & past inputs.♦Based on finite state machines.

♦UML models: model OO structures.


Combinational Model - Spin Lock [Binder Fig. 6.1]

8

7

6

5

4

3

2

0

9


Combinational Model: Use a Decision Table

♦1 of several responses selected based on distinct input variables.♦Cases can be modeled as mutually exclusive Boolean expressions of input variables.♦Response does not depend upon prior input or output.


Decision Table Construction

1. Identify decision variables & conditions.2. Identify resultant actions to be selected.3. Identify the actions to be produced in response to

condition combinations.4. Derive logic function.

Combinational Models


Auto Insurance Model

♦Renewal decision table: Table 6.1.♦Column-wise format: Table 6.2.♦Decision tree: Fig. 6.2.♦Truth table format: Fig. 6.3.♦Tractability note:

♦Decision table with n conditions has a maximum of 2 variants.

♦Some variants are implausible.

Combinational Models


Insurance Renewal Decision Table[Binder Table 6.1]

yesyes0Any5 or >7noyes200>252 to 46

noyes400<262 to 45

nono50>2514

noyes100<2613

nono25>2502

nono50<2601

CancelSend WarningPrem. Incr.Age# ClaimsVariant

Action SectionCondition Sec.


Yes

YesNoNoNoNoNoNoCancel

NoYesYesNoYesNoNoSendwarning

0200400501002550Premium increase($)

ActionSection

Any26 or older

25 or younger

26 or older

25 or younger

26 or older

25 or younger

Insuredage

5 ormore2 to 42 to 41100

Number of Claims

ConditionSection

7654321

Variant

Insurance Renewal Decision Table Column-wise Format


Implicit Variants

Variants that can be inferred, but not given.♦ In the insurance example, we don’t care about age if

there are five or more claims. The action is to cancel for any age.

♦Other implicit variants are those that cannot happen ---one cannot be both under 25 yrs old and over 25.


Test Strategies

All explicit variants at least once– When decision boundaries are systematically exercised– weak, if can’t happen conditions or undefined

boundaries result in implicit variants


Non binary Variable Domain Analysis

♦Exactly 0 claims, age 16-25♦Exactly 0 claims, age 26-85♦Exactly 1 claim, age 16-25♦Exactly 1 claim, age 26-85♦Two, 3, or 4 claims, age 16-25♦Two, 3, or 4 claims, age 26-85♦Five to 10 claims, age 16-85


NoNoNoNoNoCancel

NoNoNoYesNoSend warning

25505010050Premium increase

V2AcceptRejectAcceptRejectV3AcceptExpected result

202020InTypical

26Off

25On<= 25

15OffInsuredAge

16On>= 16

0000InTypical

-1Off(below)

1Off(above)

NumberOf Claims

0On==0

54321Variant 1 Boundaries

Test Cases


Additional Heuristics

♦Vary order of input variables♦Scramble test order♦Purposely corrupt inputs


Statistical testing /Usage based testing

Reliability estimation

Failure logging

Test execution

Test case generation

Usage specification

Test Case

1.1.3Setup1.1.4Call

Failure Report

#13

Output failure

Item 1

Item 2

Item 3

Up

Dow

n SelectShown

Hidden

Up

Up

Dow

nD

own

Select

SelectInvoke

Down


Usage specification models

<test_case> ::= <no_commands> @ <command> <select>;

<no_commands> ::= ( <unif_int>(0,2) [prob(0.9)] | <unif_int>(3,5) [prob(0.1)]);

<command> ::=(<up> [prob(0.5)] | <down> [prob(0.5)]);

State hierarchy model

Grammar modelAlgorithmic models

Usage

Usertype

Usertype

User User User

Servic e Service Se rvice Service

Subtype

SubtypeSub

type

User

Service


Usage specification models

Markov modelOperational profile

Domain based models

UserSelect menu

Function x

Function y

Normal use

System mode ZItem 1

Item 2

Item 3

10%

90%

25%

50%

25%

40%

30%

30%

Item 1

Item 2

Item 3

Up

Dow

n SelectShown

Hidden

Up

Up

Dow

nD

own

Select

SelectInvoke

Down


Operational profiles


Operational profiles


Statistical testing / Usage-based testing

Usage model

Code

Random sample


Usage Modelling

♦Each transition corresponds to an external event♦Probabilities are set according to the future use of the system♦Reliability prediction

Main Window

DialogBox

Right-click

Invoke

Terminate

Move

ResizeCANCEL or OKwith Valid Hour

Click on OK withnon-valid hour

Close Window


Markov model

♦ System states, seen as nodes

♦ Probabilities of transitions

Conditions for a Markov model:

♦ Probabilities are constants♦ No memory of past states From

Node

Transition matrixTo Node

P4400P41N4

P34P330P31N3

P240P22P21N2

P14P13P12P11N1

N4N3N2N1

N1 N2

N3N4

P12

P13

P14

P34

P24P31

P41P21


Model of a program

♦ The program is seen as a graph♦ One entry node (invoke) and one exit node (terminate)♦ Every transition from node Ni to node Nj has a

probability of Pij♦ If no connection between Ni and Nj, then Pij= 0

N1 N2

N3N4

P12

P13

P14

P34

P24P31

P21

Output

Input F


Clock SoftwareClock

Options

24 Aug 1997

AnalogDigitalClock OnlySecondsDateChange Time/Date…Info…Exit

Change Time/Date

current time: 11:10:27a new time: ________

current date: Sat 24 Aug 1996 new date: ___________

OK

Info

Vaporware Clock, version 1.0

OK


Input Domain – Subpopulations

♦Human users – keystrokes, mouse clicks♦System clock – time/date input♦Combination usage - time/date changes from the OS while the clock is executing

♦Create one Markov chain to model the input from the user


Operation modes of the clock

♦Window = {main window, change window, info window}♦Setting = {analog, digital} ♦Display = {all, clock only}♦Cursor = {time, date, none}


State of the system

♦A state of the system under test is an element of the set S, where S is the cross product of the operational modes.♦States of the clock

{main window, analog, all, none}{main window, analog, clock-only, none}{main window, digital, all, none}{main window, digital, clock-only, none}{change window, analog, all, time}{change window, analog, all, date}{change window, digital, all, time}{change window, digital, all, date}{info window, analog, all, none}{info window, digital, all, none}


Top Level Markov Chain

Window operational mode is chosen as the primary modelingmode

te rm ina ted no tinvoked

m a inw indow

in fow indow

changew indow

m a inw indow

changew indow

invoke(prob=1)

op tions .in fo

op tions.change

op tions.ex it(p rob=1 /3 )

(p rob=1 /3 )

(p rob=1 /3 )ok (p rob=1)

end (p rob=1)

Rules for Markov chainsEach arc is assigned a probability between 0 and 1 inclusive,The sum of the exit arc probabilities from each state is exactly 1.


Top Level Model – Data Dictionary

The info window will disappear and the main window will be given the focus

Press the ok button on the info windowok

The change window will disappear and the main window will be given the focus

Choose any action and return to the main window

end

The software will terminate, end of test caseSelect the “Exit” option from the “Options” menu

options.exit

The title bar must be on to apply this inputThe info window should appear and be given the focusTester should verify window appearance and modality and ensure that it accepts no illegal input

Select the “Info...” item from the “Options” menu

options.info

All window features must be displayed in order to execute this commandThe change window should appear and be given the focusTester should verify window appearance and modality and ensure that it accepts no illegal input

Select the “Change Time/Date...” item from the “Options” menu

options.change

Main window displayed in fullTester should verify window appearance, setting, and that it accepts no illegal input

Invoke the clock software invoke

Comments/Notes for TesterInput to be AppliedArc Label


Level 2 Markov Chain

options.info

options.change

invoke

ok

end

options.exit

digital, all digital,clock only

analog, all analog,clock only

double-click

double-click

options.secondsoptions.date

options.clock-only

options.digital

options.secondsoptions.date

options.clock-only

options.analog

Submodel for the Main Window


•The change window will disappear and the main window will be given the focus•Note: this action may require that the software be calibrated byissuing either an options.analog or an options.digital input

Choose any action (cancel or change the time/date) and return to the main window

end

•The software will terminate, end of test caseSelect the “Exit” option from the “Options” menu

options.exit

•The title bar must be on to apply this input•The info window should appear and be given the focus•Tester should verify window appearance and modality and ensure that it accepts no illegal input

Select the “Info...” item from the “Options” menu

options.info

•All window features must be displayed in order to execute this command•The change window should appear and be given the focus•Tester should verify window appearance and modality and ensure that it accepts no illegal input


options.change

•Main window displayed in full•Invocation may require that the software be calibrated by issuing either an options.analog or an options.digital input•Tester should verify window appearance, setting, and ensure thatit accepts no illegal input

Invoke the clock software invoke

Comments/NotesInput to be AppliedArc Label

Data Dictionary – Level 2


•The clock face should be replaced by the entire clock window

Double click, using the left mouse button, on the face of the clock

double-click

•The date should be toggled either on or off depending on its current status

Select the “Date” item from the “Options” menu

options.date

•The second hand/counter should be toggled either on or off depending on its current status

Select the “Seconds” item from the “Options” menu

options.seconds

•The clock window should be replace by a display containing only the face of the clock, without a title, menu or border

Select the “Clock Only” item from the “Options” menu

options.clock-only

•The analog display could be replaced by a digital display

Select the “Digital” item from the “Options” menu

options.digital

•The digital display should be replaced by an analog display

Select the “Analog” item from the “Options” menu

options.analog

•The info window will disappear and the main window will be given the focus•Note: this action may require that the software be calibrated by issuing either an options.analog or an options.digital input

Press the ok button on the info window

ok

Comments/NotesInput to be AppliedArc Label

Data Dictionary – Level 2


Level 2 Markov Chain

op tions .change

end

tim e da tem ove

ed it tim e

m ove

ed it da te

Submodel for the Change Window


Data Dictionary

•The valid input format is shown on the screenChange the date in the “new date” field or enter an invalid date

edit date

•The valid input format is shown on the screenChange the time in the “new time” field or enter an invalid time

edit time

•Tester should verify cursor movement and also verify both options for moving the cursor

Hit the tab key to move the cursor to the other input field or use the mouse to select the other field

move

•The change window will disappear and the main window will be given the focus•Note: this action may require that the software be calibrated by issuing either an options.analog or an options.digital input

Choose either the “Ok” button or hit the cancel icon and return to the main window

end

•All window features must be displayed in order to execute this command•The change window should appear and be given the focus•Tester should verify window appearance and modality and ensure that it accepts no illegal input


options.change

Comments/Notes for TesterInput to be AppliedArc Label


Failu

re R

ate

Time

♦Techniques♦ Markov models♦ Reliability growth models

Software Reliability


Dimensions of dependability

Dependability

Availability Reliability Security

The ability of thesystem to deliver

services whenrequested

The ability of thesystem to deliver

services as specified?

The ability of thesystem to operate

without catastrophicfailure

The ability of thesystem to protect itelfagainst accidental ordeliverate in trusion

Safety


Costs of increasing dependability

Cost

Low Medium High Veryhigh

Ultra-high

Dependability


Availability and reliability

♦Reliability♦The probability of failure-free system operation over a

specified time in a given environment for a given purpose

♦Availability♦The probability that a system, at a point in time, will be

operational and able to deliver the requested servicesBoth of these attributes can be expressed

quantitatively


Term DescriptionSystem failure An event that occurs at some point in time when

the system does not deliver a service as expectedby its users

System error Erroneous system behaviour where the behaviourof the system does not conform to itsspecification.

System fault An incorrect system state i.e. a system state thatis unexpected by the designers of the system.

Human error ormistake

Human behaviour that results in the introductionof faults into a system.

Reliability terminology


Usage profiles / Reliability

Removing X% of the faults in a system will not necessarily improve the reliability by X%!

Possibleinputs

User 1

User 3User 2

Erroneousinputs


Reliability achievement

♦Fault avoidance♦Minimise the possibility of mistakes♦Trap mistakes

♦Fault detection and removal♦ Increase the probability of detecting and correcting

faults♦Fault tolerance

♦Run-time techniques


Reliability quantities

♦Execution time ♦ is the CPU time that is actually spent by the computer in

executing the software

♦Calendar time♦ is the time people normally experience in terms of years, months,

weeks, etc

♦Clock time♦ is the elapsed time from start to end of computer execution in

running the software


Reliability metrics


Nonhomogeneous Poisson Process (NHPP) Models

♦N(t) follows a Poisson distribution. The probability that N(t) is a given integer n is:

,...2,1,0,!)]([})({ )( === − ne

ntmntNP tm

n

♦m(t) = μ(t) is called mean value function, it describes the expected cumulative number of failures in [0,t)


The Goel-Okumoto (GO) model

Assumptions♦The cumulative number of failures detected at time t follows a Poisson distribution♦All failures are independent and have the same chance of being detected♦All detected faults are removed immediately and no new faults are introduced♦ The failure process is modelled by an NHPP model with mean value function μ(t) given by:

;0,0),1()( >>−= − baeat btμ


Goel-Okumoto

The shape of the mean value function (μ(t)) and the intensity function (λ(t)) of the GO model

μ(t)

λ(t)t


S-shaped NHPP model

♦ μ(t)=a[1-(1+bt)e-bt], b>0

t

μ(t)

.)1()()( 2 btbtbt teababeebtabdt

tdt −−− =−+==μλ


The Jelinski-Moranda (JM) model

Assumptions1. Times between failures are independent, exponential

distributed random quantities2. The number of initial failures is an unknown but fixed

constant3. A detected fault is removed immediately and no new fault is

introduced4. All remaining faults contribute the same amount of the

software failure intensity


Next weeks

♦Next week (April 10)♦Lab 1 – Black-box testing

Advanced Software Engineering: Software Testingweb.cs.du.edu/~snarayan/sada/teaching/COMP3705/lecture/l2/L2_Blac… · Advanced Software Engineering: Software Testing COMP 3705(Lecture

Documents