Software Testing – Lecture #2 Thomas Ball with material from M. Young, A. Memon and MSR’s FSE group.

Software Testing – Lecture #2

Thomas Ball

with material from M. Young, A. Memon and MSR’s FSE group

Testing - The Story So Far

Two Threads

• Specification-based test generation– Foundations of Software Engineering

• Implementation-based test generation– Testing, Verification and Measurement

• Unifying Themes– create finite-state systems from infinite-state systems

via predicates– use finite-state algorithms to guide test generation

Specification-based Testing

• What does this API do?– Executable spec prescribes potential behavior

• Where do concrete tests come from?– Explore behavior of spec, generate test strategies

• Does a test execution succeed or fail?– Use spec as oracle for runtime verification

• How do we know when we’re done testing?– Cover spec (and implementation)

• What do we know when we’re done testing?– Model and implementation agree!

4 Steps to Testing Heaven

– Modeling• define infinite transition system

– Exploration• reduce to finite test graph

– Gaming• generate test strategies

– Monitoring• verify conformance

Example: Alternating Bit Protocol

• This protocol works on the producer-consumer model.

• Producer wants to send messages in a reliable manner to a consumer through unreliable channel

1st Step: Model the Infinite Transition System

Describe (transition system in Spec# of) all possible runs of ABP

Types– Msg, Ack

State– Bitstatus, …

Controllable action– Send

Observable events: – Receive – Lose Msg– Lose Ack

s0

s1

s0 s2

s3s4

s5

s3 s2

s6

Send

?LoseMsg ?Receive

?LoseAck

Send

?LoseMsg?Receive

?Timeout

Send

ABP: Data Structures

class DataMsg { // Message

readonly Data data;

readonly bool bit;

}

class Ack { // Acknowledgement

readonly bool bit;

}

ABP: State

bool

SenderBit = true;

int

SenderRecordNr = 0;

Ack

SenderInbox = null;

DataMsg

ReceiverInbox = null;

bool

ReceiverBit = true;

Seq<Data>

ReceivedFile=Seq{};

Sends Message

ReturnsAck

ABP: Transitionsvoid Send () {

require AcknowledgmentArrived() || Timeout();

if(AcknowledgmentArrived()) { if(AcknowledgmentHasTheRightBit()) { ReceiverInbox = DataMsg(SenderRecordNr + 1, !SenderBit); SenderRecordNr += 1; SenderBit = !SenderBit; } SenderInbox = null; } else if(Timeout()) { ReceiverInbox = DataMsg(SenderRecordNr, SenderBit); }}

EnablingCondition

Statechange

ABP: Modeling Demo

• Show model in SpecExplorer

• Simulate as a console application

2nd Step: Finitize the Model

Generate “all” possible behaviors (test graph):– At each state, execute any enabled method with any

allowed argument values– Nondeterministic choice from enabled method explores

possible interleavings

Control the search, using– Finite unfolding– Stochastic means– User provided predicate abstractions [ISSTA 2002]

This generates a test graph

User provided predicate abstraction

Tree can be pruned, if an equivalent state has already been visited

Example:• Generate an FSM from a stack specification.• Observation property: stack.IsEmpty()

[0,0]

Push(0)

true

false

Push(1)

Push(0)

Top()

Pop()

Pop()

[]

Push(0)

[0]

Pop()

Top()

Pop()Push(1)

[1,0]

Top()

Test graph

A test graph G is a directed graph s.t.• there are two kinds of vertices in G:

– states– choice points (CP)

• every edge e has a prob. p(e) s.t. for every CP u

∑{p(e): source of e = u}=1,• there is a non-negative cost

function c defined on edges

state vertex

p=0.3c=1

p=0.7c=2

CP vertex

c=1

ABP: Modeling Demo

• Explorer options

3rd Step: Generate Test Strategies

Cover all edges of the test graph (edge coverage)

Reach certain goal states in the test graph (reachability game) [ISSTA 2004]

1. With maximal probability and minimal cost

2. With full certainty and minimal cost

Edge Coverage Strategy

1. Produce a tour of the edges in the graph (e.g. Chinese Postman tour)

2. Divide the tour into segments starting and ending at choice points

3. At a choice point IUT selects an edge e and TT chooses randomly any segment s starting from e and follows s until the end (s ends in a choice point).

Edge Coverage Example

• The tour is[e8,e2,e1,e5,e7, e9,e4,e3,e6,e7]

• The segments are:[e8,e2,e1,e5,e7][e9,e4,e3,e6,e7]

s1

e1

e2

e4

e3

e9e8

e7e5 e6

4th Step:Execute Tests and Verify Results

• Define conformance notion: Probes

• Rewrite the IUT to insert call backs into model

• Specify upper bounds on number of repetitions

ProbesGiven an abstract domain X• A probe P m is a function from model states to X• A probe P i is a function from implementation states to X• Model state A and impl state B conform wrt the pair of

probes (P m, P i) if P m(A) = P i(B)• Different probes may be enabled at different states

A1

A2

B1

B2

B4

B3

B5

A3P1

m P1iX

P2m

P2i

IUTModel

Event Buffering• During conformance testing all events are buffered

in an unbounded queue• All enabled probes are checked at every state

e1

Events

e2

a1

Action Callss0

s4

s1

s2 s3

a1

e1 e2

a2

t0

t2

t1

a1’

(P1m,P1

i)

e1’(P2

m,P2i)

ABP Demo

• Test sequence generation

• Conformance testing

Experience

• Modeling for Testing – Models are created by testers, not designers– Models are based on informal specs, not impl.– Models are not comprehensive, only issues

• Several projects– Web services, Passport, Mediaplayer,

Distributed File replication system– Models up to 100 pages

• More than 50 people are using it on a daily basis, steep uptake

More Info

• Public release of Spec# this Summerhttp://research/microsoft.com/foundations

• Contact [email protected] if you would like to know more or are interested in a summer internship in 2005!

• Consider submitting something for ICFEM 2004http://research/microsoft.com/conferences/

ICFEM2004

MSIL Unit Test Tool a hybrid helper

• Goal capture developer knowledge ASAP via a strong set of unit teststo form a specification of the code’s behavior

• How– generate tests based on analysis of MSIL– symbolic execution + constraint satisfaction– runtime analysis to check complicated invariants

• Facets – complements specification-based test generation– positive feedback cycle with programmer

What criteria should guide unit test generation?

Predicate-complete Testing

• Predicates– relational expression such as (x<0)– the expression (x<0) || (y>0) has two predicates– predicates come from program and safe runtime semantics

• Consider a program with m statements and n predicates– predicates partition input domain– m x 2n possible observable states S

• Goal of Predicate-complete Testing:– cover all reachable observable states R S

PCT Coverage

L2: if (A || B) S else T

L3: if (C || D) U else V

• PCT requires covering all logical combinations over {A,B,C,D} at – L2 and L3– S, T, U and V

• Some combinations may not be reachable

PCT Coverage does not imply Path Coverage

L1: if (x<0)L2: skip; elseL3: x = -2;L4: x = x + 1;L5: if (x<0)L6: A;

PCT Coverage does not imply Path Coverage

L1: if (x<0)L2: skip; elseL3: x = -2;L4: x = x + 1;L5: if (x<0)L6: A;

PCT Coverage does not imply Path Coverage

L1: if (x<0)L2: skip; elseL3: x = -2;L4: x = x + 1;L5: if (x<0)L6: A;

PCT Coverage does not imply Path Coverage

L1: if (x<0)L2: skip; elseL3: x = -2;L4: x = x + 1;L5: if (x<0)L6: A;

L1: if (p)L2: if (q) L3: x=0;L4: y=p+q;

Path Coverage does not imply PCT Coverage

L1: if (p)L2: if (q) L3: x=0;L4: y=p+q;

Path Coverage does not imply PCT Coverage

Denominator Problem

• Coverage metrics require a denominator– e.g. statements executed / total statements

• Easy to define for observable states– executed observable states / (m x 2n)

• But (m x 2n) is not a very good denominator!– most observable states will not be reachable– R <<< S

Upper and Lower Bounds

m x 2n possible states S

Upper bound U

Reachable states R

Lower bound L

• Bound reachable observable states– modal transition systems and predicate abstraction

– |L| / |U| defines “goodness” of abstraction

• Test generation using lower bound L

• Refinement to increase |L| / |U| ratio

a

a’

may

MC MA

a

a’

total

MC MA

a

a’

total &

onto

a

a’

onto

Abstraction Construction

Upper Bound: May-Reachability

a

b

c

may

a

b

c

may

Upper Bound: May-Reachability

a

b

c

may

a

b

c

may

c

d

total

a

b

onto

Pessimistic Lower Bound

may

c

d

a

b

Pessimistic Lower Bound

may

onto

total

c

d

a

b

Pessimistic Lower Bound

may

onto

total

void partition(int a[]) { assume(a.length()>2); int pivot = a[0]; int lo = 1; int hi = a.length()-1; while (lo<=hi) { while (a[lo]<=pivot) lo++; while (a[hi]>pivot) hi--; if (lo<hi) swap(a,lo,hi); }}

Example

void partition(int a[]) { assume(a.length()>2); int pivot = a[0]; int lo = 1; int hi = a.length()-1; while (lo<=hi) { while (a[lo]<=pivot) lo++; while (a[hi]>pivot) hi--; if (lo<hi) swap(a,lo,hi); }}

Observation Vector

[ lo<hi, lo<=hi, a[lo]<=pivot, a[hi]>pivot ]

• lo<hi lo<=hi

lo<hi lo<=hi (a[lo]<=pivot a[hi]>pivot)

(a[lo]<=pivot a[hi]>pivot)

Only 10/16 observations possible

13 labels x 10 observations = 130 observable states

But, program constrains reachable observable statesgreatly.

void partition(int a[]) { assume(a.length()>2); int pivot = a[0]; int lo = 1; int hi = a.length()-1;

L0: while (lo<=hi) { L1: ; L2: while (a[lo]<=pivot) { L3: lo++; L4: ;} L5: while (a[hi]>pivot) { L6: hi--; L7: ;} L8: if (lo<hi) { L9: swap(a,lo,hi); LA: ;} LB: ;} LC: ;}

void partition() { decl lt, le, al, ah; enforce ( (lt=>le) & ((!lt&le)=>(al&!ah)|(!al&ah)) ); lt,le,al,ah := T,T,*,*; L0: while (le) { L1: ; L2: while (al) { L3: lt,le,al := (!lt ? F:*), lt, *; L4: ;} L5: while (ah) { L6: lt,le,ah := (!lt ? F:*), lt, *; L7: ;} L8: if (lt) { L9: al,ah := !ah,!al; LA: ;} LB: ;} LC: ;}

Boolean Program

State Space of Boolean Program

TTTT TTTF FTTF FFTF TTFT FTFT FFFT TTFF FFFF FFTTL0 x x x x xL1 x x x xL2 x x x x x x x xL3 x x x xL4 x x x x x x x xL5 x x x x xL6 x x xL7 x x x x xL8 x xL9 xLA xLB x xLC x

Upper Bound = 49 states

[ lo<hi, lo<=hi, a[lo]<=pivot, a[hi]>pivot ]

plaintext

Test Generation

• DFS of Lp generates covering set of paths

• Symbolically execute paths to generate tests

• Run program on tests to find errors and compute coverage of observable states

Array bounds violations

Generated Inputs

(L0:TTTT,L4:FTFT) { 0,-8,1 }(L0:TTTT,L4:TTFT) { 0,-8,2,1 }(L0:TTTT,L4:TTTT) { 0,-8,-8,1 }(L0:TTTF,L4:TTFF) { 1,-7,3,0 }(L0:TTTF,L4:FTTF) { 0,-7,-8 }(L0:TTTF,L4:TTTF) { 1,-7,-7,0 }(L0:TTFT,L7:TTFF) { 0,2,-8,1 }(L0:TTFT,L7:FTFT) { 0,1,2 }(L0:TTFT,L7:TTFT) { 0,3,1,2 }(L0:TTFF,L0:TTTT) { 1,2,-1,0 }

void partition(int a[]) { assume(a.length()>2); int pivot = a[0]; int lo = 1; int hi = a.length()-1;

L0: while (lo<=hi) { L1: ; L2: while (a[lo]<=pivot) { L3: lo++; L4: ;} L5: while (a[hi]>pivot) { L6: hi--; L7: ;} L8: if (lo<hi) { L9: swap(a,lo,hi); LA: ;} LB: ;} LC: ;}

Results

• Buggy partition function– U=49, L=43, Tested=42

• Fixed partition function– U=56, L=37, Tested=43

• What about the remaining 13 states?

Refinement

New Observation Vector

[ lo<hi, lo<=hi, lo=hi+1,

a[lo]<=pivot, a[hi]>pivot,

a[lo-1]<=pivot, a[hi+1]>pivot

]

Only 48/128 observations possible

For this set of predicates, Lp = U

Conclusions

• PCT coverage – new form of state-based coverage – similar to path coverage but finite

• Upper and lower bounds – computed using predicate abstraction and

modal transitions – use lower bound to guide test generation– refine bounds