L. Dillon Software Engineering & Network Systems Laboratory Michigan State University 1 Getting Results From Testing Laura K. Dillon Software Engineering.

Software Engineering & Network Systems Laboratory

Michigan State University1L. Dillon

Getting Results FromTesting

Laura K. DillonSoftware Engineering and Network Systems Lab

Michigan State University

www.cse.msu.edu/sens



Overview

Background: Testing

The Problem: Temporal Behavior

Validation

A Solution: Graphical Interval Logic-

Based Oracles



PREMISES

Software intensive systems must

be tested extensively

Testing should be done early

and often



Testing is more than running tests!

Test results

Reqs/Design/Code

Formal Specifications

Evaluateadequacy

of tests

Evaluatecorrectness

of tests

Create testplans

Test plans

Reqs/Design/Code

Reqs/Design/Code

Runtests



A specification-based test oracle

Test results

Formal Specifications

Evaluatecorrectness

of tests

Checks that behaviors exhibitedduring testing satisfy thespecifications



Overview

Background: Testing

The Problem: Temporal Behavior

Validation


Based Oracles



The Oracle Problem

Answers the question:

Were any of the test executions erroneous?

Concurrency adds a new dimension of complexity to this problem



A concurrent system …

Consists of asynchronous communicating

threads of control:

Tasks / Processes / Threads

Threads synchronize/communicate by

sending messages or through sharing data

Temporal ordering matters



GAS STATION EXAMPLE [Helmbold, Luckham: 1985] Cus(1)

Change

Cus(N)

Change

Operator

Prepay Charg

e

Pump

Start Finish

Act

. . .



Example Temporal Properties

Whenever Cus(1) prepays, it

eventually receives change, but only

after first pumping gas

If Cus(1) prepays before Cus(2), then

Cus(1) gets to pump gas before

Cus(2) gets to pump gas



Large number of behaviors can be produced

Behaviors can be long Behaviors that violate necessary

temporal constraints are easily overlooked

When validating behaviors of concurrent software systems . . .



Overview

Background: Testing

The Oracle Problem: Behavior

Validation


Based Oracles



Overview of GIL Oracles

Desired temporal properties of execution traces are expressed formally in Graphical Interval Logic (GIL)

Execution traces are generated in testing Oracle checks that execution traces satisfy

GIL specifications Displays anomalies to help user identify

faults



Programmer defines “conditions” to trace

Operator

Prepay Charg

e

Pump

Start Finish

Cus(1)

Change

Act

Cus(N)

Change

Pay1

Pay1PayN

PayN

Pump1

Pump1



Formal comments define Pay1, . . ., PayN

--% pay$n, for $n in ID_RANGE: Condition is--% triggered by--% begin accept Cus($n):Operator.prepay--% cancelled by--% begin activation Gas_Simulation--% end accept Operator:Cus($n).change

Similarly: Pump1, . . ., PumpN



An execution trace induces a state sequence

init

acc C

(1):

Op

.pre

acc C

(2):

Op

.pre

acc C

(3):

Op

.pre

acc C

(1):

Pp

.sta

rt

en

d C

(1):

Pp

.fin

i

acc C

(2):

Pp

.sta

rt

acc O

p:C

(1).

ch

an

en

d C

(2):

Pp

.fin

acc C

(1):

Op

.pre

acc C

(3):

Pp

.sta

rt

acc O

p:C

(2).

ch

an

en

d C

(3):

Pp

.fin

acc C

(2):

Op

.pre

acc C

(3):

Op

.pre

acc C

(1):

Pp

.sta

rt

Pay1Pump1Pay2Pump2

. . .

. . .

. . .

. . .

. . .



DoesPump1: Whenever Cus(1) prepays, it eventually receives change, but only after first pumping gas




[ )

Pay1

| |

|

[ )

Pay1

Pay1

Pump1




[ )

Pay1

| |

|

[ )

Pay1

Pay1

Pump1




[ )

Pay1

| |

|

[ )

Pay1

Pay1

Pump1



|


[ )

Pay1

| |

[ )

Pay1

Pay1

Pump1



|


[ )

Pay1

| |

[ )

Pay1

Pay1

Pump1



Fair1.2: If Cus(1) prepays before Cus(2), then Cus(1) gets to pump before Cus(2) gets to pump



Fair1.2:

|

[ )Pump1

Pump2

[ )

Pay1

| |

Pay1

Pay2

Pay1

| |

Pay1



Fair1.2:

[ )

Pay1

| |

Pay1

Pay2

| |

|

[ )Pump1

Pump2

Pay1 Pay1



Fair1.2:

[ )

Pay1

| |

Pay1

Pay2

| |

|

[ )Pump1

Pump2

Pay1 Pay1



Fair1.2:

[ )

Pay1

| |

Pay1

Pay2

| |

|

[ )Pump1

Pump2

Pay1 Pay1



Fair1.2:

[ )

Pay1

| |

Pay1

Pay2

| |

|

[ )Pump1

Pump2

Pay1 Pay1



init

acc C

(1):

Op

.pre

acc C

(2):

Op

.pre

acc C

(3):

Op

.pre

acc C

(1):

Pp

.sta

rt

en

d C

(1):

Pp

.fin

acc C

(2):

Pp

.sta

rt

acc O

p:C

(1).

ch

an

en

d C

(2):

Pp

.fin

acc C

(1):

Op

.pre

acc C

(3):

Pp

.sta

rt

acc O

p:C

(2).

ch

an

en

d C

(3):

Pp

.fin

acc C

(2):

Op

.pre

acc C

(3):

Op

.pre

acc C

(1):

Pp

.sta

rt

Pay1Pump1Pay2Pump2[ )

Pay1

| |

Pay1

[ )Pay1

Violates

DoesPump1



A GIL formula, f, determines a deterministic finite state automaton, Df , such that, for every

finite state sequence, s, . . .

s f

if and only if

Accept(s, Df )



5

DFA for DoesPump1 0 1

2

4 3

P1

P1

P1

P1

P1

P1

P1 M1

P1 M1

P1 M1

P1 M1

P1

P1 = Pay1M1 = Pump1

trueP1



Checking an execution trace

init

acc C

(1):

Op

.pre

acc C

(2):

Op

.pre

acc C

(3):

Op

.pre

acc C

(1):

Pp

.sta

rt

en

d C

(1):

Pp

.fin

i

acc C

(2):

Pp

.sta

rt

acc O

p:C

(1).

ch

an

en

d C

(2):

Pp

.fin

acc C

(1):

Op

.pre

acc C

(3):

Pp

.sta

rt

acc O

p:C

(2).

ch

an

en

d C

(3):

Pp

.fin

acc C

(2):

Op

.pre

acc C

(3):

Op

.pre

acc C

(1):

Pp

.sta

rt

Pay1Pump1Pay2Pump2

. . .

. . .

. . .

. . .

. . .

0 2 3 3 4 4 4 4 2 2 3 3 3 3 3 3 . . .



Summary

Testing concurrent systems is complicated by the need to check temporal properties

Temporal oracle can be produced from GIL specifications Checks that execution traces satisfy GIL

specifications Displays anomalies to help user identify faults



Yet to do

Refine heuristics for displaying failures

Conduct case studies/user studies

Integrate oracles into a proactive debugger



Acknowledgements

Thanks!

This work was supported in part by NSF grants EIA-0000433, CDA-9617310,

and CCR-9896190 Department of the Navy, Office of Naval

Research under grant N00014-01-1-0744



Meridian: An Integrated Toolkit for Developing Interactive Distributed

Applications NSF Experimental Partnership Program

Betty Cheng, Laura Dillon,

Phillip McKinley, Kurt Stirewalt



Interactive Distributed Applications (IDAs)

Examples: On-board driver/pilot navigation

systems. Computer-supported collaborative

work environments. Distributed interactive simulation.

Interact with users.Processing/data distributed across a network.



Characteristics of IDAs

Interactivity: Must interact with one or more human users. Design requires prototyping and experimentation.

Concurrency: Comprise levels of communicating, concurrent components. Analysis requires formal reasoning.

Reuse: IDAs built primarily from reusable components. E.g., comm. protocols, resource managers, data displays. Design involves selecting/specializing components.



Meridian goals Advance development technology for IDAs.

Formalize development artifacts and relationships between them

Libraries of reusable components

Provide end-to-end automation techniques.

Impact practice: Must complement existing design methods and

notations, e.g. UML



Meridian Vision

Model Editing

SpecificationAnalysis

DesignProcessing

Testing/Simulation

IDA Models IDA Constraints IDA Interface Requirements

IDA ReuseRepository

IDA ExternalParameters

SpecificationsRefinedSpecifications

Code

FeedbackUser

Req

uire

men

ts

Test Cases, Oracles

L. Dillon Software Engineering & Network Systems Laboratory Michigan State University 1 Getting Results From Testing Laura K. Dillon Software Engineering.

Documents