Lazy Annotation for Program Testing and Verification (Supplementary Materials) Speaker: Chen-Hsuan Adonis Lin Advisor: Jie-Hong Roland Jiang December 3,

mm

Lazy Annotation for Program Testing and Verification

(Supplementary Materials)

Speaker: Chen-Hsuan Adonis LinAdvisor: Jie-Hong Roland Jiang

December 3, 2010

1

mm

OutlineHow to compute Interpolants of

program sequenceConcolic Approach (without learning)

Dart: Directed Automated Random Testing

December 3, 2010

2

mm

Strongest and Weakest InterpolantsIf I and I′ are both interpolants for (F,G), then

so are I∧I′ and I∨I′Let F ∧ G be unsatisfiable. The strongest

interpolant for (F, G), denoted SI (F, G), is the unique interpolant for (F, G) that implies any other interpolant. The weakest interpolant for (F,G), denoted WI(F,G), is the unique interpolant that is implied by any other interpolant

SI (F, G) implies WI (F, G)

December 3, 2010

3

mm

Interpolants of SequencesWe want to handle program paths, therefore a

generalization of interpolant is needed.Given a sequence of formulas Γ = A1,A2,…,An, we

say that Ā 0, Ā 1,…, Ā n is in an interpolant for Γ when:Ā 0 = TRUE and Ā n = FALSE,For all 1≤i≤n, Ā i-1∧Ai implies Ā i, andFor all 1≤i≤n, Ā i is in L(A1,…,Ai)∩L(Ai+1,…,An)

If Γ is quantifier-free we can derive a quantifier-free interpolant for Γ (from the refutation of Γ )

December 3, 2010

4

mm

Interpolants for Sequences (con’t)

An intuition:

So this is a structured refutation of A1, …, Ak

(Ā i ∧ Ai+1) implies Ā i+1

December 3, 2010

5

A1 A2 A3 Ak...

Ā1 Ā2 Ā3 Āk-1...True False

mm

Iterative Computation of Interpolants

Given a formula F = F1 ∧ . . . ∧ Fn, determine whether F is unsatisfiable, and if so, find interpolants for the pairs (F ..i, F i+1..), i∈{1,...,n}, where F..i := F1∧...∧Fi and Fi+1.. :=Fi+1∧...∧Fn

Each formula Fi models a program instruction

A formula F = F1 ∧ . . . ∧ Fn models a trace through a program

In order to check if the trace is feasible or spurious, one can check if F is satisfiable or unsatisfiable

December 3, 2010

6

mm

Iterative Computation of Interpolants (con’t)

Definition (Tracking Property) Let F1 ∧ . . . ∧ Fn be unsatisfiable, and let Ki be interpolants

for (F ..i, F i+1..). We say that the family {Ki} satisfies the tracking property if ( Ki ∧ Fi+1 |= Ki+1)

Proposition: Let F1 ∧ F2 ∧ . . . ∧ Fn be unsatisfiable. Let {Ii} and {Ji} be families of predicates defined according to the following procedures: I0 := true, Ii+1 := any interpolant for (Ii ∧Fi+1, Fi+2..),

where i=0, ..., n−1 Jn := false, Ji−1 := any interpolant for (F ..i−1, ¬(Fi → Ji)),

where i=n, ..., 1{Ii} and {Ji} are interpolants for (F ..i,F i+1..) and satisfy

the tracking property

December 3, 2010

7

mm

Iterative Computation of Interpolants (con’t)

Ｉ nterpolants satisfying the tracking property “explain” the infeasibility of a trace by providing Hoare annotations

Evaluate strongest interpolants (Ii) {true} X := true {X} Y := X {X ∧ Y } assume(¬Y ∧ Z)

{false}

Evaluate weakest interpolants (Ji) {true} X := true {X ∨ ¬Z} Y := X {Y ∨ ¬Z} assume(¬Y ∧ Z)

{false}

By definition, Ii |= Ji; Ex: (X∧Y) |= (Y∨¬Z)

December 3, 2010

8

mm

Iterative Computation of Interpolants (con’t)

Evaluate strongest interpolants (Ii) {true} X := true {X} Y := X {X ∧ Y } assume(¬Y ∧ Z) {false

Evaluate weakest interpolants (Ji) {true} X := true {X ∨ ¬Z} Y := X {Y ∨ ¬Z} assume(¬Y ∧ Z) {false}

Intuitively, the strongest interpolants at node n records all facts that are established by the path leading up to n Ex: the strongest interpolant at node 2 is {X ∧ Y}

Intuitively, the weakest interpolant at n represents the disjunction of all conditions that make the trace infeasible if they hold at n Ex: the weakest interpolant at node 2 is {Y ∨¬Z}

December 3, 2010

9

mm

OutlineHow to compute Interpolants of

program sequenceConcolic Approach (without learning)

Dart: Directed Automated Random Testing

December 3, 2010

10

mm

Motivation of software testingToday, QA is mostly testing

“50% of my company employees are testers, and the rest spends 50% of their time testing!”

-- Bill Gates 1995

December 3, 2010

11

mm

Concolic ApproachCombine concrete and symbolic execution for

unit testing (Concrete + Symbolic = Concolic)DART: Directed Automated Random Testing

Proceedings of the 2005 ACM SIGPLAN conference on Programming language design and implementation

Authors Patrice Godefroid (Bell Labs)Nils Klarlund (Bell Labs)Koushik Sen (CS, UIUC)

December 3, 2010

12

mm

Example (C code)int double(int x) {

return 2 * x;

}

void test_me(int x, int y) {

int z = double(x);

if (z==y) {

if (y == x+10)

abort(); /* error */

}

}

(1) Interface extraction:• parameters of toplevel function• external variables• return values of external functions

main(){

int tmp1 = randomInt();

int tmp2 = randomInt();

test_me(tmp1,tmp2);

}

(2) Generation of test driver for random testing:

Problem: probability of reaching abort() is extremely low!December 3, 2010

13

mm

DART: Directed Searchmain(){

int t1 = randomInt();

int t2 = randomInt();

test_me(t1,t2);

}

int double(int x) {return 2 * x; }

void test_me(int x, int y) {

int z = double(x);

if (z==y) {

if (y == x+10)

abort(); /* error */

}

}

Concrete Execution

Symbolic Execution

Path Constraint

x = 36, y = 99x = 36, y = 99create create symbolicsymbolicvariables x, y variables x, y

December 3, 2010

14

mm

DART: Directed Searchmain(){

int t1 = randomInt();

int t2 = randomInt();

test_me(t1,t2);

}

int double(int x) {return 2 * x; }

void test_me(int x, int y) {

int z = double(x);

if (z==y) {

if (y == x+10)

abort(); /* error */

}

}

Concrete Execution

Symbolic Execution

Path Constraint

create symboliccreate symbolicvariables x, y variables x, y

x = 36, y = 99,x = 36, y = 99,z = 72z = 72

z = 2 * xz = 2 * x

December 3, 2010

15

mm

DART: Directed Searchmain(){

int t1 = randomInt();

int t2 = randomInt();

test_me(t1,t2);

}

int double(int x) {return 2 * x; }

void test_me(int x, int y) {

int z = double(x);

if (z==y) {

if (y == x+10)

abort(); /* error */

}

}

Concrete Execution

Symbolic Execution

Path Constraint

create symboliccreate symbolicvariables x, y variables x, y

x = 36, y = 99,x = 36, y = 99,z = 72z = 72

z = 2 * xz = 2 * x

2 * x != y2 * x != y

Solve: 2 * x == ySolve: 2 * x == y

Solution: x = 1, y = 2Solution: x = 1, y = 2

December 3, 2010

16

mm

DART: Directed Searchmain(){

int t1 = randomInt();

int t2 = randomInt();

test_me(t1,t2);

}

int double(int x) {return 2 * x; }

void test_me(int x, int y) {

int z = double(x);

if (z==y) {

if (y == x+10)

abort(); /* error */

}

}

Concrete Execution

Symbolic Execution

Path Constraint

x = 1, y = 2x = 1, y = 2create symboliccreate symbolicvariables x, y variables x, y

December 3, 2010

17

mm

DART: Directed Searchmain(){

int t1 = randomInt();

int t2 = randomInt();

test_me(t1,t2);

}

int double(int x) {return 2 * x; }

void test_me(int x, int y) {

int z = double(x);

if (z==y) {

if (y == x+10)

abort(); /* error */

}

}

Concrete Execution

Symbolic Execution

Path Constraint

create symboliccreate symbolicvariables x, y variables x, y

x = 1, y = 2, z = 2x = 1, y = 2, z = 2 z = 2 * xz = 2 * x

December 3, 2010

18

mm

DART: Directed Searchmain(){

int t1 = randomInt();

int t2 = randomInt();

test_me(t1,t2);

}

int double(int x) {return 2 * x; }

void test_me(int x, int y) {

int z = double(x);

if (z==y) {

if (y == x+10)

abort(); /* error */

}

}

Concrete Execution

Symbolic Execution

Path Constraint

create symboliccreate symbolicvariables x, y variables x, y

x = 1, y = 2, z = 2x = 1, y = 2, z = 2 z = 2 * xz = 2 * x 2 * x == y2 * x == y

December 3, 2010

19

mm

DART: Directed Searchmain(){

int t1 = randomInt();

int t2 = randomInt();

test_me(t1,t2);

}

int double(int x) {return 2 * x; }

void test_me(int x, int y) {

int z = double(x);

if (z==y) {

if (y == x+10)

abort(); /* error */

}

}

Concrete Execution

Symbolic Execution

Path Constraint

create symboliccreate symbolicvariables x, y variables x, y

2 * x == y2 * x == y

x = 1, y = 2, z = 2x = 1, y = 2, z = 2 z = 2 * xz = 2 * x

y != x + 10y != x + 10

Solve: (2 * x == y) Solve: (2 * x == y) Æ Æ (y == x +10)(y == x +10)

Solution: x = 10, y = 20Solution: x = 10, y = 20

December 3, 2010

20

mm

DART: Directed Searchmain(){

int t1 = randomInt();

int t2 = randomInt();

test_me(t1,t2);

}

int double(int x) {return 2 * x; }

void test_me(int x, int y) {

int z = double(x);

if (z==y) {

if (y == x+10)

abort(); /* error */

}

}

Concrete Execution

Symbolic Execution

Path Constraint

x = 10, y = 20x = 10, y = 20create symboliccreate symbolicvariables x, y variables x, y

December 3, 2010

21

mm

DART: Directed Searchmain(){

int t1 = randomInt();

int t2 = randomInt();

test_me(t1,t2);

}

int double(int x) {return 2 * x; }

void test_me(int x, int y) {

int z = double(x);

if (z==y) {

if (y == x+10)

abort(); /* error */

}

}

Concrete Execution

Symbolic Execution

Path Constraint

create symboliccreate symbolicvariables x, y variables x, y

x = 10, y = 20, z = 20x = 10, y = 20, z = 20 z = 2 * xz = 2 * x

December 3, 2010

22

mm

DART: Directed Searchmain(){

int t1 = randomInt();

int t2 = randomInt();

test_me(t1,t2);

}

int double(int x) {return 2 * x; }

void test_me(int x, int y) {

int z = double(x);

if (z==y) {

if (y == x+10)

abort(); /* error */

}

}

Concrete Execution

Symbolic Execution

Path Constraint

create symboliccreate symbolicvariables x, y variables x, y

x = 10, y = 20, z = 20x = 10, y = 20, z = 20 z = 2 * xz = 2 * x 2 * x == y2 * x == y

December 3, 2010

23

mm

DART: Directed Searchmain(){

int t1 = randomInt();

int t2 = randomInt();

test_me(t1,t2);

}

int double(int x) {return 2 * x; }

void test_me(int x, int y) {

int z = double(x);

if (z==y) {

if (y == x+10)

abort(); /* error */

}

}

Concrete Execution

Symbolic Execution

Path Constraint

create symboliccreate symbolicvariables x, y variables x, y

2 * x == y2 * x == y

y == x +10y == x +10z = 2 * xz = 2 * xx = 10, y = 20, z = 20x = 10, y = 20, z = 20

Program Error

December 3, 2010

24

mm

Concolic Testing: A Middle Approach

+ Complex programs

+ Efficient

- Less coverage

+ No false positive

- Simple programs

- Not efficient

+ High coverage

- False positive

Random Testing

Symbolic Testing

Concolic Testing

+ Complex programs+/- Somewhat efficient+ High coverage+ No false positive

December 3, 2010

25

mm

Limitations: A Comparative View

Concolic: Broad, shallow

Random: Narrow, deepDecember 3, 2010

26

mm

Hybrid Concolic Testing Interleave Random Testing and Concolic Testing to increase

coverage

Deep, broad, hybrid Search

December 3, 2010

27

mm

Thanks for your attention

December 3, 2010

28