1 Testing, Abstraction, Theorem Proving: Better Together! Greta Yorsh joint work with Thomas Ball and Mooly Sagiv.

1

Testing, Testing, Abstraction, Abstraction,

Theorem Proving:Theorem Proving:Better Together!Better Together!

Greta Yorsh

joint work with Thomas Ball and Mooly Sagiv

2

MotivationMotivationTraditionally,• Find errors using testing • Prove absence of errors using static analysis

– abstract interpretation – theorem proving

• Recently, finding errors by a combination of static and dynamic analyses

• Our method: leverage on existing tests to prove absence of errors

Static Analysis

Testing1.

2.

3

Main ResultMain Result

• A new method for static analysis – leverages on concrete executions

– computes program invariants

– sound

– complete with respect to abstract interpreter

– terminates

4

The ideaThe idea

• Monitor existing test set executiontest set execution

• AbstractionAbstraction to generalize from a test set

• Theorem proverTheorem prover to check soundness

5

T’

abstraction

program P

Execute

Abstract

CT

Check invariant(AT,P)

AT

Check safety properties

yes

potential error

Fabricate tests

test set T

no

verified

The Core AlgorithmThe Core Algorithm

6

void foo(int x, int y){

int *px = NULL;

A: x = x + 1;

B: if (x < 4)

C: px = &x;

D: if (px == &y)

E: x = x + 1;

F: if (x < 5)

G: *px = *px + 1;

H: return; }

[pc, x, y, px]Concrete State

pc, x<5, px==NULLAbstraction predicates:

foo(2,0) foo(6,0) foo(11,0)Test set T :

7

void foo(int x, int y){

int *px = NULL;

A: x = x + 1;

B: if (x < 4)

C: px = &x;

D: if (px == &y)

E: x = x + 1;

F: if (x < 5)

G: *px = *px + 1;

H: return; }

foo(3,0)foo(2,0) foo(6,0) foo(11,0)

pc, x<5, px==NULLAbstraction predicates:

(A,t,t)

(B,t,t)

(C,t,t)

(D,t,f)

(F,t,f)

(G,t,f)

(H,t,f)

(A,f,t)

(B,f,t)

(D,f,t)

(F,f,t)

(H,f,t)

Check invariant(AT,foo)

Test set T :

[B,4,0,NULL ]

[D,4,0,NULL ]

Fabricated States

9

T’

abstraction

program P

Execute

Abstract

CT

Check invariant(AT,P)

AT

Check safety properties

yes

potential error

Fabricate tests

test set T

no

verified

The Core AlgorithmThe Core Algorithm

10

What does it require?What does it require?

• Execution from fabricate states– fault injection or debugger

• Check invariants using a theorem prover

• Fabricate states using counter-examples from the theorem prover

11

Using Theorem ProversUsing Theorem Provers

• Calls to a theorem prover – can be expensive – timeout potentially causes loss of precision– not always generate concrete counter-example

• Our method is oriented towards finding a proof rather than detecting errors

• Checking invariants requires less theorem prover calls than computing invariants

12

void foo(int x, int y){

int *px = NULL;

A: x = x + 1;

B: if (x < 5)

C: px = &x;

D: if (px == &y)

E: x = x + 1;

F: if (x < 5)

G: *px = *px + 1;

H: return; }

foo(3,0)foo(2,0) foo(6,0) foo(11,0)

pc, x<5, px==NULLAbstraction predicates:

Test set T :

Check invariant(AT,foo)

[D,4,0,&y ]

[E,4,0,&y ]

14

• Bisimulation – every error is real• Weak reachability – every abstract state

represents a feasible concrete state• Simulation

– error in concrete system is error in abstract system

– avoids refinements unnecessary to prove the property

(our method)

Checking InvariantsChecking Invariants

Bisimulation Weak reachability Simulation

lessprecise

moreprecise

(our method)

15

2-Process Bakery2-Process Bakery

• Proof by Bisimulation– results from [C. Pasareanu et. al. - CAV’05]– 5 refinement steps

• Our method– invariant with 17 abstract states– no refinement

17

void foo(int x, int y){

int *px = NULL;

A: x = x + 1;

B: if (x < 5)

C: px = &x;

D: if (px == &y)

E: x = x + 1;

F: if (x < 5)

G: *px = *px + 1;

H: return; }

foo(3,0)foo(2,0) foo(6,0) foo(11,0)

pc, x<10 , px==NULLAbstraction predicates:

Test set T :

19

abstraction

T’

program P

Execute

Abstract

CT

Check invariant(AT,P)

AT

Check safety properties

yes

potential error

Fabricate tests

test set T

no

verified

The AlgorithmThe Algorithm

classify errors

false alarmrefine

abstraction

real error

abstraction ’

20

Abstract InterpretationAbstract Interpretation

• Potentially more efficient– execution and abstraction of concrete states is fast

– avoid abstract transformers for parametric domains

– no static abstract transition system

– one concrete state can “discover” many abstract states

• Potentially more precise – the result is the least fixpoint w.r.t.

(if all theorem prover calls were conclusive)

21

Hybrid ApproachHybrid Approach

• It is sound to stop the concrete execution at any moment and check invariant(A,P)

• Alternate between concrete execution and abstract interpretation

• Tune the performance of the analysis execution time vs. theorem proving time– timeout new abstract state not covered– while (x < 106) x++;

22

a0

...

...

23

Further Research DirectionsFurther Research Directions

• Robust implementation and evaluation– based on a SAT solver

• Fabricated states – cover more abstract states – turn fabricated traces into concrete traces – guide refinement

• Benefits for testing– new notion of coverage related to errors– eliminate redundant tests– infer procedures pre- and post-conditions– unit tests generation

24

The MostThe Most Related Work Related Work

• [T.Reps et al. – VMCAI’04 ]

Best abstract transformers for statements

• [T.Ball – FMCO’04]

Weak reachability

• [D.Lee, M.Yannakakis – STOC’92]

• [C. Pasareanu et. al. - CAV’05]

Bisimulation

25

SummarySummary

• A new method for static analysis– computing invariants– leverage existing test suite– Abstraction to generalize from a test– Theorem prover to check soundness– Model generator to create new tests

• Potentially faster and more precise than abstract interpretation

• Explain abstract interpretation in terms of concrete execution and abstraction