Aditya V. Nori, Sriram K. Rajamani Microsoft Research India.

An Empirical Study of Optimizations in Yogi

Aditya V. Nori, Sriram K. Rajamani

Microsoft Research India

What is Yogi? An industrial strength program verifier

Idea: Synergize verification and testing

Synergy [FSE ’06], Dash [ISSTA ‘08], SMASH [POPL ‘10] algorithms to perform scalable analysis

Engineered a number of optimizations for scalability

Integrated with Microsoft’s Static Driver Verifier (SDV) toolkit and used internally

Motivation

Share our experiences in making Yogi robust, scalable and industrial strength

Several of the implemented optimizations are folklore Very difficult to design tools that are bug free evaluating

optimizations is hard! Our empirical evaluation gives tool builders information about

what gains can be realistically expected from optimizations

Vanilla implementation of algorithms: (flpydisk, CancelSpinLock) took 2 hours

Algorithms + engineering + optimizations: (flpydisk, CancelSpinLock) took less than 1 second!

Outline

Overview of Yogi

Overview of optimizations

Evaluation setup

Empirical Results

Summary

Property checking

void foo(){ *p = 4; *q = 5; if (condition) error();}

QuestionIs error() unreachable for all possible inputs?

Verification: can prove the absence of bugs, but can result in false errorsTesting: finds bugs, but can’t prove their absence

The Yogi algorithm

no

no

Can extend test beyond frontier?

Refine abstraction

Construct initial abstractionConstruct random tests

Test succeeded? Bug!

Abstractionsucceeded?

τ = error path in abstraction f = frontier of error path

yes

no

yes

Proof! yes

Input:Program P

Property ψ

Example: Abstraction & Tests

no

no


Refine abstraction





yes

no

yes

Proof! yes

Input:Program P

Property ψ

void foo(int y){0: int x, lock = 0;1: do {2: lock = 1;3: x = y;4: if (*) {5: lock = 0;6: y = y+1; }7: } while (x != y);8: if (lock != 1)9: error();10:}

y = 101234

56

789

×

× ×

× ×

× ×

× ×

×

×

× ×

×

10×

Symbolic execution +

Theorem proving

Example: Refinement

no

no


Refine abstraction





yes

no

yes

Proof! yes

Input:Program P

Property ψ


01234

56

78:ρ

9

×

× ×

× ×

× ×

× ×

×

×

× ×

×

10×

8:¬ρ×

Example: Proof!

no

no


Refine abstraction





yes

no

yes

Proof! yes

Input:Program P

Property ψ


012

34:¬s5:¬s6:¬r

9

×

× ×

× ×××

××

×

×

7:¬q×

8:¬p×

4:s5:s6:r7:q8:p×

10

Optimizations

Initial abstraction from property predicates

Relevance heuristics for predicate abstraction Suitable predicates (SP) Control dependence predicates (CD)

Interprocedural analysis Global modification analysis Summaries for procedures

Thresholds for tests

Fine tuning environment models

Evaluation setup

Benchmarks: 30 WDM drivers and 83 properties (2490 runs) Anecdotal belief: most bugs in the tools are

usually caught with this test suite

Presentation methodology: Group optimizations logically such that related

optimizations are in the same group Total time taken, total number of defects found

for every possible choice of enabling/disabling each optimization in the group

Initial abstraction

state { enum {Locked = 0, Unlocked = 1} state = Unlocked;}

KeAcquireCancelSpinlock.Entry { if (state != Locked) { state = Locked; } else abort;}

KeReleaseCancelSpinlock.Entry { if (state == Locked) { state = Unlocked; } else abort;}

01

(𝑠𝑡𝑎𝑡𝑒≠𝐿𝑜𝑐𝑘𝑒𝑑)

01

(𝑠𝑡𝑎𝑡𝑒=𝐿𝑜𝑐𝑘𝑒𝑑)

01𝑇

𝑇

Empirical resultsAbstractio

n using SLIC predicates

Total time

(minutes)

#defects #timeouts

yes 2160 241 77no 2580 241 86

16%

Relevance heuristics (SP)

Avoid irrelevant conjuncts

AC

𝑇

𝑇

B𝑇

D 𝛿

AC

𝑇

¬𝜌

B𝑇

D 𝛿

C 𝜌

𝑎𝑠𝑠𝑢𝑚𝑒(𝜙)

𝑎𝑠𝑠𝑢𝑚𝑒(𝜙)

Irrelevant?

Relevance heuristics (CD)

Abstract assume statements that are not potentially relevant by skip statements

If Yogi proves that the program satisfies property, we are done.

Otherwise, validate the error trace

and refine the abstraction by putting back assume statements, if the error trace is spurious

Example: SP heuristic

int x;void foo() { bool protect = true; … if (x > 0) protect = false; … if (protect) KeAcquireCancelSpinLock(); for (i = 0; i < 1000; i++) { a[i] = readByte(i); } if (protect) KeReleaseCancelSpinLock();}

AC

𝑇

𝑇

B𝑇

D 𝑠𝑡𝑎𝑡𝑒=𝑙𝑜𝑐𝑘𝑒𝑑

AC

𝑇

¬𝜌

B𝑇


C 𝜌

𝑎𝑠𝑠𝑢𝑚𝑒( 𝑖>1000)


𝜌=(state=Locked )∧( 𝑖>1000)

Example: SP heuristic


AC

𝑇

𝑇

B𝑇


AC

𝑇

¬𝜌

B𝑇


C 𝜌



𝜌=(state=Locked )

Example: CD heuristic


Empirical resultsSP

heuristic

CD heurist

ic

Total time

(minutes)

#defects

#timeouts

yes yes 2160 241 77yes no 2580 239 91no yes 2400 238 87no no 2894 235 174

10%

Empirical resultsSP

heuristic

CD heurist

ic

Total time

(minutes)

#defects

#timeouts


16%

Empirical resultsSP

heuristic

CD heurist

ic

Total time

(minutes)

#defects

#timeouts


25%

Interprocedural analysis

Yogi performs a compositional analysis : Is it possible to execute starting from

state and reach state ?

Global modification analysis

May-Must analysis (SMASH, POPL 2010)

Example

AC

𝑇

𝑇

B𝑇


AC

𝑇

¬𝜌

B𝑇


C 𝜌

𝑓𝑜𝑜(…)

foo(…)

⟨𝜙1 , 𝑓𝑜𝑜 (…) ,𝜙2 ⟩

Empirical resultsModificat

ion analysis

Summaries

Total time

(minutes)

#defects

#timeouts


32%


ion analysis

Summaries

Total time

(minutes)

#defects

#timeouts


28%


ion analysis

Summaries

Total time

(minutes)

#defects

#timeouts


42%

Testing

Yogi relies on tests for “cheap” reachability

Long tests avoiding several potential reachability

queries results in too many states and thus

memory consumption

Test thresholds: time vs. space tradeoff

Empirical evaluation

Test threshol

d

Total time

(minutes)

#defects

#timeouts

250 2600 236 92500 2160 241 771000 2359 240 881500 2400 239 89

Modeling the environment

if (DestinationString) { DestinationString->Buffer = SourceString;

// DestinationString->Length should be set to the // length of SourceString. The line below is missing // from the original stub SDV function DestinationString->Length = strlen(SourceString);}

if (SourceString == NULL){ DestinationString->Length = 0; DestinationString->MaximumLength = 0;}

Issue type #issues

Integers used as pointers

8Uninitialized

variables15

Type inconsistencies 9

Summary

Described optimizations implemented in Yogi Evaluated optimizations on the WDM test suite

Empirical data used to decide which optimizations to include in Yogi

We believe that this detailed empirical study of optimizations will enable tool builders to decide which optimizations to include and how to engineer their tools

http://research.microsoft.com/yogi

http://research.microsoft.com/yogi

Aditya V. Nori, Sriram K. Rajamani Microsoft Research India.

Documents