Josh Berdine, Cristiano Calcagno, Peter O’Hearn 1 July ...oliveras/espai/smtSlides/peter.pdf · Proof Procedures for Separated Heap Abstractions Josh Berdine, Cristiano Calcagno,

Proof Procedures for Separated Heap Abstractions

Josh Berdine, Cristiano Calcagno, Peter O’Hearn

1 July, 2007 (Canada day)

Part 0

Pre-Intro

2

Separation Logic

x|->y * y|-> x

x y

3

Separation Logic

x|->y * y|-> x

x y

4

Separation Logic

x|->y

x y

5

Separation Logic

y|-> x

x y

6

Separation Logic

x|->y * y|-> x

x y

7

Separation Logic

x|->y * y|-> x

x y

x=10

y=4242

10 42

10

8

Separation Logic

x|->y * y|-> x

x y

x=10

y=4242

10 42

10

9

Separation Logic

x|->y

x y

x=10

y=4242

10

10

Separation Logic

y|-> x

x y

x=10

y=42

42

10

11

Separation Logic

x|->y * y|-> x

x y

12

Part I

Introduction

13

Example: DisposeTree

I procedure DispTree(p)local i , j ;if p 6=nil then

i = p�l ; j := p�r ;DispTree(i);DispTree(j);dispose(p)

14

Example: DisposeTree

II procedure DispTree(p)local i , j ;if p 6=nil then

i = p�l ; j := p�r ;DispTree(i);DispTree(j);dispose(p)

I An Unhappy Attempt to Specify

{tree(p) ∧ reach(p, n)}DispTree(p){¬allocated(n)}

14

Example: DisposeTree

I procedure DispTree(p)local i , j ;if p 6=nil then

i = p�l ; j := p�r ;DispTree(i);DispTree(j);dispose(p)

I An Unfortunate Fix

{tree(p) ∧ reach(p, n)∧¬reach(p,m) ∧ allocated(m) ∧m.f = m′ ∧ ¬allocated(q)}DispTree(p){¬allocated(n)∧¬reach(p,m) ∧ allocated(m) ∧m.f = m′ ∧ ¬allocated(q)}

14

Separation Logic

I In Separation Logic, the spec is just

{tree(p)} DispTree(p) {emp}

I Key part of proof

{p 7→[l : i , r : j ] ∗ tree(i) ∗ tree(j)}DispTree(i);{p 7→[l : i , r : j ] ∗ tree(j)}DispTree(j);{p 7→[l : i , r : j ]}dispose(p){emp}

{P}C{Q}{P ∗ R}C{Q ∗ R} Frame Rule

15

Some Background on Heap Verification

I Pointer Assertion Logic EngineI Uses MSOL. High complexity, good completeness.I (Intentionally) unsound treatment of procedures (framing)I No disposal or address arithmetic

I Boogie.I SoundI Improving treatment of frames...I Limited inductionI Class InvariantsI Relative of ESCI No disposal or address arithmetic

I Sagiv et. al. 3-valued shape analysisI Inferring invariants, good automationI Limited treatment of procedures (so far); global, and hard to make localI No disposal or address arithmetic

16

Context/Summary

I Sep logic has a lot of locality built in.

{P}C{Q}{P ∗ R}C{Q ∗ R} Frame Rule

{P1}C1{Q1} {P2}C2{Q2}{P1 ∗ P2}C1 ‖ C2{Q1 ∗ Q2}

Concurrency Rule

I Happy with disposal and address arithmetic.I Simple aim: try and see what we can do. So far..

I Static assertion checking: Smallfoot.I Program analysis: Space Invader.

17

Context/Summary

I Sep logic has a lot of locality built in.

{P}C{Q}{P ∗ R}C{Q ∗ R} Frame Rule

{P1}C1{Q1} {P2}C2{Q2}{P1 ∗ P2}C1 ‖ C2{Q1 ∗ Q2}

Concurrency Rule

I Happy with disposal and address arithmetic.

I Simple aim: try and see what we can do. So far..I Static assertion checking: Smallfoot.I Program analysis: Space Invader.

17

Context/Summary

I Sep logic has a lot of locality built in.

{P}C{Q}{P ∗ R}C{Q ∗ R} Frame Rule

{P1}C1{Q1} {P2}C2{Q2}{P1 ∗ P2}C1 ‖ C2{Q1 ∗ Q2}

Concurrency Rule

I Happy with disposal and address arithmetic.I Simple aim: try and see what we can do. So far..

I Static assertion checking: Smallfoot.I Program analysis: Space Invader.

17

Part II

Smallfoot Basics

18

Smallfoot AssertionsA special form1

(B1 ∧ · · · ∧ Bn)∧(H1 ∗ · · · ∗ Hm)

where

H ::= E 7→ρ | tree(E ) | lseg(E ,E )B ::= E=E | E 6=E

E ::= x | nil

ρ ::= f1 : E1, . . . , fn : En

B ::= E=E | E 6=E

Smallfoot also has predicates for doubly- and xor-linked lists, but I’ll ignorethose.

1assertional if-then-else as well19

Smallfoot Programs

Procedure declarationsf (~p ; ~v)[Pf ] Cf [Qf ]

with pre/post and reference params ~p and value params ~v

Commands include

x :=E�f E�f :=E x := new() dispose (E )

Loops come with invariants (inferred in Space Invader)

20

Verification = Symbolic Execution + Entailment Checking

I Inductive Definitions unrolled only on demand (on heap access)during execution.

I Rolled up only after execution, during entailment checking

I The tree definition

tree(E ) ⇐⇒ if E=nil then emp

else ∃x , y . (E 7→l : x , r : y) ∗ tree(x) ∗ tree(y)

21

Copytree Verification

Just inside the if (where p 6= nil)...

{p 6= nil ∧ tree(p)}

tree(E ) ⇐⇒ if E=nil then emp

else ∃x , y . (E 7→l : x , r : y) ∗ tree(x) ∗ tree(y)

22

Copytree Verification

Just inside the if (where p 6= nil)...

{p 6= nil ∧ tree(p)} unroll it...

tree(E ) ⇐⇒ if E=nil then emp

else ∃x , y . (E 7→l : x , r : y) ∗ tree(x) ∗ tree(y)

22

Copytree Verification

Just inside the if (where p 6= nil)...

{p 6= nil ∧ tree(p)} unroll it...{p 7→[l : x , r : y ] ∗ tree(x) ∗ tree(y)}

tree(E ) ⇐⇒ if E=nil then emp

else ∃x , y . (E 7→l : x , r : y) ∗ tree(x) ∗ tree(y)

22

Copytree Verification

Just inside the if (where p 6= nil)...

{p 6= nil ∧ tree(p)} unroll it...{p 7→[l : x , r : y ] ∗ tree(x) ∗ tree(y)}i := p�l ; j := p�r ;{p 7→[l : i , r : j ]∗tree(i) ∗ tree(j)}

tree(E ) ⇐⇒ if E=nil then emp

else ∃x , y . (E 7→l : x , r : y) ∗ tree(x) ∗ tree(y)

22

Copytree Verification

Just inside the if (where p 6= nil)...

{p 6= nil ∧ tree(p)} unroll it...{p 7→[l : x , r : y ] ∗ tree(x) ∗ tree(y)}i := p�l ; j := p�r ;{p 7→[l : i , r : j ]∗tree(i) ∗ tree(j)}tree copy(ii ; i) ; tree copy(jj ; j)s:= new() ; s�l := ii ; s�r := jj ;

tree(E ) ⇐⇒ if E=nil then emp

else ∃x , y . (E 7→l : x , r : y) ∗ tree(x) ∗ tree(y)

22

Copytree Verification

Just inside the if (where p 6= nil)...

{p 6= nil ∧ tree(p)} unroll it...{p 7→[l : x , r : y ] ∗ tree(x) ∗ tree(y)}i := p�l ; j := p�r ;{p 7→[l : i , r : j ]∗tree(i) ∗ tree(j)}tree copy(ii ; i) ; tree copy(jj ; j)s:= new() ; s�l := ii ; s�r := jj ;{p 7→[l : i , r : j ] ∗ tree(i) ∗ tree(j) ∗ s 7→[l : ii , r : jj ] ∗ tree(ii) ∗ tree(jj)}

tree(E ) ⇐⇒ if E=nil then emp

else ∃x , y . (E 7→l : x , r : y) ∗ tree(x) ∗ tree(y)

22

Copytree Verification

We are left with an entailment

p 7→[l : i , r : j ] ∗ tree(i) ∗ tree(j)∗s 7→[l : ii , r : jj ] ∗ tree(ii) ∗ tree(jj)

` tree(p)∗tree(s)

tree(E ) ⇐⇒ if E=nil then emp

else ∃x , y . (E 7→l : x , r : y) ∗ tree(x) ∗ tree(y)

22

Copytree Verification

We are left with an entailment

p 7→[l : i , r : j ] ∗ tree(i) ∗ tree(j)∗s 7→[l : ii , r : jj ] ∗ tree(ii) ∗ tree(jj)

` tree(p)∗tree(s)

let me roll it...

tree(E ) ⇐⇒ if E=nil then emp

else ∃x , y . (E 7→l : x , r : y) ∗ tree(x) ∗ tree(y)

22

Flawed Copytree Failed Verification

When we mistakenly point back into the source tree

we are left with an entailment

p 7→[l : i , r : j ] ∗ tree(i) ∗ tree(j)∗s 7→[l : i , r : j ]∗tree(ii) ∗ tree(jj)

` tree(p)∗tree(s)

that we can’t roll up...

23

Part III

Proving Entailments

24

Induction and Linked ListsList segments (list(E ) is shorthand for lseg(E , nil))

lseg(E ,F ) ⇐⇒ if E = F then emp

else ∃y .E 7→tl : y ∗ lseg(y ,F )

lseg(x , y) ∗ lseg(y , x)

x

y

25

Induction and Linked ListsList segments (list(E ) is shorthand for lseg(E , nil))

lseg(E ,F ) ⇐⇒ if E = F then emp

else ∃y .E 7→tl : y ∗ lseg(y ,F )

lseg(x , t) ∗ t 7→[tl : y ] ∗ list(y)

x t y

25

Induction and Linked ListsList segments (list(E ) is shorthand for lseg(E , nil))

lseg(E ,F ) ⇐⇒ if E = F then emp

else ∃y .E 7→tl : y ∗ lseg(y ,F )

Entailment lseg(x , t) ∗ t 7→[tl : y ] ∗ list(y) ` list(x)

x t y

25

Induction and Linked ListsList segments (list(E ) is shorthand for lseg(E , nil))

lseg(E ,F ) ⇐⇒ if E = F then emp

else ∃y .E 7→tl : y ∗ lseg(y ,F )

Non-Entailment lseg(x , t) ∗ t 7→nil ∗ list(y) 6` list(x)

x t y

25

Solution (Berdine and Calcagno)

I A proof theory oriented around Abstraction and Subtraction.

I Sample Abstraction Rule

lseg(x , t) ∗ list(t) ` list(x)

I Subtraction RuleQ1 ` Q2

Q1 ∗ S ` Q2 ∗ S

I Try to reduce an entailment to the axiom

B ∧ emp ` true ∧ emp

26

Solution (Berdine and Calcagno)

I A proof theory oriented around Abstraction and Subtraction.

I Sample Abstraction Rule

lseg(x , t) ∗ list(t) ` list(x)

I Subtraction RuleQ1 ` Q2

Q1 ∗ S ` Q2 ∗ S

I Try to reduce an entailment to the axiom

B ∧ emp ` true ∧ emp

26

Solution (Berdine and Calcagno)

I A proof theory oriented around Abstraction and Subtraction.

I Sample Abstraction Rule

lseg(x , t) ∗ list(t) ` list(x)

I Subtraction RuleQ1 ` Q2

Q1 ∗ S ` Q2 ∗ S

I Try to reduce an entailment to the axiom

B ∧ emp ` true ∧ emp

26

Solution (Berdine and Calcagno)

I A proof theory oriented around Abstraction and Subtraction.

I Sample Abstraction Rule

lseg(x , t) ∗ list(t) ` list(x)

I Subtraction RuleQ1 ` Q2

Q1 ∗ S ` Q2 ∗ S

I Try to reduce an entailment to the axiom

B ∧ emp ` true ∧ emp

26

Works great!

lseg(x , t) ∗ t 7→[tl : y ] ∗ list(y) ` list(x) Abstract (Roll)

27

Works great!

lseg(x , t) ∗ list(t) ` list(x) Abstract (Inductive)lseg(x , t) ∗ t 7→[tl : y ] ∗ list(y) ` list(x) Abstract (Roll)

27

Works great!

list(x) ` list(x) Subtractlseg(x , t) ∗ list(t) ` list(x) Abstract (Inductive)lseg(x , t) ∗ t 7→[tl : y ] ∗ list(y) ` list(x) Abstract (Roll)

27

Works great!

¨̂

emp ` emp Axiom!list(x) ` list(x) Subtractlseg(x , t) ∗ list(t) ` list(x) Abstract (Inductive)lseg(x , t) ∗ t 7→[tl : y ] ∗ list(y) ` list(x) Abstract (Roll)

27

Works great!

¨̂

emp ` emp Axiom!list(x) ` list(x) Subtractlseg(x , t) ∗ list(t) ` list(x) Abstract (Inductive)lseg(x , t) ∗ t 7→[tl : y ] ∗ list(y) ` list(x) Abstract (Roll)

lseg(x , t) ∗ t 7→nil ∗ list(y) ` list(x) Abstract (Inductive)

27

Works great!

¨̂

emp ` emp Axiom!list(x) ` list(x) Subtractlseg(x , t) ∗ list(t) ` list(x) Abstract (Inductive)lseg(x , t) ∗ t 7→[tl : y ] ∗ list(y) ` list(x) Abstract (Roll)

list(x) ∗ list(y) ` list(x) Subtractlseg(x , t) ∗ t 7→nil ∗ list(y) ` list(x) Abstract (Inductive)

27

Works great!

¨̂

emp ` emp Axiom!list(x) ` list(x) Subtractlseg(x , t) ∗ list(t) ` list(x) Abstract (Inductive)lseg(x , t) ∗ t 7→[tl : y ] ∗ list(y) ` list(x) Abstract (Roll)

_̈

list(y) ` emp Junk: Not Axiom!list(x) ∗ list(y) ` list(x) Subtractlseg(x , t) ∗ t 7→nil ∗ list(y) ` list(x) Abstract (Inductive)

27

List of abstraction rules for lseg

Rolling

emp → lseg(E ,E )

E1 6=E3 ∧ E1 7→[tl :E2, ρ] ∗ lseg(E2,E3) → lseg(E1,E3)

Induction Avoidance

lseg(E1,E2) ∗ lseg(E2, nil) → lseg(E1, nil)

lseg(E1,E2) ∗ E2 7→[t : nil] → lseg(E1, nil)

lseg(E1,E2) ∗ lseg(E2,E3) ∗ E3 7→[ρ] → lseg(E1,E3) ∗ E3 7→[ρ]

E3 6=E4 ∧ lseg(E1,E2) ∗ lseg(E2,E3) ∗ lseg(E3,E4)

→ lseg(E1,E3) ∗ lseg(E3,E4)

28

Proof Procedure for Q1 ` Q2, Normalization Phase

I Substitute out all equalities

Q1[E/x ] ` Q2[E/x ]

x = E ∧ Q1 ` Q2

I Generate disequalities. E.g., using

x 7→[ρ] ∗ y 7→[ρ′] → x 6= y

I Remove empty lists and trees: lseg(x , x), tree(nil)

I Check antecedent for inconsistency, if so, return “valid”.Inconcistencies: x 7→[ρ] ∗ x 7→[ρ′] nil 7→ − x 6= x · · ·

I Check pure consequences (easy inequational logic), if failed then“invalid”

This is cubic.29

Proof Procedure for Q1 ` Q2, Abstract/Subtract Phase

Trying to prove B1 ∧ H1 ` H2

I For each spatial predicate in H2, try to apply abstraction rules tomatch it with things in H1.

I Then, apply subtraction rule.

Q1 ` Q2

Q1 ∗ S ` Q2 ∗ S

I If you are left withB ∧ emp ` true ∧ emp

report “valid”, else “invalid”

This is cubic.

30

Completeness?

I Question: from O’Hearn to Berdine/Calcago (circa 2002):

Is your procedure complete (and if not can you proveundecidability)?

I Immediate Answer: silence

I A little later: Doh!

31

Completeness?

I Question: from O’Hearn to Berdine/Calcago (circa 2002):

Is your procedure complete (and if not can you proveundecidability)?

I Immediate Answer: silence

I A little later: Doh!

31

Completeness?

I Question: from O’Hearn to Berdine/Calcago (circa 2002):

Is your procedure complete (and if not can you proveundecidability)?

I Immediate Answer: silence

I A little later: Doh!

31

Spooky Disjunctions

I The fragment does not have disjunction in it. However,

y 6=z ∧ (ls(x , y) ∗ ls(x , z))

implies that either ls(x , y) or ls(x , z) is nonempty, but we do notknow which. So it also implies x 6= y ∨ x 6= z .

I This issue can show up in an entailment

y 6=z ∧ (ls(x , y) ∗ ls(x , z)∗x 7→−)`x 6=x

which tricks the proof procedure.

I We have never fallen foul of this incompleteness in a natural examplein program verification.

I Still...

32

Spooky Disjunctions

I The fragment does not have disjunction in it. However,

y 6=z ∧ (ls(x , y) ∗ ls(x , z))

implies that either ls(x , y) or ls(x , z) is nonempty, but we do notknow which. So it also implies x 6= y ∨ x 6= z .

I This issue can show up in an entailment

y 6=z ∧ (ls(x , y) ∗ ls(x , z)∗x 7→−)`x 6=x

which tricks the proof procedure.

I We have never fallen foul of this incompleteness in a natural examplein program verification.

I Still...

32

Spooky Disjunctions

I The fragment does not have disjunction in it. However,

y 6=z ∧ (ls(x , y) ∗ ls(x , z))

implies that either ls(x , y) or ls(x , z) is nonempty, but we do notknow which. So it also implies x 6= y ∨ x 6= z .

I This issue can show up in an entailment

y 6=z ∧ (ls(x , y) ∗ ls(x , z)∗x 7→−)`x 6=x

which tricks the proof procedure.

I We have never fallen foul of this incompleteness in a natural examplein program verification.

I Still...

32

Spooky Disjunctions

I The fragment does not have disjunction in it. However,

y 6=z ∧ (ls(x , y) ∗ ls(x , z))

implies that either ls(x , y) or ls(x , z) is nonempty, but we do notknow which. So it also implies x 6= y ∨ x 6= z .

I This issue can show up in an entailment

y 6=z ∧ (ls(x , y) ∗ ls(x , z)∗x 7→−)`x 6=x

which tricks the proof procedure.

I We have never fallen foul of this incompleteness in a natural examplein program verification.

I Still...

32

Exorcising Spooky Disjunctions

I Cubic proof procedure is complete when we know all the listsegs arenonempty (when x 6= y is there for each lseg(x , y)).

I Complete procedure for general case, using excluded middle

x = y ∧ Q1 ` Q2 x 6=y ∧ Q1 ` Q2

Q1 ` Q2

I The resulting proof procedure is exponential.

I Calcagno has a polynomial procedure which uses constraints, andwhich handles spooky disjunctions. Don’t know if complete.

33

Exorcising Spooky Disjunctions

I Cubic proof procedure is complete when we know all the listsegs arenonempty (when x 6= y is there for each lseg(x , y)).

I Complete procedure for general case, using excluded middle

x = y ∧ Q1 ` Q2 x 6=y ∧ Q1 ` Q2

Q1 ` Q2

I The resulting proof procedure is exponential.

I Calcagno has a polynomial procedure which uses constraints, andwhich handles spooky disjunctions. Don’t know if complete.

33

Exorcising Spooky Disjunctions

I Cubic proof procedure is complete when we know all the listsegs arenonempty (when x 6= y is there for each lseg(x , y)).

I Complete procedure for general case, using excluded middle

x = y ∧ Q1 ` Q2 x 6=y ∧ Q1 ` Q2

Q1 ` Q2

I The resulting proof procedure is exponential.

I Calcagno has a polynomial procedure which uses constraints, andwhich handles spooky disjunctions. Don’t know if complete.

33

Exorcising Spooky Disjunctions

I Cubic proof procedure is complete when we know all the listsegs arenonempty (when x 6= y is there for each lseg(x , y)).

I Complete procedure for general case, using excluded middle

x = y ∧ Q1 ` Q2 x 6=y ∧ Q1 ` Q2

Q1 ` Q2

I The resulting proof procedure is exponential.

I Calcagno has a polynomial procedure which uses constraints, andwhich handles spooky disjunctions. Don’t know if complete.

33

Perspective

I The interesting part is the abstraction rules replacing induction, like

lseg(x , t) ∗ list(t) ` list(x)

I We can replay this work for other data structures, but (presently)some invention is needed to choose abstraction rules.

I We can infer loop invariants (abstract interpretation) by selective useof the abstraction rules .

I Distefano et al, TACAS’06I Termination analysis: Berdine et al, CAV’06 (see Cook CAV invited)

Interprocedural shape analysis: Gotsman et al, SAS’06Pointer Arithmetic: Calcagno et al, SAS’06Thread-modular shape analysis: Gotsman et al, PLDI’07Induction Synthesis: Guo et al, PLDI’07Adaptive analysis: Bertine et al, CAV’07(see Distefano CAV talk)+ 5 SAS’07 papers

34

Perspective

I The interesting part is the abstraction rules replacing induction, like

lseg(x , t) ∗ list(t) ` list(x)

I We can replay this work for other data structures, but (presently)some invention is needed to choose abstraction rules.

I We can infer loop invariants (abstract interpretation) by selective useof the abstraction rules .

I Distefano et al, TACAS’06

I Termination analysis: Berdine et al, CAV’06 (see Cook CAV invited)Interprocedural shape analysis: Gotsman et al, SAS’06Pointer Arithmetic: Calcagno et al, SAS’06Thread-modular shape analysis: Gotsman et al, PLDI’07Induction Synthesis: Guo et al, PLDI’07Adaptive analysis: Bertine et al, CAV’07(see Distefano CAV talk)+ 5 SAS’07 papers

34

Perspective

I The interesting part is the abstraction rules replacing induction, like

lseg(x , t) ∗ list(t) ` list(x)

I We can replay this work for other data structures, but (presently)some invention is needed to choose abstraction rules.

I We can infer loop invariants (abstract interpretation) by selective useof the abstraction rules .

I Distefano et al, TACAS’06I Termination analysis: Berdine et al, CAV’06 (see Cook CAV invited)

Interprocedural shape analysis: Gotsman et al, SAS’06Pointer Arithmetic: Calcagno et al, SAS’06Thread-modular shape analysis: Gotsman et al, PLDI’07Induction Synthesis: Guo et al, PLDI’07Adaptive analysis: Bertine et al, CAV’07(see Distefano CAV talk)+ 5 SAS’07 papers

34

Coalescing

I Freeing sometimes causes adjacent nodes to be coalesced in the freelist.

I For example,

x

y z

x+1 x+2 x+z

a b

x+z+b

[x+1]:= [x+1]+[x+z+1]

35

Coalescing

I Freeing sometimes causes adjacent nodes to be coalesced in the freelist.

I For example,

x

y z

x+1 x+2 x+z

a b

x+z+b

[x+1]:= [x+1]+[x+z+1]

x

yz+b

x+1 x+2 x+z

a b

x+z+b

35

Coalescing

I Freeing sometimes causes adjacent nodes to be coalesced in the freelist.

I For example,

x

y z

x+1 x+2 x+z

a b

x+z+b

[x+1]:= [x+1]+[x+z+1]

x

yz+b

x+1 x+2 x+z+b

35

RAM to Node transit

I Abstraction for transit from

x

yz+b

x+1 x+2 x+z

a b

x+z+b

tox

yz+b

x+1 x+2 x+z+b

is an implication

(x 7→y) ∗ (x+17→z + b) ∗ blk(x+2, x+z)

∗ (x+z 7→a) ∗ (x+z+17→b) ∗ blk(x+z+2, x+z+b)

=⇒nd(x , y , z + b)

36

Program LOC Heap (KB) States (Inv) States (Post) Time (sec)2

malloc firstfit 42 240 18 3 0.05

free acyclic 55 240 6 2 0.09

malloc besttfit 46 480 90 3 1.19

malloc roving 61 240 33 5 0.13

free roving 68 720 16 2 0.84

malloc K&R 179 26880 384 66 502.23

free K&R 58 3840 89 5 9.69

2Pentium 2.3GHz, 4GB RAM37