Solving Quantied Verication Conditions using …oliveras/espai/smtSlides/clark.pdfSolving Quantied Verication Conditions using Satisability Modulo Theories Yeting Ge, Clark Barrett,

Solving Quantified Verification Conditionsusing Satisfiability Modulo Theories

Yeting Ge, Clark Barrett, Cesare Tinelli

Solving Quantified Verification Conditions using Satisfiability Modulo Theories – p.1/28

Motivation

First order logic provides a convenient formalism forspecifying verification conditions

Verification conditions often involve arithmetic and otherwell-established theories

Approaches to checking verification conditions in first orderlogic

Pure first order Automated Theorem Proving (ATP)Good at reasoning about quantified formulasNot so good at theory reasoning· Some useful theories are not finitely axiomatizable

Add ’ad-hoc’ axioms (Denney et al. IJCAR 2004)

Are there any alternatives?

Motivation

Pure first order Automated Theorem Proving (ATP)

Good at reasoning about quantified formulasNot so good at theory reasoning· Some useful theories are not finitely axiomatizable

Motivation

Pure first order Automated Theorem Proving (ATP)Good at reasoning about quantified formulasNot so good at theory reasoning

· Some useful theories are not finitely axiomatizableAdd ’ad-hoc’ axioms (Denney et al. IJCAR 2004)

Motivation

Automated theorem proving based on Satisfiability Modulo Theories(SMT)

A SMT problem is to determine the satisfiability of some formulaϕ with respect to some fixed background theory T

· Is Select(Store(arr, i, a), i) 6= a satisfiable?Many useful background (combined) theories T can be decided byefficient procedures

Good at theory reasoningTraditionally for quantifier-free formulas onlyException: Simplify

Instantiation based and incompleteShown to work for practical problemsSuccessful, but no longer supported

Motivation

A SMT problem is to determine the satisfiability of some formulaϕ with respect to some fixed background theory T· Is Select(Store(arr, i, a), i) 6= a satisfiable?

Many useful background (combined) theories T can be decided byefficient procedures

Motivation

Good at theory reasoningTraditionally for quantifier-free formulas only

Exception: SimplifyInstantiation based and incompleteShown to work for practical problemsSuccessful, but no longer supported

Motivation

Outline

Quantifier reasoning in SMTSMT solvers and Abstract DPLL Modulo Theories frameworkTriggers, matching and instantiation

ChallengesTrigger selectionInstantiation loopsEager and lazy instantiationIrrelevant axioms

Experimental resultsComparison of different heuristics (in CVC3)Comparison of SMT solversComparison of ATP and SMT solvers

Outline

Solver for SMT

Modern SMT solvers (lazy) integrate a SAT solver and one ormore theory solvers

Abstraction

SATTheorySolver

Arithmetic

SMT Example

To prove (a = b) ∧ ¬(f(a) = f(b)) is unsatisfiable

SMT Example

Solver

(a = b) ∧ ¬(f(a) = f(b))

Abstraction

TheorySolver

SMT Example

Solver

(a = b) ∧ ¬(f(a) = f(b)) b1 : f(a) = f(b)b2 : a = b

Abstraction

TheorySolver

SMT Example

Solver

(a = b) ∧ ¬(f(a) = f(b))

b2 ∧ ¬b1

b1 : f(a) = f(b)b2 : a = b

Abstraction

TheorySolver

SMT Example

Solver

(a = b) ∧ ¬(f(a) = f(b))

{b1 = F b2 = T}b2 ∧ ¬b1

b1 : f(a) = f(b)b2 : a = b

Abstraction

TheorySolver

SMT Example

Solver

(a = b) ∧ ¬(f(a) = f(b))

{b1 = F b2 = T}b2 ∧ ¬b1

b1 : f(a) = f(b)b2 : a = b

f(a) 6= f(b)

Abstraction

TheorySolver

SMT Example

Solver

(a = b) ∧ ¬(f(a) = f(b))

{b1 = F b2 = T}b2 ∧ ¬b1

a = bT-unsat

b1 : f(a) = f(b)b2 : a = b

f(a) 6= f(b)

Abstraction

TheorySolver

SMT Example

Solver

(a = b) ∧ ¬(f(a) = f(b))

{b1 = F b2 = T}b2 ∧ ¬b1

No more a = bT-unsat

b1 : f(a) = f(b)b2 : a = b

f(a) 6= f(b)

Abstraction

TheorySolver

SMT Example

Solver

T-unsat

(a = b) ∧ ¬(f(a) = f(b))

{b1 = F b2 = T}b2 ∧ ¬b1

b1 : f(a) = f(b)b2 : a = b

f(a) 6= f(b)

Abstraction

TheorySolver

No more

Quantifier Example

To prove ¬P (3) ∧ ∀x.(x > 1→ P (x)) is unsatisfiable.

Quantifier Example

Solver

¬P (3) ∧ ∀x.(x > 1 → P (x))

TheorySolver

Abstraction

b2 : ∀x.(x > 1 → P (x))b1 : P (3)

Quantifier Example

Solver

∀x.(x > 1 → P (x))

TheorySolver

b2 ∧ ¬b1

Abstraction

b2 : ∀x.(x > 1 → P (x))b1 : P (3)¬P (3) ∧ ∀x.(x > 1 → P (x))

{b1 = F b2 = T} ¬P (3)

Quantifier Example

Solver

Instantiate x with 3

TheorySolver

b2 ∧ ¬b1

Abstraction

b2 : ∀x.(x > 1 → P (x))b1 : P (3)¬P (3) ∧ ∀x.(x > 1 → P (x))

{b1 = F b2 = T} ¬P (3)∀x.(x > 1 → P (x))

Quantifier Example

Solver

3 > 1 → P (3)

TheorySolver

b2 ∧ ¬b1

Abstraction

b2 : ∀x.(x > 1 → P (x))b1 : P (3)¬P (3) ∧ ∀x.(x > 1 → P (x))

{b1 = F b2 = T} ¬P (3)∀x.(x > 1 → P (x))Instantiate x with 3

Quantifier Example

Solver

3 > 1 → P (3)

TheorySolver

b2 ∧ ¬b1

T-unsat

Abstraction

b2 : ∀x.(x > 1 → P (x))b1 : P (3)¬P (3) ∧ ∀x.(x > 1 → P (x))

Quantifier Example

Solver

3 > 1 → P (3)

TheorySolver

b2 ∧ ¬b1

No moreT-unsat

Abstraction

b2 : ∀x.(x > 1 → P (x))b1 : P (3)¬P (3) ∧ ∀x.(x > 1 → P (x))

Quantifier Example

Solver

3 > 1 → P (3)

TheorySolver

b2 ∧ ¬b1

No moreT-unsat

Abstraction

b2 : ∀x.(x > 1 → P (x))b1 : P (3)¬P (3) ∧ ∀x.(x > 1 → P (x))

Unsat{b1 = F b2 = T} ¬P (3)

∀x.(x > 1 → P (x))Instantiate x with 3

Abstract DPLL Modulo Theories

Abstract DPLL Modulo Theories is a formalism for DPLL-based smtsolvers

Describes SMT solvers as transition systems (a set of states and transitionrules)

States:

M || F (M is a set of literals assumed so far, F is a set of CNFclauses)

Final state:Fail

M || F (M is T satisfiable and M |= F )

The goal:From initial state ∅ || F0, derive a final state

States:

Final state:Fail

States:

Final state:Fail

States:

Final state:Fail

Example of Transition Rules

Unit propagation in SATUnitPropagate :

M || F, C ∨ l =⇒ M l || F, C ∨ l if

M |= ¬Cl is undefined in M

Theory propagationT-Propagate :

M || F =⇒ M l || F if

M |=T l

l or ¬l occurs in Fl is undefined in M

Rules for Quantifier Instantiation

An abstract literal is either a literal or a quantified formulaϕ[x/t] denotes the result of substituting t for all freeoccurrences of x in ϕ

Inst_∃ :

M || F =⇒ M || F, (¬(∃x. P ) ∨ P [x/sk]) if

∃ x. P is in M

sk is a fresh constant.

Inst_∀ :

M || F =⇒ M || F, (¬(∀x. P ) ∨ P [x/t]) if

∀ x. P is in M

t is a ground term.

What to instantiate

Suppose φ = ∀x.P (f(x)) is asserted to be true

Instantiate x with every ground term (naive instantiation)Too many instantiations

Instantiate x with terms relevant to φIf some subterm of φ[x/t] appears in ground formulas inM , t is relevant to φSimilar to resolution

How to find relevant terms?1. Select a subterm of φ that contains x, say f(x)

2. If f(x) matches with a ground term that appears inM , say f(a), a is relevantf(x) is called a triggerE − unification

What to instantiate

Suppose φ = ∀x.P (f(x)) is asserted to be trueInstantiate x with every ground term (naive instantiation)

Too many instantiations

Instantiate x with terms relevant to φIf some subterm of φ[x/t] appears in ground formulas inM , t is relevant to φSimilar to resolution

What to instantiate

Too many instantiationsInstantiate x with terms relevant to φ

If some subterm of φ[x/t] appears in ground formulas inM , t is relevant to φSimilar to resolution

What to instantiate

Too many instantiationsInstantiate x with terms relevant to φ

If some subterm of φ[x/t] appears in ground formulas inM , t is relevant to φSimilar to resolution

Challenges

Given a set of quantified formulas and ground formulas1. Select some subterms of a quantified formula as triggers2. Match triggers with ground terms3. Instantiate quantified formulas

ChallengesTriggersMatching (equalities, fast matching algorithm)Instantiation

Instantiation loopsEager and lazy instantiationIrrelevant axioms

Challenges

Given a set of quantified formulas and ground formulas1. Select some subterms of a quantified formula as triggers2. Match triggers with ground terms3. Instantiate quantified formulas

ChallengesTriggersMatching (equalities, fast matching algorithm)Instantiation

Instantiation loopsEager and lazy instantiationIrrelevant axioms

Triggers

Trigger selectionTriggers should contain all bound variables

Triggers can have more bound variables than thosequantified by outermost quantifiers· ∀x : (P (x)→ ∀y : Q(x, y))· (Simplify does not allow this)

Sometimes no single subterm contains all boundvariables

Multi-triggers (as in Simplify)Special trigger heuristics

TransitivityAnti-symmetryArray index

Triggers

Multi-triggers (as in Simplify)

Special trigger heuristicsTransitivityAnti-symmetryArray index

Triggers

Multi-triggers (as in Simplify)Special trigger heuristics

TransitivityAnti-symmetryArray index

Instantiation Loops

Instantiation could introduce loops1. ∀x.P (f(x), f(g(x))) (Simplify)2. ∀x.(∃y.f(x)− f(y) = 2)

3. Loops due to several formulas

Instantiation Loops

Instantiation could introduce loops1. ∀x.P (f(x), f(g(x))) (Simplify)2. ∀x.(∃y.f(x)− f(y) = 2)f(x) is selected as trigger. Suppose f(3) appearssomewhere.f(x) matches f(3)

∃y.f(3)− f(y) = 2f(3)− f(sk1) = 2f(x) matches f(sk1)f(sk2)......

Instantiation Loops

Instantiation could introduce loops1. ∀x.P (f(x), f(g(x))) (Simplify)2. ∀x.(∃y.f(x)− f(y) = 2)f(x) is selected as trigger. Suppose f(3) appearssomewhere.f(x) matches f(3)∃y.f(3)− f(y) = 2

f(3)− f(sk1) = 2f(x) matches f(sk1)f(sk2)......

Instantiation Loops

Instantiation could introduce loops1. ∀x.P (f(x), f(g(x))) (Simplify)2. ∀x.(∃y.f(x)− f(y) = 2)f(x) is selected as trigger. Suppose f(3) appearssomewhere.f(x) matches f(3)∃y.f(3)− f(y) = 2f(3)− f(sk1) = 2

f(x) matches f(sk1)f(sk2)......

Instantiation Loops

Instantiation could introduce loops1. ∀x.P (f(x), f(g(x))) (Simplify)2. ∀x.(∃y.f(x)− f(y) = 2)f(x) is selected as trigger. Suppose f(3) appearssomewhere.f(x) matches f(3)∃y.f(3)− f(y) = 2f(3)− f(sk1) = 2f(x) matches f(sk1)

f(sk2)......

Instantiation Loops

Instantiation could introduce loops1. ∀x.P (f(x), f(g(x))) (Simplify)2. ∀x.(∃y.f(x)− f(y) = 2)f(x) is selected as trigger. Suppose f(3) appearssomewhere.f(x) matches f(3)∃y.f(3)− f(y) = 2f(3)− f(sk1) = 2f(x) matches f(sk1)f(sk2)

......

Instantiation Loops

Instantiation could introduce loops1. ∀x.P (f(x), f(g(x))) (Simplify)2. ∀x.(∃y.f(x)− f(y) = 2)f(x) is selected as trigger. Suppose f(3) appearssomewhere.f(x) matches f(3)∃y.f(3)− f(y) = 2f(3)− f(sk1) = 2f(x) matches f(sk1)f(sk2)......

Instantiation Loops

Instantiation could introduce loops1. ∀x.P (f(x), f(g(x))) (Simplify)2. ∀x.(∃y.f(x)− f(y) = 2)f(x) is selected as trigger. Suppose f(3) appearssomewhere.f(x) matches f(3)∃y.f(3)− f(y) = 2f(3)− f(sk1) = 2f(x) matches f(sk1)f(sk2)......

Instantiation Loops

Loops are not always badWe experimented two kinds of loop prevention mechanism

Static loop test (as in Simplify)Dynamic loop detectionBoth are abandoned

Instantiation Loops

3. Loops due to several formulasLoops are not always bad

We experimented two kinds of loop prevention mechanismStatic loop test (as in Simplify)Dynamic loop detectionBoth are abandoned

Instantiation Loops

3. Loops due to several formulasLoops are not always badWe experimented two kinds of loop prevention mechanism

Static loop test (as in Simplify)Dynamic loop detectionBoth are abandoned

Eager and lazy instantiation

Eager instantiationInstantiate when unit propagation rule does not apply

May find contradictions earlierMay introduce useless clauses

Lazy instantiationInstantiate when no other transition rule appliesInstantiate only when necessaryMay be too late

Is there a way to have a balance between lazy and eagerinstantiation?

Eager instantiationInstantiate when unit propagation rule does not applyMay find contradictions earlierMay introduce useless clauses

Lazy instantiationInstantiate when no other transition rule applies

Instantiate only when necessaryMay be too late

Irrelevant axioms

Verification conditions are often of the form Γ ∧ ¬ϕ where ϕis a formula and Γ is a large fixed T -satisfiable collection of(quantified) axioms

Many formulas and ground terms in Γ are irrelevant to the proof ofunsatisfiability of Γ ∧ ¬ϕ

The solver may spend a lot of resources on irrelevant axiomsIt is not always easy to determine whether axioms are relevant or not

How to prevent the solver from spending too many resourceson irrelevant axioms?

Irrelevant axioms

Many formulas and ground terms in Γ are irrelevant to the proof ofunsatisfiability of Γ ∧ ¬ϕThe solver may spend a lot of resources on irrelevant axioms

It is not always easy to determine whether axioms are relevant or not

Irrelevant axioms

Many formulas and ground terms in Γ are irrelevant to the proof ofunsatisfiability of Γ ∧ ¬ϕThe solver may spend a lot of resources on irrelevant axiomsIt is not always easy to determine whether axioms are relevant or not

Irrelevant axioms

Many formulas and ground terms in Γ are irrelevant to the proof ofunsatisfiability of Γ ∧ ¬ϕThe solver may spend a lot of resources on irrelevant axiomsIt is not always easy to determine whether axioms are relevant or not

Instantiation level: three birds one stone

Definition of instantiation level IL(g) of ground term g

All terms appearing in original problem have an instantiation level of 0If ground term g matches some trigger of ∀x.P and g has aninstantiation level IL(g), then all new terms in P [x/t] (as well as newterms derived from them) have instantiation level of IL(g) + 1.

Instantiation level and matchingOnly ground terms with an instantiation level less than an upperbound are used in matching

Instantiation strategyEager instantiationThe upper bound is increased if

CVC3 runs out of ground termsNo other transition rule applies

Iterative deepening on instantiation level

Iterative deepening on instantiation levelSolving Quantified Verification Conditions using Satisfiability Modulo Theories – p.18/28

Advantages of instantiation level

Neutralizes the harmful effect of instantiation loopsNew ground terms from instantiation will not beconsidered until the upper bound is increased

Balances the eagerness of instantiationsEagerly instantiateLazily increase the upper bound

Avoids spending too many resources on irrelevant axiomsNo ground term will have more attention than others

Experimental Results

Test cases are from SMT LIB.AUFLIA/Burns 14AUFLIA/misc 29AUFLIA/piVC 42AUFLIA/RicartAgrawala 14AUFLIA/simplify 833AUFLIRA/nasa 26504AUFNIRA/nasa 1561

Only hard cases (5599 out of 29004) are selectedAMD Opteron (64 bit), 1G memory, time limit 5 minutes

CVC3 with Different Heuristics

Category B-E B-L IL-EAUFLIA/Burns 12 12 12 12AUFLIA/misc 14 12 14 14AUFLIA/piVC 29 29 29 29AUFLIA/RicAgla 14 14 14 14AUFLIA/simplify 769 497 762 768AUFLIRA/nasa 4619 4527 4131 4526AUFNIRA/nasa 142 72 46 72

B-E : No instantiation level heuristic, with eager instantiationB-L : No instantiation level heuristic, with lazy instantiationIL-E: Instantiation level with eager instantiation

CVC3, Yices and Fx7

Yices, version of SMT competition 2006Fx7, as of Nov 15, 2006CVC3, version 1.1

fx7 yices CVC3

category #case #valid time #valid time #valid time

AUFLIA/Burns 12 12 0.429 12 0.011 12 0.020

AUFLIA/misc 14 12 0.682 14 0.050 14 0.048

AUFLIA/piVC 29 15 0.517 29 0.030 29 0.106

AUFLIA/RicAgla 14 14 0.640 14 0.026 14 0.041

AUFLIA/simplify 769 760 3.218 740 1.424 768 0.739

AUFLIRA/nasa 4619 4187 0.452 4520 0.082 4526 0.014

AUFNIRA/nasa 142 48 0.410 N/A N/A 72 0.012

CVC3, Yices and Fx7

Yices, version of SMT competition 2006Fx7, as of Nov 15, 2006CVC3, version 1.1

fx7 yices CVC3

category #case #valid time #valid time #valid time

AUFLIA/Burns 12 12 0.429 12 0.011 12 0.020

AUFLIA/misc 14 12 0.682 14 0.050 14 0.048

AUFLIA/piVC 29 15 0.517 29 0.030 29 0.106

AUFLIA/RicAgla 14 14 0.640 14 0.026 14 0.041

AUFLIA/simplify 769 760 3.218 740 1.424 768 0.739

AUFLIRA/nasa 4619 4187 0.452 4520 0.082 4526 0.014

AUFNIRA/nasa 142 48 0.410 N/A N/A 72 0.012

SMT and ATP

NASA casesVerification conditions of some NASA softwareIntroduced by Denney et al. at IJCAR 2004Claim: Modern ATPs are powerful enough for “practicalapplication in program certification”

T∅ The first set generated, the hardestT∀,→

Tprop (e.g. true ∨ P ===> true)Teval (e.g. succ(pred(x)) ===> x)Tarray

Tpolicy (ad-hoc simplification, the easiest)Tarray∗ simplification of T∀,→

CVC3, Simplify, SPASS, Vampire

SimplifyThe only SMT solvers for quantifier reasoning that ispublicly available in 2004

Vampire, version 8.1One of the best ATPs, won two categories of CASCcompetition in recent years

SPASS, version 2.2The best ATP in the IJCAR 2004 paper

category #cases Vampire SPASS Simplify CVC3T∅ 365 266 302 207 343T∀,→ 6198 6080 6063 5957 6174Tprop 1468 1349 1343 1370 1444Teval 1076 959 948 979 1052Tarray 2026 2005 2000 1943 2005Tpolicy 1987 1979 1974 1917 1979Tarray∗ 14931 14903 14892 14699 14905total 28051 27541 27522 27072 27902

category Vampire SPASS Simplify CVC3T∅ 9.277 1.765 0.068 0.017T∀,→ 2.154 0.673 0.017 0.004Tprop 4.322 1.066 0.339 0.006Teval 5.603 0.760 0.042 0.008Tarray 1.444 0.270 0.011 0.005Tpolicy 1.494 0.272 0.010 0.004Tarray∗ 0.695 0.232 0.010 0.005total 1.560 0.411 0.015 0.004

Related works

SMT solvers[1] D. Detlefs, G. Nelson, and J. B. Saxe. Simplify: a theorem prover for program checking.J. ACM, 52(3):365–473, 2005.[2] B. Dutertre and L. Moura. Yices. yices.csl.sri.com/

[3] M. Moskal. Fx7. nemerle.org/ malekith/smt/en.html

SMT benchmarks[4] S. Ranise and C. Tinelli. The satisfiability modulo theories library (SMT-LIB).www.SMT-LIB.org, 2006.

[5] E. Denney, B. Fischer, and J. Schumann. Using automated theorem provers to certify

auto-generated aerospace software. In D. A. Basin and M. Rusinowitch, editors, IJCAR,

volume 3097 of Lecture Notes in Computer Science, pages 198–212. Springer, 2004.

DPLL(T)[6] H. Ganzinger, G. Hagen, R. Knowings, A. Oliveras, and C. Tinelli. DPLL(T): Fast

decision procedures. In R. Alur and D. Peled, editors, Proceedings of the 16th International

Conference on Computer Aided Verification, CAV’04 (Boston, Massachusetts), volume 3114

of LUCKS, pages 175–188. Springer, 2004.Solving Quantified Verification Conditions using Satisfiability Modulo Theories – p.27/28

Summary

Instantiation level heuristic meets several challenges inquantifier reasoningFor certain kinds of verification conditions, SMT solversmay be a better choiceFuture work

Efficient multi-trigger matching with equalitiesTechniques from ATP

Solving Quantied Verication Conditions using …oliveras/espai/smtSlides/clark.pdfSolving Quantied Verication Conditions using Satisability Modulo Theories Yeting Ge, Clark Barrett,

Documents

Media REVEALr: A social multimedia monitoring and...

Formal Verication of...

From Constraints to Tape-Out: Towards a Continuous AMS...

Generating Dynamic Verication Tools for Generalized ... ·....

Software Reliability: R untime Verication · Software...

Design and verication of digital systems -...

Markov Logic Networks -...

Satisability Solvers - Cornell University · Satisability.....

Convex Polyhedra for the Analysis and Verication of ... ·....

Towards Reliable Web Service Discovery through Behavioural.....

Verication by Theorem Proving - University of Cambridge

PRIMAL: Power Inference using Machine...

VERICATION AND CERTIFICATION REPORT

Automated Reasoning and Formal VericationAutomated Reasoning...

Bushre Verication Method

Mediarevealr: A social multimedia monitoring and...