Modular Verification of Multithreaded Software Shaz Qadeer Compaq Systems Research Center Shaz Qadeer Compaq Systems Research Center Joint work with Cormac.

Modular Verification of Multithreaded SoftwareModular Verification of Multithreaded Software

Shaz QadeerCompaq Systems Research

Center

Shaz QadeerCompaq Systems Research

CenterJoint work with

Cormac Flanagan, Stephen Freund, Sanjit Seshia

Multithreaded softwareMultithreaded software

Examples: operating systems, databases, web servers, Java libraries

Verification problem– for all inputs and interleavings, program

satisfies specification Verifying multithreaded programs is

difficult– large and complex

Modular verification of multithreaded programsModular verification of multithreaded programs

thread: verify each thread separately using specifications of other threads

procedure: in each thread, verify each procedure separately using specifications of called procedures

MethodMethod

Reduce verification of multithreaded programs to verification of sequential programs

Analysis of sequential programs well-studied

– dataflow analysis– abstract interpretation– theorem proving with verification

conditions

ActionAction

predicate on unprimed and primed variables

– (x’ = y+1 y’ = y) x := y+1

– (p x’ = x) py

– true skip

Atomic operationAtomic operation

actions p and q pq : all transitions allowed by p must also

be allowed by q (x’ = y+1 y’ = y) x’ y’ succeeds (x’ = y+1 y’ = y) x’ = y’ fails skipp assert p p abbreviates ptrue

OutlineOutline

modular verification – without procedure calls– with procedure calls

verification of Mercator

Simple multithreaded programSimple multithreaded program

Thread1

acq;x := x+1;assert x > 0;rel;

Thread2

acq;x := 0;rel;

m = i, if held by thread i0, if unheld

Init: x = 0

Simple multithreaded programSimple multithreaded program

Thread1

(m=0m’=1)m;x := x+1;assert x > 0;(m’=0)m ;

Thread2

(m=0m’=2)m;x := 0;(m’=0)m ;

Init: x = 0

Owicki-Gries-Lamport methodOwicki-Gries-Lamport method

Thread1

L1: m=0 x0 (m=0m’=1)m;L2: m=1 x0 x := x+1;L3: m=1 x1 assert x > 0;L4: m=1 x1 (m’=0)m ; L5: m=0 x0

Thread2

M1: m=0 x0 (m=0m’=2)m;M2: m=2 x0 x := 0;M3: m=2 x0 (m’=0)m ;M4: m=0 x0

Sequential correctness Non-interference

Why is Thread1 correct?Why is Thread1 correct?

Thread1 view

(m=0m’=1)m;*

*x := x+1;

*assert x > 0;

*(m’=0)m;

1-abstractionA1

;

(m=0m’=1)mA2 ;

A1;

x := x+1 A2 ;

A1;

assert x > 0 A2 ;

A1;

(m’=0)m A2 ;

(m’=0 x’ 0) (m = i m’ = i) (m = i x’ = x)

Ai Jones 83

Why is Thread1 correct?Why is Thread1 correct?

Thread1 view

(m=0m’=1)m;*

*x := x+1;

*assert x > 0;

*(m’=0)m;

1-abstraction

A1 reflexive-transitive !

Can replace A1

by A1 !

A1 ;(m=0m’=1)mA2 ;A1

;x := x+1 A2 ;A1 ;assert x > 0 A2 ;A1 ;(m’=0)m A2 ;

(m’=0 x’ 0) (m = i m’ = i) (m = i x’ = x)

Ai Jones 83

Sequentialchecker

Summary of methodSummary of method

Thread1

Thread2

Threadn

1-abs

2-abs

n-abs

Sequentialchecker

Sequentialchecker... ... ...

yes

yes

yes

Assume-guarantee decompositionAssume-guarantee decomposition

Misra-Chandy 81 – processes communicating over channels

Jones 83– parallel shared-memory programs

Abadi-Lamport 85– temporal logic

Alur-Henzinger 96, McMillan 97– Mealy machines (hardware)

. . .

OutlineOutline

modular verification – without procedure calls– with procedure calls

verification of Mercator

Simple multithreaded programSimple multithreaded program

Thread1

acq( );x := x+1;assert x > 0;rel( );

Thread2

acq( );x := 0;rel( );

Init: x = 0

Mutex implementation is not atomic !Mutex implementation is not atomic !

acq( ) { t := i; while (t 0) ACS(m,0,t);}

ACS(m,v,t) if (m=v) then m,t := t,m

rel( ) { m := 0;}

What is the specification of acq( ) and rel( )?

Modular verification of sequential programsModular verification of sequential programs

verify each procedure separately using specifications of called procedures

requires x > 0modifies xensures x’ = x+1inc( ) { x := x+1;}

. .assert x > 0;(x’ = x+1)<x>; . .

assume x > 0;x_pre := x;y_pre := y;x := x+1;assert x = x_pre+1;assert y = y_pre;

Pre/post-conditions not enough !Pre/post-conditions not enough !Thread1

acq( );x := x+1;assert x > 0;rel( );

acq( ) { t := 1; while (t 0) ACS(m,0,t);}

modifies x, mensures m=1 x0

abstraction boundary

A1 ;t := 1A2 ;while (t 0) { A1 ; ACS(m,0,t)A2 ;}A1 ;assert m=1 x0 ;

(m’=0 x’ 0) (m = i m’ = i) (m = i x’ = x)

Ai

Specification of acq() for Thread iSpecification of acq() for Thread i

Impl:acq( ) { t := i; while (t 0) ACS(m,0,t); }

Spec:skip* ; (m=0 m’=i)m

skip

(m=0 m’=i)m

m0

m=0

Often, witness can be guessed automatically !

Need witness to synchronize spec actions with impl actions !

Verification of acq() for Thread 1Verification of acq() for Thread 1

true ;t := 1 skip;true ;while (t 0) { true ; ACS(m,0,t) }

Spec: skip* ; (m=0 m’=1)m

m0 skip m=0 (m=0 m’=1)m

;

In general, assumption could involve variables in scope.

Specification of rel() for Thread iSpecification of rel() for Thread i

Impl:rel( ) { m := 0;}

Spec:(m’=0)m

(m’=0)m

Generalization of pre/post-conditionsGeneralization of pre/post-conditions

requires pmodifies Uensures q f( ) { . . .}

assert p ;

trueU

; assume q

f( ) { . . .}

Why is Thread1 correct?Why is Thread1 correct?

Thread1

skip* ; (m=0m’=1)m;x := x+1;assert x > 0;(m’=0)m;

Thread1

acq( );x := x+1;assert x > 0;rel( );

A1 ;{ skipA2 ; A1 ;};(m=0m’=1)mA2 ;A1 ;x := x+1A2 ;A1 ;assert x > 0A2 ;A1 ;(m’=0)mA2 ;

(m’=0 x’ 0) (m = i m’ = i) (m = i x’ = x)

Ai

SummarySummary

for each thread i– write assumption– write specifications for procedures– check each specification by inlining

specifications of called procedures– check assumptions of other threads

OutlineOutline

modular verification – without procedure calls– with procedure calls

verification of Mercator

ImplementationImplementation

Extended Static Checker for Java (Detlefs-Leino-Nelson-Saxe 98)

verification conditions proved by Simplify (Nelson 81)

More detailsMore details

atomic operation is read/write of a heap variable

optimizations for reducing assumptions between atomic operations

assume sequential consistency witness guessed automatically

Mercator (Heydon-Najork 99)Mercator (Heydon-Najork 99)

multithreaded web crawler written in Java in production use at Altavista synchronization based on Java monitors

and reader-writer locks

Mercator architectureMercator architecture

RAM DISK

. . .TC T1 TNUM

CheckpointingThread

Worker Threads

URL setURL queue

Reader-writer locksReader-writer locks

lock L1 :

– checkpointing thread acquires in write mode

– worker threads acquire in read mode

lock L2 :

– protects data structure on disk – worker thread acquires in write mode to

write and in read mode to read

Race detection Race detection

checkpointing thread – must hold L1 in write mode

worker thread– must hold L1 in read mode

– reading: must hold L2 in read mode

– writing: must hold L2 in write mode

VerificationVerification

analyzed ~1500 LOC Mercator’s reader-writer locks implement

specification– acq_read( ), acq_write( ), rel_read(),

rel_write( ) absence of races

– threads obey protocol for accessing shared data

found bug introduced by us

Annotation costAnnotation cost

55 annotations Scales with complexity of synchronization

patterns, not size of program

Verification of java.util.vectorVerification of java.util.vector

synchronization with Java monitors ~450 LOC 4 annotations found known race (Flanagan-Freund 00)

Annotation overheadAnnotation overhead

Category Fraction of Total

Read/Write Constraints & Data Invariants

6/55

Specification of Readers-Writer Locks 21/55

Other Interface Specifications 16/55

Environment Assumptions 12/55

File system

Web server

Disk

Buffer cache

Mutex

module Mutex { /* Data */ boolean b; int m;

/* Code */ acq() { } rel() { }}

create() { . acq(); .}

module Mutex { /* Data */ boolean b; int m;

invariant b m = 0;

/* Code */ spec …… acq() { . assert b; . }

spec …… rel() { }}

ProgramsPrograms

S SeqProg ::= atomic operation | S1 ; S2 composition | S1 S2 choice

| S1 iteration

| call p procedure call

P ParProg ::= S1 || . . . || Sn

Modular verification of multithreaded programsModular verification of multithreaded programs

one procedure active for each thread interleaving of atomic actions of active

procedures atomic action is unit of induction pre/post-conditions not enough need abstraction of each atomic action of

a procedure specification is an abstract program