Error Control for Probabilistic Model Checking

Error Control forProbabilistic Model Checking

Håkan L. S. YounesCarnegie Mellon University

Younes Error Control for Probabilistic Model Checing 2

Contributions

Framework for expressing correctness guarantees of model-checking algorithms

Enables comparison of different algorithms Improves understanding of sampling-based

algorithms New sampling-based algorithm for

probabilistic model checking Better error control through undecided results

Younes Error Control for Probabilistic Model Checing 3

Probabilistic Model Checking

Given a model , a state s, and a property , does hold in s for ?

Model: stochastic discrete event system Property: probabilistic temporal logic formula

q

arrival departure

“The probability is at least 0.1 that the queuebecomes full within 5 minutes”

Younes Error Control for Probabilistic Model Checing 4

Temporal Stochastic Logic (CSL)

Standard logic operators: , , …

Probabilistic operator: ≥ [ ] Holds in state s iff probability is at least for

paths satisfying and starting in s Until: ≤T

Holds over path iff becomes true along within time T, and is true until then

Younes Error Control for Probabilistic Model Checing 5

Property Example

“The probability is at least 0.1 that the queue becomes full within 5 minutes”

≥0.1[ ≤5 full ]

Younes Error Control for Probabilistic Model Checing 6

Possible Results ofModel Checking

Given a state s and a formula , a model-checking algorithm can:

Accept as true in s (s ) Reject as false in s (s ) Return an undecided result (s I )

An error occurs if: rejects when is true (false negative) accepts when is false (false positive)

Younes Error Control for Probabilistic Model Checing 7

Ideal Error Control

Bound on false negatives: Pr[s | s ]

Bound on false positives: Pr[s | s ]

Bound on undecided results: Pr[s I ]

Younes Error Control for Probabilistic Model Checing 8

Unrealistic Expectations

Actual probability of holding

s ≥ [ ]s ≥ [ ]

p

Pro

babi

lity

of a

ccep

ting

≥ [

] as

tru

e in

s 1 – –

Younes Error Control for Probabilistic Model Checing 9

Temporal Stochastic Logic with Indifference Regions (CSL)

Indifference region of width 2 centered around probability thresholds

Probabilistic operator: ≥ [ ] Holds in state s if probability is at least +

for paths satisfying and starting in s Does not hold if probability is at most − “Too close to call” if probability is within

distance of

Younes Error Control for Probabilistic Model Checing 10

Error Control forCurrent Solution Methods

Bound on false negatives: Pr[s | s ]

Bound on false positives: Pr[s | s ]

No undecided results: = 0 Pr[s I ] = 0

Younes Error Control for Probabilistic Model Checing 11

Probabilistic Model Checkingwith Indifference Regions

Actual probability of holding + − p

Pro

babi

lity

of a

ccep

ting

≥ [

] as

tru

e in

s 1 –

s ≥ [ ]

s ≥ [ ]s ≥ [ ]

s ≥ [ ]

Younes Error Control for Probabilistic Model Checing 12

Hypothesis TestingYounes & Simmons (CAV’02)

Single sampling plan: n, c Generate n sample execution paths Accept ≥ [ ] iff more than c paths satisfy Probability of accepting ≥ [ ] as true:

Sequential acceptance sampling

c

i

ini ppi

npncF

0

11),;(1

Younes Error Control for Probabilistic Model Checing 13

Statistical EstimationHérault et al. (VMCAI’04)

Estimate p using sample of size n: Choosing n:

Acceptance condition for ≥ [ ]:

n

iix

np

1

1~

1~Pr

2log

21

2 ppn

p~

Same as single sampling plan n, n + 1!

Younes Error Control for Probabilistic Model Checing 14

Statistical Estimation vs.Hypothesis Testing

nest nopt nest ∕ nopt

0.5 10−2 10−2 26,492 13,527 1.96

0.5 10−8 10−2 95,570 39,379 2.43

0.5 10−8 10−8 95,570 78,725 1.21

0.9 10−2 10−2 26,492 4,861 5.45

0.9 10−8 10−2 95,570 13,982 6.84

0.9 10−8 10−8 95,570 28,280 3.38

Younes Error Control for Probabilistic Model Checing 15

Numerical Transient AnalysisBaier et al. (CAV’00)

Estimate p with truncation error :

Acceptance condition for ≥ [ ]: Pr[s | s ] = 0

Pr[s | s ] = 0

ppp ~~

2

~p

2

Younes Error Control for Probabilistic Model Checing 16

Alternative Error Control

Bound on false negatives: Pr[s | s ]

Bound on false positives: Pr[s | s ]

Bound on undecided results: Pr[s I | (s ) (s )]

Younes Error Control for Probabilistic Model Checing 17

Probabilistic Model Checkingwith Undecided Results

Actual probability of holding + − p

1 –

Rejection probability

Acceptance probability

Undecided result withprobability at least 1 – –

Younes Error Control for Probabilistic Model Checing 18

Statistical Solution Method

Simultaneous acceptance sampling plans H0: p against H1: p – H0: p + against H1: p

Combining the results Accept ≥ [ ] if H0 and H0 are accepted

Reject ≥ [ ] if H1 and H1 are accepted Undecided result otherwise

Younes Error Control for Probabilistic Model Checing 19

20

15

10

5

014 14.1 14.2 14.3 14.4 14.5

= 0 = 10–2

Formula time bound (T )

Ve

rific

atio

n ti

me

(se

con

ds)

Empirical Evaluation(Symmetric Polling System)

serv1 ≥0.5[ ≤T poll1]

Younes Error Control for Probabilistic Model Checing 20

Empirical Evaluation(Symmetric Polling System)

result 14.10 14.15 14.20 14.25 14.30 14.35 14.40

accept 0 3 9 50 88 97 100

reject 100 97 91 50 12 3 0

accept 0 0 0 0 32 99 100

reject 100 99 42 1 0 0 0

undecided 0 1 58 99 68 1 0

= = = 10–2

Younes Error Control for Probabilistic Model Checing 21

Summary

Statistical estimation is never more efficient than hypothesis testing

Statistical methods are randomized algorithms for CSL model checking

Numerical methods are exact algorithms for CSL model checking

New statistical solution method with finer error control ( parameter)