Compositional Compositional Verification of Evolving Verification of Evolving Component-based Systems Component-based Systems Nishant Sinha ECE, CMU PhD Proposal Presentation
Dec 22, 2015
Compositional Verification of Compositional Verification of Evolving Component-based Evolving Component-based
SystemsSystems
Nishant Sinha
ECE, CMUPhD Proposal Presentation
2
OverviewOverview
Problem of Re-validating Evolving Systems Our Solution : Substitutability Check
• Containment• Compatibility
Assume-Guarantee for Simulation Related Approaches Proposed Research Timeline
3
Evolving Systems/DesignsEvolving Systems/Designs
Evolution is inevitable in any real hardware/software system:
• Changing requirements, designs• Bug fixes• Product changes (e.g., underlying platform)
• Upgrades of vendor-provided components
4
Re-validation on EvolutionRe-validation on Evolution
Assembly A
Component C (1.0)Component C’ (2.0)
P ?
addedfunctionalities
bug fixes
5
Re-validating Evolving SystemsRe-validating Evolving Systems
Component assembly re-validation• Necessary on upgrade of any component• High costs of complete global verification
For example, a software system of 25 components (~20K LOC) and 100+ properties might take up to a month of verification effort
Require efficient formal techniques for re-validation upon evolution
6
Re-validation on EvolutionRe-validation on Evolution
Pervasive problem in Software and Hardware Engineering communities
Evolution • Upgrades, Bug Fixes, Re-designs ..
Re-validation appears in numerous forms• Regression checks, Upgrade compatibility
checks, Re-design validation
Our proposed formulation/solution• Substitutability Check
7
Subst. Check - RequirementsSubst. Check - Requirements
Substitutability Check for Upgrades• Formal technique for statically re-
validating upgrades Local and Incremental nature
• Should avoid complete re-validation• Local : focus on modified components• Incremental : reuse previous validation
results
A “make” for component-based systems …
8
Subst. Check - Our SolutionSubst. Check - Our Solution
Our Substitutability Check procedure• Focus on Static Re-validation of upgrades• Local and Incremental• Supports changes in Component
Interfaces and Behaviors• Supports multiple component upgrades
9
ComponentC
Identical BehaviorsNew
BehaviorsLost
Behaviors
Upgraded Component
C’
Component UpgradeComponent Upgrade
Involves both addition and removal of behaviors
Compatibility Check(global)
Containment Check(local)
Two-phase Subst. Check• Local : Containment • Global : Compatibility
11
Substitutability CheckSubstitutability Check
Containment• Detects if any useful behaviors are lost• Locally checks if behaviors (services) of the
previous component are still present in new one• C C’
Compatibility• Safety with respect to other components in
assembly• Detects if the added behaviors cause any
assembly-level violations
• Does [ C’1 || C2 ||.. || Cn P ] hold?
[FM’05]
[FM’05] Dynamic Component Substitutability Analysis, S. Chaki, E. Clarke, N. Sharygina, N. Sinha, Formal Methods Conference, 2005.
12
Subst. Check for Software ComponentsSubst. Check for Software Components
We have formalized Substitutability Check in context of Component-based Software Systems [FM’05]• Components are communicating C programs• Obtain a finite behavioral model of all components by
abstraction: State-event communicating automata
Containment: • Use both Under- and Over- approximations of
components
Compatibility: • Use Dynamic Assume-Guarantee Reasoning• Focus of proposed research
13
State-event (SE) automataState-event (SE) automata
Both state and actions labeled• <Q,,T,P,L>
Composition semantics• Synchronize on shared actions
Represents abstractions
p
!q
q
!p
C ComponentComponent SE
Automaton abstraction
Predicate Abstraction
14
Component AssemblyComponent Assembly
A set of communicating concurrent C programs • No recursion, procedures inlined• Communication is abstracted into interface actions
C1 C2 C3
M1 M2 M3
Predicate Abstraction
Component Assembly
Abstract Assembly
15
Component Assembly ExampleComponent Assembly Example
Inter-process communication module
Lock server Queue
Write_to_queue Read_from_queue
Enter_crit_section,Leave_crit_section
ReadMQWriteMQ
17
Containment CheckContainment Check
Goal: Check C C’• All behaviors retained after upgrade• Cannot check directly: need
approximations Idea: Use both under- and over-
approximations Solution:
• Compute M: C M• Compute M’: M’ C’• Check for M M’
C
Containment Check
Identical New Lost
C’
18
Containment (contd.)Containment (contd.)
C C’
M M’
over-approx under-approx
M M’ ?True
C C’False, CE
CE C ?False,Refine M
CE C’ ?
True,Refine M’
! ( C C’),CE provided as feedback False
True
M
C
C’
M’
19
Containment (contd.)Containment (contd.)
Computing over- (may) approximation• Conventional predicate abstraction• Amay(a,a’) =
s, s’. [ R(s,s’) a= h(s) a’ = h(s’) ]
Computing under- (must) approximation• Modified predicate abstraction • Universal abstraction• Amust(a,a’) =
s, s’. [ R(s,s’) a= h(s) a’ = h(s’) ]
21
Compatibility CheckCompatibility Check
Goal:• Detect system-level spec violations due
to added behavior Approach: Dynamic Compositional Reasoning
• Use compositional reasoning to obtain verification proofs for the initial assembly
• Identify parts of the proof that need to be re-validated on upgrades
• Re-validate those parts while reusing as much of the previous proof as possible
C
Compatibility Check
Identical New Lost
C’
22
Compositional ReasoningCompositional Reasoning
Assume-Guarantee Reasoning (AGR)
Automatically generate assumption A • Cobleigh et. al. at NASA Ames
Use learning algorithm for regular languages, L*
M1 || A P M2 A
M1 || M2 P
AG - Non Circular
23
L* learner
Learning Regular languages: L*Learning Regular languages: L* Proposed by D. Angluin, improved by Rivest et al.
• Learning regular sets from queries and counterexamples, Information and Computation, 75(2), 1987.
• Learns the minimal DFA corresponding to an unknown regular lang.
Minimally adequate Teacher
IsMember( trace )
IsCandidate( DFA D )
a
b
a
b
UnknownRegular Language U
±Counterexample/ Yes
Modelchecker
Yes/No
Minimum DFA
• Polynomial in the number of states and length of max counterexample
24
L* : Data StructuresL* : Data Structures
Observation table : empty initially
Rows have access traces Columns have distinguishing
traces Table entries show
membership of traces in U
Iteratively updates the table using provided counterexample samples
Finally, builds a DFA from a closed table and presents it as a candidate
a ab
a
b
aa
ab
1 0
1 1 0
1 1 0
1 1 0
0 1 0
q0 q1
a,b
a
b
[q0]
[q1]
[q1]
[q1]
[q0]
25
Automated AG Reasoning Automated AG Reasoning using L*using L*
R1: M1 || A P
R2: M2 A
trueL* Assumption
GenerationA
CE
CE AnalysisActual CE
M1 || M2 P
-CE for A
+CE for A
Teacher
M1 || M2 P
true
[Cobleigh et al.]
26
Handling Multiple ComponentsHandling Multiple Components
AG-NC can be applied recursively• (Cobleigh et al.)
R1: M1 || A PR2: M2 A
M1 || M2 P
M1 || A1 P
M2 || A2 A1 M3 A2
M2 || M3 A1
M1 || M2 || M3 P
• AG Proof Tree
•Each Ai computed by a separate L* instantiation
27
Compatibility of UpgradesCompatibility of Upgrades Suppose assumptions are available from the old assembly Re-validate an upgrade
• We propose Dynamic AGR : Incremental AGR technique• Reuse previous verification results
C
Identical New Lost
C’
• Can we reuse previous assumptions directly?• NO: upgrades may change the unknown U to be learned
• Therefore, Dynamic AGR needs Dynamic L*
M1 || A1 P M2 A1
M1 || M2 P
M’1 || A’1 P M2 A’1
M’1 || M2 P
Upgrade
Reuse?
ViolatesL* invariant
28
Dynamic L*Dynamic L*
Learn assumption A corresponding to U• Corresponding Observation Table T
Unknown language U changes to U’
Goal: Continue learning from previous model A• Start learning U’ by reusing table T
Central Idea: Re-validate T against U’ to obtain T’ • T’ is valid with respect to U’
29
Dynamic L*Dynamic L*
a ab
a
b
aa
ab
1 0
1 1 0
1 1 0
1 1 0
0 1 0
q0 q1
a,b
a
b
U’ = U\{b}
Re-validationagainst U’
a ab
a
b
aa
ab
1 0
1 1 0
0 1 0
1 1 0
0 1 0
q0 q1
a
a
b
b
Table valid for U Table valid for U’
AssumptionA
AssumptionA’
30
Dynamic AGDynamic AG
M1 || A1 P M2 A1
M1 || M2 P
M’1 || A’1 P M2 A’1
M’1 || M2 P
(Dynamic L*) Re-Validate! and Reuse
Upgrade
• Multiple Components • Heuristics for choosing order of re-validating Ai’s in proof tree
31
ImplementationImplementation
ComFoRT framework for verifying concurrent C programs
Industrial benchmark • ABB Inter-process Communication (IPC) software• 4 main components – CriticalSection, IPCQueue, ReadMQ,
WriteMQ
Evaluated on single and simultaneous upgrades • WriteMQ and IPCQueue components
Properties
• P1: Write after obtaining CS lock
• P2: Correct protocol to write to IPCQueue
32
Experimental ResultsExperimental Results
Upgrade# (Property)
Torig (msec)
(AGR from scratch)
Tug (msec)
(Dynamic AGR)
Ipc1 (P1) 2260 13Ipc1 (P2) 1694 14Ipc2 (P1) 3286 17Ipc2 (P2) 805 10Ipc3 (P1) 3624 17Ipc3 (P2) 1649 14Ipc4 (P1) 1102 24
33
Summary Till NowSummary Till Now
Automated Substitutability Checking• Containment and Compatibility• Containment is a local check• Compatibility is a global check and reuses
previous verification results• Handles multiple upgrades
Implementation• ComFoRT framework• Compatibility Check applied to IPC
example
34
AGR for Simulation AGR for Simulation ConformanceConformance
Till now during AGR• Notion of conformance was trace containment
Simulation conformance : local, non-det.
• M1 || M2 ≤ P M1 || A ≤ P; M2 ≤ A
• Counterexamples are trees
We proposed automated AGR for simulation conformance of LTSs• Learn from tree CEs instead of trace CEs• Tree language of an LTS: set of trees that it
simulates
[CAV’05]
[CAV’05] Automated Assume-Guarantee Reasoning for Simulation Conformance, S. Chaki, E. Clarke, N. Sinha, P. Thati, Conference on Computer-Aided Verification, 2005.
35
AGR for Simulation ConformanceAGR for Simulation Conformance
Key contributions• Formulate simulation check as regular tree
language containment : M ≤ N T(M) T(N)• Represent tree languages with det. tree
automata• Learning algorithm LT for an unknown tree lang.
using teacher• For AGR: Teacher uses tree language
containment to answer queries Experiments with C implementations of Secure
Socket Layer (SSL) Protocol client and server
36
ResultsResults
Direct AG GainVerif. Result
T1 Mem1
T2 Mem2
Mem1/Mem2
|A| MQ #
CQ #
Invalid
* 2146 325 207 10.4 8 265 3
Valid * 2080 309 163 12.8 8 279 3Valid * 2077 309 163 12.7 16 279 3Valid * 2076 976 167 12.4 16 770 4Valid * 2075 969 167 12.4 16 767 4Invali
d* 2074 3009 234 8.9 24 1514 5
Invalid
* 2075 3059 234 8.9 24 1514 5
Invalid
* 2072 3048 234 8.9 24 1514 5
38
Related WorkRelated Work
Behavioral Subtyping : Liskov ,Wing [’01]• Extend OO subtyping to behaviors• Subtype Req: If x) is provable about
object x:T, then (y) must hold for y:S, where S is a subtype of T
• Add invariant and history constraints to object specifications
• Local substitution check for an object, independent of its use by the environment
39
Related WorkRelated Work
Interface automata formalism: Henzinger et al. [’01-’03]• Variant of Input/Output automata used to
represent component abstractions and interfaces• Composition and Refinement theory• Composition is well-defined only if under some
environment two automata don’t deadlock• M refines N if M allows more inputs and exhibits
less outputs than N
• Refinement is used as a notion of Substitution : does not capture environment use of a component
40
Related WorkRelated Work
Checking compatibility of Upgrades : McCamant, Ernst[’04]
• Compute observational summaries of components before upgrades using sample execution behaviors
• Internal and External summaries• Pre-, Post-conditions to component method calls• On an upgrade, check if new summaries refine old
ones using a theorem prover
• Unsound component behavior abstraction• No temporal information about component
behaviors
41
Related Work Related Work
Automated Assume-Guarantee Reasoning (AGR)
• Weakest assumption [GPB02]• Learning assumptions automatically
[CGP03]• Circular Rules for Automated AGR
[BGP03]• Predicate Abstraction and AGR [CCGP04]• Symbolically learning assumptions
[AMN04]
42
Related WorkRelated Work
Learning for Verification• Black-box model checking [PVY99,GPY02]• Learning for infinite-state model checking
[HV04, VSVA04]• Synthesizing interfaces of Java classes
[ACMN05]• Behavior-based model construction [HS04]• Inductive learning for shape analysis
[SLR05]
44
Proposed ResearchProposed Research
Substitutability Framework for SystemC
AGR for Liveness
Improving Substitutability Check
Learning-assisted CEGAR
45
SystemCSystemC
C++ library to simulate concurrency SystemC constructs are macros in C++ Being adopted for System-level Design by
industry• Modular computation structure• Has Communication primitives: Channels/Events• Executable specifications• Supports HW/SW Co-Design
Goal: Develop a Compatibility Check Framework for SystemC Designs
48
Program StructureProgram Structure
program = < processes , channels , variables , events>process =
( comm | ctrl flow )*comm =
event.notify() | event.notify_delayed() | signal.read() | signal.write() | channel.<transaction_name>()| port.read() | port.write()| port.<transaction_name>()
ctrl flow = if (exp) then <s1> else <s2> | while (exp) <s1>
variable types = bool | int | bitvector
49
Verifying SystemC Hardware DesignsVerifying SystemC Hardware Designs
Synchronous model of execution• Based on Discrete-event simulation
Verification easier than C++ • Finite number of concurrent processes• No dynamic objects
Available open-source parsers• PINAPA
50
Plan: Subst. Check for SystemCPlan: Subst. Check for SystemC
Flatten and Translate toSynch. Intermediate Language
Flatten and Translate toSynch. Intermediate Language
SystemCDesign
Build Verifier for Intermediate Language
Build Verifier for Intermediate Language
Build Compositional Verifier for Intermediate Language
Build Compositional Verifier for Intermediate Language
Perform Compatibility CheckPerform Compatibility Check
Delta-cycle semantics
Events, Channels, Shared Vars, Processes
Assumption automata over <Input,Output>
alphabet
51
AGR for LivenessAGR for Liveness
Till now, safety and simulation conformance• Checking liveness specs involves reasoning
about infinite system behaviors
Consider -regular language containment
• M1 || M2 P M1 || A P ; M2 A• “A” corresponds to an -regular language
• Weakest assmp. language exists; is -regular
52
AGR for LivenessAGR for Liveness
Develop a learning algorithm for general -regular languages • Previous algorithms target a subclass• Such languages have non-canonical finite
representations• Cannot extend L* directly; Myhill-Nerode
congruence is insufficient• Use a right-congruence theorem for -regular
languages (Maler and Staiger)
53
Improving Substitutability CheckImproving Substitutability Check
Till now, only behavioral changes allowed during upgrades• Interface ports of modules remain the
same
Consider changes in alphabet during Dynamic Check• Extend Dynamic L* to handle both
unknown language change and alphabet change
54
Learning-assisted CEGARLearning-assisted CEGAR
Predicate Abstraction
C1
C2
M1
M2
M1 || A P
M2 A
M1 || M2 P
Single Counterexample CE
Compositional Reasoning for Software Components C1 and C2
C1 || C2 P