Static Analysis versus Software Model Checking for bug finding Dawson Englers, Madanlal Musuvathi Stanford University, CA, USA Presented By: Prateek Agarwal ETH, Zürich
Static Analysis versus Software Model Checking for bug finding
Dawson Englers, Madanlal MusuvathiStanford University, CA, USA
Presented By:Prateek Agarwal
ETH, Zürich
Agenda
● Goal● Aim● Scope● Methodologies used
– Meta Compilation
– C Model Checker● Case Studies● Conclusions
Goal
Checking validity of general perception
Static Analysis is easy to apply, finds shallow bugs
Model Checking is harder but strictly better
Aim
● Documentation of experience● Set of case studies● Comparison of two approaches, based on
– Bugs/Lines of Code
– False positives vs Actual Bugs
– Time & Effort required
– Types of Bugs: generic vs protocol specific● Find as many bugs as possible
Scope
● Verification of system/protocol software– Critical
– Hard to test/inspect/reason about manually
– Highly optimized● Protocol codes follow event driven model
Meta Compilation (MC)
● Extension of compiler● Used for Static Analysis & Model Checking● Metal: Language for
– Slice specification by pattern matching
– Code translation
– State machines● xg++ over g++ compiler
Meta Compilation: Key Features
● Code should be highly structured– e.g. special MACRO/functions for read/write
● No source code annotation needed● Emphasis on reducing false positives● No Simulation of code
– No model of heap, tracking variable values
– Simulation tends to become model checking
Static Analysis using MC● Patterns & Actions● Small State machines for checking
Example Rule: Sync before read...start: {WAIT_FOR_DB_FULL(addr);} ==> stop|{MISCBUS_READ_DB(addr);} ==>
{err(“Buffer read not sync");};....
Start
ErrorStop
misc_bus_read_db
wait_for_db_full
Engler & Masuvathi [1]
Model Checking using MC
● Extraction & Translation– protocol code Mur→ Φ Model
● Eliminates handcrafted system models● MurΦ
– Explicit state enumeration model checker
– Each reachable state visited once
Model Checking using MCState Variable(Metal Slicer)
Translation Pattern(Metal Printer)
CorrectnessProperties
Protocol Model
Protocol Code(Implementation)
Start StateHardware Model
Error List
xg++
MurØ
Automatically Generated
Hand Crafted by User
Program
Lie, Chou, Engler, Dill [2]
Legend:
Unedited source code
Example: Model Checking using MCMetal Slicer:.../* Patterns for n/w and proc mesgs, which use length field */pat sends =
{ NI_SEND (type, data, keep,swp, wait,nl);} | { P1_SEND(type, data, keep,swp, wait,nl);} ...Metal Printer...
/* Automatically insert length assertion before send */
{N1_SEND(type, data, keep, swap, wait, null);} ==>{
if(mgk_int_cst(data) != 0)amgk_e(“assert(nh.len = len_data);”);
elsemgk_e(“assert (nh.len = len_nodata);”);
mgk_e(“ni_send(%t, %t, procNum,nh”,type,swap);...
Lie, Chou, Engler, Dill [2]
Model Checking : C Model Checker
● Motivation– xg++ user needs to know system intricacies
– MurΦ lacks many C constructs
– Eliminate need of higher level system model● Use the implementation code itself● Process scheduling & execution by CMC
CMC Key Features● Searches for all transitions● Complete system state saved and restored● Heuristics & Optimizations for state space
explosion● Correctness properties
– Assertions from implementation code
– Memory leaks etc. in built
– User specified invariants● Environment Model still handcrafted
Case Study 1: FLASH
● Cache coherence protocol for FLASH– Multiprocessor architecture
– Code runs on each cache miss● 5 FLASH protocols● Code Size: 10K-18K
John Hennessy [3]
Results
● Static Analysis– Bugs Found: 33, False Positives: 28
– Most rules on buffer management
– e.g. allocation before use, deallocation after use, not used after deallocation
● Model Checking– 8 bugs
– e.g. queue overflows, no invalidation of an exclusive line
– Handwritten model lead to delays
Summary● Static Analysis Advantage:
– Better for same set of properties
– Works best for code visible rules● Model Checking Advantage:
– Can cover complex properties, invariants
– e.g. counter overflow missed by SA● Model Checking Disadvantage:
– Simple models Less Coverage→– Detailed model Increased Time & Effort→– Erroneous model False Positives→
Case Study 2: AODV Protocol● Ad-hoc On-demand Distance Vector Protocol● Routing protocol● Simplified environment
– Input: route request, timer interrupt, packet receipt● 3 publicly available implementations used● Average 6 KLOC● Model Checking using CMC
Results
● Model Checking– 42 unique bugs, 1 bug in AODV specification
● Static Analysis: 34 bugs
Both Model Checking Static Analysis
Generic Properties 21 1 13
Protocol Specific 0 20 0
Total 21 21 13
comparison of bug count classified by properties
Comparison
● Generic Properties:– CMC found only one bug not found by SA
– SA found 13 bugs not found by CMC● Protocol Specific Properties:
– Difficult to check statically 0 bugs→● SA hit more code but CMC hit more properties!
Summary
● Static Analysis Advantage:– Wins when checking same set of properties (again)
● Model Checking Advantage: – Simulates execution of 'entire system'
– Looks for actual errors (not 'causes' of error)
– Gives actual execution trace● More checks more bugs→
– CMC executed the code
Case Study 3: TCP
● TCP Code size ~ 10 * (AODV code size)● Frequently audited, Heavily Tested code● Only Model Checking done (using CMC)● System tightly coupled with Environment
– System: TCP stack
– Environment: Linux Kernel, Kernel Modules● Incremental refinement of model
– Start simple, keep adding complexity
Case Study 3: TCP● External Functions can be included in System● Advantage
– Model Checker will find errors in them
– No effort to create stubs● Disadvantage
– Increase in state space
– Dependencies on other functions● Essential Question
– Where to draw the line between system and environment?
System Separation● Cut on narrowest interface● TCP – Kernel module
– Poorly documented
– Effectively simulating kernel
– Months of effort!
– Difficult to determine false positives● New approach: Use well defined interface
– System calls interface
– Hardware abstraction layer
– Effectively running entire kernel in CMC!
Results
● 4 bugs found● Metric to measure coverage:
– Line Coverage: Lines of code covered
– Protocol Coverage: %age of abstract protocol behavior covered
Description Line Coverage Protocol Coverage Bugs
Standard server, client 47.4 % 64.7 % 2
+ Simultaneous connect 51.0 % 66.7% 0
+ Partial Close 52.7 % 79.5% 2
+ Message corruption 50.6% 84.3% 0
Combined Coverage 55.4% 92.1%
Conclusions
● No model is as good as implementation itself– Simplification , omission missing critical errors→
● Manual work scale badly for complex systems● Use well defined interfaces● More code coverage More bugs found→
My Opinion
● Usage of live software project is a big plus● Meta-Compilation & CMC give huge advantage● Better comparison if same tools used across
case studies● Time & Effort not measured accurately● No comparison with other existing tools
Future Scope
● How to increase code coverage in meta compilation and CMC?
● Analysis of class of properties checked
References
1)Experiences using static analysis & model checking for bug finding – Dawson Engler and Madanlal Masuvathi
2)A simple method for extracting models from protocol code – David Lie, Andy Chou, Dawson Engler, David L. Dill
3)Stanford FLASH Multiprocessor: Status, Some Lessons and Plans – John Hennessy
Q & A