Finding and Preventing Run-Time Error Handling Mistakes Westley Weimer George C. Necula University of California, Berkeley {weimer,necula}@cs.berkeley.edu.

Finding and Preventing Run-Time Error Handling MistakesWestley WeimerGeorge C. NeculaUniversity of California, Berkeley{weimer,necula}@cs.berkeley.edu

Handling Errors is Difficult

Writing solid error-handling code is hard1) We have a mistake-finding analysis

o And find 800 mistakes in 4M LOC (Java)o We characterize these mistakes

2) Propose a new language featureo Existing language features: insufficiento Track obligations at run-timeo Support it by case studies

Defining Terms

Run-Time Errors: Network problems, DB access errors, OS

resource exhaustion, … Typical Error-Handling:

Resend packet, show dialog box to user, …

Application-specific Error-Handling Handling Mistakes:

One example: a network error occurs and the program forgets to release a database lock with an external shared database, …

Cleanup in Practice

Most common exception handlersDo NothingPrint Stack Trace, Abort Program

High-level invariants should be restoredAside from “handling” the error, the

program should clean up after itself Often this does not happen in practice:

Considering all (error) paths is hard!

Normal Execution Path// Copy to destinationFile without doing any byte conversion.private static void _binaryCopyURLToFile(URL sourceURL, File destinationFile) throws IOException { BufferedInputStream input = new BufferedInputStream(sourceURL.openStream()); BufferedOutputStream output = new BufferedOutputStream( new FileOutputStream(destinationFile)); // The resource pointed to might be a pdf file, which // is binary, so we are careful to read it byte by // byte and not do any conversions of the bytes. int c; while (( c = input.read()) != -1)

output.write(c); output.close(); input.close();} // line 294, ptolemy2’s JNLPUtilities.java

Error Paths - Hazards// Copy to destinationFile without doing any byte conversion.private static void _binaryCopyURLToFile(URL sourceURL, File destinationFile) throws IOException { BufferedInputStream input = new BufferedInputStream(sourceURL.openStream()); BufferedOutputStream output = new BufferedOutputStream( new FileOutputStream(destinationFile)); // The resource pointed to might be a pdf file, which // is binary, so we are careful to read it byte by // byte and not do any conversions of the bytes. int c; while (( c = input.read()) != -1)

output.write(c); output.close(); input.close();} // line 294, ptolemy2’s JNLPUtilities.java

Fix It With Try-Finally// Copy to destinationFile without doing any byte conversion.private static void _binaryCopyURLToFile(…) throws … { BufferedInputStream input; BufferedOutputStream output; input = new BufferedInputStream(…); try { output = new BufferedOutputStream(…); try {

// The resource pointed to might be a pdf file, which// is binary, so we are careful to read it byte by// byte and not do any conversions of the bytes.int c;while (( c = input.read()) != -1) output.write(c);

} finally { output.close(); } } finally { input.close(); } }

Fix It With Run-Time Checks

private static void _binaryCopyURLToFile(…) throws … { BufferedInputStream input = null; BufferedOutputStream output = null; try { input = new BufferedInputStream(sourceURL.openStream()); output = new BufferedOutputStream( new FileOutputStream(destinationFile)); // The resource pointed to might be a pdf file, … int c; while (( c = input.read()) != -1) { output.write(c); } finally { if (input != null) then try { input.close(); } catch (Exception E) { } if (output != null) then try { output.close(); } catch (Exception E) { } } }

Fix It With Flags

int f = 0; // flag tracks progresstry { openX(); f = 1; work(); openY(); f = 2; work(); openZ(); f = 3; work(); } finally { switch (f) { // note fall-through! case 3: try { closeZ(); } catch (Exception e) {} case 2: try { closeY(); } catch (Exception e) {} case 1: try { closeX(); } catch (Exception e) {} }}

Fix It With Flags (Ouch!)

int f = 0; // flag tracks progresstry { openX(); f = 1; work(); if (…) { didY = true; openY(); f = 2; } work(); openZ(); f = 3; work(); } finally { switch (f) { // note fall-through! case 3: try { closeZ(); } catch (Exception e) {} case 2: if (didY) { try { closeY(); } catch … } case 1: try { closeX(); } catch (Exception e) {} }}

Outline

1) Background

2) Bug-Finding Static Analysis

3) Proposed Language Feature

Locating Handling Mistakes

We consider four generic resourcesSockets, files, streams, database locksFrom a survey of Java code

Program should release them along all paths, even those with run-time errors

Run-time errors are rare … So use a static analysis to find

mistakes

Static Dataflow Analysis

Exceptions and run-time errors correlate

Fault model:A called method can terminate normally

or raise any of its declared exceptions Build control-flow graph Symbolically execute each method Abstracting away most data values But tracking outstanding resources

Analysis Example Program

try {Socket s = new Socket();s.send(“GET index.html”);s.close();

} finally { } // bug!

Analysis Example

startnew socket

send

close

end

Analysis Example

startnew socket

send

close

end

{ }

Analysis Example

startnew socket

send

close

end

{ }

{ socket }

Analysis Example

startnew socket

send

close

end

{ }

{ socket }

{ }

Analysis Example

startnew socket

send

close

end

{ }

{ socket }

{ } { socket }

Analysis Example

startnew socket

send

close

end

{ }

{ socket }

{ } { socket }{ socket }

Analysis Example

startnew socket

send

close

end

{ }

{ socket }

{ } { socket }{ socket }

{ }

Analysis Example

startnew socket

send

close

end

{ }

{ socket }

{ } { socket }{ socket }

{ }{ }

Example Bug Report

startnew socket

send

close

end

{ }

{ socket }

{ } { socket }{ socket }

{ }{ }

False Positives

Simple analysis gives false positivesOf 100 error reports70 would be real bugs30 would be false positives

Each false positive is one of these:if (sock != null) sock.close();socket_field_we_close_later = sock;return sock;

Bug Reporting and Filtering

Filter out reports with those 3 formsIntroduces false negativesFor every 100 real bugs reported There are ~10 real bugs we miss

Removes all false positives Manually verify the rest as bugs

Analysis ResultsProgram Name Lines of

CodeMethods withError-Handling

Mistakes

Forgotten Resources

hibernate2 57k 13 DB

jaxme 58k 6 File

axion 65k 15 File

hsqldb 71k 18 DB, Strm

cayenne 86k 7 File

sablecc 99k 3 Strm

jboss 107k 40 DB, Strm

mckoi-sql 118k 37 Strm, DB

portal 162k 39 DB, File

pcgen 178k 17 File

compiere 230k 322 DB

org.aspectj 319k 27 File, Strm

ptolemy2 362k 27 File, Strm

eclipse 1.6M 126 Strm, File

… 13 others … 347k 121 File, DB

Total 3.9M 818 File, DB

Characterizing Mistakes

Model: aX must be followed by cX1) try { a1; } finally { c1; } ; a1 ; c1; 2) try { a1; a2; } finally { c2; c1; } 3) try { a1; a1; } finally { c1; }4) for (…) { a1; work; c1; } As well as tricky flag-work for

releasing resources early …

Outline

1) Background

2) Bug-Finding Static Analysis

3) Proposed Language Feature

Destructors (C++, C#, …)

Great for stack-allocated objects Error-handling contains arbitrary

codeExample adapted from code which

has 17 unique cleanups, one 34 lines long

try { StartDate = new Date(); // a1try { StartLSN = log.getLastLSN();

// a2work(1); try { DB.getWriteLock();

// a3work(2); // a3

} finally { DB.releaseWriteLock();// c3

work(3); }// c3

} finally { StartLSN = -1; } // c2} finally { StartDate = null; } // c1

Finalizers (Java, …)

Called by the garbage collector Too late!

No ordering guarantees Socket sock = new Socket(); Stream strm = new Stream(sock); sock.finalize may be called before

strm.finalize Programs do not use them

Only 13 user-defined ones in 4M LOC Not all SDKs use them for key resources

Feature Motivation

Avoid forgetting obligations No static program restrictions Optional lexical scoping Optional early or arbitrary cleanup Database / Workflow notions:

Either my actions all succeed (a1 a2 a3)

Or they rollback (a1 a2 error c2 c1)

Compensation Stacks

Store cleanup code in run-time stacksFirst-class objects, pass them around

After “action” succeeds, push “cleanup”“action” and “cleanup” are arbitrary code

(anonymous functions) Pop all cleanup code and run it (LIFO)

When the stack goes out of scopeAt an uncaught exceptionEarly, or when the stack is finalized

Compensation Concepts

Generalized destructors No made-up classes for local cleanup Can be called early, automatic

bookkeeping Can have multiple stacks

• e.g., one for each request in a webserver Annotate interfaces to require them

Cannot make a new socket without putting “this.close()” on a stack of obligations

Will be remembered along all paths Details in paper …

Cinderella Story

CompensationStack CS = new CompensationStack();try { Input in; Output out; Input in = new Input(src); Output out = new Output(dst); while (( x = in.read()) != -1) out.write(x); out.close(); in.close(); return 0;} finally { CS.runAll(); }

“Assembly Language”

CompensationStack CS = new CompensationStack();try { Input in; Output out; compensate { in = new Input(src); } with (CS) { in.close(); } compensate { out = new Output(dst); } with (CS) { out.close(); } while (( x = in.read()) != -1) out.write(x); return 0;} finally { CS.runAll(); }

With Annotated Interfaces

CompensationStack CS = new CompensationStack();try {

Input in = new Input(CS,src); Output out = new Output(CS,dst); while (( x = in.read()) != -1) out.write(x); return 0;} finally { CS.runAll(); }

Using Most Recent Stack

CompensationStack CS = new CompensationStack();try {

Input in = new Input(src); Output out = new Output(dst); while (( x = in.read()) != -1) out.write(x); return 0;} finally { CS.runAll(); }

Using “Current Scope Stack”

Input in = new Input(src); Output out = new Output(dst); while (( x = in.read()) != -1) out.write(x); return 0;

Cinderella 2 (Before)

int f = 0; // flag tracks progresstry { openX(); f = 1; work(); if (…) { didY = true; openY(); f = 2; } work(); openZ(); f = 3; work(); } finally { switch (f) { // note fall-through! case 3: try { closeZ(); } catch (Exception e) {} case 2: if (didY) { try { closeY(); } catch … } case 1: try { closeX(); } catch (Exception e) {} }}

Cinderella 2 (After)

compensate { openX(); }with { closeX(); }work(); if (…) compensate { openY(); }

with { closeY(); }work();compensate { openZ(); }with { closeZ(); }work();// using the “current scope stack” by default

Case Studies

Extend Java with compensation stacks

Annotate key interfaces (File, DB, …) Annotate existing programs to use

compensation stacksFor library resourcesAnd for unique cleanup actionsNo new error handling!

Sun’s petstore

“Amazon.com lite” plus inventory Raises 150 exceptions over 3,900

requests Avg Response: 52.06ms (std dev 100ms)

34,608 lines of Java, 123 change sites Two hours of work Three simultaneous resources (DB)

Results: 168 lines shorter (~0.5%) 0 such exceptions over 3,900 requests Avg Response: 43.44ms (std dev 77ms)

Conclusion It is difficult to write error-handling code1) We have an analysis for finding mistakes

Dataflow analysis plus fault model We found 800 mistakes in 4M LOC (Java) We characterize these mistakes

• Programmers forget some paths

2) We propose a new feature Keep stacks of obligations at run-time Existing language features are insufficient

• Timing, bookkeeping, repeated logic … And back it up with case studies

Any Questions?

I encourage difficult questions.

Aspects

We could perhaps have used AoP to do the source-to-source transform

When we started this research, tools (e.g., AspectJ) weren’t well-suited for adding, removing and editing entire lexical scoping levels

Aspect tools also seemed like the wrong framework for a path-sensitive dataflow analysis

However, using the results of this analysis you could easily roll your own “compensation stack”-like mechanism with AoP

Do Developers Care?

Beyond the scope of this work, but … ptolemy2 Developer Rating of Reported

Bugs: 11% “in tutorials or third-party code”44% 3/5 “little used or experimental

code”19% 4/5 “definitely a bug in code that is

used more often”26% 5/5 “definitely a bug in code that is

used often”