Generating Tests from Counterexamples Jinseong Jeon ARCS, KAIST.

Generating Testsfrom

Counterexamples

Jinseong JeonARCS, KAIST

2006-11-14CS750b, KAIST 2/16

Actual Anxiety

Counterexample!

What makes him panic? What makes C.E. happen makes him panic!

2006-11-14CS750b, KAIST 3/16

Greater Goals

Counterexample!

• How can it happen?

a test vector

• Any other cases?

a test suite

2006-11-14CS750b, KAIST 4/16

The Greatest Goal

Automated Debugger!

Program

TargetPred.

Test DriverGenerator

Test SuiteGenerator

Test Driver

Test Suite

Testing

2006-11-14CS750b, KAIST 5/16

Contents

Program

TargetPred.

Test DriverGenerator

Test SuiteGenerator

Test Driver

Test Suite

Testing• How to generate a test vector?

• How to generate a test suite?

• How to generate a test driver?

2006-11-14CS750b, KAIST 6/16

from Trace to Test (1/2)

Example() { if (y == x) y++; if (z <= x) y++; a = y – z; if (a < x) LOC:}

[ Program ]

assume (y = x)y = y + 1assume !(z <= x)

a = y – zassume (a < x)

[ Trace ]

<y,0> = <x,0><y,1> = <y,0> + 1: z,0> · <x,0>

<a,2> = <y,1> - <z,0><a,2> < <x,0>

[ Trace formula ]

p , ( pc LOC )

2006-11-14CS750b, KAIST 7/16

from Trace to Test (2/2)

<y,0> = <x,0><y,1> = <y,0> + 1: z,0> · <x,0>

<a,2> = <y,1> - <z,0><a,2> < <x,0>

[ Trace formula ]

<x,0> 0<y,0> 0y,1> 1<z,0> 2<a,2> -1

[ Assignment ]

<x,0> 0<y,0> 0<z,0> 2

[ Test vector ]

integer linear programming (ILP) solver

2006-11-14CS750b, KAIST 8/16

Linear Programming• object function

– maximize c1x1 + c2x2

• problem constraints– a11x1 + a12x2 · b1

– a12x1 + a22x2 · b2

• Algorithms– Simplex, Branch and Bound, etc.

feasible

region

2006-11-14CS750b, KAIST 9/16

An ILP Application•Buffer Overrun Detection using Liner Programming and Static Analysis

int main() { char header[2048], buf[1024], *cc1, *cc2, *ptr; int i; FILE *fp; ... ptr = fgets(header, 2048, fp); cc1 = copy_buffer(header); for (i = 0; i < 10; i++) { ptr = fgets(buf, 1024, fp); cc2 = copy_buffer(buf); }}

header!alloc!max · 2048header!alloc!min ¸ 0...header!used!max · 2048header!used!min ¸ 1cc1!used!max ¸ header!used!maxcc1!used!min · header!used!min...i’!max ¸ i!max + 1I’!min · i!min + 1...

2006-11-14CS750b, KAIST 10/16

Test Suite Gen.

worklist à all locations in decreasing order of d.f.numberingReach. Tree à a single node, the roottest suite à ;

while (worklist ;) { q à pop(worklist) p-trace à MC(Reach. Tree, p, q) if Reach. Tree is complete then worklist à ; else test suite à test suite [ { test_vector(p-trace) }}return test suite

2006-11-14CS750b, KAIST 11/16

Heuristics

• A test vector can cover several locations. we can remove those locations from the worklist.

• MC’s unfolding (visiting) order uncovered first, covered last

• Time-out option

2006-11-14CS750b, KAIST 12/16

Test Driver Gen.

Original code

Lib. call

User input

Test suite

<1,1,1>

<1,0,1>

<0,1,2>

<0,1,1>

<0,0,1>

Test-feeding func.

Test Driver

<0,0,0>

2006-11-14CS750b, KAIST 13/16

A Security Example (1/2)

int saved_uid, saved_euid;

work_and_drop_priv() {L5: FILE *fp = fopen(FILENAME,”w”);L6: if (!fp) {L7: return; }L8: // workL9: seteuid(saved_uid);}

int get_root_privileges() {L1: if (saved_euid == 0) {L2: return -1; }L3: seteuid(0);L4: return 0;}

int main(int argc, char *argv[]) {L10: saved_uid = getuid();L11: saved_euid = geteuid();L12: seteuid(saved_uid);L13: // work under normal modeL14: if (get_root_privileges() == 0 ) {L15: work_and_drop_priv(); }L16: execv(argv[1], argv+1);}

2006-11-14CS750b, KAIST 14/16

A Security Example (2/2)

L10: saved_uid = getuid();L11: saved_euid = geteuid();L12: seteuid(saved_uid);L14: tmp = get_root_privileges(); L1: if (saved_euid != 0) /* fails */ L3: seteuid(saved_euid); L4: return 0;L14: if (tmp == 0) /* succeeds */L15: work_and_drop_priv(); L5: fp = fopen(FILENAME, “w”); L6: if (!fp) /* succeeds */ L7: return;L16: /* uid = 0 */

[ A trace generated by BLAST ]

2006-11-14CS750b, KAIST 15/16

Experiments

• kbfiltr, floppy, cdaudio, parport, parclass

Microsoft Windows device drivers

• ping an implementation of the ping utility

• ftpd a Linux port of the ftp daemon

2006-11-14CS750b, KAIST 16/16

Conclusions• pros

– generate a test suite using model-checker– generate an automated debugger

• dead code detection, safety verification

• cons– only integer variables– what is really affected by BLAST?