Applications of Synchronization Coverage A.Bron,E.Farchi, Y.Magid,Y.Nir,S.Ur Tehila Mayzels 1.

1

Applications of Synchronization CoverageA.Bron,E.Farchi,Y.Magid,Y.Nir,S.Ur

Tehila Mayzels

2

Outlineo Motivation o Introducting Coverageo Good Coverage Modelo Synchronized Coverage Modelo ConTest Implementationo How does it help?o Summary & Conclusions

3

Motivation Software Development Life Cycle

Design

Implementation

(Coding)

Testing

Deployment

Test plan

Implementation

Regression execution

Debugging

Code fixing

Verify?

4

Testing - Issueso Where to focus?o Should I add more tests?o Are there irrelevant tests?o When to stop testing?o Is it good enough?

5

Introducting CoverageDefinition: A metric for measuring what percentage of code/functionality has been exercised by a test suite (regression suite).

For a given system, define a set of testing tasks.For each task, check if it was actually performed (covered) in some test(s).

Measure the fraction of tasks performed.

Helps to:o Find uncovered code.o Create good regression suite.o Test the quality of the testing.o Guide code review.

6

Sequential Coverage Metrics

1. Control flow based:o Statement coverage – most popularo Branch coverage

2. Data flow based:o Define-Use

3. Input based4. Specification based

7

Characteristics Of a Good Coverage Model

o Tasks are statically generated from the code and are well understood by the user.

o Almost all tasks must be coverable (or at least a simple review is possible for the cases in which they are not).

o Uncovered tasks should yield an action item.

o Some action is taken upon 100% coverage.

8

Moving to Concurrent World…

Testing is much more complicated:o Non determinism: difficult to debug and

replay.o Sequential coverage models are not

useful for interleaving space.o Bugs appear only in specific

configurations: difficult and expensive to find, and worse – commonly found by users.

9

Example – How much is 1+10+100+1000?

static final int NUM = 4;static int Global = 0;public static void main(String[] argv) { Global = 0; threads[0] = new Adder(1); threads[1] = new Adder(10); threads[2] = new Adder(100); threads[3] = new Adder(1000); // Start Threads for(int i=0;i<NUM;i++) { threads[i].start(); } try{ Thread.sleep(1000); } catch(Exception exc) {} System.out.println(Global);}

class Adder extends Thread { int Local; Adder(int i){ Local = i; { public void add() { example.Global += Local; } public void run() { add(); }

Based on an example from Testing and Debugging Concurrent Software by Shmuel Ur

10

Example – Cont.add method

Java Source codepublic void add() {

example.Global += Local;{

Byte Code Method void add()

0 getstatic #3 <Field int Global>3 aload_04 getfield #2 <Field int Local>7 iadd8 putstatic #3 <Field int Global>11 return

11

Synchronization Coverage: 1st Attempt

Concurrent pair of events:

Piece of code1

Piece of code2execution

False – executed by the same thread True – executed by different threads

Problems:1. Too hard – too many tasks, difficult to cover and not

very important.2. Analysis is not automatic – which tasks are

coverable?

12

Synchronization Coverage: 2nd Attempt

Shared variables

Var X

Problem – like before:Analysis is not automatic – which tasks are coverable?

13

Synchronization Coverage: The Chosen Attempt

o Looking for “interesting” situations.o Handle statements with synchronization

primitives.

ConTestConcurrent Testing Tool

14

ConTest Implementationo Supports Java and C/Pthread.o The code is instrumented, a list of the

tasks is computed.o In test runtime, it keeps a

representation of the synchronization objects, e.g., in order to know when a lock is held.

o Noise injection mechanisms (yields and sleeps) help obtaining the tasks.

o Can be adopted for any set of synchronization primitives.

o Task definition for each primitive depends on semantics.

15

Mutual Exclusion – synchronize(most important)

Blocked – A thread waited there, because another thread held the lock.Blocking - A thread holding the lock there caused another thread to wait.

Example:

synchronized(lock1) { counter++; updateDB();}

synchronized(lock1) { counter--; updateDB();}

blocking

blocked

blocking

blocked

16

Mutual Exclusion – synchronize(most important)

Blocked – A thread waited there, because another thread held the lock.Blocking - A thread holding the lock there caused another thread to wait.

Example:

synchronized(lock1) { counter++; updateDB();}

synchronized(lock1) { counter--; updateDB();}

blocking

blocked

blocking

blocked

17

Mutual Exclusion – tryLock)(Failed – Another thread holds the lock.

Succeed – Succeeded to get the lock.

Lock lock = ...; if (lock.tryLock()) { try { // manipulate protected state } finally { lock.unlock(); } } else { // perform alternative actions }

18

Wait On Condition – wait)( Repeated – wait was called at least twice in the same block, in the same run, by the same thread.

public synchronized void foo} () while(!cond) }

try } wait ;()

{ catch (InterruptedException e) {} {

System.out.println(“cond happened!”) ;{

19

Semaphore wait – acquire)(

Blocked – The semaphore was not immediately available.Not-blocked – The semaphore was immediately available.

private Semaphore writeLock = new Semaphore(1);

public void getWriteLock() throws InterruptedException { writeLock.acquire();}

public void releaseWriteLock() { writeLock.release();}

20

Semaphore-try-wait –tryAcquire)(

Succeeded – Succeeded to get the semaphore.

Failed – Another thread holds the semaphore.

21

Notify – notify()/notifyAll()/signal()/signalAll )(

Had-target – There was another thread in a corresponding wait. Had-no-target – There was no other thread in a corresponding wait.

Do you remember the Lost Notify bug from

Keren’s lecture ?

What will it cover here?

22

More synchronization tasks?Interrupted task for wait():

Ways to stop wait():• Notify• Interrupt (if in use)• Timeout• Spuriously in some systems

23

Which task cannot be covered here?

public static void main (String args[]) { ... synchronized (lock) { new Thread(...).start(); ... }}

24

Coverage Requirements

o Tasks are statically generated from the code and are well understood by the user.

Each synchronization primitives leads one or two tasks .

Understanding is less trivial than for statement coverage, but not too difficult for a concurrent programmer.

25

Coverage Requirements

o Almost all tasks must be coverable

Exceptions are rare.

A synchronized block is written in order to make threads wait.

If it can’t happen, perhaps it’s redundant.

If it can – maybe it’s because the test is complicated.

26

Coverage Requirements

o Uncovered tasks should yield an action item.If the synchronization is redundant, remove it.

Otherwise, some interleaving scenario has not been tested. Strengthen the test.

If the task cannot be covered, remove the task.

27

Coverage Requirements

o Some action is taken upon 100% coverage

This can be one of the criteria for exiting the unit test when testing a concurrent program.

28

How Does It Help? Guiding Review

Dead code

Synchronization statement is unnecessaryMissing tests

Synchronization statement is suspected as unnecessary

Very hard to test

Missing application

code required

29

And After The Review: Action Items

Write more tests

Remove synchronization

statements

Change part of the design

Rewrite part of the code

Fix bugs!!!

Further review

30

Summary & Conclusionso Coverage is a useful testing technology.o Verifying concurrent programs is

difficult.o Synchronization Coverage helps testing

the concurrent code.o ConTest - a concurrent testing tool that

helps defining the tasks, running regression suite and gaining 100% coverage.

31

32

BackupSlides

33

Coverage Results – W/O ConTest

Taken from Testing and Debugging Concurrent Software by Shmuel Ur

34

Coverage Results – with ConTest

Taken from Testing and Debugging Concurrent Software by Shmuel Ur