DV Club Europe - Test and Verification Safety · PDF fileDV Club Europe Certitude TM on C/C++ ... HTML Report File Tcl DB ... Certitude: Beyond Traditional Coverage C++ example: cppcheck

© 2014 Synopsys, Inc. All rights reserved. 1

DV Club Europe

CertitudeTM on C/C++

JM Forey

2014, April the 28th


Verification Goals

• Checks whether something complies with its specification

and intended usage

– Using the lowest possible amount of resources


But Verification Is Difficult…

• Not specific to electronic or software

– One way to do something right, billions to do it wrong

– Wrong assumptions about what to verify

• False positive issues are silent

– Does the stimulus cover all scenarios effectively?

– Are the checkers able to catch all possible misbehaviors?

– Are infrastructure or process issues masking bugs?

• Inadequate verification affects productivity, quality, reputation, revenues

Must assess verification effectiveness


Effective Verification Exercise, propagate, and detect bugs

To detect a bug…

• The test must activate the bug

• An effect of the bug must propagate to an “output” of the software

• The testing infrastructure must detect the behavior difference due to the bug

C/C++ under

Verification

Testing Infrastructure

Compare

Bug

Test

Cases

Expect Results

Detection Propagation Activation


Assessing Verification Effectiveness

C/C++ under

Verification


Compare

Bug?

Expect Results

Code coverage tells about what is (not) exercised,

but nothing about propagation nor detection

??? ???

Test

Cases

Activation


Introducing Certitude

• Automatically inserts “artificial bugs” called faults into the design

• Measures the ability of the verification environment to activate, propagate, and detect faults

• Identifies missing / broken / incomplete test scenarios

• Identifies missing / broken checkers

• Provides objective measure of overall effectiveness and robustness

C/C++ under

Verification


Compare

Fault

Expect Results

Functional Qualification

Detection Propagation

Test

Cases

Activation


How Fault Injection Works?

• Modifies code to insert faults

o1 = f(i1) o1 // variable doesn’t receive value

if (a) if (TRUE) // fault forces execution of “true” branch

f1(); f1();

else else

f2(); f2();

a = b | c a = b & c // fault changes operator

• Run broken code

– Does at least one test fail? Great!

– Environment is robust enough to detect the software / model is broken

– Do all tests pass? Help!

– Original and broken software both compliant with environment


Certitude Flow with C/C++

• Leverages existing regression

environment

• Certitude steps

– Model

– Activate

– Detect

– Report analysis

• DUT can be

– Compiled as part of executable

– A separate linked shared object

Executable Executable .so

Test

Case

IDs

C

C++

Compile

Script Execute

Script

Certitude HTML

Report

Config

File

Tcl DB

Access

Verification Environment

Executable


Certitude: Beyond Traditional Coverage

C++ example:

cppcheck software


Highlighted locations:

Inserted faults

Click to get details on

fault




After regression run

Not exercise code

Conditions were

always false



After detection run

Uncaught systematic faults:

Anything not green



After detection run


Certitude Applications Today

• Fully automated fault insertion

– Software, reference models (ex: ISS), high level synthesis models (HLS)

– C, C++, SystemC

– Hardware models (RTL, netlist)

– vhdl, verilog, system verilog

– Static and / or dynamic verification

• Effectiveness of the verification

– Test selection

– Consistency between high level model and RTL implementation

• Robustness and effectiveness of safety mechanism

– In presence of systematic or random faults

– For ISO26262 can the model recover, return to a safe state or report an undesired fault


DV Club Europe - Test and Verification Safety · PDF fileDV Club Europe Certitude TM on C/C++ ... HTML Report File Tcl DB ... Certitude: Beyond Traditional Coverage C++ example: cppcheck

Documents