Technology Seminar © 1998 TransEDA Ltd 1
TransEDA Technology Seminar
System on Chip (SOC)
Functional Verification
John Perry and David Dempster
Technology Seminar © 1998 TransEDA Ltd. 2
Agenda
■ Introduction■ Overview of Reuse Methodology Manual■ Review of the Design Process■ Testing Strategies■ Testing using Functional Verification
● Unit Testing● Integration and System Testing
● Regression Testing● Test Suite Optimization
■ Users’ Experiences of TransEDA’s tools■ Benefits Summary■ Recent and Current Developments■ Information Resources & Contact Points■ Question and Answer Session
Technology Seminar © 1998 TransEDA Ltd. 3
Objectives • Introduction to RMM
(Reuse Methodology Manual)
• To review typical development sequences
• To review typical testing strategies
• Describe improved testing strategies
• Outline the benefits obtainable
• Relate some Users’ Experiences
• Explain recent & current developments
• Question and Answer session
Technology Seminar © 1998 TransEDA Ltd. 4
Overview ofReuse
MethodologyManual Presented
by
John Perry
Technology Seminar © 1998 TransEDA Ltd. 5
Review ofthe Design
Process
Requirements PhasePlanning Phase
Development PhaseVerification Phase
Rollout Phase
Technology Seminar © 1998 TransEDA Ltd. 6
DevelopmentPhasesConcept
Good
GenerateBehavior
GenerateTest bench
Simulate Synthesis
Bad
Place&
Route
Sign Off
Bad
Review of theDesign Process
Technology Seminar © 1998 TransEDA Ltd. 7
Source: Tandem Computers in IntegratedSystem Design, January 1997
Time (Effort)
Bug
Rat
e
Period 1 Period 2
Ship ?
Bug
Rat
e
Review of theDesign Process
Bug Ratedetection
withoutcode
coverage
Time and Effort
Technology Seminar © 1998 TransEDA Ltd. 8
Source: Tandem Computers in IntegratedSystem Design, January 1997
Time (Effort)
Bug
Rat
e
Period 1 Period 2
Ready to Ship
Bug
Rat
e
Review of theDesign Process
Bug Ratedetectionwith codecoverage
Technology Seminar © 1998 TransEDA Ltd. 9
Source: Tandem Computers in IntegratedSystem Design, January 1997
Time (Effort)
Bug
Rat
e
Period 1 Period 2
With Code Coverage
Without Code CoverageReady to
Ship
Bug
Rat
e
Review of theDesign Process
Comparisonof Bug Rate
detection
Ship?
Technology Seminar © 1998 TransEDA Ltd. 10
Review of theDesign Process
DevelopmentPhases
Time
Cos
t per
Bug
The cost of finding and rectifying abug increases exponentially during
the development cycle
Technology Seminar © 1998 TransEDA Ltd. 11
Review of theDesign Process
DesignEngineers
VerificationEngineers
FunctionalVerification
Introducingsome control
Functional Verification can help torestore communication and controlbetween engineering staff.
Brings the test engineers back into thedesign process.
Technology Seminar © 1998 TransEDA Ltd. 12
Why Verify?
■ Statistics show that coding, done well,has 1 - 3 defects per 100 statements
■ Studies indicate that Inspections canuncover only 60% of Total Defects
■ Studies show testing consumes at least50% of the Product’s Labor Costs
■ Few Engineers like testing!!■ 40% design time■ 60% testing time
Review of theDesign Process
Technology Seminar © 1998 TransEDA Ltd. 13
Why Verify?
Example
Assume 2 errors per 100 lines of code
2400 errors in 120K lines of code
If a success rate of 90% could be achieved,240 errors
would pass through the verification stage.
Could you afford to accept this???
Review of theDesign Process
Technology Seminar © 1998 TransEDA Ltd. 14
TestingStrategies
■ Design Reviews
■ Testing using Functional Verification■ Formal Verification■ Requirements Tracing
■ Emulation/ Prototyping
Technology Seminar © 1998 TransEDA Ltd. 15
Testingusing
FunctionalVerification
■ Levels of Testing
■ Testing Techniques
We will consider...
...and show how quality and productivitycan be improved
Technology Seminar © 1998 TransEDA Ltd. 16
Testing usingFunctional
Verification
Unit/Module testing
Integration orSub-system
testing
Systemtesting
- - - - - - Regression testing - - - - - -
Levels ofTesting
Technology Seminar © 1998 TransEDA Ltd. 17
UnitTesting
Testing usingFunctional
Verification
FSM
Testing each unit in isolation is normallyquick and easy
ALU+, x, -,/
3:1
MUX
Technology Seminar © 1998 TransEDA Ltd. 18
Integrationand Sub-
SystemTesting
Testing usingFunctional
Verification
FSM
Integration testing can concentrate onboundary and interface checking
ALU+, x, -,/
3:1
MUX
3:1
MUX
a b
OpCode
Technology Seminar © 1998 TransEDA Ltd. 19
FSM 3:1
MUX
select
mc1
mc2
always @ (select[1] or select[2] or select[3])
begin
case (select)
3’b001: mc1=0; mc2=0;
3’b010: mc1=1; mc2=0;
3’b100: mc1=0; mc2=1;
default: “error message”
endcase
end
Glue logic provides Statement and Branch information
Toggle and Variable Trace Coverage on signals/wiresTesting using
FunctionalVerification
Integrationand Sub-
SystemTesting
Technology Seminar © 1998 TransEDA Ltd. 20
Testing usingFunctional
Verification
SystemTesting
As a system is made up of sub-systems,the sub-system testing strategy can alsobe applied at this level.
System testing can concentrate onboundary and interface checking
Any section could be an IP core.
Technology Seminar © 1998 TransEDA Ltd. 21
Testing usingFunctional
Verification
RegressionTesting
Identifying which tests contribute to theoverall testing strategy and which testsduplicate other tests can saveconsiderable time and effort during theregression testing phase.
Which tests should I re-run?
Which tests can I remove for myregression runs?
Technology Seminar © 1998 TransEDA Ltd. 22
Test_b1 Results_1
Test_b2
Test_b3
Test_b4
Results_2
Results_3
Results_4
Testing usingFunctional
Verification
RegressionTesting
usingCoverPlus
CoverPlus identifies tests that areduplicated and tests that are a sub-test of
other tests
Technology Seminar © 1998 TransEDA Ltd. 23
Testing usingFunctional
Verification
RegressionTesting
usingCoverPlus
Analysis Simulation Results
X X
Time/Effort
Simulation time can be reduced duringregression testing by using CoverPlus
Technology Seminar © 1998 TransEDA Ltd. 24
Testing usingFunctional
Verification
RegressionTesting
usingCoverPlus
Analysis Simulation Results
Time/Effort
Simulation time, during regressiontesting, can be reduced further usingmultiple workstations with CoverPlus
X X
X X
X
X X
Technology Seminar © 1998 TransEDA Ltd. 25
TestingTechniques
■ Functional (or BLACK box testing)
■ Structural (or WHITE box testing)
■ Error-Oriented
■ Stress/Performance
Testing usingFunctional
Verification
Technology Seminar © 1998 TransEDA Ltd. 26
FunctionalTesting
Testing usingFunctional
Verification
Testing restricted to checking that inputsand outputs are correct.
Technology Seminar © 1998 TransEDA Ltd. 27
StructuralTesting
Out2:=1
Bus3 := Bus12 + 1
out200 := Sig2
Sig1==0
Sig4 := 0
Sig6 := 0
Bus3 := Bus12 + 2
Sig4 := 1
Sig5 := 0
Testing usingFunctional
Verification
Testing involves checking the logicalpaths and data conditions.
Technology Seminar © 1998 TransEDA Ltd. 28
StateMachineTesting
Importance of State Machines
■ 30% of an ASIC is a state machines
■ Typical state machines have 28 states
■ Typical state machines have 50 transitions
■ There is increasing demand for good quality IP
■ System on Chip state machine interactions
Critically important to verify statemachines to avoid coding errors and
operational problems
Testing usingFunctional
Verification
Technology Seminar © 1998 TransEDA Ltd. 29
Users’Experiences
withTransEDA’s
tools
Case Studiespresented
by
John Perry
Technology Seminar © 1998 TransEDA Ltd. 30
BenefitsSummary ■ Application to Hardware Design
■ Benefits of HDL Coverage
■ How it fits in the Design Flow
■ Description of Functional Verification
FunctionalVerification
Technology Seminar © 1998 TransEDA Ltd. 31
Applicationto Hardware
DesignFront-endSimulation
Regression Testing
What testsdo I need?
Have all mysignals
been toggled?Have I done
enoughsimulation?
What is leftto test?
What are thedifficult
blocks to test?
■ Sign-off
Unit
Sub-System
Integration
SystemBenefitsSummary
Technology Seminar © 1998 TransEDA Ltd. 32
■ Reduce Risk by Increasing Quality
■ Measure and Test in a controlled and thorough manner
■ Improve Productivity
■ Reduce Development Time by focusing effort on Untested Code
Benefits ofHDL
functionalverification
BenefitsSummary
Technology Seminar © 1998 TransEDA Ltd. 33
FunctionalVerification
fitsconvenientlyin the Design
Flow
ConceptGood
GenerateBehavior
GenerateTest bench
Simulate
Regression TestingCoverPlus
Synthesis
Bad
Bad
Place & Route
Sign off
Bad
BenefitsSummary Functional Verification
HDLCover VeriSure VHDLCoverStateSure
Technology Seminar © 1998 TransEDA Ltd. 34
CodeCoverage
Metrics
■ Implementation Focused
● Statement/Line
● Branch● Condition/Expression
● Path
■ Functionality Focused
● Triggering
● Toggle● Signal/Variable Trace
BenefitsSummary
Technology Seminar © 1998 TransEDA Ltd. 35
Easy toInstrument
BenefitsSummary
Select the source files
Select the coverage options for each source file
HDLCover
VeriSure
VHDLCover
Technology Seminar © 1998 TransEDA Ltd. 36
QuantitativeMeasurement
ofImplementation
Simulation
BenefitsSummary
HDLCover
VeriSure
VHDLCover
Technology Seminar © 1998 TransEDA Ltd. 37
StatementCoverageExample
BenefitsSummary
Highlights code-lines not executed
Shows number of times code-line executed
HDLCover
VeriSure
VHDLCover
Technology Seminar © 1998 TransEDA Ltd. 38
BranchCoverageExample
BenefitsSummary
Shows number of times each branch has been taken
HDLCover
VeriSure
VHDLCover
Technology Seminar © 1998 TransEDA Ltd. 39
Condition/Expression
CoverageExample
BenefitsSummary
Shows data conditions associated with decision paths
HDLCover
VeriSure
VHDLCover
Technology Seminar © 1998 TransEDA Ltd. 40
PathCoverageExample
BenefitsSummary
HDLCover
VeriSure
VHDLCover
Identifies paths that have not been visited
Technology Seminar © 1998 TransEDA Ltd. 41
TriggeringCoverageExample
BenefitsSummary
Shows activity (expressed as a percentage) ofthe items in the sensitivity list and highlights
signals that have not triggered
33%
HDLCover
VeriSure
VHDLCover
Technology Seminar © 1998 TransEDA Ltd. 42
ToggleExample
BenefitsSummary
Shows toggle-count and signals that have no activity
HDLCover
VeriSure
VHDLCover
Technology Seminar © 1998 TransEDA Ltd. 43
Signal TraceExample
BenefitsSummary
Displays combinations of user-selected signals
HDLCover
VeriSure
VHDLCover
Technology Seminar © 1998 TransEDA Ltd. 44
StateSure
■ Static analysis to highlight coding errors
■ Automatic state diagram generation
■ Verification results annotated on diagram
■ Design hierarchy annotated with
verification results
■ State machines ranked by verification
results
BenefitsSummary
Technology Seminar © 1998 TransEDA Ltd. 45
StateSure
BenefitsSummary
Technology Seminar © 1998 TransEDA Ltd. 46
StateSure
■ Verify single state machine sequences
■ Verify pairs of state machine sequences
■ Verify paired state machine transitions
■ Verify multiple state machine interactions
■ StateSure supports Verilog, VHDL or both
■ StateSure on UNIX or WindowsNT
BenefitsSummary
Technology Seminar © 1998 TransEDA Ltd. 47
StateSure
BenefitsSummary
Technology Seminar © 1998 TransEDA Ltd. 48
CoverPlus
■ Sort the most productive test cases to be run first inthe regression suite
■ Simulate the most productive test cases
■ Remove unnecessary test cases
■ Improve bug fixing productivity
■ Plan a short regression suite for the highest coverage
in the shortest possible time
■ Find the difference between test cases, so thatadditions can be made to one test to eliminate a test
case from the suite
■ Find which test cases check specific blocks of code,so increasing productivity after bug fixes or changes
BenefitsSummary
Technology Seminar © 1998 TransEDA Ltd. 49
Start CoverPlus and load simulation runs
BenefitsSummary
CoverPlus
Technology Seminar © 1998 TransEDA Ltd. 50
Sort the simulation runs for full regression suite
Required Runs
Run has identical twin
CoverPlus
BenefitsSummary
Technology Seminar © 1998 TransEDA Ltd. 51
Reduce requirement to statement only for daily regression
CoverPlus
BenefitsSummary
Technology Seminar © 1998 TransEDA Ltd. 52
Compare test cases to find code covered by only one of the two tests
Find out what code is covered by List A
CoverPlus
BenefitsSummary
Technology Seminar © 1998 TransEDA Ltd. 53
Locate which parts of the design are covered uniquely by one test
CoverPlus
Technology Seminar © 1998 TransEDA Ltd. 54
FunctionalVerification
Strategy
■ Adopting an Effective Strategywill Increases Your Productivity
■ Functional Verification Strategy:● Execute at least one Statement in 90% of source files
— add vectors and debug design
● Reach 100% of Statement and Branch coverage
— add vectors and debug design
● Reach 90% of Condition Coverage
— add vectors and debug design
● Reach 100% Condition Coverage
— optimize Regression Suite
— eliminate unnecessary tests
● Reach 100% coverage on other measurements
(Path, Toggle, Signal Trace, etc.)
BenefitsSummary
Technology Seminar © 1998 TransEDA Ltd. 55
Recent
and
Current
Developments
Recent Developments (version 4.2)
• FEC (Focussed Expression Coverage)
• Enhanced command line options/switches
• StateSure
• CoverPlus
Current Developments (version 5.0)
• HDLCover (language neutral)
• NEW improved fast GUI
• Windows NT support
• VHDL-93 and Instance-by-Instance
Current Developments (version 5.1)
• Joint language product (Verilog & VHDL)
Technology Seminar © 1998 TransEDA Ltd. 56
FocussedExpression
Coverage
Methodology that concentrates on choosingthe most effective set of test vectors for aboolean expression.
Saves time by identifying the missing orredundant test vectors.
Normally only requires N+1 vectors or 2Nvectors where N is the number of inputs.
Reduces simulation time.
HDLCover
VeriSure
VHDLCover
Technology Seminar © 1998 TransEDA Ltd. 57
FocussedExpression
Coverage
FEC methodology is based on checkingthat an input can control the output signaland drive it to ‘0’ and ‘1’
abc
& Out
Although there are 8 combinations of inputpatterns, only the following 4 test vectorsare needed to fully exercise the circuit.
a,b,c = [1,1,1], [0,1,1], [1,0,1], [1,1,0]
This methodology is referred to as N+1 asnormally the number of test vectorsrequired is one more than the number ofinputs.
A detailed Application Note regarding theoperation of FEC can be found at:
www.transeda.com/info-japan/
HDLCover
VeriSure
VHDLCover
Technology Seminar © 1998 TransEDA Ltd. 58
FocussedExpression
Coverage
Example of the diagnostic report from FEC
Example of missing test vector needed to check theoperation of input b1
HDLCover
VeriSure
VHDLCover
Technology Seminar © 1998 TransEDA Ltd. 59
FocussedExpression
Coverage
The FEC methodology can be used to define the testvectors you should use to fully validate your expression.
Simply apply one test vector, via the test-bench, to themodule under test then use the diagnostic report toidentify the vectors that are actually needed.
Finally update the test-bench with the necessary testvectors and re-run the analysis.
Below is an example of a fully validated expression.HDLCover
VeriSure
VHDLCover
Technology Seminar © 1998 TransEDA Ltd. 60
InformationResources
and
ContactPoints
A PowerPoint copy of this seminar together with theCustomer Case Studies, Application Notes on FocussedExpression Coverage and color TransEDA’s Product
Brochures can be found at:
www.transeda.com/info-japan/
TransEDA Ltdweb site: www.transeda.com
Sales and General Enquiries
John Perry [email protected]
David Dempster [email protected]
Technology Seminar © 1998 TransEDA Ltd. 61
InformationResources
and
ContactPoints
Kanematsu Electronics Ltd.,Industrial Electronics Systems Division
[email protected]: +81-3-5250-6306
Fax: +81-3-5250-6039