Topics in Verification: Reuse, Coverage, Regression Engineering, Planning, Qualification

Welcome to RTP Welcome to RTP DVClubDVClub!!

Topics in VerificationTopics in Verification

Pete LaFauciPete LaFauciOctober 18, 2006October 18, 2006

AgendaAgenda

Brief Discussion/Overview about Brief Discussion/Overview about ““the Clubthe Club””Historical Perspective of Languages & SimulatorsHistorical Perspective of Languages & SimulatorsTopics in VerificationTopics in Verification

ReuseReuseCoverageCoverageRegression EngineeringRegression EngineeringPlanningPlanningQualificationQualification

SummarySummary

DVClubDVClub Overview Overview Voluntary, Community Based Verification GroupVoluntary, Community Based Verification Group

Discuss Present Day ChallengesDiscuss Present Day ChallengesEDA neutralEDA neutralPresentation of Presentation of ““sharablesharable”” ideas, experiences, and resultsideas, experiences, and results

Advisory BoardAdvisory BoardCurrently about 10 members, spanning 6 companies (looking for Currently about 10 members, spanning 6 companies (looking for additional participants)additional participants)Planning, Logistics, Speakers, TopicsPlanning, Logistics, Speakers, Topics

Topics should be Chip Verification related, but can span Topics should be Chip Verification related, but can span adjacent areas:adjacent areas:

Design for Verification, IT, Project Management Design for Verification, IT, Project Management

Please Contact Pete LaFauci or Justin Sprague regarding interestPlease Contact Pete LaFauci or Justin Sprague regarding interestin the Advisory Board or Presenting at in the Advisory Board or Presenting at DVClubDVClub!!

Historical Perspective: WhatHistorical Perspective: What’’s dones doneProduce some test vector stimulus, and simulateProduce some test vector stimulus, and simulate

Look at the output and waveforms on workstation, debug the rest Look at the output and waveforms on workstation, debug the rest in the labin the lab

Produce a test plan and lots of directed, procedural test casesProduce a test plan and lots of directed, procedural test casesRun each test case in the test bucketRun each test case in the test bucketCompletion: each individual test case passes Completion: each individual test case passes

Produce a verification plan, which includes model Produce a verification plan, which includes model requirements, functions & coverage goalsrequirements, functions & coverage goals

Combine generation capabilities, checkers, randomness, and coverCombine generation capabilities, checkers, randomness, and coverage monitors to age monitors to gauge the simulationsgauge the simulationsRun tests with multiple Run tests with multiple ““seedsseeds””Completion: all tests pass, and coverage output is Completion: all tests pass, and coverage output is ““analyzedanalyzed””

Produce a verification methodology, Produce a verification methodology, ““architectarchitect”” a a verification environment & plan, and leverage powerful toolsverification environment & plan, and leverage powerful tools

Automate both efficiency and thoroughness, through prediction, sAutomate both efficiency and thoroughness, through prediction, steering, and proofsteering, and proofsVIP, Environment, Library, and Test VIP, Environment, Library, and Test ““ReuseReuse””““RankRank”” the test suite efficiency, directed tests for coverage closurethe test suite efficiency, directed tests for coverage closureCompletion: all tests pass, coverage goals are met using measuraCompletion: all tests pass, coverage goals are met using measurable metricsble metrics

Then

Now

Historical Prevalence: Historical Prevalence: Verification LanguagesVerification Languages

VHDLRecords / Dynamic Variable Indices & Slicing2-d ports

HVL – ‘e’ & VeraConstraint SolversEasy RTL AccessFunctional CoverageBuilt-in libraries

C, C++. PerlComplex Data StructsSuperior String Functions

VerilogConcurrencyTasks & FunctionsPLI

Early ’90s

Late ’90s –Early 00’s

Mid ’90s

Today / Tomorrow / Future?

Modeling – SystemCHigher Abstraction, TLM, HW/SWAlgorithmic Modeling & Prototype

HVDL – SystemVerilogRTL/HVL Integration/UnificationSystemC & TLM InterfacesAssertion Based VerificationDPI

ReuseReuse

Reuse: Going Beyond Traditional VIPReuse: Going Beyond Traditional VIP

““ComponentsComponents””: traditional VIP: traditional VIPCommon bus and data protocolsCommon bus and data protocolsMonitors, Predictors, DriversMonitors, Predictors, Drivers

Environments: Environments: new(ernew(er) with OOP/HVL) with OOP/HVLTestbenchesTestbenches & & ““HarnessesHarnesses””Constraints & Test CasesConstraints & Test CasesConfiguration ComponentsConfiguration ComponentsAddresses ScalabilityAddresses Scalability

“instance & program”

“inherit, derive, build, configure”

Reuse: Inheritance and AbstractionReuse: Inheritance and Abstraction

Base Class Packages

DUT templateTest Harness

Globals

MemoriesRegistersData Objects

Interface & Protocoltemplates

Harness

Drivers

Scoreboard

User Test Case & Constraints

Environment Test Methods and User InterfacesTest Benches/Harnesses

DUT VIP

Tool Packages: any_unit, any_env, any_sequence

ApplicationSpecific

VerificationEnvironment

CompanyLibrary

InheritanceAnd

Instantiations

BaseLibrary

Protocol VIPInterface VIP

Note : “Horizontal (Interface/Protocol) and Vertical (Base Classes and Methods) Reuse Strategies are Deployable using OO Design Techniques”

CoverageCoverage

Coverage: TypesCoverage: Types

Code CoverageCode CoverageAssertion & Formal CoverageAssertion & Formal CoverageFunctional CoverageFunctional CoverageIntegration CoverageIntegration CoverageSystem & Validation CoverageSystem & Validation Coverage

Coverage: Achieving Coverage: Achieving ThoroughnessThoroughness

Identify which Design Functions have not been exercisedIdentify which Design Functions have not been exercisedIdentify What Test Code has not been simulated Identify What Test Code has not been simulated Complements the Checking Code (traffic types, latency, Complements the Checking Code (traffic types, latency, performance, etc)performance, etc)

StructuralCoverage

Assertions

CodeCoverage

FunctionalCoverage

Coverage: Collection, Merging, Coverage: Collection, Merging, and Reportingand Reporting

Project Level Coverage Progress Reports

Regression 1Test Cases, Procedural

Code

Test Cases, Procedural

Code

Test Cases, Procedural

Code

Test Cases, Procedural

Code

SimulationCoverage

Output (Cover DB)

Coverage DataMerging

“Process”

Test Cases, Procedural

Code

Test Cases, Procedural

Code

Test Cases, Procedural

Code

Test Cases, Procedural

Code

SimulationCoverage

Output

Regression n “Total Coverage”Reports & HTML views

Test Cases, Procedural

Code

Test Cases, Procedural

Code

Test Cases, Procedural

Code

Test Cases, Procedural

Code

SimulationCoverage

Output(Cover DB)

Coverage: Challenges in Coverage Coverage: Challenges in Coverage Driven MethodologyDriven Methodology

Goals are manually created from the engineering Goals are manually created from the engineering Specs Specs --> Labor Intensive> Labor IntensiveRandom generation can be wasteful if overall Random generation can be wasteful if overall coverage is not increasingcoverage is not increasingSimulation cycles can be slow and/or expensive, Simulation cycles can be slow and/or expensive, especially if they are being wastedespecially if they are being wasted

Execution Predictability with Schedules & Execution Predictability with Schedules & ResourcesResources

Coverage: Execution PredictabilityCoverage: Execution Predictability

Tigr is 2.0 Test Cover age Commit

1500

1700

1900

2100

2300

2500

2700

2900

Total Planned

Wr i tten Actual

Wr i tten Commit

Cover ed Actual

Cover ed Commit

Coverage Progress Chart: “S” Curve Phenomena

Regression Regression ““EngineeringEngineering””

Regression Regression ““EngineeringEngineering””: : ChallengesChallenges

Manage thousands of simulation jobs & resultsManage thousands of simulation jobs & resultsMultiple environmentsMultiple environmentsMany Servers, Lots of output!Many Servers, Lots of output!Team environment, Individual OwnersTeam environment, Individual Owners

AnalysisAnalysisReproducing test fails easily & accurately Reproducing test fails easily & accurately --> test/bench, machine, seed, tools, > test/bench, machine, seed, tools, versionsversionsMore debug output necessary?More debug output necessary?Correctly assigned or delegated to appropriate Correctly assigned or delegated to appropriate owner(sowner(s))

EfficiencyEfficiencyDynamically Allocating Right Machines for the Right Job TypesDynamically Allocating Right Machines for the Right Job TypesMaximize Utilization of both Hardware and SoftwareMaximize Utilization of both Hardware and SoftwareCoverage Closure: Hole Analysis, RankingCoverage Closure: Hole Analysis, Ranking24/7 rebalancing, including interactive session priority managem24/7 rebalancing, including interactive session priority managementent

PlanningPlanning

Planning: Improvements in Plan Planning: Improvements in Plan Creation and Change ManagementCreation and Change Management

More Tightly Couple the Design Specification with the More Tightly Couple the Design Specification with the Verification PlanVerification Plan

Change Management SystemChange Management SystemMore Automatic Synchronization and NotificationMore Automatic Synchronization and Notification

Improve the Efficiency between Coverage Plan and Improve the Efficiency between Coverage Plan and Implementation Implementation

Reduce or Remove the Disconnect between the Functional Reduce or Remove the Disconnect between the Functional Coverage Code and the Coverage PlanCoverage Code and the Coverage PlanAutomate Functional Coverage Code Automate Functional Coverage Code ““GenerationGeneration””

QualificationQualification

Qualification: WhatQualification: What’’s this?s this?

What is Qualification?What is Qualification?WhoWho’’s checking the verification, and when?s checking the verification, and when?Error insertion to prove the accuracy and effectiveness of your Error insertion to prove the accuracy and effectiveness of your verification systemsverification systemsError insertion is placed in the DUT (not verification code)Error insertion is placed in the DUT (not verification code)Challenges in adoption: Challenges in adoption: ““how, what, whenhow, what, when””, effort levels, effort levels

Benefits for SimulationBenefits for SimulationQualify the Test BenchQualify the Test BenchQualify the Models & VIPQualify the Models & VIPQualify the Test Cases & ConstraintsQualify the Test Cases & Constraints

Benefits for Formal ProofsBenefits for Formal ProofsQualify the AssertionsQualify the AssertionsQualify the ConstraintsQualify the Constraints*Qualify the Answer the Tool is Giving you**Qualify the Answer the Tool is Giving you*

SummarySummary

Combining innovative techniques, tools, processes can Combining innovative techniques, tools, processes can yield significant results!yield significant results!

Not Not ““uncommonuncommon”” to achieve 10x productivity and to achieve 10x productivity and thoroughness improvements through methodology and thoroughness improvements through methodology and tuningtuning

Verification processes will continue to change Verification processes will continue to change considerably to keep up with managing the exponential considerably to keep up with managing the exponential growth in design complexitygrowth in design complexity

Interesting Materials, Good Food,Catch up with other Engineers in the Industry

Invite your Co-Workers!

www.dvclub.org