Verification and Validation
OMAP™ verification
Presented by: Somdipta Basu RoyTexas Instruments Inc. Dallas
[email protected]+1 214-236-4382
Verification and Validation
Outline
• OMAP™ SOC overview• OMAP organization and execution• Overall Verification Methodology • Strategy adopted
– Module level– Subsystem– Chip
• Progress tracking / metrics• Summary
Verification and Validation • OMAP™
positioning
Verification and Validation
OMAP2420™ Overview• ARM1136 Based Soc
includes– 330 MHz ARM1136– 220 MHz TI
TMS320C55xTM DSP– 2D/3D graphics
accelerator– Imaging and Video
accelerator – High-performance
system interconnects and industry-standard peripherals
• ARM1136 Based Soc includes
– 330 MHz ARM1136– 220 MHz TI
TMS320C55xTM DSP– 2D/3D graphics
accelerator– Imaging and Video
accelerator – High-performance
system interconnects and industry-standard peripherals
Verification and Validation
OMAP3430™ overview• New OMAP™ 3 architecture combines
mobile entertainment with high performance productivity applications
• Industry's first processor with advanced Superscalar ARM® Cortex™-A8 RISC core enabling 3x gain in performance
• Industry's first processor designed in 65-nm CMOS process technology adds processing performance
• IVA™ 2+ (Image Video Audio) accelerator enables multi-standard (MPEG4, WMV9, RealVideo, H263, H264) encode/decode at D1 (720x480 pixels) 30 fps
• Integrated image signal processor (ISP) for faster, higher-quality image capture and lower system cost
• Flexible system support – Composite and S-video TV output – XGA (1024x768 pixels), 16M-color (24-bit
definition) display support – Flatlink™ 3G-compliant serial display and
parallel display support – High Speed USB2.0 On-The-Go support
• Seamless connectivity to Hard Disk Drive (HDD) devices for mass storage
• Leverages SmartReflex™ technologies for advanced power reduction
• M-shield™ mobile security enhanced with ARM TrustZone™ support
• Software-compatible with OMAP™ 2 processors
• HLOS support for customizable interface
Verification and Validation
OMAP development organization
• OMAP chip level is divided into several subsystems (e.g. ARMSS/DSPSS/…)
• Each subsystem consist of key IPs– E.g. ARMSS ARM core, interrupt controller, security block,
bus converter bridges– Some IPs are reused from earlier programs, some are
developed for a target program• Each IP/group of IPs are developed and delivered to
subsystem (s) by IP teams (spanned in different continents)
• Each Subsystem integrates and tests IPs together and delivers subsystem to chip level
• Chip level integrates subsystems, peripherals, power and clock hookup and tests at chip level
Verification and Validation
How it is organized
Chip level teams InfrastructureDatabase
FlowTracking systems
Subsystem
IPIPIP …….
Validation
FPGA,HW Acc
Silicon Validation
Subsystem
IPIPIP …….
RTL Verification PD DFT RTL Verification PD DFT
RTL Verification PD DFT
• Now imagine that with ~70 IPs, 10-15 subsystems per chip and 4-5 new chips being done simultaneously (in parallel with 5 chips doing revisions) and 5 time zones
• Now imagine that with ~70 IPs, 10-15 subsystems per chip and 4-5 new chips being done simultaneously (in parallel with 5 chips doing revisions) and 5 time zones
Verification and Validation
How do we do it (and get it right most of the time!)
• AFV (Architecture for verification)• Strict IP to chip release criteria• Established IP-2-chip exchange mechanism• Automation• Common database / infrastructure / tracking
• And of course by increasing frequent flier miles
Verification and Validation
Architecture for verification
• It was all kinds of bus protocols and behaviors in OMAP1 series of products
• OMAP2/OMAP3– Standard bus protocol interconnect– All masters and slaves follow variations of same protocol– Plug-and-play
• Not everything is so perfect• Power and clock hookup / verification is challenging• Debug protocol complicated
Verification and Validation
IP to chip release
• Pre-defined RTL milestones• Ordered by RTL maturity
– Verification status– Physical design step completion
• Clear exit criteria• Same for all IPs / subsystems• But
– Exception always exists– Had to accept/integrate/test critical IPs before they have
completed
Verification and Validation
IP to Chip milestones
Physical DesignRTL verificationIntegrationDB setup/planning
>80% doneBasic testingDB set up / Planning 100% verification
Chip
IP
Reviews
Verification and Validation
IP to chip exchange
• Design delivery (standard views)– RTL– Timing related– Physical design related– …
• Verification delivery– Tests/libraries/macros from processor-based
subsystems– Test plans of subsystems for chip level review
Verification and Validation
Automation
• Automate a lot of chip level RTL coding – Hookup– I/O connection– Register configuration
• Automatically generate tests to check these features
Verification and Validation
Common database / infrastructure
• Centralized infrastructure• Common database for delivery /
exchange• IP delivery and quality tracking• Dedicated infrastructure team
Verification and Validation
Functional Verification Methodology –same established principle
• Detail verification plan
• Reviews at critical design points
• Thorough tracking
Verification and Validation
Verification Methodology
Module/Block
Constraint random testingReusable test environmentReusable stimulusExhaustive black/grey box
Subsystem
Directed and Random testingMimic chip level constraintsReuse module level environment
Chip
C/ASM based directed testingReuse from moduleSynthesizable test bench
Hardware
Same environment as chip levelApplication threadsOperating System boot up
HVL test bench / scoreboard / checker / assertions HDL test bench
Functional Coverage driven Functional Scenario driven Application driven
Design Verification Toolkit / Regression Manager / Verification Dashboard
Verification Process – checkpoints / reviews
Verification Metrics – coverage, bugs, regression, formal, cycles, efficiency tracking
Verification and Validation
Module level verification • Objective
– Validate module thoroughly before subsystem/system integration
• Goal– To achieve 100% code and functional coverage
• Strategy– Use pseudo-random test generator– Base infrastructure
• A common methodology is used for all module verification
• Common VIPs are used by modules following same protocols
– Derived components for specific modules– Black-box approach (primarily)
Verification and Validation
Module level verification
• Stimulus: Directed-random / random
• Correctness: Protocol and Data checkers (end-to-end)
• Coverage: Code and functional coverage
• Property checking for certain blocks
DESIGNUNDER
VERIFICATION
Inpu
t Por
t1
OutputPort
RegisterScoreboard
DataScoreboard
?
?
Expected data
Inpu
t Por
t2
BFM
Monitor
Coverage
Checker
BFM
Monitor
Coverage
Checker
BFM
Monitor
Coverage
Checker
Verification and Validation
Subsystem level verification • Objective
– To validate the subsystems in the design before top-level integration– Debug/isolate problems inside subsystems which are difficult to find in large
SOC• Goal
– 100% completion of directed tests as per the verification spec• Core CPU tests• Feature specific directed tests
– 100% functional coverage items re-used from IP level verification– 100% Coverage of a Manual Checklist created for test items
• Strategy– Generate test bench irritation while processor running real code– Reuse of module components– Isolate subsystem and mimic system environment to create top-level
scenarios with a much lesser simulation time
Verification and Validation
Subsystem level verification
• Stimulus: C/ASM tests for integration, boundary and functional testing• Correctness: Self-checking testing, Checkers reused from module-level• Coverage: Toggle at boundary, directed tests of all target features in the spec
CLO
CK
/R
ES
ET
INTE
RR
UP
T
Verification and Validation
Example: The ARM1136J(F)-S Subsystem test scenarios
• Reuse ARM IP test suite - Retarget CPU tests at the subsystem level- Tests that cover various AHB parameters
- Basic Boot Tests- Exception testing at subsystem context- Clock and power management tests- Feature specific testing (interrupt handling, security …) - Derivative tests
- Base tests with varying test bench parameters- Data Memory Access Tests with variable wait states in
memory- Tests run with random clock speed with allowable speed limit- Random interrupts
Verification and Validation
ARM Subsystem verification environment Components
• Mandatory components– A Clock/Reset/Idle Control Block :
• For creating multiple clock frequencies ,random/controlled reset and idle
– An Interrupt Generator BFM :• For Generating random/controlled
simultaneous interrupts and handling them– Memory interface and Memory with
variable/random wait states:• Memory model to support Instruction Read,
Data Read/Write with random latency • Optional components
– Internal Protocol Checkers• Mainly re-use from module level verification
Verification and Validation
Chip level verification • Objective
– To validate chip integration and handshaking – To validate real chip level functional scenario
• Goal– 100% scenario covered as in the plan
• Strategy– Mimic chip environment– Base SW environment for ease of reuse– Break into multiple master-slave blocks– Mix and match of real RTL and bus functional
models
Verification and Validation
Chip-Level Verification• Stimulus: C/ASM based directed tests – chip functional scenarios• Correctness: Self-checking tests, Selected checkers from module-
level• Coverage: 100% completion of all scenarios in the plan
UA
RT/ M
cBS
PD
RIV
ER
S
Trace/JTAGBFM
FlashModels
SDR/DDRModels
CameraBFM
DisplayBFM
GPIO
I/O drivers
CLK, Reset IDLE/Power Management Control Block
ARM BFM DSP BFM
Verification and Validation
Simulation environment
• Flexible environment– Replace RTL by BFMs– Software models for processors
• Test bench– Synthesizable
• Dedicated teams for environment and test bench
Verification and Validation
Software Base
• Test case use library functions• The Software Development Library
– Library routines are developed based all IP functional specs and put in a repository database to be used for all these levels of verification:
• Subsystem Level• Top Level• Chip Level actual Silicon
• A standard Format is used for all tests/subroutines/libraries
Verification and Validation
Key aspects checked at chip level
• Integration of all subsystems (achieve 100% toggle)
• Basic features• Data and control path testing• Parallel and distributed functionality• Latency / performance• Power Management• Application scenario• Debug features
Verification and Validation
Identify Critical functional arcs for OMAP2420™
Verification and Validation
Beyond RTL
• Hardware acceleration– Use at subsystem level and chip level– Stress test– Basic software checkout
• Prototyping– Use at chip level– Early software development
Verification and Validation
Verification Management• Detail test plan at every level – module/subsystem/chip• Review at critical design points with
design/spec/system teams• Tracking of
– Verification plan– Test environment development– Functional Coverage development– Coverage achievement (code, function)– Design defect– Validation defect– Test development– Test regression– Test cycles– Assertions (formal and simulation)
Verification and Validation
Metric processBug tracking
• Internal tool
Source code
• Clearcase• TDM• CVS
Runtime tools
• Modelsim• VCS• Specman• IUS
Regression engine
• Internal• others
Resource estimator
•MS Project• ????
Management request
• Trend analysis• Risk analysis• What If scenarios
Metrics Dashboard
Trend data
Verification and Validation
Verification Metrics
• Required– Bug curve (logic, DV)– Source code activity (# lines / # edits)– Cycles / bug for random testing– Passing rate
• IP level• Integration• System• ECN verification
– Code coverage (line, branch, toggle)– Functional coverage
• Level1 : Features• Level2 : Cross• Level3 : Scenario
– DV checkpoint status
• Desired– Sim farm efficiency
• Software license stall time• Setup / cleanup time• Cycles / second• % simulator / % HVL• Average / distribution for # of running
jobs• Cycles / hour
– Resource stats• Resource ramp vs. forecast• Resources invested vs. bottoms-
up plan
Verification and Validation
DV Dashboard
DV FLOW
Simulation Test Database
3rd PartyTools
Coverage monitor Database
Formal PropertyDatabase
InternalTools
Regression logs
Coverage logs
Defects SQLDB
Bug Tracking
UPLOAD (Convert to common format)
Coverage SQLDB
Regression SQLDB
Create
Simulation, Formal
Verification and Validation
Overall DV Metric System
Des
ign
A
Des
ign
B
Des
ign
C
Des
ign
D
Des
ign
E
Des
ign
F
DV MethodologyDV Status
Review DatabaseRegression/Bug/Coverage Database
DV DashboardExpectedMetrics
MethodologyCompliance
Review Status
Review SystemActual Metrics
Time
Module A, B
Coverage
Bugs
EngineeringManagement
ExecutiveManagement
Review checklists
Trend Analysis
Actual MetricsExchange
RTL DV DFT PDDesign A 80% 40% 30% 40%Design B 85% 70% 40% 50%
3rd Party Tool Engineering Analysis of coverage data
Exists: Manually collected
Exists: Automatic
Exists: Automatic
Exists: Automatic
Verification and Validation
Summary
• OMAP™ verification is a resource and time intensive task
• Detail plan and review at all levels eliminate redundancy and provide maximum coverage of functions
• Need collaboration at every level– Architecture – Design– Infrastructure– Verification
• No magic