Xilinx Confidential
Developing Video Applications on Xilinx FPGAs
Xilinx / Avnet / Mathworks Video Seminar2
Xilinx Design FlowVideo Hardware Development
Develop Executable Spec in Simulink
Create Hardware Model using
SysGen
Verify Hardware using HW Cosim
Integrate Hardware into Embedded
System
Xilinx / Avnet / Mathworks Video Seminar3
Simulink Executable Spec
• Purely algorithmic• Use abstract blocks • Define desired system response
Xilinx / Avnet / Mathworks Video Seminar4
Executable Spec Demo
Xilinx / Avnet / Mathworks Video Seminar5
Xilinx Design FlowVideo Hardware Development
Develop Executable Spec in Simulink
Create Hardware Model using
SysGen
Verify Hardware using HW Cosim
Integrate Hardware into Embedded
System
Xilinx / Avnet / Mathworks Video Seminar6
Design a Hardware Architecture
• Floating-point numbers• Defining basic elements of Hardware Architecture• Compare to Executable Spec
Xilinx / Avnet / Mathworks Video Seminar7
Define Fixed-Point Quantization
• Compare fixed-point error to golden source
Xilinx / Avnet / Mathworks Video Seminar8
Redefine Dataflow for Hardware
• Convert from frames to serial streaming• Consistent with CMOS video camera outputs
Xilinx / Avnet / Mathworks Video Seminar9
Redefine design using the Xilinx DSP Blockset
• Xilinx DSP Blockset includes over 100 DSP building blocks that have been optimized for efficient results on Xilinx devices
• Define partition using GatewayIn / GatewayOut blocks
Gateway blocks define the FPGA boundary
SysGen token block enables use of netlist generation scripts
Xilinx / Avnet / Mathworks Video Seminar10
Model Based Design Demo
Xilinx / Avnet / Mathworks Video Seminar11
Xilinx Design FlowVideo Hardware Development
Develop Executable Spec in Simulink
Create Hardware Model using
SysGen
Verify Hardware using HW Cosim
Integrate Hardware into Embedded
System
Xilinx / Avnet / Mathworks Video Seminar12
Video Hardware Verification Flow
Reference Model
Golden Input Sequences & Test Cases
Design Capture
Verification
Validation
50+ Test Sequences
Golden Test Vector Suite
Test Cases-Pedestrian Crossing-Abandon Object-Privacy Regions-Object Removal
Test Cases-Pedestrian Crossing-Abandon Object-Privacy Regions-Object Removal
Golden Test Vectors = ?
The Key is Fast Algorithm Confirmation• Simulink and the Video and Imaging blockset• Early System Architecture• Bit True, Not cycle accurate
Micro-Architecture• IO Definitions• Automated TB Generation• API Definition• Early FPGA Characterization
• HDL Simulation• Back Annotated HDL Sim• SysGen CoSim
• SysGen-VSK HwCoSim• SysGen-VSK-VFBC HwCoSim• Live Validation, Camera>VSK>Display
Xilinx / Avnet / Mathworks Video Seminar13
Accelerating Verification through Hardware
• Hardware co-verification removes simulation bottleneck– Up to 1000x simulation performance
improvement– Automates FPGA and board setup
process
DesignSimulation Time (Seconds)
Software HW Co-Sim Increase
Beamformer 113 2.5 45X
OFDM BER Test 742 .75 989X
DUC CFR 731 23 32X
Color Space Converter 277 4 69x
Video Scalar 10422 92 113X
Xilinx / Avnet / Mathworks Video Seminar14
Eliminating IO Bottlenecks using Hardware Frame Passing
System Generator includes specials “Shared Memory Read / Write” blocks to allow large amounts of data to be efficiently passed to hardware from Simulink
Boost HW co-sim performance by bundling input data samples together thus reducing simulation to hardware transactions.
- Frame size =2880
Boost HW co-sim performance by bundling input data samples together thus reducing simulation to hardware transactions.
- Frame size =2880
Xilinx / Avnet / Mathworks Video Seminar15
Simulation Runtime Improvements using Hardware Co-simulation
Abandon Object DesignSim Time(seconds)
Time / frame(seconds)
Performance Improvement
Original Simulink Abstract Model .5 .1 N/A
With SysGen block* 50 10 baseline
SysGen with HW co-sim no frames* 165 33 3X slower
SysGen with HW co-sim with frames* 15 3 3X
With input from .mat file* 10 2 5X
With in/output to .mat file* 4.2 .8 12X
* For bit-true hardware accurate simulation models
Xilinx / Avnet / Mathworks Video Seminar16
Additional Data from Xilinx Video Development Team
Frames800x600 1280x720 1920x1080
Non-Accelerated HW Accelerated Non Accelerated HW Accelerated Non-accelerated HW Accelerated
1 1243 sec 18 sec 2238 sec 25 sec 5310 sec 37 sec
2 2579 sec 25 sec 4728 sec 35 sec NA 62 sec
5 NA 43 sec NA 65 sec NA 128 sec
• Notes:– 100 Mbps Ethernet link, Effective rate ~5.1 Mbps
• ML506 Virtex-5 SXT development platform– Payload = 2 * N_Frames * Frame_Size * 32-bits– Load N frames of data, Process N Frames, Store N Frames
Xilinx / Avnet / Mathworks Video Seminar17
Hardware Co-Simulation Demo
Xilinx / Avnet / Mathworks Video Seminar18
Xilinx Design FlowVideo Hardware Development
Develop Executable Spec in Simulink
Create Hardware Model using
SysGen
Verify Hardware using HW Cosim
Integrate Hardware into Embedded
System
Xilinx / Avnet / Mathworks Video Seminar19
Why Video Systems?• Video designs generally include embedded processing
for:– Video system control and dataflow control– Table and memory updates– Low performance video processing
• Xilinx embedded processors allow high-performance video systems on a single chip– Lower cost– Higher performance– Obsolescence proof
Xilinx / Avnet / Mathworks Video Seminar20
The VSK Base System
Processor System Reset
Clock Generator
MicroBlaze Processor
MDM UARTGIP
DIP SwitchesGPIO
Push ButtonsGPIOLEDS
MPMC System ACE XPS IIC XPS IIC
DDR2
PLBARB
BlockRAM
ILMB DLMB
IXCL
DXCL
PLB
XCL
• Embedded base system provided with the VSK• Forms the framework from which video designs are created• Includes one MicroBlaze embedded processor• Customized using Platform Studio
Xilinx / Avnet / Mathworks Video Seminar21
System Generator to Embedded
• System Generator automatically generates DSP accelerators for use with the Xilinx embedded development environment (XPS)– placed into embedded IP Catalog– Supports PLB or FSL bus– Supports async clocking – Includes driver files and
documentation
• XPS project can be imported into SysGen for system debug
System Generator
Embedded Hardware
Platform Studio
Embedded Developers Kit
Embedded IP Catalog
pcores XPS Project
Xilinx / Avnet / Mathworks Video Seminar22
Video Starter Kit Reference IPCore Name
Use in Reference Designs
DVI Pass-through Camera Frame Buffer Frame Buffer
DVI_IN and DVI_OUT Yes Yes Yes
Camera processing Yes Yes
DE_GEN Yes Yes
VIDEO_TO_VFBC Yes Yes
Video Frame Buffer Controller (VFBC)
Yes Yes
• Provided as a library of “drag and drop” IP for use with the video base system
• Provides abstraction to the video interface details– Includes SW driver files
Platform Studio IP Catalog
Generated by SysGen
Reference IP included with the VSK
Xilinx / Avnet / Mathworks Video Seminar23
Abstracting the Processor Interface
• “Shared” registers, RAMs and FIFOs are used to create HW / SW abstraction – DSP design connects to a “to”
or “from” memory– Memory maps and interface
logic is added during RTL generation
– Software drivers and documentation are created for easy programming
Xilinx / Avnet / Mathworks Video Seminar24
System Design Integration Demo
Xilinx / Avnet / Mathworks Video Seminar25
Video Example #1 - VFBC
Processor System Reset
Clock Generator
MicroBlaze PRocessor
MDM UARTGIP
DIP SwitchesGPIO
Push ButtonsGPIOLEDS
MPMC
System ACE XPS IIC XPS IIC
PLBARB
BlockRAM
ILMB DLMB
PLB
Camera Input
Camera Processing
GammaVideo to
VFBCDisplay
ControllerDVI_OUTCamera Video Out
System Generator Design
VFBC
VFBC
Embedded System
User Created Video Accelerator
Video Interface Reference IP
Xilinx / Avnet / Mathworks Video Seminar26
Getting Started with VSK Reference Designs
Simplest Frame Buffer Data TransferDVI Input
Frame Buffer
DVI Output
Basic “real-time” video processingImage
ProcessingDVI Input
DVI Output
“Real-time” Frame Buffer Based Video Processing
Image ProcessingCamera
InputFrame Buffer
DVI Output