Math Works

Design of RealDesign of Real--Time Video Time Video Processing Systems in Processing Systems in Simulink With the Video & Simulink With the Video & Image Processing BlocksetImage Processing BlocksetDavid JacksonProduct Marketing ManagerVideo & Signal ProcessingThe [email protected]

2© 2005 Altera Corporation

AgendaAgenda! Video & Imaging System Design

− Imaging with MATLAB®

− Key Challenges− Model-Based Design− Simulink®

! Featured Product:− Video & Image Processing Blockset

! Executable Specifications− Benefits, ROI Study

! Video Design Example− Edge Detection

! Q&A


Intelligent Video DemonstrationIntelligent Video Demonstration! Traffic Sign Detection Example


! Headquarters: Natick, Massachusetts, USA! European Offices: UK, France, Germany,

Spain, Benelux, Italy, Switzerland, Nordic! North American Offices: CA, MI,

Washington DC, TX! Asia/Pacific Office: Korea! Worldwide Consulting ! Distributors in 20 Countries

Earth’s Topography on an Equidistant Cylindrical Projection, Created with the MATLAB Mapping Toolbox

The MathWorks at a GlanceThe MathWorks at a Glance

! Revenues of ~$300M in 2004! Privately Held! Over 1,000 Employees Worldwide,

1/3 in Product Development! Worldwide Revenue Balance:

50% North America, 50% International! More than 1,000,000 Users in 175+ Countries

Video System DesignVideo System Design


DefenseAutomotive Safety

Semiconductor

CommunicationsUAV

Medical DevicesSecurity CamerasOffice Equipment

Some Image & Video Processing SystemsSome Image & Video Processing Systems


Video System Design ChallengesVideo System Design Challenges! Extreme Computation Demands! Embedded System

Resource Constraints−Real-Time Requirements

! Designing for a Target Hardware−DSP, FPGA

! End-Product Performance, Power, Size−Roadmap for Adding Features,

Performance

! Testing & Validating Results


! Foundation for Model-Based Design, Automatic Code Generation, & Verification & Validation

! Open Architecture for Integrating Models From Other Tools

! Applications in Controls, Signal Processing, Communications, & Other System Engineering Areas

Simulink®Simulink®

The Leading Environment for Modeling, Simulating, The Leading Environment for Modeling, Simulating, & Implementing Dynamic & Embedded Systems& Implementing Dynamic & Embedded Systems


Simulink Key FeaturesSimulink Key Features! Hierarchical, Component-Based

Modeling! Open Application Program

Interface (API) ! Hybrid (Mixed-Signal), Multirate

& Multitasking System Simulation! MATLAB Integration! Extensive Library of

Predefined Blocks! Application-Specific

Libraries Available− Video & Image Processing− Signal Processing− Communications− Control − Physical Modeling


Design ImplementationRequirements & Specifications

Test & Verification

Model-Based DesignModel-Based Design

Executable Specifications

Automatic Code Generation

Continuous Verification

Design With Simulation

− Reduce Ambiguity Avoid Re-Work

− Rapid Design Iterations

− Minimizes Coding Errors

− Detect Errors Earlier

Continuous VerificationModel Elaboration


Streaming Video In/Out

Detection,Thresholding

Tracking, Counting

BackgroundEstimation

Featured Product HighlightVideo & Image Processing BlocksetFeatured Product HighlightVideo & Image Processing Blockset! Provides Over 60 Components & 100�s of Algorithms

Focused on Implementation of Embedded Systems


! Analysis & Enhancement! Conversions! Filtering! Geometric Transforms! Morphological Operations! Sinks! Sources! Statistics! Text & Graphics! Transforms! Utilities

Video & Image Processing BlocksetLibraries:

Video & Image Processing BlocksetLibraries:


! Estimation! Filtering! Math Functions! Quantizers! Signal Management! Signal Operations! Sinks! Sources! Statistics! Transforms

Signal Processing BlocksetLibraries:Signal Processing BlocksetLibraries:


! Optimize Performance & Numerics Due to Finite Word Effects

! Built-In Tools for Scaling & Modeling Finite Word Effects

! Easy to Change Parameters to Simulate Impact of Rounding, Overflow, Etc.

* Signal Processing & Video & Image Processing Blocksets require Simulink Fixed-Point for Integer & Fixed-Point Data Types

Fixed-Point & Integer Modeling*Fixed-Point & Integer Modeling*


! Generate C-Code from Video & Image Processing Models− Rapidly Prototype Your Algorithms & Systems− Test & Verify With Target Hardware− Easily Retarget Your Model to Different

Processors

Real-Time Workshop® & RTW Embedded Coder Automatic C-Code GenerationReal-Time Workshop® & RTW Embedded Coder Automatic C-Code Generation

Application ExampleApplication Example


Video & Image Processing BlocksetEdge DetectionVideo & Image Processing BlocksetEdge Detection

Live SimulinkDemo


Edge DetectionEdge Detection

Model & Simulate Double, Single, Fixed-Point Data Types

Easy to Import Streaming Video Into the Simulation

Handy Viewers for Inspecting Video at Any Point in the Algorithm

Options Important to Embedded System Designers

Executable Executable SpecificationsSpecificationsAccelerating Your Video & Image Processing System Designs


Benefits of Executable SpecificationsBenefits of Executable Specifications! Explore Design Trade-Offs in Software Before Costly

Hardware/Prototyping Stage ! Simulate With Different Data Sets (Videos) to Test

Against Multiple Environments, Situations, Etc.! Block Diagram Makes It Easy to Understand Layout of

Algorithm/Design − Reduces Specification Ambiguity− Increases Communication Between Teams/Design Stages

! Facilitates IP-Reuse Between Team Members & Projects

! Copies of return on investment (ROI) white paper available at: www.mathworks.com/roi

SummarySummary


Video & Image Processing BlocksetDesign & Simulation for Video & Image Processing Systems

Video & Image Processing BlocksetDesign & Simulation for Video & Image Processing Systems! Tools for Model-Based Design:

− Create Executable Specification− Model Floating- & Fixed-Point

Data Types− Trade-Off Analysis & Visualization − Construct Test Harnesses

for Verification− Automatic C-Code Generation for

DSP, µU

ModelModel--Based Design Based Design TutorialTutorialSimulinkSimulink, DSP Builder, DSP Builder& SOPC Builder& SOPC BuilderAlex SoohooDSP Product Marketing ManagerAltera Corporation


Altera’s Model-Based DSP Design FlowAltera’s Model-Based DSP Design Flow

DSP SystemDevelopment

SOPC Builder

DSP Builder


DSP Builder Design FlowDSP Builder Design Flow

MATLAB/Simulink Domain(System Simulation & Verification)

HDL Domain( Hardware Implementation/ RTL Simulation)


DSP Builder

DSP Builder OverviewDSP Builder Overview


Place & RoutePlace & Route

Verify in HardwareVerify in Hardware

Creates SOPC Builder Component

Download Design Hardware-in-the-Loop

Download Design Hardware-in-the-Loop

Creates HDL CodeCreates HDL Code Creates Simulation Testbench

Creates Simulation Testbench

DSP Builder OverviewDSP Builder Overview

HDL SynthesisHDL Synthesis


DSP Builder Library ComponentsDSP Builder Library Components! Arithmetic! Bus Manipulation! Complex Signals ! Logical Components! SOPC Ports! Storage! MegaCore® Functions! Rate Change! State Machine! Altera® Library! DSP Board Components


SOPC Builder: System Integration ToolSOPC Builder: System Integration Tool

Standard IPFunction

Standard IPFunction

Custom LogicCustom Logic

On-ChipMemoryOn-ChipMemory

MemoryInterfaceMemoryInterface

FlashFlash

SDRAMSDRAM

ControlControl

I/OChannel

I/OChannel

Your Value: Custom Logic

Altera Value: Standard IP Functions & Interconnect Fabric

System on a Programmable Chip


Nios II Processor OverviewNios II Processor Overview! Family of 32-Bit

RISC Processors ! Library Peripherals

& Interfaces! Import Custom

Peripherals! Royalty-Free! IDE/Debugger! RTOS! TCP/IP Stack! Performance over

200 DMIPs! Cost As Low as

35¢ of LogicYour Design Here

Nios II

Nios IICPU

On-ChipROM

On-ChipROM

UART

GPIO

Timer

SPI

SDRAMController

Cac

he

Ava

lon

Switc

h Fa

bric

Debug


Nios II IDE: Editor & Software Tool-Chain Nios II IDE: Editor & Software Tool-Chain

Integrated Editor & Software Tool-chain Functionality

� Basic Editing Capabilities

� C/C++ Syntax Highlighting

� Comprehensive Search

� Help Feature

Source Editor

� Project Navigation

� Source Files

� Compiler Based on GNU Tool Chain

� Command-Line Operation Possible

Compiler

Features:� Project Manager� Editor & Compiler� Debugger� Flash Programmer

Project Management


Building a Real-Time Video SystemBuilding a Real-Time Video System! Target Application:

− Video Edge Detection! Design Steps

− Simulink Testbench− DSP Builder Implementation− SOPC Integration − Control Software Development− Program Hardware Platform

! Hardware Demo


DMADMA

31 2RAM

Video Hardware SystemVideo Hardware System

Edge DetectionCo-Processor

Edge DetectionCo-Processor

DMADMA DMADMA

Avalon

DMADMA

SOPC BuilderSOPC Builder DSP BuilderDSP Builder


Application SpecificationApplication Specification! Prewitt Edge Detection Algorithm ! VGA Resolution 16-bit RGB(5:6:5) Format Video

− 15 Hz Input Via Camera− 60 Hz Output Via VGA Display Controller

! Sample-Based Algorithm− One Pixel per Clock

! Target Hardware: Stratix® II EP2S60 DSP Development Board


Simulink TestbenchSimulink Testbench! Build Model Using Video & Imaging Processing

(VIP) Blockset! Read Video From a File to Matrices Representing

Image (One Matrix per Color Channel)! Outputs Three Displays

− Original Image− Edge Detected Image− Edge Detected Image Overlaid on Original

! Testbench Designed to Match HW Specification− Convert Floating-Point to 5:6:5 RGB− Requires Matrix to Sample Conversion Blocks


Testbench – Top LevelTestbench – Top Level


Testbench – 5:6:5 SamplerTestbench – 5:6:5 Sampler! AVI Reader Outputs RGB Float Matrices for Image

− Specification Requires 5:6:5 Sample-Based Data− Convert Format Using Simulink Blocks


Custom Edge Detection Test CoreCustom Edge Detection Test Core! Place Wrapper Around Block From VIP Blockset to

Match HW Specification & Testbench


Testbench – OutputTestbench – Output


DSP Builder ImplementationDSP Builder Implementation! Substitute Simulink Test Core With DSP Builder Blockset


Building the Edge DetectorBuilding the Edge Detector! Consider Vertical Edge Detection

− For Each Pixel in the Image Apply the Following 3x3 Mask on the Intensity I of Adjacent Pixels

10-1

10-1

10-1


! Have to Identify 3x3 Pixel Blocks

! Use Delay & Memory Delay Blocks− Delay Uses Registers for Pixel Identification− Memory Delay Uses RAM for Line Storage

Vertical Edge DetectionVertical Edge Detection

Delay

1Z

MemoryDelay

1Z100

Target Pixel(x,y)

I(x-1,y+1)I(x-1,y+1)

I(x-1,y)I(x-1,y)

I(x-1,y-1)I(x-1,y-1)

I(x+1,y+1)I(x+1,y+1)

I(x+1,y)I(x+1,y)

I(x+1,y-1)I(x+1,y-1)


A(7:0)

Pipelined Adder

B(7:0)+ R(7:0)

Vertical Edge DetectionVertical Edge Detection! Use DSP Builder Blocks to Carry Out Arithmetic

on Pixel Intensities

− Use Pipelined Adder Blocks

⎟⎟⎟⎟⎟⎟

⎠

⎞

⎜⎜⎜⎜⎜⎜

⎝

⎛

−−+−+

+−

⎟⎟⎟⎟⎟⎟

⎠

⎞

⎜⎜⎜⎜⎜⎜

⎝

⎛

−++++

++

= −

)1,1(

),1(

)1,1(

)1,1(

),1(

)1,1(

),(

yxI

yxI

yxI

yxI

yxI

yxI

yxEdge


Adding DSP Builder BlocksAdding DSP Builder Blocks


Vertical Edge Detection BlockVertical Edge Detection Block

Add x+1 Inputs

Add x-1 Inputs

Subtract to Determine Edges

! Horizontal Edge Detection Works Using a Similar 3x3 Mask


DSP Builder Core Top LevelDSP Builder Core Top Level


! DSP Builder Contains Avalon� Port Library− Provides All Signals Required for Avalon Slave

! Other DSP Builder Blocks Used− FIFO Blocks for DMA Performance− Comparison & AND Gates for Control

System Integration – SOPC BuilderSystem Integration – SOPC Builder

Write Data7:0

Datain

Address

Address

d(7:0)

rreq

wreq

q(7:0)full

emptyUsdw(5:0)

FIFO

FIFO

a

b

Comparator

== AND

LogicalBit Operator


Creating the Co-ProcessorCreating the Co-Processor! SOPC Slave Block Built Within DSP Builder

− Connected to Tested Edge Detection Unit


Building the SOPC SystemBuilding the SOPC System1. Generate SOPC Component Inside DSP Builder

−Using Signal Compiler Block

2. Add Edge Detection Co-Processor Component With SOPC Builder System Editor

3. Add Additional SOPC Components−Nios II Processor−Camera Capture− VGA Device Controller−Memory Controller


SOPC Builder System EditorSOPC Builder System Editor

Nios IIEdge

Detector+ DMAs

VGAOut

Camera


Nios II Processor Control CodeNios II Processor Control Code! Simple C Language Program

− Rotate Frame Buffers"1 Buffer for Camera to Write To"1 Buffer for Edge Detector Block to Work On"1 Buffer for Display

− Setup DMA Transfers to Processing Module− Push Button Controls

" Toggle On/Off, Video Display


Program Hardware: Stratix II DSP Board + Camera Module Program Hardware: Stratix II DSP Board + Camera Module VGA OutputConnector

Peripheral ExpansionDaughter Card Interface

VGA ResolutionCMOS-Sensor

Camera-on-a-Chip Module

DemoDemo

Math Works

Documents

video simulink

altera corporationagendaagenda

design of realdesign

componentbased modeling

dynamic embedded systems9

matlab integration

featured product

endproduct performance