Computer Vision System Toolbox

Computer Vision System Toolbox 4.0Design and simulate computer vision and video processing systems

Computer Vision System Toolbox™ provides algorithms and tools for the design and simulation of computervision and video processing systems. These capabilities are provided as MATLAB® functions, MATLAB Systemobjects, and Simulink® blocks. The system toolbox includes algorithms for feature extraction, motion detection,object tracking, stereo vision, video processing, and video analysis. Tools include video file I/O, video display,drawing graphics, and compositing. For rapid prototyping and embedded system design, the system toolboxsupports fixed-point arithmetic and C-code generation.

Key Features▪ Algorithms available as MATLAB functions, MATLAB System objects, and Simulink blocks

▪ Feature detection, extraction, and matching for applications such as automatic image registration, videomosaicking, and video stabilization

▪ Motion estimation, including block matching, optical flow, and template matching

▪ Stereo vision, including fundamental matrix estimation and stereo image rectification

▪ Video processing, including chroma resampling and deinterlacing

▪ Image processing, including edge detection, morphology, and filtering

▪ Video file I/O, video display, graphic overlays, and compositing

▪ Algorithm support for floating-point, integer, and fixed-point data types

▪ Automatic C-code generation (with MATLAB Coder™ or Simulink Coder™)

Feature-Based Registration

Computer Vision System Toolbox supports automatic image registration by providing algorithms that usefeatures to estimate the geometric relationships between images or video frames. Typical uses include videomosaicking, video stabilization, image fusion, and stereo vision.

Feature Detection, Extraction, and Matching

Feature detection, extraction, and matching are the first steps in the feature-based registration workflow. Featuresare a set of interest points that are likely to be common across a pair of related images. Feature extraction enablesyou to derive a set of feature vectors, also called descriptors, from pixels surrounding each interest point. Thesystem toolbox offers capabilities to detect features that include corners, edges, and lines; extract descriptors fromthe features; and find the most likely paired matches of descriptors.

Estimating Geometric Relationships

With a set of matched interest points, you can infer the geometric relationship between two images or videoframes. The feature detection, extraction, and matching workflow produces many interest points and typicallyincludes outliers. To estimate the geometric relationship with this set of interest points, you can exclude outlierswith statistically robust methods such as RANSAC and least median of squares. With this workflow, the systemtoolbox can produce a projective or affine transformation that describes the geometric relationship between a pairof images or video frames.

1

Feature-based registration, as used for video stabilization. Interest points are detected in two sequential video frames(top) using corner features; the putative matches are determined with numerous outliers (bottom left); and outliers areremoved using the RANSAC method (bottom right).

Motion Estimation and Tracking

The system toolbox provides a variety of motion estimation algorithms such as optical flow, block matching,template matching, and background estimation using Gaussian mixture models (GMM). Evaluation metrics forfinding the best block match include MSE, MAD, MaxAD, SAD, and SSD. These algorithms create motionvectors, which relate to the whole image, blocks, arbitrary patches, or individual pixels, depending upon thealgorithm.

Detecting moving objects using a stationary camera. In this series of video frames, optical flow is calculated anddetected motion is shown by overlaying the flow field on top of each frame.

Video tracking is a common use for motion estimation algorithms. You can use calculated motion to identify amoving object or measure the movement of a detected object over consecutive video frames. The system toolboxalso provides Kalman filtering to predict the movement of an object in upcoming video frames.

2

Estimation of camera motion by template matching to calculate a motion vector (left). Using the motion vector, thevideo is stabilized for camera motion (right).

Stereo Vision

Stereo vision is the process of reconstructing a 3D scene from two or more views of the scene. Using the systemtoolbox, you can:

▪ Perform uncalibrated stereo image rectification on a pair of stereo images

▪ Match individual pixels along epipolar lines to compute the disparity map

▪ Calculate the distance from the camera for each point in the disparity map using knowledge about theintrinsic parameters of the camera

▪ Backproject the disparity map with the original stereo images to create a 3D reconstruction of the scene

3

Reconstructing a scene using a pair of stereo images. To visualize the disparity, the right channel is combined with theleft channel to create a composite (top left). A disparity map of the scene (top right) is then calculated by solving thepoint correspondence problem, and a 3D rendering of the scene (bottom) is reconstructed.

Stereo Image Rectification

Stereo image rectification transforms a pair of stereo images so that a corresponding point in one image can befound in the corresponding row in the other image. This process reduces the 2D stereo correspondence problemto a 1D problem, which simplifies the process of determining the depth of each point in the scene from thecamera. You can rectify a pair of stereo images with the system toolbox by determining a set of matched interestpoints, estimating the fundamental matrix, and then deriving two projective transformations.

4

Results from stereo image rectification. Non overlapping areas are show in red and cyan.

Video Processing, Visualization, and Graphics

Computer Vision System Toolbox provides algorithms and tools for video processing workflows. You can readand write from common video formats, perform common video processing algorithms such as deinterlacing andchroma-resampling, and display results with text and graphics burnt in to the video. Video processing inMATLAB uses System objects, which avoids excessive memory use by streaming data to and from video files.

Video deinterlacing in MATLAB.

Video I/O

Computer Vision System Toolbox can read and write multimedia files in a wide range of formats, such as AVI,MPEG, and WMV. You can stream video to and from MMS sources over the Internet or a local network. You can

5

acquire video directly from Web cameras, frame grabbers, DCAM-compatible cameras, and other imaging devicesusing Image Acquisition Toolbox™. Simulink users can also use the MATLAB workspace as a video source or sink.

Visualization

The system toolbox includes a video viewer with many features. You can:

▪ View video streams in-the-loop as the data is being processed

▪ View any video signal within your code or block diagram

▪ Use multiple video viewers at the same time

▪ Freeze the display and evaluate the current frame

▪ Display pixel information for a region in the frame

▪ Pan and zoom for closer inspection as the simulation is running

▪ Start, stop, pause, and step through Simulink simulations one frame at a time

Model with viewers for four videos: (from left) original, estimated background, foreground pixels, and results oftracking.

Graphics

Adding graphics to video helps with visualizing extracted information or debugging problems with a systemdesign. You can insert text in order to display the number of objects or keep track of other key information. Youcan insert graphics, such as markers, lines, and polygons to mark found features, delineate objects, or highlightother key features. Inserted text and graphics are incorporated into the data itself rather than as a separate layer.

6

You can also combine two video sources in a composite that can highlight objects or focus attention on a keyregion.

Images with text and graphics inserted. Adding these elements can help you visualize extracted information anddebug your design.

Stream Processing in MATLAB and Simulink

Computer Vision System Toolbox supports a stream processing architecture in both MATLAB and Simulink. In astream processing architecture, one or more video frames from a continuous stream are processed at a time. Thistype of processing is appropriate for analysis of large video files or systems with live video.

In MATLAB, stream processing is enabled by System objects, which use MATLAB objects to represent time-basedand data-driven algorithms, sources, and sinks. System objects implicitly manage many details of streamprocessing, such as data indexing, buffering, and management of algorithm state. You can mix System objectswith standard MATLAB functions and operators. Most System objects have a corresponding Simulink block withthe same capabilities.

Simulink handles stream processing implicitly by managing the flow of data through the blocks that make up aSimulink model. Simulink is an interactive graphical environment for modeling and simulating dynamic systemsthat uses hierarchical diagrams to represent a system model. It includes a library of general-purpose, predefinedblocks to represent algorithms, sources, sinks, and system hierarchy. Computer Vision System Toolbox provides alibrary of blocks specifically for the design of computer vision and video processing systems.

7

An abandoned object detection model (top). The three viewers (bottom) show steps in the process of detecting andtracking an abandoned object in a live video stream from a camera in a train station.

System Design and Implementation

With MATLAB and Simulink your workflow for rapid prototyping, verification, and implementation can beintegrated with algorithm development. You can:

▪ Convert floating-point algorithms into fixed-point representations to perform bit-true simulations

▪ Create system-level test benches to verify system behavior against requirements before implementinghardware and software

▪ Generate real-time C code from your MATLAB code or Simulink model and then download it onto asupported DSP board for real-time evaluation

Fixed-Point Modeling

Many real-time systems use hardware that requires fixed-point representation of your algorithm. ComputerVision System Toolbox supports fixed-point modeling in all relevant blocks and System objects with dialogs boxesand object properties that help you with configuration.

Support for fixed point in the system toolbox includes:

▪ Word sizes from 1 to 128 bits

▪ Arbitrary binary-point placement

▪ Overflow handling methods (wrap or saturation)

▪ Rounding methods, including ceiling, convergent, floor, nearest, round, simplest, and zero

8

The Fixed-Point Tool in Simulink Fixed Point™ facilitates the conversion of floating-point data types to fixedpoint. The tool tracks overflows and maxima and minima, helping you to configure fixed-point properties.

Code Generation Support

Once you have developed your algorithm or system model, you can automatically generate C code from it forverification, rapid prototyping, and implementation. Most System objects, functions, and blocks in ComputerVision System Toolbox can generate ANSI/ISO C code using MATLAB Coder, Simulink Coder, or EmbeddedCoder™. You can select optimizations for specific processor architectures and integrate legacy C code with thegenerated code to leverage existing intellectual property. You can generate C code for both floating-point andfixed-point data types.

Simulink model designed to create code for a specific hardware target. This model generates C code for a videostabilization system and embeds the algorithm into a digital signal processor (DSP).

Image Processing Primitives

Computer Vision System Toolbox includes image processing primitives for solving frequent system problems,such as interfering noise, low dynamic range, and out-of-focus optics. Unlike similar functions in ImageProcessing Toolbox™, these System objects and Simulink blocks support fixed-point data types and C-codegeneration. These primitives include:

▪ 2D spatial and frequency filtering

▪ Image pre- and post-processing algorithms

▪ Morphological operators

▪ Geometric transformations

▪ Color space conversions

9

Product Details, Demos, and System Requirementswww.mathworks.com/products/computer-vision

Trial Softwarewww.mathworks.com/trialrequest

Saleswww.mathworks.com/contactsales

Technical Supportwww.mathworks.com/support

Resources

Online User Communitywww.mathworks.com/matlabcentral

Training Serviceswww.mathworks.com/training

Third-Party Products and Serviceswww.mathworks.com/connections

Worldwide Contactswww.mathworks.com/contact

© 2011 The MathWorks, Inc. MATLAB and Simulink are registered trademarks of The MathWorks, Inc. See www.mathworks.com/trademarksfor a list of additional trademarks. Other product or brand names may be trademarks or registered trademarks of their respective holders. 10