Aerial Video Surveillance and Exploitation Roland Miezianko CIS 750 - Video Processing and Mining Prof. Latecki.

Aerial Video Surveillance and Aerial Video Surveillance and ExploitationExploitation

Roland MieziankoRoland Miezianko

CIS 750 - Video Processing and MiningCIS 750 - Video Processing and Mining

Prof. LateckiProf. Latecki

AgendaAgenda

• Aerial Surveillance ComparisonsAerial Surveillance Comparisons• Technical Challenges and the MissionTechnical Challenges and the Mission• Framework Ideas for Video SurveillanceFramework Ideas for Video Surveillance

– Alignment and Change DetectionAlignment and Change Detection– MosaicingMosaicing– Tracking Moving ObjectsTracking Moving Objects– Geo-locationGeo-location– Enhanced VisualizationEnhanced Visualization

• Image MosaicsImage Mosaics

Types of Aerial Types of Aerial SurveillanceSurveillance

• Using film and framing camerasUsing film and framing cameras– Hi-resolution still imagesHi-resolution still images– Examined by human or machineExamined by human or machine

• Video captures dynamic eventsVideo captures dynamic events– Used to detect and geo-locate moving Used to detect and geo-locate moving

objects in real-timeobjects in real-time– Follow detected motionFollow detected motion– Constantly monitor a siteConstantly monitor a site

Technical Challenges, 1Technical Challenges, 1

• Video cameras have lower Video cameras have lower resolution than framing camerasresolution than framing cameras– Video uses telephoto lens to get high Video uses telephoto lens to get high

resolution to identify objectsresolution to identify objects– Telephoto lens - Narrow field of viewTelephoto lens - Narrow field of view– Provides “soda straw” view of the Provides “soda straw” view of the

scene [2]scene [2]

Technical Challenges, 2Technical Challenges, 2

• Camera must scan the region of Camera must scan the region of interest to get the “full-picture”interest to get the “full-picture”

• Objects of interest move in and out Objects of interest move in and out of the field of viewof the field of view

• Difficulty in perceiving object Difficulty in perceiving object relative locationsrelative locations

Technical Challenges, 3Technical Challenges, 3

• Challenge in manually tracking an Challenge in manually tracking an object due to camera’s small field object due to camera’s small field of viewof view

• Video contains much more data Video contains much more data then film frames; Storage is then film frames; Storage is expensiveexpensive

The MissionThe Mission

• The new aerial surveillance The new aerial surveillance systems must provide a framework systems must provide a framework for spatio-temporal aerial video for spatio-temporal aerial video analysisanalysis

Video AnalysisVideo AnalysisFramework, 1Framework, 1

• Frame-to-Frame alignment and Frame-to-Frame alignment and decomposition of video frames into decomposition of video frames into motion layersmotion layers

• Mosaicing static background layers Mosaicing static background layers to form panoramas as compact to form panoramas as compact representations of the static scenerepresentations of the static scene

Video AnalysisVideo AnalysisFramework, 2Framework, 2

• Detecting and tracking Detecting and tracking independently moving objects in independently moving objects in the presents of background clutterthe presents of background clutter

• Geo-locating the video and tracked Geo-locating the video and tracked objects by registering it to objects by registering it to controlled reference imagery; controlled reference imagery; digital terrain maps and modelsdigital terrain maps and models

Video AnalysisVideo AnalysisFramework, 3Framework, 3

• Enhanced visualization of the video Enhanced visualization of the video by re-projecting and merging it by re-projecting and merging it with reference imagery, terrain, with reference imagery, terrain, and maps to provide a larger and maps to provide a larger contextcontext

Alignment and Change Alignment and Change Detection, 1Detection, 1

• Displacement of pixels between Displacement of pixels between video frames may occur due to the video frames may occur due to the following:following:– Motion of the video sensorMotion of the video sensor– Independent motion of objects in the Independent motion of objects in the

field of viewfield of view– Motion of the source of illuminationMotion of the source of illumination

Alignment and Change Alignment and Change Detection, 2Detection, 2

• Global motion estimationGlobal motion estimation– Displacement of pixels due to the Displacement of pixels due to the

motion of the sensor is computedmotion of the sensor is computed• Alignment of Video FramesAlignment of Video Frames

– Pyramid-ProcessingPyramid-Processing– Lock into the motion of background Lock into the motion of background

scenescene– Warp images into common coordinate Warp images into common coordinate

frameframe

Alignment and Change Alignment and Change Detection, 3Detection, 3

• Moving objects are detected by Moving objects are detected by aligning video frames and aligning video frames and detecting pixels with poor detecting pixels with poor correlation across the temporal correlation across the temporal domaindomain

MosAICING, 1MosAICING, 1

• Images are accumulated into the Images are accumulated into the mosaic as the camera pansmosaic as the camera pans

• Construction of a 2D mosaic Construction of a 2D mosaic requires computation of alignment requires computation of alignment parameters that relate all of the parameters that relate all of the images in the collection to a images in the collection to a common world coordinate systemcommon world coordinate system

MosAICING, 2MosAICING, 2

• Transformation parameters are Transformation parameters are used to warp the images into the used to warp the images into the mosaic coordinate systemmosaic coordinate system

• Warped images are then combined Warped images are then combined to form a mosaicto form a mosaic

• To avoid seams, warped frames are To avoid seams, warped frames are merged in the Laplacian pyramid merged in the Laplacian pyramid domaindomain

MosAICING, exampleMosAICING, example

MosAICING, exampleMosAICING, example

Tracking Moving Objects, Tracking Moving Objects, 11

• Scene analysis includes operations Scene analysis includes operations that interpret the source video in that interpret the source video in terms of objects and activities in terms of objects and activities in the scenethe scene

• Moving objects are detected and Moving objects are detected and tracked over the cluttered scenetracked over the cluttered scene

Tracking Moving Objects, Tracking Moving Objects, 22

• State of each moving object is State of each moving object is represented by its:represented by its:– MotionMotion– AppearanceAppearance– ShapeShape

• The state is updated at each instant of The state is updated at each instant of time using Expectation-Maximization time using Expectation-Maximization (EM) algorithm(EM) algorithm

Tracking Moving Objects, Tracking Moving Objects, exampleexample

Geo-locationGeo-location

• Video Surveillance system must Video Surveillance system must also determine the geodetic also determine the geodetic coordinates of objects within the coordinates of objects within the camera’s field of viewcamera’s field of view

• More precise geo-locations can be More precise geo-locations can be estimated by aligning video frames estimated by aligning video frames to calibrated reference imagesto calibrated reference images

Enhanced VisualizationEnhanced Visualization

• Challenging aspect of aerial video Challenging aspect of aerial video surveillance is formatting video surveillance is formatting video imagery for effective presentation imagery for effective presentation to an operatorto an operator

• The “soda straw” view makes The “soda straw” view makes direct observation tedious and direct observation tedious and disorientingdisorienting

Mosaic-Based DisplayMosaic-Based Display

• Display de-couples the observer’s Display de-couples the observer’s display from the cameradisplay from the camera

• Operator may scroll or zoom to Operator may scroll or zoom to examine one region of the mosaic examine one region of the mosaic even as the camera is updating even as the camera is updating another region of the mosaicanother region of the mosaic

Elements of Mosaic Elements of Mosaic DisplayDisplay

ED

warp mergecamera

Estimatedisplacement

Pyramid merge

Operator’s displayImage accumulating

memory

Update window

Camera InputCamera Input

Mosaic Generation, 1Mosaic Generation, 1

Mosaic Generation, 2Mosaic Generation, 2

Psuedo codes of main Psuedo codes of main algorithm [5]algorithm [5]

read(base_image);

read(unregistered_image);

base_image=expand(base_image);

confirm three pairs of matched points between

base_image and unregistered_image;

calculate initial matrix M;

Apply Levenberg-Marquardt minimization to update M;

M = inverse(M);

Resample and apply blending function to render the

mosaics;

Homogeneous CoordinatesHomogeneous Coordinates

Using homogeneous coordinates, we can describe the class of 2D planar projective transformations using matrix multiplication [4]:

Rigid TransformationRigid Transformation

The same hierarchy of transformations exists in 3D.

Rigid (Euclidean) transformation where R is a 3 × 3 orthonormal rotation matrix and t is a 3D translation vector.

Viewing MatrixViewing Matrix

The 3×4 viewing matrix:

projects 3D points through the origin onto a 2D projection plane a distance f along the z axis.

Combined EquationsCombined Equations

The combined equations projecting a 3D world coordinate p = (x, y, z, w) onto a 2D screen location u = (x', y', w') can thus be written as

where P is a 3 × 4 camera matrix. This equation is valid even if the camera calibration parameters and/or the camera orientation are unknown.

Local Image Registration, Local Image Registration, 11

• How do we compute the How do we compute the transformations relating the transformations relating the various scene pieces so that we various scene pieces so that we can paste them together?can paste them together?– We could manually identify four or We could manually identify four or

more corresponding points between more corresponding points between the two viewsthe two views

– Manual approaches are too tedious to Manual approaches are too tedious to be usefulbe useful

Local Image Registration, Local Image Registration, 22

• This has the advantages of not This has the advantages of not requiring any easily identifiable requiring any easily identifiable feature points and of being feature points and of being statistically optimal, that is, giving statistically optimal, that is, giving the maximum likelihood estimate the maximum likelihood estimate once we are in the vicinity of the once we are in the vicinity of the true solution.true solution.

Rewrite our 2D transformations

Minimizes Intensity ErrorsMinimizes Intensity Errors

Technique minimizes the sum of the squared intensity errors.

Over all corresponding pairs of pixels i inside both images I(x, y) and I’(x’, y’).

Pixels that are mapped outside image boundaries do not contribute.

MinimizationMinimization

To perform the minimization, we use the Levenberg-Marquardt iterative nonlinear minimization algorithm. This algorithm requires computation of the partial derivatives of ei with respect to the unknown motion parameters {m 0 ... m 7 }.

Complete Registration Complete Registration Algorithm: Step 1 Algorithm: Step 1 [4][4]

Complete Registration Complete Registration Algorithm: Steps 2-4Algorithm: Steps 2-4

Bryce Canyon MosaicBryce Canyon Mosaic

Wall Frame, exampleWall Frame, example

ConclusionConclusion

• The techniques presented here The techniques presented here automatically register video frames into automatically register video frames into 2D and partial 3D scene models.2D and partial 3D scene models.

• Video mosaics and related techniques Video mosaics and related techniques will enable an even more exciting range will enable an even more exciting range of interactive computer graphics, of interactive computer graphics, telepresence, and virtual reality telepresence, and virtual reality applications.applications.

ReferencesReferences

[1] Automatic Panoramic Image Construction[1] Automatic Panoramic Image Construction

Yap-Peng Tan, Sanjeev R. Kulkarni and Peter J. RamadgeYap-Peng Tan, Sanjeev R. Kulkarni and Peter J. Ramadge

Princeton University, Department of Electrical EngineeringPrinceton University, Department of Electrical Engineering

[2] Chapter 2 by Rakesh Kumar[2] Chapter 2 by Rakesh Kumar

Aerial Video Survelliance and ExploitationAerial Video Survelliance and Exploitation

[3] A Multiresolution Spline With Application to Image Mosaics[3] A Multiresolution Spline With Application to Image Mosaics

PETER J. BURT and PETER J. BURT and EDWARD H. ADELSON H. ADELSON

RCA David Sarnoff Research CenterRCA David Sarnoff Research Center

ReferencesReferences

[5] Jingbin Wang, Boston University[5] Jingbin Wang, Boston University

CS580:Advanced Graphics Project 1: Image MosaicsCS580:Advanced Graphics Project 1: Image Mosaics

[4] Richard Szeliski. Video mosaics for virtual environments. IEEE Computer[4] Richard Szeliski. Video mosaics for virtual environments. IEEE Computer

Graphics and Applications, 16(2):22--30, March 1996Graphics and Applications, 16(2):22--30, March 1996