Video stabilization

Presented by:

Soheila Sheikhbahaei

1392

Kharazmi University of Tehran

VIDEO STABILIZATION

OUTLINE

• What is video stabilization?

• Where is it useful?

• How does it work?

• What are the problems

DEFINITION

• Most amateur videos are captured using hand-held cameras. They are often very shaky

and difficult to watch.

• Video stabilization techniques have been developed to smooth shaky camera motion in

videos before viewing.

• Stabilization is the process of estimating and compensating for the background image

motion occurring due to the ego-motion of the camera.

ELIMINATING JITTER

• Although this jitter can be eliminated by anchoring the binoculars on a tripod, this is not

always feasible.

EXAMPLE

Wang, Y.S., et al., Spatially and Temporally Optimized Video Stabilization. IEEE transactions on visualization and computer graphics, 2013.

http://people.cs.nctu.edu.tw/~yushuen/VideoStabilization/

PARALLAX

• Parallax is a displacement or difference in the apparent position of an object viewed along

two different lines of sight

BIOLOGICAL MOTIVATION: INSECT NAVIGATION

• Insects have relatively small nervous system with very few neurons when compared to the

human brain, they are still capable of complex tasks, such as safe landing, obstacle

avoidance.

• Behavioral research with insects suggest that insects primarily use visual information.

• Insects have immobile eyes with fixed focal length. Moreover, they do not possess

stereoscopic vision. Insect eyes possess inferior spatial acuity but their eyes sample the

world at a significantly higher rate than human eyes do.

• The study can serve as a pure motivational tool indicating that such complex tasks, such

as stabilization, can be performed real-time, with the accuracy desired. Second, this study

can lead us into the paradigm of “active vision” or “purposive vision”.

• In fact, several researchers have used such biologically inspired mechanisms for flight

control and obstacle avoidance.

Al Bovik. The Essential Guide to Video Processing

BIOLOGICAL MOTIVATION: INSECT NAVIGATION

• Bees that fly through holes tend to fly through the center of these holes. Bees, like most

other insects, cannot measure distances from surfaces by using stereoscopic vision.

• Recent experiments have indicated that bees balance the image motion on the lateral

portion of their two eyes as they fly through openings.

• Bees were trained to fly in narrow tunnels with certain patterns on the side walls of the

tunnels. It was shown in that bees tended to fly at the center of this tunnel when the

patterns on the side walls were stationary.

• If one of these patterned side walls was moved in the direction of the bee’s flight, thereby

reducing the image motion experienced by the bee on that side, then the bees moved

closer to that side wall. Similarly, when one of the patterned side walls was moved in the

direction opposite to the direction of the bee’s flight, the bee moved away from the moving

wall.

BIOLOGICAL MOTIVATION: INSECT NAVIGATION

• Collision avoidance is another task that is visually driven in most insects. When an insect

approaches an obstacle, its image expands on it’s eyes. Insects are sensitive to this

image expansion and turn away from the direction in which the image expansion occurs,

thereby avoiding collision with obstacles.

DIFFERENT ALGORITHMS

• Depending on the type of scenario and the type of motion involved, we have different

algorithms to achieve stabilization.

• Presence of a dominant plane in the scene

• Derotation of the image sequence

• Mosaic construction

• Presence of moving objects

CLASSIFICATION OF TECHNIQUES

• Techniques are classified as two categories:

• Feature-based methods: extract and match discrete features between frames and

trajectories of these features are fit to a global motion model.

• Flow-based methods: optical flow of the image sequence is an intermediate quantity

that is used in determining the global motion.

PHASES OF STABILIZATION

• In video stabilization, we need to analyze the image motion and obtain models for the

global motion in image sequences.

• Generally the process of stabilization have to go through two phases:

• motion estimation

• motion smoothing

CAMERA MODEL

• The imaging geometry of a perspective camera:

EFFECT OF CAMERA MOTION

• The effect of camera motion can be computed using projective geometry:

EFFECT OF CAMERA MOTION

• Other popular global deformations mapping the projection of a point between two frames

are the similarity:

and affine transformations:

IMAGE FEATURES

• The basic goal in feature-based motion estimation is to use features to find maps that

relate the images taken from different view-points.

• These maps are then used to estimate the image motion by computing the parameters of

a motion model.

• Consider the case of pure rotation:

• Though various lengths, ratios, and angles formed on the images are all different, the

cross ratio remains the same. Given four collinear points A, B, C, and D on an image,

R. Hartley and A. Zisserman. Multiple View Geometry in computer vision. Cambridge University Press, Cambridge, UK, 2000.

IMAGE FEATURES

• this intuition leads to a map relating the two images.

• Given four corresponding points in general position in the two images, we can map any

point from one image to the other.

• Now, any point F on ABE will map to point F´ such that the cross ratio is preserved.

• This way one can map each point on one image to the other image. Such a map is called

homography.

IMAGE FEATURES

• In the case of planar scene:

• x1, a point on first image plane, xp, the corresponding point on the real plane, x2, the

corresponding point on the second image plane.

• Thus, homography H =H1H2 maps points from one image plane to the other.

IMAGE FEATURES

• On the other hand, when there are depth variations in the scene, such a homography

doesn’t exist between images formed by camera translation.

• In the case of depth variations, we can use structure from motion (SFM) approaches to

estimate the motion of the camera.

FEATURE BASED ALGORITHMS

• A number of features are extracted in each image and feature matching algorithms are

used to establish correspondence between the images.

• The motion parameters are found by first identifying the set of feature matches.

FEATURE TRAJECTORY SMOOTHING

• Let the ith trajectory be where pi and m and n are the start and the end frames of Pi,

respectively.

• Our goal is to solve an optimization problem that can minimize the acceleration of Pi in

each frame while constraining the offsets of neighboring trajectories to be consistent

within the input video.

Wang, Y.S., et al., Spatially and Temporally Optimized Video Stabilization. IEEE transactions on visualization and computer graphics, 2013.

BEZIER CURVES

• Bezier curves are used in computer graphics to produce curves which appear reasonably

smooth at all scales (as opposed to polygonal lines, which will not scale nicely ) in which

the interpolating polynomials depend on certain control points.

FEATURE TRAJECTORY SMOOTHING

• each smoothed trajectory is represented using a Be’zier curve and reduce the unknown

variables from all feature positions to curve control points. This reduced model also

achieves strong stabilization because the smoothed feature positions are interpolated

from the control points. We show the details of our technique in the following subsections.

DELAUNAY TRIANGULATION

• In mathematics and computational geometry, a Delaunay triangulation for a set P of

points in a plane is a triangulation DT(P) such that no point in P is inside

the circumcircle of any triangle in DT(P).

• In geometry, the circumscribed circle or circumcircle of a polygon is a circle which

passes through all the vertices of the polygon..

SPATIAL RIGIDITY PRESERVATION

• spatial rigidity is retained when stabilizing a video in order to preserve neighboring feature

trajectories to have similar treatments.

• Specifically, we compute the neighbor relations between features in each frame using the

Delaunay triangulation and enforce each triangle to undergo a rigid transformation. That

is, triangles are allowed to move and rotate but their sizes and shapes should be retained.

• This constraint works well in most videos.

OBJECTIVE FUNCTION

• we search for the control points of Bezier curves that can minimize the objective function.

RESULTS

LIMITATIONS

• Although the algorithm is robust to all challenging examples, the stabilization is not

effective if there are no background features in some frames.

MOSAICING

• Mosaicing is the process of compositing or piecing together successive frames of the

stabilized image sequence so as to virtually increase the field of view of the camera.

• Mosaics are commonly defined only for scenes viewed by a pan/tilt camera, for which the

images can be related by a projective transformation.

REFERENCES

• Al Bovik. The Essential Guide to Video Processing

• Wang, Y.S., et al., Spatially and Temporally Optimized Video Stabilization. IEEE

transactions on visualization and computer graphics, 2013.

• http://www.ics.uci.edu/~eppstein/gina/delaunay.html

• http://en.wikipedia.org/wiki/Delaunay_triangulation

• http://www.math.ubc.ca/~cass/gfx/bezier.html

• http://en.wikipedia.org/wiki/B%C3%A9zier_curve

• http://people.cs.nctu.edu.tw/~yushuen/VideoStabilization/