1 Video Processing Lecture on the image part (8+9) Automatic Perception 12+15 Volker Krüger Aalborg Media Lab Aalborg University Copenhagen [email protected].

1

Video Processing Lecture on the image part (8+9)

Automatic Perception 12+15

Volker KrügerAalborg Media Lab

Aalborg University [email protected]

2

Agenda

• Segmentation in video (finding the object(s))– MED3 applications

• Small and gentle intro to statistics

Probably next time:

• Tracking: follow the object(s) over time

• What to remember

3

Segmentation in Video

4

Videos

• Videos are Image Sequences over Time

y

x

t

•An image is a function over x and y:•A video is a function of images over time t:

• At each time step t we have an image • Framerate = the number of images per second, e.g., 25 Images/s

),(),,( yxftyxf t

),( yxf

),(),,( yxftyxf t

5

Segmentation in Video• General Segmentation is application-dependent!

Knowledge base!!• One application: Finding the object(s)

– MED3 applications– Preprocessing, Segmentation

• Tracking = follow the object(s) over time:– being able at each time t to give, e.g, position, color, orientation, etc.– Representation, Description

Knowledge base

Problemdomain Image

acquisition

Preprocessing

SegmentationRepresentationand description

Recognitionand Interpretation

Result

6

SegmentationLots of applications!! One is:• Separation of Foreground (object) and Background

(everything else = noise)• Result could be a

– Binary image, containing foreground only– Probability image, containing the likelihood of each pixel being

foreground

• Useful for further processing, such as using silhouettes, etc.• Approaches

– Motion-based– Color-based– Some approaches can learn!

(demos!)

7

Foreground-Background Segmentation using Motion and

Color

8

Foreground-Background Segmentation using Motion and Color

• Motion-based– Model-free

– No learning

– Image differencing

• Color-based– Background subtraction

• Background used as a model

• No learning

– Advanced background subtraction• Background is learned

– Very advanced background subtraction• Background is learned

9

Image Differencing

10

Image Differencing

• The motion in an image can be found by subtracting the current image from the previous image

• Algorithm1. Save image in last frame

2. Capture current camera image

3. Subtract image (= difference = motion)

4. Threshold

5. Delete noise

11

3. Subtract Image• Compute pixel-wise

• Subtract previous image from input image:

• Usually the absolute distance is applied

),(),(),( yxByxIyxF

),( yxF

1. Save image in last frame2. Capture camera image3. Subtract image4. Threshold5. Delete noise

12

4. Threshold

• Decide, when a pixel is supposed to be considered as a background pixel, or when it is to be considered as a foreground pixel:

• Pixel is foreground pixel, if

• Pixel is background pixel, if

• Problem: What TH?!?

),( yx

THyxF ),(),( yx

THyxF ),(

1. Save image in last frame2. Capture camera image3. Subtract image4. Threshold5. Delete noise

13

5. Deleting Noise

• Singular pixels are likely to appear: – Pixel-noise!!

• Apply Median filter:– Depending on filter size, bigger spots can be erased

• Alternative: Morphologic

1. Save image in last frame2. Capture camera image3. Subtract image4. Threshold5. Delete noise

(show: patch: diff)

14

Background Subtraction

15

Background Subtraction

• Foreground is moving, background is stable

• Algorithm1. Capture image containing background2. Capture camera image3. Subtract image (= difference = motion)4. Threshold5. Delete noise

(show: patch: bg_1)

16

Advanced Background Subtraction

17

Advanced Background Subtraction• What if we have small motion in the

background?– Bushed, leaves, etc. and noise in the camera/lighting– (show histo patch)

• Learn(!) the background

• Capture N images and calculate the average background image (no object present)

1. Calculate average background image 2. Capture camera image3. Subtract image (= motion)4. Threshold5. Delete noise

18

Very Advanced Background Subtraction

19

Very Advanced Background Subtraction

• Use Neighborhood relation!!– Compare pixel with its neighbors!!– Weight them!!

• Learn the background and its variations!!– E.g. Gaussian models (mean,var) for each pixel!!!– E.g. a Histogram for each Pixel Both will be revisited, later!!– The more images you train on the better!!– Idea:

• Some pixel may vary more than other pixels– Algorithm:

• Consider each pixel (x,y) in the input image and check, how much it varies with respect to the mean and variance of the learned Gaussian models?

1. Calculate mean and variance for each pixel 2. Capture camera image3. Subtract image (= motion)4. Weight the distances (new)5. Threshold according to variance6. Delete noise

20

Weight the Distances, Correlation between Pixel values

• If one pixel is considered to be a foreground pixel– Its neighbor is also likely to be a foreground pixel– If its neighbor is not considered to be a foreground

pixel, one of the two might be wrong– Neighboring pixels are highly correlated (similar)

1. Calculate mean and variance for each pixel 2. Capture camera image3. Subtract image (= motion)4. Weight the distances (new)5. Threshold according to variance6. Delete noise

21

Weight the Distances

• What does a pixel say about a foreground pixel, that is further away?– Pixels with an increasing distance to each other

are saying less about each other– The correlation between pixels degreases with

distance1. Calculate mean and variance for each pixel 2. Capture camera image3. Subtract image (= motion)4. Weight the distances (new)5. Threshold according to variance6. Delete noise

22

Weight the Distances• Use a Gaussian for Weighting• To test a pixel I(x,y)

– Center a Gaussian on this pixel and weight the neighboring pixels accordingly => Convolution!

• 1D example:

60 48 2 3 1 222 100Signal:

0.25 0.5 0.25

39.5 13.75 2.25 56.75 136.3

Gaussian filter:

Output:

23

Weight the Distances

• Likelihood image or Probability image, containing the likelihood for each pixel of being a background/foreground pixel

1. Calculate mean and variance for each pixel 2. Capture camera image3. Subtract image (= motion)4. Weight the distances (new)5. Threshold according to variance6. Delete noise

24

A little detour to Statistics

25

Statistics

• Mean– Center of gravity of the object

• Variance– The Variance measures the variations of the object pixel

positions around the center of gravity

N

iimean x

Nx

1

1

N

iimean y

Ny

1

1

N

imeanivar xx

Nx

1

2)(1

N

imeanivar yy

Ny

1

2)(1

xvar is big. yvar is small

N is the numberof object pixels

26

Statistics

• Standard deviation: sigma ()• Normal distribution = Gaussian distribution

varsigma xx

99.99 %

99.73 %

95.44 %

68.26 %

SamplesRange234

(bb)

27

Statistics

• How to use it– ”Automatic” thresholding based on statistics

• Example: the color of the hand– Training =>mean color

– Algorithm: hand pixel if: THmin < pixel < Thmax

– How do we define THmin and Thmax ?

– Use statistics: THmin = mean-2 and THmax = mean+2

28

Threshold According to Variance• Threshold can be chosen depending on the variance

– A local threshold

• Standard Deviation• For example:

– If Thmin < dist < Thmax => object pixel– Thmin = mean – – Thmax = mean +

),var(),( yxyx

99.99 %

99.73 %

95.44 %

68.26 %

SamplesRange234

1. Calculate mean and variance for each pixel 2. Capture camera image3. Subtract image (= motion)4. Weight the distances (new)5. Threshold according to variance6. Delete noise

29

Segmentation using Color

30

Segmentation using Color

• Assumption: the object has a different color than the background– As seen in Chroma keying

• Two approaches– Color histogram– Gaussian distributions

31

Segmentation using Color Histogram

32

Segmentation using Color Histogram

• Given an object, defined by its color Histogram• Algorithm:

– Segment each pixel in the input image according to the histogram

• The result: binary or probability image. – White pixels: pixel in input image had color as defined

by the histogram

– Black pixel: pixel in input image did not have color defined by the histogram

33

Learning the Color Histogram

• Recall a grayvalue histogram• The color histogram summarizes the

color distribution in an image (region)• There is a histogram bin for each

possible color!• The columns of the histogram are high

for those colors appearing often in the image (region) and low for those appearing seldom in the image.

• The more images you train on the better!!

bins

34

Color Segmentation with Histograms

• Algorithm:– For each pixel in the input I(x,y)

• Go to the bin having the same color as the pixel H( I(x,y) )

• Assign the probability value at the bin to the output image: O(x,y)

– O(x,y) = H( I(x,y) )

• Using the above leads to a probability image

• Run a threshold on the output image to get a binary image

35

Segmentation using Gaussian Distributions

36

Segmentation with Gaussian Distribution

• Given an object, calculate the mean and standard deviation of its color– Represent each color component (e.g., R,G,B) by a Gaussian

model– That is, a Gaussian distribution: (mean,sigma)

• The same principle as in “very advanced background subtraction” !!

• Algorithm: – Given an input image, segment each pixel by comparing it to the

Gaussian models of the object color• For example:

– If Thmin < dist < Thmax => object pixel– Thmin = mean – – Thmax = mean +

37

Comparing the Methods• When to use a color histogram and when

to use Gaussian models?• Plot the training data and look at it!• Gaussian models assume data is

distributed equally within a rectangle• Histograms do not assume anything• Gaussian models require less training• Histograms require a decision regarding

the resolution of the bins and the threshold.

• Thresholds in Gaussian models have a physical interpretation

r

g

r

g

38

Tracking

39

Tracking

• Follow the object(s) over time– Finding the trajectory (curve connecting the

positions over time)

• Simple tracking:

• Advanced tracking: – Cluttered background and multiple objects

40

Tracking

• For MED3 projects– Use tracking to predict where the object will be

in the next image – This allow us to specify the ROI (region of

interest)– Focus the algorithm

• Save computational resources

• Less likely that we will find an incorrect object

(show: patch: color_track)

41

Prediction

• Given the position of the object in previous images, where do we think (predict) the object will be in the current image?

• We need a motion model• The size of the search region depends on:

– The uncertain of the prediction– The framerate– How fast can the object max. move?

?

Search region

42

Motion Model• Predicted position at time t:

• Brownian Motion: According to a Gaussian model

• 0’th order:

• 1’th order:– Similar for y

• 2’th order – Similar for y

• Many other types exist: look at your application!

),( tt yx

),(),( 11 tttt yxyx

1 at time in velocity :1

11

t-xx

xtxx

t

ttt

1 at time in on accelerati :2

1

1

112

1

t-xx

xtxtxx

t

tttt

43

What to remember

• Statistics: mean and variance• Motion segmentation

– Image differencing (two images)– Background subtraction (one bg. image)– Advanced background subtraction (many bg. image)– Very advanced background subtraction (learn each pixel)

• Color segmentation– Using a histogram – Gaussian models

• Tracking– Prediction– Motion model