Video Object Tracking and Replacement for Post TV Production

Video Object Tracking Video Object Tracking and Replacement for and Replacement for

Post TV ProductionPost TV Production

LYU0303 Final Year ProjectLYU0303 Final Year Project

Fall 2003Fall 2003

OutlineOutline

Project IntroductionProject Introduction Basic parts of the purposed systemBasic parts of the purposed system Working principles of individual partsWorking principles of individual parts Future WorkFuture Work Q&AQ&A

IntroductionIntroduction

Post-TV production software changes the Post-TV production software changes the content of the original video clips.content of the original video clips.

Extensively used in video-making Extensively used in video-making industries.industries.

Why changing the content of a video?Why changing the content of a video? Reducing video production costReducing video production cost Performing dangerous actionsPerforming dangerous actions Producing effects those are impossible in realityProducing effects those are impossible in reality

Difficulties to be overcomeDifficulties to be overcome Things in video can be treated individually Things in video can be treated individually

called “video objects”.called “video objects”. Computers cannot perform object detection Computers cannot perform object detection

directly because…directly because… Image is processed byte-by-byteImage is processed byte-by-byte Without pre-knowledge about the video objects to be detectedWithout pre-knowledge about the video objects to be detected Result is definite, no fuzzy logic.Result is definite, no fuzzy logic.

Though computers cannot perform object Though computers cannot perform object detection directly, it can be programmed to detection directly, it can be programmed to work indirectly.work indirectly.

Basic parts of the purposed systemBasic parts of the purposed system

Simple bitmap reader/writerSimple bitmap reader/writer RGB/HSV converterRGB/HSV converter Edge detectorEdge detector Edge equation finderEdge equation finder Equation processorEquation processor Translation detectorTranslation detector Texture mapperTexture mapper

RGB/HSV converterRGB/HSV converter

Human eyes are more sensitive to the Human eyes are more sensitive to the brightness rather than the true color brightness rather than the true color components of an object.components of an object.

More reasonable to convert the More reasonable to convert the representation of colors into HSV (Hue, representation of colors into HSV (Hue, Saturation and Value (brightness)) model.Saturation and Value (brightness)) model.

After processing, convert back to RGB and After processing, convert back to RGB and save to disk.save to disk.

RGB/HSV converterRGB/HSV converter

RGB to HSVRGB to HSV HSV to RGBHSV to RGB

Edge detectorEdge detector

Usually, a sharp change in hue, saturation or Usually, a sharp change in hue, saturation or brightness means that there exist a boundary brightness means that there exist a boundary line.line.

HSV: (0,0,0)

HSV: (0,255,255)

Edge equation finderEdge equation finder

Derives mathematical facts out of the edge Derives mathematical facts out of the edge points.points.

Works with voting algorithm of Hough Works with voting algorithm of Hough Transform.Transform.

Automatically adjusts tolerance value to Automatically adjusts tolerance value to minimize the effect of noise points.minimize the effect of noise points.

This helps when the edge is not completely This helps when the edge is not completely straight or blurred.straight or blurred.

Edge equation finderEdge equation finder

0

4590

(x1,y1)

Angle in degree Frequency

0 1

45 3

90 1

135

135 1

Desired linear equation in point-slope form:

)1(*45tan1 xxyy

Equation processorEquation processor

After finding out the equation, constraints After finding out the equation, constraints can be applied in order to remove can be applied in order to remove redundant equations, get shadows or redundant equations, get shadows or detect occultation.detect occultation.

Find out the corner points for translation Find out the corner points for translation detector and texture mapper.detector and texture mapper.

Translation detectorTranslation detector

A simple object motion tracker.A simple object motion tracker. Collects the data of the first key frame to Collects the data of the first key frame to

accelerate the process of the remaining accelerate the process of the remaining video frames.video frames.

Can be beneficial if the video segment is Can be beneficial if the video segment is long and the scene seldom changes.long and the scene seldom changes.

Texture MapperTexture Mapper

A graphics design process in which a 2-D A graphics design process in which a 2-D surface, called a surface, called a texture maptexture map, is "wrapped , is "wrapped around" a 3-D object. around" a 3-D object.

The 3-D object acquires a surface texture The 3-D object acquires a surface texture similar to the texture map.similar to the texture map.

Texture MapperTexture Mapper

Mapping

Original position of pixel

New position of pixel

Texture MapperTexture Mapper

Every polygon is assigned 2 sets of Every polygon is assigned 2 sets of coordinatescoordinates

Image coordinates Image coordinates (r, c)(r, c) : location of pixel : location of pixel in the imagein the image

Texture coordinates Texture coordinates (u, v)(u, v) : location in : location in texture image which contains color texture image which contains color information for image coordinatesinformation for image coordinates

Texture MapperTexture Mapper

Mapping functionsMapping functions map texture map texture coordinates to image coordinates or vice coordinates to image coordinates or vice versa.versa.

They are usually determined by image They are usually determined by image points whose texture coordinates are points whose texture coordinates are given explicitly.given explicitly.

Texture MapperTexture Mapper

(r1, c1)

(u1, v1)

(r2, c2)

(u2, v2)

(r4, c4)

(u4, v4)

(r3, c3)

(u3, v3)

(u1, v1) (u2, v2)

(u3, v3)(u4, v4)

Texture MapperTexture Mapper

Scan conversionScan conversion: the process of scanning : the process of scanning all the pixels and perform the necessary all the pixels and perform the necessary calculation.calculation.

Forward mappingForward mapping maps from the texture maps from the texture space to image spacespace to image space

Inverse mapping Inverse mapping maps from the image maps from the image space to texture spacespace to texture space

Scan conversion with forward Scan conversion with forward mappingmapping

Algorithm: Algorithm: for u = umin to umaxfor u = umin to umax for v = vmin to vmixfor v = vmin to vmix r = R(u,v)r = R(u,v) c = C(u,v)c = C(u,v) copy pixel at source (u,v) copy pixel at source (u,v) to destination (r,c)to destination (r,c)

Scan conversion with forward Scan conversion with forward mappingmapping

Advantage:Advantage:Easy to compute as long as the forward Easy to compute as long as the forward mapping function is known.mapping function is known.

DisadvantageDisadvantagePixel-to-pixel mapping is not 1-1.Pixel-to-pixel mapping is not 1-1.Holes may appear.Holes may appear.Can result in aliasing.Can result in aliasing.

Scan conversion with forward Scan conversion with forward mappingmapping

Scan conversion with inverse Scan conversion with inverse mappingmapping

Algorithm: Algorithm: for (r,c) = polygon pixelfor (r,c) = polygon pixel u = TEXR(r,c)u = TEXR(r,c) v = TEXC(r,c)v = TEXC(r,c) copy pixel at source (u,v) copy pixel at source (u,v) to destination (r,c)to destination (r,c)

Scan conversion with inverse Scan conversion with inverse mappingmapping

Advantage: Advantage:

Every destination pixel is filled (no holes).Every destination pixel is filled (no holes).

Allow easy incorporation of pre-filtering & resampling Allow easy incorporation of pre-filtering & resampling operations to prevent aliasingoperations to prevent aliasing

Scan conversion with inverse Scan conversion with inverse mappingmapping

Take advantage of Take advantage of Scanline Polygon Fill AlgorScanline Polygon Fill Algorithmithm

For a row scan, maintain a list of scanline / polFor a row scan, maintain a list of scanline / polygon intersections.ygon intersections.

Intersection at scanline r+1 efficiently computeIntersection at scanline r+1 efficiently computed from row r.d from row r.

Scanline yk+1

Scanline yk{xk, yk}

{xk+1, yk+1}

Scan conversion with inverse Scan conversion with inverse mappingmapping

Coordinates at a non-boundary level are compCoordinates at a non-boundary level are computed by linearly interpolating (u,v) coordinates uted by linearly interpolating (u,v) coordinates of bounding pixels on the scanline.of bounding pixels on the scanline.

Scanline yk+1

Scanline yk{xk, yk}

{xk+1, yk+1}

Scan conversion with inverse Scan conversion with inverse mappingmapping

Suppose (ri,ci) maps to (ui,vi), i = 1,…, 5Suppose (ri,ci) maps to (ui,vi), i = 1,…, 5 (r4,c4) = s (r1,c1) + (1-s) (r3,c3) { s is known }(r4,c4) = s (r1,c1) + (1-s) (r3,c3) { s is known } (u4,v4) = s(u1,v1) + (1-s)(u3,v3) {u4,v4 are known}(u4,v4) = s(u1,v1) + (1-s)(u3,v3) {u4,v4 are known} Similarly, (u5, v5) can be found.Similarly, (u5, v5) can be found. t = (c-c4)/(c5-c4)t = (c-c4)/(c5-c4) (r,c) = t*(u5,v5) + (1-t)*(u4,v4)(r,c) = t*(u5,v5) + (1-t)*(u4,v4)

Scanline yk(r4, c4)

(r1, c1)

(r3, c3) (r2, c2)

(r5, c5)

(r, c)

image

Basic 2D linear mappingBasic 2D linear mapping

Scaling & TranslationScaling & Translationu = ar + du = ar + dv = bc + ev = bc + eupright rectangle upright rectangle uprig upright squareht square

Euclidean mappingEuclidean mappingu = (cosu = (cos)r – (sin)r – (sin)c + d)c + dv = (sinv = (sin)r + (cos)r + (cos)c + e)c + erotated unit square rotated unit square upr upright squareight square

Basic 2D linear mappingBasic 2D linear mapping

Similarity mappingSimilarity mappingu = s(cosu = s(cos)r – s(sin)r – s(sin)c + d)c + dv = s(sinv = s(sin))r + s(cosr + s(cos)c + )c + eerotated square rotated square upright u upright unit squarenit square

Affine mappingAffine mappingu = f(cosu = f(cos)r – g(sin)r – g(sin)c + d)c + dv = h(sinv = h(sin)r + i(cos)r + i(cos)c + e)c + erotated rectangle rotated rectangle uprigh upright unit squaret unit square

DEMO !

Basic 2D linear mappingBasic 2D linear mapping

Projective mappingProjective mappingThe most general 2D linear mapThe most general 2D linear mapSquare Square arbitrary quadrangle ! arbitrary quadrangle !

u = (a11r+a12c+a13) / (a31r+a32c+1)u = (a11r+a12c+a13) / (a31r+a32c+1) v = (a21r+a22c+a23) / (a31r+a32c+1)v = (a21r+a22c+a23) / (a31r+a32c+1)

The 8 variables a11,a12, … , a32 have to be The 8 variables a11,a12, … , a32 have to be found out.found out.

Basic 2D linear mappingBasic 2D linear mapping

We have a system of 8 equations solving 8 We have a system of 8 equations solving 8 unknowns.unknowns.

(x1,y1)

Future WorkFuture Work

Mapping cansMapping cans Speed optimizationSpeed optimization Movie manipulationMovie manipulation Use of 3D markersUse of 3D markers

Q & AQ & A

See the foot notes.