A Hybrid Scheme for Low Bit-Rate Coding of Stereo Images

A Hybrid Scheme for Low Bit-Rate Coding of Stereo Images

Jianmin Jiang, Eran A. EdirisingheUniversity of Bradford, Bradford Loughborough University, Leicestershire

IEEE TRANSACTIONS ON IMAGE PROCESSING, FEB 2002

OUTLINE

INTRODUCTION (Stereo Image) AN OVERVIEW OF THE HYBRID

SCHEME CONTOUR ANALYSIS OF STEREO

IMAGES FREE-FORM OBJECT ENCODING EXPERIMENTAL RESULTS

INTRODUCTION

Stereo Image Stereo images are constructed by

simulating human viewing effect upon observing objects through two horizontally separated positions

Every stereo image can be represented two frames labeled as left frame and right frame.

INTRODUCTION (cont.) If these left and right frames are to be

compressed and transmitted independently, the bandwidth required would need to be double.

Efficient compression can be achieved by exploiting redundancies Intra-frame redundancy

Two constituent images Inter-frame redundancy

Left and right frames

INTRODUCTION (cont.) Estimation for disparity or binocular para

llax Intensity based method

Look for correspondence between luminance values (eg. Block matching) Fixed blocks can’t reflect the true disparity betwe

en those free-form objects. Object based method

Determine objects in both images and seeks correspondence between the two sets (eg. The proposed scheme)

AN OVERVIEW OF THE HYBRID SCHEME

Left

Right

JPEG

iJPEG

ObjectExtraction

&match R-Obj

L-ObjBlock

Classify

Boundarycoder

Interiorcoder

DCTcoder

Backgroundcoder

Block Disparity

Object Disparity

CONTOUR ANALYSIS OF STEREO IMAGES

Contour Extraction Laplacian-of-Gaussian (LoG) operator ,σ is the standard deviation The LoG response will be positive on the dar

ker side, and negative on the lighter side

2

22

22

22

4]

21[

1),(

yx

eyx

yxLoG

The left hand graph shows a 1-D image, 200 pixels long, containing a step edge.

The right hand graph shows the response of a 1-D LoG filter with Gaussian=3 pixels

intensity

Contour Extraction (cont.) Let sx and sy be the slops of the LoG of

the image along both x and y directions for some zero-crossing point.

An edge strength at (x, y) is defined as

The contour points are chosen by using a pre-set threshold


1

463

2141

21 1

-1-2

-3

-2-4


Contour Matching Closed contours Open contours Background = + + …

We use chain codes {ai 0, 1, …, 7} to represent the contours

04


Contour Matching Closed contours (hui Li, Sanjit k. Mitra, 1995)

Shifting and smoothing the chain codes We convert the standard chain code {ai} into a modi

fied code {bi} by a shifting operation defined by

e.g. {7,0,7,0,7,0} -> {7,8,7,8,7,8} e.g.

{2,0,7,4} {4,2,1,6}

{2,0,-1,-4} {4,2,1,-2}

Mean = -1 1 {3,1,0,-3}-Mean


Chain code

Chain code

smoothing & shifting

smoothing & shifting

CONTOUR ANALYSIS OF STEREO IMAGES Contour Matching (Closed contours)

Measure of correlation between two n-point segments, one starting at index k of contour L and the other starting at index l of contour R, is defined as:

Where

{li} and {ri} be the chain code representations of two contours L and R, which corresponds to the left and right frames, and let NL and NR be their lengths

The similarity function FLR = max {MCkl} If FLR≧ FL’R, where L’ represents all similar contours to R, the b

est match is selected.

1

0

'' )(4

cos1 n

j

jljkkl rln

MC

1

0

mod)(mod)(1

'n

j

NjkNikik LL ln

ll

n pointssegments

RL

lk 80

4MCkl=1 when perfect match

L R


Contour Matching (open contours) Measure of curvature at the ith point for a co

ntour of length n with chain code {li}

The contour segments surrounding the salient points are then used as 1-D templates in finding the corresponding matches in the other image

|}}||,{max{|max 131

jijijijij

i llllC

If Ci ≧Ck for all k [i-p, i+p], where p is a constant, we say this point is a salient point

2p

2p

L R


Left

Right

JPEG

iJPEG

ObjectExtraction

&match R-Obj

L-ObjBlock

Classify

Boundarycoder

Interiorcoder

DCTcoder

Backgroundcoder

Block Disparity

Object Disparity


Contour Blocking and Classification8x8 pixels block

Internal block

External block

Boundary block

Bounding box

Object Bounding Rectangle (OBR)

We extend the bounding box such that its width and height are multiples of eight pixels

FREE-FORM OBJECT ENCODING

Padding of left OBR Extensive experiments indicated that

certain trends exist with respect to pixel value variations for boundary blocks.

We want to use a linear equation to predict those exterior pixels immediately next to to the boundary pixels in a row (projected pixel).

(E.A. Edirisinghe, J. Jiang and C. Grecos 1999)

Interior pixel

Projected pixel


Padding of left OBR (cont.) In a given row, assume there exist n (1 < n <

N) consecutive pixels inside the object,which are bounded by a projected pixel on the left or on the right . Let these n pixel values be represented by Pn.

BXAP nn Where Xn represents the column number of Pn with respect to the projected pixel

n = 2

N

N

FREE-FORM OBJECT ENCODING Padding of left OBR (cont.)

1. When Pn are bounded by two projected pixels, both projected pixels will be determined using the same linear equation.

2. If a projected pixel is flanked by interior pixels on both sides, the above process is performed in both directions and the average of the two results is taken.

3. If n = 1, the projected pixel value is taken to be equal to the single interior pixel value.

4. Remaining exterior pixels are padded with traditional technique.

Padding of left OBR (cont.) After all the boundary blocks are padded,

the exterior blocks immediately next to the boundary blocks are filled by replicating the samples at the border of boundary blocks with the priority

1. Boundary block to the left of exterior block2. Block at the top3. Block to the right4. Block at the bottom (lowest priority)5. Remaining blocks are filled with value of 128


3 P PP P

3 P 1P 2

3 P P P P P 1P 5

2 P P P P 515

5 3 P P 1 4 44 4

3 P P P P P

5 3 P P P P PP P

1


Left

Right

JPEG

iJPEG

ObjectExtraction

&match R-Obj

L-ObjBlock

Classify

Boundarycoder

Interiorcoder

DCTcoder

Backgroundcoder

Block Disparity

Object Disparity


Disparity-Based Prediction for Both Boundary Blocks and Internal Blocks Boundary blocks

Search range [-7,7]. Alpha plane (L unpadded)

Square error (L padded)

W = [-7,+7]

Left obj. Right obj.

{),(,1..,0

),(Llkif

wolkAlphaL

{

),(,1..,0

),(Rsrif

wosrAlphaR

8

1,

2)()_,_(ji

Rijij AlphayxblockLblockRSE

)_,_(min__

blockLblockRSEblockLwblockL

match

0011

0011

0111

1111

-( )2 x

RL

The best match

Since the stereo pair is produced by observation of the same scene from two

horizontally separated positions, disparity between the two frames will only occur

horizontally rather than vertically


Disparity-Based Prediction for Both Boundary Blocks and Internal Blocks Boundary blocks

Error block : The shape of the right object would be similar t

o that of its matching left object, so we won’t encode the shape of right OBRs.

R

= {sij}

)()( LijLijij AlphamAlphasc

= {mij}

L (padded)


Disparity-Based Prediction for Both Boundary Blocks and Internal Blocks Internal blocks

Use the two blocks to be pioneering blocks, one at the top and one at the left of the block to be encoded

No overhead bits needed for informing the decoder of which block is chosen in the left frame

(J. Jian, E.A. Edirisinghe and H. Schroder 1997)

Rij-1

RijRi-1j

XX


Rij-1

RijRi-1j

Lij-1

LijLi-1jBlock to be encoded

Pioneering blocks

Best match

predictor

Search window|-----N x 8-----|

L R

11

_ijji

RRR

blockP ij α,β (0,1], the weight of each block

1. Use Rij-1 and Ri-1j to compute Rij

2. Find the best match Lij in L, where L is produced by Lij-1 and Li-1j


Internal texture smooth detection Calculate the error block and MSE between

two matching regions. If MSE is less then some threshold, the error block won’t be encoded.

If MSE is too large, object extraction would be proceeded within current object.

matching coding

EXPERIMENTAL RESULTSE.C.R = (bit_NoJ – bit_NoP)/bit_NoJ

(Bit_NoJ and bit_NoP: total bits produced in compressing right frames by JPEG and by the proposed algorithm)

DCTDP : disparity compensated transform domain predictive coding(block based)

EXPERIMENTAL RESULTS

The reference image quality is poor, so matches between objects will produce larger error for both boundary and internal areas.

EXPERIMENTAL RESULTS

Visual inspection for example (right frame)

original DCTDP proposed

A Hybrid Scheme for Low Bit-Rate Coding of Stereo Images

Documents

left frame

contour r

right framesintroduction

standard chain code

d image

flr flr

right hand graph

chain code representations