IMAGE COMPRESSION IN A MULTI-CAMERA SYSTEM BASED ON … · keep the encoding complexity low, the proposed algorithm builds on PRISM (Power-efficient, Robust, hIgh compres-sion, Syndrome-based

IMAGE COMPRESSION IN A MULTI-CAMERA SYSTEM BASED ON ADISTRIBUTED SOURCE CODING APPROACH

G. Toffetti†, M. Tagliasacchi†, M. Marcon†, S. Tubaro†, A. Sarti†, K. Ramchandran∗

†Dipartimento di Elettronica e Informazione,Politecnico di Milano,

P.zza Leonardo da Vinci, 3220133 Milano, Italy

phone: + (39) 2399 3647, fax: + (39) 2399 3413email: {toffetti, tagliasacchi, tubaro, sarti}@elet.polimi.it

∗ EECS Department.,UC Berkeley

269 Cory Hall Berkeley, CA 94720phone: 510-642-2353

email: [email protected]

ABSTRACT

This paper illustrates an algorithm specifically designedfor encoding multiple views of the same scene taken fromcalibrated cameras. The assumption here is that theseviews are strongly correlated as they represent the samecontent viewed from different perspectives. In order tokeep the encoding complexity low, the proposed algorithmbuilds on PRISM (Power-efficient, Robust, hIgh compres-sion, Syndrome-based Multimedia coding), a video codingframework based on distributed source coding principles.The encoder is constrained to perform a very cheap coarse 3Dreconstruction of the scene, whereas the decoder has accessto the best 3D reconstruction to be used as side information.Preliminary results on synthetic objects demonstrate that itis possible to achieve a coding efficiency gain with respect toINTRA coding at a low encoding complexity.

1. INTRODUCTION

Traditional image and video coding are usually limited tocompress one source at a time, representing the scene from asingle viewpoint. With the proliferation of cheap acquisitiondevices it is easier to take simultaneously several looks at thesame scene from different angles. At the receiving end, theseviews can be blended together to form a 3D reconstructionof the scene. Recent results [1] show that a 3D TV mightbe a reality in the near future. An interesting applicationscenario is characterized by cameras deployed as a sensornetwork, consisting of a large number of sensing devices thatare low-power and with wireless communication capabilities.Although the potential applications of sensor networks aremanyfold, ranging from home security to environment con-trol, traffic monitoring and more, in this paper we concen-trate on camera sensor networks. Each node, equipped witha digital camera, views the scene from a different perspec-tive and it communicates the sensed images to the centralnode. As we are assuming that the communication mediumis wireless, without loss of generality we can state that theneighboring nodes can listen to what is being transmittedto the central node thus allowing some sort of exchange ofinformation among distributed nodes. In order to achieve agood coding efficiency, it is mandatory to take advantage ofthe geometrical correlation among the multiple views. Fig-ure 1 shows an example of how the cameras can be deployed.We want to encode the view taken from camera X exploitingthe views from the neighboring cameras A, B, and C. If theencoder is not power constrained, a viable solution consistsin communicating A, B and C to node X and to perform a

The authors wish to acknowledge the support pro-vided by the European Network of Excellence VISNET(http://www.visnetnoe.org)

3D reconstruction of viewpoint X using this information toform the best predictor Y . The encoder transmits the pre-diction residue N = X − Y , thus saving bits with respect toINTRA coding. As the sensing cameras usually have limitedpower, we want to move the complexity of the costly 3D vir-tual reconstruction to the decoder side, located at the centralnode. In order to achieve this we exploit the ideas of PRISM[2][3], a video coding framework built on distributed sourcecoding principles. PRISM is able to flexibly distribute thecomplexity by moving most or all of the motion estimationtask from the encoder to the decoder. In this paper we takea similar approach, where motion estimation is replaced by3D rendering. Briefly, the encoder at node X receives A,B and C and builds a low-cost coarse 3D reconstruction ofthe view, Yc. Based on this information the encoder triesto infer the correlation with the side information that willbe available at the decoder, thus deciding the bit allocation.The decoder has access to A, B and C and not being powerconstrained can build the best 3D reconstruction Y . Y isused as a side information to decode the view X. Section 4elaborates on this topic giving further details on the way thecorrelation is estimated and how the side information can beeffectively used.

The problem of coding multiple views using a distributedsource coding approach has been recently explored in the lit-erature. In [4] communication among cameras is not allowedbut some prior information about the geometry of the cam-eras and the distance of the objects is required. This workelaborates a strategy for efficiently encoding the positions ofthe objects given that only the decoder will have access tothe other views. The work in [5] is related to the algorithmproposed in this paper but the encoding process heavily re-lies on the Wyner-Ziv codec presented in [6]. Low-encodingcomplexity is achieved without communication among cam-eras but a feedback channel is needed.

The rest of this paper is organized as follows. Section 2reviews the rendering algorithm used to build the 3D virtualreconstruction. Section 3 briefly illustrates the basic ideas ofPRISM while Section 4 details the proposed algorithm. Pre-liminary experimental results on synthetic images are givenin Section 5.

2. 3D RENDERING ALGORITHM

In this section we briefly review the rendering method [7]used in our coding scheme. The algorithm receives in inputthree (or more) images from calibrated cameras. One camerais chosen as the preferred one and a depth map is estimated,indicating for each pixel of the image the depth of the object.For a given depth map f , a cost function is defined as:

E(f) = Edata(f) + Esmooth(f) (1)

Figure 1: Camera sensor network. White circles representimages to be INTRA coded while grey circles are imagesencoded with the proposed algorithm.

Our data term is defined as a non-positive value which re-sults from the differences in intensity between correspondingpixels. It is computed for every pixel p of the preferred image(we indicate this image with the index j) by these steps:

1. from p, we get the corresponding 3D-point by retro-projecting it from the preferred camera center of pro-jection with the selected depth and then we project this3D-point on each other calibrated images obtaining a setof n − 1 corresponding pixels q1, q2, ..., qi, ...qn|i 6= j

2. on every non-preferred image we compute the SSD (Sumof Square Difference) using a square window centered onqi and one centered on p, obtaining the set of valuesd1, d2, ..., di, ...dn|i 6= j

3. finally

Edata(f) = min(0,∑

p∈i=ji6=j

di − K) (2)

where K is a positive constant large enough to capturesignificant variation of the SSD function (a typical valueis K = 30).

The smoothness term is quite similar to the one used in [8]and its goal is to make neighboring pixels in the preferredimage tend to have similar depths. In order to minimize themulti-variate cost function E(f) we use an approach basedon graph cuts [8]. This methods are fast enough to be prac-tical, but unlike simulated annealing, graph cuts methodscannot be applied to arbitrary functions. We use some re-cent results [9] that give graph constructions for a quite gen-eral class of energy functions. The optimal depth map isthen filtered to remove outliers and to reduce the quantiza-tion noise. Finally, in order to create a complete model ofthe object we blend together the several surface patches ob-tained with the previous graph cuts based method. Furtherdetails can be found in [7]. Figure 2 shows as an example theviews taken from the four calibrated cameras. Figure 3 de-picts the rendered views using each of the three surroundingcameras as the preferred one. We can notice that dependingon the chosen reference the quality of the reconstruction ineach region varies. In both cases the central image is the onewe are encoding.

3. BACKGROUND ON PRISM

The PRISM video coder is based on a modification of thesource coding with side-information paradigm, where there isinherent uncertainty in the state of nature characterizing theside information. The Wyner-Ziv Theorem [10] deals withthe problem of source coding with side-information. Theencoder needs to compress a source X when the decoder hasaccess to a source Y . X and Y are correlated sources and Y

Figure 2: Original views taken from viewpoints A, B, C andX. The latter is encoded with the proposed algorithm.

Figure 3: Rendered images YA, YB and YC from viewpointX obtained using A, B and C as preferred image in therendering process.

is available only at the decoder. From information theory weknow that for the MSE distortion measure and X = Y + Nwhere N has a Gaussian distribution, the rate - distortionperformance for coding X is the same whether or not theencoder has access to Y .

For the problem of source coding with side information,the encoder needs to encode the source within a distortionconstraint, while the decoder needs to be able to decode theencoded codeword subject to the correlation noise (betweenthe source and the side-information). While the results ofWyner and Ziv are non-constructive and asymptotic in na-ture, a number of constructive methods to solve this problemhave since been proposed (such as in [11][12][6]) wherein thesource codebook is partitioned into cosets of a channel code.

For the PRISM video coder [2], the video frame to be en-coded is first divided into non-overlapping spatial blocks ofsize 8x8. The source X is the current block to be encoded.The side-information Y is the best (motion-compensated)predictor for X in the previous frame and let X = Y + N.We first encode X in the intra-coding mode to come up withthe quantized codeword for X. Now, we do the syndromeencoding, i.e., we find a channel code that is matched to the”correlation noise” N, and use that to partition the sourcecodebook into cosets of that channel code. The encodertransmits the syndrome (indicating the coset for X) and aCRC check of the quantized sequence. In contrast to MPEG,H.26x, etc., it is the decoder’s task to do motion search, asit searches over the space of candidate predictors one-by-one

to decode a sequence from the set labeled by the syndrome.When the decoded sequence matches the CRC check, decod-ing is declared to be successful. For further details pleaserefer to [3].

4. PROPOSED ALGORITHM

We refer in the following to the camera configuration of Fig-ure 1. Cameras A, B and C as well as all the other camerasmarked in gray encode images in INTRA mode, i.e. withoutreference to other cameras. Cameras marked in white usethe proposed algorithm based on distributed source coding.We focus on camera X. At the encoder A, B and C encodethe respective views and send to the central node the quan-tized versions A, B and C. Node X listens to what is beingtransmitted and uses these three views to build a coarse vir-tual reconstruction of X, Yc, using the rendering algorithmdescribed in Section 2. In order to reduce the computationalcomplexity at the encoder, the rendering of the virtual viewis carried out at reduced resolution. In our implementationwe downsample the received images by a factor of four inboth direction before feeding them to the rendering algo-rithm. The encoder processes the image on a block-by-blockbasis. For each block, the encoder checks the correlationexisting with the co-located block in the coarse virtual re-construction Yc, computing MSEc =

∑i|Xi − Y i

c |2. Basedon this measure, the encoder tries to infer the correlationthat will be observed at the decoder N = X − Y , whereY represents the best predictor that can be found at thedecoder side. The decoder is free from power constraintsand can thus use all the information available to build thebest side information. The rendering algorithm we are us-ing gives different results based on the image that is chosenas reference. This is due to the fact that it adds a depthmeasure only to the locations visible from the image chosenas the reference. In order to get a better side information,we can actually perform three different 3D reconstructions,using respectively A, B and C as reference, producing inoutput YA, YB and YC as shown in Figure 3. The decoder isable to use as a predictor any of these images. It is possibleto note that depending on the spatial region of interest, wewould pick the one that gives the best predictor. Moreover,the rendering algorithm is not able to produce a result thathas a precise reconstruction at the pixel level. This mightnot be an issue in computer vision applications, but it is notsatisfactory when it is used to build a predictor for encodingX. For this reason we allow the decoder to perform a motionsearch in small range about the co-located block in each ofthe rendered views YA, YB and YC , in order to increase thecorrelation of the encoded image with its side information.The missing link between the encoder and the decoder isassured by an off-line module that collects correlation noisestatistics defining a mapping between what is observed atthe encoder and the best predictor disclosed at the decoder.The proposed algorithm inherits most of the coding tools ofPRISM. While PRISM computes at the encoder the MSE atzero motion (co-located block in the reference frame), we cal-culate here the MSE between the block and the co-locatedblock in the coarse rendered view Yc. On the other hand,at the decoder PRISM searches for the best prediction inthe reference frame, whereas our algorithm searches in thethree high quality rendered views YA, YB and YC . Let ussummarize the steps carried out by our algorithm:

Encoder:

• receive the images A, B and C• compute a coarse low-resolution rendering of X, Yc using

A, B and C• for each block

– compute the MSE between X and the co-locatedblock in Yc

– compute the DCT transform of block X– INTRA encode block X– read the statistics collected by the classifier to infer

the correlation noise and to perform bit allocation– send syndrome bits and the CRC signature of the

quantized block

Decoder:

• receive the images A, B and C• compute three high-quality high-resolution reconstruc-

tions of X, YA, YB and YC using A, B and C• for each block

– read syndrome bits and CRC– perform a motion search around the co-located blocks

in YA, YB and YC

– when the CRC of the decoded block matches withthe CRC sent by the encoder, a decoding success isdeclared and motion search is stopped (see Section 3)

– recover the reconstructed block by IDCT

The decoder continues the motion search until it findsa predictor whose correlation with the block to be decodedis below the noise margin for which the channel code wasdesigned (see Section 3). When this happens a decodingsuccess is declared. If the motion search concludes withouta match, decoding fails and a simple error concealment tech-nique is employed by pasting the co-located block of one ofthe three rendered views. In order to increase the decodingspeed, predictors are visited in such a way that a decodingsuccess occurs as soon as possible. A spiral search searchesfor predictors in each of the images YA, YB and YC startingfrom zero motion (co-located blocks). The same candidatemotion vector is tested for all the three images before pro-ceeding to the next predictor.

5. EXPERIMENTAL RESULTS

We performed some preliminary tests on synthetic objects.Both teapot and pitbull have been modeled by a 3D softwareand several snapshots have been rendered from it. At theencoder the rendering algorithm is fed with images havingone sixteenth of the original resolution, resulting in a speed-up of a factor of 100 with respect to the full quality recon-struction performed at the decoder side. We compared therate-distortion performance of the proposed algorithm withINTRA coding. For the latter we used the H.263+ INTRAmode as a benchmark. Figures 4 and 5 shows the recon-structed image quality as a function of bit-rate for teapotand pitbull. The numbers indicated in the plot refer to thequantization parameter (QP) used (the actual quantizationstep is 2 · QP ). At low bitrates the proposed algorithm out-performs INTRA coding whereas at high bitrates there is aPSNR penalty. This is due to the fact that a couple of blocksare incorrectly decoded and the error concealment techniquepastes the co-located block from YA. This PSNR drop doesnot reflect the perceived quality though. At the same quanti-zation step size, reflecting the subjective quality better thanPSNR in this case, we observe a bit saving of around 20%.Figures 6 and 7 show the decoded teapot and pitbull imagesat QP = 8. Although we haven’t run specific simulationsin this direction, we expect to be able to exploit the robust-ness features of PRISM. Let us say for example that A isnot available at the decoder. The decoder can pick anotherneighboring camera and still perform a 3D rendering of X. Ifthe new side information is still within the noise margin thedecoder will still be able to decode. This is because PRISMis not tied to a single deterministic predictor but it encodesfor the statistical correlation between the block and its sideinformation.

0.6 0.8 1 1.2 1.4 1.6 1.8 2 2.235

36

37

38

39

40

41

42

43

44

45 H263+ intraPRISM 4

8

12

16

4

8

12

16

bitrate (bpp)

PS

NR

Figure 4: Teapot - PSNR vs. bitrate

1 1.2 1.4 1.6 1.8 2 2.2 2.4 2.6

39

40

41

42

43

44

45

46

47

48H263+ intraPRISM

4

8

10

12

4

8

10

12

bitrate (bpp)

PSN

R

Figure 5: Pitbull - PSNR vs. bitrate

6. CONCLUSIONS

We proposed a coding algorithm for multi-camera viewsbased on distributed source coding. Preliminary results showpromising results with respect to INTRA coding. We arecurrently investigating the use of other parameters otherthan MSEc (i.e. estimated surface angle, local smoothness,etc.) to drive the bit allocation at the encoder. Moreover wethink that this approach can be extended to multi-cameravideo, in such a way that both temporal (from past frames)and spatial (from rendered views) predictors can be used atthe decoder.

REFERENCES

[1] Anthony Vetro, Wojciech Matusik, Hanspeter Pfister, andJun Xin, “Coding approaches for end-to-end 3D TV sys-tems,” in Picture Coding Symposium, San Francisco, CA,December 2004.

[2] Rohit Puri and Kannan Ramchandran, “PRISM: A New Ro-bust Video Coding Architecture based on Distributed Com-pression Principles,” in Allerton Conference on Communi-cation, Control and Computing, Urbana-Champaign, IL, Oc-tober 2002.

[3] Rohit Puri and Kannan Ramchandran, “PRISM: A videocoding architecture based on distributed compression princi-ples,” Tech. Rep. No. UCB/ERL M03/6, ERL, UC Berkeley,March 2003.

Figure 6: Teapot. Decoded image (QP = 8)

Figure 7: Pitbull. Decoded image (QP = 8)

[4] Nicolas Gehrig and Pier Luigi Dragotti, “On distributed com-pression in dense camera sensor networks,” in Picture CodingSymposium, San Francisco, CA, December 2004.

[5] Anne Aaron, P. Ramanathan, and Bernd Girod, “Wyner-Zivcoding of light fields for random access,” in Proceedings ofthe IEEE Workshop on Multimedia Signal Processing, Siena,Italy, September 2004.

[6] Anne Aaron and Bernd Girod, “Compression with side in-formation using turbo codes,” in Proceedings of the IEEEData Compression Conference, Snowbird, UT, April 2002.

[7] Gianluca Dainese, Marco Marcon, Augusto Sarti, and Ste-fano Tubaro, “Complete object modeling using a volumetricapproach for mesh fusion,” in Workshop on Image Analysisfor Multimedia Interactive Services 2005, Montreux, Switzer-land, March 2005.

[8] V. Kolmogorov and R. Zabih, “Multi-camera scene recon-struction via graph cuts,” in European Conference on Com-puter Vision, Copenhagen, Danmark, May 2002.

[9] V. Kolmogorov and R. Zabih, “What energy functions canbe minimized via graph cuts?,” in European Conference onComputer Vision, Copenhagen, Danmark, May 2002.

[10] Aaron D. Wyner and Jacob Ziv, “The rate distortion func-tion for source coding with side information at the decoder,”IEEE Transactions on Information Theory, vol. 22, pp. 1–10,January 1976.

[11] Sandeep S. Pradhan and Kannan Ramchandran, “Distrib-uted source coding using syndromes (DISCUS): Design andconstruction,” in Proceedings of the IEEE Data CompressionConference, Snowbird, UT, March 1999.

[12] Angelos D. Liveris, Zixiang Xiong, and Costas N. Georghi-ades, “Distributed compression of binary sources usingconventional parallel and serial concatenated convolutionalcodes,” in Proceedings of the IEEE Data Compression Con-ference, James A. Storer and Martin Cohn, Eds., Snowbird,UT, March 2003, pp. 193–202.