Image Inpainting and Segmentation using Hierarchical Level Set Method

Image Inpainting and Segmentation using Hierarchical

Level Set Method

Xiaojun Du, Dongwook Cho, Tien D. BuiDept. Computer Science & Software Engineering

Concordia University, Montreal, QC, Canada H4V 2H8E-mail: {xiao du, d cho, bui}@cse.concordia.ca

Abstract

Image inpainting is an artistic procedure to recovera damaged painting or picture. In this paper, wepropose a novel approach for image inpainting. Inthis approach, the Mumford-Shah (MS) model andthe level set method are employed to estimate im-age structure of the damaged region. This approachhas been successfully used in image segmentationproblem. Compared to some other inpainting meth-ods, the MS model approach can detect and pre-serve edges in the inpainting areas. We propose inthis paper a fast and efficient algorithm which canachieve both inpainting and segmentation. In previ-ous works on the MS model, only one or two levelset functions are used to segment an image. Whilethis approach works well on some simple images, de-tailed edges cannot be detected on complicated im-ages. Although multi-level set functions can be usedto segment an image into many regions, the tradi-tional approach causes extensive computations andthe solutions depend on the location of the initialcurves. Our proposed approach utilizes faster hier-archical level set method and can guarantee conver-gence independent of initial conditions. Because wecan detect both the main structure and the detailededges, the approach can preserve detailed edges inthe inpainting area. Experimental results demon-strate the advantage of our method.

Keywords: image inpainting, variational method,Mumford-Shah functional, numerical PDE, imagesegmentation

1 Introduction

Image inpainting is originally an artistic procedureto recover a damaged painting or picture. It has beenintroduced in [1] and received attention from manyresearchers in computer vision and image processing.

From a technical point of view, digital image in-painting can be described as a procedure to fill a de-

fined inpainting domain (i.e. a set of damaged pixelsin a given image). In this case, we can recall some re-lated topics in image processing and computer vision,namely image interpolation, object removal or disoc-clusion problem, texture synthesis, film restoration,and damaged block recovery of compressed digitalimage or video.

Bertalmio et al. [1] formulated a third-order par-tial differential equation (PDE) which considers thecontinuity of the isophotes at the boundary and ameasure of the change in the information to be prop-agated. However, the method leads to a number ofiterative operations and fails in regions with morecomplicated image structures. They proposed animproved method for simultaneous filling-in of tex-ture and structure by combining an image decompo-sition algorithm and texture synthesis [2]. For bettercomputational efficiency, the algorithm proposed in[10] estimates a damaged region by applying aver-aging operation on neighboring pixels. Therefore, itis simple and fast. However, the results are blurredand it is hard to preserve the edges. In [3], Chanand Shen considered some mathematical models suchas total variation (TV), Green’s second formula andMumford-Shah model. Energy functional of the ge-ometric image models can be built in the Bayesianframework and the distorted pixels can be estimated.Equivalently, numerical PDE solution of functionalminimization can be applied. In [6], Criminisi et al.proposed a comparatively fast inpainting algorithmfor filling a large region occluded by an object. Themethod fills the region with a selected exemplar (asmall patch in the image) based on an edge-drivenreconstruction criterion.

In this paper, energy minimization approach isconsidered to find edges in the inpainting domain.We use the Mumford-Shah model and level setmethod, which have been successfully used in im-age segmentation. In our algorithm, image structureof the damaged region is estimated based on hierar-

Proceedings of the 3rd Canadian Conference on Computer and Robot Vision (CRV’06) 0-7695-2542-3/06 $20.00 © 2006 IEEE

chical approach of numerical level-set algorithm. Forbetter region segmentation and edge estimation in aninpainting domain, we adopt hierarchical approachwhich uses multi-level set functions. Although multi-level set functions can be used to segment an imageinto many regions, classical approaches cause exten-sive computations and the solutions also depend onthe initial conditions. This is a critical issue forpractical applications. Our proposed approach uti-lizes faster hierarchical level set method. Because wecan detect both the main structure and the detailededges, our approach can preserve the edges in theinpainting area.

The organization of this paper is as follows. InSection 2, multiphase level set algorithm using theMumford-Shah model is briefly described. Our pro-posed algorithm based on the Mumford-Shah modeland the two phase level set algorithm is in Section 3.Experimental results are shown and compared withsome existing methods in Section 4. Conclusions andfuture work are presented in Section 5.

2 Mumford-Shah Model for In-painting

The original Mumford-Shah (MS) model was pre-sented by Mumford and Shah in [9]. In this model,an energy functional is minimized to segment and de-noise an image. The MS energy functional is givenby

E(u,C) =∫

Ω

|u − u0|2 dxdy (1)

+ μ

∫Ω\C

|∇u|2 dxdy + ν · |C| .

In Eq. (1), u0 is the original image; u is the smoothapproximation of u0; C is the segmentation curve;|C| represents the length of the curve; Ω is the imagedomain; Ω\C represents the image domain excludingthe segmentation curve.

Mumford and Shah proposed that the segmenta-tion of an image can be obtained through the mini-mization of this energy functional. If the MS energyfunctional is minimized, the image will be segmentedinto regions so that: (1) u is a good approximationof u0, (2) u is smooth in each region, and (3) theboundary of each region is as short as possible. Theparameters of μ and ν are used to balance the effectsof different terms. Tsai, Yezzi, and Willsky [12], andChan and Shen [4] are the first to present the idea ofapplying the MS model to inpainting. For inpaint-ing, they modified the MS model as:

E(u,C) =∫

Ω

λ(x, y) |u − u0|2 dxdy (2)

+ μ

∫Ω\C

|∇u|2 dxdy + ν · |C| ,

where λ(x, y) = 0 if (x, y) is inside the inpaintingarea and 1 otherwise. The above equation indicatesthat only the variance of the image and the lengthof the segmentation curve are considered inside theinpainting area. The solution of the MS energy func-tional is not a trivial task. There are some alter-native solutions to this problem, such as piecewisesmooth approximation of the Mumford-Shah modelpresented by Chan and Vese [5]. If we consider thata closed curve segments an image into two regions(i.e. both inside and outside regions), the MS en-ergy functional can be written as:

E(u1, u2, C) =∫inside C

λ(x, y) |u1 − u0|2 dxdy

+ μ

∫inside C

|∇u1|2 dxdy

+∫outside C

λ(x, y) |u1 − u0|2 dxdy

+ μ

∫outside C

|∇u2|2 dxdy

+ ν · |C|, (3)

where u1 and u2 are smooth approximations of theimage inside and outside the curve. The numericalsolution of the MS energy functional can be imple-mented by the level set method. In the case thatthe image consists of two regions, the segmentationcurve can be represented by one level set function φ:

φ(x, y, t) =

⎧⎨⎩

> 0 if (x, y) is inside C= 0 if (x, y) is on C< 0 if (x, y) is outside C

(4)

Recast Eq. 3 in terms of the level set function φ thenminimize the functional energy E with respect to u1,u2, and φ, we obtain the equations for u1, u2, and φas the following [13]:

λ(x, y)(u1 − u0) = μ∇2u1 inside C∂u1∂�n = 0 on C,

(5)

λ(x, y)(u2 − u0) = μ∇2u2 outside C∂u2∂�n = 0 on C,

(6)

∂φ

∂t= δ(φ)

[ν∇ ·

( ∇φ

|∇φ|)− λ(x, y)(u1 − u0)2(7)

− μ|∇u1|2 + λ(x, y)(u2 − u0)2 + μ|∇u2|2]

The smooth image functions u1 and u2 can be ob-tained by solving the damped Poisson Eqs. (5) and

Proceedings of the 3rd Canadian Conference on Computer and Robot Vision (CRV’06) 0-7695-2542-3/06 $20.00 © 2006 IEEE

(6), and the segmentation curve can evolve accordingto Eq. (7). This is the piecewise smooth approxima-tion presented by Chan and Vese. Many advantagescan be achieved by this approach, such as simulta-neous segmentation and smoothing of noisy images,and detection of triple junctions by using multiplelevel set functions. However, because three PDEsEqs. (5), (6), and (7) needed to be solved simulta-neously, the computational cost of this approach isvery large. To overcome this difficulty, Chan andVese proposed another approximation approach us-ing the piecewise constant approximation. If the im-age intensities inside different regions are uniform,the image intensities inside different regions can beapproximated by constants. In this case, the MSenergy functional can be simplified to Eq. (8):

E(ck, C) =∑

k

∫Ωk

λ(x, y)(ck−u0)2dxdy+ν|C|, (8)

where Ωk represents the area inside each region. Thegradient term in the MS energy functional disappearsin Eq. (8) because the gradient inside each region iszero. Using the level set method [11] and the MSenergy functional of the two phase segmentation, theimage is segmented into two regions:

E(c1, c2, φ) =∫

λ(x, y)(c1 − u0)2H(φ)dxdy (9)

+∫

λ(x, y)(c2 − u0)2(1 − H(φ))dxdy

+ ν

∫δ(φ)|∇φ|dxdy

where H(x) is the Heaviside function. To minimizethe energy functional with respect to c1, c2, and φ,we obtain the following equations:

c1(φ) =∫

λ(x, y)u0H(φ)dxdy∫H(φ)dxdy

(10)

c2(φ) =∫

λ(x, y)u0(1 − H(φ))dxdy∫(1 − H(φ))dxdy

(11)

∂φ

∂t= δ(φ)

[ν∇ ·

( ∇φ

|∇φ|)

(12)

− λ(x, y)(u0 − c1)2 + λ(x, y)(u0 − c2)2]

After solving these equations, we can obtain the in-formation on c1, c2, and C. The image u0 willthen be segmented into two regions {u = c1} and{u = c2}.

3 Hierarchical Inpainting andSegmentation

In [8], Gao and Bui proposed a hierarchical methodfor image segmentation and smoothing based on theMumford-Shah variational approach and the level setmethod. The method is fast and robust with re-spect to the initial condition since the curve evolu-tion PDEs are decoupled. Following their idea, wedecouple the segmentation and diffusion. To inpaintan image, we first segment the image using piecewiseconstant approximation of the MS model, and de-termine the object boundaries inside the inpaintingarea. Then we use the diffusion technique to fill theinpainting area. To preserve the object boundary,the diffusion is only conducted from inside the objectregions towards the inpainting area but not acrossthe boundaries of the objects. Compared to thepiecewise smooth approximation of the MS model,the method decouples the solution of three PDEs,and the computation is faster.

With the level set method, the MS model workswell in many applications. However, generally onlyone or two level set functions are used to segmentan image into two or four phases. Therefore, mostexperiments have been done on simple images. Forimages with complicated structures, the regions inthe images cannot be represented by one or two levelset functions. We need n level set functions to rep-resent 2n regions. There are many difficulties forthis approach. First, we cannot define the numberof regions beforehand. We may use many level setfunctions to segment an image to avoid missing anyregion. However, this approach requires high compu-tational cost. The most important problem for theMS model using the level set method is the initialcondition. Because the MS energy functional is notconvex, the minimization result often is trapped bya local minimum. Consequently the segmentationresult depends on the initial conditions. This prob-lem is more serious for the cases with many level setfunctions. As a result, the MS model can only detectthe main structure of an image rather than detailedsegmentation.

For images with complicated or detailed struc-tures, although the MS model can produce segmen-tation results for the main structures, local segmen-tation approaches can produce better results for de-tailed structures. To combine the advantages ofthese two different approaches, we present a new hi-erarchical segmentation scheme which makes use ofboth global and local information. In the process ofthe hierarchical segmentation, we use a local window(a small area of image) over a segmentation region

Proceedings of the 3rd Canadian Conference on Computer and Robot Vision (CRV’06) 0-7695-2542-3/06 $20.00 © 2006 IEEE

to detect whether the region needs more additionalsegmentations or not. In this way, we can segmentthe image hierarchically until each object region be-comes smooth. As a result, we can keep the segmen-tation of the main structures, and we also can detectdetailed structures. The most important advantageof this approach is that the segmentation result doesnot depend on the initial conditions and the methodis relatively fast.

In the inpainting area, the detailed edges can beestimated and preserved because we can detect boththe main structure and the details of an image.

Based on the hierarchical segmentation algorithmin [7], our algorithm for image inpainting can be pre-sented as follows:

1. Define an initial closed curve. The area insidethe curve corresponds to the area threshold Ta.

2. Move the initial curve to different positions inone region (in the beginning, the region is thewhole image), and calculate the average intensi-ties inside the curve and the whole region. If atany position, the absolute difference between thetwo average intensities is smaller than the con-trast threshold Tc, the region needs no more seg-mentation. Otherwise, position the initial curveat the location where the absolute difference isthe largest.

3. In the region (in the beginning, the region is thewhole image), use piecewise constant approxi-mation of the MS model to solve the curve evo-lution and obtain the segmentation result.

4. Number the different regions in the image.5. In each region, repeat steps 2 and 3 to segment

the region into smaller regions. In this step, allthe calculations are inside one region.

6. Repeat 5 for each region obtained from differ-ent segmentation stages until all regions needno more segmentation.

7. Calculate the average intensity of each region. Ifthe absolute difference between the average in-tensities of two neighbor regions is smaller thanthe contrast threshold used in step 2, Tc, thetwo regions are merged into one region.

8. Calculate the area of each region. If the area istoo small, for example less than 10 pixels, theregion is regarded as noise, and is merged intoits neighboring region.

9. Use diffusion technique to fill the inpaintingarea. The diffusion is only conducted from in-side the object regions towards the inpaintingarea but not across the boundaries of the ob-jects.

To detect all the details of an image, we can de-fine the minimum object area as one pixel. If thereexists noise, however, the noise can also be detectedas an object. Therefore, in step 1 we need to de-fine a minimum object area to avoid the effect ofnoise. For step 2, it is obvious that if the absolutedifferences between the probe areas and the back-ground are smaller than the contrast threshold, thereis no object in the region, and the region needs nomore additional segmentations. If at some positions,the difference is bigger than the contrast threshold,we position the initial level set curve at the locationwhere the difference is the largest. At this location,the MS energy of the initial curve is the smallestcompared to all other locations.

Repeat steps 2 and 3 on all regions until no moresegmentation is needed in each region. At this point,each region is smooth enough. However, this is notthe final result. Because the curve evolution of seg-mentation does not guarantee that the curve reachesthe boundary of an object. It is possible that an ob-ject is segmented into more than one region. In thiscase, we can merge those neighboring regions whoseaverage intensities are similar. After merging, theimage intensity changes smoothly inside the objectsand the boundaries are kept because the neighboringregions with large differences are not merged. Thethreshold used in this step is the same as the contrastthreshold used in step 2.

In step 8, we remove those regions with very smallarea, which are assumed to be noise instead of ob-ject. When the inpainting area is filled, the edgesinside the inpainting area are kept clear because thediffusing is not across the boundaries.

4 Experimental Results

To show the advantage of hierarchical segmentationand inpainting approach, we implement the inpaint-ing algorithm with total variance (TV) approach, theMS model approach with one level set function, andthe MS model approach with hierarchical multi levelset functions respectively.

Fig. 1 is the inpainting results on the image ofpeppers. For the MS model approach, after segmen-tation of the image, we can detect the boundariesof objects in the inpainting area, and the edges ofobjects are kept very well. In the detailed images(c) and (f), it is obvious that the inpainting resultof the MS model approach is better than that of theTV approach. Fig. 2 shows the inpainting resultson an artificial image. In addition to the TV ap-proach and the MS model approach of hierarchicalmulti level set functions, we also show the inpaint-

Proceedings of the 3rd Canadian Conference on Computer and Robot Vision (CRV’06) 0-7695-2542-3/06 $20.00 © 2006 IEEE

(a) image with inpainting lines (b) inpainting result with TV (c) detail of (b)

(d) segmentation result with MS (e) inpainting result with MS (f) detail of (e)

Figure 1: The inpainting results on the image of peppers.

(a) image with inpainting lines(b) inpainting result with TV(c) detail of (b)

(a) (b) (c)

(d) segmentation result with MSmodel of 1 level function(e) inpainting result with MS modelof 1 level function(f) detail of (e)

(d) (e) (f)

(g) segmentation result with MS of hi-erarchical multi-level functions(h) inpainting result with MS of hier-archical multi-level functions(i) detail of (h)

(g) (h) (i)

Figure 2: The inpainting results of an artificial image.

Proceedings of the 3rd Canadian Conference on Computer and Robot Vision (CRV’06) 0-7695-2542-3/06 $20.00 © 2006 IEEE

ing result with the MS model approach of one levelset function. Because one level set function cannotdetect all the boundaries, many edges are blurred inthe inpainting area. Our hierarchical segmentationand inpainting approach can detect all the bound-aries, and keep all the edges clear in the inpaintingresult.

Fig. 3 is the inpainting results on the image ofa copy machine. Compared with the result of onelevel set function, the inpainting result of hierarchicalmulti level set functions is much better. It can keepthe edges of the paper on the table clear. Fig. 4 is theinpainting results on the image of a street scene. Be-cause the segmentation with hierarchical multi levelset functions can detect more edges than the segmen-tation with only one level set function, more edgesinside the inpainting area can be preserved. Whilethe edge of the car is blurred in detailed images (c)and (f), the edge is preserved in image (i).

5 Conclusions

Compared to some other inpainting approaches, theMS model approach can detect and preserve objectedges in the inpainting areas. In previous works ofthe MS model, only one or two level set functionsare used to segment an image. While this approachworks well on some simple images, it cannot de-tect detailed edges on complicated images. Althoughmulti-level set functions can be used to segment animage into many regions, traditional approaches re-quire extensive computations and depend on the ini-tial conditions. In this paper, we have presented ahierarchical segmentation and inpainting approachthat can keep the detailed edges in the inpaintingareas. The experimental results demonstrate the ad-vantages of this approach.

Acknowledgements: This work was supported byresearch grants from the Natural Sciences and Engi-neering Research Council of Canada.

References

[1] M. Bertalmio, G. Sapiro, V. Caselles, andC. Ballester. Image inpainting. In ACM In-ternational Conference on Computer Graph-ics and Interactive Techniques (SIGGRAPH),pages 417–424, 2000.

[2] M. Bertalmio, L. Vese, G. Sapiro, and S. Os-her. Simultaneous structure and texture imageinpainting. IEEE Transactions on Image Pro-cessing, 12(8):882–889, 2003.

[3] T. F. Chan and J. Shen. Mathematical modelsfor local non-texture inpainting. SIAM J. Appl.Math., 62(3):1019–1043, 2002.

[4] T. F. Chan and J. Shen. Image ProcessingAnd Analysis: Variational, PDE, Wavelet, AndStochastic Methods. SIAM, Philadelphia, 2005.

[5] T. F. Chan and L. A. Vese. Active contourswithout edges. IEEE Transactions on ImageProcessing, 10(2):266–277, 2001.

[6] A. Criminisi, P. Perez, and K. Toyama. Regionfilling and object removal by exemplar-basedimage inpainting. IEEE Transactions on ImageProcessing, 13(9):1200–1212, 2004.

[7] X. J. Du and T. D. Bui. A new hierarchicalimage segmentation scheme. submitted to ICPR2006.

[8] S. Gao and T.D. Bui. Image segmentationand selective smoothing by using Mumford-Shah model. IEEE Transactions on Image Pro-cessing, 14(10):1537–1549, Oct 2005.

[9] D. Mumford and J. Shah. Optimal approxima-tion by piecewise smooth functionals and asso-ciated variational problems. Comm. Pure Appl.Math., 42:577–685, 1989.

[10] M. M. Oliveira, B. Bowen, R. McKenna, and Y.-S. Chang. Fast digital image inpainting. In In-ternational Conference on Visualization, Imag-ing and Image Processing, pages 261–266, 2001.

[11] J. A. Sethian. Level set methods and fast march-ing methods. Cambridge University Press, 1999.

[12] A. Tsai, A. Yezzi, and A. S. Willsky. Curveevolution implementation of the Mumford-Shahfunctionalal for image segmentation, denoising,interpolation, and magnification. IEEE Trans-actions on Image Processing, 10(8):1169–1186,2001.

[13] L. Vese and T. F. Chan. A multiphase levelset framework for image segmentation using theMumford and Shah model. International Jour-nal of Computer Vision, 50(3):271–293, 2002.

Proceedings of the 3rd Canadian Conference on Computer and Robot Vision (CRV’06) 0-7695-2542-3/06 $20.00 © 2006 IEEE

(a) image with inpainting lines (b) inpainting result with TV

(c) segmentation result with MS of 1 level function (d) inpainting result with MS of 1 level function

(e) segmentation result with MS of hierarchicalmulti-level functions

(f) inpainting result with MS of hierarchical multi-level functions

Figure 3: The inpainting results on the image of a copy machine.

Proceedings of the 3rd Canadian Conference on Computer and Robot Vision (CRV’06) 0-7695-2542-3/06 $20.00 © 2006 IEEE

(a) image with inpainting lines (b) inpainting result with TV (c) detail of (b)

(d) segmentation result with MS (e) inpainting result with MS (f) detail of (e)

(g) segmentation result with MS ofhierarchical multi-level functions

(h) inpainting result with MS ofhierarchical multi-level functions

(i) detail of (h)

Figure 4: The inpainting results on the image of euroexpress.

Proceedings of the 3rd Canadian Conference on Computer and Robot Vision (CRV’06) 0-7695-2542-3/06 $20.00 © 2006 IEEE