-
Examplar-basedinpainting
O. Le Meur
Inpainting:context andissues
Examplar-basedinpaintingPresentationNotationCriminisi et
al.smethod
Variants ofCriminisismethodFilling ordercomputationTexture
synthesisSome examplesLimitations
Super-resolution-based inpaintingmethodProposed approachMore
than oneinpaintingLoopy BeliefPropagationSuper-resolution
Results andcomparison withexisting methodsResultsComparison
withexisting methods
Conclusion
Examplar-based inpainting
Olivier Le [email protected]
IRISA - University of Rennes 1
June 19, 20141 / 44
-
Examplar-basedinpainting
O. Le Meur
Inpainting:context andissues
Examplar-basedinpaintingPresentationNotationCriminisi et
al.smethod
Variants ofCriminisismethodFilling ordercomputationTexture
synthesisSome examplesLimitations
Super-resolution-based inpaintingmethodProposed approachMore
than oneinpaintingLoopy BeliefPropagationSuper-resolution
Results andcomparison withexisting methodsResultsComparison
withexisting methods
Conclusion
Inpainting: context and issues (1/3)
This talk is about inpainting. We will heavily rely upon these
papers:
C. Guillemot & O. Le Meur, Image inpainting: overviewand
recent advances, IEEE Signal Processing Magazine,Vol. 1, pp.
127-144, 2014.
O. Le Meur, M. Ebdelli and C. Guillemot,
Hierarchicalsuper-resolution-based inpainting, IEEE TIP,
vol.22(10), pp. 3779-3790, 2013.
O. Le Meur & C. Guillemot, Super-resolution-basedinpainting,
ECCV 2012.
2 / 44
-
Examplar-basedinpainting
O. Le Meur
Inpainting:context andissues
Examplar-basedinpaintingPresentationNotationCriminisi et
al.smethod
Variants ofCriminisismethodFilling ordercomputationTexture
synthesisSome examplesLimitations
Super-resolution-based inpaintingmethodProposed approachMore
than oneinpaintingLoopy BeliefPropagationSuper-resolution
Results andcomparison withexisting methodsResultsComparison
withexisting methods
Conclusion
Inpainting: context and issues (2/3)
InpaintingInpainting corresponds to filling holes (i.e. missing
areas) in im-ages (Bertalmio et al., 2000).
Let be an image I defined as
I : Rn 7 Rm
Let be a degradation operator M
M : 7 {0, 1}
M (x) ={
0, if x U1, otherwise
Let F the observed image:
F = M I
n = 2 for a 2D imagem = 3 for (R,G,B) image
= S U, S being the known part
of I U the unknown part of I
is the Hadamard product
3 / 44
-
Examplar-basedinpainting
O. Le Meur
Inpainting:context andissues
Examplar-basedinpaintingPresentationNotationCriminisi et
al.smethod
Variants ofCriminisismethodFilling ordercomputationTexture
synthesisSome examplesLimitations
Super-resolution-based inpaintingmethodProposed approachMore
than oneinpaintingLoopy BeliefPropagationSuper-resolution
Results andcomparison withexisting methodsResultsComparison
withexisting methods
Conclusion
Inpainting: context and issues (3/3)
Different configurations according to the definition of M :
Original image80% of the pixels
have beenremoved.
damaged portionsin black, scratches object removal
Sparsity andlow-rank methods
Diffusion-basedmethods
Examplar-basedmethods
4 / 44
-
Examplar-basedinpainting
O. Le Meur
Inpainting:context andissues
Examplar-basedinpaintingPresentationNotationCriminisi et
al.smethod
Variants ofCriminisismethodFilling ordercomputationTexture
synthesisSome examplesLimitations
Super-resolution-based inpaintingmethodProposed approachMore
than oneinpaintingLoopy BeliefPropagationSuper-resolution
Results andcomparison withexisting methodsResultsComparison
withexisting methods
Conclusion
Outline of the presentation
1 Inpainting: context and issues
2 Examplar-based inpainting
3 Variants of Criminisis method
4 Super-resolution-based inpainting method
5 Results and comparison with existing methods
6 Conclusion
5 / 44
-
Examplar-basedinpainting
O. Le Meur
Inpainting:context andissues
Examplar-basedinpaintingPresentationNotationCriminisi et
al.smethod
Variants ofCriminisismethodFilling ordercomputationTexture
synthesisSome examplesLimitations
Super-resolution-based inpaintingmethodProposed approachMore
than oneinpaintingLoopy BeliefPropagationSuper-resolution
Results andcomparison withexisting methodsResultsComparison
withexisting methods
Conclusion
Outline of the presentation
1 Inpainting: context and issues
2 Examplar-based inpaintingI PresentationI NotationI Criminisi
et al.s method
3 Variants of Criminisis method
4 Super-resolution-based inpainting method
5 Results and comparison with existing methods
6 Conclusion
6 / 44
-
Examplar-basedinpainting
O. Le Meur
Inpainting:context andissues
Examplar-basedinpaintingPresentationNotationCriminisi et
al.smethod
Variants ofCriminisismethodFilling ordercomputationTexture
synthesisSome examplesLimitations
Super-resolution-based inpaintingmethodProposed approachMore
than oneinpaintingLoopy BeliefPropagationSuper-resolution
Results andcomparison withexisting methodsResultsComparison
withexisting methods
Conclusion
Examplar-based inpainting (1/4)
Texture synthesisExamplar-based inpainting methods rely on the
assumption that theknown part of the image provides a good
dictionary which could beused efficiently to restore the unknown
part (Efros and Leung, 1999).
The recovered texture is thereforelearned from similar regions.
This can be done simply by
sampling, copying orcombining patches from theknown part of the
image;
Template Matching Patches are then stitched
together to fill in the missingarea.
7 / 44
-
Examplar-basedinpainting
O. Le Meur
Inpainting:context andissues
Examplar-basedinpaintingPresentationNotationCriminisi et
al.smethod
Variants ofCriminisismethodFilling ordercomputationTexture
synthesisSome examplesLimitations
Super-resolution-based inpaintingmethodProposed approachMore
than oneinpaintingLoopy BeliefPropagationSuper-resolution
Results andcomparison withexisting methodsResultsComparison
withexisting methods
Conclusion
Examplar-based inpainting (2/4)
Notations: a patch px is a discretized
N N neighborhoodcentered on the pixel px .This patch can be
vectorizedin a raster-scan order as amN 2-dimensional vector;
ukpx denotes the unknownpixels of the patch;
kpx denotes its knownpixels;
px(i) denotes the ithnearest neighbour of px ;
U is the front line;
8 / 44
-
Examplar-basedinpainting
O. Le Meur
Inpainting:context andissues
Examplar-basedinpaintingPresentationNotationCriminisi et
al.smethod
Variants ofCriminisismethodFilling ordercomputationTexture
synthesisSome examplesLimitations
Super-resolution-based inpaintingmethodProposed approachMore
than oneinpaintingLoopy BeliefPropagationSuper-resolution
Results andcomparison withexisting methodsResultsComparison
withexisting methods
Conclusion
Examplar-based inpainting (3/4)
Criminisi et al.s algorithmCriminisi et al. (Criminisi et al.,
2004) has brought a new momentumto inpainting applications and
methods. They proposed a new methodbased on two sequential
stages:
1 Filling order computation;2 Texture synthesis.
1 Filling order computation: P(px) = C (px)D(px)Confidence
term
C (px) =
qkpx C (q)|px |
where |px | is the area of px .
Data term
D(px) =|I(px) npx |
where is a normalizationconstant in order to ensure thatD(px) is
in the range 0 to 1.
9 / 44
-
Examplar-basedinpainting
O. Le Meur
Inpainting:context andissues
Examplar-basedinpaintingPresentationNotationCriminisi et
al.smethod
Variants ofCriminisismethodFilling ordercomputationTexture
synthesisSome examplesLimitations
Super-resolution-based inpaintingmethodProposed approachMore
than oneinpaintingLoopy BeliefPropagationSuper-resolution
Results andcomparison withexisting methodsResultsComparison
withexisting methods
Conclusion
Examplar-based inpainting (4/4)
2 Texture synthesis:
A template matching is performed within a local
neighborhood:
py = arg minqW d(kpq ,
kpx )
W S is the window search; kpx are the known pixels of the patch
px with the highest
priority; kpy are the known pixels of the nearest patch
neighbor; d(a, b) is the sum of squared differences between patches
a and
b.
The pixels of the patch ukpy are then copied into the unknown
pixelsof the patch px .
10 / 44
-
Examplar-basedinpainting
O. Le Meur
Inpainting:context andissues
Examplar-basedinpaintingPresentationNotationCriminisi et
al.smethod
Variants ofCriminisismethodFilling ordercomputationTexture
synthesisSome examplesLimitations
Super-resolution-based inpaintingmethodProposed approachMore
than oneinpaintingLoopy BeliefPropagationSuper-resolution
Results andcomparison withexisting methodsResultsComparison
withexisting methods
Conclusion
Outline of the presentation
1 Inpainting: context and issues
2 Examplar-based inpainting
3 Variants of Criminisis methodI Filling order computationI
Texture synthesisI Some examplesI Limitations
4 Super-resolution-based inpainting method
5 Results and comparison with existing methods
6 Conclusion11 / 44
-
Examplar-basedinpainting
O. Le Meur
Inpainting:context andissues
Examplar-basedinpaintingPresentationNotationCriminisi et
al.smethod
Variants ofCriminisismethodFilling ordercomputationTexture
synthesisSome examplesLimitations
Super-resolution-based inpaintingmethodProposed approachMore
than oneinpaintingLoopy BeliefPropagationSuper-resolution
Results andcomparison withexisting methodsResultsComparison
withexisting methods
Conclusion
Filling order computation (1/4)
P(px) = C (px)D(px)
Two variants are here presented: Tensor-based data term (Le Meur
et al., 2011);
Sparsity-based data term (Xu and Sun, 2010).
Many others: edge-based data term, transformation of the data
termin a nonlinear fashion, entropy-based data term...
12 / 44
-
Examplar-basedinpainting
O. Le Meur
Inpainting:context andissues
Examplar-basedinpaintingPresentationNotationCriminisi et
al.smethod
Variants ofCriminisismethodFilling ordercomputationTexture
synthesisSome examplesLimitations
Super-resolution-based inpaintingmethodProposed approachMore
than oneinpaintingLoopy BeliefPropagationSuper-resolution
Results andcomparison withexisting methodsResultsComparison
withexisting methods
Conclusion
Filling order computation (2/4)
Tensor-based data termInstead of using the gradient, (Le Meur et
al., 2011) used the structuretensor which is more robust:
D(px) = + (1 )exp( (1 2)2
)where is a positive value and [0, 1].The structure tensor is a
symmetric, positive semi-definitematrix (Weickert, 1999):
J, [I ] = K ( mi=1(Ii K)(Ii K)T
)
where Ka is a Gaussian kernel with a standard deviation a.
Theparameters and are called integration scale and noise
scale,respectively.
13 / 44
-
Examplar-basedinpainting
O. Le Meur
Inpainting:context andissues
Examplar-basedinpaintingPresentationNotationCriminisi et
al.smethod
Variants ofCriminisismethodFilling ordercomputationTexture
synthesisSome examplesLimitations
Super-resolution-based inpaintingmethodProposed approachMore
than oneinpaintingLoopy BeliefPropagationSuper-resolution
Results andcomparison withexisting methodsResultsComparison
withexisting methods
Conclusion
Filling order computation (3/4)
D(px) = + (1 )exp( (1 2)2
)
When 1 ' 2, the data term tends to . It tends to 1 when1
>> 2.
14 / 44
-
Examplar-basedinpainting
O. Le Meur
Inpainting:context andissues
Examplar-basedinpaintingPresentationNotationCriminisi et
al.smethod
Variants ofCriminisismethodFilling ordercomputationTexture
synthesisSome examplesLimitations
Super-resolution-based inpaintingmethodProposed approachMore
than oneinpaintingLoopy BeliefPropagationSuper-resolution
Results andcomparison withexisting methodsResultsComparison
withexisting methods
Conclusion
Filling order computation (4/4)
Sparsity-based data termSparsity-based data term (Xu and Sun,
2010) is based on the sparse-ness of nonzero patch
similarities:
D(px) =
|Ns(px)||N (px)| pjWs w2px ,pjwhere Ns and N are the numbers of
valid and candidate patches inthe search window.Weight wpx ,pj is
proportional to the similarity between the two patchescentered on
px and pj (
j wpx ,pj = 1).
A large value of the structure sparsity term means sparse
similaritywith neighboring patches a good confidence that the input
patch is on some structure.
15 / 44
-
Examplar-basedinpainting
O. Le Meur
Inpainting:context andissues
Examplar-basedinpaintingPresentationNotationCriminisi et
al.smethod
Variants ofCriminisismethodFilling ordercomputationTexture
synthesisSome examplesLimitations
Super-resolution-based inpaintingmethodProposed approachMore
than oneinpaintingLoopy BeliefPropagationSuper-resolution
Results andcomparison withexisting methodsResultsComparison
withexisting methods
Conclusion
Texture synthesis (1/4)
Texture synthesis with more than one candidateFrom K patches
px(i) which are the most similar to the known partkpx of the input
patch, the unknown part of the patch to be filled ukpxis then
obtained by a linear combination of the sub-patches ukpx(i) .
ukpx =Ki=1
wiukpx(i)
How can we compute the weightswi of this linear combination?
Note: K is locally adjusted by usingan -ball including patches
within acertain radius.
16 / 44
-
Examplar-basedinpainting
O. Le Meur
Inpainting:context andissues
Examplar-basedinpaintingPresentationNotationCriminisi et
al.smethod
Variants ofCriminisismethodFilling ordercomputationTexture
synthesisSome examplesLimitations
Super-resolution-based inpaintingmethodProposed approachMore
than oneinpaintingLoopy BeliefPropagationSuper-resolution
Results andcomparison withexisting methodsResultsComparison
withexisting methods
Conclusion
Texture synthesis (2/4)
ukpx =Ki=1
wiukpx(i)
Different solutions exist (Guillemot et al., 2013): Average
template matching: wi = 1K , i; Non-local means approach (Buades et
al., 2005):
wi = exp(d(pkx , pkx(i))
h2
)
Least-square method minimizing
E(w) = kpx Aw22,aw = arg min
wE(w)
17 / 44
-
Examplar-basedinpainting
O. Le Meur
Inpainting:context andissues
Examplar-basedinpaintingPresentationNotationCriminisi et
al.smethod
Variants ofCriminisismethodFilling ordercomputationTexture
synthesisSome examplesLimitations
Super-resolution-based inpaintingmethodProposed approachMore
than oneinpaintingLoopy BeliefPropagationSuper-resolution
Results andcomparison withexisting methodsResultsComparison
withexisting methods
Conclusion
Texture synthesis (3/4)
ukpx =Ki=1
wiukpx(i)
Constrained Least-square optimization with the
sum-to-oneconstraint of the weight vector LLE method (Saul
andRoweis, 2003)
E(w) = kpx Aw22,aw = arg min
wE(w) s.t. wT1K = 1
Constrained Least-square optimization with positive weights NMF
method (Lee and Seung, 2001)
w = arg minw
E(w) s.t. wi 0
18 / 44
-
Examplar-basedinpainting
O. Le Meur
Inpainting:context andissues
Examplar-basedinpaintingPresentationNotationCriminisi et
al.smethod
Variants ofCriminisismethodFilling ordercomputationTexture
synthesisSome examplesLimitations
Super-resolution-based inpaintingmethodProposed approachMore
than oneinpaintingLoopy BeliefPropagationSuper-resolution
Results andcomparison withexisting methodsResultsComparison
withexisting methods
Conclusion
Texture synthesis (4/4)
Similarity metrics: Using a Gaussian weighted Euclidean
distance
dL2(px , py ) = px py22,awhere a controls the decay of the
Gaussian functiong(k) = e
|k|2a2 , a > 0;
A better distance introduced in (Bugeau et al., 2010, Le Meurand
Guillemot, 2012):
d(px , py ) = dL2(px , py ) (1 + dH (px , py ))where dH (px , py
) is the Hellinger distance
dH (px , py ) =
1k
p1(k)p2(k)
where p1 and p2 represent the histograms of patches px , py
,respectively.
19 / 44
-
Examplar-basedinpainting
O. Le Meur
Inpainting:context andissues
Examplar-basedinpaintingPresentationNotationCriminisi et
al.smethod
Variants ofCriminisismethodFilling ordercomputationTexture
synthesisSome examplesLimitations
Super-resolution-based inpaintingmethodProposed approachMore
than oneinpaintingLoopy BeliefPropagationSuper-resolution
Results andcomparison withexisting methodsResultsComparison
withexisting methods
Conclusion
Some Examples (1/2)
Inpainted pictures with (Criminisi et al., 2004)s method
(Courtesy ofP. Perez):
20 / 44
-
Some Examples (2/2)
Results from (Le Meur et al., 2011).
-
Examplar-basedinpainting
O. Le Meur
Inpainting:context andissues
Examplar-basedinpaintingPresentationNotationCriminisi et
al.smethod
Variants ofCriminisismethodFilling ordercomputationTexture
synthesisSome examplesLimitations
Super-resolution-based inpaintingmethodProposed approachMore
than oneinpaintingLoopy BeliefPropagationSuper-resolution
Results andcomparison withexisting methodsResultsComparison
withexisting methods
Conclusion
Limitations
Very sensitive to the parameter settings such as the filling
orderand the patch size:
Examplar-based methods are a one-pass greedy algorithms.
22 / 44
-
Examplar-basedinpainting
O. Le Meur
Inpainting:context andissues
Examplar-basedinpaintingPresentationNotationCriminisi et
al.smethod
Variants ofCriminisismethodFilling ordercomputationTexture
synthesisSome examplesLimitations
Super-resolution-based inpaintingmethodProposed approachMore
than oneinpaintingLoopy BeliefPropagationSuper-resolution
Results andcomparison withexisting methodsResultsComparison
withexisting methods
Conclusion
Outline of the presentation
1 Inpainting: context and issues
2 Examplar-based inpainting
3 Variants of Criminisis method
4 Super-resolution-based inpainting methodI Proposed approachI
More than one inpaintingI Loopy Belief PropagationI
Super-resolution
5 Results and comparison with existing methods
6 Conclusion23 / 44
-
Examplar-basedinpainting
O. Le Meur
Inpainting:context andissues
Examplar-basedinpaintingPresentationNotationCriminisi et
al.smethod
Variants ofCriminisismethodFilling ordercomputationTexture
synthesisSome examplesLimitations
Super-resolution-based inpaintingmethodProposed approachMore
than oneinpaintingLoopy BeliefPropagationSuper-resolution
Results andcomparison withexisting methodsResultsComparison
withexisting methods
Conclusion
Proposed approach (1/1)
Objectives of the proposed methodWe apply an examplar-based
inpainting algorithm several times andfuse together the inpainted
results.
less sensitive to the inpainting setting;relax the greedy
constraint.
The inpainting method is applied on a coarse version of the
inputpicture:
less demanding of computational resources;less sensitive to
noise;K candidates for the texture synthesis without introducing
blur.
Need to fuse the inpainted images and to retrieve the
highestfrequencies
Loopy Belief Propagation and Super-Resolution algorithms.24 /
44
-
Examplar-basedinpainting
O. Le Meur
Inpainting:context andissues
Examplar-basedinpaintingPresentationNotationCriminisi et
al.smethod
Variants ofCriminisismethodFilling ordercomputationTexture
synthesisSome examplesLimitations
Super-resolution-based inpaintingmethodProposed approachMore
than oneinpaintingLoopy BeliefPropagationSuper-resolution
Results andcomparison withexisting methodsResultsComparison
withexisting methods
Conclusion
More than one inpainting (1/1)
The baseline algorithm is anexamplar-based method: Filling
order
computation; Texture synthesis.
Decimation factor n = 3 13 sets of parameters
Table: Thirteen inpainting configurations.Setting Parameters
1Patchs size 5 5
Decimation factor n = 3Search window 80 80Sparsity-based filling
order
2 default + rotation by 180 degrees3 default + patchs size 7 74
default + rotation by 180 degrees+ patchs size 7 75 default +
patchs size 11 116 default + rotation by 180 degrees+ patchs size
11 117 default + patchs size 9 98 default + rotation by 180
degrees+ patchs size 9 99 default + patchs size 9 9+ Tensor-based
filling order
10 default + patchs size 7 7+ Tensor-based filling order11
default + patchs size 5 5+ Tensor-based filling order12 default +
patchs size 11 11+ Tensor-based filling order
13default + rotation by 180 degrees
+ patchs size 9 9+ Tensor-based filling order
25 / 44
-
Examplar-basedinpainting
O. Le Meur
Inpainting:context andissues
Examplar-basedinpaintingPresentationNotationCriminisi et
al.smethod
Variants ofCriminisismethodFilling ordercomputationTexture
synthesisSome examplesLimitations
Super-resolution-based inpaintingmethodProposed approachMore
than oneinpaintingLoopy BeliefPropagationSuper-resolution
Results andcomparison withexisting methodsResultsComparison
withexisting methods
Conclusion
Loopy Belief Propagation (1/5)
. . . . . .Loopy Belief Propagation is used to fuse together the
13 inpainted
images.
Let be a finite set of labels L composed of M = 13 values.
E(l) =p
Vd(lp) +
(n,m)N4Vs(ln, lm)
where, lp the label of pixel px , represents the pixel in U and
N4 isa neighbourhood system. is a weighting factor.
26 / 44
-
Examplar-basedinpainting
O. Le Meur
Inpainting:context andissues
Examplar-basedinpaintingPresentationNotationCriminisi et
al.smethod
Variants ofCriminisismethodFilling ordercomputationTexture
synthesisSome examplesLimitations
Super-resolution-based inpaintingmethodProposed approachMore
than oneinpaintingLoopy BeliefPropagationSuper-resolution
Results andcomparison withexisting methodsResultsComparison
withexisting methods
Conclusion
Loopy Belief Propagation (2/5)
E(l) =p
Vd(lp) +
(n,m)N4Vs(ln, lm)
Vd(lp) represents the cost of assigning a label lp to a pixel px
:
Vd(lp) =nL
u
{I (l)(x + u) I (n)(x + u)
}2 Vs(ln, lm) is the discontinuity cost:
Vs(ln, lm) = (ln lm)2
The minimization is performed iteratively (less than
15iterations) (Boykov and Kolmogorov, 2004, Boykov et al.,
2001,Yedidia et al., 2005).
27 / 44
-
Examplar-basedinpainting
O. Le Meur
Inpainting:context andissues
Examplar-basedinpaintingPresentationNotationCriminisi et
al.smethod
Variants ofCriminisismethodFilling ordercomputationTexture
synthesisSome examplesLimitations
Super-resolution-based inpaintingmethodProposed approachMore
than oneinpaintingLoopy BeliefPropagationSuper-resolution
Results andcomparison withexisting methodsResultsComparison
withexisting methods
Conclusion
Loopy Belief Propagation (3/5)
LBP convergence: 13 inpainted image in
input; 25 iterations; resolution=80 120.
28 / 44
-
Examplar-basedinpainting
O. Le Meur
Inpainting:context andissues
Examplar-basedinpaintingPresentationNotationCriminisi et
al.smethod
Variants ofCriminisismethodFilling ordercomputationTexture
synthesisSome examplesLimitations
Super-resolution-based inpaintingmethodProposed approachMore
than oneinpaintingLoopy BeliefPropagationSuper-resolution
Results andcomparison withexisting methodsResultsComparison
withexisting methods
Conclusion
Loopy Belief Propagation (4/5)
LBP convergence: 13 inpainted image in
input; 25 iterations; resolution=120 80.
29 / 44
-
Examplar-basedinpainting
O. Le Meur
Inpainting:context andissues
Examplar-basedinpaintingPresentationNotationCriminisi et
al.smethod
Variants ofCriminisismethodFilling ordercomputationTexture
synthesisSome examplesLimitations
Super-resolution-based inpaintingmethodProposed approachMore
than oneinpaintingLoopy BeliefPropagationSuper-resolution
Results andcomparison withexisting methodsResultsComparison
withexisting methods
Conclusion
Loopy Belief Propagation (5/5)
LBP convergence: 13 inpainted image in
input; 25 iterations; resolution=200 135.
30 / 44
-
Examplar-basedinpainting
O. Le Meur
Inpainting:context andissues
Examplar-basedinpaintingPresentationNotationCriminisi et
al.smethod
Variants ofCriminisismethodFilling ordercomputationTexture
synthesisSome examplesLimitations
Super-resolution-based inpaintingmethodProposed approachMore
than oneinpaintingLoopy BeliefPropagationSuper-resolution
Results andcomparison withexisting methodsResultsComparison
withexisting methods
Conclusion
Super-resolution (1/2)
For the LR patch correspondingto the HR patch having thehighest
priority: We look for its best
neighbour; Only the best candidate is
kept; The corresponding HR
patch is simply deduced. Its pixel values are then
copied into the unknownparts of the current HRpatch.
31 / 44
-
Examplar-basedinpainting
O. Le Meur
Inpainting:context andissues
Examplar-basedinpaintingPresentationNotationCriminisi et
al.smethod
Variants ofCriminisismethodFilling ordercomputationTexture
synthesisSome examplesLimitations
Super-resolution-based inpaintingmethodProposed approachMore
than oneinpaintingLoopy BeliefPropagationSuper-resolution
Results andcomparison withexisting methodsResultsComparison
withexisting methods
Conclusion
Super-resolution (2/2)
To speed-up the process, we can perform thesearch:
within a search window;
within a dictionary (as illustrated on theright) composed of LR
patches withtheir corresponding HR patches.
32 / 44
-
Examplar-basedinpainting
O. Le Meur
Inpainting:context andissues
Examplar-basedinpaintingPresentationNotationCriminisi et
al.smethod
Variants ofCriminisismethodFilling ordercomputationTexture
synthesisSome examplesLimitations
Super-resolution-based inpaintingmethodProposed approachMore
than oneinpaintingLoopy BeliefPropagationSuper-resolution
Results andcomparison withexisting methodsResultsComparison
withexisting methods
Conclusion
Outline of the presentation
1 Inpainting: context and issues
2 Examplar-based inpainting
3 Variants of Criminisis method
4 Super-resolution-based inpainting method
5 Results and comparison with existing methodsI ResultsI
Comparison with existing methods
6 Conclusion
33 / 44
-
Examplar-basedinpainting
O. Le Meur
Inpainting:context andissues
Examplar-basedinpaintingPresentationNotationCriminisi et
al.smethod
Variants ofCriminisismethodFilling ordercomputationTexture
synthesisSome examplesLimitations
Super-resolution-based inpaintingmethodProposed approachMore
than oneinpaintingLoopy BeliefPropagationSuper-resolution
Results andcomparison withexisting methodsResultsComparison
withexisting methods
Conclusion
Results (1/4)
34 / 44
-
Examplar-basedinpainting
O. Le Meur
Inpainting:context andissues
Examplar-basedinpaintingPresentationNotationCriminisi et
al.smethod
Variants ofCriminisismethodFilling ordercomputationTexture
synthesisSome examplesLimitations
Super-resolution-based inpaintingmethodProposed approachMore
than oneinpaintingLoopy BeliefPropagationSuper-resolution
Results andcomparison withexisting methodsResultsComparison
withexisting methods
Conclusion
Results (2/4)
35 / 44
-
Examplar-basedinpainting
O. Le Meur
Inpainting:context andissues
Examplar-basedinpaintingPresentationNotationCriminisi et
al.smethod
Variants ofCriminisismethodFilling ordercomputationTexture
synthesisSome examplesLimitations
Super-resolution-based inpaintingmethodProposed approachMore
than oneinpaintingLoopy BeliefPropagationSuper-resolution
Results andcomparison withexisting methodsResultsComparison
withexisting methods
Conclusion
Results (3/4)
36 / 44
-
Examplar-basedinpainting
O. Le Meur
Inpainting:context andissues
Examplar-basedinpaintingPresentationNotationCriminisi et
al.smethod
Variants ofCriminisismethodFilling ordercomputationTexture
synthesisSome examplesLimitations
Super-resolution-based inpaintingmethodProposed approachMore
than oneinpaintingLoopy BeliefPropagationSuper-resolution
Results andcomparison withexisting methodsResultsComparison
withexisting methods
Conclusion
Results (4/4)
37 / 44
-
Examplar-basedinpainting
O. Le Meur
Inpainting:context andissues
Examplar-basedinpaintingPresentationNotationCriminisi et
al.smethod
Variants ofCriminisismethodFilling ordercomputationTexture
synthesisSome examplesLimitations
Super-resolution-based inpaintingmethodProposed approachMore
than oneinpaintingLoopy BeliefPropagationSuper-resolution
Results andcomparison withexisting methodsResultsComparison
withexisting methods
Conclusion
Comparison with existing methods (1/5)
Three methods have been tested:
[Komodakis] N. Komodakis, and G. Tziritas, Image Completionusing
Global Optimization. in CVPR 2007 (Komodakis andTziritas,
2007);
[Pritch] Y. Pritch, E. Kav-Venaki, S. Peleg, Shift-Map
ImageEditing. in ICCV 2009 (Pritch et al., 2009);
[He] K. He and J. Sun, Statistics of Patch Offsets for
ImageCompletion. in ECCV 2012 (He and Sun, 2012).
38 / 44
-
Examplar-basedinpainting
O. Le Meur
Inpainting:context andissues
Examplar-basedinpaintingPresentationNotationCriminisi et
al.smethod
Variants ofCriminisismethodFilling ordercomputationTexture
synthesisSome examplesLimitations
Super-resolution-based inpaintingmethodProposed approachMore
than oneinpaintingLoopy BeliefPropagationSuper-resolution
Results andcomparison withexisting methodsResultsComparison
withexisting methods
Conclusion
Comparison with existing methods (2/5)
From left to right: Komodakis, Pritch, He, Ours.39 / 44
-
Examplar-basedinpainting
O. Le Meur
Inpainting:context andissues
Examplar-basedinpaintingPresentationNotationCriminisi et
al.smethod
Variants ofCriminisismethodFilling ordercomputationTexture
synthesisSome examplesLimitations
Super-resolution-based inpaintingmethodProposed approachMore
than oneinpaintingLoopy BeliefPropagationSuper-resolution
Results andcomparison withexisting methodsResultsComparison
withexisting methods
Conclusion
Comparison with existing methods (3/5)
From left to right: Komodakis, Pritch, He, Ours.40 / 44
-
Examplar-basedinpainting
O. Le Meur
Inpainting:context andissues
Examplar-basedinpaintingPresentationNotationCriminisi et
al.smethod
Variants ofCriminisismethodFilling ordercomputationTexture
synthesisSome examplesLimitations
Super-resolution-based inpaintingmethodProposed approachMore
than oneinpaintingLoopy BeliefPropagationSuper-resolution
Results andcomparison withexisting methodsResultsComparison
withexisting methods
Conclusion
Comparison with existing methods (4/5)
Much more results on the
link:http://people.irisa.fr/Olivier.Le_Meur/publi/2013_TIP/
indexSoA.html
41 / 44
-
Examplar-basedinpainting
O. Le Meur
Inpainting:context andissues
Examplar-basedinpaintingPresentationNotationCriminisi et
al.smethod
Variants ofCriminisismethodFilling ordercomputationTexture
synthesisSome examplesLimitations
Super-resolution-based inpaintingmethodProposed approachMore
than oneinpaintingLoopy BeliefPropagationSuper-resolution
Results andcomparison withexisting methodsResultsComparison
withexisting methods
Conclusion
Comparison with existing methods (5/5)
Limitations and failure cases:
From left to right: original, Hes method and proposed one.
No semantic information are used... No objective quality
metric.
42 / 44
-
Examplar-basedinpainting
O. Le Meur
Inpainting:context andissues
Examplar-basedinpaintingPresentationNotationCriminisi et
al.smethod
Variants ofCriminisismethodFilling ordercomputationTexture
synthesisSome examplesLimitations
Super-resolution-based inpaintingmethodProposed approachMore
than oneinpaintingLoopy BeliefPropagationSuper-resolution
Results andcomparison withexisting methodsResultsComparison
withexisting methods
Conclusion
Outline of the presentation
1 Inpainting: context and issues
2 Examplar-based inpainting
3 Variants of Criminisis method
4 Super-resolution-based inpainting method
5 Results and comparison with existing methods
6 Conclusion
43 / 44
-
Examplar-basedinpainting
O. Le Meur
Inpainting:context andissues
Examplar-basedinpaintingPresentationNotationCriminisi et
al.smethod
Variants ofCriminisismethodFilling ordercomputationTexture
synthesisSome examplesLimitations
Super-resolution-based inpaintingmethodProposed approachMore
than oneinpaintingLoopy BeliefPropagationSuper-resolution
Results andcomparison withexisting methodsResultsComparison
withexisting methods
Conclusion
Conclusion
A new framework to perform inpainting of still color
pictures:coarse inpainting + super-resolution.Binary file could be
downloaded:http://people.irisa.fr/Olivier.Le_Meur/publi/2013_TIP/index.html
A natural extension is to deal with video inpainting.A paper
dealing with video inpainting under revision in IEEE TIP.
44 / 44
-
Examplar-basedinpainting
O. Le Meur
References
ReferencesM. Bertalmio, G. Sapiro, V. Caselles, and C.
Ballester. Image inpainting. In SIGGRPAH 2000, 2000.Y. Boykov and
V. Kolmogorov. An experimental comparison of min-cut/max-flow
algorithms for energy minimization in vision.
IEEE Trans. On PAMI, 26(9):11241137, 2004.Y. Boykov, O. Veksler,
and R.Zabih. Efficient approximate energy minimization via graph
cuts. IEEE Trans. On PAMI, 20(12):
12221239, 2001.A. Buades, B. Coll, and J.M. Morel. A non local
algorithm for image denoising. In IEEE Computer Vision and Pattern
Recognition
(CVPR), volume 2, pages 6065, 2005.A. Bugeau, M. Bertalmo, V.
Caselles, and G. Sapiro. A comprehensive framework for image
inpainting. IEEE Trans. on Image
Processing, 19(10):26342644, 2010.A. Criminisi, P. Perez, and K.
Toyama. Region filling and object removal by examplar-based image
inpainting. IEEE Trans. On
Image Processing, 13:12001212, 2004.A. A. Efros and T. K. Leung.
Texture synthesis by non-parametric sampling. In IEEE Computer
Vision and Pattern Recognition
(CVPR), pages 10331038, 1999.C. Guillemot, M. Turkan, O. Le
Meur, and M. Ebdelli. Object removal and loss concealment using
neigbor embedding methods.
Signal processing: image communication, 28:14051419, 2013.K. He
and J. Sun. Statistics of patch offsets for image completion. In
ECCV, 2012.N. Komodakis and G. Tziritas. Image completion using
efficient belief propagation via priority scheduling and dynamic
pruning.
IEEE Trans. On Image Processing, 16(11):2649 2661, 2007.O. Le
Meur and C. Guillemot. Super-resolution-based inpainting. In ECCV,
pages 554567, 2012.O. Le Meur, J. Gautier, and C. Guillemot.
Examplar-based inpainting based on local geometry. In ICIP, 2011.D.
D. Lee and H. S. Seung. Algorithms for non-negative matrix
factorization. In In NIPS, pages 556562. MIT Press, 2001.Y. Pritch,
E. Kav-Venaki, and S. Peleg. Shift-map image editing. In ICCV09,
pages 151158, Kyoto, Sept 2009.L.K. Saul and S.T. Roweis. Think
globally, fit locally: Unsupervised learning of low dimensional
manifolds. Journal of Machine
Learning Research, 4:119155, 2003.J. Weickert.
Coherence-enhancing diffusion filtering. International Journal of
Computer Vision, 32:111127, 1999.Z. Xu and J. Sun. Image inpainting
by patch propagation using patch sparsity. IEEE Trans. on Image
Processing, 19(5):
11531165, 2010.J.S. Yedidia, W.T. Freeman, and Y. Weiss.
Constructing free energy approximations and generalized belief
propagation algorithms.
IEEE Transactions on Information Theory, 51:22822312, 2005.44 /
44
Inpainting: context and issuesExamplar-based
inpaintingPresentationNotationCriminisi et al.'s method
Variants of Criminisi's methodFilling order computationTexture
synthesisSome examplesLimitations
Super-resolution-based inpainting methodProposed approachMore
than one inpaintingLoopy Belief PropagationSuper-resolution
Results and comparison with existing methodsResultsComparison
with existing methods
Conclusion*
0.0: 0.1: 0.2: 0.3: 0.4: 0.5: 0.6: 0.7: 0.8: 0.9: 0.10: 0.11:
0.12: 0.13: 0.14: 0.15: 0.16: 0.17: 0.18: 0.19: 0.20: 0.21: 0.22:
0.23: 0.24: 0.25: 0.26: 0.27: 0.28: 0.29: 0.30: 0.31: 0.32: 0.33:
0.34: 0.35: 0.36: 0.37: 0.38: 0.39: 0.40: 0.41: 0.42: 0.43: 0.44:
0.45: 0.46: 0.47: 0.48: 0.49: 0.50: 0.51: 0.52: 0.53: 0.54: 0.55:
0.56: 0.57: 0.58: 0.59: 0.60: 0.61: 0.62: 0.63: 0.64: 0.65: 0.66:
0.67: 0.68: 0.69: 0.70: 0.71: 0.72: 0.73: 0.74: 0.75: 0.76: 0.77:
0.78: 0.79: 0.80: 0.81: 0.82: 0.83: 0.84: 0.85: 0.86: 0.87: 0.88:
0.89: 0.90: 0.91: 0.92: 0.93: 0.94: 0.95: 0.96: 0.97: 0.98: 0.99:
0.100: 0.101: 0.102: 0.103: 0.104: 0.105: 0.106: 0.107: 0.108:
0.109: 0.110: 0.111: 0.112: 0.113: 0.114: 0.115: 0.116: 0.117:
0.118: 0.119: 0.120: 0.121: 0.122: 0.123: 0.124: anm0: 1.0: 1.1:
1.2: 1.3: 1.4: 1.5: 1.6: 1.7: 1.8: 1.9: 1.10: 1.11: 1.12: 1.13:
1.14: 1.15: 1.16: 1.17: 1.18: 1.19: 1.20: 1.21: 1.22: 1.23: 1.24:
anm1: 2.0: 2.1: 2.2: 2.3: 2.4: 2.5: 2.6: 2.7: 2.8: 2.9: 2.10: 2.11:
2.12: 2.13: 2.14: 2.15: 2.16: 2.17: 2.18: 2.19: 2.20: 2.21: 2.22:
2.23: 2.24: anm2: 3.0: 3.1: 3.2: 3.3: 3.4: 3.5: 3.6: 3.7: 3.8: 3.9:
3.10: 3.11: 3.12: 3.13: 3.14: 3.15: 3.16: 3.17: 3.18: 3.19: 3.20:
3.21: 3.22: 3.23: 3.24: anm3: 4.0: 4.1: 4.2: 4.3: 4.4: 4.5: 4.6:
4.7: 4.8: 4.9: 4.10: 4.11: 4.12: 4.13: 4.14: 4.15: 4.16: 4.17:
4.18: 4.19: 4.20: 4.21: 4.22: 4.23: 4.24: anm4: 5.0: 5.1: 5.2: 5.3:
5.4: 5.5: 5.6: 5.7: 5.8: 5.9: 5.10: 5.11: 5.12: 5.13: 5.14: 5.15:
5.16: 5.17: 5.18: 5.19: 5.20: 5.21: 5.22: 5.23: 5.24: anm5: 6.0:
6.1: 6.2: 6.3: 6.4: 6.5: 6.6: 6.7: 6.8: 6.9: 6.10: 6.11: 6.12:
6.13: 6.14: 6.15: 6.16: 6.17: 6.18: 6.19: 6.20: 6.21: 6.22: 6.23:
6.24: anm6: 7.0: 7.1: 7.2: 7.3: 7.4: 7.5: 7.6: 7.7: 7.8: 7.9: 7.10:
7.11: 7.12: 7.13: 7.14: 7.15: 7.16: 7.17: 7.18: 7.19: 7.20: 7.21:
7.22: 7.23: 7.24: 7.25: 7.26: 7.27: 7.28: 7.29: 7.30: 7.31: 7.32:
7.33: 7.34: 7.35: 7.36: 7.37: 7.38: 7.39: 7.40: 7.41: 7.42: 7.43:
7.44: 7.45: 7.46: 7.47: 7.48: 7.49: 7.50: 7.51: 7.52: 7.53: 7.54:
7.55: 7.56: 7.57: 7.58: 7.59: 7.60: 7.61: 7.62: 7.63: 7.64: 7.65:
7.66: 7.67: 7.68: 7.69: 7.70: 7.71: 7.72: 7.73: 7.74: 7.75: 7.76:
7.77: 7.78: 7.79: 7.80: 7.81: 7.82: 7.83: 7.84: 7.85: 7.86: 7.87:
7.88: 7.89: 7.90: 7.91: 7.92: 7.93: 7.94: 7.95: 7.96: 7.97: 7.98:
7.99: 7.100: 7.101: 7.102: 7.103: 7.104: 7.105: 7.106: 7.107:
7.108: 7.109: 7.110: 7.111: 7.112: 7.113: 7.114: 7.115: 7.116:
7.117: 7.118: 7.119: 7.120: 7.121: 7.122: 7.123: 7.124: 7.125:
7.126: 7.127: 7.128: 7.129: 7.130: 7.131: 7.132: 7.133: 7.134:
7.135: 7.136: 7.137: 7.138: 7.139: 7.140: 7.141: 7.142: 7.143:
7.144: 7.145: 7.146: 7.147: 7.148: 7.149: 7.150: 7.151: 7.152:
7.153: anm7: