1. No category

Blending Methods and Other Improvements for
Exemplar-based Image Inpainting Techniques
Maxime Daisy, Pierre Buyssens,
David Tschumperlé and Olivier Lézoray
GREYC - CNRS UMR 6072, Image team
9th of April 2015
Context
D. Tschumperlé (GREYC, Caen - FRANCE)
Journée Imagerie Rouen
9th of April 2015
2 / 45
Context
D. Tschumperlé (GREYC, Caen - FRANCE)
Journée Imagerie Rouen
9th of April 2015
2 / 45
Context
D. Tschumperlé (GREYC, Caen - FRANCE)
Journée Imagerie Rouen
9th of April 2015
2 / 45
Context
D. Tschumperlé (GREYC, Caen - FRANCE)
Journée Imagerie Rouen
9th of April 2015
2 / 45
Context
D. Tschumperlé (GREYC, Caen - FRANCE)
Journée Imagerie Rouen
9th of April 2015
2 / 45
Context
Geometry-based methods
[Masnou et al. ’98, Bertalmio et al. ’00, Chan et al. ’01, Tschumperlé
and Deriche ’03, . . .]
→ boundary geometry interpolation
• Good global geometry reconstruction
• Not able to create complex textures
• Time consuming
D. Tschumperlé (GREYC, Caen - FRANCE)
Journée Imagerie Rouen
9th of April 2015
3 / 45
Masked image
D. Tschumperlé (GREYC, Caen - FRANCE)
Journée Imagerie Rouen
9th of April 2015
4 / 45
Inpainted with Diffusion PDE’s [Tschumperlé and Deriche ’03]
D. Tschumperlé (GREYC, Caen - FRANCE)
Journée Imagerie Rouen
9th of April 2015
4 / 45
Context
Pattern-based methods
[Efros and Leung’99, Criminisi et al. ’04, Lemeur et al. ’11, . . .]
→ copy/paste of image chunks
• Large area reconstruction
• Quite fast
• Not enough local/global coherence e.g. block-effect artifacts
⇒ Best compromise between time and quality
D. Tschumperlé (GREYC, Caen - FRANCE)
Journée Imagerie Rouen
9th of April 2015
5 / 45
Context
Criminisi et al. 2004, "Region Filling and Object Removal by
Exemplar-Based Image Inpainting"
D. Tschumperlé (GREYC, Caen - FRANCE)
Journée Imagerie Rouen
9th of April 2015
6 / 45
Masked image
D. Tschumperlé (GREYC, Caen - FRANCE)
Journée Imagerie Rouen
9th of April 2015
7 / 45
Inpainted with [Criminisi et al. ’04]
D. Tschumperlé (GREYC, Caen - FRANCE)
Journée Imagerie Rouen
9th of April 2015
7 / 45
Exemplar-based inpainting :
Proposed improvements
D. Tschumperlé (GREYC, Caen - FRANCE)
Journée Imagerie Rouen
9th of April 2015
8 / 45
Exemplar-based Inpainting Algorithm
Proposed improvements
1
priority accuracy enhancement
⇒ more global geometry consistency
2
better lookup statregy
⇒ more local geometry consistency
3
spatial patch blending
⇒ strongly reduced block-effect artifacts
D. Tschumperlé (GREYC, Caen - FRANCE)
Journée Imagerie Rouen
9th of April 2015
9 / 45
A better data term accuracy
Criminisi et al. data term
Dp =
−→ →
−
|∇Ip ⊥ . np |
α
(1)
"The gradient ∇Ip is computed as the maximum value of the image
gradient in Ψp ∩ I."
−
→
−
→⊥
∇I ⊥
p = {∇I q |
D. Tschumperlé (GREYC, Caen - FRANCE)
arg max
−
→
k∇I q k}
(2)
q ∈((I−Ω)∩ψp )
Journée Imagerie Rouen
9th of April 2015
10 / 45
A better data term accuracy
Proposed data term
→
−
˜ p = kGp . n p k
D
with
Gp =
X
(3)
−
→ −
→
wp (q ) ∇I q ∇I Tq
(4)
q ∈ψp ∩(I−Ω)
wp : normalized 2d gaussian centered at p
D. Tschumperlé (GREYC, Caen - FRANCE)
Journée Imagerie Rouen
9th of April 2015
11 / 45
A better lookup strategy
Criminisi et al. 2014 lookup strategy
• window search ?
⇒ faster
⇒ much less global
• all over the image search ?
⇒ slow
⇒ not so much good results
D. Tschumperlé (GREYC, Caen - FRANCE)
Journée Imagerie Rouen
9th of April 2015
12 / 45
A better lookup strategy
Window search
Iteration 0
D. Tschumperlé (GREYC, Caen - FRANCE)
Journée Imagerie Rouen
9th of April 2015
13 / 45
A better lookup strategy
Window search
Iteration n
D. Tschumperlé (GREYC, Caen - FRANCE)
Journée Imagerie Rouen
9th of April 2015
13 / 45
A better lookup strategy
Our lookup strategy
• inspired from [Ashikhmin ’01, PatchMatch ’09]
• local/global search scheme
• use search sites of surrounding inpainted patches
⇒ more local coherence
D. Tschumperlé (GREYC, Caen - FRANCE)
Journée Imagerie Rouen
9th of April 2015
14 / 45
A better lookup strategy
D. Tschumperlé (GREYC, Caen - FRANCE)
Journée Imagerie Rouen
9th of April 2015
15 / 45
Joshua Tree - 512 × 384
Masked image
D. Tschumperlé (GREYC, Caen - FRANCE)
Journée Imagerie Rouen
9th of April 2015
16 / 45
Joshua Tree - 512 × 384
Inpainted with windowed search (4.5s)
D. Tschumperlé (GREYC, Caen - FRANCE)
Journée Imagerie Rouen
9th of April 2015
16 / 45
Joshua Tree - 512 × 384
Inpainted with smart search (3.2s)
D. Tschumperlé (GREYC, Caen - FRANCE)
Journée Imagerie Rouen
9th of April 2015
16 / 45
Fake Owl - 800 × 450
Masked image
D. Tschumperlé (GREYC, Caen - FRANCE)
Journée Imagerie Rouen
9th of April 2015
17 / 45
Fake Owl - 800 × 450
Inpainted with windowed search (1.7s)
D. Tschumperlé (GREYC, Caen - FRANCE)
Journée Imagerie Rouen
9th of April 2015
17 / 45
Fake Owl - 800 × 450
Inpainted with smart search (0.8s)
D. Tschumperlé (GREYC, Caen - FRANCE)
Journée Imagerie Rouen
9th of April 2015
17 / 45
block-effect artifacts ?
D. Tschumperlé (GREYC, Caen - FRANCE)
Journée Imagerie Rouen
9th of April 2015
18 / 45
Spatial Patch Blending
D. Tschumperlé (GREYC, Caen - FRANCE)
Journée Imagerie Rouen
9th of April 2015
19 / 45
Spatial Patch Blending
Masked image
D. Tschumperlé (GREYC, Caen - FRANCE)
Journée Imagerie Rouen
9th of April 2015
20 / 45
Spatial Patch Blending
Inpainted with [Criminisi et al. ’04]
D. Tschumperlé (GREYC, Caen - FRANCE)
Journée Imagerie Rouen
9th of April 2015
20 / 45
Spatial Patch Blending
Inpainted with diffusion PDE’s [Tschumperlé and Deriche ’03]
D. Tschumperlé (GREYC, Caen - FRANCE)
Journée Imagerie Rouen
9th of April 2015
20 / 45
Spatial Patch Blending
Inpainted with [Criminisi et al. ’04] + Our spatial patch blending
D. Tschumperlé (GREYC, Caen - FRANCE)
Journée Imagerie Rouen
9th of April 2015
20 / 45
Artifact Detection
Masked image
D. Tschumperlé (GREYC, Caen - FRANCE)
Journée Imagerie Rouen
9th of April 2015
21 / 45
Artifact Detection
Inpainted with [Criminisi et al. ’04]
D. Tschumperlé (GREYC, Caen - FRANCE)
Journée Imagerie Rouen
9th of April 2015
21 / 45
Artifact Detection
1
Map of the artifact location probabilities
2 hypothesis for artifact locations
• local sharp variations in the inpainted image
• source patches come from very different locations
PA = k ∇I k . div (φ)
φ : inpainting correspondence map
2
Strongest artifact locations
LA = {p | PA (p) > τ }
3
Blending amplitude map
P
σ(p) = ρ .
w (p,q )
P
w (p , q )
max
q ∈LA
with
r ∈I q ∈L
A
D. Tschumperlé (GREYC, Caen - FRANCE)
w (p, q ) = exp −
Journée Imagerie Rouen
kp−q k2
PA (q )2
9th of April 2015
(5)
22 / 45
Artifact Detection
Inpainted image.
D. Tschumperlé (GREYC, Caen - FRANCE)
Journée Imagerie Rouen
9th of April 2015
23 / 45
Artifact Detection
Break field image.
D. Tschumperlé (GREYC, Caen - FRANCE)
Journée Imagerie Rouen
9th of April 2015
23 / 45
Artifact Detection
Break points.
D. Tschumperlé (GREYC, Caen - FRANCE)
Journée Imagerie Rouen
9th of April 2015
23 / 45
Artifact Detection
Blending amplitude map.
D. Tschumperlé (GREYC, Caen - FRANCE)
Journée Imagerie Rouen
9th of April 2015
23 / 45
Patch Blending
D. Tschumperlé (GREYC, Caen - FRANCE)
Journée Imagerie Rouen
9th of April 2015
24 / 45
Patch Blending
Weighted sum of overlapping patches pixels values
P
J (p) =
w (p, q ) . ψq (p − q )
ψq ∈Ψp
P
(6)
w (q , p )
ψq ∈Ψp
2
p−q k
• w (p, q ) = exp − kσ(
p)2
• Ψp = {ψq | ψq ∩ ψp 6= ∅}
→ one gaussian function for each possible blending amplitude
⇒ quite slow
D. Tschumperlé (GREYC, Caen - FRANCE)
Journée Imagerie Rouen
9th of April 2015
25 / 45
A faster patch blending algorithm
D. Tschumperlé (GREYC, Caen - FRANCE)
Journée Imagerie Rouen
9th of April 2015
26 / 45
Result
Masked image
D. Tschumperlé (GREYC, Caen - FRANCE)
Journée Imagerie Rouen
9th of April 2015
27 / 45
Result
Inpainted image
D. Tschumperlé (GREYC, Caen - FRANCE)
Journée Imagerie Rouen
9th of April 2015
28 / 45
Result
Inpainted Image + Spatial patch blending
D. Tschumperlé (GREYC, Caen - FRANCE)
Journée Imagerie Rouen
9th of April 2015
29 / 45
Block-effect artifacts ?
Masked image
D. Tschumperlé (GREYC, Caen - FRANCE)
Journée Imagerie Rouen
9th of April 2015
30 / 45
Block-effect artifacts ?
Inpainted Image
D. Tschumperlé (GREYC, Caen - FRANCE)
Journée Imagerie Rouen
9th of April 2015
30 / 45
Block-effect artifacts ?
Blended Image (isotropic)
D. Tschumperlé (GREYC, Caen - FRANCE)
Journée Imagerie Rouen
9th of April 2015
30 / 45
Geometry-guided patch blending : process
Anisotropic image regularization
D. Tschumperlé (GREYC, Caen - FRANCE)
Journée Imagerie Rouen
9th of April 2015
31 / 45
Geometry-guided patch blending : process
Anisotropic patch blending
D. Tschumperlé (GREYC, Caen - FRANCE)
Journée Imagerie Rouen
9th of April 2015
32 / 45
Geometry-guided patch blending
1
Structure eigen values normalization
ˆ S(p)i =
λ
λS(p)
max λS(p)i
p∈I
2
Blending tensors eigen values computation
λBi =
3
1
ˆ S1 + λ
ˆ S2 )γi
(1 + λ
Blending tensor building
T ⊥
⊥ T ⊥
B = λσB1 e⊥
S1 . eS1 + λσ B2 eS2 . eS2
(7)
• eSi : structure tensor eigen vectors
• λσBi = σB λBi
• σB : maximum blending bandwidth
D. Tschumperlé (GREYC, Caen - FRANCE)
Journée Imagerie Rouen
9th of April 2015
33 / 45
Geometry-guided patch blending
Weighted sum of overlapping patches pixels values using Eq. (6) with

2
exp − kp−q k2
isotropic
T σ(p)−1 w (p , q ) =
exp X B(p2) X
anisotropic
2σB
where X = q − p
D. Tschumperlé (GREYC, Caen - FRANCE)
Journée Imagerie Rouen
9th of April 2015
34 / 45
block-effect artifacts ?
Masked image
D. Tschumperlé (GREYC, Caen - FRANCE)
Journée Imagerie Rouen
9th of April 2015
35 / 45
block-effect artifacts ?
Inpainted image
D. Tschumperlé (GREYC, Caen - FRANCE)
Journée Imagerie Rouen
9th of April 2015
35 / 45
block-effect artifacts ?
Blended image (isotropic)
D. Tschumperlé (GREYC, Caen - FRANCE)
Journée Imagerie Rouen
9th of April 2015
35 / 45
block-effect artifacts ?
Blended image (geometry-guided)
D. Tschumperlé (GREYC, Caen - FRANCE)
Journée Imagerie Rouen
9th of April 2015
35 / 45
Woman
Masked image
D. Tschumperlé (GREYC, Caen - FRANCE)
Journée Imagerie Rouen
9th of April 2015
36 / 45
Woman
Without blending
D. Tschumperlé (GREYC, Caen - FRANCE)
Journée Imagerie Rouen
9th of April 2015
36 / 45
Woman
Isotropic patch blending
D. Tschumperlé (GREYC, Caen - FRANCE)
Journée Imagerie Rouen
9th of April 2015
36 / 45
Woman
Geometry-guided patch blending
D. Tschumperlé (GREYC, Caen - FRANCE)
Journée Imagerie Rouen
9th of April 2015
36 / 45
Copter
Masked image
D. Tschumperlé (GREYC, Caen - FRANCE)
Journée Imagerie Rouen
9th of April 2015
37 / 45
Copter
Copter + mask
D. Tschumperlé (GREYC, Caen - FRANCE)
Journée Imagerie Rouen
9th of April 2015
37 / 45
Copter
Inpainted + anisotropic patch blending
D. Tschumperlé (GREYC, Caen - FRANCE)
Journée Imagerie Rouen
9th of April 2015
37 / 45
Opera
Masked image
D. Tschumperlé (GREYC, Caen - FRANCE)
Journée Imagerie Rouen
9th of April 2015
38 / 45
Opera
Inpainted + anisotropic patch blending
D. Tschumperlé (GREYC, Caen - FRANCE)
Journée Imagerie Rouen
9th of April 2015
38 / 45
Opera
Inpainted without blending [Lemeur et al’11]
D. Tschumperlé (GREYC, Caen - FRANCE)
Journée Imagerie Rouen
9th of April 2015
38 / 45
Video Inpainting
D. Tschumperlé (GREYC, Caen - FRANCE)
Journée Imagerie Rouen
9th of April 2015
39 / 45
Adaptation to videos
• priorities computed frame by frame
• cobblestone patches (e.g. of size 5 × 5 × 3) and lookup windows
Shown to be working in the state of the art
• Wexler et al. TPAMI 2007, "Space-time completion of Videos"
• Newson et al. SIAM J. Imaging sciences 2014, "Video inpainting of
complex scenes"
D. Tschumperlé (GREYC, Caen - FRANCE)
Journée Imagerie Rouen
9th of April 2015
40 / 45
Space-time artifacts
Space-time patch blending
D. Tschumperlé (GREYC, Caen - FRANCE)
Journée Imagerie Rouen
9th of April 2015
41 / 45
Adaptation to videos
Space-time patch blending
D. Tschumperlé (GREYC, Caen - FRANCE)
Journée Imagerie Rouen
9th of April 2015
42 / 45
Masked video : (x,y) and (x,t) plans
D. Tschumperlé (GREYC, Caen - FRANCE)
Journée Imagerie Rouen
9th of April 2015
43 / 45
Inpainted video (without blending)
D. Tschumperlé (GREYC, Caen - FRANCE)
Journée Imagerie Rouen
9th of April 2015
43 / 45
Blended Image (geometry-guided)
D. Tschumperlé (GREYC, Caen - FRANCE)
Journée Imagerie Rouen
9th of April 2015
43 / 45
Conclusion
Contributions
1
Improvements of a reference inpainting method
2
Method to reduce block-effect artifacts
3
Adaptation of (1) and (2) to video data
Recent advances
• Depth-aware patch blending for stereoscopic inpainted results
More results at
https://daisy.users.greyc.fr/@research
D. Tschumperlé (GREYC, Caen - FRANCE)
Journée Imagerie Rouen
9th of April 2015
44 / 45
References and publications
References
• Wexler et al., Space-Time Video Completion, IEEE TPAMI 2007
• Newson et al., Video Inpainting of Complex Scenes, SIAM J. IMAGING
SCIENCES 2014
Publications
• Daisy M., Tschumperlé D. and Lézoray O., Spatial Patch Blending for
Artefact Reduction in Pattern-Based Inpainting Techniques, CAIP’13
• Daisy M., Tschumperlé D. and Lézoray O., A Fast Spatial Patch
Blending Algorithm for Artefact Reduction in Pattern-based Image
Inpainting, SIGGRAPH-ASIA’13
• Daisy M., Buyssens P., Tschumperlé D. and Lézoray O., A smarter
exemplar-based inpainting algorithm using local and global heuristics
for more geometric coherence, ICIP’14
D. Tschumperlé (GREYC, Caen - FRANCE)
Journée Imagerie Rouen
9th of April 2015
45 / 45