Top Banner
1 Graphcut Textures Image and Video Synthesis Using Graph Cuts Vivek Kwatra Arno Schödl Irfan Essa Greg Turk Aaron Bobick Presented by Benjamin Sigg
38

Graphcut Textures - ETHZgraphics.ethz.ch/.../former/seminar/handouts/Sigg_GraphCutTextures.… · 1 Graphcut Textures Image and Video Synthesis Using Graph Cuts Vivek Kwatra Arno

Sep 17, 2018

Download

Documents

tranthu
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Page 1: Graphcut Textures - ETHZgraphics.ethz.ch/.../former/seminar/handouts/Sigg_GraphCutTextures.… · 1 Graphcut Textures Image and Video Synthesis Using Graph Cuts Vivek Kwatra Arno

1

Graphcut TexturesImage and Video Synthesis Using

Graph Cuts

Vivek KwatraArno Schödl

Irfan EssaGreg Turk

Aaron Bobick

Presented by Benjamin Sigg

Page 2: Graphcut Textures - ETHZgraphics.ethz.ch/.../former/seminar/handouts/Sigg_GraphCutTextures.… · 1 Graphcut Textures Image and Video Synthesis Using Graph Cuts Vivek Kwatra Arno

2

Topics

• Whats the goal?• Graph Cut• Patch placement• Extensions, Video• Authors conclusions• My conclusions

Page 3: Graphcut Textures - ETHZgraphics.ethz.ch/.../former/seminar/handouts/Sigg_GraphCutTextures.… · 1 Graphcut Textures Image and Video Synthesis Using Graph Cuts Vivek Kwatra Arno

3

Creating Textures

• Input is a small image• Output is a large image

InputOutput

Page 4: Graphcut Textures - ETHZgraphics.ethz.ch/.../former/seminar/handouts/Sigg_GraphCutTextures.… · 1 Graphcut Textures Image and Video Synthesis Using Graph Cuts Vivek Kwatra Arno

4

Idea behind

• Take a patch from the input• Place the patch somewhere on the output

such that no seams are visible

InputOutput

Page 5: Graphcut Textures - ETHZgraphics.ethz.ch/.../former/seminar/handouts/Sigg_GraphCutTextures.… · 1 Graphcut Textures Image and Video Synthesis Using Graph Cuts Vivek Kwatra Arno

5

Idea behind

• Iterate over– Step 1: Patch matching & placement– Step 2: Graph cut

1

2

Page 6: Graphcut Textures - ETHZgraphics.ethz.ch/.../former/seminar/handouts/Sigg_GraphCutTextures.… · 1 Graphcut Textures Image and Video Synthesis Using Graph Cuts Vivek Kwatra Arno

6

Graph Cut

• Cut two images, such that they can be seamless merged

• Minimize a costfunction for cut and merge

Page 7: Graphcut Textures - ETHZgraphics.ethz.ch/.../former/seminar/handouts/Sigg_GraphCutTextures.… · 1 Graphcut Textures Image and Video Synthesis Using Graph Cuts Vivek Kwatra Arno

7

Graph Cut

• A cut is more than a line

+ =

Page 8: Graphcut Textures - ETHZgraphics.ethz.ch/.../former/seminar/handouts/Sigg_GraphCutTextures.… · 1 Graphcut Textures Image and Video Synthesis Using Graph Cuts Vivek Kwatra Arno

8

Graph Cut

• Input– One graph– One costfunction

• Output– The cut with minimal costs

Page 9: Graphcut Textures - ETHZgraphics.ethz.ch/.../former/seminar/handouts/Sigg_GraphCutTextures.… · 1 Graphcut Textures Image and Video Synthesis Using Graph Cuts Vivek Kwatra Arno

9

Graph Cut: Input

• Two images seen as graphs

Page 10: Graphcut Textures - ETHZgraphics.ethz.ch/.../former/seminar/handouts/Sigg_GraphCutTextures.… · 1 Graphcut Textures Image and Video Synthesis Using Graph Cuts Vivek Kwatra Arno

10

Graph Cut: Input

• But only one graph required• Use an “embracing” graph

Page 11: Graphcut Textures - ETHZgraphics.ethz.ch/.../former/seminar/handouts/Sigg_GraphCutTextures.… · 1 Graphcut Textures Image and Video Synthesis Using Graph Cuts Vivek Kwatra Arno

11

Graph Cut: Input

• Costfunction

M A , B , s , tA s B s A t B t

AB

s t

Page 12: Graphcut Textures - ETHZgraphics.ethz.ch/.../former/seminar/handouts/Sigg_GraphCutTextures.… · 1 Graphcut Textures Image and Video Synthesis Using Graph Cuts Vivek Kwatra Arno

12

Graph Cut - Input

M A , B , s , tA s B s A t B t

AB

s t

AB

s t

AB

s t

Page 13: Graphcut Textures - ETHZgraphics.ethz.ch/.../former/seminar/handouts/Sigg_GraphCutTextures.… · 1 Graphcut Textures Image and Video Synthesis Using Graph Cuts Vivek Kwatra Arno

13

Graph Cut - Output

• The minimal cut of the graph

Page 14: Graphcut Textures - ETHZgraphics.ethz.ch/.../former/seminar/handouts/Sigg_GraphCutTextures.… · 1 Graphcut Textures Image and Video Synthesis Using Graph Cuts Vivek Kwatra Arno

14

Graph Cut - Output

• Merge the two images along the cut• (Hopefully) no visible seams

Page 15: Graphcut Textures - ETHZgraphics.ethz.ch/.../former/seminar/handouts/Sigg_GraphCutTextures.… · 1 Graphcut Textures Image and Video Synthesis Using Graph Cuts Vivek Kwatra Arno

15

Graph Cut: Example

• Two merged images• No seams are visible

Page 16: Graphcut Textures - ETHZgraphics.ethz.ch/.../former/seminar/handouts/Sigg_GraphCutTextures.… · 1 Graphcut Textures Image and Video Synthesis Using Graph Cuts Vivek Kwatra Arno

16

Graph Cut

• Algorithms are well known• Many possible Costfunctions• What about many images?

Page 17: Graphcut Textures - ETHZgraphics.ethz.ch/.../former/seminar/handouts/Sigg_GraphCutTextures.… · 1 Graphcut Textures Image and Video Synthesis Using Graph Cuts Vivek Kwatra Arno

17

Many Images

• Merge first two images• Merge resulting image with a third

image...

+ =+ +

Page 18: Graphcut Textures - ETHZgraphics.ethz.ch/.../former/seminar/handouts/Sigg_GraphCutTextures.… · 1 Graphcut Textures Image and Video Synthesis Using Graph Cuts Vivek Kwatra Arno

18

Many Images

• Merge with a third image• Result is very bad• Don't throw away information!

+ =

Page 19: Graphcut Textures - ETHZgraphics.ethz.ch/.../former/seminar/handouts/Sigg_GraphCutTextures.… · 1 Graphcut Textures Image and Video Synthesis Using Graph Cuts Vivek Kwatra Arno

19

Many Images

• Store old seam costs

Page 20: Graphcut Textures - ETHZgraphics.ethz.ch/.../former/seminar/handouts/Sigg_GraphCutTextures.… · 1 Graphcut Textures Image and Video Synthesis Using Graph Cuts Vivek Kwatra Arno

20

Many Images

• Add the old seam costs to the new cut

old cut new cut

cost costnew costold

Page 21: Graphcut Textures - ETHZgraphics.ethz.ch/.../former/seminar/handouts/Sigg_GraphCutTextures.… · 1 Graphcut Textures Image and Video Synthesis Using Graph Cuts Vivek Kwatra Arno

21

Patch Placement

• Place input somewhere on the output

• Cut out a patch

• Random placement• Entire patch matching• Sub-patch matching

t

Page 22: Graphcut Textures - ETHZgraphics.ethz.ch/.../former/seminar/handouts/Sigg_GraphCutTextures.… · 1 Graphcut Textures Image and Video Synthesis Using Graph Cuts Vivek Kwatra Arno

22

Random placement

• Place the input randomatically on the output

• Graph cut decides, which part of the input will be seen

• The fastest approach• Results are good for random textures

Page 23: Graphcut Textures - ETHZgraphics.ethz.ch/.../former/seminar/handouts/Sigg_GraphCutTextures.… · 1 Graphcut Textures Image and Video Synthesis Using Graph Cuts Vivek Kwatra Arno

23

Entire patch matching

• Place the input such that it matches the output best

• Costfunction for Placement (SSD):

• Results are good for (semi-) structured textures

C t 1At p At

I p O p t 2I : InputO : OutputA: Intersection I Op : Pointt : Translation

Page 24: Graphcut Textures - ETHZgraphics.ethz.ch/.../former/seminar/handouts/Sigg_GraphCutTextures.… · 1 Graphcut Textures Image and Video Synthesis Using Graph Cuts Vivek Kwatra Arno

24

Sub patch matching

• Choose a small patch in the output• Search a patch in the input that fits well• Same Costfunction as “entire patch

matching”

• Most general technique• Good for videos

Page 25: Graphcut Textures - ETHZgraphics.ethz.ch/.../former/seminar/handouts/Sigg_GraphCutTextures.… · 1 Graphcut Textures Image and Video Synthesis Using Graph Cuts Vivek Kwatra Arno

25

Results

Page 26: Graphcut Textures - ETHZgraphics.ethz.ch/.../former/seminar/handouts/Sigg_GraphCutTextures.… · 1 Graphcut Textures Image and Video Synthesis Using Graph Cuts Vivek Kwatra Arno

26

Results

• Input, Output and Error

Page 27: Graphcut Textures - ETHZgraphics.ethz.ch/.../former/seminar/handouts/Sigg_GraphCutTextures.… · 1 Graphcut Textures Image and Video Synthesis Using Graph Cuts Vivek Kwatra Arno

27

Results

• Not every image is a good input

Page 28: Graphcut Textures - ETHZgraphics.ethz.ch/.../former/seminar/handouts/Sigg_GraphCutTextures.… · 1 Graphcut Textures Image and Video Synthesis Using Graph Cuts Vivek Kwatra Arno

28

Extensions

• Other costfunctions• Blending techniques to omit seams• FFT for patch placing, dramatically fastens

the algorithm• Use not only translations, use rotations,

mirroring and scaling.

Page 29: Graphcut Textures - ETHZgraphics.ethz.ch/.../former/seminar/handouts/Sigg_GraphCutTextures.… · 1 Graphcut Textures Image and Video Synthesis Using Graph Cuts Vivek Kwatra Arno

29

Extensions

• Perspective by scaling the input• Additional Constraints for scaling

Page 30: Graphcut Textures - ETHZgraphics.ethz.ch/.../former/seminar/handouts/Sigg_GraphCutTextures.… · 1 Graphcut Textures Image and Video Synthesis Using Graph Cuts Vivek Kwatra Arno

30

Extensions

• Interactive Merging and Blending

1-2 hours of work

Page 31: Graphcut Textures - ETHZgraphics.ethz.ch/.../former/seminar/handouts/Sigg_GraphCutTextures.… · 1 Graphcut Textures Image and Video Synthesis Using Graph Cuts Vivek Kwatra Arno

31

Video

• Can be used to generate infinite videos• Just see a video as 3-dimensional graph

with axis: x, y, t.• Loop the video

Page 32: Graphcut Textures - ETHZgraphics.ethz.ch/.../former/seminar/handouts/Sigg_GraphCutTextures.… · 1 Graphcut Textures Image and Video Synthesis Using Graph Cuts Vivek Kwatra Arno

32

Video Transition

• Search two good matching frames• Make the cut through this two frames• Cut is a plane

Page 33: Graphcut Textures - ETHZgraphics.ethz.ch/.../former/seminar/handouts/Sigg_GraphCutTextures.… · 1 Graphcut Textures Image and Video Synthesis Using Graph Cuts Vivek Kwatra Arno

33

Random Temporal offset

• For small input• No periodicity• Requires continous part in time

Page 34: Graphcut Textures - ETHZgraphics.ethz.ch/.../former/seminar/handouts/Sigg_GraphCutTextures.… · 1 Graphcut Textures Image and Video Synthesis Using Graph Cuts Vivek Kwatra Arno

34

Spatio-Temporally Stationary Videos

• For videos with a one-directional movement (e.x. Smoke)

• Pixels moved in time and space

Page 35: Graphcut Textures - ETHZgraphics.ethz.ch/.../former/seminar/handouts/Sigg_GraphCutTextures.… · 1 Graphcut Textures Image and Video Synthesis Using Graph Cuts Vivek Kwatra Arno

35

Temporal Constraints

• Moving pixels in time and place causes problems, continuity is lost

• First and last n frames fixed by input• Last n frames removed when output

created

Page 36: Graphcut Textures - ETHZgraphics.ethz.ch/.../former/seminar/handouts/Sigg_GraphCutTextures.… · 1 Graphcut Textures Image and Video Synthesis Using Graph Cuts Vivek Kwatra Arno

36

Authors conclusions

• Algorithm is fast• No restrictions for the shape• Method for adding constraints• Easely generalized

Page 37: Graphcut Textures - ETHZgraphics.ethz.ch/.../former/seminar/handouts/Sigg_GraphCutTextures.… · 1 Graphcut Textures Image and Video Synthesis Using Graph Cuts Vivek Kwatra Arno

37

My conclusions

• Very flexible technique:– User interaction is possible– Different costfunctions applicable– Different pach placing/matching algorithms

available• Good for random images• Still problems with videos

Page 38: Graphcut Textures - ETHZgraphics.ethz.ch/.../former/seminar/handouts/Sigg_GraphCutTextures.… · 1 Graphcut Textures Image and Video Synthesis Using Graph Cuts Vivek Kwatra Arno

38

The End