Generating Classic Mosaics with Graph Cuts Y. Liu, O. Veksler and O. Juan University of Western Ontario, Canada Ecole Centrale de Paris, France.

Generating Classic Mosaics with Graph Cuts

Y. Liu, O. Veksler and O. JuanUniversity of Western Ontario, Canada

Ecole Centrale de Paris, France

Different Types of Mosaics

A. Hausner [SIGGRAPH 01]

K. Smith et. al [SIGGRAPH 05]

J. Orchard et. al [NPAR 08]

Jesus at Basilica of Sant' Apollinare Nuovo

Classic Mosaic: Main constraints• Tile orientation should emphasize important

scene edges• Tiles should be packed tightly while preserving

their completeness

Bad TilingGood Tiling

Previous Work• User Interaction– Hausner [SIGGRAPH2001]– Elber and Wolberg[Visual Computer 2003]

• Explicit Edge detection– Di Blasi et. al [Visual Computer 2005]– Battiato et. al [WSCG 2006]

• Implicit Methods– Battiato et. al [EG 2008]– Liu et. al [EMMCVPR 2007]

Previous Work Cont.

• Requires user interaction• Discontinuities in tile

orientation filed• No control over tile overlap

• Local optimization for tile orientation

• Heuristics for tile packing

Hausner [SIGGRAPH 2001] Battiato et. Al [EG 2008]

Our Method: Overview• We address the problem in a global

optimization framework– pixel labelling problem

• Formulate energy which encodes constraints for a good mosaic– Encourage tiles to align to image edges– Smoothly varying tile orientations– Tight tile packing– Overlapping tiles prohibited

High Energy

Low Energy

Mosaic as Pixel Labelling • Each pixel p is assigned a pairwise label

(vp , ϕp)

• ϕp is tile orientation– if tile centered at p is visible,

orientation is ϕp• vp is visibility– 1 means tile is present– 0 means tile is hidden

0

0

1

1 ϕp

Energy Function( , )E v (1 )p

p P

v

( )p p pp P

v D

{ , }

( , , , )pq p q p qp q N

V v v

encourages dense packing

encourages tiles to align to image edges

encourages smooth tile orientations and prohibits tile overlap

high cost

low cost

high cost

high cost

low cost

high cost

low cost

Optimization with Graph cut • NP-hard to optimize all variables

simultaneously • We use step-wise optimization– First, ignore the visibility labels and optimize the

orientation variables– Then, fix the orientation labels and optimize the

visibility variables

Optimizing Orientation Variables

• Assign an orientation label ϕp to each pixel– don’t know where the tiles

will be placed, so optimize for all possible locations

• approximately optimized by alpha-expansion [Boykov et.al. PAMI01]

( )E ( )p pp P

D

{ , }

( , )pq p qp q N

V

Optimizing the visibility variables• Heuristically build many mosaic layers

according to the orientation labels– Each layer consists of non-overlapping tiles– Each layer is as dense as possible

Maintain current solution Randomly pick mosaic

layer Find the smallest energy

stitching

1

10

0 110

1 01

101

1

formulated as binary labelling problem energy roughly corresponds to the area of the

gap space, therefore encouraging reduction in gaps

tiles will not overlap since energy would be infinite

Submodular energy, optimized with a graph cut [Kolmogorov et. al. PAMI2004]

Stitching Layers

Layer Stitching Illustration• Tiles may cross the image edges and create a

blurry effect

Current Solution Random Layer

Layer Stitching illustration• Tiles overlapping intensity edges are removed

Good stitching

Bad stitching

Overview of Layer Stitching• Start from a random layer• In each iteration, stitch the current solution

and a candidate mosaic layer together• Repeat until the result is visually pleasing

Experimental Results

Our Approach

Battiato et. Al [EG 2008]

Our Approach

Battiato et. Al [EG 2008]

Experimental Results

Experimental Results

Experimental Results

Mosaic with Different Tile Size

Input Image 10 by 10 20 by 20

Number of Iterations in Stitching

Starting Layer 27 iteration 156 iteration

Edge Avoidance

Small Weight Large Weight

Conclusion and future work• We address classic mosaic problem in an energy

optimization framework• Our energy function encodes desired mosaic properties

• Tiles align to image edges, minimize gap space, prohibit tile overlap

• We eliminate monotonous user interaction of drawing feature edges

• Take advantage of powerful discrete optimization methods based on graph cuts

• Results are visually pleasing• Extensions:

• tiles of different shapes and sizes• video mosaics