Composition-Guided Image Acquisition Serene Banerjee Ph.D. Defense, April 28 th , 2004 http://www.ece.utexas.edu/~seren e Committee Members: Prof. Ross Baldick Prof. Alan C. Bovik Prof. Brian L. Evans (Advisor) Prof. Wilson S. Geisler Prof. Joydeep Ghosh Prof. Robert W. Heath, Jr. Computer Engineering Curriculum Track Dept. of Electrical and Computer Engineering The University of Texas at Austin
38
Embed
Composition-Guided Image Acquisition Serene Banerjee Ph.D. Defense, April 28 th, 2004 serene Committee Members: Prof. Ross Baldick.
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Composition-Guided Image Acquisition
Serene BanerjeePh.D. Defense, April 28th, 2004 http://www.ece.utexas.edu/~serene
Committee Members: Prof. Ross Baldick
Prof. Alan C. Bovik Prof. Brian L. Evans (Advisor)
Prof. Wilson S. GeislerProf. Joydeep Ghosh
Prof. Robert W. Heath, Jr.
Computer Engineering Curriculum TrackDept. of Electrical and Computer EngineeringThe University of Texas at Austin
4/28/2004 Composition-Guided Image Acquisition 2
“One day Alice came to a fork in the road and saw a Cheshire cat in a tree.
‘Which Road do I take?’ she asked.
‘Where do you want to go?’ was his response.
‘I don’t know,’ Alice answered.
‘Then,’ said the cat, ‘it doesn’t matter.”
Lewis Carroll Alice in Wonderland
4/28/2004 Composition-Guided Image Acquisition 3
Outline
Introduction Motivation Overview of contributions Summary of previous research for main subject detection
Contributions Online main subject detection Aesthetic enhancements, given main subject Blur background objects merging with main subject
Conclusions
4/28/2004 Composition-Guided Image Acquisition 4
Motivation
Problem: Amateur photographers take unappealing pictures (e.g. personal and business use)
Help users take better pictures with digital cameras
Software control Shutter aperture and speed Focus Zoom White balance
Additional hardware could control Camera angle Aspect ratio: landscape or
portrait
4/28/2004 Composition-Guided Image Acquisition 10
Outline
Introduction Contributions
Online main subject detection In-camera segmentation of the main subject Low-complexity one-pass algorithm Amenable to implementation in digital still cameras
Aesthetic enhancement, given main subject Mitigation of mergers with background objects
Conclusions
4/28/2004 Composition-Guided Image Acquisition 11
Online Main Subject Detection
Auto-focus main subject Take supplementary picture
Open shutter aperture (takes 1s) to blur objects not in focus
In-focus edges stronger than out-of-focus edges
Process supplementary picture to find main subject mask Enhance in-focus edges Detect strong edges Close boundary
Contribution #1
3x3 Highpass
filter
Detect sharper
edges
Close boundary
Auto-focus filter
Open shutter for blur
Scene
Binary main subject mask
Compute intensity
Supplementary picture
4/28/2004 Composition-Guided Image Acquisition 12
Supplementary picture has intensity function, I IH and IL are highpass and lowpass versions
For background image, contribution from IL is greater
Goal: Identify pixels contributing high frequencies I is modeled as mixture of IH and IL
Highpass filtering of I enhances main subject edges
Main Subject Detection: Formulation
LH Ik
kI
kI
11
1
Contribution #1
where k 1
4/28/2004 Composition-Guided Image Acquisition 13
Step 1: Enhance In-focus Edges
Subtract smoothed image from sharpened one Strong edges in main subject, weak edges in background
Σ
Supplementary image
Lowpass image
Highboost image
+
-
Contribution #1
Edge-enhanced image with
stronger main subject edges
4/28/2004 Composition-Guided Image Acquisition 14
Step 2: Detect Strong Edges
Canny edge detector detects strong edges [Canny; 1986] Selects weak edges only if they are connected to strong
edges
Laplacian of Gaussian detector [Burt & Adelson; 1983] Selects edges based on zero crossings of second derivative Either detects weak and strong edges or eliminates weak
Frequency-based features not applicable if Main subject does not have enough high frequencies Background not blurry enough
Could incorporate region-based features
Example of an image where the proposed algorithm fails to detect the main subject, the flower
Contribution #1
4/28/2004 Composition-Guided Image Acquisition 20
Outline
Introduction Contributions
Main subject detection Aesthetic enhancement, given main subject
Reposition main subject to follow rule-of-thirds Simulate background blur for motion or clarity
Mitigation of mergers with background objects
Conclusions
4/28/2004 Composition-Guided Image Acquisition 21
Rule-of-Thirds
Better interaction of main subject with image background
Center of mass of main subject at 1/3 or 2/3 picture width (or height) from the left (or top) edge
Contribution #2
Main subject in center of picture
Main subject follows rule-of-thirds
Outdoor setting; the flower is main subject
4/28/2004 Composition-Guided Image Acquisition 22
Rule-of-Thirds Algorithm
Compute center-of-mass of main subject 2 multiply-accumulates, 1 memory read per pixel 1 division per image
Locate closest one-third corner 8 compares per image (4 comparisons of (x,y) points)
Shift picture so center-of-mass falls at desired corner Mirror undefined boundary pixels Best case: no change to image Worst case: 1/3 rows/columns need to be shifted Average (main subject in middle): shift 1/6 rows/columns 0 to 2 memory accesses per pixel
Contribution #2
4/28/2004 Composition-Guided Image Acquisition 23
Ideal Background Blur Example
Contribution #2
Background blur emphasizes main subject, the shell, and aids in constrained image communication
Indoor setting; no humans in picture
4/28/2004 Composition-Guided Image Acquisition 24
Simulated Background Blur
Possible camera blurs Background blur: shutter aperture Linear blur: subject or camera motion Radial blur: camera rotation Zoom: change in zoom
Digital alternatives Original image masked with detected main subject mask Region of interest filtering performed on non-masked pixels Complexity: 9 multiply-accumulates and 4 memory accesses
per pixel for convolution with symmetric 3x3 filter
Contribution #2
4/28/2004 Composition-Guided Image Acquisition 25
Results (1)
Supplementary image with main
subject(s) in focus
Detected main subject mask
Rule-of-Thirds: Main subject repositioned
Simulated background blur
Contribution #2
Outdoor setting; human main subject
4/28/2004 Composition-Guided Image Acquisition 26
Results (2)
Supplementary image with main
subject(s) in focus
Detected main subject mask
Rule-of-Thirds: Main subject repositioned
Simulated background blur
Contribution #2
Outdoor setting; human main subject
4/28/2004 Composition-Guided Image Acquisition 27
Results (3)
Supplementary image with main
subject(s) in focus
Detected main subject mask
Rule-of-Thirds: Main subject repositioned
Simulated background blur
Contribution #2
Indoor setting; no human subjects
4/28/2004 Composition-Guided Image Acquisition 28
Outline
Introduction Contributions
Main subject detection Aesthetic enhancement, given main subject Mitigation of mergers with background objects
Framework for background analysis and merger detection Low-complexity one-pass algorithm for merger mitigation
Conclusions
4/28/2004 Composition-Guided Image Acquisition 29
Ideal Merger Mitigation Example
Contribution #3
Unwanted mergers avoided
Background bar merges with gymnast’s hand
4/28/2004 Composition-Guided Image Acquisition 30
Mitigation of Mergers: Overview
Goal: Identify background objects merging with main subject In-focus background object Connected to main subject mask Large area relative to image size
Merger detection Color segmentation based on hue Identify distracting background
object based on distance to main subject and frequency content
Blur merging background objects to induce a sense of distance
Contribution #3
Merging background objects: trees and bush
over right shoulder
4/28/2004 Composition-Guided Image Acquisition 31
Segmentation of Background Objects
Hues above histogram average are dominant hues Background is a mixture of dominant hues Thresholds: average of two consecutive dominant hues
Contribution #3
Background hues
Histogram of background hues and identified objects
Thresholds = {87, 151}
4/28/2004 Composition-Guided Image Acquisition 32
Merger Object Detection Define Frequency Inverse Distance Measure for
each disjoint background object Oi Decreases with nearest distance (di) from main subject Increases with high spatial frequency coefficients (ωi
H)
Merged object: Object with highest transform value
form lExponentia ),(
formDivision ),(
),(
formLinear y)(x,)),(1(
i
i
i
Oy)(x,
),(
Oy)(x,
Oy)(x,
yxdHii
i
Hi
i
Hiii
ieyx
yxd
yx
yxd
Contribution #3
4/28/2004 Composition-Guided Image Acquisition 33
Measure Selection
Linear, division, and exponential forms to combine High frequencies computed with residual in Gaussian
pyramid decomposition Euclidean distance measured from main subject mask
Attribute Linear Divisional Exponential
Computational complexity
Low High High
Merged object’s size
Large Small Small
Contribution #3
4/28/2004 Composition-Guided Image Acquisition 34
Merger Mitigation ResultsBackground tree and bush merging with main subject
High frequency and inv. distance values for
background
Blurred tree and bush appear to be farther away
Contribution #3
4/28/2004 Composition-Guided Image Acquisition 35
Per-pixel Implementation Complexity
Contribution #3
Process /Operation Multiply-accumulates
Compares Memory accesses
RGB to hue 3 6 4
Histogram and thresholding 1 2
RGB to intensity 2
Gaussian pyramid 9 4
Approx. inv. distance measure 2 1 2
Detect merged object 1 1
Gaussian pyramid reconstruction
9 1 5
TOTAL 27 11 15For comparison, JPEG compression takes ~60 operations/pixel
4/28/2004 Composition-Guided Image Acquisition 36
System Prototype
Generated picture with blur Merger mitigated picture
Measure how close rule-of-thirds
followed
Scene
Automate rule-of-thirds
Simulate background blur
Generated picture with rule-of-thirds
Binary main subject mask
Intensity Gaussian pyramid
Background segmentation
Inverse distance
transform
Grayscale image
X
Detect merging object
Grayscale image
Reconstruct color pyramid
Color Gaussian pyramid
Transform coefficients
3x3 Highpass
filter
Detect sharper
edges
Close boundary
Auto-focus filter
Open shutter for blur
Compute intensity
Original color image
Supplementary image
4/28/2004 Composition-Guided Image Acquisition 37
Conclusion
Contributions Combined optical/digital image acquisition Provide online feedback to amateur photographers Low-complexity one-pass method for main subject detection Rule-of-thirds: placement of the main subject on the canvas Simulated background blur: motion and depth-of-field Mitigation of mergers with background objects
Deliverables Prototype development for digital still image acquisition Copies of MATLAB code, slides, and papers, available at