A Framework for Photo-Quality Assessment and Enhancement based on Visual Aesthetics Subhabrata Bhattacharya Rahul Sukthankar Mubarak Shah.

A Framework for Photo-Quality Assessment and

Enhancement based on Visual Aesthetics

Subhabrata Bhattacharya Rahul Sukthankar Mubarak Shah

Reference http://www.cs.ucf.edu/~subh/

Outline

Introduction

Learning Aesthetics

Enhancing Composition

Experimental Results

Introduction

Assessing the quality of photographs is challenging .

Experienced photographers adhere to several rules of composition .

Rule of Thirds

Visual Weight Balance

Subject of interest is aligned to one of the stress points.

Rule of Thirds

Rule of Thirds : Example

In a well composed image the visual weights of different regions satisfy the Golden Ratio .

Visual Weight Balance

Sea

Sky

k

~1.618k

Visual Weight Balance : Example

Introduction

In this paper, will use these two rules to assess an image .

Formulate photo quality evaluation as a machine learning problem .

Overview

Learning Aesthetics

Dataset

User Survey

Aesthetic Features

Learning and Prediction

Dataset

Single subject Compositions (384)

Landscapes/Seascapes (248)

User Survey

15 participants were asked to assign integer rank from 1 to 5.

Each user was asked to rank no more than 30 images.

Generate single ground truth for each image (Fa).

User Survey

Aesthetic Feature

Extract a relative foreground position feature for images with single-foreground compositions.

A visual weight ratio feature for photographs of seascapes or landscapes.

Defined as the normalized Euclidean distance between foreground’s mass to each four stress points.

Relative foreground position

Relative foreground position

Relative foreground position

The ratio of the sky region, to that in the support region ( ground or sea).

Visual weight ratio

Yg

Yk

Visual weight ratio

Visual weight ratio

Learning and Prediction

We use SVR to learn the mappings.

150 random images for training and resting for testing.

Enhancing Composition

Relocate the foreground object to increase the predicted appeal factor.

Better balancing the visual weights of the sky and support region.

Why Cropping does not work?

Optimal Crop

Semantic Segmentation

Input ImageGeometric

Context Classifier*

*D. Hoiem, A.A. Efros, and M. Hebert, "Geometric Context from a Single Image", ICCV 2005

Sky

Support

Post Processin

gHorizon

Segmented

Foreground

Optimal object placement

Support Neighborhoo

d

s.t. neighbors stay “like neighbors”

+Intensity Term Gradient Term

Example

PAF = 3.22

Original Image

PAF = 4.53

Optimal Solution

Rescaling

Scaling Factor

Vanishing Point

Optimal location

Visual Attention Center

Inpainting Foreground Hole

Yunjun Zhang. Jiangjian Xiao. Mubarak Shah, “Region Completion in a Single Image”, EUROGRAPHICS 04

Inpaint Hole

Balancing visual weights

Yk

Yg

Ratio of Current extents

Yk +h

Yg

h = vertical extent of the balanced image

Experimental Results

PAF = 3.77 PAF = 4.25

Before After

Experimental Results

PAF = 3.92 PAF = 4.11

Before After

PAF = 3.98

PAF = 4.46

Experimental Results

Before After

PAF = 4.02

PAF = 4.34

Before After

Experimental Results

PAF = 3.13

PAF = 4.19

Experimental Results

Before After

PAF = 3.83

PAF = 4.02

Experimental Results

Before After

PAF = 3.92

PAF = 4.38

Experimental Results

Before After

Experimental Results

PAF = 4.02

PAF = 4.71

Before After

Experimental Results

PAF = 4.17

PAF = 4.49

Before After

Optimal Placement

Visual Weights

Failure case

PAF = 2.34Fa = 2.41 (Ground Truth)Before

PAF = 3.63Fa = 2.54 (Ground Truth)After

Thank You