"Designing and Implementing Camera ISP Algorithms Using Deep Learning and Computer Vision," a Presentation from Motorola

Copyright © 2017 Motorola Mobility LLC 1

Val Marchevsky

May 2017

Designing and Implementing Camera ISP

Algorithms Using Deep Learning and

Computer Vision


Quality Image Preserves Your Memories

Examples


Subtle Differences Matter


People Want Top Notch Cameras

Camera is a top priority (Unaided)2015 PUF Study

BrazilUS


When Your Camera Is Good, Phone is Good

• Customers get it. They

want a good camera on

their smartphones.


What is Image Quality?

• Focus

• Sharpness

• Preservation of Texture

• White Balance

• Contrast

• Exposure

• Noise

• Artifacts

• Stabilization

Image Credit DXO Labs


What makes Image Quality Challenging?

• Subjectivity

• Competing goals (sharpness /

texture)

• Lab performance vs. real-

world performance

• Corner cases (wrong focus,

green dogs)


How does Lenovo/Motorola use machine

learning to make better cameras?

• No Reference Image Quality Analysis (NR-IQA)

• SVM-based HDR trigger

• Focus Failure Detection

• Estimated MOS

• DxOMark analytics

• Neural Network AWB


• Optimizers

• AdaBoost

• Support Vector Machines

• Linear Regression

• Neural Networks, CNNs and Stochastic Gradient Descent

• Frameworks

• Caffe

• TorchFlow

• MXNet

• Moto-proprietary

What Machine Learning Technologies Does Lenovo/Motorola Use?


• Motorola developed General

Image Quality Index (GIQI),

a regression model that

computes an estimate of

mean opinion score (MOS).

GIQI is a CNN application,

with its own network

definition, “MOSNet”

No Reference Image Quality Analysis (NR-IQA)

MOS Per Scene


GIQI Examples


GIQI and MOSNet

MOSNet is trained on a large body of

artificially distorted images

To speed convergence, it uses transfer-

learning and borrows its initial weight values

from a standard pre-trained Caffe model

x 2


GIQI Performance on Public Datasets

• Comparison against the top performing algorithms on the public datasets

including LIVE in the Wild Challenge Database

• 100 random train-test (80/20) splits


Using GIQI Estimated MOS for Product Evaluation

• Motorola uses GIQI to

compute probability

distributions of estimated

MOS for comparing

product performance


GIQI Estimated MOS vs. Psychometric Evaluation

Psychometric

Evaluation

(man)

GIQI

(machine)

Man vs. Machine


Machine Assessed Image Quality

• Effective Analysis Of Corner Cases

• Rapid Iteration of Development Process

• Competitive Analysis

• Relative Parity with Human Observers


Auto Focus Failure Detection

Problem: We spend way too much time analyzing

perceived Auto Focus failures. Can we have a machine filter

gross or all failures out?

Vector: Software

Solution: Use machine-learning based classifier to improve auto-trigger

performance. Initial support-vector based solution improves recall by over

30% with no degradation in precision. Results improved with expanded

dataset.

Markets: all

End Users:

• Better quality focus solution where we can analyze true failures and

concentrate on real issues

Iterations Chart credit DXO Labs


AF-FD Examples

Image

Quality

Details:

Focus Class:

Out of focus

Focus Score:

0.6843

Image

Quality

Details:

Focus Class:

Out of focus

Focus Score:

0.7237


• Motorola developed an

autofocus failure detection

classifier

• Based on engineered features

including natural scene

statistics

• Best results were obtained

with AdaBoost optimization

GIQI and MOSNet

Image distortions deform the Gaussian

shape of natural scene statistics

Figure credit: Mittal, Anish, Anush Krishna Moorthy, and

Alan Conrad Bovik. "No-reference image quality

assessment in the spatial domain." IEEE Transactions on

Image Processing 21.12 (2012): 4695-4708.


AF-FD: Sample Results after Training


AF-FD: Real-world Results on User Trial Data


• Huge reduction in escaped focus defects

• Higher complexity algorithms without higher risk penalty

• Error classification and data mining

• Best focus software stack in industry

Auto Focus Failure Detection


HDR Trigger

Problem: Image quality suffers

when HDR trigger is too

conservative (poor recall)

Solution:

Use machine-learning based classifier to

improve auto-trigger performance.

Initial support-vector based solution improves

recall by over 30% with no degradation in

precision.

Results improved with expanded dataset.


We want to use HDR when we can!


• Sometimes best

technology to use

depends on scene

content!

HDR Trigger Comparison

% o

f corr

ect

HD

R T

riggers


• Laboratory for Image and Video Engineering (http://live.ece.utexas.edu)

• Mittal, Anish, Anush Krishna Moorthy, and Alan Conrad Bovik. "No-

reference image quality assessment in the spatial domain." IEEE

Transactions on Image Processing 21, no. 12 (2012): 4695-4708.

Resources

http://live.ece.utexas.edu

http://live.ece.utexas.edu


• Motorola Mobility LLC is a proud partner of the Laboratory of Image and Video Engineering

(LIVE) at the University of Texas @ Austin

• Professor Alan Bovik and his students have generously shared their ideas and talents with

Motorola. Their expertise was crucial in the development and to the success of GIQI and

AF-FD

• LIVE developed Natural Scene Statistics (NSS) and continues to pioneer research and

advances in the field of NR-IQA

• Motorola Mobility thanks DXO for being an advocate for consumer Image Quality and letting

us use their public data

Acknowledgements


Note: Moto branded products are designed and

manufactured by or for Motorola Mobility LLC,

a wholly owned subsidiary of Lenovo.

"Designing and Implementing Camera ISP Algorithms Using Deep Learning and Computer Vision," a Presentation from Motorola

Technology