Elaborazioni di Immagini per Dispositivi Mobilebattiato/EI_MOBILE0708/... · WB Statistic ADVANCED SYSTEM TECHNOLOGY 13 March 2008 8 Exposure ... Feature extraction Correction curve

1

STMicroelectronics

Advanced System Technology13 March 2008

Elaborazioni di Immagini per Dispositivi Mobile

Ing. Alessandro Capra

ADVANCED SYSTEM TECHNOLOGY13 March 2008 2

Agenda� Introduction

� Mobile camera Devices

� Pre-processing: Auto Focus, Auto Exposure

� Color interpolation� Color interpolation� Noise management� Noise estimation� Croma management� Codecs still/video� Codecs still/video� Applications� Red eye

2


Miniature camera: inside view

Sensor CMOS

IR Filter

Lens

Lens Holder

Substratus


System: The Yellow Duck

3


EXPOSURE CORRECTION


Exposure

One of main problems affecting image quality comes from improper exposure adjustment.

Most existing techniques to improve tonal quality (i.e. histogram equalization, gray level slicing…) are blind tovisual content of image.

There’s no way to define exactly a correct exposure but ideally most relevant features of the scene should occupymid tone gray levels.

4


CMOS Sensor

Exposure

CFAB

GR

Integrationtime

Analoggain

Digitalgain

EC

WB

Statistic


Exposure

Anti Flickering Filter: reduce the light flickering during the videoacquisition

Leaky Integrator

FinFout

1

)1

1()1(1

)()(

≥

−⋅−+⋅=

KK

tFK

tFtF outinout

5


Visibility Image 1/3

Since extraction of features can be difficult in highly underexposed or overexposed images, the mean gray level of the Y channel is forced to be ≈128.

ORIGINAL CORRECTED


Visibility Image 2/3

Classical EC can be considered as a “roughly” correction. Correction based on relevant zones in most cases produces better results.

FLAT CORRECTION RELEVANT ZONES CORRECTION

6


Computation Of Measures

N

igradtrhF

N

i

)|)((|1∑

==

• N number of pixels in the block • grad(i) output of Laplacian (or Sobel)

3××××3 filter for pixel i• Trh operator discards low values

(noise)

∑

∑

=

=⋅−

= 255

0

255

0

)(

)(

k

k

kH

kHMkC

• M mean gray level of block• H(k) histogram bin for gray value k

Contrast

Focus

-1

4

-1

0

-1

0

0

-1

0

Laplacian filter


Relevant blocks identification

Y channel

FC FC FC FC FC FC FC








Measures on non-overlapping blocks









Relevant blocks identification:VS=a*F+b*C>T (a+b=1)

All measures are normalized in the range [0:1].

7


Relevant regions example


Examples

input output

8


Examples

input output


Skin detection 1/2

• Probabilistic model of distribution of Cr, Cb channels used to assign reliability of skin pixels classification

• Pixels with probability of being skin>T are used by exposure correction

INPUT SCALED SKIN PROBABILITIES

9


Skin detection 2/2

( ) [ ]

−=

−

2

)x(dexp

2

1s|xp

22

1 rr Σ

π

[ ] )x()'x()x(d1

2 µµ Σ rrrrr−−=

−

Skin PDF over Cr,Cb plane can be modeled as a 2D gaussian function. Skin probabilities are well identified on chromatic plane.

Skin probabilities

Possible solutions to reduce complexity:1. Store portions of skin probabilities on 2D LUTS (indexed on Cr, Cb);2. Store bounds (on CrCb plane) of skin region.

vectorechrominanc matrix, covariance xr

∑

rmean vecto metric, or,color vect usMahalanobidxrr


Example

input output

10


Possible speed improvement 1/3

Input imageFeature extraction

Correctioncurve

Output image

Scaled image(i.e. preview,

RGB from BPmacro pixels)


Possible Speed improvement 2/3

INPUT

FULL RESOLUTION

1/4 RESOLUTION

11


Possible Speed improvement 3/3

FULL RESOLUTION

1/4 RESOLUTION

INPUT


Local exposure correctionExposure correction often works poorly especially in presence ofareas of very different illumination.

A better strategy would be applying different corrections in different areas.

INPUT FAILED OUTPUT

12


Local Correction 1/4

INPUT OUTPUT



input output

13



input output



input output

14


Comparison 1

OLD NEW


Comparison 2

OLD NEW

15


Comparison 3

OLD NEW


AUTOFOCUS

16


Optical principles

f1

D1

B1 =+


Optical principles

e= focus error

c= diameter of circle of confusion

g= N·max(c)

17


Optical principles

e= focus error

c= diameter of circle of confusion

g= N·max(c)


Lens: the focus system

Most single-lens reflex cameras use an autofocus method called the "phase detection system." Using a separator lens in autofocus module, this system produces two images from the image information of the subject captured through the lens. It then measures the distance between those two images using a line sensor, and detects defocus amount.

18


Lens: the focus system

Autofocus of compact cameras uses a mechanism called the "contrast detection system." Based on the principle that "in focus = highest contrast," this system analyzes the image information of the subject obtained by an image sensor. Then, by moving the lens, this system seeks the lens position where the image contrast is highest.


AutoFocus: Development� What is the end-users desire?

� To not know that an Auto-Focus algorithm is operating!

� Perfect pictures every time at a push of a button� The same intelligence as you and me!!!

� Why?

� They don’t care

� Often hi-res pictures are viewed later, they do not know the picture is in/out of focus until it is too late

� Latency to snap is the number 1 reason for bad pictures (particularly of children and animals)

� They think it is easy (just like their eyes) …. So do you?

19


AutoFocus

RequirementsRequirements� In-focus positioning related to frequency information

� Stable video focusing

� Object tracking

� Multi-objects multi-focus management

� Fast execution and low complexity

Disturbance ElementsDisturbance Elements� Light condition

� Number of edges

� Noise

� Homogeneous regions

� Motion Blur


Statistics Examples

∑∑

∑∑

=

=

n

x

m

y

n

x

m

y

yxureaplaceMeas

yxureaplaceMeas

),(LAbsoluteL

),(LSquareL 2

0-10

-14-1

0-10

Classical Laplacian:

-10-1

040

-10-1

Diagonal Laplacian

∑∑

∑∑

=

=

n

x

m

y

n

x

m

y

yxureaplaceMeas

yxureaplaceMeas

),(LagonalLAbsoluteDi

),(LonalLSquareDiag 2

20


Focus Measure

Motor Steps

Measure window

140 160 180 200 220 240 2602

2.2

2.4

2.6

2.8

3

3.2

3.4

3.6x 10

7

Step motor

Foc

us M

easu

re Square Laplace Measure

140 160 180 200 220 240 2605

5.1

5.2

5.3

5.4

5.5

5.6

5.7x 10

5

Step motor

Foc

us M

easu

re

Absolute Laplace Measure

Step motor

Foc

us M

easu

re Square Diagonal Laplace Measure

140 160 180 200 220 240 2601.3

1.4

1.5

1.6

1.7

1.8

1.9

2

2.1

2.2

2.3x 10

7

Absolute Diagonal Laplace Measure

Step motorF

ocus

Mea

sure

140 160 180 200 220 240 2604.5

4.55

4.6

4.65

4.7

4.75

4.8

4.85

4.9

4.95

5x 10

5


Advanced AF

BB

BB

FF

FF

FF

BB

BB

�Multi-zone analysis�Tracking�Object detection & Motion Analysis

21


AF Functional View

Actuator

DefCor GStatistics

AE CtrlLoop

AF CtrlLoop

Param

s


Statistics

GZoning HP Filters Accum Weighting Fm

22


AF Control loop

Fm

Do_Nothing

Start End

LoopSupervisor

PeakSearch

LeakyIntegrat.

TrackingDecision

Position Internal Representation

Mapping to Actuator command

Reset

Params

Do_Nothing

Coarse

Fine

Tracking


Actuator Review

1. Electromagnetic Actuators1. Stepper Motors2. Simple Solenoids3. Voice Coil Solenoids

2. Piezoelectric Actuators1. Stacked Piezo Devices2. Bimorphs3. Disk Translators4. Moonie Motors5. Helimorphs6. Oscillating Bimorphs7. Inch Worms8. Ultrasonic disk motors

3. Electrostatic Actuators4. Electrostrictive Actuators5. Magnetostrictive Actuators6. Shape memory alloys7. MEMS Actuators

Available Technologies For electro-mechanical Micro Actuation:

23


Stepper Motor: MaxEmil, ALPS

Largely used in DSC and SRL where actuator size is not relevant. Quick and precise actuator


Piezo motion solution: Miniswys

24


Established technologies: voice coil solution

Type of motor used for most Speakers for Hi-Fi

�Used in Sony-Ericsson K750i camera phone�Proposed by SEMCO (Samsung Electro-Mechanics)

Magnet

Winding / sheath


What is MSM?

� MSM is a voicecoil driven actuator with a twist – it has a static holding force

� Technology developed by Philips Applied Technologies

� Available to Philips High Technology plastics for use in camera modules

� A very clever and simple trick is used…

25


MSM concept

Lens carriage

Module housing

Guidance pole (ferrous)

Coil

Magnet


MSM concept – no current in coil

Magnet attracted to poles – holding force generated by pole/carriage friction

26


MSM concept – pulsed current in coil


1Limited actuator� Actuator is based on piezo in 2 or more layers

� By putting a voltage on the layers the different expansion of the layers can cause a bend

� 1Limited use a patented shape, the helimorph, to magnify the deflection to make it larger than a simple bender

27


1Limited piezo design


Electro-wetting: Varioptic

Varioptic lens technology

Innovative solution. Low cost and high miniaturisation. Still under development to improve speed, HQ optical response, mechanical resilience.

28


DGITAL AUTOFOCUS


CDM

29



� Conventional concept

� Eyesquad

Object Best lens ImageSensors Processing

ObjectDistorted lens

ImageSensorsEyesquad’s

Processing

Both parts are inter-related

30



Q&A

31


Bibliography

� http://www.dpreview.com� http://www.nikon.co.jp/main/eng/portfolio/about/technology/index.htm� http://www.canon.com/technology/index.html� http://www.kodak.com� http://www.learningcenter.sony.us� http://www.st.com/stonline/products/families/imaging/imaging.htm� http://www.samsung.com� http://www.philips.com

� http://www.trenholm.org/hmmerk/index.html

� Image Sensor Architectures for Digital Cinematography, DALSA Digital Cinema� Active Pixel Sensor (Aps) Design - From Pixels To Systems, Orly Yadid-Pecht� CCD vs CMOS, Nicolas Blanc� An advanced video camera system with robust AF, AE, and AWB control, IEEE� Selecting the Optimal Focus Measure for Autofocusing and Depth-From-Focus,

Murali Subbarao and Jenn-Kwei Tyan

Elaborazioni di Immagini per Dispositivi Mobilebattiato/EI_MOBILE0708/... · WB Statistic ADVANCED SYSTEM TECHNOLOGY 13 March 2008 8 Exposure ... Feature extraction Correction curve

Documents