Photorealistic Models for Pupil Light Reflex and Iridal Pattern Deformation

Photorealistic Models for Pupil Light Reflex and Iridal Pattern Deformation

Vitor F. Pamplona Manuel M. Oliveira Gladimir V. G. Baranoski

• The most important feature in facial animation

Eyes

© Marco Ruiz - http://bit.ly/bba1qw

Pupil Light Reflex (PLR)• Involuntary movement of the pupil• Deforms the iris patterns

4

Close-Ups Require Natural Looking Eyes

Image: http://wallpaper-s.org

5

Contributions• A Physiologically-based Model for PLR– First to simulate pupil dynamics under variable lighting– All parameters derived from experimental data– Real-time predictable animations– Support for individual variability

• A Model Iridal Pattern Deformation

6

Contributions• A Physiologically-based Model for PLR– First to simulate pupil dynamics under variable lighting– All parameters derived from experimental data– Real-time predictable animations– Support for individual variability

• A Model Iridal Pattern Deformation

• Realistic animations from physically meaningful data

7

Our Models in Action

8

Related Work• Moon & Spencer’s model (1944)

• Fitting of Experimental Data

Moon, P. and Spencer, D.. On the stiles-crawford effect. J. Opt. Soc. Am. 1944

104.9 3tanh(0.4(log ( ) 0.5))mm bD L

9

Related Work• Moon & Spencer’s model (1944)

• Fitting of Experimental Data• Static


104.9 3tanh(0.4(log ( ) 0.5))mm bD L

10

The Dynamics of PLR

Brain

Retina

11Brain

Retina

The Dynamics of PLR

12Brain

Retina

The Dynamics of PLR

13Brain

Retina

The Dynamics of PLR

14Brain

Retina

The Dynamics of PLR

15Brain

Retina

The Dynamics of PLR

16Brain

Retina

The Dynamics of PLR

17Brain

Retina

The Dynamics of PLR

18

Related Work (Cont.)• Longtin & Milton’s model (1989)

• Tries to model the dynamics of PLR• Uses a delay differential equation

Longtin, A. and Milton, J. G.. Modelling autonomous oscilations in the human pupil light reflex using non-linear delay-differential equations. Bulletin of Math. Bio. 1989.

min

)(ln)( 2

tAg

dt

dA

dA

dgmm

19Longtin, A. and Milton, J. G.. Modelling autonomous oscilations in the human pupil light reflex

using non-linear delay-differential equations. Bulletin of Math. Bio. 1989.

Pupil Area

min

)(ln)( 2

tAg

dt

dA

dA

dgmm



20

Retinal Light Flux


min

)(ln)( 2

tAg

dt

dA

dA

dgmm

Pupil Area



21

Retinal Light Flux


Latency

min

)(ln)( 2

tAg

dt

dA

dA

dgmm

Pupil Area



22

• Longtin & Milton’s model (1989)• Tries to model the dynamics of PLR• Uses a delay differential equation

Retinal Light Flux


Muscular Activity

Pupil Area

Latency

min

)(ln)( 2

tAg

dt

dA

dA

dgmm

Related Work (Cont.)

23Longtin, A. and Milton, J. G.. Modelling autonomous oscilations in the human pupil light reflex

using non-linear delay-differential equations. Bulletin of Math. Bio. 1989.

• Longtin & Milton’s model (1989)• Several unknowns

Function

Constant

Latency

Constant

Related Work (Cont.)

min

)(ln)( 2

tAg

dt

dA

dA

dgmm

24

Outline

• Physiologically-based Model for PLR

• Iridal Pattern Deformation Model

• Results

• Summary

25

Summary• Physiologically-based model from Longtin & Milton

• Static model of Moon & Spencer

min

)(ln)( 2

tAg

dt

dA

dA

dgmm

))5.0)((log4.0tanh(39.4 10 bmm LD

26

Pupil Light Reflex Model• Longtin & Milton’s (L&M) Model

min

)(ln)( 2

tAg

dt

dA

dA

dgmm

27

Pupil Light Reflex Model• L&M model under constant illumination condition

becomes

min

)(ln)( 2

tAg

dt

dA

dA

dgmm

]ln[ln)( min2 mmAg Longtin & Milton

28

Pupil Light Reflex Model

min

)(ln)( 2

tAg

dt

dA

dA

dgmm

]ln[ln)( min2 mmAg

Moon & Spencer

Longtin & Milton

))5.0)((log4.0tanh(39.4 10 bmm LD

• L&M model under constant illumination condition

becomes

29


min

)(ln)( 2

tAg

dt

dA

dA

dgmm

]ln[ln)( min2 mmAg

]1513.1)[ln(4.03

9.4atanh3.2

bmm LD Moon & Spencer

Rewritten

Longtin & Milton


becomes

30


min

)(ln)( 2

tAg

dt

dA

dA

dgmm

]ln[ln)( min2 mmAg

]1513.1)[ln(4.03

9.4atanh3.2


Rewritten

Longtin & Milton


becomes

31


min

)(ln)( 2

tAg

dt

dA

dA

dgmm

]ln[ln)( min2 mmAg

]1513.1)[ln(4.03

9.4atanh3.2


Rewritten

Longtin & Milton


becomes

32


min

)(ln)( 2

tAg

dt

dA

dA

dgmm

]ln[ln3

9.4atanh3.2 min

mmD

]1513.1)[ln(4.03

9.4atanh3.2


Rewritten


becomes

33


min

)(ln)( 2

tAg

dt

dA

dA

dgmm

]1513.1)[ln(4.03

9.4atanh3.2


Rewritten


becomes

]ln[ln3

9.4atanh3.2 min

mmD

34


min

)(ln)( 2

tAg

dt

dA

dA

dgmm

]1513.1)[ln(4.03

9.4atanh3.2

bmm LD

From Moon & Spencer

model]ln[ln

3

9.4atanh3.2 min

mmD


becomes

35


min

)(ln)( 2

tAg

dt

dA

dA

dgmm

]1513.1)[ln(4.03

9.4atanh3.2

bmm LD

]4.8x10 ln[ln3

9.4atanh3.2 10-

mmD


becomes

36


min

)(ln)( 2

tAg

dt

dA

dA

dgmm

]1513.1)[ln(4.03

9.4atanh3.2

bmm LD

]4.8x10 ln[ln3

9.4atanh3.2 10-

mmD


becomes

37


min

)(ln)( 2

tAg

dt

dA

dA

dgmm

]1513.1)[ln(4.03

9.4atanh3.2

bmm LD

2.5]4.8x10 ln[ln45.03

9.4atanh3.2 10-

mmD


becomes

38


min

)(ln)( 2

tAg

dt

dA

dA

dgmm

2.5]4.8x10 ln[ln45.03

9.4atanh3.2 10-

mmD


becomes

39

Static Model Comparison

Moon & SpencerOur Model

40


Moon & SpencerOur Model

Just for the equilibrium case

41


min

)(ln)( 2

tAg

dt

dA

dA

dgmm

2.5]4.8x10ln[ln45.03

9.4atanh3.2 10-

mmD


becomes

42


min

)(ln)( 2

tAg

dt

dA

dA

dgmm

2.5]4.8x10ln[ln45.03

9.4atanh3.2 10-

mmD


becomes

43

Pupil Light Reflex Model• The Dynamic Model for PLR

1010x8.4

)(ln45.02.5)(3.2

tDg

dt

dD

dD

dgmm

44

Pupil Light Reflex Model• The Dynamic Model for PLR

3

9.4atanh)( mm

mm

DDg

1010x8.4

)(ln45.02.5)(3.2

tDg

dt

dD

dD

dgmm

45

PLR Model: Individual Sensitivity to Light• Mapped as iso-curves in Moon & Spencer’s data


46



Cb

Ct

Cm

47



Cb

Ct

Cm

Subject X

PLR Model Validation• Videos captured while changing the illumination– The illumination was measured (lux meter)– The pupil size in mm was estimated for each frame

49

PLR Model: Individual Variability

50


51


52


53

Outline

• Physiologically-based Model for PLR


• Results

• Summary

54

Iris Texture

55

Feature Tracking

56

Feature Tracking

57

Feature Tracking

58

Feature Tracking

59

Feature Tracking

60

Tracking Results

61

Relative Position to the Iris Border

rp

r

pR

62


rp

r

pR

63


rp

r

pR

64


rp

r

pR

65


r

pr

pR

Simulation of Iris Deformation

Comparison Against Photographs

67Rendering Picture

Orig. Texture

Comparison Against Photographs

68Rendering Picture

Orig. Texture

69

Final Result

70

Summary• A Physiologically-based Pupil Light Reflex (PLR) Model– First to simulate pupil dynamics under variable lighting– All parameters derived from experimental data– Real-time predictable animations– Support for individual variability


• Produce photorealistic animations in real time

71

Possible Applications• Screening tool for – Eye diseases such as … XXXX– Intoxication by alcohol/drugs– An average healthy subject model for comparison

• Ophthalmologic tool for simulations and training

• Iris recognition without controlled illumination

Acknowledgments• Volunteers and Collaborators• Dr. Jacobo Melamed Cattan, MD• Prof. Roberto Silva• Prof. Luis A. V. Carvalho• Leandro Fernandes, Marcos Slomp and Renato Silveira• CNPq-Brazil fellowship (305613/2007-3)• NSERC-Canada grant (238337)• Microsoft Brazil


Vitor F. Pamplona Manuel M. Oliveira Gladimir V. G. Baranoski


Vitor F. PamplonaManuel M. Oliveira

Gladimir V. G. Baranoski

76

PLR Model: Light Bulb Validation

77

PLR Model: Light Bulb Validation

78

PLR Model: Flash Light Validation

* Light intensity obtained from the inverse of Moon and Spencer

79

Applications• Photorealistic animations

– Non-adhoc automatic animation of the iris-pupil system

• Tool for diagnostics– An average healthy subject model for comparison– Ophthalmologic simulations

• Iris recognition– Controlled illumination is not needed anymore

80

Approximating Hippus Involuntary variations on the pupil size

Under constant illumination conditions Frequency of 0.05 a 0.3Hz Aplitude about 0.2mm

Implementation

Random perturbations in the light source From -100.3 to 100.3 Blondels In a frequency range of 0.05 a 0.3Hz

Stark, L. W., Sherman, P. M. A servoanalytic study of consensual pupil reflex to light. J. Neurophysiol, 1959. Hachol, A. et al. Measurement of pupil reactivity using fast pupillometry. Physiol. Meas., 2007.

81

Wyatt’s model for Iris Structure• Model for iris’ collagen fibbers

• Linear and radial deformation plus a non linear component.

Wyatt, H. J. A minimum-wear-and-tear meshwork for the iris. Vision Research. 2000.

82


83

PLR Model: Velocities• Included in the numerical step• Constriction is 3 times faster than dilation

• S is a constant that may change among individuals

dtconstriction=Tms

current−Tmsprevious

S

dtdilation=Tmscurrent−Tms

previous

3S

Ellis, C. J. The pupillary light reflex in normal subjects. British J. of Ophthalmology, 1981.Bergamin, O. et al. The influence of iris color on the pupillary light reflex. Graefes Arch. Clin. Exp. Ophthalmol., 1998.

84

PLR Model: Individual Variability• Mapped as iso-curves in Moon and Spenser’s data


85

Defformation model: Individual Variability

86

Defformation Model: Feature Tracking

87

Modelo 2: Tracking das Saliências

Copyright Vitor Pamplona 88

The problem of an static latency Model

89

Photorealistic Facial Animations

91

Challenges• Under variable lighting conditions:– How does the pupil change?– How does the iris pattern deform?


Eyes and Facial Animation


93

Physiologically-based models [Animar o fluxo]

Privitera, C. M., Stark L. W. A. Binocular Pupil Model for Simulation of RelativeAfferent Pupil Defects and the Swinging Flashlight Test. Bio. Cyber. 2006.

94

Contributions• Pupil Light Reflex (PLR) model– First practical model to simulate pupil dynamics under

variable lighting conditions– First integrated model with support to individual

variability, latency, and constriction/dilation velocities

• Iridal pattern deformation model– First model validated with human eyes.

95

Summary• Physiologically-based model from Longtin & Milton– Dynamic– Constants and functions undefined

• Experiment-based model from Moon & Spencer– Static– Average subject

2

min

( )( ) ln

mm

dg dA tg A

dA dt

104.9 3tanh(0.4(log ( ) 0.5))mm bD L

96

Summary• Physiologically-based model from Longtin & Milton– Dynamic– Constants and functions (g) undefined

• Experiment-based model from Moon & Spencer– Static– Average subject

2

min

( )( ) ln

mm

dg dA tg A

dA dt

104.9 3tanh(0.4(log ( ) 0.5))mm bD L

Eyes and Emotions

Photorealistic Models for Pupil Light Reflex and Iridal Pattern Deformation

Technology

pupil dynamics

plr model

based model

pupil diameters

pupil movements

brainretinathe dynamics

human pupil light reflex

longtin miltons model