An Affordable Solution for Binocular Eye Tracking and
Calibration in Head-mounted Displays
An Affordable Solution for Binocular Eye Trackingand Calibration
in Head-mounted DisplaysMichael Stengel (TU Braunschweig), Steve
Grogorick (TU Braunschweig),Elmar Eisemann (TU Delft), Martin
Eisemann (TH Kln), Marcus Magnor (TU Braunschweig)ACM Multimedia
2015, Brisbane
Michael Stengelstengel @ cs.tu-bs.de
1
Immersive media is getting popular.Photographic footage by
shutterstock.com
2
Unreal Engine and footage. Copyright by Epic Games,
International.Support for Real-time gaze-tracking is missing !
3
binocular eye-trackinginexpensive components< 400 US$low
latency ( 12-17 ms)modular designefficient user calibrationfull
field of view (100 per eye)uniform tracking accuracyfully
adjustable lensesno external camerasOur solution.
4
BINOCULAR EYE TRACKING PROTOTYPEHMD TECHNOLOGY - TODAY AND
TOMORROWCALIBRATION AND TRACKINGVR APPLICATIONS
5
HMD Technology Today.Wide Field of View
LEFT EYERIGHT EYEFOVEAL REGIONHMD SCREEN> 100 FoV per eye to
create immersionbiconvex aspheric lensesfight chromatic abberation
in renderinglarge viewing angles require higher FoV
6
HMD Technology Today.Wide Field of ViewHigh temporal
resolution
refresh rate at least above retinal integration ratestrobe
displays largely reduce display motion bluroptimal display : 1000
HzTemporal Video Filtering for Perceptual Motion Blur[Stengel et
al. 14]
7
HMD Technology Today.Wide Field of ViewHigh temporal
resolutionHigh spatial resolution
Apparent Display Resolution Enhancement[Didyk11, Templin12,
Stengel13]2.5k per eye is state-of-the-artnot sufficient for foveal
region (0.6 0.9 arc minutes) requires pixel resolution of 16k x
16kLena Sderberg. Copyright by USC.
8
HMD Technology Today.Wide Field of ViewHigh temporal
resolutionHigh spatial resolutionInstant visual feedback
Kite. Copyright by Epic Games, Inc.Foveated Gaze-Contingent
Displays[Dorr et al. 06, Duchowski et al. 07, Targino da
Costa,14]tolerable latency depends on application< 20 to 40 ms
to avoid motion sicknessrequires serious render optimization
features
9
HMD Technology Today.Wide Field of ViewHigh temporal
resolutionHigh spatial resolutionInstant visual feedbackAccurate
physical viewpoint
Low latency head trackingnot sufficiently solved for mobile
systemsLarge-Scale Direct Monocular SLAM [Engel et al. 14]
10
HMD Technology Today.Wide Field of ViewHigh temporal
resolutionHigh spatial resolutionInstant visual feedbackAccurate
physical viewpointNatural depth perception (accomodation)
Pursuit Gaze Calibration [Pfeuffer et al.13]Focus Cues Displays
[Narain et al. 15, Huang et al. 15]interpupillary distance
estimation (IPD)stereo rendering using IPDnatural accomodation
currently not supported (no natural depth of field)
11
HMD Technology Today.Wide Field of ViewHigh temporal
resolutionHigh spatial resolutionInstant visual feedbackAccurate
physical viewpointNatural depth perception (accomodation)High
dynamic range imaging
A computational model of afterimages[Ritschel et al.
2012]Simulating the Visual Experience, Tone-Mapping[Benoit et al.
09 , Ritschel et al. 12, Jacobs et al. 15]Currently global tone
mapping
12
HMD Technology Today.Wide Field of ViewHigh temporal
resolutionHigh spatial resolutionInstant visual feedbackAccurate
physical viewpointNatural depth perception (accomodation)High
dynamic range imagingPerceptual realismSmart actors; collaborative
VR with gaze support[Marsella et al. 13, Debevec et al. 12,
Duchowski et al. 05, Steptoe et al. 08]
Digital Ira Project [Activision, Inc. and USC, 2012]
13
HMD Technology Today.Wide Field of ViewHigh temporal
resolutionHigh spatial resolutionInstant visual feedbackAccurate
physical viewpointNatural depth perception (accomodation)High
dynamic range imagingPerceptual realism
Benefits fromgaze-tracking !
Gaze-tracking is the technologyyou are looking for.
14
ERGONEERSFOVESMI DK1 EYE TRACKING UPGRADEARRINGTON RESEARCH EYE
TRACKING KIT>20,000 USD50,000 USD15,000 USDNo details. Not yet
available.
15
Construction blueprints will be freely available.
16
BINOCULAR EYE TRACKING PROTOTYPEHMD TECHNOLOGYCALIBRATION AND
TRACKINGVR APPLICATIONS
17
18
3D-printed body caseFixation using head strapsOrientation
tracker5.6 HD DisplayAspheric lenses
19
3D-printed body caseFixation using head strapsOrientation
tracker5.6 HD DisplayAspheric lensesEye-tracking cameras
20
3D-printed body caseFixation using head strapsOrientation
tracker5.6 HD DisplayAspheric lensesDichroic mirrorsEye-tracking
cameras
21
3D-printed body caseFixation using head strapsOrientation
tracker5.6 HD DisplayFlexible Lens-holderswith illumination
unitsAspheric lensesDichroic mirrorsEye-tracking cameras
22
3D-printed body caseFixation using head strapsOrientation
tracker5.6 HD DisplayFlexible Lens-holderswith illumination
unitsAspheric lensesPosition-tracking cameraDichroic
mirrorsEye-tracking cameras
23
BICONVEX LENSESSCREENCASEUSER
24
INFRARED ILLUMINATIONEYE TRACKINGCAMERAEYE
TRACKINGCAMERADICHROICMIRRORDICHROICMIRROR
25
Infrared LightVisible Light
Dichroic mirror behavior
26
BINOCULAR EYE TRACKING PROTOTYPEHMD TECHNOLOGYCALIBRATION AND
TRACKINGVR APPLICATIONS
27
PRE-CALIBRATION / HMD
Reprojection error aftercamera calibrationUsed lenses from
DK1IOR estimation andphysical renderingLens shape
reconstruction
28
Unkown parametersLens settingsEye locations
29
User specificUnkown parametersLens settingsEye locations
30
CALIBRATION / USER
Average eye assumptionUnknown propertyRelative location
PHYSICAL EYE MODELA photon accurate model of the human eye
[Deering05]The Physiology of the Eye [Adler et al., 08]
31
CALIBRATION / LENS LOCATION
SIMULATED MODELCAMERA CAPTURELEFT EYERIGHT EYE
32
CALIBRATION / LENS LOCATION
CAMERA CAPTURELEFT EYERIGHT EYESIMULATED MODEL
33
CALIBRATION / EYE BALL LOCATION
CAMERA CAPTURELEFT EYERIGHT EYESIMULATED MODEL Details in the
paper
34
CALIBRATION / USER
Interpupillary Distance (IPD)IPD
35
PUPIL TRACKINGPHYSICAL EYE MODEL
Pupil Size
Pupil Position
SCREEN CAPTURE
36
PUPIL TRACKINGLarge occlusionLess occlusionPupilpositionGaze
mapping
37
GAZE ACCURACY
ANGULAR GAZE ERROR DISTRIBUTION0.750.500.000.251.00
38
TRACKING PERFORMANCE / DESKTOPProcessing stepTimingsCamera frame
capture11 msOcclusion metric3 msPupil Detection (occluded)2 msPupil
Detection (visible)1.5 msGaze computation< 0.1ms~ 17.5 ms
(total)
may be skippedfor many users~ 12.5 ms (total)
< 2 ms
< 8 mstwo cores on [email protected] algorithmallows
gaze-contingency
39
TRACKING PERFORMANCE / MOBILEProcessing stepTimingsCamera frame
capture20 msPupil Detection (visible)13.5 msGaze computation<
0.1ms~ 33.5 ms (total)
RASPBERRY PI2quad core processorone core per eye and camera
grabberimplementation for NEON instruction set
40
BINOCULAR EYE TRACKING PROTOTYPEHMD TECHNOLOGYCALIBRATION AND
TRACKINGVR APPLICATIONS
41
FOVEATED RENDERINGUnreal Engine and footage.Copyright by Epic
Games, International.
42
SIMULATED EFFECTS / ACCOMODATIONUnreal Engine and
footage.Copyright by Epic Games, International.
43
SIMULATED EFFECTS / ACCOMODATION
Unreal Engine and footage.Copyright by Epic Games,
International.
44
AVATAR ANIMATION / TELE PRESENCE
Unreal Engine and footage.Copyright by Epic Games,
International.
45
USER STUDIES OF VIRTUAL EXPERIENCES
46
GAZE ANALYSIS IN PANORAMIC VIDEOS[Loewe and Stengel et
al.,ETVIS15]
47
MORE APPLICATIONS!Interaction MetaphorSelection /
manipulationHands-free navigationAdaptive Tone MappingAdaptive
VisualizationFoveated video/streamingExtended interfacesAugmented
VRMobile applications
EVE: Valkyrie
48
Thank you.
Michael Stengelstengel @
cs.tu-bs.degraphics.tu-bs.de/people/stengel
Steve Grogorickgrogorick @
cs.tu-bs.degraphics.tu-bs.de/people/grogorick
Elmar Eisemann (TU Delft)Martin Eisemann (TH Kln)Marcus Magnor
(TU Braunschweig)
49
REFERENCESApparent Display Resolution Enhancement for Moving
Images [Didyk11]Apparent Resolution Enhancement for Animations
[Templin12]Optimizing Apparent Display Resolution Enhancement for
Arbitrary Videos [Stengel13]A computational model of afterimages
[Ritschel et al. 12]Simulating the Visual Experience of Very Bright
and Very Dark Scenes [Jacobs et al. 15]Spatio-temporal tone mapping
operator based on a retina model [Benoit et al. 09]Foveated
Gaze-Contingent Displays for Peripheral LOD Management,3D
Visualization, and Stereo Imaging [Duchowski et al. 07]A
Retina-Based Perceptually Lossless Limit and a Gaussian
FoveationScheme With Loss Control [Targino da
Costa,14]Gaze-Contingent Spatio-Temporal Filtering in a
Head-Mounted Display [Dorr et al. 06]Pursuit Calibration: Making
Gaze Calibration Less Tedious and More Flexible [Pfeuffer et al.
13]Temporal Video Filtering and Exposure Control for Perceptual
Motion Blur [Stengel et al. 14]SmartBody, Cerebella Project
[Marsella et al., USC]LSD-SLAM: Large-Scale Direct Monocular SLAM
[Engel et al., 14]
50