Binocular Eye Trackingand Calibration in Head-mounted Displays
Post on 17-Jan-2017
1364 Views
Preview:
Transcript
An Affordable Solution for Binocular Eye Tracking and Calibration in Head-mounted Displays
An Affordable Solution for Binocular Eye Trackingand Calibration in Head-mounted DisplaysMichael Stengel (TU Braunschweig), Steve Grogorick (TU Braunschweig),Elmar Eisemann (TU Delft), Martin Eisemann (TH Kln), Marcus Magnor (TU Braunschweig)ACM Multimedia 2015, Brisbane
Michael Stengelstengel @ cs.tu-bs.de
1
Immersive media is getting popular.Photographic footage by shutterstock.com
2
Unreal Engine and footage. Copyright by Epic Games, International.Support for Real-time gaze-tracking is missing !
3
binocular eye-trackinginexpensive components< 400 US$low latency ( 12-17 ms)modular designefficient user calibrationfull field of view (100 per eye)uniform tracking accuracyfully adjustable lensesno external camerasOur solution.
4
BINOCULAR EYE TRACKING PROTOTYPEHMD TECHNOLOGY - TODAY AND TOMORROWCALIBRATION AND TRACKINGVR APPLICATIONS
5
HMD Technology Today.Wide Field of View
LEFT EYERIGHT EYEFOVEAL REGIONHMD SCREEN> 100 FoV per eye to create immersionbiconvex aspheric lensesfight chromatic abberation in renderinglarge viewing angles require higher FoV
6
HMD Technology Today.Wide Field of ViewHigh temporal resolution
refresh rate at least above retinal integration ratestrobe displays largely reduce display motion bluroptimal display : 1000 HzTemporal Video Filtering for Perceptual Motion Blur[Stengel et al. 14]
7
HMD Technology Today.Wide Field of ViewHigh temporal resolutionHigh spatial resolution
Apparent Display Resolution Enhancement[Didyk11, Templin12, Stengel13]2.5k per eye is state-of-the-artnot sufficient for foveal region (0.6 0.9 arc minutes) requires pixel resolution of 16k x 16kLena Sderberg. Copyright by USC.
8
HMD Technology Today.Wide Field of ViewHigh temporal resolutionHigh spatial resolutionInstant visual feedback
Kite. Copyright by Epic Games, Inc.Foveated Gaze-Contingent Displays[Dorr et al. 06, Duchowski et al. 07, Targino da Costa,14]tolerable latency depends on application< 20 to 40 ms to avoid motion sicknessrequires serious render optimization features
9
HMD Technology Today.Wide Field of ViewHigh temporal resolutionHigh spatial resolutionInstant visual feedbackAccurate physical viewpoint
Low latency head trackingnot sufficiently solved for mobile systemsLarge-Scale Direct Monocular SLAM [Engel et al. 14]
10
HMD Technology Today.Wide Field of ViewHigh temporal resolutionHigh spatial resolutionInstant visual feedbackAccurate physical viewpointNatural depth perception (accomodation)
Pursuit Gaze Calibration [Pfeuffer et al.13]Focus Cues Displays [Narain et al. 15, Huang et al. 15]interpupillary distance estimation (IPD)stereo rendering using IPDnatural accomodation currently not supported (no natural depth of field)
11
HMD Technology Today.Wide Field of ViewHigh temporal resolutionHigh spatial resolutionInstant visual feedbackAccurate physical viewpointNatural depth perception (accomodation)High dynamic range imaging
A computational model of afterimages[Ritschel et al. 2012]Simulating the Visual Experience, Tone-Mapping[Benoit et al. 09 , Ritschel et al. 12, Jacobs et al. 15]Currently global tone mapping
12
HMD Technology Today.Wide Field of ViewHigh temporal resolutionHigh spatial resolutionInstant visual feedbackAccurate physical viewpointNatural depth perception (accomodation)High dynamic range imagingPerceptual realismSmart actors; collaborative VR with gaze support[Marsella et al. 13, Debevec et al. 12, Duchowski et al. 05, Steptoe et al. 08]
Digital Ira Project [Activision, Inc. and USC, 2012]
13
HMD Technology Today.Wide Field of ViewHigh temporal resolutionHigh spatial resolutionInstant visual feedbackAccurate physical viewpointNatural depth perception (accomodation)High dynamic range imagingPerceptual realism
Benefits fromgaze-tracking !
Gaze-tracking is the technologyyou are looking for.
14
ERGONEERSFOVESMI DK1 EYE TRACKING UPGRADEARRINGTON RESEARCH EYE TRACKING KIT>20,000 USD50,000 USD15,000 USDNo details. Not yet available.
15
Construction blueprints will be freely available.
16
BINOCULAR EYE TRACKING PROTOTYPEHMD TECHNOLOGYCALIBRATION AND TRACKINGVR APPLICATIONS
17
18
3D-printed body caseFixation using head strapsOrientation tracker5.6 HD DisplayAspheric lenses
19
3D-printed body caseFixation using head strapsOrientation tracker5.6 HD DisplayAspheric lensesEye-tracking cameras
20
3D-printed body caseFixation using head strapsOrientation tracker5.6 HD DisplayAspheric lensesDichroic mirrorsEye-tracking cameras
21
3D-printed body caseFixation using head strapsOrientation tracker5.6 HD DisplayFlexible Lens-holderswith illumination unitsAspheric lensesDichroic mirrorsEye-tracking cameras
22
3D-printed body caseFixation using head strapsOrientation tracker5.6 HD DisplayFlexible Lens-holderswith illumination unitsAspheric lensesPosition-tracking cameraDichroic mirrorsEye-tracking cameras
23
BICONVEX LENSESSCREENCASEUSER
24
INFRARED ILLUMINATIONEYE TRACKINGCAMERAEYE TRACKINGCAMERADICHROICMIRRORDICHROICMIRROR
25
Infrared LightVisible Light
Dichroic mirror behavior
26
BINOCULAR EYE TRACKING PROTOTYPEHMD TECHNOLOGYCALIBRATION AND TRACKINGVR APPLICATIONS
27
PRE-CALIBRATION / HMD
Reprojection error aftercamera calibrationUsed lenses from DK1IOR estimation andphysical renderingLens shape reconstruction
28
Unkown parametersLens settingsEye locations
29
User specificUnkown parametersLens settingsEye locations
30
CALIBRATION / USER
Average eye assumptionUnknown propertyRelative location
PHYSICAL EYE MODELA photon accurate model of the human eye [Deering05]The Physiology of the Eye [Adler et al., 08]
31
CALIBRATION / LENS LOCATION
SIMULATED MODELCAMERA CAPTURELEFT EYERIGHT EYE
32
CALIBRATION / LENS LOCATION
CAMERA CAPTURELEFT EYERIGHT EYESIMULATED MODEL
33
CALIBRATION / EYE BALL LOCATION
CAMERA CAPTURELEFT EYERIGHT EYESIMULATED MODEL Details in the paper
34
CALIBRATION / USER
Interpupillary Distance (IPD)IPD
35
PUPIL TRACKINGPHYSICAL EYE MODEL
Pupil Size
Pupil Position
SCREEN CAPTURE
36
PUPIL TRACKINGLarge occlusionLess occlusionPupilpositionGaze mapping
37
GAZE ACCURACY
ANGULAR GAZE ERROR DISTRIBUTION0.750.500.000.251.00
38
TRACKING PERFORMANCE / DESKTOPProcessing stepTimingsCamera frame capture11 msOcclusion metric3 msPupil Detection (occluded)2 msPupil Detection (visible)1.5 msGaze computation< 0.1ms~ 17.5 ms (total)
may be skippedfor many users~ 12.5 ms (total)
< 2 ms
< 8 mstwo cores on i7@3.5Ghzparallelized algorithmallows gaze-contingency
39
TRACKING PERFORMANCE / MOBILEProcessing stepTimingsCamera frame capture20 msPupil Detection (visible)13.5 msGaze computation< 0.1ms~ 33.5 ms (total)
RASPBERRY PI2quad core processorone core per eye and camera grabberimplementation for NEON instruction set
40
BINOCULAR EYE TRACKING PROTOTYPEHMD TECHNOLOGYCALIBRATION AND TRACKINGVR APPLICATIONS
41
FOVEATED RENDERINGUnreal Engine and footage.Copyright by Epic Games, International.
42
SIMULATED EFFECTS / ACCOMODATIONUnreal Engine and footage.Copyright by Epic Games, International.
43
SIMULATED EFFECTS / ACCOMODATION
Unreal Engine and footage.Copyright by Epic Games, International.
44
AVATAR ANIMATION / TELE PRESENCE
Unreal Engine and footage.Copyright by Epic Games, International.
45
USER STUDIES OF VIRTUAL EXPERIENCES
46
GAZE ANALYSIS IN PANORAMIC VIDEOS[Loewe and Stengel et al.,ETVIS15]
47
MORE APPLICATIONS!Interaction MetaphorSelection / manipulationHands-free navigationAdaptive Tone MappingAdaptive VisualizationFoveated video/streamingExtended interfacesAugmented VRMobile applications
EVE: Valkyrie
48
Thank you.
Michael Stengelstengel @ cs.tu-bs.degraphics.tu-bs.de/people/stengel
Steve Grogorickgrogorick @ cs.tu-bs.degraphics.tu-bs.de/people/grogorick
Elmar Eisemann (TU Delft)Martin Eisemann (TH Kln)Marcus Magnor (TU Braunschweig)
49
REFERENCESApparent Display Resolution Enhancement for Moving Images [Didyk11]Apparent Resolution Enhancement for Animations [Templin12]Optimizing Apparent Display Resolution Enhancement for Arbitrary Videos [Stengel13]A computational model of afterimages [Ritschel et al. 12]Simulating the Visual Experience of Very Bright and Very Dark Scenes [Jacobs et al. 15]Spatio-temporal tone mapping operator based on a retina model [Benoit et al. 09]Foveated Gaze-Contingent Displays for Peripheral LOD Management,3D Visualization, and Stereo Imaging [Duchowski et al. 07]A Retina-Based Perceptually Lossless Limit and a Gaussian FoveationScheme With Loss Control [Targino da Costa,14]Gaze-Contingent Spatio-Temporal Filtering in a Head-Mounted Display [Dorr et al. 06]Pursuit Calibration: Making Gaze Calibration Less Tedious and More Flexible [Pfeuffer et al. 13]Temporal Video Filtering and Exposure Control for Perceptual Motion Blur [Stengel et al. 14]SmartBody, Cerebella Project [Marsella et al., USC]LSD-SLAM: Large-Scale Direct Monocular SLAM [Engel et al., 14]
50
top related