"Implementing Eye Tracking for Medical, Automotive and Headset Applications," a Presentation From Xilinx and EyeTech Digital Systems
Post on 19-Aug-2015
6 Views
Preview:
Transcript
Copyright © 2015 eyetech DIGITAL SYSTEMS 1
Robert Chappell: eyetech Digital Systems & Dan Isaacs: Xilinx
12 May 2015
Implementing Eye Tracking for Medical,
Automotive and Headset Applications (Example Shown for Virtual Reality Headset)
Copyright © 2015 eyetech DIGITAL SYSTEMS 2
• Measures eye rotation & gaze point
• Near IR illumination
• Corneal reflections & pupil position
• History
• Dodge and Cline (1901)
• Photographic film & beams of light
• Today—Electronic imagers & processors
Eye Tracking Basics
Copyright © 2015 eyetech DIGITAL SYSTEMS 4
• “Stand-alone” capability—no PC required
• Compact size
• Low power (< 5 W)
• Low cost (< $200)
• Improved processing
• Multi–OS support
• Field upgrades
• Reasonable Development time and cost
Design Goals for New Eye Tracking Camera
Copyright © 2015 eyetech DIGITAL SYSTEMS 5
1. Complete PCB design and layout
2. Port legacy software to new hardware
3. Develop new algorithm (AEye) in Matlab
4. Implement AEye algorithm in programmable logic (PL)
5. Gradually replace legacy blocks with PL blocks
Team varied from 2 to 4 engineers. Work done while
multitasking with other projects. ~2 years
Step by Step Design Process
Copyright © 2015 eyetech DIGITAL SYSTEMS 7
AEye
• Parallel processing
• PL implementation
• More extensive RT
processing—e.g. Full frame
Canny edge det.
• Higher success rate
• Faster acquisition
• Only gaze point sent to PC
• Great embedded solution
Legacy
• Sequential processing
• Video sent to PC
• Processing done on host
• Difficult to support new
hosts—e.g. Android
New Algorithm Possibilities
Copyright © 2015 eyetech DIGITAL SYSTEMS 8
Processing Flow
Image
Generation Contrast
Enhancement
Canny Edge
Detection
Feature
Detection
Gaze
Calculations
Copyright © 2015 eyetech DIGITAL SYSTEMS 9
Hardware / Software Partitioning
Complexity
# of Operations per Frame
Algo Control
Gaze Calculations
Sensor
Interface
Feature
Finding
Smoothing
Filter
Edge Det.
Implemented
in PL
Implemented
in SW
Copyright © 2015 eyetech DIGITAL SYSTEMS 10
• Generic “Cardboard” VR System
• (NDAs prevent showing actual systems)
Application Example—VR Headset (1)
Copyright © 2015 eyetech DIGITAL SYSTEMS 11
Eye
HMD Objective Lens Hot Mirror
Display
• Option 1: Hot mirror between eye and objective lens
• Ideal angle for eye imaging
• No objective lens distortion
• Must be space between eye and objective lens
Application Example—VR Headset (2)
Copyright © 2015 eyetech DIGITAL SYSTEMS 12
Eye
HMD Objective Lens
Display
Micro Camera Head
• Option 2: Micro camera with direct view
• Severe angle
• No objective lens distortion
• Multiple camera heads may be needed
• Must be space between eye and objective lens
Application Example—VR Headset (3)
Copyright © 2015 eyetech DIGITAL SYSTEMS 13
• Option 3: Hot mirror behind objective lens
• Ideal angle
• Eye can be close to objective lens
• Enough space needed between lens and screen
• Processing must tolerate distortion
Application Example—VR Headset (4)
Eye
HMD Objective Lens Hot Mirror
Display
AEye Tracking Camera
Copyright © 2015 eyetech DIGITAL SYSTEMS 14
• “Stand-alone” capability — yes
• Low power (< 5 W) — 3.7 W
• Low cost (< $200 ) — yes
• Highly robust processing e.g.. Instant acquisition, high
tracking success rate. — yes
• Multi – OS support — yes — Windows, Android
• Field upgrades — yes
• Development time and cost — 2 years
Goals Met
Copyright © 2015 eyetech DIGITAL SYSTEMS 16
• Having a flexible HW / SW boundary is a great benefit
• Rapid development of minimum viable product (MVP)
• Gradual shifting from legacy to new algorithms
• Ease of C++ for complex calculations & control
• Speed of PL where needed
Conclusions
Copyright © 2015 eyetech DIGITAL SYSTEMS 19
• Flexible SW/HW boundary helped with integration and allowed easy feature
enhancements
• Locate the image sensor on a separate PCB to support more applications
• Processing at the edge opens new opportunities—host free operation—multi
sensor systems
• Having better / easier power management features in the hardware (Newer
MPSoC from Xilinx should address some of these concerns)
• We were early adopters of the Xilinx ZYNQ and experienced some early tool
issues
• Great vendor and distributor support helped us get through
• Using the latest tools now would have been nice up front
Lessons Learned
Copyright © 2015 eyetech DIGITAL SYSTEMS 20
XEye Tracker—Main Camera Board
Expansion / Test (80 Pin)
Processor JTAG
MIO Spares
PL Spares XADC (x2 Ch) /
PWM Control
Image
Sensor
Quad SPI
NOR Flash
128 Mb
4bit
Data
Data &
Control
Clock
Generation
USB
Phy ULPI (8bit data)
Power & LED
Interface
Power Regulation & Control
Copyright © 2015 eyetech DIGITAL SYSTEMS 21
• Real-time image processing computations
• Contrast Enhancement
• Gaussian & Sobel Filtering
• Eye finding & gaze angle detection algorithms
• Results output to main memory
• Processor interrupted to retrieve result data
Programmable Logic Functions
Copyright © 2015 eyetech DIGITAL SYSTEMS 22
• Processor computes final gaze angle calculations from processed results
• Reports data for the image frame across USB
• Hardware DMA offloads data movement from ARM processor
Processor Specific Functions
Sensor
Interface
Lin
e B
uff
er
Sensor I/F
Clock zone
Image Processing
Clock Zone
Command and Control Clock zone
Image Filtering
Contrast Enhancement
Gaussian
Sobel
Frame Buffer
Interface
LPDDR2
Memory
256 MB
Cross point Switch
Results Buffer
Interface Command
& Control
Test Buffer
Interface
ARM A9
Core0
HP0 HP1 HP2 HP3 GP1
Eye Finding
Feature Extraction
Central
Interconnect
ARM A9
Core1
Memory
Controller
GP0
Test
Interface
Sensor
PL PS
On-Chip
Memory
Hard IP ZYNQ APSoC 7020
USB0
I2C0
GIC
SMC QSPI
Nor Flash Clocks/PLL
USB PHY
XADC
top related