"Implementing Eye Tracking for Medical, Automotive and Headset Applications," a Presentation From Xilinx and EyeTech Digital Systems

Copyright © 2015 eyetech DIGITAL SYSTEMS 1

Robert Chappell: eyetech Digital Systems & Dan Isaacs: Xilinx

12 May 2015

Implementing Eye Tracking for Medical,

Automotive and Headset Applications (Example Shown for Virtual Reality Headset)


• Measures eye rotation & gaze point

• Near IR illumination

• Corneal reflections & pupil position

• History

• Dodge and Cline (1901)

• Photographic film & beams of light

• Today—Electronic imagers & processors

Eye Tracking Basics


Eye Tracking Applications


• “Stand-alone” capability—no PC required

• Compact size

• Low power (< 5 W)

• Low cost (< $200)

• Improved processing

• Multi–OS support

• Field upgrades

• Reasonable Development time and cost

Design Goals for New Eye Tracking Camera


1. Complete PCB design and layout

2. Port legacy software to new hardware

3. Develop new algorithm (AEye) in Matlab

4. Implement AEye algorithm in programmable logic (PL)

5. Gradually replace legacy blocks with PL blocks

Team varied from 2 to 4 engineers. Work done while

multitasking with other projects. ~2 years

Step by Step Design Process


Hardware Design Base on Xilinx Zynq


AEye

• Parallel processing

• PL implementation

• More extensive RT

processing—e.g. Full frame

Canny edge det.

• Higher success rate

• Faster acquisition

• Only gaze point sent to PC

• Great embedded solution

Legacy

• Sequential processing

• Video sent to PC

• Processing done on host

• Difficult to support new

hosts—e.g. Android

New Algorithm Possibilities


Processing Flow

Image

Generation Contrast

Enhancement

Canny Edge

Detection

Feature

Detection

Gaze

Calculations


Hardware / Software Partitioning

Complexity

# of Operations per Frame

Algo Control

Gaze Calculations

Sensor

Interface

Feature

Finding

Smoothing

Filter

Edge Det.

Implemented

in PL

Implemented

in SW


• Generic “Cardboard” VR System

• (NDAs prevent showing actual systems)

Application Example—VR Headset (1)


Eye

HMD Objective Lens Hot Mirror

Display

• Option 1: Hot mirror between eye and objective lens

• Ideal angle for eye imaging

• No objective lens distortion

• Must be space between eye and objective lens



Eye

HMD Objective Lens

Display

Micro Camera Head

• Option 2: Micro camera with direct view

• Severe angle

• No objective lens distortion

• Multiple camera heads may be needed

• Must be space between eye and objective lens



• Option 3: Hot mirror behind objective lens

• Ideal angle

• Eye can be close to objective lens

• Enough space needed between lens and screen

• Processing must tolerate distortion


Eye

HMD Objective Lens Hot Mirror

Display

AEye Tracking Camera


• “Stand-alone” capability — yes

• Low power (< 5 W) — 3.7 W

• Low cost (< $200 ) — yes

• Highly robust processing e.g.. Instant acquisition, high

tracking success rate. — yes

• Multi – OS support — yes — Windows, Android

• Field upgrades — yes

• Development time and cost — 2 years

Goals Met


Status

In Progress In Production


• Having a flexible HW / SW boundary is a great benefit

• Rapid development of minimum viable product (MVP)

• Gradual shifting from legacy to new algorithms

• Ease of C++ for complex calculations & control

• Speed of PL where needed

Conclusions


Video Showing FPGA Pupil Tracking


Appendix


• Flexible SW/HW boundary helped with integration and allowed easy feature

enhancements

• Locate the image sensor on a separate PCB to support more applications

• Processing at the edge opens new opportunities—host free operation—multi

sensor systems

• Having better / easier power management features in the hardware (Newer

MPSoC from Xilinx should address some of these concerns)

• We were early adopters of the Xilinx ZYNQ and experienced some early tool

issues

• Great vendor and distributor support helped us get through

• Using the latest tools now would have been nice up front

Lessons Learned


XEye Tracker—Main Camera Board

Expansion / Test (80 Pin)

Processor JTAG

MIO Spares

PL Spares XADC (x2 Ch) /

PWM Control

Image

Sensor

Quad SPI

NOR Flash

128 Mb

4bit

Data

Data &

Control

Clock

Generation

USB

Phy ULPI (8bit data)

Power & LED

Interface

Power Regulation & Control


• Real-time image processing computations

• Contrast Enhancement

• Gaussian & Sobel Filtering

• Eye finding & gaze angle detection algorithms

• Results output to main memory

• Processor interrupted to retrieve result data

Programmable Logic Functions


• Processor computes final gaze angle calculations from processed results

• Reports data for the image frame across USB

• Hardware DMA offloads data movement from ARM processor

Processor Specific Functions

Sensor

Interface

Lin

e B

uff

er

Sensor I/F

Clock zone

Image Processing

Clock Zone

Command and Control Clock zone

Image Filtering

Contrast Enhancement

Gaussian

Sobel

Frame Buffer

Interface

LPDDR2

Memory

256 MB

Cross point Switch

Results Buffer

Interface Command

& Control

Test Buffer

Interface

ARM A9

Core0

HP0 HP1 HP2 HP3 GP1

Eye Finding

Feature Extraction

Central

Interconnect

ARM A9

Core1

Memory

Controller

GP0

Test

Interface

Sensor

PL PS

On-Chip

Memory

Hard IP ZYNQ APSoC 7020

USB0

I2C0

GIC

SMC QSPI

Nor Flash Clocks/PLL

USB PHY

XADC

"Implementing Eye Tracking for Medical, Automotive and Headset Applications," a Presentation From Xilinx and EyeTech Digital Systems

Technology

objective lens ideal

new eye tracking camera

eye tracking applications

pc processing

aeye tracking camera

screen processing

robust processing

new hardware