37.3: Dynamic Magnification of Video for People with Visual Impairment Robert B. Goldstein, Henry Apfelbaum, Gang Luo and Eli Peli The Schepens Eye Research.

37.3: Dynamic Magnification 37.3: Dynamic Magnification of Video for People with of Video for People with

Visual ImpairmentVisual Impairment

Robert B. Goldstein, Henry Robert B. Goldstein, Henry Apfelbaum, Gang Luo and Eli PeliApfelbaum, Gang Luo and Eli Peli

The Schepens Eye Research Institute, The Schepens Eye Research Institute, Harvard Medical School, Boston, MA, USAHarvard Medical School, Boston, MA, USA

Population of Visually ImpairedPopulation of Visually ImpairedThere are about 3 million visually

impaired people in the USAExpected to double by year 2020People with visual impairment

complain about reading or face recognition

They also watch as much television as the rest of us

SimulationSimulation

Solutions for Video VisibilitySolutions for Video Visibility

EnhancementEnhancement Multiple types (next talk)Multiple types (next talk)

Substitution (voice-over description on 3rd Substitution (voice-over description on 3rd channel)channel)

MagnificationMagnification Moving CloserMoving Closer Bigger ScreenBigger Screen Optical magnification (telescopes)Optical magnification (telescopes) DynamicDynamic MagnificationMagnification

MagnificationMagnification

Zoom commonly available in Zoom commonly available in DVD players, TVs and Video DVD players, TVs and Video ConferencingConferencing

Patient can dynamically Patient can dynamically control (as the video is control (as the video is playing)playing)

Problems with Control of Problems with Control of MagnificationMagnification

Restriction of field of view means that only Restriction of field of view means that only part of the scene is on the screenpart of the scene is on the screen

Rapid changes in scenes in most movies Rapid changes in scenes in most movies does not allow for optimal manual controldoes not allow for optimal manual control

Center of magnification therefore must be Center of magnification therefore must be at the at the center of interestcenter of interest

The selection of the The selection of the center of interestcenter of interest is is of critical importanceof critical importance

POR (Point of Regard)POR (Point of Regard)

The problem is how to determine the The problem is how to determine the center of interest and make it the center center of interest and make it the center of interest (center of magnification)of interest (center of magnification)

It is done now for Movie-to-TV editingIt is done now for Movie-to-TV editing We measure the eye movements of We measure the eye movements of

normally sighted people to determine normally sighted people to determine where they are looking (Point of Regard)where they are looking (Point of Regard)

Record Eye Movements To Get Point of RegardRecord Eye Movements To Get Point of Regard

Subject viewing video in a comfortable seat without a bite bar

Remote ISCAN

Infrared pupil tracking device

DVD

RS232

Remote ISCAN RecordingRecording

Data File contains frame number and x,y coordinates

Subject at 74 inches

16×9 format on a 4x3 NTSC

27” TV

PlaybackPlayback

X,YDVD

Data File

Zoom and

Roam

27” TVZoom

Video Clip Selection TableVideo Clip Selection Table

CategorCategoryy

TitleTitle TimeTime

Talk ShowTalk Show Quiz Show (1994)Quiz Show (1994) 6:406:40

RomanceRomance Shakespeare in Love Shakespeare in Love (1998)(1998)

7:067:06

SportsSports Any Given Sunday Any Given Sunday (1999)(1999)

4:124:12

DocumentaDocumentaryry

Blue Planet (2001)Blue Planet (2001) 8:148:14

NewsNews Network (1976)Network (1976) 4:024:02

ComedyComedy Big (1988)Big (1988) 6:296:29

Total (min:sec)Total (min:sec) 37:2937:29

We Need to Record Eye We Need to Record Eye Movements from Several PeopleMovements from Several People

People do not blink at same timePeople do not blink at same time Loss of tracking occurs at different timesLoss of tracking occurs at different times Eyes “jump” (saccades) at different timesEyes “jump” (saccades) at different times People may look at different objectsPeople may look at different objects Should we use a single observer watching Should we use a single observer watching

multiple times? multiple times? different viewing strategy different viewing strategy

Merging of these multiple eye coordinate filesMerging of these multiple eye coordinate files

Types of Eye MovementsTypes of Eye MovementsDefinitionDefinition ParameterParameter

SaccadeSaccadeHigh velocity jump High velocity jump from one position to from one position to anotheranother

> 30> 30oo/second/second

PursuitPursuitSmooth movement Smooth movement tracking a moving tracking a moving targettarget

< 30< 30oo/second/second

FixationFixationEye position remains Eye position remains constant and constant and centered on a targetcentered on a target

Segment of Segment of

small motions small motions of of (max (max x or x or y y <50 or r<0.5) <50 or r<0.5) terminated by terminated by Saccade, Pursuit Saccade, Pursuit or Artifact (blink)or Artifact (blink)

3 Types of Calibrations3 Types of Calibrations

Internal 5-Point ISCAN Calibration for POR Internal 5-Point ISCAN Calibration for POR calculationscalculations

External Calibration to equate POR Values to External Calibration to equate POR Values to screen positionsscreen positions

POR recordings of purposeful pursuits, POR recordings of purposeful pursuits, fixations and saccadesfixations and saccades

Purposeful Pursuit, Fixation and Saccades

-300

-100

100

300

500

700

900 1100 1300 1500 1700 1900

FRAME

X p

osi

tio

n

Saccade And Artifact RemovalSaccade And Artifact Removal

Artifacts caused by Artifacts caused by BlinksBlinks Loss of trackingLoss of tracking Incorrectly handled timing Incorrectly handled timing

interactions between ISCAN and interactions between ISCAN and DVDDVD

Filter to remove theseFilter to remove these

File of Fixation/Pursuit SegmentsFile of Fixation/Pursuit Segments

AfterAfter the initial filtering the initial filtering step, we are left with a file step, we are left with a file that defines segments of that defines segments of fixationsfixations

Time

Fixation Fixation Fixationgap gap

Detection of Detection of TimeTime Overlapped Fixation Segments Overlapped Fixation Segments

Five time overlapped segments from three observers

Time Overlapped Fixation SegmentTime Overlapped Fixation Segment

Subject A

Subject B

Subject C

Other SubjectsOther Subjects

1

2 3

4 5

Arbitrary reference segment

Position Overlap DetectionPosition Overlap Detection

Outlier segments determined by having mean 2 SD Outlier segments determined by having mean 2 SD away from overall meanaway from overall mean

Box represents ¼ screen dimension around the mean Box represents ¼ screen dimension around the mean PORPOR

256

0 256Horizontal ISCAN “pixels”

4 Fixations Overlap

Outlier

Ve

rtic

al I

SC

AN

“p

ixe

ls” 512

512

Smoothing FilterSmoothing Filter Successive POR values that differ by a small Successive POR values that differ by a small

amount from each other cause small shifts in amount from each other cause small shifts in center of magnification that are unnecessary center of magnification that are unnecessary and disturbing to the viewerand disturbing to the viewer

Smoothing Filter was implemented that Smoothing Filter was implemented that eliminates small shifts in POR. eliminates small shifts in POR.

““Jump Threshold” set at 1/8Jump Threshold” set at 1/8 thth total screen total screen dimensiondimension

Successive PORs that move less than this Successive PORs that move less than this jump threshold grouped together and jump threshold grouped together and averaged.averaged.

2x Magnified

Rejection StatisticsRejection Statistics

Gender and Gender and AgeAge

NN RejectedRejected # # Fixations Fixations

usedused

# # Pursuits Pursuits detecteddetected

Male<40Male<40 77 52%52% 1603216032 14041404

Female<40Female<40 55 68%68% 96449644 752752

Female>45Female>45 44 69%69% 67776777 471471

Male>45Male>45 33 70%70% 49144914 465465

Why Not Always Use the Center?Why Not Always Use the Center?

-8

-4

0

4

8

-12 -8 -4 0 4 8 12

Degrees from Center of Screen

Degr

ees

"Big"

-8

-4

0

4

8

-12 -8 -4 0 4 8 12

Degrees from Center of Screen

Deg

rees

"Network"

Do People Look in The Same Place ?Do People Look in The Same Place ?

0

200

400

600

800

0 2 4 6 8 10 12 14

Position Overlaps Per Segment

Co

un

t

7 Males < 40 years

Other subject groups had similar results

A single fixation segment cannot be counted multiple times

Outlier

““Picture Over Picture” Can Address Picture Over Picture” Can Address Loss of Context Loss of Context

Edge-detected (cartoon) image Edge-detected (cartoon) image (original size) superimposed on (original size) superimposed on magnified image (POP)magnified image (POP) Edge-detection of original size image in Edge-detection of original size image in

real timereal time User controls level of magnification User controls level of magnification

and on/off of edge-detected imageand on/off of edge-detected image

PlaybackPlayback

X,YDVD

Data File

Zoom and

Roam

27” TVZoom

EdgeFilter

VideoMixer

Picture Over PicturePicture Over Picture

Viewer can see that there are two people in the Viewer can see that there are two people in the scenescene

Viewer can turn edges on and offViewer can turn edges on and off

POP VideoPOP Video

Issues To AddressIssues To Address

Improved data analysis proceduresImproved data analysis procedures Cross-group analysis to investigate Cross-group analysis to investigate

gender/age differencesgender/age differences Subject satisfaction experimentSubject satisfaction experiment

User control over magnificationUser control over magnification User control over edgesUser control over edges

AcknowledgmentsAcknowledgments

Shabtai LernerShabtai Lerner

Avni VoraAvni Vora

James BarabasJames Barabas

Dan StringerDan Stringer

Rob GiorgiRob Giorgi

Russell WoodsRussell Woods

Alex BowersAlex Bowers

Jeonhoon KimJeonhoon Kim

Doris ApfelbaumDoris Apfelbaum

Morey WaltuckMorey Waltuck

•Supported by NIH Grants EY05957 and EY12890•Over 30 people who watched videos•Lab Staff