Etra 2010 Gaze Enhanced Visual Search

Understanding the Benefits of Gaze

Enhanced Visual Search

Pernilla Qvarfordt, Jacob T. Biehl, Gene Golovchinsky and Tony Dunnigan

FX Palo Alto LaboratoryPalo Alto, California, USA

Inspecting images is common:

• Radiologist inspect medical images• Airport security inspects x-rays of

luggage• Satellite images are inspect for

threats• Quality control of products often

include visual inspection

Visual search is error prone

• We miss looking everywhere– Radiologist overall error

rate ~20%• (Goddard et al., 2001)

• Current solutions:– Systematic inspection for

all parts of the image– Documentation of review

process– Second reviewer– Pattern recognitions

models (e.g CAD)

(From Mello-Thoms et al. ETRA 2002)

Past research on improving visual

inspection• Training

– Prescribed scan paths• Kollera, Drury and Schwaninger (2009), Nickles, Melloy and

Gramapadhye (2003)

– Scan paths from expert to guide novices • Sadasvian et al. (2005)

• Improving user interfaces– Augementing display of images

• Haiman et al (2004)

– Segmentation of images• Forlines and Balakrishnan (2009)

– Re-presentation of viewed but not selected regions • Nodine and Kundel (1987)

Two phase inspection method

Phase 1 Phase 2

Gaze Data

Detect fixations

Cluster fixations

Determine clusters to exclude

Experimental design• 2 x 2 within-subject design & 8

participants

• 24 images: 6 images per condition– 1 training image per condition

• 260-300 shapes– ~25 x 25 pixels

• 5-20 targets per image (random)– 10-40 close distractors

• 67.5 sec per phase– Each segment shown 7.5 sec

• Gaze block: 270 ms threshold to block cluster

• Tobii X120 Eye tracker & 18” CRT Monitor

Gaze blockNo block

Segm

entationF

ull image

Target Close distractors

Results: Performance

• Overall no difference in True Positive identifications after both phases

• Increase in True Positive rate in 2nd phase (Block + full image)– Near sig. interaction

• Increase in FN not viewed in 1st phase transitioning to TP in 2nd phase (Block + full image)– Sig. interaction

• Significant reduced mental workload (TLX) for Gaze Block

Results: Performance

• Overall no difference in True Positive identifications after both phases

• Increase in True Positive rate in 2nd phase (Block + full image)– Near sig. interaction

• Increase in FN not viewed in 1st phase transitioning to TP in 2nd phase (Block + full image)– Sig. interaction

• Significant reduced mental workload (TLX) for Gaze Block

Results: Gaze Behavior

• Longer durations on True Positives than on False Negatives– Inline with previous research:

• (Nodine and Kundel, 1987; Manning, Ethell and Donovan, 2001)

• Adopt to fixation length– Longer fixation in phase 2

– Sig. shorter fixation on FN viewed in phase 1 with gaze block

550 ms 1032 ms

Future work

• How to use gaze patterns to guide inspectors to better performance?– Optimize use of the two phases

• How to combine information from gaze and image processing to guide inspectors to important parts of the image?

Conclusion

• Two phase inspection method– Reduces workload (with gaze block)– Have positive effect on FN not viewed

transitioning to TP during– Possible to estimate targets benefiting

fromsecond review

Now for your questions…

Pernilla Qvarfordt, Jacob T. Biehl, Gene Golovchinsky and Tony Dunnigan

FX Palo Alto LaboratoryPalo Alto, California, USA