Eye Tracking and Performance Evaluation · – Important area of HCI research where eye tracking can have a significant impact on the ... • Content-Based Image Retrieval – Similar

Eye Tracking and

Performance Evaluation

Automatic Detection of User Outcomes

Allen Harper Proposal Talk

Department of Computer Science City University of New York Graduate Center

August 22, 2014

Outline •  Part 1: Novel Framework for Eye Tracking •  Part 2: Introduce Research Questions •  Part 3: Pilot Study •  Part 4: Machine Learning Analysis •  Part 5: Proposed Work and Timeline •  Part 6: Discussion

2

Wouldn’t it be nice if computer systems could…

•  Automatically and unobtrusively determine if a user belongs to a low or high task performance group

•  Using only eye-movement data

3

Why Eye Movement Data? •  Eye movements are the overt expression of

our visual attention system – and therefore related to covert cognitive

activities •  Eye tracking metrics can then be seen as

proxy variables –  from which cognitive qualities can be inferred

4

Traditional Uses of Eye Tracking •  HCI-Eye Tracking (HET) perspective •  Descriptive Tool •  Reports percentage and distribution of

visual attention across a visual stimuli

5

Novel Use for Eye Tracking •  Eye Tracking-Performance Connection

(EPC) perspective •  Predictive Role •  Combined with machine learning techniques

in order to classify users into performance groups

6

Ideal EPC Framework

•  Given eye movement data of an unknown subject –  accurately predict their

performance level

7

?

Level of performance

Eye

Met

rics

D

Goal of Literature Survey •  What can we learn about conducting eye

tracking experiments – Which reveal strong relationships between eye

movement patterns and user performance •  “Metric hunt”

– Locate candidate eye tracking measures consistently correlated with user performance

•  Discovering which task types improved the strength of the connection between eye movement patterns and performance

8

Experimental Issues

9

Level of performance

Eye

Met

ric

A

Level of performance

Eye

Met

ric

B

Level of performance

Eye

Met

ric

C

Level of performance

Eye

Met

ric

D

Low dispersion Task too easy

Eye movement not necessary

EPC

Hints of EPC •  Eye movements Performance

•  When… –  Users viewed consistent information content –  Visual stimuli unaltered during the experiment –  Eye tracking aligned with task execution –  Performance measure aligned with task execution –  Tasks required “full” attention of users

10

Literature Survey Summary

11

Lessons Learned •  EPC experiments must have controls for:

– Content Homogeneity – Visual Homogeneity

•  And respect the alignments of task execution and performance measurement

•  Provide an appropriately difficult user task

12

Take Home •  Eye tracking community has found it difficult

to consistently and reliably connect eye movements with user performance

•  First step: apply EPC concepts to develop a baseline experiment—EPC verification

•  Second step: is to check the validity of each EPC component individually

13

Outline •  Part 1: Novel Framework for Eye Tracking •  Part 2: Introduce Research Questions •  Part 3: Pilot Study •  Part 4: Machine Learning Analysis •  Part 5: Proposed Work and Timeline •  Part 6: Discussion

14

RQ1: Establishing EPC Baseline • Control for:

– Content homogeneity – Visual homogeneity – Performance and task alignments

• Predicted Outcome: – Successful classification of subjects

into performance groups

15

RQ2: Content Homogeneity • Relax control of:

– Content homogeneity restrictions

• While still controlling for: – Visual homogeneity – Performance and task alignments

• Predicted Outcome: – Decline in model accuracy

16

RQ3: Visual Homogeneity • Relax control of:

– Visual homogeneity • While still controlling for:

– Content homogeneity – Task and performance alignments

• Predicted Outcome: – Decline in model accuracy

17

RQ4: Alignments • Relax control of:

– Alignment restrictions • While still controlling for:

– Content homogeneity – Visual Stimuli

• Predicted Outcome: – Decline in model accuracy

18

Outline •  Part 1: Novel Framework for Eye Tracking •  Part 2: Introduce Research Questions •  Part 3: Pilot Study •  Part 4: Machine Learning Analysis •  Part 5: Proposed Work and Timeline •  Part 6: Discussion

19

Pilot Study •  Goal was to establish an EPC baseline •  Operationalize our EPC concepts •  Select application domain •  Choose user interface style

– Correct stimuli for visual homogeneity •  Produce information content

– Correct for content homogeneity •  Develop performance measure

20

Application Domain •  Education Learning Systems

– Following the current trend towards increased use of online learning materials both within academia and from external services

–  Important area of HCI research where eye tracking can have a significant impact on the improvement of educational methodologies

21

Interface Style •  Special characteristics

–  Multiple regions •  Visual dispersion

–  Complex information •  Foveal Stress

–  Dynamic information •  Speed stress

22

Educational Learning System

Correct for Visual Homogeneity •  All slides have one line titles and three

bullets •  Fonts stay the same in each AOI •  No use of color, markup (e.g., italics, bold) •  No use of animations •  Stationary speaker with identical

background for all stories

23

Developing Information Content •  Obscure Wikipedia biographies

–  In order to avoid pre-exposure to material •  Limited to five minutes in duration

– Both IRB constraints and user fatigue •  Each story contains ten PowerPoint slides

– 30s timing intervals per slide and each slide contains four 7.5s subintervals

24

Correct for Content Homogeneity •  Discovered significant differences in the

distribution of information across slides •  Developed a normalization rubric using

2:1:1:1 ratios for names, dates, numbers, ideas

•  Applying a Latin Square design to the set –  {Name, Name, Number, Date, Idea}

25

Normalization Rubric

26

Performance Measure Design •  Information-recall questionnaire •  Engineered to match the same patterns as

found in information content •  Limited to 25 items

– Both IRB constraints and subject fatigue

27

Design of Performance Measure

28

Camtasia Studio Layout

29

Outline •  Part 1: Novel Framework for Eye Tracking •  Part 2: Introduce Research Questions •  Part 3: Pilot Study •  Part 4: Machine Learning Analysis •  Part 5: Proposed Work and Timeline •  Part 6: Discussion

30

Machine Learning Analysis •  Due to a lack of success found in eye

tracking literature using traditional statistical approaches we adopted a machine learning approach

31

Research Areas •  Interactive Information Retrieval (IIR)

– Eye movements indicate relevance of items to the search query term

•  Content-Based Image Retrieval – Similar to IIR, but in terms of image search

•  Object Relevance in 3-D environments – Search occurs in more naturalistic settings

32

Machine Learning Literature

33

Feature Extraction

34

Feature Groups •  Content-dependent Fixation-based •  Content-dependent Dwell-based •  Content-independent Fixation-based •  Content-independent Dwell-based •  Distance measures •  Completeness of Scan •  Eye shape characteristics

35

Data Processing Steps •  Eye tracker produces proprietary text format •  Python scripts read and convert proprietary

file structure to Excel spreadsheet •  1,197 features constructed

36

Machine Learning Platform •  Waikato Environment for Knowledge

Analysis (WEKA) •  Availability of current algorithms •  Ease of use either from GUI or command

line interface

37

Machine Learning Tools •  Machine Learning Algorithms

– Naïve Bayes – Logistic Regression – Support Vector Machine (SVM) – J48 – Random Forest

•  Attribute Selection Methods – Best First Forward – Linear Forward Selection

38

Evaluation Metrics •  Accuracy: The percentage of predictions

that are correct •  Precision: The percentage of positive

predictions that are correct •  Recall: The percentage of positive labeled

instances that were predicted as positive •  F-Measure: Harmonic mean of precision

and recall

39

Pilot Study Results

40

Outline •  Part 1: Novel Framework for Eye Tracking •  Part 2: Introduce Research Questions •  Part 3: Pilot Study •  Part 4: Machine Learning Analysis •  Part 5: Proposed Work and Timeline •  Part 6: Discussion

41

Proposed Work •  Having established EPC baseline result •  Test the limits of

– Content Homogeneity – Visual Homogeneity – Alignments of performance and task

42

RQ2: Testing Content Homogeneity

• Relax control of: – Content homogeneity restrictions

• While still controlling for: – Visual homogeneity – Performance and task alignments

• Predicted Outcome: – Decline in model accuracy

43

RQ3: Testing Visual Homogeneity • Relax control of:

– Visual homogeneity • While still controlling for:

– Content homogeneity – Task and performance alignments

• Predicted Outcome: – Decline in model accuracy

44

RQ4: Testing Alignments • Relax control of:

– Alignment restrictions • While still controlling for:

– Content homogeneity – Visual Stimuli

• Predicted Outcome: – Decline in model accuracy

45

Timeline

46

Outline •  Part 1: Novel Framework for Eye Tracking •  Part 2: Introduce Research Questions •  Part 3: Pilot Study •  Part 4: Machine Learning Analysis •  Part 5: Proposed Work and Timeline •  Part 6: Discussion

47

Expected Contributions •  Central goal of our research

– Advance the understanding of how eye movement patterns are related to user performance during task execution

48

Contribution 1 •  Preliminary Result (RQ1)

– Pilot study indicates that in a rigorously designed EPC experiment verification experiment it is possible to use eye movement metrics in order to classify users into performance groups

– This result opens new possibilities for the types of questions that can be addressed within the field of eye tracking

49

Contribution 2 •  EPC Framework

– Provides guidance for eye tracking researchers in the design and implementation of experiments attempting to connect eye movements and performance

–  In addition, RQ2-4 will provide additional data on this relationship is impacted by changes in experimental design

50

Contribution 3 •  Machine Learning Results

– Provide guidance for best practices in terms of data handling as well as comparative data on model accuracies

51

Contribution 4 •  Availability of experimental materials

– By making the visual stimuli, code, and all auxiliary research materials available to the research community this project will facilitate similar research

52

Future Applications •  Improving Usability Evaluation

– Automating evaluations – Removing the time consuming – Reducing the need for expert reviews

•  Adaptive User Interfaces –  Interfaces could employ EPC to provide a

greater user personalization based on the detected performance level

53

THANK YOU!

QUESTIONS?