Student Research Week 2006 Image-based Evaluation of Video-acquired Research Skills Unmil Karadkar, Marlo Nordt Richard Furuta Cody Lee Christopher Quick.

Student Research Week 2006

Image-based Evaluation of Video-acquired Research Skills

Unmil Karadkar, Marlo Nordt

Richard Furuta

Cody Lee

Christopher Quick

Texas A&M University

Center for the Study of Digital Libraries

The Department of Computer Science

Michael E. DeBakey Institute

Veterinary Physiology & Pharmacology

Student Research Week 2006

Outline

• Cardiovascular Research

• Research Skill Testing Interfaces

• Design of the Evaluation

• Results and Analysis

• Future Work

Student Research Week 2006

Outline

• Cardiovascular Research

• Research Skill Testing Interfaces

• Design of the Evaluation

• Results and Analysis

• Future Work

Student Research Week 2006

Cardiovascular Research

• Related to the blood circulation system

• Involves live bats– Non-destructive, in-vivo studies– Pallid Bats – transparent wings

• Observe bat wings under a microscope

• Use these observations for research– Normal conditions– Modified conditions (occlusion, pressure)

Student Research Week 2006

Research Skills

• Arteries and Veins

• Order of blood vessel– Vein– Venule– Capillary

• Lymphatic vessels

Student Research Week 2006

Working with videos

• Single layout of images

Montage– Facilitates comparison of

consecutive images

• Motion cues

But experienced researchers

often don’t need these cues

Student Research Week 2006

Still Images for Testing Skills

• Validated by a pilot subject

• Properties of image layoutsSpatial organization

Temporal presentation

• Constrain or facilitate contextual informationComparison of images

– Images of blood vessels at different locations– Images of a specific location over time

Student Research Week 2006

Outline

• Cardiovascular Research

• Research Skill Testing Interfaces

• Design of the Evaluation

• Results and Analysis

• Future Work

Student Research Week 2006

Montage Layout

• Spatial Dimensions (0)

• Context (user dependent)

• Temporal Dimension (used)

• Slowed down video

Student Research Week 2006

Scrolling Layout

• Spatial Dimensions (1)

• Context (4 images)

• Temporal Dimension (used)

• Smooth scrolling

Student Research Week 2006

Thumbnail Layout• Spatial Dimensions (2)

• Context (all images in set)

• Temporal Dimension (unused)

Student Research Week 2006

Outline

• Cardiovascular Research

• Research Skill Testing Interfaces

• Design of the Evaluation

• Results and Analysis

• Future Work

Student Research Week 2006

Experimental Design 1• 3 Layouts

– Thumbnail– Scrolling– Montage– 16 images in a set

A E L

Time/image 3s 4s 2s

Answers per image per image per set

Sets 2 2 3

• 3 Tasks– Artery/Vein recognition– Size Estimation– Lymphatic Vessel Wall

Identification

Student Research Week 2006

Experimental Design 2• Balanced across subjects

– Ordering of tasks (AEL, ELA, LAE)– Ordering of image layouts (TMS, MST, STM)

• Powerpoint for presenting layouts

• Feedback– Verbal– Powerpoint’s Pointer feature

• Validation of answers– Panel of experts– Accurate measurements

Student Research Week 2006

Subjects

• Trained researchers in the bat lab– About 30 subjects available world-wide

• 9 experts– Research experience of more than 2 semesters

• 6 novices– Research experience of 1 semester only

• Age group 18 to 35

• 4 female, 11 male

Student Research Week 2006

Data Collection

• Answers for the tasks

• Demographic Questionnaire– Self evaluation of expertise

• Task Questionnaire– For each task

• Most preferred layout• Perceived best performance

• Short interview

Student Research Week 2006

Outline

• Cardiovascular Research

• Research Skill Testing Interfaces

• Design of the Evaluation

• Results and Analysis

• Future Work

Student Research Week 2006

• N=16, x=9.75 σ2=9.73• No learning• Constrained by knowledge• Preferred layout T

• N=16, x=13.59 σ2=2.13• Learning (p=0.0004)• Constrained by layout• Preferred layout T

Artery/Vein RecognitionExperts

0%

20%

40%

60%

80%

100%

T M S

Layout

Per

form

ance

Round 1

Round 2

Novices

0%

20%

40%

60%

80%

100%

T M S

Layout

Pe

rfo

rma

nc

e

Round 1

Round 2

(p=0.005)89.6%

91.4

%

Validated conventional wisdom

Student Research Week 2006

Size EstimationExperts

0%

20%

40%

60%

80%

M1 M2 S1 S2 T1 T2

Round

Pe

rfo

rma

nc

e +/- 40%

+/- 30%

+/- 20%

+/- 10%

Novices

0%

10%

20%

30%

40%

M1 M2 S1 S2 T1 T2

Round

Pe

rfo

rma

nce +/- 40%

+/- 30%

+/- 20%

+/- 10%

• N=16, x=9.03 σ2=14.11• Preferred layout T

• N=16, x=4.06 σ2=12.51 • Preferred layout T

General understanding of the order of blood vessels does not translate into accurate estimation of size

(p<0.002)

(p<0.002)

Student Research Week 2006

Lymphatic Vessel Wall Identification

Experts

0%

20%

40%

60%

80%

100%

M S T

Layout

Per

form

ance

Novices

0%

20%

40%

60%

M S T

Layout

Per

form

ance

• Worked best with layout M (p=0.035)

• Preferred layout M

(p=0.03)87%

55%

Experts and Novices use different strategies for applying contextual information

• Worked best with layout T

• Preferred layout M

Student Research Week 2006

Outline

• Cardiovascular Research

• Research Skill Testing Interfaces

• Design of the Evaluation

• Results and Analysis

• Future Work

Student Research Week 2006

Future Work

• Train subjects on new interfaces– Subjects preferred familiar interfaces but

performed well with the unfamiliar ones as well

• Investigate mechanisms for conveying size of objects within images– Subjects could not accurately estimate the size

of blood vessels in spite of possessing the required knowledge

• Explore other image layouts

Student Research Week 2006

http://ebat.tamu.edu

Center for the Study of Digital Libraries

Micahel E. DeBakey Institute

Further information

Unmil Karadkar

Marlo Nordt

Richard Furuta

[email protected]

[email protected]

[email protected]

Christopher Quick [email protected]