Neural basis of human visual motion perception Alex Huk Neurobiology & Center for Perceptual Systems UT-Austin
Neural basis of human visualmotion perception
Alex HukNeurobiology &Center for Perceptual SystemsUT-Austin
Neural basis of motion perception
CognitiveHow is this sensory evidence accumulated, remembered,and combined with other information to guide behavior?
SensoryHow does the brain represent sensory stimuli?
2) Direction selectivity• Neurons sensitive to direction in human visual cortex
1) Motion aftereffect • Neural basis of a direction-selective visual illusion
3) Direction of moving objects• Neurons that represent direction of object motion
Goals
fMRI
Resolution • spatially coarse, temporally sluggishbut• proportional to average spike rate (Rees et al, 2000; Heeger, Huk, Geisler, Albrecht, 2000; Logothetis et al, 2001)
BOLD signal• neural activity oxygen demand proportion of deoxygenated hemoglobin MRI signal
Average spiking activity • space: across each visual cortical area• time: across several seconds
Human visual cortex
V1
V2
V2
V3
V3
V3a
V4v
Motion responses in human visual cortex
moving stationary
vs
MT+
1Neural basis of the motion aftereffect
Motion aftereffect
Addams, 1834
*
Motion aftereffect
Motion aftereffect
Motion aftereffect
Motion aftereffect
Motion aftereffect
Motion aftereffect
MAE MAE control control MAEcontrol
fMRI experiment
MT+ responses on MAE trials
MAE MAE control control MAEcontrol
fMRI experiment
Average MAE time series
MAE MAE control control MAEcontrol
Adapt Blank TestMT+ responses on MAE trials
fMRI experiment
Motion aftereffect
Difference in MT+ activity during MAE vs control?
Motion aftereffect
MAE > control
Tootell et al, Nature (1995)He et al, Curr Bio (1998) Culham et al, J Neurophys (1999)Taylor et al, Neuroimage (2000)Hautzel et al, Brain Res (2001)Berman & Colby, Cog Br Res (2001)
Huk, Ress, & Heeger, Neuron (2001)
L
R
Direction-selective neurons
Monkey MT, MST:> 90% of neurons aredirection-selective
stationary
neuralresponse
perceiveddirection
stimulus
L R
NONE
Motion aftereffect: Theory
rightwardstationary
L R L R
NONE RIGHT
neuralresponse
perceiveddirection
stimulus
Motion aftereffect: Theory
stationary
L R L R
NONE RIGHT
adaptationneuralresponse
perceiveddirection
stimulus rightward
Motion aftereffect: Theory
stationary
L R L R
NONE RIGHT
adaptationneuralresponse
perceiveddirection
stimulus rightward
Motion aftereffect: Theory
stationary
L R L R
NONE RIGHT
adaptationneuralresponse
perceiveddirection
stimulus rightward
Motion aftereffect: Theory
stationarystationary
L R L R
NONE RIGHT LEFT
L Rneuralresponse
perceiveddirection
stimulus rightward
Motion aftereffect: Theory
Motion aftereffect: Theory
Direction-selective reduction in response
stationary
LEFT
L R
Cat Primary Visual CortexGiaschi et al 1993Hammond et al 1985,1986,1988Marlin et al 1988Saul & Cynader 1989Vautin & Berkeley 1877von der Heydt et al 1973
Monkey MTPetersen et al 1985van Wezel & Britten, 2001,2002
MAEillusory motion
controlno motion
attention increases MT+ responsese.g., Beauchamp et al 1997 Treue & Martinez Trujillo 1999 Huk & Heeger 2000
fMRInet increase ? theory + single neurons
direction-selective decrease
Motion aftereffect: Puzzle
+ attention
Equating attention during test period
equate attention
task
5 sec test
motion task:which patch movedoutward faster?
• performed on both MAE and control trials
• equal, threshold difficulty
Equating attention during test period
Motion aftereffect
Difference in MT+ activity during MAE vs controlwhen attention is equal?
Motion aftereffect with controlled attention
Huk, Ress, & Heeger, Neuron (2001)
MAE = control
Summary: Motion aftereffect
MT+ responses are affected by attention to MAE
Motion aftereffect does not necessarily depend on anet increase in MT+ response
Direction-selectivity?
2Direction-selectivity inhuman visual cortex
Direction-selective adaptation
Direction-selective response decrease• Compare Repeated vs Mixed
(e.g.,Grill-Spector et al, 1999;Kourtzi & Kanwisher, 2001)
X 12
X 12
Mixed direction
Direction-selective adaptation
Adapted direction
Direction-selective adaptation
X 12
Mixed direction
Adapted direction
Direction-selective adaptation
X 12
X 12
Mixed direction
Adapted direction
Direction-selective adaptation
X 12
X 12
Mixed direction
Adapted direction
Direction-selective adaptation
X 12
X 12
Mixed direction
Adapted direction
Control attentionspeed task
mixeddirection
adapteddirection
mixeddirection
adapteddirection 6
adaptationresponseamplitude
time (sec)
0 18 36 54 72
fMRI experiment
time (sec)
mixeddirection
adapteddirection
mixeddirection
adapteddirection 6
adaptationresponse
mea
nfMRI response(% BOLD change)
0 18 36
Average fMRI response
Direction-selectiveadaptation?
Direction-selective adaptation
Huk, Ress, & Heeger, Neuron (2001)
fMRI response(% BOLD change)
adaptation baseline response
Direction-selectivity in human visual cortex
Huk, Ress, & Heeger, Neuron (2001)
direction-selectivity index(adaptation response /
baseline response)
Summary: Direction-selectivity
Direction-selective neurons,increasing across visual cortex,
strongest in MT+
Direction of moving objects?
3Direction of moving objects
+ =
Gratings and plaids
componentgrating 1
componentgrating 2
plaidpattern
≠
component-motion cells direction of local components
Component and pattern motion cells
grating component moving strong response
=
pattern-motion cellsdirection of objects and patterns
=
component-motion cellsdirection of local components
Component and pattern motion cells
grating component moving strong response
=
pattern moving strong response
Movshon et al, 1986Albright, 1984
pattern-motion cellsdirection of objects and patterns
?
Human cortexcomponent-motion cells pattern-motion cells
+
Selective adaptation protocol
Adapted direction
Mixed direction
Selective adaptation protocol
Adapted direction
Mixed direction
Selective adaptation protocol
Adapted direction
Mixed direction
…
Selective adaptation protocol
Adapted direction
Mixed direction
…
Selective adaptation protocol
Adapted direction
Mixed direction
…
Selective adaptation protocol
Adapted direction
Mixed direction
…
Selective adaptation protocol
Adapted direction
Mixed direction
…
Selective adaptation protocol
Adapted direction
Mixed direction
…
Selective adaptation protocol
Adapted direction
Mixed direction
…
…
Selective adaptation protocol
Adapted direction
Mixed direction
1.3 sec 0.7 sec
plaid motion
Trials
1.3 sec 0.7 sec
plaid motion
response 8
Trials
time (sec)
0 16 32 48
mixeddirection
adapteddirection
mixeddirection
adapteddirection 6
fMRI experiment
64
time (sec)
mixeddirection
adapteddirection
mixeddirection
adapteddirection 6
adaptationresponse
mea
nfMRI response(% BOLD change)
0 16 32
Average fMRI response
mea
n
mixeddirection
adapteddirection
time (s)0 16 32
fMRI response(% BOLD
signal change)
Modulationof pattern-motion
response
Plaid direction
Predictions: Pattern-motion activity
mixeddirection
adapteddirection
mea
n
time (s)0 16 32
fMRI response(% BOLD
signal change)
No modulationof component-motion
response
Component direction
Predictions: Component motion activity
0
0.2
-0.2
time (s)0 16 32
fMRI response(% BOLD
signal change)
mixeddirection
adapteddirection
Pattern motion adaptation
Modulation of patternmotion response?
0
0.2
-0.2
time (s)0 16 32
fMRI response(% BOLD
signal change)
mixeddirection
adapteddirection
Pattern-motion adaptation
Pattern-motion response
⇓
MT+
Pattern motion adaptation
Huk & Heeger, Nature Neurosci (2002)
0
0.2
-0.2
time (s)0 16 32
fMRI response(% BOLD
signal change)
mixeddirection
adapteddirection
MT+
Pattern motion adaptation
Huk & Heeger, Nature Neurosci (2002)
no adaptation when plaiddirection varies (p = .35 - .74)
pattern-motion
0
0.1
0.2
V1 V2 V3 V3A V4v MT+
visual area
Pattern motion selectivity across visual cortex
pattern-motiondirection-selectivity index
(adaptation response / baseline response)
Huk & Heeger, Nature Neurosci (2002)
+ =strong
pattern-motionpercept
Pattern motion perception
+ =
+ =
strongpattern-motion
percept
weakpattern-motion
percept
Pattern motion perception
Adelson & Movshon, 1982
Pattern motion perception
Adapted direction
Mixed direction
Pattern motion perception
Adapted direction
Mixed direction
…
…
Adapted direction
Mixed direction
Pattern motion perception
0
0.2
-0.2
time (s)0 16 32
mixeddirection
adapteddirection
fMRI response(% BOLD
signal change)
MT+
Pattern motion perception
weak pattern-motion response
weak percept
⇓
Huk & Heeger, Nature Neurosci (2002)
strong percept,strong adaptation
0
0.2
-0.2
time (s)0 16 32
mixeddirection
adapteddirection
fMRI response(% BOLD
signal change)
MT+
Pattern motion perception
weak percept,weak pattern motion
response
Huk & Heeger, Nature Neurosci (2002)
equally strong adaptationwith high + low SF plaids
(p = .93)
0
0.2
-0.2
time (s)0 16 32
mixeddirection
adapteddirection
fMRI response(% BOLD
signal change)
MT+
Pattern motion perception
Huk & Heeger, Nature Neurosci (2002)
0
0.2
-0.2
time (s)0 16 32
mixeddirection
adapteddirection
fMRI response(% BOLD
signal change)
MT+
Pattern motion perception
Huk & Heeger, Nature Neurosci (2002)
strong pattern-motion perceptweak pattern-motion percept
0
0.1
0.2
V1 V2 V3 V3A V4v MT+
visual area
patternmotionpercept
Relation between fMRI response and percept
pattern-motiondirection-selectivity index
(adaptation response / baseline response)
Huk & Heeger, Nature Neurosci (2002)
Summary: Pattern motion
Pattern-motion selectivity in human visual cortex
Early visual areas represent component motions,later areas (MT+) represent pattern motion
Strength of pattern-motion activitycorresponds to strength of pattern-motion percept
Conclusions
Neural basis of human motion perception
• Neurons: Direction-selectivity in human cortex
• Processing: Component-motion pattern-motion
• Perception: Relative response strengths correspond to motion percepts
Conclusions
Human / monkey homology• Strong direction-selectivity, pattern-motion in:
human MT+ ⇔ monkey MT/MST
• Human MT+ subdivisible into MT and MST (Huk, Dougherty, & Heeger, J Neurosci, 2002)
Passive viewing vs controlled attention
passive viewing controlled attention
Response to moving test stimulus 70% larger (p ~0) Responses were not saturated during task performance
trial type viewing condition, p = 0.001
Future directions: Decision
Decision [LIP]Sensory [MT]
Accumulation of evidence
?
Future directions: Decision
Combination of sensory evidence and other knowledge
Decision [LIP]
Future directions: Decision
Decision [LIP]
?Combination of sensory evidence and other knowledge
Bias
Future directions: Decision
?
1) start point: boost baseline?Decision [LIP]
Combination of sensory evidence and other knowledge
Bias
Future directions: Decision
?
1) start point: boost baseline?2) weighting: steepen slope?
Decision [LIP]
Combination of sensory evidence and other knowledge
Bias
Future directions: Decision
?
1) start point: boost baseline?2) weighting: steepen slope?3) threshold: lower maximum?
Decision [LIP]
Combination of sensory evidence and other knowledge
Bias
Collaborators
Monkey single-units and psychophysicsMike Shadlen, John Palmer
Retinotopy, gray matter segmentation & flatteningBrian Wandell, Alex Wade, Alyssa Brewer
MR Physics Gary Glover
fMRIDavid HeegerDavid RessBob Dougherty Geoff Boynton
x
y t
Future directions: Sensory
Speed
spacetime
selective adaptation: separate sensitivity to speed from temporal and spatial frequencies
The aperture problem
The aperture problem
The aperture problem
“Component motion” cells aperture problem
“Pattern motion” cells object motion
Future directions: DecisionAccumulation of evidence
?
Functional subdivision of humanareas MT and MST
Monkey MT / MST complex
MT translationsmaller receptive fieldsretinotopic organization
MST optic flowlarger receptive fieldsno clear retinotopy
MTMST
Strategy: MT+ subdivision
Localizer [MT+]• all motion responsive neurons
Retinotopy [MT]• small receptive fields• retinotopic organization
Ipsilateral [MST]• large receptive fields• no retinotopy
Localizing MT+
0 0.4 0.8correlation
vs
Retinotopy stimulus
Wedge of moving dots
motion-wedge rotates slowly through visual field
Retinotopy and receptive field size
small RF[MT]
large RF[MST]
Ipsilateral stimulus
10 deg
Ipsilateral motion
Ipsilateral stationary
Ipsilateral stimulus and receptive fieldsize
small RF[MT]
no response
large RF[MST]
ipsilateral response
Strategy: MT+ subdivision
Localizer [MT+]• all motion responsive neurons
Retinotopy [MT]• small receptive fields• retinotopic organization
Ipsilateral [MST]• large receptive fields• no retinotopy
Subdividing human MT+
Localizer [MT+] Retinotopy [MT] Ipsilateral [MST]
Subdividing human MT+
Localizer [MT+] Retinotopy [MT] Ipsilateral [MST]r rθ
Summary: Subdivision of MT+
• Human MT+ is subdivisible into MT, MST• MT contains a retinotopic map of motion direction• MST neurons summate motion over larger regions of space
Direction-selective adaptation
Direction-selective adaptation
Direction-selective adaptation
Speed task: control attention
Direction-selective adaptation
Direction-selective adaptation
Speed task: control attention
X 3
X 3
oppositedirection
adapteddirection
oppositedirection
adapteddirection 6
adaptationresponseamplitude
time (sec)
0 16 32 48 54
fMRI experiment
Direction-selective adaptation
Huk, Ress, & Heeger, Neuron (2001)
Direction-selectiveadaptation
Direction-selectivity⇓
Opposite-direction > Adapted-direction
MT+
Pattern-motion controls
Weak effects just due to SFs? strong
weak
Pattern-motion controls
Weak effects just due to SFs? No. Strong adaptation with
coherent high and low-SF plaids.(F1,55 = 0.07, p = .93)
strong
weak
Pattern-motion controls
Weak effects just due to SFs? No. Strong adaptation with
coherent high and low-SF plaids.(F1,55 = 0.07, p = .93)
Were responses really due todirection-selective adaptation?
Pattern-motion controls
Yes. Rotating plaid direction from trial to trial produced no response modulation.
(p = .35 - .74)
Weak effects just due to SFs? No. Strong adaptation with
coherent high and low-SF plaids.(F1,55 = 0.07, p = .93)
Were responses really due todirection-selective adaptation?
Conclusions
Adaptation
• Characterize neural response properties
• Quantify selectivity across visual areas
• Selectively isolate hierarchical stages of processing