Single-Trail Analysis of EEG during Rapid Visual Discrimination: Enabling Cortically Coupled Computer Vision Authors: Paul Sajda, Adam D. Gerson, Marios G. Philiastides Dept. of Biomedical Engineering, Columbia University, USA Lucas Parra Dept. of Biomedical Engineering City College of New Work USA Thusday, June 16, 2009 1
38
Embed
Authors: Paul Sajda , Adam D. Gerson , Marios G. Philiastides
Single-Trail Analysis of EEG during Rapid Visual Discrimination: Enabling Cortically Coupled Computer Vision. Authors: Paul Sajda , Adam D. Gerson , Marios G. Philiastides Dept. of Biomedical Engineering, Columbia University, USA Lucas Parra Dept. of Biomedical Engineering - PowerPoint PPT Presentation
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Single-Trail Analysis of EEG during Rapid Visual Discrimination: Enabling Cortically Coupled Computer Vision
Authors:Paul Sajda, Adam D. Gerson, Marios G. Philiastides
Dept. of Biomedical Engineering, Columbia University, USALucas Parra
Dept. of Biomedical EngineeringCity College of New Work
USA
Thusday, June 16, 2009 1
Outlines
• Introduction• Linear Methods for Single-Trail Analysis• EEG Correlates of Perceptual Decision Making• Identifying Cortical Process leading to
Response Time Variability• EEG-Based Image Triage• Conclusion
Thusday, June 16, 2009 2
Introduction• Linear discrimination of multichannel electroencephalography
(EEG) has used for single-trail detection of neural signatures of visual recognition events.
• Demonstrate: relating neural variability to response variability
• Studies for response accuracy
• Response latency during visual target detection Example: Running in the park with your head phones on,
listening to your favorite tune, & concentrating on your stride, you look up & see a face that you immediately recognize as a high school friend.
Thusday, June 16, 2009 3
Introduction (Con..) She is wearing a hat, glasses & has aged fifteen years since you
last saw her. You & she are running in opposite directions so you only see
her for a fleeting moment, yet you are sure it was her. Your visual system has just effortlessly accomplished a feat that
has thus far baffled best computer vision systems. Such ability for rapid processing of visual information is even
more impressive in light of the fact that neurons are relatively slow processing elements to digital computers, where individual transistors can switch a million times faster than a neuron can spike.
Thusday, June 16, 2009 4
Introduction (Con..) Noninvasive neuroimaging has provided a means to peer into
brain during rapid visual object recognition. Analysis of trial-averaged event-related potentials in EEG has
enabled us to assess speed of visual recognition & discrimination in terms of timing of underlying neural process.
Recent work has used single-trial analysis of EEG to characterize neural activity directly correlated with behavioral variability during tasks involving rapid visual discrimination.
Components extracted from EEG can capture neural correlates of visual recognition & decision-making processes on a trail-by-trail basis.
Thusday, June 16, 2009 5
Introduction (con..)• Noninvasive BCI paradigms: Having a subject consciously modulate brain rhythms Having a subject consciously generate a motor
plan/visual imagery Directly modulating subjects cortical activity by
stimulus frequency (SSVEP) Exploiting specific ERPs (novelty/oddball P300) Focus: Single trail detection of ERPs & relationship to visual
slow drifts (<0.5 Hz), no info carry S2: Reduce dimensionality by grouping signal into L sample
blocks & does not change within time window. S3: Increase # of trails by L redundant samples/window L=50 signal of interest at 10 Hz>> signal: noise Transform original data for each trail with TD dimensions into
L trails: TD/L L=50, N=100, # training exam=5000 train classifier with 640-dimensional feature vector at 10 Hz
Thusday, June 16, 2009 9
Linear Methods for Single-Trail Analysis: LDA (Con..)
• linear classifier: TD/L dimension, good results
• Single training window: L=T, D-dimensional feature vector
• Linear Classification:
• Feature vector x is projected onto an orientation defined by vector w
• Projection, y = wTx, optimally differentiates bet’n two classes.
Thusday, June 16, 2009 10
Thusday, June 16, 2009 11
Linear Methods for Single-Trail Analysis (Con..)
• Adv: linear voltage combination-immediate
interpretation (current) Coefficients that coupled current with
observed voltages given for linear model by a=<xTy>/<y2>
Angular bracket indicate average over trials & samples.
Coefficients a describe coupling (& correlation) of discrimination component y with sensor activity x.
a & x: D-dim vector(row/column) Strong coupling: low attenuation &
visualized intensity map-sensor projections
Easy to implement Fast Permit real-time operation
• Disadv: does not capture synchronized
activity >10 Hz Neither does it capture activity
that is not a fixed distance in time from stimulus
Only phase-locked activity detection
Thusday, June 16, 2009 12
EEG Correlates of Perceptual Decision Making
• Single-trail LDA to identify cortical correlates of decision-making during rapid discrimination of images.
• Experiment: Psychophysical performance is measured for several subjects during RSVP task
• A series of target (faces) and non-target (cars) trials presented in rapid succession (Fig. 25.2a)
• Simultaneously recording neuronal activity from 64-channel EEG electrode array.
• Stimulus evidence varied with phase coherence (Fig. 25.2b)
Thusday, June 16, 2009 13
EEG Correlates of Perceptual Decision Making (Con..)
• Within trail block: face & car images over a range of phase coherence presented random order.
• 60/120 Hz notch filter (minimize line noise) Record: EOG signals, remove motion & blink Motor response & stimulus events recorded on separate
channels
Thusday, June 16, 2009 15
EEG Correlates of Perceptual Decision Making (Con..)
• Identify EEG components that maximally discriminate bet’n 2 conditions
• Identify 2 time window for discrimination
• Identify early (170 ms following stimulus) & late component (>300 ms)
• LD trained by integrating data across both time windows (2D-feature vector)
• Performance improved (higher Az)
• Fig. 25.3: comparison of psychometric & neurometric functions for 1 sub in dataset.
• All subjects a single function can fit behavioral & neuronal datasets & 2 sep functions.
Thusday, June 16, 2009 16
EEG Correlates of Perceptual Decision Making (Con..)
• Data analyzed both stimulus & response-locked conditions
• Both face selective components appear to be more correlated with onset of visual stimulation than response for one subject (Fig. 25.4)
• Discriminate activity significant down to 30% phase coherence (early & late component)
• Randomize Az: significant level of p<0.01.
• Results: neural correlates of perceptual decision making identified using high-spatial density EEG & corresponding component activities
Thusday, June 16, 2009 17
EEG Correlates of Perceptual Decision Making (Con..)
Thusday, June 16, 2009 18
Identifying Cortical Process leading to Response Time Variability
• Significant variability in response time observed across trials in many visual discrimination & recognition.
• Factors: difficulty in discriminating an object on any given trial
• Trial-by-trial variability of subject’s engagement
• Intrinsic variability of neural processing
• Identifying neural activity correlated with response time variability may shed light on the underlying cortical networks responsible
for perceptual decision-making processes & processing latencies that these networks may introduce for a given task
Thusday, June 16, 2009 19
Identifying Cortical Process leading to Response Time Variability (Con..)
Visual target detection: use RSVP & single trial spatial integration of high-density EEG
to identify time course & cortical origins leading to response time variability.
• RSVP paradigm (Fig 25.5)
• Varied scale, pose & position of target objects (people) requires subjects to recognize rather than low-level features
• Continuous sequence of natural scenes
• 4 blocks (50 sequences) rest period: no more than 5 mins.
Thusday, June 16, 2009 20
Identifying Cortical Process leading to Response Time Variability (Con..)
• Each sequences: 50 images, 50% chance containing one target image with one or more people in a natural scene.
• Target appear: middle 30 images/50 sequences
• Detractor image: remaining natural scene without a person
• Each image: 100 ms
• Fixation cross: 2s bet’n sequences.
• Instructed to press left button of a 3-button mouse with right index finger while fixation cross present, release as soon as recognize target image.
• Used LD: to determine spatial weighting coefficients that optimally discriminate bet’n EEG resulting from different RSVP task conditions (target vs. distractor) over specific temporal window bet’n stimulus & response.
Thusday, June 16, 2009 21
Identifying Cortical Process leading to Response Time Variability (Con..)
• Integration across sensors enhances signal quality without loss of temporal precision common to trail averaging in ERP.
Resulting discrimination components describe activity specific to target recognition & subsequent response for individual trails.
Intertrial variability estimated by extracting feature from discrimination components.
• Peaks of spatially integrated discriminating components found by fitting a parametric function to extracted component y(t)
Thusday, June 16, 2009 22
Identifying Cortical Process leading to Response Time Variability (Con..)
• Gaussion Profile: parameterized by its height , width, delay and baseline offset :
• Response-locking of discriminating components determined by computing linear regression coefficients that predict latency of component activity, measured by from response time given by r.
• `j = rj + b [`j-peak latencies, rj-response time for j-th trial, b-offset term for regression] (Fig. 25.6)
• Proportionality factor from response time peak latency regression (): degree of response-locking (%) for each com.
Quantify extent to which component is correlated with response across trails.
Thusday, June 16, 2009 23
0%: pure stimulus 100%: pure response lock=100% : slow responses late activity fast responses early activity=0%: timing activity does not change with response time & therefore stimulus locked.
Thusday, June 16, 2009 24
Identifying Cortical Process leading to Response Time Variability (Con..)
• Group results: discriminating component activity across 9 participants (Fig. 25.7)
• Scalp projections normalized prior averaging
• A shift of activity from frontal to parietal regions over the course of 200 ms.
• Scaled response times and component peak time are concatenated across subjects
• These registered group response times then projected onto scaled component peak times to estimate degree of response-locking across subjects.
• Group response lock increased from 28% (200ms) to 78% (50 ms) after response.
Thusday, June 16, 2009 25
Thusday, June 16, 2009 26
Identifying Cortical Process leading to Response Time Variability (Con..)
Significant processing delay introduced by early processing stages
Within 200 ms prior to response (250 ms following stimulus), activity is already, on average, bet’n 25 & 35% response-locked
It is possible that some of this early response-locking may be due to early visual processes (0-250 ms post stimulus
For 9 subjects: correlation Analysis reveal that Discrimination Component activity progressively becomes more response-locked with subsequent Processing stages
Thusday, June 16, 2009 27
EEG-Based Image Triage• EEG system capable of using neural signatures detected
during RSVP to triage sequences of images, reordering them so target images are placed near the beginning of sequence.
• Called: “Cortically coupled computer vision”Robust recognition capabilities of human visual system
(invariance to pose, lighting, scale) use a noninvasive cortical interface (EEG) to intercept
interface detects the result (decision) of processingThusday, June 16, 2009 28
Image Triage (Con..)• Experimentation: Series of self-paced feedback slides were
presented indicating position of target images within sequence before & after EEG triage.
• Participants completed 2 blocks (50 sequences) with brief rest period lasting no more than 5 minutes bet’n blocks
• During 2nd block: participants were instructed to quickly press left button of 3-button mouse with right index finger as soon as they recognized target images.
• Button press twice if one target immediately followed other.
• Not respond with a button press during 1st block.
• Classify EEG online: used Fisher linear discriminator to estimate a spatial weighting vector that maximally discriminate
between sensor array signals evoked by target & non-target images.
Thusday, June 16, 2009 29
Image Triage (Con..)• 5000 images presented to subject in sequences of 100 images
• Training window: 400-500 ms following stimulus onset was used to extract training data
• Weights updated adaptively with each trail during training
• Classification threshold adjusted to give optimum performance for observed prevalence (Class-prior)
• These weights & threshold were fixed at end of training period & applied to subsequent testing dataset (images: 2501-5000)
• Offline triage: after experiment multiple classifier with different training window onsets were used.
Thusday, June 16, 2009 30
Image Triage (Con..)• Training window: 0~900 ms in steps of 50 ms
• Duration: 50 ms
• After trained, optimal weighting outputs found using logistic regression to discriminate target & non-target images.
• EEG data evoked by 2500 images to train & find inter-classifier weights.
applied to testing dataset evoked by 2nd set 2500 images (2501~5000)
Following experiments: all image sequences were concatenate to create training & testing sequences that each contain 2500 images (target# 50, non-targets# 2450).
Image sequences are restored according to output of classifier with multiple training windows for EEG evoked by every image.
Thusday, June 16, 2009 31
Image Triage (Con..)• Comparison: Sequence triaged based on button response.
• Image restored:
• RT-onset of button response (occurs within1 sec)
• P(target/RT)=0, no response
• Priors P(target) = 0.02
• P(non-target)=0.98
• P(RT/target): Gaussian distributions with mean & variance determined from response times from training sequences
• If target appeared 1st in sequence & 2 button response occurred within 1 sec-assigned to target images
• 2nd response to 2nd target image
Thusday, June 16, 2009 32
Image Triage (Con..)• Testing sequences: if 2 or more within 1 sec, response with
greatest p(target/RT) assigned to image.
• P(RT/non-target) is a mixture of 13 Gaussians (same variance as p(RT/target))
• Means: shifting mean from p(RT/target) 600 ms in past to 700 ms in future, increments of 100 ms, exclude actual mean of p(RT/target)
• Triage results (subject 2): Fig. 25.8
• # of targets as a function of # of distracter images both before & after triage based button press & EEG.
• area under curve generated by plotting fraction of targets as a function of fraction of distracter images presented is used to quantify triage performance.
Thusday, June 16, 2009 33
Thusday, June 16, 2009 34
Image Triage (Con..)
• Area is 0.50 for all unsorted image sequences since target images are randomly distributed throughout sequences.
• Ideal: 1.00
• No significant diff in performance (button-based & EEG-based) triage (0.930.06, 0.92 0.03, p=0.69, N=5)
• position of target images (black squares) & non-target images (white squares) in images sequences (Raster 25.8b-f)
• Performance (5 subjects): Table 25.1
Thusday, June 16, 2009 35
Image Triage (Con..)
• Performance (5 subjects): Table 25.1
• High-level performance
• Button-based triage performance begins to fail, when subjects do not consistently respond to target stimuli (response time>1 sec), subject 2: only 74%
• Invasive & noninvasive EEG recording obtained during RSVP of natural image stimuli have shed on speed, variability & spatiotemporal dynamics of visual processing.
• Future issues: learning/priming/habituation, effect of subject expertise, image type & category
• Application level: intercepting neural correlates of visual discrimination & recognition events that effectively bypass “slow & noisy” motor response loop.