PLS’09 Beijing, China, September 7, 2009 Sponsored by US Army Research Office STIR: Advanced Estimation of Cognitive Status Contract No. W911NF-08-C-0121.

PLS’09 Beijing, China, September 7, 2009

Sponsored by US Army Research Office STIR: Advanced Estimation of Cognitive StatusContract No. W911NF-08-C-0121 15-SEP-2008 TO14-MAR-2009

PLS Tools in ElectroencephalographyLeonard J. Trejo

PDT InstitutePalo Alto, CA 94303, USA

The 6th International Conference on Partial Least Squares and Related Methods

Sept. 4th – 7th, 2009Beijing, China


PDT and PDT Institute

• PDT• Neuroergonomics Models and Applications

• Human-System Integration• Human Performance Optimization

• Robust Biomedical Signal Processing• Embedded and Real-time Systems for Bio-Sensing

• PDT Institute• PhD/Masters/Undergraduate Training• University Partners (UC Santa Cruz, Tsinghua

University, UC San Diego, Univ. of West Florida)


When I am not working…


Outline• Problem: stress, workload, fatigue and performance• Response: Neuroergonomic models and control systems

– Create useful definitions of cognitive states– Model, estimate and control cognitive states

• Background– Multimodal sensor-state models using PLS and KPLS Algorithms– Successes and failures: fatigue / BCI / engagement / workload

• New directions– Truly multidimensional sensor-process models– PARAFAC, N-PLS

• Summary


Estimation of Cognitive States

Aroused or Overloaded

Fatigued

RestingEngaged

Working

Other States

Behavior and Performance

Rewardsystem

Executivecontrol

WorkingmemorySensation

& perception

Autonomicsystem

Other Processes

Internal Processes

Biosignals


Useful Definitions

• Engagement: selection of a task as the focus of attention and effort

• Workload: significant commit-ment of attention and effort to task

• Overload: task demands outstrip performance capacity

• Mental Fatigue: desire to with-draw attention and effort from a task

Work-load

MentalFatigue

Non-specificFactors

Engage-ment

GeneralCognitive

Status


ElectroencaphalogramCerebral Cortex

• the outermost layers of brain• 2-4 mm thick (human)


EEG Sources


EEG Sources


Other Elements of Sensor-State Models

Modality Effect of Workload

Heart rate Increase

Heart rate variability (and HFQRS) Decrease

Vertical and horizontal EOG (eye movements)

Increase

Blinks May decrease for intake

Pupil diameter Increase

Skin conductance, SCR, GSR Increase

EMG (frontalis, temporalis, trapezius) Increase


Successful Application 1: Mental Fatigue Black = Alert Red = Mentally Fatigued

Fz Pz

FrontalTheta

ParietalAlpha


Robust EEG-Based Classification of Mental Fatigue

2300 (Day 1) vs. 1900 Hrs (Day 2)

40

50

60

70

80

90

100

Signal-to-noise Ratio (dB)

Te

st

Pro

po

rtio

n

Co

rre

ct

21 Channels 1001009994795650

12 Channels 98989688655150

4 Channels 87889088765450

0-3-6-9-12-15-18


Successful Application 2: BCI



Central sulcus Primary motor areaSecondary motor area

Secondarymotor area

Lateral sulcus

Primary motor area

Resting State Real or imaginary motion

(Adapted from Beatty, 1995)

EEG from Motor Areas



Control System for Target PracticeTrial-by-trial classification (left, right)

– 250 ms display updateDual adaptive controller design

– Adaptive PLS pattern recognition– Adaptive gain control for motion




Partly Successful Application: Mental Workload Estimation

Trejo, et al. ACI 2007Trejo, et al. ACI 2007


Stress, Workload, Fatigue and Performance

Trejo, et al. ACI 2007


Stabilizing Classifiers

Spectral Normalization

EEG Bandwidth Limiter

EEG Spectral Features

Classifiers

2s ECG Epoch

MV EOG/ECG Regression

Filters

R-wave Detector

PSDEMG

FeaturesRMS20,

Burst Duration, Burst

Frequency…

Innovations in EEG Algorithm Stabilization

ECG

FeaturesR-wave detectHR,

HRV, STD-IBI,

…

EOG

FeaturesRMS20, Blinks, EMs ,

…

AVAS Thresholds

Filters

4- 20s ECG Epoch

Gauges

PLS Algorithm


Multimodal Overload Patterns

0 50 100 150 200 250 300 350 400 450

50

100

RT - blue; HRstd - red

0 50 100 150 200 250 300 350 400 4500

5

10

15

Left temporalis EMG - blue; Right temporalis EMG - red

0 50 100 150 200 250 300 350 400 4500

5

10

15

Fz/theta - blue; Pz/alpha - red

0 50 100 150 200 250 300 350 400 4500

100

200

300

vEOG - blue; hEOG - redTime (s)

Val

ue


Workload-related EEG Sources

Passive viewing: theta alpha

Engaged 5: theta alpha

10.55 Hz

10.30 Hz5.79 Hz

5.79 Hz

Anterior Cingulate Inferior Parietal Precuneus


Application Summary• Models of engagement, fatigue & BCI:

• 90-100% accurate• Stable within a day• Stable from day to day

• Models of mental workload:• 60-90% accurate• Moderately stable within a day• Unstable from day to day


Recommended Directions

1. Deployable multimodal sensors (EEG, fNIR, EOG,

gaze, HRV, EMG, SCR, SpO2, BP, core body

temperature, gesture, posture facial expression, ...)

2. Multimodal experimental designs and operational tests

3. Advanced neurocognitive process models

4. Multimodal sensor-process mapping algorithms


“Atomic” EEG Elements

Atoms

Molecule

Basic Sources“atoms”

CoherentSystems

“molecules”

Coherence BondsCovalent Bonds


“Molecular” EEG Processes

Coherence BondsAtoms


Familiar (bilinear) Mapping Algorithms

Factor Analysis

Principal Component Analysis (PCA)

ijjf

F

fifij ebax

1

F

f 1 af

bf

0ije


Multimodal MappingHow to generalize bilinear models to systems with more dimensions?

1. Unfolding a bilinear modela. Represent all experimental factors in one dimensionb. Observations (trials) is second dimensionc. Contrast each dimension vs. pairs of the other two

2. Multidimensional modela. Assume orthogonal factors: PARAFACb. Allow interacting factors: Tucker 3

3. Modeling approacha. Unsupervised extraction: PARAFAC, CANDECOMP, Tucker 3b. Supervised extraction: N-PLS


Unfolding a Bilinear Model

Unfolding

Dim 1 Dim 2 Dim 3

XX

X1 X2 X3


Multidimensional Modeling (Tucker 3 Model, unsupervised)

ijklmnknjm

F

nil

F

m

F

lijk egcbax

321

111

• xijk is an element of (l x m x n) multidimensional array

• F1, F2, F3 are the number of components extracted on the 1st, 2nd and 3rd mode

• a, b, c are elements of the A, B, C loadings matrices for the 1st, 2nd and 3rd mode

• g are the elements of the core matrix G which defines how individual loading vectors in different modes interact

• eijk is an error element (unexplained variance)


PARAFAC (Parallel Factor Analysis, unsupervised)

ijkkfjf

F

fifijk ecbax

1

F

f 1 af

bf

cf

PARAFAC is a special case of the Tucker 3 model where F1= F2 = F3=F and G = I For a 3-way array:


N-way PLS(supervised)

X

frequency

workload condition

X

F

f 1 af

cfbf

F

f 1 vf

uf

time

time

max. covariance

electrodes

EEG

Labels

af – spectral atom bf – spatial atom cf – temporal atom

vf – workload atom uf – temporal atom


Demo: Workload / PARAFAC EEG

Workl

oad co

ndtions

(e.g.

, trial

s, time)

Elec

trod

es

EEG Frequency


Summary•Successes and Failures

• Fatigue, BCI, engagement: accurate, stable• Workload: variably accurate, unstable

•Useful models of state-related EEG sources• “Atomic” EEG sources• “Molecular” EEG systems

•Approaches to multidimensional models and algorithms• Tradtional bilinear methods (PCA, factor analysis, ICA)• Truly multidimensional methods

•Correlated factors (Tucker 3)•Uncorrleated factors (PARAFAC, CANDECOMP, N-PLS)•Supervised algorithms (N-PLS)

PLS’09 Beijing, China, September 7, 2009 Sponsored by US Army Research Office STIR: Advanced Estimation of Cognitive Status Contract No. W911NF-08-C-0121.

Documents

pls09 beijing

bci slide

ment slide

trapeziusincrease slide

thick human slide

west florida slide

npls summary slide

successful application