High Transfer Rate, Real-time Brain-Computer Interface

1

High Transfer Rate, Real-time Brain-Computer Interface

Machine-based learning techniques towards a practical spelling device for

the completely paralyzed

April, 2005 ThinQ Innovation 2

Agenda

Brain Computer Interfaces – brief intro. Our system

– Overview, technical details– Machine learning – Support Vector Machines– Additional Bandwidth – Word Prediction – Results

Future Improvements, Q&A Demonstration at Psychology Lab


BCIs – the Need

‘Locked-in’ patients

Example: J.D. Bauby, “The Diving Bell and the Butterfly”

Persistence of life –“butterfly” Extreme physical disability –

“diving bell”


BCIs – the Need Amyotrophic Lateral Sclerosis

(ALS), aka Lou Gherig’s

– Degeneration of motor neurons, paralysis of voluntary muscles

– 120,000 diagnosed each year worldwide

– 2000 Canadians live with ALS right now

– Can leave patients ‘locked-in’– Cognitive and sensory functions

remain intact


BCI(1): Slow Cortical Potentials (SCPs)

• Extensive training ~ 3 months using biofeedback mechanism

• Tested on ALS patients, learned to control SCPs

Ref: N. Birbaumer et al., “The thought translation device (TTD) for completely paralyzed patients,” IEEE Trans. Rehab. Eng., Vol. 8, pp. 190-193,June 2000.


BCI(1): SCPs cont.• Most successful subject – artificially fed and respirated for 4 years

• After 3 months of training, wrote letter below

-Took 16 hours to write ~ 2 letters/minute

-Expresses thanks, wants to have a party


BCI(2): Implants - Cyberkinetics Inc.

• BrainGate Neural Interface System: Mkt. cap ~$45mil.

• Control of cursor on PC using implant in motor cortex

• Undergoing limited clinical trials

• Limb movement possibilities


P300 Spelling Device – the P300 Event Related Potential

• Known as ‘oddball’ or ‘surprise’ paradigm• Inherent

300ms

8-40 uV avg. deflection


P300 Spelling Device – the System• Non-invasive

•Inherent Response


P300 Speller Terminology Epoch = One flash of any row or column Trial = 1 complete set of epochs - all rows and

columns Symbol = Alphanumeric characters or pictures


BCI Competition 2003

• Provided pre-collected data for competition

• P300 Spelling Paradigm:

-Winners included Kaper et al.

-Used Support Vector Machines

-Achieved high transfer rate with real-time implementation possibilities


System Operation Steps

– Training (approximate 1hr)• Provide visual stimuli (flashing of rows/columns)• Record data with known classification label• Run data through pattern recognition algorithm (SVM)• Create customized models for each individual

– Spelling• Load customized model for individual• Provide visual stimuli (flashing of rows/columns)• Record data with unknown classification label• Run data through SVM classifier• Sum up decision values• Feedback most probable letter


Display Flexible matrix size Flexible matrix contents

– Alphanumeric Characters– Words– Symbols


Display cont… Random and exhaustive flashing of all of the rows

and columns on display Flashing cycle: 300ms

– 100ms intensification period– 200ms de-intensification period

10 second rest period at the end of each symbol


Data Collection Collect data from DAQ sampled at 240Hz 600ms after intensification Buffer overlap Flexible data collection delay Flexible data recording time


Data Collection – cont. 10 channels collected simultaneously Data from each channel concatenated together Data stored into program memory Collected until end of a symbol

– Converted to array– Memory cleared for next symbol

System is timing critical


Timing Issue Purpose

– Process within 300ms window

Bottleneck– Online SVM processing

• Old design = 340ms/Epoch• New design = 17.67ms/Epoch

Requirement– Pentium4 or equivalent is sufficient


Matlab Interface Why we use Matlab? VB–Matlab interface

using APIs Common functions

– Pass matrix array to Matlab workspace

– Get matrix array from Matlab workspace

– Execute command line or script


Support Vector Machines Pattern recognition Algorithm SVM used for:

– Creating models for different individuals (train)– Getting discriminant scores (spelling)

Detailed information covered later


Score Matrix

Coordinate Index 1 2 3 4 5 6

7 0.000 0.000 0.000 0.000 0.000 0.0008 1.100 0.000 0.000 0.000 0.000 0.0009 0.000 0.000 0.000 0.000 0.000 0.000

10 0.000 0.000 0.000 0.000 0.000 0.00011 0.000 0.000 0.000 0.000 0.000 0.00012 0.000 0.000 0.000 0.000 0.000 0.000


7 0.000 0.000 0.000 0.000 0.000 0.0008 1.100 1.100 0.000 0.000 0.000 0.0009 0.000 0.000 0.000 0.000 0.000 0.000

10 0.000 0.000 0.000 0.000 0.000 0.00011 0.000 0.000 0.000 0.000 0.000 0.00012 0.000 0.000 0.000 0.000 0.000 0.000


7 0.000 0.000 0.000 0.000 0.000 0.0008 1.100 1.100 1.100 0.000 0.000 0.0009 0.000 0.000 0.000 0.000 0.000 0.000

10 0.000 0.000 0.000 0.000 0.000 0.00011 0.000 0.000 0.000 0.000 0.000 0.00012 0.000 0.000 0.000 0.000 0.000 0.000


7 0.000 0.000 0.000 0.000 0.000 0.0008 1.100 1.100 1.100 1.100 0.000 0.0009 0.000 0.000 0.000 0.000 0.000 0.000

10 0.000 0.000 0.000 0.000 0.000 0.00011 0.000 0.000 0.000 0.000 0.000 0.00012 0.000 0.000 0.000 0.000 0.000 0.000


7 0.000 0.000 0.000 0.000 0.000 0.0008 1.100 1.100 1.100 1.100 1.100 0.0009 0.000 0.000 0.000 0.000 0.000 0.000

10 0.000 0.000 0.000 0.000 0.000 0.00011 0.000 0.000 0.000 0.000 0.000 0.00012 0.000 0.000 0.000 0.000 0.000 0.000


7 0.000 0.000 0.000 0.000 0.000 0.0008 1.100 1.100 1.100 1.100 1.100 1.1009 0.000 0.000 0.000 0.000 0.000 0.000

10 0.000 0.000 0.000 0.000 0.000 0.00011 0.000 0.000 0.000 0.000 0.000 0.00012 0.000 0.000 0.000 0.000 0.000 0.000


7 0.000 0.000 0.000 0.000 2.011 0.0008 1.100 1.100 1.100 1.100 3.111 1.1009 0.000 0.000 0.000 0.000 2.011 0.000

10 0.000 0.000 0.000 0.000 2.011 0.00011 0.000 0.000 0.000 0.000 2.011 0.00012 0.000 0.000 0.000 0.000 2.011 0.000


7 0.000 0.000 0.000 0.000 2.011 0.0008 1.100 1.100 1.100 1.100 3.111 1.1009 0.000 0.000 0.000 0.000 2.011 0.000

10 0.000 0.000 0.000 0.000 2.011 0.00011 -2.500 -2.500 -2.500 -2.500 -0.489 -2.50012 0.000 0.000 0.000 0.000 2.011 0.000


7 -8.236 -6.458 -4.596 -1.987 -0.234 -0.2398 -9.300 -8.985 -6.235 -1.256 13.234 -0.8729 -9.632 -7.372 -6.999 -0.945 -1.300 -2.386

10 -8.123 -7.458 -4.563 -2.397 -2.011 -3.02411 -7.985 -8.001 -5.335 -3.598 -3.501 -5.07512 -9.234 -9.021 -6.346 -4.871 2.011 -5.897


Word Prediction Idea: predict intended words based on previous

spelling. Similar to cellular phone ‘smart text’ Extract top ranked words

– SQL for fast searching– Dynamic database

Selection updated on

the bottom of the display Words chosen same way


System Design Modular Design Approach


What is SVM?

Developed by Vapnik in 1992 at Bell Labs Broad applications Based on concept of ‘learn from examples’ Key concepts:

– Linear Decision Boundary with Margin– Nonlinear feature transformation


Basic Concept

{x1, ..., xn} be our training data set

yi {1,-1} be the class label of xi then,

Find a decision boundary Make a decision on disjoint test data


Decision Boundary (linear)

Infinite possibility

Class -1

Class 1


Bad Decision Boundary

Class -1

Class 1

Class -1

Class 1


Good Decision Boundary

Class -1

Class 1

m Want to maximize m Boundary found using

constrained optimization problem


Optimization Problem

Optimization Problem


After Training xi’s on the decision boundary are called

SUPPORT VECTORS Support vectors and b defines the

decision boundary


Geometrical Interpretation

6=1.4

Class -1

Class 1

1=0.8

2=0

3=0

4=0

5=07=0

8=0.6

9=0

10=0


Non-separable Samples

Use of Soft Margin Separation Kernel Transformation


Soft Margin Separation

Class -1

Class 1


Soft Margin Separation

Idea: simultaneous maximization of margin and minimization of training error


Nonlinear Samples Some Samples are inherently nonlinear in

input space

No linear boundary is sufficiently accurate


Solution?


Kernel Transformation Idea: map input space into feature space

such that samples become linearly separable


Gaussian Kernel


SVM Implementation Matlab interface to libsvm Kernel: RBF with = 6.6799e-4 C parameter: 20.007


SVM Implementation

Average Method (61.538%) Multi-Model Method (65.22%) Concatenation Method (82.418%) Weighted Concatenation Method

(max. 86.264%)


Possible Improvements Weighted concatenation method

Customized Kernel Parameters

Weighted Concatenation Method

0.50x0.75x

1.25x

1.50x 1.75x 2.00x5x

78

80

82

84

86

88

weighting at (Pz, PO7, PO8 sites)

Accu

racy (

%)


Measure of Performance Bit Rate

– N: number of available symbols – p: prediction accuracy– t: number of seconds taken to choose one

symbol Letters per minute

1

1log1loglog

60222 N

pp)(+pp+N

t


Cont… Resulting Transfer Rates

– Without using dictionary

– With using dictionary


More Accurate Measure Resulting Transfer Rates

– Without using dictionary

– With using dictionary


Cont… Mechanism

– Receives a chosen letter from control module– Appends the letter to current letters in the word– Searches SQL database– Return list of most probable target words based

on ranking


Result Analysis

1. Accuracy across subjects

2. Accuracy over time, same subject

3. Accuracy over number of trials

4. Accuracy versus model size


Accuracy Across Subjects

Date Subject # of Trials Accuracy

(letters)

Percentage

March 26 Jack 15 12/14 86%

March 28 Min 15 12/21 57%

March 30 Brian 15 19/19 100%

March 31st Jyh-Liang 15 26/26 100%

April 2 Lucky 15 25/28 89%


Accuracy Across Subjects

Date Subject # of Trials Accuracy

(letters)

Percentage





Accuracy Over Time, Same Subject Subject: Jack

Date Time Change

from Model Made

# of Trials Accuracy

(letters)

Percentage

March 26 0 15 12/14 86%

March 28 2 days 15 19/19 100%

March 31 5 days 15 19/22 86%


Accuracy Over Number of Trials Subject: Jyh-Liang

Accuracy Vs Number of Trials

0%

20%

40%

60%

80%

100%

120%

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

Number of Trials

Acc

ura

cy


Accuracy Versus Model Size

Accuracy Vs Model Size [5 Trials]

0.00%

20.00%

40.00%

60.00%

80.00%

100.00%

120.00%

6.00 10.00 14.00 21.00 25.00 30.00

Size of Models (# of Symbols)

Ac

cu

rac

y (

%)

Subject: Jyh-Liang


Questions?

High Transfer Rate, Real-time Brain-Computer Interface

Documents

rowscolumnsrecord data

precollected data

practical spelling device

als patients

months of training

paralyzed patients

exhaustive flashing

thought translation