BMI Principles

BMI Principles

Jose C. PrincipeUniversity of Florida

Adapted from Hayrettin Gürkök, U. of Twente, NL

Literature

Difficulties in Invasive BMIs

• BCIs offer an easy entry to research – Non invasiveness straight forward data collection– Closer to cognition– Conventional signal processing

• BMIs research infrastructure is much harder– Work with animals (ethics)– Difficult instrumentation– Unclear signal processing

Choice of Scale for Neuroprosthetics

Bandwidth (approximate)

Localization

Scalp Electrodes

0 ~ 80 Hz Cortical SurfaceVolume Conduction3-5 cm

Electro-corticogram (ECoG)

0 ~ 500Hz Cortical Surface0.5-1 cm

Micro Electrodes

0 ~ 500Hz

500 ~ 7kHz

Local Fields1mm

Single Neuron200 mm

footing

two polyimide cables

Electrode Arrays

0 0.01 0.02 0.03 0.04-40

-30

-20

-10

0

10

20

30

40

50

Time (s)

Micr

ovol

ts

J. C. Sanchez, N. Alba, T. Nishida, C. Batich, and P. R. Carney, "Structural modifications in chronic microwire electrodes for cortical neuroprosthetics: a case study," IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2006

Utah array

Brain Gate

Michigan probes

Technical Issues with BMIs

• An implantable BMI requires beyond of state of the art technology:– Ultra low power– Ultra miniaturized– Huge data bandwidth/power form factor– Packaging

28mm

15mm

12mm Thru vias to RX/Power Coil

+

12.5 mm

Coil winding

3.5 mm

50µm pitch Electrodes

Coin Battery(10 x 2.5 mm)

Thru vias to Battery

Supportingscrews

Flexiblesubstrate

TX antenna

ModularElectrodes

Electrodeattachment

sites

IF-IC

RFIC

18 mm

Coil

Battery

PatternedSubstrate

SupportingSubstrate

ElectrodeArray

IC

Flip-chipconnection

Specifications:16 flexible microelectrodes (40 dB, 20 KHz)Wireless (500 Kpulse/sec)2mW of power (72-96 hours between charges)

FWIRE: Florida Wireless Implantable Recording Electrodes

RatPack Low-Power, Wireless, Portable BMIs

• Requirements– Total Weight: < 100g– Battery Powered: Run for 4 hours

• Implantable– Biocompatible– Heat flux: < 50 mW/cm2

– Power dissipation should not exceed a few hundred milliwatts

• Backpack– Small form factor– Speed vs. Low Power

UF PICO SystemPICO system = DSP + Wireless

Generation 3

J.R. Wolpaw et al. 2002

BCI (BMI) bypasses the brain’s normal pathways of peripheral nerves (and muscles)

General Architecture

BMIs: How to put it together?• NeoCortical Brain Areas Related to

Movement Posterior Parietal (PP) – Visual to motor transformation

Premotor (PM) and Dorsal Premotor (PMD) -

Planning and guidance (visual inputs)

Primary Motor (M1) – Initiates muscle contraction

Motor Tasks Performed

-40 -30 -20 -10 0 10 20 30 40-40

-30

-20

-10

0

10

20

30

40

Task

1Ta

sk 2

Data• 2 Owl monkeys – Belle, Carmen• 2 Rhesus monkeys – Aurora, Ivy• 54-192 sorted cells• Cortices sampled: PP, M1, PMd, S1, SMA• Neuronal rate (100 Hz) and behavior is time synchronized and downsampled to 10Hz

100 msec Binned Counts Raster of 105 neurons (spike sorted)

Firing Rates

Time

Neu

ron

Num

ber

200 400 600 800 1000 1200 1400 1600 1800 2000

10

20

30

40

50

60

70

80

90

100

Ensemble Correlations – Local in Time – are Averaged with Global Models

Computational Models of Neural Intent

• Three different levels of neurophysiology realism

– Black Box models – function relation between input - desired response (no realism!)

– Generative Models –state space models using neuroscience elements (minimal realism).

– White models – significant realism (wish list!)

Optimal Linear Model

• The Wiener (regression) solution

• Normalized LMS with weight decay is a simple starting point.

• Four multiplies, one divide and two adds per weight update

• Ten tap embedding with 105 neurons• For 1-D topology contains 1,050

parameters (3,150)

)()()(

)()1( 2 nxnenx

nwnw

pw 1)( IR

Z-1 delay of 1 sampleS adderwi(n) parameter i at time n

w0

w9

3-D, 2-D Trajectory Modeling and Robot Control

• Collaboration with Miguel Nicolelis, Duke University

• Sponsored by DARPA

Time-Delay Neural Network (TDNN)

• The first layer is a bank of linear filters followed by a nonlinearity.

• The number of delays to span I second

• y(n)= Σ wf(Σwx(n))• Trained with

backpropagation• Topology contains a ten tap

embedding and five hidden PEs– 5,255 weights (1-D)

Principe, UF

Multiple Switching Local Models

• Multiple adaptive filters that compete to win the modeling of a signal segment.

• Structure is trained all together with normalized LMS/weight decay

• Needs to be adapted for input-output modeling.• We selected 10 FIR experts of order 10 (105 input channels)

d(n)

Recurrent Multilayer Perceptron (RMLP) – Nonlinear “Black Box”

• Spatially recurrent dynamical systems

• Memory is created by feeding back the states of the hidden PEs.

• Feedback allows for continuous representations on multiple timescales.

• If unfolded into a TDNN it can be shown to be a universal mapper in Rn

• Trained with backpropagation through time

))1()(()( 1111 byWxWy ttft f

2122 )()( byWy tt

Generative Models for BMIs

• Use partial information about the physiological system, normally in the form of states.

• They can be either applied to binned data or to spike trains directly.

• Here we will only cover the spike train implementations.

Difficulty of spike train Analysis: Spike trains are point processes, i.e. all the information is

contained in the timing of events, not in the amplitude of the signals!

Particle Filters for Point Processes

)(iitt

Xkf

Kinematic State

Neural Tuning function spike trains

Prediction

it

itt

it vXFX 11

Updating

)|( )(1

it

jt

it

it Npww

)( jtN

NonGaussian

P(state|observation)

N

i

itt

it

jtt xxkwNxp

1:0:0

)(:1:0 )()|(

N

i

ikk

ikkk kWNp

1:1 )()|( xxx

Linear filter

nonlinearity f

Poisson model

kinematics

spikes

)( tt Poissonspike

)( lagtt kf x

Instantaneous tuning modelt

-0.1 -0.05 0 0.05 0.1 0.15 0.2 0.250

0.05

0.1

0.15

0.2

0.25

KX

(K

X)

neuron 80 nonlinear estimation

optimum delay

[-300, 500] ms

[-250, 450] ms

[-200, 400] ms

[-150, 350] ms

[-100, 300] ms

[-50, 250] ms

[0,200] ms

-50 -40 -30 -20 -10 0 10 20 30 40-50

-40

-30

-20

-10

0

10

20

30

40

x

y

position

desiredKalmanPPMonte Carlo PP

Generative Data Modeling

…..

…..

…..

…..

…..

…..

…..

…..

…..

…..

…..

…..

…..

…..

…..

…..

…..

…..

…..

…..

…..

…..

…..

…..

…..

…..

…..

…..

Neu

ral C

hann

els

Time

ObservableProcesses

(probed neurons)

HiddenProcesses

(Brain areas)

BMI lessons learnedBMIs are beyond the Proof of Concept

stage, but….Present systems are signal translators

and will not be the blue print for clinical applications

Current decoding methods use kinematic training signals - not available in the paralyzed

I/O models cannot contend with new environments without retraining

BMIs should not be simply a passive decoder – incorporate cognitive abilities of the user

BMI lessons learnedBMIs are beyond the Proof of

Concept stage, but….Present systems are signal

translators and will not be the blue print for clinical applications

Current decoding methods use kinematic training signals - not available in the paralyzed

I/O models cannot contend with new environments without retraining

BMIs should not be simply a passive decoder – incorporate cognitive abilities of the user

A Paradigm Shift for BMIs!

• During training the user actions create a desired response to the DSP algorithm.

• During testing the DSP algorithm creates an approximation to the desired response.

DSP algorithm

Desired response

Neural Signal Processing

• The control algorithm learns through reinforcement to achieve common goals in the environment.

• Shared control with user to enhance learning in multiple scenarios and acquire the net benefits of behavioral, computational, and physiological strategies

X

Control Algorithm

Learning Algorithm

Neural Signal Processing

A Paradigm Shift for BMIs!

Construction of a New FrameworkHow to capitalize on the perception-action cycle?

• The brain is embodied and the body is embedded

• Need to quantify Brain State at different time resolutions

• Intelligent behavior arises from the actions of an individual seeking to maximize received reward in a complex and changing world.

• The BMI must engage and dialogue with the user:– Exploits better engineering knowledge– Utilizes cognitive states– Dissects behavior top-down– Exploits rewards– Learns with use

• Propose Reinforcement Learning to train the BMI.

FUTUREPAST

INTERNALREPRESENTATION

EXTERNALWORLD

LIMBICSYSTEM

ORGANIZED PASTEXPERIENCE

PREDICTIVEMODELING

DOES ACTIONMEET

FUTURE REALITY?

SENSORYSTIMULUS

Causality line

Body line

Reward Learning Involves a Dialogue

• Relation between the agent and its environment.

• Environment: You are in state 14. You have 2 possible actions.

• Agent: I'll take action 2.• Environment: You received a

reinforcement of 17.8 units. You are now in state 13. You have 2 possible actions.

• Agent: I'll take action 1.

• repeat

AGENT

ENVIRONMENT

actions

rew

ards

stat

es

Goal

Start

CABMI involves TWO intelligent agents in a cooperative dialogue!!!

stat

es

ROBOT

actions

rew

ards

RAT’S BRAIN

environment

RAT’S BRAIN

COMPUTER AGENTUser’s

neuromodulationsets the value function for the

CA

Both the CA and the user have the same reward in 3D space

Features of co-adaptive BMI

• Enables intelligent system design in BMIs• Both systems adapt in close loop in a very tight

coupling between brain activity and computer agent ( CA states are specified by brain activity).

• User must incorporate the CA in its world (can a rat learn this?)

• CA must decode brain activity for its value function (can

it model the signature of behavior?). • In fact CABMI is a “symbiotic” biological-computer

hybrid system.

31

Experiment workspace [top view]

The user learns first to associate levers with water reward in a training phase.

In brain control, it progressively associates the blue guide LED of the robotic arm with the target lever LEDs. Only when the robot presses the target lever it will get reward.

Experiment workspace [top view]

Experimental CABMI Paradigm

-3 -2 -1 0 1 2 3

012

3

0

1

2

3

IncorrectTarget

CorrectTarget

StartingPosition

Match LEDs

Grid-space

Match LEDs

Rat’s PerspectiveWater Reward

Map workspace to grid

Rat

Robot Arm

Left Lever Right Lever

27 discrete actions 26 movements

1 stationary

Experimental CABMI Paradigm

• CA rewards are defined in 3D at the target lever positions.

• RL is used to train the CA in brain control (tabula rasa, i.e. no a priori information).

• During brain control, shaping of the reward field increases the level of difficulty across multiple sections with an adjustable threshold target.

35

-0.1-0.05

0

0.150.2

0.250.3

0.35

0.15

0.2

0.25

0.3

z

Left Target

xy

00.05

0.10.15

0.20.25

0.30.35

0.15

0.2

0.25

0.3

z

Right Target

xy

00.05

0.10.15

0.20.25

0.30.35

0.15

0.2

0.25

0.3

z

Right Target

xy-0.1

-0.050

0.150.2

0.250.3

0.35

0.15

0.2

0.25

0.3

z

Left Target

xy

Neuromodulation defines the States

Sampling rate 24.4 kHz

Hall, Brain Research (1974)

32 channelsSpike sorted data

Bilateral Premotor/motorAreas

Performance metrics

Performance metrics:1. Percentage of trials earning reward2. Average control time required to reach a target

4 sessions were simulated using random action selection to estimate chance performance for the CABMI in increasing difficulty tasks.

% trials earning reward time to achieve reward

Performance in 4 tasks of increasing difficulty

Closed-Loop RLBMI

Non-functional levers

Functional levers

Robot workspace in rat visual field of view.

BLUE – Robot

GREEN - Lever

Top-view of the rat behavioral cage.

• It is well established that preparation, execution, and also imagination of movement produce an event-related desynchronization (ERD) over the sensorimotor areas, with maxima in the alpha band (mu rhythm, 10 Hz) and beta band (20 Hz).

• The mu ERD is most prominent over the contralateral sensorimotor areas during motor preparation and extends bilaterally with movement initiation

• ERD during hand motor imagery is very similar to the pre-movement ERD, i.e., it is locally restricted to the contralateral sensorimotor areas

Event Related Desynchronization (ERD) and synchronization (ERS)

• During movement preparation and execution, an increase of synchronization (ERS) in the 10-Hz band normally appears over areas not engaged in the task (idling)

• ERS can also be observed after the movement, over the same areas that had displayed ERD earlier

Event Related Desynchronization (ERD) and synchronization (ERS)

Beta rebound following movement and somatosensory stimulation

• The general finding is that beta oscillations are desynchronized during preparation, execution, and imagination of a motor act

• After movement offset, the beta band activity recovers very fast (<1 s) and short-lasting beta bursts appear.

• The occurrence of a beta rebound related to mental motor imagery implies that this activity does not necessarily depend on motor cortex output.

• A number of experiments have also shown beta oscillations to be sensitive to somatosensory stimulation

ERS (Blue) and ERD (Red)

ERD

ERS 12.0 Hz +/- 1.0

Pfurtscheller

10.9 Hz +/- 0.9

ERS (Blue) and ERD (Red)

Pfurtscheller

Beta ERS

Pfurtscheller

Alpha and Beta ERS

Pfurtscheller

Signal Processing for ERD/ERS

• Bandpass filtering between 9-13 Hz will emphasize this component.

• Estimate the power • Place a statistical threshold for detection.

• Alternatively use PSD and threshold the appropriate frequency band.

Paradigm 1

(http://www.dcs.gla.ac.uk/~rod/Videos.html)

Paradigm 2

Event Related Potentials

• ERPs are a signature of cognition. They signal a massive communication amongst brain areas (kind of the brain’s impulse response to an internal stimulus).

• This is very good, but the problem is that it is normally much smaller than the ongoing EEG activity (i.e. the SNR is negative).

Event Related Potentials

• The ERP shape is well known and pretty stable across individuals, and has a known distribution across the channels.

• The P300 is the most used for BMIs because it is task relevant

N100-P200 complex is pre-attentive response appearing over sensory areasP300 signals a rare tasks relevant event (Cz)N400 signals an unexpected event (Cz)

Event Related Potentials

• In order to deal with the negative SNR, we use averaging of the stimulus.

• If you have a transient that appears in white Gaussian noise, align the transient and average across trielas you obtain an increase of SNR by , where N is the number of trials.

• This is normally done but has three shortcomings:– It is not real time– It assumes that the shape of the ERP is the same– It assumes that the latency is constant

N

P300 Event Related Potentials

P300 Event Related Potentials

Negative SNR so need averaging (i.e. repeated presentation of stimuli)

P300 Paradigm

P300 Paradigm

P300 Paradigm 2

The Cortical Mouse In 1990 the CNEL proposed a new computer interface that would control cursor movement in the screen using directly brain activity (EEG) and implemented in a NeXT Computer

• Decision based on single ERPs (N400) in real time • Neural network classifier implemented in DSP chip• Overall control (synch, screen, data flow) by the OS

Left/RightYES NO

Konger, C., Principe, J., ANN classification of ERPs for a new computer interface IEEE IJCNN, 1990Sina Eatemadi, A new computer interface using event related potentials University of Florida, 1992.

4.5bits/min

Slow Cortical Potentials

SCP Paradigm

Steady State Evoked Potential (ssEP)

ssVEP Paradigms

ssVEP Paradigms

ssVEP Paradigms

• One of the most reliable effects. • Need to do FFT of occipital channels and pick

the highest frequency.• Car race (winner of the first BCI competition)

Taxonomy of BCI paradigms

Taxonomy of BCI paradigms

Taxonomy of BCI paradigms

Taxonomy of BCI paradigms

Mu Rhythm

• When a subject imagines movement or sees movement made by others a burst of activity in the 8-12Hz range appears over the sensorimotor areas in the brain

• The subject can synchronize the rhythm and by moving desynchronize it, hence it ia good signal to be used for motor BMI tasks.