Brain Signal and in Brain Machine Interface (BMI) Tassos ...phdsch11/wp-content/... · Brain Signal Processing and Applications in Brain Machine Interface (BMI) Tassos Bezerianos,

Brain Signal Processing and Applications in Brain Machine Interface (BMI)

Tassos Bezerianos, BS,MS,PhD

Head of Biosignal Lab, Dept of Medical Physics

School of Medicine, University of PatrasPATRAS, GREECE

1

1st PhD School on Complexity Sciences July 18‐29, 2011

ContentsBrain Computer Interface (BCI) (Definitions, Features and Limitations)BCI ArchitectureBCI CategoriesBCI ApplicationsBCI at Game MarketFuture Developments

Biosignal Lab, Dept of Medical Physics School of Medicine, University of Patras, Greece 2

IntroductionInterface Between Brain and computer

Definition:Wolpaw et al.: “A direct brain‐computer interface is a device that provides the brain with a new, non‐muscular communication and control channel”

Biosignal Lab, Dept of Medical Physics School of Medicine, University of Patras, GreeceBiosignal Lab, Dept of Medical Physics School of Medicine, University of Patras, Greece 3

What is BCI (in General)A BCI enables communication without movement Most BCIs translate your brain’s electrical activity (EEGs) into messages or commands.

BCI Rely on

Mental Activities

Imagine Movement and Emotional

Imaginery

Selective attention (P300 and Steady

State Evoked


What is NOT a BCIBCI cannot read mindsInterpret mental activityWrite to the brainBiofeedbackProstheticsRetinal or cochlear implantsMedical EEGsEEG or fMRI Lie DetectionNeuromarketingEmployee screeningAttention or fatigue monitors


Who can use BCIWhy use a BCI if you’re healthy?BCIs:Only provide communication.Provide the same information available via

conventional interfaces.Are exclusive interfaces.Thus are of no practical value to people who canotherwise communicate.

Biosignal Lab, Dept of Medical Physics School of Medicine, University of Patras, Greece6Biosignal Lab, Dept of Medical Physics School of Medicine, University of Patras, Greece 6

Who can use BCIWho are able to use a BCI? People with disabilities of sensorimotor system .Due to a phenomenon named by the most of research groups as “BCI Illiteracy”, about 20% of people are not able to control a conventional BCI.

We Conclude that:Will not attain wider adoption without dramaticimprovements in information transfer rate (ITR).


A Definition for BCI and Neuroprosthetics

BCI NEUROPROSTHETICS

A speficificated device directly connected to any part of nervous

system

Connect the Brain with a Computer

System


Different Kind of Subjects

Different Kind of

Recordings

One or more signal Processing

Unit (or Combination)

Feature/Classifier Selection

ArchitectureFocus on the Basic Components

9Biosignal Lab, Dept of Medical Physics School of Medicine, University of Patras, Greece 9

Stimulus, Visual/Auditory Evoked Potentials

Command Execution

A BCI Example

Diagram of the BCI developed by Miguel Nicolelis and colleagues for use on Rhesus monkeys


How BCI works IThere are several brain areas with distinct functionality which is known.

Biosignal Lab, Dept of Medical Physics School of Medicine, University of Patras, Greece

PRESS cCLICK HERE

ToPreview The Video


How BCI works IIFor each BCI Implementation we are rely on the corresponding Brain region.

z

Imagine Movement and Emotional

Imaginery

Selective attention (P300 and Steady

State Evoked

Mental Activities


How BCI works IIIBrain Anatomy and Fuctions

For each BCI Implementation we rely on the corresponding Brain region.E.g. Finger Movement results in Event Related Desynchronization, while foot movement Event Related Synchronization at specific Frequency band (mu Rhythm) over the hand area


The beginning of the BCI

14

IEEE TRANSACTIONS ON BIOMEDICAL ENGINEERING, VOL. 51, NO. 6, JUNE 2004

Toward a Direct Brain Interface Based on HumanSubdural Recordings and Wavelet‐Packet Analysis


Functionally event results

ERD(Event-related

desynchronization)

ERS(Event-related

synchronization)

ERP(Event-related

potentials)

not phase-locked not phase-locked phase-locked

decreaseHand movement: in Mu rhythm

(9-13 Hz)

Closing eyes and relaxation: 12 Hz)-in alpha (9increase


Event related Synchronization/ Desynchronization

Nitish Thakor 16Biosignal Lab, Dept of Medical Physics School of Medicine, University of Patras, Greece 16

Blind Source Separation‐Motor Imaginary Enhacement


BCI Categories ‐Formally


BCI Categories

ECOG Microelectrodes Intracortical

SUA, MUA, LFPsEEG

NON INVASIVE SYSTEMS

INVASIVE SYSTEMS


Brief Introduction to Electroencephalography (EEG)

20

Electroencephalography (EEG)EEG Records the current flow in cortical areas, tracking tiny electrical impulses that caused by brain cells communication.


EEG Recording

Frequency Rage 0‐ 40Hz (Due to lowpass filter effect of the scalpand intervening tissues)

No Localized (spatial Resolution of few centimeters)

Amplitude Signals 10–20 microVolts

NON‐ INVASIVE

SUMMARYBOARD

22

Objectives of EEG studyThe EEG is used in the evaluation of brain disorders. Most commonly it is used to show the type and location of the activity in the brain during a seizure. It also is used to evaluate people who are having problems associated with brain function. These problems might include confusion, coma, tumors, long‐term difficulties with thinking or memory, or weakening of specific parts of the body (such as weakness associated with a stroke).An EEG is also used to determine brain death. It may be used to prove that someone on life‐support equipment has no chance of recovery.


CharacteristicsThe EEG is typically described in terms of rhythmic activity. The rhythmic activity is divided into bands by frequency: Delta, Theta, Alpha, Beta and Gamma.

Frequency Location

Delta < 4Hz Frontally in adults, posteriorly in children; high amplitude waves Adults slow wave sleep; Babies

Theta 4 – 7Hz Found in locations not related to task at hand

Young children; Drowsiness or arousal in older childen and adults; idling

Alpha 8 – 12Hz

posterior regions of head, both sides, higher in amplitude on dominant side. Central sites (c3‐c4) at rest .

Relaxed/reflecting; Closing the eyes

Beta 12 – 30 Hzboth sides, symmetrical distribution, most evident frontally; low amplitude waves

Alert/Working; Active busy or anxious thinking, active concertration

Gamma > 30Hz Somatosensory cortexShort term memory matching of recognized objects, sounds or tactile sensations; Cross‐modal sensory processing


Delta waves < 4Hz

Theta waves 4 – 7Hz

Alpha waves 8 – 12Hz

Beta waves 12 – 30 Hz

Gamma waves > 30Hz

EEG rhythmic activity


RecordingsEEG

An EEG recording net (Electrical Geodesics, Inc. ) being used on a participant in a brain wave study.

Montages•Bipolar •Referential•Average reference•Laplacian

EEG Electrode Positions


1 second, 1 Channel Recording

EEG Recording

An EEG recording net (Electrical Geodesics, Inc. ) being used on a participant in a brain wave study.

EEG Electrodes The Neuroscan SynAmps2amplifier, power supply,

and 70‐channel headbox (left)


EEG BCI – Recent Progress

Brice Rebsamen, et al., Controlling a wheelchair indoors using thought, Intelligent Systems, 2007.

Wireless EEG systemBCI controlled wheelchair

Chin‐Teng Lin, et al., Noninvasive Neural Prostheses Using Mobile and Wireless EEG, Proceedings of the IEEE, 2008

Gerolf Vanacker et al., Context‐based filtering for assisted brain‐actuated wheelchair driving EEG, Computational Intelligence and Neuroscience, 2007


EEG BCI ‐ Non‐invasiveBCI controlled FES

BCI combined with Virtual Reality

Leeb, R., et al., A tetraplegic patient controls a wheelchair in virtual reality, BRAINPLAY 2007

Pfurtscheller, et al., EEG‐based asynchronous BCI controls functional electrical stimulation in a tetraplegic patient, Eurasip Journal on Applied Signal Processing, 2005.


EEG BCI Visual Evoked Potentials Applications

Brain to Brain Communication

1010101110011 1010101110011


EEG BCIApplication for control a robotic hand


BCI Categories

ECOG Microelectrodes IntracorticalSUA, MUA

EEG


INVASIVE SYSTEMS


ECoG Recording

Frequency Rage 0‐ 200 Hz (reported usefull information 300Hz‐6Khz

More Localized than EEG (spatial Resolution of milimeters)

ECoG provides higher amplitude signals with values in the rangeof 50–100 microVolts compared to 10–20microVolts for EEG

INVASIVE

SUMMARYBOARD

33

ECoG electrodes‐ Platinum‐ Diameter: 4mm‐ Exposed electrode area diameter: 2.3mm‐ Grids of 8x8, 2x8, 4x5, etc.‐ Strips of 2‐16‐ Depth electrodes reach deeper structures (e.g.,

hippocampus)


MicroECoG‐ Higher spatial resolution‐ 0.04mm microECoG depth electrodes


ECoG InstrumentationHeadbox:

70 input channels24 bit A/D conversionActive noise cancellation110 dB CMRR10 GΩ input impedance

‐ System Unit: Synchronization of samplingTransmits data and trigger information to computer20KHz maximum sampling rate

‐ Power Unit: Isolated power supply necessaryfor patient protection


Use of ECoG for identification of functional brain areas

ECoG stimulations: determine critical location by disrupting the function.

ECoG recordings: mapping endogenous cortical function, reflecting normal cortical function.


ECoG Recording


Low frequency decrease in power (inhibitory) and high frequency increase with activity

Kai Miller et.al., 2007, J Neuroscience

Functional Brain mapping using ECoG


Functional Brain mapping using ECoG

Spectogram from a single electrode for 15s hand movement vs. 10s baseline. Decrease in mu rhythm in hand movement

Kai Miller et.al., 2007, J Neuroscience40Biosignal Lab, Dept of Medical Physics School of Medicine, University of Patras, Greece 40

Offline hand motor area mapping. The bar plots indicate the sum ofsuprathreshold activity for each electrode.

ECoG Based BCI system


Example of ECoG based BCI system

Finger flexion

ECoG features related to finger flexion

Brain signal changes between rest and thumb movement for subjects

J Kub´anek et.al., 2009, J Neural Eng 42Biosignal Lab, Dept of Medical Physics School of Medicine, University of Patras, Greece 42

Example of ECoG based BCI system

Actual and decoded movement trajectories

Relationship of brain areas with thumb flexion

J Kub´anek et.al., 2009, J Neural Eng

Interdependence of actual and decoded finger flexion. (thumb: first row/column, little finger: last row/column)

Discrete classification of finger movements. (from left to right: thumb, index, middle, ring and little finger)


Event Related Task Detected by ECoGMotor Imaginery


BCI Categories

ECOGMicroelectrodes IntracorticalSUA, MUA

EEG


INVASIVE SYSTEMS


Microelectrode Recording

Local Field Potentials Frequency Rage 0‐ 300 Hz. Well Spatial Localized.

Single and Multi Unit Activity (usually threshold 350Hz‐3500Hz)

INVASIVE

SUMMARYBOARD

46

Overview of all Methods

Gerwin Schalk et al: Sensor Modalities for Brain‐Computer Interfacing. 47Biosignal Lab, Dept of Medical Physics School of Medicine, University of Patras, Greece 47

Microelectrode RecordingsMicroelectrode recordings from within the brain represent activity from one or multiple neurons.These systems use:1. firing rates of individual or multiple neurons or2. or the overall neuronal activity (local field potentials (LFPs)) of multiple neurons recorded within the brain.


Microelectrode RecordingsHowever, the stability of recordings from electrodesimplanted within the brain is currently uncertain because electrodes are subject to different tissue responses that lead to encapsulationAlso, tiny movements of theelectrodes can move them away from individual neurons

MOSTLY LFPs ARE USED


Microelectrodes


Microelectrodes


Cell RecordingWhat we measure?


Multi‐ Single Unit Activity and Local Field Potentials

Spike ClusteringSpike Sorting

Spike Sorting


Typical Example of Animal SUA Activity Recording

~15msecond

EEG

Stimulus

Spikes‐MUA


Decoding spoken words using local field otentials recorded from the cortical surface

Spencer Kellis1, Kai Miller2, Kyle Thomson3 Richard Brown1, Paul House and Bradley Greger


An example of BCIThe four components of a closed‐loop, neural interface system: (1) a recording array that extracts neuralsignals, (2) a decoding algorithm that translates these neural signals into a set of command signals, (3) anoutput device that is controlled by these command signals, and (4) sensory feedback in the form of vision andpotentially other sensory modalities. Transparent head image is courtesy of c iStockphoto.com/KiyoshiTakahase Segundo.


Recording Technology

SfN Spike and ECoG Workshop g.tec


BCI based on Implants


BCI Implant and Muscle Stimulator

Mini‐Symposium Biomimetic Brain Machine Interfaces for the Controlof Movement Andrew H. Fagg,1 Nicholas G. Hatsopoulos,3,4,5 Victor de Lafuente,6 Karen A. Moxon,7 Shamim Nemati,2 James M. Rebesco,8 Ranulfo Romo,6 Sara A. Solla,8,9 Jake Reimer,3 Dennis Tkach,4 Eric A. Pohlmeyer,8,10 and Lee E. Miller8,10a


SUA, MUA and LFP RecordingsA real recording from Dr. N. Hatsopoulos Laboratory during an experiment similar to that is depicted in the figure above. PRESS

CLICK HERE

ToPreview The Video


Hardware and recording technology

Neurochip, Wikipedia

A wireless multi‐channel neural amplifier for freely moving animals


Use Dissolvable films

Dissolvable films of silk fibroin for ultrathinconformal bio‐integrated electronicsDae‐Hyeong Kim and Jonathan Viventi et al.*


Bio Integrated Electronics

Dissolvable films of silk fibroin for ultrathinconformal bio‐integrated electronicsDae‐Hyeong Kim and Jonathan Viventi et al.*


Other BCI Applications

Rats implanted with BCIs in Theodore Berger's experiments


Applications‐MultimediaRecently a system for rapid Image retrieval were proposed.


Other BCI ApplicationsDirect Neural Control of Anatomically Correct Robotic Hands‐Interface Technologies‐‐Control Strategies: Population Decoding


Games based on EEG BCI


The Future In BCIThe question now is:

“What would be the futureFor Brain Computer and Brain Machine Interface?”

New approachesFully implantable BMI.


Memristors A new approach to artificial Inteligence?

An array of 17 purpose-built oxygen-depleted titanium dioxide memristors built at HP Labs, imaged by an atomic force microscope. The wires are about 50 nm, or 150 atoms, wide.[1] Electric current through the memristors shifts the oxygen vacancies, causing a gradual and persistent change in electrical resistance.[2]


MoNETA: A Mind Made from Memristors


MoNETA: A Mind Made from Memristors


Hybrid BCIHYBRID BCI SYSTEMSSIMULTANEOUS ERD/SSVEP BCI TO IMPROVE ACCURACY

COMBINING EYE GAZE AND ERD BCI

“Is this Enough?”


Recent Activities of our Laboratory

October 20105th International Summer School on Emerging Technologies in Biomedicine “High Throughput Communication between Brain and Machines(http://heart.med.upatras.gr/school2010)


Summer School,Invited Speakers

Fabio BabiloniDepartment of Human Physiology and Pharmacology,Biophysics Interest Group, University of Rome "La Sapienza", Italy

Anastasios BezerianosDepartment of Medical Physics, School of Medicine,University of Patras, GreeceNicho Hatsopoulos

Dept. of Organismal Biology & Anatomy, Committees on Computational Neuroscience and Neurobiology, Center for Integrative Neuroscience and Neuroengineering Research (CINNR) University of Chicago , US



Andreas IoannidesAAI Scientific Cultural Services Ltd, Nicosia, Cyprus

Christopher J. JamesProfessor of Healthcare Technology and Director of a the Institute of Digital Healthcare, University of Warwick, UK

Jürgen KurthsProfessor and Chair of Nonlinear Dynamics, Institute for Physics, University of Potsdam, DE

Pedro LarrañagaProfessor at the Department of Artificial Intelligence at the Polytechnic University of Madrid.



Nikos LogothetisDirector of the Dept. "Physiology of Cognitive Processes" at the Max Planck Institute for

Biological Cybernetics (MPIK), in Tübingen, DEFivos PanetsosProfessor, Escuela Universitaria de Óptica, Universidad Complutense de MadridPanos M. Pardalos

Distinguished Professor Director, Center for Applied Optimization, Department of Industrialand Systems Engineering, University of Florida



Selma SupekProfessor of Physics, Department of Physics, University of Zagreb, Croatia

Nitish ThakorThe JHU Biomedical InstrumentationLabSchool of Medicine Johns Hopkins University, Baltimore, US

Marcin ByczukInstitute of Electronics, Medical ElectronicsDivision, Technical University of Lodz, PolandGunther Krausz – g‐tec

Guger Technologies OG, Graz, Austria


Experiments‐ Demonstration

Two Experiments Took Place1. Steady state Evoked Potentials2. Visual Evoked Potentials P300 Based SpellerThe procedure is aparted from the three folowing steps

78

TRAININGPREPARATION

OF THE SUBJECT

TESTING


Expetiments‐ Demonstration


P300 TRAIN P300 TEST and SSEP TRAIN

SSEP TEST

80

Local OrganizingCommittee

Chairman,T. Bezerianos

SUMMER SCHOOL 2010 FAMILY

THANK YOU

Biosignal Lab Dept of Medical Physics School of Medicine, University of Patras2010

81

Brain Signal and in Brain Machine Interface (BMI) Tassos ...phdsch11/wp-content/... · Brain Signal Processing and Applications in Brain Machine Interface (BMI) Tassos Bezerianos,

Documents