Abhishek_BTP2x

PROJECT No. : 23

DESIGN & DEVELOPMENT OF HUMAN COMPUTER INTERFACE

BASED CONTROLLERS

A PROJECT REPORT

Submitted in partial fulfillment of the requirements for the award of the degree

of

BACHELOR OF TECHNOLOGY

In

ELECTRICAL ENGINEERING

Submitted By: ABHISHEK

Guided By: Dr. VINOD KUMAR

DEPARTMENT OF ELECTRICAL ENGINEERING INDIAN INSTITUTE OF TECHNOLOGY ROORKEE

ROORKEE – 247667 (INDIA) MAY, 2010

ii

DECLARATION BY THE CANDIDATE I hereby certify that the work which is presented in this project titled “Design and Development of Human Computer Interface based Controllers” in partial fulfillment for the award of the degree of Bachelor of Technology (Electrical) submitted to the Department of Electrical Engineering, I.I.T. Roorkee is an authentic record of my own work carried out during the period from August 2010 to May 2010 under the guidance of Dr. Vinod Kumar, Department of Electrical Engineering, I.I.T. Roorkee. The matter embodied in this project report to the best of my knowledge has not submitted for the award of any other degree elsewhere. Date: Abhishek Place: Roorkee (B.Tech IV Year)

This is to certify that the above statement made by the candidate is correct to the best of my knowledge Dr. Vinod Kumar Professor & Head Department of Electrical Engineering IIT Roorkee Roorkee-247667

iii

ACKNOWLEDGEMENT It is my pleasure to take this opportunity to express my heartfelt gratitude to my guide, Dr. Vinod Kumar, Professor, Department of Electrical Engineering, I.I.T. Roorkee, for his valuable guidance and constant encouragement at every stage of this project, sparing his time, sharing knowledge and experience and providing guidance. It would not have been possible for me to complete this project in time without his suggestion, constant inspiration and timely help. I would like to thank Department of Electrical Engineering, I.I.T. Roorkee for resources that helped me to complete our work. I am especially thankful to lab staff for their cooperation. I take this opportunity to thank Mr. Atul Bansal, S.R.F. (Biomedical Laboratory, IIT Roorkee) for his valuable help throughout my project. I express my heartiest gratitude and love to my parents, friends for their patience, encouragement and constant inspiration. Above all I thank God for his blessings.

Abhishek

iv

ABSTRACT This project is for cursor control in a real world environment, where people with motor disabilities face many challenges to operate a personal computer. The project’s main objective is to demonstrate the ability of ocular signals in controlling the cursor of the computer. Ocular signals from FP1 and FP2 positions are used to simulate horizontal cursor movement and cursor clicks. The main challenges of the technology demonstrator are difference in the EEG signal patterns of different people, computational complexity and data acquisition. Signal is acquired from the FP1 and FP2 electrode positions using BIOPAC MP150 and EEG100C systems. Acquisition program was developed in Dev C++ and data analysis program and GUI were developed in MATLAB. Communication between both the compilers was established using text files which act as interrupts for both the programs. A special filter is used to remove the power line frequency and noise. Data analysis is done by finding the differential of the signal using curve fitting or non linear regression technique. Maxima and minima points are calculated from the differential vector and then analyzed in order to reduce the number of maxima and minima so that they represent maximum speed of eyelid closure and opening. The indices of maxima and minima are used to differentiate between an eyeblink or eyeball movement. A GUI was developed which acted as the computer interface with a defined event for getting Visually Evoked Event Related Potentials required by the analysis tools. The event consists of a figure with five rectangular boxes, out of which one is to be selected by moving the cursor over it and then clicking. Cursor can be clicked by the blinking of eyelid while discrete movement of cursor can be done by moving the eyeball in the desired direction. This interface provides a simple method of communicating with the computer for persons with motor disabilities.

1

Table of Contents DECLARATION BY THE CANDIDATE ii ACKOWLEGEMENT iii ABSTRACT iv Table of Contents 1 List of Figures 3 Introduction 4 Background Theory 6 1. Electroencephalography 6 1.1. Introduction 6 1.2. Source of EEG Activity 6 1.3. Methodology 7 1.3.1 10-20 System (EEG) 7 1.3.2 Recording EEG Data 8 1.4. EEG Waves 9 1.5. Limitations 11 1.5.1 Artifacts in EEG 11 2. Non-Linear Regression 13 2.1. Introduction 13 2.2. Mathematical Analysis 13 2.2.1 Non-Linear Least Squares (Theory) 14 2.2.2 Non-Linear Least Squares (Geometric Implementation) 15 2.3. Computational Methods 16 2.3.1 Vanilla Descent Method 16 2.3.2 Gauss Newton Method 16 2.3.3 Levenberg Marquardt Algorithm 17 2.4. An Example 18 3. EEG Recording System 21 3.1. Introduction 21 3.2. MP150 Data Acquisition System 21 3.3. UIM100C 22 3.4. EEG100C 23 3.5. EEG Electrodes 24 4. EEG Signal Acquisition 25 4.1. Placement of Electrodes 25 4.2. Eyeblink 25 4.3. Eyeball Movement 26 4.4. Filtering 28 4.4.1 Low Pass Filter 28 4.4.2 High Pass Filter 29 4.5. Algorithm 30 4.5.1. Main Process 31 4.5.2. Data Analysis 31 Project Work 34 Stage1: Data Acquisition 34

2

1. MP150acq 34 2. MATLAB GUI 35 3. Trial Run 35 Stage2: Eyeblink Detection and Eyeball Tracking 38 1. List of Main Functions 38 2. Calculation of Differential Data-avgresults 38 2.1 Purpose and Prototype 38 2.2 Run and Demonstration 38 3. Locating Maxima-locmaxmin 39 3.1 Purpose and Prototype 39 3.2 Run and Demonstration 39 4. Locate Final Peaks-maxminanalyse 40 4.1 Purpose and Prototype 40 4.2 Run and Demonstration 40 5. Sorting time series-sort1 41 5.1 Purpose and Prototype 41 5.2 Run and Demonstration 41 6. trackingeye 42 6.1 Purpose and Prototype 42 7. runderiv 42 7.1 Purpose and Prototype 42 8. Cursor Initialization and Movement in MATLAB 42 Stage3: Implementation of Graphical User Interface (GUI) 43 1. Graphical User Interface 43 2. Description of Different Controls of GUI 43 Practical Application 48 Human Computer Interf ace 48 1. Introduction 48 2. Main Features 48 3. Block diagram of Human Computer Interface 49 4. Requirements 49 4.1 Hardware Requirements 49 4.2 Software Requirements 49 5. Operating Instructions 49 5.1 Starting the System 49 5.2 Normalization 50 5.3 Running in Real-Time Mode 50 5.4 Running in Offline Mode 50 6. The Experiment 50 6.1 Instructions for Cursor Control 50 Future Work 51 Conclusion 52 Bibliography & References 53

3

List of Figures Figure 1: A Typical BCI System 4 Figure 2: International 10-20 System 8 Figure 3: Delta Wave 9 Figure 4: Theta Wave 9 Figure 5: Alpha Wave 10 Figure 6: Mu Rhythm 10 Figure 7: Beta Wave 10 Figure 8: Gamma Wave 11 Figure 9: Initial Condition 18 Figure 10: Condition After 10 Iterations 19 Figure 11: Final Condition 19 Figure 12: MP150 Data Acquisition System 21 Figure 13: UIM 100C 23 Figure 14: EEG 100C 23 Figure 15: Placement of Electrodes 25 Figure 16: Eyeblink Data 26 Figure 17: Movement towards Left 27 Figure 18: Movement towards Right 27 Figure 19: Sequence of Right and Left Movements 28 Figure 20: Result of Filtering 29 Figure 21: Flowcharts 30 Figure 22: Function Fitting Using Curve Fitting Toolbox 32 Figure 23: Block Diagram of Data Acquisition and Communication 35 Figure 24: Output of .exe at start 36 Figure 25: Output after end of acquisition 36 Figure 26: Differential of Data 39 Figure 27: vmaxmin on differential data 40 Figure 28: Erroneous Data 41 Figure 29: Main GUI Window 43 Figure 30: Exit Dialog Box 44 Figure 31: Start Stop Control 44 Figure 32: Menu Items 45 Figure 33: Mode Dialog Box 45 Figure 34: Eyeblink Normalization 46 Figure 35: Eyeball Normalization 47 Figure 36: Experimental Setup 48

4

Introduction Brain Computer Interface is one of the hopeful technologies for communication between humans and machines. Electroencephalogram-based Brain Computer Interfaces have become a major topic of research in the field of neural engineering, rehabilitation, and brain science. Brain computer interface (BCI) [1][2][3] technology has been used as a means for providing individuals with severe motor disabilities, an ability to communicate via computers and to have command and control over their environment. Figure 1 shows a typical setup for BCI. An important issue in BCI research is cursor control, where the objective is to map the EEG signal to movements of a cursor on a computer screen. It has many potential applications, not only in the control of computer cursors, but also in other devices like robotic arms or neuroprostheses. BCI can be divided into two categories i.e. invasive and noninvasive. Most noninvasive BCI systems use electroencephalogram (EEG) signals; i.e., the electrical brain activity recorded from electrodes placed on the scalp. Measuring the EEG is a simple noninvasive way to monitor electrical brain activity, but it does not provide detailed information on the activity of single neurons (a few µVolts) and noisy environment (especially if recording outside shield rooms). In invasive BCI systems, the activity of single neurons (their spiking rate) is recorded from microelectrodes implanted in the brain. These invasive BCIs face substantial technical difficulties and entail significant clinical risks as they require that recording electrodes be implanted in the cortex and function well for long periods, and they risk infection and other damage to the brain. For humans, therefore, noninvasive BCI systems are applied due to the clinical risks and ethics.

Fig. 1 A Typical BCI Setup [5]

Level of motor disability in patients largely depends upon the clinical causes of impairment. In the most affected patients, the ‘locked-in’ group, there is no residual

5

motor ability. As there are no other means available to the patient to communicate with outside world, both invasive and non-invasive BCI can be useful. The relative number of these cases with no residual disability is rare: much more common are cases in which arm function is lost, but facial and eye muscle control remain intact. In such cases, non-invasive means of restoration of reaching and grasping promises to offer significant benefits. Incorporation of new features in BCI is being carried out by researchers all over the world. This project uses the eyeblink detection and eyeball tracking methodologies to establish cursor control in 1 dimension. Eyeblinks and eyeball movement generates a potential in the FP1 and FP2 electrode positions of EEG. This visually evoked potential can be translated into cursor movement and clicks. The human computer interface that has been developed in this project is a non-invasive type of BCI and instead of relying on the thinking process in the brain; it uses ocular muscle movements to guide a cursor.

6

Background Theory 1. Electroencephalogram

1.1 Introduction The electroencephalogram (EEG) [9] is a record of the oscillations of brain electric potentials recorded from perhaps 20 to 256 electrodes attached to the human scalp. The recorded signals are transmitted to an EEG system composed of amplifiers, filters, and paper chart or computer monitor. The first human EEG recordings were accomplished by the German psychiatrist Hans Berger in 1924. Berger’s subjects (typically his children) revealed robust changes when the eyes were closed and when the subjects performed mental arithmetic. 1.2 Source of EEG Activity The electrical activity of the brain can be described in spatial scales from the currents within a single dendritic spine to the relatively gross potentials that the EEG records from the scalp. Neurons, or nerve cells, are electrically active cells which are primarily responsible for carrying out the brain's functions. Neurons create action potentials, which are discrete electrical signals that travel down axons and cause the release of chemical neurotransmitters at the synapse, which is an area of near contact between two neurons. This neurotransmitter then activates a receptor in the dendrite or body of the neuron that is on the other side of the synapse, the post-synaptic neuron. The neurotransmitter, when combined with the receptor, typically causes an electrical current within the dendrite or body of the post-synaptic neuron. Thousands of post-synaptic currents from a single neuron's dendrites and body then sum up to cause the neuron to generate an action potential. This neuron then synapses on other neurons, and so on.

EEG reflects correlated synaptic activity caused by post-synaptic potentials of cortical neurons. The ionic currents involved in the generation of fast action potentials may not contribute greatly to the averaged field potentials representing the EEG. More specifically, the scalp electrical potentials that produce EEG are generally thought to be caused by the extracellular ionic currents caused by dendritic electrical activity, whereas the fields producing magneto-encephalographic signal are associated with intracellular ionic currents.

The electric potentials generated by single neurons are far too small to be picked by EEG. EEG activity therefore always reflects the summation of the synchronous activity of thousands or millions of neurons that have similar spatial orientation, radial to the scalp. Currents that are tangential to the scalp are not picked up by the EEG. The EEG therefore benefits from the parallel, radial arrangement of apical dendrites in the cortex. Because voltage fields fall off with the fourth power of the radius, activity from deep sources is more difficult to detect than currents near the skull.

Scalp EEG activity shows oscillations at a variety of frequencies. Several of these oscillations have characteristic frequency ranges, spatial distributions and are associated

7

with different states of brain functioning (e.g., waking and the various sleep stages). These oscillations represent synchronized activity over a network of neurons. The neuronal networks underlying some of these oscillations are understood, while many others are not. Research that measures both EEG and neuron spiking finds the relationship between the two is complex with the power of surface EEG only in two bands that of gamma and delta relating to neuron spike activity.

1.3 Methodology In conventional scalp EEG, the recording is obtained by placing electrodes on the scalp with a conductive gel or paste, usually after preparing the scalp area by light abrasion to reduce impedance due to dead skin cells. Many systems typically use electrodes, each of which is attached to an individual wire. Some systems use caps or nets into which electrodes are embedded; this is particularly common when high-density arrays of electrodes are needed.

Electrode locations and names are specified by the International 10–20 system for most clinical and research applications (except when high-density arrays are used). This system ensures that the naming of electrodes is consistent across laboratories.

1.3.1 10-20 system (EEG) The "10-20" system or "International 10-20" system is an internationally recognized method to describe and apply the location of scalp electrodes in the context of an EEG test or experiment. This method was developed to ensure standardized reproducibility so that subjects’ studies could be compared over time and subjects could be compared to each other. This system is based on the relationship between the location of an electrode and the underlying area of cerebral cortex. The "10" and "20" refer to the fact that the actual distances between adjacent electrodes are either 10% or 20% of the total front-back or right-left distance of the skull.

Each site has a letter to identify the lobe and a number to identify the hemisphere location. The letters F, T, C, P and O stand for Frontal, Temporal, Central, Parietal and Occipital respectively. Note that there exists no central lobe; the "C" letter is only used for identification purposes only. A "z" (zero) refers to an electrode placed on the midline. Even numbers (2, 4, 6, 8) refer to electrode positions on the right hemisphere, whereas odd numbers (1, 3, 5, 7) refer to those on the left hemisphere.

Two anatomical landmarks are used for the essential positioning of the EEG electrodes: first, the nasion which is the point between the forehead and the nose; second, the inion which is the lowest point of the skull from the back of the head and is normally indicated by a prominent bump.

When recording a more detailed EEG with more electrodes, extra electrodes are added utilizing the spaces in-between the existing 10-20 system. This new electrode-naming-system is more complicated giving rise to the Modified Combinatorial Nomenclature (MCN). This MCN system uses 1, 3, 5, 7, 9 for the left hemisphere which represents 10%, 20%, 30%, 40%, and 50% of the inion-to-nasion distance respectively. The introduction

8

of extra letters allows the naming of extra electrode sites. Note that these new letters do not necessarily refer to an area on the underlying cerebral cortex.

Fig. 2 International 10-20 System (Courtesy: http://frontalcortex.com/)

1.3.2 Recording EEG Data In most clinical applications, 19 recording electrodes (plus ground and system reference) are used. A smaller number of electrodes are typically used when recording EEG from neonates. Additional electrodes can be added to the standard set-up when a clinical or research application demands increased spatial resolution for a particular area of the brain. High-density arrays (typically via cap or net) can contain up to 256 electrodes more-or-less evenly spaced around the scalp.

Each electrode is connected to one input of a differential amplifier (one amplifier per pair of electrodes); a common system reference electrode is connected to the other input of each differential amplifier. These amplifiers amplify the voltage between the active electrode and the reference (typically 1,000–100,000 times, or 60–100 dB of voltage gain). In analog EEG, the signal is then filtered, and the EEG signal is output as the deflection of pens as paper passes underneath. Most EEG systems these days, however, are digital, and the amplified signal is digitized via an analog-to-digital converter, after being passed through an anti-aliasing filter. Analog-to-digital sampling typically occurs at 256–512 Hz in clinical scalp EEG; sampling rates of up to 20 kHz are used in some research applications.

During the recording, a series of activation procedures may be used. These procedures may induce normal or abnormal EEG activity that might not otherwise be seen. These procedures include hyperventilation, photic stimulation (with a strobe light), eye closure, mental activity, sleep and sleep deprivation. During epilepsy monitoring, a patient's typical seizure medications may be withdrawn.

The digital EEG signal is stored electronically and can be filtered for display. Typical settings for the high-pass filterrespectively. The high-electrogalvanic signals and movement artifact, whereas the lowhigh-frequency artifacts, such as typically used to remove artifact caused by electrical power lines. A typical adult human EEG signal is about 10µV to 100 µV in amplitude when measured from the scalp and is about 10–20 mV when measured from subdural electrodes.

Since an EEG voltage signal represents a difference between the voltages at two electrodes, the display of the EEG for the reading encephalographer may be set up in one of several ways. The representation of the EEG channels is referred to as a

1.4 EEG Waves [9] Following is the description of different types of waves in EEG signal:

• Delta is the frequency range up to 4 Hz. It tends to be the highest in amplitude and the slowest waves. It is seen normally in adults in normally in babies. It may occur distribution with diffuse lesions, metabolic encephalopathy hydrocephalus or deep midline lesions.

(Courtesy:

• Theta is the frequency range from 4 Hz to 7 Hz. Theta is seen normally in young children. It may be seen in drowsiness or arousal in older children and adults; it can also be seen in It can be seen as a focal disturbance in focal subcortical lesions

(Courtesy:

• Alpha is the frequency range from 8 rhythmic EEG activity he saw, the "alpha wave." This is activity in the 8

9

The digital EEG signal is stored electronically and can be filtered for display. Typical pass filter and a low-pass filter are 0.5-1 Hz

-pass filter typically filters out slow artifact, such as signals and movement artifact, whereas the low-pass

frequency artifacts, such as electromyographic signals. An additional typically used to remove artifact caused by electrical power lines. A typical adult human EEG signal is about 10µV to 100 µV in amplitude when measured from the scalp and is

20 mV when measured from subdural electrodes.

EEG voltage signal represents a difference between the voltages at two electrodes, the display of the EEG for the reading encephalographer may be set up in one of several ways. The representation of the EEG channels is referred to as a

Following is the description of different types of waves in EEG signal:

is the frequency range up to 4 Hz. It tends to be the highest in amplitude and the slowest waves. It is seen normally in adults in slow wave sleepnormally in babies. It may occur focally with subcortical lesions and in general distribution with diffuse lesions, metabolic encephalopathy hydrocephalus or deep

Fig. 3 Delta Wave (Courtesy: http://en.wikipedia.org/wiki/Electroencephalography

is the frequency range from 4 Hz to 7 Hz. Theta is seen normally in young children. It may be seen in drowsiness or arousal in older children and adults; it can also be seen in meditation. Excess theta for age represents abnormal activity. It can be seen as a focal disturbance in focal subcortical lesions.

Fig. 4 Theta Wave (Courtesy: http://en.wikipedia.org/wiki/Electroencephalography

is the frequency range from 8 Hz to 12 Hz. Hans Bergerrhythmic EEG activity he saw, the "alpha wave." This is activity in the 8

The digital EEG signal is stored electronically and can be filtered for display. Typical Hz and 35–70 Hz,

pass filter typically filters out slow artifact, such as pass filter filters out

signals. An additional notch filter is typically used to remove artifact caused by electrical power lines. A typical adult human EEG signal is about 10µV to 100 µV in amplitude when measured from the scalp and is

EEG voltage signal represents a difference between the voltages at two electrodes, the display of the EEG for the reading encephalographer may be set up in one of several ways. The representation of the EEG channels is referred to as a montage.

is the frequency range up to 4 Hz. It tends to be the highest in amplitude and slow wave sleep. It is also seen

focally with subcortical lesions and in general distribution with diffuse lesions, metabolic encephalopathy hydrocephalus or deep

http://en.wikipedia.org/wiki/Electroencephalography)

is the frequency range from 4 Hz to 7 Hz. Theta is seen normally in young children. It may be seen in drowsiness or arousal in older children and adults; it

. Excess theta for age represents abnormal activity. .

wikipedia.org/wiki/Electroencephalography)

Hans Berger named the first rhythmic EEG activity he saw, the "alpha wave." This is activity in the 8–12 Hz

range seen in the posterior regions of the head on both sides, being higher inamplitude on the dominant side. It is brought out by closing the eyes and by relaxation.

(Courtesy:

• Mu rhythm is alphacharacteristically attenuates with movement of the imagery of movement of the contralateral arm).

(Courtesy:

• Beta is the frequency range from 12 Hz to about 30 Hz. It is seen usually on both sides in symmetrical distribution and is most evident frontally. Beta activity is closely linked to motor behavior and is generally attenuated during active movements. Low amplitassociated with active, busy or anxious thinking and active concentration.

(Courtesy: http://en.wikipedia.org/wiki/Electroencephalography)

• Gamma is the frequency range approximately 30thought to represent binding of different populations of neurons together into a network for the purpose of carrying out a certain cognitive or motor fun

10

range seen in the posterior regions of the head on both sides, being higher inamplitude on the dominant side. It is brought out by closing the eyes and by

Fig. 5 Alpha Wave

(Courtesy: http://en.wikipedia.org/wiki/Electroencephalography

s alpha-range activity that is seen over the sensorimotor cortexcharacteristically attenuates with movement of the contralateralimagery of movement of the contralateral arm).

Fig. 6 Mu Rhythm (Courtesy: http://en.wikipedia.org/wiki/Electroencephalography

is the frequency range from 12 Hz to about 30 Hz. It is seen usually on both sides in symmetrical distribution and is most evident frontally. Beta activity is closely linked to motor behavior and is generally attenuated during active movements. Low amplitude beta with multiple and varying frequencies is often associated with active, busy or anxious thinking and active concentration.

Fig. 7 Beta Wave (Courtesy: http://en.wikipedia.org/wiki/Electroencephalography)

is the frequency range approximately 30–100 Hz. Gamma rhythms are thought to represent binding of different populations of neurons together into a network for the purpose of carrying out a certain cognitive or motor fun

range seen in the posterior regions of the head on both sides, being higher in amplitude on the dominant side. It is brought out by closing the eyes and by

http://en.wikipedia.org/wiki/Electroencephalography)

sensorimotor cortex. It contralateral arm (or mental

http://en.wikipedia.org/wiki/Electroencephalography)

is the frequency range from 12 Hz to about 30 Hz. It is seen usually on both sides in symmetrical distribution and is most evident frontally. Beta activity is closely linked to motor behavior and is generally attenuated during active

ude beta with multiple and varying frequencies is often associated with active, busy or anxious thinking and active concentration.

(Courtesy: http://en.wikipedia.org/wiki/Electroencephalography)

100 Hz. Gamma rhythms are thought to represent binding of different populations of neurons together into a network for the purpose of carrying out a certain cognitive or motor function.

(Courtesy:

Type Frequency (Hz)

Delta upto 4

Theta 4-7 Alpha 8-12

Beta 12-30

Gamma 30-100+ 1.5 Limitations EEG has several limitations. Most important is its poor spatial resolution. EEG is most sensitive to a particular set of postsuperficial layers of the cortex, on the crests of to the skull. Dendrites which are deeper in the cortex, inside structures (such as the hippocampusskull, have far less contribution to the EEG signal.

The meninges, cerebrospinal fluidintracranial source. It is mathematically current source for a given EEG signal, as some currents produce potentials that cancel each other out. This is referred to as the done to produce remarkably good estimates of, at least, a localized represents the recorded currents.

1.5.1 Artifacts in EEG Eye-induced artifacts are caused by the potential difference between the retina, which is quite large compared to cerebral potentials. When the eye is completely still, this does not affect EEG. But there are nearly always small or large reflexive eye movements, which generates a potential which is picked up in the frontopolar and frleads. Involuntary eye movements, known as which also generate electromyographicalso generates electromyographic

11

Fig. 8 Gamma Wave (Courtesy: http://en.wikipedia.org/wiki/Electroencephalography

Table 1 Description of EEG Waves

Location

Frontally in adults, posterior in children; high amplitude waves. Found in locations not related to task at hand.Posterior regions of head, both sides, higher in amplitude on dominant side. Central sites (c3-c4) at rest.Both sides, symmetrical distribution, most evident frontally; low amplitude waves. Somatosensory cortex

EEG has several limitations. Most important is its poor spatial resolution. EEG is most sensitive to a particular set of post-synaptic potentials: those which are generated in superficial layers of the cortex, on the crests of gyri directly abutting the skull and radial to the skull. Dendrites which are deeper in the cortex, inside sulci, in midline or deep

hippocampus), or producing currents which are tangential to the skull, have far less contribution to the EEG signal.

cerebrospinal fluid and skull "smear" the EEG signal, obscuring its intracranial source. It is mathematically impossible to reconstruct a unique intracranial current source for a given EEG signal, as some currents produce potentials that cancel each other out. This is referred to as the inverse problem. However, much work has been done to produce remarkably good estimates of, at least, a localized electric dipolerepresents the recorded currents.

induced artifacts are caused by the potential difference between the , which is quite large compared to cerebral potentials. When the eye is completely

still, this does not affect EEG. But there are nearly always small or large reflexive eye movements, which generates a potential which is picked up in the frontopolar and frleads. Involuntary eye movements, known as saccades, are caused by ocular muscles,

electromyographic potentials. Purposeful or reflexive eye blinking electromyographic potentials, but more importantly there is reflexive

http://en.wikipedia.org/wiki/Electroencephalography)

Frontally in adults, posterior in children; high amplitude

locations not related to task at hand. Posterior regions of head, both sides, higher in amplitude

c4) at rest. Both sides, symmetrical distribution, most evident

EEG has several limitations. Most important is its poor spatial resolution. EEG is most synaptic potentials: those which are generated in

directly abutting the skull and radial , in midline or deep

), or producing currents which are tangential to the

and skull "smear" the EEG signal, obscuring its impossible to reconstruct a unique intracranial

current source for a given EEG signal, as some currents produce potentials that cancel . However, much work has been

electric dipole that

induced artifacts are caused by the potential difference between the cornea and , which is quite large compared to cerebral potentials. When the eye is completely

still, this does not affect EEG. But there are nearly always small or large reflexive eye movements, which generates a potential which is picked up in the frontopolar and frontal

, are caused by ocular muscles, potentials. Purposeful or reflexive eye blinking

potentials, but more importantly there is reflexive

12

movement of the eyeball during blinking which gives a characteristic artifactual appearance of the EEG.

Eyelid fluttering artifacts of a characteristic type were previously called Kappa rhythm (or Kappa waves). It is usually seen in the prefrontal leads, that is, just over the eyes. Sometimes they are seen with mental activity. They are usually in the Theta (4–7 Hz) or Alpha (8–13 Hz) range. They were named because they were believed to originate from the brain. Later study revealed they were generated by rapid fluttering of the eyelids, sometimes so minute that it was difficult to see. They are in fact noise in the EEG reading, and should not technically be called a rhythm or wave. Therefore, current usage in electroencephalography refers to the phenomenon as an eyelid fluttering artifact, rather than a Kappa rhythm (or wave).

13

Background Theory 2. Non-linear Regression

2.1 Introduction Regression analysis is a method used to find the ‘best fit’ line or curve for the series of data points. It can be used for prediction (including forecasting of time-series data), inference, and hypothesis testing, and modeling of causal relationships. Nonlinear regression [4][5] aims to describe the relationship between a response variable and one or more explanatory variables in a non-linear fashion. Nonlinear regression is more general than linear regression and can fit data to any equation that defines Y as a function of X and one or more parameters. It finds the values of those parameters that generate the curve that comes closest to the data.

Table 2 Comparison between non-linear and linear regression

Non Linear Regression Linear Regression Dependent variable, yi is not a linear combination of the parameters

Dependent variable, yi is a linear combination of the parameters

It is not possible to directly derive an equation to compute the best-fit values from the data, hence the sum of squares function must be minimized by an iterative procedure.

Minimization of squared error results in a set of simultaneous linear equations in the parameters, which are solved to yield the parameter estimators

Non-convergence may be a problem.

It is globally concave and hence convergent

Uses matrix algebra as basis Uses simple algebra as basis

2.2 Mathematical Analysis Nonlinear regression model can be given by a general equation:

�� � ��β, xi1� � εi

, where the function f() relating the response to the predictors is not necessarily linear. β is the vector of parameters. The likelihood for this model is given by:

��, ��� � 1

�2��� �� exp ��

∑ ��� � ��β, xi1��

�����

2��

14

is minimized.

Differentiating S(β) with respect to parameters we get:

�����

��� �2 �� �� � ��β, xi1�� d��β, xi1���

Setting the partial derivatives to zero produces estimating equations for the parameters. These equations are generally non-linear and require numerical optimization methods for their solution. Coefficient variances may be estimated from the linearised version of the model. Let,

!�" � d��β, xi1��� , and F={F ij}. Then the estimated asymptotic covariance matrix of the regression parameters is #��� � $��!�!�%�

,where s2 is the estimated error variance. 2.2.1 Non Linear Least Squares (Theory) It is desired to find the vector ß of parameters such that the curve fits best the given data in the least squares sense, that is, the sum of squares

� � � &��'

���

,where residual &� � �� � ��β, xi � for i = 1.....m. For minimum error gradients should be zero. ����" � 2 � &� �&���"�

� 0 �) � 1, … . . , ,� In a non-linear system, the derivatives are functions of both the independent variable and the parameters, so these gradient equations do not have a closed solution. Instead, initial values must be chosen for the parameters. Then, the parameters are refined iteratively, that is, the values are obtained by successive approximation, �" ~ �"./� � �". � ∆�"

15

At each iteration, the model is linearized by approximation to a first-order Taylor series expansion about ß as given by

��1� , ��~��1� , �.� � � d�213, βk5��"

6βj � βk8 ~ ��1� , �.� � � 9�"

"∆�"

The Jacobian, J, is a function of constants, the independent variable and the parameters, so it changes from one iteration to the next. Thus, in terms of the linearized model, �&���" � �9�"

&� � ∆�� � � 9�"�

"��∆�"; ∆�� � �� � �213, β

k5

On substitution, gradient equations become:

�2 � 9�"�

"���∆�� � � 9�;∆�;� � 0

�

;��

which on rearrangement gives the normal simultaneous equations �9<9�∆�" � 9<∆� When the observations are not equally reliable, a weighted sum of squares may be minimized,

� � � =��&���

���

Each element of the diagonal weight matrix, W should, ideally, be equal to the reciprocal of the variance of the measurement. The normal equations are then �9<9�∆� � 9<=∆� 2.2.2 Non Linear Least Squares (Geometric Interpolation) In Nonlinear Least Squares, the objective function is quadratic with respect to the parameters only in a region close to its minimum value, where the truncated Taylor series is a good approximation to the model.

� ~ � =����� � � 9�""

�

����)�

�

16

The more the parameter values differ from their optimal values, the more the contours deviate from elliptical shape. A consequence of this is that initial parameter estimates should be as close as practicable to their optimal values. It also explains how divergence can come about as the Gauss-Newton algorithm is convergent only when the objective function is approximately quadratic in the parameters. 2.3. Computational Methods Next task in non-linear regression is to arrive at the solution of the least square equations that we have formed. This task requires iterative techniques so that the squared error is least. There are many ways of solving these equations. 2.3.1 Vanilla Descent Method

• Simplest, most intuitive technique • Parameter update is performed by adding the negative of the scaled gradient at

each step 1�/� � 1� � >∆�

• Convergence problems like different curvature at different points and update rule tends to rattle out minima sometimes.

• We should take large steps where gradient descent is gentle and small steps where gradient descent is more but this method does the opposite.

2.3.2 Gauss Newton Method

• Quadratic convergence method, and utilizes curvature information • ß is updated as

�;/� � �; � H�1g where H = Hessian Matrix given by

A". � 2 �� �&���"�&���.

'

���� &� ��&���"��.

g= gradient vector of S(ß)

B" � 2 � &� �&���"'

���

• Ignore second order derivative terms. This gives

A". ~ 2 � 9�"9�.'

���

17

,where 9�" � �&3��)

• This leads to Gauss-Newton method given by: �;/� � �; � ��; �� � �� 9C<9C�%�9C<&

Convergence of the Gauss–Newton method is not guaranteed in all instances. The approximation:

D&� �2&3��)��ED ≪ D�&3��)�&3��ED

needs to hold to be able to ignore the second-order derivative terms may be valid in the following two cases, for which convergence is to be expected:

• The function values ri are small in magnitude, at least around the minimum.

• The functions are only "mildly" non-linear, so that GHCIGJKGJL is relatively small in

magnitude. 2.3.3 Levenberg Marquardt Method [10] This algorithm is by far the most widely used algorithm as it can change its characteristics according to the type of error. For cases, where vanilla descent is efficient, Levenberg-Marquardt (LM) [10] algorithm approaches vanilla descent and for cases, where gauss-newton method is fast and efficient, LM algorithm approaches the gauss-newton method. LM algorithm was proposed in two steps. Levenberg made an algorithm which was modified by Marquardt and this modified algorithm is known as LM algorithm. Contributions by Levenberg

• Noticed that GNA and gradient descent are complimentary to each other. • Modified the update rule as:

�9<9 � >M�∆ � 9<�� � �����

• Algorithm to use the update rule:

i. Do an update as directed by the rule above. ii. Evaluate the error at the new parameter vector. iii. If the error has increased as a result the update, then retract the step and

increase λ by a factor of 10 or so. Then go to (1) and try an update again. iv. If the error has decreased as a result of the update, then accept the step (i.e.

keep the weights at their new values) and decrease l by a factor of 10 or so.

18

There were some limitations in the algorithm proposed by Levenberg. If the value of λ is large then the Hessian matrix is not used at all. This results in extra calculations for Hessian matrix which becomes irrelevant in this case. It was not able to resolve the classical error valley problem i.e. it didn’t result in larger movement at places where error gradient is very small. Contributions by Marquardt

• Provided insight to solve the classical 'valley' problem • Modified the update rule as

29<9 � >�3NB�9<9�5∆� 9<�� � �����

• Since the Hessian is proportional to the curvature of f, this implies a large step in the direction with low curvature (i.e., an almost flat terrain) and a small step in the direction with high curvature (i.e. a steep incline).

Hence, the limitations of Levenberg algorithm were removed by Marquardt algorithm and this resulted in the formation of LM algorithm which is considered to be the most robust method for non-linear regression. 2.4. An Example [9] Let us study an example that has been taken from Wikipedia. Original data is provided by the user and now we need to form a model which when used can regenerate the data within the permissible error limits. Let the model be y=acos(bx) + bsin(ax) Initial values of parameters: a=100, b=102

Fig. 9 Initial Condition (Courtesy: http://en.wikipedia.org/wiki/Electroencephalography)

Green line represents original data and blue line represents the fitted data. With initial values of parameters the plot looks like one shown in Fig. 9. Our aim is to change the

19

value of parameters iteratively to arrive at a point where the squared error is within the tolerance values. At this point the fitted curve would start touching the original data and we can visually observe that both the curves are almost identical to each other. There are various parameters like confidence intervals, chi squared error etc. which can help us analyze the goodness of fit and careful study of these parameters help us in choosing the right model for our data.

Fig. 10 Condition after 10 iterations (Courtesy: http://en.wikipedia.org/wiki/Electroencephalography)

Fig. 11 Final Condition (Courtesy: http://en.wikipedia.org/wiki/Electroencephalography)

20

Following are the steps involved in performing non-linear regression for a given data:

• Clarify your goal, is nonlinear regression appropriate • Prepare the data • Choose a model • Decide which model parameters to fit and which to constrain • Choose a weighing scheme (if any) • Choose initial values • Perform curve-fit and analyze the results.

21

Background Theory 3. EEG Recording System

3.1 Introduction BIOPAC EEG System [11] is a system that consists of three different components which work in tandem to produce the required EEG signal from the user. This system can be used to acquire, communicate and analyze the data. Three different components are:

• MP150 • UIM100C • EEG100C

3.2 MP150 Data Acquisition System The MP150 Data Acquisition System is a device that acquires data and communicates with the computer by sending the data from the amplifier to the computer for analysis purposes. AcqKnowledge is an interactive, intuitive program that lets the user instantly view, measure analyze and transform data from the MP150 System. Following are the salient features of this device:

• Ethernet-ready data acquisition and analysis • Record multiple channels with differing sample rates • Record at speeds up to 400 kHz

Applications of this device include amplifiers and interfaces, biomechanics, ECG, EEG, EMG, EOG, exercise physiology, sleep studies etc. MP150 System components include:

• Ethernet-ready data acquisition unit • Universal Interface Module • Network interface card • Ethernet crossover cable • 12V Power transformer with cord

Fig 12 MP150 Data Acquisition System (Courtesy: http://www.biopac.com)

22

Table 3 Specifications of MP150 System

3.3 UIM100C The UIM100C (Universal Interface Module) is used to connect amplifier modules and signal cables to an MP system. Sixteen analog input channels plus 2 analog output channels are provided via standard 3.5 mm phone jacks on the front panel. The back panel provides access to 16 digital I/O lines and an external trigger input via screw terminals. Amplifier modules snap onto the side of the UIM100C for signal routing to the MP data acquisition unit. The output of any amplifier module snapped to the UIM100C can be monitored during acquisition via the corresponding 3.5 mm analog channel phone jack. The UIM100C also provides a direct link to the analog and digital I/O lines of the MP device when collecting or sending data to external equipment.

Analog Input

Number of Channels 16 Absolute Maximum Input ±15 V Operational Input Voltage ±10 V A/D Resolution 16 Bits Accuracy (% of FSR) ±0.003 Input impedance 1.0 MOhm

Analog Output

Number of Channels 2 Max number of waveforms output with acquisition

2

Output Voltage Range ±10 V D/A Resolution 16 Bits Accuracy (% of FSR) ±0.003 Output Drive Current ±5 mA Output Impedance 100 Ohm

Digital I/O

Number of Channels 16 Voltage Levels TTL, CMOS Output Drive Current ±20 mA (max) External Trigger Input TTL, CMOS

Time Base Min Sample Rate 2 samples/hour Trigger Options Internal, External or Signal

Level

Power Amplifier Module Isolation Provided by the MP unit CE Marking EC Low Voltage and EMC

Directives Leakage current <8 µA (Normal), <400 µA

(Single Fault) Fuse 2 A (fast blow)

Serial Interface Ethernet UDP (10M bits/sec)

Miscellaneous

Power Requirements 12VDC @ 2 amp Transmission Type Ethernet Analog Output Buffer 512 Ks per analog output

channel Internal Buffer 6 M samples

23

Fig. 13 UIM 100C (Courtesy: http://www.biopac.com)

Table 4 Specifications of UIM100C Module

Analog I/O Access 16 channels (front panel) - 3.5 mm phone jacks D/A Outputs 2 channels (front panel) - 3.5 mm phone jacks

Digital I/O Access 16 channels (back panel) - screw terminals Ext. Trigger Access 1 channel (back panel) - screw terminal

Isolated Power Access ±12 V, ±5 V @ 100 ma (back panel) - screw terminals

3.4 EEG100C The EEG100C Electroencephalogram (EEG) amplifier amplifies bioelectric potentials associated with neuronal activity of the brain and can be used to perform unipolar or bipolar EEG measurements. The amplifier output can be switched between normal EEG output and Alpha wave detection. The Alpha detection mode outputs a smoothed wave with a peak indicating maximal alpha activity (signal energy in the 8-13 Hz frequency range).

Fig. 14 EEG 100C Amplifier (Courtesy: http://www.biopac.com)

Table 5 EEG 100C Amplifier

Gain 5000, 10000, 20000, 50000 Output selection Normal, Alpha wave indicator Low Pass Filter 35 Hz, 100 Hz High Pass Filter 0.1 Hz, 1.0 Hz

Notch Interference Filter 50 dB rejection @ 50/60 Hz Zin 2 Mohm (Differential)

1000 Mohm (Common mode) CMRR 110 dB min (50/60 Hz)

24

3.5 EEG Electrodes Disc type Silver Silver Chloride Electrodes are used for taking EEG data. These electrodes are disposable and come with their own conducting paste.The corresponding equations for such an electrode can be presented as follows:

Ag+ + 1e− ↔ Ag0(s) Ag+ + Cl− ↔ AgCl(s)

or an overall reaction can be written:

Ag0(s) + Cl− ↔ AgCl(s) + e-

Common Mode Input Voltage Range

±10 V (referenced to amplifier ground) ±1500 VDC (referenced to mains ground)

Output Range ±10 V (analog)

Input Voltage Range

Gain 5000 : Vin ±2 (mV) Gain 10000: Vin ±1 (mV) Gain 20000: Vin ±0.5 (mV) Gain 50000: Vin ±0.2 (mV)

4. EEG Signal Acquisition and Event Detection Before selecting the algorithm, it is imperative to know the characteristics of EEG data for the visually evoked potentials. Ocular data or any other EEG data depends largely on the subject and placement of electrodes. This section deals with analysis of 4.1 Placement of ElectrodesTo get desired ocular data, placement of electrodes is very important. Data has been acquired using disposable connections. Disposable electrodes provide with taccording to the need of the experiment. Figure below depicts the placement of electrodes on a subject.

Connections of electrodes with EEG Machine

• Ground1, Ground2• FP1, FP2---� Vin+• Reference1, Reference2

4.2 Eyeblink During blinking of an eyeasymmetric data with the values moving up at some slope and then again settling to base level with some different slopes. During eyeblinks, time index for attaining peaks in both FP1 and FP2 is almost the same.

25

Background TheoryEEG Signal Acquisition and Event Detection

Before selecting the algorithm, it is imperative to know the characteristics of EEG data for the visually evoked potentials. Ocular data or any other EEG data depends largely on the subject and placement of electrodes. This section deals with analysis of

1 Placement of Electrodes To get desired ocular data, placement of electrodes is very important. Data has been acquired using disposable silver-silver chloride electrodes and plug in type of connections. Disposable electrodes provide with the flexibility of adjusting electrodes according to the need of the experiment. Figure below depicts the placement of electrodes

Fig. 15 Placement of Electrodes

Connections of electrodes with EEG Machine Ground2---�Ground

Vin+ Reference1, Reference2---� Vin-

blinking of an eye, eye muscles contract and expand. This results in an asymmetric data with the values moving up at some slope and then again settling to base level with some different slopes. During eyeblinks, time index for attaining peaks in both

e same.

ackground Theory EEG Signal Acquisition and Event Detection

Before selecting the algorithm, it is imperative to know the characteristics of EEG data for the visually evoked potentials. Ocular data or any other EEG data depends largely on the subject and placement of electrodes. This section deals with analysis of ocular data.

To get desired ocular data, placement of electrodes is very important. Data has been electrodes and plug in type of

he flexibility of adjusting electrodes according to the need of the experiment. Figure below depicts the placement of electrodes

eye muscles contract and expand. This results in an asymmetric data with the values moving up at some slope and then again settling to base level with some different slopes. During eyeblinks, time index for attaining peaks in both

26

Fig. 16 Eyeblink Data

4.3 Eyeball Movement During eyeball movement, same contraction and expansion of muscles takes place but there is a difference between the times of attainment of peaks. One of the peaks occurs before the other according to the movement. Both types of data can be analyzed by finding the time index when slope becomes highest and then using this index with certain threshold to check whether a blink has occurred or eyeballs have moved in some direction. One of the major consideration in the experiment was that of filtering of data, as data was corrupted with significant amount of noise because of the use of make shift electrodes. Presence of power line frequency and noise can result in deviation of results from the desired targets. Special care should also be taken so as to use a type of filter which doesn’t provide a large amount of delay element and is also quite computationally light since real time analysis of data needs to take place. During real time analysis and operation, aim is to provide minimum computational delay during analysis so that data loss is very low and data correspondence with the event is quite high.

27

Fig. 17 Movement towards Left

Fig. 18 Movement towards Right

28

Fig. 19 Sequence of Right and Left Movements

4.4 Filtering Pan and Tompkins[7][8] in their research paper on “A Real Time QRS Peak Detection Algorithm” have presented a simple and computationally less intensive filter which can be used for filtering the data. It is composed of a mixture of two cascaded low pass and high pass filters and effectively reduces the power line frequency components as well as the noise components. 4.4.1 Low Pass Filter The transfer function of second order low pass filter is

A�O� � �1 � O%P���1 � O%���

The amplitude response is given by

|A�RS�| � $3,�3RS$3,�RS/2

,where T is the time period. The difference equation for this LPF is given by ��,S� � 2��,S � S� � ��,S � 2S� � 1�,S� � 21�,S � 6S� � 1�,S � 12S� ,where cutoff frequency is about 11 Hz and gain is 36 with a delay of 6 samples.

29

4.4.2 High Pass Filter The design of high pass filter is based on subtracting the output of a first order low pass filter from an all pass filter. The transfer function of such a high pass filter is given by

A�O� � �1 � 32O%�P � O%W�1 � O%�

The amplitude response is given by

|A�RS�| � �256 � $3,�16RS���cos �RS2 �

,where T is the time period. The difference equation for this LPF is given by ��,S� � �321�,S � 16S� � ���,S � S� � 1�,S� � 1�,S � 32S�� ,where low cutoff frequency is about 3 Hz and gain is 32 with a delay of 16 samples.

Fig. 20 Result of Filtering

30

4.5 Algorithm Flowchart below shows the overall algorithm for this project:

a) b) Fig. 21 Flow Charts a) Main Process (MATLAB) b) Eyeblink Detection and Eyeball Tracking

31

4.5.1 Main Process Data acquisition is started in C++ as soon as the start push button is pressed in the GUI. MATLAB waits for the first epoch of data and when it receives its first epoch it starts the analysis process. After the analysis process is over cursor is initialized and according to the results of the analysis, cursor performs blink and movement operations. 4.5.2 Data Analysis Figure 21 b) shows the flowchart for data analysis. It is a basic skeleton which consists of all the major operations that need to be performed for detecting eyeblinks and eyeball movement. Following are the major tasks that need to be performed:

• Function Fitting: Function fitting is performed using sum of two Gaussians model:

f(x) = a1*exp(-((x-b1)/c1)^2) + a2*exp(-((x-b2)/c2)^2)) Before choosing the model, we should keep in mind that the model should be such that information is not lost and the characteristic of data is also maintained. Figure 22 shows the function fitting on eyeblink data from FP1.

• Calculating first derivative (point-wise): A point-wise first derivative needs to be calculated to be able to locate the local maxima and minima which prevents the usage of offset and onset calculations. This method is quite fast as after performing function fitting derivative can be easily calculated. Point-wise derivative is also called running derivative of a time-series and is one of the very important analysis tools in ECG and EEG data analysis.

• Locating the Peaks: Peaks on the derivative data can be located by a simple analysis method. First all the differential values above a calculated threshold are arranged in a matrix. Then all the negative values are arranged and all the positive values are sorted according to their incidence on time-axis. Now an experimentally calculated window of 200ms is used to group all the maxima and minima values which lie in this window. Maximum and minimum values are then taken out of this group, which give the peaks in the ocular data. Here minima values are just minimum in terms of their numerical value (least negative value) but in physical sense these minimum values are maximum speeds of any change in muscular tension while moving the eyeball or blinking.

32

Fig. 22 Function Fitting Using Curve Fitting Toolbox

While calculating the peaks in a data, a window of half the length of epoch is used. There can be cases when the data gets divided in between the two windows. In such cases, instead of alternate negative and positive values of local maxima, there can be only one positive maxima in one window and a negative maxima in other window. To overcome this problem, a shifting window is used when such a condition arises. In this way, loss of characteristics in the data is minimized.

• Time index evaluation for detection: Data presented in section IV (background theory) has some characteristics which are quite common in all the three cases. There are increasing voltage levels with some slope and decreasing voltage levels with different slope. Computation time can be greatly reduced if single detection algorithm is used for all the three cases. Since the increasing and decreasing characteristics of the data are similar, peaks can be detected easily in all the three types of data. The indices of peaks can be used to differentiate between the type of muscular movement. It can be seen that peaks occur at almost the same time in blinking while there is a delay in one of the peaks when eyeball movement occurs. This information can be used to complete the detection.

33

• Cursor Initialization and Movement: Cursor initialization is done using a simple code in MATLAB which calls the java class object for initializing mouse pointer. After initialization, from the data that we have got and analysis we have done, it is possible to guide the cursor in horizontal direction and that too in discrete steps because the data doesn’t contain any information on how much eyeballs have travelled in one particular direction. To accomplish this task of continuous tracking more electrodes need to be placed around the eye, particularly below the eye and on one side of the eye. This would make the computations more cumbersome and therefore, in this technology demonstrator only discrete cursor movements have been done.

34

Project Work Stage 1: Data Acquisition

BIOPAC EEG Data Acquisition System needs to communicate with the computer to transfer data which can be utilized for analysis purposes. One of the major drawbacks with this system is that data acquisition with other interface programs can be done only in serial programming. This means that during data analysis, loss of data can occur because while completing the data analysis part, data acquisition lags. To overcome this problem, data acquisition can be done using Dev C++ and data analysis can be done using MATLAB . A proper communication network should be established between MATLAB and C++ so that there can be no loss of data and MATLAB can also acquire the data as soon as analysis of previously acquired data completes. Following are the main components of data acquisition:

• MP150acq.exe • MATLAB GUI

1. MP150acq It is an executable file which has been developed in Dev C++ for data acquisition. This is the main interface with Biopac EEG Data acquisition system. It calls the dll, header file and library files necessary for interface with the EEG device. It keeps on waiting for the interrupt from MATLAB GUI. As soon as the star button is pressed with mode selected to realtime, it reads the acquisition file which is again generated by MATLAB as soon as the acquisition parameters are set. As soon as MP150acq gets a 1 in the interrupt file an infinite loop breaks up and data acquisition starts. Data gets stored in two separate text files: fp1data.txt and fp2data.txt. Another text file start.txt contains the number of epochs that have been written on the file and this text file acts as a counter for MATLAB. The counter tells MATLAB to wait if the value of its own index counter exceeds the value in start file. Until the value in start file is zero, MATLAB doesn’t start acquiring data. In this way communication gets established between data acquisition and data analysis. Description of files necessary for MP10acq.exe:

• mpdev.dll: Communication and data acquisition is established with BIOPAC MP Systems by calling this dll file. It is created and distributed by BIOPAC systems.

• mpdev.h: A header file for MP Systems and contains a list of all the important functions necessary for setting up the dll file.

• mpdev.lib: Library file for MP systems • MP150acq.cpp: C++ source file which can be edited according to our need.

Loads the library file and dll files for data acquisition.

35

Fig. 23 Block Diagram of Data Acquisition and Communication

2. MATLAB GUI Data analysis is done in MATLAB environment. The GUI performs various important functions for data acquisition. Acquisition parameters are set in MATLAB using ‘Set Acquisition Parameters’ drop down menu. When start push button is pressed, the value in interrupt text file goes 1. This value acts as an interrupt for C++ Program to start data acquisition. Then MATLAB GUI goes in a loop to check if first sample of data has been acquired. Since it can become an infinite loop if data acquisition fails or executable file is not working, a check was put inside the loop. If the loop goes on for more than 10 seconds without breaking, then the GUI automatically closes and error dialog box for the same appears. This GUI also prepares .mat files for whole data acquired and also for individual epochs of data. Therefore data acquisition takes place in following steps:

• Switch on the data acquisition device • Run MP150acq.exe • Start MATLAB GUI, choose mode=realtime and set acquisition parameters • Push start button on MATLAB GUI • Data Acquisition Starts.

3. Trial Run Initial values of communication files: execfile.txt=0 start.txt=0

Graphical User Interface

Data Analysis (MATLAB)

Data Acquisition

Counter

Data Acquisition

(C++)

BIOPAC MP150 System

36

Fig. 24 Output of .exe at start Start Push Button is pressed execfile.txt=1 acq.txt=5 1000 1(runtime, frequency, length of epoch in seconds)

Fig. 25 Output after end of acquisition

37

After data acquisition is completed start.txt=5 .mat files created: fp1data_1, fp1data_2, fp1data_3, fp1data_4, fp1data_5, fp2data_1, fp2data_2, fp2data_3, fp2data_4, fp2data_5, fp1data, fp2data

38

Project Work Stage 2: Eyeblink Detection and Eyeball Tracking

For eyeblink detection and eyeball tracking, MATLAB is used as a programming platform. When an epoch of data from the BIOPAC machine is available, analysis starts in MATLAB. For nonlinear regression, Curve Fitting Toolbox is used and a .m file generated from the toolbox is modified to find the differential of the data. 1. List of Main Functions

• avgresults • locmaxmin • maxminanalyse • sort1 • trackingeye • RunDeriv

The complete code is included in Compact Disk (CD) attached with this report. Following is a brief description and demonstration of these functions. 2. Calculation of Differential Data – avgresults 2.1 Purpose and Prototype This function generates differential of the data using non-linear regression technique. Sum of two Gaussians is used as the model that needs to be fitted. Following is the prototype of this function:

diff = avgresults(data) ‘data’ is the EOG data ‘diff’ is the differential of EOG data 2.2 Run and Demonstration >> diff = avgresults(data); >> subplot(2,1,1) plot(fp1data) subplot(2,1,2) plot(diff)

39

Fig. 26 Differential of Data

3. Locate Maxima - locmaxmin 3.1 Purpose and Prototype This function locates initial maxima and minima in the differential of data. Following is the prototype of this function:

maxmin = locmaxmin(diff,threshold_upper, threshold_ lower) ‘diff’ is the differential of EOG data ‘threshold_upper’ is the threshold for positive differential values ‘threshold_lower’ is the threshold for negative differential values 3.2 Run and Demonstration >> maxmin = locmaxmin(diff,0.002, 0.002) >> size(maxmin) ans = 178 2

40

4. Locate Final Peaks - maxminanalyse 4.1 Purpose and Prototype This function locates final maxima and minima in the differential of data. The values returned from this function are the maximum speeds of eyelid opening and closing. Following is the prototype of this function:

vmaxmin = maxminanalyse(maxmin,window) ‘window’ is the number of samples around which only one maxima can occur. Study of ocular data reveals that this value can be taken to be 200ms. ‘maxmin’ is the matrix of initial maxima and minima values. 4.2 Run and Demonstration >> vmaxmin = maxminanalyse(maxmin,200) vmaxmin = 0.0048 895.0000 -0.0048 995.0000 >> figure, hold on >> plot(diff) >> plot(vmaxmin(:,2),vmaxmin(:,1),'*r')

Fig. 27 vmaxmin on differential data

41

Fig. 28 Erroneous Data

To correct such errors a shifting window is used which shifts when size of vmaxmin is odd or one. In this way whole data is collected in a window and then analysis can take place in a better way. 5. Sorting time series – sort1 5.1 Purpose and Prototype This is a complementary function to sort the vectors as required. This function has two forms. If number of input argument is one, then sorting takes place according to the first column and if input arguments are two, then sorting takes place according to the second column. Following is the prototype of this function:

a=sort1(maxmin,d) ‘maxmin’ is an array of maxima points in derivative data ‘d’ is an optional parameter and is number of rows of vector maxmin 5.2 Run and Demonstration >> maxmin=[1 3;5 2;2 7]; >> a=sort1(maxmin) a = 1 3 2 7 5 2 >> a=sort1(maxmin,3)

42

a = 5 2 1 3 2 7

6. trackingeye 6.1 Purpose and Prototype This function finds out the blinks, left movement and right movement in the data. It compares the indices of peaks in FP1 and FP2 data and then according to the algorithm, returns the values for all types of movements. These values are checked and cursor is commanded to work accordingly. Following is the prototype of this function:

[early late blinks c1,c2]=trackingeye(vmaxmin1,vmax min2, last1, last2]

It outputs 1 if a movement has occurred or not. It checks various cases of appearance of peaks. Suppose that delay in a peak is quite large that it passes out of the present window but appears in the next window, then last1 and last2 are updated accordingly and they are checked against other peak. 7. runderiv 7.1 Purpose and Prototype This function calculates the running derivative of an unevenly spaced time-series with a flat weighing function. Following is the prototype for this function:

RunDeriv=runderiv(Data,WindowWidth)

This matlab function has been adapted from a MATLAB function by Eran O. Ofek (http://wise-obs.tau.ac.il/~eran/matlab.html) 8. Cursor Initialization and Movement in MATLAB The ability to control mouse pointer position, motion and clicks is not available in MATLAB. To work around this issue, Java class java.awt.Robot can be used. Below is a simple example that illustrates how one can move the mouse with MATLAB code using Java functionality. import java.awt.Robot; mouse = Robot; mouse.mouseMove(0, 0); screenSize = get(0, 'screensize'); for i = 1: screenSize(4) mouse.mouseMove(i, i); pause(0.00001); end

43

Project Work Stage 3: Implementation of Graphical User Interface

(GUI) 1. Graphical User Interface GUI provides a convenient way to perform the experiment. This GUI is built in MATLAB 7.6.0. This GUI provides all the capabilities starting from Normalization to Cursor Clicks and Movement. It is shown in Figure 1. This GUI and its all files are available on the CD attached with this report. 2. Description of Different Controls of GUI

Fig. 29 Main GUI Window This GUI has the following sections:

1. Cursor Bar: The Cursor bar is the section where mouse movement and clicks occur. It consists of a figure and a text bar. The text bar displays the color of the

44

box selected and gives instructions on when to start the experiment. The figure contains five rectangular boxes of different colors. Mouse pointer is initialized at the blue color box and as the mouse moves and clicks on some other box; the color of that box is displayed in the text box.

2. Start/Stop Control: This section consists of the Start and Stop pushbuttons and a table component that displays the calculated parameters for every epoch of data collected. The parameters displayed are mean voltage, rms voltage and variance of the signal. As soon as the Start pushbutton is pressed data acquisition and analysis starts. Pressing of the Start pushbutton checks whether mode of operation is selected. If selected mode is real-time, then another check is put in place which ensures that the acquisition parameters are set. Violation of any of these conditions produces an error dialog box. The Stop button press executes a GUI component which is shown in figure 2. This is a dialog box which asks user if he wants to quit or not. When ‘yes’ is pressed, the GUI window closes.

Fig. 30 Exit Dialog Box

Fig. 31 Start Stop Control

3. Plot: Plot section plots the incoming data from FP1 and FP2 channels on the two axes components. This helps the user to visually analyze the data and is meant for visual analysis purposes.

45

4. Menu Bar: This section is contains the menu bar of the GUI. The figure shows all the menu items used in the GUI

Fig. 32 Menu Items

Following is the description of the menu items: • Mode: Allows the user to select the mode of operation. When the user

clicks on this tab a dialog box appears with two radio buttons pertaining to offline and real-time mode.

Fig.33 Mode Dialog Box

• Record: Records the data from FP1 and FP2 channels in a specified MAT file.

• Record as Text: Records the data from FP1 and FP2 channels in a specified text file. It uses dlmwrite function to write data in text file.

• Exit: Closes the GUI. • Acquisition Properties: Sets the acquisition parameters of the data

acquisition device. The parameters that can be set are runtime in

46

seconds, sampling frequency and length of epoch in seconds with default values set at 5, 1000 and 1 respectively. It uses an input dialog box to input the values.

• Eyeblink Parameters: Sets the eyeblink parameters value for analysis of eyeblinks. The parameters that can be set are lower threshold and upper threshold. It also uses an input dialog box to input the values.

• Eyeblink: This menu item is used to analyze the eyeblinks of the experimenter so that the eyeblink parameters can be set. The popup menu has two axes for plotting the data and differential of data. As soon as start is pressed, an input dialog box appears and user can set the acquisition parameters for the normalization process.

• Eyeball: This menu item is used to analyze the movement of eyeball of the experimenter so that the thresholds for cursor movements can be set. The popup menu has two axes to display the data from the two channels and three radio buttons out of which one needs to be selected and eyeball moved in the selected direction to start the normalization process. Set parameters push button sets the parameters for acquisition. As start is pressed, data acquisition starts; mean voltage, rms voltage and energy spectrums are stored in the corresponding variables according to the direction selected.

Fig. 34 Eyeblink Normalization

47

Fig. 35 Eyeball Normalization

• Help: Opens the help file of the GUI. • Documentation: Opens the documentation file of the GUI.

Table 6 Keyboard Shortcuts of Menu Items

Menu Item Keyboard Shortcut Mode Ctrl + M Record Ctrl + S

Record as Text Ctrl + T Exit Ctrl + X

Acquisition Properties Ctrl + A Eyeblink Parameters Ctrl + E

Instructions Ctrl + H Documentation Ctrl + D

Eyeblink Ctrl + O Eyeball Ctrl + N

48

Practical Application Human Computer Interface

1. Introduction Human Computer Interface is a technology demonstrator to establish the usage of Visually Evoked Potentials in guiding a cursor. These types of technologies are in advanced stages of development in some countries because they provide a person with severe motor disabilities a way to communicate with machines and computers. These types of machines are often aimed at assisting, augmenting or repairing human cognitive or sensory-motor functions. They have also gained widespread acceptance in neuroprosthetics applications that aim at restoring damaged hearing, sight and movement. U.S. Military is has been exploring applications for such devices, to enhance troop performance and to communicate with the adversaries. A documentation report and all the necessary files and complete codes are available on the CD attached with this report.

Fig. 36 Experimental Setup

2. Main Features

• Non-invasive type of technique, so no implantation is necessary. • Fast and robust method of controlling cursor movement as compared to image

processing based methods. • Offline Average Computation Time of 0.0007 seconds and Online Average

Computation Time of 0.0157 seconds. • Minimal number of electrodes required to control the mouse pointer

49

• Lots of future prospects for further improvements.

3. Block Diagram of Human Computer Interface

4. Requirements Following are the software and hardware requirements: 4.1. Hardware Requirements

1. Computer at least 1.3 GHz and 1 GB RAM 2. BIOPAC Data Acquisition System 3. EEG Electrodes

4.2. Software Requirements

1. Operating System: Windows XP 2. MATLAB 7.6 with Curve Fitting Toolbox 3. MP150acq.exe 4. All necessary MATLAB codes for data analysis and acquisition.

5. Operating Instructions 5.1. Starting the System

1. First turn on the computer. 2. Connect the BIOPAC machine to the Ethernet port. 3. Wait for some time for connection.

Graphical User Interface

User Commands

Eyeblink Detection and

Eyeball Tracking

Data Acquisition

Normalization

Cursor Initialization

Cursor Clicks and Movement

50

4. Start MATLAB. 5. Start GUI named BTP2.

5.2. Normalization 1. Go to Normalization and Click on Eyeblink 2. Observe differential of eyeblink data and left, right as well. 3. Set eyeblink parameters by pressing Ctrl + E or Settings�Set Eyeblink

Parameters 4. Go to Normalization again and Click on Eyeball 5. Take the data by clicking on the radio button. Set a particular value of thresholds

and then observe the peaks by moving your eyeball in the specified direction selected by radio button.

6. Set the parameters before pressing acquire. 5.3. Running in Real-Time Mode

1. Select the mode by pressing Ctrl + M or File->Mode 2. Set acquisition parameters by pressing Ctrl + A or Settings-> Acquistion

Parameters 3. Start MP150acq.exe 4. Push Start button on MATLAB GUI and observe the cursor movement as you

move your eyeballs in the desired direction and mouse click as you blink your eyes.

5.4. Running in Offline Mode

1. Select the mode by pressing Ctrl + M or File->Mode 2. Press Start button. Load the .txt file that contains this type of data. 3. Observe the data that has been preset and relate it to cursor clicks and blinks.

6. The Experiment 6.1 Instructions for Cursor Control

1. Cursor movement is a discrete movement. When user wants to move cursor to the left, he/she should see towards extreme left and then come back to centre position. If he/she wants to go left again, then perform the same thing, otherwise move your eyes to the right side.

2. Blink eyes for cursor clicks.

51

Future Work

1. The project can be extended for two dimensional cursor controls.

2. With parallel programming capabilities, more electrodes can be used which can detect the signal with other algorithms that might prove fast.

3. This technology demonstrates the capability of the technique to control mouse

cursors. The developed programs can be run in the background and whole computer can be controlled using such technology.

4. Time factor is crucial for real time applications. The code (refer enclosed CD) can be made more efficient to execute large algorithms involved in least possible time.

5. The idea can be used to detect sleep events and level of alertness in a person by

measuring the dynamic parameters like mean speed of eye lid closure, stability of the eyeball at the farthest left corner etc. This experiment would prove to be a cornerstone in development of handheld portable devices for measuring alertness levels and prevent road accidents due to sleep events.

52

Conclusion The project ‘Design and Development of Human Computer Interface Based Controllers’ was completed successfully. I implemented all the algorithms and provided a very useful practical application in the field of biomedical signal processing. My system works in robust and reliable way and is successful in recognizing movements of the eyeball and eyeblinks. The system has been provided with the capability to normalize and hence, this system is invariant to different persons. A data acquisition interface has been made which is able to communicate between two different compilers and acquire data while analysis is going on as well. Various checks have been placed that help the system to be stable and versatile. Single algorithmic approach has been chosen for identification purposes. This makes the program fast and computationally light. The provided Graphical User Interface (GUI) is very interactive and easy to operate. All the instructions are clear and users can easily perform the desired task in this project.

53

Bibliography and References

1. Yuanqing Li, Chuanchu Wang, Haihong Zhang and Cuntai Guan, “An EEG Based

System for Two Dimensional Control”, International Joint Conference on Neural Networks (IJCNN 2008)

2. Sanjay Gupta and Harvinder Singh. “Preprocessing EEG Signals for Direct Human-System Interface”, IEEE 1996

3. Aleksander Kostov and Mark Polak "Parallel Man–Machine Training in Development of EEG-Based Cursor Control" IEEE Transactions On Rehabilitation Engineering, VOL. 8, NO. 2, June 2000

4. G.A.F Seber and C.J. Wild. “Nonlinear Regression”. New York: John Wiley and Sons, 1989.

5. R.J.Oosterbaan, “Frequency and Regression Analysis”. In: H.P.Ritzema (ed.), Drainage Principles and Applications, Publ. 16, pp. 175-224, ILRI, Wageningen, The Netherlands, 1994

6. Curve Fitting Toolbox, http://www.mathworks.com/products/curvefitting/

7. Pan and Tompkins, “A Real Time QRS Peak Detection Algorithm” IEEE Transactions on Biomedical Engineering, March, 1985

8. Rangaraj M. Rangayyan, “Biomedical Signal Analysis: A Case Study Approach”.Singapore: John Wiley and Sons, 2005

9. http://en.wikipedia.org/ 10. Kenneth Levenberg (1944). "A Method for the Solution of Certain Non-Linear

Problems in Least Squares". The Quarterly of Applied Mathematics 164–168. 11. http://www.biopac.com/