Class separation of target and non-target responses using ...

Class separation of target and non-target responses

using unsupervised learning P300 speller-based BCIs

Completed in partial fulfillment of graduation requirements for an Honours Bachelor of Science –

Computer Science degree.

Connor Flood

Algoma University

Supervised by Dr. George Townsend

April 09, 2015

2 | P a g e

Abstract

Implementing various techniques to improve the classification of target and non-target responses

for P300 spellers shows great potential for increasing the accuracy of spelled letters and words.

There is a balanced limitation in the parameters associated with using a P300 speller, such as the

number of flashes per character, and the total number of characters on the board. Some

classification techniques, such as a brute-force style guessing approach for every character per

letter spelled, did not show to make substantial improvements to the existing classifier. However,

after modifying the P300 speller parameters to increase the number of flashes, and decrease the

number of letters on the board, meaningful results were found where letters were often spelled

correctly without any problems. The success of the classification of the target and non-target

letters were attributed both to the classifying techniques, as well as to the modification of the

parameters of the P300 speller.

Acknowledgements

I would like to sincerely thank my thesis supervisor, Dr. George Townsend, for introducing me

to the research area of BCIs, and for all of his patience and dedication in helping me through all

steps of this thesis. I would also like to thank Valerie Platsko for all of her help, and sharing the

lab resources.

Finally, a huge thank you to my parents (Glenn and Kristen Flood), and my brother Alex, for

their continued support both during the time spent on this thesis, as well throughout my

undergraduate degree and planning of further education and career paths.

3 | P a g e

Table of Contents

Abstract ................................................................................................................................................. 2

Acknowledgements ................................................................................................................................ 2

List of Figures ........................................................................................................................................ 5

List of Tables ......................................................................................................................................... 5

Part 1 – Introduction ............................................................................................................................. 6

Background Information ................................................................................................................... 6

Brain Computer Interfaces............................................................................................................ 6

EEG and 10-20 System .................................................................................................................. 7

Event Related and Steady State Visually Evoked Potentials ........................................................ 7

P300 Speller ................................................................................................................................... 8

Row-Column Paradigm ................................................................................................................. 8

Checkerboard Paradigm ............................................................................................................. 10

Amyotrophic Lateral Sclerosis (ALS) ......................................................................................... 10

Feature Extraction ....................................................................................................................... 11

Class Separation .......................................................................................................................... 12

Unsupervised Learning................................................................................................................ 12

Objective .......................................................................................................................................... 13

Rationale .......................................................................................................................................... 13

Research Question ........................................................................................................................... 14

Scope ................................................................................................................................................ 14

Limitations ....................................................................................................................................... 14

Timetable ......................................................................................................................................... 15

Methods ............................................................................................................................................... 16

Materials .......................................................................................................................................... 16

Procedure ......................................................................................................................................... 16

Procedure - Generating Features ................................................................................................ 17

Feature #1: Symmetry .................................................................................................................... 21

Feature #2: Convexity .................................................................................................................... 22

Feature #3: Median-Max Index Difference ..................................................................................... 23

Feature #4: Direction Changes ....................................................................................................... 24

Combining Features: ...................................................................................................................... 25

4 | P a g e

Procedure - Brute-Force Guessing .............................................................................................. 25

Procedure - Modifying P300 Parameters .................................................................................... 30

Results .................................................................................................................................................. 33

Results - Generating Features ......................................................................................................... 33

Results – Brute-Force Guessing ...................................................................................................... 34

Results – Modifying P300 Parameters ............................................................................................ 35

Discussion ............................................................................................................................................ 41

Discussion – Generating Features ................................................................................................... 41

Discussion - Brute-Force Guessing .................................................................................................. 42

Discussion – Modifying P300 Parameters ....................................................................................... 44

Conclusion ........................................................................................................................................... 47

Future Work ........................................................................................................................................ 48

References ............................................................................................................................................ 50

Appendix A - Graphs .......................................................................................................................... 52

Appendix B – Code .............................................................................................................................. 57

5 | P a g e

List of Figures

Figure 1: Score of Targets and Non-Targets vs Time – Page 19

Figure 2: Score of Targets and Non-Targets vs Time (zoomed in) – Page 20

Figure 3: Classifier Score vs Character Value – Page 36

Figure 4: Letters Correct vs Number Sequences – Page 38

Figure 5: Score Rank of ‘U’ vs Number Sequences – Page 39

Figures 6-13: Character Rank vs Data Sequence Number in “COMPUTER” – Appendix A

List of Tables

Table 1: Feature Score Results – Page 33

Table 2: “WADSWORTH” Target Letter Ranking – Page 35

Table 3: “COMPUTER” Target Letter Ranking – Page 36

Table 4: Letters Correct Per Sequence – Page 37

Table 5: Words Guessed Per Sequence – Page 40

6 | P a g e

Part 1 – Introduction

Background Information

1.1 Brain Computer Interfaces

1.2 EEG and 10-20 System

1.3 Event Related and Steady State Visually Evoked Potentials

1.4 P300 Speller

1.5 Row-Column Paradigm

1.6 Checkerboard Paradigm

1.7 Amyotrophic Lateral Sclerosis (ALS)

1.8 Feature Extraction

1.9 Class Separation

1.10 Unsupervised Learning

Brain Computer Interfaces

Dennis McFarland and Jonathan Wolpaw discuss in their publication “Brain-Computer

Interfaces for Communication and Control” that brain activity produces electrical signals which

are detectable on the scalp [1]. A Brain Computer Interface (BCI) is able to translate these

signals into outputs which allow users to communicate without participation of peripheral nerves

or muscles [1]. McFarland and Wolpaw further explain that electrodes on the surface of the scalp

are still fairly distant from brain tissue and are separated by coverings of the brain, skull,

subcutaneous tissue, and scalp [1]. This results in the signal being considerably degraded [1].

Only synchronized activity of large numbers of neural elements can be detected, thus limiting the

7 | P a g e

resolution with what the brain activity can be monitored in this non-invasive approach [1]. In

addition, scalp electrodes also detect activity from sources other than the brain, such as

environmental and biological noise [1].

EEG and 10-20 System

Obtaining and measuring electroencephalogram (EEG) signals is necessary in many techniques

for recording brain signals, and is important due to its non-invasive property and easy

implementation [2]. Evoked potentials are one paradigm of EEG, which contains transient

waveforms in ongoing activity [1]. These waveforms are phase-locked to an event [1]. An

international standard for arranging the electrodes placed on the scalp is referred to as the 10-20

system [3]. The 10-20 system is conventionally based on the identification of anatomical

landmarks, with consecutive placement of electrodes at fixed distances [4].

Event Related and Steady State Visually Evoked Potentials

Zhenghua Wu states that when a visual stimulus with a steady frequency appears in the visual

field of a person, that an electrical signal can be observed via the scalp, with the same frequency

as that of the stimulus [5]. This signal is referred to as steady-state visually evoked potential

(SSVEP) [5]. While there are differences in the powers of different frequency SSVEPs, all of

them are concentrated on the occipital area [5]. Wu also makes the connection that even though

SSVEP has been used in many areas since its discovery, one important area concerns BCI, in

which many flickers with different frequencies are used to stimulate the subject’s eyes [5]. By

recognizing the frequency components in the subject’s SSVEP, the flicker that the subject is

8 | P a g e

staring at is determined [5]. As a result, the task corresponding to that flicker can be completed

automatically [5].

An event-related potential (ERP) is a signal observed on the scalp of the subject that occurs when

an event such as a visual stimulus happens in a short period [5]. The ERP for each trial is

hypothesized to be synchronous with the event and similar between trials [5].

P300 Speller

The P300-based speller was originally introduced by Farwell and Donchin in 1988 [6]. Subjects

using the P300 speller select characters from a matrix presented on a computer screen by

analyzing and classifying EEG signals [6]. The flashing of a character being focused on elicits an

event-related potential (ERP) that distinguishes between target and non-target characters [6]. A

“target” refers to the character being focused on and attempting to spell, and “non-targets” are

the remaining characters. Classification usually includes features both from the occipital lobe

(visually-evoked potentials), as well as from the P300 response [6]. The P300 response is a

positive deflection in the EEG over the parietal cortex that appears around 300 ms after a

stimulus [6]. Groups of characters are successively and repeatedly flashed, but only the group

that contains the target character will elicit a response [7]. A central limitation of this application

lies in the need of a high signal-to-noise ratio in order to make an accurate decision [7].

Row-Column Paradigm

In the original implementation of a P300 BCI, characters items are grouped for flashing as rows

and columns [8]. Thus, this orientation is referred to the row-column paradigm (RCP) [8]. The

9 | P a g e

computer is able to identify the target character as the intersection of the row and column that

elicited the largest P300 [8]. The RCP however is subject to errors that slow communication,

cause frustration, and diminish attentional resources [8]. Townsend, et al grouped these errors to

have two primary causes.

The first of these causes are that errors typically occur with the greatest frequency in locations

adjacent to the target character, and nearly always in the same row or column [8]. Since a P300 is

produced for every item in a row or column, this error occurs every time a target item flashes [8].

Errors also arise when flashes of non-target rows or columns that are adjacent to the target,

attract the subject’s attention, and as a result produce P300 responses [8]. This is often referred to

as “adjacency-distraction errors.” [8]

The second of these causes are that sets of items must be intensified in random order. However,

this allows target items to sometimes flash consecutively [8]. This occurs when a row flash is

followed by a column flash (or vice-versa), and the target item is at the intersection of that

particular row and column [8]. One error that is caused by a double-flash is that the second flash

may go unnoticed by the subject, and thus no P300 response is produced [8]. The second error is

that even if the second flash is perceived, the P300 responses to the two flashes will temporarily

overlap [8].

The above mentioned double-flash issues associated with CBP are collectively referred to as the

“double-flash problem” [8]. A solution to this problem is shown in the next section, entitled the

“checkerboard paradigm.”

10 | P a g e

Checkerboard Paradigm

In recognizing the issues associated with the double-flash problem, an alternative stimulation

paradigm that is faster, more accurate, and more reliable than the RCP was designed by

Townsend, et, al.. This new paradigm is called the “checkerboard paradigm.” (CBP) [8]

The CBP uses an 8x9 matrix which is virtually superimposed on a checkerboard, that the

subjects never actually see [8]. Items in white cells of the 8x9 matrix are segregated into a white

6x6 matrix, and items in the black cells are segregated into a black 6x6 matrix [8]. The

respective items of each of these 6x6 matrices are randomly populated before each sequence of

flashes [8].

This results in the subject seeing random groups of 6 items flashing, opposed to entire rows or

columns in traditional RCP [8]. More importantly, CBP ensures that an item cannot flash again

for a minimum of 6 intervening flashes, and a maximum of 18 intervening flashes [8]. Thus, this

eliminates the double-flash problem. By maximizing the time between successive flashes of the

target item, the CBP should increase the amplitude of the P300 responses, and resultantly also

improve BCI speed and accuracy [8].

Amyotrophic Lateral Sclerosis (ALS)

Amyotrophic lateral sclerosis (ALS) is a progressive disease of the upper and lower motor

neurons that often leads to complete paralysis within 2-5 years [9]. As of 2013, about 30,000

people in the United States are living with ALS [10]. As the disease progresses, many assistive

communication devices that were once a necessity, may become ineffective [9]. BCI-technology

11 | P a g e

allows people with severe motor disabilities to use brain signals, rather than muscles, to

communicate (via a P300 speller) and control their environments [9].

Feature Extraction

Wolpaw and Wolpaw define two important concepts that are important to understand when

discussing feature extraction [11]. First, signal-to-noise ratio (SNR) is the simply the ratio to

signal power to the noise power [11]. A high SNR indicates minimal corruption of the signal of

interest by background noise [11]. An important concept is understanding the difference between

noise and artifacts. Noise is due solely to background neurological activity [11]. Artifacts

however are due to biological or external sources unrelated to neurological activity, such as eye

blinks or artificial respirator activity [11]. Both noise and artifacts can contaminate the signal

[11].

A fundamental signal feature is simply a direct measurement of a signal [11]. Alone,

fundamental signal features often provide limited relevant information about complex brain

signals [11]. But when used in linear/non-linear combinations, ratios, statistical measures, or

other transformations of multiple signal features, can provide meaningful data [11]. Most

features in BCI applications are based on spatial, temporal, and spectral analyses of brain signals

[11]. In order to determine the subject’s wishes when using a BCI to be as accurate as possible, a

number of features must be extracted simultaneously [11].

12 | P a g e

Class Separation

Classification involves separating data into two or more classes, where each data member must

belong to exactly one class. Many papers in this area explore classification techniques with

domains up to tens of thousands of features [12]. Selecting the most relevant features is usually

suboptimal for building a predictor, especially if the features are redundant [12]. However, a

subset of useful features may exclude many redundant, but also relevant, features [12]. Many

feature selection algorithms include feature ranking as a principal or auxiliary selection

mechanism because of its simplicity, scalability, and empirical success [12]. The ranking of these

features directly relates to a feature’s respective weights [12]. Noise reduction and consequently

better class separation may be obtained by adding features that are presumably redundant [12]. In

terms of unsupervised feature selection, feature ranking criteria that have been found useful

include entropy, smoothness, density, and reliability of the features.

Unsupervised Learning

Before details of unsupervised learning are discussed, a brief explanation of what supervision

learning should be defined. Supervised learning is learning in the presence of ground truth, such

that training data is labelled with respect to their class membership [13]. Thus unsupervised

learning is learning from unlabelled data [13]. In the context of classification, the goal of

unsupervised learning is to divide a set of unlabelled data into classes [13]. Unsupervised

classification can directly fit a model of a multi-modal probability density to the training data

[13]. In principle, it is not difficult to extend any probability model to a mixture model [13].

Unsupervised learning can only focus on representing the training data as well as possible [13].

13 | P a g e

Objective

This thesis will attempt to implement various class separation techniques in order to more

effectively classify target and non-target responses derived from a P300 speller-based BCI. By

being able to generate meaningful features and apply appropriate weights to those features, the

classification of targets and non-targets will be more successful, and improve the usability and

success of the P300 speller.

The analysis component will focus on the role and overall effect that added class separation

techniques play in the classification stage of the P300 speller, and consequently the measure to

which these techniques will perform better or worse than existing classifying techniques. The

analysis and results of this will be tested on unseen data, in order to mimic the functionality of

unsupervised learning.

Rationale

Proper classification of targets and non-targets is essential and is the basis for a P300 speller-

based BCI’s success in allowing its subjects to effectively communicate. Not only does this have

the potential to increase the reliability of the classification, but could also result in subjects

requiring fewer flash sequences in order for the BCI to confidently decide the character they are

attempting to spell. If the subject using the BCI suffers from late stages of ALS, the effectiveness

of the P300 speller will greatly improve their ability to communicate, and thus their quality of

life. By focusing on proper classification for unsupervised learning, results generated will be

more applicable to the BCI spelling the subject’s target letters in real time.

14 | P a g e

Research Question

This thesis will attempt to answer the following research question:

“To what extent do newly implemented class separation techniques improve the functionality of

classifying targets and non-targets in unsupervised learning, as applied to the P300 speller-based

BCIs?”

Scope

The scope will focus strictly on unsupervised learning, and implementing techniques for class

separation. Although new data was generated during the time frame of this thesis, there will be

no discussion of data acquisition methods as this was not a focus for this topic. With the

exception of the above mentioned, all data was previously generated before research and

implementation stages began, and is independent of experiments later discussed.

Limitations

A major limitation encountered was the amount of test data that the implemented classification

techniques were tested with, due to time constraints. There was much “manual labour” involved

when using the P300 GUI application with the modified code, which was very time consuming.

Future work on this topic could allow a greater amount of time to be spent towards automating

this process, and thus allowing more time to conduct further trials.

15 | P a g e

Timetable

16 | P a g e

Methods

Materials

Analysis:

Laptop/Computer (running Windows XP or later) for performing analysis

MATLAB r2014a

Sample EEG data from previous experiments

P300 GUI Application

Data Acquisition (not performed by me):

8-channel electrode cap

Conductive gel

Toothpick

Computer (running Windows XP) and monitor

BCI 2000 Software

Procedure

The overall procedure for this thesis can be broken down into three sections, with each section

examining a different purpose and producing various forms of results. This allowed for a breadth

of understanding in various sub-areas of data classification, and a greater number of conclusions

to be made.

17 | P a g e

The first section will detail manually generating features in an attempt to better classify target

and non-target responses, based on the classifier scores already obtained.

The second section will focus on a brute-force method of analysis in which each possible

character was “guessed” to be the target letter at each flash.

The third section will modify two P300 parameters: number of sequences, and number of

characters to choose from, to determine both its overall effect, as well as the effect on

increasing/decreasing number of sequences towards the success of proper classification.

Procedure - Generating Features

The process for manually generating features can be difficult, and often requires some trial and

error. The goal was to create and implement features which would be able to score targets and

non-targets in an unsupervised learning fashion. Features, in this case, would be mathematical

ways to score trials based on corresponding data. However, the data these features are scoring

wouldn’t be directly dependent on the raw EEG values. Instead, these features would be

computing scores that an existing classifier had already assigned after generating its own weights

and features. While this may seem like an unnecessary step, the accuracy of the previous

classification is not very high, as there are many target and non-target responses which were

incorrectly labelled. Therefore, it is the goal that these newly created features will be able to

improve the previous classification based on its own results.

When a trial is classified to be either a target or non-target, a numeric score is returned to

determine which class it belongs to. A positive score indicates that the trial is classified as a

18 | P a g e

target, and a negative score indicates that the trial is classified as a non-target. The range of a

score is from negative infinity to positive infinity. More realistically, the scores being examined

in the sample data given ranged between -6000 and +5000.

It would not make sense to generate a feature to improve classification based on a single score

being returned for a given trial. A single score is 1-dimensional and does not provide any

additional data or patterns which can be extracted from to generate a feature. Continuing what

will be discussed below, the classification will be based on consecutive target scores, and

consecutive non-target scores, and attempting to find consistent and distinct patterns for targets

and non-targets.

The input data used for this experiment will contain 3192 flashes, consisting of 266 targets and

2926 non-targets. The dynamic that is added to increment the dimension size of this data is

obtained from “shifting” EEG. The concept of shifting EEG refers to shifting the indices of EEG

values so that the position of those values relative to the time after a flash occurs changes. After

this shift occurs, then the data can be completely reclassified. For the data that was provided for

this experiment, there were 16 shifts of indices incrementing by 1 backwards (i.e. to the left), and

16 shifts forwards (i.e. to the right). With 32 shifts, plus the origin of the starting position, that

gives a total of 33 classification scores per trial. The research and investigation of the

effectiveness of these shifts goes outside the scope of this thesis. The implementation to generate

the data with shifted EEG classification values was previously done, and was simply provided to

perform this experiment.

19 | P a g e

Given that the input data contains 3192 flashes, and each one of those flashes also contains 33

classification scores, the structure being dealt with is a 2-dimensional 3192x34 array of type

‘double’. Below is a figure of this structure being plotted, with some additional parameters to be

discussed:

In the above figure, the blue data points are the actual targets, and the red data points are the

actual non-targets. While this seems contradictory to know what the actual targets/non-targets

are for unsupervised learning classification, this knowledge is never used to help or support the

classification. Instead, it was used as more of a visual inspection to try and detect patterns in the

data visually to make some general inferences. The above figure also contains large dots. These

dots represent the middle data point (16), out of the range of 33 scores given for each flash letter.

In reality, it was this data point which was used to perform the classification, depending on

whether it was positive or negative, as previously discussed.

20 | P a g e

What the above figure doesn’t illustrate very well, is that the data is actually graphed in

parabola-like shapes for each trial. By zooming in to a more visible scope, this is apparent in the

figure below, using a subset of the same data as above:

Here the patterns for the 33 data-points for each trial are more clearly illustrated. The challenge

now is to find consistent patterns in these “parabolas” that are both unique and contrasting

between targets and non-targets. In basic terms, the objective is to find mathematical ways to

determine which data points are red, and which data points are blue, in the above two figures.

Determining what patterns to find and implement are crucial towards the success of generating

useful features for successful classification. After visual inspection of the graphed data, the

following conclusions were made on target and non-target response score patterns:

21 | P a g e

Targets:

Often shaped like an upside-down ‘U’

An upside-down ‘U’ contains the following mathematical properties:

o Perfectly symmetrical

o Completely convex

o Highest peak occurs at middle index

o Direction changes twice (i.e. slope at point of direction change = 0)

Non-Targets:

No consistent patterns

Mostly noise with no visible resulting artifacts detected

Based on mathematical properties listed above, this inspired the features which would be used at

an attempt for improved classification. Since the non-targets had no consistent visible pattern,

then it did not make sense to create features for non-targets. Instead, features based on the

patterns of the target data was used for generating features.

Below describes the theory and implementation behind the four features which were developed,

based on the visual properties of the target score patterns.

Feature #1: Symmetry

Given 33 data points, a feature is needed to return a score on how symmetric those data points

were. The range of the score this feature would return would be between 0 and 1, where 0 was

completely asymmetric and 1 was completely symmetric. Theoretically, a perfect upside-down

22 | P a g e

‘U’ would return a value of 1, and a perfectly linear line segment with a slope of 45° would

return a value of 0.

A mathematical representation of how this algorithm is expressed is:

1 −∑ [𝑥(𝑛) − 𝑥(33 − 𝑛)16

𝑛=1 ]

16, 𝑤ℎ𝑒𝑟𝑒 𝑥 𝑖𝑠 𝑡ℎ𝑒 𝑛𝑜𝑟𝑚𝑎𝑙𝑖𝑧𝑒𝑑 𝑠𝑒𝑡 𝑜𝑓 33 𝑑𝑎𝑡𝑎 𝑝𝑜𝑖𝑛𝑡𝑠

This allows the data to be immediately normalized between 0 and 1. The differences are then

taken between each data point’s respective data point, and the sum for all 16 combinations are

found, and divided by 16 to determine the average. Since a symmetric pair would expect a

difference of 0 (normalized), the average difference is then subtracted from 1, so that as the

symmetry of the data increases, then the value being returned also increases between the range of

0 and 1.

Feature #2: Convexity

Given 33 data points, a feature is needed to return how convex those data points were. The range

of the score this feature would return would be between 0 and 1, where 0 was completely

concave and 1 was completely convex. Theoretically, a perfect upside-down ‘U’ would return a

value of 1, a perfectly shaped “U” would return a value of 0, and a horizontal line would return a

value of 0.5.

A mathematical representation of how this algorithm is expressed is:

23 | P a g e

32 − [∑ [(𝑥(𝑛) > 𝑥(𝑛 + 1)) +

𝑥(𝑛) == 𝑥(𝑛 + 1)2

] + ∑ [(𝑥(𝑛) < 𝑥(𝑛 + 1)) +𝑥(𝑛) == 𝑥(𝑛 + 1)

2]32

𝑛=1615𝑛=1

32] ,

𝑤ℎ𝑒𝑟𝑒 𝑒𝑎𝑐ℎ 𝑏𝑜𝑜𝑙𝑒𝑎𝑛 𝑠𝑡𝑎𝑡𝑒𝑚𝑒𝑛𝑡 𝑟𝑒𝑡𝑢𝑟𝑛𝑠 𝑎 𝑣𝑎𝑙𝑢𝑒 𝑜𝑓 1 𝑤ℎ𝑒𝑛 𝑡𝑟𝑢𝑒, 𝑎𝑛𝑑 0 𝑤ℎ𝑒𝑛 𝑓𝑎𝑙𝑠𝑒

The logic for this algorithm is that for the first half of the data points, every time a proceeding

point increases in value, then that increases the concavity of the series of points. For the second

half of the data points, every time a proceeding point decreases in value, then that also increases

the concavity of the data. As well, any time a proceeding point is equal to the previous point,

then that makes the data series more neutral. In the above described event of a value increasing

the concavity of the series, a value of 1 is summed to a running total. A value of 0.5 is also added

to this total when the neutral case occurs. This total is then subtracted from the total maximum

pairs of points (32), and then divided by that same value of 32, in order to return a value between

0 and 1 for convexity.

Feature #3: Median-Max Index Difference

Given 33 data points, a feature is needed which detects how close the maximum point of the

series is to the middle of the plot. Therefore, this feature would be finding the difference between

the index of the median of the un-ordered numbers (17), and the index of the maximum point.

This would then need to be returned in the range of 0 and 1, where 0 represents the furthest

possible distance between the maximum index and the centre, and 1 represents the maximum

index being equal to the centre index. An example of 0 being returned would be data shaped like

a “U”, where the maximum points are either the first or last index of the series. An example of 1

being returned would be data shaped like an upside-down “U”, there the maximum value occurs

directly in the centre.

24 | P a g e

A mathematical representation of how this algorithm is expressed is:

17 − |17 − 𝑖𝑛𝑑𝑒𝑥(max(𝑥))|

17

This allows a normalized value between 0 and 1 to be returned, based on the conditions listed

above.

Feature #4: Direction Changes

Given 33 data points, a feature is needed to return a score on how many times the direction (or

sign of the slope), changes. The range of the score this feature would return would be between 0

and 32. A score of 0 would occur for any plot, where the slope between any two points remains

either positive or negative for all combination of points. A score of 32 would occur for a

completely jagged line, where the sign of the slope switches after every data point. It would not

make sense to normalize the score of this feature between 0 and 1, because this wouldn’t

accurately represent relative differences of scores where the number of changes are low. For

example, since for the perfect upside-down “U” shape that we are predicting target scores to look

like, there would be 1 direction change in this case. However, to normalize this score, we have to

assume that it’s possible that a score of 32 can be returned. In reality, the scores are more likely

to be around the 2-5 range, and so a difference of 1/32 between each direction change does not

do the score justice. This is why just the absolute value will be returned.

25 | P a g e

Given the complexity required for iteration through the points, it is difficult to write a proper

mathematical expression to represent the algorithm without writing pseudo code. The algorithm

for this feature can be found in the appendices, entitled “num_changes.m”.

Combining Features:

With the above four features that were created, plus including the given score from the original

classifier, we now have a five-feature set to use for classification. By choosing the combination

of these five features, new classifications can be made, which will be compared to the original

classification results. The results of this new classification with the generated features are shown

in the first part of the results section.

Procedure - Brute-Force Guessing

The second section of the methods performed for this thesis is entitled “brute-force guessing”.

The reason for this title is that the logic for how the modified classifier will determine which

character is the target-letter, will be through an approach similar to a brute force algorithm.

This is accomplished by implementing a utility which alters various marking channels for 72

iterations, so each iteration will leave only the data pertaining to that character for that iteration

only. In other words, for each letter being spelled, the utility will try and give a score for that

letter by guessing which letter it is. For example, if the utility is on the first iteration and

guessing that the target letter is an ‘A’, then data corresponding to the remainder of the letters in

26 | P a g e

the word are deleted so that the classification is focused on simply the letter being spelled in the

word, and the letter which the utility is guessing the target letter is.

One of the first tasks that must be done is to generate a structure which contains all of the flash

locations for a given character in a word. One of the sample data files being used was for the

spelling of the word ‘WADSWORTH’. For this data file, there were 72 possible characters

which were flashing in the matrix (8x9) to choose from. Each character flashed twice per

sequence, and five sequences occurred for the spelling of each letter in the word. Since the word

being spelled is 9 letters long, then there are (2 flashes/sequence * 5 sequences/letter * 9

letters/word =) 90 flashes for each character throughout the entire spelling of the word. With 90

flashes occurring for each character in a word, and 72 characters total, then the structure that

contains all the flash locations would be a 2-dimensional array of size 72x90.

A function was created to handle most of the logic for generating the above mentioned structure.

The utility being created is called ‘FlashDriver.m’, and the new function created is called

‘flash.m’. In FlashDriver, a structure needs to be created that simply contains the indices for

every flash occurrence, regardless of the letter. This is accomplished by parsing through the

structure sts.StimulusType and finding where an increase occurs, as this represents the

occurrence of a flash. The ‘flash’ function is called from FlashDriver, with three arguments

being passed: ‘letter’, ‘file’, and ‘AllFlashInx’. The argument ‘letter’ represents a character

(1:72), ‘file’ represents the file location of the data to be loaded form, and ‘AllFlashInx’

represents the structure mentioned above which contains the locations of all flashes. With this

information passed in, the ‘flash’ function can return the location of all (90) flashes that occur for

27 | P a g e

that given letter in a word of a given length (9). FlashDriver uses a loop to parse through all 72

characters in order to populate the 72x90 structure. One challenge of the ‘flash’ function was that

the data being referenced was obtained from 6 different channels. The data from each channel

was found in the variable ‘sts.Targ0x’, where x ranges for the integers between 1 and 6. The

variable ‘sts.Targ01’ contains all flashes occurring for the ‘lower’ characters, such that all

flashes for the letter ‘A’ are found only in sts.Targ01 and not in any of the other channels. The

character number 72 however would be found only in sts.Targ06, and not any of the previous

channels. Characters in the middle can be found in multiple channels. This is due to the fact that

6 letters are flashing at a time, and each channel focuses on only one of those letters during a

flash. So not only does the ‘flash’ function need to find the common elements between

AllFlashInx and sts.Targ0x==letter, but then also take the union of all these results from the

various sts.Targ0x structures so that it contains indices from all of these channels without

repetition. Once implemented correctly, the cardinality of this set will be the proper length, in

this case, 90. This set will then be what is returned to FlashDriver to populate the 72x90 structure

for all 72 characters. This 72x90 structure will be referred to as ‘flash_inx_letter’.

The remainder of the work being done in FlashDriver is the ‘cropping’ of various structures so

only the letter currently being spelled is focused. For example, in the event the ‘D’ in

‘WADSWORTH’ is being spelled, then the appropriate values in the ‘sts’ variables will need to

be cropped everywhere except for where it references the ‘D’ being spelled. The following

variables are what needed to be cropped for each iteration of letters in a word:

sts.PhaseInSequence

sts.StimulusBegin

28 | P a g e

sts.Targ0x (x = range of 1:6)

sts.StimulusType

This is accomplished by defining beginning (‘beg’), and ending (‘fin’), markers for which data to

keep for a given letter. In this case, the value of letter does not range from 1:72, but instead

ranges from 1 to the length of the word being spelled. In MATLAB, these index markers are

calculated using the following syntax:

beg = 1+find(sts.PhaseInSequence(1:end-1)~=1 &

sts.PhaseInSequence(2:end)==1); fin = find(sts.PhaseInSequence(1:end-1)==3 & sts.PhaseInSequence(2:end)~=3);

This allows the ‘beg’ markers to be indices where the value in sts.PhaseInSequence is not equal

to 1, and the proceeding index is equal to 1, as this indicates the beginning of a sequence. The

‘fin’ markers are the indices in sts.PhaseInSequence which are equal to 3, and the proceeding

index is not equal to 3, as this indicates the end of a sequence. This will generate a ‘beg’ 1-D

array of the length of the word and a ‘fin’ 1-D array also of the same length.

As for actually cropping the structures, consider the following 2 lines of MATLAB code:

sts.PhaseInSequence(1:beg(letter)-1)=0; sts.PhaseInSequence(fin(letter)+1:end)=0;

Using the ‘beg’ and ‘fin’ values that were defined, they will act as the markers for what will be

cropped, or zeroed out. Since ‘beg’ and ‘fin’ are the length of the word being spelt

(‘WADSWORTH’), the index of the ‘beg’ and ‘fin’ arrays correspond to the letter being spelled.

For example, beg(4) would refer to the beginning index of the 4th letter in a word. The first line

of the above code excerpt zeroes out all of the data in sts.PhaseInSequence that occurs before the

29 | P a g e

‘beg’ index. The following line then zeroes out all of the data in sts.PhaseInSequence that occurs

after the ‘fin’ index. What remains after those two lines are executed is the data that pertains only

what ‘letter’ represents.

The cropping shown above is repeated in the exact same manner for sts.StimulusBegin, and

sts.Targ0x (x ranging from 0 to 6). The remaining structure that must be modified is

sts.StimulusType. This is done in a slightly different manner. For sts.StimulusType, we begin by

zeroing out the entire structure right away:

sts.StimulusType(:) = 0;

Now to repopulate some of the data in sts.StimulusType, consider the following two lines of

MATLAB code:

sts.StimulusType(flash_inx_letter(guess,(1+(letter-1)*NumTargFlashes):

letter*NumTargFlashes))= 1;

sts.StimulusType(flash_inx_letter(guess,(1+(letter-1)*NumTargFlashes):

letter*NumTargFlashes)+1) = 1;

The repopulation of this structure is using the 72x90 ‘flash_inx_letter’ structure generated

before. The variable ‘guess’ that is shown above, refers to a value between 1:72 that is being

guessed in a brute-force manner, in pretending that the character which ‘guess’ represents is the

target letter. I.e. if guess==7, then we are pretending that the target letter is ‘G’, and making

classifications accordingly. ‘NumTargFlashes’ is equal to the total number of target flashes that

occur for a given word.

Most of the other implementation for this section of the Method deals with the storage and

organization of data as it interacts with the P300 GUI. In order to keep track of the ‘guess’ (i.e.

30 | P a g e

the character being guessed for being the target letter at each iteration), this variable is

incremented and stored in a reloadable structure every iteration. During each iteration, the score

that this ‘guess’ letter received is also appended to a structure that is added on to every iteration.

An iteration is defined as manually selecting “Generate Feature Weights” in the P300 GUI. This

process is repeated for all 72 characters of a letter, and that process is then repeated for all the

letters in a word.

In order to generate results for this section of the Method, the scores for all 72 characters for all 9

letters of the word ‘WADSWORTH’ will be examined. This will be accomplished by

normalizing the data for each letter, and then recording how the target character for each letter in

the spelled word ranked among the 71 non-targets.

Procedure - Modifying P300 Parameters

The final method of this thesis was inspired by the results of the previous method. While the

results of the previous method will be discussed in the next section, it brought up some

interesting ideas that could be further explored. Mainly, this dealt with modifying two of the

more crucial parameters of the P300 speller in order to see how that effects results. The first one

of these parameters is the number of flash sequences per letter being spelled. In the previous

method experiment, each target letter being spelled had 5 sequences per letter, with 2 flashes per

sequence, giving a total of 10 flashes per target letter. This parameter of 5 sequences per target

letter can be changed to any desired value. The second of the parameters to be modified is the

total number of characters from the array to choose from. In the previous method experiment, an

8x9 matrix was used for data acquisition, giving a total of 72 characters to choose from. By

31 | P a g e

increasing the number of flash sequences, and decreasing the P300 matrix size, it is hypothesized

that the classification results will dramatically improve More specifically, the number of flash

sequences per target letter will increase from 5 to 15, and the matrix size will decrease from 8x9

to 5x5. With the new matrix size, the letter ‘Z’ will be removed. While this would not be desired

in a real application, this new matrix size is simply used for research and better understanding

and a better understanding of classification techniques.

After the results for spelling the word ‘COMPUTER’ will be found in a similar way to the

previous method will address, one final inquiry will be further examined. This will examine how

successful the classifier is at spelling the target word as a whole after each individual sequence.

The results of that experiment will be beneficial in discovering exactly what the minimal number

of flash sequences are required for the successful spelling of each letter.

Modifying these two parameters for data acquisition can easily be done in the BCI2000 software.

However, the utility which will be performing the classification will need to be slightly modified

to handle these new parameters. Mainly, any loops that iterate 72 times for total characters will

need to be changed to 25, and any loops that iterate 5 times for sequences will need to be

changed to 15. The ‘flash.m’ function is what will need the most modifications. All of the

experiments conducted so far have been using the checkerboard-paradigm (CBP). Thus a 9x8

matrix containing 72 total characters, is virtually split into two 6x6 black and white boards.

However, a 5x5 matrix is more difficult to split into two boards since 25 is not evenly divisible

by two. The solution is one of the two virtual boards containing empty spaces, so that there is

still a total of 25 characters. However, this makes the sts.Targ0x variables slightly more

32 | P a g e

complicated. With the new 5x5 board, some flash groups will contain 3 flashes, and other flash

groups will contain 4 flashes. As a result, the sts.Targ0x variables that we care about are between

the range of x=1:4. But while sts.Targ04 will be fully populated, some of those values are

repeated from a previous flash group, and make it appear that there were always 4 flashes per

group, even though that’s not true. The useful variable to solve this problem is found within the

contents of sts.Targ00. The value found at each index of this structure indicates the number of

letters that flashed at a given index. Thus the values of this structure will either be 3 or 4 at the

index of a flash group. Therefore, sts.Targ04 can be modified so that the values of this structure

can be set to 0 whenever the value of sts.Targ00 is equal to 3 at the same indices. This can be

accomplished in a single line of MATLAB code which is shown below:

sts.Targ04(find(sts.Targ00~=4))=0;

One final modification that must be made is for ‘sts.StimulusBegin’. This is necessary for when

we want to examine the iterative process of removing the latest flash sequence, until there is only

one flash sequence remaining. These modifications will also occur for the sts.Targ0x variables.

New structures will be saved after a sequence is removed after each iteration. These structures

will then be normalized and ranked in classification scores in the same way as the previous

methods. The results of the classification will then be compared among each sequence.

33 | P a g e

Results

The below 3 sections contain results for each of the 3 methods explained in the previous section.

Results - Generating Features

In order to obtain the results from the various features generated, a script titled

‘classifier_driver.m’ was implemented. This script outputs separate averages for targets and non-

targets for each one of the scores given for all of the features. It also outputs the scores given

from the original classifier. Based on the output of this script, the below table displays these

results:

Feature Name Target Score Non-Target Score

Original Classifier 894.1154 -65.2690

Symmetry 0.6296 0.5630

Convexity 0.6215 0.7348

Median-Max Distance 0.6798 0.3714

Direction Changes 2.2444 2.9843

The original classifier features shows a substantial difference between the target and non-target

score, as suspected. The features that have the most interest are the bottom four, the ones which

were implemented.

34 | P a g e

The symmetry feature, median-max distance feature, and direction changes feature all gave

better target scores, than the non-target scores. Even though the target average for direction

changes was lower than the non-target average, this was expected and hoped for since it was

predicted the target average would be 1, given its shape was a perfect upside-down “U”.

The only feature which produced negative results was the convexity feature. The non-target

average for the convexity feature was 0.7248 while the target average for the same feature was

0.6215. This is likely due to the unpredictability of non-target brain activity. Both background

and biological noise can cause scores to vary, and thus form patterns that cannot be easily

predicted. In this case, it seems that these random scores seemed to match a convex pattern. If

this is found to be consistent, then the knowledge of this feature can help improve classifiers in

the future.

Results – Brute-Force Guessing

Using the guessing method similar to a brute force approach, the success of this classification

was determined using obtained data for spelling the word “WADSWORTH”. For this data being

used, there were 5 sequences (10 flashes) per letter, with a total of 72 characters to choose from.

35 | P a g e

Below is a table outlining how each target letter was ranked out of the 72 possible characters for

each letter in the target word.

Target Letter Rank (out of 72)

W 1

A 8

D 26

S 7

W 32

O 15

R 34

T 49

H 24

As promising as this classification method looked after the first letter, the overall results were

fairly disappointing. Besides the ‘W’, none of the other letters were even a close contender to

being the chosen target-letter with this approach. On the other hand, a thought that should also be

considered is the importance of ranks when it’s not 1st. In terms of the end application, the

chosen letter is either right or wrong, so how important is it how close or far off it is from being

1st? This idea will be further explored in the discussion section.

Results – Modifying P300 Parameters

The results of this final method focused on spelling the newly generated data for the word

“COMPUTER”. This is different from the previous method, since the number of sequences

increased from 5 to 15, and the number of characters to choose from on the P300 matrix

decreased from 72 to 25.

36 | P a g e

Similar to the previous experiment results, below is a table outlining how each target letter was

ranked out of the 25 possible characters for each letter in the target word.

Target Letter Rank (out of 25)

C 1

O 1

M 1

P 1

U 2

T 1

E 1

R 1

It is clear from these results that by incrementing the flash sequences and decrementing the size

of the target array, this classification method worked very well. Every one of these target letters

was ranked 1st, with the exception of the letter ‘U’. While the ‘U’ was ranked 2nd, the following

graph displays how close ‘U’ (letter 21) was from being given a score higher than the winner

(letter 6):

37 | P a g e

It was not a surprise that the results obtained from this experiment were a drastic improvement

from the previous experiment, given the substantial change in its parameters. However, an

interesting component of this experiment that was to be investigated was how the results change

when sequences are removed. The iterative removal of the last sequence simulates how the

subject would have spelled the word if they were given fewer sequences (i.e. fewer flashes), per

target letter.

The below table and graph show how many letters correct (out of a possible 8), the classifier

guessed correctly after each sequence. This begins after the first sequence, and goes until

sequence number 15.

Sequence # # Letters Correct (out of 8)

1 0

2 1

3 0

4 3

5 2

6 4

7 2

8 3

9 3

10 5

11 5

12 7

13 7

14 6

15 7

38 | P a g e

Further diagrams in the appendices will show the ranks of all the target letters in “COMPUTER”

after each sequence. However, one diagram that should be analyzed now is the diagram depicting

the rank of the target letter ‘U’, since that was the letter which was incorrectly classified after the

15th sequence.

39 | P a g e

Below is the graph of the target letter ‘U’, and the rank (out of 25) it was given after each

sequence:

As displayed in the above graph, the ‘U’ was classified as the target letter (rank 1) after the 6th

sequence. The ‘U’ remained the winner at rank 1 for the following 8 sequences. It was then at the

15th sequence, that the ‘U’ become ranked #2. This is unfortunate since it would be expected that

the “confidence” or reliability of the classifier would continue to increase as the number of

sequences increase. However, for some reason this wasn’t the case for this particular target letter.

Possible reasons for this will be examined in the discussion section.

40 | P a g e

One final display of results for this experiment will show the actual word that was spelled after

each sequence. This will display realistic results in that this is what would actually be output by

the P300 speller for each sequence. This also allows focus to be on which letters were

consistently guessed correctly throughout the sequences. The display of these results are shown

below:

Sequence # Word Guessed (for ‘COMPUTER’)

1 UHIWTQXS

2 KLCGKOSR

3 KGKWIVTI

4 CBMWQTHC

5 CDMUNDDP

6 OTMEUTCR

7 JPEXUTHV

8 CUEXUTHA

9 LUMEUMWR

10 CMMUUTOR

11 CTMUUTWR

12 COMTUTER

13 CUMPUTER

14 CEMPUTTR

15 COMPFTER

Based on a visual inspection of the above table, it is clear that the word being spelled does not

begin to correctly form until after about the 10th sequence. Letters such as the ‘C’, “M’, ‘T’, and

‘R’ become consistently correct guesses after this point. However the ‘U’ is the only letter to

have been consistently correctly guessed for so many consecutive sequences, and then be

replaced by an incorrect guess. The number of appropriate sequences will be further examined in

the discussion section.

41 | P a g e

Discussion

This discussion section will be broken down into the three areas that have been outlined in the

Methods and Results sections. While these areas will be discussed separately, a joined

conclusion will be made in the Conclusion section to represent the research and experiments of

the entire paper.

Discussion – Generating Features

As shown in the Results section for ‘Generating Features’, 3 out of the 4 features generated and

implemented showed positive results. While these results were positive, they were not ground-

breaking in the sense that this would be improving the existing classifier. These were 4 features

that were manually implemented based on a visual inspection of the data. In reality, the existing

classifier chooses from thousands of possible features, and then applies weights to the features

that it wants in order generate strong results to perform an accurate classification. It would be

very difficult to manually implement generated features and expect those features to both

accurately and consistently classify this large amount of data. In reality, more focus should be

drawn to further operations and calculations that can be performed on the output of the existing

classifier, and using those results to try and improve the final classification. More than anything,

this initial experiment was helpful in better understanding the interaction between features and

the classifiers, and provide a solid direction in what future experiments can be conducted in an

attempt to improve the classification of target and non-target responses.

42 | P a g e

Discussion - Brute-Force Guessing

The results obtained from the initial brute-force guessing method were quite disappointing. With

only 1 out of 8 letters being spelled correctly, this method is nowhere near plausible for success

as a needed application. Since the first character was spelled correctly with great results, a

significant amount of time was spent attempting to detect a bug that was preventing the

remainder of the letters being spelled correctly. Specifically, there was curiosity on whether there

was a miscalculation in how the indices for cropping out data were derived, but unfortunately no

such bug or error was found.

With this brute force approach for guessing the correct target letter, each ‘guess’ had appropriate

features and generated weights created to make this ‘guess’ the classified winner. It was hoped

that even though most of the 72 characters would be shown as winners depending on their

appropriate weights and features, that the real target letter would be a much clearer winner than

the other ‘fake winners’. For example, consider the letter ‘W’ in ‘WADSWORTH’. In the brute

force approach, all 72 characters were made to look like they were the actual target letter.

However, what made the ‘W’ stand out as the clear winner was that even once all the scores

were normalized, the target score for the ‘W’ was so good that it stood out from all of the other

winners. Theoretically, despite most of the 72 characters being initially classified as the target

letter, the real target letter would still stand out from all of the other guessed letters, similar to

how the ‘W’ was classified as the target letter. If this worked for the ‘W’, then why did it fail for

the remainder of the letters? The reason is not because the real target letter was scored poorly for

each letters in the target word. The real reason is that the features and weights generated to make

43 | P a g e

the other 72 characters look like the target letter was done so successfully, that it was difficult to

differentiate between all of these plausible winners because they were all scored so well. The

hope was that even if all the guessed letters were scored highly, the real target letter would stand

out, but this was not the case since all of the guessed letters were scored just as well as the real

target letter.

Despite the above method not being successful in making a clear classification of the target letter

and non-targets, it was successful in eliminating several choices. Out of a rank of 72 choices, the

average rank for target letters in the word ‘WADSWORTH’ was around 25. While this is clearly

nowhere close to consistently guessing the letter correctly, it does consistently eliminate about

two thirds of the possibilities. This is slightly comforting, but at the same time, does it really

matter how close the guess was to being correct? Regardless of whether the guessed target letter

was ranked #2 or #72, in the end the guess was incorrect. If the subject misspells a letter, they are

likely not to care what the incorrect guess was. From their view point, the guessed letter was

either correct or incorrect. With that being said, even though the guessed letters in

‘WADSWORTH’ were ranked in the top third of possible characters, from the subject’s

standpoint it only guessed 1 out of 9 letters correctly. The only usefulness of knowing how close

in rank the real target letter was from being the chosen classified letter is being able to measure

the improvements or detriments in the classifier. If the classifier continues to be modified, even

if the end result in terms of the number of correctly guessed letters does not change, small

improvements can be detected by knowing that the target letter is getting closer to being the

classified letter.

44 | P a g e

In its current implementation, this brute force method does not work under standard conditions.

Future research could try and find unique features or techniques to improve this method. As a

proof of concept to show that this method has the potential to theoretically succeed as a

classification technique, the experiment was repeated with new data and parameters This will be

analyzed in the next section of the discussion to determine the overall effect modifying two

important parameters of the P300 had on the success for classifying targets and non-targets using

the brute-force approach.

Discussion – Modifying P300 Parameters

Due to the unsatisfactory results of the brute force method in the previous conditions, the

experiment as repeated with new test data and a modification in two P300 parameters: number of

flash sequences, and size of speller matrix. The results of this modification made a substantial

difference, as 8 out of the 9 letters were guessed correctly. The changes in parameters were quite

substantial, so it was expected to receive positive results. As mentioned in the previous section,

this was done as more of a proof of concept, but it does create a few discussion areas. One of

these is how realistic these new parameters have towards the real-life application of the P300

speller.

The first parameter that was modified which needs to be discussed is the size of the target matrix.

In the previous experiment, an 8x9 matrix containing 72 characters was used. The reduction of

45 | P a g e

the matrix to size 5x5 containing 25 characters was crucial in improving results. Of course, this

is due to there being less competition that the real target letter is facing for all of the non-target

letters. An initial thought of this experiment questioned why 72 characters was necessary to

begin with for real-life application. If a subject suffers from late stages of ALS and is using the

P300-speller as a method of communication, it is not likely necessary for the matrix board to

contain characters such as ‘Ctrl’ or “Pg. Down’. For this case, it would make more sense to

contain just the letters of the alphabet, since they are relying on this as the only form of

communication, and there would be no point in overcomplicating this process. However, in the

case where the subject using the P300-speller as a way modifying a document in a word

processor, then all of the extra options on the matrix may be necessary. In either case, if the

strength and accuracy of the classifier is high enough, then the target letters would be

consistently guessed in both situations anyway.

The second modified parameter to consider is increasing the number of target flashes from 10 to

30. It is intuitive that increasing the number of flashes per target letter will also increase the

reliability of the classification of targets and non-targets. The drawback however is the amount

of time that is required to now spell each letter. It would not make sense to force a subject to sit

through 30 flashes per target letter if another classification method can do it in far fewer flashes.

While each subject will require a different amount of target flashes to correctly spell a letter, it

makes more sense to over-estimate the amount of flashes required than it is to under-estimate

this amount. By under-estimating, the success of the classification could vary and cause the P300

to spell out gibberish. By over-estimating the required number of flashes, it may be worth the

extra amount of time when using the P300 speller if this will ensure successful results.

46 | P a g e

Therefore, this balance must be considered when deciding the number of target flashes required

by each individual subject.

The latter half of this experiment focused on a similarity to the discussion above. Using the same

data and results generated from this experiment (i.e. spelling the word “COMPUTER”), the exact

effect of the number of flash sequences required for successful classification was explored.

When the word “WADSWORTH” was being spelled, 5 sequences of flashes (2

flashes/sequence) was used, resulting in 1/9 letters spelled correctly. When the word

“COMPUTER” was spelled, 15 sequences of flashes (2 flashes/sequence) was used, resulting in

7/8 letters spelled correctly. However, the success of the spelling of this second experiment can

be broken down so that it can become apparent how well the word would have been spelled had

there been less flash sequences. For example, if only 5 flash sequences were used for spelling

“COMPUTER”, then only 2 out of the 8 letters in the word would have been spelled correctly at

this point. This improvement is very minimal to the first experiment, and the only reason for

eventual success was that 10 more flash sequences followed this.

It has already been discussed that increasing the number of flash sequences should consequently

increase the reliability of the classification. However, this was not the case for the letter ‘U’ in

“COMPUTER” since it become classified as a non-target at flash sequence #15, despite being

classified as the target letter for the previous 8 flash sequences. This is a bit of an anomaly, since

it disagrees with the theory of classification improving, given more data. One possible

explanation is that due to the subject having to concentrate for 30 flashes per letter when spelling

47 | P a g e

the target word, distraction could become more difficult to hold for a longer period of time. A

solution to this could be to have the number of flash sequences be dynamic, and calculated at

run-time of the speller. In the case of the letter ‘U’, if the classifier realizes that after 3 flash-

sequences the target letter has not changed, then it could consider the possibility of locking into

this choice and moving onto the next letter without having to complete the rest of the flash

sequences. Since the brute force method was shown to not be very successful to begin with, the

data obtained from this experiment would not be ideal to use to test this new hypothesis, but

could be worth exploring in future work.

Conclusion

Several conclusions can be drawn based on the three main experiments that were conducted. The

first conclusion is that successful classification of targets and non-targets using generated

features is a very complicated task. While manually creating features based on a visual

inspection gave positive results when contrasting the difference of scores for targets and non-

targets, this would not be near enough to base the success an entire classifier on. The existing

classifier is able to choose from a pool of thousands of different features, and then apply various

weights to the hundreds of features that it selects, in order to generate good results. Trying to

manually reproduce this task with implemented features is not only near-impossible, but goes

outside the scope of this thesis. The purpose of that experiment was to become comfortable with

how the classifier works, and to understand the steps involved in generating intelligent features.

The second conclusion that can be made is that the brute force method approach of guessing the

target letter was unsuccessful under standard conditions. Using an 8x9 matrix and 5 flash

48 | P a g e

sequences/letter, successfully classifying only 1/9 letters is nowhere near ideal. The concept of

this approach was interesting enough to conduct this experiment, but in the end was simply not

conclusive in showing any improvements in classification. This led directly into the third

experiment which discussed the importance of various parameters of the P300 speller, and the

impacts that modifying those parameters may have. While for this final experiment it was

beneficial to modify these parameters in order to obtain ideal results, in reality this would not

make sense as it is unjustifiable to hinder aspects of the speller in order to obtain positive results

from a classification method that was already proven to be unsuccessful. An overall conclusion

from the results of these experiments suggest that there is still much room for improvements to

be made in the classification of targets and non-targets for the P300-speller, both in accuracy as

well as number flash sequences required for successful classification.

Future Work

Future work on this topic could continue to explore new methods or techniques for improving

class separation. Specifically, the technique of data clustering could be worth exploring in order

see how well target and non-target responses cluster, as a means of separation. Future

consideration could also focus on how well the 2nd and 3rd ranked letters are scored relative to the

winners. More specifically, experiments could be conducted to detect if there is a larger gap in

the scores of the winner and runner-up of guessed characters for target responses, than there are

for non-target responses. One final improvement that could be made for future work could be on

improving automation of the P300 GUI. Due to many modifications made to the functionality of

the classifier, there was a lot of manual work done in order to calculate results when using the

49 | P a g e

GUI. A substantial amount of time could be saved for future result generation and analysis if the

P300 GUI application was slightly modified to directly fit the needs of the work that is being

done.

50 | P a g e

References

[1] D. McFarland, J. Wolpaw, “Brain-Computer Interfaces for Communication and Control,”

in Communications of the ACM. vol. 53, no.5, pp 60-66, 2015.

[2] A. Momennezhad, M. Shamsi, H. Ebrahimnezhad, H. Saberkari, “Classification of EEG-

P300 Signals Extracted from Brain Activities in BCI Systems using ʋ-SVM and BLDA

Algorithms,” in Applied Medical Informatics. vol. 32 no. 2, pp 23-35, 2014.

[3] C. King, P. Wang, L. Chui, A. Do, Z. Nenadic, “Operation of a Brain-Computer Interface

Walking Simulator by Users with Spinal Cord Injury,” in arXiv. 2015.

[4] U. Herwig, P. Satrapi, C. Schonfeldt-Lecuona, “Using the International 10-20 EEG

system for positioning of transcranial magnetic stimulation,” in Brain Topography. vol.

16, no. 2, pp 95-99, 2015.

[5] Z. Wu, “Studying modulation on simultaneously activated SSVEP neural networks by a

cognitive task,” in Journal of Biological Physics, no. 1, pp 55, 2014.

[6] J. Lu, W. Speier, X. Hu, N. Pouratian, “The effects of stimulus timing features on P300

speller performance,” in Clinical Neurophysiology. vol. 124, pp 306-214, 2013.

[7] J. Mattout, M. Perrin, O. Bertrand, E. Maby, “Update article: Improving BCI

performance through co-adaptation: Applications to the P300-speller,” in Annals of

Physical and Rehabilitation Medicine. 2014.

51 | P a g e

[8] G. Townsend, B. LaPallo, C. Boulay, D. Krusienski, G. Frye, C. Hauser, N. Schwartz, T.

Vaughan, J. Wolpaw, E. Sellers, “A novel P300-based brain-computer interface stimulus

presentation paradigm: Moving beyond rows and columns,” in Clinical Neurophysiology.

vol. 121, pp 1109-1120, 2010.

[9] L. McCane, S. Heckman, D. McFarland, G. Townsend, J. Mak, E. Sellers, D. Zeitlin, L.

Tenteromano, J. Wolpaw, T. Vaughan, “P300-based brain-computer interface (BCI)

event-related potentials (ERPs): People with amyotrophic lateral sclerosis (ALS) vs. age-

matched controls,” in Clinical Neurophysiology: Official Journal of the International

Federation of Clinical Neurophysiology. 2015.

[10] ALS Association, “Quick Facts about ALS & The ALS Assocation,” at

www.alsa.org/news/media/quick-facts.html. 2012.

[11] J. Wolpaw, E. Wolpaw, “Brain-Computer Interfaces: Principles and Practice,” in Oxford

University Press. 2012.

[12] I. Guyon, A. Elisseeff, “An Introduction to Variable and Feature Selection,” in Journal of

Machine Learning Research. vol. 3, no. 7/8, pp 1157-1182, 2003.

[13] M. Webber, “Unsupervised Learning of Models for Object Recognition,” at

http://www.vision.caltech.edu/publications/MarkusWeber-thesis.pdf. 2000.

52 | P a g e

Appendix A - Graphs

Character Rank vs. Data Sequence number for ‘C’ in “COMPUTER”

53 | P a g e

Character Rank vs. Data Sequence number for ‘O’ in “COMPUTER”

Character Rank vs. Data Sequence number for ‘M’ in “COMPUTER”

54 | P a g e

Character Rank vs. Data Sequence number for ‘P’ in “COMPUTER”

Character Rank vs. Data Sequence number for ‘U’ in “COMPUTER”

55 | P a g e

Character Rank vs. Data Sequence number for ‘T’ in “COMPUTER”

Character Rank vs. Data Sequence number for ‘E’ in “COMPUTER”

56 | P a g e

Character Rank vs. Data Sequence number for ‘R’ in “COMPUTER”

57 | P a g e

Appendix B – Code

symmetry.m

% this function returns a value between 0 and 1, % where 0 represents complete asymmetry of the points % and 1 represents complete symmetry. function [sym_value] = symmetry(targs) x = targs(1,:); x=x-min(x); x=x/max(x); sym_value = sum(abs(fliplr(x(1:16))-x(18:end))); sym_value = sym_value/16; sym_value = 1-sym_value; end

convexity.m

% this function returns value between 0 and 1 for convexity % of target array. 0 represents concave, and 1 represents % convex. the function assumes that target is convex by % giving a score of 33/33. Every time that a preceding point % to the left of the median decreases, so does the score. % Every time that the succeeding point to the right of the % median increases, then the score decreases. % The value from 0-1 is calculated by dividing the score by % the number of targets (33).

function [convex_value] = convexity(targs) sample = targs(1,:); num_samples = 33; convex_value = num_samples;

for i = (num_samples-1)/2 : 2 if sample(i-1) > sample(i) convex_value = convex_value - 1; end if sample(i-1) == sample(i) convex_value = convex_value - 0.5; end end

for j = (num_samples-1)/2 : num_samples - 1 if sample(j+1) < sample(j) convex_value = convex_value - 1; end if sample(j+1) == sample(j) convex_value = convex_value - 0.5; end end convex_value = (convex_value-1) / (num_samples-1); end

58 | P a g e

med_max_distance_function.m

% this function returns a value between 0 and 1, % where 0 represents the maxiumum distance between % the median index and maximum value indices of a sample, and 1 % represents no distance. this is calculated by finding % the index of the maxiumum value of a sample, and % calculating the difference between that and the median % index of a sample. this is subtrated from the median index, % and divided by the median index to return the result % between 0 and 1. function [med_max_distance_value] = med_max_distance(targs) sample = targs(1,:); num_samples = 33; median = 17;

max_targ_value = max(sample); max_targ_index = find(sample == max_targ_value);

distance = abs(median - max_targ_index); med_max_distance_value = (median-distance)/median; end

num_changes.m

% this returns the number of changes in direction the graph of % a sample takes.

function [num_changes_value] = num_changes(targs) sample = targs(1,:); num_samples = 33; num_changes_value = 0;

if sample(2)>sample(1) dir = 1; else dir = 0; end

for i = 2 : (num_samples-1) if dir == 1 if sample(i+1)<sample(i) num_changes_value = num_changes_value + 1; dir = 0; end else if sample(i+1) > sample(i) num_changes_value = num_changes_value + 1; dir = 1; end end end end

59 | P a g e

classifier_driver.m

% clear; close all; figure(1); hold on; cd('C:\Users\Connor Flood\Documents\School Work\4th Year\BCI Thesis\jitter'); load;

%mscore contains 266 items of targets, and 2926 items of non-targets. Each %item contains 33 values, or raw features, where the median (17), was the %value of classifier output used to decide if an item was target or not.

mscore(:,34) = []; targs = mscore(trigs,:); %targs contains all target items nontargs = mscore(nons,:); %nontargs contains all non-target items

% ---------------------------- % EXISTING FUNCTION BELOW % ---------------------------- for i=1:length(targs) control_targ(i) = targs(i); end

for i=1:length(nontargs) control_nontarg(i) = nontargs(i); end

control_targ = control_targ'; control_nontarg = control_nontarg'; targ_control_avg = mean(control_targ); targ_control_avg nontarg_control_avg = mean(control_nontarg); nontarg_control_avg

% plot(targs(:,17),control_targ,'b.'); % plot(nontargs(:,17),control_nontarg,'r.');

% ---------------------------- % SYMMETRY FUNCTION BELOW % ---------------------------- for i=1:length(targs) sym_targ(i) = symmetry(targs(i,:)); end

for i=1:length(nontargs) sym_nontarg(i) = symmetry(nontargs(i,:)); end

sym_targ = sym_targ'; sym_nontarg = sym_nontarg'; targ_sym_avg = mean(sym_targ); targ_sym_avg nontarg_sym_avg = mean(sym_nontarg);

60 | P a g e

nontarg_sym_avg

% plot(targs(:,17),symtarg,'b.'); % plot(nontargs(:,17),symnontarg,'r.');

% ---------------------------- % CONVEXITY FUNCTION BELOW % -----------------------------

for i=1:length(targs) convex_targ(i) = convexity(targs(i,:)); end

for i=1:length(nontargs) convex_nontarg(i) = convexity(nontargs(i,:)); end

convex_targ = convex_targ'; convex_nontarg = convex_nontarg'; targ_convex_avg = mean(convex_targ); targ_convex_avg nontarg_convex_avg = mean(convex_nontarg); nontarg_convex_avg

% plot(targs(:,17),convex_targ,'b.'); % plot(nontargs(:,17),convex_nontarg,'r.');

% ---------------------------- % MED_MAX_DISTANCE FUNCTION BELOW % ---------------------------- for i=1:length(targs) med_max_distance_targ(i) = med_max_distance(targs(i,:)); end

for i=1:length(nontargs) med_max_distance_nontarg(i) = med_max_distance(nontargs(i,:)); end

med_max_distance_targ = med_max_distance_targ'; med_max_distance_nontarg = med_max_distance_nontarg'; targ_med_max_distance_avg = mean(med_max_distance_targ); targ_med_max_distance_avg nontarg_med_max_distance_avg = mean(med_max_distance_nontarg); nontarg_med_max_distance_avg

% plot(targs(:,17),med_max_distance_targ,'b.'); % plot(nontargs(:,17),med_max_distance_nontarg,'r.');

% ---------------------------- % NUM_CHANGES FUNCTION BELOW % ----------------------------

61 | P a g e

for i=1:length(targs) num_changes_targ(i) = num_changes(targs(i,:)); end

for i=1:length(nontargs) num_changes_nontarg(i) = num_changes(nontargs(i,:)); end

num_changes_targ = num_changes_targ'; num_changes_nontarg = num_changes_nontarg'; targ_num_changes_avg = mean(num_changes_targ); targ_num_changes_avg nontarg_num_changes_avg = mean(num_changes_nontarg); nontarg_num_changes_avg

% plot(targs(:,17),num_changes_targ,'b.'); % plot(nontargs(:,17),num_changes_nontarg,'r.');

% ----------------------------------- % 2-DIMENSIONAL FUNCTION COMPARISON % ----------------------------------- % plot(symtarg, convex_targ, 'b.'); % plot(symnontarg, convex_nontarg, 'r.');

% ----------------------------------- % MDBC Trainer % ----------------------------------- % targ_class=[control_targ, sym_targ, convex_targ, ... % med_max_distance_targ, num_changes_targ]; % nontarg_class=[control_nontarg, sym_nontarg, convex_nontarg, ... % med_max_distance_nontarg, num_changes_nontarg];

targ_class=[control_targ, sym_targ, convex_targ, ... med_max_distance_targ, num_changes_targ]; nontarg_class=[control_nontarg, sym_nontarg, convex_nontarg, ... med_max_distance_nontarg, num_changes_nontarg];

[m,ico]=mdbc_train(targ_class , nontarg_class);

num_correct_targ = 0; sum_correct_targ = 0; sum_incorrect_targ = 0; for i=1:266 keep_targ=(1:266); keep_targ=setdiff(keep_targ,i); % now "keep" represents all targets

except for target #i [m,ico]=mdbc_train(targ_class(keep_targ,:),nontarg_class); %train with

all targets except #i res_targ(:,i)=mdbc_classify(m,ico,targ_class(i,:)); % test on target #i

and report the two probabilities into res(:,i) if (res_targ(1,i) >= res_targ(2,i)) num_correct_targ = num_correct_targ + 1; end sum_correct_targ = sum_correct_targ + res_targ(1,i); sum_incorrect_targ = sum_incorrect_targ + res_targ(2,i);

62 | P a g e

end

num_wrong_targ = 266 - num_correct_targ; percent_correct_targ = num_correct_targ/266 * 100; percent_correct_targ

average_correct_targ = sum_correct_targ/266; average_incorrect_targ = sum_incorrect_targ/266; average_correct_targ average_incorrect_targ

sort_correct_targ = sort(res_targ(1,:)); sort_incorrect_targ = sort(res_targ(2,:)); median_correct_targ = sort_correct_targ(266/2); median_incorrect_targ = sort_incorrect_targ(266/2);

median_correct_targ median_incorrect_targ

num_correct_nontarg = 0; sum_correct_nontarg = 0; sum_incorrect_nontarg = 0; for i=1:2926 keep_nontarg=(1:2926); keep_nontarg=setdiff(keep_nontarg,i); % now "keep" represents all targets

except for target #i [m,ico]=mdbc_train(nontarg_class(keep_nontarg,:),targ_class); %train with

all targets except #i res_nontarg(:,i)=mdbc_classify(m,ico,nontarg_class(i,:)); % test on

target #i and report the two probabilities into res(:,i) if (res_nontarg(1,i) >= res_nontarg(2,i)) num_correct_nontarg = num_correct_nontarg + 1; end sum_correct_nontarg = sum_correct_nontarg + res_nontarg(1,i); sum_incorrect_nontarg = sum_incorrect_nontarg + res_nontarg(2,i); end

num_wrong_nontarg = 2926 - num_correct_nontarg; percent_correct_nontarg = num_correct_nontarg/2926 * 100; percent_correct_nontarg

average_correct_nontarg = sum_correct_nontarg/2926; average_incorrect_nontarg = sum_incorrect_nontarg/2926; average_correct_nontarg average_incorrect_nontarg

sort_correct_nontarg = sort(res_nontarg(1,:)); sort_incorrect_nontarg = sort(res_nontarg(2,:)); median_correct_nontarg = sort_correct_nontarg(2926/2); median_incorrect_nontarg = sort_incorrect_nontarg(2926/2);

median_correct_nontarg median_incorrect_nontarg

63 | P a g e

FlashDriver.m

%utility for locating flash indices and cropping appropriate areas of %various structures to focus on specific letters load(fname); sig = eeg; num_letters = size(prm.TargetDefinitions,1);

trials=length(find(sts.PhaseInSequence(1:end-1)==1 &

sts.PhaseInSequence(2:end)==2)); NumTargFlashes=length(find(sts.StimulusType(1:end-

1)<sts.StimulusType(2:end)))/trials;

AllFlashInx=find(sts.StimulusBegin(1:end-1)<sts.StimulusBegin(2:end))+1; for i=1:num_letters flash_inx_letter(i,:) = flash(i, fname, AllFlashInx); end

word_length = length(prm.TextToSpell{1}); increment = length(AllFlashInx)/word_length; letter = 1; start_inx = (letter-1)*increment +1; end_inx = letter*increment;

beg = 1+find(sts.PhaseInSequence(1:end-1)~=1 &

sts.PhaseInSequence(2:end)==1); fin = find(sts.PhaseInSequence(1:end-1)==3 & sts.PhaseInSequence(2:end)~=3);

sts.PhaseInSequence(1:beg(letter)-1)=0; sts.PhaseInSequence(fin(letter)+1:end)=0;

sts.StimulusBegin(1:beg(letter)-1)=0; sts.StimulusBegin(fin(letter)+1:end)=0;

sts.Targ00(1:beg(letter)-1)=0; sts.Targ00(fin(letter)+1:end)=0;

sts.Targ01(1:beg(letter)-1)=0; sts.Targ01(fin(letter)+1:end)=0;

sts.Targ02(1:beg(letter)-1)=0; sts.Targ02(fin(letter)+1:end)=0;

sts.Targ03(1:beg(letter)-1)=0; sts.Targ03(fin(letter)+1:end)=0;

sts.Targ04(1:beg(letter)-1)=0; sts.Targ04(fin(letter)+1:end)=0;

sts.Targ05(1:beg(letter)-1)=0; sts.Targ05(fin(letter)+1:end)=0;

sts.Targ06(1:beg(letter)-1)=0; sts.Targ06(fin(letter)+1:end)=0;

64 | P a g e

if exist('guess.mat', 'file') load guess else guess=0; save guess guess end

guess = guess +1; save guess guess

sts.StimulusType(:) = 0; sts.StimulusType(flash_inx_letter(guess,(1+(letter-

1)*NumTargFlashes):letter*NumTargFlashes))= 1; sts.StimulusType(flash_inx_letter(guess,(1+(letter-

1)*NumTargFlashes):letter*NumTargFlashes)+1) = 1;

flash.m

%Function which returns a vector of indices for all of the occurences %of 'flashes' for a given letter function [LetterFlashInx] = flash(letter, file, AllFlashInx) load(file); %/AllLetterFlashInx1 = find((sts.Targ01==letter)&AllFlashInx);

% sts.Targ04 = int16( sts.Targ04 & sts.Targ00>=4);

sts.Targ04(find(sts.Targ00~=4))=0;

for i=1:length(sts.Targ04) if sts.Targ00(i)<4 sts.Targ04(i) = 0; end end

AllLetterFlashInx1 = find(sts.Targ01==letter); AllLetterFlashInx2 = find(sts.Targ02==letter); AllLetterFlashInx3 = find(sts.Targ03==letter); AllLetterFlashInx4 = find(sts.Targ04==letter);

%indices for all values of a letter occuring on a flash FlashInx1 = intersect(AllFlashInx, AllLetterFlashInx1); FlashInx2 = intersect(AllFlashInx, AllLetterFlashInx2); FlashInx3 = intersect(AllFlashInx, AllLetterFlashInx3); FlashInx4 = intersect(AllFlashInx, AllLetterFlashInx4);

%union of all 6 targets for a letter LetterFlashInx = union(FlashInx1, FlashInx2); LetterFlashInx = union(LetterFlashInx, FlashInx3); LetterFlashInx = union(LetterFlashInx, FlashInx4);

65 | P a g e

% BELOW FOR 9X8 GRID % % %indices for all values of a letter % %/AllLetterFlashInx1 = find((sts.Targ01==letter)&AllFlashInx); % AllLetterFlashInx1 = find(sts.Targ01==letter); % AllLetterFlashInx2 = find(sts.Targ02==letter); % AllLetterFlashInx3 = find(sts.Targ03==letter); % AllLetterFlashInx4 = find(sts.Targ04==letter); % AllLetterFlashInx5 = find(sts.Targ05==letter); % AllLetterFlashInx6 = find(sts.Targ06==letter); % % %indices for all values of a letter occuring on a flash % FlashInx1 = intersect(AllFlashInx, AllLetterFlashInx1); % FlashInx2 = intersect(AllFlashInx, AllLetterFlashInx2); % FlashInx3 = intersect(AllFlashInx, AllLetterFlashInx3); % FlashInx4 = intersect(AllFlashInx, AllLetterFlashInx4); % FlashInx5 = intersect(AllFlashInx, AllLetterFlashInx5); % FlashInx6 = intersect(AllFlashInx, AllLetterFlashInx6); % % %union of all 6 targets for a letter % LetterFlashInx = union(FlashInx1, FlashInx2); % LetterFlashInx = union(LetterFlashInx, FlashInx3); % LetterFlashInx = union(LetterFlashInx, FlashInx4); % LetterFlashInx = union(LetterFlashInx, FlashInx5); % LetterFlashInx = union(LetterFlashInx, FlashInx6); % % % hold on; % % plot(sts.Targ01) % % plot(AllFlashInx,0,'r.') % % plot(LetterFlashInx,0,'black.')

end

P3Classify.m (added modifications)

function

[predicted,result,score,resultthresh,tar]=P3Classify(Responses,Code,Type,MUD,

NumberOfSequences,SetsPerSequence,trialnr,NumMatrixRows,NumMatrixColumns,char

vect,wind,states,extra);

lenflash=length(states.Flashing); ind=find(states.Flashing(1:lenflash-1)==0 &

states.Flashing(2:lenflash)==1)+1; % from the state information, make the following: % targets(len,MAXFLASH) showing the targets currently flashing % flashgroup(len,NumMatrixRows*NumMatrixColumns) showing target flash NumTargets=NumMatrixRows*NumMatrixColumns; MAXFLASH=19; THRESHRATIO=NumTargets/(NumMatrixRows+NumMatrixColumns); NumInFlash=states.Targ00(ind); len=length(NumInFlash); flashgroup=zeros(len,MAXFLASH); targets=zeros(len,NumTargets);

66 | P a g e

for t=1:MAXFLASH ts=num2str(t); if length(ts)<2 ts=['0',ts]; end flashgroup(:,t)=states.(['Targ',ts])(ind); flashgroup(NumInFlash<t,t)=0; for f=1:NumTargets targets(flashgroup(:,t)==f,f) = 1; end end

windowlen=wind(2)-wind(1); numresponse=size(Responses,1); numchannels=size(Responses,3);

MUD(:,2)=MUD(:,2)-wind(1)+(MUD(:,1)-1)*windowlen;

nsamp=size(MUD(:,2),1); response1=reshape(Responses,numresponse,numchannels*windowlen); pscore=response1(:,MUD(:,2)+1)*MUD(:,3); if exist('pscore_rec.mat', 'file') load pscore_rec; end

load guess

pscorex(:,guess) = pscore;

Typex(:,guess) = Type;

save pscore_rec pscorex Typex; fprintf(1, 'Classifying Responses...\n\n'); %savemscore;where-1 % test for finding peaks % avg responses by row/col choices=NumTargets; numletters=max(trialnr); NumTargFlashes=length(Type(Type>0))/numletters;

cflash=zeros(choices,NumTargFlashes); score=zeros(numletters,choices); lrv=zeros(1,choices);

cc=1;

for bb=1:numletters cflash=zeros(choices,NumTargFlashes); range=find(trialnr==bb); if ~isempty(range) code=targets(range,:); code(Type(range)==0,:)=[]; [junk,code]=max(sum(code));

67 | P a g e

if code>0 targ(cc)=code; for aa=1:choices rv=pscore(range(find(targets(range,aa))));

lrv(aa)=length(rv);

cflash(aa,1:NumTargFlashes)=rv;

cflash(aa,:)=cumsum(cflash(aa,:)); end

numseq(bb)=min(lrv);

score(cc,:)=cflash(:,NumTargFlashes);

[valt,maxtarg]=max(cflash); predictedtarg(cc,:)=maxtarg;

cc=cc+1; end end %------------------------------------------- % Dynamical Thresholding codes %------------------------------------------- for ii=1:NumTargFlashes auxtarg=cflash(:,ii); sorted=sort(auxtarg); m=mean(sorted(1:(end-1))); sd=std(sorted(1:(end-1))); ci(bb,ii)=(sorted(end)-m)/sd; end; %------------------------------------------------------- %savepscore; %cmd=0;errors; end

tar=charvect(targ); targ=repmat(targ',1,single(NumTargFlashes)); numcorrect=[predictedtarg==targ]; predicted=charvect(predictedtarg); [rrr,ccc]=size(predictedtarg); if rrr==1 predicted=predicted'; end

%cmd=1;errors; %----------------------- % Alain's modification %------------------------ numcorrectmat=numcorrect; %------------------------- if size(numcorrect,1)>1

68 | P a g e

numcorrect=sum(numcorrect); end

result=numcorrect/double(cc-1)*100;

guiout=fopen('gui_result.txt','a'); fprintf(guiout,'Timestamp-> %04d-%02d-%02d

%02d:%02d:%02d\r\n\r\n',fix(clock)); fprintf(guiout, ' Target: %s \r\n',tar); fprintf(guiout, '\r\n Flashes | %% Correct | Predicted Symbols' ); fprintf(guiout, '\r\n --------|-----------|------------------\r\n' ); fprintf(1, ' Target: %s \n',tar); fprintf(1, '\n Flashes | %% Correct | Predicted Symbols' ); fprintf(1, '\n --------|-----------|------------------\n' ); for kk=1:size(result,2); pred=predicted(:,kk)'; rind=find(numseq<kk); if ~isempty(rind) for uu=1:length(rind) pred(rind(uu))=' '; end end fprintf(guiout, ' %02d | %03d%% | %s

\r\n',kk,round(result(kk)),pred); fprintf(1, ' %02d | %03d%% | %s

\n',kk,round(result(kk)),pred); end fprintf(guiout, '\r\n\r\n') fprintf(1, '\n\n') fclose(guiout);

%----------------------------------- % George's modification %----------------------------------- fprintf('--------------------------------------------------------------------

-------------------------\n'); fprintf(' Dynamical Target Flashes \n'); fprintf('%s \t\t %s \t\t %s \t\t %s \t\t %s\n','Flash', 'ConfInt', 'WSR %max'

, 'No Feedback', 'Feedback (wrong=~,none=BLANK)'); fprintf('--------------------------------------------------------------------

-------------------------\n');

for ff=2:NumTargFlashes threshvec=sort(ci(:,ff)); threshvec=[0;(threshvec(1:end-

1)+threshvec(2:end))/2;max([ceil(ci(:,ff)*2);100.0])]; bestWSR=-1.0; for ii=1:length(threshvec) truepos=length(find(predictedtarg(:,ff)==targ(:,ff) &

ci(:,ff)>threshvec(ii))); falsepos=length(find(predictedtarg(:,ff)~=targ(:,ff) &

ci(:,ff)>threshvec(ii))); noreports=length(find(ci(:,ff)<=threshvec(ii))); if double(numletters) == noreports perr=1.0;

69 | P a g e

timefact=double(numletters); % NOT SURE ABOUT THIS factor

representing time wasted on no-reports else perr=(falsepos)/(double(numletters)-noreports); timefact=double(numletters)/(double(numletters)-noreports); %

factor representing time wasted on no-reports end if perr>=0.5 perr=0.499999; end selfact=1.0/(1-2*perr); % factor representing selections required to

make one good selection percentWSR=100.0/(selfact*timefact)/(double(ff)/2); %p=1-perr; %n=NumMatrixRows*NumMatrixColumns; %br=log2(n)+p.*log2(p)+(1-p).*log2((1-p)/(n-1)); %bx=br/log2(n); % symbols per trial %bx=bx/selfact; %percentWSR=100.0*bx/(double(ff)/2); if percentWSR>bestWSR; bestWSR=percentWSR; bestii=ii; noreps=noreports; str=charvect(predictedtarg(:,ff))'; str(predictedtarg(:,ff)~=targ(:,ff))='~'; str((ci(:,ff)<=threshvec(ii)))=' '; end end; if bestWSR<=0 bestWSR=0; end resultthresh(ff-1,:)=[double(ff),threshvec(bestii),bestWSR]; fprintf('%2d\t\t\t\t%4.2f\t\t\t%5.2f\t\t\t%2d\t\t\t<%s>\n', ff,

threshvec(bestii),bestWSR,noreps,str); end;

Classify_Driver.m

%script to determine the rankings of target letters in a target word %and produce graph and statistics on that data word = 'COMPUTER'; for seq=1:15 for letter=1:length(word) file = sprintf('%s_%s_%02d',word,word(letter),seq); [score,winner] = new_function(file); targ_letter = abs(word(letter)) - 64; [junk,I]=sort(-score); place=find(I==targ_letter);

winners(letter,seq) = winner; sequence_letter_places(letter,seq) = place; end end

70 | P a g e

for letter=1:length(word) min_flash(letter) = 30; for seq=15:-1:1 x = sequence_letter_places(letter,seq); if x == 1 min_flash(letter) = seq; else break; end end end

for letter=1:length(word) figure plot(sequence_letter_places(letter,:)) end hold off; figure

for seq=1:15 num_correct(seq) = 0; for letter=1:length(word) if sequence_letter_places(letter,seq)==1 num_correct(seq) = num_correct(seq) + 1; end end end

for seq=1:15 word_guess_build = ' '; for letter=1:length(word) letter_guess = char(winners(letter,seq)+64); word_guess_build = strcat(word_guess_build,letter_guess); end word_guess_seq(seq,:) = word_guess_build; end

word_guess_seq plot(num_correct)

new_function.m

%function which returns the normalized score for a target letter, %and the score of the winning letter function [normalized_cumulative_score, winner] = new_function(saveScore) load(saveScore); x = size(pscorex); if x(2) ~= 75 pscorex(:,25) = 0; Typex(:,25) = 0; end

for letter=1:25 norm_factor=mean(pscorex(pscorex(:,letter)>0,letter)); pscorex(:,letter)=pscorex(:,letter)/norm_factor;

71 | P a g e

normalized_cumulative_score(letter)=sum(pscorex(find(Typex(:,letter)),letter)

); end winner =

find(normalized_cumulative_score==max(normalized_cumulative_score(:))); end

xpre.m

%script which makes appropriate crops to sts.StimulusBegin in order to %remove data sequences load Computer2

for i=14:-1:1 fin = find(sts.PhaseInSequence(1:end-1)==3 &

sts.PhaseInSequence(2:end)~=3)+1; for j=1:8 beg = find(sts.StimulusBegin(1:end-1)< sts.StimulusBegin(2:end)); beg = beg(beg<fin(j)); beg = beg(end-14); sts.StimulusBegin(beg:fin(j))=0; sts.StimulusType(beg:fin(j))=0;

for k=0:6 cmd=sprintf('sts.Targ%02d(beg:fin(j))=0;',k); eval(cmd); end end save(sprintf('Computer2-%02d',i),'sig','sts','prm'); end