Adaptive Stimulation Design for the Treatment of Epilepsy Joelle Pineau School of Computer Science, McGill University Montreal, QC CANADA Jointly with.

Adaptive Stimulation Design for the Treatment of Epilepsy

Joelle PineauSchool of Computer Science, McGill University

Montreal, QC CANADA

Jointly with Robert Vincent, Aaron Courville, Massimo Avoli

SAMSI Program on Challenges in Dynamic Treatment Regimes and Multistage Decision-Making

June 21, 2007

SAMSI 2007: Adaptive Stimulation Design for the Treatment of Epilepsy

Background

Vagus nerve and deep brain stimulation are used to treat various neurological disorders, including epilepsy.

Images from www.epilepsyfoundation.org and www.cyberonics.com


Project goal

• Problem: Existing devices offer limited control and do not adapt to the patient’s condition.

• Idea: Create an improved class of devices with closed-loop control.


A reinforcement formulation

• Objective: Minimize occurrence of seizures and overall amount of stimulation.

• The MDP model:

– States, st : recordings of electrical activity

– Actions, at : stimulation (frequency, voltage, location)

– Transitions, P(st|st-1, at) : unknown

– Rewards, rt : large cost for seizures, small cost for stimulation

st-1

at at+1

…

rt-1

st

rt

st+1

rt+1

at-1


Technical challenges

1. Investigate supervised learning methods for automatic seizure detection

to inform choice of good state representation.

2. Design a computational (generative) model of epilepsy.

3. Run reinforcement learning using online data from the computational

model.

4. Run reinforcement learning using batch data from an in-vitro model.


Recordings of electrical activity

• Recorded from single sensing electrode in in-vitro model of epilepsy.

• Raw data: 4096-sample frames windowed, normalized, FFT

• xt = 83 real-valued features: mean, range, energy, 80 FFT magnitudes

• yt ={normal, spike, seizure} hand-labeled for each frame.


Seizure detection

• Long literature on seizure detection using supervised learning (e.g. RBFs, wavelets, neural networks, energy methods).

» Usually uninterpretable results.

• Related literature in time-series prediction (e.g. HMMs, CRFs).

» Requires feature selection.

• Instead, focus on ensemble boosting methods:

– Extend standard multi-class Adaboost [Schapire&Singer, 1999] to a

recurrent formulation, in which yt = f (xt , yt-1 , yt-2 , …, yt-k).

» Relatively interpretable results (assuming simple learners).

» Requires no prior model of the distribution of features.


Classification accuracy

yt = f (xt , yt-1 )


Feature extraction = insight into state design

• In all recurrent examples, first weak hypothesis recruited was frequency band 62 or 63 (= 76-77 Hz).

– High value normal

– Low value spike

• Frequency bands 6-8 (=7-10 Hz) also consistently recruited early.

– High value spike

– Low value normal

• Often recruited in the first 20 rounds is Energy.

– High value spike

• In recurrent Adaboost, prior label often recruited early and acts as memory.

– High spike prior spike

etc.







model.



A computational model of epilepsy

• Aims of computational modeling:

– To understand basic mechanisms of epilepsy sufficiently to design

a good state representation.

– To understand the appropriate class of policies to consider.

– To have an inexpensive testing environment for RL algorithms.

• A word of caution:

– Epilepsy is a complex disease. Many animal models are found in

the literature. We focus on model by [Avoli et al., 2002], which

we will use for online exploration in-vitro.


Model overview

(a) Assume a simple stochastic neuron model (leaky integrate-and-fire).

(b) Connect many neurons in small-world network configuration.

(a) Leaky integrate-and-fire neuron model (b) Network structure


Model overview (cont’d)

(c) Define a sensor model:

– Voltage measurement over a patch of adjacent neurons (neuron

contribution falls off in inverse-square relationship with distance

from the patch center).

(d) Define the stimulation model:

– Input current applied uniformely to a “patch” of neural units.

istim

vsensor


Traces from the computational model

# neurons

firing

sensor

voltage


Results from fixed stimulation strategies

0 Hz

0.5 Hz

1.0 Hz

2.0 Hz

4.0 Hz

5.0 Hz







model.



Reinforcement learning agent

• States:

st = {sensor reading} x {# time steps since last stimulation}

discretized into small number of (independent) features

• Actions:

a1 = Stimulator on for 1 time-step.

a2 = Stimulator off for 1 time-step.

• Rewards:

R(s,a) = -100 for a seizure (i.e. firing count > 50)

R(s,a) = -10 for a stimulation

R(s,a) = 0 otherwise

• Training details:» Online data, finite horizon (60 x 100-sec traces), -greedy exploration, Sarsa()

with eligibility


Results: Learning

(a) After 1 learning episode (b) After 10 learning episodes


Results: Expected return


Results: Policy







model.



Learning from batch in-vitro data

We have data of the type used in the classification task, but which

includes fixed-policy stimulation.

• States: 83 real-valued features (electrical signal sampled + FFT)

• Actions: {0 Hz, 0.5 Hz, 1.0 Hz, 2.0 Hz, 5.0 Hz}

• Training: batch (~30 x 60-sec traces), tree-based regression.

No results


Wrap-up

• Preliminary evidence for:

– ability to detect seizures using in-vitro data.

– generation of synthetic data with epileptiform behavior.

– controllability of the stimulator in the computational model.

• Results apply to a specific model of epilepsy - generalization to other models is unknown.

• Issues I did not discuss today:

– Designing the reward function.

– Using the features found in Adaboost within the RL agent.

– Transferring what we have learned in the computational model to the biological model.

– Learning from very few data points.


Available data

• Recordings used for classification (no stimulation)

– http://www.cs.mcgill.ca/~jpineau/datasets/epilepsy.tar.gz

• Recordings used for batch reinforcement learning (fixed policy stimulation)

– Subject to approval by neuroscientists (1-2 weeks delay).

• Generative model of epilepsy

– Still under development, but publicly available soon (1-2 months).

Adaptive Stimulation Design for the Treatment of Epilepsy Joelle Pineau School of Computer Science, McGill University Montreal, QC CANADA Jointly with.

Documents

stimulation model

adaptive stimulation

stimulation s t

vitro model of epilepsy

treatment of epilepsy

ps t s t

stimulation frequency

state design