Detection of K-complexes and Sleep Spindles (DETOKS) using ...€¦ · DETOKS by applying it on an online EEG database in Section 5. Speci cally, we compare spindle and K-complex

Detection of K-complexes and Sleep Spindles (DETOKS) using Sparse Optimization

Ankit Parekhalowast Ivan W Selesnickb David M Rapoportc Indu Ayappac

aDept of Mathematics School of Engineering New York UniversitybDept of Electrical and Computer Engineering School of Engineering New York University

cDept of Medicine Division of Pulmonary Critical Care and Sleep Medicine School of Medicine New York University

Abstract

Background - This paper addresses the problem of detecting sleep spindles and K-complexes in human sleep EEGSleep spindles and K-complexes aid in classifying stage 2 NREM human sleepNew Method - We propose a non-linear model for the EEG consisting of a transient low-frequency and an oscillatorycomponent The transient component captures the non-oscillatory transients in the EEG The oscillatory componentadmits a sparse time-frequency representation Using a convex objective function this paper presents a fast non-linearoptimization algorithm to estimate the components in the proposed signal model The low-frequency and oscillatorycomponents are used to detect K-complexes and sleep spindles respectivelyResults and comparison with other methods - The performance of the proposed method is evaluated using anonline EEG database The F1 scores for the spindle detection averaged 070 plusmn 003 and the F1 scores for the K-complexdetection averaged 057 plusmn 002 The Matthews Correlation Coefficient and Cohenrsquos Kappa values were in a range similarto the F1 scores for both the sleep spindle and K-complex detection The F1 scores for the proposed method are higherthan existing detection algorithmsConclusions - Comparable run-times and better detection results than traditional detection algorithms suggests thatthe proposed method is promising for the practical detection of sleep spindles and K-complexes

Keywords Sparse signal convex optimization sleep spindle detection K-complex detection

1 Introduction

Sleep spindles comprise of a group of rhythmic wavesthat progressively increase and decrease in amplitude [1]They are of at least 05 seconds in duration and have fre-quencies in the range of 12 Hz to 14 Hz [2] Recent studieshave suggested an extended frequency range from 11 Hz to16 Hz [3 4] Sleep spindles are believed to play an impor-tant role in synaptic plasticity and memory consolidationduring sleep [5] Alteration in the density of sleep spindlesis observed in patients with disorders such as schizophrenia[6] [7] autism [8] and other neurodegenerative and sleepdisorders [9] This leads to mounting belief that sleep spin-dles may be valuable as diagnostic biomarkers [4]

The K-complex is a transient waveform with a bipha-sic morphology characterized by a negative sharp wavefollowed by a positive one [10] The K-complex is a rela-tively large waveform having a duration between 05 and15 seconds with an amplitude larger than 75 microV

The detection of sleep spindles and K-complexes aidin the scoring of stage N2 of NREM sleep Traditionallythese morphologically distinct waveforms have been iden-tified manually by trained experts in sleep clinics This issubjective time consuming and prone to errors The low

lowastCorresponding author Email address ankitparekhnyuedu

and widely varying inter-rater agreement for sleep spin-dle and K-complex detection adds to the complexity ofthe overall scoring process and diagnostic utility The Co-henrsquos κ coefficient for inter-rater manual scoring rangesfrom 046 to 089 [11] Some studies have reported aneven lower κ coefficient [12 13] Solving the above issuesrequire a reliable automated detector for sleep spindles andK-complexes

11 Detection Algorithms

Numerous automated detectors have been developedover the past few years for detecting sleep spindles andK-complexes At the core of most of the spindle detectionalgorithms is the use of a constant or adaptive thresh-old after bandpass filtering the input EEG [7 14 15 16]Few detectors have been designed to simultaneously detectspindles and K-complexes whereas most are designed todetect one or the other [17] The bandpass filter is alsoexcited by transients present in the input EEG Much ofthe effort of these algorithms is to either pre-process [18]or post-process [19] the bandpass filtered data so as todistinguish spindles from transients

To reduce this overhead algorithms have been designedusing neural networks [20] sleep spindle morphology [21]support vector machines [22] time-frequency methods viathe short-time Fourier transform (STFT) [13 19] and adap-

Preprint submitted to Elsevier April 3 2015

tive time-frequency methods [23 24] The drawback of us-ing machine learning methods is that they may suffer fromover-learning This increases the number of falsely detectedspindles as compared to scoring by experts (based on thespecificity values) [20] A non-linear pre-processor for theEEG using convex optimization was shown to successfullyseparate the transients [25] However that approach iscomputationally inefficient as it utilizes two STFTrsquos withhigh overlap between consecutive windows

The Teager-Kaiser energy operator (TKEO) has beenused for the automatic detection of K-complexes in con-juction with the time-frequency and neural network meth-ods highlighted above [26 10 27] The TKEO is helpfulin extracting the sharp rising and falling edges of the K-complex However the presence of transients adverselyaffects the performance of these algorithms as the TKEOis not able to successfully extract the K-complex activity inthe input EEG signal The spindle and K-complex activitycan be made more prominent by suppressing transients inthe EEG However this suppression is difficult using linearfilters thereby motivating non-linear methods [28]

12 Contribution

In this paper we propose a non-linear method for thedetection of sleep spindles and K-complexes (DETOKS)We model the input EEG as a sum of three components

1) Transient The transient component is modeled asa sparse signal possessing a sparse first-order derivativeEssentially the transient component is comprised of spikeson a baseline of zero

2) Low-frequency The low-frequency component of theEEG signal

3) Oscillations The rhythmic oscillations in the EEGsignal that admit a sparse time-frequency representation

To estimate the three components from the input EEGsignal we propose an optimization problem utilizing a con-vex objective function and also derive a fast algorithmfor its solution Post estimation of the components weuse the low-frequency and the oscillatory components forthe detection of K-complexes and sleep spindles respec-tively Since the proposed method separates the transientsfrom the low-frequency and oscillatory components usedfor detection the bandpass filter reveals the spindle activ-ity much more prominently with respect to the baselineThus the proposed approach aims to make traditionalspindle detection methods robust by prefacing them withnon-linear transient removal

We list the preliminaries in Section 2 and formulatethe DETOKS problem in Section 3 We illustrate the sup-pression of the transients in the input EEG with variousexamples in Section 4 We evaluate the performance ofDETOKS by applying it on an online EEG database inSection 5 Specifically we compare spindle and K-complexdetection results using the proposed DETOKS method andexisting automated detectors [14 15 7 29 16 13 12]

2 Preliminaries

21 Notation

We denote vectors and matrices by lower and uppercase letters respectively The N -point signal y is repre-sented by the vector

y = [y(0) y(N minus 1)]T y isin RN (1)

where [middot]T represents the transpose The `1 and `2 normof the vector y are defined as

||y||1 =sumn

|y(n)| ||y||2 =

(sumn

|y(n)|2)12

(2)

The matrix D is defined as

D =

minus1 1

minus1 1

minus1 1

(3)

Using the matrix D the first order difference of an N -point discrete signal x is given by Dx Here D is of size(N minus 1) times N The soft-threshold function [30] for λ gt0 λ isin R is defined as

soft(x λ) =

xminus λx

|x| |x| gt λ

0 |x| le λx isin C (4)

The soft-threshold function as defined in (4) is valid forcomplex valued x The notation soft(x λ) implies thatthe soft-threshold function is applied element-wise to xwith a threshold of λ The Teager-Kaiser Energy Operator(TKEO) T (middot) for a discrete-time signal y is defined as

[T (y)]n = y2(n)minus y(nminus 1) middot y(n+ 1) (5)

22 Sparse Optimization

To estimate a signal x possessing a sparse or approxi-mately sparse derivative from a noisy mixture

y = x + w (6)

it is common to minimize the `1 norm of the first-order dif-ference Dx subject to a data fidelity constraint [31] Thusa suitable optimization problem for estimating x is

x = arg minx

1

2||y minus x||22 + λ||Dx||1

(7)

The value λ gt 0 controls the sparsity of the derivative ofx where the `1 norm is used as a convex proxy for sparsityThe problem in (7) is known as total variation denoising(TVD) whose solution in linear time is given in [32] If x isalso sparse ie it comprises spikes on a baseline of zero

2

then an appropriate optimization problem is given by

x = arg minx

1

2||y minus x||22 + λ1||x||1 + λ2||Dx||1

(8)

This problem is known as the fused lasso [33] [34] and itssolution is given by

x = soft(tvd(y λ2) λ1) (9)

where tvd(middot middot) represents the solution to the TV denoisingproblem (7)

23 Short-time Fourier Transform

Signals that admit a sparse representation can be de-scribed by a small number of coefficients using an appropri-ate transform Such representations when they exist canaccount for most of the energy contained in a signal [35]Recently over-complete transforms have been exploited forthe sparse representation of signals [36] A signal consist-ing of oscillatory pulses can be represented sparsely usingthe STFT The STFT requires specification of several pa-rameters such as window length overlapping factor andthe discrete Fourier transform (DFT) length

In this work we use a window length of 128 seconds forthe STFT and a DFT length equal to the window lengthThis choice of window length ensures that the DFT lengthis a power of 2 when the sampling frequency is 50 Hz100 Hz or 200 Hz In case the EEG is sampled at 128Hz we use a one-second window We use 75 overlappingbetween windows ie a hop size of one quarter of thewindow length Hence the STFT is 4-times over-sampled

Precisely using a time-frequency array for STFT coef-ficients of size MtimesK for a signal y of length N we defineΦ CMtimesK 7rarr CN as

Φc = STFTminus1(c) (10)

whereas ΦH CN 7rarr CMtimesK is defined as

ΦHy = STFT(y) (11)

Using a sine window we implement the STFT to have theperfect reconstruction property specifically

ΦΦH = I (12)

where ΦH represents the Hermitian transpose of Φ [37]

3 Simultaneous Detection using DETOKS

31 Problem Formulation

We model the EEG signal as

y = f + x + s + w f x sw isin RN (13)

where f represents a low-frequency signal x is a sparse sig-nal with sparse first-order derivative s consists of rhytmic

oscillations and is sparse with respect to Φ and w repre-sents the residual This kind of a signal model is similarto the one used for transient removal and suppression in[38 25] In contrast to [25] we model the transient com-ponent using a sparse first-order derivative rather than anSTFT

We seek estimates for the components x f and s fromthe given signal y in (13) The component s can be mod-eled as

s = Φc (14)

where c isin CMtimesK is the STFT coefficient array Using alowpass filter L we define the highpass filter H as

H = Iminus L (15)

assuming that the frequency response of the lowpass filteris zero-phase or at least approximately zero-phase [38] Fora highpass filter having a 2dminusorder zero at z = 1 thematrix H is of size (N minus 2d)timesN when applied to a signalof length N Applying the highpass filter to the signalmodel in (13) we have

H(y minus xminus Φc) asymp w (16)

In order to estimate the components x and c from a giveny and minimize the energy of the residual w we proposethe following unconstrained optimization problem

arg minxc

1

2||H(y minus xminus Φc)||22 + λ0||x||1

+ λ1||Dx||1 + λ2||c||1 (17)

The objective function in (17) promotes the sparsity ofthe signal x its first-order derivative Dx and the STFTcoefficient array c using the `1 norm The scalars λ0 λ1and λ2 are regularization parameters We set the highpassfilter H to be a zero-phase recursive discrete-time filterthat we write as

H = Aminus1B (18)

where A and B are banded1 Toeplitz matrices as describedin Sec VI of [38] Note that A is of size (N minus 2d)times (N minus2d) and B is of size (N minus 2d) times N when the 2dminusorderhighpass filter is applied to a signal of length N Thebanded structure of A and B aids in the computationalefficiency of the algorithm that will be developed in thenext sub-section

32 AlgorithmVarious problems arising in signal processing and bio-

medical engineering are formulated as convex optimization

1A banded matrix is a type of sparse matrix whose non-zeroentries are restricted on the main diagonal and one or more diagonalson either side of the main diagonal

3

problems The objective function in (17) is also convexand can be minimized via convex optimization algorithmsThere exists a wealth of algorithms for solving problemsof type (17) with two or more sparsity inducing penalties[39 40] In particular we apply Douglas-Rachford split-ting [41] to solve (17) The Douglas-Rachford splittingapproach results in the well-known alternating directionmethod of multipliers (ADMM) [42 43] Applying vari-able splitting we rewrite (17) as

arg minu1u2xc

1

2||H(y minus u1 minus Φu2)||22 + λ0||x||1

+ λ1||Dx||1 + λ2||c||1

(19a)

st u1 = x u2 = c (19b)

Note that (19) is equivalent to (17) Using the scaled aug-mented Lagrangian [44] we can minimize (19) by the fol-lowing iterative procedure

Repeat

u1u2 larr arg minu1u2

1

2||H(y minus u1 minus Φu2)||22

+micro

2||u1 minus xminus d1||2 +

micro

2||u2 minus cminus d2||22

(20a)

x clarr arg minxc

λ0||x||1 + λ1||Dx||1 + λ2||c||1

+micro

2||u1 minus xminus d1||2 +

micro

2||u2 minus cminus d2||22

(20b)

d1 larr d1 minus (u1 minus x) (20c)

d2 larr d2 minus (u2 minus c) (20d)

where micro gt 0 The minimization in (20b) is separable thusx and c can be minimized independently Thus we canwrite (20b) as

xlarr arg minx

λ0||x||1 + λ1||Dx||1 +

micro

2||u1 minus xminus d1||2

(21a)

clarr arg minc

λ2||c||1 +

micro

2||u2 minus cminus d2||22

(21b)

The solutions to (21a) and (21b) are readily implementedusing the solution to the fused lasso problem as describedin Sec 22 and the soft threshold function (4) Specificallythe solutions to (21a) and (21b) are given by

xlarr soft(tvd(u1 minus d1 λ1micro) λ0micro) (22)

clarr soft(u2 minus d2 λ2micro) (23)

To derive the solution to (20a) we make the following

substitutions

u = [u1u2]T d = [d1d2]T (24a)

x = [x c]T M = [I Φ] (24b)

where u d x isin R2N Due to the substitutions we rewrite(20a) as

ularr arg minu

1

2||Hy minusHMu||22 +

micro

2||uminus xminus d||22

(25)

The solution to (25) can be written explicitly as

ularr[MT HT HM + microI2N

]minus1 [(MT HT Hy + micro(x + d)

]

(26)

where I2N is the (2N times 2N) identity matrix Due to theform (18) of the highpass filter H and the perfect recon-struction property (12) of Φ we have

HT H = BT (AAT )minus1B (27)

MMT = 2I (28)

Using the matrix inverse lemma [45] (27) and (28) wewrite(

MT HT HM + microI2N

)minus1

=

(MT BT (AAT )minus1BM + microI2N

)minus1

(29)

=1

micro

(I2N minusMT BT

[microAAT + 2BBT

]minus1

BM

) (30)

The matrix microAAT + 2BBT is banded since the matricesA and B are banded This is advantageous as the itera-tion now consists of solving a banded system of equationsrather than a dense system in (29) Combining (30) and(26) we obtain the following procedure for solving (20a)

Glarr(microAAT + 2BBT

)(31a)

g1 larr1

microBT (AAT )minus1By + (x + d1) (31b)

g2 larr1

microΦBT (AAT )minus1By + (c + d2) (31c)

u1 larr g1 minus BT Gminus1B(g1 + Φg2) (31d)

u2 larr g2 minus ΦHBT Gminus1B(g1 + Φg2) (31e)

Note that the vector (1micro)BT (AAT )minus1By is used in ev-ery iteration Since the signal y is not updated in eachiteration we need only compute it once before the loopCombining the routines discussed above we obtain theDETOKS algorithm After we obtain the estimates forx and c we calculate the signal components s and f as

4

DETOKS algorithm

inputs

y isin RN micro gt 0 λi gt 0 i = 0 1 2

hlarr (1micro)BT (AAT )minus1By

repeat

Glarr(microAAT + 2BBT

)g1 larr h + x + d1

g2 larr Φh + (c + d2)

u1 larr g1 minus BT Gminus1B(g1 + Φg2)

u2 larr g2 minus ΦHBT Gminus1B(g1 + Φg2)

xlarr soft(tvd(u1 minus d1 λ1micro) λ0micro)

clarr soft(u2 minus d2 λ2micro)

d1 larr d1 minus (u1 minus x)

d2 larr d2 minus (u2 minus c)

until convergence

slarr Φc

f larr (y minus xminus s)minusAminus1B(y minus xminus s)

return x s f

follows

slarr Φc (32)

f larr (y minus xminus s)minusAminus1B(y minus xminus s) (33)

where we use the highpass filter defined in (15)Figure 1 portrays the decomposition of an EEG sig-

nal into the signal components f s and x Notably thelow-frequency component f contains the K-complex Theoscillatory component s contains the sleep spindles Thenon-oscillatory transients are contained in the sparse com-ponent x

33 Detection of sleep spindles and K-complexes

The oscillatory component s is used to detect sleepspindles We bandpass filter s to remove any non-spindleoscillations using a 4th order Butterworth filter with apassband of 115 Hz to 155 Hz We denote this bandpassfiltered signal as BPF(s) The TKEO T (middot) as defined in(5) is then applied to the signal BPF(s) Using a constantvalue (c1) for the threshold a binary signal bspindle(t) isdefined as

bspindle(t) =

1 T(BPF(s)) gt c1

0 T(BPF(s)) le c1(34)

Any detected spindle of duration less than 05 seconds isrejected We allow the maximum duration of a detectedspindle to be 3 seconds as in [4]

For detecting K-complexes we apply the TKEO tothe low-frequency component f and define a binary signal

673 674 675 676 677 678

minus10

10

minus10

10

minus50

0

50

minus10

10

minus50

0

50

Time (sec)

Amplitude(micro

V)

EEG (y)

Transient (x)

Lowminusfrequency (f)

Oscillatory (s)

Residual (w)

Figure 1 Decomposition of the raw EEG into the signal componentsx f and s using the DETOKS algorithm

EEG DETOKS

Bandpassfilter

x

TKEO Threshold

TKEO ThresholdSleep

Spindle

K-complex

s

f

Proposed detection method

Figure 2 The detection of sleep spindles and K-complexes using theDETOKS algorithm

bK-complex(t) as in (34) with a constant threshold value(c2) as

bK-complex(t) =

1 T(f) gt c2

0 T(f) le c2(35)

The detected K-complexes of duration less than 05 sec-onds are rejected

The DETOKS algorithm uses f and s to detect sleepspindles and K-complexes simultaneously in sleep EEGFigure 2 depicts the proposed detection method Figures 3and 4 show the detection of a sleep spindle and K-complexrespectively using DETOKS The example EEG shown inFig 3 is different from the one in Fig 4 as the expertannotation for K-complexes was not available for the EEG

5

673 674 675 676 677 678

01

0

20

minus10

10

0

40

Time (s)

Am

plitu

de

Expert scored spindle Raw EEG

BPF(s)

TKEO applied to BPF(s)

Detected spindle bspindle

Figure 3 The detection of spindles using a Butterworth bandpassfilter (BPF) and the TKEO Operator The value of the constantthreshold (c1) was fixed at 003

80 81 82 83 84 85

01

0

20

0

50

minus50

0

50

Time (s)

Am

plitu

de

Raw EEG

Lowminusfrequency (f)

TKEO applied to (f)

bKminuscomplex

Expert annotated Kminuscomplex

Detected Kminuscomplex

Figure 4 The detection of K-complex using the TKEO The valueof the constant threshold (c2) was fixed at 1

in Fig 3

4 Examples

We demonstrate the suppression of transients and de-tection of sleep spindles and K-complexes using DETOKSby applying it to the C3-A1 channel of an EEG We com-pare the performance of DETOKS for sleep spindle de-tection with the methods proposed by Wendt et al [14]and by Martin et al [15] We refer the reader to [4] for asummary of their detection processes

Recall that the DETOKS algorithm calls for the specifi-cation of the parameters λ0 λ1 λ2 and micro The parametersλ0 and λ1 influence the sparse nature of the component xand its derivative Dx respectively Similarly λ2 controlsthe time-frequency sparsity of the oscillatory componentFor the examples in Fig 1 and the ones that follow weuse λ0 = 06 λ1 = 7 and λ2 isin [75 85] The parame-ters λ0 λ1 and λ2 were set empirically to ensure that the

5 10 15 20 25 30 35 40 45 501

2

3

Number of iterations

05|

|H(y

minusxminus

Φc)

|| 22 + λ

0||x|| 1 +

λ1||D

x|| 1 +

λ2||c

|| 1

micro = 1

micro = 05

micro = 01

Figure 5 The convergence of the cost function (17) for differentvalues of micro

818 819 820 821 822 823 824 825

020

020

020

0

50

Time (s)

Am

plitu

de

Raw EEG

Martin BPF

Wendt BPF

BPF(s)

Expert annotated spindles

Proposed method spindle detection

Martin spindle detection

Wendt spindle detectionTransientactivity

Figure 6 Comparison of the spindle detection using Wendt Martinand the proposed detection method

oscillatory component was free of transients and the de-tected spindle duration nearly matched the duration of adesignated spindle We use threshold values c1 = 003 andc2 = 10 Although there is no precise definition for thefrequency of K-complexes they are usually within 05 Hzto 4 Hz [26 46 47] Therefore we set the lowpass filtercut-off frequency to 4 Hz

We use the value micro = 05 This value of micro gives thelowest value of the objective function in (17) at the 20thiteration (Fig 5) Note that micro does not affect the solutionto which the DETOKS algorithm converges it only af-fects the rate of convergence Though several parametersmust be specified λ2 is the only parameter that is variedbetween [75 85] by experimental observation in order toemulate the duration of the detected spindle to that of adesignated spindle

41 Sleep Spindle detection

We compare the spindle detection of the Wendt Mar-tin and the proposed DETOKS method in Fig 6 Experts

6

130 132 134 136 138 140

020

020

020

0

50

Time (s)

Am

plitu

de

Raw EEG

BPF(s)

Wendt BPF

Martin BPF

Expert annotated spindles

False positive spindle

False positive spindle

Proposed method spindle detection

Figure 7 False detection of spindles by the Wendt and Martin algo-rithm due to the transient activity in the bandpass filtered data

have annotated three sleep spindles at 8181 8208 and at8239 seconds The Martin algorithm detects only two ofthe spindles whereas the Wendt algorithm detects onlyone spindle On the other hand all the three spindles aredetected by the proposed detection method The effect ofthe suppression of transients can be seen in the bandpassfilter activity in Fig 6 The bandpass filter is excited onlyduring spindle activity in the EEG In contrast for theWendt and Martin algorithms the low amplitude spindleactivity is masked by the transients which excite the band-pass filter Thus the suppression of transients increases thenumber of true positives for spindle detection

As an another example consider the issue of falsely de-tected spindles Figure 7 illustrates three spindles scoredby experts at 1316 1334 and at 1385 seconds Howeverthe Martin and Wendt algorithms each detect an addi-tional spindle which is not annotated by experts Thetransient activity excited the bandpass filters again tothe point that the Wendt and Martin algorithms detectfalse spindles On the other hand DETOKS avoids thesefalse detections Moreover the duration of the spindles de-tected by DETOKS better matches those of the experts

If the parameters λ0 λ1 and λ2 for the DETOKS al-gorithm are increased or decreased then the duration ofthe spindles detected will also be affected Thus one canset the parameters λ0 λ1 and λ2 so that the duration ofa detected spindle emulates the duration of a designatedspindle

42 K-complex detection

Conventional methods for K-complex detection applya lowpass filter directly to the EEG then use the TKEOHowever this yields a large number of false positives asseen in Fig 8 There is only one true positive K-complexat 2544 seconds and four false positives Removal of these

251 253 255 257 259

01

0

20

0

50

minus50

0

50

Time (s)

Am

plitu

de

Raw EEG (y)

LPF(y)

TKEO applied to LPF(y)

bKminuscomplex

True positive Kminuscomplex

TransientActivity

Figure 8 Detecting K-complexes with a 4th order Butterworth low-pass filter LPF(middot) and the TKEO There are four false positive K-complexes and only one true positive

false positives requires additional processing stages ratherthan simply using the TKEO on the lowpass filtered EEG[26] The transients in the EEG excite the lowpass fil-ter which are then mis-identified as K-complexes afterthe TKEO stage Moreover increasing the threshold forTKEO will not only result in low true positive valuesbut the duration of the detected K-complexes will also beshorter compared to the expert annotation In contrastthe component x obtained using the DETOKS algorithmcaptures the transient activity thereby reducing the num-ber of false positives while accurately detecting the desig-nated K-complexes

Figure 9 compares the K-complexes detected using theproposed method and the method by Devuyst et al [12]Specifically we compare the annotations provided by De-vuyst et al online2 for their method and the K-complexdetection by DETOKS The experts have annotated twoK-complexes - at 240 and at 254 seconds The compo-nent f obtained by the proposed method is also shown Itcan be seen that the Devuyst algorithm [12] detects onlythe K-complex at 255 seconds Moreover the durationof the detected K-complex is less than that indicated bythe expert On the other hand DETOKS detects bothK-complexes which better match the expert detection

5 Evaluation of DETOKS

We assess the performance of DETOKS by applying itto an online EEG database We use two publicly avail-able databases3 - one for spindle detection and one for

2httpwwwtctsfpmsacbe~devuystDatabases

DatabaseKcomplexes3httpwwwtctsfpmsacbe~devuystDatabases

7

239 241 243 245 247 249 251 253 255

0101

0

50

0

50

minus50

0

50

Time (s)

Am

plitu

de

Raw EEG

Transient component (x)

Lowminusfrequency component (f)

Detected Kminuscomplexes

Devuyst detected Kminuscomplexes

Expert scored Kminuscomplex

Figure 9 Comparison of the K-complex detection by Devuyst et aland the proposed simultaneous detection method The transients inthe EEG are captured in the component x

K-complex detection The databases4 are made availableonline by Devuyst et al [13] For both spindle and K-complex detection we use the central channel C3-A1 ofthe EEG We use these databases for the evaluation studybecause they are readily available online and provide ex-pert annotations for the scoring of sleep spindles and K-complexes

51 Database

As per [13] the EEG database for sleep spindle detec-tion was acquired in a sleep laboratory of a Belgium hospi-tal using a digital 32-channel polygraph (BrainnetTM Sys-tem of MEDATEC Brussels Belgium) The patients pos-sessed different pathologies (dysomnia restless legs syn-drome insomnia apnoea hypopnoea syndrome) [13] ThreeEEG channels (CZ-A1 or C3-A1 FP1-A1 and O1-A1)two EOG channels and one submental EMG channel wererecorded A 30-minute segment of the central EEG chan-nel was extracted from each whole-night recording for sleepspindle scoring These excerpts were given to two expertswho independently scored spindles Out of the 8 excerptswe conduct the study using the 5 that were annotated byboth experts

The EEG database for K-complex detection was alsoacquired in a sleep laboratory of a Belgium hospital us-ing a digital 32-channel polygraph (BrainnetTM Systemof MEDATEC Brussels Belgium) [12] As in the case ofthe sleep spindles a 30-minute segment from each whole-night recording was independently given to two experts

4University of MONS - TCTS Laboratory (S Devuyst T Du-toit) and Universite Libre de Bruxelles - CHU de Charleroi SleepLaboratory (M Kerkhofs)

for scoring of K-complexes according to the manual [2]and the recommendations in [12] For this study we usethe 5 excerpts that were scored by both experts

52 Existing detection methods

For sleep-spindle detection we compare DETOKS tothe existing algorithms by Wendt et al (S1) [14] Martinet al (S2) [15] Wamsley et al (S3) [7] Bodizs et al (S3)[29] Molle et al (S3) [16] and Devuyst et al (S6) [13]A summary of these algorithms is provided in [4 13] Wereject any spindles detected by S1-S6 of less than 05 sec-onds in duration For K-complex detection we compareDETOKS to the annotations provided by Devuyst et alfor their detection method [12]

53 Measure of Performance

We use the detection by experts for the sleep spin-dles and the K-complexes as the gold standard A samplepoint of the EEG is recorded as a sleep spindle or a K-complex if it was scored as such by either expert Thisis the by-sample analysis of a detector [4] using a lsquounionrsquorule of the expert detection [13] We create a contingencytable for sleep spindles and K-complexes to calculate thevalues of true positive (TP) false positive (FP) true neg-ative (TN) and false negative (FN) These values are thenused to calculate the recall and precision values for the de-tector We use F1 scores to evaluate the detectors The F1

score ranges from 0 to 1 with 1 denoting the perfect detec-tor Using the contingency table the statistical measuresof Cohenrsquos κ [48] and Matthews Correlation Coefficient(MCC) [49] can also be calculated

54 Results

The C3-A1 channel of the EEG was processed usingthe proposed DETOKS algorithm and the estimates of thecomponents s and f were then used for sleep spindle andK-complex detection respectively We used the same pa-rameters for λ0 λ1 λ2 and the threshold values c1 andc2 as in Sec 4 We used the STFT parameters given inSec 23

Figure 10 displays the F1 scores for sleep spindle de-tection recall and precision values for the algorithms inSec 52 and DETOKS Table 1 lists the F1 scores forK-complex detection for the Devuyst algorithm [12] andDETOKS The recall and precision values are also listed

Further statistical measures of performance are listedin the Appendix The proposed DETOKS method takesabout 4 minutes to run 20 iterations on approximately 8hours of EEG at sampling frequency 100 Hz

55 Discussions

The proposed DETOKS method achieves high F1 scoresfor spindle detection due to the suppression of transientsAs seen in Appendix A the values of κ and MCC were sim-ilar to the F1 scores obtained The TP values are compara-tively higher than those of algorithms S1-S6 The number

8

S1 S2 S3 S4 S5 S6 DETOKS0

02

04

06

08

1

F1 S

core

Algorithm

a

049 050

006

034040

062070

0 02 04 06 08 10

02

04

06

08

1

Pre

cisi

on

Recall

DETOKS

S4

S6

S1

S2

S5

S3

b

Figure 10 Results of the evaluation study in Sec 5 (a) Comparisonof the F1 scores of the spindle detection algorithms (b) Recall andPrecision values for the algorithms S1-S6 and the proposed DETOKSmethod The error bars represent standard error of the mean

Devuyst et al DETOKS

F1 Score 051 057

Recall 040 061

Precision 074 056

Table 1 K-complex detection performance evaluation

of false positives are also reduced The reduction in FPvalues and an increase in TP values result in the F1 scoreof the proposed detection method for sleep spindles averag-ing 07plusmn003 over the five excerpts Note that an extensivestudy on sleep spindle detection by automated detectors[4] showed that the F1 score of 24 individual experts aver-aged 069plusmn006 on their crowd-sourcing generated spindlegold standard The proposed DETOKS method achievesthe average F1 score of an expert consistently thus makingit a reliable spindle detector

A balanced detector shows high recall and precisionvalues Figure 10 shows that DETOKS is the most bal-anced among the 6 algorithms S1-S6 for spindle detec-tion The recall and precision values for DETOKS aver-aged 071 plusmn 001 and 068 plusmn 003 respectively The algo-rithm by Bodizs et al [29] has a higher recall than theDETOKS but low precision indicating the detection of alarge number of false positives in contrast to DETOKSThe proposed DETOKS method is not only reliable butalso a balanced spindle detector

For the detection of K-complexes the proposed methodobtained an average F1 score of 057 plusmn 0019 over the 5excerpts The performance of DETOKS for K-complexdetection was also balanced compared to the Devuyst al-gorithm [12] It can be seen in Table 1 that the proposed

6 62 64 66 68 7 72 74 76 7804

05

06

07

08

09

1

Change in F1 score for one epoch for λ

2 isin [6 8]

λ2

F1 S

core

λ1 = 6

λ1 = 7

λ1 = 8

Figure 11 Change in the F1 score of an epoch on changing theparameters λ1 and λ2 in (17) λ0 is fixed at 06

detection method has better recall and precision valuescompared to the Devuyst algorithm Note that the recallprecision and F1 scores for the Devuyst method were cal-culated based on the TP FP TN and FN values providedin [12]

The algorithm by Devuyst et al employs K-complexdetection using a training dataset [12] It tunes the detec-tion process based on visually identified K-complexes Incontrast the proposed detection method needs no train-ing data-set Moreover for different databases based onthe sampling frequency the only parameters that needto be tuned are the regularization parameters λ0 λ1 andλ2 The other parameters for example the STFT windowlength are based on the definition of sleep spindles andhence need no tuning Figure 11 shows that the F1 scorefor sleep spindle detection for one EEG epoch is affectedmarginally when the parameters λ1 and λ2 are changed insmall increments with λ0 fixed Similar change is observedwhen λ0 is varied in addition to λ1 and λ2

6 Conclusion

This paper proposes an EEG signal model comprisingof 1) a transient component 2) a low-frequency compo-nent and 3) an oscillatory component with sparse time-frequency representation We propose a convex optimiza-tion algorithm for the detection of K-complexes and sleepspindles (DETOKS) which estimates the three compo-nents in the signal model The proposed DETOKS methodutilizes the Teager Kaiser Energy Operator (TKEO) as ameans to obtain the envelope of the bandpass filtered oscil-latory component and the low-frequency component Theenvelopes are then used for the detection of sleep spindlesand K-complexes

An assessment of the proposed DETOKS method yieldsF1 scores of average 070 plusmn 003 for sleep spindle detec-tion and 057 plusmn 0019 for K-complex detection The pro-posed signal model and the DETOKS algorithm give bet-ter results for the detection of both sleep spindles andK-complexes compared to existing algorithms specifically

9

aimed at one or the other The average F1 score of the pro-posed detector is about the same as the F1 score attainedby individual experts annotating crowd-sourced spindledata [4] Comparable run-times and better detection re-sults than traditional spindle and K-complex detection al-gorithms suggest that the proposed DETOKS method ispractical for the detection of sleep spindles and K-comp-lexes

7 References

[1] M H Silber S Ancoli-Israel M H Bonnet S ChokrovertyM M Grigg-Damberger M Hirshkowitz S Kapen S aKeenan M H Kryger T Penzel M R Pressman C IberThe visual scoring of sleep in adults Journal of clinical sleepmedicine JCSM official publication of the American Academyof Sleep Medicine 3 (2) (2007) 121ndash31 ISSN 1550-9389

[2] R Berry R Brooks C Gamaldo S Harding R Lloyd C Mar-cus B Vaughn The AASM Manual for the Scoring of Sleep andAssociated Events Rules Terminology and Technical Specifi-cations 2013

[3] S Devuyst T Dutoit J Didier F Meers E Stanus P Ste-nuit M Kerkhofs Automatic sleep spindle detection in patientswith sleep disorders Proc IEEE Int Conf Eng in Medicineand Biology (EMBC) (2006) 3883ndash3886ISSN 1557-170X doi101109IEMBS2006259298

[4] S C Warby S L Wendt P Welinder E G S Munk O Car-rillo H B D Sorensen P Jennum P E Peppard P PeronaE Mignot Sleep-spindle detection crowdsourcing and evaluat-ing performance of experts non-experts and automated meth-ods Nature methods 11 (4) (2014) 385ndash92 ISSN 1548-7105doi101038nmeth2855

[5] S M Fogel C T Smith The function of the sleep spin-dle a physiological index of intelligence and a mechanismfor sleep-dependent memory consolidation Neuroscience andbiobehavioral reviews 35 (5) (2011) 1154ndash65 ISSN 1873-7528doi101016jneubiorev201012003

[6] F Ferrarelli R Huber M J Peterson M Massimini M Mur-phy B a Riedner A Watson P Bria G Tononi Reducedsleep spindle activity in schizophrenia patients The Americanjournal of psychiatry 164 (3) (2007) 483ndash92 ISSN 0002-953Xdoi101176appiajp1643483

[7] E Wamsley M Tucker A Shinn Reduced sleep spindles andspindle coherence in schizophrenia mechanisms of impairedmemory consolidation Biol Psychiatry 71 (2) (2012) 154ndash161doi101016jbiopsych201108008Reduced

[8] E Limoges L Mottron C Bolduc C Berthiaume R God-bout Atypical sleep architecture and the autism phenotypeBrain a journal of neurology 128 (Pt 5) (2005) 1049ndash61 ISSN1460-2156 doi101093brainawh425

[9] D Petit J F Gagnon M L Fantini L Ferini-StrambiJ Montplaisir Sleep and quantitative EEG in neurodegener-ative disorders doi101016jjpsychores200402001 2004

[10] C Richard R Lengelle Joint time and time-frequency opti-mal detection of K-complexes in sleep EEG Computers andbiomedical research an international journal 31 (1998) 209ndash229ISSN 0010-4809 doiS0010480998914768[pii]

[11] C Stepnowsky D Levendowski D Popovic I Ayappa D MRapoport Scoring accuracy of automated sleep staging from abipolar electroocular recording compared to manual scoring bymultiple raters Sleep medicine 14 (11) (2013) 1199ndash207 ISSN1878-5506 doi101016jsleep201304022

[12] S Devuyst T Dutoit P Stenuit M Kerkhofs Automatic K-complexes detection in sleep EEG recordings using likelihoodthresholds in Proc IEEE Int Conf Eng in Medicine andBiology (EMBC) ISBN 9781424441235 ISSN 1557-170X 4658ndash4661 doi101109IEMBS20105626447 2010

[13] S Devuyst T Dutoit P Stenuit M Kerkhofs Automaticsleep spindles detection - overview and development of a stan-

dard proposal assessment method Proc IEEE Int Conf Engin Medicine and Biology (EMBC) (2011) 1713ndash1716ISSN 1557-170X doi101109IEMBS20116090491

[14] S L Wendt J E Christensen J Kempfner H L LeonthinP Jennum H B D Sorensen Validation of a novel automaticsleep spindle detector with high performance during sleep inmiddle aged subjects Proc IEEE Int Conf Eng in Medicineand Biology (EMBC) (2012) 4250ndash4253ISSN 1557-170X doi101109EMBC20126346905

[15] N Martin M Lafortune J Godbout M Barakat R RobillardG Poirier C Bastien J Carrier Topography of age-relatedchanges in sleep spindles Neurobiology of aging 34 (2) (2013)468ndash76 ISSN 1558-1497 doi101016jneurobiolaging201205020

[16] S Gais M Molle K Helms J Born Learning-dependent in-creases in sleep spindle density The Journal of neuroscience the official journal of the Society for Neuroscience 22 (15) (2002)6830ndash4 ISSN 1529-2401 doi20026697

[17] T A Camilleri K P Camilleri S G Fabri Automatic detec-tion of spindles and K-complexes in sleep EEG using switch-ing multiple models Biomedical Signal Processing and Control10 (2014) 117ndash127 ISSN 17468094 doi101016jbspc201401010

[18] A Jaleel B Ahmed R Tafreshi D B Boivin L StreletzN Haddad Improved spindle detection through intuitive pre-processing of electroencephalogram Journal of neurosciencemethods 233 (2014) 1ndash12 ISSN 1872-678X doi101016jjneumeth201405009

[19] S Motamedi-Fakhr M Moshrefi-Torbati M Hill C M HillP R White Signal processing techniques applied to humansleep EEG signals A review Biomedical Signal Processing andControl 10 (2014) 21ndash33 ISSN 17468094 doi101016jbspc201312003

[20] D Gorur U Halici H Aydin Sleep spindles detection usingshort time Fourier transform and neural networks Int Jt ConfNeural Networks (IJCNN) (2002) 1631ndash1636

[21] J Costa M Ortigueira A Batista L Paiva An automaticsleep spindle detector based on WT STFT and WMSD WorldAcademy of Science Engineering and Technology (2012) 1833ndash1836

[22] N Acir C Guzeli Automatic recognition of sleep spindles inEEG via radial basis support vector machine based on a mod-ified feature selection algorithm Neural Computing and Ap-plications 14 (1) (2005) 56ndash65 ISSN 0941-0643 doi101007s00521-004-0442-z

[23] P Durka K Blinowska Matching pursuit parametrization ofsleep spindles Engineering in Medicine and Biology 3 (1996)1011ndash1012 doi101109IEMBS1996652685

[24] P Durka D Ircha K Blinowska Stochastic time-frequencydictionaries for matching pursuit IEEE Transactions on SignalProcessing 49 (3) (2001) 507ndash510 ISSN 1053587X doi10110978905866

[25] A Parekh I W Selesnick D M Rapoport I Ayappa Sleepspindle eetection using time-frequency sparsity in Proc IEEESignal Processing in Medicine and Biology Symposium (SPMB)doi101109SPMB20147002965 2014

[26] A Erdamar F Duman S Yetkin A wavelet and teager energyoperator based method for automatic detection of K-Complexin sleep EEG Expert Systems with Applications 39 (1) (2012)1284ndash1290 ISSN 09574174 doi101016jeswa201107138

[27] C Strungaru M S Popescu Neural network for sleep EEGK-complex detection Biomedizinische Technik Biomedical en-gineering 43 Suppl 3 (1998) 113ndash116 ISSN 0013-5585 doi101515bmte199843s3113

[28] I W Selesnick Resonance-based signal decomposition Anew sparsity-enabled signal analysis method Signal Processing91 (12) (2011) 2793ndash2809 ISSN 01651684 doiDOI101016jsigpro201010018

[29] R Bodizs J Kormendi P Rigo A S Lazar The individualadjustment method of sleep spindle analysis methodologicalimprovements and roots in the fingerprint paradigm Journal

10

of neuroscience methods 178 (1) (2009) 205ndash13 ISSN 0165-0270doi101016jjneumeth200811006

[30] D L Donoho De-noising by soft-thresholding IEEE Transac-tions on Information Theory 41 (1995) 613ndash627 ISSN 00189448doi10110918382009

[31] L I Rudin S Osher E Fatemi Nonlinear total variation basednoise removal algorithms Physica D Nonlinear Phenomena60 (1992) 259ndash268 ISSN 01672789 doi1010160167-2789(92)90242-F

[32] L Condat A Direct Algorithm for 1-D Total Variation Denois-ing IEEE Signal Processing Letters 20 (11) (2013) 1054ndash1057ISSN 1070-9908 doi101109LSP20132278339

[33] A Rinaldo Properties and refinements of the fused lasso TheAnnals of Statistics 37 (5B) (2009) 2922ndash2952 ISSN 0090-5364doi10121408-AOS665

[34] R Tibshirani M Saunders S Rosset J Zhu K Knight Spar-sity and smoothness via the fused lasso Journal of the RoyalStatistical Society Series B Statistical Methodology 67 (2005)91ndash108 ISSN 13697412 doi101111j1467-9868200500490x

[35] S Mallat Chapter 1 - Sparse Representations in A wavelettour of signal processing The sparse way Academic PressISBN 978-0-12-374370-1 1ndash31 doihttpdxdoiorg101016B978-0-12-374370-100005-7 2009

[36] S Chen D L Donoho M A Saunders Atomic decompositionby basis pursuit SIAM journal on scientific computing 43 (1)(1998) 129ndash159 doi101137S1064827596304010

[37] I W Selesnick Short-time Fourier transform and speech de-noising httpcnxorgcontentm32294

[38] I W Selesnick H L Graber D S Pfeil R L Barbour Simul-taneous low-pass filtering and total variation denoising IEEETransactions on Signal Processing 62 (5) (2014) 1109ndash1124ISSN NA doi101109TSP20142298836

[39] A Chambolle T Pock A first-order primal-dual algorithmfor convex problems with applications to imaging J MathVis 40 (1) (2011) 120ndash145 ISSN 09249907 doi101007s10851-010-0251-1

[40] P L Combettes J-C Pesquet Proximal thresholding algo-rithm for minimization over orthonormal bases SIAM Journalon Optimization 18 (4) (2007) 1351ndash1376 ISSN 1052-6234 doi101137060669498

[41] P L Combettes J-C Pesquet Proximal splitting methods insignal processing in H H Bauschke Others (Eds) Fixed-Point Algorithms for Inverse Problems in Science and Engineer-ing Springer-Verlag 185ndash212 2011

[42] M Afonso J Bioucas-Dias M Figueiredo Fast image recov-ery using variable splitting and constrained optimization IEEETransactions on Image Processing 19 (2010) 2345ndash2356 ISSN1941-0042 doi101109TIP20102047910

[43] D Gabay Applications of the method of multipliers to vari-tional inequalities in Augmented Lagrangian methods Appli-cations to the numerical solution of boundary-value problemsElsevier 299ndash331 1983

[44] S Boyd N Parikh E Chu J Eckstein Distributed Opti-mization and Statistical Learning via the Alternating Direc-tion Method of Multipliers Foundations and Trends in Ma-chine Learning 3 (1) (2010) 1ndash122 ISSN 1935-8237 doi1015612200000016

[45] D Tylavsky G Sohie Generalization of the matrix inversionlemma Proceedings of the IEEE 74 (1986) 1050ndash1052 ISSN0018-9219 doi101109PROC198613587

[46] G Bremer J R Smith I Karacan Automatic detection of theK-complex in sleep electroencephalograms IEEE transactionson bio-medical engineering 17 (1970) 314ndash323 ISSN 0018-9294doi101109TBME19704502759

[47] M Woertz T Miazhynskaia P Anderer G Dorffner Auto-matic K-complex detection comparison of two different ap-proaches Abstracts of the ESRS JSR 13 (1)

[48] J Cohen A coefficient of agreement for nominal scales Edu-cational and Psychological Measurement 20 (1) (1960) 37ndash46doi101177001316446002000104

[49] B W Matthews Comparison of the predicted and observed

secondary structure of T4 phage lysozyme Biochimica et Bio-physica Acta (BBA) - Protein Structure 405 (2) (1975) 442ndash451doi1010160005-2795(75)90109-9

Appendix A K-complex detection

Performance and comparison of the proposed methodfor K-complex detection with Devuyst et al

Devuyst DETOKS Devuyst DETOKS Devuyst DETOKS Devuyst DETOKS Devuyst DETOKS

TP 2646 4777 4800 7824 1676 1973 6672 10404 4840 7140

TN 349221 341758 348236 343917 355906 355150 335746 331796 347611 345900

FP 627 8089 1337 5655 889 1644 2674 6623 1869 3580

FN 7506 5375 5627 2603 1529 1232 14908 11176 5680 3380

Recall 0261 0471 0460 0750 0523 0616 0309 0482 0460 0679

Precision 0808 0371 0782 0580 0653 0545 0714 0611 0721 0666

F1 Score 0394 0415 0580 0655 0581 0578 0431 0539 0562 0672

Specificity 0998 0977 0996 0984 0998 0995 0992 0980 0995 0990

NPV 0979 0985 0984 0992 0996 0997 0957 0967 0984 0990

Accuracy 0977 0963 0981 0977 0993 0992 0951 0951 0979 0981

Kappa 0386 0396 0570 0643 0578 0574 0410 0513 0552 0652

MCC 0451 0399 0591 0649 0581 0575 0450 0517 0566 0661

Excerpt 3 Excerpt 4 Excerpt 5Excerpt 1 Excerpt 2

11

Appendix B Sleep spindle detection

Performance of the proposed DETOKS method for spin-dle detection and comparison with other widely used algo-rithms

Excerpt 1 Wendt Martin Wamsley Bodizs Moelle Devuyst DETOKS

TP 8357 4121 2374 9550 4991 9692 9354

TN 156774 164771 166242 137540 164911 159737 162632

FP 9751 1754 283 28985 1614 6788 3892

FN 5118 9354 11101 3925 8484 3783 4121

Recall 0620 0306 0176 0709 0370 0719 0694

Precision 0462 0701 0893 0248 0756 0588 0706

F1 Score 0529 0426 0294 0367 0497 0647 0700

Specificity 0941 0989 0998 0826 0990 0959 0977

NPV 0968 0946 0937 0972 0951 0977 0975

Accuracy 0917 0938 0937 0817 0944 0941 0955

Kappa 0485 0399 0276 0288 0471 0615 0676

MCC 0491 0437 0381 0343 0505 0619 0676

Excerpt 2 Wendt Martin Wamsley Bodizs Moelle Devuyst DETOKS

TP 9881 7954 0 11045 7006 9862 10307

TN 320771 339807 344818 268002 340517 336140 339091

FP 24754 5718 707 77523 5008 9385 6433

FN 4594 6521 14475 3430 7469 4613 4168

Recall 0683 0549 0000 0763 0484 0681 071

Precision 0285 0582 0000 0125 0583 0512 062

F1 Score 0402 0565 0000 0214 0529 0585 067

Specificity 0928 0983 0998 0776 0986 0973 098

NPV 0986 0981 0960 0987 0979 0986 099

Accuracy 0918 0966 0958 0775 0965 0961 097

Kappa 0366 0548 ‐0004 0156 0511 0565 065

MCC 0407 0548 ‐0009 0246 0514 0571 065

Excerpt 3 Wendt Martin Wamsley Bodizs Moelle Devuyst DETOKS

TP 1005 680 12 000 0 1400 1707

TN 86050 87234 86741 000 87717 86882 86772

FP 1667 483 976 000 0 835 944

FN 1278 1603 2271 000 2283 883 576

Recall 0440 0298 0005 000 0000 0613 0748

Precision 0376 0585 0012 000 0000 0626 0644

F1 Score 0406 0395 0007 000 0000 0620 0692

Specificity 0981 0994 0989 000 1000 0990 0989

NPV 0985 0982 0974 000 0975 0990 0993

Accuracy 0967 0977 0964 000 0975 0981 0983

Kappa 0389 0384 ‐0008 000 0000 0610 0683

MCC 0390 0407 ‐0009 000 0000 0610 0685

Excerpt 5 Wendt Martin Wamsley Bodizs Moelle Devuyst DETOKS

TP 10461 9672 0 14779 0 10332 14164

TN 330466 333514 337779 295704 340039 335566 333646

FP 9573 6525 2260 44335 0 4473 6392

FN 9500 10289 19961 5182 19961 9629 5797

Recall 0524 0485 0000 0740 0000 0518 0710

Precision 0522 0597 0000 0250 0000 0698 0689

F1 Score 0523 0535 0000 0374 0000 0594 0699

Specificity 0972 0981 0993 0870 1000 0987 0981

NPV 0972 0970 0944 0983 0945 0972 0983

Accuracy 0947 0953 0938 0862 0945 0961 0966

Kappa 0495 0511 ‐0011 0317 0000 0574 0681

MCC 0495 0514 ‐0019 0377 0000 0581 0681

Excerpt 6 Wendt Martin Wamsley Bodizs Moelle Devuyst DETOKS

TP 12810 11724 0 17988 11928 13381 15435

TN 327975 332715 336704 288379 331078 332761 332467

FP 9588 4848 859 49184 6485 4802 5095

FN 9627 10713 22437 4449 10509 9056 7002

Recall 0571 0523 0000 0802 0532 0596 0688

Precision 0572 0707 0000 0268 0648 0736 0752

F1 Score 0571 0601 0000 0401 0584 0659 0718

Specificity 0972 0986 0997 0854 0981 0986 0985

NPV 0971 0969 0938 0985 0969 0974 0979

Accuracy 0947 0957 0935 0851 0953 0962 0966

Kappa 0543 0579 ‐0005 0340 0559 0639 0701

MCC 0543 0586 ‐0013 0407 0562 0643 0701

12