Research Article Gait Biomarkers Classification by Combining ...

Research ArticleGait Biomarkers Classification by Combining AssembledAlgorithms and Deep Learning Results of a Local Study

Eddy Sanchez-DelaCruz 1 Roberto Weber2 R R Biswal3 Jose Mejıa 4

Gandhi Hernandez-Chan5 and Heberto Gomez-Pozos6

1Departamento de Posgrado Instituto Tecnologico Superior de Misantla Veracruz Mexico2Servicios Medicos Universidad Juarez Autonoma de Tabasco Villahermosa Mexico3Tecnologico de Monterrey Escuela de Ingenierıa y Ciencias Mexico4Universidad Autonoma de Ciudad Juarez Ciudad Juarez Mexico5Consejo Nacional de Ciencia y Tecnologıa Centro de Investigacion en Ciencias de la Informacion GeoespacialMexico City Mexico6Universidad Autonoma del Estado de Hidalgo Pachuca Mexico

Correspondence should be addressed to Eddy Sanchez-DelaCruz eddsacxgmailcom

Received 3 June 2019 Revised 12 October 2019 Accepted 21 November 2019 Published 19 December 2019

Academic Editor Zoran Bursac

Copyright copy 2019 Eddy Sanchez-DelaCruz et al -is is an open access article distributed under the Creative CommonsAttribution License which permits unrestricted use distribution and reproduction in anymedium provided the original work isproperly cited

Machine learning one of the core disciplines of artificial intelligence is an approach whose main emphasis is analytical modelbuilding In other words machine learning enables an automaton to make its own decisions based on a previous training processMachine learning has revolutionized every research sector including health care by providing precise and accurate decisionsinvolving minimal human interventions through pattern recognition -is is emphasized in this research which addresses theissue of ldquosupport for diabetic neuropathy (DN) recognitionrdquo DN is a disease that affects a large proportion of the globalpopulation In this research we have used gait biomarkers of subjects representing a particular sector of population located insouthern Mexico to identify persons suffering from DN To do this we used a home-made body sensor network to capture rawdata of the walking pattern of individuals with and without DN-e information was then processed using three sampling criteriaand 23 assembled classifiers in combination with a deep learning algorithm-e architecture of the best combination was chosenand reconfigured for better performance -e results revealed a highly acceptable classification with greater than 85 accuracywhen using these combined approaches

1 Introduction

In Mexico diabetes affects 60 of the population (httpfmdiabetesorgwp-contentuploads201411diabetes2013INEGIpdf) Diabetic neuropathy (DN) is a major consequenceof diabetes mellitus and may have a detrimental effect on thepatientrsquos manner of walking also known as ldquogaitrdquo One variantof DN diabetic peripheral neuropathy (DPN) is a peripheralpathology that causes the patient to show disorder in gait andprogressive deterioration Diagnosis of this pathology requiresmedical evaluation but the use of computational techniques hasalso been proposed for its detection to reduce the margin of

error of classification [1] -e present research involved the useof a network of sensors to acquire gait biomarkers for samplepatients with DN and healthy individuals -ese samples wereused to create a model that contains the characteristics ofhealthy persons as well as patients suffering from DN and tagstheir state of health Subsequently a set of test data with theknown health status of each case was used but without tagging-e test data confirmed the efficiency of the models followingthe implementation of an exhaustive search that combinedvarious algorithms (assembled classifiers+deep learning) andselection of the one with the maximum percentage of correctlyclassified instances -ese instances showed with a high degree

HindawiComputational and Mathematical Methods in MedicineVolume 2019 Article ID 3515268 14 pageshttpsdoiorg10115520193515268

of certainty the existence of atrophy in muscles leading to anabnormal gait due to DN

Machine learning has been widely used in several areasIn health research it has been applied for disease diagnosisand the subsequent timely treatment of progressive diseasesincluding DN [2ndash5] which affects a high percentage of theworld population -e present research focuses on therecognition of persons affected by DN through the classi-fication of gait biomarkers For this purpose the followingmethodology was used (i) A group of individuals with andwithout DN was selected (ii) -e sensors were placed andthe biomarkers data of gait were obtained (iii) Each of thecases was tagged as positive or negative for DN dependingon whether the person presented the condition (iv) -ecollected data were divided into two groups the first wasused as training data and the second one as test data (v) Amodel that describes the behavior of the gait in both caseswas built and trained with the training dataset (vi) -emodel was evaluated using the test dataset (without tagging)and different classification algorithms (classifiers) (vii) -eassembled classifiers were combined with a deep learningalgorithm to find the one that generates the highest accuracyindexes

In the state-of-the-art scientific literature no method hasyet combined these approaches to solve the problem pre-sented here In addition due to the successive refinementobtained using this combined approach the combination ofan assembled classifier + deep learning algorithm appears tobe a promising option for increasing the percentage ofcorrectly classified instances by categorizing gait biomarkersin patients with DN against those of healthy controls

2 State of the Art

DN is a consequence of degradation of the peripheral andautonomous nervous system It is probably the most fre-quent complication of diabetes affecting more than 50 ofpatients after 20 years of the disease course depending onthe severity and duration of hyperglycemia -e prevalenceincreases with years of progression hyperglycemia andestablished cardiovascular disease [6] About 60 to 70 per-cent of people with diabetes suffer from some type ofneuropathy and these nerve disorders can develop at anytime however the risk increases with age and with theduration of the disease -e highest DN incidence rates arefound in people who have been suffering from diabetes for atleast 25 years DN also seems to be more common in peoplewho have problems controlling their blood glucose (bloodsugar) as well as in people with high levels of body fat orelevated blood pressure or who are overweight [7]-eDN ispresent in 40 to 50 of diabetic patients at 10 years after theonset of both type 1 and type 2 diabetes although less than50 of these patients show DN symptoms DN prevalenceincreases with the time of evolution of the disease and withthe age of the patient with its extent and severity related tothe degree and duration of hyperglycemia [8]

-ere are several studies that propose the use of hard-ware devices to gather information from patients sufferingfrom diseases that affect gait In addition a wide variety of

machine learning algorithms have been used to categorizethese diseases some of which are described below

Several studies have proposed the use of hardware de-vices to gather information about patients suffering fromdiseases that affect gait In addition a wide variety of ma-chine learning algorithms have been used to categorize thesediseases For example Mueller et al compared gait char-acteristics including torsional flexor pairs for feet and therange of ankle motion of subjects with diabetes mellitus andperipheral neuropathy -ey found that patients with di-abetes showed less mobility and lower ankle power speedand length of stride during walking as well as a significantdecrease in ankle strength and mobility which seemed to bethe key factors contributing to patterns of altered walk inthese patients [9]

Similarly Sacco and Amadio used sensitive time trackingin neuropathic and non-neuropathic diabetic patients as ameasure of sensory deficit focusing on dynamic and tem-poral parameters -e aim of their study was to investigatewhether neuropathic patients develop changes in dynamicsduring walking to compensate for sensory deficits -eycompared the results of neuropathic patients to those of anondiabetic group to determine the relationships betweenthe maximum plantar pressure cronaxie and sensitiveness inselected plantar areas as they speculated that neuropathicpatients develop compensatory musculoskeletal mecha-nisms to make up for their sensory deficit [10] -ey basedtheir research on an innovative thematic approach involvingDPN and described and interpreted a treadmill self-healingsystem by neuropathic diabetic subjects using biomechanicsand somatosensory considerations-eir innovation was theuse of electromyography (EMG) and a treadmill instru-mented in a clinical application to study and interpretmotor control during gait in neuropathic diabetic patients-ey found significantly higher somatosensory responsesand pain tolerance thresholds in the diabetic neuropathicgroup these responses were considered far from normalpatterns -e EMG responses of the thigh and leg musclesand especially the tibialis anterior and vastus lateralis weredelayed in the diabetic neuropathic group when compared tothe normal pattern -e study showed that long-term sen-sory and motor defects altered muscle activation patternsduring neuropathic walking on the treadmill [11]

Kwon et al comparedmuscle activity and joint momentsin the lower extremities when walking among nine subjectswith DN and nine control subjects -ey found that con-traction of agonist and antagonist muscles occurred in theankle and knee joints in subjects with DN during the supportphase and they concluded that these contractions may berelated to an adaptive gait strategy that compensates for thedecrease in sensory information from the ankle and foot-econtractions may contribute to a more stable gait but theincreased muscle activity probably has a higher energy cost-e differences in joint moments and electromyographicactivity moment when walking in subjects with DN could beexplained by several factors including the loss of sensoryperception decreased muscle strength decreased anklemobility and slow speed -e results also showed thatsubjects with DN had less ankle mobility slower walking

2 Computational and Mathematical Methods in Medicine

speeds longer posture phases and greater dorsiflexion of thelower peak ankle ankle plantar flexion and extensionmoments of knee when compared with the control subjects[12]

Yavuser et al defined gait deviation in patients withdiabetes mellitus by studying the associations betweenelectrophysiological findings and gait characteristics -eirgait analysis showed a slow gait shorter steps limited kneeand ankle mobility lower plantar flexor moment of theankle and lower power in the diabetic group and the dif-ferences were statistically significant In addition wave levelsand latency were significantly correlated with ankle mobilityand the plantar flexionmoment of the ankle-ey concludedthat neuropathy might not be the only reason for gait de-viations in patients with diabetes mellitus [13]

Akashi et al compared the electromyographic activity ofthe thigh and calf muscles during gait in nondiabetic subjectsand patients with DN at two stages of disease those with andwithout previous experience of ulcers in their clinical his-tory -ey also investigated whether the changes in elec-tromyography were due to some alteration in the reactionforce on the floor during gaiting -ey found that long-termneuropathic deficits represented by a clinical history of atleast one foot ulcer in the last two years caused a late ac-tivation of the lateral vastus and lateral gastrocnemius and alower propulsion of the vertical reaction force of the floorduring barefoot walking [14]

Sawacha et al investigated the muscle activity of de-viations during gait even in the early stages of diabeteswhen neuropathy is absent -is study involved 50 subjects10 controls (body mass index 244 + 28 age 612 + 507) 20diabetics (body mass index 264 + 25 age 5653 + 1329) and20 neuropathic (body mass index 268 + 34 age 612 + 77)-e electrical activity of six muscles was collected bilaterallyin the lower extremity during the motion gluteus mediusrectus femoris tibialis anterior long peroneus gastrocne-mius lateralis and extensor digitorum communis and theelectromyographic activity was represented through a linearmodel -e time and space parameters were also evaluatedby means of two Bertec force plates and a six-camera motioncapture system (BTS 60ndash120Hz) In the initial contact andload response an early response peak of rectus femorisactivity occurred in diabetic subjects with and withoutneuropathy -e results suggest that important deviations ofmuscle activity are present in diabetic subjects althoughthese are not directly related to neuropathy -e authors keyfinding can be considered as the presence of statisticallysignificant alterations in non-neuropathic subjects -eresults also suggest that important deviations of muscleactivity are present in diabetic subjects although these arenot directly related to the neuropathy -e authors believethat these results indicate that changes in the muscles of thefoot occur before changes in nerve function can be detected[15]

Deschamps et al indicated that the reduction in themobility of the foot was a key factor in the biomechanicalalteration of the foot in individuals with diabetes mellitus-e aim of their study was to compare the kinematics andcoupling in adult patients with diabetes but with and

without neuropathy based on age sex and walking speedDifferences in the range of movement were quantified withthe Rizzoli multisegment standing model and differentphases of the gait cycle were analyzed by repeated one-waymeasures using analysis of variance ANOVA -e groupswith diabetes showed significantly lower values of move-ment compared to the control group -ese findings sug-gested an alteration in the kinematics and segmentalcoupling during gait in diabetic patients with and withoutneuropathy [16]

Fernando et al carried out a detailed review of electronicdatabases by searching for articles studying the effects of DNon gait -eir analysis of the spatial-temporal parameterskinematics of lower limbs kinetics muscle activation andplantar pressure showed that patients with DN had elevatedplantar pressures and occupied a greater length of time in thestance phase with maximum contact in the flat feet positionduring gaiting when compared to healthy controls [17]

Patterson and Caulfield used accelerometers to detectdifferent gait conditions in people with normal and rigidankles -ey used an algorithm that quantifies the relevantcharacteristics of the swing phase in the foot and found aclear distinction between gait patterns in the ankle move-ments [18]

Gomes et al studied patients with DN who suffered gaitdisturbances related to plantar ulcerations -ey corrobo-rated this relationship by designing computational simu-lations based on the gait muscle excitation patterns andfound that their simulation was able to represent the hipposture adopted by patients with DN during movement asan adaptation to the loss of function in the distal muscles[19]

Sanchez-DelaCruz et al proposed a classification modelusing gait information derived from data from a publicrepository for their tests and implementing various machinelearning algorithms -e best result was obtained by com-bining the algorithms LogitBoost+RandomSubSpace andthey showed that assembled classifiers are a good alternativefor binary classification [20] Based on these results theydesigned a sensor network for collecting gait biomarkers andbuilt a database of patients with neurodegenerative diseases[21]

Camargo et al designed a study to assess aspects ofbalance ankle strength and parameters of spatiotemporalgait in persons with DPN and to verify whether deficits in theparameters of the spatiotemporal gait were associated withmuscular strength and ankle balance Spatiotemporal mo-bility functional mobility balance performance and anklemuscle strength were affected in individuals with DPN -eperformance of the time up and go test and the isometricmuscle strength of the ankle were associated with changes inspatiotemporal gait especially during the condition ofmaximum gait velocity [22]

Berki and Davis collected pressure and tension data from26 diabetic subjects and healthy controls using a new in-strumentation that measures the vertical and horizontalforce vectors of the plantar contact surface in the gait cycle-ey applied two-dimensional discrete Fourier transform ineach dataset for each of the ten sensor sizes -e results

Computational and Mathematical Methods in Medicine 3

showed that the sensor measuring 96mmtimes 96mm causedsignificant reductions in the three tension components(plt 0001) while the sensors measuring 16mmtimes 16mmup to 48mmtimes 48mm can capture the entire spatial rangeof frequencies in the pressure and voltage data [23]

Anjaneya and Holi proposed a method that considerstime and signal characteristics frequencies for DN classifi-cation using a neural network -eir approach was based onthe fact that diabetes risks have increased among childrenand adults in the last decade and that existing methods forearly detection showed potential classification opportunitieswith an accuracy of 9705 [24]

Al-Angari et al used measures of shape and entropy tointroduce new characteristics for capturing the variations inplantar pressure in a study of patients with DPN retinop-athy and nephropathy compared with a diabetic controlgroup without complications -e change in the position ofthe peak pressure of the plant with each step for both feet wasrepresented as a convex polygon asymmetry index area ofthe convex polygon second wavelet moment and entropy ofthe sample [25]

Kavakiotis et al carried out a systematic review ofelectronic information records of scientific articles of the lastfive years through the following queries ldquoMachine LearningAND Diabetesrdquo ldquoData Mining AND Diabetesrdquo and ldquoDi-abetesrdquo whose revision was made in the PubMed and theDBLP Computer Science Bibliography databases As a resultthey found that different algorithms have been implementedwith different datasets of diabetes In their work theypresented a comparison of the percentages obtained in thesestudies [1]

-e current state-of-the-art information indicates thefollowing

(i) -e gait biomarkers acquired by cameras or sen-sors are a reliable source for the collection of gaitinformation in people suffering from gait atrophy

(ii) A large variety of machine learning algorithms havebeen used separately to classify disorders of thehuman gait

(iii) Reliable and competitive classification percentageshave been obtained

Given these observations the classification of gait bio-markers of subjects with DN is an area that is expected toexpand in such a way that reliable and accurate percentagesof classification will be obtained In the present study weassumed that a sensor network would be a promising optionfor collecting gait information to build a dataset on which toimplement an appropriate combination of machine learningalgorithms

3 Materials and Methods

31 Instrument to Collect Data A sensor network consistingof five 3-axis ADXL-335 accelerometer was built validatedand connected to an Arduino MEGA-2560 card -e to-pological connections consisted of Cartesian coordinates xy and z of the ground (GND) and a voltage of 33 V

(Figure 1(a)) -e sensors were distributed as follows asensor was placed on each ankle on each knee and on thehip (close to the gravity center) Data were acquired directlyfrom the accelerometers and no filter was used

-e ADXL-3351 accelerometer (httpwwwanalogcommediaentechnical-documentationdata-sheetsADXL335pdf)is an analog sensor that detects movement ie it is able torespond with an electrical signal to a disturbance inducedby the application of a force or gravity -is device mea-sures the acceleration on a 3G scale and uses a voltage levelof 33 V -e Arduino MEGA-25602 (httpswwwarduinoccenMainArduinoBoardMega2560) is a card that con-tains among others 16 analog inputs 4 UARTs (serialports) a USB connection a power connector and a resetbutton -ese electronic devices allowed the developmentof a useful and above all low-cost sensor network 3827USD (Table 1)

A prototype of the sensor network was validated with asociocultural gender group boys and girls (Figure 1(b)) -edata captured were clean ie noise-free data were obtainedthus allowing an acceptable classification by combining theLogitBoost+RandomForest algorithms as reported else-where [5]

32 Creation of the Database -e selection of subjects wasbased on the work presented in [26] In that work theauthors referred to the creation of a dataset with human gaitinformation and the effect of mechanical perturbations offifteen subjects walking at three speeds on an instrumentedtreadmill

Due to the characteristics of the subjects for our studywe opted to use the purposive sampling technique describedin [27] -is is a nonprobability sampling that is highlyeffective when researchers need to study a certain domain asit allows them to use only those elements from the pop-ulation that best suits the purpose of the study -is kind ofsampling method is fundamental for the quality of datagathered because the reliability and competence of thesource is controlled by the researchers thereby providing aneffective selection of the limited resources

In accordance with the gait cycle or stride as shown inFigure 2 the database was created for patients suffering fromDN using a sensor network -e data represented a par-ticular region of the state of Tabasco located in the southernzone of Mexico For this purpose a gait laboratory wascreated consisting of a 20m 3m space with 8m labelled forthe track (Figure 3(a)) in the premises of the Medical Ser-vices Unit of the Autonomous University of Tabasco-e labalso had seating arrangements to allow the patientsrsquo care-givers to wait and to sign the consent report forms

We worked with 10 patients who presented abnormalityin gait due to DN in addition to 5 healthy subjects (con-trols) -e distribution of characteristics such as gender ageweight height years of suffering and cause is shown inTable 2 -e inclusion criteria were any gender age equal toor greater than 15 years and ambulatory ie they movedwithout support We excluded patients who had experiencedfalls due to their condition patients who did not sign

4 Computational and Mathematical Methods in Medicine

Informed Report pregnant women and patients with medicalconditions that visibly did not allow them to walk for 5minutes Similar studies for gait analysis in patients havebeen published for 13 subjects with amyotrophic lateralsclerosis [29] 14 subjects with Huntingtonrsquos disease [30] 15subjects related to Parkinsonrsquos disease [31] and 17 subjectswith stroke [32]

-e study subjects were instructed to walk normally toperform two familiarization trials with the sensor on prior toconducting the real test involving the capture of gait bio-markers (Figure 3(b))

-erefore one file was created for each patient with theraw data of the x y and z axes of each of the 5 acceler-ometers -ese data were then used as inputs for the clas-sifiers In addition to each file the attribute ldquocaserdquo was addedwhich refers to patients with DN pathologies or controlsubjects (Table 3) -is resulted in the classes of binary setsdiseased control with a total of 16 attributes

33 Data Segmentation For a visual quantitative analysisthe 10 files of the patients and the 5 files of the healthycontrols were integrated into a single dataset from whichsome statistical measurements (Table 4) and correlation(Figure 4) were obtained

-ese measures minimum maximum mean and stan-dard deviation facilitating correct data collection ie thevalues oscillated in the same ranges indicating no ldquooutlierrdquonoise A relationship analysis of the attributes allowed thegeneration of correlation graphs of each sensor for all 15study subjects (Figure 4)

Figure 4(a) which corresponds to the center of gravityshows that no definite correlation exists between the Carte-sian coordinates Instead the hip axes are grouped due to thelinear displacement during gait In relation to the knees theright extremity (Figure 4(b)) shows a positive correlation andthe left extremity (Figure 4(c)) depicts a grouping that cor-responds to a weak relationship In the right ankle(Figure 4(d)) a positive tendency is noted while the left ankle(Figure 4(e)) denotes the presence of clustering -ese ob-servations confirm the assumption derived from Table 4 thatno addition or removal of attributes is required from thedataset

34 Sampling Criteria From the binary dataset diseasedcontrol was used to construct three subsets of data thatconsidered the sampling criteria cross-validation 23ndash13and representative sample

(i) Cross-validation -e data were divided into Ksubsets (folds) One subset is used as test data and therest (K minus 1) as training data -e process was re-peated during K iterations with each of the possibletest set -e error was calculated as the arithmeticmean of each iteration error to obtain a single resulttherefore if MSEi (mean squared error) denotes theerror in the ith iteration then the cross-validationerror is estimated by CV(k) (ik)1113936

ki1MSEi

ArdunioMEGA25602

1514131211100908

0706050403020100

xyz

xyz

xyz

xyz

xyz

ADXL-335

x

yz

ADXL-335

x

yz

ADXL-335

x

yz

ADXL-335

x

yz

ADXL-335

x

yz

Righ ankle

Left ankle

Righ knee

Left knee

Chest

(a)

Arduino

Accelerometers

(b)

Figure 1 Sensor network (a) topology (b) validation

Table 1 Cost of materials for the sensor network

Device Amounts Price TotalsAccelerometer ADXL-335 5 9000 45000Arduino MEGA-2560 1 25000 25000Wire-UTP cat 5 10m 200 em 2000Total in MXN $ 72000Total in USD $ 3827

Computational and Mathematical Methods in Medicine 5

(ii) 23ndash13 Another way to divide the data is in atraining set Dtrain and its corresponding test setDtest such that Dtrain cupDtest D andDtrain capDtest 0 -e model is trained in Dtrain toobtain 1113954f Dtrain and calculate the generalizationerror using the data points in Dtest -e GE estimate(generalization error) is given by 1113955GEholdminus out 1113955GE(fDtrain

Dtest) -is approach is also known as thehold-out method

(iii) Representative Sample Statistical measure to obtainthe test subset -is is obtained with the equation

n (y2pqN)(e2(N minus 1) + y2pq) where n samplesize y confidence level p probability of occurrence(050) q probability of nonoccurrence (050) N

total population and e permissible error(01)

35 Classifiers For each sampling subset (cross-validation23ndash13 and representative sample) 23 assembled algo-rithms were tested by combining them with the deep RNAMultilayer perceptron known as the Dl4jMlpClassifier al-gorithm in Waikato Environment for Knowledge Analysis(WEKA)

Double support I Single support

Initialcontact

Loadingresponse Midstance Terminal

stance

Double support II

PreswingInitialswing

Mediumswing

Terminalswing

Stance SwingStride

(gait cycle)

Figure 2 Gait cycle

Acquisition instrumentData base

Discrimination model

Result

Classifier

00101010011001000010101000101010

1 2 3 4 5 6 7

Sensors

(a) (b)

Figure 3 Gait laboratory [28] (a) General classification model (gait lab) (b) capture of biomarkers

6 Computational and Mathematical Methods in Medicine

(1) AdaBoostM1+Dl4jMlpClassifier(2) AdditiveRegression+Dl4jMlpClassifier(3) AttributeSelectedClassifier+Dl4jMlpClassifier(4) Bagging+Dl4jMlpClassifier(5) ClassificationViaClustering+Dl4jMlpClassifier(6) ClassificationViaRegression+Dl4jMlpClassifier(7) CostSensitiveClasifier+Dl4jMlpClassifier(8) CVParameterelection+Dl4jMlpClassifier(9) FilteredClassifier+Dl4jMlpClassifier(10) LogitBoost+Dl4jMlpClassifier(11) MetaCost+Dl4jMlpClassifier(12) MultiClassClassifier+Dl4jMlpClassifier(13) MultiClassClassifierUpdateable+Dl4jMlpClassifier(14) MultiScheme+Dl4jMlpClassifier(15) MultiSearch+Dl4jMlpClassifier(16) OneClassClassifier+Dl4jMlpClassifier(17) OrdinalClassClassifier+Dl4jMlpClassifier(18) RandomCommittee+Dl4jMlpClassifier(19) RandomizableFilteredClassifier+Dl4jMlpClassifier(20) RandomSubSpace+Dl4jMlpClassifier(21) Stacking+Dl4jMlpClassifier(22) -resholdSelector+Dl4jMlpClassifier(23) WeightedInstancesHandlerWrapper+Dl4jMlp

Classifier

-e combinations 2 5 7 11 13 14 and 16 were discardedsince the required nature of parameters could not be

implemented -e tests with the other combinations revealedthe best result with the representative sample test set and withthe combination of FilteredClassifier+Dl4jMlpClassifier clas-sifiers which are described below

(i) FilteredClassifier -is refers to a class in order toexecute an arbitrary base classifier (in this case theDl4jMlpClassifier) in data that have been passedthrough an arbitrary filter (in this case Discretize[33 34] which discretizes a range of numeric attri-butes in the dataset in nominal attributes) Like theclassifier the filter structure is based exclusively onthe training data and the test instances are processedby the filter without changing its structure If unequalinstance weights or attribute weights are present andthe filter or classifier cannot deal with them theinstances andor attributes are resampled with re-placement based on the weights before passing themto the filter or classifier (as appropriate)

(ii) Dl4jMlpClassifier -is is based on the multilayerperceptron (Algorithm 1) and is an artificial neuralnetwork made of multiple layers -e neurons ofthe hidden layer use the weighted sum of the in-puts with the synaptic weights wij as a rule ofpropagation and on that weighted sum a transferfunction of sigmoid type or hyperbolic tangent isapplied which is bounded in response -elearning that is usually used in this type of net-works is called backpropagation of the error Bothare increasing functions with two saturation levelsthe maximum which provides output 1 and theminimum which provides output 0 for the

Table 2 Characteristic distribution of the study subjects

Patient Gender Age Weight (kg) Height (cm) Suffering years Cause1 M 54 89 170 5 Hereditary2 M 60 108 165 10 Nutrition3 F 56 99 160 4 Hereditary4 M 56 815 162 6 Hereditary5 M 62 73 157 15 Nutrition6 F 50 70 159 8 Hereditary7 M 58 102 161 6 Nutrition8 M 57 877 158 8 Nutrition9 F 61 90 165 3 Hereditary10 M 50 832 163 5 Hereditary11 F 35 72 161 0 Healthy12 M 38 82 165 0 Healthy13 M 45 95 167 0 Healthy14 M 40 75 159 0 Healthy15 F 29 59 155 0 Healthy

Table 3 Dataset attributes with gait biomarkers fragment

rodDer-X rodDer-Y rodDer-Z rodIzq-X rodIzq-Y [ ] cad-Z Case187011821 208441092 206435782 23633454 Control216604624 214561741 207332365 23833754 Control206336545 213543299 207334487 23633567 Diseased175786965 205353455 204234677 23733436 Diseased

Computational and Mathematical Methods in Medicine 7

Tabl

e4

Statistical

measurements

ofnu

merical

attributes

ValEstad

rodD

er-X

rodD

er-Y

rodD

er-Z

rodIzq-X

rodIzq-Y

rodIzq-Z

tobD

er-X

tobD

er-Y

tobD

er-Z

tobIzq-X

tobIzq-Y

tobIzq-Z

cad-X

cad-Y

cad-Z

Minim

um0005

0005

0005

1045

1152

0337

1372

1328

125

1045

1152

0337

0786

1284

0684

Maxim

um3359

3379

3472

2568

2202

2705

1982

2202

2681

2568

2202

2705

2852

2471

2778

Mean

0464

0448

0446

2029

1711

1702

1745

1638

1952

2029

1711

1712

1779

1762

195

EstD

ev0595

0576

0563

013

009

0144

0075

0096

0173

013

009

0144

0206

0076

0215

8 Computational and Mathematical Methods in Medicine

sigmoidal function and output minus 1 for the hy-perbolic tangent

36 Generation of Random Weights A synaptic weightcalled wij is assigned for each input value Although thevalues are assigned randomly several methods exist in theliterature to generate these values One of them is Xavierrsquosmethod [35] which was implemented in this study asfollows Given a set of inputs x1 x2 xn1113864 1113865 the weights ofa distribution with zero mean and specific variance are

initialized Var(W) (2(nin + nout)) where Var(W) is thevariance of the initialized weights with a normal distri-bution (usually Gaussian or uniform) for the neuron inquestion and nin and nout are the input and output numberof neurons of a layer

37 Base Function -e base function f 1113936ni1wixi is ap-

plied to the input values with their assigned weights Inrelated work the base function is also called the summationof initial values the aggregation function and the network

14

10

20

15

25

10

20

15

25

10

2015

25

10

2015

25

16

18

20

22

24

141618202224

colx

colz

coly

(a)

00

10

05

20

15

25

30

10

00

20

30

10

0005

2015

253035

10

00

20

30

10

0005

2015

2530

10

00

05

20

15

25

30

35

rodDerx

rodDery

rodDerz

(b)

10

05

20

15

25

1005

2015

25

10

20

15

25

10

20

15

25

14

12

16

18

20

22

1412

16182022

rodlzqx

rodlzqy

rodlzqz

(c)

14

16

18

20

22

14

15

16

17

18

19

20

14151617181920

16

20

1416182022

12

24

16

20

12

24

tobDerx

tobDery

tobDerz

(d)

10

05

20

15

25

1005

2015

25

10

20

15

25

10

20

15

25

14

12

16

18

20

22

1412

16182022

toblzqz

toblzqy

toblzqx

(e)

Figure 4 Correlation of Cartesian coordinates of each sensor (a) gravity center (b) right knee (c) left knee (d) right ankle (e) left ankle

Computational and Mathematical Methods in Medicine 9

function among others and in general can have differentexpressions

38 Activation Function Given the sum of initial values theactivation function is obtained which is chosen according tothe task to be performed by the neuron For the multilayerperceptron the most used activation functions are thesigmoidal function and the hyperbolic tangent function

-ese functions have as image a continuous interval ofvalues within the intervals [1 1] and [0 1] and they aregiven by the following equations fsigm(x) (1(1 + eminus x))

and fthip(x) ((1 minus eminus x)(1 + eminus x)) -e activation functionused in this research is discussed in Section 422

39 Output Function -e output is given by theY F(X W) function where Y is the vector formed by theoutputs of network (y1 z2 y3 yn) X is the input vectorto network W is the set of all the network parameters ieweights and thresholds and F is a nonlinear function

310 Validation Metrics To validate the results the fol-lowing techniques were used

(i) -rough the confusion matrix each column rep-resents the predictions of each class while each rowrepresents the instances in the real class One of thebenefits of the confusion matrix is that it allows tosee if the model is confusing two classes that isrecognizing one class A as other class B

(ii) -rough the ROC space (receiver operating char-acteristic) which is elaborated from the sensitivity

and specificity values(iii) Validation of the medical specialist

4 Results and Discussion

41CombinationofAssembledAlgorithmsandDeepLearning-e raw data from the dataset described in Section 32 wereused for Creation of the database and the binary tests wereconducted diseased control as shown in Table 5 To dothis each assembled classifier of the WEKA family ofmetaclassifiers was combined with the deep learning algo-rithm multilayer perceptron with backward propagationDl4jMLPClassifier -e best result of the combination of

FilteredClassifier+Dl4jMlpClassifier was obtained with thecriterion of the representative sample

-e tests were performed using a Lenovo laptop G470Intel (R) Celeron (R) CPU B800 150Hz RAM 200GB64 bit Operating System Windows 7 Professional with theWEKA (available from httpwwwwekaorg) tool de-veloped by Witten and Frank [36]

42 Parameters Configuration of theDeep LearningAlgorithm

421 Iterations Table 5 shows that the best accuracy was850829 with 10 iterations (epochs) for training which isthe preset configurational parameter in WEKA -e resultswere confirmed or improved by conducting the tests byincreasing the iteration number to 20 30 40 50 60 70 8090 100 200 300 400 500 600 700 800 900 and 1000(Figure 5) Figure 5 does not show an elbow graph becausethe graph does not represent the search for the optimalnumber of elements for analysis rather it shows themaximum number of iterations of the algorithm needed toobtain the best performance

-e trend shows that with 40 iterations the percentageincreases to 8646 and does not show an increase in ac-curacy with higher iterations thus 40 iterations wereconsidered as the ideal value

422 Activation Functions -e preset activation functionin the WEKA tool is Softmax which was used to obtain themaximum classification percentage as mentioned before insection above It was also tested with Cube for 40 iterationsand the percentage of classified instances decreased (seeTable 6)

43 ValidationMetrics -e results were validated using thefollowing techniques

431 Confusion Matrix Accuracy was calculated from theequation ((TP + TN)total)(100) where TP are the truepositives TN are the true negatives and total is the numberof instances used for the test that is ((228 + 85)362)

(100) 8646 (see values in Table 7)Of the total number of test instances for the diseased

class 228 were classified correctly and 25 were confused withhealthy controls By contrast 85 instances were correctlyclassified out of the control class and 24 were confused

Input x1 x2 xn1113864 1113865

(1) begin(2) Synaptic weights are initialized wij to x1 x2 xn1113864 1113865 with Xavierrsquos method (see Section 36)(3) -e base function f is applied obtaining ui (see Section 37)(4) -e Softmax activation function is implemented on ui (see Section 38)(5) It is generalized by mean output function (see Section 39)(6) end

Output result y

ALGORITHM 1 Multilayer perceptron

10 Computational and Mathematical Methods in Medicine

432 ROC Space Sensitivity and Specificity -e ROC spacewas elaborated considering the values of sensitivity andspecificity which were calculated from the confusionmatrixas follows sensitivity (TP(TP + FN)) and specificity

1minus (FP(FP + TN)) where TP were true positives FN werefalse negatives and FP were false positives -e aboveequations gave a specificity of 077 and a sensitivity of 090

433 Expert Opinion -e medical specialist (Dr RobertoGerman Weber Burque Palacios) who validated this re-search based on his experience notes that at least for thestudy region the precision of 8646 is satisfactory for a firstapproach in this type of study concerning gait biomarkers inpatients with DN -is corroborates the Swets affirmationldquoIn clinical diagnosis when the sensitivity and specificityvalues represented in the Cartesian plane (or ROC space)

exceed 08 to the left (y axis) it can be considered appro-priaterdquo [37]

In this research patients and healthy individuals havebeen categorized with a high percentage of precision byapplying a combination of assembled classifiers and deeplearning to a dataset with gait biomarkers of DN-e expertsuggested a future collection of more gait information ofpatients affected by DN more healthy controls and pa-tients with another related disease that affects gait toobserve the performance of algorithm combination in amulticlass set

72

74

76

78

80

82

84

86

88

8508

7872

8232

8646

8508

779

8259

8378314

8259

8011

8232

84538563

8287

7707

8093

7784

8591

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19

Figure 5 Iterations for training

Table 5 Results of combining machine learning algorithms

Metaclassifier Deep learning 10-fold cross-validation 23ndash13 Representative sampleDl4jMlpClassifier 757 7905 801105

AdaBoostM1 Dl4jMlpClassifier 766333 755 803867AttributeSelectedClassifier Dl4jMlpClassifier 773833 774 781768Bagging Dl4jMlpClassifier 794167 7965 814917ClassificationViaRegression Dl4jMlpClassifier 666667 675 698895CVParameterelection Dl4jMlpClassifier 757 7905 801105FilteredClassifier Dl4jMlpClassifier 817333 845 850829LogitBoost Dl4jMlpClassifier 666667 675 698895MultiClassClassifier Dl4jMlpClassifier 757 7905 801105MultiSearch Dl4jMlpClassifier 757 7905 801105OrdinalClassClassifier Dl4jMlpClassifier 757 7905 801105RandomCommittee Dl4jMlpClassifier 791833 813 801105RandomizableFilteredClassifier Dl4jMlpClassifier 739 7675 798343RandomSubSpace Dl4jMlpClassifier 797333 807 787293Stacking Dl4jMlpClassifier 366667 675 698895-resholdSelector Dl4jMlpClassifier 729167 7665 477901WeightedInstancesHandlerWrapper Dl4jMlpClassifier 757 7905 801105

Table 6 Activation functions implemented and performance

Activation functions Softmax 8646Cube 7927

Computational and Mathematical Methods in Medicine 11

Another recent study has shown a positive predictive valueof 87 for detection of neuropathy in patients [38] -eclassification is based on pseudomotor dysfunction however itrequires a more expensive setup of equipment when comparedwith the cost of the sensors used here One of the objectives ofthis study was to provide a low-cost tool for early identificationof possible neuropathy A limitation of the present study whichcould be improved in future work involves the details of theclinical characterization of the patients such as the presence ofdiabetic complications -is information is important sincecomplications can bias the results

5 Conclusions and Future Work

-e results presented here confirm the assumption that acombination of metaclassifiers with deep learning cangenerate a reliable and acceptable classification percentage ofmore than 85 by categorizing the gait biomarkers of af-fected subjects with DN and healthy controls -e best resultobtained for the present study corresponds to the repre-sentative sample with 40 iterations In addition the con-vergence of disciplines is confirmed to help in solvingcomplex problemsmdashin this case the categorization of DN

-e results were obtained from patients suffering fromDN at different stages Diagnosis of patients with DN at theearly stages of disease is crucial and the high sensitivity ofthe motion sensors can allow the detection of gait patternsthat are otherwise imperceptible to the specialist

-e following seven efforts are considered worthwhilefor the continuation and improvement of this research (i)To corroborate the study with patients from other regions ofMexico taking into consideration both DN cases andhealthy controls in order to build a dataset of greater di-mensions and containing more information about gaitbiomarkers (ii) To add sensors that record other parameterssuch as heart rate temperature or others that provide ad-ditional relevant attributes and if possible that permitfeature selection (iii) To include information from otherbody limbs such as the arms and neck (iv) To develop an adhoc expert system to support studies of diabetic diseases withatrophy factors in the patientrsquos gait andor to assist thespecialist in predicting DN in persons given the efficiencyachieved by combining the metaclassifier with the deeplearning algorithm FilteredClassifier+Dl4jMlpClassifier-is proposed expert system motivated by the biometricrecognition of Hernandez et al [39] could be used onlinewith only basic and standard network protocols withoutrequirements for advanced network mechanisms (ie fromthe perspective of ubiquitous computing for a better ex-perience for study subjects) (v) To improve the results byconsidering the implementation of the use of the method ofCombined selection and optimization of hyperparameters ofclassification algorithms [40 41] to explore the behavior of

this method and to increase the maximum percentage of8646 achieved in the present research (vi) To extend thisstudy to other ailments that cause immobility such as os-teoarthritis as many other diseases are associated withmovement disorders (vii) To expand the database with morecases in future work

Data Availability

-e database used to support the findings of this study isavailable from the corresponding author upon request

Conflicts of Interest

-e authors declare that there are no conflicts of interest

Acknowledgments

-e authors would like to thank Dr Roberto Weber spe-cialist in diabetes mellitus for his advice and support withthe tests carried out on the volunteers in the gait laboratoryat the Universidad Juarez Autonoma de Tabasco MexicoAlso the authors are very thankful to personnel employeesand patients for their invaluable collaborations and to thestudent Fabiola Monrraga for technical support

References

[1] I Kavakiotis O Tsave A Salifoglou N MaglaverasI Vlahavas and I Chouvarda ldquoMachine learning and datamining methods in diabetes researchrdquo Computational andStructural Biotechnology Journal vol 15 pp 104ndash116 2017

[2] P Turaga R Chellappa V S Subrahmanian and O UdrealdquoMachine recognition of human activities a surveyrdquo IEEETransactions on Circuits and Systems for Video Technologyvol 18 no 11 pp 1473ndash1488 2008

[3] L-F Liu W Jia and Y-H Zhu ldquoSurvey of gait recognitionrdquoEmerging Intelligent Computing Technology and ApplicationsWith Aspects of Artificial Intelligence Springer Berlin Ger-many pp 652ndash659 2009

[4] D Gafurov ldquoA survey of biometric gait recognition ap-proaches security and challengesrdquo in Proceedings of the NIK-2007 Conference Washington DC USA September 2007

[5] F Monrraga Bernardino E Sanchez-DelaCruz andI V Meza Ruız ldquoKnee-ankle sensor for gait characterizationgender identification caserdquo in Intelligent Computing Systemspp 31ndash40 Springer Berlin Germany 2018

[6] A D Madrid ldquoiquestQue es la neuropatıa diabeticardquo 2016httpsdiabetesmadridorgneuropatia-diabetica

[7] N I of Diabetes Digestive and K Diseases NeuropatıasDiabeticas El dantildeo de los nervios NIDDK Bethesda MAUSA 2011 httpswwwniddknihgovhealth-informationinformacion-de-la-saluddiabetesinformacion-generalprevenir-problemasneuropatias-diabeticas

[8] P Ramırez-Lopez O Acevedo Giles and A Gonzalez PedrazaAviles ldquoNeuropatıa diabetica frecuencia factores de riesgo y

Table 7 Confusion matrix for the binary set diseased control

Diseased Healthy control Classified as228 25 Diseased24 85 Healthy control

12 Computational and Mathematical Methods in Medicine

calidad de vida en pacientes de una clınica de primer nivel deatencionrdquo Archivos en Medicina Familiar vol 19 pp 105ndash111 2017

[9] M J Mueller S D Minor S A Sahrmann J A Schaaf andM J Strube ldquoDifferences in the gait characteristics of patientswith diabetes and peripheral neuropathy compared with age-matched controlsrdquo Physical Gerapy vol 74 no 4 pp 299ndash308 1994

[10] I C N Sacco and A C Amadio ldquoA study of biomechanicalparameters in gait analysis and sensitive cronaxie of diabeticneuropathic patientsrdquo Clinical Biomechanics vol 15 no 3pp 196ndash202 2000

[11] I C N Sacco and A C Amadio ldquoInfluence of the diabeticneuropathy on the behavior of electromyographic and sen-sorial responses in treadmill gaitrdquo Clinical Biomechanicsvol 18 no 5 pp 426ndash434 2003

[12] O-Y Kwon S D Minor K S Maluf and M J MuellerldquoComparison of muscle activity during walking in subjectswith and without diabetic neuropathyrdquo Gait amp Posturevol 18 no 1 pp 105ndash113 2003

[13] G Yavuzer I Yetkin F B Toruner N Koca andN Bolukbasi ldquoGait deviations of patients with diabetesmellitus looking beyond peripheral neuropathyrdquo EuropaMedicophysica vol 42 pp 127ndash133 2006

[14] P M H Akashi I C N Sacco RWatari and E Hennig ldquo-eeffect of diabetic neuropathy and previous foot ulceration inEMG and ground reaction forces during gaitrdquo Clinical Bio-mechanics vol 23 no 5 pp 584ndash592 2008

[15] Z Sawacha F Spolaor G Guarneri et al ldquoAbnormal muscleactivation during gait in diabetes patients with and withoutneuropathyrdquo Gait amp Posture vol 35 no 1 pp 101ndash105 2012

[16] K Deschamps G A Matricali P Roosen et al ldquoComparisonof foot segmental mobility and coupling during gait betweenpatients with diabetes mellitus with and without neuropathyand adults without diabetesrdquo Clinical Biomechanics vol 28no 7 pp 813ndash819 2013

[17] M Fernando R Crowther P Lazzarini et al ldquoBiomechanicalcharacteristics of peripheral diabetic neuropathy a systematicreview and meta-analysis of findings from the gait cyclemuscle activity and dynamic barefoot plantar pressurerdquoClinical Biomechanics vol 28 no 8 pp 831ndash845 2013

[18] M R Patterson and B Caulfield ldquoUsing a foot mountedaccelerometer to detect changes in gait patternsrdquo in Pro-ceedings of the 2013 35th Annual International Conference ofthe IEEE Engineering inMedicine and Biology Society (EMBC)pp 7471ndash7475 Osaka Japan July 2013

[19] A A Gomes A Forner-Cordero M Ackermann andI C Sacco ldquoDynamic simulation of hip strategy of diabeticneuropathic individuals during gaitrdquo in Proceedings of the 5thIEEE RASEMBS International Conference on BiomedicalRobotics and Biomechatronics pp 211ndash215 Sao Paulo BrazilAugust 2014

[20] E Sanchez-Delacruz F Acosta-Escalante M A WisterJ A Hernandez-Nolasco P Pancardo and J J Mendez-Castillo ldquoGait recognition in the classification of neurode-generative diseasesrdquo in Proceedings of the 8th InternationalConference on Ubiquitous Computing and Ambient In-telligence pp 128ndash135 Springer Belfast UK December 2014

[21] E Sanchez-DelaCruz F Acosta-Escalante C Boll-Woehrlenet al ldquoCategorizacion de enfermedades neurodegenerativas apartir de biomarcadores de la marchardquo Komputer Sapiensvol 2 pp 17ndash20 2015

[22] M R Camargo J A Barela A J L NozabieliA M Mantovani A R Martinelli and C E P T Fregonesi

ldquoBalance and ankle muscle strength predict spatiotemporalgait parameters in individuals with diabetic peripheral neu-ropathyrdquo Diabetes amp Metabolic Syndrome Clinical Researchamp Reviews vol 9 no 2 pp 79ndash84 2015

[23] V Berki and B L Davis ldquoSpatial frequency content of plantarpressure and shear profiles for diabetic and non-diabeticsubjectsrdquo Journal of Biomechanics vol 49 no 15pp 3746ndash3748 2016

[24] L Anjaneya and M S Holi ldquoMultilayer machine learningalgorithm to classify diabetic type on knee datasetrdquo in Pro-ceedings of the 2016 IEEE International Conference on RecentTrends in Electronics Information amp Communication Tech-nology (RTEICT) pp 584ndash587 Bangalore India May 2016

[25] H M Al-Angari A H Khandoker S Lee et al ldquoNoveldynamic peak and distribution plantar pressure measures ondiabetic patients during walkingrdquo Gait amp Posture vol 51pp 261ndash267 2017

[26] J K Moore S K Hnat and A J van den Bogert ldquoAnelaborate data set on human gait and the effect of mechanicalperturbationsrdquo PeerJ vol 3 p e918 2015

[27] M D C Tongco ldquoPurposive sampling as a tool for informantselectionrdquo Ethnobotany Research and Applications vol 5pp 147ndash158 2007

[28] T Stockel R Jacksteit M Behrens R Skripitz R Bader andA Mau-Moeller ldquo-e mental representation of the humangait in young and older adultsrdquo Frontiers in Psychology vol 6p 943 2015

[29] J M Hausdorff A Lertratanakul M E CudkowiczA L Peterson D Kaliton and A L Goldberger ldquoDynamicmarkers of altered gait rhythm in amyotrophic lateral scle-rosisrdquo Journal of Applied Physiology vol 88 no 6pp 2045ndash2053 2000

[30] S J Pyo H Kim I S Kim et al ldquoQuantitative gait analysis inpatients with huntingtonrsquos diseaserdquo Journal of MovementDisorders vol 10 no 3 pp 140ndash144 2017

[31] M H -aut G C McIntosh R R Rice R A MillerJ Rathbun and J M Brault ldquoRhythmic auditory stimulationin gait training for Parkinsonrsquos disease patientsrdquo MovementDisorders vol 11 no 2 pp 193ndash200 1996

[32] A W Titus S Hillier Q A Louw and G Inglis-Jassiem ldquoAnanalysis of trunk kinematics and gait parameters in peoplewith strokerdquoAfrican Journal of Disability vol 7 pp 1ndash6 2018

[33] U Fayyad and K Irani ldquoMulti-interval discretization ofcontinuous-valued attributes for classification learningrdquo inProceedings of the 13th International Joint Conference onArtificial Intelligence pp 1022ndash1027 Chambery FranceAugustndashSeptember 1993

[34] I Kononenko ldquoOn biases in estimating multi-valued attri-butesrdquo in Proceedings of the 13th International Joint Con-ference on Artificial Intelligence pp 1034ndash1040 QuebecCanada August 1995

[35] X Glorot and Y Bengio ldquoUnderstanding the difficulty oftraining deep feedforward neural networksrdquo in Proceedings ofthe 13th International Conference on Artificial Intelligence andStatistics pp 249ndash256 Sardinia Italy May 2010

[36] I H Witten E Frank M A Hall and C J Pal Data MiningPractical Machine Learning Tools and Techniques MorganKaufmann Burlington MA USA 2016

[37] J A Swets Signal Detection Geory and ROC Analysis inPsychology and Diagnostics Collected Papers PsychologyPress London UK 2014

[38] A Carbajal-Ramırez J A Hernandez-DomınguezM A Molina-Ayala M M Rojas-Uribe and A Chavez-Negrete ldquoEarly identification of peripheral neuropathy based

Computational and Mathematical Methods in Medicine 13

on sudomotor dysfunction in Mexican patients with type 2diabetesrdquo BMC Neurology vol 19 no 1 p 109 2019

[39] J A Hernandez A O Ortiz J Andaverde and G BurlakldquoBiometrics in online assessments a study case in high schoolstudents electronics communications and computersrdquo inProceedings of the 18th International Conference on CON-IELECOMP 2008 pp 111ndash116 Cholula Puebla MexicoMarch 2008

[40] C -ornton F Hutter H H Hoos and K Leyton-BrownldquoAuto-WEKA combined selection and hyperparameter op-timization of classification algorithmsrdquo in Proceedings of the19th ACM SIGKDD International Conference on KnowledgeDiscovery and Data Mining pp 847ndash855 Association forComputing Machinery New York NY USA 2013

[41] L Kotthoff C -ornton H H Hoos F Hutter andK Leyton-Brown ldquoAuto-WEKA 20 automatic model se-lection and hyperparameter optimization in WEKArdquo Journalof Machine Learning Research vol 17 pp 1ndash5 2016

14 Computational and Mathematical Methods in Medicine