Page 1
Department of Computer Science,
University of Waikato, New Zealand
Eibe Frank
WEKA: A Machine
Learning Toolkit
The Explorer
• Classification and
Regression
• Clustering
• Association Rules
• Attribute Selection
• Data Visualization
The Experimenter
The Knowledge
Flow GUI
Conclusions
Machine Learning with WEKA
Page 2
5/6/2018 University of Waikato 2
WEKA: the bird
Copyright: Martin Kramer ([email protected] )
Page 3
5/6/2018 University of Waikato 3
WEKA: the software
Machine learning/data mining software written in
Java (distributed under the GNU Public License)
Used for research, education, and applications
Complements “Data Mining” by Witten & Frank
Main features:
Comprehensive set of data pre-processing tools,
learning algorithms and evaluation methods
Graphical user interfaces (incl. data visualization)
Environment for comparing learning algorithms
Page 4
5/6/2018 University of Waikato 4
WEKA: versions
There are several versions of WEKA:
WEKA 3.0: “book version” compatible with
description in data mining book
WEKA 3.2: “GUI version” adds graphical user
interfaces (book version is command-line only)
WEKA 3.3: “development version” with lots of
improvements
This talk is based on the latest snapshot of WEKA
3.3 (soon to be WEKA 3.4)
Page 5
5/6/2018 University of Waikato 5
@relation heart-disease-simplified
@attribute age numeric
@attribute sex { female, male}
@attribute chest_pain_type { typ_angina, asympt, non_anginal, atyp_angina}
@attribute cholesterol numeric
@attribute exercise_induced_angina { no, yes}
@attribute class { present, not_present}
@data
63,male,typ_angina,233,no,not_present
67,male,asympt,286,yes,present
67,male,asympt,229,yes,present
38,female,non_anginal,?,no,not_present
...
WEKA only deals with “flat” files
Page 6
5/6/2018 University of Waikato 6
@relation heart-disease-simplified
@attribute age numeric
@attribute sex { female, male}
@attribute chest_pain_type { typ_angina, asympt, non_anginal, atyp_angina}
@attribute cholesterol numeric
@attribute exercise_induced_angina { no, yes}
@attribute class { present, not_present}
@data
63,male,typ_angina,233,no,not_present
67,male,asympt,286,yes,present
67,male,asympt,229,yes,present
38,female,non_anginal,?,no,not_present
...
WEKA only deals with “flat” files
Page 7
5/6/2018 University of Waikato 7
Page 8
5/6/2018 University of Waikato 8
Page 9
5/6/2018 University of Waikato 9
Page 10
5/6/2018 University of Waikato 10
Explorer: pre-processing the data
Data can be imported from a file in various
formats: ARFF, CSV, C4.5, binary
Data can also be read from a URL or from an SQL
database (using JDBC)
Pre-processing tools in WEKA are called “filters”
WEKA contains filters for:
Discretization, normalization, resampling, attribute
selection, transforming and combining attributes, …
Page 11
5/6/2018 University of Waikato 11
Page 12
5/6/2018 University of Waikato 12
Page 13
5/6/2018 University of Waikato 13
Page 14
5/6/2018 University of Waikato 14
Page 15
5/6/2018 University of Waikato 15
Page 16
5/6/2018 University of Waikato 16
Page 17
5/6/2018 University of Waikato 17
Page 18
5/6/2018 University of Waikato 18
Page 19
5/6/2018 University of Waikato 19
Page 20
5/6/2018 University of Waikato 20
Page 21
5/6/2018 University of Waikato 21
Page 22
5/6/2018 University of Waikato 22
Page 23
5/6/2018 University of Waikato 23
Page 24
5/6/2018 University of Waikato 24
Page 25
5/6/2018 University of Waikato 25
Page 26
5/6/2018 University of Waikato 26
Page 27
5/6/2018 University of Waikato 27
Page 28
5/6/2018 University of Waikato 28
Page 29
5/6/2018 University of Waikato 29
Page 30
5/6/2018 University of Waikato 30
Page 31
5/6/2018 University of Waikato 31
Page 32
5/6/2018 University of Waikato 32
Explorer: building “classifiers”
Classifiers in WEKA are models for predicting
nominal or numeric quantities
Implemented learning schemes include:
Decision trees and lists, instance-based classifiers,
support vector machines, multi-layer perceptrons,
logistic regression, Bayes’ nets, …
“Meta”-classifiers include:
Bagging, boosting, stacking, error-correcting output
codes, locally weighted learning, …
Page 33
5/6/2018 University of Waikato 33
Page 34
5/6/2018 University of Waikato 34
Page 35
5/6/2018 University of Waikato 35
Page 36
5/6/2018 University of Waikato 36
Page 37
5/6/2018 University of Waikato 37
Page 38
5/6/2018 University of Waikato 38
Page 39
5/6/2018 University of Waikato 39
Page 40
5/6/2018 University of Waikato 40
Page 41
5/6/2018 University of Waikato 41
Page 42
5/6/2018 University of Waikato 42
Page 43
5/6/2018 University of Waikato 43
Page 44
5/6/2018 University of Waikato 44
Page 45
5/6/2018 University of Waikato 45
Page 46
5/6/2018 University of Waikato 46
Page 47
5/6/2018 University of Waikato 47
Page 48
5/6/2018 University of Waikato 48
Page 49
5/6/2018 University of Waikato 49
Page 50
5/6/2018 University of Waikato 50
Page 51
5/6/2018 University of Waikato 51
Page 52
5/6/2018 University of Waikato 52
Page 53
5/6/2018 University of Waikato 53
Page 54
5/6/2018 University of Waikato 54
Page 55
5/6/2018 University of Waikato 55
Page 56
5/6/2018 University of Waikato 56
Page 57
5/6/2018 University of Waikato 57
Page 58
5/6/2018 University of Waikato 58
Page 59
5/6/2018 University of Waikato 59
Page 60
5/6/2018 University of Waikato 60
Page 61
5/6/2018 University of Waikato 61
Page 62
5/6/2018 University of Waikato 62
Page 63
5/6/2018 University of Waikato 63
Page 64
5/6/2018 University of Waikato 64
Page 65
5/6/2018 University of Waikato 65 QuickTime™ and a TIFF (LZW) decompressor are needed to see this picture.
Page 66
5/6/2018 University of Waikato 66 QuickTime™ and a TIFF (LZW) decompressor are needed to see this picture.
Page 67
5/6/2018 University of Waikato 67 QuickTime™ and a TIFF (LZW) decompressor are needed to see this picture.
Page 68
5/6/2018 University of Waikato 68
Page 69
5/6/2018 University of Waikato 69
Page 70
5/6/2018 University of Waikato 70
Page 71
5/6/2018 University of Waikato 71
Page 72
5/6/2018 University of Waikato 72
Page 73
5/6/2018 University of Waikato 73
Page 74
5/6/2018 University of Waikato 74
Page 75
5/6/2018 University of Waikato 75
Quic k Time™ and a TIFF (LZW) dec ompres s or are needed to s ee this pic ture.
Page 76
5/6/2018 University of Waikato 76
Page 77
5/6/2018 University of Waikato 77
Page 78
5/6/2018 University of Waikato 78
Page 79
5/6/2018 University of Waikato 79
Page 80
5/6/2018 University of Waikato 80
QuickTime™ and a TIFF (LZW) decompressor are needed to see this picture.
Page 81
5/6/2018 University of Waikato 81
QuickTime™ and a TIFF (LZW) decompressor are needed to see this picture.
Page 82
5/6/2018 University of Waikato 82
Page 83
5/6/2018 University of Waikato 83
QuickTime™ and a TIFF (LZW) decompressor are needed to see this picture.
Page 84
5/6/2018 University of Waikato 84
Page 85
5/6/2018 University of Waikato 85
Page 86
5/6/2018 University of Waikato 86
Page 87
5/6/2018 University of Waikato 87
Page 88
5/6/2018 University of Waikato 88
Page 89
5/6/2018 University of Waikato 89
Page 90
5/6/2018 University of Waikato 90
Page 91
5/6/2018 University of Waikato 91
Page 92
5/6/2018 University of Waikato 92
Explorer: clustering data
WEKA contains “clusterers” for finding groups of
similar instances in a dataset
Implemented schemes are:
k-Means, EM, Cobweb, X-means, FarthestFirst
Clusters can be visualized and compared to “true”
clusters (if given)
Evaluation based on loglikelihood if clustering
scheme produces a probability distribution
Page 93
5/6/2018 University of Waikato 93
Page 94
5/6/2018 University of Waikato 94
Page 95
5/6/2018 University of Waikato 95
Page 96
5/6/2018 University of Waikato 96
Page 97
5/6/2018 University of Waikato 97
Page 98
5/6/2018 University of Waikato 98
Page 99
5/6/2018 University of Waikato 99
Page 100
5/6/2018 University of Waikato 100
Page 101
5/6/2018 University of Waikato 101
Page 102
5/6/2018 University of Waikato 102
Page 103
5/6/2018 University of Waikato 103
Page 104
5/6/2018 University of Waikato 104
Page 105
5/6/2018 University of Waikato 105
Page 106
5/6/2018 University of Waikato 106
Page 107
5/6/2018 University of Waikato 107
Page 108
5/6/2018 University of Waikato 108
Explorer: finding associations
WEKA contains an implementation of the Apriori
algorithm for learning association rules
Works only with discrete data
Can identify statistical dependencies between
groups of attributes:
milk, butter bread, eggs (with confidence 0.9 and
support 2000)
Apriori can compute all rules that have a given
minimum support and exceed a given confidence
Page 109
5/6/2018 University of Waikato 109
Page 110
5/6/2018 University of Waikato 110
Page 111
5/6/2018 University of Waikato 111
Page 112
5/6/2018 University of Waikato 112
Page 113
5/6/2018 University of Waikato 113
Page 114
5/6/2018 University of Waikato 114
Page 115
5/6/2018 University of Waikato 115
Page 116
5/6/2018 University of Waikato 116
Explorer: attribute selection
Panel that can be used to investigate which
(subsets of) attributes are the most predictive ones
Attribute selection methods contain two parts:
A search method: best-first, forward selection,
random, exhaustive, genetic algorithm, ranking
An evaluation method: correlation-based, wrapper,
information gain, chi-squared, …
Very flexible: WEKA allows (almost) arbitrary
combinations of these two
Page 117
5/6/2018 University of Waikato 117
Page 118
5/6/2018 University of Waikato 118
Page 119
5/6/2018 University of Waikato 119
Page 120
5/6/2018 University of Waikato 120
Page 121
5/6/2018 University of Waikato 121
Page 122
5/6/2018 University of Waikato 122
Page 123
5/6/2018 University of Waikato 123
Page 124
5/6/2018 University of Waikato 124
Page 125
5/6/2018 University of Waikato 125
Explorer: data visualization
Visualization very useful in practice: e.g. helps to
determine difficulty of the learning problem
WEKA can visualize single attributes (1-d) and
pairs of attributes (2-d)
To do: rotating 3-d visualizations (Xgobi-style)
Color-coded class values
“Jitter” option to deal with nominal attributes (and
to detect “hidden” data points)
“Zoom-in” function
Page 126
5/6/2018 University of Waikato 126
Page 127
5/6/2018 University of Waikato 127
Page 128
5/6/2018 University of Waikato 128
Page 129
5/6/2018 University of Waikato 129
Page 130
5/6/2018 University of Waikato 130
Page 131
5/6/2018 University of Waikato 131
Page 132
5/6/2018 University of Waikato 132
Page 133
5/6/2018 University of Waikato 133
Page 134
5/6/2018 University of Waikato 134
Page 135
5/6/2018 University of Waikato 135
Page 136
5/6/2018 University of Waikato 136
Page 137
5/6/2018 University of Waikato 137
Page 138
5/6/2018 University of Waikato 138
Performing experiments
Experimenter makes it easy to compare the
performance of different learning schemes
For classification and regression problems
Results can be written into file or database
Evaluation options: cross-validation, learning
curve, hold-out
Can also iterate over different parameter settings
Significance-testing built in!
Page 139
5/6/2018 University of Waikato 139
Page 140
5/6/2018 University of Waikato 140
Page 141
5/6/2018 University of Waikato 141
Page 142
5/6/2018 University of Waikato 142
Page 143
5/6/2018 University of Waikato 143
Page 144
5/6/2018 University of Waikato 144
Page 145
5/6/2018 University of Waikato 145
Page 146
5/6/2018 University of Waikato 146
Page 147
5/6/2018 University of Waikato 147
Page 148
5/6/2018 University of Waikato 148
Page 149
5/6/2018 University of Waikato 149
Page 150
5/6/2018 University of Waikato 150
Page 151
5/6/2018 University of Waikato 151
Page 152
5/6/2018 University of Waikato 152
The Knowledge Flow GUI
New graphical user interface for WEKA
Java-Beans-based interface for setting up and
running machine learning experiments
Data sources, classifiers, etc. are beans and can
be connected graphically
Data “flows” through components: e.g.,
“data source” -> “filter” -> “classifier” -> “evaluator”
Layouts can be saved and loaded again later
Page 153
5/6/2018 University of Waikato 153
Page 154
5/6/2018 University of Waikato 154
Page 155
5/6/2018 University of Waikato 155
Page 156
5/6/2018 University of Waikato 156
Page 157
5/6/2018 University of Waikato 157
Page 158
5/6/2018 University of Waikato 158
Page 159
5/6/2018 University of Waikato 159
Page 160
5/6/2018 University of Waikato 160
Page 161
5/6/2018 University of Waikato 161
Page 162
5/6/2018 University of Waikato 162
Page 163
5/6/2018 University of Waikato 163
Page 164
5/6/2018 University of Waikato 164
Page 165
5/6/2018 University of Waikato 165
Page 166
5/6/2018 University of Waikato 166
Page 167
5/6/2018 University of Waikato 167
Page 168
5/6/2018 University of Waikato 168
Page 169
5/6/2018 University of Waikato 169
Page 170
5/6/2018 University of Waikato 170
Page 171
5/6/2018 University of Waikato 171
Page 172
5/6/2018 University of Waikato 172
Page 173
5/6/2018 University of Waikato 173
Conclusion: try it yourself!
WEKA is available at
http://www.cs.waikato.ac.nz/ml/weka
Also has a list of projects based on WEKA
WEKA contributors:
Abdelaziz Mahoui, Alexander K. Seewald, Ashraf M. Kibriya, Bernhard
Pfahringer , Brent Martin, Peter Flach, Eibe Frank ,Gabi Schmidberger
,Ian H. Witten , J. Lindgren, Janice Boughton, Jason Wells, Len Trigg,
Lucio de Souza Coelho, Malcolm Ware, Mark Hall ,Remco Bouckaert ,
Richard Kirkby, Shane Butler, Shane Legg, Stuart Inglis, Sylvain Roy,
Tony Voyle, Xin Xu, Yong Wang, Zhihai Wang