Data mining techniques using weka

Data Mining Techniques using WEKA

VINOD GUPTA SCHOOL OF MANAGEMENT, IIT KHARAGPUR

In partial fulfillmentOf the requirements for the degree of

MASTER OF BUSINESS ADMINISTRATION

SUBMITTED BY: Prashant Menon 10BM60061 VGSOM, IIT KHARAGPUR

Introduction to WEKA

WEKA is a collection of open source of many data mining and machine learning algorithms, including– pre-processing on data– Classification:– clustering– association rule extraction

• Created by researchers at the University of Waikato in New Zealand

• Java based (also open source).

Main features of WEK

• 49 data preprocessing tools

• 76 classification/regression algorithms

• 8 clustering algorithms

• 15 attribute/subset evaluators + 10 search algorithms for feature selection.

• 3 algorithms for finding association rules

• 3 graphical user interfaces

– “The Explorer” (exploratory data analysis)

– “The Experimenter” (experimental environment)

– “The KnowledgeFlow” (new process model inspired interface)

Weka: Download and Installation

• Download Weka (the stable version) from http://www.cs.waikato.ac.nz/ml/weka/

– Choose a self-extracting executable (including Java VM)

– (If one is interested in modifying/extending weka there is a developer version that includes the source code)• After download is completed, run the self extracting file to install Weka, and use the default set-ups.

Weka Application Interfaces

• Explorer– preprocessing, attribute selection, learning, visualiation• Experimenter– testing and evaluating machine learning algorithms• Knowledge Flow– visual design of KDD process– Explorer• Simple Command-line A simple interface for typing commands

Weka Functions and Tools

• Preprocessing Filters• Attribute selection• Classification/Regression• Clustering• Association discovery• Visualization

Load data file and Preprocessing

• Load data file in formats: ARFF, CSV, C4.5, binary• Import from URL or SQL database (using JDBC)• Preprocessing filters– Adding/removing attributes– Attribute value substitution– Discretization– Time series filters (delta, shift)– Sampling, randomization– Missing value management– Normalization and other numeric transformations

Feature Selection

• Very flexible: arbitrary combination of search and evaluation methods• Search methods

– best-first– genetic– ranking ...• Evaluation measures– ReliefF– information gain– gain ratio

Classification

• Predicted target must be categorical• Implemented methods– decision trees(J48, etc.) and rules– Naïve Bayes– neural networks– instance-based classifiers …• Evaluation methods– test data set

– crossvalidation

Clustering• Implemented methods– k-Means– EM– Cobweb– X-means– FarthestFirst…• Clusters can be visualized and compared to “true” clusters (if given)

Regression

• Predicted target is continuous• Methods– Linear regression– Simple Linear Regression– Neural networks– Regression trees …

Weka: Pros and cons

Pros– Open source,• Free• Extensible• Can be integrated into other java packages

– GUIs (Graphic User Interfaces)• Relatively easier to use– Features• Run individual experiment, or• Build KDD phases

Cons

– Lack of proper and adequate documentations– Systems are updated constantly (Kitchen Sink Syndrome)

3. WEKA data formats

• Data can be imported from a file in various formats:– ARFF (Attribute Relation File Format) has two sections:• the Header information defines attribute name, type and relations.• the Data section lists the data records.– CSV: Comma Separated Values (text file)– C4.5: A format used by a decision induction algorithmC4.5, requires two separated files• Name file: defines the names of the attributes• Date file: lists the records (samples)– binary• Data can also be read from a URL or from an SQL database (using JDBC)

This term paper will demonstrate the following two data mining techniques using WEKA:

Clustering (Simple K Means)

Linear regression

Clustering

Clustering allows a user to make groups of data to determine patterns from the data. Clustering has its advantages when the data set is defined and a general pattern needs to be determined from the data. One can create a specific number of groups, depending on business needs. One defining benefit of clustering over classification is that every attribute in the data set will be used to analyze the data. A major disadvantage of using clustering is that the user is required to know ahead of time how many groups he wants to create. For a user without any real knowledge of his data, this might be difficult. It might take several steps of trial and error to determine the ideal number of groups to create.However, for the average user, clustering can be the most useful data mining method one can use. It can quickly take the entire set of data and turn it into groups, from which one can quickly make some conclusions.

Data set for WEKA

This data set consists of three types of entities: (a) the specification of an auto in terms of various characteristics, (b)its assigned insurance risk rating, (c) its normalized losses in use as compared to other cars. The second rating corresponds to the degree to which the auto is more risky than its price indicates. Cars are initially assigned a risk factor symbol associated with its price. Then, if it is more risky (or less), this symbol is adjusted by moving it up (or down) the scale. A value of +3 indicates that the auto is risky, -3 that it is probably pretty safe.The third factor is the relative average loss payment per insured vehicle year. This value is normalized for all autos within a particular size classification (two-door small, station wagons, sports/speciality, etc...), and represents the average loss per car per year.

A part of the saved arff file.

@relation autos@attribute normalized-losses real@attribute make { alfa-romero, audi, bmw, chevrolet, dodge, honda, isuzu, jaguar, mazda, mercedes-benz, mercury, mitsubishi, nissan, peugot, plymouth, porsche, renault, saab, subaru, toyota, volkswagen, volvo}@attribute fuel-type { diesel, gas}@attribute aspiration { std, turbo}@attribute num-of-doors { four, two}

@attribute body-style { hardtop, wagon, sedan, hatchback, convertible}@attribute drive-wheels { 4wd, fwd, rwd}@attribute engine-location { front, rear}@attribute wheel-base real@attribute length real@attribute width real@attribute height real@attribute curb-weight real@attribute engine-type { dohc, dohcv, l, ohc, ohcf, ohcv, rotor}@attribute num-of-cylinders { eight, five, four, six, three, twelve, two}@attribute engine-size real@attribute fuel-system { 1bbl, 2bbl, 4bbl, idi, mfi, mpfi, spdi, spfi}@attribute bore real@attribute stroke real@attribute compression-ratio real@attribute horsepower real@attribute peak-rpm real@attribute city-mpg real@attribute highway-mpg real@attribute price real@attribute symboling { -3, -2, -1, 0, 1, 2, 3}

@data?,alfa-romero,gas,std,two,convertible,rwd,front,88.6,168.8,64.1,48.8,2548,dohc,four,130,mpfi,3.47,2.68,9,111,5000,21,27,13495,3?,alfa-romero,gas,std,two,convertible,rwd,front,88.6,168.8,64.1,48.8,2548,dohc,four,130,mpfi,3.47,2.68,9,111,5000,21,27,16500,3?,alfa-romero,gas,std,two,hatchback,rwd,front,94.5,171.2,65.5,52.4,2823,ohcv,six,152,mpfi,2.68,3.47,9,154,5000,19,26,16500,1164,audi,gas,std,four,sedan,fwd,front,99.8,176.6,66.2,54.3,2337,ohc,four,109,mpfi,3.19,3.4,10,102,5500,24,30,13950,2164,audi,gas,std,four,sedan,4wd,front,99.4,176.6,66.4,54.3,2824,ohc,five,136,mpfi,3.19,3.4,8,115,5500,18,22,17450,2?,audi,gas,std,two,sedan,fwd,front,99.8,177.3,66.3,53.1,2507,ohc,five,136,mpfi,3.19,3.4,8.5,110,5500,19,25,15250,2158,audi,gas,std,four,sedan,fwd,front,105.8,192.7,71.4,55.7,2844,ohc,five,136,mpfi,3.19,3.4,8.5,110,5500,19,25,17710,1?,audi,gas,std,four,wagon,fwd,front,105.8,192.7,71.4,55.7,2954,ohc,five,136,mpfi,3.19,3.4,8.5,110,5500,19,25,18920,1158,audi,gas,turbo,four,sedan,fwd,front,105.8,192.7,71.4,55.9,3086,ohc,five,131,mpfi,3.13,3.4,8.3,140,5500,17,20,23875,1?,audi,gas,turbo,two,hatchback,4wd,front,99.5,178.2,67.9,52,3053,ohc,five,131,mpfi,3.13,3.4,7,160,5500,16,22,?,0192,bmw,gas,std,two,sedan,rwd,front,101.2,176.8,64.8,54.3,2395,ohc,four,108,mpfi,3.5,2.8,8.8,101,5800,23,29,16430,2192,bmw,gas,std,four,sedan,rwd,front,101.2,176.8,64.8,54.3,2395,ohc,four,108,mpfi,3.5,2.8,8.8,101,5800,23,29,16925,0188,bmw,gas,std,two,sedan,rwd,front,101.2,176.8,64.8,54.3,2710,ohc,six,164,mpfi,3.31,3.19,9,121,4250,21,28,20970,0188,bmw,gas,std,four,sedan,rwd,front,101.2,176.8,64.8,54.3,2765,ohc,six,164,mpfi,3.31,3.19,9,121,4250,21,28,21105,0?,bmw,gas,std,four,sedan,rwd,front,103.5,189,66.9,55.7,3055,ohc,six,164,mpfi,3.31,3.19,9,121,4250,20,25,24565,1

?,bmw,gas,std,four,sedan,rwd,front,103.5,189,66.9,55.7,3230,ohc,six,209,mpfi,3.62,3.39,8,182,5400,16,22,30760,0?,bmw,gas,std,two,sedan,rwd,front,103.5,193.8,67.9,53.7,3380,ohc,six,209,mpfi,3.62,3.39,8,182,5400,16,22,41315,0?,bmw,gas,std,four,sedan,rwd,front,110,197,70.9,56.3,3505,ohc,six,209,mpfi,3.62,3.39,8,182,5400,15,20,36880,0121,chevrolet,gas,std,two,hatchback,fwd,front,88.4,141.1,60.3,53.2,1488,l,three,61,2bbl,2.91,3.03,9.5,48,5100,47,53,5151,298,chevrolet,gas,std,two,hatchback,fwd,front,94.5,155.9,63.6,52,1874,ohc,four,90,2bbl,3.03,3.11,9.6,70,5400,38,43,6295,181,chevrolet,gas,std,four,sedan,fwd,front,94.5,158.8,63.6,52,1909,ohc,four,90,2bbl,3.03,3.11,9.6,70,5400,38,43,6575,0118,dodge,gas,std,two,hatchback,fwd,front,93.7,157.3,63.8,50.8,1876,ohc,four,90,2bbl,2.97,3.23,9.41,68,5500,37,41,5572,1118,dodge,gas,std,two,hatchback,fwd,front,93.7,157.3,63.8,50.8,1876,ohc,four,90,2bbl,2.97,3.23,9.4,68,5500,31,38,6377,1118,dodge,gas,turbo,two,hatchback,fwd,front,93.7,157.3,63.8,50.8,2128,ohc,four,98,mpfi,3.03,3.39,7.6,102,5500,24,30,7957,1148,dodge,gas,std,four,hatchback,fwd,front,93.7,157.3,63.8,50.6,1967,ohc,four,90,2bbl,2.97,3.23,9.4,68,5500,31,38,6229,1148,dodge,gas,std,four,sedan,fwd,front,93.7,157.3,63.8,50.6,1989,ohc,four,90,2bbl,2.97,3.23,9.4,68,5500,31,38,6692,1

Clustering in WEKA

Load the data file AUTOS.arff into WEKA using the same steps we used to load data into the Preprocess tab. Take a few minutes to look around the data in this tab. Look at the columns, the attribute data, the distribution of the columns, etc. The screen should look like the figure shown below after loading the data.

With this data set, we are looking to create clusters, so instead of clicking on the Classify tab, click on the Cluster tab. Click Choose and select SimpleKMeans from

the choices that appear (this will be our preferred method of clustering for this article). WEKA Explorer window should look like the following figure at this point.

Finally, we want to adjust the attributes of our cluster algorithm by clicking SimpleKMeans (not the best UI design here, but go with it). The only attribute of the algorithm we are interested in adjusting here is the numClusters field, which tells us how many clusters we want to create. (Remember, one need to know this before start.) Let's change the default value of 2 to 4 for now, but keep these steps in mind later if one wants to adjust the number of clusters created. WEKA Explorer should look like the following at this point. Click OK to accept these values.

At this point, we are ready to run the clustering algorithm. Remember that this much rows of data with four data clusters would likely take a few hours of computation with a spreadsheet, but WEKA can spit out the answer in less than a second. The output should look like the figure shown below.

Time taken to build model (full training data) : 0.02 seconds

=== Model and evaluation on training set ===

Clustered Instances

0 60 ( 29%)1 33 ( 16%)2 55 ( 27%)3 57 ( 28%)

Based on the values of cluster centroids as shown in the above figure, we can state the characteristics of each of the clusters. For explaniantion we are taking Cluster 1 and Cluster 2

Cluster 2

This group will always look for the premium segment car ‘Peugot’ . Has the larget wheel base, length, height, curb weight, engine size. As the engine size is inversely proportional to the mileage , it has the lowest city and high way mileage. It has the highest number of cylinders.Compression ratio , horse power, peak rpm all have the highest value which make it a highest priced Car.

Cluster 1

This group will always look for the ‘VALUE FOR MONEY’ car. It belongs to the mass segment. As the engine power is inversely proportional to the mileage , we can see it has

the highest highway and city mileage and low compression ration, horse power and RPM. For this segment price is one of the important criteria before buying the car.

The Cluster analysis will help the car company which segment it should target before start of the new product development/bringing the car into the market.

Visualization of Clustering Results A more intuitive way to go through the results is to visualize them in the graphical form. To do so:

Right click the result in the Result list panel

Select Visualize cluster assignments

By setting X-axis variable as Cluster, Y-axis variable as Instance_number and Color as aspiration, we get the following output:

Here we can see all the clusters (segments) have mixed response to the aspiration.

Similarly we can change the variables in X-axis, Y-axis and color to visualize other aspects of result. Note that WEKA has generated an extra variable named “Cluster” (not present in original data) which signifies the cluster membership of various instances. We can save the output as an arff file by clicking on the save button. The output file contains an additional attribute cluster for each instance. Thus besides the value of twenty six attributes for any instance, the output also specifies the cluster membership for that instance.

Creating the regression model with WEKA

To create the model, click on the Classify tab. The first step is to select the model we want to build, so WEKA knows how to work with the data, and how to create the appropriate model:

1. Click the Choose button, then expand the functions branch.2. Select the LinearRegression leaf.

This tells WEKA that we want to build a regression model. As one can see from the other choices, though, there are lots of possible models to build.This should give a good indication of how we are only touching the surface of this subject. There is another choice called SimpleLinearRegression in the same branch. Do not choose this because simple regression only looks at one variable, and we have six.

The used attributes are as follows:

The used attributes are : MYCT: machine cycle time in nanoseconds (integer) MMIN: minimum main memory in kilobytes (integer) MMAX: maximum main memory in kilobytes (integer)CACH: cache memory in kilobytes (integer)CHMIN: minimum channels in units (integer)CHMAX: maximum channels in units (integer) PRP: published relative performance (integer) (target variable)

A part of the data file is as follows:

@relation machine_cpu@attribute MYCT numeric@attribute MMIN numeric@attribute MMAX numeric@attribute CACH numeric@attribute CHMIN numeric@attribute CHMAX numeric@attribute class numeric@data125,256,6000,256,16,128,19829,8000,32000,32,8,32,26929,8000,32000,32,8,32,22029,8000,32000,32,8,32,17229,8000,16000,32,8,16,13226,8000,32000,64,8,32,31823,16000,32000,64,16,32,36723,16000,32000,64,16,32,48923,16000,64000,64,16,32,636

When we've selected the right model, WEKA Explorer should look like the following figure.

Now that the desired model has been chosen, we have to tell WEKA where the data is that it should use to build the model. Though it may be obvious to us that we want to use the data we supplied in the ARFF file, there are actually different options, some more advanced than what we'll be using. The other three choices are Supplied test set, where one can supply a different set of data to build the model; Cross-validation, which lets WEKA build a model based on subsets of the supplied data and then average them out to create a final model; and Percentage split, where WEKA takes a percentile subset of the supplied data to build a final model. These other choices are useful with different models, which we'll see in future articles. With regression, we can simply choose Use training set. This tells WEKA that to build our desired model, we can simply use the data set we supplied in our ARFF file.Finally, the last step to creating our model is to choose the dependent variable (the column we are looking to predict). We know this should be the Class, since that's what we're trying to determine. Right below the test options, there's a combo box that lets us choose the dependent variable. The column Class should be selected by default. If it's not, please select it.

Now we are ready to create our model. Click Start. The following figure shows what the output should look like.

=== Run information ===

Scheme: weka.classifiers.functions.LinearRegression -S 0 -R 1.0E-8

Relation: machine_cpu

Instances: 209

Attributes: 7

MYCT

MMIN

MMAX

CACH

CHMIN

CHMAX

class

Test mode: evaluate on training data

=== Classifier model (full training set) ===

Linear Regression Model

class =

0.0491 * MYCT +

0.0152 * MMIN +

0.0056 * MMAX +

0.6298 * CACH +

1.4599 * CHMAX +

-56.075

Time taken to build model: 0 seconds

=== Evaluation on training set ===

=== Summary ===

Correlation coefficient 0.93

Mean absolute error 37.9748

Root mean squared error 58.9899

Relative absolute error 39.592 %

Root relative squared error 36.7663 %

Total Number of Instances 209

Here class represents PRP(Published Relative Performance)

Interpreting the regression model

It puts the regression model right there in the output, as shown in Listing below:

Class(PRP) = 0.0491 * MYCT + 0.0152 * MMIN + 0.0056 * MMAX + 0.6298 * CACH + 1.4599 * CHMAX -56.075

Listing 2 shows the results, plugging in the values for my house.

Listing 2. PRP value using regression model

Class(PRP)= 0.0491 * 29 + 0.0152 * 8000 + .0056 * 32000 + .6298 * 32 + 1.4599 * 32 – 56.075

PRP= 1.4239 + 121.6 + 179.2 + 20.1536 + 46.7168-56.075

PRP = 313.0193

However, looking back to the top of the article, data mining isn't just about outputting a single number: It's about identifying patterns and rules. It's not strictly used to produce an absolute number but rather to create a model that lets us detect patterns, predict output, and come up with conclusions backed by the data

Minimum channel units doesn't matter — WEKA will only use columns that statistically contribute to the accuracy of the model (measured in R-squared). It will throw out and ignore columns that don't help in creating a good model. So this regression model is telling us that minimum channels doesn't affect PRP.

We can also visualize the classifier error i.e. those instances which are wrongly predicted by regression equation by right clinking on the result set in the Result list panel and selecting Visualize classifier errors.

The X-axis has Price (actual) and the Y-axis has Predicted Price.