Data Mining Algorithms Prof. S. Sudarshan CSE Dept, IIT Bombay Most Slides Courtesy Prof. Sunita Sarawagi School of IT, IIT Bombay.

Data Mining Algorithms

Prof. S. Sudarshan CSE Dept, IIT Bombay

Most Slides Courtesy Prof. Sunita Sarawagi

School of IT, IIT Bombay

Overview

Decision Tree classification algorithms

Clustering algorithmsChallengesResources

Decision Tree Classifiers

Decision tree classifiers

Widely used learning method Easy to interpret: can be re-represented as if-then-else

rules Approximates function by piece wise constant regions Does not require any prior knowledge of data

distribution, works well on noisy data. Has been applied to:

classify medical patients based on the disease, equipment malfunction by cause, loan applicant by likelihood of payment.

Setting

Given old data about customers and payments, predict new applicant’s loan eligibility.

AgeSalaryProfessionLocationCustomer type

Previous customers Classifier Decision rules

Salary > 5 L

Prof. = Exec

New applicant’s data

Good/bad

Tree where internal nodes are simple decision rules on one or more attributes and leaf nodes are predicted class labels.

Decision trees

Salary < 1 M

Prof = teaching

Good

Age < 30

BadBad Good

Topics to be coveredTree construction:

Basic tree learning algorithm Measures of predictive ability High performance decision tree construction: Sprint

Tree pruning: Why prune Methods of pruning

Other issues: Handling missing data Continuous class labels Effect of training size

Tree learning algorithms

ID3 (Quinlan 1986)Successor C4.5 (Quinlan 1993)SLIQ (Mehta et al)SPRINT (Shafer et al)

Basic algorithm for tree building

Greedy top-down construction.

Gen_Tree (Node, data)

make node a leaf?Yes Stop

Find best attribute and best split on attribute

Partition data on split condition

For each child j of node Gen_Tree (node_j, data_j)

Selectioncriteria

Split criteria

Select the attribute that is best for classification.

Intuitively pick one that best separates instances of different classes.

Quantifying the intuitive: measuring separability:

First define impurity of an arbitrary set S consisting of K classes

1

Impurity Measures

Information entropy:

Zero when consisting of only one class, one when all classes in equal number

Other measures of impurity: Gini:

k

iii ppSEntropy

1

log)(

k

iipSGini

1

21)(

Split criteria

K classes, set of S instances partitioned into r subsets. Instance Sj has fraction pij instances of class j.

Information entropy:

Gini index:

r

j

k

iijij

j ppS

S

1 1

log

)(1 1

21

r

j

k

iij

j pS

S

0 1Impurity

1/4

Gini

r =1, k=2

Information gain

Information gain on partitioning S into r subsets

Impurity (S) - sum of weighted impurity of each subset

r

jj

jr SEntropy

S

SSEntropySSSGain

11 )()()..,(

Information gain: example

S

K= 2, |S| = 100, p1= 0.6, p2= 0.4E(S) = -0.6 log(0.6) - 0.4 log

(0.4)=0.29

S1 S2

| S1 | = 70, p1= 0.8, p2= 0.2E(S1) = -0.8log0.8 - 0.2log0.2 = 0.21

| S2| = 30, p1= 0.13, p2= 0.87

E(S2) = -0.13log0.13 - 0.87 log 0.87=.16

Information gain: E(S) - (0.7 E(S1 ) + 0.3 E(S2) ) =0.1

Meta learning methods

No single classifier good under all casesDifficult to evaluate in advance the conditionsMeta learning: combine the effects of the

classifiers Voting: sum up votes of component classifiers Combiners: learn a new classifier on the outcomes of

previous ones: Boosting: staged classifiers

Disadvantage: interpretation hard Knowledge probing: learn single classifier to mimic

meta classifier

SPRINT (Serial PaRallelizable INduction of decision Trees)

Decision-tree classifier for data mining

Design goals: Able to handle large disk-resident

training sets No restrictions on training-set size Easily parallelizable

Example

Example DataAge Car Type42 family

18 truck

57 sports

21 sports

28 family

72 truck

Age < 25

CarType in {sports}

High

High Low

RiskLow

High

High

High

Low

Low

Building tree

GrowTree(TrainingData D) Partition(D);

Partition(Data D) if (all points in D belong to the same class) then return; for each attribute A do evaluate splits on attribute A; use best split found to partition D into D1 and D2; Partition(D1); Partition(D2);

Data Setup: Attribute Lists One list for each attribute Entries in an Attribute List consist of:

attribute value class value record id

Lists for continuous attributes are in sorted order Lists may be disk-resident Each leaf-node has its own set of attribute lists

representing the training examples belonging to that leaf

Age Risk RID17 High 120 High 5

23 High 0

32 Low 4

43 High 2

68 Low 3

Example list:

Attribute Lists: ExampleAge Car Type Risk23 family High17 sports High43 sports High

68 family Low

32 truck Low

20 family High

Car Type Risk RIDfamily High 0sports High 1sports High 2

family Low 3

truck Low 4

family High 5

Age Risk RID23 High 017 High 143 High 2

68 Low 3

32 Low 4

20 High 5

Age Risk RID17 High 120 High 5

23 High 0

32 Low 4

43 High 2

68 Low 3

Car Type Risk RIDfamily High 0sports High 1sports High 2

family Low 3

truck Low 4

family High 5

Initial Attribute Lists for the root node:

Evaluating Split Points

Gini Index if data D contains examples from c

classesGini(D) = 1 - pj2

where pj is the relative frequency of class j in D If D split into D1 & D2 with n1 & n2 tuples each

Ginisplit(D) = n1* gini(D1) + n2* gini(D2) n n

Note: Only class frequencies are needed to compute index

Finding Split Points

For each attribute A do evaluate splits on attribute A using

attribute list

Keep split with lowest GINI index

Finding Split Points: Continuous Attrib.

Consider splits of form: value(A) < x Example: Age < 17

Evaluate this split-form for every value in an attribute list

To evaluate splits on attribute A for a given tree-node:

Initialize class-histogram of left child to zeroes;Initialize class-histogram of right child to same as its parent;

for each record in the attribute list doevaluate splitting index for value(A) <

record.value;using class label of the record, update class

histograms;

Finding Split Points: Continuous Attrib.

Age Risk RID23 High 017 High 1

43 High 2

68 Low 3

32 Low 4

20 High 5

Attribute List

High Low4 2

High Low0 0

High Low4 2

High Low4 2

High Low4 2

High Low0 0

High Low4 2

Position of cursor in scan

0: Age < 17

3: Age < 32

6

State of Class Histograms:

Left Child Right Child

1: Age < 20

High Low0 0

High Low0 0

GINI Index:

GINI = undef

GINI = 0.4

GINI = 0.222

GINI = undef

Finding Split Points: Categorical Attrib.

Consider splits of the form: value(A) {x1, x2, ..., xn} Example: CarType {family, sports}

Evaluate this split-form for subsets of domain(A) To evaluate splits on attribute A for a given tree

node:initialize class/value matrix of node to zeroes;for each record in the attribute list do

increment appropriate count in matrix;evaluate splitting index for various subsets using the constructed matrix;

Finding Split Points: Categorical Attrib.

Attribute List

High Low

family 2 1

sports 2 0

truck 0 1

class/value matrix

Car Type Risk RIDfamily High 0

sports High 1

sports High 2

family Low 3

truck Low 4

family High 5

CarType in {family}High Low2 1

High Low2 1

Left Child Right Child GINI Index:

High Low2 1

High Low2 1

CarType in {truck}

GINI = 0.444

GINI = 0.267

High Low2 0

High Low2 0

CarType in {sports} GINI = 0.333

Performing the Splits

The attribute lists of every node must be divided among the two children

To split the attribute lists of a give node:for the list of the attribute used to split this node do

use the split test to divide the records;collect the record ids;

build a hashtable from the collected ids;

for the remaining attribute lists douse the hashtable to divide each list;

build class-histograms for each new leaf;

Performing the Splits: Example

Age < 32

Age Risk RID17 High 120 High 5

23 High 0

32 Low 4

43 High 2

68 Low 3

Car Type Risk RIDfamily High 0sports High 1sports High 2

family Low 3

truck Low 4

family High 5

Age Risk RID17 High 120 High 5

23 High 0

Age Risk RID32 Low 443 High 2

68 Low 3

Car Type Risk RIDfamily High 0sports High 1

family High 5

Car Type Risk RIDsports High 2

family Low 3

truck Low 4

Hash Table0 Left1 Left2 Right3 Right4 Right5 Left

Sprint: summary

Each node of the decision tree classifier, requires examining possible splits on each value of each attribute.

After choosing a split attribute, need to partition all data into its subset.

Need to make this search efficient. Evaluating splits on numeric attributes:

Sort on attribute value, incrementally evaluate gini Splits on categorical attributes

For each subset, find gini and choose the best For large sets, use greedy method

Approaches to prevent overfitting

Stop growing the tree beyond a certain point

First over-fit, then post prune. (More widely used) Tree building divided into phases:

Growth phasePrune phase

Hard to decide when to stop growing the tree, so second appraoch more widely used.

Criteria for finding correct final tree size:

Cross validation with separate test data Use all data for training but apply statistical

test to decide right size. Use some criteria function to choose best size

Example: Minimum description length (MDL) criteria

Cross validation approach: Partition the dataset into two disjoint parts:

1. Training set used for building the tree.2. Validation set used for pruning the tree

Build the tree using the training-set. Evaluate the tree on the validation set and at each

leaf and internal node keep count of correctly labeled data.

Starting bottom-up, prune nodes with error less than its children.

Cross validation..

Need large validation set to smooth out over-fittings of training data. Rule of thumb: one-third.

What if training data set size is limited? Generate many different parititions of data. n-fold cross validation: partition training data into

n parts D1, D2…Dn. Train n classifiers with D-Di as training and Di as

test instance. Pick average.

Rule-based pruning

Tree-based pruning limits the kind of pruning. If a node is pruned all subtrees under it has to be pruned.

Rule-based: For each leaf of the tree, extract a rule using a conjuction of all tests upto the root.

On the validation set, independently prune tests from each rule to get the highest accuracy for that rule.

Sort rule by decreasing accuracy..

MDL-based pruning

Idea: a branch of the tree is over-fitted if the training examples that fit under it can be explicitly enumerated (with classes) in less space than occupied by tree

Prune branch if over-fitted philosophy: use tree that minimizes

description length of training data

Regression trees

Decision tree with continuous class labels:

Regression trees approximates the function with piece-wise constant regions.

Split criteria for regression trees: Predicted value for a set S = average of all

values in S Error: sum of the square of error of each

member of S from the predicted average. Pick smallest average error.

Issues

Multiple splits on continuous attributes [Fayyad 93, Multi-interval discretization of continuous attributes]

Multi attribute tests on nodes to handle correlated attributes multivariate linear splits [Oblique trees, Murthy 94]

Methods of handling missing values assume majority value take most probable path

Allowing varying costs for different attributes

Pros and Cons of decision trees

� Cons Cannot handle complicated relationship between features simple decision boundaries problems with lots of missing data

� Pros+ Reasonable training time+ Fast application+ Easy to interpret+ Easy to implement+ Can handle large number of features

More information: http://www.recursive-partitioning.com/

Clustering or Unsupervised learning

Distance functions

Numeric data: euclidean, manhattan distances Minkowski metric: [sum(xi-yi)^m]^(1/m) Larger m gives higher weight to larger distances

Categorical data: 0/1 to indicate presence/absence Euclidean distance: equal weightage to 1 and 0 match Hamming distance (# dissimilarity) Jaccard coefficients: #similarity in 1s/(# of 1s) (0-0

matches not important Combined numeric and categorical data:weighted normalized

distance:

Distance functions on high dimensional data

Example: Time series, Text, Images Euclidian measures make all points equally far Reduce number of dimensions:

choose subset of original features using random projections, feature selection techniques

transform original features using statistical methods like Principal Component Analysis

Define domain specific similarity measures: e.g. for images define features like number of objects, color histogram; for time series define shape based measures.

Define non-distance based (model-based) clustering methods:

Clustering methods

Hierarchical clustering agglomerative Vs divisive single link Vs complete link

Partitional clustering distance-based: K-means model-based: EM density-based:

Partitional methods: K-meansCriteria: minimize sum of square of distance

Between each point and centroid of the cluster.Between each pair of points in the cluster

Algorithm: Select initial partition with K clusters: random,

first K, K separated points

Repeat until stabilization:Assign each point to closest cluster centerGenerate new cluster centersAdjust clusters by merging/splitting

Properties

May not reach global optimaConverges fast in practice: guaranteed for

certain forms of optimization function Complexity: O(KndI):

I number of iterations, n number of points, d number of dimensions, K number of clusters.

Database research on scalable algorithms: Birch: one/two pass of data by keeping R-tree

like index in memory [Sigmod 96]

Model based clustering

Assume data generated from K probability distributions

Typically Gaussian distribution Soft or probabilistic version of K-means clustering

Need to find distribution parameters.EM Algorithm

EM Algorithm

Initialize K cluster centersIterate between two steps

Expectation step: assign points to clusters

Maximation step: estimate model parameters

)),,() |Pr(

)Pr(/) |Pr()Pr() |Pr() (

ikkki

ikikikki

dNcd

dcdcdccdP

m

ik

ji

kiik cdP

cdPd

m 1 ) (

) (1

Properties

May not reach global optimaConverges fast in practice:

guaranteed for certain forms of optimization function

Complexity: O(KndI): I number of iterations, n number of

points, d number of dimensions, K number of clusters.

Scalable clustering algorithms

Birch: one/two pass of data by keeping R-tree like index in memory [Sigmod 96]

Fayyad and Bradley: Sample repetitively and update summary of clusters stored in memory (K-mean and EM) [KDD 98]

Dasgupta 99: Recent theoretical breakthrough, find Gaussian clusters with guaranteed performance Random projections

To Learn More

Books

Ian H. Witten and Frank Eibe,Data mining : practical machine learning tools and techniques with Java implementations, Morgan Kaufmann, 1999

Usama Fayyad et al. (eds), Advances in Knowledge Discovery and Data Mining, AAAI/MIT Press, 1996

Tom Mitchell, Machine Learning, McGraw-Hill

SoftwarePublic domain

Weka 3: data mining algos in Java (http://www.cs.waikato.ac.nz/~ml/weka)classification, regression

MLC++: data mining tools in C++mainly classification

Free for universities try convincing IBM to give it free!

Datasets: follow links from www.kdnuggets.com to UC Irvine site

Resources

http://www.kdnuggets.com Great site with links to software, datasets etc. Be

sure to visit it.

http://www.cs.bham.ac.uk/~anp/TheDataMine.html OLAP: http://altaplana.com/olap/ SIGKDD: http://www.acm.org/sigkdd Data mining and knowledge discovery journal:

http://www.research.microsoft.com/research/datamine/

Communications of ACM Special Issue on Data Mining, Nov 1996

Resources at IITB

http://www.cse.iitb.ernet.in/~dbms IITB DB group home page

http://www.it.iitb.ernet.in/~sunita/it642 Data Warehousing and Data Mining

course offered by Prof. Sunita Sarawagi at IITB