Data Mining. Jim Jim ’ s cows Which cows should I buy??

Data MiningData Mining

JimJim Jim’s cowsJim’s cows

Which cows should I

buy??

Jim’s cowsJim’s cows

NameNameMilk Milk Avg. Avg. (MA)(MA)

AGEAGERatingRating

MonaMona6655GoodGood

LisaLisa4466BadBad

MaryMary8833GoodGood

QuirriQuirri6655BadBad

PaulaPaula2266GoodGood

AbdulAbdul101077BadBad

Cows on Cows on salesale

NameNameMilk Milk Avg. Avg. (MA)(MA)

AGAGEE

PhilPhil5533

CollinsCollins3322

LarryLarry9955

BirdBird2255

Which cows should I

buy??

Which cows should I buy??

And suppose I know their behavior, And suppose I know their behavior, preferred mating months, milk preferred mating months, milk production, nutritional habits, production, nutritional habits, immune system data…?immune system data…?

Now suppose I have 10,000 cows…Now suppose I have 10,000 cows…

““understanding” dataunderstanding” data Trying to find patterns in data is not new: Trying to find patterns in data is not new:

hunters seek patterns in animal migration, hunters seek patterns in animal migration, politicians in voting habits, people in their politicians in voting habits, people in their partner’s behavior, etc. partner’s behavior, etc.

However, the amount of available data is However, the amount of available data is increasing very fast (exponentially?).increasing very fast (exponentially?).

This gives greater opportunities to extract This gives greater opportunities to extract valuable information from the data.valuable information from the data.

But it also makes the task of But it also makes the task of “understanding” the data with conventional “understanding” the data with conventional tools very difficult.tools very difficult.

Data MiningData Mining Data Mining:Data Mining: The process of discovering The process of discovering

patterns in data, usually stored in a patterns in data, usually stored in a Database. The patterns lead to Database. The patterns lead to advantages (economic or other).advantages (economic or other).

Very fast growing area of researchVery fast growing area of research Because:Because:

Huge databases (Walmart-20 mil Huge databases (Walmart-20 mil transactions/day)transactions/day)

Automatic data capture of transactions (Bar Automatic data capture of transactions (Bar code, satellites, scanners, cameras, etc.)code, satellites, scanners, cameras, etc.)

Large financial advantageLarge financial advantage Evolving analytical methodsEvolving analytical methods

Data Mining techniques in Data Mining techniques in some huji coursessome huji courses

TechniqueTechniqueCourseCourse

Decision TreesDecision TreesArtificial IntelligenceArtificial Intelligence

EM, Perceptron, SVM, EM, Perceptron, SVM, PCAPCA……

Intro. to Machine Intro. to Machine LearningLearning

Intro. to information Intro. to information processing and processing and

LearningLearning

Neural NetworksNeural NetworksNeural Networks 1, 2Neural Networks 1, 2..

K-Nearest NeighborK-Nearest NeighborComputational Computational GeometryGeometry

Data MiningData Mining

Two extremes for the expression of Two extremes for the expression of the patterns:the patterns:

1.1. ““Black Box”:Black Box”: “Buy cow Zehava, Petra “Buy cow Zehava, Petra and Paulina”and Paulina”

2.2. ““Transparent Box” (Structural Transparent Box” (Structural Patterns):Patterns): “Buy cows with age<4 and “Buy cows with age<4 and weight >300 or cows with calm weight >300 or cows with calm behavior and >90 liters of milk behavior and >90 liters of milk production per month”production per month”

The weather exampleThe weather exampleOutlookOutlookTemp.Temp.HumidityHumidityWindyWindyPlayPlay

SunnySunnyHotHotHighHighFalseFalseNoNo

SunnySunnyHotHotHighHighTrueTrueNo No

OvercastOvercastHotHotHighHighFalseFalseYesYes

RainyRainyMildMildHighHighFalseFalseYesYes

RainyRainyCoolCoolNormalNormalFalseFalseYesYes

RainyRainyCoolCoolNormalNormalTrue True NoNo

OvercastOvercastCoolCoolNormalNormalTrueTrueYesYes

SunnySunnyMildMildHighHighFalseFalseNoNo

SunnySunnyCoolCoolNormalNormalFalseFalseYesYes

Today is Overcast, mild temperature, high humidity, and windy. Will we play?Today is Overcast, mild temperature, high humidity, and windy. Will we play?

Questions one can askQuestions one can ask A set of rules learned from this data could be A set of rules learned from this data could be

presented in a presented in a Decision ListDecision List:: If outlook=sunny and humidity=high then If outlook=sunny and humidity=high then play=noplay=no ElseIf outlook=rainy and windy=true then ElseIf outlook=rainy and windy=true then play=noplay=no ElseIf outlook=overcast then ElseIf outlook=overcast then play=yesplay=yes ElseIf humidity=normal then ElseIf humidity=normal then play=yesplay=yes Else Else play=yesplay=yes

This is an example of This is an example of Classification RulesClassification Rules We could also look forWe could also look for Association Rules: Association Rules:

If temperature=cool then humidity=normalIf temperature=cool then humidity=normal If windy=false and play=no then outlook=sunny If windy=false and play=no then outlook=sunny

and and humidity=highhumidity=high

Example ContExample Cont..

The previous example is very simplified. The previous example is very simplified. Real Databases will probably:Real Databases will probably:

1.1. Contain Numerical values as well.Contain Numerical values as well.

2.2. Contain “Noise” and errors.Contain “Noise” and errors.

3.3. Be a lot larger.Be a lot larger. And the analysis we are asked to And the analysis we are asked to

perform might not be of Association perform might not be of Association Rules, but rather Decision Trees, Neural Rules, but rather Decision Trees, Neural Networks, etc.Networks, etc.

CautionCaution David Rhine was a parapsychologist in the 1930-David Rhine was a parapsychologist in the 1930-

1950’s1950’s He hypothesized that some people have Extra-He hypothesized that some people have Extra-

Sensory Perception (ESP)Sensory Perception (ESP) He asked people to say if 10 hidden cards are red He asked people to say if 10 hidden cards are red

or blue.or blue. He discovered that almost 1 in every 1000 people He discovered that almost 1 in every 1000 people

has ESP !has ESP ! He told these people that they have ESP and called He told these people that they have ESP and called

them in for another testthem in for another test He discovered almost all of them had lost their ESP He discovered almost all of them had lost their ESP

!! He concluded that…He concluded that… You shouldn’t tell people they have ESP, it caused You shouldn’t tell people they have ESP, it caused

them to loose it.them to loose it.[Source: J. Ullman][Source: J. Ullman]

Another ExampleAnother Example A classic example is a Database which holds A classic example is a Database which holds

data concerning purchases in a supermarket.data concerning purchases in a supermarket. Each Each Shopping BasketShopping Basket is a list of items that is a list of items that

were bought in a single purchase by some were bought in a single purchase by some customer. customer.

Such huge DB’s which are saved for long Such huge DB’s which are saved for long periods of time are called periods of time are called Data Data WarehousesWarehouses..

It is extremely valuable for the manager of It is extremely valuable for the manager of the store to extract Association Rules from the store to extract Association Rules from the huge Data Warehouse. the huge Data Warehouse.

It is even more valuable if this information It is even more valuable if this information can be associated with the person buying, can be associated with the person buying, hence the Club Memberships… hence the Club Memberships…

Supermarket ExampleSupermarket Example For example, if Beer and Diapers are found For example, if Beer and Diapers are found

to be bought together often, this might to be bought together often, this might encourage the manager to give a discount encourage the manager to give a discount for purchasing Beer, Diapers and a new for purchasing Beer, Diapers and a new product together.product together.

Another example: if older people are found Another example: if older people are found to be more “loyal” to a certain brand than to be more “loyal” to a certain brand than young people, a manager might not promote young people, a manager might not promote a new brand of shampoo, intended for older a new brand of shampoo, intended for older people.people.

transiditem

111111penpen

111111inkink

111111milkmilk

111111juicejuice

112112penpen

112112inkink

112 112 milkmilk

113113penpen

113113milkmilk

114114penpen

114 114 inkink

114114juicejuice

The Purchases RelationItemset: A set of

items

Support of an itemset: the fraction of transactions that contain all items in the itemset.What is the Support

of:

1.{pen}?

2.{pen, ink}?

3.{pen, juice}?

Frequent ItemsetsFrequent Itemsets We would like to find items that are We would like to find items that are

purchased together in high frequency- purchased together in high frequency- Frequent ItemsetsFrequent Itemsets. .

We look for itemsets which have a We look for itemsets which have a support > minSupport.support > minSupport.

If minSupport is set to 0.7, then the If minSupport is set to 0.7, then the frequent itemsets in our example would be:frequent itemsets in our example would be:

{pen}, {ink}, {milk}, {pen, ink}, {pen, {pen}, {ink}, {milk}, {pen, ink}, {pen, milk}milk}

The A-Priori property of frequent The A-Priori property of frequent itemsets: itemsets: Every subset of a frequent Every subset of a frequent itemset is also a frequent itemset.itemset is also a frequent itemset.

Algorithm for finding Frequent Algorithm for finding Frequent itemsetsitemsets

Suppose we have n items.Suppose we have n items. The naïve approach: for every subset of items, The naïve approach: for every subset of items,

check if it is frequent.check if it is frequent. Very expensiveVery expensive Improvement (based on the A-priori property): Improvement (based on the A-priori property):

first identify frequent itemsets of size 1, then try first identify frequent itemsets of size 1, then try to expand them.to expand them.

Greatly reduces the number of candidate Greatly reduces the number of candidate frequent itemsets.frequent itemsets.

A single scan of the table is enough to determine A single scan of the table is enough to determine which candidate itemsets, are frequent.which candidate itemsets, are frequent.

The algorithm terminates when no new frequent The algorithm terminates when no new frequent itemsets are found in an iteration.itemsets are found in an iteration.

Algorithm for finding Frequent Algorithm for finding Frequent itemsetsitemsets

foreach item, check if it is a frequent itemset; foreach item, check if it is a frequent itemset; (appears in >minSupport of the transactions)(appears in >minSupport of the transactions)

k=1;k=1;repeatrepeat

foreach new frequent itemset Iforeach new frequent itemset Ikk with k items: with k items:Generate all itemsets IGenerate all itemsets Ik+1 k+1 with k+1 items, such with k+1 items, such

that Ithat Ik k is contained inis contained in IIk+1.k+1.

scan all transactions once and add itemsets scan all transactions once and add itemsets that have support > minSupport.that have support > minSupport.

k++k++until until no new frequent itemsets are foundno new frequent itemsets are found

Finding Frequent itemsets, on table Finding Frequent itemsets, on table “Purchases”, with minSupport=0.7“Purchases”, with minSupport=0.7

In the first run, the following single itemsets are In the first run, the following single itemsets are found to be frequent: {found to be frequent: {penpen}, {}, {inkink}, {}, {milkmilk}.}.

Now we generate the candidates for k=2: {Now we generate the candidates for k=2: {penpen, , inkink}, {}, {penpen, , milkmilk}, {}, {penpen,, juice juice}, {}, {inkink, , milkmilk}, }, {{inkink,, juice juice} and {} and {milkmilk,, juice juice}.}.

By scanning the relation, we determine that the By scanning the relation, we determine that the following are frequent: {following are frequent: {penpen, , inkink}, {}, {penpen, , milkmilk}.}.

Now we generate the candidates for k=3: {Now we generate the candidates for k=3: {penpen, , inkink, , milkmilk}, {}, {penpen, , milkmilk, , juicejuice}, {}, {penpen, , inkink, , juicejuice}.}.

By scanning the relation, we determine that By scanning the relation, we determine that none of these are frequent, and the algorithm none of these are frequent, and the algorithm ends with: { {ends with: { {penpen}, {}, {inkink}, {}, {milkmilk}, {}, {penpen, , inkink}, {}, {penpen, , milkmilk} }} }

Algorithm refinementAlgorithm refinement:: One important refinement:One important refinement: after the candidate- after the candidate-

generation phase, and before the scan of the relation generation phase, and before the scan of the relation (A-priori), eliminate candidate itemsets in which (A-priori), eliminate candidate itemsets in which there is a subset which is not frequent. This is due to there is a subset which is not frequent. This is due to the A-Priori property.the A-Priori property.

In the second iteration, this means we would In the second iteration, this means we would eliminate {eliminate {penpen,, juice juice}, {}, {inkink,, juice juice} and {} and {milkmilk,, juice juice} } as candidates since {as candidates since {juicejuice} is not frequent. So we } is not frequent. So we only check {only check {penpen, , inkink}, {}, {penpen, , milkmilk} and {} and {inkink, , milkmilk}. }.

So only {So only {penpen, , ink,ink, milkmilk} is generated as a candidate, } is generated as a candidate, but it is eliminated before the scan because {but it is eliminated before the scan because {inkink, , milkmilk} is not frequent.} is not frequent.

So we don’t perform the 3So we don’t perform the 3rdrd iteration of the relation. iteration of the relation. More complex algorithms use the same tools: More complex algorithms use the same tools:

iterative generationiterative generation and and testing of candidate testing of candidate itemsetsitemsets..

Association RulesAssociation Rules Up until now we discussed identification of Up until now we discussed identification of

frequent item sets. We now wish to go one step frequent item sets. We now wish to go one step further.further.

An association rule is of the structure An association rule is of the structure {pen}=> {ink}{pen}=> {ink}

It should be read as: “if a pen is purchased in a It should be read as: “if a pen is purchased in a transaction, it is likely that ink will also be transaction, it is likely that ink will also be purchased in that transaction”.purchased in that transaction”.

It describes the data in the DB (past). It describes the data in the DB (past). Extrapolation to future transactions should be Extrapolation to future transactions should be done with caution.done with caution.

More formally, an Association Rule is LHS=>RHS, More formally, an Association Rule is LHS=>RHS, where both LHS and RHS are sets of items, and where both LHS and RHS are sets of items, and implies that if every item in LHS was purchased implies that if every item in LHS was purchased in a transaction, it is likely that the items in RHS in a transaction, it is likely that the items in RHS are purchased as well.are purchased as well.

Measures for Association Measures for Association RulesRules

1.1. Support of “LHS=>RHS”Support of “LHS=>RHS” is the support is the support of the itemset (LHS of the itemset (LHS UU RHS). In other RHS). In other words: the words: the fraction of transactions that contain all items in (LHS LHS U U RHS)RHS) .

2. Confidence of “LHS=>RHS”:“LHS=>RHS”: Consider Consider all transactions which contain all items in all transactions which contain all items in LHS. The fraction of these transactions LHS. The fraction of these transactions that also contain all items in RHS, is the that also contain all items in RHS, is the confidence of RHS. confidence of RHS. =S(LHS U RHS)/S(LHS) =S(LHS U RHS)/S(LHS)

The confidence of a rule is an indication The confidence of a rule is an indication of the strength of the rule.of the strength of the rule.

What is the support of {pen}=>{ink}? And the confidence?

What is the support of {ink}=>{pen}? And the confidence?

Finding Association rulesFinding Association rules

A user can ask for rules with minimum A user can ask for rules with minimum support minSup and minimum confidence support minSup and minimum confidence minConf.minConf.

Firstly, all frequent itemsets with Firstly, all frequent itemsets with support>minSup are computed with the support>minSup are computed with the previous Algorithm.previous Algorithm.

Secondly, rules are generated using the Secondly, rules are generated using the frequent itemsets, and checked for minConf.frequent itemsets, and checked for minConf.

Finding Association rulesFinding Association rules

Find all frequent itemsets using the previous Find all frequent itemsets using the previous alg.alg.

For each frequent itemset X with support For each frequent itemset X with support S(X):S(X):For each division of X into 2 itemsets:For each division of X into 2 itemsets:

Divide X into 2 itemsets LHS and RHS.Divide X into 2 itemsets LHS and RHS.

The Confidence of LHS=>RHS is S(X)/S(LHS). The Confidence of LHS=>RHS is S(X)/S(LHS).

We computed S(LHS) in the previous We computed S(LHS) in the previous algorithm (because LHS is frequent since X is algorithm (because LHS is frequent since X is frequent). frequent).

Generalized association Generalized association rulesrulestransiddateitem

1111111.5.991.5.99penpen

1111111.5.991.5.99inkink

1111111.5.991.5.99MilkMilk

1111111.5.991.5.99juicejuice

11211210.5.9910.5.99penpen

11211210.5.9910.5.99inkink

112 112 10.5.9910.5.99milkmilk

11311315.5.9915.5.99PenPen

11311315.5.9915.5.99milkmilk

1141141.6.991.6.99PenPen

114 114 1.6.991.6.99InkInk

1141141.6.991.6.99juicejuice

We would like to know if the rule {pen}=>{juice} is different on the first day of the month compared to other days. How?

What are its support and confidence

generally ?

And on the first days of the

month ?

Generalized association Generalized association rulesrules

By specifying different attributes to group by By specifying different attributes to group by (date in the last example), we can come up (date in the last example), we can come up with interesting rules which we would with interesting rules which we would otherwise miss.otherwise miss.

Another example would be to group by Another example would be to group by location and check if the same rules apply for location and check if the same rules apply for customers from Jerusalem compared to Tel customers from Jerusalem compared to Tel Aviv. Aviv.

By comparing the support and confidence of By comparing the support and confidence of the rules we can observe differences in the the rules we can observe differences in the data on different conditions. data on different conditions.

Caution in predictionCaution in prediction

When we find a pattern in the data, we wish to When we find a pattern in the data, we wish to use it for prediction (that is in many case the use it for prediction (that is in many case the point).point).

However, we have to be cautious about this. However, we have to be cautious about this. For example: suppose {pen}=>{ink} has a high For example: suppose {pen}=>{ink} has a high

support and confidence. We might give a support and confidence. We might give a discount on pens in order to increase sales of discount on pens in order to increase sales of pens and therefore also in sales of ink.pens and therefore also in sales of ink.

However, this assumes a However, this assumes a causal link causal link between between {pen} and {ink}. {pen} and {ink}.

Caution in predictionCaution in prediction Suppose pens and pencils are always sold Suppose pens and pencils are always sold

together together We would then also get the rule We would then also get the rule

{pencil}=>{ink} with the same support and {pencil}=>{ink} with the same support and confidence as {pen}=>{ink}confidence as {pen}=>{ink}

However, it is clear there is no causal link However, it is clear there is no causal link between buying pencils and buying ink. between buying pencils and buying ink.

If we promoted pencils it would not cause an If we promoted pencils it would not cause an increase in sales of ink, despite high support and increase in sales of ink, despite high support and confidence.confidence.

The chance to infer “wrong” rules (rules which The chance to infer “wrong” rules (rules which are not causal links) decreases as the DB size are not causal links) decreases as the DB size increases, but we should keep in mind that such increases, but we should keep in mind that such rules do come up.rules do come up.

Therefore, the generated rules are a only good Therefore, the generated rules are a only good starting point for identifying causal links. starting point for identifying causal links.

Classification and Regression Classification and Regression rulesrules

Consider the following relation: Consider the following relation: InsuranceInfo(InsuranceInfo(ageage integer, integer, carTypecarType string, string, highRiskhighRisk

bool)bool) The relation holds information about current The relation holds information about current

customers. customers. The company wants to use the data in order to The company wants to use the data in order to

predict if a new customer, whose age and carType predict if a new customer, whose age and carType are known, is at high risk (and therefore charge are known, is at high risk (and therefore charge higher insurance fee of course).higher insurance fee of course).

Such a rule for example could be “if age is Such a rule for example could be “if age is between 18 and 23, and carType is either ‘sports’ between 18 and 23, and carType is either ‘sports’ or ‘truck’, the risk is high”.or ‘truck’, the risk is high”.

Classification and Regression Classification and Regression rulesrules

Such rules, where we are only interested in Such rules, where we are only interested in predicting one attribute are special.predicting one attribute are special.

The attribute which we predict is called the The attribute which we predict is called the DependentDependent attribute. attribute.

The other attributes are called the The other attributes are called the PredictorPredictor attributes.attributes.

If the dependant attribute is If the dependant attribute is categoricalcategorical, we , we call such rules call such rules classification rulesclassification rules..

If the dependent attribute is If the dependent attribute is numericalnumerical, we call , we call such rules such rules regression rulesregression rules..

Regression in a nutshellRegression in a nutshell

NameName

BlooBlood d

pres. pres. (BP)(BP)

Milk Milk AveraAvera

ge ge (MA)(MA)

AGEAGENOCNOCRatinRatingg

MonaMona727266552299

LisaLisa797944661177

MarryMarry898988334433

QuirriQuirri565666552299

PaulaPaula777722664477

AbdulAbdul90901010778833

VickyVicky6969445533??

Jim’s cows

(training set)

new cow (test set)

Regression in a nutshellRegression in a nutshell Assume that the Rate is a linear Assume that the Rate is a linear

combination of the other attributes:combination of the other attributes:Rate= wRate= w0 0 + w+ w11*BP + w*BP + w22*MA + w*MA + w33*AGE + *AGE +

ww44*NOC*NOC

Our goal is thus to find wOur goal is thus to find w0, 0, ww1, 1, ww2, 2, ww3, 3, ww4 4

(which actually means how strongly each (which actually means how strongly each attribute affects the Rate)attribute affects the Rate)

We thus want to minimize:We thus want to minimize:

ΣΣ(Rate(Rate(i)(i)-[w-[w00++ ww1*1*BPBP(i)(i) ++ ww2*2*MAMA(i)(i)

++ ww3*3*AGEAGE(i)(i) ++

ww4*4*NOCNOC(i) (i) ])])

i

Prediction of Rate using

w0-w4

Real Rate

i=Cow number

Regression in a nutshellRegression in a nutshell This minimization is pretty This minimization is pretty

straightforward (though outside the straightforward (though outside the scope of this course).scope of this course).

It will give better coefficients the larger It will give better coefficients the larger the “training set” is.the “training set” is.

Of course, the rate is not deterministic.Of course, the rate is not deterministic. The assumption that the sum is linear The assumption that the sum is linear

is wrong in many cases. Hence the use is wrong in many cases. Hence the use of SVM, Neural Networks, etc.of SVM, Neural Networks, etc.

Notice this only deals with the case of Notice this only deals with the case of all attributes being numerical.all attributes being numerical.