Association Rule Mining - Fordham Universitygweiss/classes/cisc6930/slides/10 Association Rule...Association Rule: Basic Concepts Given: (1) database of transactions, (2) each transaction

1

Association Rule Mining

What Is Association Rule Mining?

Association rule mining is finding frequent patterns or associations among sets of items or objects, usually amongst transactional data

Applications include Market Basket analysis, cross-marketing, catalog design, etc.

2

Association Mining

Examples. Rule form: “Body ead [support, confidence]”.

buys(x, “diapers”) buys(x, “beers”) [0.5%, 60%]

buys(x, "bread") buys(x, "milk") [0.6%, 65%]

major(x, "CS") /\ takes(x, "DB") grade(x, "A") [1%, 75%]

age(X,30-45) /\ income(X, 50K-75K) buys(X, SUVcar)

age=“30-45”, income=“50K-75K” car=“SUV”

Market-basket Analysis & Finding Associations

Do items occur together?

Proposed by Agrawal et al in 1993.

It is an important data mining model studied extensively by the database and data mining community.

Assumes all data are categorical.

Initially used for Market Basket Analysis to find how items purchased by customers are related.

Bread Milk [sup = 5%, conf = 100%]

Association Rule: Basic Concepts

Given: (1) database of transactions, (2) each transaction is a list of items (purchased by a customer in a visit)

Find: all rules that correlate the presence of one set of items with that of another set of items

E.g., 98% of people who purchase tires and auto accessories also get automotive services done

Applications

* Maintenance Agreement (What the store should do to boost Maintenance Agreement sales)

Home Electronics * (What other products should the store stocks up?)

Detecting “ping-pong”ing of patients, faulty “collisions”

5

Association Rule Mining

Given a set of transactions, find rules that will predict the occurrence of an item based on the occurrences of other items in the transaction

6

Market-Basket transactions

TID Items

1 Bread, Milk

2 Bread, Diaper, Beer, Eggs

3 Milk, Diaper, Beer, Coke

4 Bread, Milk, Diaper, Beer

5 Bread, Milk, Diaper, Coke

Example of Association Rules

{Diaper} {Beer},

{Milk, Bread} {Eggs,Coke},

{Beer, Bread} {Milk},

Implication means co-occurrence,

not causality!

An itemset is simply a set of items

Examples from a Supermarket

Can you think of association rules from a supermarket?

Let’s say you identify association rules from a supermarket, how might you exploit them?

That is, if you are the store manager, how might you make money?

Assume you have a rule of the form X Y

7

Supermarket examples

If you have a rule X Y, you could:

Run a sale on X if you want to increase sales of Y

Locate the two items near each other

Locate the two items far from each other to make the shopper walk through the store

Print out a coupon on checkout for Y if shopper bought X but not Y

8

Association “rules”–standard format

Rule format: (A set can consist of just a single item)

If {set of items} Then {set of items}

Condition implies Results

If {Diapers, Baby Food}

Condition

{Beer, Chips}

Results

Then

Customer

buys diaper Customer buys both

Customer

buys beer

Right side very often is a single item Rules do not imply causality

What is an Interesting Association?

Requires domain-knowledge validation

Actionable, non-trivial, understandable

Algorithms provide first-pass based on statistics on how “unexpected” an association is

Some standard statistics used:

C R

support ≈ p(R&C) percent of “baskets” where rule holds

confidence ≈ p(R|C) percent of times R holds when C holds

Support and Confidence

Find all the rules X Y with minimum confidence and support Support = probability that a transaction

contains {X,Y}

i.e., ratio of transactions in which X, Y occur together to all transactions in DB.

Confidence = conditional probability that a transaction having X contains Y

i.e., ratio of transactions in which X, Y occur together to those in which X occurs.

Thel confidence of a rule LHS => RHS can be computed as the support of the whole itemset divided by the support of LHS:

Confidence (LHS => RHS) = Support(LHS RHS) / Support(LHS)

Customer

buys diaper Customer buys both

Customer

buys beer

Definition: Frequent Itemset

Itemset

A collection of one or more items

Example: {Milk, Bread, Diaper}

k-itemset: itemset with k items

Support count ()

Frequency count of occurrence of itemset

E.g. ({Milk, Bread,Diaper}) = 2

Support

Fraction of transactions containing the itemset

E.g. s({Milk, Bread, Diaper}) = 2/5

Frequent Itemset

An itemset whose support is greater than or equal to a minsup threshold

TID Items

1 Bread, Milk





12

Support and Confidence Calculations

TID Items

1 Bread, Milk





Beer}Diaper,Milk{

4.05

2

|T|

)BeerDiaper,,Milk(

s

67.03

2

)Diaper,Milk(

)BeerDiaper,Milk,(

c

Given Association Rule

– {Milk, Diaper} {Beer}

Rule Evaluation Metrics

– Support (s)

Fraction of transactions that contain both X and Y

– Confidence (c)

Measures how often items in Y appear in transactions that contain X

Now Compute these two metrics

Support and Confidence – 2nd Example

Transaction ID Items Bought

1001 A, B, C

1002 A, C

1003 A, D

1004 B, E, F

1005 A, D, F

Itemset {A, C} has a support of 2/5 = 40% Rule {A} ==> {C} has confidence of 50% Rule {C} ==> {A} has confidence of 100% Support for {A, C, E} ? Support for {A, D, F} ? Confidence for {A, D} ==> {F} ? Confidence for {A} ==> {D, F} ?

Goal: Find all rules that satisfy the user-specified minimum support (minsup) and minimum confidence (minconf).

Example

Transaction data Assume: minsup = 30% minconf = 80%

An example frequent itemset: {Chicken, Clothes, Milk} [sup = 3/7]

Rules from the itemset are partitions of the items Association rules from above itemset: Clothes Milk, Chicken [sup = 3/7, conf = 3/3]

… …

Clothes, Chicken Milk, [sup = 3/7, conf = 3/3]

15

t1: Beef, Chicken, Milk t2: Beef, Cheese t3: Cheese, Boots t4: Beef, Chicken, Cheese t5: Beef, Chicken, Clothes, Cheese, Milk t6: Chicken, Clothes, Milk t7: Chicken, Milk, Clothes

Mining Association Rules

TID Items

1 Bread, Milk





Example of Rules:

{Milk,Diaper} {Beer} (s=0.4, c=0.67) {Milk,Beer} {Diaper} (s=0.4, c=1.0) {Diaper,Beer} {Milk} (s=0.4, c=0.67) {Beer} {Milk,Diaper} (s=0.4, c=0.67) {Diaper} {Milk,Beer} (s=0.4, c=0.5) {Milk} {Diaper,Beer} (s=0.4, c=0.5)

Observations:

• All the above rules are binary partitions of the same itemset: {Milk, Diaper, Beer}

• Rules originating from the same itemset have identical support (by definition) but may have different confidence values

Drawback of Confidence

Coffee

Coffee

Tea 15 5 20

Tea 75 5 80

90 10 100

Association Rule: Tea Coffee

Confidence= P(Coffee|Tea) = 0.75

but P(Coffee) = 0.9

Although confidence is high, rule is misleading

P(Coffee|Tea) = 0.9375

Mining Association Rules

Two-step approach:

1. Frequent Itemset Generation

– Generate all itemsets whose support minsup

2. Rule Generation – Generate high confidence rules from each frequent itemset,

where each rule is a binary partitioning of a frequent itemset

Frequent itemset generation is still computationally expensive

Transaction data representation

A simplistic view of “shopping baskets”

Some important information not considered:

the quantity of each item purchased

the price paid

19

Many mining algorithms

There are a large number of them

They use different strategies and data structures.

Their resulting sets of rules are all the same.

Given a transaction data set T, and a minimum support and a minimum confident, the set of association rules existing in T is uniquely determined.

Any algorithm should find the same set of rules although their computational efficiencies and memory requirements may be different.

We study only one: the Apriori Algorithm

20

The Apriori algorithm

The best known algorithm

Two steps:

Find all itemsets that have minimum support (frequent itemsets, also called large itemsets).

Use frequent itemsets to generate rules.

E.g., a frequent itemset {Chicken, Clothes, Milk} [sup = 3/7]

and one rule from the frequent itemset

Clothes Milk, Chicken [sup = 3/7, conf = 3/3]

21

Step 1: Mining all Frequent Itemsets

A frequent itemset is an itemset whose support is ≥ minsup.

Key idea: The Apriori property (downward closure property): any subsets of a frequent itemset are also frequent itemsets

22

AB AC AD BC BD CD

A B C D

ABC ABD ACD BCD

Steps in Association Rule Discovery

Find frequent itemsets

Itemsets with at least minimum support

Support is “downward closed” so a subset of a frequent itemset must be frequent if {AB} is a frequent itemset, both {A} and {B} are frequent itemsets

If an itemset doesnot satisfy minimum support, none of its supersets will either (this is key point that allows pruning of search space)

Iteratively find frequent itemsets with cardinality from 1 to k (k-itemsets)

Use the frequent itemsets to generate assoc. rules

Generate all binary partitions, but may have to fit template

E.g., only one item on right side or only two items on left side

Frequent Itemset Generation

24

null

AB AC AD AE BC BD BE CD CE DE

A B C D E

ABC ABD ABE ACD ACE ADE BCD BCE BDE CDE

ABCD ABCE ABDE ACDE BCDE

ABCDE

Given d items, there

are 2d possible

candidate itemsets

Mining Association Rules—An Example

For rule A C:

support = support({A ,C}) = 50%

confidence = support({A ,C})/support({A}) = 66.6%

The Apriori principle:

Any subset of a frequent itemset must be frequent

25

Transaction ID Items Bought

2000 A,B,C

1000 A,C

4000 A,D

5000 B,E,F

Frequent Itemset Support

{A} 75%

{B} 50%

{C} 50%

{A,C} 50%

Min. support 50%

Min. confidence 50%

User specifies these

26

Illustrating the Apriori Principle

Found to be

Infrequent

null


A B C D E



ABCDE

null


A B C D E



ABCDE

Pruned

supersets

The Apriori Algorithm

Terminology:

Ck is the set of candidate k-itemsets

Lk is the set of k-itemsets

Join Step: Ck is generated by joining two elements from Lk-1

There must be a lot of overlap for the join to only increase length by 1

Prune Step: Any (k-1)-itemset that is not frequent cannot be a subset of a frequent k-itemset

This is a bit confusing since we want to use it the other way. We prune a candidate k-itemset if any of its k-1 itemsets are not in our list of frequent k-1 itemsets

To utilize this you simply start with k=1, which is single-item itemsets and they you work your way up from there!

27

The Algorithm

Iterative algo. (also called level-wise search): Find all 1-item frequent itemsets; then all 2-item frequent itemsets, and so on.

In each iteration k, only consider itemsets that contain some k-1 frequent itemset.

Find frequent itemsets of size 1: F1

From k = 2 Ck = candidates of size k: those itemsets of size k that

could be frequent, given Fk-1

Fk = those itemsets that are actually frequent, Fk Ck (need to scan the database once).

28

Apriori Candidate Generation

The candidate-gen function takes Lk-1 and returns a superset (called the candidates) of the set of all frequent k-itemsets.

There are two steps: join step: Generate all possible candidate itemsets

Ck of length k

prune step: Remove those candidates in Ck that cannot be frequent.

29

How to Generate Candidates?

Suppose the items in Lk-1 are listed in an order

Step 1: self-joining Lk-1

The description below is a bit confusing– all we do is splice two sets

together so that only one new item is added (see next slide)

insert into Ck

select p.item1, p.item2, …, p.itemk-1, q.itemk-1

from Lk-1 p, Lk-1 q

where p.item1=q.item1, …, p.itemk-2=q.itemk-2, p.itemk-1 < q.itemk-1

Step 2: pruning

forall itemsets c in Ck do

forall (k-1)-subsets s of c do

if (s is not in Lk-1) then delete c from Ck

30

Self-Joining Step

All items in the itemset to be self joined are in a consistent order– any order

Such as lexicographic (alphabetical) order

Two items in the itemset can be joined only if they differ in the last position

Then when you join them the size of the itemset goes up by one

See example on next slide

31

Example of Generating Candidates (1)

L3={abc, abd, acd, ace, bcd}

Self-joining: L3*L3

abc and abd yields abcd

acd and ace yields acde

We do not join abd and acd

Even though it would give abcd which is a candidate

If the product were a candidate it would have already been generated given the ordering

This may not be obvious at first glance 32


Note that for abcd to be frequent by the Apriori property abc, bcd, and abd must be frequent

abc and abd are alphabetically before bcd

So if we see abc and bcd we do not need to generate abcd because if abd were there it would have already been generated

If it is not there then it would be pruned later

33


Given abde we go to the pruning phase

acde is removed because ade is not in L3

Merge step does not ensure all subsets are frequent

C4={abcd}

34

The Apriori Algorithm — Example (minsup = 30%)

35

TID Items

100 1 3 4

200 2 3 5

300 1 2 3 5

400 2 5

Database D itemset sup.

{1} 2

{2} 3

{3} 3

{4} 1

{5} 3

itemset sup.

{1} 2

{2} 3

{3} 3

{5} 3

Scan D

C1

L1

itemset

{1 2}

{1 3}

{1 5}

{2 3}

{2 5}

{3 5}

itemset sup

{1 2} 1

{1 3} 2

{1 5} 1

{2 3} 2

{2 5} 3

{3 5} 2

itemset sup

{1 3} 2

{2 3} 2

{2 5} 3

{3 5} 2

L2

C2 C2

Scan D

C3 L3 itemset

{2 3 5}Scan D itemset sup

{2 3 5} 2

Warning: Do Not Forget Pruning

Rules get pruned in two ways Apriori property violated If Apriori not violated, still must scan database and if

minsup not exceeded then prune Apriori property is necessary but not sufficient to keep a

rule

If you forget to prune via Apriori property, you will get same results since will catch on the scan But I will take off points on an exam. Make it clear when

prune using Apriori property (do not fill in count when crossing off)

Apriori property cannot be violated until k=3. Begins go get trickier at k=4 since more subsets to check

36

Step 2: Rules from Frequent Itemsets

Frequent itemsets association rules

One more step is needed

For each frequent itemset X,

For each proper nonempty subset A of X, Let B = X - A

A B is an association rule if

Confidence(A B) ≥ minconf,

support(A B) = support(AB) = support(X)

confidence(A B) = support(A B) / support(A)

37

Generating Rules: an Example

Suppose {2,3,4} is frequent, with sup=50% Proper nonempty subsets: {2,3}, {2,4}, {3,4}, {2}, {3}, {4}, with sup=50%,

50%, 75%, 75%, 75%, 75% respectively These generate these association rules:

2,3 4, confidence=100% 2,4 3, confidence=100% 3,4 2, confidence=67% 2 3,4, confidence=67% 3 2,4, confidence=67% 4 2,3, confidence=67% All rules have support = 50%

Then apply confidence threshold to identify strong rules Rules that meet the support and confidence requirements If confidence threshold is 80% we are left with 2 strong rules

38

Generating Rules: Summary

To recap, in order to obtain A B, we need to have support(A B) and support(A)

All the required information for confidence computation has already been recorded in itemset generation. No need to see the data T any more.

This step is not as time-consuming as frequent itemset generation Hint: I almost always ask this on the exam

39

On Apriori Algorithm

Seems to be very expensive

Level-wise search

K = the size of the largest itemset

It makes at most K passes over data

In practice, K is bounded (10).

The algorithm is very fast. Under some conditions, all rules can be found in linear time.

Scale up to large data sets

40

Granularity of items

One exception to the “ease” of applying association rules is selecting the granularity of the items.

Should you choose: diet coke?

coke product?

soft drink?

beverage?

Should you include more than one level of granularity? Some association finding techniques allow you to represent

hierarchies explicitly

Multiple-Level Association Rules

Items often form a hierarchy

Items at the lower level are expected to have lower support

Rules regarding itemsets at appropriate levels could be quite useful

Transaction database can be encoded based on dimensions and levels

Food

Milk Bread

Skim 2% Wheat White

Mining Multi-Level Associations

A top-down, progressive deepening

approach

First find high-level strong rules: milk bread [20%, 60%]

Then find their lower-level “weaker” rules: 2% milk wheat bread [6%, 50%]

Usually requires different thresholds at

different levels to find meaningful rules

lower support at lower levels

Interestingness Measurements

Objective measures

Two popular measurements:

Support

Confidence

Subjective measures (Silberschatz & Tuzhilin, KDD95)

A rule (pattern) is interesting if

it is unexpected (surprising to the user); and/or

actionable (the user can do something with it)

44

Criticism to Support and Confidence

Example 1:

Among 5000 students 3000 play basketball 3750 eat cereal 2000 both play basket ball and eat cereal

play basketball eat cereal [40%, 66.7%] is misleading because the overall percentage of students eating cereal is 75% which is higher than 66.7%.

play basketball not eat cereal [20%, 33.3%] is far more interesting, although with lower support and confidence

45

basketball not basketball sum(row)

cereal 2000 1750 3750

not cereal 1000 250 1250

sum(col.) 3000 2000 5000

Lift of A => B = P(B|A)/P(B)

and a rule is interesting if lift is

not near 1.0

What is the lift of this rule?

(1/3)/(1250/5000) = 1.33

Customer Number vs. Transaction ID

In the homework you may have a problem where there is a customer id for each transaction

You can be asked to do association analysis based on the customer id If this is so, you need to aggregate the transactions to the

customer level

If a customer has 3 transactions then you just create an itemset containing all of the items in the union of the 3 transactions

Note we will ignore the frequency of purchase

46

Virtual items

If you’re interested in including other possible variables, can create “virtual items”

gift-wrap, used-coupon, new-store, winter-holidays, bought-nothing,…

Associations: Pros and Cons

Pros

can quickly mine patterns describing business/customers/etc. without major effort in problem formulation

virtual items allow much flexibility

unparalleled tool for hypothesis generation

Cons

unfocused not clear exactly how to apply mined “knowledge”

only hypothesis generation

can produce many, many rules! may only be a few nuggets among them (or none)

Association Rules

Association rule types:

Actionable Rules – contain high-quality, actionable

information

Trivial Rules – information already well-known by

those familiar with the business

Inexplicable Rules – no explanation and do not

suggest action

Trivial and inexplicable rules occur most often

Association Rule Mining - Fordham Universitygweiss/classes/cisc6930/slides/10 Association Rule...Association Rule: Basic Concepts Given: (1) database of transactions, (2) each transaction

Documents