Spectral Clustering Eyal David Image Processing seminar May 2008.

Spectral Clustering

Eyal David

Image Processing seminar

May 2008

2

Lecture Outline

Motivation Graph overview and construction Demo Spectral Clustering Demo Cool implementations

3

A Tutorial on Spectral Clustering\Arik Azran

4

Spectral Clustering Example – 2 Spirals

-2

-1.5

-1

-0.5

0

0.5

1

1.5

2

-2 -1.5 -1 -0.5 0 0.5 1 1.5 2

Dataset exhibits complex Dataset exhibits complex cluster shapescluster shapes

K-means performs very K-means performs very poorly in this space due poorly in this space due bias toward dense bias toward dense spherical clusters.spherical clusters.

-0.8

-0.6

-0.4

-0.2

0

0.2

0.4

0.6

0.8

-0.709 -0.7085 -0.708 -0.7075 -0.707 -0.7065 -0.706In the embedded space In the embedded space given by two leading given by two leading eigenvectors, clusters eigenvectors, clusters are trivial to separate.are trivial to separate.

Spectral Clustering - Derek Greene

5

Lecture Outline

Motivation Graph overview and construction Graph demo Spectral Clustering Spectral Clustering demo Cool implementation

6

Matthias Hein and Ulrike von Luxburg August 2007

7


8


9


10


11


12


13


14


15


16


17


Demo

(Live example)

19

Lecture Outline

Motivation Graph overview and construction Demo Spectral Clustering Demo Cool implementations

20


21


22


23


24


25


26


27


28


29

Eigenvectors & Eigenvalues

30


31


Demo

(Live example)

33

Spectral Clustering Algorithm Ng, Jordan, and Weiss

Motivation Given a set of points

We would like to cluster them into k subsets

1,...,l

nS s s R

Slides from Spectral Clustering by Rebecca Nugent, Larissa Stanberry based on Ng et al On Spectral clustering: analysis and algorithm

34

Algorithm

Form the affinity matrix Define if

Scaling parameter chosen by user

Define D a diagonal matrix whose

(i,i) element is the sum of A’s row i

nxnW Ri j

0iiW

2 2|| || / 2i js s

ijW e


35

Algorithm

Form the matrix

Find , the k largest eigenvectors of L These form the the columns of the new

matrix X Note: have reduced dimension from nxn to nxk

1/ 2 1/ 2L D WD

1 2, ,..., kx x x


36

Algorithm

Form the matrix Y Renormalize each of X’s rows to have unit length Y

Treat each row of Y as a point in Cluster into k clusters via K-means

2 2/( )ij ij ijj

Y X X kR

nxkR


37

Algorithm

Final Cluster Assignment Assign point to cluster j iff row i of Y was

assigned to cluster jis


38

Why?

If we eventually use K-means, why not just apply K-means to the original data?

This method allows us to cluster non-convex regions


39

Some Examples

40

Ng et al On Spectral clustering: analysis and algorithm

41


42


43


44


45


46


47


48

User’s Prerogative

Affinity matrix construction Choice of scaling factor

Realistically, search over and pick value that gives the tightest clusters

Choice of k, the number of clusters Choice of clustering method

2


49

0

5

10

15

20

25

30

35

40

45

50

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

K

Eig

enva

lue

Largest Largest eigenvalueseigenvalues

of Cisi/Medline of Cisi/Medline datadata

λ1

λ2

How to select k? Eigengap: the difference between two consecutive eigenvalues. Most stable clustering is generally given by the value k that

maximises the expression

1k k k

Choose Choose k=2k=2

12max k

Spectral Clustering - Derek Greene

50


Recap – The bottom line

51

Summary

Spectral clustering can help us in hard clustering problems

The technique is simple to understand The solution comes from solving a simple

algebra problem which is not hard to implement

Great care should be taken in choosing the “starting conditions”

The End

Spectral Clustering Eyal David Image Processing seminar May 2008.

Documents

algorithm slide

spectral clustering

matthias hein

stable clustering

rebecca nugent

larissa stanberry

cluster j slides

matrix y