Cluster Analysis Hal Whitehead BIOL4062/5062. What is cluster analysis? Non-hierarchical cluster analysis –K-means Hierarchical divisive cluster analysis.

Cluster Analysis

Hal Whitehead

BIOL4062/5062

• What is cluster analysis?

• Non-hierarchical cluster analysis– K-means

• Hierarchical divisive cluster analysis

• Hierarchical agglomerative cluster analysis– Linkage: single, complete, average, …– Cophenetic correlation coefficient

• Additive trees

• Problems with cluster analyses

Cluster Analysis

“Classification”

Maximize within cluster homogeneity

(similar individuals within cluster)

“The Search for Discontinuities”Discontinuities: places to put divisions between clusters

4 5 6 7 81

2

3

4

5

?

Discontinuities:

Discontinuities generally present:

taxonomy

social organization

community ecology??

Types of cluster analysis:

• Uses: data, dissimilarity, similarity matrix

• Non-hierarchical– K-means

• Hierarchical– Hierarchical divisive (repeated K-means, network

methods)

– Hierarchical agglomerative• single linkage, average linkage, ...

• Additive trees

Non-hierarchical Clustering Techniques:K-Means

• Uses data matrix with Euclidean distances

• Maximizes between-cluster variance for given number of clusters– i.e. Choose clusters to maximize F-ratio in 1-

way MANOVA

K-Means

Works iteratively:1. Choose number of clusters

2. Assigns points to clustersRandomly or some other clustering technique

3. Moves each point to other clusters in turn--increase in between cluster variance?

4. Repeat step 3. until no improvement possible

K-means with three clusters

0.0 0.2 0.4 0.6 0.8 1.0X

0.0

0.2

0.4

0.6

0.8

1.0

Y

0.0 0.2 0.4 0.6 0.8 1.0X

0.0

0.2

0.4

0.6

0.8

1.0

Y

0.0 0.2 0.4 0.6 0.8 1.0X

0.0

0.2

0.4

0.6

0.8

1.0

Y

0.0 0.2 0.4 0.6 0.8 1.0X

0.0

0.2

0.4

0.6

0.8

1.0

Y

0.0 0.2 0.4 0.6 0.8 1.0X

0.0

0.2

0.4

0.6

0.8

1.0

Y

0.0 0.2 0.4 0.6 0.8 1.0X

0.0

0.2

0.4

0.6

0.8

1.0

Y

0.0 0.2 0.4 0.6 0.8 1.0X

0.0

0.2

0.4

0.6

0.8

1.0

Y


Variable Between SS df Within SS df F-ratio

X 0.536 2 0.007 7 256.163

Y 0.541 2 0.050 7 37.566

** TOTAL ** 1.078 4 0.058 14

0.0 0.2 0.4 0.6 0.8 1.0X

0.0

0.2

0.4

0.6

0.8

1.0

Y


Cluster 1 of 3 contains 4 cases

Members Statistics

Case Distance | Variable Minimum Mean Maximum St.Dev.

Case 1 0.02 | X 0.41 0.45 0.49 0.04

Case 2 0.11 | Y 0.03 0.19 0.27 0.11

Case 3 0.06 |

Case 4 0.05 |


Members Statistics


Case 7 0.06 | X 0.11 0.15 0.19 0.03

Case 8 0.03 | Y 0.61 0.70 0.77 0.07

Case 9 0.02 |

Case 10 0.06 |


Members Statistics


Case 5 0.01 | X 0.77 0.77 0.78 0.01

Case 6 0.01 | Y 0.33 0.35 0.36 0.02

0.0 0.2 0.4 0.6 0.8 1.0X

0.0

0.2

0.4

0.6

0.8

1.0

Y

Disadvantages of K-means

• Reaches optimum, but not necessarily global

• Must choose number of clusters before analysis– How many clusters?

Example: Sperm whale codas

Patterned series of clicks:

| | | | |ic1 ic2 ic3 ic4

For 5-click codas: 681 x 4 data set

5-click codas:

| | | | |ic1 ic2 ic3 ic4

0.0 0.1 0.2 0.3 0.4 0.5 0.6ic1

0.1

0.2

0.3

0.4

0.5

0.6

0.7

ic4

-6 -2 2 61st Principal Component

-7

-3

1

5

2nd

Prin

cipa

l Com

pone

nt

93% of variance in 2 PC’s

5-click codas:K-means with 10 clusters

-6 -2 2 61st Principal Component

-7

-3

1

5

2nd

Prin

cipa

l Com

pone

nt

0.0 0.1 0.2 0.3 0.4 0.5 0.6ic1

0.1

0.2

0.3

0.4

0.5

0.6

0.7

ic4

Hierarchical Cluster Analysis

• Usually represented by:– Dendrogram or tree-diagram

Cluster Tree

0.0 0.1 0.2 0.3Distances

Case 1

Case 2

Case 3

Case 4

Case 5

Case 6

Case 7

Case 8

Case 9

Case 10

Hierarchical Cluster Analysis

• Hierarchical Divisive Cluster Analysis

• Hierarchical Agglomerative Cluster Analysis

Hierarchical Divisive Cluster Analysis

• Starts with all units in one cluster, successively splits them– Successive use of K-Means, or some other divisive

technique, with n=2– Either: Each time use the cluster with the greatest

sum of squared distances– Or: Split each cluster each time.

• Hierarchical divisive are good techniques, but rarely used, outside network analysis

0.70.60.50.40.30.20.1

1

14

15

4

5

11

9

19

2

12

20

Association index

Hierarchical Agglomerative Cluster Analysis

• Start with each individual units occupying its own cluster

• The clusters are then gradually merged until just one is left

• The most common cluster analyses

0.7 0.6 0.5 0.4 0.3 0.2 0.1

1

14

15

4

5

11

9

19

2

12

20

Association index


Works on dissimilarity matrix or negative similarity matrixmay be Euclidean, Penrose, … distances

At each step:1. There is a symmetric matrix of dissimilarities between clusters 2. The two clusters with least dissimilarity are merged3. The dissimilarity between the new (merged) cluster and all

others is calculated

Different techniques do step 3. in different ways:


A B C D E

A 0 . . . .

B 0.35 0 . . .

C 0.45 0.67 0 . .

D 0.11 0.45 0.57 0 .

E 0.22 0.56 0.78 0.19 0

AD B C E

AD 0 . . .

B ? 0 . .

C ? 0.67 0 .

E ? 0.56 0.78 0

First link A and D How to calculate new disimmilarities?

Hierarchical Agglomerative Cluster AnalysisSingle Linkage

A B C D E

A 0 . . . .

B 0.35 0 . . .

C 0.45 0.67 0 . .

D 0.11 0.45 0.57 0 .

E 0.22 0.56 0.78 0.19 0

AD B C E

AD 0 . . .

B 0.35 0 . .

C ? 0.67 0 .

E ? 0.56 0.78 0

d(AD,B)=Min{d(A,B), d(D,B)}

Hierarchical Agglomerative Cluster AnalysisComplete Linkage

A B C D E

A 0 . . . .

B 0.35 0 . . .

C 0.45 0.67 0 . .

D 0.11 0.45 0.57 0 .

E 0.22 0.56 0.78 0.19 0

AD B C E

AD 0 . . .

B 0.45 0 . .

C ? 0.67 0 .

E ? 0.56 0.78 0

d(AD,B)=Max{d(A,B), d(D,B)}

Hierarchical Agglomerative Cluster AnalysisAverage Linkage

A B C D E

A 0 . . . .

B 0.35 0 . . .

C 0.45 0.67 0 . .

D 0.11 0.45 0.57 0 .

E 0.22 0.56 0.78 0.19 0

AD B C E

AD 0 . . .

B 0.40 0 . .

C ? 0.67 0 .

E ? 0.56 0.78 0

d(AD,B)=Mean{d(A,B), d(D,B)}

Hierarchical Agglomerative Cluster AnalysisCentroid Clustering

(uses data matrix, or true distance matrix)V1 V2 V3

A 0.11 0.75 0.33

B 0.35 0.99 0.41

C 0.45 0.67 0.22

D 0.11 0.71 0.37

E 0.22 0.56 0.78

F 0.13 0.14 0.55

G 0.55 0.90 0.21

V1(AD)=Mean{V1(A),V1(D)}

V1 V2 V3

AD 0.11 0.73 0.35

B 0.35 0.99 0.41

C 0.45 0.67 0.22

E 0.22 0.56 0.78

F 0.13 0.14 0.55

G 0.55 0.90 0.21

Hierarchical Agglomerative Cluster AnalysisWard’s Method

• Minimizes within-cluster sum-of squares

• Similar to centroid clustering

0.7 0.6 0.5 0.4 0.3 0.2 0.1

1

14

15

4

5

11

9

19

2

12

20

Association index

Average Linkage

1 1.00

2 0.00 1.00

4 0.53 0.00 1.00

5 0.18 0.05 0.00 1.00

9 0.22 0.09 0.13 0.25 1.00

11 0.36 0.00 0.17 0.40 0.33 1.00

12 0.00 0.37 0.18 0.00 0.13 0.00 1.00

14 0.74 0.00 0.30 0.20 0.23 0.17 0.00 1.00

15 0.53 0.00 0.30 0.00 0.36 0.00 0.26 0.56 1.00

19 0.00 0.00 0.17 0.21 0.43 0.32 0.29 0.09 0.09 1.00

20 0.04 0.00 0.17 0.00 0.14 0.10 0.35 0.00 0.18 0.25 1.00

1 2 4 5 9 11 12 14 15 19 20

0.75 0.7 0.65 0.6 0.55 0.5 0.45 0.4 0.35 0.3

1

14

15

4

5

11

9

19

2

12

20

Association index

Single Linkage

0.7 0.6 0.5 0.4 0.3 0.2 0.1

1

14

15

4

5

11

9

19

2

12

20

Association index

Average Linkage

0.7 0.6 0.5 0.4 0.3 0.2 0.1 0

1

14

15

4

2

12

5

11

9

19

20

Association index

Complete Linkage

0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0

1

14

15

4

2

12

20

5

11

9

19

Association index

Ward's

Hierarchical Agglomerative Clustering Techniques

• Single Linkage– Produces “straggly” clusters– Not recommended if much experimental error– Used in taxonomy– Invariant to transformations

• Complete Linkage– Produces “tight” clusters– Not recommended if much experimental error– Invariant to transformations

• Average Linkage, Centroid, Ward’s– Most likely to mimic input clusters– Not invariant to transformations in dissimilarity measure

Cophenetic Correlation Coefficient CCC

• Correlation between original disimilarity matrix and dissimilarity inferred from cluster analysis

• CCC >~ 0.8 indicate a good match

• CCC <~ 0.8, dendrogram not a good representation– probably should not be displayed

• Use CCC to choose best linkage method (highest coefficient)

1 1.00

2 0.00 1.00

4 0.53 0.00 1.00

5 0.18 0.05 0.00 1.00

9 0.22 0.09 0.13 0.25 1.00

11 0.36 0.00 0.17 0.40 0.33 1.00

12 0.00 0.37 0.18 0.00 0.13 0.00 1.00

14 0.74 0.00 0.30 0.20 0.23 0.17 0.00 1.00

15 0.53 0.00 0.30 0.00 0.36 0.00 0.26 0.56 1.00

19 0.00 0.00 0.17 0.21 0.43 0.32 0.29 0.09 0.09 1.00

20 0.04 0.00 0.17 0.00 0.14 0.10 0.35 0.00 0.18 0.25 1.00

1 2 4 5 9 11 12 14 15 19 20

0.7 0.6 0.5 0.4 0.3 0.2 0.1

1

14

15

4

5

11

9

19

2

12

20

Association index

Average Linkage

0.75 0.7 0.65 0.6 0.55 0.5 0.45 0.4 0.35 0.3

1

14

15

4

5

11

9

19

2

12

20

Association index

Single Linkage

0.7 0.6 0.5 0.4 0.3 0.2 0.1

1

14

15

4

5

11

9

19

2

12

20

Association index

Average Linkage

0.7 0.6 0.5 0.4 0.3 0.2 0.1 0

1

14

15

4

2

12

5

11

9

19

20

Association index

Complete Linkage

0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0

1

14

15

4

2

12

20

5

11

9

19

Association index

Ward's

CCC=0.83

CCC=0.75

CCC=0.77

CCC=0.80

Additive trees• Dendrogram in which path

lengths represent dissimilarities

• Computation quite complex (cross between agglomerative techniques and multidimensional scaling)

• Good when data are measured as dissimilarities

• Often used in taxonomy and genetics

Additive Tree

A

B

C

D

E

A B C D EA . . . . .B 14 . . . .C 6 12 . . .D 81 7 13 . .E 17 1 6 16 .

Problems with Cluster Analysis• Are there really biologically-meaningful clusters in

the data?• Does the dendrogram represent biological reality

(web-of-life versus tree-of-life)?• How many clusters to use?

– stopping rules are arbitrary

• Which method to use?– best technique is data-dependent

• Dendrograms become messy with many units

10.90.80.70.60.50.40.30.20.1 0

9179221367740757149316422515316539501501151215131515120613824103313111461019147054123956480133211161181426413170303115722246114351202252102222715980413642811141130711511181418371241431040114211431144832952961114910091025102010421045171081267409466106410752979861835787916818115731332515940644076451024727315231614157450250421394313489290509518165583051015088531563903991628829191271311132561931712121219122012685076747232353806512429591248134413451966213121441930165167186269649224623092310

Association index

Social Structure of 160 northern bottlenose whales

Clustering TechniquesType Technique Use

Non-hierarchical K-Means Dividing data sets

Hierarchical divisive Repeated K-means Good technique on

small data sets

Network methods...

Hierarchical agglomerative

Single linkage Taxonomy

Complete linkage Tighter Clusters

Average linkage,

Centroid, Ward’s Usually Preferred

Hierarchical Additive trees Excellent for displaying dissimilarity; taxonomy, genetics

Cluster Analysis Hal Whitehead BIOL4062/5062. What is cluster analysis? Non-hierarchical cluster analysis –K-means Hierarchical divisive cluster analysis.

Documents

clusters cluster

cluster variance

types of cluster analysis

cluster analyses

network analysis slide

data set slide

number of clusters

treediagram slide