So Far…… Clustering basics, necessity for clustering, Usage in various fields : engineering and industrial fields Properties : hierarchical, flat,

So Far……

Clustering basics, necessity for clustering, Usage in various fields :

engineering and industrial fields

Properties : hierarchical, flat, iterative, hard, soft, disjunctive

Types: Supervised and unsupervised

K-means algorithm – for unsupervised clustering

Vector quantization – for supervised clustering

CONTINUE with fuzzy and neural net algorithms – supervised

clustering

Topics for today

Basics of fuzzy systems, and fields of application

Residual analysis methods

Fuzzy C-means algorithm – Matlab illustration

Artificial neural network – basics

Structure and working of artificial neuron

Hyper plane analysis of the output

FUZZY LOGIC SYSTEMS

IN

SUPERVISED CLUSTERING

Fuzzy Systems

Last decade increase in Fuzzy system implementation

More popular in the field of control systems and pattern recognition

Consumer products (washing machine,camcorders, palm pilot…) and

industrial systems (to provide decision support and expert system with

powerful reasoning capabilities bound by a minimum of rules)

Classical set (non-fuzzy) – either belongs to or does not belong to the set

(crisp membership)

Fuzzy set – allows degree of membership for each element to range over

the unit interval [0, 1]

Fuzzy membership represent similarities of objects to imprecisely

defined properties whereas, probabilities convey information about

relative frequencies

Probability : some kind of likelihood or degree of certainty or if it the

outcome of clearly defined but randomly occurring events

Major feature of fuzzy : expresses the amount of ambiguity in human

thinking

When to use fuzzy logic ?

Continuous phenomenon, not easily breakable into discrete

segments

Cannot model a process

Residual Analysis

Fuzzy reasoning :IF-THEN reasoning based on the sign of the residual Ex: IF residual-1 is positive and residual-2 is negative THEN fault1 is

Present IF residual-1 is zero and residual-2 is zero THEN system is fault free

:

Fuzzy Clustering : each data point belongs to all classes with a certain

degree of membership. The degree is dependant upon the distance to all

cluster centers. For fault diagnosis, each class could correspond to a

particular fault

(1) Fuzzy centers computed by minimizing the following partition formula:

Fuzzy C-means algorithm

ikm

N

kik

C

if dumCJ ,

1,

1

)(),(

11

,

C

ikiusubject to

C - Number of clusters

N - Number of data points

kiu ,

ikd ,

),1( m

- fuzzy membership of the k-th point to the i-th cluster

- Euclidean distance between the data point and the cluster center

fuzzy weighting factor which defines the degree of fuzziness of the results (normally chosen m =2, to get analytical solution)

-

(2) The cluster centers v, (centroids or prototypes) are defined as the

fuzzy weighted center of gravity of the data ,

N

k

mik

N

kk

mik

i

u

xuv

1,

1,

)(

)(Ci .....2,1

(3) The minimization of the partition functional (1) will give the

following expression for the membership

C

j

m

jk

ik

ki

d

d

u

1

1

2

,

,

,

1

(4) Euclidean distance defined as :

22, )( ikik vxd

Two steps of Fuzzy Clustering

Off-line phase: Learning phase, determines cluster centers of the classes

(this is done by iteratively calculating membership function). A learning

data set is necessary, which must contain residuals for all known faults.

On-line phase: Calculates the membership degree of the current residuals

to each of the known classes.

1. Choose the number of classes C , ; Chose m=2,

Initialize ( start with some arbitrary values for cluster centers

and corresponding partition matrix values )

2. Calculate the cluster centers using Eq. in (2)

3. Calculate new partition matrix using Eq. in (3)

4. Compare and . If the variation of the membership degree,

calculated with an appropriate norm, is smaller than a given

threshold, stop the algorithm, otherwise go back to step 2.

Off-line phase Cluster centers determination

nC 2

iv

)0(U

)1(U)( jU )1( jU

iku ,

)0(U

Matlab – fuzzy logic tool box – illustration of fuzzy C-means algorithm

For the incoming data, calculate the degree of membership to all the

centers using the following formula

On-line phase

C

j

m

jk

ik

ki

d

d

u

1

1

2

,

,

,

1

22, )( ikik vxd

Ci .....2,1

kiu ,

ikd ,

- fuzzy membership of the k-th point to the i-th cluster

- Euclidean distance between the data point and the cluster center

Artificial Neural Net systems

Based on low level microscopic biological models

Originated from modeling of human brain and evolution

Collective behavior of NN, like a human brain, demonstrates the

ability to learn, recall and generalize from training patterns of data

Consists of large number of highly interconnected processing

elements (nodes)

Application areas: speech recognition, speech to text conversion,

image processing, investing, trading……

Model specified by three basic elements:

Models of the processing element

Models of interconnections and structures (network topology)

Learning rules (the way information is stored in the network)

Each node collects values from all its input connections, performs a

predefined mathematical operation and produces a single output

Net input to the node = integration function (typically dot product)

combining all inputs

Each input connection is weighed. These weights (can be positive or

negative values) are determined during learning process. Because

of this adjustments, NN is able to learn.

Activation output of node = activation function ( net input ), usually

a non-linear function

Ex: Activation function Unipolar ( output takes the value 0 or 1 )

Output y = 0 , if sum < threshold (b)

= 1 , if sum > threshold (b)

sum is dot product of input vector x= ( ) and weight vector w = ( )

N

nnn xws

1

Nxx ...............1

Nww .................1

Some activation functions (unipolar and bipolar)

Input vector x = ( ) Linearly combined with weights

Then s is activated by a threshold function T(-) to produce the output y = T(s) = 1 when s > 0, else y = T(s) = -1.

Then all the input vectors x such that

forms a Hyperplane H in the input vector space. H partitions the feature vector space into right and left half spaces, H+ (when sum is > b) and H- (when sum < b)

The Perceptron as Hyperplane separator

bxwxwS NN .........11 , where b is the threshold

0.........11 bxwxwSx NN

Nxx ...............1

Ex : consider a single perceptron with two inputs

Let w1 = 2 andw2 = -1, b=0, then 2x1 - x2 = 0 determines H

the points (0,0) and (1,2) belong to H

The feature vector x = (x1,x2) = (2,3) is summed into

S = 2(2) - 1(3) = 1 > 0, so that the activated output is y = T(1) = 1

(corresponds to H+ in the plane, i.e right half)

(x1,x2) = (0,2) activates the output y = T(2(0) - 1(2)) = T(-1) = -1,

which indicates that (0,2) is in the left halfspace H-. The figure shows

these points.

Mapping in Hyperplane. (example of linear mapping between input and output

Non-linear mapping between input and output

Example : XOR logic function or 2- bit parity problem.

N = 2 inputs, M = 1 output, and Q = 4 sample vector (input/output)

pairs for training, and K= 2 clusters (even and odd).

Hyperplane diagram for 2-bit parity problem

XOR function implementation in three layered network

Take:

result is two parallel hyperplanes that yield three convex regions. The

hyperplanes are determined by

The threshold at the first neuron in the hidden layer yields

The threshold at the second hidden neuron yields

hyperplanes yield three convex regions. The four sets of above inputs

yield the three unique vectors

Corresponding to three regions in the hyperplane

Hyperplanes showing three regions for 2-bit parity problem

Regions 1 and 3make up the odd parity (Class 2),while Region 3 is even parity (Class 1).

For the second layer (output layer), the equations are as follows:

Choosing

Threshold = 1/2