Chapter 8 The k-Means Algorithm and Genetic Algorithm.

Chapter 8

The k-Means Algorithm and Genetic Algorithm

Chapter 82Data Warehouse and Data Mining

Contents

• k-Means algorithm

• Genetic algorithm

• Rough set approach

• Fuzzy set approaches


The K-Means AlgorithmThe K-Means AlgorithmThe K-Means AlgorithmThe K-Means Algorithm

The K-Means algorithm is a simple yet effective statistical clustering technique.

Here is the algorithm:

1. Choose a value for K, the total number of clusters to be determined.

2. Choose K instances (data points) within the dataset at random. These are the initial cluster centers.

3. Use simple Euclidean distance to assign the remaining instances to their closest cluster center.


4. Use the instances in each cluster to calculate a new mean for each cluster.

5. If the new mean values are identical to the mean values of the previous iteration the process terminates. Otherwise, use the new means as cluster centers and repeat steps 3-5.

The K-Means AlgorithmThe K-Means AlgorithmThe K-Means AlgorithmThe K-Means Algorithm


The K-Means AlgorithmThe K-Means AlgorithmAn Example Using K-Means






The K-Means AlgorithmThe K-Means AlgorithmGeneral Considerations






The k-Nearest Neighbor Algorithm

• All instances correspond to points in the n-D space.• The nearest neighbor are defined in terms of

Euclidean distance.• The target function could be discrete- or real- valued.

.

_+

_ xq

+

_ _+

_

_

+

.

..

. .


The k-Nearest Neighbor Algorithm

• For discrete-valued, the k-NN returns the most common value among the k training examples nearest to xq.

• Vonoroi diagram: the decision surface induced by 1-NN for a typical set of training examples.

.

_+

_ xq

+

_ _+

_

_

+

.

..

. .


Discussion on the k-NN Algorithm

• The k-NN algorithm for continuous-valued target functions– Calculate the mean values of the k nearest

neighbors

• Distance-weighted nearest neighbor algorithm– Weight the contribution of each of the k

neighbors according to their distance to the query point xq

• giving greater weight to closer neighbors– Similarly, for real-valued target functions

wd xq xi

12( , )


Genetic LearningGenetic LearningHere we present a basic genetic learning algorithm.

1. Initialize a population P of n elements, often referred to as chromosomes, as a potential solution.

2. Until a specified termination condition is satisfied:

a. Use a fitness function to evaluate each element of the current solution. If an element passes the fitness criteria, it remains in P.

b. The population now contains m elements (m<=n). Use genetic operators to create (n-m) new elements. Add the new elements to the population.


Genetic LearningGenetic LearningGenetic Algorithms and Supervised Learning












Genetic LearningGenetic LearningGenetic Algorithms and... Supervised Learning

Genetic LearningGenetic LearningGenetic Algorithms and... Supervised Learning


Genetic LearningGenetic LearningGenetic Algorithms and ..Unsupervised Clustering

Genetic LearningGenetic LearningGenetic Algorithms and ..Unsupervised Clustering


Genetic LearningGenetic LearningGenetic Algorithms and Unsupervised Clustering

Genetic LearningGenetic LearningGenetic Algorithms and Unsupervised Clustering


Genetic LearningGenetic LearningGeneral Considerations


Here is a list of considerations when using a problem-solving approach based on genetic learning:

Genetic algorithms are designed to find globally optimized solutions. However, there is no guarantee that any given solution is not the result of a local rather than a global optimization.

The fitness function determines the computational complexity of a genetic algorithm. A fitness function involving several calculations can be computationally expensive.




Genetic algorithms explain their results to the extent that the fitness function is understandable.

Transforming the data to form suitable for a genetic algorithm can be a challenge.


Genetic Algorithms• GA: based on an analogy to biological evolution• Each rule is represented by a string of bits• An initial population is created consisting of

randomly generated rules• Based on the notion of survival of the fittest, a

new population is formed to consists of the fittest rules and their offsprings

• The fitness of a rule is represented by its classification accuracy on a set of training examples

• Offsprings are generated by crossover and mutation


Genetic Algorithms

• Population-based technique for discovery of ....knowledge structures

• Based on idea that evolution represents search for optimum solution set

• Massively parallel


The Vocabulary of GAs

• Population– Set of individuals, each represented

by one or more strings of characters

• Chromosome– The string representing an individual


011010

Chromosome

Gene(Allele="0")

Locus=5

•Locus : The ordinal place... on a chromosome where a specific gene is found

•Allele :The value of a specific gene

•Gene The basic informational unit on a chromosome

The vocabulary of GAs, contd.


Genetic operators

• Reproduction– Increase representations of strong

individuals

• Crossover– Explore the search space

• Mutation– Recapture “lost” genes due to crossover


Genetic operators illustrated...

011010Parent 1:

Parent 2:

Offspring 1:

Offspring 2:000110011010000110

Simple reproduction

011010 Offspring 1:Offspring 2:

000110011110000010

Reproduction with crossover at locus 3

011010 Offspring 1:

Offspring 2:000110010010000110

Simple reproduction with mutation at locus 3 for offspring 1

Parent 1:

Parent 2:

Parent 1:

Parent 2:


GAs rely on the concept of “fitness”

• Ability of an individual to survive into the next generation

• “Survival of the fittest”• Usually calculated in terms of an

objective fitness function• Maximization• Minimization• Other functions


Genetic Programming

• Based on adaptation and evolution

• Structures undergoing adaptation are computer programs of varying size and shape

• Computer programs are genetically “bred” over time


The Learning Classifier System

• Rule-based knowledge discovery and concept learning tool

• Operates by means of evaluation, credit assignment, and discovery applied to a population of “chromosomes” (rules) each with a corresponding “phenotype” (outcome)


Components of a Learning Classifier System

• Performance– Provides interaction between environment and rule

base– Performs matching function

• Reinforcement– Rewards accurate classifiers– Punishes inaccurate classifiers

• Discovery– Uses the genetic algorithm to search for plausible rules


Rough Set Approach• Rough sets are used to approximately or “roughly”

define equivalent classes • A rough set for a given class C is approximated by

two sets: – a lower approximation (certain to be in C) – an upper approximation (cannot be described as not

belonging to C)


Fuzzy Set Approaches

• Fuzzy logic uses truth values between 0.0 and 1.0 to represent the degree of membership (such as using fuzzy membership graph)

• Attribute values are converted to fuzzy values– e.g., income is mapped into the discrete categories

{low, medium, high} with fuzzy values calculated


Fuzzy Set Approaches

• For a given new sample, more than one fuzzy value may apply

• Each applicable rule contributes a vote for membership in the categories

• Typically, the truth values for each predicted category are summed.


Chapter SummaryChapter SummaryChapter SummaryChapter Summary

• The K-Means algorithm is a statistical unsupervised clustering technique.

•All input attributes to the algorithm must be numeric and the user is required to make a decision about..... how many clusters are to be discovered.

•The algorithm begins by randomly choosing one data point to represent each cluster.

•Each data instance is then placed in the cluster to which it is most similar.

•New cluster centers are computed and the process continues until .....the cluster centers do not change.



•The K-Means algorithm is easy to implement and understand. However,

•the algorithm is not guaranteed to converge to a globally optimal solution,

•lacks the ability to explain what has been found,

•unable to tell which attributes are significant in determining the formed clusters.

•Despite these limitations, the K-Means algorithm is among the most widely used clustering techniques.



• Genetic algorithms apply the theory of evolution to inductive learning.

• Genetic learning can be supervised ...or ...unsupervised

• typically used for problems that cannot be solved with traditional techniques.

•A standard genetic approach to learning applies a fitness function to a set of data elements to determine...... which elements survive from one generation to the next.



•Those elements not surviving are used to create new instances to replace deleted elements.

•In addition to being used for supervised learning and unsupervised clustering, genetic techniques can be employed in conjunction with other learning techniques.


Key TermsKey TermsKey TermsKey Terms

Affinity analysis. The process of determining which things are typically grouped together.

Confidence. Given a rule of the form “If A then B,” confidence is defined as the conditional

probability that B is true when A is known to be true.

Crossover. A genetic learning operation that creates new population elements by combining parts of

two or more elements from the current population.


Key TermsKey TermsKey TermsKey Terms

Genetic algorithm. A data mining technique based on the theory of evolution.

Mutation. A genetic learning operation that creates a new population element by randomly modifying a portion of an existing element.

Selection. A genetic learning operation that adds copies of current population elements with high fitness scores to the next generation of the population.


Reference

Data Mining: Concepts and Techniques (Chapter 7 Slide for textbook), Jiawei Han and Micheline Kamber, Intelligent Database Systems Research Lab, School of Computing Science, Simon Fraser University, Canada

Chapter 8 The k-Means Algorithm and Genetic Algorithm.

Documents

data miningthe

data miningcontentsk

data miningdiscussion

data mining4

instances data points

meansdata warehouse

new means

approachesdata warehouse