Self-organizing incremental neural network and its application

ContentsWhat is SOINN

Why SOINNDetail algorithm of SOINN

SOINN for machine learningSOINN for associative memory

References

Self-organizing incremental neural network and its

application

F. Shen1 O. Hasegawa2

1National Key Laboratory for Novel Software Technology, Nanjing University

2Imaging Science and Engineering Lab, Tokyo Institute of Technology

June 14, 2009

F. Shen, O. Hasegawa Self-organizing incremental neural network and its application

References

Contents of this tutorial

1 What is SOINN

2 Why SOINN

3 Detail algorithm of SOINN

4 SOINN for machine learning

5 SOINN for associative memory

6 References

References

What is SOINN

1 What is SOINN

2 Why SOINN

6 References

References

What is SOINN

SOINN: Self-organizing incremental neural network

Represent the topological structure of the input data

Realize online incremental learning

References

What is SOINN

References

What is SOINN

References

What is SOINN

References

BackgroundCharacteristics of SOINN

1 What is SOINN

2 Why SOINN

6 References

References

Background: Networks for topology representation

SOM(Self-Organizing Map): predefine structure and size ofthe network

NG(Neural Gas): predefine the network size

GNG(Growing Neural Gas): predefine the network size;constant learning rate leads to non-stationary result.

References

Background: Networks for incremental learning

Incremental learning: Learning new knowledge without destroyof old learned knowledge (Stability-Plasticity Dilemma)

ART(Adaptive Resonance Theory): Need a user definedthreshold.

Multilayer Perceptrons: To learn new knowledge will destroyold knowledge

Sub-network methods: Need plenty of storage

References

Characteristics of SOINN

Neurons are self-organized with no predefined networkstructure and size

Adaptively find suitable number of neurons for the network

Realize online incremental learning without any prioricondition

Find typical prototypes for large-scale data set.

Robust to noise

References

Robust to noise

References

Robust to noise

References

Robust to noise

References

Robust to noise

References

Robust to noise

References

Architecture of SOINNTraining process of SOINNSimilarity threshold for judging input dataLearning rateSimple version of SOINNSimulation results

1 What is SOINN

2 Why SOINN

6 References

References

Structure: Two-layer competitive network

Two-layer competitivenetwork

First layer: Competitivefor input data

Second layer: Competitivefor output of first-layer

Output topology structureand weight vector ofsecond layer

References

Training flowchart of SOINN

Adaptively updatedthreshold

Between-classinsertion

Update weight ofnodes

Within-classinsertion

Remove noise nodes

References

Remove noise nodes

References

Remove noise nodes

References

Remove noise nodes

References

Remove noise nodes

References

Remove noise nodes

References

Remove noise nodes

References

First layer: adaptively updating threshold Ti

Basic idea: within-class distance ≤ T ≤ between-class distance

1 Initialize: Ti = +∞ when node i is a new node.2 When i is winner or second winner, update Ti by

If i has neighbors, Ti is updated as the maximum distancebetween i and all of its neighbors.

Ti = maxc∈Ni

||Wi − Wc || (1)

If i has no neighbors, Ti is updated as the minimum distanceof i and all other nodes in network A.

Ti = minc∈A\{i}

||Wi − Wc || (2)

References

Ti = maxc∈Ni

||Wi − Wc || (1)

Ti = minc∈A\{i}

||Wi − Wc || (2)

References

Ti = maxc∈Ni

||Wi − Wc || (1)

Ti = minc∈A\{i}

||Wi − Wc || (2)

References

Ti = maxc∈Ni

||Wi − Wc || (1)

Ti = minc∈A\{i}

||Wi − Wc || (2)

References

Ti = maxc∈Ni

||Wi − Wc || (1)

Ti = minc∈A\{i}

||Wi − Wc || (2)

References

Ti = maxc∈Ni

||Wi − Wc || (1)

Ti = minc∈A\{i}

||Wi − Wc || (2)

References

Second layer: constant threshold Tc

Basic idea 1: within-class distance ≤ T ≤ between-classdistance

Basic idea 2: we already have some knowledge of input datafrom results of first-layer.

Within-class distance:

(i ,j)∈C

||Wi − Wj || (3)

Between-class distance of two class Ci and Cj :

db(Ci ,Cj) = mini∈Ci ,j∈Cj

||Wi − Wj || (4)

References

(i ,j)∈C

||Wi − Wj || (3)

||Wi − Wj || (4)

References

(i ,j)∈C

||Wi − Wj || (3)

||Wi − Wj || (4)

References

(i ,j)∈C

||Wi − Wj || (3)

||Wi − Wj || (4)

References

(i ,j)∈C

||Wi − Wj || (3)

||Wi − Wj || (4)

References

Second layer: constant threshold Tc (continue)

1 Set Tc as the minimum between-class distance.

Tc = db(Ci1 ,Cj1) = mink,l=1,...,Q,k 6=l

db(Ck ,Cl) (5)

2 If Tc is less than within-class distance dw , set Tc as the nextminimum between-cluster distance.

Tc = db(Ci2 ,Cj2) = mink,l=1,...,Q,k 6=l ,k 6=i1,l 6=j1

db(Ck ,Cl ) (6)

3 Go to step 2 to update Tc until Tc is greater than dw .

References

db(Ck ,Cl) (5)

db(Ck ,Cl ) (6)

References

db(Ck ,Cl) (5)

db(Ck ,Cl ) (6)

References

db(Ck ,Cl) (5)

db(Ck ,Cl ) (6)

References

Updating learning rate ǫ1(t) and ǫ2(t)

Update of weight vector

∆Ws1 = ǫ1(t)(ξ − Ws1) (7)

∆Wi = ǫ2(t)(ξ − Wi ) (∀i ∈ Ns1) (8)

After the size of network becomes stable, fine tune the network

stochastic approximation: a number of adaptation steps witha strength ǫ(t) decaying slowly but not too slowly, i.e.,∑∞

t=1 ǫ(t) = ∞, and∑∞

t=1 ǫ2(t) < ∞.

The harmonic series satisfies the conditions.

ǫ1(t) =1

t, ǫ2(t) =

100t(9)

References

∆Ws1 = ǫ1(t)(ξ − Ws1) (7)

∆Wi = ǫ2(t)(ξ − Wi ) (∀i ∈ Ns1) (8)

t=1 ǫ(t) = ∞, and∑∞

t=1 ǫ2(t) < ∞.

ǫ1(t) =1

t, ǫ2(t) =

100t(9)

References

∆Ws1 = ǫ1(t)(ξ − Ws1) (7)

∆Wi = ǫ2(t)(ξ − Wi ) (∀i ∈ Ns1) (8)

t=1 ǫ(t) = ∞, and∑∞

t=1 ǫ2(t) < ∞.

ǫ1(t) =1

t, ǫ2(t) =

100t(9)

References

∆Ws1 = ǫ1(t)(ξ − Ws1) (7)

∆Wi = ǫ2(t)(ξ − Wi ) (∀i ∈ Ns1) (8)

t=1 ǫ(t) = ∞, and∑∞

t=1 ǫ2(t) < ∞.

ǫ1(t) =1

t, ǫ2(t) =

100t(9)

References

Single-layer SOINN

For topologyrepresentation,first-layer is enough

Within-classinsertion slightlyhappened infirst-layer

Using subclass anddensity to judge ifconnection isneeded.

References

Single-layer SOINN

References

Single-layer SOINN

References

Single-layer SOINN

References

Single-layer SOINN

References

Single-layer SOINN

References

Artificial data set: topology representation

Stationary and non-stationary

Stationary: all training data obey same distributionNon-stationary: next training sample maybe obey differentdistribution from previous one.

Original data Stationary Non-stationary

References

Artificial data set: topology representation (continue)

Original data Two-layer SOINN Single-layer SOINN

Conclusion of experiments: SOINN is able to

Represent topology structure of input data.

Realize incremental learning.

Automatically learn number of nodes, de-noise, etc.F. Shen, O. Hasegawa Self-organizing incremental neural network and its application

References

Unsupervised learningSupervised learningSemi-supervised learningActive learning

1 What is SOINN

2 Why SOINN

6 References

References

Some objectives of unsupervised learning

Automatically learn number of classes of input data

Clustering with no priori knowledge

Topology representation

Realize real-time incremental learning

Separate classes with low density overlapped area

References

SOINN for unsupervised learning: If two nodes connectedwith one path, the nodes belong to one class

1 Do SOINN for input data, output topology representation ofnodes

2 Initialize all nodes as unclassified.

3 Randomly choose one unclassified node i from node set A.Mark node i as classified and label it as class Ci .

4 Search A to find all unclassified nodes that are connected tonode i with a “path.” Mark these nodes as classified and labelthem as the same class as node i .

5 Go to Step3 to continue the classification process until allnodes are classified.

References

Artificial data set: 5 classes with 10% noise

Original data Clustering result

Conclusion of experiments

Automatically reports number of classes.

Perfectly clustering data with different shape and distribution.

Find typical prototypes; incremental learning; de-noise; etc.F. Shen, O. Hasegawa Self-organizing incremental neural network and its application

References

Face recognition: AT&T face data set

Experiment results

Automatically reports there are 10 classes.

Prototypes of every classes are reported.

With such prototypes, recognition ratio (1-NN rule) is 90%.

References

Experiment results

References

Experiment results

References

Experiment results

References

Experiment results

References

Prototype-based classifier: based on 1-NN or k-NN rule

Nearest Neighbor Classifier (NNC): all training data asprototypesNearest Mean Classifier (NMC): mean of each class asprototypesk-means classifier (KMC), Learning Vector Quantization(LVQ), and others: predefine number of prototypes for everyclass.

Main difficulty

1 How to find enough prototypes without overfitting2 How to realize Incremental learning

Incremental of new data inside one class (non-stationary orconcept drift);Incremental of new classes.

References

Main difficulty

References

Main difficulty

References

Main difficulty

References

Main difficulty

References

Main difficulty

References

Main difficulty

References

Main difficulty

References

Main difficulty

References

Main difficulty

References

SOINN for supervised learning: Targets

Automatically learn the number of prototypes needed torepresent every class

Only the prototypes used to determine the decision boundarywill be remained

Realize both types of incremental learning

Robust to noise

References

Robust to noise

References

Robust to noise

References

Robust to noise

References

Robust to noise

References

Adjusted SOINN Classifier (ASC)

SOINN learns k fork-means.

Noise-reduction removesnoisy prototypes

Center-cleaning removesprototypes unuseful fordecision

References

ASC: noise-reduction & center-cleaning

Noise-reduction

If the label of a node differs from the label of majority voting of itsk-neighbors, it is considered an outlier.

Center-cleaning

If a prototype of class i has never been the nearest prototype ofother classes, remove the prototype.

References

Noise-reduction

Center-cleaning

References

Noise-reduction

Center-cleaning

References

Noise-reduction

Center-cleaning

References

Noise-reduction

Center-cleaning

References

Experiment results: artificial data (I)

Original data SOINN results ASC results

Test results of ASC

No. of prototypes = 6; Recognition ratio = 100%.

References

Test results of ASC

References

Test results of ASC

References

Test results of ASC

References

Test results of ASC

References

Experiment results: artificial data (II)

Test results of ASC

References

Test results of ASC

References

Test results of ASC

References

Test results of ASC

References

Test results of ASC

References

Experiment results: artificial data (III)

Test results of ASC

No. of prototypes = 87; Recognition ratio = 97.8%.

References

Test results of ASC

References

Test results of ASC

References

Test results of ASC

References

Test results of ASC

References

Experiment results: optdigits

ASC with different parameter sets (ad , λ), displayed with averageof 10 times training and standard deviation

Parameter set of {ad , λ}(50, 50) (25, 25) (10, 10)

recognition ratio (%) 97.7 ± 0.2 97.4 ± 0.2 97.0 ± 0.2

No. of prototypes 377 ± 12 258 ± 7 112 ± 7

Compression ratio (%) 9.9 ± 0.3 6.8 ± 0.2 2.9 ± 0.2

Compare with SVM and 1-NN

LibSVM: 1197 support vectors; Recognition ratio = 96.6%.

1-NN: best classifier (98%). All 3823 samples as prototypes.

References

No. of prototypes 377 ± 12 258 ± 7 112 ± 7

References

No. of prototypes 377 ± 12 258 ± 7 112 ± 7

References

No. of prototypes 377 ± 12 258 ± 7 112 ± 7

References

Experiment results: UCI repository data sets

Comparison results of ASC and other classifiers: recognition ratio

Data set ASC (ad , λ) NSC (σ2max) KMC (M) NNC (k) LVQ (M)

Iris 97.4 ± 0.86 96.3 ± 0.4 96.2 ± 0.8 96.7 ± 0.6 96.1 ± 0.6Breast cancer 97.4 ± 0.38 97.2 ± 0.2 95.9 ± 0.3 97.0 ± 0.2 96.3 ± 0.4Ionosphere 90.4 ± 0.64 91.9 ± 0.8 87.4 ± 0.6 86.1 ± 0.7 86.4 ± 0.8

Glass 73.5 ± 1.6 70.2 ± 1.5 68.8 ± 1.1 72.3 ± 1.2 68.3 ± 2.0Liver disorders 62.6 ± 0.83 62.9 ± 2.3 59.3 ± 2.3 67.3 ± 1.6 66.3 ± 1.9Pima Indians 72.0 ± 0.63 68.6 ± 1.6 68.7 ± 0.9 74.7 ± 0.7 73.5 ± 0.9

Wine 82.6 ± 1.55 75.3 ± 1.7 71.9 ± 1.9 73.9 ± 1.9 72.3 ± 1.5

Average 82.3 ± 0.93 80.4 ± 1.2 78.3 ± 1.1 81.1 ± 0.99 79.9 ± 1.2

In average, ASC has best recognition performance.

References

Experiment results: UCI repository data sets

Comparison results of ASC and other classifiers: recognition ratio

Data set ASC (ad , λ) NSC (σ2max) KMC (M) NNC (k) LVQ (M)

Iris 97.4 ± 0.86 96.3 ± 0.4 96.2 ± 0.8 96.7 ± 0.6 96.1 ± 0.6Breast cancer 97.4 ± 0.38 97.2 ± 0.2 95.9 ± 0.3 97.0 ± 0.2 96.3 ± 0.4Ionosphere 90.4 ± 0.64 91.9 ± 0.8 87.4 ± 0.6 86.1 ± 0.7 86.4 ± 0.8

Glass 73.5 ± 1.6 70.2 ± 1.5 68.8 ± 1.1 72.3 ± 1.2 68.3 ± 2.0Liver disorders 62.6 ± 0.83 62.9 ± 2.3 59.3 ± 2.3 67.3 ± 1.6 66.3 ± 1.9Pima Indians 72.0 ± 0.63 68.6 ± 1.6 68.7 ± 0.9 74.7 ± 0.7 73.5 ± 0.9

Wine 82.6 ± 1.55 75.3 ± 1.7 71.9 ± 1.9 73.9 ± 1.9 72.3 ± 1.5

Average 82.3 ± 0.93 80.4 ± 1.2 78.3 ± 1.1 81.1 ± 0.99 79.9 ± 1.2

In average, ASC has best recognition performance.

References

Experiment results: UCI repository data sets (continue)

Comparison results of ASC and other classifiers: compression ratio

Data set ASC (a∗

d , λ∗) NSC (σ2max

) KMC (M∗) NNC (k∗) LVQ (M∗)

Iris 5.2 (6, 6) 7.3 (0.25) 8.0 (4) 100 (14) 15 (22)Breast cancer 1.4 (8, 8) 1.8 (35.0) 0.29 (1) 100 (5) 5.9 (40)Ionosphere 3.4 (15, 15) 31 (1.25) 4.0 (7) 100 (2) 6.8 (24)

Glass 13.7 (15, 15) 97 (0.005) 17 (6) 100 (1) 45 (97)Liver disorders 4.6 (6, 6) 4.9 (600) 11 (19) 100 (14) 8.4 (29)Pima Indians 0.6 (6, 6) 1.7 (2600) 1.0 (4) 100 (17) 3.4 (26)

Wine 3.2 (6, 6) 96 (4.0) 29 (17) 100 (1) 32 (57)

Average 4.6 34.2 10.0 100 16.6

In average, ASC has best compression ratio.

References

Experiment results: UCI repository data sets (continue)

Comparison results of ASC and other classifiers: compression ratio

Data set ASC (a∗

d , λ∗) NSC (σ2max

) KMC (M∗) NNC (k∗) LVQ (M∗)

Iris 5.2 (6, 6) 7.3 (0.25) 8.0 (4) 100 (14) 15 (22)Breast cancer 1.4 (8, 8) 1.8 (35.0) 0.29 (1) 100 (5) 5.9 (40)Ionosphere 3.4 (15, 15) 31 (1.25) 4.0 (7) 100 (2) 6.8 (24)

Glass 13.7 (15, 15) 97 (0.005) 17 (6) 100 (1) 45 (97)Liver disorders 4.6 (6, 6) 4.9 (600) 11 (19) 100 (14) 8.4 (29)Pima Indians 0.6 (6, 6) 1.7 (2600) 1.0 (4) 100 (17) 3.4 (26)

Wine 3.2 (6, 6) 96 (4.0) 29 (17) 100 (1) 32 (57)

Average 4.6 34.2 10.0 100 16.6

In average, ASC has best compression ratio.

References

Requirement of Semi-supervised learning

Labeled instances are difficult, expensive, or time consumingto obtain.

How can a system use large amount of unlabeled data withlimited labeled data to built good classifiers?

New data are continually added to an already huge database

How can a system learn new knowledge without forgettingprevious learned knowledge?

References

SOINN used for Semi-supervised learning

1 SOINN:represent topology,incremental learning;

2 Labeled data: label nodes(winner);

3 Division of a cluster

Condition of division

Rc−1 ≤ Rc&Rc > Rc+1 (10)

Rc =∑

a∈Nc

dis(wa, wc) (11)

c-1: former nodec+1: unlabeled neighbors.

References

Rc−1 ≤ Rc&Rc > Rc+1 (10)

Rc =∑

a∈Nc

dis(wa, wc) (11)

References

Rc−1 ≤ Rc&Rc > Rc+1 (10)

Rc =∑

a∈Nc

dis(wa, wc) (11)

References

Rc−1 ≤ Rc&Rc > Rc+1 (10)

Rc =∑

a∈Nc

dis(wa, wc) (11)

References

Rc−1 ≤ Rc&Rc > Rc+1 (10)

Rc =∑

a∈Nc

dis(wa, wc) (11)

References

Experiment: original data

5%, 15%, or 40% overlap

training samples 500, validation samples 5,000, and testsamples 5,000

labeled samples: 10% and 20%

light blue: unlabeled data; others: labeled data

- - - - - ideal decision boundary

References

5%, 15%, or 40% overlap

References

5%, 15%, or 40% overlap

References

5%, 15%, or 40% overlap

References

5%, 15%, or 40% overlap

References

5%, 15%, or 40% overlap

References

Experiment results

Separate classeswith few labeledsamples.

For UCI data sets,work better thanother typicalmethods.

References

Experiment results

References

Experiment results

References

SOINN used for active learning

Targets: Actively ask for label of some samples to label allclasses

Idea:1 Use SOINN to learn the topology structure of input data.2 Actively label the vertex nodes of every class3 Use vertex nodes to label all nodes.4 Actively label the nodes lie in the overlapped area.

References

Experiment: artificial data set under stationaryenvironment

Original data: Four classes in all, with 10% noise.

Results: under stationary environment; 10 teacher vectors.

References

Experiment: artificial data set under non-stationaryenvironment

16 teacher vectors are asked.

References

Experiment: artificial data set under non-stationaryenvironment

16 teacher vectors are asked.

References

BackgroundSOINN-AMExperimentsGeneral Associative Memory

1 What is SOINN

2 Why SOINN

6 References

References

Background: typical associative memory systems

Distributed Learning Associative Memory:Hopfield Network: most famous network, for auto-associativememoryBidirectional Associative Memory (BAM), forhetero-associative memory

Competitive Learning Associative MemoryKFMAM: Kohonon feature map associative memory.

Difficulties

Forget previously learned knowledge when learning newknowledge incrementally.

Storage limitation.

Memory real-valued data.

Many-to-Many associate.F. Shen, O. Hasegawa Self-organizing incremental neural network and its application

References

Difficulties

Storage limitation.

References

Difficulties

Storage limitation.

References

Difficulties

Storage limitation.

References

Difficulties

Storage limitation.

References

Difficulties

Storage limitation.

References

Difficulties

Storage limitation.

References

Difficulties

Storage limitation.

References

Difficulties

Storage limitation.

References

Difficulties

Storage limitation.

References

Difficulties

Storage limitation.

References

Objectives of SOINN-AM

Incremental learning of memory pairs.

Robust for noise data.

Dealing with real-valued data.

Many-to-many association.

References

Architecture of SOINN-AM

References

Algorithms of SOINN-AM

Basic idea of memory phase

1 Combine key vector and associate vector as input data.

2 Use SOINN to learn such input data.

Basic idea of recall phase

1 Using key part of nodes to find winner node for key vector,the distance is d .

2 If d ≤ ǫ, output the associative part of winner as the recallresults.

3 If d > ǫ, report unknown for key vector.

References

Original data

Binary data

Real-valued data

References

Comparison with typical AM systems

References

Robustness of noise

References

Many-to-Many associate testing

SOINN-AM recalls all patterns perfectly.

References

Architecture and basic idea of GAM

Input layer: key vectorand associate vector.

Memory layer: Memorypatterns with classes.

Associate layer: Buildassociation betweenclasses.

References

Features of GAM

Memory classes: not just memory patterns.

Real-valued data: not limited with binary data.

Many-to-many association: not limited with one-to-oneassociation.

Robust for noisy data.

Memory and recall temporal sequences.

Incremental learning: static patterns or temporal sequences.

References

Features of GAM

References

Features of GAM

References

Features of GAM

References

Features of GAM

References

Features of GAM

References

Features of GAM

References

1 What is SOINN

2 Why SOINN

6 References

References

References about SOINN

SOINN for unsupervised learning:

Furao Shen and Osamu Hasegawa, ”An Incremental Network for On-lineUnsupervised Classification and Topology Learning”, Neural Networks,Vol.19, No.1, pp.90-106, (2005)

Furao Shen, Tomotaka Ogura and Osamu Hasegawa, ”An enhancedself-organizing incremental neural network for online unsupervisedlearning”, Neural Networks, Vol.20, No.8, pp.893-903, (2007)

SOINN for Supervised learning:

Furao Shen and Osamu Hasegawa, ”A Fast Nearest Neighbor ClassifierBased on Self-organizing Incremental Neural Network”, Neural Networks,Vol.21, No.10, pp1537-1547, (2008)

References

SOINN for Semi-supervised and active learning

Youki Kamiya, Toshiaki Ishii, Furao Shen and Osamu Hasegawa: ”AnOnline Semi-Supervised Clustering Algorithm Based on a Self-organizingIncremental Neural Network,” IJCNN 2007, Orlando, FL, USA, August2007

Furao Shen, Keisuke Sakurai, Youki Kamiya and Osamu Hasegawa: ”AnOnline Semi-supervised Active Learning Algorithm with Self-organizingIncremental Neural Network,” IJCNN 2007, Orlando, FL, USA, August2007

SOINN for Associative Memory:

Sudo Akihito; Sato Akihiro; Hasegawa Osamu, ”Associative Memory forOnline Learning in Noisy Environments Using Self-organizing IncrementalNeural Network”, IEEE Transactions on Neural Networks, (2009) in press

References

Download papers and program of SOINN

http://www.isl.titech.ac.jp/˜ hasegawalab/soinn.html

Self-organizing incremental neural network and its application

Documents

Self-organizing neural networks for learning air combat...

Error-Driven Incremental Learning in Deep Convolutional...

Self-Organizing Neural Networks for Perception of...

Surface Reconstruction from Unorganized Points Using ...

A fast nearest neighbor classifier based on self-organizing...

Incremental Hierarchical Clustering of Text Documents ·...

2806 Neural Computation Self-Organizing Maps Lecture 9 2005....

Chapter 6 Neural Network Implementations. Neural Network...

Neural Networks: Self-Organizing Maps (SOM)

Neural networks Self Organizing Map by Engr. Edgar Carrillo....

Neocognitron: A self-organizing neural network model for a.....

An enhanced self-organizing incremental neural network for.....

Hierarchical Evolutionary Learning of Growing Incremental...

Unsupervised Learning with Self-Organizing Spiking Neural...

A Hybrid Self-organizing Neural Gas Based Network

Artificial Neural Network as an Incremental Non-linear