Talaska Tomacz - Analog-Counter-Based Conscience Mechanism 20060126

8/3/2019 Talaska Tomacz - Analog-Counter-Based Conscience Mechanism 20060126

http://slidepdf.com/reader/full/talaska-tomacz-analog-counter-based-conscience-mechanism-20060126 1/6

Analog-Counter-Based Conscience Mechanism inKohonen’s Neural Network

Implemented in CMOS 0.18 µm Technology

Tomasz Talaśka1, Ryszard Wojtyna1, Rafał Długosz2,3,4,*, Krzysztof Iniewski2, Witold Pedrycz2

1 University of Technology and Agriculture, Faculty of Telecomm. and Electrical Engineering, Kaliskiego 7, 85-791, Bydgoszcz,Poland, [email protected], [email protected]

2 University of Alberta, Department of Electrical and Computer Engineering, W2-079 ECERF Building, Edmonton, T6G 2V4,Canada, [email protected], [email protected], [email protected]

3 University of Neuchâtel, Institute of Microtechnology, Rue A.-L. Breguet 2, CH-2000, Neuchâtel, Switzerland4 Poznań University of Technology, Department of Computing Science and Management, Piotrowo 3A, 60-965 Poznań, Poland

* fellow of the Foundation for Polish Science

Abstract -In this study, we present a hardware

implementation of the conscience mechanism in Kohonen

self-organizing maps. The proposed realization of the

conscience mechanism is important to the functioning of the

neural network as it eliminates so-called dead (inactive)neurons. As a result the network learning, the level

quantization error can be reduced. The conscience

mechanism and the Winner Take All (WTA) block have been

implemented in 0.18 µm CMOS process. The implementation

of the conscience mechanism itself occupies 1200 µm2 and its

maximum power consumption is 9.5 µW. The WTA block

together with the conscience mechanism occupies 0.024 mm2

and dissipates 55 µW.

1. I NTRODUCTION

Continuous progress in microelectronics and Integrated

Circuit (IC) manufacturing allows for implementation of

large neural networks due to ever increasing packingdensity. Large neural networks are required to model biological systems and can be consider in numerous areasapplications [1, 3]. One of the key challenges of theeffective neural network implementation concerns a low power dissipation of the underlying hardware.

In this work we are present an IC implementation of the

Kohonen’s neural network (self-organizing map) anddiscuss several enhancements to its learning mechanism.As a result the proposed technique is faster whencompared to supervised training and does not require high precision (as being required in the case of gradient-basedlearning).

It is instructive to start with a couple of introductorycomments about the paradigm of self-organized learningas being supported by the Kohonen's maps. Typically themap is organized in a form of a grid of p x p processingunits (neurons) endowed with a vector of connections. Thenumber of inputs (dimension of input space) is equal to

“n”. The underlying principle is that of a topological preservation of a data structure in a highly dimensional

space when being mapped onto a low, two- or three-dimensional space. As usual, let us assume that at the beginning of learning all weights (connections) areinitialized to some random values. This competitive

learning process using winner-take-all (WTA) block relies

on presenting the neural network with learning vectors inorder to make the weight vectors resemble to presentedexamples. As a result of this process only one neuron, theclosest to the learning vector (where the distance is

expressed in same metric space) wins the competition [5]. Next this winning neuron adjust its connections in thefollowing way [5]:

( ) ( ))()(1 k w xk wk w −⋅+=+ η (1a)

Other neurons remain unchanged:

( ) )(1 k wk w =+ (1b)

The purpose of these changes is to make the weight

vector w resemble the learning vector x. However, allother neurons do not change their weights until they winat some point in time.

In the above formula w(k ) are given values of neuronweights, w(k +1) are new values of neurons weights,

calculated in the Kohonen's algorithm and η is so called

learning coefficient. Initial value of this coefficient must be in the range (0, 1) and its value vanishes to 0 with progress of the learning process [5].

When many similar learning vectors are applied to the

input of the neural network, one neuron wins all the timeand its weights values will converge to the average valuesof the learning vectors presented previously to thenetwork. In the case when learning vectors are differentenough, in each learning cycle other neuron wins thecompetition. In result we obtain self-organization processof the neural network. These types of networks are

frequently called SOMs (Self-Organizing Maps) [5].An important problem in Kohonen's networks concerns

with so called dead neurons. Those are the neurons thatnever win a competition and as a result their weightsnever change. Dead neurons decrease efficiency of neuralnetwork's training process, because smaller number of neurons take part in the competition [4].

In this work, we are presenting CMOS 0.18μmimplementation of the conscience mechanism, which due

to improvement of the learning algorithm, allows for reduction of the so-called dead neurons. The presentedalgorithm is one of the many variants that are currentlyunder consideration by authors. Recently we have shownthe implementation of the conscience mechanism usingdigital counters [7]. The counters implemented using

digital techniques are stable, but have limitation: theyexhibit fixed counting limits. The solution presented here proposes a novel concept of analog counters, that enabletuning of the counting range in a large range and offer

Published in IEEE Workshop on Signal Processing Systems (SIPS) IT5, LW2-2, 420-425, 2006which should be used for any reference to this work

1



much lower power dissipation levels than their digitalcounterparts.

The paper is organized as follows. In Section 2 wediscuss an essence of the learning process and thecorresponding ASIC architecture that implements WTA

block using Kohonen’s rule.In Section 3, given are details of the implementation of theconscience mechanism. A collaboration between the

conscience mechanism and the WTA block is discussed.

Section 4 presents a CMOS layout implementation and post-layout simulation results. Finally, conclusions andfuture work are converted in section 5.

2. ASIC IMPLEMENTATION OF KOHONEN’S NETWORK

Model responsible for neural network training processis shown in Figure 1. EDC sub-circuits represent blocksresponsible for calculating a degree of similarity between

training and weight vectors for each neuron. In order to

1

n

Input

learnig

vectors

[ X ]

w11(k )

w2n(k ) w21(k )

Σ

W T

A

I sq1(k ) I cons1(k )

I count1(k )

EDC

EDC

W mn(k ) W m1(k )

EDC

CNR

AWC

Σ

I sq2(k ) I cons2(k )

CNR

Σ

I sq m(k ) I cons m(k )

CNR

I count2(k )

I count m(k )

w1n(k ) w11(k ) w1n(k )

w11(k+1)

w1n(k+1)

Enable1

AWC

w21(k ) w2n(k )

w21(k+1)

w2n(k+1)

Enable2

AWC

W m1(k ) W mn(k )

W m1(k+1)

W mn(k+1)

Enablem

CONSC

Fig. 1 Diagram of the proposed learning block of Kohonen's neural network for m neurons and n inputs: EDC - Euclidian’s distance calculation blocks, CONSC - conscience mechanism blocks, CNR - analog counter, AWC - adaptation (Kohonen's algorithm) weight correction block

V DD

V SS

V DD

V SS

2

1cons1consd I =

V ref

C

V DD

Q1D1

NQ1

reset

clk

QmDm

NQm

reset

clk

DFF1

DFFm

Enable1

Enablem

reset

clock

clk

1 2

3

2

conscons mmd I =

1cons I

m I

cons

ref

I

ref I

Fig. 2 Winner Takes Block (1) Current comparators (2) Digital control block (3) Source of reference signal [7]

2



calculate the degree of similarity between two vector wehave used Euclidian’s measure. Calculation of thatmeasure in hardware is performed in differentialtransconductance circuits [7] described in the form

( )2

squar W X V V A I −−= (2)

In this expression, A is a constant parameter determined

by transistor sizing, V X and V W are voltages that representinputs and weights of the neuron, while I squer is an singlesquarer's output current.

Figure 2 presents a WTA circuit used to detect awinning neuron, viz the neuron for which a degree of similarity with the input of the networks is the highest.The principles behind circuit operation have been

described in [7], and are presented shortly in Section 4.

3 IMPLEMENTATION OF THE CONSCIENCE MECHANISM

The task of the conscience mechanism is artificiallyincreasing of the Euclidean's distance between input

signals (vector X ) and weights of the neuron (vector W ),that have won competition in previous iteration of the

training process. This decreases the probability for win of this neuron in the next iteration. As a result neurons thathave never won achieve certain ability to win thecompetition. It is important to ensure that when thetraining process is advanced and all neurons do take partin the competition, this mechanism could be disconnected.

This allows the network to perform its functionindependently on the number of neuron wins.

A schematic diagram of the proposed consciencemechanism is illustrated in Figure 3.

wins counter (CNR)

d sq ( X , W ) d cons2( X , W )

Σ

z·d count2 Enable

Fig. 3 Idea of the proposed conscience mechanism

Fig. 4 Analog counter used in conscience mechanism

The conscience mechanism is realized as follows. Notethat (3) and (4) represent a relationship between Euclidian

distance between training vector x and weight vector w for the winning neuron, increased by a factor proportional to

number this neurons wins a competition.

)(),(),( cons

2

count

2

sq

2

cons k I z d W X d W X d i=⋅+= (3)

( ) )()( count

)(

1

2

cons

sq

k I w x Ak I i

k I

n

j

ij ji

i

+⎥⎦

⎤⎢⎣

⎡−⋅= ∑

=

4 4 4 84 4 4 76

(4)

M6M1

I countM3

V DD

M4

M2 M8

V con

M5M10M11

V countM7 M9M12

M13 M14

V SS

V G1V G2

Fig. 5 Differential transconductance U-I converter used in conscience

mechanism to convert voltage from analog counter (Vcount from C2

capacitor) to current I count

Fig. 6 Illustration of the range's flexibility of the analog counter used in

conscience mechanism (up) for Vcontr = 0.77V counting to 3 (bottom) for

Vcontr = 0.6V counting to about 400

In these equations d sq2

is the real Euclidean's distance

between vectors X and W , d cons2 is the distance artificially

increased by factor d count2

from conscience mechanism.All these variables are represented by currents.

3



In above equations index j determines the input nodeand the weight of a given neuron, index i is the number of the following neuron ( x j is the following input while w j isthe following weight), n describes number of weights,

which is equal to number of inputs of the neural network.

Fig. 7 Post-layout simulations of the conscience mechanism (analog

counter and U-I converter) for various values of the control voltages

Vcontr (Fig. 4) and Vcon (Fig. 5)

Parameter z is used to control of the influence of theconscience mechanism the so-called multiplier that can be

used for additional operation. This parameter, in hardware

implementation represented by DC voltage, enables for example, the conscience mechanism can be turned-off.

The counter of the neuron wins can be implemented asanalog or digital block. Because digital counters become

power and area inefficient for large neural networks [7],we have decided to implement the counter in analogdomain, which offers a far higher flexibility solution.

A basic idea of the proposed solution for the analogcounter is shown in Figure 4. The counter operation is based in the following principle. Successive negated input

pulses arriving from the WTA block open the currentsource MP1 that charges capacitor C2. When the voltagestored on C2 becomes larger than the threshold voltage thevoltage at the NOT1 output changes from V DD to V SS. As a

result NOT2 output changes to V DD when the switch K1 isclosed. When the CLK becomes V SS and CLK negated becomes V DD the reset signal goes high. This in turncauses turning on the process of discharging C2 throughMN2, which implies zeroing of the counter. The switchK1 allows for synchronization of the RESET signal with

CLK in order for the discharging impulse to last longenough. The voltage stored on C2 (proportional to number of wins) is converted to current that is added to the outputcurrent of the squaring circuits.

In the analog counter circuit shown in Figure 4 thecounter input signal CLK is connected to the output of the

WTA circuit (signals Enable in Figs 1 and 2). Using thecontrol voltage V contr the resolution of the counter can becontrolled, as illustrated later in Figure 6. The output of

the analog counter V count has to be converted to current I count. This can be realized in the U-I converter proposed by authors [6], which is a differential transconductor shown in Figure 5.

The following equation represents the relationship between the output current and input voltages:

( )( )thssCONG1G2count 2V V V V V K I −−−≅ (5)

In the above equation K is a constant dependent ontransistor sizing, V th is a transistor threshold voltage. V CON voltage is used to control multiplier gain, V G1 and V G2 arevoltages at the gates of M1 and M2 dependent on the

voltages V in1 and V in2 (output V count from the counter).Figures 6 and 7 represent post-layout simulation results

for the conscience circuit for few values of V contr and V con.It is apparent that the proposed circuit is very flexible.

Figure 6 shows the analog counter output for variouscounting ranges (e.g. for 3 and for about 400). Figure 7

represents the conscience circuit output current I count increasing Euclidian’s distance combined with increased

number of wins represented as V count voltage in the C2capacitor.

4 K OHONEN'S TRAINING NETWORK IMPLEMENTATION

Layout of the entire Kohonen's block is presented

schematically in Figure 1, implemented in CMOS

0.18 μm technology is shown Figure 8.

Experimental results shown in the subsequent part of this section relate to collaboration of the consciencemechanism block with blocks responsible for calculatingEuclidian’s distance for the neurons and WTA block.

4



Fig. 8 Layout of the WTA block implemented in 0.18 µm process. Block 1 calculates Euclidian’s distance, block 2 implements consciencemechanism, and block 3 provides detection mechanism of the winning neuron

Fig. 9 WTA signals: (up) signals in analog counters in conscience mechanism for neurons A, B, C

(bottom) WTA block outputs - signals "Enable" in Fig 1.

Fig. 10 Signals in the winning neuron detection circuit (Fig 2) that illustrates competition between neurons A, B, C - NOT gates (block 1) input

voltages

5



To verify Kohonen’s algorithm and the proposedimplementation numerous test case simulations have beenundertaken. In one experiment input and weight vectorshave been fixed and assigned as follows: x1 = 1 V, x2 = 1 V, w11 = 1.2 V, w12 = 0.9 V, w21 = 1.3 V,w22 = 0.8 V, w31 = 1.3V, w32 = 0.3 V.

The results of these simulations are shown in Figures 9and 10. After applying the training vector X thecompetition is won by neuron A (Figure 9, bottom-A). k presentations of the same training vector X decrease

winning probability of the neuron A as a result of actionsof the implemented conscience mechanism. Consequently,

at some point the neuron B can win (Figure 9, bottom-B).After several presentations of the training vector X winning probability of neurons A and B is decreased tothe level, which enables neuron C also to win (Figure 9, bottom-C) although its real Euclidean’s distance betweenits weight vector and training vector is significantly larger

than neurons A and B.Figure 10 illustrates operation of the WTA block

presented in Figure 2 and, in fact, competition betweenneurons due to operation of the conscience mechanism.WTA's input currents are in fact conscience mechanism'soutput currents I cons. Neuron whose weights W are theclosest to the input signals X generates the smallest current I sq. The reference current (dependent on reference voltageV ref ) from block number 3 (Fig 2) is increasing and

compared to all I cons simultaneously.Output of the comparator corresponding to the winning

neuron will fall down as first, what is detected by digitalcontrol circuit and interpreted as a win of this neuron.

It is important to point out that if the consciencemechanism would not be implemented, the neuron A

would be always winning when presented with the giveexample training vector X . As a result neurons B and C

would be dead in this case. In the Figure 11 we see thatalthough neurons A and B win alternately because their weights are the closets to the training vector X , the timedistance between neuron C and neurons A, B decreasesafter each presentation of the training vector X as a resultof actions of the implemented conscience mechanism. In

In Figure 10 we see that after many presentations of thetraining vector X , time distance between switching of theoutputs of the comparators associated with particular neurons diminishes.

This is the result of the constant training vector X in this

experiment. Different values stored in a given time instantin particular counters compensate the real Euclidean’sdistances between vectors. These real distances are

constant due to the constant input values X and W , but inthe real network training vector X will not be constant.Constant value of X is assumed here only to better illustrate operation of the proposed WTA block withconscience mechanism.

5. CONCLUSIONS

The implemented conscience mechanism in the Winner

Take All (WTA) block represents a next step towards acomplete IC implementation of the Kohonen’s neuralnetwork. The proposed circuit of the consciencemechanism, implemented in 0.18 µm technology usinganalog counters, occupies 1200 µm2 (40 µm x 30 µm) anddissipates 9.5 µW.

The analog counter has certain leaking propertieswording due its implementation as analog voltage storedon a capacitor. Charge leaking process in the analogcounter causes conscience mechanism to become weaker as the timing distance between subsequent winnersincreases. As a result, this parasitic effect may notnecessarily be viewed as a disadvantage from the training process point of view. The penalized neurons slowly

recover full rights to competition.The entire Kohonen's training block occupies

0.024 mm

2

(120 μm by 200 μm) and in the case of 3output neurons and two-dimensional input vector X

dissipates 55 µW. The chip is currently in fabrication atTaiwan Semiconductor Corporation (TSMC). The power

dissipation is small enough that even in very large neuralnetworks (in which one may require several hundredsWTA blocks), the total power consumption would bemanageable.

The final step required for full implementation of thetraining process shown in Figure 1 is adaptation weight

correction block (AWC). Such a mechanism is currentlyunder implementation and will be reported elsewhere.

R EFERENCES

[1] Cauwenberghs G., Bayoumi M.: Learning on silicon, adaptive

VLSI neural systems, Kluwer Academic Publishers, 1999.

[2] Deboeck G., Kohonen T.: Visual explorations in finance with self-

oranizing maps, Spinger-Verlag,1998.

[3] Maass W., Bishop C.: Pulsed neural networks, Massachusetts

Institute of Technology, The MIT Press, 1999.[4] Zurada J: Introduction to artificial neural systems, West

Publishing Company, USA, 1992.

[5] Kohonen T.: Self-organizing maps, Springer Verlag, Berlin 2001[6] Wojtyna R., Talaska T.: Improved Power-Saving Synapse for

Hardware Implemented ANN’s, Int. Conf. on Signals and

Electronic Systems ICSES’04, Poznań, Poland 2004, pp. 27-30.

[7] Talaśka T., Wojtyna R., Długosz R., Iniewski K.: Implementationof the conscience mechanism for Kohonen’s neural network in

CMOS 0.18 µm technology, International Conference Mixed Design of Integrated Circuits and Systems, Gdynia, Poland 2006, pp. 319-315.

[8] Wawryn K, Strzeszewski B.: Low power VLSI neuron cells for

artificial neural networks, in Proc. ISCAS, vol.3. Atlanta 1996, pp. 372-375.

____________________ This work was supported partly by

the KBN grant No. 1580/T11/2005/29.

6

Talaska Tomacz - Analog-Counter-Based Conscience Mechanism 20060126

Documents