Equilibrium characterization of dynamical neural networks for synthesis of associative memories

IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 2, NO. 5, SEPTEMBER 1991 509

Brief Papers

Equilibrium Characterization of Dynamical Neural Networks and a Systematic Synthesis Procedure for Associative Memories

Subramania I. Sudharsanan and Malur K. Sundareshan

Abstract-Several new results concerning the charac te r iza- tion of the equi l ibr ium conditions of a cont inuous- t ime dynamical neural network model a n d a systematic p rocedure for synthesizing associative memory networks with nonsymmetr ical interconnection matr ices a r e presented. T h e equi l ibr ium char - acterization focuses on the exponent ia l stabil i ty a n d instability properties of the network equilibria a n d on equi l ibr ium confinement, viz., ensuring the uniqueness of a n equi l ibr ium in a specific region of the s ta te space. While the equi l ibr ium confinement result involves a s imple test , the stabil i ty results given here obtain explicit estimates of the degree of exponent ia l stability a n d the regions of a t t rac t ion of the stable equi l ibr ium points. Employing these results a s valuable guidelines, a systematic synthesis procedure for construct ing a dynamical neura l network tha t stores a given set of vectors as the stable equilib- r ium points is developed. I n cont ras t t o cer ta in known procedures for the synthesis of dynamical neura l network associative memories, the present p rocedure not only expands t h e scope of memory s torage by removing the restrictions of symmetry on the interconnection mat r ix b u t also constructively exploits the roles of the var ious network pa rame te r s (such a s the nonl inear functions in addi t ion to the weight matr ices a n d bias inputs) in identifying a scheme fo r systematically tailoring these p a r a m - e te rs for a n efficient synthesis.

I. INTRODUCTION HE problem of developing efficient procedures for de- T signing neural networks that store a prespecified set

of memory vectors as stable equilibrium points has at- tracted considerable attention in the recent past. Among the several diverse approaches that have been proposed in the literature, a few representative ones are the Hebbian learning schemes used by Hopfield [l], Kosko [2], and Pineda [3]; methods employing least-squares solutions of matrix equations used by Kohonen [4], Anderson [6], and Michel and coworkers [7], [37]; approaches using the projection learning rule developed by Personnaz et al. [3 11, [32] ; master-slave formulations of optimization problems from Lapedes and Farber [8] and Sudharsanan and Sundareshan [9] ; and contraction mapping arguments used by Stubberud and Thomas [IO]. An excellent survey of several synthesis techniques which gives a comparative evaluation of their strengths has recently been given by Michel and Farrell [37].

Manuscript received July 2, 1990; revised March 26, 1991. S . I . Sudharsanan was with the Department of Electrical and Computer

Engineering, University of Arizona, Tucson, A 2 8572 1. He is now with the Advanced Development System Design, IBM Corporation, Boca Ra- ton. FL 33429.

M. K . Sundareshan is with the Department of Electrical and Computer Engineering, University of Arizona, Tucson, AZ 85721.

IEEE Log Number 9100486.

Although both static and dynamic network models have been widely considered for various applications, a study of dynamical neural networks is of greater interest not only because of the significantly extended range of capabilities they provide but also for the more complex and challenging problems encountered in formulating satisfactory design procedures. Fundamental to the development of systematic design procedures for dynamical neural networks to serve as associative memories are the qualitative properties of the network. A detailed qualitative analysis fo- cusing on a characterization of the network equilibria and the stability properties of the equilibrium conditions is highly useful in these designs [7], [26]. In particular, conditions on the network parameters that ensure sufficiently large basins of attraction for each stable equilibrium point serve as valuable guidelines.

In conducting a study of the qualitative properties of the network, certain specific types of questions attain a higher degree of importance in the context of the intended application of the network as an associative memory as distinct from the other possible applications, for instance, optimization. For solving optimization problems one would like to have a network with a single stable equilibrium point that corresponds to the global minimum of the objective function, whereas in the case of an associative memory, in order to be useful at all, one requires that there exist many stable equilibria. These issues have been discussed in the context of synthesizing a neural network for optimization applications in [ 131, where conditions for the existence of a unique equilibrium in the entire state space are given. For a neural network to function as an associative memory, however, one should be interested in a different set of conditions to meet the requirement that the network store the specified memory vectors as stable equilibria confined to specified quadrants of the state space. Moreover, there are several additional complexi- ties that one should contend with. When the vectors are stored upon completion of the design, it is not guaranteed that the network will have only the stored vectors as the stable equilibria; many other vectors may also be introduced as stable equilibrium points. This will in general result in smaller regions of attraction of the stored vectors and also lead to spurious stable outputs, rendering the memory recall rather unreliable.

The specific concern for ensuring large regions (basins) of attraction for the stored memories is closely related to the memory capacity of the network, and a number of

1045.922719 1/0900-0509$01 .OO 0 199 1 IEEE

510 IEEE TRANSACTIONS ON NEURAL NETWORKS. VOL 2. NO. 5. SEPTEMBER 1991

researchers have paid attention to the capacity evaluations of particular types of associative memory models. While considerable literature exists on such evaluations for the discrete-time Hopfield model (i.e., the recognition of a binary memory with respect to its Hamming distance) [ 141-[ 191, corresponding evaluations for continuous-time network models are rather few. For the specific class of continuous-time models considered in this paper, the maximum number of stable equilibria, i.e. the capacity, was first shown to be 2” by Li et al. [7], where y1 is the number of neurons characterizing the network. Some questions on the accuracy of this evaluation were, however, raised by Salam and Wang [20], who generated some counterexamples via numerical simulations. The inaccu- racy in the computation in [7] was later traced to an er- roneous application of the mean value theorem in the course of demonstrating the existence of only one equilibrium point in a quadrant of the state space [21]. One of the principal results in the present article enunciates simple conditions for ensuring the uniqueness of a stable equilibrium in a given quadrant.

Our focus in this paper will be on the synthesis of a continuous-time neural network model described by

U = -Au + Wg(u) + b

where U: [0, 00) -+ Bfl , A E Bff ”’ 3 A = diag [ a , , u2, . . . , a , ] , ai > 0, W E Bfl ’ I , b E B f f , and g( . ) : B” +

@if’ is a vector-valued function with sigmoidal elements, to efficiently serve as an associative memory. To aid in this development, we shall present some basic results from a qualitative study of the equilibrium characterization of dynamic network models described by (1) . It should be mentioned that the stability properties of neural networks of the present type and their generalizations have been analyzed quite extensively in the context of associative memory designs (without claiming to be exhaustive, one may mention the works [7], [ 1 11, [ 121, [22], and [26] for representative results). However, several of the existing design procedures make essential use of the symmetry of the interconnection matrix W in order to associate with the system an energy function, E , whose derivative along the system trajectories can be constrained in an appropriate manner in order to draw conclusions on the stability status of the equilibrium points through a standard Lya- punov analysis. Requiring W to be symmetric, however, is a rather severe restriction and imposes serious limita- tions on the capability of the network to store specified sets of memory vectors. Hence, in the entire development given in this paper, we shall exclude such restrictions and consider networks where W is not required to be symmetric.

To be fair, however, we should mention that use of symmetric W enables one to guarantee the global stability of the network, i.e., a guaranteed convergence to an equilibrium point starting from any arbitrary initial state and also an absence of limit cycle oscillations. Such considerations, as pointed out in [31], [37], are of particular importance in analyzing the dynamics of information re-

trieval from the stored memory. Nonsymmetrical interconnection matrices, in contrast, lead to networks which can store information as the stable equilibrium points, but which can also lead to limit cycles. Limit cycles may not be preferred in some application scenarios. A more detailed and complete discussion of the implications of symmetric versus nonsymmetric interconnection matrices on the process of association and information retrieval can be found in [37].

In this paper we shall present simple conditions for the exponential stability of the equilibrium points of the network together with explicit estimates of the rate of decay of the trajectories to these equilibrium points (i.e., degree of exponential stability). Procedures for estimating the regions of attraction of these equilibrium points will also be outlined. Since, in addition to the stable equilibrium points, a dynamical network described by (1) can have a number of unstable equilibrium points, we shall briefly state a representative instability result which may be regarded as the counterpart of a corresponding stability result. It must be noted that conditions that prove the instability of an equilibrium state are not merely of aca- demic interest and are valuable not only in unambiguously drawing conclusions on the stability status of that equilibrium but also in obtaining an understanding of the trajectory behavior before convergence to other stable equilibrium points takes place. By following approaches that are different from the ones used here, similar questions concerning the stability properties of dynamical neural networks have been investigated in [7], [21], [22], [26], and

The study of equilibrium characterization conducted here will include devising a simple test that ensures the confinement of network equilibria to distinct quadrants of the state space. This test involves checking the signs of the principal minors of a matrix and is different from the types of conditions derived in [7], [2 11, and [38] for other classes of neural networks. It is also of significant importance in associative memory applications, since it ensures that a design executed to store a given set of vectors as the stable equilibrium points of a dynamical neural network will not introduce certain other (spurious) stable equilibrium points in close proximity to the to-be-stored memories.

The results of the qualitative analysis conducted here can be used in devising efficient synthesis procedures. As an illustration, we shall develop a systematic procedure for identifying the various parameters of a neural network whose dynamics are described by ( l ) , to store a given set of vectors as its stable equilibrium points. It is shown that the present procedure has various advantages over existing schemes of its type, which are explicitly illustrated through a numerical example. A particularly important feature of the design that must be underscored is that it permits a systematic tailoring of the nonlinear functions (in terms of their slopes at specified points) in addition to the selection of self-activation terms, interconnection weights, and bias inputs. In contrast, a majority of the

1361.

IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 2. NO. 5, SEPTEMBER 1991 51 1

available synthesis procedures start with arbitrarily selected sigmoidal functions and focus on the selection of interconnection weights and bias inputs only. It should be noted that the possibility for the use of a constructive selection of the nonlinear input-output characteristics for expanding the scope of memory storage does not seem to have been widely appreciated.

The paper is organized as follows. Section I1 outlines some basic properties of the neural network model and introduces certain concepts that will be used later. In Sec- tion 111, results for the exponential stability and instability of the equilibrium points are given and procedures for estimating the regions of attraction of the stable equilibrium points are outlined. Section IV deals with the equilibrium confinement problem and presents a sufficient condition for locating the equilibrium points in distinct quadrants of

s the state space. In Section V, a systematic synthesis procedure for constructing a neural network to serve as an associative memory is presented and some important features of this design are identified. The paper is concluded with a few pertinent remarks in Section VI. It must be emphasized that all the results presented in this paper are developed for a dynamical neural network with an asym- metric interconnection structure and hence considerably enlarge the scope of the associative memory design when compared with several established procedures for this problem.

11. NETWORK PROPERTIES AND SOME BASIC CONCEPTS The continuous-time neural network model defined by

( I ) is equivalent to the set of n scalar equations

(2)

where U,, b,, and w,, are elements of U: [0, 03) -+ (R", b E (R", and W E (R" ' I respectively, n denoting the number of neurons. It should be emphasized that in contrast to several earlier works, we do not require W to be symmetric.

The nonlinear functions g,(. ), i = 1, 2 , , n are assumed to satisfy the following conditions:

(i) u,g,(u,) > 0 VU, E 63

(ii) lim = sgn ( U , ) 1 ( I 1 I - 03

(iii) g,(u,)/u, 2 g,(u,)/u, v I4 I 5 I U , I (iv) g,'(u,) = dg,(u,)/du, > 0 V U , E (R. (3)

Remark 1: It is easily seen that the above conditions are satisfied for the commonly used sigmoidal nonlinear- ities g,(u,) = tan-' (7rhu,/2)2/7r and g,(u,) = tanh (hu,) [21, ~ 7 1 , [ i l l .

For a network described by (2), (or (l)), when ai, wij, bi, and gi( ) are specified, the equilibrium points are given

by U* E (Rn 3 U* = [U:, U:,

i = 1, 2 , * *

equations

* , ufIT, where U:, , n , are solutions of the nonlinear algebraic

Since there exist, in general, multiple equilibria and we are interested in the qualitative properties of each equilibrium point, we shall assume that a given equilibrium point U* being analyzed is an isolated equilibrium condition; i.e., there exists an r > 0 such that the region @(U*, r) C (Rf7 3 63 = {U: 11 U - u*ll < r } contains no equilibria other than U*. For the class of neural network models considered here, it can be shown, following [7], that this is a reasonable assumption. Here and in the following, 11 * 11 denotes the L 2 norm.

, u f I T , with a coordinate transformation x, = U, - U:, we can rewrite (2) as

For a given equilibrium point U* = [U;, U;, *

n

I = I XI = -a,x, + C w,g,(x,>, i = I , 2, * . 9 n ( 5 )

where g,(x,) = g,(x, + U:) - gJ(uJ*). It is simple to see that g,(O) = 0 and hence x = [ x , , x2, * . , x,JT, x, = 0, i = 1 , 2 , e . . , n, defines an equilibrium point of (3, which is the mapping of U* under the above transformation. Furthermore, it can be shown, under the assump- tions given in (3), that there exist positive real numbers a,, p,, and r, such that

with pi > & ( U T ) and a; < g,! (U:). For the sake of precision in the development given in

this paper, we will use the following notation and define a few terms. Given a matrix M E (R'n xn, M T denotes its transpose. For M E (Rn Spec ( M ) denotes the set of its eigenvalues. For M E (R ' l x " 3 M = M T , h,&V) denotes the minimum eigenvalue of M and h,(M) denotes its maximum eigenvalue. For M E ( R ' l X ' n , I[MII denotes the spectral norm h g 2 ( M T M ) . We shall use LHP to denote the left half of the complex plane.

Dejinition I : A matrix M E (R" 3 M = M T is said to be positive dejinite (positive semidejinite) if

x T M x > O ( 2 0 ) V X E &",x # 0.

DeJinition 2: A matrix M E (Rn x n is said to be a 6 matrix if all its principal minors are positive.

Many useful properties of 6 matrices are given in texts on linear algebra [28]. Among these, two specific properties are of special interest in our further development.

512 IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 2 , NO. 5 , SEPTEMBER 1991

Property 1: If M E @ ” x ” is a 6 matrix, then there exists a diagonal matrix D, E CR” ”’ with positive diagonal entries for each x # 0 such that

xTD,Mx > 0, vx E all, x # 0.

Property 2: If M E R“’” admits a decomposition M = M I - M2, where M I is nonsingular, then IIM,lMZII < 1 * M is a 6 matrix.

111. STABILITY ANALYSIS OF EQUILIBRIUM POINTS

In this section we shall present two theorems which state the conditions that guarantee exponential convergence with a specified degree of the network trajectories to certain equilibrium points. In contrast to the stability results given in [ 1 11, [ 121, and [22], our results explicitly identify the degree of exponential stability. As an illustration of the development of conditions which are sufficient to conclude that a given equilibrium point is unstable, we shall present an instability result. We shall also outline a few procedures for the estimation of the region of attraction for a stable equilibrium point. Similar questions concerning the characterization of the qualitative properties of continuous-time dynamical neural networks are investigated in [7] and [26] by following approaches that are different from the one used here.

A. Exponential Stability-Some Basics Let us define a region 63 C a” specified by the positive

constant r such that 63 = {x: 11 x 11 I r } . Also assume that the equilibrium point x* = 0 of the dynamical system

x = f ( x ( t ) ) , f ( 0 ) = 0 (7) where x ( . ): + CR” andf( ): a’’ -+ af’, is an isolated equilibrium contained in 63 (i.e., no other equilibria of (7) are inside 63). We then have the following definition

Dejnition 3: The isolated equilibrium x* = 0 of the system described by (7) is exponentially stable in 63 with degree r if every trajectory starting at any feasible initial state x ( t o ) = x,, E 63 satisfies the condition

Ilx(t)ll 5 dlx~11 exp ( - r ( f - to)) v t 2 to, vx E 03

(8)

where P and 77 are positive constants independent of the initial conditions (to, xo) .

It should be noted that there is no loss of generality in the consideration of the equilibrium point at the origin, since if x* # 0, a coordinate transformation 2 = x - x* readily transfers this equilibrium point to the origin which will be an equilibrium point of the transformed system.

We shall now briefly state a basic result from the stability theory of dynamical systems [23], which will be used in the establishment of the exponential stability conditions for the equilibria of the neural network under consideration in this paper.

~ 3 1 .

Proposition 1: The equilibrium point x* = 0 of (7) is exponentially stable in 03 with degree 7 if there exists a function V : af’ + a, called a Lyapunov function, that satisfies the following conditions:

a) V(x) has a continuous partial derivative with respect

b) V( ) is positive definite in @ (i.e., V(x) > 0 V x E

c) the time derivative of V( . ) along the trajectories of

to each element of x E R”;

03, x # 0 and V(0) = 0);

system (7) satisfies

9x9 I -2qV(x) dt

vx E a, V t 2 to.

B. Stability Results for the Neural Network

alently, the n-dimensional system Let us consider the transformed network ( 5 ) or, equiv-

(9) where A = diag [al, a2, * * . , a,,] and g ( x ) = [ g l ( x l ) , g2(x2)> * * , gll (xJ]T. Recall that x* = 0 is an isolated equilibrium of (9). Let us define G E & ” x ” 3 G = diag

x = -Ax + Wg(x) = -Ax + h(x)

[ g ; , si, . . . , S,‘~I, where

For a precise statement of a result enunciating conditions for the exponential stability of the equilibrium point x* = 0 of the system described by (9), let us make use of the following hypotheses:

(H-1) Spec(WG - A ) C LHP. (H-2) For a specified K , > 0, there exists r > 0 that

defines the region 03(Kl) C a” 3 @ ( K , ) = {x: I/xII < r } such that

1 1 h(x) - WGXII I K1 IIxll V X E 03(K,). (1 1)

Remark 2: Hypothesis (H-2) holds trivially for nonlinear functions g(x) whose elements satisfy the conditions stated in (3). Observe that since a Taylor series expansion around x = 0 results in h(x) = W g ( x ) = WGx + R(x) , where R(x) contains the higher order terms (i.e., lirn,+,,

1 1 R(x)II / I 1 x 11 = 0, V 11 x 11 < 6, for some 6 > 0), for any specified K , > 0 it is possible to find a neighborhood 03(K,) of x = 0 where 11 R(x)II is bounded by the linear term K , II x II .

We then have the following result.

Theorem 1: For the system described by (9), let (H-1) and (H-2) hold with K , = 1 /((2 + E ) A,(P)) for some E

> 0, P E CRfi “I being the symmetric positive definite solution of the matrix equation

(WG - A)TP + P(WG - A ) + I = 0. (12)

Then, the equilibrium x* = 0 of system (9) is exponentially stable with degree 7 = 0.5cKI in @ ( K l ) .

Proof: Since Spec(WG - A ) C LHP, the solution P of (12) such that P = P T and P is positive definite exists


and is unique [30]. Consider the Lyapunov function candidate V(x) = x'Px. Then, along the trajectories of (9),

V(X) x ' [ (WG - A)TP + P(WG - A)] x + 2xTPR(x)

= - x T x + 2xTPR(x)

-27x'Px - [xT(Z - 2 y P ) x - 2xTPR(x)]

where R(x) = h(x) - WG(x) . Hence,

V(X) I -2vV(x) VX E @(KI), V t 2 to (13) if x T ( I - 27P)x - 2xTPR(x) 2 0 V X E @(K,) . (14)

From (H-2) , it follows that

l12xTPR(x)ll 5 2KIhM(P) Ilxl12

Hence, (14) holds if 1 - 27XM(P) L 2 / ( 2 + E ) , which trivially holds for the specified values of y and K , . Thus, (14), and hence (13), are true and the theorem is

Remark 3: It is evident that the degree of exponential stability, 7, is dependent on K I , which also determines the size of & ( K l ) . Although the degree of stability computed here gives a lower bound on the actual rate of decay of the network trajectories, the observation made above is useful to underscore a behavior that was observed in simulation experiments, namely that as the trajectory gets closer to the equilibrium point in question it experiences a faster decay.

Remark 4: As an interesting corollary to Theorem 1, a more direct way of estimating the region of attraction of the equilibrium point x* = 0 can be given. Let hypothesis ( H - I ) hold and let p = 1 / ( 2 X M ( P ) (1 Wll), where P E 01" is the symmetric positive definite solution of ( 1 2 ) . Now, from (6) it is evident that there exist positive numbers r l , r2,

proved. 1

* , r, such that

where g,! = (dg , (x , ) ldx , ) ( x , = o . Then, with regard to the region 63 c a" 3 , n } , we have the following result.

Corollary I : 63 is contained in the region of attraction of x* = 0.

Proof: Observe from the proof of Theorem 1 that

= {x: Ix,( I r,, i = I , 2 ,

V(x) = - x T x + 2xTPR(x)

and hence t(x) is negative definite if

which establishes that 63 is contained in the region of attraction of x* = 0.

Now, since R(x) = W ( g ( x ) - Gx), the inequality

ensures ( 1 6 ) . However, the set of elementwise inequali- ties

It?,(X,) - s:x,II < PlX, l

V l x , l 5 r,, i = 1, 2 , , N (18)

(15 ) . 1 is sufficient for (17) to hold, which is further satisfied by

Remark 5: Since, in general, a neural network defined by (1) will have multiple equilibrium points, some of which could be unstable, establishing conditions for the instability of an equilibrium point will be highly useful to serve as guidelines in the synthesis of associative memories. Toward this end, we will give a result which could be regarded as the instability counterpart of Theorem 1.

Theorem 2: For the system described by (9), let WG - A have at least one eigenvalue with positive real part. Then, the equilibrium x* = 0 is unstable.

Proof: The proof runs along lines similar to that of Theorem 1. Since WG - A has at least one eigenvalue with positive real part, the matrix equation

(A - W G ) T P , + Pl(A - W G ) + I = 0 (19) has a solution P I E 01" " such that P I = PT and PI has at least one positive eigenvalue [ 2 4 ] . Now, using V(x) = x T P l x, which attains both positive and negative values arbitrarily close to the equilibrium at x* = 0, one obtains

V(x) = x T x + 2 x T P 1 R ( x ) .

Since R(x) = h(x) - W G x satisfies limx+o ( 1 1 R(x)II /I1 x 11 = 0, it is evident that there exist positive constants c I and c2 such that

V(x) > c l x T x VIlxll < c2 (20) which, by Cetaev's instability theorem [ 2 5 ] , establishes that x* = 0 is unstable. 1

Remark 6: In the result developed in Theorem 1 , the value of K I , and hence the size of @(K,) , is selected to obtain an upper bound on 11 h(x) - W G x 1 1 . An alternative result that in general yields a different estimate of the region of attraction CR for the same value of v can be developed, as described in the following, by requiring a specific symmetry in the arrangement of the interconnection pattern without any limits on the size of the elements. Since both are only sufficient conditions for exponential stability with a specified degree, they are complementary and considerably enlarge the scope of the synthesis procedures for determining the interconnection parameters of the neural network whose stable equilibrium points are exponentially stable with this degree.

Theorem 3: For the system described by (9), let h( ) : 01" + 01" be factored in the form

(21) h(x) = [U@) - S(x) ]Px

514 lEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 2, NO. 5 , SEPTEMBER 1991

where P E @"'" is the symmetric and positive definite solution of the matrix equation

(22) - A T P - PA + Q = 0

for an arbitrarily selected Q E C R " " " such that Q = Q' and Q is positive definite, U( . ) : @" -+ @ ' I is an arbitrary skew-symmetric matrix, and S( . ) : @'I + @" " ' I is a symmetric matrix that satisfies the inequality

(23) for some y > 0 and for all x E a, where 63 C CR" 3 d3 = {x: IIxI/ < r } for some r > 0. Then, the equilibrium point at x = 0 of the system described by (9) is exponentially stable with degree y in 63.

Pro08 Let the conditions of the theorem hold. Since Spec(-A) C LHP, the solution P of (22) such that P = P T and P is positive definite exists and is unique. Con- sider the Lyapunov function candidate V(x) = x'px. Then, along the trajectories of (9),

x'[(Q - 2 y P ) + 2 P S ( x ) P ] x I 0

V(x) = x'[-A'P - P A ] x + x'Ph(x) + h T ( x ) P x

- 2 y x ' P ~ - xT [ (Q - 2 y P ) + 2 P S ( x ) P ] x

using (21) and ( 2 2 ) . Hence,

V(x) I -2yV(x) v x E os if (23) holds for all x E a, which establishes the exponential stability with degree y.

Remark 7: It must be carefully observed that the size of 63 depends not only on y but also on the selection of Q, U , and S matrices. Hence the above result is quite general. Specific selections of Q , U , and S can be made in different ways, which correspondingly give explicit values of y and the size of (B. For the sake of illustration, we shall identify in the following a specific construction of Q, U , and S matrices in order to determine the corresponding stability degree and the region of attraction.

Let us define the number

p = min (a,)X,,,[Z - i ( A - ' W G + GW'A-I)] (24)

where G is defined, as before, as G = diag [g;, gi, . . ,

r, be positive constants that define the region 63 C @ ' I E 63 = {x: 1 x, I I r , } , where r, is determined from the inequality

1

. , g; ] , g,' = (dg,(x , ) /dx,) Also, let r l , r2,

with E being a positive constant given by

X,,l(Y'+ Y') I / I WII

E = min (a;)

1 1 2 2

Y = -(I - y A - l ) - - A - ' W G . (27)

We then have the following criterion, which results from Theorem 3.

Corollary 2: Let p and 03 be defined by (24) and (25) and let M E @"", defined by

M = I - i (A-' WG + GW'A-') (28)

be positive definite. Then the equilibrium point at x = 0 of system (9) is exponentially stable with degree y < p in (B.

Proof: We will show this result by an application of Theorem 3 . Consider a selection of Q = 1. Then P = $ A - ' , from solving ( 2 2 ) . Now let the elements of U(x) : @" + @" ' ' I and S(x) : @ ' I + CR', ' ' I be selected in the form

and

wherep,,, i = 1, 2 , * . * , n , denote the diagonal elements of P . It is evident that U,,@) = -u,,(x) V i , j = 1, 2 ,

, n , and sY(x) = s,,(x) V i , j = 1 , 2 , . . . , n . . . . The result is proved if

x T I I - 2 y P + 2 P S ( x > P ] x 2 0 V X E 03. (30)

Now,

I - 2 7 8 + 2PS(x)P

= 1 - yA-1 - 4 (A- lWD(x) + D'(x)WA-I)

where

D(x): &'I -+ @ ' I 3 D(x)

Then, using (6), D(x) can be expressed as D(x) = G + E@), where E@): @" + @'"" 3 E(x) = diag [ e l ( x l ) , E&?), . . . , e,,(x,,>l, e,(x,) being bounded by I e,(x,) I < E V x E 63 for some E > 0. Thus,

I - 2 y P + 2PS(x)P = I - yA-l - [A-' W(G + E(x))

+ ( G + E(x))WA-' ]

= Y + Y' - [A- lWE(x) + E(x)WA-l ] . (31)

Now,

Y + Y' = 1 - $ ( A - ' WG + GW'A-I) - TA-'

= M - yA-'

is positive definite if h,,,(M) > yXM(A-') and X,,,(M) > 0 (which is true since M is positive definite). Hence, it is easily seen that for y < p defined by ( 2 4 ) , Y + Y' is positive definite. Now, using ( 3 1 ) , it can be shown that (30) is satisfied if

IIA-'WE(x) + E@) WA-'II I X,,,(Y + Y')


or

4 f M ( A - ' ) II WII 5 h,,V + yT> (32)

which is further ensured by the selection of E given in

Remark 8: It should be noted that X,,,(Y + Y T ) depends on the value of 7 and, in particular, that smaller values of 17 correspondingly increase X,,(Y + Y T ) . This in turn in- creases the value of E and hence the size of the region of attraction 63. This is consistent with a similar phenome- non observed from the stability result in Theorem 1, as noted in Remark 3.

Remark 9: It should be emphasized that the above corollary is just one illustration of the type of results that can be developed from an application of Theorem 2. Specific selections of Q = I and U and S given by (29) are made in this development. Alternative selections of Q, U , and S can similarly be made to obtain corresponding conditions on the network parameters for ensuring exponential stability of a certain degree and also for estimating the region of attraction of the equilibrium point under consideration.

Remark 10: In comparison with certain previously re- ported stability results, the presently developed conditions, viz. Theorem 1 and Theorem 3, are less restrictive and/or yield additional useful information on the stability properties of equilibrium points. Theorem 1 gives conditions which are similar to the analysis given by Guez et al. [22]; however, the degree of exponential stability and the region of attraction are explicitly identified in the present result. An alternative approach of treating the neural network as a large-scale system composed of in- terconnections of individual neurons has been used by Michel et al. [26]; it involves the construction of a test matrix from the bounds on the interconnection parameters and the nonlinear functions, which is further checked for being an 317. matrix. This, however, is equivalent to requiring the satisfaction of generalized diagonal domi- nance conditions [27], which are more restrictive than the present results.

(26), thus proving the result.

IV. EQUILIBRIUM CONFINEMENT Towards the objective of realizing an enlarged basin of

attraction for a stable equilibrium point and to permit re- liable recall of the corresponding memory, it is desirable to conduct the design of the associative memory network such that there exists only one equilibrium point in a specific region of the state space. For the sake of precision, let us define the following.

Definition 3: Given a fixed point z = [zl, z2, * , z,,] E (R"such that zi # 0 V i = 1 , 2, * , n , the quadrant containing z, denoted r((, z) C (R", is defined by

, n } . r(,$, z) = {.$:(g,; > 0 , vi = 1, 2,

The following result enunciates a condition for confining the various equilibria to different quadrants in the

sense that an equilibrium point in a particular quadrant is the unique equilibrium in that quadrant. Let us consider a specific equilibrium point U* of the neural network defined by (1) and the quadrant r(,$, U*). Define the matrix F E an x n by

whereu?, i = 1, 2, * , n, are the elements of U*.

We then have the following result.

Theorem 4: For a given equilibrium point U*, let A - WF be a 6 matrix. Then there can exist no other equilibria of the neural network defined by (1) in the quadrant ( 4 , U*).

Proof: We shall show this by contradiction. Let us assume that there exists another equilibrium point ii* in the quadrant r((, U*). Evidently, U* and ii* satisfy the equations

Au* = Wg(u*) + b

Aii* = Wg(li*) + b. (34)

The elements of ii*, viz. ci$, ci:, * * , U,, -* can have mag- nitudes greater than, equal to, or less than the corresponding elements of U*, viz. U?, u t , - - * , U:. Let us use the indices I, k, and m to identify these elements such that IciTl > Iu$I , Icitl < [ u t i , and&; = U:.

Since the nonlinear functions g,( ) are monotonically increasing and satisfy condition (iii) in (3), for each 1 one can find a,? E (R such that

lci,*l > [U",*/ > 1 ~ 7 1 and st($) = PtQT (35)

where PI = gt (uT) /uT. Similarly, for each k one can find a,* E (R such that

Icik*l > la,*[ > IuZI and g k ( c i t ) = P k E ; (36)

where Pk = g k ( u t ) / u f . Also, for each m, let a,* E (R such that

i; = U"; = and grn(d3 = Ptn (a ;> (37)

where P,, = g,,(u;)/u;. It is easily seen that U* = [a:, a,*, , U":]' E r(t, U*). A graphical depiction of the locations of 6; and U",* relative to u t , &:, U?, and is given in Fig. 1.

, &I, g(u*) = Fu*, and g(i i*) = FU*, (34) may be rewritten in the form

Arc* = WFu* + b

AU* = WFU* + b

Now observing that F = Diag [P I , P2, * *

(38)

(39)

or, equivalently,

A i * - WFU" = (A - WF)u*. For expressing ii* in terms of U*, we may note from the elementwise relations (35)-(37) the following: (i) for ele-

516 IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 2, NO. 5. SEPTEMBER 1991

//-

Fig. 1 . Relative locations of network equilibria. (a) Type 1 element, I uf I < 1i;l. (b) Typekelement, Iu$I > I l i f l .

ments with indices of type 1 for which 1 Li? I > 112: 1 , there (A - WF)(IZ" - U") = -AAa* (40) exist numbers d, > 0 such that Li; = ( 1 + d , ) $ ; ( i i ) for elements with indices of type k for which I Lit 1 < I L$ 1 , there exist numbers dL in the range - 1 < dL < 0 such

which 1Li;l = I C i I , one may write 12: = ( 1 + d, , , )12~ , where d, = 0.

where A = diag [ d , , d 2 , * * , d,,]. Now, if ( A - WF) is a 6 matrix, there exists a diagonal

that Cif = (1 + dk).@; and (iii) for indices of type m for matrix D,, = diag [d,,,, d,,z, . . . , d,,,,l, d,,, > 0, i = 1 , 2 , * , n , such that (see Property 1)

Using these relations, one may rewrite (39) as (12" - u * ) ~ D , ~ ( A - W F ) ( a * - U") > 0 (41) [A(Z + A ) - WF]a* = (A - WF)U*

for any U* and U* such that a* # U*. However, from or, equivalently, (40) 3


(a* - u * ) ~ D ~ ~ ( A - WF)(a* - U*)

= -(U* - U*)TD,AA-a*

r

where the expansion is obtained by separating the terms with indices of type 1 and type k . Now observing that the corresponding elements of U* and a* have the same sign since a* E r ( f , U*) and dl > 0 when 1 a: 1 > 1 U: 1 , while dL < 0 when < Iu: 1 , one can conclude that the R . H . S . of (42) is strictly negative for (a* - U*) # 0.

Thus we have a contradiction with (41), which implies that, if A - WF is a 6 matrix, there cannot exist any other equilibrium points of the network in the quadrant I'((,

Remark 11: At the outset, the principal condition for equilibrium confinement, viz. A - WF being a 6 matrix, appears quite distinct from the conditions for exponential stability given in Theorems 1 and 2. Nevertheless, there exist some interrelations between these conditions. Al- though such relations are difficult to establish in the most general cases, under special conditions on the neural network parameters, some useful relations can be explicitly identified. For illustration; consider the case where W has nonnegative diagonal elements. Then, it is possible to show that E , = A - ( W F + F W T ) / 2 positive definite 3 A - WFis a 6 matrix [27]. However, E, positive definite =) E 2 = A - (WG + GWT)/2 is positive definite since, by the properties of the nonlinear functions, each diagonal element of F is greater than the corresponding element of G (for a proof, see [9]). Hence, we have Spec(WG - A ) C LHP. Thus, under these conditions, any design procedure that ensures selection of A and W to make E, positive definite not only places the to-be-stored memory vectors in distinct quadrants but also ensures that these are stored as exponentially stable equilibrium points. It is possible to establish several other interrelations between these conditions, under different scenarios, which are useful in developing specific synthesis procedures. For the sake of conciseness, these details will not be given here.

The use of the present equilibrium characterization result can be better explained by the following example.

Example: Consider a three-neuron network with parameter values

U". 1

1 2.521739 0.0 -2.521739

w = [ 0.0 0.0 0.0

-2.521739 0.0 2.521739

-0.265

b = [ 0.6 ] -0.265

and A = I . Let the nonlinear sigmoidal functions be selected as g,(u, = 2/7r tan-' (O.lnu,), i = 1, 2, 3. It can be found by simulation that this network has two stable equilibrium points, v I = [-0.03, 0.6, -O.5lT and v2 = [-0.5, 0.6, -0.03IT in the quadrant I?(.$, z), z = [-1, 1, -1IT. Both matrices A - WF, and A - WF, corresponding to v , and v 2 have the same principal minors, given by 4.95 x lop1, 4.95 x lo-', and -4.59 x lop3. Hence, A - WF, (or A - WF,) is not a 6 matrix and by applying the present result, two equilibrium points in the quadrant r(.$, z) cannot be ruled out. When b is changed to b = [-0.865, 0.6, -O.865lT, one can again find by simulation that there is only one equilibrium point, v = [-0.865, 0.6, -0.865IT, in I'(.$, z) and this equilibrium point is a stable one. The corresponding A - WF matrix in this case can be determined to be a 6 matrix since its principal minors are 0.5075, 0.5075, and 0.0105088. This, from our result, ensures the uniqueness of v in the quadrant F ( f , z). The importance of the present result, which very simply gives assurance of the uniqueness of the equilibrium point within the region under consideration without extensive and laborious simulation, is clearly apparent.

V. SYNTHESIS PROCEDURE In this section, we shall present a systematic procedure

for synthesizing a neural network whose dynamics are described by (1) to function as an associative memory. The synthesis problem can be briefly stated as follows: Given a set of vectors { v I , v2, . , VI,}, v, E (R", i = 1, 2, . . . , m, obtain A = diag [ a l , a,, - , a,], W = [ ~ 1 1 1 1 , , = ~ , 2 , * , g,( * )IT, and b = [ b , , b,, * - , b,JT for the network such that v , , v2, . . . , v,, are stable equilibrium points of the designed network. To ensure reasonably large regions of attraction for these equilibrium points, we would also like to store these, if possible, as the unique equilibrium points in their respective quadrants of the state space.

Observing that a necessary condition for the vectors v,, i = 1 , 2 ,

g = [si( * 1, g2( * 1,

, m, to be the stable equilibria of (1) is

Auk = Wg(vk) + b, k = 1, 2, * * , m (43)

a reformulation of the synthesis problem into a problem of solving a set of simultaneous linear equations can be given as

A(vk - V I ) = W[g(vk) - g ( v , ) ] , k = 2, 3, * * , m

(44)

AN = W\k (45) where N E (Rn

9 VI,

3 \k = [g(v2) - g(v,), g(v3) - g(vl), * * , g(v,,) - g(vl)]. If A and g( * ), and hence \k, are fixed from other considerations, the synthesis problem reduces to solving (45) for the interconnection matrix W and then obtaining b by substitution in (43).

or, equivalently,

('" - I ) 3 N = [v , - V I , v3 - V I , *

- and \k E (R" ( I J J I )

518 IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 2. NO. 5 . SEPTEMBER 1991

It should be emphasized here that a number of prior works [4], [7]-[9], [21], [26], [31]-[38] have addressed the associative memory synthesis problem following this approach. While a majority of these studies employ methods that attempt to obtain a least-squares solution of ( 4 3 , the methods of [8] and [9] employ a master neural network that forces a solution to (45) by optimizing an appropriately posed objective function. However, it should be noted that (43) is a condition to be satisfied by all equilibrium points, both stable and unstable. Thus, although a solution obtained as above guarantees that u L , k = 1 , 2, . . . , m, turn out to be the equilibrium points of the designed network, a separate check on the stability status of each equilibrium needs to be made after the design and there is a possibility that some of the to-be-stored memories come out as unstable equilibrium points. This problem can be overcome by tailoring a design process that includes the use of an appropriate stability condition for fixing the design parameters, as we shall present now. Another important point is that our procedure does not begin with an arbitrary selection of A and g( ); rather, precise considerations will be spelled out for an organized selection of these parameters.

The principal conditions to be satisfied for establishing the exponential stability of a given equilibrium point U* as given in the two stability results in Section I11 are

(i) Spec(A - W G ) C LHP or, alternatively, (ii) M = I - + ( A - ' WG + G W T A - ' ) is positive defi-

nite, where G = diag [g;, gi, * * * , g;J ]> 81 =

Note that a sufficient condition for ensuring both of the above (see Property 2 in Section 11) is

(dgi(ui) /'dui) I ui = i t : .

IIA-'WGII < 1 (46) and for reasons of simplicity we shall employ this in formulating the design procedure. Also, for the sake of explicitly identifying the considerations that guide the selection of the various parameters, we will separate the question of selecting Wand b from that of selecting A and g ( 1.

A. Selection of W a n d b

For the sake of simplicity, let us assume that A and g( . ) are already selected. Then a simple solution to the set of equations in (45) can be obtained using generalized inverses [28] as

(47) where W is an estimate of W, \k + is the generalized inverse of *, and Y is an arbitrary n X n matrix. When Rank (\E) = n, I - WP + = 0 and hence I@ = W will be uniquely determined. However, in a majority of applications for the synthesis of associative memories, the number of to-be-stored memories, rn, will be less than the di- mension of the state space, tz, and hence Rank (9) < n . This factor contributes a degree of freedom in the selec-

W = AN*+ + Y(I - \E*+)

tion of Y , which has been exploited in the design procedure proposed by Li et al. [7]. As will become evident from the subsequent development, obtaining a minimum value for 1 1 is desirable in the interests of satisfying the stability requirements; hence we shall proceed with the selection of Y = 0 which yields [28] the minimum norm solution of (45).

An efficient procedure for computing the generalized inverse \k + involves the use of singular value decomposition of \E as

* = UCVT (48)

are unitary matrices (i.e., U T = U - ' and VT = V-I) and C E @ ' l x " ' i - ' )

has the form

where e @ J J x '2 and I/ e @(' ' I ~ 1 ) x ( '11 - 1 )

where C , E arxJ-3 C l = diag [al, az, . . . , a,.], r I n being the rank of 9. 9 + is then obtained as

*+ = V C + U T (49)

where C + e ~ I ) " has the form

With Y = 0 and \Ir + given by (49), the estimate of Wcan be obtained as

W = ANVC+UT. (50) Once kf' is determined, b can be readily obtained from (43) as

b = AV1 - Wg(vl) . (5 1 )

A few words on the accuracy of this solution are useful. For the case when r < min (n , rn - 1 ) , the set of equations (45) admits an exact solution. Also, for the case when r = min (n , rn - I ) , an exact solution is possible if rn - 1 5 n. However, in the other case, viz. r = min ( n , m - 1) and rn - 1 > n , the solution, in general, will be inexact. Inexact solutions can also result when (45) constitutes an inconsistent set; the occurrence of this sit- uation in practical synthesis applications is rather rare. In the latter cases leading to inexact solutions, the degree of approximation can be determined by computing the resid- uals as follows. Let w, e @" denote the ith row of W E @ ' I ' I and let cJ E aJu - I denote the jth column of NTA E

. Then, from (45) we have @ (n1 - I ) X I 1

N ~ A = q T w T (52) or, equivalently, by an expansion in terms of columns,

cJ = q T w J , j = 1, 2, . . . , n

w, = ( * T ) + C , .

Now, from (50), a least-squares solution for w, results in the residual terms (for details, one may refer to [28])

IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 2, NO. 5 , SEPTEMBER 1991 519

11, - I

(W - I@jT(w - I@) = RTJ = ( v T C ; ) ~ i = r + I

where U ; E (R”’ I denotes the ith column of V. The total error can then be computed as

I1 111 - 1

11 W - PII; = c c (V:cJ)2 (53) j=l i = r + l

where 11 I I F denotes the Frobenius norm. Evidently, when the R.H.S. of (53) is zero, the solution given by (50) is exact.

B. Selection of A and g( )

While (50) gives the selection of W which ensures that V I , v2, - * * , v,,l are the equilibrium points of the network, a selection of A and g( * ) can now be attempted to ensuring that these vectors are stored as stable equilibrium points. Observing that since Vand U are unitary matrices,

11 WII = IIANVCfUTll

(54)

where o,,, is the largest singular value of N and (T, is the smallest diagonal element of C l , one may conclude from (46) that if A and g( ) are selected such that

(55)

for a G computed at an equilibrium point U* = v,, then the corresponding memory v, will be stored as an exponentially stable equilibrium point. Equation (54) can be simplified further by noting that g;, the jth diagonal element of G, specifies the slope of the nonlinear function g, ( ) at the jth element of U*. From the specified to-be- stored memories v I , v2, * , v,,,, a bound for 11 GI1 can hence be given as

/

where vkJ E 63 is thejth element of vk. Furthermore, since A is diagonal and 11 A 11 1 1 A-’ 11 is the condition number of A, the L.H.S. of (55) can be minimized with the selection A = cl, where I is an identity matrix of size n X n and c > 0.

From the above discussion, it is evident that with A = cl, a selection of g( ) that satisfies

(57)

would in turn satisfy (55), and hence would ensure that each of the given vectors vi , i = 1, 2 , * , m , would be stored as an exponentially stable equilibrium point by determining appropriate W and b as discussed in the pre- vious section.

Inequality (57) points the way to an intelligent selection of the nonlinear functions in g( ). Since (T,,,, is essentially

fixed from the problem specification (i.e., the vectors de- sired to be stored), it is of interest to ensure smaller values for r. One may, however, observe from (56) that smaller values of f imply that the equilibrium points are to be stored in regions where the nonlinear functions have rel- atively smaller slopes, i.e., closer to the saturation region. Consequently, selection of sigmoidal functions such as g,(u,) = tanh ( Xu,), with a large parameter X, is desirable in the interests of satisfying (57), which in turn guarantees the stability of the equilibrium point in question. Indeed, the nonlinear function can be appropriately tai- lored to the problem specified according to the above considerations. It must be emphasized that this is one of the important features of the present design; earlier synthesis procedures have mostly ignored this possible freedom by arbitrarily assuming a nonlinear function a priori and at- tempting to answer the problem of storing a given set of vectors by the selection of only W (which is further constrained to be symmetric in many designs) and b. The ex- panded scope of the present solution is clearly evident.

The selection of A = c l in the present design is made in an attempt to satisfy inequality (55), which guarantees stability of the stored vectors. It must be noted that this selection is also beneficial in confining the equilibrium point in question to its respective quadrant such that a spurious stable equilibrium point will not be introduced within the same quadrant. This follows from the observation that a sufficient condition for A - WF to be a 6 matrix is 11 A-’ WF I/ < 1, (see Property 2 in Section 11), which further is ensured by requiring (as before)

C. An Algorithm for Synthesis and an Example

In this subsection, we shall summarize the discussion given above by precisely identifying the steps followed in the synthesis procedure. A numerical example will be given to illustrate the applicability of the present procedure.

Given vectors { v I , v2, , v,,,}, v, E @”, i = 1 , 2 , . m: . . .

Step 1: Check if for a n y j # i , v/ E F((, v,). If yes, it may not be possible to store them efficiently since v, and vl are in the same quadrant. Step 2: Select A = I. Also, make an initial selection of the nonlinear functions g,( e ) , i = 1, 2 , , n , that satisfy the conditions in (3). Step 3: Compute N = [v2 - v l , v3 - v l , * * 5 VI, - VI13 9 = [g(v2) - g(vJ, g(v3) - g(v,>, * * 9 g(v,,,) - g(v,)]. Also, compute’ the largest singular value (T~, of N and the singular value decomposition of 9:

‘For efficient procedures for computing the singular values and the singular value decomposition of a matrix, one may refer to [27] or [28].

520 IEEE TRANSACTIONS ON NEURAL NETWORKS. VOL. 2 , NO. 5, SEPTEMBER 1991

* = U C V T

Step 4: Find { = maxk, { g; ( v k J ) } , where vk/ is the j t h element of vk. Check if ((T~,,/(T,) { < 1. If yes, go to Step 5. If no, go to Step 2 and modify g,( ) such that { has a smaller value. Step 5: Compute W = NVC + U T where

and b = vl - Wg(vI). Step 6: Check if uTcJ = 0 Vi = r + 1, r + 2, * * * , m - 1 a n d j = 1, 2, . . * , n , where U , is the ith column of I/ and cl is the jth row of N . If yes, the vectors v I , v2, , v,, are stored exactly as the stable equilibrium points of the neural network. If not, the storage is ap- proximate. Step 7: Compute for each i = 1, 2, 9 m ,

and check if A - WFl is a 6 matrix. If yes, v, is stored as the unique equilibrium point in the quadrant I'((, v,).

Example: Five vectors in as, v I = [2.1, 2.0, -4.2, -5.1, -3.1IT, ~2 = [2.0, 2.8, 4.6, 2.9, 3.4IT, vi = [-2.3, -3.0, 2.8, 1.9, 4.75IT, ~4 = [4.0, 3.9, -2.4, 2.7, 4.OITand vs = [-3.1, -4.2, -1.9, 4.2, -2.1IT, are required to be stored as memory vectors.

Step I : The vectors are in distinct quadrants. Step 2: Let A = I and determine g,( ) through a few iterations as gl(u,) = 2 / ~ tan-' ( 1 . 8 7 5 ~ ~ , ) . This was conducted by initially starting with g,(u,) = 2 / ~ tan-' (TU,) and iteratively adjusting the argument of the tan- I

( ) function by executing steps 3 and 4 and returning to step 2 if the condition in step 4 is not satisfied. This will be made more clear by further discussion at the end of the procedure. Step 3: Rank (\k) = 4 with (T, = 0.6917 and (T,,, = 23.014 1. Step 4: { = maxk,J {g ; (vk , ) } = 2.97 X and the ratio (uSn/o,.) { = 0.988 < 1. Hence the given vectors can be stored as stable equilibria by this design. Step 5: Computed network parameters are

- I. 138881 I. 128695 - 1.009177 -0.4324422 1.393342

1.520285 1.524130 -0.5366862 -0.1872082 I . I33 193

0.4100377 0.4765162 3.626802 1.999820 -I 119399

0,1980552 0.2646499 9,73728902 - 02 5.224637 - 1.209066

Step 6: Residual error is zero and hence the storage is exact. Step 7: The matrices A - WF corresponding to v I , v2, and v3 belong to class 6 , guaranteeing their uniqueness in the respective quadrants. For a nonlinearity given by g,(u,) = 2 tan-' ( T U , ) / T ,

the value of (T, can be computed to be 0.68 and [ = 0.0546. This results in ((T,,,/(T,){ = 1.848 and we will not be able to guarantee the given vectors to be stable equilibrium points of the network. Hence, a nonlinearity with a higher gain as in step 2 was chosen.

VI. CONCLUSIONS The principal contributions of this paper are the various

results characterizing the equilibrium conditions of a very useful dynamical neural network model and a systematic synthesis procedure for designing associative memories with nonsymmetrical weight matrices. The equilibrium characterization study has focused on developing specific results for the exponential stability and instability of the equilibrium points of the network and on obtaining conditions that ensure that uniqueness of an equilibrium point in a specific region of the state space. While the equilibrium confinement result involves a very simple test, the stability results given here are more general than the existing results and specifically identify the degree of exponential stability and procedures for estimating the regions of attraction of stable equilibrium points. Deriving guidance from the obtained analytical results, a systematic synthesis algorithm has been developed for constructing a dynamical neural network that stores a given set of vectors as the stable equilibrium points. The associative memory synthesis given here considerably extends the scope of the memory storage by permitting the interconnection matrix to be nonsymmetric and in constructively exploiting the roles of the various network parameters (viz., slopes of the nonlinear functions in addition to the interconnection matrix and the bias inputs) to offer systematic tailoring procedures for an efficient synthesis. While the qualitative properties established in this paper are of particular interest in the associative memory design shown here, they are also of fundamental importance in other applications of dynamical neural networks as useful computational devices.

The present results can be used to seek further enhance- ments through further research in several directions. One challenging line of research is to utilize the exponential stability rates identified here in order to tailor gradient descent learning procedures with desirable convergence properties and to train a dynamical network to store a given set of vectors. Some preliminary results in this di- rection have been recently obtained [29].

L-0.3193269 -0.3637822 -0 3073451 -0.1863765 3.861677 1 REFERENCES

and = [ 1099343, -o.4965794, 6292369, [ I ] J . J . Hopfield, "Neural networks and physical systems with emergent collective computational abilities," Proc. Nar. Acad. Sci. U . S . . vol.

- 1.496394, 0.79241 16IT. 79, pp. 2554-2559. 1982.


[2] B. Kosko, “Adaptive bidirectional associative memories,” Appl. Opt., vol. 26, no. 23, pp. 4947-4960, 1987.

[3] F. J . Pineda, “Dynamics and architecture for neural computation,” J . Complexity, vol. 4, pp. 216-245, 1988.

[4] T. Kohonen, Self-organization and Associative Memory. Berlin, Germany: Springer-Verlag. 1984.

[SI J. A. Anderson. “Cognitive and psychological computation with neural networks,” IEEE Trans. Syst. Man, Cybern., vol. SMC-13,

161 S . Grossberg, Studies of Mind and Brain: Neural Principles of Learn- ing, Perception, Development, Cognition, and Motor Control. Bos- ton: Reidel, 1982.

171 J:H. Li, A. N. Michel and W. Porod,“Qualitative analysis and synthesis of a class of neural networks,” IEEE Trans. Circuits Sysr., vol.

[8] A. Lapedes and R . Farber, “Programming a massively parallel computation universal system: static behavior,” in Neural Networks fo r Computing, AIP Con$ Proc., 1986, pp. 283-298.

191 S . I . Sudharsanan and M. IC. Sundareshan, “Exponential stability and a systematic synthesis of a class of neural networks for quadratic min- imization,” Neural Networks, to be published.

[ 101 A. R . Stubberud and R. J. Thomas, “Associative recall using a contraction operator,” IEEE Trans. Circuits Syst., vol. 36, pp. 682-686, 1989.

[ I I ] J . J . Hopfield, “Neurons with graded response have collective computational properties like those of two-state neurons.” Proc. Nut. Acrid. Sci: U.S. , vol. 81, pp. 3088, 1984.

1121 M . A. Cohen and S . Grossberg, “Absolute stability of global pattern formation and parallel memory storage by competitive neural networks,” IEEE Trans. Syst., Man, Cybern., vol. SMC-13, pp. 815- 826, 1983.

1 131 S . I. Sudharsanan and M. K. Sundareshan, “Neural network computational algorithms for least squares estimation problems,” presented at the 1989 Int. Joint Conf. on Neural Networks, Washington, DC, June 1989.

[ 141 Y . S . Abu-Mostafa and J.-M. St. Jacques, “Information capacity of the Hopfield model,” IEEE Trans. Inform. Theory, vol. IT-31, pp.

[I51 R . J. McEliece, E. C. Posner, E. R. Rodemich, and S . S . Venkatesh, “The capacity of the Hopfield associative memory,” IEEE Trans. Inform. Theory, vol. IT-33, no. 4. pp. 461-482, 1987.

[I61 A. Dembo. “On the capacity of associative memories with linear threshold functions,” IEEE Trans. Inform. Theory, vol. 35, no. 4,

[ 171 A . Kuh and B. Dickinson, “Information capacity of associative memories,” IEEE Trans. Inform. Theory, vol. 35, no. I . pp. 59-68, 1989.

1181 M. Kam. R. Cheng and A. Guez, “Pattern retrieval and learning in nets of asynchronous binary threshold elements,” IEEE Trans. Cir- cuits Syst., vol. 36, pp. 353-364, Mar. 1989.

[ 191 E. Blum, “Mathematical aspects of outer-product asynchronous content-addressable memories,” Biol. Cybern., vol. 62, pp. 337-348, 1990.

1201 F. M. A . Salam and Y. Wang, “Some properties of dynamic feed-

pp. 799-815, 1983.

35, pp. 976-986, 1988.

461-464, 1985.

pp. 709-720, 1989.

back neural nets,” in Proc. 27fh IEEE Conf. Decision and Control (Austin, TX), 1988, pp. 337-342.

[21] J.-H. Li, A. N. Michel, and W. Porod, “Analysis and synthesis of a class of neural networks: variable structure systems with infinite gain,” IEEE Trans. Circuits Syst., vol. 36, pp. 713-731, May 1989.

[22] A. Guez, V. Protopopsecu, and J. Barhen, “On the stability, storage capacity and +sign of nonlinear continuous neural networks,” IEEE Trans. Syst. , Man, Cybern., vol. 18, no. 1, 1988.

[23] W. Hatin, Srability of Motion. [24] 0. Taussky, “A generalization of a theorm of Liapunov,” SIAM J .

Appl. Math., vol. 9, pp. 640-643, 1961. [25] N. N. Krasovskii, Stability of Motion (translated by J. L. Brenner).

Stanford, CA: Stanford University Press, 1963. [261 A. N. Michel, J. A. Farrel, and W. Porod, “Qualitative analysis of

neural networks,” IEEE Trans. Circuits Syst., vol. 36, pp. 229-243, 1989.

[27] M. Fiedler, Special Matrices and Their Applications in Numerical Mathematics. Prague, Czechoslovakia: Martinus Nijhoff Publish- ers, 1986.

[28] G. Golub and C. Van Loan, Matrix Computations. Baltimore, MD: Johns Hopkins University Press, 1989.

[29] S. I. Sudharsanan and M. K. Sundareshan, “Systematic design of associative memory networks: equilibrium confinement, exponential stability and gradient descent learning,” in Proc. In/. Joint Con5 Neural Networks (San Diego, CA), 1990.

[30] R. W. Brockett, Finite Dimensional Linear Systems. New York: Wiley, 1970.

[31] L. Personnaz, I. Guyon, and G. Dreyfus, “Information storage and retrieval in spin glass like neural networks,” J . Phys. Lett. . vol. 46.

[32] L. Personnaz, I. Guyon, and G. Dreyfus, “Collective computational properties of neural networks: New learning mechanisms,” Phys. Rev.

[33] L. Personnaz, I. Guyon, and G . Dreyfus, “High-order neural networks: Information storage without errors,” Europhys. Letters, vol.

[34] I . Guyon, L. Personnaz, J. P. Nadal, and 0. Dreyfus, “Storage and retrieval of complex sequences in neural networks,” Phys. Rev. A ,

[35] J. A. Farrell and A. N. Michel, “A synthesis procedure for Hop- field’s continuous-time content addressable memory,” IEEE Trans. Circuits Syst., vol. 37, pp. 877-884, 1990.

[36] A. N. Michel, J. A. Farrell, and H. F. Sun, “Analysis and synthesis techniques for Hopfield type synchronous discrete-time neural networks with application to associative memory,” IEEE Trans. Cir- cuits Syst., vol. 37, pp. 1356-1366, 1990.

[37] A. N. Michel and J. A. Farrell, “Associative memories via artificial neural networks,” IEEE Coizrrol Sysrems Magazine, vol. 10, pp. 6- 17, 1990.

[38] J. H. Li, A. N. Michel, and W. Porod, “Analysis and synthesis of a class of neural networks: Linear systems operating on a closed hy- percube,” IEEE Trans. Circuirs Syst., vol. 36, pp. 1405-1422, 1989.

New York: Springer-Verlag. 1968.

pp. L359-365, 1985.

A , vol. 34, pp. 4217-4228, 1986.

4, pp. 863-868, 1987.

vol. 38, pp. 6365-6372, 1988.

Equilibrium characterization of dynamical neural networks for synthesis of associative memories

Documents