Exponential Transients in Continuous-Time Symmetric Hopfield Nets

We establish a fundamental result in the theory of continuous-time neural computation, by showing that so called continuoustime symmetric Hopfield nets, whose asymptotic convergence is always guaranteed by the existence of a Liapunov function may, in the worst case, possess a transient period that is exponential in the network size. The result stands in contrast to e.g. the use of such network models in combinatorial optimization applications.

Because of the apparent simplicity of symmetric Hopfield network dynamics, one might also assume that they always converge rapiãly-an assumption that seems to often be implicitly made in e.g. discussing the potentiat of sucñ networks as "fast analog solvers" for optimization problemJ. contrary to this expectation, we shall in this paper construcü for every r¿ a Hopfield network cn'of.6n+t units with a s¡'rnmetric coupling weight matrix a¡rd a saturated-[nãar (,activa-tion function" that simulates an (n + l)-bit binary counter a¡rd thus produces a sequence of.2' -I well-controlled oscillations before it converges. Bãsides suggesting some caution in applying neural networks to optimization problems, thîs result provides to our knowledge the first known example of a continuous-time, Liapunov-function controlled dynamical system with an exponential transient period. such an exponential-transient oscillator can also bsused to support a general r\rring machine simulation by symmetric Hopfield networks [g].
In terms of bit representations, our convergence time lower bound ca,n be compared to a general upper bound for discrete Hopfietd networks [10]. It turns out that the continuous-time system c, converges iater than arry dìsc.ete sy*metric Hopfield network of the same description length, assuming that the time interval between two subsequent discrete updates corresponds tã a continuous time unit. This suggests that continuous-time analog models of computation may be worth investigating mgre for their gains in represãntational efficiãncy than for their (theoretical) capability for arbitrary-precision real number compuiation þ].
A Hopfield network C Cnwithm 6r¿* 1 neurons will now be constructed which simulates an (n + l)-bit binary counter, a¡rd thus has a transient period that is exponential in the parameter m. The original idea for a corresponding discretetime counter network stems from [3]. In our simulation , the binary states of the counter will be represented by excitations of the corresponding real-valued units in C that are either above the upper saturation threshold of 1 or below the lower saturation threshold of0 for the activation function ø. For brevit¡ we shall simply say that a unit p is saturated, at 0 or 1 at time ú if its excitation satisfies (o(¿) S 0 or {o(ú) ) 1, respectively. We also say that p is unsaturated when 0 < {o(ú) < 1. (Note that we use the encitations, not the actual staúes of the units to represent binary values.) The following theorem summarizes the result: Theorem L. For euery integer n ) 0 there enists a continuous-t'ime symmetric Hopf,eld, net C with m = 6n * \ neurons whose global state transition from saturation at 0 to saturat'ion at 1 requires cont'inuous time Q(2n'/a f e), for any 0 ( e < 0.05 such that 2 /2 < e2L/u. Th'is conuergence bound translates to zakjuI)) time units, where M represents the number of bits that are sufficient for encoding the weights 'in C and g(M) is an arbitrarg cont'inuous function such Prool. (Sketch.) The construction of symmetric Hopfield nel C = Cn with m: 6n * 1 units and zero initial state y(0) : 0simulating an (n + l)-bit binary counter will be describe<l by induction on r¿. The operation of the network will first be discussed intuitively, a¡rd its correctness will then be formally verified. The induction sta¡ts with a network C¡ containing only a single unit c¡, with bias tr(O, co) = e and feedback coupling u(co,co) = L +e. This represents the first counter bit of "order 0". Because of its positive feedback the state of ca graduaily grows from initial 0 towards 1. Eventually c0 saturates at 1, at which point we say that the unit co becomes actiueor 7Íres. This trick ofgradual transition from 0 to 1 (see Lemma 3 below) is used repeatedly throughout our construction of C.
For the induction step depicted in Figure 1 (the edges in this graph drawn without an originating unit correspond to the biases), assume that an "order (k -1)" counter network C n-t (1 < h < n) has been constructed' containing the first k counter units cot... tc¡-1, together with auxiliary units a,¿,r¿,b¿,d,¿,21 ((. : Ir. . . ,lc-!), for a total of m¡, -6k -5 units. Then the next counter unit c¡.
is connected to all the ruft units p € Cx-t via unit weights which, together with c¡'s bias, make c¡ to fire shortly after all these units are active, i.e. when the simulated counting from 0 to 2k -L has been accomplished. In addition, unit c¡ is connected to a sequence of five auxiliary units Qk¡frk¡b¡a,d,¡,2¡, which are being, one by one, activated after c¿ fires (Lemma 3). The purpose of the auxiliary units a¡, bn,dx is only to slow down the continuous-time state flow. The unit rr is used to reset all the lower-order units in Cn-t back to values nea,r 0 a,fter c/c fires (Lemma 2.2b). To achieve this effect, r¡ is linked with each p € Cn-t via a large negative weight u(rn,p) --lu("n,p) *DqeCn_rio(q,e)>ou(e,P)l that exceeds the mutual positive influence of units in Cnt U {c¿}. The value of parameter Vn :L-Ðpe¿^_ru(rxrp) is determined so that the state of rr is independent of the states of p e. C¡"-1. Finall¡ unií z¡, balances the negative influence of ør on C*-r so thai the first & counter bits can again count from 0 to 2k -I but now with c¡ being active. This is achieved by exact weights u(zx,p) : -u(u*,P) -1 for p e Cn¡ in which the -1 compensates for u(c¡,p): 1. Clearly, units p e Cx-t cannot reversely afrect z¡, since their maximal contribution DpeC^_ru(p,z¡r) :  (1) and Lemma 1. Furthermore, the transfer of the activity in C from a unit to a subsequent one, when all the incident units are saturated, will be analyzed explicitly and its duration time will be calculated in Lemma 3. (But note that the analysis for cs at f : 0 slightly differs.) The result is also generalized to the case when some of the incident units may become unsaturated.
Similarl¡ the stronger defect bounds in Lemma 3.2 are met since time 2úr is guaranteed before unit rfr unsaturates. The lower bound AQn le) : Q()m/6 le) on the total simulation time follows immediately from the previous time analysis.
From the proof of Lemma 1, the maximum integer weight parameter in C is of order 2o(^) . This corresponds to O(rn) bits per weight that is repeated O(*') times, and thus yields at most O(m") bits in the representation. In addition, the biases and feedbacks of the rn units include fraction e (or e/3), and taking this into account requires A(mbg(Lle)) additional bits, say at least rcrnlog(l/e)bits for some constant rc ) 0. By choosing u -2-r(nt)/("-) in which / is a continuous increasing function whose inverse is defined as /-1(¿¿) : t"r/50.1), where g satisfies SQù = njt'/3) (implying l(m): A@\) and s(p) = o(tù, it follows that M : O(l(*)), especially M 2 l(m) from M 2 nmlog(Ile). The convergence time nQ*/6 le) can be translated b A(zr@)/(nm)+m/6) -2aff@)/m) which can be rewritten as 2e(M/f-l(u\ -2a@Qø)) since Í(m): A(M) from M : O(l@)) and f -t (M) ) rn from M 2 f (m).This completes the proof of the theorem. tr

A Simulation Example
A computer program HCOUNT has been created to automate the construction from Theorem 1. For input n ) 0, the program generates system (1) describing the Hopfield net dynamics in the form of a FORiIRAN subroutine corresponding to the (rz + 1)-bit binary counter to be simulated. This FORTRAN procedure is then presented to a solver from the NAG library that provides a numerical solution for the system. For example, implementing a 4-bit counter on the HCOUNT generator results in a continuous-time symmetric Hopfield net C3 with 19 variables. Figure 2 shows the state evolution of counter units c0, c1 , c2, ca for a period of23 -1 :7 simulated discrete steps confirming the correctness ofthe construction. A parameter value of e = 0.1 was used in this numerical simulation, showing that the theoretical estimate of e in Theorem 1 is actually quite conservative.