## Abstract

In this work, we shall put forward a novel chaos memory retrieval model with a Chebyshev-type activation function as an artificial chaos neuron. According to certain numerical analyses of the present association model with autocorrelation connection matrix between neurons, the dependence of memory retrieval properties on the initial Hamming distance between the input pattern and a target pattern to be retrieved among the embedded patterns will be presented to examine the retrieval abilities, i.e. the memory capacity of the associative memory.

### Keywords

- Chebyshev Chaos neuron
- associative memory
- computer simulation
- numerical analysis methods
- memory retrieval

## 1. Introduction

Over the past quarter century, it has been extensively reported that there may exist inherently chaotic dynamics in the human electroencephalogram (EEG) in the variety of biological experiments [1, 2, 3, 4, 5, 6, 7, 8]. In addition from the viewpoints of the artificial neuron models including chaos neurons, many researchers have investigated the association memory models in terms of the various types of activation functions of artificial neurons [9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33] so as to improve their memory retrieval capabilities as reviewed in brief in the following section.

First, as an artificial neuron model, Tsuda reported a dynamic retrieval model as well as dynamic linking of associative memories and pointed out a crucial role of chaos in those dynamics [9, 10, 11]. In addition, Davis and Nara * et al*. put forward a new memory search model with a chaos control [12, 13]. Subsequently, several applications of the chaos neural networks with the sigmoidal, i.e. monotonous, activation function have been reported by Aihara

*. [14] and Nakamura and Nakagawa [15]. In practice, however, as was confirmed by Kasahara and Nakagawa [16], the chaotic dynamics in association process with such a sigmoidal, i.e. monotonous, activation function encounters an inevitable troublesome such that the complete association of the embedded patterns becomes considerably difficult if the loading rate, i.e.*et al

In contrast to the aforementioned monotonous chaos neuron model [14, 15, 16], the neurodynamics with a nonmonotonous activation function was investigated by Morita [19], in which he reported that the nonmonotonous mapping in a neurodynamics may possess a certain advantage of the critical loading rate,

From the earlier-noted aspects, in this work, we shall propose a novel chaos auto-association model with the Chebyshev-type activation function recently proposed [35] which may have an advantage to promote the memory capacity of the associative memory beyond the conventional models, proposed up to date, e.g. the sinusoidal activation function and the signum one [9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 27, 28, 29, 30, 31, 32, 33, 34].

In §2, a theoretical framework of the present association model will be described making use of the Chebyshev-type chaos activation function [35]. Therein, a chaos control related to the chaos simulated annealing as previously mentioned [21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35] will be introduced to transfer the system from a strongly chaotic initial state to a moderate one which may lead the associative model to a memory retrieval point. Furthermore, some computer simulation results for the memory retrieval characteristics will be given in §3, in which an apparent advantage of the present chaos associative model will be elucidated in comparison with the previously proposed sinusoidal activation function model [21, 22, 23, 24, 27, 28, 29, 30, 31, 32, 33] as well as the Associatron [17]. Finally, §4 is devoted to a few concluding remarks on the presently proposed chaos associative model as well as the future works to be investigated further.

## 2. Theory

Denoting the * i-th* component of the

*embedded, or memorized, vector as*r-th

where

Therefore, one may evaluate the memory retrieval performance depending on the loading rate * i-th* neuron,

where * i-th* neuron

here

In the present work, we shall focus our interests on the recently proposed Chebyshev-type activation function as follows [35]:

where

To review the chaotic property of the present chaotic neurons [35], let us present the fundamental properties of a single neuron in the following section. Several profiles of the activation function of Eq. (5) for several

as is confirmed in Figure 1. Herein, the subscripts

Then, in order to examine a chaotic dynamics of the present chaos neuron model, replacing * i-th* neuron [35]:

As an example, the chaotic behavior of the internal state

Therein, one may confirm the variation of the chaos strength, or the complexity of dynamics, depending on

which corresponds to an identity mapping such that

Through the updating rule of Eq.(7), the corresponding Lyapunov exponent

where

To derive Eq. (10), we have noticed the following derivations:

where _{,} and

respectively [35].

The dependence of the Lyapunov exponents

Therein, one may confirm that the aforementioned expression holds approximately in comparison with the numerical results with Eqs. (7) and (10). Hence, according to Eqs. (10) and (16), the following relation is considered to be approximately satisfied [35]:

Especially for a specific case of

which is found to be exactly same with that of the logistic mapping because of the topological conjugacy of the Chebyshev-type mapping as Eq. (5) with the conventional logistic one [35]. In the aforementioned work [35], the corresponding invariant measure was also investigated further in comparison with the sinusoidal and the monotonous activation functions [37].

From these results, the strength of chaos is found to be properly controlled, through the parameter

In the similar manner to the previous models [21, 22, 23, 24, 27, 28, 29, 30, 31, 32, 33, 34, 37, 38, 41], the control dynamics of the parameter

where

where

Updating

## 3. Computer simulation results

In this section, we shall show some numerical results derived from the aforementioned chaos auto-association model defined by Eqs. (1), (3), (5), (8), and (19).

For numerical simulations, the embedded vectors

where

and

where

The initial conditions for

where

setting as

For convenience, the ratio of the distorted components according to Eq. (25), or the Hamming distance

Hence, the directional cosine, i.e. the inner product,

It is well known that the succeeded memory retrieval becomes apparently troublesome if

Then, in similar manner to the previous works [21, 22, 23, 24, 27, 28, 29, 30, 31, 32, 33, 34, 41] for the associative memories, the overlaps

If one of the overlaps

and

as

Thus through the present numerical simulations for certain values of

Thus, the ability of the memory retrievals for the associative memory model may be evaluated in terms of the score of the succeeded memory retrievals rate to satisfy the conditions and Eqs. (30) and (31) for a certain fixed value of

In the following practical simulations, the aforementioned parameters,

In Figure 5a–e, the retrieval characteristics, i.e. the dependence of the succeeded association rate defined by Eq. (32),

In Figure 7, let us present some examples of time dependences of the internal states

and

respectively; here

Therefore, the chaos dynamics is expected to be properly controlled by the parameter

The resultant retrieval characteristics for Eqs. (34) (sinusoidal activation function) and (35) (signum activation function) are given in Figures 8 and 9, respectively. First, in Figure 8 for the sinusoidal activation function as Eq. (34), according to these

Then, according to

In Figure 6, let us present the dependence of the memory capacities * .* the Chebyshev-type activation function as Eq. (5) [35], the sinusoidal activation function as Eq. (34) [21, 22, 23, 26, 27, 28, 29, 30, 31, 32, 33, 34], and the Associatron as Eq. (35) [17] for comparison. From these results, one may find the advantage of the presently proposed Chebyshev-type activation function as Eq. (5) beyond such previous models as Eqs. (34) [21, 22, 23, 26, 27, 28, 29, 30, 31, 32, 33, 34] and (35) [17] as well as the other association models with the autocorrelation connection matrix

## 4. Concluding remarks

In this chapter, we have proposed a chaos associative memory model with the Chebyshev-type activation function [35] instead of the conventional ones including such as the sinusoidal one [21, 22, 23, 24, 27, 28, 29, 30, 31, 32, 33, 34] as well as the signum one [17]. From the present computer simulation results, we have confirmed the apparent advantage of the present chaos associative model beyond the aforementioned conventional models from the viewpoint of the memory capacities related to the robustness for the included noise, i.e. the distorted components in an input vector for the memory retrieval. In practice for the extreme case such as

Several investigations on an optimization scheme of the included model parameters such as

Moreover, it seems to be also interesting to investigate the orthogonal learning model [42, 43] as well as the continuous time model [20]. It seems to be worthwhile to introduce the present Chebyshev-type activation function to the chaotic leaning problems as previously reported [38, 39, 40] as well as the design of the classifier applied to the sensibility measurement scheme [44] in connection with an application to the communication technology, e.g. a silent speech interface as seen in the human-brain interfaces. In addition, a chaos learning model with the present Chebyshev-type activation function is to be investigated in comparison with the other chaos learning models with the sinusoidal activation functions [38, 39, 40].

## Acknowledgments

The present work was supported in part by START program by Japan Science and Technology Agency in 2014–2016. The present author would like to greatly appreciate for the Grants-in-Aid for Scientific Research of MEXT, No.21300081, No.23650109, and No.24300084. The present author also would like to sincerely express his gratitude to his academic staffs and also to his family for their continuously heartwarming supports.

## References

- 1.
Babloyanz JM, Nicolis C. Evidence of chaotic dynamics of brain activity during the sleep cycle. Physics Letters. 1985; 111A :152-156 - 2.
Siska J. On some problems encountered in the estimation of the correlation dimension of the EEG. Physics Letters. 1986; A118 :63-66 - 3.
Mayer-Kress G, Layne SP. Dimensionality of the human electroencephalogram. Annals of the New York Academy of Sciences. 1987; 504 :62-87 - 4.
Meyer-Kress G, Yates FE, Benton L, Keidel M, Tirsch W, Poppl SJ, et al. Dimensional analysis of nonlinear oscillations in brain, heart, and muscle. Mathematical Biosciences. 1988; 90 :155-182 - 5.
Nan X, Jinghua X. The fractal dimension of EEG as a physical measure of conscious human brain activities. Bulletin of Mathematical Biology. 1988; 50 :559-565 - 6.
Nakagawa M. On the chaos and fractal properties in EEG data. IEICE Transactions on Fundamentals. 1995; 78 :161-168 - 7.
Nakagawa M. Chaos and fractal properties in EEG data. Electronic Communication Japan. 1995; 78 :27-36 - 8.
Duke DW, Pritchard WS. Measuring Chaos in the Human Brain Proceedings of the Conference. World Scientific Inc; 1991 - 9.
Tsuda I. Chaotic neural networks and thesaurus, neurocomputers and attention. In: Holden AV, Kryukov VI, editors. Poc. of the International conference on Neurocomputers and attention. Manchester University Press; 1991. pp. 405-424 - 10.
Tsuda. Dynamic link of memory—Chaotic memory map in non-equilibrium neural networks. Neural Networks. 1992; 5 :313-326 - 11.
Tsuda E, Shimizu H. Memory dynamics in asynchronous neural networks. Progress in Theoretical Physics. 1987; 78 :51-71 - 12.
Davis P. Application of optical Chaos to temporal pattern search in a nonlinear optical resonator. Japanese Journal of Applied Physics. 1990; 29 (7A):L1238 - 13.
Nara S, Davis P, Totsuji H. Memory search using complex dynamics in a recurrent neural network model. Neural Networks. 1993; 6 :963-973 - 14.
Aihara T, Toyoda M. Chaotic neural networks. Physics Letters. 1990; A144 :333-340 - 15.
Nakamura K, Nakagawa M. On the associative model with parameter controlled Chaos neurons. Journal of the Physical Society of Japan. 1993; 62 :2942-2955 - 16.
Kasahara T, Nakagawa M. Parameter-controlled chaos neural networks. Electronics and Communications in Japan, Part III Fundamentals. 1995; 78 :1-10 - 17.
Nakano K. Associatron-A Model of Associative Memory. In: IEEE Transactions on Systems, Man, and Cybernetics, SMC-2. 1972. pp. 380-388 - 18.
Nozawa H. A neural network model as a globally coupled map and applications based on chaos. Chaos. 1992; 2 :377-386 - 19.
Morita. Associative memory with nonmonotone dynamics. Neural Networks. 1993; 6 :115-126 - 20.
Fukai T. Self-consistent signal-to-noise analysis of the statistical behavior of analog neural networks and enhancement of the storage capacity. Physics Review. 1993; E48 :867-897 - 21.
Nakagawa M. A model of Chaos neuro-dynamics with a periodic activation function, in proc . In: 1994 International Conference on Neural Information Processing (ICONIP'94). Seoul; 1994. pp. 609-613 - 22.
Nakagawa M. A Model of Chaos Neuro-Dynamics. In: Proc. 1994 International Conference on Dynamical Systems and Chaos (ICDC'94). Tokyo; 1995. pp. 603-607 - 23.
Nakagawa M. An artificial neuron model with a periodic activation function. Journal of Physics Society Japan. 1994; 64 :1023-1031 - 24.
Kasahara T, Nakagawa M. A study of association model with periodic Chaos neurons. Journal of the Physical Society of Japan. 1995; 64 :4964-4977 - 25.
Nakagawa M. A synergetic neural network. IEICE Transactions on Fundamentals. 1995; E78-A :412-423 - 26.
Nakagawa M. Chaos synergetic neural network. Journal of the Physical Society of Japan. 1995; 64 :3112-3119 - 27.
Nakagawa M. Periodic Chaos Neural Networks. In: Proc. 1995 International Conference on Neural Networks (ICNN'95). Australia; 1995. pp. 3028-3033 - 28.
Nakagawa M. A novel Chaos neuron model with a periodic mapping. Journal of the Physical Society of Japan. 1997; 66 :263-267 - 29.
Nakagawa M. A parameter controlled Chaos neural network. Journal of the Physical Society of Japan. 1996; 65 :1859-1867 - 30.
Nakagawa M. An Autonomously Controlled Chaos Neural Network. In: Proc. 1996 International Conference on Neural Networks (ICNN'96). Washington DC. pp. 862-867 - 31.
Tanaka T, Nakagawa M. A Chaos association model with a time-dependent periodic activation function. IEICE Transactions on Fundamentals. J79-A :1826 - 32.
Nakagawa M. A super memory retrieval with Chaos associative model. Journal of the Physical Society of Japan. 1999; 68 :2457-2465 - 33.
Nakagawa M. A Chaos associative model with a sinusoidal activation function. Chaos, Soliton and Fractals. 1999; 10 :1437-1452 - 34.
Nakagawa M. Chaos and Fractals in Engineering. World Scientific Inc.; 1999 - 35.
Nakagawa M. On the Chaos neuron models with Chebyshev type activation functions. Journal of the Physical Society of Japan. 2021; 90 :014001 - 36.
Courant R, Hilbert D. Methoden der Mathematischen Physik. Berlin: Springer; 1962 - 37.
Nakagawa M. On the Chaos neurons and invariant measure from a symmetry consideration. IEICE. 2003; 86 (2):103-108 - 38.
Okada T, Nakagawa M. A study of Back propagation learning with periodic Chaos neurons. Chaos Sollitons and Fractals. 1999; 10 :77-97 - 39.
Onozuka Y, Nakagawa M. IEICE Transactions on Fundamentals. 2001; J84-A :33-41 - 40.
Maeda H, Nakagawa M. A Back Propagation Model with Periodic Chaos Neurons. In: Proc 1996 International conference on neural information processing (ICONIP’96). Hong Kong; 1996. pp. 567-571 - 41.
Nakagawa M. Statistical properties of Chaos associative memory. Journal of Physics Society of Japan. 2002; 71 :2316-2325 - 42.
Personnaz L, Guyon I, Dreyfus G. Information storage and retrieval in spin-glass like neural networks. Journal of Physics Letters. 1985; 46 :L359-L365 - 43.
Kanter I, Sompolinski H. Associative recall of memory without errors. Physical Review A. 1987; 35 :380-392 - 44.
Nakagawa M. Chaos and Fractals Sensibility Information Engineering. Nikkan-Kogyo Shinbun Inc; 2010