A Chaos Auto-Associative Model with Chebyshev Activation Function

Abstract

In this work, we shall put forward a novel chaos memory retrieval model with a Chebyshev-type activation function as an artificial chaos neuron. According to certain numerical analyses of the present association model with autocorrelation connection matrix between neurons, the dependence of memory retrieval properties on the initial Hamming distance between the input pattern and a target pattern to be retrieved among the embedded patterns will be presented to examine the retrieval abilities, i.e. the memory capacity of the associative memory.

1. Introduction

Over the past quarter century, it has been extensively reported that there may exist inherently chaotic dynamics in the human electroencephalogram (EEG) in the variety of biological experiments [1, 2, 3, 4, 5, 6, 7, 8]. In addition from the viewpoints of the artificial neuron models including chaos neurons, many researchers have investigated the association memory models in terms of the various types of activation functions of artificial neurons [9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33] so as to improve their memory retrieval capabilities as reviewed in brief in the following section.

First, as an artificial neuron model, Tsuda reported a dynamic retrieval model as well as dynamic linking of associative memories and pointed out a crucial role of chaos in those dynamics [9, 10, 11]. In addition, Davis and Nara et al. put forward a new memory search model with a chaos control [12, 13]. Subsequently, several applications of the chaos neural networks with the sigmoidal, i.e. monotonous, activation function have been reported by Aihara et al. [14] and Nakamura and Nakagawa [15]. In practice, however, as was confirmed by Kasahara and Nakagawa [16], the chaotic dynamics in association process with such a sigmoidal, i.e. monotonous, activation function encounters an inevitable troublesome such that the complete association of the embedded patterns becomes considerably difficult if the loading rate, i.e. $\alpha =L/N$, in which $L$ and $N$ are the number of embedded patterns and the neurons, respectively, increases beyond a certain critical values ${\alpha }_c$, i.e. ${\alpha }_c$ ∼ 0.05, for the autocorrelation learning model.

In contrast to the aforementioned monotonous chaos neuron model [14, 15, 16], the neurodynamics with a nonmonotonous activation function was investigated by Morita [19], in which he reported that the nonmonotonous mapping in a neurodynamics may possess a certain advantage of the critical loading rate, ${\alpha }_c$∼0.27, superior than ${\alpha }_c$∼0.15 for the conventional association model as the Associatron with such a signum activation function [17]. Later, Shiino and Fukai examined the memory capacity for a somewhat simplified step-like nonmonotonous activation function with the continuous time model, instead of the aforementioned discrete time ones, in an appropriate statistical manner, and concluded that the complete association could be achieved up to the critical loading rate such as ∼0.42 [20]. Subsequently, several models of nonmonotonous activation functions have been proposed including a sinusoidal activation function to control chaos dynamics [21, 22, 23, 24, 25, 26, 27, 28, 29]. In practice, the present author reported a chaos neuron model with a periodic activation function in order to construct a chaos association model with a discrete time as well as the orthogonal learning scheme [21, 22, 23, 24]. Therein, the memory capacity was found to be increased up to ${\alpha }_c$ ∼0.4, even in the search mode without any key information for the input vector, beyond the previously proposed monotonous dynamic models with the discrete time [14, 15, 16]. As the further possible applications, such a chaotic dynamics was involved in the synergetic neural network [25, 26]. The earlier-noted chaos neuron models with such sinusoidal periodic activation functions, which are referred as the sinusoidal chaos neuron model in the present work, have been extensively applied to the chaos association models to mainly evaluate their memory capacities [21, 22, 23, 24, 27, 28, 29, 30, 31, 32, 33, 34].

From the earlier-noted aspects, in this work, we shall propose a novel chaos auto-association model with the Chebyshev-type activation function recently proposed [35] which may have an advantage to promote the memory capacity of the associative memory beyond the conventional models, proposed up to date, e.g. the sinusoidal activation function and the signum one [9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 27, 28, 29, 30, 31, 32, 33, 34].

In §2, a theoretical framework of the present association model will be described making use of the Chebyshev-type chaos activation function [35]. Therein, a chaos control related to the chaos simulated annealing as previously mentioned [21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35] will be introduced to transfer the system from a strongly chaotic initial state to a moderate one which may lead the associative model to a memory retrieval point. Furthermore, some computer simulation results for the memory retrieval characteristics will be given in §3, in which an apparent advantage of the present chaos associative model will be elucidated in comparison with the previously proposed sinusoidal activation function model [21, 22, 23, 24, 27, 28, 29, 30, 31, 32, 33] as well as the Associatron [17]. Finally, §4 is devoted to a few concluding remarks on the presently proposed chaos associative model as well as the future works to be investigated further.

2. Theory

Denoting the i-th component of the r-th embedded, or memorized, vector as $e_i^{\left(r\right)}$, the autocorrelation connection matrix can be defined as follows in accordance with the conventional autocorrelation learning model [9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19]:

$$w_{\mathit{ij}}=\frac{1}{N}\sum _{r=1}^L{e}_i^{\left(r\right)}{e}_j^{\left(r\right)},$$E1

where $N$ and $L$ are the numbers of the neurons and the embedded vectors, respectively. Then the loading rate, which corresponds to a ratio of the number of the embedded vectors to that of neurons, $\alpha$ is to be defined by:

$$\alpha =L/N.$$E2

Therefore, one may evaluate the memory retrieval performance depending on the loading rate $\alpha$ for an initial vector which may involve a certain noise in a certain vector among the embedded ones, as will be explained later in detail. Then the updating rule of the internal state of the i-th neuron, ${\sigma }_i\left(t\right)$, in the presently concerned discrete time model may be introduced by:

$${\sigma }_i\left(t+1\right)=\left(1-k_s\right){\sigma }_i\left(t\right)+k_s\sum _{j=1}^N{w}_{\mathit{ij}}{s}_j\left(t\right),$$E3

where $k_s\left(0<k_s\le 1\right)$ is considered to be a relaxation parameter of the updating process of the internal state ${\sigma }_i\left(t\right)$ [34], and the output of the i-th neuron $s_i\left(t\right)$ is to be related to ${\sigma }_i\left(t\right)$ in terms of

$$s_i\left(t\right)=f\left({\sigma }_i\left(t\right)\right);$$E4

here $f\left(•\right)$ is an activation function of the neuron [9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35], which is closely related to the dynamics of the neural networks [34] as has been reported in the conventional models with several nonmonotonous activation functions [9, 10, 11, 12, 13, 18, 19, 20] as well as the chaos associative models [21, 22, 23, 24, 27, 28, 29, 30, 31, 32, 33, 34] with periodic activation function.

In the present work, we shall focus our interests on the recently proposed Chebyshev-type activation function as follows [35]:

$$s_i\left(t\right)={tanh}^{-1}\left\{sin\left(\nu {sin}^{-1}\left(tanh{\sigma }_i\left(t\right)\right)\right)\right\},$$E5

where $\nu$ may be regarded as a chaos control parameter to be related to the strength of chaos as will be mentioned later [35]. If we ignore the operations of ${tanh}^{-1}\left(•\right)$ and $tanh\left(•\right)$ in Eq. (5), the present activation function is readily reduced to the original Chebyshev function of the second kind [36] as was previously argued in detail [35]. That is, the activation function as Eq. (5) is regarded as a modified version of the original Chebyshev function [36] so as to apply it to the practical applications where the range of the internal states ${\sigma }_i\left(t\right)$ may not be restricted in general as $\left|{\sigma }_i\left(t\right)\right|\le 1$, which is to be forced in the original Chebyshev function [35, 36].

To review the chaotic property of the present chaotic neurons [35], let us present the fundamental properties of a single neuron in the following section. Several profiles of the activation function of Eq. (5) for several $\nu$ are depicted in Figure 1. As a specific case, assuming that ${\sigma }_i\to \pm \infty$ in Eq. (5), one finds

Figure 1.

The presently introduced Chebyshev-type activation function [35]. Herein, the control parameter $\nu$ is set to $\nu =2,3,4,5$ for each profile of the activation function defined by Eq. (5). The solid, dashed, dashed-dotted, and dotted lines are for $\nu =2$, $\nu =3$, $\nu =4$_, and $\nu =5$, respectively.

$$\begin{array}{rcl}{s}_i\left(t\right)& =& \underset{{\sigma }_i\left(t\right)\to \pm \infty }{lim}{tanh}^{-1}\left\{sin\left(\nu {sin}^{-1}\left(tanh{\sigma }_i\left(t\right)\right)\right)\right\}\\ & =& {tanh}^{-1}\left\{sin\left(\pm \nu \frac{\pi }{2}\right)\right\}=\pm {tanh}^{-1}\left\{sin\left(\nu \frac{\pi }{2}\right)\right\},\end{array}$$E6

as is confirmed in Figure 1. Herein, the subscripts $i$ of ${\sigma }_i\left(t\right)$ and $s_i\left(t\right)$ are ignored for the moment for a single neuron.

Then, in order to examine a chaotic dynamics of the present chaos neuron model, replacing $s_i\left(t\right)$ with ${\sigma }_i\left(t+1\right)$ in Eq. (5), we have the following updating rule for the internal state ${\sigma }_i\left(t\right)$ of the i-th neuron [35]:

$${\sigma }_i\left(t+1\right)={tanh}^{-1}\left\{sin\left(\nu {sin}^{-1}\left(tanh{\sigma }_i\left(t\right)\right)\right)\right\}.$$E7

As an example, the chaotic behavior of the internal state ${\sigma }_i\left(t\right)$ in terms of Eq. (7) for a single chaos neuron, in which the subscript $i$ of ${\sigma }_i\left(t\right)$ is ignored again, is presented in Figure 2, where $\nu$ was set to $\nu =2$. Moreover, the corresponding bifurcation diagram is depicted in Figure 3 over $\tau ={\tau }_0\sim 2\left(0<{\tau }_0<<1\right)$, or $\nu =1\sim +\infty$, in the abscissa, where $\nu$ is related to $\tau$ in terms of [35]:

Figure 2.

An example of chaotic behavior of the internal state $\sigma \left(t\right)$ according to Eq. (7) where $\nu =2$ or $\tau =2/\nu =1$.

Figure 3.

The bifurcation diagram for the Chebyshev-type mapping given by Eq. (7) [35].

$$\nu =2/\tau .$$E8

Therein, one may confirm the variation of the chaos strength, or the complexity of dynamics, depending on $\tau$. As a special case with $\nu =1$ or $\tau =2/\nu =2$, one may readily confirm the following relation:

$${\sigma }_i\left(t+1\right)={tanh}^{-1}\left\{sin\left({sin}^{-1}\left(tanh{\sigma }_i\left(t\right)\right)\right)\right\}={\sigma }_i\left(t\right),$$E9

which corresponds to an identity mapping such that $s_i\left(t\right)={\sigma }_i\left(t\right)$.

Through the updating rule of Eq.(7), the corresponding Lyapunov exponent $\lambda$ has to be derived as [35]:

$$\begin{array}{l}\lambda =\underset{T\to \infty }{lim}\frac{1}{T}\sum _{n=0}^{T-1}log\left|\frac{\mathit{d\sigma }\left(t+1\right)}{\mathit{d\sigma }\left(t\right)}\right|\\ =\underset{T\to \infty }{lim}\frac{1}{T}\sum _{n=0}^{T-1}log\left|\frac{\nu cosz\left(t\right)}{cos\mathit{\nu z}\left(t\right)}\right|\\ =log\nu +\underset{T\to \infty }{lim}\frac{1}{T}\sum _{n=0}^{T-1}log\left|\frac{cosz\left(t\right)}{cos\mathit{\nu z}\left(t\right)}\right|,\end{array}$$E10

where

$$z\left(t\right)={sin}^{-1}\left\{tanh\sigma \left(t\right)\right\}.$$E11

To derive Eq. (10), we have noticed the following derivations:

$$\begin{array}{l}\frac{\mathit{d\sigma }\left(t+1\right)}{\mathit{d\sigma }\left(t\right)}=\frac{{\mathit{dg}}_{\nu }\left(\eta \left(t\right)\right)}{\mathit{d\eta }\left(t\right)}\frac{\mathit{d\eta }\left(t\right)}{\mathit{d\sigma }\left(t\right)}\\ =\frac{{\mathit{dg}}_{\nu }\left(\eta \left(t\right)\right)}{{\mathit{df}}_{\nu }\left(\eta \left(t\right)\right)}\frac{{\mathit{df}}_{\nu }\left(\eta \left(t\right)\right)}{\mathit{d\eta }\left(t\right)}\frac{\mathit{d\eta }\left(t\right)}{\mathit{d\sigma }\left(t\right)}\\ =\nu \frac{cos\left(z\left(t\right)\right)}{cos\left(\mathit{\nu z}\left(t\right)\right)},\end{array}$$E12

where $g_{\nu }$, $f_{\nu }$, $\eta \left(t\right)$_, and $z\left(t\right)$ are defined as:

$$g_{\nu }\left(\eta \left(t\right)\right)={tanh}^{-1}\left[f_{\nu }\left(\eta \left(t\right)\right)\right],$$E13

$$f_{\nu }\left(\eta \left(t\right)\right)=sin\left\{\nu {sin}^{-1}\left(\eta \left(t\right)\right)\right\}=sin\left\{\mathit{\nu z}\left(t\right)\right\},$$E14

$$\eta \left(t\right)=sinz\left(t\right)=tanh\sigma \left(t\right),$$E15

respectively [35].

The dependence of the Lyapunov exponents $\lambda$ derived by Eq. (10) on $\tau =2/\nu$ are illustrated with dot symbols in Figure 4, where $\lambda$ is found to be decreased as $\tau$ increases toward $\nu =2/\tau =1$ as Eq. (9). In addition, the following approximated expression [34] of the Lyapunov exponents is also depicted as a solid curve in Figure 4:

Figure 4.

The Lyapunov exponents for the Chebyshev-type mapping given by Eq. (7) for $0<\tau =2/\nu <2$. Herein, the Lyapunov exponents in the ordinate are normalized in terms of log2 for convenience. The dots and the solid line are for the numerical results derived by Eqs. (7) and (10) and the approximated expression, $log\left(2/\tau \right)/log2$ as in Eq. (16), respectively.

$$\lambda \simeq log\nu =log\left(\frac{2}{\tau }\right).$$E16

Therein, one may confirm that the aforementioned expression holds approximately in comparison with the numerical results with Eqs. (7) and (10). Hence, according to Eqs. (10) and (16), the following relation is considered to be approximately satisfied [35]:

$$\underset{T\to \infty }{lim}\frac{1}{T}\sum _{t=0}^{T-1}log\left|\frac{cosz\left(t\right)}{cos\mathit{\nu z}\left(t\right)}\right|\simeq 0.$$E17

Especially for a specific case of $\nu =2$, or $\tau =1$, an analytical expression of the Lyapunov exponent $\lambda$ has been derived as [35]

$$\lambda =log2,$$E18

which is found to be exactly same with that of the logistic mapping because of the topological conjugacy of the Chebyshev-type mapping as Eq. (5) with the conventional logistic one [35]. In the aforementioned work [35], the corresponding invariant measure was also investigated further in comparison with the sinusoidal and the monotonous activation functions [37].

From these results, the strength of chaos is found to be properly controlled, through the parameter $\tau =2/\nu$ as in the previous chaos associative memories [21, 22, 23, 24, 27, 28, 29, 30, 31, 32, 33, 34] as well as the learning models with the back-propagation algorithm [38, 39, 40].

In the similar manner to the previous models [21, 22, 23, 24, 27, 28, 29, 30, 31, 32, 33, 34, 37, 38, 41], the control dynamics of the parameter $\tau \left(t\right)$ may be simply introduced as follows:

$$\tau \left(t+1\right)=\tau \left(t\right)+\kappa \left({\tau }_{\infty }-\tau \left(t\right)\right),$$E19

where $0<\kappa <1$ is regarded as a parameter to control the number of the steps, or the speed, of the transient retrieval process from an initial chaotic state with a relatively small $\tau \left(0\right)={\tau }_0\left(<<1\right)$ to a memory retrieval point with $\tau \left(t\right)\sim {\tau }_{\infty }$. Since $\nu =2/\tau$, $\nu$ varies from $2/{\tau }_0\left(>>1\right)$ to $2/{\tau }_{\infty }$ as $\tau \left(t\right)$ changes from $\tau \left(0\right)={\tau }_0\left(<<1\right)$ to $\tau \left(\infty \right)={\tau }_{\infty }$ according to Eq. (19). Solving the difference equation as Eq. (19) with an initial value of $\tau \left(0\right)={\tau }_0\left(<<1\right)$, one may readily derive the following analytical expression for $\tau \left(t\right)$:

$$\begin{array}{l}\tau \left(t\right)={\tau }_{\infty }-\left({\tau }_{\infty }-\tau \left(0\right)\right){\left(1-\kappa \right)}^t\\ ={\tau }_{\infty }-\left({\tau }_{\infty }-{\tau }_0\right)exp\left(-t/T_c\right),\end{array}$$E20

where $T_c$ is regarded as a time constant for the retrieval process from $\tau \left(0\right)={\tau }_0\left(0<{\tau }_0<<1\right)$ to $\tau \left(\infty \right)={\tau }_{\infty }$ and given by:

$$T_c=-\frac{1}{log\left(1-\kappa \right)}=\frac{1}{log\left(\frac{1}{1-\kappa }\right)}.$$E21

Updating $\tau \left(t\right)$ with a certain initial value as $\tau \left(0\right)={\tau }_0\left(0<{\tau }_0<<1\right)$ and solving Eq. (3) with Eqs. (5), (8), and (19) simultaneously under an initial vector set $s_i\left(0\right)\left(1\le i\le N\right)$ which has a Hamming distance with respect to a target vector $e_i^{\left(s\right)}\left(1\le i\le N,1\le s\le L\right)$ to be retrieved among the embedded vectors, we may construct a chaos auto-associative model with the Chebyshev-type activation function [35].

3. Computer simulation results

In this section, we shall show some numerical results derived from the aforementioned chaos auto-association model defined by Eqs. (1), (3), (5), (8), and (19).

For numerical simulations, the embedded vectors $e_i^{\left(r\right)}\left(1\le i\le N,1\le r\le L\right)$ will be defined as random bipolar vectors as follows:

$$e_i^{\left(r\right)}=\mathit{sign}\left(z_i^{\left(r\right)}\right),$$E22

where $\mathit{sign}\left(•\right)$ is the signum function such that

$$\mathit{sign}\left(x\right)=\left\{\begin{array}{c}+1\left(x>0\right)\\ 0\left(x=0\right)\\ -1\left(x<0\right)\end{array}\right.,$$E23

and $z_i^{\left(r\right)}\left(1\le i\le N,1\le r\le L\right)$ are the pseudo-random numbers between −1 and + 1. In addition, $e_i^{\left(r\right)}\left(1\le r\le N\right)$ are assumed to be linearly independent each other to avoid the redundancy among the embedded vectors such that

$$det\left(\sum _{i=1}^N{e}_i^{\left(r\right)}{e}_i^{\left(s\right)}\right)\ne 0,$$E24

where $det\left(•\right)$ means the determinant of a matrix $•$. Then the connection matrix, i.e. the autocorrelation matrix, between the neurons, $w_{\mathit{ij}}$ is straightforwardly derived in terms of Eq. (1).

The initial conditions for $s_i\left(t\right)\left(1\le i\le N\right)$ may be set [21, 22, 23, 24, 27, 28, 29, 30, 31, 32, 33] in the following manner:

$$s_i\left(0\right)=\left\{\begin{array}{c}-e_i^{\left(s\right)}\left(1\le i\le H_d\right)\\ e_i^{\left(s\right)}\left(H_d+1\le i\le N\right)\end{array}\right.,$$E25

where $H_d$ is the Hamming distance between the initial vector $s_i\left(0\right)$ and a concerned target vector $e_i^{\left(s\right)}\left(\forall s\in \left\{1,2,•••,L\right\}\right)$ to be retrieved which is randomly chosen among the embedded vectors $e_i^{\left(r\right)}\left(1\le r\le L\right)$. Of course, since the presently introduced embedded patterns are defined as the random bipolar vectors as in Eq. (22), Eq. (25) is substantially equivalent to random choice of the bit-reversed components in an initial vector $s_i\left(0\right)$ with certain amounts of bit-reversed $H_d$ components corresponding to the involved noise in $s_i\left(0\right)$. For a minimum noise, or a minimum Hamming distance, included in the initial vector $s_i\left(0\right)$, $H_d$ is to be set to $1$, which corresponds to a single-bit reverse in the initial vector$s_i\left(0\right)$ from $e_i^{\left(s\right)}\left(\forall s\in \left\{1,2,•••,L\right\}\right)$. For the initial values of the vector, $s_i\left(0\right)$, the corresponding initial values for the internal states ${\sigma }_i\left(0\right)$ may be set to as:

$${\sigma }_i\left(0\right)=\sum _{j=1}^N{w}_{\mathit{ij}}{s}_j\left(0\right),$$E26

setting as ${\sigma }_i\left(t+1\right)={\sigma }_i\left(t\right)={\sigma }_i\left(0\right)$ and $s_i\left(t\right)=s_i\left(0\right)$ in Eq. (3).

For convenience, the ratio of the distorted components according to Eq. (25), or the Hamming distance $H_d$, to the number of neurons, $N$, may be introduced by:

$$r_d=H_d/N.$$E27

Hence, the directional cosine, i.e. the inner product, ${\varsigma }^{\left(s\right)}$ between the input vector $s_i\left(0\right)$ and a target vector $e_i^{\left(s\right)}\left(\forall s\in \left\{1,2,•••,L\right\}\right)$ can be evaluated in terms of $r_d$ as follows:

$${\varsigma }^{\left(s\right)}=\frac{1}{N}\sum _{i=1}^N{e}_i^{\left(s\right)}{s}_i\left(0\right)=1-\frac{2H_d}{N}=1-2r_d.$$E28

It is well known that the succeeded memory retrieval becomes apparently troublesome if ${\varsigma }^{\left(s\right)}$ is decreased from ${\varsigma }^{\left(s\right)}=1$ to ${\varsigma }^{\left(s\right)}\sim 0$ as an initial condition, since the involved information in order to retrieve a target vector $e_i^{\left(s\right)}\left(\forall s\in \left\{1,2,•••,L\right\}\right)$ tends to be depressed as $r_d$ increases from $r_d=0$ to $r_d\sim 0.5$ [9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 27, 28, 29, 30, 31, 32, 33, 34, 41].

Then, in similar manner to the previous works [21, 22, 23, 24, 27, 28, 29, 30, 31, 32, 33, 34, 41] for the associative memories, the overlaps $o^{\left(r\right)}\left(t\right)$ with respect to the embedded vector $e_i^{\left(r\right)}\left(1\le r\le L\right)$ for the output vector $s_i\left(t\right)$ may be defined as:

$$o^{\left(r\right)}\left(t\right)=\frac{1}{N}\sum _{i=1}^N{e}_i^{\left(r\right)}\mathit{sign}\left\{s_i\left(t\right)\right\}.$$E29

If one of the overlaps $o^{\left(r\right)}\left(t\right)\left(1\le r\le L\right)$, e.g.$o^{\left(s\right)}\left(t\right)$, leads to $+1$ or $-1$, then one may determine that a complete memory retrieval corresponding to $e_i^{\left(s\right)}$ is achieved with $s_i\left(t\right)=e_i^{\left(s\right)}$. It has to be noted here that there exists an uncertainty for the sign of $o^{\left(s\right)}\left(t\right)$ because of the invariant property of the dynamic equation, Eq. (3), and the activation function Eq. (5), under $s_i\left(t\right)\to -s_i\left(t\right)$ as well as ${\sigma }_i\left(t\right)\to -{\sigma }_i\left(t\right)$. Thus, the success rate for the different random embedded sets $e_i^{\left(r\right)}=\mathit{sign}\left(z_i^{\left(r\right)}\right)\left(1\le r\le L\right)$ may be evaluated by examining the succeeded memory retrieval according to the following criteria:

$$\left|\underset{1\le r\le L}{\mathit{Max}}\left|o^{\left(r\right)}\left(t\right)\right|-1\right|=\left|\underset{1\le r\le L}{\mathit{Max}}\left|\frac{1}{N}\sum _{i=1}^N{e}_i^{\left(r\right)}\mathit{sign}\left\{s_i\left(t\right)\right\}\right|-1\right|<\epsilon \left(<<1\right),$$E30

and

$$\left|\tau \left(t+1\right)-\tau \left(t\right)\right|<\epsilon \left(<<1\right),$$E31

as $\tau \left(t\right)$ approaches to ${\tau }_{\infty }$ starting from $\tau \left(0\right)={\tau }_0\left(<<1\right)$ in accordance with Eq. (19) [33, 34]. Provided that both conditions such as Eqs. (30) and (31) are satisfied simultaneously, the current memory retrieval may be decided as a succeeded one. Alternatively, if only the latter condition as Eq. (31) is satisfied, the retrieval trial is judged as a failed one. In any cases, the next retrieval process with Eq. (3) will be restarted with the other random set of embedded vectors as Eq. (22) to constitute the connection matrix $w_{\mathit{ij}}$ as Eq. (1) resetting $\tau \left(t\right)$ to the initial value of $\tau \left(0\right)={\tau }_0$, $s_i\left(0\right)$, and ${\sigma }_i\left(0\right)$ to Eqs. (25) and (26) with fixed $\alpha$ and $r_d$. This retrieval process will be successively repeated up to the total number of trials, i.e. $T_s$.

Thus through the present numerical simulations for certain values of $\alpha =L/N$ and $r_d=H_d/N$, one may evaluate the number of the succeeded trials $N_s\left(\alpha ,r_d\right)$ among $T_s$ trials with the aforementioned different random bipolar vector sets $e_i^{\left(r\right)}\left(1\le i\le N,1\le r\le L\right)$ and the connection matrices $w_{\mathit{ij}}$ as defined in terms of Eqs. (22) and (1). Here, the number of trials $T_s$ in the retrieval simulations corresponds to the number of the retrieval processes from $\tau \left(0\right)={\tau }_0\left(0<{\tau }_0<<1\right)$ to $\tau \left(\infty \right)={\tau }_{\infty }$ according to Eq. (19) or Eq. (20) with the criterion Eq. (31). Then, counting up the total number of the succeeded trials as $N_s\left(\alpha ,r_d\right)$ among $T_s$ trials, the succeeded association rate $S_r\left(\alpha ,r_d\right)$ in accordance with the aforementioned criteria with Eqs. (30) and (31) for a pair of fixed parameters, $\alpha =L/N$ and $r_d=H_d/N$, will be introduced as follows:

$$S_r\left(\alpha ,r_d\right)=\frac{N_s\left(\alpha ,r_d\right)}{T_s}.$$E32

Thus, the ability of the memory retrievals for the associative memory model may be evaluated in terms of the score of the succeeded memory retrievals rate to satisfy the conditions and Eqs. (30) and (31) for a certain fixed value of $r_d=H_d/N$, i.e. $S_r\left(\alpha ,r_d\right)$ vs. $\alpha =L/N$ characteristics as will be presented later. Hence, the memory capacity $M_c\left(r_d\right)$ may be defined as the total area of $S_r\left(\alpha ,r_d\right)$ (the ordinate) vs. $\alpha =L/N$ (the abscissa) curves as follows:

$$M_c\left(r_d\right)={\int }_0^1\mathit{d\alpha }{S}_r\left(\alpha ,r_d\right).$$E33

In the following practical simulations, the aforementioned parameters, $N$, $k_s$, $\kappa$, $T_s$, $\tau \left(\infty \right)={\tau }_{\infty }$, $\tau \left(0\right)={\tau }_0$, and $\epsilon$ (see Eqs. (30) and (31).) are set to 1000, 0.2, 0.5, 100, 1, ${10}^{-10}$, and ${10}^{-10}$, respectively, if not mentioned.

In Figure 5a–e, the retrieval characteristics, i.e. the dependence of the succeeded association rate defined by Eq. (32), $S_r\left(\alpha ,r_d\right)$, on the loading rate $\alpha =L/N$ are depicted for $r_d=1/N=0.001$(${\varsigma }^{\left(s\right)}=0.998$), $r_d=0.1$(${\varsigma }^{\left(s\right)}=0.8$), $r_d=0.2$ (${\varsigma }^{\left(s\right)}=0.6$), and $r_d=0.3$(${\varsigma }^{\left(s\right)}=0.4$), respectively. From these results, the memory retrieval abilities of the presently proposed Chebyshev-type associative memory model are found to be apparently superior than the conventional association models [9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 27, 28, 29, 30, 31, 32, 33, 34, 41]. By increasing $r_d=H_d/N$, i.e. decreasing ${\varsigma }^{\left(s\right)}$ as Eq. (28), the memory capacities $M_c\left(r_d\right)$ defined by Eq. (33) are found to be decreased in similar to the conventional associative models [9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 27, 28, 29, 30, 31, 32, 33, 34, 41]. In practice, the memory capacities $M_c\left(r_d\right)$ were evaluated as $M_c\left(r_d=0.001\right)=$ 0.4858, $M_c\left(r_d=0.1\right)=$ 0.4292, $M_c\left(r_d=0.2\right)=$ 0.3680, $M_c\left(r_d=0.3\right)=$ 0.2472, and $M_c\left(r_d=0.4\right)=$ 0.1168. From these findings, one may confirm the advantage of the present chaos associative model with the Chebyshev-type activation function as Eq. (5) beyond the conventional association models [21, 22, 23, 24, 27, 28, 29, 30, 31, 32, 33, 34, 41]. This point will be mentioned later in detail as shown in Figure 6.

Figure 5.

Memory retrieval characteristics for the Chebyshev-type activation function as in Eq. (5). Here, the abscissa and the ordinate are for the loading rate $\alpha =L/N$ and the succeeded association rate $S_r\left(\alpha ,r_d\right)$ defined by Eq. (32). (a) $r_d=0.001$, (b) $r_d=0.1$, (c) $r_d=0.2$, and (d) $r_d=0.3$.

Figure 6.

Comparison of the memory capacities defined by Eq. (33), i.e. the total area $S_r\left(\alpha ,r_d\right)$ vs. $\alpha =L/N$ curves given in Figures 5, 8 and 9, depending on $r_d=H_d/N$ between the presently examined three models, in which the solid, the dashed-dotted, and the dotted lines are for the Chebyshev-type activation function as Eq. (5), the sinusoidal activation function as Eq. (34) and the Associatron with signum activation function as Eq. (35), respectively.

In Figure 7, let us present some examples of time dependences of the internal states ${\sigma }_i\left(t\right)$ in Eq. (3), the outputs $s_i\left(t\right)$ derived from Eq. (5), and the overlaps $o^{\left(r\right)}\left(t\right)$ defined by Eq. (29) under an updating process of $\tau \left(t\right)$ according to Eq. (19) with an initial value $\tau \left(0\right)={\tau }_0$ as shown in Figure 7d for a certain loading rate $\alpha =L/N=0.3$ and $r_d=H_d/N=0.1,0.2,0.3$. From these results, one may confirm the success of the memory retrievals for (a-1)–(a-3) with $r_d=0.1$ as well as (b-1)-(b-3) with $r_d=0.2$, whereas the memory retrieval is no longer to be attained for (c-1)–(c-3) with $r_d=0.3$. This tendency is found to coincide with the previously shown results for the success rates for $r_d=0.1,0.2,0.3$ as shown in Figure 5b–d, respectively. In addition even for $\alpha =0.3$ and $r_d=0.2$, one may find the succeeded memory retrievals in which $o^{\left(s\right)}\left(t\right)=\underset{r}{\mathit{Max}}\left\{o^{\left(r\right)}\left(t\right)\right\}$ approaches to +1 or − 1 to satisfy Eq. (30). Of course, it is well appreciated that, for such a case with $\alpha =0.3$ and $r_d=0.2$, the succeeded memory retrievals considerably become troublesome in the conventional association models [9, 10, 11, 12, 13, 14, 18, 19, 20, 21, 22, 23, 24, 27, 28, 29, 30, 31, 32, 33, 34]. Comparing the time dependences of ${\sigma }_i\left(t\right)$ or $s_i\left(t\right)$ between the succeeded cases(see Figure 7a and b.) and the failed one (see Figure 7c.), it may be found that, in such a failed retrieval case as in Figure 7(c-3), there exists no longer separated regime as ${\sigma }_i\left(t\right)<0$, ${\sigma }_i\left(t\right)>0$, or $s_i\left(t\right)<0$, $s_i\left(t\right)>0$ as $\tau \left(t\right)\to {\tau }_{\infty }$ as can been seen in Figure 7(c-1), (c-2). In fact this tendency in Figure 7(c-1), (c-2) is obviously different from Figure 7(a-1), (a-2) and Figure 7(b-1), (b-2). As can be confirmed in Figure 7(c-3), $\underset{r}{\mathit{Max}}\left\{o^{\left(r\right)}\left(t\right)\right\}$ does no longer approach to $1$ under such behaviors of ${\sigma }_i\left(t\right)$ and $s_i\left(t\right)$, in which one may confirm no longer the separated regime as ${\sigma }_i\left(t\right)<0$, ${\sigma }_i\left(t\right)>0$, or $s_i\left(t\right)<0$, $s_i\left(t\right)>0$ as $\tau \left(t\right)\to {\tau }_{\infty }$. Now we are at the position to compare the present results with the conventional models with the sinusoidal and the signum activation functions. In practice, we shall compare the present numerical results of the Chebyshev-type activation function with the previously reported sinusoidal activation function model [21, 22, 23, 24, 27, 28, 29, 30, 31, 32, 33, 34] and the Associatron with the signum activation function proposed by Nakano [17], which are given by:

Figure 7.

Some examples of time dependences of the internal states ${\sigma }_i\left(t\right)$ in Eq. (3), the outputs $s_i\left(t\right)$ derived from Eq. (5), and the overlaps $o^{\left(r\right)}\left(t\right)$ defined by Eq. (29) under an updating process of $\tau \left(t\right)$ according to Eq. (19) with an initial value $\tau \left(0\right)={\tau }_0$ for $\alpha =L/N=0.3$ and $r_d=H_d/N=0.1,0.2,0.3$. (a-1) the time dependence of the internal states ${\sigma }_i\left(t\right)$ in Eq. (3) for $r_d=0.1$. (a-2) the time dependence of the outputs $s_i\left(t\right)$ according to Eq. (5) for $r_d=0.1$. (a-3) the time dependence of overlaps $o^{\left(r\right)}\left(t\right)$ defined by Eq. (29) for $r_d=0.1$. (b-1) the time dependence of the internal states ${\sigma }_i\left(t\right)$ in Eq. (3)$r_d=0.2$. (b-2) the time dependence of the outputs $s_i\left(t\right)$ according to Eq. (5) for $r_d=0.2$. (b-3) the time dependence of overlaps $o^{\left(r\right)}\left(t\right)$ defined by Eq. (29)$r_d=0.2$. (c-1) the time dependence of the internal states ${\sigma }_i\left(t\right)$ in Eq. (3) for $r_d=0.3$. (c-2) the time dependence of the outputs $s_i\left(t\right)$ according to Eq. (5) for $r_d=0.3$. (c-3) the time dependence of overlaps $o^{\left(r\right)}\left(t\right)$ defined by Eq. (29) for $r_d=0.3$. (d) the time dependence of $\tau \left(t\right)$, which follows Eq. (19), whose behavior is common to all figures, (a-1)–(c-3).

$$s_i\left(t\right)=sin\left(\frac{\pi }{2\tau }{\sigma }_i\left(t\right)\right),$$E34

and

$$s_i\left(t\right)=\mathit{sign}\left({\sigma }_i\left(t\right)\right),$$E35

respectively; here $\tau$ ranges over ${\tau }_0\le \tau \le {\tau }_{\infty }$ in Eq. (34), which may be controlled in terms of Eq. (19), and $\mathit{sign}\left(•\right)$ is again the signum function defined by Eq. (23). The fundamental chaotic property of a sinusoidal chaos neuron defined by Eq. (34) has been reported in the previous paper [35], in which, in similar manner to the present Chebyshev-type activation function, the strength of chaos, i.e. the Lyapunov exponents are found to be approximated as [34, 35, 37]:

$$\lambda \sim log\left(\frac{\pi }{4\tau }\right).$$E36

Therefore, the chaos dynamics is expected to be properly controlled by the parameter $\tau$ [35] which is to be updated through Eq. (19) or Eq. (20) in similar to the aforementioned Chebyshev-type association model.

The resultant retrieval characteristics for Eqs. (34) (sinusoidal activation function) and (35) (signum activation function) are given in Figures 8 and 9, respectively. First, in Figure 8 for the sinusoidal activation function as Eq. (34), according to these $\alpha$ vs. $S_r\left(\alpha ,r_d\right)$ curves, the memory capacities as Eq. (33) were evaluated as $M_c\left(r_d=0.001\right)=$0.3234, $M_c\left(r_d=0.1\right)=$0.2078, $M_c\left(r_d=0.2\right)=$0.1254, $M_c\left(r_d=0.3\right)=$0.1028, and $M_c\left(r_d=0.4\right)=$0.0560. Here, the control parameter $\tau =\tau \left(t\right)$ in Eq. (34) was updated in terms of Eq. (19) as in the same manner to the previously mentioned Chebyshev-type model in Figure 5.

Figure 8.

Memory retrieval characteristics for the sinusoidal activation function as in Eq. (34) instead of Eq. (5). Here the abscissa and the ordinate are for the loading rate $\alpha =L/N$ and the success rate $S_r\left(\alpha ,r_d\right)$ defined by Eq. (32). (a) $r_d=0.001$_, (b) $r_d=0.1$_, (c) $r_d=0.2$, and (d) $r_d=0.3$.

Figure 9.

Memory retrieval characteristics for the signum function as in Eq. (35) instead of Eq. (5). Here the abscissa and the ordinate are for the loading rate $\alpha =L/N$ and the association rate defined by Eq. (32). (a) $r_d=0.001$_, (b) $r_d=0.1$_, (c) $r_d=0.2$_, and (d) $r_d=0.3$.

Then, according to $\alpha$ vs. $S_r\left(\alpha ,r_d\right)$ curves in Figure 9, the memory capacities $M_c\left(r_d\right)$ as Eq. (33) for the Associatron with Eq. (35) were evaluated as $M_c\left(r_d=0.001\right)=$0.2282, $M_c\left(r_d=0.1\right)=$0.1454, $M_c\left(r_d=0.2\right)=$0.1402, and $M_c\left(r_d=0.3\right)=$0.1278, $M_c\left(r_d=0.4\right)=$0.070. These values of the memory capacities $M_c\left(r_d\right)$ are found to be lower than the aforementioned Chebyshev-type associative model with Eq. (5) over $0.001<r_d=H_d/N<0.4$ and lower than the sinusoidal associative one with Eq. (34), especially for a relatively low $r_d$ such that $r_d=H_d/N<0.1$.

In Figure 6, let us present the dependence of the memory capacities $M_c\left(r_d\right)$ on $r_d=H_d/N$ for the presently concerned three models, i.e. the Chebyshev-type activation function as Eq. (5) [35], the sinusoidal activation function as Eq. (34) [21, 22, 23, 26, 27, 28, 29, 30, 31, 32, 33, 34], and the Associatron as Eq. (35) [17] for comparison. From these results, one may find the advantage of the presently proposed Chebyshev-type activation function as Eq. (5) beyond such previous models as Eqs. (34) [21, 22, 23, 26, 27, 28, 29, 30, 31, 32, 33, 34] and (35) [17] as well as the other association models with the autocorrelation connection matrix $w_{\mathit{ij}}$ and the discrete time updating scheme as Eq. (3) [9, 10, 11, 12, 13, 14, 15, 18, 20]. Especially, for relatively small values of $r_d$ such that $r_d=H_d/N<0.1$, it is confirmed that the chaos retrieval models with the Chebyshev-type and the sinusoidal activation functions possess a remarkable advantage beyond the non-chaos association model such as the Assocciatron [17]. Alternatively, for relatively large $r_d$ such that $r_d=H_d/N>0.2$, the memory capacities of the sinusoidal chaos associative memory are found to be depressed and become slightly lower than those of the Associatron [17]. As a whole, however, the Chebyshev-type chaos association model shows an apparent advantage beyond these models, i.e. the sinusoidal chaos model [21, 22, 23, 26, 27, 28, 29, 30, 31, 32, 33, 34] and the Associatron [17] over $r_d=0.001\sim 0.4$. Therefore, one may conclude that the chaos retrieval memories such as the Chebyshev-type and sinusoidal models have an great advantage, especially for a relatively small $r_d$ rather than for a relatively large $r_d$. This finding may indicate that the information involved in the input vector $s_i\left(0\right)$ related to $r_d$ as defined by Eq. (25) is to be closely related to the retrieval ability even for the chaos memory retrieval processes in similar to the conventional non-chaos association model such as the Assosiatron [17, 34]. It seems to be instructive to note here that the memory capacity $M_c\left(r_d\to 1/N\sim 0\right)\sim 0.486$ for the present Chebyshev-type association model with autocorrelation connection matrix $w_{\mathit{ij}}$ is considerably close to ∼0.5 which is the critical value of the memory capacity for the orthogonal learning models with the signum-type activation function as Eq. (35), investigated by means of a statistical approach in the analogy with the spin glass system as previously reported [42, 43].

4. Concluding remarks

In this chapter, we have proposed a chaos associative memory model with the Chebyshev-type activation function [35] instead of the conventional ones including such as the sinusoidal one [21, 22, 23, 24, 27, 28, 29, 30, 31, 32, 33, 34] as well as the signum one [17]. From the present computer simulation results, we have confirmed the apparent advantage of the present chaos associative model beyond the aforementioned conventional models from the viewpoint of the memory capacities related to the robustness for the included noise, i.e. the distorted components in an input vector for the memory retrieval. In practice for the extreme case such as $r_d=1/N\left(<<1\right)$, the memory capacities of the Chebyshev-type activation function as Eq. (5), the sinusoidal activation function as Eq. (34), and the signum function as Eq. (35) are found to be 0.486, 0.341, and 0.228, respectively. Moreover, it has been found that the presently focused Chebyshev-type activation function has a remarkable advantage beyond the sinusoidal association models [21, 22, 23, 24, 27, 28, 29, 30, 31, 32, 33, 34] as well as the non-chaos association model as the Assocciatron [17] as can be confirmed by $r_d$ vs. $M_c\left(r_d\right)$ characteristics in Figure 6. Moreover, for relatively large values of $r_d$ such that $r_d>0.2$, the memory capacities of the sinusoidal chaos associative memory were found to be slightly lower than those of the Associatron [17] as can be seen in Figure 6, whereas that of the Chebyshev-type model has shown a relatively large memory capacities beyond both the sinusoidal chaos model [21, 22, 23, 24, 27, 28, 29, 30, 31, 32, 33, 34] and the Associatron [17]. Especially, it has to be borne in mind here again that the memory capacity of the Chebyshev-type model for $H_d=1$, or $r_d=1/N$, is promoted up to $M_c\left(r_d\right)$∼0.486 which is remarkably close to that of the Associatron with the orthogonal learning model [42, 43] in which $M_c\left(r_d\right)$∼0.5.

Several investigations on an optimization scheme of the included model parameters such as $k_s$, $\kappa$, ${\tau }_0$, and ${\tau }_{\infty }$ are now in progress and then will be reported elsewhere in the near future.

Moreover, it seems to be also interesting to investigate the orthogonal learning model [42, 43] as well as the continuous time model [20]. It seems to be worthwhile to introduce the present Chebyshev-type activation function to the chaotic leaning problems as previously reported [38, 39, 40] as well as the design of the classifier applied to the sensibility measurement scheme [44] in connection with an application to the communication technology, e.g. a silent speech interface as seen in the human-brain interfaces. In addition, a chaos learning model with the present Chebyshev-type activation function is to be investigated in comparison with the other chaos learning models with the sinusoidal activation functions [38, 39, 40].

Acknowledgments

The present work was supported in part by START program by Japan Science and Technology Agency in 2014–2016. The present author would like to greatly appreciate for the Grants-in-Aid for Scientific Research of MEXT, No.21300081, No.23650109, and No.24300084. The present author also would like to sincerely express his gratitude to his academic staffs and also to his family for their continuously heartwarming supports.