3.1. Introduction

Ability of spiking neurons to respond to incoming signal selectively was originally discovered by J. Hopfield in 1995 (Hopfield, 1995). He found that spiking neuron soma behaviour was similar to a radial basis function: the neuron fired as earlier as higher degree of coincidence of incoming spikes was; if the degree was sufficiently low, the neuron did not fire at all. And spiking neuron synapses appeared to be acting as a spike pattern storing unit: one was able to get a spiking neuron to fire to a certain spike pattern by adjusting synaptic time delays the way that they evened out (in temporal sense) incoming signal and made it to reach the neuron soma simultaneously. Spike pattern encoded in synaptic time delays of a neuron was called a center of spiking neuron in the following. Here it is worth to note that synchronization phenomena is of primary importance in nature (Pikovsky et al., 2001), particularly in the brain functioning (Malsburg, 1994).

The discovered capabilities of spiking neurons provided the basis for constructing self-learning networks of spiking neurons. Original architecture of self-learning spiking neural network and its learning algorithm, namely, a temporal Hebbian rule were introduced in (Natschlaeger & Ruf, 1998). The proposed self-learning network was able to separate clusters as long as their number was not greater than dimensionality of input signal. If clusters number exceeded number of input signal dimensions, spiking neural network performance decreased. This drawback was overcome by using population coding of incoming signal based on pools of receptive neurons in the first hidden layer of the network (Bohte et al., 2002). Such spiking neural network was shown to be considerably powerful and significantly fast in solving real life problems. Henceforward we will use this network as a basis for its further improvements and hybrid architectures designing.

3.2. Architecture

Self-learning spiking neural network architecture is shown on Fig. 3. As illustrated, it is heterogeneous two-layered feed-forward neural network with lateral connections in the second hidden layer.

The first hidden layer consists of pools of receptive neurons and performs transformation of input signal. It encodes an (n×1) -dimensional input sampled pattern x(k) (here n is the dimensionality of input space, k=1,N¯ is a patter number, N is number of patterns in incoming set) into (hn×1) -dimensional vector of input spikes t[0](x(k)) (here h is the number of receptive neurons in a pool) where each spike is defined by its firing time.

Spiking neurons form the second hidden layer of the network. They are connected with receptive neurons with multiple synapses where incoming vector of spikes transforms into postsynaptic potentials. Number of spiking neurons in the second hidden layer is set to be equal to the number of clusters to be detected. Each spiking neuron corresponds to a certain cluster. The neuron that has fired to the input pattern defines cluster that the pattern belongs to. Thus, the second hidden layer takes (hn×1) -dimensional vector of input spikes t[0](x(k)) and outputs (m×1) -dimensional vector of outgoing spikes t[1](x(k)) that defines the membership of input pattern x(k)

This is the basic architecture and behaviour of self-learning spiking neural network. The detailed architecture is stated below.

3.3. Population coding and receptive neurons

The first hidden layer is constructed to perform population coding of input signal. It acts in such a manner that each dimensional component xi(k) i=1,n¯ , of input signal x(k) is processed by a pool of h receptive neurons RNli l=1,h¯ . Obviously, there can be different number of receptive neurons hi in a pool for each dimensional component in the general case. For the sake of simplicity, we will consider here that the number of neurons is equal for all pools.

As a rule, activation functions of receptive neurons within a pool are bell-shaped (Gaussians usually), shifted, overlapped, of different width, and have dead zone. Generally firing time of a spike emitted by a receptive neuron RNli upon incoming signal xi(k) lies in a certain interval {−1}∪[0,tmax[0]] referred to as coding interval and is described by the following expression:

Figure 3.

Self-learning spiking neural network (spiking neurons are depicted the way to stress they act in integrate-and-fire manner)

where ⌊•⌋ is the floor function, ψ(••) cli[0] σli and θr.n. are the receptive neuron’s activation function, center, width and dead zone, respectively ( r.n. in the last parameter means ‘receptive neuron’), -1 indicates that the neuron does not fire. An example of population coding is depicted on Fig. 4. It is easily seen that the closer xi(k) is to the center cli[0] of receptive neuron RNli the earlier the neuron emits spike tli[0](xi(k))

Figure 4.

An example of population coding. Incoming signal xi(k) fires receptive neurons RN2,i and RN3,i . It is considered here that width of activation function of all receptive neurons within the pool is the same

In this work, we used Gaussian as activation function of receptive neurons:

There can be several ways to set widths and centers of receptive neurons within a pool. As a rule, activation functions can be of two types – either ‘narrow’ or ‘wide’. Centers of each width type of activation function are calculated in different ways but in either case they cover date range uniformly. More details can be found in (Bohte et al., 2002).

3.4. Spiking neurons layer

Spiking neuron is considered to be formed of two constituents, they are: synapse and soma.

As it was mentioned above, synapses between receptive neurons and spiking neurons are multiple structures. As shown on Fig. 5, a multiple synapse MSjli consists of a set of q subsynapses with different time delays dp dp−dp−10 dq−d1tmax[0] and adjustable weights wjlip (here p=1,q¯ ). It should be noted that number of subsynapses within a multiple synapse are fixed for the whole network. Having a spike tli[0](xi(k)) from the li -th receptive neuron, the p -th subsynapse of the j -th spiking neuron produces delayed weighted postsynaptic potential

where ε(•) is a spike-response function usually described by the expression (Natschlaeger & Ruf, 1998)

Figure 5.

Multiple synapse

Each incoming total postsynaptic potential contributes to membrane potential of spiking neuron SNj as follows:

Spiking neuron SNj generates at most one outgoing spike tj[1](x(k)) during a simulation interval (the presentation of an input pattern x(k) ), and fires at the instant the membrane potential reaches firing threshold θs.n. ( s.n. means here ‘spiking neuron’). After neuron has fired, the membrane potential is reset to the rest value urest (0 usually) until the next input pattern is presented.

It is easily seen that the descried behaviour of spiking neuron corresponds to the one of the leaky integrate-and-fire model.

Spiking neurons are linked with lateral inhibitory connections that disable all other neurons to fire after the first one has fired. Thus, any input pattern makes only one spiking neuron to fire that is only one component of the vector of outgoing spikes has non-negative value. There can be cases when a few spiking neurons fire simultaneously for an input pattern. Such cases are rare enough, and their appearance depends directly on initial synaptic weights distribution.

3.5. Crisp data clustering

As it was mentioned above, a spiking neuron acts similarly to a radial basis function, and its response depends on degree of coincidence of the input. There was considered a spiking neuron center to describe such neuron behaviour in a convenient way (Natschlaeger & Ruf, 1998). In the general case, it is considered to possess the following property: the closer input pattern is to the neuron’s center, the earlier output spike fires. Hence, a spiking neuron firing time reflects the similarity (Natschlaeger & Ruf, 1998) (or distance (Bohte et al., 2002)) between the input pattern and the neuron center. Degree of coincidence is utilized here as a similarity (distance) measure.

Center of spiking neuron is encoded in the synaptic time delays. They produce coincidence output (and in its terns it makes the soma to fire at the earliest possible time) if incoming pattern is similar to the encoded one. Thus, the learned spiking neuron can respond selectively to the input set of patterns. Data clustering in the described neural network rests on this property of spiking neuron. Input pattern fires the neuron whose center is the most similar (the closest) to it, and the fired spike prevents the rest neurons to fire through the lateral inhibitory connections. This way self-learning spiking neural network performs clusters separation if classes to be detected do not overlap one another.

One can readily see that the unsupervised pattern classification procedure of the spiking neurons layer is identical with the one of self-organizing maps (Kohonen, 1995).

4.1. Winner-takes-all

The purpose of an unsupervised learning algorithm of spiking neural network is to adjust centers of spiking neurons so as to make each of them to correspond to centroid of a certain data cluster. Such learning algorithm was introduced on the basis of two learning rules, namely, ‘Winner-Takes-All’ rule and temporal Hebbain rule (Natschlaeger & Ruf, 1998, Gerstner et al., 1996). The first one defines which neuron should be updated, and the second one defines how it should be updated. The algorithm updates neuron centers through synaptic weights adjusting, whereas synaptic time delays always remain constant. The concept here is that significance of the given time delay can be changed by varying corresponding synaptic weight.

Each learning epoch consists of two phases. Competition, the first phase, defines a neuron-winner. Being laterally linked with inhibitory connections, spiking neurons compete to respond to the pattern. The one wins (and fires) whose center is the closest to the pattern. Following competition phase, weights adjusting takes place. The learning algorithm adjusts synaptic weights of the neuron-winner to move it closer to the input pattern. It strengthens weights of those subsynapses which contributed to the neuron-winner’s firing (i.e. the subsynapses produced delayed spikes right before the neuron firing) and weakens ones which did not contribute (i.e. the delayed spikes appeared right after the neurons firing or long before it). Generally, the learning algorithm can be expressed as

where K is the current epoch number, ηw(•)0 is the learning rate (while it is constant in (Natschlaeger & Ruf, 1998), it can depend on epoch number in the general case; w means ‘weights’), L(•) is the learning function (Gerstner et al, 1996), j˜ is the number of neuron that has won on the current epoch, Δtjlip is the time delay between delayed spike tli[0](xi(k))+dp produced by the p -th subsynapse of the li -th synapse and spiking neuron firing time tj[1](x(k)) :

As a rule, the learning function has the following form (Berredo, 2005):

where α0 β0 ν are the shape parameters of the learning function L(•) that can be obtained empirically (Berredo, 2005, Natschlaeger & Ruf, 1998). The learning function and its shape parameters are depicted on Fig. 6. The effect of the shape parameters on results of information processing performed by spiking neural network can be found in (Meftah et al., 2008).

Upon the learning stage, center of a spiking neuron represents centroid of a certain data cluster, and spiking neural network can successfully perform unsupervised classification of the input set.

4.2. Winner-takes-more

The learning algorithm (7) updates only neuron-winner on each epoch and disregards other neurons. It seems more natural to update not only spiking neuron-winner, but also its neighbours (Bodyanskiy & Dolotov, 2009). This approach is known as ‘Winner-Takes-More’

Figure 6.

Learning function L(•)

rule. It implies that there is a cooperation phase before weights adjustment. Neuron-winner determines a local region of topological neighbourhood on each learning epoch. Within this region, the neuron-winner fires along with its neighbours, and the closer a neighbour is to the winner, the more significantly its weights are changed. The topological region is represented by the neighbourhood function ϕ(|Δtjj˜|) that depends on difference |Δtjj˜| between the neuron-winner firing time tj˜[1](x(k)) and the neighbour firing time tj[1](x(k)) (distance between the neurons in temporal sense) and a parameter that defines effective width of the region. As a rule, ϕ(•) is a kernel function that is symmetric about its maximum at the point where Δtjj˜=0 . It reaches unity at that point and monotonically decreases as Δtjj˜ tends to infinity. The functions that are the most frequently used as neighbourhood function are Gaussian, paraboloid, Mexican Hat, and many others (Bodyanskiy & Rudenko, 2004).

For self-learning spiking neural network, the learning algorithm based on ‘Winner-Takes-More’ rule can be expressed in the following form (Bodyanskiy & Dolotov, 2009):

where temporal distance Δtjj˜ is

Obviously, expression (11) is a generalization of (7).

Analysis of competitive unsupervised learning convergence showed that width parameter of the neighbourhood function should decrease during synaptic weights adjustment (Cottrell & Fort, 1986). For Gaussian neighbourhood function

width parameter ρ can be adjusted as follows (Ritter & Schulten, 1986):

where γ0 is a scalar that determines rate of neuron-winner effect on its neighbours.

Noteworthily that exponential decreasing of width parameter can be achieved by applying the simpler expression instead of (14) (Bodyanskiy & Rudenko, 2004):

Learning algorithm (11) requires modification of self-learning spiking neural architecture. Lateral inhibitory connections in the second hidden layer should be replaced with excitatory ones during the network learning stage in order to implement ‘Winner-Takes-More’ rule.

In the following sections, it will be shown that the learning algorithm based on ‘Winner-Takes-More’ rule is more natural than the one based on ‘Winner-Takes-All’ rule to learn fuzzy spiking neural network.

5.1. Introduction

Hardware implementations of spiking neural network demonstrated fast processing ability that made it possible to apply such systems in real-life applications where processing speed was a rather critical parameter (Maass, 1997a, Maass & Bishop 1998, Schoenauer et al., 2000, Kraft et al., 2006). From theoretical point of view, the current research works on spiking neurons hardware implementation subject are very particular, they lack for a technically plausible description on a general ground. In this section, we consider a spiking neuron as a processing system of classical automatic control theory (Feldbaum & Butkovskiy, 1971, Dorf & Bishop, 1995, Phillips & Harbor, 2000, Goodwin et al., 2001). Spiking neuron functioning is described in terms of the Laplace transform. Having such a general description of a spiking neuron, one can derive various hardware implementations of self-learning spiking neural network for solving specific technical problems, among them realistic complex image processing.

Within a scope of automatic control theory, a spike t(x(k)) can be represented by the Dirac delta function δ(t−t(x(k))) . Its Laplace transform is

where s is the Laplace operator. Spiking neuron takes spikes on its input, performs spike–membrane potential–spike transformation, and produces spikes on its output. Obviously, it is a kind of analog-digital system that processes information in continuous-time form and transmits it in pulse-position form. This is the basic concept for designing analog-digital architecture of self-learning spiking neural network. Overall network architecture is depicted on Fig. 7 and is explained in details in the following subsections.

5.2. Synapse as a second-order critically damped response unit

Multiple synapse MSjli of a spiking neuron SNj transforms incoming pulse-position signal δ(t−tli[0](xi(k))) to continuous-time signal ujli(t) . Spike-response function (4), the basis of such transformation, has form that is similar to the one of impulse response of second-order damped response unit. Transfer function of a second-order damped response unit with unit gain factor is

where τ1,2=τ32±τ324−τ42 τ1≥τ2 τ3≥2τ4 and its impulse response is

Putting τ1=τ2=τPSP (that corresponds to a second-order critically damped response unit) and applying l'Hôpital's rule, one can obtain

Comparing spike-response function (4) with the impulse response (19) leads us to the following relationship:

Thus, transfer function of the second-order critically damped response unit whose impulse response corresponds to a spike-response function is

where SRF means ‘spike-response function’.

Now, we can design multiple synapse in terms of the Laplace transform (Bodyanskiy et al., 2009). As illustrated on Fig. 7, multiple synapse MSjli is a dynamic system that consists of a set of subsynapses that are connected in parallel. Each subsynapse is formed by a group of time delay, second-order critically damped response unit, and gain. As a response to incoming spike δ(t−tli[0](xi(k))) the subsynapse produces delayed weighted postsynaptic potential ujlip(s) and the multiple synapse produces total postsynaptic potential ujli(s) that arrives to spiking neuron soma.

Taking into account (21), transfer function of the p -th subsynapse of MSjli takes the following form:

and its response to a spike δ(t−tli[0](xi(k))) is

So finally, considering transfer function of multiple synapse MSjli

the Laplace transform of the multiple synapse output can be expressed in the following form:

Here it is worth to note that since it is impossible to use δ -function in practice (Phillips & Harbor, 2000), it is convenient to model it with impulse of a triangular form (Feldbaum & Butkovskiy, 1971) as shown on Fig. 8. Such impulse is similar to δ -function under the following condition

Figure 8.

Triangular impulse a(tΔ)

In this case, numerator of (21) should be revised the way to take into account finite peak of a(tΔ) (in contrast to the one of the Dirac delta function).

5.3. Soma as a threshold detection unit

Spiking neuron soma performs transformation that is opposite to one of the synapse. It takes continuous-time signals ujli(t) and produces pulse-position signal δ(t−tj[1](x(k))) . In doing so, soma responds each time membrane potential reaches a certain threshold value. In other words, spiking neuron soma acts as a threshold detection system and consequently it can be designed on the base of relay control systems concept (Tsypkin, 1984). In (Bodyanskiy et al., 2009), mechanisms of threshold detection behaviour and firing process were described. Here we extend this approach to capture refractoriness of spiking neuron.

Threshold detection behaviour of a neuron soma can be modelled by an element relay with dead zone θ_s.n. that is defined by the nonlinear function

where sign (•)is the signum function. Soma firing can be described by a derivative unit that is connected with the element relay in series and produces a spike each time the relay switches. In order to avoid a negative spike that appears as a response to the relay resetting, a conventional diode is added next to the derivative unit. The diode is defined by the following function:

where trelay[1] is a spike produced by the derivative unit upon the relay switching.

Now we can define the Laplace transform of an outgoing spike tj[1](x(k)) , namely,

As it was mentioned above, the leaky integrate-and-fire model disregards the neuron refractoriness. Anyway, the refractory period is implemented in the layer of spiking neurons indirectly. The point is that a spiking neuron cannot produce another spike after firing and until the end of the simulation interval since the input pattern is provided only once within the interval. In the analog-digital architecture of spiking neuron, the refractoriness can be modelled by a feedback circuit. As shown on Fig. 7, it is a group of a time delay, a second-order critically damped response unit, and a gain that are connected in series. The time delay defines duration of a spike generation period dspike (usually, dspike→0 ). The second-order critically damped response unit defines a spike after-potential. Generally, spike after-potential can be represented by a second-order damped response unit, but for the sake of simplicity, we use critically damped response unit as it can be defined by one parameter only, namely, τSAP ( SAP means here ‘spike after-potential’). This parameter controls duration of the refractory period. Finally, the gain unit sets amplitude of the spike after-potential wSAP . Obviously, wSAP should be much greater than any synaptic weight.

Thus, transfer function of the feedback circuit is

where F.B. means ‘feedback circuit’, and transfer function of the soma is

where G _F.F. is defined by (29) (F.F. means ‘feed-forward circuit’).

It is easily seen that the functioning of spiking neuron analog-digital architecture introduced above is similar to the spike-response model.

6.1. Fuzzy receptive neurons

A common peculiarity of artificial neural networks is that they store dependence of system model outputs on its inputs in the form of ‘black box’. Instead, data processing methods based on fuzzy logic allow of system model designing and storing in analytical form that can be substantially interpreted in a relatively simple way. This fact arises interest in designing of hybrid systems that can combine spiking neural networks computational capabilities with capability of fuzzy logic methods to conveniently describe input-output relationships of the system being modelled. The present section shows how receptive neuron layers, a part of spiking neural network, can be ‘fuzzified’.

One can readily see that the layer of receptive neuron pools is identical to a fuzzification layer of neuro-fuzzy systems like Takagi-Sugeno-Kang networks, ANFIS, etc. (Jang et al., 1997). Considering activation function ψli(xi(k)) as a membership function, the receptive neurons layer can be treated as the one that transforms input data set to a fuzzy set that is defined by values of activation-membership function ψli(xi(k)) and is expressed over time domain in form of firing times tli[0](xi(k)) (Bodyanskiy et al., 2008a). In fact, each pool of receptive neurons performs zero order Takagi-Sugeno fuzzy inference (Jang et al., 1997)

where Xli is the fuzzy set with membership function ψli(xi(k)) . Thus, one can interpret a receptive neurons pool as a certain linguistic variable and each receptive neuron (more precisely, fuzzy receptive neuron) within the pool – as a linguistic term with membership function ψli(xi(k)) (Fig. 9). This way, having any a priori knowledge of data structure, it is possible beforehand to adjust activation functions of the first layer neurons to fit them and thus, to get better clustering results.

6.2. Fuzzy clustering

Conventional approach of data clustering implies that each pattern x(k) can belong to one cluster only. It is more natural to consider that a pattern can belong to a several clusters with different membership levels. This case is the subject matter of fuzzy cluster analysis that is heading in several directions. Among them, algorithms based on objective function are the most mathematically rigorous (Bezdek, 1981). Such algorithms solve data processing tasks by optimizing a certain preset cluster quality criterion.

One of the commonly used cluster quality criteria can be stated as follows:

where μj(x(k))∈[0,1] is the membership level of the input pattern x(k) to the j -th cluster, vj is the center of the j -th cluster, ζ≥0 is the fuzzifier that determines boundary between

Figure 9.

Terms of linguistic variable for the i-th input. Membership functions are adjusted to represent a priori knowledge of input data structure. Incoming signal x _i (k) fires fuzzy receptive neurons FRN _2,i and FRN _3,i

clusters and controls the amount of fuzziness in the final partition, ‖x(k)−vj‖A is the distance between x(k) and vj in a certain metric, A is a norm matrix that defines distance metric. By applying the method of indefinite Lagrange multipliers under restrictions

minimization of (33) leads us to the following solution:

that originates the methods of so-called fuzzy probabilistic clustering (Bezdek et al., 2005). In the case when norm matrix A is the identity matrix and ζ=1 , equations (36), (37) present hard c-means algorithm, and for ζ=2 they are conventional fuzzy c-means algorithm.

Efficiency of fuzzy probabilistic clustering decreases in the presence of noise. Algorithm (36), (37) produces unnaturally high degree of membership for outliers that are equidistant from clusters centers. This drawback is avoided by applying fuzzy possibilistic approach that is based on the following objective function:

where λj0 is the scalar parameter that defines the distance at which membership level takes the value 0.5, i.e. if ‖x(k)−vj‖A2=λj , then μj(x(k))=0.5 . Minimization of (38) with respect to μj(x(k)) vj , and λj yields the following solution:

that gives conventional possibilistic c-means algorithm if ζ=2 and A is the identity matrix (Bezdek et al., 2005).

After spiking neural network learning has been done, center cj[1] of a spiking neuron SNj represents center vj of a certain data cluster, and its firing time tj[1](x(k)) reflects distance ‖x(k)−vj‖A in temporal sense (Natschlaeger & Ruf, 1998, Bohte et al., 2002). This notion allows us of using self-learning spiking neural network output in fuzzy clustering algorithms described above. In order to implement fuzzy clustering on the base of the spiking neural network, its architecture is modified in the following way: lateral connections in the second hidden layer are disabled, and output fuzzy clustering layer is added next to spiking neuron layer. Such modification is applied to the spiking neural network on data clustering stage only. Output fuzzy clustering layer receives information on the distances of input patter to centers of all spiking neurons and produces fuzzy partition using either probabilistic approach (36), (37) as follows (Bodyanskiy & Dolotov, 2008a-2008b):

or possibilistic approach (39)-(41) as follows (Bodyanskiy et al., 2008b):

Obviously, the learning algorithm (11) is more natural here then (7) since response of each spiking neuron within the second hidden layer matters for producing fuzzy partition by the output layer.

The advantage of fuzzy clustering based on self-learning spiking neural network is that it is not required to calculate centers of data clusters according to (37) or (40) as the network finds them itself during learning.