A Continuous-Time Recurrent Neural Network for Joint Equalization and Decoding – Analog Hardware Implementation Aspects

Abstract

Equalization and channel decoding are “traditionally” two cascade processes at the receiver side of a digital transmission. They aim to achieve a reliable and efficient transmission. For high data rates, the energy consumption of their corresponding algorithms is expected to become a limiting factor. For mobile devices with limited battery’s size, the energy consumption, mirrored in the lifetime of the battery, becomes even more crucial. Therefore, an energy-efficient implementation of equalization and decoding algorithms is desirable. The prevailing way is by increasing the energy efficiency of the underlying digital circuits. However, we address here promising alternatives offered by mixed (analog/digital) circuits. We are concerned with modeling joint equalization and decoding as a whole in a continuous-time framework. In doing so, continuous-time recurrent neural networks play an essential role because of their nonlinear characteristic and special suitability for analog very-large-scale integration (VLSI). Based on the proposed model, we show that the superiority of joint equalization and decoding (a well-known fact from the discrete-time case) preserves in analog. Additionally, analog circuit design related aspects such as adaptivity, connectivity and accuracy are discussed and linked to theoretical aspects of recurrent neural networks such as Lyapunov stability and simulated annealing.

1. Introduction

Energy efficiency has been increasingly attracting more interest due to economical and environmental reasons. Mobile communications sector has currently a share of 0.2% in global carbon emissions. This share is expected to double between 2007 and 2020 due to the ever-increasing demand for wireless devices [1, 2]. The sustained interest in higher data rate transmission is strengthening this impact. While major resources are being invested in increasing the energy efficiency of digital circuits, there is, on the other hand, a growing interest pointing at alternatives to the digital realization [3], including a mixed (analog/digital) approach. In such an approach, specific energy consuming (sub)tasks are implemented in analog instead of a “conventional” digital realization. The analog implementation possesses a high potential to significantly improve the energy efficiency [4] because of the inherent parallel processing of signals that are continuous in both time and amplitude. This has been shown in the field of error correction coding with a focus on decoding of low-density parity-check (LDPC) codes. Our ongoing research on equalization reveals similar results. We do not intend “analog” for linear signal processing with all its disadvantages like component inaccuracies and susceptibility to noise and temperature dependency [5] but for nonlinear processing instead. The work of Mead [6] and others on Neuromorphic analog very-large-scale integration (VLSI) has shown that “analog signal processing systems can be built that share the robustness of digital systems but outperform digital systems by several orders of magnitude in terms of speed and/or power consumption” [5].

The nonlinearity makes the analog implementation of an algorithm as robust as its digital counterpart [3, 5]. This profits from the match between the needed nonlinear operations for the algorithm and the physical properties of analog devices [7].

The capability of artificial neural networks (in the following neural networks) to successfully solve many scientific and engineering tasks has been shown oftentimes. Moreover, mapping algorithms to neural network structures can simplify the circuit design because of the regular (and repetitive) structure of neural networks and their limited number of well-defined arithmetic operations. Digital implementations can be considered precise (reproducibility of results under similar circumstances) but accurate (closeness of a result to the “true” value) only to the extent to which they have enough digits to represent [8]. This means, accuracy in digital implementations is achieved at the cost of efficiency (e.g., relatively larger chip area and more power consumption) [9]. An analog implementation is usually efficient in terms of chip area and processing speed [9], however, at the price of an inherent lack of the reproducibility of results [8] (because of a limited accuracy of the network components as an example [9]). However, by exploiting the distributed nature of neural structures the precision of the analog implementation can be improved despite inaccurate components and subsystems [8] ¹

For a clear distinction between accuracy and precision when used in hardware implementation context, we refer to [8].

. In other words, it is the distributed massively parallel nonlinear collective behavior of an analog implementation (of neural networks) which offers the possibility to make it as robust as its digital counterpart but more energy efficient ²

Energy efficiency is defined later as appropriate.

(additionally to smaller chip area). Particularly for recurrent neural networks (the class we focus on when considered as nonlinear dynamical systems), the robustness can be additionally achieved by exploiting “attracting” equilibrium points. In the light of this discussion, we map in this chapter a joint equalization and decoding algorithm into a novel continuous-time recurrent neural network structure. This class of neural networks has been attracting a lot of interest because of their widespread applications. They can be either trained for system identification [10], or they can be considered as dynamical systems (dynamical solver). In the latter case, there is no need for a computationally complex and time-consuming training phase. This relies on the ability of these networks (under specific conditions) to be Lyapunov stable.

Equalization and channel decoding (together, in the following detection) are processes at the receiver side of a digital transmission. They aim to provide a reliable and efficient transmission. Equalization is needed to cope with the interference caused by multipath propagation, multiusers, multisubchannels, multiantennas and combinations thereof [11]. Channel (de)coding is applied for further improving the power efficiency. Equalization and decoding are nonlinear discrete optimization problems. The optimum solutions, in general, are computationally very demanding. Therefore, suboptimum solutions are applied, often soft-valued iterative schemes because of their good complexity-performance trade-off.

For high data rates, the energy consumption of equalization and decoding algorithms is expected to become a limiting factor. The need for floating-point computation and the nonlinear and iterative nature of (some of) these algorithms revive the option of an analog electronic implementation [12, 13], embedded in an essentially digital receiver. This option has been strengthened since the emergence of the “soft-valued” computation in this context [4] since soft-values are a natural property of analog signals. In contrast to analog decoding, analog equalization did not attract that amount of attention.

Furthermore, joint equalization and decoding (a technique where equalizer and decoder exchange their local available knowledge) further improves the efficiency of the transmission as an example in terms of lower bit error rates, however, at the cost of more computational complexity [14]. Most of the work related to joint equalization and decoding is limited to the discrete-time realization. One of the very few contributions focusing on continuous-time joint equalization and decoding is given in [13]. The consideration in [13] is not “neural networks-based”. Stability and convergence are observed but not “deeply” considered.

We introduce in this chapter a novel continuous-time joint equalization and decoding structure. For this purpose, continuous-time single-layer recurrent neural networks play an essential role because of their nonlinear and recursive characteristic, special suitability for analog VLSI and since they serve as promising computational models for analog hardware implementation [15]. Both, equalizer and decoder are modeled as continuous-time recurrent neural networks. An additional proper feedback between equalizer and decoder is established for joint equalization and decoding. We also review individually, both continuous-time equalization and continuous-time decoding based on recurrent neural network structures. No training is needed since the recurrent neural network is serving as a dynamical solver or a computational model [15, 16]. This means, transmission properties are used to define the recurrent neural network (number of neurons, weight coefficients, activation functions, etc.) such that no training is needed. In addition, we highlight challenges emerging from the analog hardware implementation such as adaptivity, connectivity and accuracy. We also introduce our developed circuit for analog equalization based on continuous-time recurrent neural networks [3]. Characteristic properties of recurrent neural networks such as stability and convergence are addressed too. Based on the introduced model, we show by simulations that the superiority of joint equalization and decoding can be preserved in the analog “domain”.

The main motivation for performing joint equalization and decoding in analog instead of using conventional digital circuits is to improve the energy efficiency and to minimize the area consumption in the VLSI chips [17]. The proposed continuous-time recurrent neural network serves as a promising computational model for analog hardware implementation.

The remainder of this chapter is organized as follows: In Section 2, we describe the block transmission model. Sections 3 and 4 are dedicated to the equalization process, the application of continuous-time recurrent neural networks and the analog circuit design and its corresponding performance and energy efficiency. Sections 5 and 6 are devoted to the channel decoding and the application of continuous-time recurrent neural networks for belief propagation (a decoding algorithm for LDPC codes). For both equalization and decoding cases, analog hardware design aspects and challenges and the behavior of the continuous-time recurrent neural network as a dynamical system are discussed. The continuous-time joint equalization and decoding based on recurrent neural networks is presented in Sections 7 and 8. Simulation results are shown in Section 9. We finish this chapter with a conclusion in Section 10.

Throughout this chapter, bold small and bold capital letters designate vectors (or finite discrete sets) and matrices, respectively. ³

Except for L, $\stackrel{⌣}{\mathit{L}}$ and ${\mathit{L}}_{ch}$ which are vectors.

All nonbold letters are scalars. diag_m{B} returns the matrix B where the nondiagonal elements are set to zeros. ${\text{diag}}_{\upsilon }\left\{\mathit{\text{b}}\right\}$ returns a matrix where the vector b is put on the diagonal. 0_N is the all-zero vector of length N. 0, 1 and I represent the all-zero, all-one and the identity matrix of suitable size, respectively. We consider column vectors. ${(\cdot )}^H$ represents the conjugate transpose of a vector or a matrix, whereas ${(\cdot )}^T$ represents the transpose. $z_r=\Re \left(z\right)$ , $z_i=\Im \left(z\right)$ returns the real and imaginary part of the complex-valued argument $z=z_r+\iota z_i$ , respectively. $\iota =\sqrt{-1}$ . t and l are designated to the continuous-time variable and the discrete-time index, respectively.

2. Block transmission model

The block transmission model for linear modulation schemes is shown in Figure 1. For details, see [18]:

Figure 1.

Block transmission model for linear modulation schemes. SRC (SNK) represents the digital source (sink). DET is the detector. COD performs the encoding process (adding redundancy). $\mathcal{M}$ maps encoded bits to complex-valued symbols. $\mathit{R}$ is the block transmit matrix.

• SRC (SNK) represents the digital source (sink). SRC repeatedly generates successive streams of k bits, i.e., q₁, q₂, ⋯ ,q_M.

• $\mathit{q}(\widehat{\mathit{q}})\in {\left\{\mathrm{0,1}\right\}}^k$ is the vector of source (detected) bits of length k.

• ${\mathit{q}}_c\in {\left\{\mathrm{0,1}\right\}}^n$ is the vector of encoded source bits of length $n>k$ . For an uncoded transmission ${\mathit{q}}_c=\mathit{q}$ (and thus $k=n$ ).

• COD performs a bijective map from q to q_c where $n>k$ (adding redundancy). We consider in this chapter binary LDPC codes. Only 2^k combinations of n bits out of overall 2ⁿ combinations are used. The set of the 2^k combinations represent the code book $\mathcal{C}$ . $r_c=k/n$ is the code rate.

• x ∈ ψ^N is the transmit vector of length N.

• N is the block size. Successive transmit vectors are separated by a guard time to avoid interference between different blocks. Thus, Figure 1 describes the transmission for a single block and stays valid for the next block (possibly with a different R).

• ψ$=\left\{{\psi }_1\mathrm{,}{\psi }_2\mathrm{,}\dots \mathrm{,}{\psi }_{2^m}\right\}$ , $m\in \mathbb{N}/\left\{0\right\}$ is the symbol alphabet. There exist $2^{m\cdot N}$ possible transmit vectors. The set of all possible transmit vectors is χ. The mapping from q_c to x is performed by $\mathcal{M}$ . Each symbol $\psi$ represents m bits. A special class of symbol alphabets are the so-called separable symbol alphabet ${\mathit{\psi }}^{(s)}$ [19, 20].

• $\tilde{\mathit{x}}$ is the receive vector of length N. In general $\tilde{\mathit{x}}\in {ℂ}^N$ .

• We distinguish:

  1. – For an uncoded transmission $M\times k=m\times N$ .

  2. – For a coded transmission and $N<n/m$ : One codeword lasts over many transmit blocks.

  3. – For a coded transmission and $N=n/m$ : One codeword lasts exactly over a single transmit block.

  4. – For a coded transmission and $N=M\times n/m$ : $M$ codewords are contained in a single transmit block.

• R$=\{r_{ij}:i\mathrm{,}j\in \{\mathrm{1,2,}\cdots \mathrm{,}N\left\}\right\}$ is the block transmit matrix of size $N\times N$ . R is hermitian and positive semidefinite. The block transmit matrix R contains the whole knowledge about the transmission scheme (transmit and receive filters) and the physical propagation channel between transmitter(s) and receiver(s) [18].

• $\tilde{\mathit{n}}$ is a sample function of an additive Gaussian noise vector process of length N with zero mean and covariance matrix ${\mathbf{\Phi }}_{\tilde{\mathit{n}}\tilde{\mathit{n}}}=\frac{N_0}{2}\cdot \mathit{R}$ where $\frac{N_0}{2}$ is the double-sided noise power spectral density.

• DET is the detector including equalization and decoding.

The model in Figure 1 is a general model and fits to different transmission schemes like orthogonal frequency division multiplexing (OFDM), code division multiple access (CDMA), multicarrier CDMA (MC-CDMA) and multiple-input multiple-output (MIMO). The relation with the original continuous-time (physical) model can be found in [11, 18]. The model in Figure 1 can be described mathematically as follows [11]:

$\tilde{\mathit{x}}=\mathit{R}\cdot \mathit{x}+\tilde{\mathit{n}}.$ E1

By decomposing R into a diagonal part R_d = diag_m{R} and a nondiagonal part R_\d = R−R_d, Eq. (1) can be rewritten as:

$\tilde{\mathit{x}}=\underset{\text{signal}}{\underbrace{{\mathit{R}}_d\cdot x}}+\underset{\text{interference}}{\underbrace{{\mathit{R}}_{\backslash d}\cdot \mathit{x}}}+\underset{\text{additive noise}}{\underbrace{\tilde{\mathit{n}}}}\mathrm{.}$ E2

For the j-th element of the receive vector $j\in \mathrm{\{}\mathrm{1,2,}\cdots \mathrm{,}N\mathrm{\}}$ Eq. (2) can be expressed as

${\tilde{x}}_j=r_{jj}\cdot x_j+{\displaystyle \sum _{\begin{array}{c}m=1\\ m\ne j\end{array}}^N}{r}_{jm}\cdot x_m+{\tilde{n}}_j\mathrm{.}$ E3

We notice from Eqs. (2), (3) that the nondiagonal elements of R describe the interference between the elements of the transmit vector at the receiver side. For interference-free transmission R_\d = 0. For an interference-free transmission over an additive white Gaussian noise (AWGN) channel R = I.

Figure 2 shows the channel matrix for a MIMO transmission scheme for different number of transmit/receive antennas. Figure 3 shows the channel matrix for OFDM with/without spreading. Figure 4 shows the channel matrix for MIMO-OFDM. In Figures 2–4, the darker the elements, the larger the absolute values of the entries of the corresponding matrix R, and hence larger the interference [21].

Figure 2.

Visualization of the channel matrix for a MIMO transmission scheme with eight transmit antennas and different receive antennas.

Figure 3.

Visualization of the channel matrix for OFDM with 16 subcarriers and spreading over four subcarriers with/without interleaving.

Figure 4.

Visualization of the channel matrix for a MIMO-OFDM transmission scheme with eight subcarriers and three transmit antennas.

Remark 1. For a clear distinction between channel matrix and block transmit matrix, we refer to [11, 18]. Generally speaking, the block transmit matrix R is a block diagonal matrix of “many” channel matrices.

The detector DET in Figure 1 has to deliver a vector $\widehat{\mathit{q}}$ with a minimum bit error rate compared to q (conditional to the available computational power) given that COD, $\mathcal{M}$ and R are known at the receiver side. The optimum detection (maximum likelihood detection) for realistic cases is often infeasible. Therefore, suboptimum schemes are used, mainly based on separating the detection into an equalization EQ (to cope with interference caused by ${\mathit{R}}_{\backslash d}$ ) and a decoding DEC (to utilize the redundancy added by COD). In this case, we distinguish between separate and joint equalization and decoding, cf. Figure 5. The superiority of the latter one is widely accepted: The separate equalization and decoding as in Figure 5(a) in general leads to a performance loss since the equalizer does not utilize the knowledge available at the decoder [14]. Each of the components DET, EQ and DEC can be seen as a pattern classifier. By separating the detection into equalization and decoding, an optimum detection in general cannot be achieved anymore (even if optimum equalization and optimum decoding are individually applied). Nevertheless, this is a common practice.

Figure 5.

Detection: EQ is the equalizer, DEC is the decoder, DECI is a hard decision function. Notice the feedback from the decoder to the equalizer in (b), i.e., the turbo principle.

DECI in Figure 5 is a hard decision function. For a coded transmission, DECI is a unit step function. For an uncoded transmission, COD and DEC are removed from Figure 1 and Figure 5, respectively. DECI in this case is a stepwise function depending on the symbol alphabet ψ which maps the (in general complex-valued) elements of the equalized vector $\stackrel{⌣}{\mathit{x}}$ to the vector of detected symbols $\widehat{\mathit{x}}\in {\mathit{\psi }}^N$ cf. Figure 8. The map from $\widehat{\mathit{x}}$ to $\widehat{\mathit{q}}$ is then straightforward. In summary

• For an uncoded transmission DECI: ${ℂ}^N\to {\mathit{\psi }}^N$ .

• For a coded transmission DECI: ${ℝ}^k\to {\left\{\mathrm{0,1}\right\}}^k$ .

3. Vector equalization

For an uncoded transmission, the detection DET reduces to a vector equalization EQ as shown in Figure 6.

Figure 6.

Uncoded block transmission model. Neither encoding at the transmitter nor decoding at the receiver. The detection reduces to a vector equalization EQ.

The optimum vector equalization rule (the maximum likelihood one) is based on the minimum Mahalanobis distance and is given as [21]

${\widehat{\mathit{x}}}_{\text{ML}}=\underset{\mathit{\xi }\in \mathit{\chi }}{\text{arg}\text{min}}\left\{\frac{1}{2}\cdot {\mathit{\xi }}^H\cdot \mathit{R}\cdot \mathit{\xi }-\Re \left\{{\mathit{\xi }}^H\cdot \tilde{\mathit{x}}\right\}\right\}\mathrm{.}$ E4

For each receive vector $\tilde{\mathit{x}}$ , the optimum vector equalizer calculates the Mahalanobis distance Eq. (4) to all possible transmit vectors $\mathit{\chi }$ of cardinality $2^{m\cdot N}$ and decides in favor of that possible transmit vector ${\widehat{\mathit{x}}}_{\text{ML}}$ with the minimum Mahalanobis distance to the receive vector $\tilde{\mathit{x}}$ , i.e., exhaustive search is required in general. This can be performed for small $2^{m\cdot N}$ which is usually not the case in practice. Therefore, suboptimum equalization schemes are applied, which trade-off performance against complexity.

4. Continuous-time single-layer recurrent neural networks for vector equalization

The dynamical behavior of continuous-time single-layer recurrent neural networks of dimension N′, abbreviated in the following by RNN ⁴

The abbreviation RNN in this chapter inherently includes the continuous-time and the single-layer properties.

, is given by the state-space equations [22]:

$\begin{array}{l}{\mathrm{\Upsilon }}_e\cdot \frac{\text{d}\mathit{u}(t)}{\text{d}t}=-\mathit{u}(t)+\mathit{W}\cdot \mathit{v}(t)+{\mathit{W}}_0\cdot \mathit{e},\\ \mathit{v}(t)=\mathit{\phi }(\mathit{u}(t))=[{\phi }_1(u_1(t)),{\phi }_2(u_2(t)),\cdots ,{\phi }_{N^{\prime }}(u_{N^{\prime }}(t))]T.\end{array}$ E5

In Eq. (5), ϒ_e is a diagonal and positive definite matrix of size N′ × N′. v(t) is the output, u(t) is the inner state, e is the external input. $\mathit{v}\mathrm{,}\mathit{u}\mathrm{,}\mathit{e}\in {ℂ}^{N^{\prime }}$ .φ_j (·) : j ∈ {1, 2, …, N′} is the j-th activation function. $\mathit{W}=\left\{w_{j{j}^{\prime }}:j\mathrm{,}{j}^{\prime }\in \{\mathrm{1,2,}\cdots \mathrm{,}{N}^{\prime }\}\right\}\in {ℂ}^{N^{\prime }\times N^{\prime }}$ , ${\mathit{W}}_0={\text{diag}}_v\left\{{\left[w_{10}\mathrm{,}{w}_{20}\mathrm{,}\cdots \mathrm{,}{w}_{N^{\prime }0}\right]}^T\right\}\in {ℝ}^{N^{\prime }\times N^{\prime }}$ are the weight matrices. The real-valued RNN (all variables and functions in Eq. (5) are real-valued) is shown in Figure 7, which is known as “additive model” or “resistance-capacitance model” [23]. In this case, $w_{j{j}^{\prime }}=\frac{R_j}{R_{j{j}^{\prime }}}$ is the weight coefficient between the output of the j′-th neuron and the input of the j-th neuron, $w_{j0}=\frac{R_j}{R_{j0}}$ is the weight coefficient of the j-th external input. We also notice that the feedback W ⋅ v in Eq. (5) and Figure 7 is a linear function of the output v. Moreover, ϒ_e can be given in this case as ϒ_e$={\text{diag}}_v\left\{{\left[R_1\cdot C_1\mathrm{,}{R}_2\cdot C_2\mathrm{,}\dots \mathrm{,}{R}_{N^{\prime }}\cdot C_{N^{\prime }}\right]}^T\right\}$ .

Figure 7.

Continuous-time single-layer real-valued recurrent neural network. v(t) is the output, u(t) is the inner state, e is the external input and φ(⋅) is the activation function. This model is known as “additive model” or “resistance-capacitance model” [23].

As a nonlinear dynamical system, the stability of the RNN is of primary interest [16]. This has been proven under specific conditions by Lyapunov’s stability theory in [24] for real-valued RNN and in [22, 25] for complex-valued ones, among others. The RNN in Eq. (5) represents a general purpose structure. Based on N′, φ, W, W₀ a wide range of optimization problems can be solved. First and most well-investigated applications of the RNN include the content addressable memory [24, 26], analog-to-digital converter (ADC) [27] and the traveling salesman problem [28]. In all these cases, no training is needed since the RNN is acting as a dynamical solver. This feature is desirable in many engineering fields like signal processing, communications, automatic control, etc., and has first been exploited by Hopfield in his pioneering work [24, 29], where information has been stored in a dynamically stable RNN. We focus in the following on the vector equalization.

Remark 2. The dimension of a real-valued RNN is the same as the number of neurons.

Remark 3. Two real-valued RNNs each of N′ neurons are required to represent one complex-valued RNN (with dimension N′). This is possible by separating Eq. (5) into real and imaginary parts. However, this doubles in general the number of connections per neuron (and hence the number of multiplications) because of the required connections (represented by W_i) between the two real-valued RNNs as it can be seen from the following equation:

${\mathbf{\Upsilon }}_e\cdot \frac{\text{d}}{\text{d}t}\left[\begin{array}{c}{\mathit{u}}_r(t)\\ {\mathit{u}}_i(t)\end{array}\right]=-\left[\begin{array}{c}{\mathit{u}}_r(t)\\ {\mathit{u}}_i(t)\end{array}\right]+\left[\begin{array}{cc}{\mathit{W}}_r& -{\mathit{W}}_i\\ {\mathit{W}}_i& {\mathit{W}}_r\end{array}\right]\cdot \left[\begin{array}{c}{\mathit{v}}_r(t)\\ {\mathit{v}}_i(t)\end{array}\right]+\left[\begin{array}{cc}{\mathit{W}}_0& 0\\ 0& {\mathit{W}}_0\end{array}\right]\cdot \left[\begin{array}{c}{\mathit{e}}_r\\ {\mathit{e}}_i\end{array}\right]\mathrm{.}$ E6

ϒ_e in this case is a diagonal positive definite matrix of size 2 · N′ × 2 · N′ and

$\begin{array}{ccc}\mathit{u}(t)={\mathit{u}}_r(t)+\iota {\mathit{u}}_i(t)& \mathrm{,}& \mathit{e}={\mathit{e}}_r+\iota {\mathit{e}}_i\\ \mathit{v}(t)={\mathit{v}}_r(t)+\iota {\mathit{v}}_i(t)& \mathrm{,}& \mathit{W}={\mathit{W}}_r+\iota {\mathit{W}}_i.\end{array}$

A. Vector equalization based on RNN

The usage of the RNN for vector equalization became known for multiuser interference cancellation in CDMA environments [30, 31]. However, this was limited to the binary phase-shift keying (BPSK) symbol alphabet ψ = {−1, +1}. This has been generalized to complex-valued symbol alphabets in [21] by combining the results of references [20], [22], [32] ⁵

For discrete-time single-layer recurrent neural networks for vector equalization, we refer to references [19, 33].

. Based thereon, it has been proven that the RNN ends in a local minimum of Eq. (4) if the following relations are fulfilled [21], cf. Eqs. (1), (2), (5) and Figures 6 and 7.

$\begin{array}{ccc}\mathit{e}=\tilde{\mathit{x}}& \mathit{v}=\stackrel{⌣}{\mathit{x}}& {\mathit{N}}^{\text{'}}=\mathit{N}\\ {\mathit{W}}_0={\mathit{R}}_d^{-1}& \mathit{W}=\mathit{I}-{\mathit{R}}_d^{-1}\cdot \mathit{R}& \mathit{\phi }(\cdot )={\mathit{\theta }}^{(opt)}(\cdot )\end{array}$ E7

and therefore $\widehat{\mathit{x}}=\text{DECI}(\mathit{v})$ . Figure 8

shows an example of an eight quadrature amplitude modulation (8 QAM) symbol alphabet and its corresponding DECI function. The relations in Eq. (7) are obtained by the comparison between the maximum likelihood function of the vector equalization and the Lyapunov function of the RNN.

Figure 8.

An example of an 8 QAM symbol alphabet and its corresponding DECI function. Each element of the symbol alphabet (marked with ×) has its own “decisions region” visualized by different colors. The function DECI delivers that element of the symbol alphabet, where the input argument lies in its corresponding decision region.

The dynamical behavior of the vector equalization based on RNN can be given as, cf. Eqs. (1), (5), (7)

$\begin{array}{c}{\mathbf{\Upsilon }}_e\cdot \frac{\text{d}\mathit{u}\left(t\right)}{\text{d}t}=-\mathit{u}\left(t\right)+\stackrel{⌣}{\mathit{x}}\left(t\right)+{\mathit{R}}_d^{-1}\cdot \mathit{R}\cdot \left[\mathit{x}-\stackrel{⌣}{\mathit{x}}(t)\right]+{\mathit{R}}_d^{-1}\cdot \tilde{\mathit{n}}\mathrm{,}\\ \stackrel{⌣}{\mathit{x}}\left(t\right)={\mathit{\theta }}^{(opt)}\left(\mathit{u}\left(t\right)\right)={\left[{\mathit{\theta }}_1^{(opt)}\left({\mathit{u}}_1\left(t\right)\right)\mathrm{,}{\mathit{\theta }}_2^{(opt)}\left({\mathit{u}}_2\left(t\right)\right)\mathrm{,}\cdots \mathrm{,}{\mathit{\theta }}_N^{(opt)}\left({\mathit{u}}_N\left(t\right)\right)\right]}^T\mathrm{.}\end{array}$ E8

The locally asymptotical stability of Eq. (8) based on Lyapunov functions has been proved in [21] (based on [22]) for separable symbol alphabets ψ^(s). When Eq. (8) reaches an equilibrium point u_ep, i.e., $\frac{\text{d}\mathit{u}(t)}{\text{d}t}{=\mathbf{0}}_N\Rightarrow \mathit{u}={\mathit{u}}_{ep}$ , Eq. (8) can be rewritten as

${\mathit{u}}_{ep}={\stackrel{⌣}{\mathit{x}}}_{ep}+{\mathit{R}}_d^{-1}\cdot \mathit{R}\cdot \left[\mathit{x}-{\stackrel{⌣}{\mathit{x}}}_{ep}\right]+{\mathit{R}}_d^{-1}\cdot \tilde{\mathit{n}}.$ E9

If additionally, a correct equalization is achieved, i.e., ${\stackrel{⌣}{\mathit{x}}}_{ep}=\mathit{x}$ , the inner state is

${\mathit{u}}_{ep}=\mathit{x}+\underset{{\mathit{n}}_e}{\underbrace{{\mathit{R}}_d^{-1}\cdot \tilde{\mathit{n}}}}\mathrm{.}$ E10

Thus, the RNN as vector equalizer, Eq. (8) acts as “analog dynamical solver” and there is no need for a training. The covariance matrix of n_e is ${\mathbf{\Phi }}_{{\mathit{n}}_e{\mathit{n}}_e}=\frac{N_0}{2}\cdot {\mathit{R}}_d^{-1}\cdot \mathit{R}\cdot {\mathit{R}}_d^{-1}$ . We define

${\mathbf{\Sigma }}_{{\mathit{n}}_e}={\text{diag}}_v\left\{{\left[{\sigma }_1^2\mathrm{,}{\sigma }_2^2\mathrm{,}\cdots \mathrm{,}{\sigma }_N^2\right]}^T\right\}={\text{diag}}_m\left\{{\mathbf{\Phi }}_{{\mathit{n}}_e{\mathit{n}}_e}\right\}=\frac{N_0}{2}\cdot {\mathit{R}}_d^{-1}\mathrm{.}$ E11

In Eq. (7), θ^(opt) (·) is the optimum activation function and depends on the symbol alphabet ψ. For BPSK (a real-valued case)

${\theta }^{(opt)}\left(u\right)=\tanh \left(\frac{u}{{\sigma }^2}\right)\mathrm{.}$ E12

where σ² is given in Eq. (11).

Remark 4. For separable symbol alphabets,
$\mathit{\psi }={\mathit{\psi }}^{(s)}\Rightarrow {\theta }^{(opt)}\left(u=u_r+\iota u_i\right)={\theta }_r^{(opt)}\left(u_r\right)+\iota {\theta }_i^{(opt)}\left(u_i\right)$ [19].

B. Analog hardware implementation aspects: equalization

The analog signal processing as a matter of topical importance for modern receiver architectures was recognized in [34], where an analog vector equalizer—designed in BiCMOS technology—was considered as a promising application for the analog processing of baseband signals. The equalizer accepts sampled vector symbols in analog form with an advantage that the equalizer does not require an ADC at the input interface. At very high data rates, the exclusion of an ADC softens the trade-off between chip area requirement and overall power consumption. We discuss in the following section the main features/challenges of the analog implementation of the vector equalizer based on RNN.

Structure: An RNN of dimension N′ (in general 2 ·N′ neurons) is capable to act as a vector equalizer as long as the block size at the transmitter side N (over all possible symbol alphabets, coding schemes and block sizes) is as maximum as N′, i.e., N ≤ N′.

Activation function: The definition of the optimum activation function θ^(opt)(·) is not general, but depends on the symbol alphabet under consideration. Different symbol alphabets need different activation functions. However, we have proven in [20] that for square QAM symbol alphabets—the most relevant ones in practice—θ^(opt) (·) can be approximated as a sum of a limited number of shifted and weighted hyperbolic tangent functions. Square QAM symbol alphabets are separable ones, cf. Remark 4. The analog implementation of the hyperbolic tangent well befits the large-signal transfer function of transconductance stages based on bipolar differential amplifiers [3, 34].

Adaptivity: A vector equalizer must be capable to adapt to different and time-variant interference levels. The adaptivity is regulated by the measurement of the block transmit matrix R, a task performed by a “channel estimation unit” (CEU). The weight matrices W and W₀ are then computed as in Eq. (7) and forwarded to the RNN (Figure 9). Thus, the weight matrices W and W₀ are not the outcome of any training algorithm but related directly to R, cf. Eq. (7). This represents a typical example for the mixed-signal integrated circuit, where the weight coefficients are (obtained and) stored digitally, converted into analog values, later used as weight coefficients for the analog RNN [8].

Figure 9.

Uncoded block transmission model. The detection reduces to a vector equalization EQ. The channel estimation unit (CEU) estimates the block transmit matrix R.

For the j-th neuron in the additive model Figure 7, the ratio between two resistors R_j and R_jj′ (R_j and R_j0) is used to configure each weight coefficient w_jj′ (w_j0). According to the additive model, R_jj′ and R_j0 can assume both positive and negative values, and the absolute value theoretically extends from R_j to infinite (for w_jj′ ∈ [−1, +1]). This puts serious limitations to the direct implementation of the model. In [3], we showed how this difficulty can be overcome by using a Gilbert cell as a four-quadrant analog multiplier. A Gilbert cell [35] is composed of two pairs of differential amplifiers with cross-coupled collectors, and is controlled by a differential voltage input G_ji applied at the base gate of the transistors. When biased with a differential tail current ${\mathit{I}}_{ji}={\mathit{I}}_{ji}^{+}-{\mathit{I}}_{ji}^{-}$ , the differential output current ${\mathit{I}}_{ji\mathrm{,}w}={\mathit{I}}_{ji\mathrm{,}w}^{+}-{\mathit{I}}_{ji\mathrm{,}w}^{-}$ is a fraction w of the tail current I_ji, as a function of the input voltage G_ji:

$I_{ji\mathrm{,}w}={\mathit{I}}_{ji\mathrm{,}w}^{+}-{\mathit{I}}_{ji\mathrm{,}w}^{-}=f_{Gc}\left({\mathit{I}}_{ji}={\mathit{I}}_{ji}^{+}-{\mathit{I}}_{ji}^{-}\mathrm{,}{G}_{ji}\right)=w\cdot {\mathit{I}}_{ji}\in \left[-{\mathit{I}}_{ji}\mathrm{,}+{\mathit{I}}_{ji}\right]\mathrm{.}$ E13

Accuracy: Locally asymptotical Lyapunov stability can be guaranteed for the RNN in Eqs. (5), (8) if, among others, the hermitian property is verified for the weight matrix W (the symmetric property in the real-valued case). Inaccuracies in the weights’ representation may jeopardize the Lyapunov stability and impact the performance of the vector equalizer. The first cause of weights’ inaccuracy may arise from the limited accuracy of the analog design in terms of components’ parasitics, devices’ mismatch, process variation, just to name a few. Those inaccuracies (if modest) are expected to slightly degrade the performance without causing a catastrophic failure, thanks to the high nonlinearity of the equalization algorithm. Moreover, it has been shown in [8], [36] that in some cases, they produce beneficial effects: These imperfections incorporate some kind of simulated annealing which enables escaping local minima by allowing occasionally “uphill steps” since the Lyapunov stable RNN is a gradient-like system. This feature is emulated in discrete-time by stochastic Hopfield networks [23]. Non-precision of the weights may also arise from an insufficient resolution of the digital-to-analog converter (DAC) (Figure 9). On the other hand, an overzealous DAC design increases the chip area, the power consumption and adds complexity to the interface between the analog vector equalizer and the digital CEU. In this case, a conservative approach suggests to use a DAC with enough resolution to match the precision used by the CEU.

Interneuron connectivity and reconfigurability: Scaling the architecture of an analog VLSI design is not straightforward. A vector equalizer based on recurrent neural networks is composed by the repetition of equal sub-systems, i.e., the neurons. Using a bottom-up approach, the first step to scale the system involves the redesign of the single neuron in order to handle more feedback inputs. In a successive step, the neurons are connected together and a system-level simulation is performed to check the functionality of the system. However, several design choices must be made during the process and it is not guaranteed that the optimum architecture for a certain number of neurons is still the best choice when the number of neurons changes. For large N, the block transmit matrix R, defining the weight matrix W, is usually sparse. If a maximum number of nonzero elements over the rows of R is assumed, the requirement for a full connectivity between the neurons in Figure 7 can be relaxed, and only a maximum number of connections per neuron will be necessary. In this case, however, in addition to the “adaptivity”, the RNN must be reconfigured according to the position of the nonzero elements in R. The hardware simplification given by the partial connectivity may be counterbalanced by the necessity of a further routing (e.g., multiplexing/demultiplexing) of the feedback. For special cases, where the block transmit matrix can be reordered around the diagonal, more independent RNNs can be simply used in parallel. In Figures 3(b) and 3(c), four independent RNNs, each of dimension four, can be used in parallel. Additionally, for specific transmission schemes such as MIMO-OFDM in Figure 4, the connectivity can be assumed limited (number of transmit antennas minus one) and fixed (crosstalk only between same subcarriers, when used simultaneously on different transmit antennas).

Example 1. In Figure 4, eight RNNs (number of subcarriers) each of dimension of three (number of transmit antennas) can be used in parallel. Each neuron has two feedback inputs.

C. Circuit design

We review here the main features of the analog circuit design of an RNN as vector equalizer working with the BPSK symbol alphabet and composed of four neurons. Detailed explanation can be found in reference [3]. The RNN is realized in IHP 0.25 μm SiGe BiCMOS technology (SG25H3). A simplified schematic of a neuron is shown in Figure 10. Schematics of gray boxes are presented in Figure 11.

Figure 10.

A simplified schematic of a single neuron as a part of a (four neurons) RNN analog vector equalizer. ${u^{\prime }}_j$ is the inner state, ${e^{\prime }}_j$ is the external input and $G_{ji}$ is used for adapting the weight coefficient $w_{ji}$ from the output of the i-th neuron to the input of the j-th neuron. The circuit is fully differential [3].

Figure 11.

Details of the circuit building blocks. Gilbert cell used as a four-quadrant analog multiplier, buffer stages, BJT differential pairs for the generation of the hyperbolic tangent function and a metal-oxide-semiconductor field-effect transistor (MOSFET) switch used as a sequencer [3].

The dynamical behavior of the circuit in Figures 10 and 11 is described as [3]

$\begin{array}{ccc}\mathbf{\Upsilon }\cdot \frac{\text{d}{\mathit{u}}^{\prime }\left(t\right)}{\text{d}t}& =& -{\mathit{u}}^{\prime }\left(t\right)+\mathit{W}\cdot {\mathit{v}}^{\prime }\left(t\right)+{\mathit{W}}_0\cdot {\mathit{e}}^{\prime }\mathrm{,}\\ \frac{R\cdot I_t}{N-1}\cdot \tanh \left(\frac{{\mathit{u}}^{\prime }\left(t\right)}{2\cdot V_t}\right)& =& {\mathit{v}}^{\prime }(t)\mathrm{,}\\ \tau \cdot \mathit{I}& =& \mathbf{\Upsilon }\mathrm{.}\end{array}$ E14

which is equivalent to Eq. (5). τ = R · C is the time constant of the circuit. R is shown in Figure 10 and C is a fictitious capacitance between the nodes and $u_j^{+}$ and $u_j^{-}$ .V_t is the thermal voltage and I_t is the tail current in Figure 11. The circuit is fully differential and the differential currents and voltages are denoted as, cf. Figures 10 and 11:

$\begin{array}{ccc}{\mathit{I}}_{ji}={\mathit{I}}_{ji}^{+}-{\mathit{I}}_{ji}^{-}& {\mathit{I}}_j={\mathit{I}}_j^{+}-{\mathit{I}}_j^{-}& {\mathit{I}}_o={\mathit{I}}_o^{+}-{\mathit{I}}_o^{-}\mathrm{,}\\ {\mathit{I}}_{ji\mathrm{,}w}={\mathit{I}}_{ji\mathrm{,}w}^{+}-{\mathit{I}}_{ji\mathrm{,}w}^{-}& {{\mathit{u}}^{\prime }}_j={\mathit{u}}_j^{+}-{\mathit{u}}_j^{-}& {{\mathit{e}}^{\prime }}_j={\mathit{e}}_j^{+}-{\mathit{e}}_j^{-}\end{array}$ E15

(1) Performance: Simulation results based on the above described analog RNN are shown in Figure 12. The interference is described by the channel matrix R_test.

Figure 12.

BER vs. $E_b/N_0$ for the analog RNN vector equalizer. Evolution time equals $10\cdot \tau$ . BPSK symbol alphabet and channel matrix ${\mathit{R}}_{test}$ .

${\mathit{R}}_{test}=\left[\begin{array}{cccc}1& 0.24& -0.34& -0.57\\ 0.24& 1& 0.32& 0.29\\ -0.34& 0.32& 1& 0.25\\ -0.57& 0.29& 0.25& 1\end{array}\right]$

The black dashed line shows the bit error rate (BER) for a BPSK symbol alphabet in an AWGN channel (an interference-free channel). Performance achieved by the maximum likelihood algorithm in Eq. (4) is included as a solid black line. The performance of the analog RNN vector equalizer ⁶

Analog RNN vector equalizer refers to the described analog hardware-implemented RNN for vector equalization.

is presented in a solid red line with square markers. Compared to the optimum algorithm, the signal-to-noise ratio (SNR) loss for the analog RNN vector equalizer can be quantified in approximately 1.7 dB at a BER of 10⁻⁴. This loss in SNR emphasizes the suboptimality of the RNN as vector equalizer and depends on the channel matrix. Figure 13 shows an example of a transient simulation for the analog RNN vector equalizer. The time constant is approximately τ = 40 ps. The SNR ratio is set to 2 dB and a series of three receive vectors are equalized in sequence. Because of the channel matrix and noise, the sampled vectors at the input of the equalizer $\tilde{\mathit{x}}$ present different signs and values, compared to the sent vectors x (shown in square brackets). The equalization of each receive vector lasts $10\cdot \tau$ . First half of this interval (evolution time) is used to reach a stable state, while the second half of the interval (reset time) is used to return to a predefined inner state (all-zero state) before the equalization of a new vector starts. At the end of the evolution time, a decision is made based on the sign of the output vector (the decision function DECI for BPSK is a sign function). In our example, a comparison between the sent and the recovered bits shows an error of one bit out of twelve, equivalent to a $\text{BER}\approx \frac{1}{12}$ , a result in line with the BER shown in Figure 12.

Figure 13.

An example of a transient simulation for the analog RNN vector equalizer.

(a) Inputs $\tilde{\mathit{x}}$ (b) Outputs: soft decisions $\stackrel{⌣}{\mathit{x}}$ and hard decisions $\stackrel{⌣}{\mathit{x}}$ .

Remark 5. The evolution and reset times are the two limiting factors for the maximum throughput of the analog RNN vector equalizer. However, they cannot be unlimitedly minimized since the RNN needs a minimum evolution time to reach an equilibrium point representing a local minimum of the Lyapunov function, i.e., a local minimum of Eq. (4).

(2) Energy efficiency: The energy efficiency of a hardware “architecture” is the ratio between the power requirement (Watt) of the architecture and its achievement in a given time period. In our case, the throughput of the equalizer represents the achievement. Combining the value of τ and the power consumption, the abovementioned analog vector equalizer is expected to win the competition versus common digital signal processing, thanks to three to four orders of magnitude better energy efficiency [3].

5. Channel coding

Channel coding (including encoding at the transmitter side COD and decoding at the receiver side DEC) aims to enable an error-free transmission over noisy channels with maximum possible transmit rate. This is done by adding redundancy (extra bits) at the transmitter side, i.e., the bijective map from q to q_c (Figure 14,) such that the codewords q_c are sufficiently distinguishable at the receiver side even if the noisy channel corrupts some bits during the transmission. Figure 14 shows a coded transmission over an AWGN channel.

For every received codeword, the optimum decoding (the maximum likelihood one) needs to calculate the distance between the received codeword and all possible codewords $\mathcal{C}$ , which makes it infeasible for realistic cases (except for convolutional codes which are not considered here). We focus on binary LDPC codes and their corresponding suboptimum decoding algorithm: the belief propagation with BPSK symbol alphabet. LDPC codes [37] belong to the class of binary linear block codes and have been shown to achieve an error rate very close to the Shannon limit (a performance lower bound) for the AWGN channel and have been implemented in many practical systems such as the satellite digital video broadcast (DVB-S2) [38]. A binary linear block code is characterized by a binary parity check matrix H of size (n − k) × n for n > k.

Figure 14.

Coded transmission over an BER channel.

6. Continuous-time single-layer high-order recurrent neural networks for belief propagation

One of the largest drawbacks of RNNs is their quadratic Lyapunov function [39]. Optimization problems associated with cost functions of higher degree cannot be solved “satisfactorily” by RNNs. Increasing the order of the Lyapunov function leads to a nonlinear feedback in the network. In doing so, we obtain the single-layer high-order recurrent neural network, named differently in literature, depending on the nonlinear feedback [39–42].

Remark 6. High-order recurrent neural networks are in the literature exclusively real-valued.

Figure 15 shows the continuous-time single-layer high-order recurrent neural network, abbreviated in the following by HORNN ⁷

The abbreviation HORNN in this chapter inherently includes the continuous-time, single-layer and real-valued properties.

.

The dynamical behavior is given by

$\begin{array}{ccc}{\mathbf{\Upsilon }}_d\cdot \frac{\text{d}\check{\mathit{u}}\left(t\right)}{\text{d}t}& =& -\check{\mathit{u}}\left(t\right)+\check{\mathit{W}}\cdot \check{\mathit{f}}\left(\check{\mathit{v}}\left(t\right)\right)+{\check{\mathit{W}}}_0\cdot \check{\mathit{e}}\mathrm{,}\\ \check{\mathit{v}}\left(t\right)& =& \check{\mathit{\phi }}\left(\check{u}\left(t\right)\right)={\left[{\check{\phi }}_1\left({\check{u}}_1\left(t\right)\right)\mathrm{,}{\stackrel{⌣}{\phi }}_2\left({\check{u}}_2\left(t\right)\right)\mathrm{,}\cdots \mathrm{,}{\check{\phi }}_{\stackrel{⌣}{n}}\left({\check{u}}_{\check{n}}\left(t\right)\right)\right]}^T\mathrm{,}\\ {}_{\mathbf{\Upsilon }}& =& {\text{diag}}_v\left\{{\left[{\check{R}}_1\cdot {\check{C}}_1\mathrm{,}{\check{R}}_2\cdot {\check{C}}_2\mathrm{,}\dots \mathrm{,}{\check{R}}_{\check{n}}\cdot {\check{C}}_{\check{n}}\right]}^T\right\}\mathrm{.}\end{array}$ E16

Figure 15.

Continuous-time single-layer real-valued high-order recurrent neural network. $\check{v}(t)$ is the output, $\check{u}(t)$ is the inner state, $\check{e}$ is the external input and $\check{\phi }(\cdot )$ is the activation function. $\check{\mathit{f}}\left(\check{\mathit{v}}\right)$ is a real-valued continuously differentiable vector function with $\check{\mathit{f}}({\mathbf{0}}_{\check{n}})={\mathbf{0}}_{\check{n}}$ [21].

The parameters in Eq. (16) can be linked to Figure 15 in the same way as Eq. (5) linked to Figure 7. $\check{\mathit{f}}\left(\check{\mathit{v}}\right)$ is a real-valued continuously differentiable vector function. In addition, $\check{\mathit{f}}({\mathbf{0}}_{\check{n}})={\mathbf{0}}_{\check{n}}$ . It is worth mentioning that the term “high-order” in this case refers to the interconnections between the neurons rather than the degree of the differential equation describing the dynamics. As for RNNs, this is still of first order, cf. Eq. (16).

Remark 7. In the special case $\check{\mathit{f}}(\check{\mathit{v}})=\check{\mathit{v}}$ , the HORNN reduces to the (real-valued) RNN.

In order to apply HORNNs to solve optimization tasks, their stability has to be investigated. A property without which the behavior of dynamical systems is often suspected [39]. This was the topic of many publications [39–42]. A common denominator of the locally asymptotical stability proof of the HORNN based on Lyapunov functions is

• $$\check{\phi }\mathrm{(}\cdot \mathrm{)}$$ is continuously differentiable and a strictly increasing function.

• The right side of the first line of Eq. (16) can be rewritten as a gradient of a scalar function.

Figure 16.

Tanner graph of the systematic Hamming code n = 7 and k = 4.

A. Belief propagation based on HORNN

Originally proposed by Gallager [37], belief propagation is a suboptimum graph-based decoding algorithm for LDPC codes. The corresponding graph is bipartite (n parity nodes and n – k check nodes) and known as Tanner graph [43]. This is shown in Figure 16 for the Hamming code with the parity check matrix $H_{\text{Hamming}}$ Eq. (17) where n = 7, k = 4. The belief propagation algorithm iteratively exchanges “messages” between parity and check nodes.

$H_{\text{Hamming}}=\left[\begin{array}{cccc}0& 1& 1& 1\\ 1& 0& 1& 1\\ 1& 1& 0& 1\end{array}|\begin{array}{ccc}1& 0& 0\\ 0& 1& 0\\ 0& 0& 1\end{array}\right]$ E17

For every binary linear block code characterized by the binary parity check matrix H of size (n − k) × n for n > k, three binary matrices $P_{n_h\times n_h}$ , $S_{n_h\times n_h}$ and $B_{n_h\times n}$ can be uniquely defined [44, 45] such that Eq. (16) and Figure 15 perform continuous-time belief propagation if the following relations are fulfilled:

$\check{\mathit{u}}=L,$ E18a

$\check{\mathit{e}}=B\cdot L_{ch},$ E18b

$\check{\phi }(\cdot )=\tanh \left(\frac{\cdot }{2}\right)\mathrm{,}$ E18c

$\check{\mathit{v}}=\check{\mathit{\phi }}(L),$ E18d

$\check{W}=P,$ E18e

${\check{W}}_0=I_{n_h\times n_h},$ E18f

${\check{f}}_j=2\cdot \text{atanh}\left\{{\displaystyle \mathbf{\prod }_{j^{\prime }\in \text{pos}\left[\mathit{S}(j,:)=1\right]}}{\check{v}}_{j^{\prime }}\right\}\text{for}j\mathrm{,}{j}^{\prime }\in \left\{\mathrm{1,}\mathrm{2,}\dots \mathrm{,}{n}_h\right\}\mathrm{,}$ E18g

$\check{n}=n_h\mathrm{.}$ E18h

In Eq. (18) ⁸

L, $\stackrel{⌣}{\mathit{L}}$ and ${\mathit{L}}_{ch}$ are vectors.

,

• $k$ is the length of the information word (q in Figures 1 and 14).

• n is the length of the codeword (${\mathit{q}}_c$ in Figures 1 and 14).

• $n_h{=\mathbf{1}}_{\left(1\times \left(n-k\right)\right)}^T\cdot H\cdot {\mathbf{1}}_{\left(n\times 1\right)}\in \left[k+\mathrm{1,}n\cdot \left(n-k\right)\right]$ is the number of nonzero elements in H.

• $L_{ch\mathrm{,}(n\times 1)}$ is the vector of intrinsic log-likelihood ratio (LLR), which depends on the transition probability of the channel. For $q_{c\mathrm{,}j}$ (the j-th element of $q_c$ for $j\in \{\mathrm{1,2,}\cdots \mathrm{,}n\}$ ) it is given as

  $L_{ch,j}=\text{ln }\frac{p\dot{x}j={\stackrel{}{\tilde{x}}}_j|q_{c,j}=0)}{p({\dot{x}}_j={\stackrel{}{\tilde{x}}}_j|q_{c,j}=1).}$ E19

  In the last relation, ${\dot{x}}_j$ is the variable of the conditioned probability density function $p\left({\dot{x}}_j|q_{c\mathrm{,}j}\right)$ . ln(⋅) is the natural logarithm. For an AWGN channel, $\mathcal{N}(0,{\sigma }_n^2):L_{ch\mathrm{,}j}=\frac{{\tilde{x}}_j}{2\cdot {\sigma }_n^2}$.

• $L_{(n_h\times 1)}$ is the “message” sent from the variable nodes to the check nodes.

• ${\check{f}}_{(n_h\times 1)}$ is the “message” sent from check nodes to variable nodes.

• $I_{(n_h\times n_h)}$ is an identity matrix of size $n_h\times n_h$ .

• $\text{pos[}S(j,:)=1]$ delivers the positions of the nonzero elements in the j-th row of the matrix S.

The dynamical behavior of belief propagation can be described based on Eqs. (16), (18) and Figures 14, 15, and 16 [45]

$\begin{array}{ll}{\mathbf{\Upsilon }}_d\cdot \frac{\text{d}L\left(t\right)}{\text{d}t}\hfill & =-L\left(t\right)+P\cdot \check{\mathit{f}}\left(\check{\mathit{v}}\left(t\right)\right)+\mathit{B}\cdot L_{ch}\mathrm{,}\hfill \\ \check{\mathit{v}}(t)\hfill & =\mathbf{tanh}\left(\frac{L\left(t\right)}{2}\right)\mathrm{,}\hfill \\ \stackrel{⌣}{L}\left(t\right)\hfill & =B^T\cdot \check{f}\left(\check{\mathit{v}}\left(t\right)\right)+L_{ch}\mathrm{.}\hfill \end{array}$ E20

$\stackrel{⌣}{L}(t)$ is the soft-output of the decoding algorithm, cf. Figures 5 and 14. The discrete-time description is given as [44]

$\begin{array}{ll}L[l+1]\hfill & =P\cdot \check{\mathit{f}}\left(\check{\mathit{v}}[l]\right)+\mathit{B}\cdot L_{ch}\mathrm{,}\hfill \\ \check{\mathit{v}}\left[l\right]\hfill & =\mathbf{tanh}\left(\frac{L\left[l\right]}{2}\right)\mathrm{,}\hfill \\ \stackrel{⌣}{\mathit{L}}\left[l\right]\hfill & =B^T\cdot \check{f}\left(\check{\mathit{v}}[l]\right)+L_{ch}\mathrm{.}\hfill \end{array}$ E21

B. Dynamical behavior of belief propagation

In a series of papers, Hemati et. al. [12, 17, 46–49] also modeled the dynamics of analog belief propagation as a set of first-order nonlinear differential equations Eq. (20). This was motivated from a circuit design aspect, where ${\mathbf{\Upsilon }}_d$ (the same is valid for ϒ_e) can be seen as a bandwidth limitation of the analog circuit, realized taking advantage of the low-pass filter behavior of transmission lines Figure 17. We have shown in [45] that the model in Figure 17 also has important dynamical properties when compared with the discrete-time belief propagation Eq. (21) [44]. Particularly, the equilibrium points of the continuous-time belief propagation of Eq. (20) coincide with the fixed points of the discrete-time belief propagation of Eq. (21). This has been proved in [45]. In both cases

$L_{ep}=P\cdot \check{f}\left(\mathbf{tanh}\left(\frac{L_{ep}}{2}\right)\right)+B\cdot L_{ch}\mathrm{.}$ E22

The absolute stability of belief propagation Eqs. (20), (21) was proven for repetition codes (one of the simplest binary linear block codes) in [44, 45]. In this case

${\stackrel{⌣}{L}}_{ep}{=\mathbf{1}}_{(n\times n)}\cdot L_{ch}\mathrm{.}$ E23

Figure 17.

A simple model for analog decoding as presented in [46].

Far away from repetition codes, it has been noticed that iterative decoding algorithms (belief propagation is one of them) exhibit depending on the SNR a wide range of phenomena associated with nonlinear dynamical systems such as existence of multiple fixed points, oscillatory behavior, bifurcation, chaos and transit chaos [50]. Equilibrium points are reached at “relatively” high SNR. The analysis in reference [50] is limited to the discrete-time case.

Remark 8. The HORNN in Figure 15, Eqs. (18), (20) for belief propagation acts as a computational model.

C. Analog hardware implementation aspects: decoding

Many analog hardware implementation aspects have been already mentioned in Section 4-B. We mention here only additional aspects exclusively related to the analog belief propagation based on HORNN.

Structure: In practice, different coding schemes (different parity check matrices H) with various (k, n) constellations are applied to modify the code rate $r_c=k/n$ depending on the channel state. The HORNN in Figure 15 is capable to act as a continuous-time belief propagation (decoder) as long as the number of neurons $\check{n}$ in Figure 15 equals (or is larger than) the maximum number of nonzero elements over all parity check matrices and all ($k\mathrm{,}n$ ) constellations, i.e., $\check{n}\ge {{\displaystyle \max }}_{\mathit{H}}{n}_h$ .

Adaptivity: No training is needed. ${\check{\mathit{W}}}_0$ and ${\check{\mathit{W}}}_{}$ are directly related to the parity check matrix H. In contrast to the analog RNN vector equalizer, the weight coefficients are binary, i.e., the weight matrices ${\check{\mathit{W}}}_0$ and ${\check{\mathit{W}}}_{}$ define a feedback to be either existent or not. In such a case for Figure 15, ${\check{R}}_{j{j}^{\prime }}$ , ${\check{R}}_{j0}\in \left\{{\check{R}}_j\mathrm{,}\infty \right\}$ . Moreover, there is no need for high-resolution DAC for the weight coefficients.

Interneuron connectivity: No full connection is needed since the matrix P for LDPC codes is sparse. The number of connections per neuron must equal the maximum number of nonzero elements in P row-wise over all considered coding schemes and equals ${\max }_{\mathit{H}}P\cdot 1_{\left(n_h\times 1\right)}$ . If this is fulfilled and if interneuron connectivity control is available, the structure in Figure 15 becomes valid for all considered coding scheme.

Vector function connectivity: For different coding schemes, the number of the arguments ${{\check{\mathit{v}}}^{\prime }}_j$ to evaluate the function ${\check{f}}_j$ changes, cf. Eq. (18g). The maximum number of the arguments depends on the number of the nonzero elements in S row-wise and equals ${\max }_{\mathit{H}}S\cdot 1_{\left(n_h\times 1\right)}$ . Thus, implementing the function ${\check{f}}_j$ according to this maximum number enables evaluating the function ${\check{f}}_j$ for all considered coding schemes.

Remark 9. For a specific coding scheme, the interneuron connectivity can be made fixed. The resulted HORNN structure in this case is valid also for all codeword lengths resulted after performing a puncturing of the original code.

Remark 10. Both, the interneuron connectivity and the weight adaptation play a significant role, in the equalization as well as in the decoding. It can safely be said that they represent the major challenge of the circuit, since the analog circuit must be capable to perform equalization and decoding for a given number of possible combinations of block size, symbol alphabet, coding scheme, etc. Particularly for the decoding, the advantage of having a non-full connectivity is counterbalanced by a double (and very complex) (de)multiplexing of the signals (once for the vector function $\check{f}$ and once for the interneuron connectivity).

7. Joint equalization and decoding

Figure 18.

Two examples for joint equalization and decoding. Notice the feedback from the decoder to the equalizer, i.e., turbo principle.

Figure 19.

Joint equalization and decoding as described in [52]. ${\mathit{L}}_{ext}$ represents the “knowledge” obtained by exploiting the redundancy of the code.

Turbo equalization is a joint iterative equalization and decoding scheme. In this case, a symbol-by-symbol maximum aposteriori probability (s/s MAP) equalizer exchanges in an iterative way reliability values L with a (s/s MAP) decoder [51, 52]. This concept is inspired from the decoding concept of turbo codes, where two (s/s MAP) decoders exchange iteratively reliability values [53]. Despite its good performance, the main drawback of the turbo equalizer is the very high complexity of the s/s MAP-equalizer for multipath channels with long impulse response (compared with symbol duration) and/or symbol alphabets with large cardinality. Therefore, a suboptimum equalization (and a suboptimum decoding) usually replace the s/s-MAP ones (Figure 18).

One discrete-time joint equalization and decoding approach has been introduced in [52] and is shown in Figure 19. $\tilde{\mathit{x}}$ , ${\mathit{R}}_d$ and R are as in Eq. (2) and z⁻¹ is a delay unit. We notice that there are two different (iteration) loops in Figure 19: the equalization loop (the blue one) on symbol basis (in the sense of ψ) and the decoding loop (the dashed one) on bit basis. $a=\left\{\mathrm{1,2,3,}\cdots \right\}\mathrm{,}\rho \in \mathbb{N}$ , i.e., after each ρ equalization loops, one decoding loop is performed. The conversion between symbol basis and bit basis (u to ${\mathit{L}}_{ch}$ ) is performed by ${\mathit{\theta }}_{S/L}(\cdot )$ , the way around ($\stackrel{⌣}{\mathit{L}}$ to $\stackrel{⌣}{\mathit{x}}$) by ${\mathit{\theta }}_{L/S}(\cdot )$ . The expressions for ${\mathit{\theta }}_{L/S}(\cdot )$ and ${\mathit{\theta }}_{L/S}(\cdot )$ can be found in [52]. However, for BPSK, they are given as

$\begin{array}{ll}{\theta }_{S/L}\left(u\right)\hfill & =\frac{2}{{\sigma }^2}\cdot u\mathrm{,}\hfill \\ {\theta }_{L/S}\left(\stackrel{⌣}{L}\right)\hfill & =\tanh \left(\frac{\stackrel{⌣}{L}}{2}\right)\mathrm{.}\hfill \end{array}$ E24

σ² is given in Eq. (11). If we consider only the equalization loop in Figure 19, we notice that it describes exactly the dynamical behavior of discrete-time recurrent neural networks [19, 25, 33, 54–56]

$\begin{array}{ccc}\mathit{u}\left[l+1\right]& =& \left[\mathit{I}-{\mathit{R}}_d^{-1}\cdot \mathit{R}\right]\cdot \stackrel{⌣}{\mathit{x}}\left[l\right]+{\mathit{R}}_d^{-1}\cdot \tilde{\mathit{x}}\mathrm{,}\\ \stackrel{⌣}{\mathit{x}}\left[l\right]& =& {\mathit{\theta }}_{L/S}\text{(}{\mathit{\theta }}_{S/L}(\mathit{u}[l]))\mathrm{.}\end{array}$ E25

Remark 11. If ${\mathit{\theta }}_{L/S}\left({\mathit{\theta }}_{S/L}\left(\mathit{u}\right)\right)={\mathit{\theta }}^{(opt)}(\mathit{u})$ , Eqs. (8), (25) share the same equilibrium/fixed points. For BPSK, it can be easily shown based on Eqs. (12), (24) that this is fulfilled.

8. Continuous-time joint equalization and decoding

Motivated by the expected improvement of the energy efficiency by analog implementation compared with the conventional digital one, we map in this section the joint equalization and decoding structure given in Figure 19 to a continuous-time framework. s/s MAP DEC in Figure 19 is replaced by a suboptimum decoding algorithm: the belief propagation. Moreover, equalization and decoding loops in Figure 19 are replaced by RNN and HORNN as discussed previously in Sections 4-A and 6-A, respectively. The introduced structure serves as a computational model for an analog hardware implementation and does not need any training.

Figure 20 shows a novel continuous-time joint equalization and decoding based on recurrent neural network structures. The dynamical behavior of the whole system is described by the following differential equations:

${\mathbf{\Upsilon }}_e\cdot \frac{\text{d}\mathit{u}\left(t\right)}{\text{d}t}=-\mathit{u}\left(t\right)+\mathit{W}\cdot \stackrel{⌣}{\mathit{x}}\left(t\right)+{\mathit{W}}_0\cdot \tilde{\mathit{x}}\mathrm{,}$ E26a

${\mathit{L}}_{ch}\left(t\right)={\mathit{\theta }}_{\text{S/L}}\left(\mathit{u}\left(t\right)\right)\mathrm{,}$ E26b

${\mathbf{\Upsilon }}_d\cdot \frac{\text{d}\mathit{L}\left(t\right)}{\text{d}t}=-\mathit{L}\left(t\right)+\mathit{P}\cdot \mathit{f}\left(\mathit{L}\left(t\right)\right)+\mathit{B}\cdot {\mathit{L}}_{ch}\left(t\right)\mathrm{,}$ E26c

$\stackrel{⌣}{\mathit{L}}\left(t\right)=\underset{{\mathit{L}}_{ext}\left(t\right)}{\underbrace{{\mathit{B}}^T\cdot \mathit{f}\left(\mathit{L}\left(t\right)\right)}}+{\mathit{L}}_{ch}\left(t\right)\mathrm{,}$ E26d

$\stackrel{⌣}{\mathit{x}}\left(t\right)={\mathit{\theta }}_{\text{L/S}}\left(\stackrel{⌣}{\mathit{L}}\left(t\right)\right)\mathrm{,}$ E26e

$f_j=2\cdot \text{atanh}\left\{{\displaystyle \mathit{\prod }_{j^{\prime }\in \text{pos}\left[\mathit{S}(j,:)=1\right]}}\tanh \left(\frac{L_{j^{\prime }}}{2}\right)\right\}\mathrm{.}$ E26f

Figure 20.

Continuous-time joint equalization and decoding based on a recurrent neural network structure for real-valued symbol alphabet (for complex-valued ones, cf. Remark 3). $\stackrel{⌣}{\mathit{L}}(t)$ , $\stackrel{⌣}{\mathit{x}}(t)$ are the soft output of the decoder and the equalizer, respectively. (a) The equalization part. (b) The decoding part.

• Eq. (26a) and Figure 20(a) describe the continuous-time vector equalization, cf. Eqs. (1), (5), (7), (8).

• Eq. (26c) and Figure 20(b) describe the continuous-time belief propagation, cf. Eqs. (16), (18), (20).

Comparing Figure 20 with Figures 7 and 15, we notice that

• The function $\stackrel{}{\mathit{\phi }}$

  
  (⋅) in Figure 7 (the optimum activation function ${\mathit{\theta }}^{(opt)}(\cdot )$ ) has been split into two functions ${\mathit{\theta }}_{\text{S/L}}(\cdot )$ and ${\mathit{\theta }}_{\text{L/S}}(\cdot )$ in Figure 20(a). For BPSK symbol alphabet and based on Eqs. (12), (24), it can be easily shown that ${\mathit{\theta }}^{(opt)}\left(\mathit{u}\right)={\mathit{\theta }}_{\text{L/S}}\left({\mathit{\theta }}_{\text{S/L}}\left(\mathit{u}\right)\right)$ , cf. Remark 11.

• The functions $\check{\mathit{\phi }}(\cdot )$ and $\check{\mathit{f}}(\cdot )$ in Figure 15, Eq. (18) have been merged to one function $\stackrel{}{\mathit{f}}(\cdot )$ in Figure 20(b), Eq. (26f) $\mathit{f}\left(\mathit{L}\right)=\check{\mathit{f}}\left(\check{\mathit{\phi }}\left(\mathit{L}\right)\right)$. $\stackrel{⌣}{\mathit{L}}\left(t\right)$ $\stackrel{⌣}{\mathit{x}}\left(t\right)$ are the soft output of the decoder and the equalizer, respectively.

A. Special cases

The novel structure in Figure 20 is general and stays valid for the following cases:

Figure 21.

Continuous-time separate equalization and decoding based on a recurrent neural network structure for real-valued symbol alphabet (for complex-valued ones, cf. Remark 3). $\stackrel{⌣}{\mathit{x}}(t)$ is the soft output of the equalizer.

• Separate equalization and decoding: In this case, Figure 20(a) is modified such that no feedback from decoder to equalizer is applied. This is shown in Figure 21(a). Only at the end of the separate equalization and decoding process, the output is given as $\stackrel{⌣}{\mathit{L}}={\mathit{L}}_{ext}+{\mathit{L}}_{ch}$ . We distinguish between two cases

  1. Equalization and decoding take place separately at the same time.

  2. Successive equalization and decoding: only after the end of the equalization process, ${\mathit{L}}_{ch}$ are forwarded to the decoder and the decoder starts the evolution. We focus on this case.

• Coded transmission over an AWGN channel: In this case, $\mathit{R}=\mathit{I}$ and hence based on Eq. (7) $\mathit{W}=\mathbf{0}$ , ${\mathit{W}}_0=\mathit{I}$ . Under these conditions, Eq. (26a) becomes linear and can easily be solved $\mathit{u}\left(t\right)\to \tilde{\mathit{x}}$ and ${\mathit{L}}_{ch}\left(t\right)$ in Eq. (26b) becomes time independent ${\mathit{L}}_{ch}$ . In this case, Eq. (26c) reduces to Eqs. (16), (18).

• Uncoded transmission over an “interference-causing” channel: In this case, $\mathit{P}=0$ , $\mathit{B}=0$ and Eq. (26c) becomes $\mathit{L}=0_{n_h}$ . Under these conditions, Eq. (26a) reduces to Eqs. (5), (7), (8) (notice, however, Remark 11).

B. Throughput, asynchronicity and scheduling

The diagonal elements in ${\mathbf{\Upsilon }}_d$ define the duration of the transient response the HORNN needs in order to converge eventually (in case of convergence). The larger they are, the longer is the transient response and consequently the less is the decoding throughput. The same is valid for ${\mathbf{\Upsilon }}_e$ . The diagonal elements of ${\mathbf{\Upsilon }}_e$ based on our analog RNN vector equalizer are in the range of a few tens of picoseconds.

Unequal diagonal elements in ϒ_e (and ϒ_d) represent some kind of continuous-time asynchronicity [46]. Asynchronicity in discrete-time RNNs is desirable since it provides the ability to avoid limit cycles, which can probably occur in synchronous discrete-time RNNs [54, 57].

Assuming ${\mathbf{\Upsilon }}_d={\tau }_d\cdot \mathit{I}$ and ${\mathbf{\Upsilon }}_e={\tau }_e\cdot \mathit{I}$ , we notice that the ratio ${\tau }_e/{\tau }_d$ is comparable to the scheduling problem in the discrete-time joint equalization and decoding case. More precisely, how many iterations ρ within the equalizer should be performed before a decoding process takes place, cf. Figure 19. This is optimized usually by simulations and is case dependent.

From a dynamical point of view, the case ${\tau }_e/{\tau }_d\ll 1$ (or ${\tau }_d/{\tau }_e\ll 1$ ) could be seen as a singular perturbation (in time). In this case, one part of Figure 20 can be seen as “frozen” compared with the other part.

Remark 12. We notice that the parameters of the transmission model (block transmit matrix, symbol alphabet, block size, channel coding scheme) are utilized to define the parameters of the continuous-time recurrent neural network structure in Figure 20 such that no training is needed. This represents in practice a big advantage especially for analog hardware. However, to enable different coding schemes and symbol alphabets, either a full connectivity or a vector and interneuron connectivity controls are needed. Both structures are challenging from a hardware implementation point of view.

Remark 13. For the ease of depiction, Figures 20 and 21 assume that one transmitted block contains exactly one codeword. This is not necessarily the case in practice. As an example, if one transmitted block contains two codewords, one RNN and two parallel HORNNs will be needed. On the other hand, if one codeword lasts over two transmitted blocks, two parallel RNNs and one HORNN is needed.

9. Simulation results

We simulate the dynamical system as given in Eq. (26) and Figure 20 based on the first Euler method [58]. We assume:

Figure 22.

BER vs. $E_b/N_0$ for evolution time equals $20\cdot \tau$ . Continuous-time (joint) equalization and decoding. BPSK symbol alphabet.

• BPSK modulation scheme (symbol alphabet $\mathit{\psi }=\{-\mathrm{1,}+1\}$ ).

• Each transmitted block contains one codeword, cf. Remark 13.

• ${\mathbf{\Upsilon }}_d={\mathbf{\Upsilon }}_e=\mathit{\tau }\cdot \mathit{I}$ .

• Channel coding scheme: An LDPC code with $k=204$ , code rate $0.5$ ($n=408$ ) and column weight $3$ taken from [59].

• Multipath channels [60]:

  1. - Proakis-a abbreviated in the following by its channel impulse response $h_a$ leading to a small interference.

  2. - Proakis-b abbreviated in the following by its channel impulse response $h_b$ leading to a moderate interference.

The impulse response of $h_a$ and $h_b$ are

$\begin{array}{l}{h}_a=\left[\begin{array}{ccccccccccc}0.04& -0.05& 0.07& -0.21& -0.5& 0.72& 0.36& 0& 0.21& 0.03& 0.07\end{array}\right]\mathrm{,}\\ h_b=[\begin{array}{ccc}0.407& 0.815& 0.407\end{array}].\end{array}$

The block transmission matrix R is a banded Toeplitz matrix of the autocorrelation function of the channel impulse response [61]. The following cases are considered:

• Uncoded transmission over AWGN channel. The bit error rate can be obtained analytically and is given as $\frac{1}{2}\cdot \text{erfc}\left\{\frac{E_b}{N_0}\right\}$ [62]. $\text{erfc}(\cdot )$ is the complementary error function and ${\mathit{E}}_b$ is the energy per bit.

• Coded transmission over AWGN channel and continuous-time decoding at the receiver (HORNN-belief propagation).

• Uncoded transmission over (the abovementioned) multipath channels and continuous-time equalization at the receiver (RNN-equalization).

• Coded transmission over (the abovementioned) multipath channels. We distinguish between joint equalization and decoding (Figure 20) and separate equalization and decoding (Figure 21). In the latter case, equalization is performed firstly, and consequently the decoding.

The evolution time for the whole system in all cases is 20 ⋅ τ, i.e., all simulated scenarios deliver the same throughput. For separate equalization and decoding, the evolution time of the equalization equals the evolution time of the decoding and equals 10 τ. The simulation results are shown in Figure 22. We notice the following:

• Joint equalization and decoding outperforms the separate one, which is a fact we know from the discrete-time case. Our proposed model in Figure 20 is capable of “transforming” this advantage to the continuous-time case.

• For the channel $h_a$ , the BER (for continuous-time joint equalization and decoding) is close to the coded BER curve.

• For the channel $h_b$ , there exists a gap between the obtained results and the coded AWGN curve. This was expected, since $h_b$ represents a more severe multipath channel compared with $h_a$ .

• If only equalization performance is considered, we compare between “Uncoded & EQ” curves and “Uncoded BER” curves. In Figure 22(a), the vector equalizer based on continuous-time recurrent neural networks is capable to remove already all interferences caused by the multipath channel $h_a$ , whereas in Figure 22(b), the “Uncoded & EQ” curve approaches an error floor.

Remark 14. Interleaving and antigray mapping often encountered in the context of iterative equalization and decoding can be easily integrated in the proposed model in Figure 20. Antigray mapping will influence the functions ${\mathit{\theta }}_{\text{S/L}}(\cdot )$ and ${\mathit{\theta }}_{\text{L/S}}(\cdot )$ , whereas interleaving affects the matrix B.

10. Conclusion

Joint equalization and decoding is a detection technique which possesses the potential for improving the bit error rates of the transmission at the cost of additional computational complexity at the receiver. Joint equalization and decoding is being considered only for the discrete-time case. However, for high data rates, the energy consumption of a digital implementation becomes a limiting factor and shortens the lifetime of the battery. Improving the energy efficiency revives the analog implementation option for joint equalization and decoding algorithms, particularly taking advantage of the nonlinearity of the corresponding algorithms.

Continuous-time recurrent neural networks serve as promising computational models for analog hardware implementation and stand out due to their Lyapunov stability (the proved existence of attracting equilibrium points under specific conditions) and special suitability for analog VLSI. They have often been applied for solving optimization problems even without the need for a training. The drop of the training is particularly favorable for analog hardware implementation.

In this chapter, we introduced a novel continuous-time recurrent neural network structure, which is capable to perform continuous-time joint equalization and decoding. This structure is based on continuous-time recurrent neural networks for equalization and continuous-time high-order recurrent neural networks for belief propagation, a well-known decoding algorithm for low-density parity-check codes. In both cases, the behavior of the underlying dynamical system has been addressed, Lyapunov stability and simulated annealing are a few examples. The parameters of the transmission system (channel matrix, symbol alphabet, block size, channel coding scheme) are used to define the parameters of the proposed recurrent neural network such that no training is needed.

Simulation results showed that the superiority of joint equalization and decoding preserves, if this is done in analog according to our proposed model. Compared with the digital implementation, the analog one is expected to improve the energy efficiency and consume less chip area. We confirmed this for the analog hardware implementation of the equalization part. In this case, the analog vector equalization achieves an energy efficiency of a few picojoule per equalized bit, which is three to four orders of magnitude better than the digital counterparts. Additionally, analog hardware implementation aspects have been discussed. We showed as an example the importance of the interneuron connectivity, especially pointing out the challenges represented either by the hardware implementation of a massively distributed network, or by the routing of the signals using (de)multiplexers.