In this chapter, we present an overview of the recent advances in analog-to-digital converter (ADC) neural networks. Biological neural networks consist of natural binarization reflected by the neurosynaptic processes. This natural analog-to-binary conversion ability of neurons can be modeled to emulate analog-to-digital conversion using a set of nonlinear circuit elements and existing artificial neural network models. Since one neuron during processing consumes on average only about half nanowatts of power, neurons can perform highly energy-efficient operations, including pattern recognition. Analog-to-digital conversion itself is an example of simple pattern recognition where input analog signal can be presented in one of the 2N different patterns for N bits. The classical configuration of neural network-based ADC is Hopfield neural network ADC. Improved designs, such as modified Hopfield network ADC, T-model neural ADC, and multilevel neurons-based neural ADC, will be discussed. In addition, the latest architecture designs of neural ADC such as hybrid complementary metal-oxide semiconductor (CMOS)-memristor Hopfield ADC are covered at the end of this chapter.
- neural networks
- analog-to-digital converters
- Hopfield network
This chapter presents a review of the advancements in the application of neural network (NN) systems in analog-to-digital converter (ADC) design. Analog-to-digital (A/D) conversion is an essential part of all microelectronic systems design that serves as a link between analog sensors and digital-processing circuitry . The dominant period of the ADC design development came with the maturity of complementary metal-oxide semiconductor (CMOS) technologies . At present, there is a huge variety of high-speed and high-resolution ADCs based on the most advanced CMOS processes that are applicable for different applications . In fact, even though the ADC design field is mature, the complexity of the construction of properly operating ADC system that fits certain applications is still high. In conventional CMOS ADCs, a number of appropriately designed analog circuitries, such as switches, operational amplifiers, voltage converters, and so on, are required . However, with modern advancements in computational systems and processing applications, the demand for faster processing and more flexible architectures that can perform a variety of tasks in the most efficient manner has increased. Artificial neural network (ANN) technology is a well-known candidate that can resolve such demands in high-performance A/D conversion as it divides the task between a number of simple processing elements (neurons) . Further, neurons can perform highly energy-efficient operations of pattern recognition, in particular, one neuron during processing consumes on average only about half nanowatts of power .
Since the early twentieth century, scientists and engineers have been trying to explain how human brain functions, and a number of models that are aimed to mimic some features of the biological neural networks were proposed. The work presented by McCulloch and Pitts  is one of the first examples of mathematical modeling of ANN that is based on a two-state neuron model. The information processing that is performed in biological neural networks incorporates memorization, learning, classification, and so on and is performed by natural binarization mechanism reflected by the neurosynaptic processes. Brain associative property that is used in processing information is being discussed widely since the 1960s and later. Based on the works presented by [5, 6, 7, 8], Hopfield proposed a neural network model that incorporates associative memory property. The idea that he presented is actually a model of content addressable memory (CAM) that can be implemented in hardware [9, 10, 11]. He discovered that such a type of network has collective computational properties so that it can be used in solving different optimization problems [9, 10, 11].
One of the applications of such CAM-based neural network (NN) that was introduced by Hopfield and Tank includes solving simple optimization problem such as analog-to-digital (A/D) conversion, where the dynamics of the system is described by an energy function (or cost function) . The main concept behind the proper operation of the Hopfield NN is based on minimization of the energy function, so that when the minimum value is achieved, the network reaches its stable state [9, 10, 11, 12]. In general, A/D conversion can be classified as an example of simple pattern recognition where input analog signal can be presented in one of the different patterns for N bits. In Hopfield NN-based ADC, these digital patterns are stored as a memory and are retrieved when the network reaches stable state after conversion period .
The NN model proposed by Hopfield represents a network of interconnected processing units (neurons) connected through a symmetric connection matrix with zero diagonal elements [9, 10, 11, 13]. The interconnection nodes between neurons can be viewed as synaptic strength values. The strength of each synapse is represented by the conductance value at each node. The network dynamics is governed by the behaviour of energy function, so that when the energy function is of the minimum value, the network reaches stable state and gives digital output [9, 10, 11, 13].
Therefore, in Section 2, a comprehensive discussion on Hopfield NN in general and the ADC based on the Hopfield NN design is presented. The section addresses such topics as the theory of the Hopfield NN, the description of how to construct an ADC structure and the problems that appear in the Hopfield NN ADC. In Section 3, a review of different designs based on original Hopfield ADC such as modified Hopfield neural ADC, NN-based ADCs with non-symmetrical weight matrix, NN-based ADC with multilevel neurons and level-shifted neural ADC is presented. In Section 4, recent CMOS-memristor-based ADC architectures are reviewed. The last section summarizes and gives a conclusion for this chapter.
2. Hopfield neural network ADC
2.1. The Hopfield ADC theory
In his early works, Hopfield introduced the ideas behind the emergent collective computational properties of highly interconnected associative networks [9, 10]. The neural network models that were presented earlier were of Perceptron type and were implemented by feedforward architecture . By contrast, Hopfield presented a different type of architecture with fully interconnected neurons, where each neuron translates its output to the inputs of the remaining neurons through feedback connections [9, 10]. The strength of each feedback connection is represented by its weight (or synapse). In a later work, Hopfield and Tank  presented methods of how the network can be applied in solving optimization problems, such as A/D conversion, signal decomposition and linear programming.
One of the earliest works on artificial neural networks (ANNs) by McCulloch and Pitts  described a two-state (on-state and off-state) stochastic neuron model that simplifies biological neural function to simple logical operation. However, this model was not applicable for analog processing as it did not have the continuous behaviour as of biological neurons [10, 11, 13]. Hopfield proposed the NN model with continuous neuron response [10, 11, 13], which has computation properties of the stochastic model  that can be implemented in hardware. Continuous neuron response in Hopfield NN can be interpreted as an analogy of graded dependence of firing rate produced by the soma of biological neuron as the input signal to the neuron membrane [10, 11, 13] without considering action potential signal details. Two states in neuron model are considered as ‘0’ for not firing state and ‘1’ for firing at a maximum rate [10, 11, 13]. The graded response function that describes such dependence is neuron’s activation function that is represented by monotonically increasing sigmoid function (Eq. (1))
where is the input voltage to neuron, so the neuron’s output signal will be equal to .
The neuron output can be either logic high or logic low depending on whether the effective input voltage to neuron is higher or lower than the neuron threshold, as it can be observed from Figure 1.
The hardware implementation of 4-bit Hopfield NN ADC proposed in  is shown in Figure 2. As it is described in , at each analog input level, the network creates an energy function surface that consists of local minima states with one global minimum for this particular analog input. The global minimum for each input level represents the correct digital representation for the input signal . The dynamics of the system can be viewed as a flow in energy state space that tends to minimize , so that when the network reaches minimum it stops searching process [10, 13]. When the ADC network arrives at an energy minimum state, it should produce an output code that best represents the corresponding analog input. Thus, the function is a Lyapunov stability function of the system . The proper operation of the Hopfield ADC is achieved when the voltage level of the output code is equal to the value of the analog input, Eq. (2).
The ADC network consists of four neurons that are interconnected by a synaptic weight matrix. The network dynamics is highly dependent on the values of synaptic matrix elements. This dependency was analysed by Hopfield in his work [9, 10], where it is deduced that for the system to reach a stable state two conditions should be maintained: (1) the symmetrical synaptic weight matrix and (2) the diagonal synaptic weights that correspond to feedbacks from neurons to their own inputs should be equal to zero . Following this condition, as it is shown in [9, 10, 11, 13], the Hopfield neural network should converge to a stable state. The energy function for the Hopfield network with symmetric weight matrix is shown by Eq. (3)
The NN proposed by Hopfield has the features that correlate with biological NNs and so it represents a simplified analogy of biological NNs. The system dynamic change can be described by the first-order differential equation of the rate of change of neuron input potential, Eq. (4). The capacitance that is present at the neuron input is a circuit representation of neuron cell membrane capacitance, while the term in which can be viewed as neuron cell transmembrane resistance 
From Eq. (4), it is seen that neuron is charged by integrating the current flowing into the neuron with charging time constant [10, 13]. The current that flows into the neuron consists of three components formed prior to the neuron input, which are postsynaptic current from neuron analog input current and constant reference current [10, 13].
The ADC operation also can be described by the energy function shown by Eq. (5) . The first term of Eq. (5) shows the squared difference between analog input voltage and the corresponding digital output voltage. As it was previously assumed, the value of analog input voltage should be close to the voltage level of the corresponding output code, see Eq. (2). If for particular the output code is the most correct digital representation, the first term of Eq. (5) should be equal to zero . The second term in Eq. (5) is added to ensure that the digital output voltages will be of logic ‘0’ and ‘1’ 
2.2. The local minima problem
As it is already discussed, the stability of the Hopfield NN is achieved when the energy function is at its minimum in the state space. The dynamics of the system is moving towards decreasing the energy function. Thus, for the Hopfield NN the energy state space will have multiple local minima, where each of these local minima states is able to stabilize the system dynamics. In theory, the ADC structure proposed by Tank and Hopfield , which is based on the Hopfield NN with symmetric weight matrix, has to retrieve correct digital response of the analog input voltage by means of the energy local minima states that are assigned for each correct digital output. However, in practice, this concept does not work as is expected. It appears that the local minima states corrupt the correct operation of the network [14, 15, 16, 17, 18, 19].
In the original Hopfield’s work , it was proposed to implement neurons with the CMOS operational amplifiers. The results that were obtained exhibit not ideal ADC behaviour with incorrect output states (Figure 3).
It is found that after each A/D conversion cycle, the threshold voltage of each neuron circuitry differs from the pre-set value of V , such that the response of comparators exhibits offset. The authors in  proposed that the behaviour caused by the hysteresis of CMOS neurons in addition to the local minima states is a dominant contributor to the wrong network response. This hysteresis change in thresholds makes the system to stabilize at the local minima state, which is located closer to the network’s present energy state at the moment of conversion [14, 15, 16, 17, 18, 19]. In order to solve this problem, one of the solutions is to reset the neuron state to the initial threshold value after each conversion . However, the main disadvantage of this method is that it requires more power.
Alternatively, in the works [11, 12, 13, 14, 15, 16], the authors proposed to change the Hopfield ADC network architecture itself in order to eliminate the local minima states, which cause errors in the ADC outputs. Thus, different methods on eliminating local minima problem are proposed in [14, 15, 16, 17, 18, 19], which are discussed in more detail in Section 3.
3. Hopfield neural network-based ADCs
The design presented by Hopfield and Tank is a first example of ADC task implementation with neural networks. This idea became very popular later, as it appeared very simple compared to the conventional designs and, moreover, it opens up the possibilities to explore its phenomenological computational abilities which is a good contribution for science and engineering by itself.
As it was previously described, the existence of local minima in the dynamics of the original Hopfield network ADC design corrupts its digital output by generating spurious states. This problem was addressed by several works that presented the ways of eliminating the local minima states by changing the structure of the ADC network [14, 15, 16, 17, 18, 19]. In the following subsection, the two methods that are claimed to eliminate the problem of local minima of the energy function are presented.
3.1. Eliminating the local minima problem of Hopfield ADC
3.1.1. Modified Hopfield architecture with correction currents
In the study by , the authors analyzed the stability of the output codes of Hopfield network ADC in terms of overlap of input currents between two adjacent output codes which is defined as . According to Lee and Sheu , in order to avoid the local minima state, this parameter should be higher or equal to zero. Thus, it was deduced that in order to eliminate this current overlap condition, the correction currents can be applied back to the inputs of Hopfield network through the additional set of conductance weights .
The modified Hopfield network ADC schematic diagram is shown in Figure 2. The correction currents are generated by inverting amplifiers in order to compensate the overlap and to maintain system dynamics converging to a stable state. In Eq. (8), the dynamics of network in a stable state with applied correction current, is described.
The energy function of the modified Hopfield network can be described by adding an additional term that represents the correction currents, Eq. (8). The correcting energy eliminates the local minima states and gives the network one global minimum energy state .
There are certain conditions, according to , that should be followed while selecting the correction currents and conductance values. The first condition is to avoid the state when the parameter is less than zero so that to avoid the two adjacent codes be stable simultaneously. The second condition states that the network dynamics must be sustained in the operation that minimizes energy function of the system. The last condition is to maintain the input current range in appropriate for the global minimum value. For the detailed description of the method, please refer to .
3.1.2. Non-symmetric Hopfield architecture
The type of architecture based on Hopfield network is built with non-symmetric connection weight matrix, which is another example that is aimed to solve the local minima states problem. In the designs by [15, 16, 17, 18, 19], the properties of the triangular connections are analyzed. In , the authors prove that by triangular interconnection matrix the network operates without spurious states and that this type of architecture can be a good alternative for the original Hopfield design. Similar network type was analyzed by Sun et al. , and it is proven that the local minima problem can be mitigated by using this architecture. Taking into account the structure of the model , the learning component can be applied to the network making this type of architecture advantageous over the original one.
In Figure 3, the non-symmetric T-model ADC is shown. The input current at each raw represents the current flowing from the external analog input source and from the reference.
This section presents an overview of the designs of neural network ADC of Hopfield network type that solves the problem of the local minima of energy function that creates the digital error at the output of the ADC. We introduced a brief explanation of the two methods of elimination of the local minima.
3.2. Hopfield ADC with multilevel neurons
An interesting alternative design is proposed by [20, 21] in which the authors focus on implementing analog neurons to be of multiple states. The design named as Multilevel Neural Network is applied to the original Hopfield neural network ADC by replacing the conventional two-state sigmoidal neurons by multiple-state (or multiple threshold) neurons . The motivation under this idea is to create a type of neural ADC with better resolution but with the same number (or even less number) of synaptic weights as in the original Hopfield ADC design . This method reduces the complexity of weight matrix and makes it easier to implement the ADC with improved resolution in hardware .
The schematic diagram of the ADC proposed in  is shown in Figure 4. Being a distinguishable alternative neural networks-based ADC design, it still does not solve the problems of the local minima states of the Hopfield associative network. In , the authors considered this case and proposed to solve the local minima by additional correction current method .
The multilevel neuron dynamics is described by the block diagram shown in Figure 5 . In the original Hopfield ADC, continuous neuron model dynamics is described by the first-order differential equation (Eq. (4)). The two-state neuron activation function is expressed by Eq. (1). The neuron output is then equal to , and it can take two states logic high and logic low (refer to Section 2). In multilevel neuron model, a two-state activation function neuron is replaced with the multiple-state nonlinearity block (Eq. (10)) (Figures 6 and 7)
The nonlinearity function shown in Eq. (10) is described as a sum of monotonically nondecreasing step functions with different threshold values , where the state of the neuron changes. Each step function is multiplied by the positive coefficient parameter that can be seen as an offset parameter . The sigmoidal multilevel nonlinearity can be observed in Figure 8
The neuron dynamics can be expressed by the block diagram shown in Figure 7. The term translates information about the current state to its own input so that when the input current value is higher than the , the state of the neuron is increased. In this design, the additional value is present as a diagonal element in the weight matrix . Therefore, the system is described by Eq. (12)
The energy function for the multilevel ADC architecture can also be found by the square of difference expression, Eq. (14). The number of levels in the multilevel nonlinearity block of the neuron is and represents the base of conversion . The system tends to find the correct digital representation with base of analog input signal with the minimum energy function value . After expanding Eq. (14), Eq. (15) is obtained, which gives the synaptic weight values of the network, Eq. (16)
In the ADC with multilevel neurons, the design system suffers from the local minima problem, which they solve by applying a similar technique that was proposed by  described in the previous subsection . Another method of eliminating incorrect output response for multilevel neuron-based ADC was presented in , where the parallel hardware-annealing technique was introduced.
3.3. Hopfield neural network-based level-shifted ADC
In the previous subsections, we discussed various types of architectures that are the modified versions of Hopfield ADC, such as the ADC with correction current, the ADC with non-symmetric weight matrix and the ADC with multilevel neurons. All these designs are based on the original Hopfield ADC structure. However, in this subsection we discuss a type of Hopfield-based ADC that is different from the earlier architectures discussed. The level-shifted neural ADC  is a new type of architecture that is constructed with multiple 2-bit Hopfield ADCs and voltage level shifters (Figure 9). The ADC design proposed by Hopfield and Tank  produces a 4-bit digital output, which is not very much practical in modern technologies. In order to increase the number of neurons in Hopfield NN ADC , the corresponding scaling of input and output voltages should be made according to Eq. (2). Therefore, if the goal is to increase the resolution by increasing the number of neurons of Hopfield ADC, the binary output voltage values from neurons will be reduced. Furthermore, the resolution change will require appropriate scaling of the weight matrix. These two problems were addressed in [20, 21, 22] and methods that solve these problems were presented. The level-shifted neural ADC is another method that can solve the resolution improvement issue of Hopfield NN ADC.
The operation principle of the proposed level-shifted neural ADC  is not very complicated compared to the designs in [14, 15, 16, 17, 18, 19, 20, 21, 22]. As it was mentioned, the design consists of multiple 2-bit Hopfield ADC blocks that operate in parallel. Each successive 2-bit ADC block receives input signal that is DC-shifted to some small positive voltage level. The design parameters can be adjusted depending on the application of the ADC.
The preliminary results of the level-shifted neural ADC for 16-quantization level ADC are presented in . As the design consists of multiple operation in parallel 2-bit Hopfield ADCs, the number of output bits in the digital code is larger compared to the conventional Hopfield ADC. Therefore, it is proposed to use a feedforward neural network encoder so that the digital output will be of a 4-bit format and also to reduce the error in computation due to the local minima and circuit nonidealities.
4. CMOS/memristor hybrid network-based ADC
Since memristor, the fourth fundamental circuit element , was discovered by HP Labs in 2008 , the device is receiving very high attention as it has a potential to emulate the functionality of biological synapses. During the past decade, many scientists have shown a variety of methods of memristor application in hardware design of ANN systems. For instance, in  the hybrid CMOS-memristor Hopfield network-based associative memory is demonstrated. While in the work conducted by Guo et al. , the CMOS-memristor hybrid architecture is applied in the design of 4-bit Hopfield neural ADC. Figure 10 reflects the schematic of the system proposed in .
The CMOS-memristor hybrid Hopfield ADC  consists of memristor-based weight matrix and sigmoidal CMOS neurons. The advantage of implementing constant synapses (in Hopfield NN for ADC design synaptic weights a preset and kept unchanged ) with memristors is that being a nanoscale device, memristors consume much less power . Moreover, they significantly reduce the on-chip area compared to CMOS-based synaptic weights . In their work, Guo et al.  demonstrated the simulation of the proposed system and also successfully implemented their circuit in hardware.
The tuning of memristors is performed by applying either voltage or current pulses with gradually changing amplitude (and/or width) continuously until the device reaches a desired resistance state . In order to sustain the pre-programmed resistances in memristive weight matrix, the network-operating region (analog input and neuron maximum output voltage) was scaled down so as to prevent any resistance state fluctuations in memristors . The CMOS-memristor hybrid ADC applied resetting the neuron states technique similar to that demonstrated in  for reduction of the effects of the local minima states.
Another type of CMOS-memristor hybrid neural ADC is a T-model neural ADC architecture proposed by . In the design by Wang et al. , the additional least mean square (LMS) training algorithm is applied in order to optimize the system operation to certain conditions. The LMS algorithm that was used in  allows flexibility to ADC in terms of voltage operation region. The training algorithm is implemented by means of digital training block connected to the T-model weight matrix. The works presented in [2, 27] introduce architectures of neural ADC that utilize memristors as a synaptic weight elements. They demonstrate that the lower power consumption of the memristive devices can be applied in the Hopfield NN ADC design. By contrast, the Hopfield network still requires additional circuitry to eliminate the local minima-related errors.
The Hopfield network-based ADCs represent a compact approach for the implementation of analog-to-digital conversion task. However, if trying to implement the model in hardware, the multiple circuit nonidealities create errors in the digital output that somehow must be corrected. For instance, as it was discussed previously, the offset response (hysteresis) of comparators after each conversion creates condition for the network to develop incorrect patterns. The possible solution for eliminating offset is to reset the comparators periodically after each conversion to the initial 0-V threshold state, as it was already mentioned . However, this method is not preferable in terms of circuit implementation, as such circuit requires more power. Another problem, as it was previously discussed, was the local minima behaviour of Hopfield network that creates spurious states so that the output does not correspond to the desired response. The existence of local minima of the network is deduced by circuit analysis techniques in , and it was proposed to add a feedback current that will balance the network and create a single energy minimum for the whole system dynamics. Thus, the Hopfield NN-based ADC examples discussed in this chapter are still not adapted into practical use. Even though the local minima problem was mitigated, there is not much analysis on resolution improvement. In [20, 21, 22] by means of multilevel neuron structure, the 8-bit of resolution was achieved. However, ADC structure becomes much more complex since it incorporates multilevel nonlinearity blocks in each neuron and also uses correction current technique as in . Therefore, in order to achieve performance as better as possible from such type of designs as Hopfield network-based ADCs, the complexity of system components must be increased and many parameters must be taken into account, such as circuit mismatches and offsets since they can affect the output significantly. In addition, the analog structure of Hopfield network-based ADCs creates limitations to resolution improvement and thus makes these designs difficult to be implemented and to be compatible with conventional ADCs.
The alternative ADC structure based on Neural Engineering Framework (NEF) was demonstrated in , where it is proposed to shift system parts as much as possible into the digital domain, and only the front end of the ADC incorporates feedforward-type neural network encoder that passes signal to analog neurons, and the rest of the processing is done in digital form. Since the design uses a huge population of neurons in the input, even some amount of neurons will fail and the system is robust to such failures. Moreover, the stability issue is no longer valid in this type of architecture, as the neural network used in the design is purely feedforward. The NEF ADC is generally flexible and scalable, as it mostly consists of digital circuitry, and therefore, it can be adapted to any system requirements and technologies. However, the unresolved issue of the design is a very high power consumption of the network .
This chapter presents a review of existing technologies of neural network-based ADC designs. A/D conversion is an essential process in microelectronic systems that create a connection between analog systems (e.g., sensors) and digital-processing circuitry . With the modern advancements in submicron CMOS technologies, a variety of high-speed and high-resolution ADCs that are used in different applications have increased . In fact, considering the maturity of the field, the complexity of building an ADC has not been reduced. Moreover, due to the applications that require higher performance and flexibility, the resources of conventional ADC architectures may not be enough. Artificial intelligence is considered to tackle such high requirements on speed and performance. The A/D conversion is not excluded from the list of operations that can be done by means of ANN.
In classical works presented by Hopfield [9, 10], he proposed a mathematical CAM model that consists of a group of two-state neurons interconnected between each other that exhibit collective computational behaviour. He further presented the design that can solve optimization problems . The A/D conversion in his work  was considered as a simple optimization problem in which it was desired to minimize the value of energy function that describes the dynamics of the ADC system. He presented a 4-bit NN ADC architecture that can be implemented in hardware . However, the ADC architecture has intrinsic imperfection due to multiple local minima of energy function that creates digital error in the output of the ADC [14, 15, 16, 17, 18, 19].
To solve the local minima problem, several methods are proposed in [14, 15, 16, 17, 18, 19]. As it is discussed in Section 3, there are two main methods of eliminating the local minima states and obtaining one global minimum. In the modified Hopfield ADC design, it is proposed to apply correction currents back to the input of each neuron in order to reduce the overlapping current occurring between adjacent output codes . This method eliminates local minima and creates one global minimum towards which the network flow is attracted . Another interesting method that also reduces the effects of local minima is the neural ADC with non-symmetrical weight matrix connection [15, 16, 17, 18, 19]. According to this method, ADC architectures with non-symmetrical weight matrix do not create multiple energy minima states; as a result, such networks are also attracted to a global minimum energy state [15, 16, 17, 18, 19].
Multilevel neural ADC architecture  is based on the original Hopfield ADC structure but with modified neuron model. The authors in  proposed a multiple-state neuron implementation that is aimed to improve the resolution of the ADC. A similar goal, to improve resolution, was pursued by the level-shifted neural ADC architecture  that is built with multiple Hopfield ADC blocks and voltage level shifters.
In addition to the presented CMOS-based neural ADC structures in Section 3, examples of CMOS-memristor-based neural ADC architecture [2, 27] are discussed in Section 4. The memristor device is a promising technology that is aimed to expand the capabilities of traditional CMOS-based systems. The application of memristors in neuromorphic circuits and the development of new memristor-based architectures are currently being widely discussed. However, in [2, 27], traditional implementation of neural ADC architecture was modified by the addition of memristors. Thus, the demonstrated results in [2, 27] have shown that there is a potential in the application of memristors in CMOS-based systems, as memristors consume less power and save on-chip area, which makes memristor-based neural ADC an attractive alternative to traditional NN-based ADC designs that are discussed previously. To sum up, a general overview on the NN-based ADC design area is presented in this chapter.