Applications of General Regression Neural Networks in Dynamic Systems

Abstract

Nowadays, computational intelligence (CI) receives much attention in academic and industry due to a plethora of possible applications. CI includes fuzzy logic (FL), evolutionary algorithms (EA), expert systems (ES) and artificial neural networks (ANN). Many CI components have applications in modeling and control of dynamic systems. FL mimics the human reasoning by converting linguistic variables into a set of rules. EA are metaheuristic population-based algorithms which use evolutionary operations such as mutation, crossover, and selection to find an optimal solution for a given problem. ES are programmed based on an expert knowledge to make informed decisions in complex tasks. ANN models how the neurons are connected in animal nervous systems. ANN have learning abilities and they are trained using data to make intelligent decisions. Since ANN have universal approximation abilities, they can be used to solve regression, classification, and forecasting problems. ANNs are made of interconnected layers where every layer is made of neurons and these neurons have connections with other neurons. These layers consist of an input layer, hidden layer/layers, and an output layer.

1. Introduction

Nowadays, computational intelligence (CI) receives much attention in academic and industry due to a plethora of possible applications. CI includes fuzzy logic (FL), evolutionary algorithms (EA), expert systems (ES), and artificial neural networks (ANN). Many CI components have applications in modeling and control of dynamic systems. FL mimics the human reasoning by converting linguistic variables into a set of rules. EA are metaheuristic population-based algorithms which use evolutionary operations such as mutation, crossover, and selection to find an optimal solution for a given problem. ES are programmed based on an expert knowledge to make informed decisions in complex tasks. ANN model how the neurons are connected in animal nervous systems. ANN have learning abilities and they are trained using data to make intelligent decisions. Since ANN have universal approximation abilities [1], they can be used to solve regression, classification, and forecasting problems. ANNs are made of interconnected layers where every layer is made of neurons, and these neurons have connections with other neurons. These layers consist of an input layer, hidden layer/layers, and an output layer. ANN have two major types as shown in Figure 1: feed-forward neural network (FFNN) and recurrent neural network (RNN). In FFNN, the data can only flow from the input to hidden layer, while in RNN, the data can flow in any direction. The output of a single-hidden-layer FFNN can be written as

$$Y=\left(W_{\mathit{HO}}h\left(\mathbf{x}{W}_{\mathit{IH}}+{\mathbf{b}}_{\mathbf{I}}\right)\right)+{\mathbf{b}}_{\mathbf{O}}$$E1

where $Y$ is the network output, $W_{\mathit{HO}}$ is the hidden-output layers weights matrix, $h$ is the hidden layer activation function, $\mathbf{x}$ is the input vector, $W_{\mathit{IH}}$ is the input-hidden layers weights matrix, ${\mathbf{b}}_{\mathbf{I}}$ is the input layer bias vector, and ${\mathbf{b}}_{\mathbf{O}}$ is the hidden layer bias vector.

Figure 1.

Feed-forward and recurrent networks.

The output of a single-hidden-layer RNN with a recurrent hidden layer can be written as

$$Y=\left(W_{\mathit{HO}}h\left(\mathbf{x}{W}_{\mathit{IH}}+{\mathbf{h}}_{\mathbf{t}-\mathbf{1}}{W}_{\mathit{HH}}+{\mathbf{b}}_{\mathbf{I}}\right)\right)+{\mathbf{b}}_{\mathbf{O}}$$E2

The training of neural networks involves modifying the neural network parameters to reduce a given error function. Gradient descent (GD) [2, 3] is the most common ANN training method:

$${\theta }_{\mathit{new}}={\theta }_{\mathit{old}}-\lambda \frac{\partial E}{\partial \theta }$$E3

where $\theta$ are the network parameters, $\lambda$ is the learning rate, and E is the error function:

$$E=\frac{1}{N}\sum _{I=1}^N{\left(y-t\right)}^2$$E4

where $N$ is the number of samples, $y$ is the network output, and $t$ is the network target.

2. General regression neural network (GRNN)

The general regression neural network (GRNN) is a single-pass neural network which uses a Gaussian activation function in the hidden layer [4]. GRNN consists of input, hidden, summation, and division layers.

The regression of the random variable $y$ on the observed values $X$ of random variable $x$ can be found using

$$E\left[y|X\right]=\frac{{\int }_{-\infty }^{\infty }\mathit{yf}\left(X,y\right)\mathit{dy}}{{\int }_{-\infty }^{\infty }f\left(X,y\right)\mathit{dy}}$$E5

where $f\left(X,y\right)$ is a known joint continuous probability density function.

When $f\left(X,y\right)$ is unknown, it should be estimated from a set of observations of $x$ and $y$. $f\left(X,y\right)$ can be estimated using the nonparametric consistent estimator suggested by Parzen as follows:

$$\widehat{f}\left(X,Y\right)=\frac{1}{2{\pi }^{\left(p+1\right)/2}{\sigma }^{\left(p+1\right)}}\frac{1}{n}\sum _{i=1}^n{e}^{-\frac{{\left(X-X^i\right)}^T\left(X-X^i\right)}{2{\sigma }^2}}{e}^{-\frac{{\left(Y-Y^i\right)}^2}{2{\sigma }^2}}$$E6

where $n$ is the number of observations, $p$ is the dimension of the vector variable, and $x$ and $\sigma$ are the smoothing factors.

Substituting (6) into (5) leads to

$$\widehat{Y}\left(X\right)=\frac{{\sum }_{i=1}^n{e}^{\frac{-{\left(X-X^i\right)}^T\left(X-X^i\right)}{2{\sigma }^2}}}{{\sum }_{i=1}^n{e}^{\frac{-{\left(X-X^i\right)}^T\left(X-X^i\right)}{2{\sigma }^2}}}\frac{{\int }_{-\infty }^{\infty }{\mathit{ye}}^{-\frac{{\left(Y-Y^i\right)}^2}{2{\sigma }^2}}\mathit{dy}}{{\int }_{-\infty }^{\infty }{e}^{-\frac{{\left(Y-Y^i\right)}^2}{2{\sigma }^2}}\mathit{dy}}$$E7

After solving the integration, the following will result:

$$\widehat{Y}\left(X\right)=\frac{{\sum }_{i=1}^n{\mathit{ye}}^{\frac{-{\left(X-X^i\right)}^T\left(X-X^i\right)}{2{\sigma }^2}}}{{\sum }_{i=1}^n{e}^{\frac{-{\left(X-X^i\right)}^T\left(X-X^i\right)}{2{\sigma }^2}}}$$E8

2.1. Previous studies

GRNN was used in different applications related to modeling, system identification, prediction, and control of dynamic systems including: feedback linearization controller [5], HVAC process identification and control [6], modeling and monitoring of batch processes [7], cooling load prediction for buildings [8], fault diagnosis of a building’s air handling unit [9], intelligent control [10], optimal control for variable-speed wind generation systems [11], annual power load forecasting model [12], vehicle sideslip angle estimation [13], fault diagnosis for methane sensors [14], fault detection of excavator’s hydraulic system [15], detection of time-varying inter-turn short circuit in a squirrel cage induction machine [16], system identification of nonlinear rotorcraft heave mode [17], and modeling of traveling wave ultrasonic motors [18].

Some significant modifications of GRNN include using fuzzy c-means clustering to cluster the input data of GRNN [19], modified GRNN which uses different types of Parzen estimators to estimate the density function of the regression [20], density-driven GRNN combining GRNN, density-dependent kernels and regularization for function approximation [21], GRNN to model time-varying systems [22], adapting GRNN for modeling of dynamic plants [23] using different adaptation approaches including modifying the training targets, and adding a new pattern and dynamic initialization of $\sigma$.

2.2. GRNN training algorithm

GRNN training is rather simple. The input weights are the training inputs transposed, and the output weights are the training targets. Since GRNN is an associative memory, after training, the number of the hidden neurons is equal to the number of the training samples. However, this training procedure is not efficient if there are many training samples, so one of the suggested solutions is using a data dimensionality reduction technique such as clustering or principal component analysis (PCA). One of the novel solutions to data dimensionality reduction is using an error-based algorithm to grow GRNN [24] as explained in Algorithm 1. The algorithm will check whether an input is required to be included in the training, based on prediction error before training GRNN with that input. If the prediction error without including that input is more than the certain level, then GRNN should be trained with it.

2.2.1. Reducing data dimensionality using clustering

Clustering techniques can be used to reduce the data dimensionality before feeding it to the GRNN. k-means clustering is one of the popular clustering techniques. The k-means clustering algorithm is explained in Algorithm 2. Also, results of comparing GRNN performance before and after applying k-means algorithm are shown in Table 1. Although the training and testing errors will increase, there are large reductions in the network size.

Dataset	Training error after/before k-means MSE	Testing error after/before k-means MSE	Size reduction %
Abalone	0.0177/0.002	0.0141/0.006	99.76
Building energy	0.047/3.44e-05	0.0165/0.023	99.76
Chemical sensor	0.241/0.016	0.328/0.034	97.99
Cholesterol	0.050/4.605e-05	0.030/0.009	92

Table 1.

Using GRNN with k-means clustering.

The aim of the algorithm is to minimize the distance objective function:

$$J=\sum _{i=1}^N\sum _{j=1}^M{‖x_i-c_j‖}^2$$E9

2.2.2. Reducing data dimensionality using PCA

PCA can be used to reduce a large dataset into a smaller dataset which still carries most of the important information from the large dataset. In a mathematical sense, PCA converts a number of correlated variables into a number of uncorrelated variables. PCA algorithm is explained in Algorithm 3.

Dataset	Training error after/before PCA MSE	Testing error after/before PCA MSE	Size reduction %
Abalone	0.197/0.002	0.188/0.006	99.8
Building energy	0.061/3.44e-05	0.049/0.023	99.6
Chemical sensor	0.241/0.016	0.328/0.034	98.3
Cholesterol	0.026/4.605e-05	0.028/0.009	92

Table 2.

Using GRNN with PCA.

2.3. GRNN output algorithm

After GRNN is trained, the output of GRNN can be calculated using

$$D={\left(X-W_i\right)}^T\left(X-W_i\right)$$E10

$$\widehat{Y}=\frac{{\sum }_{i=1}^N{W}_o{e}^{(}}{{\sum }_{i=1}^N{e}^{(}D/2{\sigma }^2)}$$E11

where $D$ is the Euclidean distance between the input $X$ and the input weights $W_i$, $W_o$ is the output weight, and $\sigma$ is the smoothing factor of the radial basis function.

GRNN output calculation is explained in Algorithm 4.

Other distance measures can be also used such as Manhattan (city block), so (10) will become

$$D=X-W_i$$E12

3. Estimation of GRNN smoothing parameter ($\mathbf{\sigma }$)

Since $\sigma$ is the only free parameter in GRNN and suitable values of it will improve GRNN accuracy, it should be estimated. Since there is no optimal analytical solution for finding $\sigma$, numerical approaches can be used to estimate it. The holdout method is one of the suggested methods. In this method, samples are randomly removed from the training dataset; then using the GRNN with a fixed $\sigma$, the output is calculated using the removed samples; then the error is calculated between the network outputs and the sample targets. This procedure is repeated for different $\sigma$ values. The smoothing parameter ($\sigma$) with the lowest sum of errors is selected as the best $\sigma$. The holdout algorithm is explained in Algorithm 5.

Other search and optimization methods might be also used to find $\sigma$. For instance, genetic algorithms (GA) and differential evolution (DE) are suitable options. Algorithm 6 explains how to find $\sigma$ using DE or GA. Also, the results of using DE and GA are depicted in Figure 2. Both of GA and DE can find a good approximation of $\sigma$ within 100 iterations only; however, DE converges faster since it is a vectorized algorithm.

Figure 2.

DE and GA used to estimate GRNN $\sigma$. (a) Estimation of $\sigma$ using DE (b) MSE evolution when using DE to estimate s (c) Estimation of $\sigma$ using GA (d) MSE evolution when using GA to estimate $\sigma$.

4. GRNN vs. back-propagation neural networks (BPNN)

There are many differences between GRNN and BPNN. Firstly, GRNN is single-pass learning algorithm, while BPNN needs two passes: forward and backward pass. This means that GRNN consumes significantly less training time. Secondly, the only free parameter in GRNN is the smoothing parameter $\sigma$, while in BPNN more parameters are required such as weights, biases, and learning rates. This also indicates that GRNN quick learning abilities and its suitability for online systems or for system where minimal computations are required. Also, another difference is that since GRNN is an autoassociative memory network, it will store all the distinct input/output samples while BPNN has a limited predefined size. This size growth issue is resolved by either using clustering or PCA (read Sections 2.21 and 2.2.2). Finally, GRNN is based on the general regression theory, while BPNN is based on gradient-descent iterative optimization method.

To show the advantages of GRNN over BPNN, a comparison is held using standard regression datasets built inside MATLAB software [25]. For all the datasets, they are divided 70% for training and 30% for testing. After training the network with the 70% training data, the output of the neural network is found using the remaining testing data. The most notable advantage of GRNN over BPNN is the shorter training time which confirms its selection for dynamic systems modeling and control. Also, GRNN has less testing error which means it has better generalization abilities than BPNN. The comparison results are summarized in Table 3.

Type	Dataset	Training time (sec)	Training error (MSE)	Testing error (MSE)
GRNN	Abalone	0.621	0.342	0.384
BPNN	Abalone	1.323	0.436	0.395
GRNN	Building energy	0.630	0.0731	0.628
BPNN	Building energy	1.880	0.1152	0.631
GRNN	Chemical sensor	0.701	0.888	1.316
BPNN	Chemical sensor	1.473	0.228	1.584
GRNN	Cholesterol	0.801	0.037	0.172
BPNN	Cholesterol	2.099	0.061	0.215

Table 3.

GRNN vs. BPNN training and testing performance.

5. GRNN in identification of dynamic systems

System identification is the process of building a model of unknown/partially known dynamic system based on observed input/output data. Gray-box and black-box identification are two common approaches of system identification. In the gray-box approach, a nominal model of a dynamic system is known, but its exact parameters are unknown, so an identifier is used to find these parameters. In the black-box approach, the identification is based only on the data. Examples of black-box identification include fuzzy logic (FL) and neural networks (NN). GRNN can be used to identify dynamic systems quickly and accurately. There are two methods to use GRNN for system identification: the batch mode (off-line training) and sequential mode (online training). In the batch mode, all the observed data is available before the system identification, so GRNN can be trained with a big chunk of the data, while in the sequential mode only a few data samples are available for identification.

5.1. GRNN identification in batch training mode

In the batch mode, the observed data should be divided into training, validation, and testing. GRNN will be fed with all the training data to identify the system. Then in the validation stage, the network should be tested with different data, usually randomly selected, and the error is recorded for every validation test. Then the validation process is repeated several times. Usually 10 times is standard. And then the average validation error is found based on all the validation tests. This validation procedure is called k-fold cross validation a standard technique in machine learning (ML) applications. To test the generalization ability of an identified model, a new dataset is used called testing dataset. Based on the model performance in the testing stage, one can decide whether the model is suitable or not.

5.1.1. Batch training GRNN to identify hexacopter attitude dynamics

In this example, GRNN is used to identify the attitude (pitch/roll/yaw) of a hexacopter drone based on real flight test data in the free flight mode. The data consist of three inputs: rolling, pitching, and yawing control values and three outputs: rolling, pitching, and yawing rates. The dataset contains 6691 data samples with a sample rate of 0.01 seconds. A total of 4683 samples are used to train GRNN in the batch mode, and the remaining data samples (2008) are used for testing. The results of hexacopter attitude identification are shown in Figure 3(a–c). The results are accurate with very low error. MSE in training stage is 0.001139 and 0.00258 in the testing stage. Also, the training time was only 0.720 seconds.

Figure 3.

Attitude identification of hexacopter in batch training: (a) rolling rate identification, (b) pitching rate identification, and (c) yawing rate identification.

5.2. GRNN identification in sequential training mode

In sequential training, the data flow once at a time which makes using the batch training procedures impossible. So GRNN should be able to find the system model from only the current and past measurements. So it is a prediction problem. Since GRNN converges to a regression surface even with a few data samples and since it is accurate and quick, it can be used in the online dynamic systems identification.

5.2.1. Sequential training GRNN to identify hexacopter attitude dynamics

To use GRNN in sequential mode, it is preferred to use the delayed output of the plant as an input in addition to the current input as shown in Figure 4. The same data which was used for batch mode is used in the sequential training. The inputs to GRNN are the control values of rolling, pitching, and yawing and the delayed rolling, pitching, and yawing rates. The results of using GRNN in the sequential training mode are shown in Figure 5(a–c). The results of sequential training are more accurate than the results in batch training.

Figure 4.

Sequential training GRNN.

Figure 5.

Attitude identification of hexacopter in sequential training: (a) rolling rate identification, (b) pitching rate identification, and (c) yawing rate identification.

6. GRNN in control of dynamic systems

The aim of adding a closed-loop controller to the dynamic systems is either to reach the desired performance or stabilize the unstable system. GRNN can be used in controlling dynamic systems as a predictive or feedback controller. GRNN in control systems can be used as either supervised or unsupervised. When GRNN is trained as a predictive then the controller input and output data are known, so this is a supervised problem. On the other hand, if GRNN is utilized as a feedback controller (see Figure 6) without being pretrained, only the controller input data is known so GRNN have to find the suitable control signal $u$.

Figure 6.

Unsupervised learning problem in control.

6.1. GRNN as predictive controller

To utilize GRNN as a predictive controller, it should be trained with input-output data from another controller. For example, training a GRNN with a proportional integral derivative (PID) controller input/output data as shown in Figure 7. Then the trained GRNN can be used as a controller.

Figure 7.

Training GRNN as predictive controller.

6.1.1. Example 1: GRNN as predictive controller

If we have a discrete time system Liu [26] described as

$$y\left(k+1\right)=0.8\ast \mathit{sin}\left(y\left(k\right)\right)+15\ast u\left(k\right)$$E13

The desired reference is $y_d\left(k\right)=2^{\ast }\mathit{sin}\left(0.1\mathit{\pi t}\right)$.

The perfect control law can be written as

$$u\left(k\right)=\frac{y_d\left(k+1\right)}{15}-\frac{0.8\ast \mathit{sin}\left(y\left(k\right)\right)}{15}$$E14

To train GRNN as a predictive controller, the system described in (13) and (14) is simulated for 50 seconds. Then the controller output $u$ and the plant output $y$ were stored. GRNN is trained with the plant output as input and the controller output as output. For any time step the plant output is fed to GRNN, and the controller output $u$ is estimated. The estimated controller output by GRNN and the perfect controller output are almost identical as shown in Figure 8. Also, the tracking performance after using GRNN as a predictive controller is very accurate as shown in Figure 9.

Figure 8.

Perfect vs. estimated GRNN controller output.

Figure 9.

GRNN tracking performance.

6.2. GRNN as an adaptive estimator controller

Since GRNN has robust approximation abilities, it can be used to approximate the dynamics of a given system to find the control law especially if the system is partially known or unknown.

Assume there is a nonlinear dynamic system written as

$$\dot{x}=f\left(x,t\right)+\mathit{bu}+d$$E15

where $\dot{x}$ is the derivative of the states, $f\left(x,t\right)$ is a known function of the states, $b$ is the input gain, and $d$ is the external disturbance.

The perfect control law can be written as

$$u=\frac{1}{b}\left(\dot{x}-f\left(x,t\right)-d\right)$$E16

If $f\left(x,t\right)$ is unknown, then the control law in (16) cannot be found; hence, the alternative is using GRNN to estimate the unknown function $f\left(x,t\right)$. To derive the update law of GRNN weights, let us define the objective function as MSE error function as follows:

$$E=\frac{1}{2}{\left(\widehat{y}-y\right)}^2$$E17

where $\widehat{y}$ is the estimation of GRNN and $y$ is the optimal value of $f\left(x,t\right)$. To derive the update law of the GRNN weights, the error should be minimized with respect to GRNN weights $W$:

$$\frac{\partial E}{\partial W}=\left(\widehat{W}H-y\right)\ast H$$E18

where $\widehat{W}$ is the GRNN current hidden-output layers weights and $H$ is the hidden layer output, so the update law of GRNN weights will be

$$W_{i+1}=W_i+H\left(\widehat{W}H-y\right)$$E19

6.3. Example 2: using GRNN to approximate the unknown dynamics

Let us consider the same discrete as in example 1:

$$y\left(k+1\right)=f\left(k\right)+15\ast u\left(k\right)$$E20

The desired reference is $y_d\left(k\right)=2^{\ast }\mathit{sin}\left(0.1\mathit{\pi t}\right)$

where $f\left(k\right)$ is unknown nonlinear function.

The perfect control law can be written as

$$u\left(k\right)=\frac{-f\left(k\right)}{15}+\frac{y_d\left(k\right)}{15}$$E21

GRNN is used to estimate the unknown function $f\left(k\right)$. With applying the update law in (19), $f\left(k\right)$ is estimated with an acceptable accuracy as shown in Figure 10. MSE between the ideal and the estimated $f\left(k\right)$ is 0.0033. The accurate controller tracking performance is also shown Figure 11.

Figure 10.

Using GRNN to estimate the unknown dynamics.

Figure 11.

GRNN tracking performance for example 2.

6.4. GRNN as an adaptive optimal controller

GRNN has learning abilities which means it is suitable to be an adaptive intelligent controller. Rather than approximating the unknown function in the control law (16), one can use GRNN to approximate the whole controller output as shown in Figure 12. The same update law as in (19) can be used to update GRNN weights to approximate the controller output $u$.

Figure 12.

Training GRNN as an adaptive controller.

6.4.1. Example 3: using GRNN as an adaptive controller

Let us consider the same discrete system as in (13):

$$y\left(k+1\right)=0.8\ast \mathit{sin}\left(y\left(k\right)\right)+15\ast u\left(k\right)$$

with the same desired reference $y_d\left(k\right)=2\ast \mathit{sin}\left(0.1\mathit{\pi t}\right)$, but in this case GRNN is used to estimate the full controller output u as shown in Figure 14 and the tracking performance is shown in Figure 13.

Figure 13.

GRNN tracking performance for example 3.

Figure 14.

GRNN Estimated control law for example 3.

6.4.2. Example 4: using GRNN as an adaptive controller

Let us use GRNN to control a more complex discrete plant [27] described as

$$\begin{array}{l}y\left(k+1\right)=0.2\mathit{cos}\left(0.8\left(y\left(k\right)+y\left(k-1\right)\right)\right)+0.4\mathit{sin}\left(0.8\left(y\left(k-1\right)+y\left(k\right)+2u\left(k\right)+u\left(k-1\right)\right)\right)\\ +0.1\left(9+y\left(k\right)+y\left(k-1\right)\right)+\frac{2\left(u\left(k\right)+u\left(k-1\right)\right)}{(1+\mathit{cos}\left(y\left(k\right)\right)}\end{array}$$E22

The desired reference in this case is

$$y_d\left(k\right)=0.8+0.05\left(\mathit{sin}\left(\mathit{\pi k}/50\right)+\mathit{sin}\left(\mathit{\pi k}/100\right)+\mathit{sin}\left(\mathit{sin}\left(\mathit{\pi k}/150\right)\right)\right)$$

The tracking performance of adaptive GRNN is shown in Figure 15.

Figure 15.

GRNN as an adaptive controller in Example 4.

7. MATLAB examples

In this section, GRNN MATLAB code examples are provided.

7.1. Basic GRNN Commands in MATLAB

In this example, GRNN is trained to find the square of a given number.

To design a GRNN in MATLAB:

Firstly, create the inputs and the targets and specify the spread parameter:

Secondly, create GRNN:

To view GRNN after creating it:

The results are shown in Figure 16.

Figure 16.

View GRNN in MATLAB.

To find GRNN output based on a given input:

The result is 17.

7.2. The holdout method to find σ