## 1. Introduction

This chapter describes the development of voltage control system of a synchronous generator based on neural networks. Recurrent (dynamic) neural networks (RNN) are used, as a type that has great capabilities in approximation of dynamic systems [1]. Two algorithms are used for training – Dual Heuristic Programming (DHP) and Globalized Dual Heuristic Programming (GDHP). The algorithms have been developed for the optimal control of nonlinear systems using dynamic programming principles.

Neural voltage controller is developed in MATLAB and Simulink environment. For training purposes a mathematical model of synchronous generator is designed and applied in Simulink. DHP and GDHP algorithms are designed in Simulink, with matrix calculations in S-functions. Algorithms are used for offline training of neural networks (NN). In the second part, the same functions are redesigned as real time controllers, based on the Real Time Windows Target Toolbox.

DHP or GDHP algorithms provide a significant improvement over conventional PI controller with, in some cases, power system stabilizer (PSS). Conventional linear controller can only provide optimal control in single operating point. Algorithms described in the chapter provide significantly better control over the whole operating range by minimization of the user defined cost function during the training of the neural network.

Digital voltage controller is implemented on real system in two basic steps. First step is neural network design and training in Matlab environment on desktop computer. Second step consists of transfer of controller with computed coefficients to the digital control system hardware.

For the controller to have optimal performance in all real operating conditions, neural networks must be trained for the whole working range of the plant. Procedure described in the chapter can be applied on standard voltage control systems by replacing the PI controller with the offline-trained neural network. Most modern digital control systems have sufficient hardware and software support for neural network implementation.

## 2. Recurrent neural networks

Neural networks are computational models capable of machine learning and pattern recognition. They can also be used as universal approximators of continuous real functions. Typical dynamic neural network is shown in Figure 1.

Neural network is formed from interconnected layers of neurons. Output of each neuron is calculated from:

where

* b* is bias.

Typical activation function maps input values * logsig*,

*and Gaussian.*tansig

Dynamic behavior is added by introducing a memory element (step delay) that stores neuron output and reintroduces it as input to all neurons in the same layer in next time step.

This type of network is named recurrent or dynamic neural network. An example of tansig (* Hyperbolic Tangent Sigmoid Transfer Function*) activation function,

Input values are in range of

Use of function with simple derivative significantly improves calculation times of numerical procedures.

Dynamic (recurrent) neural networks provide good approximation of time-dependent functions. However, training of recurrent neural networks is more complex than training static networks.

### 2.1. Recurrent neural networks training

There are many methods for training neural networks, and most of them rely on some form of backpropagation – calculation of partial derivative of system output over network weight coefficients

Described procedure is valid for static neural networks, but is of no use in recurrent networks, as each neuron has inputs from previous time step from the same layer. Two procedures are often used in calculation of partial derivatives: * Backpropagation Through Time* (

*) and*BPTT

*(*Real Time Recurrent Learning

*).*RTRL

In BPTT recurrent neuron inputs are replaced with the same neural network delayed by one time step. Process is iterated N times. Thus, recurrent network is approximated by N feedforward networks with delayed outputs. Network weight coefficients, recurrent inputs and outputs must be preserved for past N time steps. Obtained derivative

This chapter will describe use of RTRL procedure in training dynamic networks. Output of recurrent network with one hidden and one output layer has the following form:

where

y(t) – network output,

Equation (2) can easily be expanded for multi layer recurrent networks. Partial derivative of network output over output layer weight coefficients is

For hidden layer, partial derivatives are

Partial derivative of hidden layer output over weight coefficients is

where

Value of

Matrix of output derivatives over weight coefficients is of the form

### 2.2. Neural network training algorithms

#### 2.2.1. Gradient descent

Gradient descent is the most commonly used method in neural networks training. Difference between real and desired network output can be expressed as

Differentiation of (5) provides update values of weight coefficients:

Parameter

#### 2.2.2. Kalman filter based training

Kalman filter [5] was developed as a recursive procedure of optimal filtering of linear dynamic state-space systems. For RNN training extended Kalman filter (EKF) is used. It is also applicable to nonlinear systems using linearization around operating point. The algorithm is used as a predictive-corrective procedure.

Kalman filter based recurrent network training is described in [6] and [7].

Training is performed in two steps. The first step is calculation of prediction values of weight coefficients. Second step is correction of predicted variables based on measured state space.

Algorithm is defined by the following relations:

where

To escape local minima during training, a diagonal matrix

Updates to weight coefficients are determined from (9), as a product of Kalman gain and

Training based on Kalman filter is fast, has good rate of convergence and is often the only practical solution for recurrent networks training. However, the procedure is demanding in terms of computational power and memory requirements.

## 3. System identification

Nonlinear dynamic systems are identified as input-output system model, or as a state space system. For the selected process, identification is performed by minimization of the cost function

where

Selected cost function represents squared error of the identified model.

Dynamic system identification can be performed according to [9], using either series-parallel or parallel structure.

In series-parallel structure Figure 3, model neural network input vector is expanded with system outputs from previous steps. In parallel structure Figure 3, network output from previous states is added to input vector. Series-parallel identification provides estimated system output for next time step only, while parallel identification can predict multiple future iterations. However, parallel structure is harder to train and is susceptible to state drifting.

To verify the validity of the RTRL algorithm, neural networks were trained on several discrete mathematical functions. Supervised training has been used, with control value expressed as difference between desired and actual network output.

A parallel identification structure was used on dynamic SISO system

Figure 4a compares initial response of model output

Input signal used for training and verification is

In second test, a system with one input, two state variables and one output has been identified:

System was identified using series-parallel structure with RNN with 5 neurons in hidden layer, and one linear neuron in output layer. Identification results are shown in Figure 5.

Input signal used for training and verification is the same as in previous example:

Both examples were designed in Matlab and Simulink. Neural network was trained using c-language S-function (Appendix B). Although the training took 10000 steps, all important system dynamics were visible on trained network after 2000 steps.

### 3.1. Determination of partial derivatives of functions using neural networks

One of the most important features of neural network modeling is the possibility to determine system output derivatives over inputs

Partial derivative is calculated from (2)

where

For nonlinear function

Results from Figure 6b show good match between calculated function derivatives and values obtained from neural network of identified model.

In S-function for training RNN (Appendix B) a procedure is shown for online calculation of partial derivatives of

## 4. Adaptive critic design

Adaptive critic design (ACD) is a set of optimal control procedures of nonlinear systems. It is based on dynamic programming and neural networks. Optimal control is achieved by training neural networks and using them in the structure according to the selected ACD algorithm. ACD algorithms are implemented using a heuristic approach – system state space variables, control values and cost function are selected based on process characteristics. This text focuses on the Heuristic Dynamic Programming (HDP), Dual Heuristic Programming (DHP) and General Dual Heuristic Programming (GDHP) [1,2].

### 4.1. Heuristic dynamic programming

Heuristic dynamic programming (* HDP*) is the simplest (by structure) ACD algorithm. The basis of all HDP algorithms is minimization of Bellman dynamic programming function

where

Control signal that minimizes Bellman sum is generated using one of the Adaptive Critic algorithms. In practical control systems, HDP optimization is realized with three neural networks with a structure as shown in Figure 7.

Model neural network in * 7a* is used for model identification. Network inputs are control signal

Model neural network is trained to minimize the quadratic criterion:

where:

For adaptive control systems, model NN is trained continuously, allowing for adaptation to change of system parameters, and optimal control. Model neural network is trained simultaneously with other networks over time.

Critic neural network in * 7b* is used for identification of Bellman sum

Training is used to minimize

where:

Critic neural network training is performed with a copy of the same network which is fed with predicted future system outputs

Action neural network training is based on model and critic network outputs, by minimizing criterion:

where:

Partial derivatives

Minimization of scalar product

Action neural network is trained from the output network (unsupervised algorithms). Training of neural networks is carried out using (19) with the training concept shown in Figure 7c.

### 4.2. Dual heuristic programming

Dual Heuristic Programming optimal control algorithm is developed with a goal of increasing procedure efficiency by increasing training speed.

Model network training is the same as in HDP procedure. The improvement is in the direct calculation of the Bellman sum partial derivative over state space variables

Structure of DHP control system is shown in Figure 8.

Training of critic neural network minimizes scalar product:

with

where

where

* n* is size of model network output vector

* m* is size of control vector

Third part of equation (22) is determined from deriving

Somewhat simplified structure of DHP is shown in Figure 8a. Partial derivatives

Action neural network is trained to minimize Bellman sum,

Thus, equation (22) can be transformed to

Training of Action Network minimizes scalar product:

with

Simplified training procedure of action NN is shown in Figure 8a.

Cross training of critic and action networks results in slower approximation. In order to reduce training time, it is recommended to begin the procedure with action network pre-trained as linear controller.

### 4.3. Global dual heuristic programming

Globalized Dual heuristic optimal control algorithm uses both algorithms described.

Critic neural network provides approximation of Bellman sum

Action network is trained using the same procedure as described for HDP and DHP.

## 5. Neural networks based control of excitation voltage

Main elements of electric power system are generators, transmission and distribution network and consumers. Transmission and distribution network form a power grid, which is usually very large in geographic terms. Grid control is implemented on both local and global level, mostly on the power generation side of the grid. Specific to power grid control is large number of closed control loops governing the power and voltage of synchronous generators. Transmission network is governed by setting voltage on transformers in order to minimize transmission losses. Governing of generator power and voltage achieves balance of power as well as reactive power flow on the transmission system. Voltage control also has a large impact on the dynamic system behavior.

Generator power output is governed on the generator unit. For water and steam turbine generators, it defines local power transmission to the grid, while also influencing system frequency. Considering the inertness of the generator systems, power output is not difficult to govern. Voltage control manages the balance between generator and grid voltage, and determines reactive power flow. A change in generator voltage setpoint has impact on system state variables and parameters. Disregarding system transients, a change of generator voltage causes a change on generator busbars, generator current and reactive power.

If we also consider system dynamics, generator voltage change also changes system parameters influencing system damping. Synchronous generator connected to the power grid is dominantly nonlinear in characteristics. Generator output power is proportional to generator and grid voltage, while the relation to angle between generator and grid voltage is a sinus function. Maximum theoretical output power is realized with a load angle being

Operating point of generator connected to the grid is determined by steady state excitation current. Excitation voltage and current values dictate generator voltage, reactive power and load angle.

Frequent small disturbances expected and part of normal operating conditions, resulting in oscillations of generator electromechanical variables. Especially pronounced is the swinging of generator power, reaching several percents of nominal value, with a frequency around 1Hz. It is caused by low system damping and outer disturbances on the grid. Also present on the grid are oscillations caused by the system characteristic frequencies, usually around 0.1Hz.

Power grid can be represented as a single machine connected to an infinite bus system. For each operating point a characteristic frequency and damping can be determined.

In case of small disturbances with a synchronous generator working around predefined operating point, system can be considered linear, but only if it returns to the same operating point after each disturbance. For those conditions, power system stabilizer (* PSS*) is added to AVR. Based on change of output power and load angle, it modifies voltage regulator setpoint and increases system damping. PSS is configured using linear control theory, and is presently a standard part of automatic voltage regulator systems [10].

Large disturbances occur after short circuits on the power grid, transmission and generator outages, and connection and disconnection of large loads.

Electromechanical oscillations occurring as a consequence of disturbances cause damped oscillations of all electrical and mechanical values (generator voltages and currents, active and reactive power, speed and load angle). Oscillations are also propagated by AVR to excitation system values.

Large system disturbances can cause significant changes to local parameters relevant for generator stability. Due to the linear nature of the voltage regulator system, a change in grid voltage or configuration leads to generator working in non optimal state. Even if the system returns to the same operating conditions, passing through nonlinear states during disturbance causes non optimal control, diminishing the stabilizing effects of the voltage regulator on the system. This can in turn cause another disturbance or, in case of larger units, disruption of the whole power grid. Limitations of the use of linear control systems call for an improved system which can provide optimal control of generator excitation system [13]- [15].

There is a lot of research effort directed toward design of excitation control system based on neural networks. However, very few of the demonstrated solutions have been implemented on real plants.

The goal of the chapter is to develop a practical-to-use system. This is mainly achieved by separating the design on preparation and implementation phase. Almost all algorithm steps are executed in off-line mode, up to the final step of digital voltage regulator implementation.

To implement the voltage regulator described in the chapter following equipment is needed:

data acquisition hardware and software

personal computer with Matlab and Simulink environment

automatic voltage regulator with neural network support (no training support is needed)

Described controller is not adaptive. Adaptive critic design (ACD) algorithm is used for optimal control.

Classic voltage regulators (AVR) equipped with PSS can be very effective in voltage control and power system stabilization if it is used in well configured transmission grid. In such conditions the generator stays in the linear operating range and the controller maintains optimal configuration on change of active and reactive power. Voltage regulator accuracy, balance of reactive power, stabilizing effect during large disturbances and active stabilization in normal operating conditions are achieved.

If the generator is connected to relatively weak transmission grid, with possible configuration changes during operation, use of neural network based control of excitation voltage leads to better results.

### 5.1. HDP algorithm implementation

In order to achieve optimal control of synchronous generator connected to the power grid, it is necessary to select a minimum of two system variables – generator voltage and either output power or angular velocity. Selected variables must reflect system oscillations during disturbances. It is also necessary to choose cost function that is minimized during training. System optimization is achieved by minimization of squared generator voltage governing error

It is important to notice that the two demands are in conflict. Increase in gain in main control loop reduces static control error, but also reduces system damping and increases power output deviation

System identification is carried by training model neural network. Figure 8 shows training process of model neural network. Identification of nonlinear system is made by series-parallel structure.

For ACD algorithm optimal control two output state space variables are chosen – generator voltage and output power,

where

Model NN has 9 inputs and 2 outputs.

Figure 9 shows simulation block diagram of identification procedure, implemented in Matlab and Simulink environment.

A third order nonlinear mathematical model was used as a synchronous generator [20]. Synchronous generator is modeled as a single machine connected to an infinite bus. Although the model ignores initial magnetic flows, it proves to be a good approximation of an actual generator. A predictor-corrector method is used to solve differential equations.

Selected mathematical model, used with well conditioned state space variables and described methods, achieves good numerical stability during simulation, allowing for relatively large sample time (T_{D}=0.02s). Described model of synchronous generator is implemented in C-language S-function and used for further simulations.

For model identification, recurrent neural network is used, with five neurons in hidden layer and two neurons in output layer. Hidden layer is implemented with full recursion and * tansig* activation function. Output layer uses linear activation function.

Initial learning rate value

In the same experiment efficiency of the algorithm was verified. Training was stopped after 100000 steps. For the rest of the simulation, network output is calculated without the change of weight coefficient, thus eliminating the recency effect.

Once the model neural network is trained, it is possible to create simulation for training critic and action neural networks. Simulink model of the simulation is shown in Figure 11.

Simulation uses expressions (21), (22) and (23).

Neural network training is implemented in Matlab S-functions, written in C programming language.

Input of the S function is formed of system signals and desired value. S-function returns RNN output signals and partial derivatives of outputs

Training results are shown in Figure 12. Figure 12a compares generator voltage response using conventional automatic voltage regulator (AVR) to neural network based controller trained using HDP algorithm. It can be seen that voltage rate of change was not degraded as a result of system damping. It is important to emphasize that generator voltage rate of change is key to system stability during large disturbances.

Figure 12b compares active power output oscillation of conventional and HDP controller. Figure 12c shows load angle, which is indicative of system stability

HDP algorithm is also verified on voltage control of synchronous generator from Simulink SimPower System block library (Synchronous Machine).

Model network is designed with one hidden recursion layer with 5 neurons. Critic and action networks use the same structure, but with four neurons in hidden layer.

HDP algorithm shown in Figure 11 is applied. Training of neural networks is performed in two steps– in first step action NN is trained to mimic classic voltage regulator, and in second step DHP algorithm is used to train both action and critic NN. It is important to get initial values of action NN in first step, as DHP needs network partial derivatives in training critic NN. Figure 13 shows training results.

Figure 13a shows comparison of classic and NN controller responses. Generator voltage rate of change is almost the same for both controllers.

Figure 13c and 13e show comparison of output power and load angle change. It is obvious that the HDP neural network controller shows significant improvement over classic regulator.

Figure 13b shows comparison of voltage transient in the condition generator voltage reduced to 90% of nominal value. Rate of change of generator voltage is maintained even during such extreme operating conditions.

Figure 13d shows conventional automatic voltage regulator (AVR) operating on the edge of stability, while the DHP controller is completely stable. Comparison of load angle values demonstrates significantly better behavior of DHP controller. On large load angles DHP controller maintains stability while conventional AVR provides low system damping, leading to instability and possible outages.

Neural network HDP voltage controllers were developed using DHP algorithm. Experimental results showed that DHP algorithm is more efficient than ADC algorithm. Implementation of GDHP does not bring significant improvements over DHP.

Application of developed S-functions enables simple use of described algorithms, as well as experimenting with different function parameters.

## 6. Real time windows target implementation

DHP optimal control algorithm has also been implemented using Matlab Real Time Windows Target Toolbox (RTWT), based on [18] and [19]. Simulation hardware consists of desktop PC equipped with National Instruments data acquisition card NI DAQ 6024.

Use of RTWT platform enables direct implementation of developed controllers on the real system.

In Figure 14, a system consisting of synchronous generator and RTWT controller is shown. Signal acquisition is achieved using DAQ 6062E Input, with AD conversion of the input values. Line voltage and phase current signals are processed in “Clark” S-function, using Clarke transformation. Transformed values of generator voltage * P* are used as control feedbacks. Digital to analog conversion is performed by output module DAQ 6062 Output. All simulation blocks are part of a standard Simulink library except Clark transformation and RNN S-functions.

ACD algorithm is implemented in two steps. Classic voltage controller is used in the first step (Figure 14). In the second step it is replaced with NN controller.

During testing, generator active power and voltage were changed for a wide range of values. A pulse signal was superimposed to the generator voltage set-point value, resulting in large disturbances on the generator. To obtain measurement records RTWT Signal Visualization was used, with measured data saved in Workspace Simulink block * yP*. Recorded data was used to train neural network in offline mode.

A RNN was used with one hidden layer containing 5 tansig neurons. Model NN had 9 inputs and 2 outputs. Results of NN training are displayed in Figure 15.

Obtained model NN was used to train critic and action RNN using procedure described in 3. The trained action NN was transferred to Simulink block (Figure 14) and prepared for use in real time. Using RTWT module system was run in real-time mode, with trained action RNN as a voltage controller.

Simulation results are shown in Figure 16, comparing classic and DHP voltage controllers. Figure 16a represents comparison between generator voltage responses and shows satisfactory rate of generator voltage change and high control accuracy. In Figure 16b, generator active power outputs are compared. Considerable damping was achieved using DHP controller. Improved system stability during large disturbances is visible in Figure 16c, where the maximum load angle deviation is significantly lower for DHP controller.

RTWT controller was implemented in sample-time mode with sample time set to

Matlab Simulink RTWT Toolbox simplifies the transfer of developed procedures to the hardware platform.

## 7. Conclusion

In the chapter, a complete process of development and implementation of neural network based controller is described. Optimal control is achieved by training neural network using data obtained from real system. Developed algorithms were tested on voltage control of synchronous generator connected to the power grid.

Optimal control algorithm is implemented in Matlab and Simulink environment. Neural network training is implemented in C programming language Matlab S-function. S-function execution time was sufficiently low to achieve real time performance.

Developed algorithm is tested using Matlab Real Time Windows Target Toolbox, using the same models with minimal modifications.

Experiments performed on laboratory system show possibility of building optimal controller based on special network training procedure with no need for adaptive features.

Obtained simulation results show advantages of neural networks based controller over classic controllers. Developed procedure can relatively easily be adapted for use on real systems.

## Appendices

### Appendix A

Matlab function for calculating hidden layer derivatives of outputs over weight coefficients of hidden layer.

### Appendix B

S-function for RNN training with one hidden layer.

The first part defines sample time, number of inputs, outputs and neurons. Input is formed of values used for training. Output 1 is RNN output. Output 2 is partial derivative of output over input. Training is performed using Kalman filter.

## References

- 1.
P.Werbos "Aproximate dynamic programming for real-time control and neural modeling" in Handbook of Intelligent Control, White and Sofge Eds. New York: Van Nostrand Reinhold, pp493-525. - 2.
D. Prokhorov: "Adaptive critic design", IEEE Trans. Neural Networks, vol. 8, pp. 997-1007, Sept 1997. - 3.
Wiliams, R. J. and Zisper D: "A learning agorithm for continually running fully recurrent neural networks", Neural Copmutation, 1:270-280. - 4.
Pedro DeLima: "Application of Adaptive Critic Designs for Fault Tolerant Control", IEEE Computational intelligence Society Walter J Karplus Summer Research Grant 2004. - 5.
R.E. Kalman, ‘‘A new approach to linear filtering and prediction problems,’’Transactions of the ASME, Ser. D, Journal of Basic Engineering, 82, 35–45 (1960). - 6.
G.V. Puskorius and L. A. Feldkamp: "Neurocontrol of Nonlinear Dynamic Systems with Kalman Filter Trained Recurrent Networks", IEEE Trans. on Neural Networks, 5(14), 279-297. - 7.
Simon S. Haykin: “Kalman Filtering and Neural Networks”, 2001 - 8.
M. Cernasky, L. Benuskova:“ Simple recurrent network trained by RTRL and extended Kalman filter algorithms“ Neural Network World 13(3) (2003) 223–234 - 9.
Narendra K.S., Parthasarathy K. “Identification and Control of Dynamical Systems Using Neural Networks”, IEEE transaction on Neural Networks, 1:4-27, March 1990. - 10.
IEEE recommended practice for excitation system models for power system stability studies, : IEEE St. 421.5-2002 (Section 9). - 11.
W. Mielczarski and A. M. Zajaczkowski "Nonlinear Field Voltage Control of a Synchronous Generator using Feedback Linearization", Automatica Vol 30, pp1625-1630, 1994. - 12.
P. Shamsollahi, O. P. Malik, "Real time Implementation and Experimental Studies of a Neural Adaptive Power System Stabilizer", IEEE Trans. on Energy Conversion, Vol. 14. No. 3, September 1999. - 13.
G. K. Venayagamoorthy, Ronald G. Harley, and Donald Wuncsh "Implementation off an adaptive neural network for efective control of turbogenerators” IEEE Budapest Power Tech. Conf. 1999. paper BPT99 pp 431 – 23. - 14.
T. Kobayashi and A. Yokoyama, "An Adaptive Neuro-Control System of Synchronous Generator for Power system stabilization", IEEE Trasaction on Energy Conversion, Vol. 11, No. 3, September 1996. - 15.
D.Flynn, S. McLonne, G. W. Irwin, M. D. Brown, E. Swidenbank, and B. W. Hogg, "Neural control turbogeneraroe systems", Automatica, vol 33, no,11, pp. 1961 – 1973, 1997. - 16.
Q.H. Wu, B.W. Hogg, G.W. Irwin " A neural network regulator for turbogenerators," IEEE Trans. on Neural Network, vol. 3, no. 1, pp. 95-100, Jan 1992. - 17.
G. K. Venayagamoorthy, Ronald G. Harley, and Donald Wuncsh "Implementation off an adaptive neural network for efective control of turbogenerators” IEEE Budapest Power Tech. Conf. 1999. paper BPT99 pp 431 – 23. - 18.
MATLAB the Language of Technical Computing, http://www.mathworks.com/help/rtwin/index.html - 19.
MATLAB the Language of Technical Computing, http://www.mathworks.com/products/matlab - 20.
M.Miskovic " Extended Synchronous Generator Operation Region Using Neural Network Based Adaptive Control ", PhD thesis, FER Zagreb 2007. - 21.
M. Miskovic, M.Mirosevic, G.Erceg “ Load angle estimation of a synchronous generator using dynamical neural networks “//Energija 02 (2009.) ; 174-191. - 22.
Miskovic, Mato; Mirosevic, Marija; Miskovic, Ivan. On-line Identification of Synchronous Generator Mathematical Model // ICRERA 2012 Proceedings / Kurokawa, Fujio (ur.). Nagasaki, Japan : University of Nagasaki, 2012. 1-3 - 23.
Matuško, Jadranko; Petrović, Ivan; Perić, Nedjeljko. Neural network based tire/road friction force estimation. Engineering Applications of Artificial Intelligence. (2007), 15 pages - 24.
Sumina, Damir; Bulic, Neven; Miskovic, Mato. Parameter tuning of power system stabilizer using eigenvalue sensitivity. // Electric power systems research. 81 (2011), 12; 2171-2177