This chapter describes the development of voltage control system of a synchronous generator based on neural networks. Recurrent (dynamic) neural networks (RNN) are used, as a type that has great capabilities in approximation of dynamic systems . Two algorithms are used for training – Dual Heuristic Programming (DHP) and Globalized Dual Heuristic Programming (GDHP). The algorithms have been developed for the optimal control of nonlinear systems using dynamic programming principles.
Neural voltage controller is developed in MATLAB and Simulink environment. For training purposes a mathematical model of synchronous generator is designed and applied in Simulink. DHP and GDHP algorithms are designed in Simulink, with matrix calculations in S-functions. Algorithms are used for offline training of neural networks (NN). In the second part, the same functions are redesigned as real time controllers, based on the Real Time Windows Target Toolbox.
DHP or GDHP algorithms provide a significant improvement over conventional PI controller with, in some cases, power system stabilizer (PSS). Conventional linear controller can only provide optimal control in single operating point. Algorithms described in the chapter provide significantly better control over the whole operating range by minimization of the user defined cost function during the training of the neural network.
Digital voltage controller is implemented on real system in two basic steps. First step is neural network design and training in Matlab environment on desktop computer. Second step consists of transfer of controller with computed coefficients to the digital control system hardware.
For the controller to have optimal performance in all real operating conditions, neural networks must be trained for the whole working range of the plant. Procedure described in the chapter can be applied on standard voltage control systems by replacing the PI controller with the offline-trained neural network. Most modern digital control systems have sufficient hardware and software support for neural network implementation.
2. Recurrent neural networks
Neural networks are computational models capable of machine learning and pattern recognition. They can also be used as universal approximators of continuous real functions. Typical dynamic neural network is shown in Figure 1.
Neural network is formed from interconnected layers of neurons. Output of each neuron is calculated from:
is activation function,
and are weights and outputs from previous layer and
Typical activation function maps input values to output or . Activation function must be derivable over the whole range. Among typically used nonlinear activation functions are
Dynamic behavior is added by introducing a memory element (step delay) that stores neuron output and reintroduces it as input to all neurons in the same layer in next time step.
This type of network is named recurrent or dynamic neural network. An example of tansig (
Input values are in range of , with output values in range .
Use of function with simple derivative significantly improves calculation times of numerical procedures.
Dynamic (recurrent) neural networks provide good approximation of time-dependent functions. However, training of recurrent neural networks is more complex than training static networks.
2.1. Recurrent neural networks training
There are many methods for training neural networks, and most of them rely on some form of backpropagation – calculation of partial derivative of system output over network weight coefficients . Partial derivatives are determined using chain rule, by finding derivatives of weight coefficients in output layer, using intermediate results to calculate next layer, with respect to network structure. Calculation is repeated until all layers are processed.
Described procedure is valid for static neural networks, but is of no use in recurrent networks, as each neuron has inputs from previous time step from the same layer. Two procedures are often used in calculation of partial derivatives:
In BPTT recurrent neuron inputs are replaced with the same neural network delayed by one time step. Process is iterated N times. Thus, recurrent network is approximated by N feedforward networks with delayed outputs. Network weight coefficients, recurrent inputs and outputs must be preserved for past N time steps. Obtained derivative is used to determine new weight coefficients. One advantage of BPTT is that all methods used for training feedforward networks can be utilized in training recurrent networks.
This chapter will describe use of RTRL procedure in training dynamic networks. Output of recurrent network with one hidden and one output layer has the following form:
– hidden layer input, ,
– hidden layer output,
y(t) – network output,
– hidden layer weight coefficients,
– output layer weight coefficients,
Equation (2) can easily be expanded for multi layer recurrent networks. Partial derivative of network output over output layer weight coefficients is
For hidden layer, partial derivatives are
Partial derivative of hidden layer output over weight coefficients is
– number of neurons in hidden layer,
– number of neurons in input layer.
Value of is calculated using four nested loops (Appendix A).
Matrix of output derivatives over weight coefficients is of the form .
2.2. Neural network training algorithms
2.2.1. Gradient descent
Gradient descent is the most commonly used method in neural networks training. Difference between real and desired network output can be expressed as
Differentiation of (5) provides update values of weight coefficients:
Parameter defines learning rate and must be in range .
2.2.2. Kalman filter based training
Kalman filter  was developed as a recursive procedure of optimal filtering of linear dynamic state-space systems. For RNN training extended Kalman filter (EKF) is used. It is also applicable to nonlinear systems using linearization around operating point. The algorithm is used as a predictive-corrective procedure.
Training is performed in two steps. The first step is calculation of prediction values of weight coefficients. Second step is correction of predicted variables based on measured state space.
Algorithm is defined by the following relations:
– Kalman gain matrix, determined form network output derivatives over weight coefficients . System linearization is accomplished by calculating .
– Error covariance matrix, calculated in every iteration
To escape local minima during training, a diagonal matrix is added to covariance matrix in each time step.
Updates to weight coefficients are determined from (9), as a product of Kalman gain and in current step.
Training based on Kalman filter is fast, has good rate of convergence and is often the only practical solution for recurrent networks training. However, the procedure is demanding in terms of computational power and memory requirements.
3. System identification
Nonlinear dynamic systems are identified as input-output system model, or as a state space system. For the selected process, identification is performed by minimization of the cost function
where is the difference of the measured system outputs and output values of the model.
Selected cost function represents squared error of the identified model.
Dynamic system identification can be performed according to , using either series-parallel or parallel structure.
In series-parallel structure Figure 3, model neural network input vector is expanded with system outputs from previous steps. In parallel structure Figure 3, network output from previous states is added to input vector. Series-parallel identification provides estimated system output for next time step only, while parallel identification can predict multiple future iterations. However, parallel structure is harder to train and is susceptible to state drifting.
To verify the validity of the RTRL algorithm, neural networks were trained on several discrete mathematical functions. Supervised training has been used, with control value expressed as difference between desired and actual network output.
A parallel identification structure was used on dynamic SISO system . Training has been performed on Recurrent Neural Network with four neurons in hidden layer, and one neuron in linear output layer. Identification results are shown in Figure 4.
Figure 4a compares initial response of model output and network output . Figure 4b represents the same values with trained network. There is a perfect match between function output values and output of the trained NN. Figure 4c displays training rate as error square . Training was completed in 20000 steps.
Input signal used for training and verification is , with sample time .
In second test, a system with one input, two state variables and one output has been identified:
System was identified using series-parallel structure with RNN with 5 neurons in hidden layer, and one linear neuron in output layer. Identification results are shown in Figure 5.
Input signal used for training and verification is the same as in previous example:
Both examples were designed in Matlab and Simulink. Neural network was trained using c-language S-function (Appendix B). Although the training took 10000 steps, all important system dynamics were visible on trained network after 2000 steps.
3.1. Determination of partial derivatives of functions using neural networks
One of the most important features of neural network modeling is the possibility to determine system output derivatives over inputs . After finding output derivatives, various optimization algorithms can be used.
Partial derivative is calculated from (2)
where is number of hidden layer inputs. Partial derivative is obtained from network backpropagation.
For nonlinear function , partial derivative was first calculated analytically, and then obtained from model network using (12). Results are compared in Figure 6a and 6b.
Results from Figure 6b show good match between calculated function derivatives and values obtained from neural network of identified model.
In S-function for training RNN (Appendix B) a procedure is shown for online calculation of partial derivatives of , using (12).
4. Adaptive critic design
Adaptive critic design (ACD) is a set of optimal control procedures of nonlinear systems. It is based on dynamic programming and neural networks. Optimal control is achieved by training neural networks and using them in the structure according to the selected ACD algorithm. ACD algorithms are implemented using a heuristic approach – system state space variables, control values and cost function are selected based on process characteristics. This text focuses on the Heuristic Dynamic Programming (HDP), Dual Heuristic Programming (DHP) and General Dual Heuristic Programming (GDHP) [1,2].
4.1. Heuristic dynamic programming
Heuristic dynamic programming (
represents discount factor with value in range , ensuring the convergence of Bellman sum and cancellation of old values,
is user defined utility function based on system state space variables.
Control signal that minimizes Bellman sum is generated using one of the Adaptive Critic algorithms. In practical control systems, HDP optimization is realized with three neural networks with a structure as shown in Figure 7.
Model neural network in
Model neural network is trained to minimize the quadratic criterion:
For adaptive control systems, model NN is trained continuously, allowing for adaptation to change of system parameters, and optimal control. Model neural network is trained simultaneously with other networks over time.
Critic neural network in
Training is used to minimize
Critic neural network training is performed with a copy of the same network which is fed with predicted future system outputs , thus approximating .
Action neural network training is based on model and critic network outputs, by minimizing criterion:
Partial derivatives and are determined from model network using backpropagation in the manner described in section 2.5.
Minimization of scalar product in (18) by training of action network results in minimization of Bellman sum in in (13). Minimum of is achieved when , as action network training criterion approaches zero, thus ensuring end of training process.
Action neural network is trained from the output network (unsupervised algorithms). Training of neural networks is carried out using (19) with the training concept shown in Figure 7c.
4.2. Dual heuristic programming
Dual Heuristic Programming optimal control algorithm is developed with a goal of increasing procedure efficiency by increasing training speed.
Model network training is the same as in HDP procedure. The improvement is in the direct calculation of the Bellman sum partial derivative over state space variables as output of critic network.
Structure of DHP control system is shown in Figure 8.
Training of critic neural network minimizes scalar product:
where represents partial derivative of Bellman sum over state space variables, and is estimated from critic network. The second part of (21) is determined by deriving
Third part of equation (22) is determined from deriving
Somewhat simplified structure of DHP is shown in Figure 8a. Partial derivatives , , are determined from model network, and partial derivatives from action network. Prediction of states and is determined from model network.
Action neural network is trained to minimize Bellman sum, , from the criterion .
Thus, equation (22) can be transformed to
Training of Action Network minimizes scalar product:
Simplified training procedure of action NN is shown in Figure 8a.
Cross training of critic and action networks results in slower approximation. In order to reduce training time, it is recommended to begin the procedure with action network pre-trained as linear controller.
4.3. Global dual heuristic programming
Globalized Dual heuristic optimal control algorithm uses both algorithms described.
Critic neural network provides approximation of Bellman sum and derivative
Action network is trained using the same procedure as described for HDP and DHP.
5. Neural networks based control of excitation voltage
Main elements of electric power system are generators, transmission and distribution network and consumers. Transmission and distribution network form a power grid, which is usually very large in geographic terms. Grid control is implemented on both local and global level, mostly on the power generation side of the grid. Specific to power grid control is large number of closed control loops governing the power and voltage of synchronous generators. Transmission network is governed by setting voltage on transformers in order to minimize transmission losses. Governing of generator power and voltage achieves balance of power as well as reactive power flow on the transmission system. Voltage control also has a large impact on the dynamic system behavior.
Generator power output is governed on the generator unit. For water and steam turbine generators, it defines local power transmission to the grid, while also influencing system frequency. Considering the inertness of the generator systems, power output is not difficult to govern. Voltage control manages the balance between generator and grid voltage, and determines reactive power flow. A change in generator voltage setpoint has impact on system state variables and parameters. Disregarding system transients, a change of generator voltage causes a change on generator busbars, generator current and reactive power.
If we also consider system dynamics, generator voltage change also changes system parameters influencing system damping. Synchronous generator connected to the power grid is dominantly nonlinear in characteristics. Generator output power is proportional to generator and grid voltage, while the relation to angle between generator and grid voltage is a sinus function. Maximum theoretical output power is realized with a load angle being . Exceeding of maximum value of load angle causes large disturbances and disconnection from power grid.
Operating point of generator connected to the grid is determined by steady state excitation current. Excitation voltage and current values dictate generator voltage, reactive power and load angle.
Frequent small disturbances expected and part of normal operating conditions, resulting in oscillations of generator electromechanical variables. Especially pronounced is the swinging of generator power, reaching several percents of nominal value, with a frequency around 1Hz. It is caused by low system damping and outer disturbances on the grid. Also present on the grid are oscillations caused by the system characteristic frequencies, usually around 0.1Hz.
Power grid can be represented as a single machine connected to an infinite bus system. For each operating point a characteristic frequency and damping can be determined.
In case of small disturbances with a synchronous generator working around predefined operating point, system can be considered linear, but only if it returns to the same operating point after each disturbance. For those conditions, power system stabilizer (
Large disturbances occur after short circuits on the power grid, transmission and generator outages, and connection and disconnection of large loads.
Electromechanical oscillations occurring as a consequence of disturbances cause damped oscillations of all electrical and mechanical values (generator voltages and currents, active and reactive power, speed and load angle). Oscillations are also propagated by AVR to excitation system values.
Large system disturbances can cause significant changes to local parameters relevant for generator stability. Due to the linear nature of the voltage regulator system, a change in grid voltage or configuration leads to generator working in non optimal state. Even if the system returns to the same operating conditions, passing through nonlinear states during disturbance causes non optimal control, diminishing the stabilizing effects of the voltage regulator on the system. This can in turn cause another disturbance or, in case of larger units, disruption of the whole power grid. Limitations of the use of linear control systems call for an improved system which can provide optimal control of generator excitation system - .
There is a lot of research effort directed toward design of excitation control system based on neural networks. However, very few of the demonstrated solutions have been implemented on real plants.
The goal of the chapter is to develop a practical-to-use system. This is mainly achieved by separating the design on preparation and implementation phase. Almost all algorithm steps are executed in off-line mode, up to the final step of digital voltage regulator implementation.
To implement the voltage regulator described in the chapter following equipment is needed:
data acquisition hardware and software
personal computer with Matlab and Simulink environment
automatic voltage regulator with neural network support (no training support is needed)
Described controller is not adaptive. Adaptive critic design (ACD) algorithm is used for optimal control.
Classic voltage regulators (AVR) equipped with PSS can be very effective in voltage control and power system stabilization if it is used in well configured transmission grid. In such conditions the generator stays in the linear operating range and the controller maintains optimal configuration on change of active and reactive power. Voltage regulator accuracy, balance of reactive power, stabilizing effect during large disturbances and active stabilization in normal operating conditions are achieved.
If the generator is connected to relatively weak transmission grid, with possible configuration changes during operation, use of neural network based control of excitation voltage leads to better results.
5.1. HDP algorithm implementation
In order to achieve optimal control of synchronous generator connected to the power grid, it is necessary to select a minimum of two system variables – generator voltage and either output power or angular velocity. Selected variables must reflect system oscillations during disturbances. It is also necessary to choose cost function that is minimized during training. System optimization is achieved by minimization of squared generator voltage governing error and minimization of output power change obtained from:
It is important to notice that the two demands are in conflict. Increase in gain in main control loop reduces static control error, but also reduces system damping and increases power output deviation . Use of cost function (27) ensures training process that increases system damping without sacrificing controller responsiveness.
System identification is carried by training model neural network. Figure 8 shows training process of model neural network. Identification of nonlinear system is made by series-parallel structure.
For ACD algorithm optimal control two output state space variables are chosen – generator voltage and output power, . Vector is used to form the cost function. Input vector is
represents generator voltage setpoint,
– exciter control signal,
is system output.
Model NN has 9 inputs and 2 outputs.
Figure 9 shows simulation block diagram of identification procedure, implemented in Matlab and Simulink environment.
A third order nonlinear mathematical model was used as a synchronous generator . Synchronous generator is modeled as a single machine connected to an infinite bus. Although the model ignores initial magnetic flows, it proves to be a good approximation of an actual generator. A predictor-corrector method is used to solve differential equations.
Selected mathematical model, used with well conditioned state space variables and described methods, achieves good numerical stability during simulation, allowing for relatively large sample time (TD=0.02s). Described model of synchronous generator is implemented in C-language S-function and used for further simulations.
For model identification, recurrent neural network is used, with five neurons in hidden layer and two neurons in output layer. Hidden layer is implemented with full recursion and
Initial learning rate value (21) is set to . It was reduced during training to . Training was completed in ~10000 steps. Results are shown in Figure 10. Signals and represent generator voltage at the start and at the end of RNN training. Figure 10a compares responses of control error of generator model and neural network. Voltage setpoint is generated as a pulse signal with a magnitude of 10%. In figure 10b, response to the change of generator power is shown. Figure 10c shows squared sum of RNN error during training.
In the same experiment efficiency of the algorithm was verified. Training was stopped after 100000 steps. For the rest of the simulation, network output is calculated without the change of weight coefficient, thus eliminating the recency effect.
Once the model neural network is trained, it is possible to create simulation for training critic and action neural networks. Simulink model of the simulation is shown in Figure 11.
Simulation uses expressions (21), (22) and (23).
Neural network training is implemented in Matlab S-functions, written in C programming language.
Input of the S function is formed of system signals and desired value. S-function returns RNN output signals and partial derivatives of outputs . Critic and action networks are trained in parallel.
Training results are shown in Figure 12. Figure 12a compares generator voltage response using conventional automatic voltage regulator (AVR) to neural network based controller trained using HDP algorithm. It can be seen that voltage rate of change was not degraded as a result of system damping. It is important to emphasize that generator voltage rate of change is key to system stability during large disturbances.
Figure 12b compares active power output oscillation of conventional and HDP controller. Figure 12c shows load angle, which is indicative of system stability . System damping is significantly increased by use of neural network controller.
HDP algorithm is also verified on voltage control of synchronous generator from Simulink SimPower System block library (Synchronous Machine).
Model network is designed with one hidden recursion layer with 5 neurons. Critic and action networks use the same structure, but with four neurons in hidden layer.
HDP algorithm shown in Figure 11 is applied. Training of neural networks is performed in two steps– in first step action NN is trained to mimic classic voltage regulator, and in second step DHP algorithm is used to train both action and critic NN. It is important to get initial values of action NN in first step, as DHP needs network partial derivatives in training critic NN. Figure 13 shows training results.
Figure 13a shows comparison of classic and NN controller responses. Generator voltage rate of change is almost the same for both controllers.
Figure 13c and 13e show comparison of output power and load angle change. It is obvious that the HDP neural network controller shows significant improvement over classic regulator.
Figure 13b shows comparison of voltage transient in the condition generator voltage reduced to 90% of nominal value. Rate of change of generator voltage is maintained even during such extreme operating conditions.
Figure 13d shows conventional automatic voltage regulator (AVR) operating on the edge of stability, while the DHP controller is completely stable. Comparison of load angle values demonstrates significantly better behavior of DHP controller. On large load angles DHP controller maintains stability while conventional AVR provides low system damping, leading to instability and possible outages.
Neural network HDP voltage controllers were developed using DHP algorithm. Experimental results showed that DHP algorithm is more efficient than ADC algorithm. Implementation of GDHP does not bring significant improvements over DHP.
Application of developed S-functions enables simple use of described algorithms, as well as experimenting with different function parameters.
6. Real time windows target implementation
DHP optimal control algorithm has also been implemented using Matlab Real Time Windows Target Toolbox (RTWT), based on  and . Simulation hardware consists of desktop PC equipped with National Instruments data acquisition card NI DAQ 6024.
Use of RTWT platform enables direct implementation of developed controllers on the real system.
In Figure 14, a system consisting of synchronous generator and RTWT controller is shown. Signal acquisition is achieved using DAQ 6062E Input, with AD conversion of the input values. Line voltage and phase current signals are processed in “Clark” S-function, using Clarke transformation. Transformed values of generator voltage and generator power
ACD algorithm is implemented in two steps. Classic voltage controller is used in the first step (Figure 14). In the second step it is replaced with NN controller.
During testing, generator active power and voltage were changed for a wide range of values. A pulse signal was superimposed to the generator voltage set-point value, resulting in large disturbances on the generator. To obtain measurement records RTWT Signal Visualization was used, with measured data saved in Workspace Simulink block
A RNN was used with one hidden layer containing 5 tansig neurons. Model NN had 9 inputs and 2 outputs. Results of NN training are displayed in Figure 15.
Obtained model NN was used to train critic and action RNN using procedure described in 3. The trained action NN was transferred to Simulink block (Figure 14) and prepared for use in real time. Using RTWT module system was run in real-time mode, with trained action RNN as a voltage controller.
Simulation results are shown in Figure 16, comparing classic and DHP voltage controllers. Figure 16a represents comparison between generator voltage responses and shows satisfactory rate of generator voltage change and high control accuracy. In Figure 16b, generator active power outputs are compared. Considerable damping was achieved using DHP controller. Improved system stability during large disturbances is visible in Figure 16c, where the maximum load angle deviation is significantly lower for DHP controller.
RTWT controller was implemented in sample-time mode with sample time set to . Control loop sample time is .
Matlab Simulink RTWT Toolbox simplifies the transfer of developed procedures to the hardware platform.
In the chapter, a complete process of development and implementation of neural network based controller is described. Optimal control is achieved by training neural network using data obtained from real system. Developed algorithms were tested on voltage control of synchronous generator connected to the power grid.
Optimal control algorithm is implemented in Matlab and Simulink environment. Neural network training is implemented in C programming language Matlab S-function. S-function execution time was sufficiently low to achieve real time performance.
Developed algorithm is tested using Matlab Real Time Windows Target Toolbox, using the same models with minimal modifications.
Experiments performed on laboratory system show possibility of building optimal controller based on special network training procedure with no need for adaptive features.
Obtained simulation results show advantages of neural networks based controller over classic controllers. Developed procedure can relatively easily be adapted for use on real systems.
Matlab function for calculating hidden layer derivatives of outputs over weight coefficients of hidden layer.
S-function for RNN training with one hidden layer.
The first part defines sample time, number of inputs, outputs and neurons. Input is formed of values used for training. Output 1 is RNN output. Output 2 is partial derivative of output over input. Training is performed using Kalman filter.