Training Deep Neural Networks with Reinforcement Learning for Time Series Forecasting

2.1 The structure of the model

The model of time series forecasting is given as the following:

Denote t = 1, 2, 3, …, where T is the time, n is the dimensionality of the input of function f(x), xtis the time series data, and xt+1is unknown data in the future as well as the output of model.

A deep belief net (DBN) composed by restricted Boltzmann machines (RBMs) and multilayer perceptron (MLP) is shown in Figure 1.

2.2 RBM

Restricted Boltzmann machine (RBM) is a kind of probabilistic generative neural network which composed by two layers of units: visible layer and hidden layer (see Figure 2).

Units of different layers connect to each other with weights wij=wji, where i=1,2,…,nand j=1,2,…,mare the numbers of units of visible layer and hidden layer, respectively. The outputs of units vi,hjare binary, i.e., 0 or 1, except for the initial value of visible units which is given by the input data. The probabilities of 1 of a visible unit and a hidden unit are according to the following:

Here bi,bjare the biases of units. The learning rules of RBM are given as follows:

where 0<ε<1is a learning rate, pij=<vihj>data,pij'<vihj>modeland <vi>,<hj>indicate the expectations of the first Gibbs sampling (k = 0), and <v˜i>,<h˜j>the kth Gibbs sampling, and it works when k = 1.

2.3 MLP

Multilayer perceptron (MLP) is the most popular neural network which is generally composed by three layers of units: input layer, hidden layer, and output layer (see Figure 3).

The output of the unit y=fzand unit zk=fxis given as the following logistic sigmoid functions:

Here n is the dimensionality of the input, K is the number of hidden units, and xn+1=1.0,zK+1=1.0are the support units of biases vjn+1,wK+1.

The learning rules of MLP using error back-propagation (BP) method [5] are given as follows:

where 0<ε<1is a learning rate and y˜is the teacher signal.

The learning algorithm of MLP using BP is as follows:

Step 1. Observe an input xt=xtxt−1…xt−n+1;

Step 2. Predict a future data yt=xt+1according to Eqs. (7) and (8).

Step 3. Calculate the modification of connection weights, Δwj,Δvjiaccording to Eqs. (9) and (10).

Step 4. Modify the connections,

Step 5. For the next time step t+1, return to step 1.

2.4 The training method of DBN

As same as the training process proposed in [10], the training process of DBN is performed by two steps. The first one, pretraining, utilizes the learning rules of RBM, i.e., Eqs. (4–6), for each RBM independently. The second step is a fine-tuning process using the pretrained parameters of RBMs and BP algorithm. These processes are shown in Figure 4 and Eqs. (11)–(13).

In the case of reinforcement learning (RL), the output is decided by a probability distribution, e.g., the Gaussian distribution y∼πμσ2. So the output units are the mean μand the variance σinstead of one unit y.

The learning algorithm of stochastic gradient ascent (SGA) [7] is as follows.

Step 1. Observe an input xt=xtxt−1…xt−n+1.

Step 2. Predict a future data yt=xt+1according to a probability yt∼πxtwwith ANN models which are constructed by parameters wwμjwσjwijvji.

Step 3. Receive a scalar reward/punishment rtby calculating the prediction error:

where ζis an evaluation constant greater than or equal to zero.

Step 4. Calculate characteristic eligibility eitand eligibility trace D¯it:

where 0≤γ<1is a discount factor and widenotes ith internal variable of DBN.

Step 5. Calculate the modification Δwit:

where b≥0denotes the reinforcement baseline (it can be set as zero).

Step 6. Improve the policy Eq. (16) by renewing its internal variable wiby Eq. (21):

where 0≤ε≤1is a learning rate.

Step 7. For the next time step t+1, return to step 1.

Characteristic eligibility eit, shown in Eq. (18), means that the change of the policy function concerns with the change of system internal variable vector. In fact, the algorithm combines reward/punishment to modify the stochastic policy with its internal variable renewing by Step 4 and Step 5.

The calculation of ewμjt,ewσjt,evijtin MLP part of DBN is induced as follows;

The eitof the RBM of Lth layer in the case of the DBN is given as follows:

The learning rate εin Eq. (21) affects the learning performance of fine-tuning of DBN. Different values to result different training error (mean squared error (MSE)) as shown in Figure 5. An adaptive learning rate as a linear function of learning error is proposed as in Eq. (27):

where is 0≤βa constant.

2.5 Optimization of meta-parameters

The number of RBM that constitute the DBN and the number of neurons of each layer affects prediction performance seriously. In [9], particle swarm optimization (PSO) method is used to decide the structure of DBN, and in [13] it is suggested that random search method [16] is more efficient. In the experiment of time series forecasting by DBN and SGA shown in this chapter, these meta-parameters were decided by the random search, and the exploration limits are shown as the following.

• The number of RBMs: [0–3]

• The number of units in each layer of DBN: [2–20]

• Learning rate of each RBM in Eqs. (4)–(6): [10⁻⁵–10⁻¹]

• Fixed learning rate of SGA in Eq. (21): [10⁻⁵–10⁻¹]

• Discount factor in Eq. (19): [10⁻⁵–10⁻¹]

• Coefficient in Eq. (27) [0.5–2.0]

The optimization algorithm of these meta-parameters by the random search method is as follows:

Step 1. Set random values of meta-parameters beyond the exploration limitations.

Step 2. Predict a future data yt≈xt+1by MLP or DBN using the current weighted connections.

Step 3. If the error between yt,xt+1is reduced enough, store the values of meta-parameters,

or else if the error is not changed,

stop the exploration,

else return to step 1.

3.1 CATS benchmark time series data

CATS time series data is the artificial benchmark data for forecasting competition with ANN methods [3, 4].This artificial time series is given with 5000 data, among which 100 are missed (hidden by competition the organizers). The missed data exist in five blocks:

• Elements 981 to 1000

• Elements 1981 to 2000

• Elements 2981 to 3000

• Elements 3981 to 4000

• Elements 4981 to 5000

The mean square error E1is used as the prediction precision in the competition, and it is computed by the 100 missing data and their predicted values as the following:

where y¯tis the long-term prediction result of the missed data. The CATS time series data is shown in Figure 6.

The prediction results of different blocks of CATS data are shown in Figure 7. Comparing to the conventional learning method of DBN, i.e., using Hinton’s RBM unsupervised learning method [6, 8] and back-propagation (BP), the proposed method which used the reinforcement learning method SGA instead of BP showed its superiority in the sense of the average prediction precision E₁ (see Figure 7f). In addition, the proposed method, DBN with SGA, yielded the highest prediction (E1 measurement) comparing to all previous studies such as MLP with BP, the best prediction of CATS competition IJCNN’04 [4], the conventional DBNs with BP [9, 11], and hybrid models [13]. The details are shown in Table 1.

The meta-parameters obtained by random search method are shown in Table 2. And we found that the MSE of learning, i.e., given by one-ahead prediction results, showed that the proposed method has worse convergence compared to the conventional BP training. In Figure 8, the case of the first block learning MSE of two methods is shown. The convergence of MSE given by BP converged in a long training process and SGA gave unstable MSE of prediction. However, as the basic consideration of a sparse model, the better results of long-term prediction of the proposed method may successfully avoid the over-fitting problem which is caused by the model that is built too strictly by the training sample and loses its robustness for unknown data.

3.2 Real time series data

Three types of natural phenomenon time series data provided by Aalto University [17] were used in the one-ahead forecasting experiments of real time series data.

• CO₂: Atmospheric CO₂ from continuous air samples weekly averages atmospheric CO₂ concentration derived from continuous air samples, Hawaii, 2225 data

• Sea level pressures: Monthly values of the Darwin sea level pressure series, A.D. 1882–1998, 1300 data

• Sunspot number: Monthly averages of sunspot numbers from A.D. 1749 to the present 3078 values

The prediction results of these three datasets are shown in Figure 9. Short-term prediction error is shown in Table 3. DBN with the SGA learning method showed its priority in all cases.

The efficiency of random search to find the optimal meta-parameters, i.e., the structure of RBM and MLP, learning rates, discount factor, etc. which are explained in Section 2.5 is shown in Figure 10 in the case of DBN with SGA learning algorithm. The random search results are shown in Table 4.

We also used seven types of natural phenomenon time series data of TSDL [18]. The data to be predicted was chosen based on [19] which are named as Lynx, Sunspots, River flow, Vehicles, RGNP, Wine, and Airline. The short-term (one-ahead) prediction results are shown in Figure 11 and Table 5.

From Table 5, it can be confirmed that SGA showed its priority to BP except the cases of Vehicles and Wine. From Table 6, an interesting result of random search for meta-parameter showed that the structures of DBN for different datasets were different, not only the number of units on each layer but also the number of RBMs. In the case of SGA learning method, the number of layer for Sunspots, River flow, and Wine were more than DBN using BP learning.