Open access peer-reviewed chapter

Water Allocation Improvement in River Basin Using Adaptive Neural Fuzzy Reinforcement Learning Approach

By Abolpour B., Javan M. and Karamouz M.

Published: January 1st 2008

DOI: 10.5772/5285

Downloaded: 3920

1. Introduction

Optimal use of water is an important objective of water resource development projects all over the world. An integrated approach toward better water resources management in river basins for irrigation planning is needed to find optimal water use policies. In the past, researchers used variables affecting crop pattern and reservoir releases as decision variables (Yeh, 1985). Labadie, 1993, found discrepancies in simulation and optimization models which are important factors in non-adaptive and weak system managements in river basins. These models become more complicated considering conflicting objectives, stochastic hydrology behavior, and uncertain consumptive water use. Labadie, 1993, presented a combined simulation-optimization strategy for river system management. In his studies, decision variable was reservoir release and objective function was maximization of power generation. However, the objective of his study was to assess directly the optimal water use.

The other group of studies is concerned with indirect optimization of water use by selecting the best strategies or alternatives in the river basin or even on the farms. Multi-objective methods have been widely used in different water resource projects. Bogardi & Nachtnebel, 1994, used multicriteria decision analysis in the study of water resources management. Other applications of this group can be found in the works of.Karamouz et al., 1992, and Owen et al., 1997.

The theory of fuzzy logic provides a mechanism to represent the degree of satisfaction of reservoir objective through the use of fuzzy membership function measures that can be combined in an integrated fashion. The fuzzy approach, alluding to the vagueness or imprecision inherent in problems of this type, has found increasing application in many fields. Fontane et al., 1997, applied reservoir operation based on Fuzzy Logic concept in order to deal with imprecise objectives for the reservoirs located in the monographic area on the Cache la Poudre river basin in the northern Colorado. Sasikumar and Mujumdar, 1998, developed a Fuzzy Waste-Load Allocation Model (FWLAM) for water quality management of a river system using fuzzy multiple objective optimization. Dubrovin et al., 2002, used a new methodology for fuzzy inference and compared it with a traditional (Sugeno style) method, for multipurpose real-time reservoir operation. In these researches, it is implicitly assumed that current decisions are independent of future events and decisions beyond the planning horizon. Besides, stochastic nature of hydrologic parameters, imprecise water demand, uncertainty of relationship between variables in groundwater and surface water resources, can not be completely incorporated into membership functions (Tilmant et al., 2002, and Karamouz and Mousavi, 2003).

Molden and Gates, 1990, Gates and Ahmed, 1995, developed an approach for assessing the alternative strategies for improving irrigation water delivery system in the context of multiple planning criteria. Alternatives that involve structural, managerial and policy changes have also been discussed. The model takes into account the parameter of uncertainty on both supply and demand sides of the system resulting from temporal and spatial variability and inadequate data. The objective of adequacy, efficiency, dependability and equity of water delivery were used to evaluate system performance under each alternative considered. Techniques of Multicriterion Decision Making (MCDM) were also presented. The part of historical data is created by the decisions of experts, users (farmers), designers, and managers and is defined as “Human effects” (Belaineh et al, 2003). In these researches, the effects are not completely incorporated into membership functions and the results of this method are in conflict by application of this approach. This approach has also problems in defining objectives, constraining functions or implementing models.

Increasing demands for agricultural products with limited water resources lead to water allocation and management problems. In addition, the conflicting objectives of individual monetary benefits and social benefits make the problems rather more complex. For efficient and scientific solutions of these problems, groundwater is also to be optimally extracted and combined with surface water to meet the requirements. On the other hand, uncertainty, vagueness and random factors make water allocation problems more complex in the form of unexpected droughts and floods, uncertainty in conjunctive use of surface and ground water, vagueness in water use efficiency and variation of inflows from month to month. As control problems become more complex in these applications, the use of traditional control techniques requiring mathematical models of the plant becomes more difficult to apply. Intelligent controllers have several important advantages, such as shorter development time, and less assumption about the dynamical behavior of the plant, that makes them attractive for application to these problems. Fuzzy set theory provides a mathematical framework for modeling vagueness and imprecision. Neural networks have the ability to learn complex mappings, generalize information, and classify inputs. Hybrid controllers utilize the advantages of each, as well as other novel techniques, creating a powerful tool for intelligent control (Sasaki and Gen, 2003).

The methodology that can be used in selecting the optimum decision of water allocation for each sub-basin from the previous decisions (historical data) is the basic modeling approach in this study. This method includes two steps: the first step is to prepare the simulation models of water use, and the second step is development of the optimization models of water allocation for each sub-basin. Usually, these steps are separated in the literature. In this study, models of each step are not only obtained based on compatible methodologies, but the results of each optimization model are also obtained based on the optimal values of input predictor variables which are selected from the results of simulation models over historical data. Therefore, the output values of the simulation models remain constant. In other words, the simulation models learn to minimize the error between the output and real values (observed values) by using Adaptive Neural Fuzzy Inference System (ANFIS) method. The optimization models are reinforcement learning that seeks to maximize the values of the input predictor variables subject to the fixed output values of simulation models.

For all sub-basins, river outflow was the sole prediction variable for the all simulation models. ANFIS method used different sets of input predictor variables for each sub-basin as dictated by the hydrologic factors. For example, if groundwater extraction occurred, this variable was also used as an input predictor variable, as well as decision variable.

The abilities and advantages of presented method can be explained as: 1) The direct effects of uncertain, vague and random factors over water resources system, water demand estimated and hydrological regime can be incorporated into membership function that are considered in developing the simulation and optimization models. 2) The Human effects are incorporated into membership functions, and the results of this approach will not be conflicted in the future conditions. Therefore, these effects can be quantified by using the reliabilities of previous and optimum conditions of the decision variables in this study. 3) This method does not have problems like MCDM or Economical methods in defining objectives, constraining functions or implementing models.


2. Methods

2.1. Adaptive Neural Fuzzy Inference System

An adaptive network is a network structure consisting of a number of nodes connected through direct links. Each node represents a process unit, and the links between nodes specify the causal relationship between the connected nodes. All or parts of the nodes are adaptive, which means the outputs of theses nodes depend on modifiable parameters pertaining to these nodes. The learning rule specifies how these parameters should be updated to minimize a prescribed error measure, which is a mathematical expression that measures the discrepancy between the network’s actual output and a desired output (Papadrakakis and Lagaros, 2003). Neuro-fuzzy systems are multi-layer feed forward adaptive networks that realize the basic elements and functions of traditional fuzzy logic systems (Oh et al., 2002). Since it has been shown that fuzzy logic systems are universal approximators, neuro-fuzzy control systems, which are isomorphic to traditional fuzzy logic control systems in terms of their functions, are also universal approximators. Adaptive Neural Fuzzy Inference System (ANFIS), developed by Jang et al., 1997, is an extension of the Takagi, Sugeno, and Kang (TSK) fuzzy model (Li et al., 2001). The TSK fuzzy model was known as the first fuzzy model that was developed to generate fuzzy rules from a given input-output data set. This model allows the fuzzy systems to learn the parameters using adaptive backpropagation learning algorithm. In general, ANFIS is much more complicated than fuzzy inference systems (Li et al., 2001). A fuzzy inference system (FIS) can be considered to be a parameterized nonlinear map or a crisp function in a consequence called f, namely:


Where yl is a part of output if Mamdani reasoning is applied or a constant if Sugeno reasoning is applied (Jang et al., 1997). The membership functionμAil(xi)corresponds to the input x = [x1 ,…,xn ]of the rule land mis the number of fuzzy rules. For the ith input predictor variable, xi is the real data (for example- the measured values of inflow and storage volume) in one point from the set of observed values. The output values, f(x)are the estimated values (for example- the estimated value of release) of simulation function within the range of input set. The center of gravity method is used for defuzzification. This can be further written as:


Where wl=yl and


If FS is a set of continuous estimated value functions on domain D, then fcan approximate FS to any desired accuracy. Let FS be a bounded function on [a,b] and D={x1 ,…,xh} a set of points in [a,b]. Then there exists the Least Squares Polynomial of degreerbetween FS and Qh, which minimizes the following expression:


Overall polynomial’s degree is equal to or less than r. Where Qh is real data of output values over hth point of input set (For each input predictor variable i=1,2,…,nand for each point of real world data j=1,2,…,h).

Simulation model. In the Mamdani type of fuzzy system, the real data of the output values can be classified into classes such that the length of each class is equal to [a,b]. But in the Sugeno type, the length of [a,b] is only determined over input data set (D), and fcan be approximately equal to FS; hence, FS is the output values of simulation model. Consider a Sugeno type of fuzzy system, the following rule base is developed:

If x1 isA11and x2 isA21, …, and xn isAn1, Then f1=p01+p11x1 +p21x2 + … +pn1xn.

If x1 isA12and x2 isA22, …, and xn isAn2, Then f2=p02+p12x1 +p22x2 +… +pn2xn.

If x1 isA1mand x2 isA2m, …, and xn isAnm, Then fm=p0m+p1mx1 +p2mx2 + … +pnmxn.

If the membership function of fuzzy sets ( i=1,2,…,m, l=1,2,…,n) isμil, mis the number of rules and nis the number of variables. In the water resources system,μilcan be the numeral value of membership function of input predictor variable such as agricultural water demand. Also,Ailis the real world data where the agricultural water demand is one of the input predictor variables. Using product for T-normor logical and, evaluation of the rules can be written as (Jang et al., 1997):

1. Evaluating the rule premises results in


2. Evaluating the implication and the rule consequences gives


This can be separated to phases by defining




wl is the connection weights and is updated only after presentation of the entire data set. This process is called "Learning", (Jang et al., 1997).

2.2. Adaptive Neural Fuzzy Reinforcement Learning

On the traditional optimization models of reservoir operation and river basin systems, net benefit has been maximized or costs have been minimized. Applications can be found in the work of Jacobs and Vogel, 1998, and Malek, 1998. Most of the operation models are not consistent in dealing with the objectives of the group of farmers, designers, and decision makers with conflicting points of views. Multiobjective uses of water, different strategies and natural factors have added complexity to these models. The natural factors can be included by considering drought or spring periods. Because of these factors, in recent years, efforts are devoted to the development of objective functions and optimization methods of water use on large river basins. Main objectives in this research include distributed water, excess water in the sub-basins, and allocated water in downstream sub-basins.

Reinforcement Learning (RL) is one of the major approaches to solve Markov decision problems with unknown transition probabilities. RL, one of the most studied reinforcement learning algorithms, maintains estimates of the average reward ρ and of the relative value function R(s,x) of choosing decision xin state s, from which an optimal strategy can be derived (Jouffe, 1998). It is assumed that the reinforcement learning agent obtains inputs from a continuous state space S of dimension NS and may perform actions taken from a continuous action space Xof dimension NX. The sets of dimensions of the state space and the action space will be denoted as DS :={1,..,NS }and DX :={1,…,NX }, respectively. Considering, for each statesSand each actionxX,h˜(t|s,x), there is a probability density function giving the distribution of the successor state tif action xis executed in state s. Furthermore, letf˜(s,x,t)Rbe the (unknown) reward the agent gets for executing action xin state sif the action causes a transition to state t. The agent is supposed to select actions at discrete points in time.

The goal of the learning task then is to find a stationary policyμ:SX,i.e. a mapping from states to actions, such that the expected sum of future rewards


is maximized for eachsS, where sK+1 is determined from sK usingh˜(sK+1|sK,μ(sK)). Let


be the sum of discounted future rewards the agent may expect if it executes action xin state sand behaves according to the policy μafterwards. Then, the optimal Q-values,Q˜μ*(s,x), are given by the fixed-point solution of the Bellman equation


and the optimal policy μ* is to execute in each state sthe action xthat maximizes these Q-values (Apple and Brauer, 2000):


Optimization model.In this study, the optimal values of decision variables are obtained by combining Fuzzy Reinforcement Learning and Adaptive Neural Fuzzy Inference Systems (ANFIS). Simulation model is developed based on ANFIS method and input predictor variables (observed values) xi. Optimization model is developed based on two groups of variables. First group is known variables and their values can be obtained from the sets of input data (historical data). Second group is decision variables that have been unknown in the optimization process and will be estimated at the end of optimization process. Hence, fl for each rule is written as:


Where l=1,2,…,mis the number of rules, i=1,2,..,kis number of input predictor variables which m, nand kare the numbers of rules, decision variables, and known variables, respectively.pilis the modifiable parameter for each rule and the input predictor variables that were obtained from ANFIS method. In the first step, it is assumed that wl is constant, independent of xi and can be estimated based on the known variables. Substituting Eq. 14 into Eq. 9 results in:


Where FO is the estimated values of objective function in optimization model. Defining Cl for independent values of decision variables for each rule, Eq.15 can be written as:


In this study, Gaussian membership function is used in the simulation and optimization process. It is written as (Harris, 2000, and Odhiambo et al., 2001):


WhereμAil(x)is the membership value for fuzzy set, xis the input predictor variables (for example- inflow and storage volume in the sub-basin No. 4), ϕdescribes the ‘center‘ of the membership function, and σis the spread of the membership function. Also by using this equation the value of variable xcan be obtained assuming thatμAil(x)is known.


Equation 17 is the objective function and the value of FO (for example- release from the dam) in Eq. 16 depends on the value of decision (for example- inflow) and known variables (for example- storage volume) xi. If the goal with the membership function (μG) is to find maximum value of FO based on the known variables and given modifiable parameters, then value of decision variables can be obtained based on maximizing the objective function. This process will be completely adjusted with Reinforcement Learning method (Eq. 12). But, in this study, it is assumed that value of FS is fixed and can be given by the sets of input data (historical data) or it can be the set of decision-makers (in the future). In other words, the goal is to estimate the best values of decision variables that have been obtained from given value of FS. Therefore, the optimal value of decision variables must be found based on objective function and simulation model. The objective function and constraints can be written as:

Max  FOE20

Subject to:


Equation 22 is developed by fuzzy rule base system that can be derived by ANFIS method using historical observation data (the sets of input data in simulation process). Equation 22 can be used for control value of FS, and will be divided into rule base number land input predictor variables number i.

In the first step, it is assumed that wl is constant and independent of xi, but these connection weights (wl) are not constant and depend on xi, as can be seen in Equations 5 and 7. Therefore, using trial and error methods, these parameters are found in the presented method using fuzzy linear programming with crisp objective function developed by Zimmermann, 1996, for solving equations 20 to 22. An algorithm was developed based on combining ANFIS method and fuzzy linear programming. The state variables are the values of membership function for each decision variable (μAil(xi)). In this study, this algorithm and solution process is called “ANFRL” method, and equations 20 to 22 are the basic modeling approach in this method. The optimal values of these variables can be found by the solution process, subject to minimizing the error of the estimated value of membership function for each decision variable, which is computed by simulation and optimization phases. The parameters of membership functionand ϕ)are the constraints in the optimization process. Figure 1 shows the algorithm of solution process, which is presented in Appendix I.

Quantifiable parameter for method results justification.Reliability is defined as the probability that a state of the system zr is in satisfactory state Z (Hashimoto et al., 1982).


In this paper, there are two satisfactory states. First, in each month, the water resources discharge is equal to water demand in downstream sub-basin. The water resources discharge includes the release from dam or the excess water of upstream sub-basin, groundwater pumping and drainage water reused in the downstream sub-basin. Second, in each month, the residual storage volume is equal or greater than inflow. The two satisfactory states were chosen so as to reflect concerns on how the system will satisfy the two major purposes such as water supply and flood control. Hence, the reliability of the first satisfactory state for the primary water resources management is obtained based on water resources discharge toward water demand. The reliability of the second satisfactory state is obtained based on the residual storage volume toward inflow. The reliability for the results of each optimization model is computed too.

Figure 1.

The algorithm of the solution process (ANFRL method) based on combined ANFIS and fuzzy linear programming.

Figure 2.

River network and sub-basins of Kor and Seevand river basin.

3. Case study: the Kor and Seevand river basin

General features.The Kor and Seevand river basin is located in the northern part of Fars province in Iran and lies between 51o, 45′ to 54o, 30′ eastern latitude and 29o, 01′ to 31o, 15′ northern longitude. Total river basin area is 31511 km2 with 16630 km2 of mountains and 14881 km2 of plains and lakes. Kor river with two branches called Kor and Seevand are the artery of this river basin. These two branches join in Marvdasht area and form the main Kor River. The downstream reach flows into Bakhtegan Lake and is called Korbal river. River network of Kor and Seevand basin is shown in Fig. 2. Doroodzan Dam with 993 million cubic meters of capacity is located on Kor river. This dam supplies irrigation demands of Ramjerd and Marvdasht plains, domestic water for Shiraz City, and hydropower generation.

Sub-basins characteristics.In this study, the river basin is divided into seven sub-basins. Six diversion dams are built on Korbal reach. Some of these ancient diversion dams like Feizeabad and Amir are currently under rehabilitation program and play an important role in the distribution of irrigation water system. In the future, there will be two more storage dams. One will be located near Tang-e-Boraq hydrometeric station on the Kor river (Mollasadra Dam), and the other will be located near Ghaderabad hydrometeric station on Seevand river (Seiboyeh Dam).

Sub-basin No. 4 is Doroodzan Lake that is the only available reservoir in Kor and Seevand river basin. This sub-basin is considered as a single basin because there is a balance between inflow, release, and volume of reservoir that can be evaluated well for periods during which observed data are available. Sub-basin No.5 is located between Doroodzan Dam and Pol-e-Khan hydrometeric station, the irrigation and drainage network lie in this area, too. In this sub-basin, there are different water resources such that it is a complete water resource system. The amount of water required in this sub-basin is used for agricultural, domestic, industrial, and hydropower uses. Release from Doroodzan Dam supplies such demands in two downstream sub-basins (No. 6 and No. 7). These water demands have not been included in the water demands of sub-basin No. 5 (DEM5). These demands would be input predictor variables in the developing simulation models and known variables in the optimization analysis of sub-basins No. 5 and No. 6.

Simulation data characteristics.Simulation of a large-scale river basin can often be very difficult considering different factors affecting the hydrologic characteristics of the basin. This is mainly due to the fact that water use and water resource systems characteristics can significantly vary in different parts of the basin. Therefore, the simulation methods of water resources are used on small-scale basin (sub-basin). The simulation models developed for this river basin are capable of simulating each sub-basin, separately. The basic modeling approach is included in seven simulation models for each sub-basin so that this river basin could be simulated by combination of these models. For all sub-basins, the monthly values of river flows at each of the downstream hydrometer station are estimated by using the simulation models that were developed from the ANFIS method. Hence, seven models are obtained in the step of developing simulation models. Observed monthly values were used to develop the simulation models from October 1975 to September 2001 that were the sets of input data (real world data). The accuracy of the results of each simulation model with the real world data is evaluated in another step that is called “ verification modeling”. Each simulation model is verified by using observed value of years 1982-83, 1995-96, and 1999-2000 (36 months). These three years were selected based on normal, dry, and spring periods. Since Doroodzan Dam became operational on October 1975, this date was selected as the starting date for all of the analysis in this study. Some observed or measured values were incorrect; therefore, these input data were omitted from the analysis. Table 1 shows the simulation results in Kor and Seevand river basins obtained from ANFIS methods.

Sub- Basin No.1234567
NameAspasTang-e-BoraqKamfiroozDoroodzanDoroodzanKorbal upKorbal Down
Simulated Data (Month)29526924624627112096
Verified Data (Month)363636363600
Input predictor VariablesSW1
River Flow at H.S.* (Output values)RF1RF2RF3RF4RF5RF6RF7
Fuzzy Rules5478766

Table 1.

Properties and ANFIS method based simulation results in the Kor and Seevand river basin. (*Hydrometeric Station; **Root Mean Square Error; *** Slope of Regression Line).

3.1. Developing simulation models

Cross validation.In order to attain statistically significant results, a 10-fold cross validation was carried out in the sub-basin No. 5 such that ten different splitting of the data set could be considered. The data set had 271 monthly data of input predictor variables that ninety percent of the set is the training set and 10% of the set is the test set for each fold. The process of the developing simulation model was repeated ten times, for each fold, with different rules number and variform membership functions. The six, seven and eight rules respectively with the one of membership function shapes like Gaussian, Bell and Pi Shaped were assumed for each time. The Gaussian Combination Shaped for membership function with seven rules was also assumed in the 10th time. For all folds, the prediction ability of each model was evaluated both on the training set and the test set in terms of Root Mean Squared Error (RMSE). For example in the 5th fold, assuming Gaussian membership function for each input predictor variables and 6, 7 and 8 rules, RMSE for training set equal 17.68, 10.81 and 14.86 for 17680, 22800 and 31386 Epoch, respectively.

Error on Training Set10.8310.1411.5310.1711.0311.5810.2011.4211.6411.0710.96
Error on Test Set29.8327.5010.0220.6413.6710.0124.5811.5613.5813.9817.54

Table 2.

RMSE of the 10th simulation model identified from each fold.

Results of such experiments can be summarized in a table, in which 10 rows are identified as errors of 10 simulation models for each fold and the 10 columns are identified errors on the 10 fold for each simulation model. The average of RMSE in each row is reported, as an estimate of the prediction capability of each simulation model. For example, the RMSE of the 10th simulation model is identified for each fold and is shown in Table 2. The averages of RMSE equal 10.96 and 17.54 for training and test data in this simulation model. There is not a statistically significant difference between the means or distributions of error on the training and test data at the 99.0 % confidence level. For all simulation models (in each row), these means or distributions have not statistically significant differences either. However, at this confidence level in each fold there is a statistically significant difference between the means of error on the training and test data of each simulation model (in each column). On the other hand, the process of developing simulation model is independent of splitting the data set, and is dependent on rules number and membership function shape. Therefore, Gaussian membership function with seven rules is the best setting of simulation model and has the minimum error on training and test data. Note that 10-fold cross validation is only considered in the sub-basin No. 5, and results, which have been presented in Table 1, are the simulation results in Kor and Seevand river basin for the entire text of this paper.

Sub-basins simulation models.For all sub-basins, the parameters of membership function (σ,ϕ) and the modifiable parameters (pil) in the Sugeno type of fuzzy system for each model are obtained by using water resources factors (input data) that are only shown in Table 3 for the sub-basin No. 5. For example in sub-basin No. 7, the excess water of sub-basin No. 6 (RF6) and agricultural water demand (DEM7) in this sub-basin are the input predictor variables for estimating the river flow at Jahanabad hydrometeric station (RF7). The unit of these variables is million cubic meters per month (MCMM) for all sub-basins. The river flow can be estimated by using these parameters as follows, that is one of the ANFIS models in this study:


Table 3.

The estimated values of the modifiable parameters (Pi1) and the membership function parameters (ϕ and σ) obtained from ANFIS simulation method in the sub-basin No. 5. The unit of membership function parameters is million cubic meters per month (MCMM) and the modifiable parameters are linear coefficient.

Rule 1. If x1 is RF6 over the input set with σ=12.03, ϕ=45.7 (membership function parameters); and x2 is DEM7 over the input set with σ=0.7, ϕ=16.06; then f1= -0.15 + 1.07RF6 - 2.04DEM7.

Rule 6. If x1 is RF6 over the input set with σ=11.76, ϕ=47.18; and x2 is DEM7 over the input set with σ=0.64, ϕ=10.87; then f6= 122.98 + 1.06RF6 – 12.95DEM7.

The simulation of sub-basin No. 5 is achieved by using relationship between input predictor variables and river flow of Pol-e-Khan hydrometeric station or spilled water in this sub-basin (RF5). Input predictor variables were demand (DEM5), release (RF4), inflow to the dam (RF3), storage volume (VOL), groundwater pumping (GW5), surface water (SW5), and drainage water reused (DW5). In the sub-basin No.4, release values (RF4) are simulated using inflow (RF3) and volume of stored water in the lake (VOL). The detailed overview and the type of input predictor variables for other sub-basins are listed in Table 1. Other simulation models can be rewritten similar to the presented approach in sub-basin No. 7. Abolpour, 2005, presented more detail of simulation models in the case study.

Sub- Basin No.1234567
ANFRL Model No.11112312345123451
Length of real world data (Month)29526924624624624627127127127127112012012012012096
Length of total data (Month)300**324273288288288288288288288288300300300300300300
Number of optimum values106223151160182121160204165153164148115183132174171

Table 4.

Known and decision variables in each optimization scenario for all sub-basins, and optimization results in the Kor and Seevand river basin. (*Known and decision variable, and j is sub-basin index.; ** Including real world data, for example 295 months of observed data, and predicted values, for example 5 months of simulated data by using ANFIS method).

Membership function properties.A property of ANFIS method is the development of membership functions for each input predictor variable (Jang et al., 1997). These membership functions can be used for the evaluation of input predictor variables. For example, in the downstream of Doroodzan dam (sub–basin No. 5), membership functions are developed for each input predictor variables. In this sub-basin, for each of seven input predictor variables, seven membership functions are obtained. Because the values of the input and output variables are vague or uncertain over time and / or space, they are classified into classes (e.g. low, mean, very high, etc.) for seven different climate season (e.g. Drought – Spring) using fuzzy membership functions. Based on 10-fold cross validation in the ANFIS process, the historical data follows the seven formulated fuzzy rules. Each rule pertains to a single climate season, adaptively adjusting the midpoints and ranges of the membership functions so as to minimize the prediction error. By using these fuzzy membership functions, the water resources management policies could be evaluated in the real time operation of the system and the results can be compared with the historical records of water supply in the study area (Abolpour, 2005, Abolpour & Javan, 2007).

3.2. Using optimization methods for different scenarios

The ANFRL method is used to develop optimization models for each sub-basin that has obtained the optimum values of decision variables. These models are conducted with simulation models developed by using ANFIS method. The membership function parameters (σ,ϕ) and the modifiable parameters (Pi2) in optimization models are the same values of the simulation models. But, the input predictor variables for each simulation models are divided into the known and unknown variables where unknown variables are the decision variables in the optimization models. Also, the output values in simulation models are one of the known variables in the optimization models. In some of sub basins, the ANFRL method may develop several optimization models for each scenario so that they are only conducted with one of the simulation models. Therefore, the total number of optimization models is 17 in this study and their properties are presented in Table 4.

In each sub-basin, the optimization models find the optimum values of decision variables for the period of past 25 years. The values of known variables are obtained from the sets of input data (real world data) that have been used in the process of developing simulation models. If the values of known variables that are output values (river flow) in simulation models do not exist in the sets of input data, then the predicted values of these variables are used in the optimization models. The predicted values are estimated by using the results of simulation models. In this manner, the optimization models can be completely implemented in each month of the period. For all sub-basins, known and decision variables in each optimization scenario are presented in Table 4. The length of real world data is the number of input data (historical data) that are used to develop simulation models. The length of total data includes the length of real world data and predicted values, which are estimated by using ANFIS method. The number of optimum values is the results of optimization models that yield the optimal values by using the ANFRL method. The lengths of real world data, total data, and the number of optimum values are shown in Table 4, too.

As an introduction to the problem, we will consider representative sub-basin No. 4, which has a surface water reservoir. For this portion of the river basin, we must balance reservoir inflows (RF3), outflows (release from dam - RF4), and storage volumes (VOL). The ANFIS method uses the formulated fuzzy rule system to predict the single output variable, outflow, in response to the two input predictor variables, reservoir inflow and storage volume. A different set of decision variables is used for three different optimization scenarios, and they are 1) inflow into dam; 2) reservoir storage volume; 3) both inflow and storage volume. In the optimization model of scenario No. 1, the inflow value is one of the input data and the release value (downstream of this sub-basin) is the output value in the simulation process. In the optimization process, the inflow value is decision variable and the optimal value of this variable must be found subject to a fixed release value.

Because the release values are fixed in the modeling of the optimization process, this variable is defined as “known” variable. The values of storage volume (input data) are used to develop the simulation model; hence, the specified value of this variable is required in using the optimization model of scenario No. 1. Therefore, the sets of input data (observed values) are used to find the given values of release and storage volume, and these variables are defined as “known” variables (Table 4). The values of inflows that are used as decision variables in the process of optimization modeling are called “unknown” variables (Table 4). The state variable is the value of membership function for each decision variable and is obtained from ANFIS method from simulation process over monthly management periods (Table 3). Therefore, in this sub-basin, three optimization models are used and the results of optimization model No. 1 and 2 are shown in Fig. 3.

Figure 3.

Results of optimization models in sub-basin No. 4. Optimal and observed data of inflow (a-RF3) and storage volume (b-VOL) in Doroodzan Dam.

Optimization models in sub-basin No.5 are developed under five scenarios. In all models, objective functions are defined so that they optimize river flows at Pol-e-Khan hydrometeric station (RF5), using ANFRL method. Optimization model No.1 is developed for condition in which release of dam (RF4) is the decision variable. In this model, surface water (SW5), water demand (DEM5), inflow (RF3), storage volume (VOL), groundwater pumping (GW5) and drainage water reused (DW5), are the known variables. Properties of other optimization models are presented in Table 4. All areas in this sub-basin have been under cultivation during the past 25 years and no new development plans are available for this area. There have been a considerable number of dry and spring periods with different severity during the past 25 years. Therefore, the results of optimization models can definitely be used for future conditions. The results of optimization model No. 1, 2 and 3 are shown in Fig. 4. For the other sub-basins, the characteristics of optimization models are presented in Table 4, but the optimum values of decision variables are not shown.

Figure 4.

Results of optimization models in sub-basin No.5. Optimal and observed data of release (a-RF4), groundwater pumping (b-GW5), and drainage water reused (c-DW5).

4. Results and discussion

An important objective of this study was to maximize the volume of excess water in each sub-basin or river flow in each hydrometeric station. Decision variables of optimization models included release from the dam, storage volume, river flow in the upstream sub-basin, and groundwater pumping or drainage water reused. Results of these models are presented in Table 4. In some months, optimum values of decision variables could not be found. Optimum values of decision variables were found from the algorithm presented in Fig. 1. This process consists of two phases. In the simulation phase, the possible values of decision variables are determined from simulation models of the ANFIS method. If the possible values for decision variables could be found from simulation model, these values would be compared with the primary values obtained from optimization phases. If simulation model had a better correlation with real world data, the possible values of variables could be obtained for more months. If the values of known variables were out of range for the physical conditions of sub-basin, then optimization phase would not yield reasonable values for decision variables. Therefore, as it can be seen in Figures 3 and 4, the results of the optimization model are only presented for the month in which the model yields the optimum value.

For all sub-basins, the ANFRL based numerical results are the optimal values of decision variables such as excess water, release from dam, groundwater pumping and drainage water reused. The justification of applying these values instead of primary water resources management should be considered by using a quantifiable parameter. Hence, the reliabilities of previous and optimum conditions of the decision variables are obtained based on the observed data and the results of optimization models (Eq. 23). In the sub-basin No. 4, the storage volume used in computing flood control reliability for observed data and the optimal value, is the decision variables in the upstream Doroodzan Dam. In sub-basin No. 5, the release from dam, groundwater pumping and drainage water reused are the decision variables, used in computing water supply reliability for observed data and the optimal value. The reliabilities of previous and optimum conditions for each month are shown in Table 5.

In sub-basin No.5, the annual water supply reliability equals 0.42 based on the observed data of release form dam, groundwater pumping and drainage water reused in the past 25 years. Also, the variation range of monthly reliability is 0.19 to 0.75 (Table 5). The decision variable is the release from dam, groundwater pumping and drainage water reused in scenarios No. 1, 2 and 3, respectively. The annual reliability equals 0.44, 0.45 and 0.40 based on the results of scenarios No. 1, 2 and 3, respectively. In scenario No. 4 (Model-4), the decision variables are release from dam and groundwater pumping. The annual reliability equals 0.47 based on the results of this scenario. The release from dam, groundwater pumping and drainage water reused are the decision variables in scenario No. 5. The water supply reliability, which is based on this scenario result equals 0.5 for each year. Therefore, the optimization model results obtained based on the scenarios Nos. 1, 2, 4 and 5 yields reliability increment of about 4, 9, 13, and 21 percent respectively (Table 5).

For each month, the variation range is 0.18 to 1.0 in the optimization model No. 5 whose average is equal to 0.5 has been greater than what was obtained from other optimization models. The maximum value of the reliability increment can be related to the integration management that is obtained in scenario No. 5. Besides, in this study, the reliability is defined based on the satisfactory state that the water resources discharge is only equal to water demand. This satisfactory state is created by assuming that the water demand is determinate. Hence, the present approach for developing simulation and optimization models can enable us to consider the effects of uncertainty, vague and random factors over water resources discharge. For example in sub-basin No. 5, these effects are 21 percent that are considered in developing models of scenario No. 5.

The reason of considering the agent non-increment of the reliability more than 0.5 is that, the water supply reliability recalculated based on another satisfactory state. At this state the water resources discharge is equal or greater than water demand and these reliability are shown in Table 5. In this way, the satisfactory state is created by assuming that the water demand is not determinate. The annual water supply reliability in sub-basin No. 5 is equal to 0.86 based on the observed data for the period of the past 25 years. For all scenarios, the variation range of the annual reliability of water supply is 0.86 to 0.96 and is very close to one.

MonthP. W.R.M.Scenario No.
Increasing (%)-----4.349.020.0013.5020.86
Increasing (%)-----6.370.000.9311.074.56
Increasing (%)-----1.86-----0.00----------
Wet Months0.790.91-----0.77----------
Increasing (%)-----15.35-----0.00----------

Table 5.

The water resources and flood control reliabilities for each scenario in the sub-basins No. 4 and No. 5.

In scenario No. 4, the decision variables are release from the dam and groundwater pumping. The increment of the water supply reliability is about 11 percent based on the results of this scenario and is greater than what was obtained from other scenarios. The annual release from the dam in this scenario is equal to 1070 MCMM that is the maximum value of discharge compared to other scenarios. Therefore, in the previous and optimum conditions of water resources management, water resources discharge is usually more than water demand. This is due to the existence of the effects of the uncertainty and imprecise factors such as irrigation efficiency on estimated water demand. Hence, the present approach for developing simulation and optimization models can enable us to consider these effects which are about 11 percent in sub-basin No. 5.

In sub-basin No. 4, the annual flood control reliability is equal to 0.91 based on the observed data of storage volume for the period of the past 25 years (Table 5). The annual reliability is equal to 0.93 based on the results of scenario No. 1, and this optimal value of decision variables is only obtained for storage volume. In scenario No. 3 (Model-3), the decision variables are storage volume and inflow, and the annual reliability is equal to 0.90 based on the results of this scenario. In this case study, most of the previous floods occurred during March to May. The residual storage volume is very important during these months, and the flood control reliability must be obtained for these months. The variation range of the flood control reliability is 0.75 to 0.82 form March to May, and the average value is equal to 0.79 during this period (Table 5). In scenario No. 1, the variation range is 0.82 to 1.0 whose average is equal to 0.91. This value has more than what was obtained from other optimization models. In this scenario, the reliability increment is about 15 percent by considering the effects of random factors over hydrological regime in the upstream sub-basin.

5. Summary and conclusions

In recent years, fuzzy logic has become a strong tool in water resources studies. The main objective of this study is to use this approach in the optimization of water use in river basins. An approach is presented for considering spatial and temporal variation in allocating water on a large-scale river basin. Using simulation models is very important in developing an optimization model in this study. The simulation model used for this purpose consisted of smaller multi-process simulation models. The ability of fuzzy control systems or fuzzy rule based on water resources systems have been presented in the previous studies (Nguyen and Prasad, 1999, Oldhiambo et al., 2001, and Dubrovin, 2002). ANFIS method is a modified form of these methods that can simulate uncertainty, vagueness and other factors affecting the input predictor variables. Although this method is not a complete reasoning model, the development ability of Gaussion membership functions based on the conjunction of univariate fuzzy sets which is defined on the individual components of the input domain, is the reason of the application of this method. Monthly data for developing simulation model has been used in this study. The selection of these time interval and input predictor variables, which had the suitable effects on water balance in each sub-basin, may have impact on the quality of model results in this application. However, ANFIS and Fuzzy Reinforcement Learning concepts are combined to derive the ANFRL method for developing the optimization models.

Water Balance (WB), Linear Regression (LR), Autoregressive Integrated Moving Average (ARIMA), and ANFIS methods are used to simulate seven interconnected sub-basins in this case study. By using the quantitative parameters like modeling efficiency, the accuracy of the ANFIS methodology was considered in the simulation of the behavior of complex river basin systems within the context of uncertainty. Although, WB and ARIMA methods were better methods in upstream sub-basins, ANFIS model was the only method that could be used for simulation of all sub-basins (Abolpour, 2005, Abolpour & Javan 2007).

The presented approach offers two important advantages. First, this method can analyze the direct effects of uncertain, vague, conflicting, and random nature variables and parameters in a water resources system. In sub-basins No. 4 and 5, the present approach for developing simulation and optimization models have the ability of considering the effects of uncertainty factors over water resources system, imprecise factors over water demand estimated and random factors over hydrological regime. The quantitative values of these effects are 21,11 and 15 percent, respectively. The average value is about 16 percent, which can be considered as water allocation improvement in these sub-basins. Second, this method does not show any problem in defining the objective or constraint functions, and the solution process is simpler in comparison with other methods like Genetic Algorithm or Multi-Criteria Decision Making (MCDM). However, two important disadvantages in using this approach are: First, this method requires relatively long periods of historical data for deriving a robust rule set. Second, if the ANFIS model cannot yield suitable estimation of water resources variability then the results of ANFRL model will not be accurate.

Moreover, multi-processes optimization models for each sub-basin on a large scale river basin are developed too. Combination of the results of these optimization models can yield the spatial and temporal optimum values for allocating water. For example, in the Kor and Seevand river basins, the manager of water resources system can find the optimum value for allocating water in each sub-basin. The results obtained from this analysis enable the manager to allocate water for river flow, environmental needs of Bakhtegan Lake and other uses in the sub-basin. In the future, this analysis will be performed by using the expected values of monthly input data obtained from historical record based on Markov chain approach. The analysis could start from anywhere in the sub-basin. Therefore, if the expected value of each input predictor variable is given for each sub-basin, the optimum value of decision variables could be determined in any other part of the sub-basins.

The results of ANFIS method were obtained based on the assumption of simulating primary water resources management. The results obtained from ANFRL method were based on the assumption of selecting optimum strategies from primary water resources management. Therefore, if the results of ANFIS method are only used, the sixteen percent improvement in water allocation will not be attained for the same conditions in the future. The ANFRL, Stochastic Programming Problems with Recourse (SPPR) and Fuzzy Stochastic Dynamic Programming (FSDP) methods are used to optimize water allocation in these sub-basins. The results of ANFRL method based on utilization of conjunctive use strategy of surface and ground water, showed that about 100 percent improvement in water supply reliability as compared to the previous decision of water resources management during dry periods (Abolpour, 2005, Abolpour & Javan, 2007). The imprecise factors like random, vague an uncertainty does not only affect the balance variables of water resources in each sub-basin, but are also related to each other. Therefore, if the simulation models based on ANFIS method could accurately simulate the relationships between factors and their effects on water use modeling in each river basin, the optimization models based on ANFRL method could also achieve the same goal in other case study.



The research leading to this paper was conducted at the Shiraz University, Iran. The measured data were collected by Fars Regional Water Authority – Iran, and Fars Agricultural Research Center – Iran. The authors are grateful to Dr. B. Zahraie from University of Tehran, Sh. Araghi – Nejhad and Reza Karachian Ph. D. Candidates of Amir Kabir University. Water and Environment Research and Development (WE-R&D) office is also greatly appreciated.

Solution Process

The solution process of ANFRL method can be summarized as follows:

  1. Simulation model is derived by using ANFIS method and observed data.

  2. Simulation phase which consists of:

    1. An initial value ofPi3is assumed for input predictor variables in one of the rules. These are decision variables in optimization phase.

    2. In other rules, a value ofPi4is computed based on possible value of decision variables.

    3. The values of connection weights wl andPi5are computed for each rule (Equations 5 and 7).

    4. In the rule l, the estimated value of crisp function (fl ) is calculated based on the possible values of decision and known variables (Eq. 8).

    5. The estimated value of output function FS is computed by using Eq. 9.

    6. The initial values ofPi6are used for other rules and steps 2.1 to 2.6 are also repeated.

    7. The estimated values of FS are compared with observed values, for selecting the possible values of decision variables, which are calculated in the simulation phase.

  3. Optimization phase which consists of:

    1. The modifiable parametersPi7, and Cl are computed by using the results of simulation phase.

    2. The constraints are formulated by assuming the initial value of μc (Equations 21 and 22).

    3. Fuzzy linear programming with crisp objective function is used to compute the estimated value of membership functions of goal (μG ).

    4. The estimated value of μG and the initial value of μc are compared. If the difference is negligible, the primary values of decision variables are estimated. Otherwise, another value of μc is used and steps 3.2 to 3.3 are also repeated.

    5. The possible and primary values of decision variables are compared. If the difference is negligible, then optimum value is obtained and the solution process is stopped.


Pil= Fuzzy set of i th input variable for rule l.

bl(x) = Relative value (connection weights) of membership function for rule l.

CONS = Constant value of simulation model.

DEMj = Water demand variable in sub-basin No. j.

DWj = Drainage water reused in sub-basin No. j.

FS = The output values of simulation model in ANFIS method.

FO = The estimated values of objective function in optimization model (ANFRL method).

PilThe reward the agent gets for executing action x in state s if the action causes a transition to state t.

GWj = Groundwater pumping in sub-basin No. j.

μAil= A probability density function.

μAilThe value of expected sum of discounted future rewards for state s corresponding to μ.

w¯l= The modifiable parameters for each rules and variables that are obtained from ANFIS method.

Qh = The real value of output variable over h th point of input set.

μAil= Q-value in reinforcement learning algorithms corresponding to μ.

pil= The optimal Q-value corresponding to μ*.

RF1 = River flow at Tang-e-Boraq Hydrometeric Station (Spilled water of sub-basin No.1).

RF2 = River flow at Chamriz Hydrometeric Station (Spilled water of sub-basin No.2).

RF3 = Inflow of Doroodzan Dam (Spilled water of sub-basin No.3).

RF4 = Releases from Doroodzan Dam (Spilled water of sub-basin No.4).

RF5 = River flow at Pol-e-Khan Hydrometeric Station (Spilled water of sub-basin No.5).

RF6 = River flow at Kheirabad Hydrometeric Station (Spilled water of sub-basin No.6).

RF7 = River flow at Jahanabad Hydrometeric Station (Spilled water of sub-basin No.7 or discharged into Bakhtegan Lake).

SWj = Surface water in sub-basin No. j.

VOL = Storage volume of Doroodzan Dam for each month.

AilThe connection weights of membership function for lth fuzzy rule system.

xi = The real value of ith input predictor variables in simulation analysis, and the values of known or unknown variables in the optimization analysis.

yl =A place of output, which is a constant value in rule l.

zr = A state of the system.

Z = Satisfactory state.

f˜(s,x,t)== The membership function of i th input predictor variable in rule l on fuzzy set A.

μ* = The optimal value of membership function.

ϕ = The " center" of membership function.

σ = The spread of the membership function.

© 2008 The Author(s). Licensee IntechOpen. This chapter is distributed under the terms of the Creative Commons Attribution-NonCommercial-ShareAlike-3.0 License, which permits use, distribution and reproduction for non-commercial purposes, provided the original is properly cited and derivative works building on this content are distributed under the same license.

How to cite and reference

Link to this chapter Copy to clipboard

Cite this chapter Copy to clipboard

Abolpour B., Javan M. and Karamouz M. (January 1st 2008). Water Allocation Improvement in River Basin Using Adaptive Neural Fuzzy Reinforcement Learning Approach, Reinforcement Learning, Cornelius Weber, Mark Elshaw and Norbert Michael Mayer, IntechOpen, DOI: 10.5772/5285. Available from:

chapter statistics

3920total chapter downloads

More statistics for editors and authors

Login to your personal dashboard for more detailed statistics on your publications.

Access personal reporting

Related Content

This Book

Next chapter

Reinforcement Learning for Building Environmental Control

By Konstantinos Dalamagkidis and Dionysia Kolokotsa

Related Book

First chapter

Different Tools on Multi-Objective Optimization of a Hybrid Artificial Neural Network – Genetic Algorithm for Plasma Chemical Reactor Modelling

By Nor Aishah Saidina Amin and I. Istadi

We are IntechOpen, the world's leading publisher of Open Access books. Built by scientists, for scientists. Our readership spans scientists, professors, researchers, librarians, and students, as well as business professionals. We share our knowledge and peer-reveiwed research papers with libraries, scientific and engineering societies, and also work with corporate R&D departments and government entities.

More About Us