A Framework for Bridging the Gap Between Symbolic and Non-Symbolic AI

FRS (Fuzzy Rule Systems) and ANNs (Artificial Neural Networks) have gained much popularity due to their capabilities in modeling human knowledge and in learning from data, respectively. Fuzzy systems have the advantage of allowing users to incorporate available experts' knowledge directly in the fuzzy model [2]. Thus, decisions made by fuzzy systems are transparent to the user (i.e. the reasons behind the decisions made are clearly understood by tracing the decision and finding out which rules fired and contributed to it). However, there are many parameters whose values are arbitrary. These values have to be "guessed" by a fuzzy system designer, yet they largely influence the system behavior. Thus, a fuzzy system is as good as its programmer.


Introduction
FRS (Fuzzy Rule Systems) and ANNs (Artificial Neural Networks) have gained much popularity due to their capabilities in modeling human knowledge and in learning from data, respectively. Fuzzy systems have the advantage of allowing users to incorporate available experts' knowledge directly in the fuzzy model [2]. Thus, decisions made by fuzzy systems are transparent to the user (i.e. the reasons behind the decisions made are clearly understood by tracing the decision and finding out which rules fired and contributed to it). However, there are many parameters whose values are arbitrary. These values have to be "guessed" by a fuzzy system designer, yet they largely influence the system behavior. Thus, a fuzzy system is as good as its programmer.
On the other hand, ANNs have the advantage of being universal functions approximators, requiring only sample data points and no expert knowledge [3]. Despite their advantages, they are essentially black box models. This means that the reasons behind their decisions are concealed in the knowledge acquired in the trained weights. However, these usually have no clear logical interpretation. Thus, their reliability is questionable.
To combine the advantages of both systems while overcoming their disadvantages, two approaches have been proposed in literature. The first is rule extraction from weights of trained ANNs [4]- [7]. However, the proposed approaches often yield some "un-plausible" rules, thus rule pruning and retraining is often required. For examples, some rules may be impossible i.e. their firing depends on conditions that can never occur in reality (impossible antecedents). For example, a rule dictating that a certain action is to be taken in case time is negative. The second approach is ANFIS [8], which attempts to cast the fuzzy system as an ANN with five layers. Although, only two of these layers are adaptable, this model is still more complicated to build and train than a conventional feed-forward ANN for two main reasons. First, the user's expertise is required to choose appropriate consequent (output) membership functions. Second, the desired output needs to be known a priori. This may not be possible for several applications including design problems, inverse problems and high dimensional problems. For example, in a robot path tracking problem, the ANN is required to predict the correct control input. In such application, the desired performance is known but no real-solid rules exist, especially, if the robot is required to be self-adaptive. Similarly, It has been proved in [1] that the sigmoid response to a sum of inputs is equivalent to combining the sigmoid response to each input using the fuzzy logic operator "ior" (interactive or

The proposed approach
In linguistic terms, an antecedent formed by "ior-ing" several conditions, is equivalent to replacing the conventional phrase: "if A & B & ---then" with "So long as none of the conditions A, B, … are violated ---then". Throughout the paper we will use the Mnemonic "SLANCV" as a shortcut for this phrase. Thus we can say that Eq. 1 can be restated as a set of rules taking the following format:  Adding a penalty term to the objective function used in the ANN training so as to impose a maximum limit to its output.
In this research, we adopted the first approach.
To apply the proposed approach to a particular design problem, there are essentially three phases

Case study: Robot path tracking
In this example, an ANN is adapted to assist a differential-wheel robot in tracking an arbitrary path. Fig. 2 illustrates the geometry of the problem. The robot kinematic model, the need for a closed loop solution, the choice of a suitable representation for the ANN's inputs and outputs as well as the initialization, training and interpretation of the weights of the obtained ANN, are discussed below.

The robot kinematic model
The kinematic model of the robot is described by the following equations: represent the horizontal, vertical components of the robot linear velocity v, its orientation angle and its angular velocity, respectively. Once a suitable control algorithm determines and v ω , it is straightforward to determine the corresponding ν r , ν l values by solving the following 2 simultaneous equations: where l is the distance between the right and left wheels to yield: It is clear from (7) that an average positive (negative) velocity indicates forward (reverse) motion along the robot current axis orientation. Similarly, it is clear from (8), that a positive (negative) difference between ν r and ν l indicates a rotate-left or counter-clockwise (rotateright or clockwise) action. In case both wheels speeds are equal in magnitude and opposite in sign, the robot rotates in-place. However, to assist the ANN in learning the concept of a path (not an instance of a path) as well as making it robust against disturbances, we need to find a closed-loop control law. This is not straightforward because the kinematics model is nonlinear. In what follows, our objective is to show how the proposed ANN-based framework can provide reliable closedloop control; of the form shown in Fig. (3); compared to the direct forcing case. In Fig. (3), the role of the Robot kinematics simulator is to predict the robot location at the current time step given its current control inputs.

Open vs closed loop solutions of the path tracking problem
For the purpose of illustration, we will restrict our case study to the following family of paths:   3 01 a n d 11 The ANN is trained on randomly chosen 11 members of this family and tested on different 11 members of the same family. To demonstrate the robustness of the proposed approach an additive disturbance (of uniform distribution) is added to both and v ω . The value of the disturbance can reach up to 200% of v value and 100% of ω value.

Choice of the ANN's inputs and outputs
Several possible input-output choices exist. The first has been reported in [11]. The time is the input and the speeds are the output. An alternative choice is to consider the coordinates (x, y) of each point on the path; as inputs; and the corresponding actions (speeds); as output. The third choice is to input the path as a whole as a single input vector and the corresponding sequence of actions (speeds) as a single output vector. All these choices share two fundamental disadvantages. First, it is impossible to interpret the trained weights as fuzzy rules. Furthermore, the ANN does not learn the "concept" of path tracking in general. Instead, it learns to track a single path only. In addition, the first and second choices do not explicitly capture the relation between consecutive path points. To overcome these limitations, we investigated different combinations of ANN inputs and outputs.
Only the two input-output combinations, that produced the best results, are discussed: i. Case "A": The inputs to the ANN are chosen to be: the angle '  ' that the robot needs to rotate to orient itself in the direction of the goal.
Refering to Fig. (2),  is calculated using the following equations: the deviations between the actual and reference robot linear and angular velocities The outputs are the corrections (increase/ decrease) ( inc inc v, ω ) needed for (v, ω ) to keep following the required path. Fig. 4 shows the block diagram of this system.
ii. Case "B" The inputs and outputs are as in case A, with the following two additional inputs:  the previous control inputs (v, ω ).

Training phase
To train the ANN, the following objective function is minimized: where ' p i ρ ' is the distance between the desired location on the p th path and the corresponding robot's actual location at the ith time instant, i K is the number of points along the path to be tracked and p N is the number of paths used in training. Since the objective function is not a closed form function (in the ANN weights and biases) and can only be computed through a simulator, a numerical optimization algorithm is used [10], [11], where gradients are computed using a numerical version of the Broyden-Fletcher-Goldfarb-Shanno (BFGS) optimization algorithm. The gradients are calculated numerically using the following formula: www.intechopen.com where ε is a very small number, typically 11 10 ε =  . i Par is the i th element of the vector Par For case 'A', the number of adjustable parameters is 2* (4 (inputs) * 4 (hidden neurons/ rules)) + 4 (hidden neurons)* 2 (outputs), which equals 40 parameters. For case B, the number of adjustable parameters is 2* (6 (inputs) * 7 (hidden neurons/ rules)) + 7 (hidden neurons)* 2 (outputs), which equals 98 parameters . Figs. (6 and 7) show the results for cases 'A' and 'B', respectively.
It is clear that case 'B' is more robust to disturbance. However, both cases clearly outperform the open loop (direct forcing) case.
The fact that case 'B' produced better results is to be expected. Certainly, feedback provides memory to the system and helps in accumulating knowledge, which is an essential aspect of learning. We tried two types of feedback; direct feedback in which the ANN learns the link between its own outputs ( inc inc v, ω ) and the errors in performance and indirect feedback in which the ANN learns the link between a certain action (v, ω ) and the errors in performance. The second memory type produced better results. Appendix A compares these results obtained using our approach with those obtained using conventional ANN [1,6,7] and ANFIS [8].

Rule extraction, analysis and post-processing
Following the guidelines given in section 3.4, rule extraction from the trained weights and biases becomes straightforward.   In order to improve system performance and remove any inconsistencies, the rules above must be analysed. The following rule analysis procedure is limited to case 'B'. This procedure is assisted by a plot of the DOF (Degree-Of-Firing) of each rule (output of sigmoid) versus time. Such plots help in visualizing which rules the ANN is applying at each time instant and judging the decision/ performance made at this particular time. In particular, it is highly useful to identify dominant rules (rules having relatively high outputs) at the time a wrong decision is made. Correction is then possible by retraining the ANN keeping all rules fixed except the faulty dominant one. For example, as shown in Fig. 9, at the time the deviation from the desired path becomes maximum, rule 5 is dominant followed by rule 4. Accordingly, three different strategies, have been attempted, to retrain the ANN, to improve its performance. In all three strategies, all rules have been kept fixed except: for the first strategy: rule 5 (5 th column of Wh c ,5 th column of B c and the 5 th row of Wo c ), -for the second strategy: both the degrees of firing of rules 4 and 5 (4 th and 5 th column of Wh c ), and their then part (4 th and 5 th row of Wo c ), -for the third strategy: the degree of firing of rule 5 and its then part (5 th column of Wh c and the 5 th row of Wo c ), with the addition of a new rule (rule 8) whose SLANCV part is the same as that of rule 5 (8 th column of B c is same as its 5 th column) but whose DOF (8 th column of Wh c ) and consequent part (or then part) (8 th row of Wo c ) are to be determined by the training algorithm. The idea is to insert a new rule that co-fires with the malfunctioning rule to provide some correcting action. For our case study, this last strategy gave us the best results. The results after retraining are shown in Fig. 10. www.intechopen.com Fig. 10. Results of case B after retraining with only the parameters corresponding to rules 5 degree of firing and its "then" part allowed to vary in addition to adding a new rule 8 with the same SLANCV part as rule 5 to co-fire with it and to provide corrective action. It is clear that this retraining improved the results.

Discussion
Although the rule analysis method presented, helped in improving the results, it still needs improvement. In fact, rule analysis is a rather complex task. Understanding the logic behind the rules (why each rule recommends a particular consequent given a certain antecedent) is not simple for the following reasons: 1. Rules operate in parallel and collectively. Each rule recommends a certain consequent depending on the context of the neighbouring rules. Collectively, the overall decision helps achieving the desired objective. 2. According to the training algorithm, rules recommend certain consequents in order to minimize the desired objective function which is a function of the robot kinematics. 3. Rules are derived through batch (off-line) training. Therefore, the overall objective function is minimized over time and not at a particular instant (rules are derived based on a global point of view). Therefore, a rule may not sound reasonable to employ at a certain moment. This what makes it necessary to train an ANN over a certain family of curves. Different families of curves are expected to require different global rules. 4. Rule analysis is a trial and error process. Its complexity is proportional to the dimensionality (number of independent variables).

Directions for future research: Solving general optimization problems
The proposed approach can be applied to general optimization problems and not just pathtracking. From an abstract point of view, any optimization problem can be mapped to a goal-tracking problem in which:


The goal point is the desired performance or desired objective function value.  The input is the absolute difference between the objective function value at the current solution and the desired objective function value.  The output is the correction, to the current solution, recommended by the ANN. This is fed-back as an input to the ANN at the next iteration.  Time evolution corresponds to iterations.
Hence, over the different iterations the ANN is expected to suggest a sequence of corrections that helps in approaching the required objective function value.
This approach to optimization is expected to be less prune to local minima trapping, provides better insight into the nature of the investigated problem and can easily deal with multi-objectives/ errors optimization. For example, in the robot path tracking problem, were error measures that, ideally are required to be all zeros. In this case, the robot perfectly tracks the desired path and remains tangent to it at all times. This property (being tangent at all times) is desirable, from a practical point of view, because even if the robot path is close to the desired path but with too many frequent changes of orientation, wear out will occur to the robot parts and it will be questionable whether the robot can, practically, makes these moves).   To apply the proposed ANN-based approach to find the global minimum of this function, the closed loop system in Fig. 12 has been adopted. The inputs to the ANN are the values of 12 and xx at the previous iteration, the absolute difference between the desired value of the objective function at the current iteration and its current actual value. The outputs of the ANN are the recommended corrections to 12 and xx (increase or decrease). The ANN weights and biases were estimated numerically as before such that the ANN minimize the camel objective function. The desired value for the objective function at any iteration is chosen to be slightly less than its current value (For example, for decrratio=0.85, the desired objective function at a certain iteration =decrratio* the actual objective function value corresponding to the current solution). This idea is borrowed from [13], where it has been recommended to be adopted with any optimization method. The ANN has been trained, as before, using a numerical BFGS algorithm, where the objective function to be minimized is defined as : where, F ρ is the absolute difference between the desired objective function value and its actual value at the last iteration. The rules extracted from the trained ANN are as follows:  ). When using the BFGS technique to minimize the camel objective function, with the same learning rate, the algorithm completely diverged. For a lower learning rate, direct BFGS reached the global optimal. However, ANN-based optimization offers greater promise for higher dimensions/ multi-objective/ error problems and provides insight into the solution through the analysis of the derived rules. A key reason for the robustness of the ANN-based optimization over direct optimization is that it explores the objective function and derives problem-dependent heuristics (as opposed to meta-heuristics).
An important direction for future investigation is the use of concepts borrowed from "robust optimization" to enhance the quality of the rules extracted from the ANN. Robust optimization concepts can be applied at two different levels:  At the action level, we can include noise during ANN training. The error in pathtracking can be defined as a function of the difference between the average expected location and the desired location as well as the variance (or standard deviation of this error). This will make the ANN develop a decision strategy that is more prudent and is less likely to cause divergence from the desired path, in case of disturbance.
www.intechopen.com  At the weights/biases level, robust optimization can be used to develop robust rules. Rules are robust when they are reachable from different weight initializations (i.e. they are not sensitive to a particular initialization) and lead to acceptable performance when subject to small perturbations.

Conclusion
We believe that merging symbolic AI (logic) with non-symbolic AI (ANNs) through our new proposed framework can achieve the following advantages: 1. The resulting learning system is transparent to the user and its reliability can be easily assessed. A suitable strategy has been outlined for improving its reliability based on the analysis of the extracted rules. 2. The system is robust to noisy inputs and disturbances. 3. The logic-based approach to optimization can be less prune to local minima trapping. 4. The approach is applicable to a broad class of engineering problems where the corresponding correct output to a certain example input is not necessarily available (but a means of assessing the fitness of the output is available through simulation or practical measurements).
We do not claim that the proposed approach can outperform existing approaches in all problems, however, we can certainly claim that we offered researchers, a framework truly worthy of investigation for complex optimization, control and design problems. The best approach will always remain problem-dependent which is the charm and challenge of engineering optimization.

Acknowledgement
After thanking Allah almighty for giving the authors the stimulus required to complete this work, the first author would like to thank her students for their valuable inquiries during a summer course on ANNs that she gave. Their eagerness to understand, queries and continuous criticisms helped her a lot to better formulate her thoughts and served as a valuable encouragement, especially during the early part of this work. It is true that, at many occasions, we learn from our students even more than we teach them! The second author acknowledges the support of EJUST.

Appendix A
In this appendix, we apply the conventional ANN formulation ( [1] , [6] , [7]) and ANFIS [8] to case 'B' of the adopted case study. It is noteworthy that strict application of the conventional ANN or ANFIS to this case study is not possible because the desired system output is unknown (it is not possible to evaluate the objective function except by using the robot simulator). Therefore, the numerical BFGS has been used, as before, in the training phase with the same objective function defined in Eq. (9). As said earlier, with the conventional ANN formulation the output of the ANN described in Eq. (1) can be interpreted as fuzzy rules of the form:  The fact that the antecedent of a rule depends on both the bias of the corresponding hidden neuron and the weights from inputs-to this hidden neuron makes it difficult the use of known inputs constraints in weights/biases initialization. Thus, we are forced to use small random weights and biases initially, train the ANN, extract the rules and then re-train in case some of the rules yield un-plausible antecedents. For our case study, 3 out of the 7 rules had un-plausible antecedents. For example, one of the rules stated: Clearly comparing ρ to a negative threshold is not logical.  Fig. 14). Similarly, to be able to compare our approach to ANFIS [8], we use the same block diagram given in Fig. 5 but replacing the typical feed-forward ANN with an ANFIS. The ANFIS formulation does not impose restrictions on membership function's choices. Therefore, sigmoid membership functions have been chosen, for the purpose of comparison with our approach. In this case, the membership function of the j th rule takes the form: Our approach can be viewed as a modified ANFIS system with the 'Product' operator replaced by the 'ior' operator and with 0 ijk p = . As indicated by the results (Fig. 15), these modifications enhance the performance considerably. ANFIS training involves the estimation of the parameters , ijk jk p r for each rule contributing to the output as well as the membership functions parameters ij ij a, c of each rule. The extracted rules after training are as follows: A Framework for Bridging the Gap Between Symbolic and Non-Symbolic AI 47 Fig. 15. Shows the results for using conventional ANNs and ANFIS instead of the proposed formulation for case B. By comparing these results with those in Fig. 7 , the superiority of the proposed approach-thanks to Allah-is clear.