Management of Software-Defined Networking Powered by Artificial Intelligence

Separating data and control planes by Software-Defined Networking (SDN) not only handles networks centrally and smartly. However, through implementing innovative protocols by centralized controllers, it also contributes flexibility to computer networks. The Internet-of-Things (IoT) and the implementation of 5G have increased the number of heterogeneous connected devices, creating a huge amount of data. Hence, the incorporation of Artificial Intelligence (AI) and Machine Learning is significant. Thanks to SDN controllers, which are programmable and versatile enough to incorporate machine learning algorithms to handle the underlying networks while keeping the network abstracted from controller applications. In this chapter, a software-defined networking management system powered by AI (SDNMS-PAI) is proposed for end-to-end (E2E) heterogeneous networks. By applying artificial intelligence to the controller, we will demonstrate this regarding E2E resource management. SDNMS-PAI provides an architecture with a global view of the underlying network and manages the E2E heterogeneous networks with AI learning.


Introduction
Due to the rapid development of Internet technology, network terminals have been widely spread. However, traditional network architectures have failed to adapt to future advances in communication and Internet technologies, resulting in heterogeneous networks. As a result, the existing network infrastructure was unable to keep up with the rapid changes of the Internet. A key feature of traditional network architectures is that the data and control planes are tightly coupled, which has some limitations. For example, if you want to change the network configuration, you need to configure each device independently across the entire network which is a daunting task.
Similarly, vendors are reluctant to provide the internal details of the device to developers and users, as changes in the configuration of existing networking devices can lead to malfunctions in the network. In addition, the protocol is strongly built into the firmware of network devices. These limitations hinder Another issue is the end-to-end (E2E) quality-of-service (QoS) performance of heterogeneous network providers. If the same provider manages SDN controllers, user applications and forwarding devices on the enterprise network, then, the network status of the underlying devices is readily available for upper-layer applications. However, the Internet consists of different providers where end-users, applications and service providers are often heterogeneous. As a result, the status of the network is not directly available for applications running on the upper layers.
Several solutions have been proposed to address the issue of the allocation of E2E resources [17][18][19][20][21][22][23]. However, they depend on the traditional and manual configuration of the control plane. i.e., once a policy has been defined for the underlying network. The behavior of the network is then controlled accordingly, regardless of the scale of the network or the dynamic changes. The policy of controlling the network is therefore not always optimal. Moreover, these solutions do not provide effective management of the SDN due to scaling up, increasing network complexity and dynamic changes. There is therefore a need to find a global optimal solution with an excellent value for the objective functions. We therefore propose a software-defined networking management system powered by AI (SDNMS-PAI) architecture to auto-configure policy management and E2E resource allocation.
The advantage of AI based architecture is that the AI agent will interact with the underlying network through the SDN controller for pushing the global optimal policy flow rules in the forwarding devices. The controller will share the network status information with the AI agent and based on real time status of the network the AI agent will find the most appropriate actions to be taken. The actions will be pushed as the flow rules in the forwarding devices. AI can be used to bring a closedloop control of the SDN. The closed-loop control incorporates collection of data, analytics, and subsequent actions that are all based on the results of the analytics [24]. All components of the closed loop can be improved and enhanced by means of AI to improve the speed, accuracy and, ultimately, the effectiveness of the closed loop control.
The main contributions of this chapter are summarized as follows: • We leverage the hierarchical SDN architecture to provision the E2E QoS for heterogeneous networks and build a centralized intelligent agent with global E2E view aiming at learning the global optimum policy through interaction with the data plane.
• We apply Q-learning where the learning agent obtains the states of the underlying network and provisions the E2E resource allocations for a service request in the heterogeneous network domains with several QoS classes on the E2E path.
• We demonstrate the proposed SDNMS-PAI with a use case for E2E resource allocation i.e. E2E QoS provisioning.
• Moreover, we evaluate the E2E delay, jitter, packet loss ratio (PLR), and E2E degree of correspondence (DC) [25] ratio for service requests in a hierarchical SDN architecture with an AI agent.

AI powered SDN architecture
In this section, an overview of the proposed SDNMS-PAI is provided. First, we introduce the three planes of the SDN architecture and explain them with a pictorial Computer-Mediated Communication diagram. Then, we introduce the hierarchical SDN architecture for the allocation of E2E resources and the deployment of AI enabled learning. The hierarchical control plane consists of two levels of hierarchy of local and global controllers. Then we develop the SDNMS-PAI architecture for the E2E view and the resource allocation leveraging Q-learning. The proposed architecture consists of a hierarchical control plane with a global E2E view and leverages Q-learning to manage E2E resources in SDN in a smart way.

Hierarchical control plane SDN architecture powered by AI
In this subsection, we first introduce an SDN and a hierarchical architecture followed by an AI powered SDN architecture. The SDN consists of data, control, and application planes. Figure 1 [26] shows the typical SDN architecture. Forwarding devices like routers and switches are part of the data plane. The centralized controller is part of the control plane. At the top is the application plane where different applications can be deployed and executed for a variety of purposes, such as routing, load balancing, security, and monitoring. The controller shall act as a strategic control point for the underlying network. However, several issues arise from a single controller in the SDN. For example, if the controller fails due to a software or hardware problem, the entire network that depends on the controller will collapse.
In addition, the controller will experience a performance bottleneck if the number of switches in its domain increases or the request messages towards it increases. Furthermore, traffic loads are not evenly distributed over the network. As a result,  multiple controllers should be used for viewing the E2E network. However, if there are multiple heterogeneous domains, there is a need for consistency and collaboration between domains for the provisioning of E2E QoS. Figure 2 shows the hierarchical control plane SDN architecture. In the proposed architecture there are local controllers which has access of the data planes of the local domains. Global controllers (GCs) in the hierarchical control plane architecture have access to the global view of physically distributed local data plane switches. The hierarchical architecture of SDN controllers integrates autonomous domains with hierarchical associations. Multiple domains are integrated with the hierarchical architecture of the controller, where the local domain controllers (LCs) coordinate via the GC. By applying hierarchical architecture, new services can be easily managed and deployed in domains that coexist on the E2E path between the source and the destination [27] nodes.
The tasks handled by the controller are propagated from the lower LC layer to the upper GC layer, which reduces computational complexity. The hierarchical control plane with a global view reduces the E2E delay as the network scales [28]. In the proposed architecture, the GC acts proactively to set up the E2E path and therefore reduces the delay in setting up the path (the delay in setting up the path and pushing the flow entries into the switches) [29]. The hierarchical architecture enables communication between multiple LCs with a variety of equipment. The effectiveness of the hierarchical control plane for effective collaboration between heterogeneous tactical networks with a guaranteed QoS has been demonstrated in [30,31]. The rewards for state action pairs in the Q-learning are therefore more accurate than the local view states because these rewards with a hierarchical architecture reflect the E2E view of the underlying network.  [32,33].
Computer-Mediated Communication 6 In our proposed SDNMS-PAI, a hierarchical control plane architecture is employed to construct a completely global view and control for geographical distributed network and build a global AI agent through the global control plane to generate a network control policy via reinforcement learning algorithms. The SDNMS-PAI can intelligently control and optimize a network to meet the differentiated network requirements in a large-scale dynamic network. In the following subsections, we describe the proposed AI enabled SDN architecture from bottom to top. The SDNMS-PAI is shown in Figure 3.

Data plane
Data plane in the SDNMS-PAI consists of the forwarding devices (known as the infrastructure or the underlying network). The matching of the packets in the data plane and the actions take place according to the forwarding rules that are defined in a flow Table. A flow table comprises of several flow entries. The packet header information is matched with the flow entries in the flow table. Each flow entry has three mandatory fields, i.e., header, action, and counter. Table 1 is an example of a flow table in which the first row contains header fields and second and onward rows contain flow entries.
When a new packet arrives on the ingress port of a switch, the matching process starts, if a packet has a destination IP address starting with 172.10.X.X then forward it to port number 8 and counter 201 will be updated. Similarly, the third row (with source IP address: 10.10.1.X) explains if a packet has the same source and destination port number (X) then drop it. If the rules for the new packet do not exist in the flow table then the switch sends a Packet_In message to the controller and the destination will be returned by the controller to the forwarding device (Packet_Out message) and the flow rules will be updated in the flow table, respectively. In contrast to traditional networks where the decision about the routing takes place in the tightly couple distributed networking devices. Herein, in the SDNMS-PAI, the information of the network is collected via the LCs which is used by the AI enabled

SB-API
The Southbound Application Programming Interface (SB-API) provides an interface for data interaction with the local control plane. There are several protocols available for the interaction of the two planes, but the most popular is OpenFlow. OpenFlow provides a secure interface for communication between the controller and the switch. The status of the network topology and the policies for action from the global control plane are communicated to the data plane via the SB-API in the SDNMS-PAI. The White Paper [34] describes the advantages and flexibility of OpenFlow for the programming of forwarding devices. The concept of OpenFlow originated from Stanford University, and the OpenFlow Networking Foundation (ONF) consortium now performs the standardization tasks of OpenFlow.

Local control plane
The data plane switches of each domain are connected to the LCs on the E2E path. The LCs interact with the data plane through SB-APIs. The AIGCP dynamically obtains the underlying network status from the LCs; therefore, it has access to the global topology. As a result, the AIGCP will provide resources from local controllers upon the arrival of a service request. LCs work together through GC, and service level agreements (SLAs) are exchanged through it. Each LC is equipped with a traffic flow template (TFT) module [35] containing the source and destination port numbers, the Internet Protocol (IP) addresses and the QoS parameters. The data collected will be used by the AIGCP for the allocation of E2E resources.

NB-API
The northbound application programming interface (NB-API) functions as a communication interface between the local control and AIGCP. The local control plane functions as a bridge between the forwarding devices and AIGCP utilizing the representational state transfer (REST) API. Similarly, the operational statistics (e.g., about the flow entries) from the data plane are available via this API to the global control plane AI agent. Reinforcement learning algorithms running in the global control plane communicates with the local control plane through this API and the corresponding actions are delegated to the data plane. These actions represent the behavior of the reinforcement learning algorithms executed in the global control plane. For example, a firewall application implements policies for Computer-Mediated Communication 8 controlling the ingress and egress packets passing through the network. Therefore, the data plane devices will forward or block the traffic according to the rules defined in the application. Similarly, a load balancing algorithm will control the traffic through monitoring congestion in different paths of the network. Herein, we employ the Q-learning for E2E QoS provisioning.

AI enabled global control plane
The purpose of the AIGCP is to generate global optimum policies leveraging the global view from the hierarchical SDN architecture. In the SDNMS-PAI paradigm, the AIGCP leverage of hierarchical SDN architecture to obtain the global view as well as control of the E2E network. The state detection module in the global control plane has the global view of the E2E network status which helps the AI agent to make decisions about the global optimum policy based on the E2E view. It feeds the AI agent with the information about the states of the E2E network.

Optimal policy learning mechanism
The local controllers obtain the QoS information (such as the delay, jitter, and PLR) from the data plane devices for all the service requests and the service classes on the E2E paths. The service requests and service classes are shown in Table 2 [36] and Table 3 [37]. The service request is a combination of the E2E delay, jitter, and PLR for an application. An example of the offered service classes in 5 E2E domains is shown in Table 3. Each local controller shares this information with the global controller. Thus, global controller has the E2E view of the network.
Reinforcement learning with Q-learning enabled AI agent is used to maximize the rewards for an agent. Q-learning is one of the methodologies to leverage reinforcement learning. It does not require a model of the environment, and it can cope with problems utilizing stochastic transitions with rewards, without demanding adaptations. For a finite Markov decision process (FMDP), Q-learning computes an optimal policy aiming to maximize the expected value of the accumulated reward over every as well as all successive steps, beginning from current state. Q-learning can find an optimal action-selection policy for any given FMDP, given infinite exploration time along with partly-random policy [38]. Q is the function name that the algorithm learns with the maximum expected rewards for an action taken in a given state [39].
If the service request meets the end-to-end QoS demand for a state action pair, a high reward factor is assigned. For this purpose, the DC ratio is checked for the state action pair. The DC ratio denotes whether the QoS requirements are meeting for a service request or not. For example, if the application service request E2E demand for delay is 150 and the service classes offer a delay of 40, 20, 15, 0 and 45 on the E2E path, then the ratio will be 150/120 i.e., 1  a high Q value for the service request. On the contrary if the DC≺1, the reward is low for the state action pair for that service request. This process continues until all the possible source to destination paths are explored and checked for the DC value against each state action pair.

Use case
Herein, we describe a scenario in which we can employ our proposed SDNMS-PAI for modeling the behavior of the network. We provide an example in the context of QoS service classes allocation, where the SDNMS-PAI is used to make smart choices in order to choose the best service classes on the E2E routing path to meet the E2E QoS requirements. Moreover, based on the Q-learning rewards more excellent service classes are selected in future. The traditional design of the internet mainly focusses on the reliability of services [16]. However, with 5G and beyond networks the requirements for applications have changed, and the applications demands for low latency with high data rates. Further, it is imperative whether the E2E QoS is according to the application service requests. Moreover, with heterogeneous networks on the path from source to destination, there exists several service classes in each domain. Hence, meeting the E2E QoS requirements for the applications service requests is a challenging problem.
Service class mapping mainly involves service classes allocation on the E2E path that meets the QoS demands of different service requests. The typical E2E service classes request for each application are different as shown in Table 2. For example, for application 1 the service requests are different than from application 2 and so on. Several solutions [40][41][42] have been proposed by researchers for service class mapping to meet the E2E QoS requirements for the applications. Furthermore, the mapping of the service classes is a challenging task with respect to meeting the E2E service needs due to the local view of the network state information in the domains.

Results and discussion
Results of the proposed SDNMS-PAI are compared with existing ones i.e., software-defined networking with no artificial intelligence (SDN-NAI) [32]. There are 5 domains on the E2E path and two layers of the controllers i.e., local controllers and a global controller. We consider delay, jitter, and PLR as the primary QoS parameters in every domain. Controllers of the five domains are assigned to 50 nodes according to the controller placement in [43]. Figure 4 compares the E2E delay (in milliseconds (ms)) from source to destination for the SDN-NAI i.e., SDN with no artificial intelligence enabled global control plane and our proposed SDNMS-PAI with. We can see that the delay for the initial service requests is greater for the SDNMS-PAI because the AI agent explores the E2E paths from source to destination for the optimal service classes. However, as the AI agent learns about the global optimum policy, then the delay decrease as compared to SDN-NAI which is shown in the 3rd, 4th , and 5th domains. Initially the service request rates are smaller hence the delay is low however with increasing the service request rate the delay increases because of the consumption of the available bandwidth resources on the E2E paths.
The results in Figure 5 show that E2E jitter (ms) from source to destination for an SDN-NAI compared with SDNMS-PAI. The figure reveals that the jitter for the initial service requests is greater for the SDNMS-PAI due to the AI agent exploring the E2E paths from source to destination to find the optimal service classes. However, as the AI agent becomes more proficient in learning about the global optimum policy, then the jitter decreases as compared to SDN-NAI, which is shown in the 3rd, 4th, and 5th domains. Initially, with lower service request rate the jitter is low since each service request requires only a portion of the available bandwidth on an E2E path. With increasing the service request rate, however, the jitter will increase because of the bandwidth resources used in each service request.   Figure 6 compares the PLR with increasing the service request rate. Herein, the PLR is the ratio of the number of received packets divided by the total number of packets against each service request from source to destination. We can see from Figure 6 that the PLR is initially high for the SDNMS-PAI however as the AI agent obtains a global optimum then the PLR does not increase in the same rate with SDN-NAI. However, the overall PLR increase with increasing the service request rate because the available resources in the network gets occupied. Figure 7 shows a comparison of the E2E DC ratio for SDNMS-PAI and SDN-NAI. We can see from the figure that the SDN-NAI DC ratio was initially higher than the SDNMS-PAI. However, as the AI agent learns, the DC ratio for the proposed scheme is much higher than the SDN-NAI ratio. The basic reason is that, as the service  requests increase, the overall DC ratio becomes low due to the consumption of the available bandwidth on the E2E pathways. Nevertheless, the E2E DC ratio is still 1 or greater than 1 for the proposed SDNMS-PAI, which means that it satisfies the QoS requirements for the application service request. In addition, it overcomes the SDN-NAI in E2E DC ratio.

Conclusions
In this chapter we proposed SDNMS-PAI for the E2E resource allocation i.e., service classes allocation for the E2E service requests. As the distributed management and tight coupling of control and data planes limit the control and global view of network resources. Moreover, the E2E resources in heterogeneous networks cannot be provisioned. Hence, in this chapter we proposed the hierarchical SDN architecture because a single controller with manual configuration of the control plane led to failure and restricts the optimal policy. Moreover, we provided a use case example with service requests and service classes. Furthermore, the SDNMS-PAI scheme employed in a hierarchical SDN architecture with AI agent in the global control plane overcomes the SDN-NAI in terms of E2E delay, jitter, PLR, and DC ratio.