## Abstract

The analysis of the current situation in the wireless communication market shows an increase in the workload, which leads to an increase in the need in additional resources. However, the uneven loading of the infrastructure nodes leads to their loss of use; so, there is a need in introducing technologies that both do not lead to downtime of equipment and ensure the quality of load service during the day. An overview of the NFV virtualization technology has shown that it is appropriate to build wireless networks, since it provides the necessary flexibility and scalability. The method for determining the location and capacity of reserved computer resources of virtual network functions in the data centers of the mobile communication operator, method for determining the size of computing resources constant configuration time interval, and distributed method of local reconfiguration of the virtual network computing resources in the case of a failure or overload are proposed. Thus, configuration, operation, and reconfiguration processes in mobile core network with virtualized functions are described.

### Keywords

- network function virtualization
- evolved packet core
- resource allocation
- reconfiguration
- mobile network

## 1. Introduction

In the mobile cellular network, the rapid development has been observed. Modern telecommunication systems are being constructed as complex networks that involve various types of devices united into a single complex, operating in conditions of large load flows and large number of connections [1]. They can offer higher data transfer rates, with the integration of more services and guarantee of high quality of experience. Nevertheless, this development also means that the amount of data that is transferred in the mobile network is increasing and the volume of signaling traffic is increasing, respectively. According to [2], it is expected that total mobile data traffic will have increased to 77 exabytes per month by 2022, almost seven times more compared to 2017. Mobile data traffic will grow at an average annual growth rate (CAGR) equal to 46% from 2017 to 2022.

According to Shimojo et al. [3], vehicles, houses, personal devices, robots, sensors, etc. will be connected wirelessly. It means that an automatic and intelligent control system will be achieved. An increase in the number of devices will affect the IoT market, which is estimated to be $19 trillion [4], and is expected to reach 50 billion [5]. In addition, rich content services, such as real-time streaming movies that require high resolution and tele-surgery requiring small delay must be provided (Figure 1).

In addition, the average signaling requirement per subscriber is up to 42% higher in LTE compared to the standard of the past generation communication [6].

Furthermore, market competition requires faster deployment of services and elasticity of changing service criteria as well as the ability to cope with higher service requirements. Therefore, there is a need to manage the signaling traffic in order to provide the necessary quality of service to end users and the proper use of resources of the network operator.

In such circumstances, operators are forced to build up the network infrastructure to ensure the process of service of telecommunication services at a given level of quality. During the day, the load differs, and according to [7], up to 80% of the computing capacity of the base stations and up to half of the capacity of the core network are unused. This leads to a low usage of resources as well as a high level of energy consumption, which reduce the cost-effectiveness of the network for mobile operators.

The emergence of the concept of network functions virtualization opens up new opportunities for the world of telecommunication systems. At the same time, there is a need for new approaches, models, and methods for organizing service handling. The use of virtual servers to solve the tasks of the mobile core network can greatly simplify the process of organizing resources on the service server and ensure its scalability and fault tolerance.

The principle of network function virtualization (NFV) [8] is aimed at transforming network architectures by deploying network functions into software that can run on a standard hardware platform. According to the ETSI [9], the network function is a functional block within a network infrastructure that has defined external interfaces and a defined functional behavior. Network functions are components of the LTE evolved packet core (EPC) network, such as MME, HSS, PGW, and SGW, which for the NFV case will be deployed on the basis of data center system, with the use of leased computing resources (CPU core, memory, disk space, and network interface card), which can be allocated and reallocated in the process of operation depending on actual load requirements.

Thus, the features of NFV can be characterized as follows [10]:

Separation of software from hardware. Since the network element is no longer an aggregate of integrated hardware and software entities, the evolution of both is independent of each other. This allows having separate terms of development and maintenance of software and hardware.

Flexible deployment of network functions. The separation of software from hardware helps to reallocate and share infrastructure resources; thus, together hardware and software can perform various functions at different times. It helps network operators to deploy new network services faster on the same physical platform. Consequently, the components can be created in any NFV-compliant device on the network and their connections can be installed on a flexible basis.

Dynamic scaling. Dividing the functionality of the network function into created software components provides greater flexibility in scaling the actual performance of the virtual network function (VNF) more dynamically and with greater details, for example, according to the actual traffic for which the network operator should provide capacity.

At present, numbers of problems remain unresolved. You need to consider hybridity of the service environment, where flexible, well-scalable, virtual servicing entities located in rented cloud-based databases operate along with specialized hardware with limited features. Therefore, the task to organize the computing resources of service nodes and flows between them in a hybrid environment, which consists of hardware telecommunications and virtual computing entities, is important.

Unlike the existing static architecture of the LTE EPC network, a system (Figure 2) in which service flows are processed by hardware, and in the case of expected overload, the redistribution of flows happens and takes into account the expansion of the service network by adding virtual service facilities located in the leased clouds of the data centers is proposed (Figure 3). After organizing a hybrid service environment, there is a need to adapt the computing resources of the system in the process of operation to ensure a high-quality service, and also it is necessary to consider the features of the reconfiguration process and the costs associated with it. So far, there has not been any comprehensive solution to the task of controlling the computing resources of the hybrid telecommunication environment. The peculiarities of the load distribution of resources of network elements, hardware or virtual ones, have it been considered yet either.

Thus, the chapter proposes a structured approach to the management of resources of network functions through sequential control of the following stages: monitoring, forecasting, controlling the sufficiency of resources, and controlling the given level of quality of telecommunication services.

Having analyzed the research and development processes of telecommunication networks of the next generation, we may argue that the existence of powerful data centers greatly expands the possibilities of organizing the process of providing services. One of the key aspects of network virtualization is the allocation of physical resources to virtual network functions. This involves mapping virtual networks on physical networks, as well as managing dedicated resources throughout the life cycle of a virtual network. The optimality and flexibility of resource allocation are key factors for successful network virtualization.

Most of the existing methods for solving the tasks of organizing hardware and virtual resources offer a static distribution of resources, in which, when computing and telecommunication environment is organized, the reallocation of resources does not occur throughout its life cycle. As the network traffic is not static, this may result in improper use of shared computing resources. It is important to organize monitoring of virtual nodes and provide the resources on the basis of their real needs.

## 2. Method for determining the location and required capacity of virtual reserved computing resources in case of an overload of the physical network

The method is based on the shared embedding concept [11] of the individual virtualized services of the core network on the physical network. We suppose that the virtual network functions of the mobile core network have the same functionality and interfaces as the network components of the 3GPP LTE EPC architecture.

The number of service chains must be determined in advance. The extreme case would be consideration of one service chain for the mobile phone/eNodeB. Since realistic scenarios for mobile networks are up to 10,000 eNodeBs, the resulting optimization model will be enormous and quite long computation time is required to solve it. Therefore, we accept reasonably large clusters of eNodeBs and assume that each of these eNodeB clusters refers to a single service chain of the core network.

Consider the situation when the provider of telecommunication services already has an existing topology of base stations. You need to determine a subset of the network nodes where the load aggregation blocks will be placed which will generate the requests to the same virtualized EPC service. After that, for each base station site, we assign a node of aggregation (traffic aggregation point – TAP).

Let * x*be a binary variable that is equal to 1 if we need to place TAP at point

_{i}

*and equals 0 in the other case. In addition, we define*i

*as a binary variable that is equal to 1 if the base station*y

_{ji}

*sends the load to the*j

*TAP and equals 0 in the other case. We need to define the values of*i

*and*x

_{i}

*in order to find the optimal value of the objective function.*y

_{ji}

Objective function (1) aims to minimize network latency. Objective function (2) represents the total cost of placing aggregation nodes and the cost of establishing channels between base stations and the respective TAPs. The objective function (3) aims to leave more free bandwidth on each physical channel. The residual bandwidth of all channels is maximized, since high-downloaded channels can lead to network overload; so, it is advisable to get a solution where more free channels are left.

These optimization goals can be useful for network operators to plan the best deployment strategy.

where * L*is the delay of the communication channel between the site

_{ji}

*and TAP*j

*;*i

* cost*is the cost that consists of two parts: the fixed initial cost

_{i}

*, which is responsible for fixed investments such as space and installation of equipment, and the additional costs –*f

_{i}

*– per unit of processing power on the computing node, where*costN

_{i}

*is the amount of computational resources of processing:*d

_{i}

* costl*is the cost of establishing a connection between the site

_{ji}

*and TAP*j

*, and it is determined as a linear combination of the initial fixed cost*i

*and the variable part dependent on the bandwidth*fl

_{ji}

*, which is necessary for the channel, and the cost of the unit of capacity*B

_{ji}

*:*costLj

* c*– available bandwidth throughput.

_{ji}

It is possible to use the linear combination (4) of Eqs. (1)–(3) with weights * a*,

*, and*b

*, which can be applied not only to give importance the component but also in order to scale the values of these equations for the purpose of converting to comparable values and have meaningful summation:*c

Subject to:

Restriction (5) ensures that each base station will be connected only one TAP. Restriction (6) ensures that a channel is created between the base station site * j*and the TAP

*only if*i

*was placed.*i

Restriction (7) ensures that the maximum TAP does not exceed budget * p*, while (8) is a capacity limit that ensures that the general requirements for processing of all base stations assigned to a specific TAP do not exceed the actual physical resources installed. Restriction (9) makes sure the sufficiency of channel resources for the establishment of channels, and (10) ensures admissibility of delay value, i.e., not exceeding the threshold

*.*T

_{j}

Below we describe the method for solving the problem of placement and the capacity of reserved computing resources of virtual network functions.

Physical network is given in the form of graph * SN* =

*, where*(N, NE)

*is a set of physical nodes and*N

*is a set of channels. Each channel*L

*∈*l = (n

_{1}, n

_{2})

*∈*NE, n

_{1},n

_{2}

*has a maximum capacity of*N

*and each node*c(n

_{1}, n

_{2})

*∈*n

*is associated with certain resources*N

*, i ∈ R, where*c

_{n}

^{i}

*is the set of resource types (CPU cores, memory, disk space, and network interface card). The set of all traffic aggregation points (TAPs), i.e., eNodeB clusters, is denoted as*R

*⊆*K

*. For each node*N

*∈*n

*,*N

*is a binary parameter that indicates whether it is administratively possible to deploy a function*suit

_{n}

^{k,j}

*∈*j

*on the node*V

*, where*n

*is the set of types of network functions,*V

*service, where*k

*∈*k

*.*K

A virtual mobile core network is represented by a set of services (one service per TAP) which are embedded in the physical network.

The requirements to the bandwidth between two functions, * j1*and

*, referring to the TAP*j2, (j1,j2)∈E

*service are denoted as*k∈K

*.*d

_{k}

^{(j1,j2)}

*is the amount of computing resource type*d

_{k}

^{j,i}

*allocated to the network function*i

*in the service*j

*.*k

*specifies the processing time for the type resource*s

_{n,i}

^{k,j}

*of the virtual network function*i

*for the service*j

*with one resource unit on node*k

*. The requirements to the admissible processing time of the network function*n

*related to the service*j

*are designated as*k

*.*P

_{k}

^{j}

*– the maximum delay for*T

_{k}

*∈*k

*,*K

*is the network latency for the channel*L(n

_{1},n

_{2})

*∈*(n

_{1},n

_{2})

*.*NE

The goal of optimization is to find the location of the virtualized services of the core network (i.e., the allocation of network functions and the allocation of resources, as well as definition of the ways to transfer traffic between them), so as to minimize the cost of the occupied resources of channels and nodes in the physical network, while satisfying the load requirements * λ*. Let us formulate an objective function (Eq. (11)) in the form of a linear combination of two value expressions: the occupied capacity of computing node resources, where the value of resource unit

^{k,j}

*on node*i

*is denoted by*n

*, and the occupied bandwidth of the channels, where*costN(i,n)

*is the cost of the unit of bandwidth of the physical channel*costL(n

_{1},n

_{2})

*∈*(n

_{1},n

_{2})

*.*NE

The following Eqs. (11)–(20) represent the formulation of the optimization problem of mixed integer nonlinear programming. The Boolean variables * x*indicate whether the network function

_{n}

^{k,j}

*associated with the service*j

*is located on the physical node*k

*. For*n

*= TAP,*j

*are not variables but input parameters that indicate where TAP*x

_{n}

^{k,TAP}

*is, i.e.,*k

Similarly, Boolean variables * (n*∈

_{1},n

_{2})

*is used for the path between*NE

*and*j

_{1}

*for service*j

_{2}

*.*k

Eq. (12) ensures that for each TAP/service, only one network function of each type is placed. Eq. (13) ensures that the allocation of resources is carried out on physical nodes, which have an administrative opportunity to locate the corresponding network functions. Eqs. (14)–(16) represent restriction for the available resources of physical nodes and channels. Eq. (17) represents a restriction for flow conservation of all paths in the physical network. Eq. (18) ensures that the variables in the task of locating network functions and displaying a path are Boolean.

In order to limit the delays on channels, the delay limit shown in Eq. (19) is also added. And to take into account the necessary performance of the virtual network function, the restrictions for the value of the processing time of the request determined in Eq. (20) are necessary.

It is supposed to solve the problem (11)–(20) in the offline mode at the initial stage. According to the solution, each network function reserves a certain number of resources of the virtual network function based on the assessment of its greatest resource requirements. The instantaneous needs of different network functions are dynamically satisfied by activating the necessary configuration of virtual machines during execution in such a way as to satisfy the guarantees provided for each network function.

## 3. Method for determining the size of the time interval of the constant configuration of computing resources

The decision when to provide resources depends on the dynamics of traffic loads. Telecommunication loads undergo long-term changes, such as hourly effects or seasonal effects, as well as short-term fluctuations such as unexpected crowds. While long-term fluctuations can be predicted in advance, observing changes in the past, short-term fluctuations are less predictable, and in some cases, unpredictable. The proposed method uses two different approaches for working in conditions of changes that are observed at different time scales. Proactive resource management is used to assess the load and corresponding management, as well as reactive resource management is used to correct long-term errors or to respond to unforeseen overload.

We propose to apply a mechanism which implies dynamic change in the duration of the constant configuration of the resources of the virtual network function, depending on the difference between the maximum load value at a certain base interval and the minimum one. Eq. (21) describes the principle:

* Int*is the interval during which the appropriate specified resources will be allocated;

*is the base value of the interval calculated according to the load discretization approach described below;*Int

_{base}

*is the coefficient of the duration change of constant configuration determined by the network operator according to the experiment;*K

*is the average predicted arrival rate in the period*λ

_{basepred}(t)

*, and*t

*is the minimum acceptable value of the base interval.*Int

_{minbase}

To do this, you need to define the base interval. The goal is to present a daily load pattern, sampling its requests into successive, non-overlapping time intervals with a single representative value in each interval. Load discretization: having a time series * X*in the interval [

*,*v

*], time series*τ

*on the same interval is the discretization of the load*Y

*, if [*X

*,*v

*] can be divided into*τ

*consecutive non-overlapping time intervals, {[*m

*,*v

*],[*τ

_{1}

*,*τ

_{1}

*],…,[*τ

_{2}

*,*τ

_{m-1}

*]}, so that*τ

*(*X

*) =*j

*, for all*r

_{i}

*in*j

*-th interval, [*i

*,*τ

_{i-1}

*].*τ

_{i}

The solution for time series discretization (Eq. 22) is given as follows:

Eq. (22) is an objective function which has to be minimized, where * X*is the time series and

*is a function of the value of the number of changes or intervals,*f(m)

*. The purpose of Eq. (22) is to simultaneously minimize the load representation error and number of changes. Basic interval is calculated as*m

At the same time, it is proposed to continuously monitor the values of the request arrival rate and use the predicted values if the load does not exceed the threshold; otherwise, current trends are evaluated and resources are scaled on the basis of the new forecast.

Load forecasting for the next time interval is carried out by taking into account long-term statistics and adjusting it according to the model of exponential smoothing, where errors of more recent past periods have a greater importance factor:

* α*(smoothing constant) is the coefficient that characterizes the weight rate reduction and takes values from 0 to 1; the closer the value of this parameter is to 1, the better is the consideration of the influence of the last levels of the series during the forecast. The model parameters are set by the network administrator according to the experiment.

*is the request arrival rate on interval*λ

_{obs}(t)

*,*t

*is the interval of previous observations, which is considered by the algorithm, and*h

*denotes*x

^{+}

*.*max(0,x)

## 4. Method of local reconfiguration of network computing resources in case of failure or overload

There might be situations when the resources available on the nodes will be insufficient or if the node fails. Potential failures can be physical nodes failures, failures of servers that have higher failure rates than telecommunication hardware or the infrastructure provider will perform node maintenance tasks and this will require the migration of nodes.

For this case, the methods of reconfiguration are used which seek to find the places for migration of network functions from the affected nodes, minimizing the cost of recovering the node after failure and maintaining a high level of physical performance of the network. The proposed improved recovery methods differ from existing ones by taking into account the cost of resources on the nodes and the final quality of service, as well as the case of node overload. In addition, in previous research, the problem of locating management nodes, which are coordinators of the movement of virtual network functions, remained unsolved.

* MN*represents a set of control nodes (hereinafter–managers), where managers

*are responsible for the operation of the proposed recovery mechanism after the failure. Each control node is connected to one or more nodes in the physical network and performs the steps required to recover from the failure. Let us assume that managers can be located in nodes*MN⊆N

*. For a given number of managers*N

*, there is a finite set of possible*A

*, so that the general cost function*A

*({*U

_{A}

*:*p

_{n}

*}) can be minimized, where*n∈N

*is a Boolean variable equal to 1 if the manager is placed at the point*p

_{n}

*. The task of optimization will be given as follows:*n

The main purpose of the optimal placement of managers is to minimize delays between nodes and managers in the network. However, considering only delays is not enough. The placement of managers should also take into account certain restrictions of stability. Figure 4 shows different issues that need to be considered when evaluating the stability of the placement. Below we will briefly explain these issues and what is needed to be sustainable in relation to them. Figure 4 shows normalized delays between nodes and arrival rate at nodes.

Let us presume that the nodes are assigned to their closest manager, using as the metric of delay, i.e., the shortest path * dl*between the node

_{g1,g2}

*and the manager*g1

*. The number of nodes per manager may be unbalanced – the more nodes the manager has to control, the greater is the load per this manager. If the number of site requests to the manager in the network increases, additional delays probability due to the queues in the control system increases too. In order to be resilient to manager overload, the assignment of nodes to different managers should be balanced properly.*g2

It is obvious that one manager is not enough to achieve network resilience. On the other hand, when multiple managers are hosted in the network, the logic of network management is distributed across multiple managers, and these managers must be synchronized to maintain a consistent global state. Depending on the frequency of synchronization between managers, the delay between individual managers plays an important role.

Based on the * dl*matrix, which contains the distance of the shortest paths between all nodes, the maximum transmission latency between the node and the manager for a certain placement of managers can be calculated as follows:

U_{A}^{latency}(p) = max(ddc_{n}),

* ddc*is the maximum transmission delay from the network node to the manager at the point

_{n}

*;*n

* ddc*is calculated as follows:

_{n}

where * latency*is the delay between manager and node

_{g}

*,*g

* π*is a Boolean variable equal to 1 if node

_{g,n}

*is served by a manager located at the point*g

*.*n

We consider not the average, but the maximum delay value, since the average hides the values of the worst case which are important when resiliency needs to be improved.

Depending on the situation, it may be desirable to have an approximately equal load for all managers, so that no manager is overloaded, while others have little work. Next, we consider the balanced distribution of nodes between managers. As a formal metric, we introduce the balance of placement, or rather, the imbalance, * U*, i.e., the deviation from the fully balanced distribution, as the difference between the load for the most downloaded manager and the least downloaded manager.

_{A}

^{imbalance}

* U*is calculated as follows:

_{A}

^{imbalance}

* U*, де

_{A}

^{imbalance}(p) = max(ldc

_{n})– min(ldc

_{n})

*,*ldc

_{n}> 0

* ldc*– manager load at

_{n}

*;*n

* ldc*is given as follows:

_{n}

where * load*is the load factor for the node

_{g}

*.*g

As the last aspect of a resilient placement of managers, let us consider how the delay between managers can be taken into account when choosing managers. Formally, the delay between managers * U*is defined as the greatest delay between any two managers at a given placement:

_{A}

^{interlatency}

In general, placement with a delay between managers’ considerations tends to place all managers closer to each other. This increases the maximum delay from nodes to managers.

Thus, the target optimization function is given by:

where * wu*is the set of importance coefficients.

The recovery algorithm is based on prototype described in [12] but considers modified problem formulation and expands the solution on node overload case.

The physical network is given in the form of a graph * SN = (N,NE)*, where

*is a set of physical nodes and*N

*is a set of channels. Each channel*NE

*∈*(n

_{1},n

_{2})

*,*NE

*∈*n

_{1},n

_{2}

*has a maximum throughput of*N

*and a network delay*c(n

_{1},n

_{2})

*and each node*L(n

_{1},n

_{2}),

*is associated with certain resources*n∈N

*,*c

_{n}

^{i}

*∈*i

*, where*R

*is the set of types of resources. The communication network is represented by a set of services (or virtual network requests)*R

*that are embedded into the physical network. The virtual network request*K

*∈*k, k

*can be given as a graph*K,

*, where*G

_{k}= (V

_{k},E

_{k})

*is the set of virtual nodes containing*V

_{k}

*elements and denoted as*h

_{k}

*, where*V

_{k}= (v

_{k,1},v

_{k,2},…,v

_{k,hk})

*indicates the*v

_{k,j}

*network function in the service chain of*j-th

*.*k

*is the set of virtual channels*E

_{k}

*∈*e

_{k}(v

_{k,j},v

_{k,g})

*. The channel throughput requirements between the two functions,*E

_{k}

*and*j1

*, referring to the*j2

*∈*k

*service are marked as*K

*,*d

_{k}

^{(j1,j2)}

*is the number of resource type*d

_{k}

^{j,i}

*allocated to the network function*i

*in the*j

*service. The Boolean variables*k-th

*indicate whether the network function*x

_{n}

^{k,j}

*associated with*j

*∈*k

*is located on the physical node*K

*, and the variables*n

*determine whether the physical channel*f

_{(n1,n2)}

^{k,(j1,j2)}

*is used in the path between*(n1, n2)

*and*j1

*for request*j2

*.*k

*is the maximum delay for request*L

_{k}

*.*k

*is the cost of the occupied resource unit on the physical node*costN(i,n)

*, and*n

*is the cost per unit occupied bandwidth on the physical channel*costL(n

_{1},n

_{2})

*∈*(n

_{1},n

_{2})

*.*NE

*means that the*suit

_{n}

^{k,j}

*function*j

*can be placed on node*k

*.*n

The process of moving the nodes of the virtual network hosted on the failed node, * v*, starts when the system sends a recovery request to the corresponding host manager. The recovery process for each failed virtual node proceeds as follows: the manager sends the recovery request to all nodes of the physical network, which hosts the virtual nodes adjacent to the affected virtual nodes. Each of these nodes builds the shortest path tree (SPT) to all nodes of the physical network at a distance of not more than

_{k,j}

^{fail}

*(the threshold is set by the service provider) from the node, where the SPT root is the node. The manager uses these paths to select the node with the optimal distance to all nodes in the physical network, where the nodes of the virtual network are located adjacent to the failed node. This node eventually becomes the best candidate for hosting the affected virtual host. In addition, the capacity of the end nodes of the paths with the SPT should be at least the capacity of the virtual node located on the failed node. We select a node with a minimum cost of the path to all root nodes in the SPT trees and the minimum processing cost. Figure 5 contains a description of the pseudo-code of the recovery algorithm (Figure 6) after failure and is applied for all*l

*:*{v

_{k,j}

*.*x

_{n}

^{k,j}= 1 & n = failed}

There is also a probability of the node failure due to overload. To perform a recovery in an overloaded network, the reconfiguration procedure is performed to migrate the virtual nodes hosted on the overloaded physical node.

The recovery process begins with sorting all the virtual nodes located on the overloaded physical node. The criterion (CRT in Figure 7) used to sort these nodes in a virtual network is the capacity of the virtual nodes. Then, the recovery procedure is performed on the first sorted virtual network node, which has a capacity equal to the overloaded, to migrate to the new node of the physical network.

When the load or resources change, some virtual network functions (VNFs) may have to be moved. There is a probability that finding a new node candidate for a node of a virtual network hosted on a failed site will not be possible. In this case, the reconfiguration procedure is performed to migrate one or more virtual nodes. Let us consider the problem of migration as an optimization problem, which is aimed at minimizing the general migration costs with the limits of permissible delay and computational resources.

The goal of optimization is to find the location of virtual network functions (i.e., the location of network functions and resource allocation as well as channels to transfer traffic between them), so as to minimize the cost of the occupied resources of channels and nodes in the physical network, while satisfying the requirements of traffic. Let us give the objective function (26) in the form of a linear combination (with weighted coefficients * a*,

*,*b

*, and*c

*) of the cost expressions.*e

Let us determine the binary variable * x*∈{0,1} to indicate that VNF

_{n}

^{k,j}

*is associated with the service chain*j

*placed on the node*k

*after migration. The indicator*n

*= 0 means that VNF*x

_{n}

^{k,j}

*is not placed on node*j

*after migration; otherwise,*n

*is placed on node*j

*after migration.*n

Then, we enter the binary variable * y*to display the network status before migration. It is similar to variable

_{n}

^{k,j}

*,*x

_{n}

^{k,j}

*= 0 means that VNF is not located on node*y

_{n}

^{k,j}

*before migration; otherwise,*n

*is located on node*j

*before migration.*n

Thus, we can use the * I*indicator to indicate whether the VNF

^{k,j}

*by*j

*service was moved in the current migration process.*k

* I* = 0 indicates that the VNF has been moved in the current migration process, and

^{k,j}

*= 1 indicates that the VNF has not been moved.*I

^{k,j}

* x*denotes whether the

_{n}

*physical server works or not after migration.*n

* y*indicates whether the

_{n}

*physical server works or not before migrating.*n

In order to consider the resources that are consumed while migrating, we introduce the following equations:

* B*indicates the required

_{n}

*costs to launch the*b

_{n}

*server:*n-th

* i*for the migration of VNF

*from the service chain*j

*from the server*k

*to the server*n

*’:*n

where * l*is the function of using resource

_{i}(x)

*for migration from the server and*i

*is the use of resource*l’

_{i}(x)

*for migration to the server.*i

The objective function will be calculated as follows:

Taking everything into account, we formulate the problem as follows.

Objective function:

Min MCost.

With constraints:

Hence, the objective function (26) is a linear combination of four equations which aims to minimize: the cost of starting and using a server, using server resources, communication channels, and resources for migration. Eq. (27) ensures the one-time allocation of network functions, and Eq. (28) is the administrative possibility of placement on the node. Eqs. (29) and (30) represent a limit for the resources of physical nodes and channels, i.e., they ensure that the amount of resources involved in a node does not exceed the amount of available resources. Eq. (31) represents a flow conservation limit, i.e., the input stream at the node is equal to the output stream. Eq. (32) ensures that the variables in the problem are Boolean. Eqs. (33) and (34) represent a limit for the time of transmission by telecommunication channels and time of processing by service nodes, respectively, and ensure compliance with the specified time requirements for the service.

## 5. Operating scheme of the resource management system

Thus, before operation starting, it is necessary to have statistics on the requests arrival rate for the network function and the probability characteristics of the request servicing. According to the allocation method, the binding of each network function of the traditional network to the data center and the amount of resources that should be reserved for the corresponding virtualized network function is determined. Next, it is necessary to divide the lifecycle of the network function into intervals during which its configuration will remain unchanged and a certain amount of resources will be activated in accordance with the method of determining the size of the resources constant configuration time interval, while taking into account the expected load. When a mobile network operates, a physical node may not be able to continue to handle an incoming load due to lack of resources or due to its failure, and in this case, a distributed local reconfiguration of resources that re-distributes virtual nodes is triggered.

The general resource management system is shown in Figure 8.

The monitoring system tracks traffic and counts the number of requests. The monitoring system sets the threshold for the number of requests and sends a message to the coordinator if detecting an overload. When the coordinator accepts an overload message from the monitoring system, the resource allocation unit calculates the required amount of resources to process the applications properly and dynamically distributes the estimated volume. Then, the coordinator redirects the requests and the overload is eliminated.

The coordinator is launched periodically. To predict the base load, one can take the average value of historic daily load. The coordinator sends an incoming load to a data center, which maintains excessive workload, and also exchanges data with a resource allocation unit to provide information about the predicted input load.

The resource distribution module is responsible for distributing the appropriate amount of resources needed to handle the load with the specified quality indicators. During the direct operation of the system, this module is started when the actual load exceeds the base predicted value of the load in order to provide additional resources for excessive load. Since the resource distribution module and coordinator do not start when the actual load is lower than the predicted one, the resource reconfiguration procedure creates minimal additional costs associated with this process.

The general operating scheme of the resource management system is shown in Figure 9.

## 6. Analysis

Quantitative and qualitative analysis (Figure 10) of the proposed methods showed a reduction of the cost associated with reserved resources up to 15%, which contributes to increasing the efficiency of load processing, saving computing resources.

The examples of representation of time series values, i.e., loads that illustrate the accuracy of representation, depending on the selected interval of the constant configuration, are presented in Figure 11, where the representation error for the case of intervals in 10 minutes is 7%, and for the case of 60 minutes—19%.

The results of simulation of the method of determining the size of the resources constant configuration time interval (Figure 12) showed that the difference between representational value and actual one can be 9%. If you do not apply a dynamic adjustment system to the value of the constant configuration interval, then the deviation will be 18%, i.e., 9% more, and the resources will be spent more.

In order to assess the proposed approach, the average amount of free resources per day was determined as the difference between fixed allocations, i.e., when 100% of resources were always allocated during the day, and dynamically allocated resources by using NFV. According to the results of simulation, the volume of resources allocated dynamically on average is 42% less than in the case of using the traditional distribution approach. Figure 13 depicts the result of the dynamic distribution of resources in the virtualized EPC of the mobile network in a graphical form. A gray line illustrates the fixed allocations for the worst case scenario. The black curve shows the amount of resources distributed dynamically according to the proposed method.

According to the simulation results (Figure 14), the proposed local reconfiguration method showed up to 27% lower costs compared to a strategy aimed at minimizing delay, the delay being within the permissible limits but by 20% greater.

## 7. Conclusions

The main result of the study has become the development of the method for reconfiguring resources of the core network by means of virtualization technology. As a result of the research, the following basic scientific results have been obtained.

The analysis of the current situation in the wireless communication market shows an increase in the workload, which leads to an increase in the need in additional resources. However, the uneven loading of the infrastructure nodes leads to their loss of use; so, there is a need in introducing technologies that both do not lead to downtime of equipment and ensure the quality of load service during the day.

An overview of the NFV virtualization technology has shown that it is appropriate to build wireless networks, since it provides the necessary flexibility and scalability.

We have developed the method for determining the location and capacity of reserved computer resources of virtual network functions in the data centers of the mobile communication operator, which guarantees the quality of providing telecommunication services with the minimum necessary resources by determining their sufficient configuration in a heterogeneous environment of available resources. This allows reducing costs by 13% compared to the randomly selected monocloud and by 47% compared with the traditional approach to deploying the network.

In addition, we have developed the method for determining the size of computing resources constant configuration time interval, which involves its changing and the consideration of both the cost of reconfiguration and the use of resources, as well as provides a flexible use of resources in the virtualized environment, which reduces the percentage of free resources by 42% compared to the dedicated equipment and by 9% compared to existing analogs and reducing the workload on the network.

Furthermore, we have improved the distributed method of local reconfiguration of the virtual network computing resources in the case of a failure or overload, which uses decentralized management and considers migration costs, that redistributes virtual network functions in normal and emergency modes while providing rational resource usage and reducing costs on average by 21%.