The results of monitoring and analyzing traffic accidents, fixed by an intelligent monitoring system with photoradar complexes, are considered. The system works with a network of distributed photoradar vehicle detectors for road accidents, video surveillance cameras, vehicle information and communication systems, built-in car navigation equipment and mobile communication equipment. A multiagent approach developed to address the tasks of sensor data collecting and processing. The system functionality is implemented by several agents that perform data collecting, cleaning, clustering, comparing time series, retrieving data for visualization, preparing charts and reports, performing spatial and intellectual analysis, etc. Convergent approach is the convergence of cloud, fog and mobile data processing technologies. The diagnostic system is necessary for remote maintenance of photoradar equipment. The structure of the neural network is adapted to the diagnosing problems and forecasting. The tasks of intellectual analysis and forecasting traffic accidents are solved. The hybrid fuzzy neural network is synthesized. Because of the comparison of time series of traffic accidents and time series of meteorological factors, the presence of factors to become determinants for an abnormal change in the traffic situation in controlled areas is established.
- smart road environment
- intelligent system
- multiagent system
- data mining
- machine learning
- convergent model
- smart&safe city
- decision support
- wireless sensor networks
- big sensor data
Smart&Safe City means the development and implementation of projects such as Smart Manufacturing, Smart Houses, Smart Light, Smart Energy, Intelligent Transportation System, Smart Road, and so on [1, 2]. The goal of Smart Technology Development & Safe City is to ensure the comfort and safety of human life in the urban infrastructure and efficient production in the industrial sector. Smart&Safe City components are integrated into a multimodal smart environment . It provides interaction of cyberphysical devices, cloud computing resources and mobile communication systems. Smart Environment helps the artificial intelligence system to solve problems of automatic control or to support decision-making based on big data monitoring about the surrounding reality. It is based on the Internet of Things network platform for the collection and processing of sensor data. The platform includes the following:
Intelligent sensors (sensors, measuring devices, photo and video fixation devices).
Telecommunication networks of broadband data transmission (fiber-optic and wireless) and mobile communication systems.
Satellite navigation systems.
The paradigm of an intelligent multimodal environment includes three basic concepts such as ubiquitous (pervasive) computing and networking ; intellectual assistance (ambient intelligence)  and smart environments .
The creation of smart road environment (SRE) is an important direction in the Smart&Safe City concept . Environment is needed for the interaction of satellite vehicle monitoring systems, intelligent transport systems (ITS) , unmanned vehicles, intelligent road infrastructure components and mobile communication users. SRE includes a built-in intelligent functionality in the vehicles, objects of road transport infrastructure and intelligent system for monitoring and traffic management. It is based on the methods of monitoring and managing traffic flows , provides information and safety to road users. Research in this area relates to the creation of traffic monitoring systems , for example, using radio tags  or embedded monitoring complexes . Monitoring technology includes stream sensor data processing (photos, video streams, telemetry data, user information), data mining, machine learning, forecasting, multiagent processing  and the convergence of computing models (clouds, fog and mobile computing) . The monitoring tasks are as follows: monitoring the condition of the pavement, meteorological monitoring, monitoring of traffic flows and monitoring violations of traffic rules.
Modern road transport infrastructure consists of a system of satellite navigation, traffic signal control, regulation of cargo transportation, information boards, detection systems of car numbers, registration of traffic accidents and violations. The intellectualization of the road transport infrastructure is to develop intelligent systems for monitoring and surveillance, parking management system, decision-making systems for traffic flows regulation, intelligent transport systems, and so on. The purpose of the SRE elements is to influence the behavior of cars, drivers and pedestrians in terms of optimizing transport routes and passenger flows, reducing security risks by preventing emergency situations.
The main elements of the SRE are as follows:
Intelligent real-time monitoring system
Real-time traffic information system for alerting and warning road users
System of accounting and analysis of road users’ social reactions 
Interactive journey planner system
Intelligent traffic lights systems
Intelligent signaling system
Surveillance cameras (CCTV) and photoradar complexes
Satellite systems of transport monitoring
Parking and loading areas information systems
Sensor systems for the movement of unmanned vehicles
Intelligent vehicle transport systems
Electronic payment systems for road services
An important element of SRE is an intelligent monitoring system for decision-making on the management of the road infrastructure objects. The system works with a network of spatially distributed photoradar vehicle detectors for road accidents, video surveillance cameras, vehicle information and communication systems (VICS), built-in car navigation equipment and mobile communication equipment. It is designed for the collection and sensor data processing. The monitoring objectives are analysis, assessment and forecast of changes in traffic situations to control the behavior of vehicles and road users and to alert police, emergency services, ambulance, maintenance and other services.
2. Photoradar complexes for data collection in SRE
Monitoring of objects and incidents in the road infrastructure is carried out on the basis of the collection and sensor data processing obtained from ground platforms, aerial and space surveillance facilities. The main ground platforms in SRE are CCTV cameras and photoradar vehicle detector complexes (Figure 1).
Photoradar complexes allow in an automatic mode to fix incidents on objects of road transport infrastructure, to collect and accumulate sensor data . A lot of complexes receive a huge amount of data, which cannot be processed by a person in real time. Complexes can recognize objects in photos and in a video stream, measure the speed of vehicles in the control zone, automatically capture and save photos of violators, recognize license plates, collect and transfer data to the data center. However, the complexes do not have the capabilities of intellectual analysis and forecasting in real-time mode.
Creation of a heterogeneous transport environment is required for the interaction of the complexes and the transfer of sensor data to the data center. The trend in the field of telecommunications consists of replacing wired networks with wireless channels for monitoring distributed objects . A wireless network is necessary for the interaction of mobile and fixed elements of the SRE. It includes a segment of the Internet of Things for the data exchange between complexes, intelligent transport systems, surveillance systems, a segment of the cellular network for data exchange between users and a segment of the satellite navigation system. The heterogeneous network is realized through technologies of wireless sensor networks (WSN) , cellular networks, Wi-Fi networks and satellite networks.
3. Convergent model for sensor data processing in the smart road environment
Modern approaches to distributed computing and storage of sensor data are based on the concept of convergence . Convergence is defined as the interlinking of computing and storage technologies such as media, content and communication networks. Convergence in relation to network technologies means the process of telecommunication technologies’ convergence with the appearance of similar characteristics in network equipment, communication channels, network standards and protocols and data transfer processes. For example, the technology integration of mobile and cloud computing is the result of the convergence . Another example is the convergence of cloud and fog computing models in a wireless sensor network , which is proposed to create a computing platform for distributed sensor data processing in the SRE. The convergent model of cloud, fog and mobile computing (Figure 2) is designed for sensor data processing, obtained from spatially distributed photoradar complexes, a video surveillance camera, navigation equipment, intelligent transport systems and mobile equipment.
The convergence network platform may include some hardware and software levels that are as follows:
The sensor nodes are associated with industrial controllers and sensors, directly implementing fog computing.
Clusters network segments with coordinators, cellular modems, router, which collects and transfers sensor data into the data warehouse.
Cloud computing clusters.
Warehouse of sensor data and monitoring results.
The user mobile devices for the organization of access to computing and information resources.
The first level of the platform is a fog computing model. It provides the collection and sensor data processing on distributed nodes of the sensor network, in measuring devices and automation devices. Fog computing model is also the platform for data storage services on end-terminal devices and network services for data transmission. Computation is performed by terminal devices with limited computing and energy resources, including WSN nodes, controllers, industrial equipment, household appliances with microprocessors equipment and sensor network nodes. Modern WSN nodes have sufficient processing power to organize distributed computing . The fog computing model is the basis of the Internet of Things . Fog computing platform is necessary for realization of multiagent processing of sensor data and consolidated storage of calculated results on sensor network nodes [24, 25].
The second level of the convergent platform is implemented on the basis of the cloud computing model. Cloud platforms are now used in almost all areas of activity . It is used for the ubiquitous network for access to a common pool of configurable resources (software, server, information, platform, etc.) at any time. The user uses the technology of “thin” client as a means of access to applications and data. The infrastructure of the information system is located at the provider of cloud services. The information is stored in cloud storage on the servers of the network. It is temporarily cached by the analytical processing . The trend is the creation of distributed storage for BigData processing .
The third level of the convergent platform is related to the data processing on smart phones and tablets for presentation of monitoring results to users with the visualizing events and making decisions to reduce road incidents . Mobile computing model is the platform for human-computer interaction. It involves mobile communication, mobile hardware and software. Communication issues include ad hoc networks and infrastructure networks as well as communication properties, protocols, data formats and mobile technologies.
4. Multiagent data collection, data mining and forecasting
Monitoring of road infrastructure includes procedures that are as follows:
Vehicle detection and identification in a controlled section of the road with the measurement of its speed
Photography and video fixation of traffic rules violations
Collect data on the traffic flow parameters in all monitored areas and transfer to the data processing center via communication channels
Vehicle detection on demand and tracking them with visualization of routes of their movements on a cartographic basis
Photographs and video materials processing about violations
Accumulation and statistical data processing on offenses for periods of time to identify and analyze the dependencies of changes in violations and road accidents from the influence of various factors (weather conditions, traffic volume, repair work, city events, time of day, seasonal factors, etc.)
Spatial analysis of offenses to identify critical areas and “bottlenecks” in the road transport infrastructure and their dependence on changes in traffic conditions with visualization on the map
Intellectual data mining and forecasting of road traffic situations for making decisions to improve traffic safety
Multiagent approach is advisable for the implementation of monitoring procedures. It involves the use of software agents for data collection, data mining and forecasting, as well as to alert road users about road traffic situation via mobile and navigation equipment . The data collection and initial data processing are realized in the fog computing layer by means of agents loaded into sensor nodes. Sensor units are connected to the photoradar complexes. Agents interact with server components of the monitoring system.
The hypervisor is used to manage agents. It is consolidated computing resources for distributed data processing. Software agents are operated on the sensor nodes. Agents respond to requests, decide on the selection of data processing functions, clone and migrate to other network nodes. A feature of the agents is the behavior realization. The behavior is determined by the mathematical function, which implements the steps of sensor data processing. Other options determine the agent behavior in case of certain kinds of situations. The model of brokers is offered to agent interaction with server applications at the data center. Broker is an agent that runs on routers and realizes the storage, data protection, transmission and warehouse loading functions.
The multiagent system includes the following software agents:
Agent for the synthesis and control of the photoradar devices queue for inquiry.
Agent for creating threads for asynchronous device polling.
Agents of data polling from devices, separated by geographic zones and by types (devices Cordon-Temp, KrisP, Parkon, etc.). The data polling from device sensors is carried out by the agents from different zones using the SNMP protocol.
Agents that keep event logs directly on the complexes. Each complex maintains a local database, recording events in the log files. A lot of local databases represent a distributed hierarchical data warehouse. Agents keep a log file of vehicle passages, a log file of traffic violations, a log file of telemetry parameters for device diagnostics, and so on.
Agents for uploading data from device logs to central storage. Agents work through the web interface and generate a lot of files in XML format. One file contains the data of one violation and is associated with a digital signature file and violation pictures.
Agent for aggregating data about recorded violations for a period of time. This agent generates a comma separated values (CSV) file containing rows with parameters of all violations for a given period. It is an element of a distributed fog database. A lot of CSV files on different devices form a distributed hypertable of summary data on violations over a period of time.
Agent for aggregating the values of the complex parameters over a period of time. This agent creates a CSV file containing rows with the values of the complex parameters. It is also an element of a distributed database. A lot of CSV files with parameters of different devices form a hypertable for their diagnosis over a period of time.
Agent for parsing the files with violation parameters and parameters of the complexes for loading data into the central cloud storage.
Data mining agents for the analysis of data violations. This group of agents analyzes the time series of the uploaded violations data over a time period to identify the dependencies of growth or to reduce violations from various factors.
Data mining agents for the analysis of device parameter. This group performs analysis of time series of device parameters to detect parameter deviations from the required and reference values (benchmarking). The tools include data visualization agent, data aggregation agent, data selection agent, data mining agent and data analyze agent [31, 32].
Agents for forecasting violations of traffic rules and agents for forecasting failures and errors in the operation of complexes. Forecasting is performed using the technique of deep machine learning based on the synthesis of a fuzzy neural network, its training and forecasting changes in the operating parameters of the complex.
Agents for data visualization on computers in the data center and on mobile units. A variety of agents form a distributed content management system. Agents are downloadable php and js scripts. They allow in standard browsers to present historical, current and forecast data in the form of graphs, tables and dashboards. The data correspond to the polling time and geographical coordinates of the complex location.
The data aggregation agent is needed to support the technology of work with database in the aggregation mode for the selection and visualization of hypertable data mart. When the mode is setup, the user should define a set of object properties (columns of values) that will be shown in the hypertable. Available properties can be selected from the drop-down list.
The visualization agent allows to see information in the hypertable data mart. The hypertable is a nonstandard user interface for data visualization. It combines the functionality of a classic table with a tree structure. Elements of the hypertable can be located on distributed sensory nodes. Elements allow to view the dynamic changes of the values changes in real time. The data are grouped according to the parameters and levels of aggregation. A distinctive feature of the hypertable is that the number of rows is not a static value, a row character and functionality are not equal and some of them are the aggregates. The aggregates are nodal and show summary information on the relevant columns of the lower-level aggregation rows. The actual number of the hypertable rows varies dynamically, depending on the grouping of rows. Another feature of the hypertable is the ability to view quickly and analyze changes. The user can view hypertable change of any selected index for the period, as well as the predicted values for the specified forecast horizon. An example of data visualization, photoradar complex parameters, is shown in Figure 3.
The data analyze agent allows choosing the data needed for the analysis of a concrete situation. The data marts selection criteria can be quite complex. For this purpose, the system uses multilevel queries and filters that limit the data choice. The agent allows the personnel easily create queries to choose the right information.
5. Prognostic models and machine learning methods for multiagent diagnostics of photoradar complexes and road incidents
The diagnostic system is necessary for remote maintenance of photoradar equipment. The system should monitor the complex parameters, transmit telemetric information, predict possible malfunctions and automatically report about failures. The complex has a set of parameters such as supply voltage, response time, housing temperature, ambient temperature, and so on. Since the complexes are distributed over a large area, a multiagent remote diagnostic system is being developed to monitor their operation. Key element of a diagnostic system is the mechanism of forecasting a change, depending on its current parameters, level of external indignations and the influences. The data mining tasks and failures’ forecasting tasks are solved using deep machine learning and fuzzy neural networks based on the analysis of time series of complex parameters. The monitoring task for the equipment is determined by the high requirements for the uninterrupted operation of devices. In the event of emergency situations, the minimum time is allocated to correct the malfunctions. For a short time, it is necessary to determine the order of repair work and the required amount of resources such as the working time of specialists, the need to operate machinery and the required spare modules. The evaluation of the reliability of the complex is based on the data analysis on the device state at times and data analysis on the violations in the complex operations. Data for assessing the reliability of the complex include the following:
The work time of complex for the reporting data
The uninterrupted work time of complex
The time of fixing traffic accidents
The number of fixed objects
The number of recorded traffic accidents
The complex downtime for failure
The failure frequency
The cost of repairs
The number of errors in fixing traffic accidents
The purpose of the analysis is to identify the deviations of the complex parameters and to detect errors and malfunctions. The deviations of the parameters are exceeding thresholds, deviation of values from normative and normalized previous data. The results of forecasting are used to plan an unscheduled repair work in order to prevent possible failures. The work schedule depends on the following parameters: location of the complex; density of traffic on the repair location; availability of spare parts; nature of the malfunction; types of repair work carried out earlier with the device; frequency of malfunctions and required resources to restore functionality.
We consider the system of forecasting of a qualitative condition of the photoradar complex on the basis of indistinct implication . In case of
If to consider that is available
Rule 1: If is and is …. and is , then, is ,
Rule 2: If is and is …. and is , then, is ,
We present further sequences of the functioning of the diagnostics system of photoradar complexes with a conclusion of Mamdani-Zade in the form of the following stages:
The first stage is the aggregation of the reasons for failures in the systems: The arriving value of the function
The second stage is the aggregation effects of disruption of complexes: Each implication of the unique value of function
The third stage is the aggregation of results: At this stage, the operator of the sum is applied to aggregation of results of implication of many rules.
In final part of the conclusion of Mamdani-Zade, the procedure of a defuzzification allowing to receive accurate value of an output variable—the predicted condition of photoradar complex is carried out (Figure 4).
The main weak spot in an implication method with a conclusion of Mamdani-Zade is subjectivity of the creation of a grid of rules and functions of accessory. This defect method can be eliminated by creation of the hybrid computing mechanism where implication of Mamdani-Zade is mediated by work of the neural network (NN), with the training mechanism inherent in it. Forecasting is the process of making predictions of the future based on past and present data. Forecasting accuracy is constantly being improved with the continual introduction of machine learning techniques. Time series sensor data are any data set that collects telemetry information regularly over a period of time. The fundamental problem for machine learning and time series is the same: to predict new outcomes based on previously known results. Time series and machine learning can be combined together in order to give the benefits of each approach. Time series does a good job at decomposing data into trended and seasonal elements. This analysis can then be used as an input for an NN model, which can incorporate the trend and seasonal information into its algorithm. The NN represents the parallel computing system consisting of a large number of elementary units of information processing—the neurons, accumulating experimental knowledge and providing them for the subsequent processing. The term “training” is understood as ability of NN to receive reasonable results on the basis of the data, which were not found in the course of training. The sequence of training on the basis of procedure of the return distribution is presented in Figure 5.
This property is used at realization of hybrid indistinct neural network (HINN). We consider the sequence of functioning of HINN (Figure 6).
On the first layer, the fuzzification is carried out. The formula of a fuzzifikation looks as follows:
It is necessary to consider that generally, the number of functions of accessory does not coincide with the number of rules. Therefore, if each
In the second layer, the aggregation of values of the
Thus, the calculated parameters
The third layer when using a conclusion of Mamdani-Zade calculates the centers for k-rules for a formula: , where
After that aggregation of a consequence with the use of operation of algebraic work is carried out: .
The fourth layer is presented by two neurons
The fifth layer is presented by the unique neuron, which is carrying out a defuzzification:
The algorithm of HINN training can conditionally be shared into two stages. At the first stage, parameters of the center of output functions of accessory in the third layer are subject to training. For this purpose, parameters of scales for the fixing of parameters of functions of accessory on the first layer (center, width and form) were determined as:
It should be noted that output signals
Further, the decision of system of the equations is carried out on the basis of pseudo-inversion of matrixes:
At the second stage, after fixing of values of linear parameters calculated the actual exits of HINN
6. The results of the intellectual analysis
This method was also used to forecast traffic accidents, based on the processing of accident statistics on controlled road sections. A method for forecasting road accidents was implemented depending on three factors: the amount of traffic flow per unit of time, the number of road accidents and the temperature indicators in the control areas. Before we begin to analyze how to conduct traffic accident inference with location and time information, a proper data structure is needed. When analyzing such spatial and temporal data, the use of matrix is widely accepted as the first choice. For temporal dimension, in order to match the time interval of traffic accident data, we select 1 hour as the time interval and divide 1 day into 24 slices. For spatial dimension, we mesh location into
To accumulate the statistics, their spatial and intellectual analysis, synthesis of graphs and reports to support decision-making, the system employs a special agent for remote polling of photo-video fixing complexes and automatic unloading of data on driver offenses and road accidents. Statistical data are presented in the form of time series or function graphs (Figure 8) of incidents, changes in speed and density of the flow of vehicles in controlled areas, ambient temperatures and are input parameters for training the neural network.
In the process of analyzing time series with the moments of road incidents, time intervals were chosen in which the number of incidents deviated from the average indicators. As an example, we present graphs of statistics on incidents collected from six complexes during the month (Figure 9).
Analysis of the data presented in the graphs showed anomalies. It is seen that for a month on five complexes (KD0173, KD0174, KD0183, KD0122 and KD0180) that the number of road incidents is fixed, which on the average is about 60–70 units with the exception of the KD0201 complex. However, after April 17, there is a decrease in the number of road incidents simultaneously on all the complexes.
To determine the causes of anomalies and the forecast of incidents, meteorological data (temperatures, atmospheric pressure and precipitation) were collected at anomalous areas at similar time intervals.
Indicators in the form of the number of incidents, temperature values and traffic density values have become input parameters for training the neural network. The number of neurons of the first layer of the network was set to 18 and the number of rules as 9. After training the network, a forecast was made for road accidents (Figure 10).
The results of the network showed an acceptable error in the forecast of an average of 13%. The model made it possible to determine the dependence of the number of incidents on the changes in traffic and on the temperature regime in the controlled sections of the road. In particular, the prognostic model showed the dependence of the level of incidents recorded by the Kordon-Temp complexes on the M-5 (Ural, Russia) route from changes in temperature and precipitation. It can be concluded that the neural network and the prediction system provide sufficient accuracy for the prediction model.
The results of monitoring and analysis of traffic accidents, fixed by an intelligent monitoring system with photoradar complexes, are considered. A multiagent approach was developed to address the tasks of collecting and processing sensor data. Functionality of agents and brokers is defined as a mathematical function that determines the action to sensor data processing and the selection of behaviors to respond to emerging events. The system functionality is implemented by several agents that perform data collecting, cleaning, clustering, comparing time series, retrieving data for visualization in the dynamic hypertable form, preparing charts and reports, performing spatial and intellectual analysis, generating push notifications to mobile client, and so on. To accumulate the statistics, their spatial and intellectual analysis, synthesis of graphs and reports to support decision-making, the system employs special agents for remote polling of photo-video fixing complexes and automatic upload of data. The agent collects and downloads multimedia data such as photos and frames from the video stream, as well as various sensor data on traffic parameters.
Convergent approach is the convergence of distributed data processing technologies (cloud, fog and mobile computing). The model is designed for the collection, processing and integration of sensor data obtained in the process of monitoring and control of spatially distributed objects and processes. Convergent model of distributed computing includes three levels of data processing. The first level is fog computing. Here, processing and aggregation of sensor data is realized by migrating software agents in heterogenic sensor networks. At the next level (cloud computing), sensor data and aggregates are implemented in the server cluster. The cluster includes the main server to control the hypervisor and network servers at local network. The third level is implemented on mobile systems, where agents are to retrieve and visualize the results of monitoring and intellectual analysis with geo-information technologies.
The tasks of intellectual analysis and forecasting using methods of deep machine learning are solved. As a prognostic model, a hybrid fuzzy neural network was synthesized and its training was performed. The structure of the neural network is adapted to the problems of diagnosing and forecasting the operation of photoradar complexes, as well as for analysis and prediction of road accidents. As an example, consider the results of the intellectual analysis of unloading data collected from complexes in a month’s time in comparison with meteorological data in order to reveal the patterns of variation in the number and severity of road incidents. In the process of spatial analysis, similar sections of the road and transport infrastructure are identified by the number and type of traffic accidents. Clustering of such areas allows to define the most emergency areas. In the process of intellectual analysis of time series, time intervals are determined, in which an abnormal deviation of the incidents number from average indicators are occurred. A comparison of the time series of road accidents and time series of meteorological factors has shown that changes in the traffic situation in controlled areas are strongly dependent on weather conditions.
The reported study was funded by Russian Foundation for Basic Research (RFBR) according to the research projects № 18-07-00975, 16-07-00031, 17-307-50010, 17-37-50033.