Advanced Driving Assistance System for an Electric Vehicle Based on Deep Learning

Abstract

This chapter deals with a design of a new speed control method using artificial intelligence techniques applied to an autonomous electric vehicle. In this research, we develop an Advanced Driver Assistance System (ADAS) which aims to enhance the driving manner and the safety, especially when traveling too fast. The proposed model is a complete end-to-end vehicle speed system controller that proceeds from a detected speed limit sign to the regulation of the motor’s speed. It recognizes the speed limit signs before extracting from them, a speed information that will be sent, as reference, to a NARMA-L2 based controller. The study is developped specially for electric vehicle using Brushless Direct Current (BLDC) motor. The simulation results, implemented using Matlab-Simulink, show that the speed of the electric vehicle is controlled successfully with different speed references coming from the image processing unit.

1. Introduction

Nowadays, internal combustion engine vehicles are the major source of air pollution and damage to our health. To solve these problems, electric vehicles are one of the most encouraging energy saving and environmental protection solutions. However, such solutions show luck of autonomy. The current trend in vehicle transportation is to implement solution where the driver and an intelligent system coexist and communicate to improve road safety and security. Advanced driver assistance systems (ADAS) are bridging the gap between traditional electric vehicles and the vehicles of tomorrow which are intelligent, autonomous and safe for the drivers, the passengers and everyone else on the road.

Generally, an autonomous electric vehicle consists of 4 main modules: the energy source module, the auxiliary module, the intelligent module composed of computers, sensors and actuators, and an electrical propulsion module [1], which consists of an electronic controller, a power converter, a mechanical transmission and an electric motor of either AC or DC, as shown in Figure 1. The motors convert the electrical energy that comes from the battery into a mechanical energy that allows the vehicle to move. They can also be considered as generators when sending energy back to the source of energy.

Figure 1.

Block of diagram of an Autonomous electric vehicle.

According to the specific requirements needed for autonomous electric vehicle’s applications, such as, robustness, power, speed range and level of noise, the choice of motors can vary depending on their types. DC motors have been widely used since they fulfill some of the requirements mentioned before [2]. However, in recent years, in order to compensate for their electrical losses and mechanical friction, they have been replaced by Brushless DC Motors (BLDC), given their high efficiency and low noise [3, 4, 5]. Besides, BLDC are known to have a big initial torque with small size and low weight. They also can be built in the tires to reduce the complexity and weight of the driving mechanism.

Speed regulation is an important control challenge for any BLDC motor. In [6], the authors proposed an implementation of Space Vector Pulse Width Modulation (SVPWM) for the control of the power converter and the BLDC motor. They showed that using SVPWM methodology, offers the minimum switching losses, reduces harmonics compared with the other Pulse Width Modulation (PWM) methods.

In order to enhance and perfect the speed adaptation of the autonomous electric vehicle, while assisting at the same time the driver, and increasing its safety and security, ADAS has been developed, relying on inputs from multiple data sources, including automotive imaging, image processing, and in-vehicle networking.

In fact, It has been said that 80–90% of the driver’s performance depends on visual information [2]. This is why, a huge number of collisions occur when the driver is not looking forward or unable to stop at an intersection due to excessive speed. It will be very important to automatically detect the speed limit sign and control the vehicle’s speed when it is traveling too fast. Various ADAS are now designed with a large number of sensors and actuators to analyze the environment and take the appropriate actions. Among the existing ADAS systems, the traffic sign recognition holds a lot of potential as they enhance the driver’s safety by notifying him about possible dangers related to speed.

The automatic recognition of these signs, however, is not easy to carry due to the weather’s conditions, the blur resulting from moving vehicles and the lighting conditions. Figure 2 shows some of these factors that make it difficult to identify the road signs.

Figure 2.

Difficulties that affect the traffic sign recognition systems.

To handle these challenges, researchers recommended the use of image processing and machine learning techniques. The automatic recognition of traffic signs includes, mainly, the traffic sign detection and the traffic sign classification.

Traffic signs have several distinctive features like colors, shapes and symbols. In the detection stage, the input images are preprocessed, enhanced and then, segmented according to their color or geometry. Color-based methods usually use normalized RGB space [7, 8, 9, 10] or HSV space [11, 12, 13, 14, 15, 16] or YUV space [17] to distinguish between traffic and non traffic signs. However, these methods are generally affected by the weather conditions and the illumination variations.

Geometry-based methods are, on the contrary, robust to the illumination changes since they characterize the shape of the traffic sign. Mainly, authors used corner detection [11, 18], distance transform [19], Hough transform [20, 21] or radial geometry voting [22, 23, 24].

Recently, many researchers combined the color-based methods with the shape-based methods to achieve better results [10, 13, 16, 25, 26]. For instance, authors in [16], used color segmentation to roughly identify the sign before applying a matching technique based on the logical XOR operator using the shape extracted details. Similarly, authors in [10] used color segmentation to roughly locate the signs and then, used the shape information to eliminate the false candidates.

As for the classification stage, many methods have been used to identify the class of the traffic signs, such as, Support Vector Machine (SVM) [8, 13, 16], Viola-Jones detector [27, 28], neural networks [2, 15] and random forest [29, 30].

As we can see, all the presented researches focused only on recognizing traffic signs without trying to integrate and test them into a complete system controlling mechanical and electrical part. In [2], the authors designed an end-to-end system, where they proposed a NARMA-L2 neuro controller for speed regulation based on steerable decomposition and Bayesian neural networks. Despite the acceptable accuracy rate (about 0.975), it failed, unfortunately, to accurately recognize noisy signs.

In this chapter, we continue to design an end-to-end system, based on deep learning approaches, that enhances autonomous vehicle speed control and presents better performances in the presence of noisy inputs.

The rest of this chapter is organized as follows: in Section 2, we describe the proposed model. Section 3 will be reserved to the experimental results and finally, Section 4 will conclude the paper.

2. The proposed speed controller based on traffic sign recognition

The proposed speed control system comprises two main modules, an image processing module and a control module. The first module recognizes the speed limit signs before extracting from them, a speed information that will be sent to the control module. An overview of these units is shown in Figure 3.

Figure 3.

The proposed speed control system.

2.1 The image processing module

This first module aims to recognize the speed limit traffic sign whatever the weather conditions are. Figure 4 presents the proposed model of the speed limit sign recognition system. This system will perform two important tasks: detecting the sign and then identifying the speed. As it is shown in Figure 4, this unit receives an input image, performs grayscale transformation before normalization and noise removal. Lastly, each preprocessed image will be fed to a CNN unit for a classification process.

Figure 4.

The proposed speed limit sign recognition module.

2.1.1 The preprocessing stage

In order to prepare the deep neural network to learn relevant features from speed-limit images, additional processing is required: first of all, we expand the training images, then, we normalize the augmented images, and finally, we filter them with a median filter.

• Data normalization:

  In this step, we normalized the gray scale images in order to reduce poor lighting variations observed in the database. Let $\mathit{Im}\left(i,j\right)$ denotes the grayscale value of pixel $\left(i,j\right)$, $\mathit{Me}$ and $\mathit{Std}$ denote the estimated mean and standard deviation of $\mathit{Im}$, respectively, and $\mathit{Nor}\left(i,j\right)$ denotes the normalized grayscale value of pixel $\left(i,j\right)$. The normalized image is defined using Eq. (1):

$$\mathit{Nor}\left(i,j\right)=\frac{\left(\mathit{Im}-\mathit{Me}\right)\ast {\mathit{\text{Constant}}}_1}{\mathit{Std}+{\mathit{\text{Constant}}}_2}$$E1

${\mathit{\text{Constant}}}_1$ and ${\mathit{\text{Constant}}}_2$ are two constants set experimentally to $50$ and $100$ respectively. Figure 5 shows the images obtained by 1. Finally, a median filter is applied to the normalized input image to obtain an enhanced speed sign image.

• Data augmentation: Deep neural networks require a huge learning database to perform the speed limit recognition task. However, most publicly available databases suffer from lack of data. Increasing these databases is, therefore, a crucial step for an accurate sign recognition. Moreover, an augmentation on the training data makes the proposed model more robust to geometric changes. Figure 6 shows a sample speed limit image with different augmentation techniques applied to it: vertical flipping, rotation with small angles $\in \left[-8^{\circ },8^{\circ }\right]$ and horizontal translation of 1 unit to both sides right and left.

Figure 5.

Data normalization: a. Original images. b. After Grayscale transformation, c. After normalization by Eq. (1) and d. after median filtering.

Figure 6.

Some of the transformations used for data augmentation.

2.1.2 The deep learning stage

In this chapter, in order to extract features and organize speed signs in categories, a Convolutional Neural Network was therefore used. The architecture of the proposed CNN, which has three convolution layers, and one output layer, is presented in Figure 7. The first, third and fifth layers are convolution layers with 8, 16 and 32 kernels, respectively, with a size of $5x5$. The activation function in the CNN is a rectified linear unit (Relu) function. Sub-sampling is presented in the second, fourth and sixth layers. We used also a max pooling layer with a kernel size of $2x2$ and a step size of $2$. We flatted the sub-sampling to a 1152-dimensional vector and directly connected the output layer with a soft-max activation function.

Figure 7.

Proposed CNN’s structure.

2.2 The speed control module

The aim of this unit is to control the speed of the autonomous electric vehicle using information coming from the image processing unit as speed reference and then, input it to the control unit. The studied traction system is composed of BLDC motor, three phase Mosfets inverter, a gearbox bloc associated with mechanical differential used for speed adaptation for the shaft and the two wheels (see Figure 8). The traction system can be changed by a single wheel [31] if we don’t take in consideration the mechanical differential part.

Figure 8.

Control strategy block diagram of the EV drive system using BLDC motor.

Figure 8 shows the control strategy of the AEV using BLDC motor. The feedback signals are the motor speed signal measured by the speed sensor and the rotor’s position taken from the three hall sensors. The speed controller unit receive the reference speed signal from the digital processing unit (detected from the trafic speed sign) and the actual speed of the motor (actual vehicle speed). Then, the generated PWM refers to the error between the reference speed and the measured speed as well as the commutation sequence of three hall sensor signals $H_{\mathit{sa}}$, $H_{\mathit{sb}}$ and $H_{\mathit{sc}}$ (Figure 9) in order to control the three phase inverter switches [32].

Figure 9.

Hall sensors output signals in a 360 electrical degrees cycle.

The three phase inverter consists of six Mosfet switches $Q_{i,i=1,..,6}$ and six freewheeling diode as shown in Figure 10. Considering that the motor is in clockwise revolution, the state of the six switches ($S_{w1}$, $S_{w2}$, $S_{w3}$, $S_{w4}$, $S_{w5}$ and $S_{w6}$), depending to the three Hall sensors state ($H_{\mathit{as}}$, $H_{\mathit{sb}}$ and $H_{\mathit{sc}}$), are shown in Table 1.

Figure 10.

Equivalent circuit of the BLDC motor associated with three-phase inverter.

2.2.1 Mathematical model of the studied BLDC motor

The BLDC motor has three windings coupled in Y-connected on the stator with a permanent magnets rotor with. If we neglect any saturation effects with a constant parameters in the three phase.

The electrical equations of the BLDC motor are described by:

$$\left[\begin{array}{c}{V}_{\mathit{as}}\\ V_{\mathit{bs}}\\ V_{\mathit{cs}}\end{array}\right]=\left[\begin{array}{ccc}{R}_s& 0& 0\\ 0& R_s& 0\\ 0& 0& R_s\end{array}\right]+\frac{d}{\mathit{dt}}\left[\begin{array}{ccc}{L}_{\mathit{aa}}& L_{\mathit{ab}}& L_{\mathit{ac}}\\ L_{\mathit{ba}}& L_{\mathit{bb}}& L_{\mathit{bc}}\\ L_{\mathit{ca}}& L_{\mathit{cb}}& L_{\mathit{cc}}\end{array}\right]\left[\begin{array}{c}{i}_a\\ i_b\\ i_c\end{array}\right]+\left[\begin{array}{c}{e}_a\\ e_b\\ e_c\end{array}\right]$$E2

Where:

• $V_{\mathit{as}}$, $V_{\mathit{bs}}$ and $V_{\mathit{cs}}$ are the Stator voltages,

• $R_s$ is the stator resistance ($R_a$ = $R_b$ = $R_c$ = $R_s$),

• $i_a$, $i_b$ and $i_c$ are the stator currents,

• $L_{\mathit{aa}}$, $L_{\mathit{bb}}$ and $L_{\mathit{cc}}$ are Self Inductances of phases $a$, $b$ and $c$, respectively,

• $L_{\mathit{ab}}$, $L_{\mathit{ac}}$ and $L_{\mathit{bc}}$ are Mutual Inductances,

• $e_a$, $e_b$ and $e_c$ are back EMF’s.

$$\mathit{\text{If}}\left\{\begin{array}{l}{L}_{\mathit{aa}}=L_{\mathit{bb}}=L_{\mathit{cc}}=L\\ L_{\mathit{ab}}=L_{\mathit{ba}}=L_{\mathit{ac}}=L_{\mathit{ca}}=L_{\mathit{bc}}=L_{\mathit{cb}}=M\end{array}\right.$$E3

The state space representation of motor becomes:

$$\left[\begin{array}{c}{V}_{\mathit{as}}\\ V_{\mathit{bs}}\\ V_{\mathit{cs}}\end{array}\right]=\left[\begin{array}{ccc}{R}_s& 0& 0\\ 0& R_s& 0\\ 0& 0& R_s\end{array}\right]+\frac{d}{\mathit{dt}}\left[\begin{array}{ccc}L& M& M\\ M& L& M\\ M& M& L\end{array}\right]\left[\begin{array}{c}{i}_a\\ i_b\\ i_c\end{array}\right]+\left[\begin{array}{c}{e}_a\\ e_b\\ e_c\end{array}\right]$$E4

In addition, at balanced condition of motor phase, we have:

$$i_a+i_b+i_c=0$$E5

and

$$L_s=L-M$$E6

so the state space representation is:

$$\left[\begin{array}{c}{V}_{\mathit{as}}\\ V_{\mathit{bs}}\\ V_{\mathit{cs}}\end{array}\right]=\left[\begin{array}{ccc}{R}_s& 0& 0\\ 0& R_s& 0\\ 0& 0& R_s\end{array}\right]+\frac{d}{\mathit{dt}}\left[\begin{array}{ccc}{L}_s& 0& 0\\ 0& L_s& 0\\ 0& 0& L_s\end{array}\right]\left[\begin{array}{c}{i}_a\\ i_b\\ i_c\end{array}\right]+\left[\begin{array}{c}{e}_a\\ e_b\\ e_c\end{array}\right]$$E7

The three back EMF (have trapezoidal form) are represented by:

$$\left[\begin{array}{c}{e}_a\\ e_b\\ e_c\end{array}\right]={\omega }_m.{\lambda }_m\left[\begin{array}{c}{f}_{\mathit{as}}\left({\theta }_r\right)\\ f_{\mathit{bs}}\left({\theta }_r\right)\\ f_{\mathit{cs}}\left({\theta }_r\right)\end{array}\right]$$E8

Where:

• ${\omega }_m$ is the rotor speed (rad/s),

• ${\theta }_m$ is the rotor position (rad),

• The functions $f_{\mathit{as}}\left({\theta }_r\right)$, $f_{\mathit{bs}}\left({\theta }_r\right)$ and $f_{\mathit{cs}}\left({\theta }_r\right)$ are represented by Table 2.

Cycle	Hall Sensors			Switches State
	$H_a$	$H_b$	$H_c$	$S_{w1}$	$S_{w2}$	$S_{w3}$	$S_{w4}$	$S_{w5}$	$S_{w6}$
1	1	0	1	1	0	0	1	0	0
2	1	0	0	1	0	0	0	0	1
3	1	1	0	0	0	1	0	0	1
4	0	1	0	0	1	1	0	0	0
5	0	1	1	0	1	0	0	1	0
6	0	0	1	0	0	0	1	1	0

Table 1.

Hall sensors output and the switch state.

${\theta }_e$	$f_{\mathit{as}}\left({\theta }_r\right)$	$f_{\mathit{bs}}\left({\theta }_r\right)$	$f_{\mathit{cs}}\left({\theta }_r\right)$
$0^{\circ }$-${60}^{\circ }$	1	−1	$1-\frac{6{\theta }_e}{\pi }$
${60}^{\circ }$-${120}^{\circ }$	1	$\frac{6{\theta }_e}{\pi }-3$	−1
${120}^{\circ }$-${180}^{\circ }$	$5-\frac{6{\theta }_e}{\pi }$	1	−1
${180}^{\circ }$-${240}^{\circ }$	−1	1	$\frac{6{\theta }_e}{\pi }-7$
${240}^{\circ }$-${300}^{\circ }$	−1	$9-\frac{6{\theta }_e}{\pi }$	1
${300}^{\circ }$-${360}^{\circ }$	$\frac{6{\theta }_e}{\pi }-11$	−1	1

Table 2.

Functions $f_{\mathit{as}}\left({\theta }_r\right)$, $f_{\mathit{bs}}\left({\theta }_r\right)$ and $f_{\mathit{cs}}\left({\theta }_r\right)$.

2.2.2 Implementation of NARMA-L2 neuro controller for speed regulation

Many speed controllers have been frequently used in literature, such as PI (proportional integral) and PID (Proportional integral derivative) (PID), given their simple structure, rapid-reaction and reasonable cost. However, they exhibit a slow response when associated with dynamic loads. Recently, intelligent-based controller, such as neural networks control (NNC), genetic algorithms and fuzzy logic control, were exploited in the speed control of BLDC [2]. Among these techniques, the neural networks are considered in this chapter, because they are the most suitable to handle the non-linearity of the BLDC system that contains uncertainties. Thus, an intelligent neuronal controller is proposed, based on Nonlinear Auto-regressive Moving Average Level-2 model (NARMA L2).

There are two steps involved in the control process. The first step is the feedback linearization to identify the system to be controlled, while the second step is the training of the system’s dynamics. Generally, the NARMA L2 nonlinear description of the system is represented by a discrete-time $n^{\mathit{th}}$ order equation (Eq. (9))

$$\begin{array}{l}y\left(k+d\right)=f(y\left(k\right),y\left(k-1\right),\dots y\left(k-n+1\right),u\left(k\right),u\left(k-1\right),\\ \dots ,u\left(k-n+1\right))+g(y\left(k\right),y\left(k-1\right),\dots y\left(k-n+1\right),\\ u\left(k\right),u\left(k-1\right),\dots ,u\left(k-n+1\right)).u\left(k\right)\end{array}$$E9

Where $u\left(k\right)$ is the system input, $y\left(k\right)$ the system output and $d$ is the system delay. $f\left(.\right)$ and $g\left(.\right)$ are the additive and the multiplicative non-linear terms respectively, to be approximated in the training step. The Figure 11 shows the structure of NARMA-L2 Model (Figure 12).

Figure 11.

NARMA-L2 Model.

Figure 12.

Specifications of the plant model.

3. Experiments and discussion

In this section we present the dataset and the details of the experiments performed in this study. Experiments were performed to select the best number of folds, the best number of layers and the best optimization’s function. The final proposed model was implemented using Matlab 2018 with an Intel Core i7 1.8 GHz CPU working in a windows 10 environment. The simulink model of the proposed algorithm is presented by Figure 13.

Figure 13.

Simulink model of our algorithm.

3.1 Used database

In order to validate the proposed method, we used the German traffic sign recognition benchmark (GTSRB). To solve the traffic sign recognition problem, this database has been made with different visual indications. The image qualities vary depending on the illumination, the contrast and the color. Table 3 shows some details of the data-set used in our work.

Speed limit sign	Class	Nb of Training image	Nb of Testing image
30	1	2220	17
50	2	2250	27
60	3	1410	229
70	4	1980	304
80	5	1858	573
100	6	1439	281
120	7	1410	450
Total images		12567	1881

Table 3.

Characteristics of the used database.

3.2 Speed sign recognition result

The partitioning of the input data was performed randomly according to 10-folds cross-validation procedure: we divided the training database into ten folders. Each time, we used one folder as a validation set and we trained the nine remaining folders. The process will continue until the validation error starts increasing. At this moment, the training procedure will be stopped, and then saved. After running the 10-folds, we have selected the best neural network obtained with the best validation performances. An example can be shown in Figure 14. Concerning the final recognition rate, it will be calculated after trying the test database with the selected network that assigns the value of 1 for exact speed limit sign and 0 for all other signs (Figure 15). The average speed limit sign recognition rate obtained after several tries is 90.8% with a validation accuracy 0f 94.75% (Table 4).

Figure 14.

Example of the best validation performance. The training stops when the validation error is increasing.

Figure 15.

Confusion matrix of the deep learning process.

K-Folds	Validation accuracy	Testing accuracy	AUC
3-folds	93.005 (%)	86.44 (%)	0.98
5-folds	93.95 (%)	87.9 (%)	0.998
8-folds	95.41 (%)	90.1 (%)	0.993
10-folds	94.75 (%)	90.8 (%)	0.999

Table 4.

Speed limit recognition accuracy with different k-folds.

As can be seen from Figure 16, 100 and 60 are two of the most difficult signs to classify, whereas 30 and 50 are the easiest to recognize. Table 4 summarizes the recognition rates obtained with different folds: 3, 5, 8 and 10 as well as the Area-Under-Curve of the ROC curve. The best recognition rates were obtained with 10-folds. Besides, we show in Table 5 that the adam function is a best choice for the optimization of the deep learning network.

Figure 16.

Average recognition rates per speed limit sign.

Function	Testing accuracy
Sgdm	86.28 (%)
Rmsprop	84.68 (%)
adam	90.8 (%)

Table 5.

Speed limit recognition accuracy with different optimization function, K = 10 folds.

3.3 The NARMA L2 controller’s result

The speed detected from the sign recognition process will be taken as a reference input to the NARMA L2 controller. Here, we must also train the controller to adjust the vehicle speed. The training data obtained from the NARMA-L2 controller are illustrated in Figure 17.

Figure 17.

Training data of NARMA L2 controller.

Figure 18, displays the speed response of the proposed system with both increase and decrease of speed reference. We notice that the speed reaches rapidly the desired value. The proposed controller presents a good behaviour upon disturbances, and thus it is stabilized after duration of 0.05 s with an almost no static error.

Figure 18.

Desired Speed curve with the NARMA L2 controller.

4. Conclusions

Deep learaning, image processing and NARMA-L2 controllers have been successfully developed and simulated using MATLAB to control the speed of a BLDC Motor by recognizing the traffic sign images. Simulation results show effectiveness of the proposed controllers for dealing with the motor system with nonlinearity under wide dynamic operation regimes.