Particle-Filter-Based Intelligent Video Surveillance System

Abstract

In this study, an intelligent video surveillance (IVS) system is designed based on the particle filter. The designed IVS system can gather the information of the number of persons in the area and hot spots of the area. At first, the Gaussian mixture background model is utilized to detect moving objects by background subtraction. The moving object appearing in the margin of the video frame is considered as a new person. Then, a new particle filter is assigned to track the new person when it is detected. A particle filter is canceled when the corresponding tracked person leaves the video frame. Moreover, the Kalman filter is utilized to estimate the position of the person when the person is occluded. Information of the number of persons in the area and hot spots is gathered by tracking persons in the video frame. Finally, a user interface is designed to feedback the gathered information to users of the IVS system. By applying the proposed IVS system, the load of security guards can be reduced. Moreover, by hot spot analysis, the business operator can understand customer habits to plan the traffic flow and adjust the product placement for improving customer experience.

1. Introduction

Video surveillance systems are often utilized at some specific places such as exits, entrances, parking lots, convenient stores, etc. for management. Traditionally, security guards watch screens of monitored places for surveillance. However, according to statistics, one security guard can only watch four monitors at the same time, and the concentration can last only for 10 minutes such that more than 50% of key information is lost. Moreover, it is not economically worthwhile to hire multiple security guards to monitor one video surveillance system. Therefore, intelligent video surveillance (IVS) systems have become more and more important in commercial sector and have attracted a lot of attention in research area as well [1, 2, 3].

IVS system can be defined as the real-time monitoring of both persistent and transient objects within a specific environment. IVS is also referred to as video analytics (VA) which involves the use of software to automatically detect the objects of specific interest and analyze their behaviors. For finding the objects of interest, it is usually done by detecting the movements or changes in the image that can be achieved by background subtraction technique. To do background subtraction, an effective way to build up the background is the Gaussian mixture model [4, 5]. After the object of interest is detected, the goal is to analyze their behavior that sometimes can be done by tracking them. Usually, particle filters and Kalman filters are employed for the purpose of tracking objects in IVS systems [6, 7, 8, 9, 10].

IVS systems have been applied for different kinds of purposes. Ref. [11] presents an architecture for a perimeter security system dedicated to critical transport infrastructure protection. Ref. [12] addresses a framework for event decision of vision-based intelligent surveillance system based on the fuzzy model. A shape-perceived algorithm using the building block-based matching method is presented in [13] for object tracking of intelligent surveillance applications. A way of unification of flame and smoke detection algorithms by merging the common steps into a single processing flow is proposed in [14] for IVS systems.

The aim of this study is to design a IVS system based on the particle filter. The designed IVS system can gather the information of the number of persons being in the area, the number of persons having been in the area, and hot spots (places of more than usual interest, activity, or popularity) of the area. The Gaussian mixture background model is utilized to detect moving objects by background subtraction in the designed IVS system. The moving object appearing in the margin of the video frame is considered as a new unit (person). When a new person is detected, a new particle filter is established and assigned to track the new person. For saving the computational load, the particle filter is terminated when the corresponding tracked person leaves the video frame. Moreover, the Kalman filters is utilized to estimate the position of the person when the person is occluded. Information of the number of persons in the area (having been in the area) and hot spots is gathered by tracking persons in the video frame. Finally, a user interface is designed to feedback the gathered information to users of the IVS system. By applying the proposed IVS system, the load of security guards can be reduced. Moreover, by hot spot analysis, the business operator can understand customer habits to plan the traffic flow and adjust the product placement for improving customer experience.

2. Preliminary

In this section, three well-known techniques, i.e. adaptive Gaussian mixture model, particle filter, and Kalman filter, are presented for constructing the IVS system.

2.1. Adaptive Gaussian mixture model

The recent history of each pixel, $\left\{X_1,X_2,\cdots X_t\right\}$, is modeled by a mixture of k Gaussian distributions. The probability of observing the current pixel value is given as Eq. [4]:

$$P\left(X_t\right)=\sum _{i=1}^k{\omega }_{i,t}\eta \left(X_t,{\mu }_{i,t},{\mathrm{\Sigma }}_{i,t}\right)$$E1

where k is the number of distributions, ${\omega }_{i,t}$ is an estimate of the weight of the ith Gaussian distribution in the mixture at time t, ${\mu }_{i,t}$ is the mean value of the ith Gaussian distribution in the mixture at time t, ${\mathrm{\Sigma }}_{i,t}$ is the covariance matrix of the ith Gaussian distribution in the mixture at time t, and $\eta$ is a Gaussian probability density function:

$$\eta \left(X_t,\mu ,\mathrm{\Sigma }\right)=\frac{1}{2\pi {\left|\mathrm{\Sigma }\right|}^{\frac{1}{2}}}{e}^{-\frac{1}{2}{\left(X_t-\mu \right)}^T{\mathrm{\Sigma }}^{-1}\left(X_t-\mu \right)}.$$E2

The updating rules for the parameters of the adaptive Gaussian mixture model can be found in [5]. After the Gaussian mixture model is established, the foreground pixels (representing the moving objects) can be obtained by applying the Mahalanobis distance:

$$D_i\left(X_t\right)=\sqrt{{\left(X_t-{\mu }_{i,t}\right)}^T{\mathrm{\Sigma }}_{i,t}\left(X_t-{\mu }_{i,t}\right)}.$$E3

The adaptive Gaussian mixture model has been applied for different kinds of applications, such as automatic speech emotion recognition [15], tracking targets on long-range radar systems [16], fast sampling-based motion planning [17], etc.

2.2. Particle filter

The key idea of particle filtering is to approximate the probability distribution by a weighted sample set [18]:

$$\mathbf{S}=\left\{\left.\left({\mathbf{s}}^{\left(n\right)},{\pi }^{\left(n\right)}\right)\right|n=1,\cdots ,N\right\}.$$E4

Each sample consists of an element $\mathbf{s}$ which represents the hypothetical state of the object and a corresponding discrete sampling probability $\pi$ where ${\sum }_{n=1}^N{\pi }^{\left(n\right)}=1$. The evolution of the sample set is calculated by propagating each sample according to a system model. Each element of the set is then weighted in terms of the observations, and N samples are drawn with replacement. The mean state of the object is estimated at each time step by

$$E\left(\mathbf{S}\right)=\sum _{n=1}^N{\pi }^{\left(n\right)}{\mathbf{s}}^{\left(n\right)}.$$E5

Particle filter provides a robust tracking framework.

The particle filter has been successfully applied to many applications. An algorithm to track the vehicle with the adaptively changed scale based on particle filter is propose in [19]. The vehicle guidance with control action computed by a Rao-Blackwellized particle filter is proposed in [20]. The localization of indoor robot based on particle filter with EKF proposal distribution is proposed in [21].

2.3. Kalman filter

The Kalman filter [22] addresses the general problem of trying to estimate the state of a discrete-time controlled process that is governed by the linear stochastic difference equation:

$${\mathbf{x}}_t={\mathbf{Fx}}_{t-1}+{\mathbf{w}}_t$$E6

with measurement equation

$${\mathbf{\phi }}_t={\mathbf{Hx}}_t+{\mathbf{v}}_t$$E7

where ${\mathbf{x}}_t$ is the state vector, $\mathbf{F}$ is the transition matrix, ${\mathbf{\phi }}_t$ is the measurement output, $\mathbf{H}$ is the output matrix, ${\mathbf{w}}_t$ is the process noise, and ${\mathbf{v}}_t$ is the measurement noise. The process noise and measurement noise are assumed to be independent of each other, white, and with normal probability distributions:

$$p\left(\mathbf{\omega }\right)\sim N\left(0,\mathbf{Q}\right)$$E8

$$p\left(\mathbf{v}\right)\sim N\left(0,\mathbf{R}\right)$$E9

where $\mathbf{Q}$ is the process noise covariance matrix and $\mathbf{R}$ is measurement noise covariance matrix. Moreover, ${\widehat{\mathbf{x}}}_t^{-}$ is defined to be the a priori state estimate at step t, and ${\widehat{\mathbf{x}}}_t$ is defined to be the a posteriori state estimate. Then, ${\mathbf{e}}_t^{-}\equiv {\mathbf{x}}_t-{\widehat{\mathbf{x}}}_t^{-}$ is defined to be a priori estimate error, and ${\mathbf{e}}_t\equiv {\mathbf{x}}_t-{\widehat{\mathbf{x}}}_t$ is defined to be the a posteriori estimate error. The a priori estimate error covariance is then

$${\mathbf{P}}_t^{-}=E\left[{\mathbf{e}}_t^{-}{\left({\mathbf{e}}_t^{-}\right)}^T\right]$$E10

and the a posteriori estimate error covariance is

$${\mathbf{P}}_t=E\left[{\mathbf{e}}_t^{-}{\left({\mathbf{e}}_t\right)}^T\right].$$E11

The equations for the Kalman filter fall into two groups: time update (predictor) equations and measurement update (corrector) equations. The time update equations are given as

$${\widehat{\mathbf{x}}}_t^{-}=\mathbf{F}{\widehat{\mathbf{x}}}_{t-1}$$E12

$${\mathbf{P}}_t^{-}={\mathbf{FP}}_{t-1}{\mathbf{F}}^T+\mathbf{Q}.$$E13

The measurement update equations are given as

$${\mathbf{K}}_t={\mathbf{P}}_t^{-}{\mathbf{H}}^T{\left({\mathbf{HP}}_t^{-}{\mathbf{H}}^T+\mathbf{R}\right)}^{-1}$$E14

$${\widehat{\mathbf{x}}}_t={\widehat{\mathbf{x}}}_t^{-}+{\mathbf{K}}_t\left({\mathbf{\phi }}_t-\mathbf{H}{\widehat{\mathbf{x}}}_t^{-}\right)$$E15

$${\mathbf{P}}_t=\left(\mathbf{I}-{\mathbf{K}}_t\mathbf{H}\right){\mathbf{P}}_t^{-}.$$E16

Figure 1 shows the operation architecture of Kalman filter.

Figure 1.

The operation architecture of Kalman filter.

The Kalman filter has been widely applied to time series analysis and statistical modeling problems. This study [23] improves the navigation performance, when refraction starlight is used to compute the position and velocity of a satellite in unscented Kalman filter. An anti-spoofing algorithm based on adaptive Kalman filter for high dynamic positioning in global positioning system is proposed in [24]. In this work [25], the robust Kalman filter is applied to the people occupancy estimation problem, and an iterative algorithm is developed to handle the state-dependent model uncertainties.

3. IVS system design

The IVS system design is presented in this section. Figure 2 shows the operation process of the IVS system. Firstly, the adaptive Gaussian mixture model is applied for constructing the background model to detect moving objects in the video image frame. The moving object appearing in the margin of the video image frame is considered as a new unit (person), and then a new particle filter is established and designated to track the new person. Moreover, the Kalman filter is utilized to correct the position obtaining by the particle filter and to estimate the position during occlusion. After that, the information of the number of persons in the area, the number of persons having been in the area, and hot spots are obtained by analyzing the tracking paths. Finally, the information is fed back to the user by the user interface. Each block of the IVS system design shown in Figure 2 will be illustrated in the following subsections.

Figure 2.

The operation process of the IVS system.

3.1. Gaussian mixture model for detecting new units

By applying the adaptive Gaussian mixture model described in Subsection 2.1, the moving objects can be detected by using the Mahalanobis distance of Eq. (3). Figure 3 illustrates the foreground pixels representing the moving objects obtained by the adaptive Gaussian mixture model. Here, we assume that a new unit (person) will appear only from the border of the monitored place. Therefore, for a moving object detected in the margin of the monitoring video frame, we need to determine that it is a new person or not.

Figure 3.

Foreground pixels representing the moving objects obtained by the adaptive Gaussian mixture model: (a) background, (b) moving object, and (c) foreground pixels.

Figure 4 shows the checking process for determining the object detected in the margin of the monitoring video frame as a new person or not. At first we need to check the size and ratio of the detected object to identify that the detected object is a person or not. If the size and ratio of the detected object are identified as a person, then we have to check that the detected person is new or not. In the case that there is no tracked person in the video frame, the detected person in the margin of the video frame is determined to be a new person. In the case that there has (have) been tracked person(s) in the video frame, we need to calculate the distance(s) between the detected person and tracked person(s) to check that the detected person is new or not. If the distance(s) is (are all) longer than a predefined threshold T_d, the detected person is considered as a new person. If some distances are shorter than T_d, we need to apply Eq. (18), which will be described in the following subsection, to calculate the similarities of color distribution between the detected object and the tracked units with distance shorter than T_d. If all similarities between the detected object and the tracked units with distance shorter than T_d are lower than a predefined threshold T_s, the detected object is determined as a new person.

Figure 4.

The checking process for determining the object detected in the margin of the monitoring video frame as a new person or not.

It is noted that there are several people detection methods [26, 27]. However, in the designed IVS system, we assume that the only moving objects are persons. Hence, we choose a simple method, which is adaptive Gaussian mixture model, to detect that the persons appear from the border of the monitored place for reducing the computational load.

3.2. Particle filter for tracking units

For a new detected person, a new particle filter is established and designated to track the new person. In the design of the particle filter, the target model of target region (the detected person) is the color distribution which is represented by histograms calculated in the HS (Hue, Saturation) space using 8 × 8 bins. A popular measure between two color distributions is the Bhattacharyya coefficient. Considering discrete densities such as two color histograms

$$p={\left\{p^{\left(u\right)}\right\}}_{u=1\dots m},q={\left\{q^{\left(u\right)}\right\}}_{u=1\dots m}$$E17

the Bhattacharyya coefficient is defined as

$$\rho \left[p,q\right]=\sum _{u=1}^m\sqrt{p^{\left(u\right)}{q}^{\left(u\right)}}.$$E18

The larger $\rho$ is, the more similar the two distributions are. For two identical histograms, we obtain $\rho =1$ indicating a perfect match. The target region of the detected person is represented by a rectangle so that a sample is given as

$$\mathit{s}={\left[xy{H}_x{H}_y\right]}^T$$E19

where x and y represent the center location of the rectangle and H_x and H_y are the width and length of the rectangle, respectively, as shown in Figure 5. The sample set is propagated through the application of a dynamic model:

$${\mathit{s}}_t={\mathit{As}}_{t-1}+{\mathit{w}}_{t-1}$$E20

Figure 5.

The target region of the detected person represented by a rectangle.

where A is an 4×4 identity matrix and ${\mathit{w}}_{t-1}$ is a random vector drawn from the noise distribution of the system. Assuming that the target histogram is $q$ and the histogram of the sample ${\mathbf{s}}^{\left(n\right)}$ is $p_{{\mathbf{s}}^{\left(n\right)}}$, the observation probability of each sample is given as

$${\pi }^{\left(n\right)}=\rho \left[p_{{\mathbf{s}}^{\left(n\right)}},q\right].$$E21

The tracking result can be calculated by Eq. (5). During filtering, samples with a high weight may be chosen several times, leading to identical copies, while others with relatively low weights may not be chosen at all. Figure 6 illustrates persons tracked by particle filter.

Figure 6.

Persons tracked particle filter.

Although the particle filter is a robust method for tracking objects, it cannot deal with some special cases. Since we use color distribution for the target model of the particle filter, it may lose tracking when the color of the background is similar to the color of the tracking object. Moreover, if the tracking object is occluded, still the particle filter will lose tracking.

3.3. Kalman filter for correcting and estimating positions

In the IVS system design, the Kalman filter is utilized to correct the position obtained by the particle filter and to estimate the position during occlusion. Here, the uniform linear movement is considered. Hence, the linear stochastic difference equation is given as

$$\left[\begin{array}{l}{\widehat{x}}_t\\ {\widehat{y}}_t\\ {\dot{\widehat{x}}}_t\\ {\dot{\widehat{y}}}_t\end{array}\right]=\left[\begin{array}{l}\begin{array}{ll}1& 0\\ 0& 1\end{array}\begin{array}{ll}1& 0\\ 0& 1\end{array}\\ \begin{array}{ll}0& 0\\ 0& 0\end{array}\begin{array}{ll}1& 0\\ 0& 1\end{array}\end{array}\right]\left[\begin{array}{l}{\widehat{x}}_{t-1}\\ {\widehat{y}}_{t-1}\\ {\dot{\widehat{x}}}_{t-1}\\ {\dot{\widehat{y}}}_{t-1}\end{array}\right]+{\mathit{w}}_{t-1}$$E22

and the measurement equation is given as

$$\left[\begin{array}{l}{\widehat{x}}_t\\ {\widehat{y}}_t\end{array}\right]=\left[\begin{array}{ll}1& 0\\ 0& 1\end{array}\begin{array}{ll}0& 0\\ 0& 0\end{array}\right]\left[\begin{array}{l}{\widehat{x}}_t\\ {\widehat{y}}_t\\ {\dot{\widehat{x}}}_t\\ {\dot{\widehat{y}}}_t\end{array}\right]+{\mathit{v}}_t$$E23

where $\left(\widehat{x},\widehat{y}\right)$ is the estimating position and $\dot{\widehat{x}}$ and $\dot{\widehat{y}}$ represent the estimating speeds on $x$ and $y$ directions, respectively. In the measurement update equation of Eq. (15), the measurement ${\mathit{\phi }}_t={\left[\begin{array}{ll}{x}_t& y_t\end{array}\right]}^T$ is obtained from the tracking result of the particle filter.

For improving the tracking results of particle filter, after propagation by Eq. (20), the estimating speeds ${\dot{\widehat{x}}}_{t-1}$ and ${\dot{\widehat{y}}}_{t-1}$ of the Kalman filter are added to the position of each sample of the particle filter such that

$${\mathit{s}}_t^{\left(n\right)}=\left[\begin{array}{llll}{x}_t^{\left(n\right)}+{\dot{\widehat{x}}}_{t-1},& y_t^{\left(n\right)}+{\dot{\widehat{y}}}_{t-1},& H_x,& H_y\end{array}\right].$$E24

When the tracked object is occluded, the Kalman filter is applied to estimate the position of the occluded object. Therefore, for the case that all the similarities between samples of particle filter and the target are lower than lower than a predefined threshold T_p, the Kalman filter doesn’t use the measurement correction and only takes the filter prediction as object position. Moreover, all samples of the particle filter are uniformly distributed around the estimating position such that the particle filter can retrieve tracking after the object recovering from occluded. However, if the occluded object is stayed in the back of obstacle without moving, the Kalman filter will still lose the tracking.

3.4. User interface for feeding back information

Finally, by analyzing the tracking paths, the information of the tracking result, the number of persons in the area, the number of persons having been in the area, and hot spots are obtained and then fed back to the user through the user interface as shown in Figure 7. Furthermore, several parameters can be adjusted through the user interface for adapting different environments.

Figure 7.

User interface of the IVS system.

4. Experiment results

The experiment is done in the San Shia Campus of the National Taipei University, Taipei, Taiwan. Figure 8 illustrates the operation process of the IVS system. In Figure 8, the up-left frame is the original image; the up-right frame is the background subtraction binary image applying Gaussian mixture model; the left-down frame shows the tracking result by applying particle filter (green rectangle) and the result corrected by Kalman filter (white rectangle), and the right-down frame is the final tracking result. Figure 9 makes a comparison to show the utility of adding the estimating speeds of the Kalman filter to the position of each sample of the particle filter, i.e., to show the utility of Eq. (24). It can be seen that a fast-moving bicycle can be tracked by the particle filter with Eq. (24) as shown in Figure 9(a). However, the fast-moving bicycle cannot be tracked by the particle filter without Eq. (24) as shown in Figure 9(b). Figure 10 makes a comparison to show the utility of applying Kalman filter to estimate the position of the occluded object. It can be seen that the particle filter can retrieve tracking after occlusion with applying Kalman filter to estimate the position of the occluded object as shown in Figure 10(a). However, it can be seen that the particle filter cannot retrieve tracking after occlusion without applying Kalman filter to estimate the position of the occluded object as shown in Figure 10(b). Moreover, it can also be seen in Figure 10(a) that all samples of the particle filter are uniformly distributed around the estimating position such that the particle filter can retrieve tracking after the object recovering from occluded.

Figure 8.

Illustration of the IVS operation process.

Figure 9.

A fast-moving bicycle tracked by the particle filter (a) with Eq. (24) and (b) without Eq. (24).

Figure 10.

(a) Applying Kalman filter to estimate the position of the occluded object and (b) without applying Kalman filter to estimate the position of the occluded object.

5. Conclusion

Based on the particle filter, an IVS system has been designed in this study. Utilizing the Gaussian mixture background model, the moving objects appearing in the margin of the video frame can be detected and considered as a new person. Then, a new particle filter is established and designated to track the new considered person. Moreover, the Kalman filter is applied to correct the tracking result and estimate the position when the tracked person is occluded. By analyzing the tracking paths, the information of the number of persons in the area, the number of persons having been in the area, and hot spots can be obtained. Finally, the information is fed back to the user through the user interface.