Open access peer-reviewed chapter

Particle-Filter-Based Intelligent Video Surveillance System

By Ying-Jen Chen

Submitted: October 19th 2017Reviewed: April 3rd 2018Published: November 5th 2018

DOI: 10.5772/intechopen.76960

Downloaded: 200

Abstract

In this study, an intelligent video surveillance (IVS) system is designed based on the particle filter. The designed IVS system can gather the information of the number of persons in the area and hot spots of the area. At first, the Gaussian mixture background model is utilized to detect moving objects by background subtraction. The moving object appearing in the margin of the video frame is considered as a new person. Then, a new particle filter is assigned to track the new person when it is detected. A particle filter is canceled when the corresponding tracked person leaves the video frame. Moreover, the Kalman filter is utilized to estimate the position of the person when the person is occluded. Information of the number of persons in the area and hot spots is gathered by tracking persons in the video frame. Finally, a user interface is designed to feedback the gathered information to users of the IVS system. By applying the proposed IVS system, the load of security guards can be reduced. Moreover, by hot spot analysis, the business operator can understand customer habits to plan the traffic flow and adjust the product placement for improving customer experience.

Keywords

  • intelligent video surveillance (IVS)
  • Gaussian mixture model
  • particle filter
  • Kalman filter

1. Introduction

Video surveillance systems are often utilized at some specific places such as exits, entrances, parking lots, convenient stores, etc. for management. Traditionally, security guards watch screens of monitored places for surveillance. However, according to statistics, one security guard can only watch four monitors at the same time, and the concentration can last only for 10 minutes such that more than 50% of key information is lost. Moreover, it is not economically worthwhile to hire multiple security guards to monitor one video surveillance system. Therefore, intelligent video surveillance (IVS) systems have become more and more important in commercial sector and have attracted a lot of attention in research area as well [1, 2, 3].

IVS system can be defined as the real-time monitoring of both persistent and transient objects within a specific environment. IVS is also referred to as video analytics (VA) which involves the use of software to automatically detect the objects of specific interest and analyze their behaviors. For finding the objects of interest, it is usually done by detecting the movements or changes in the image that can be achieved by background subtraction technique. To do background subtraction, an effective way to build up the background is the Gaussian mixture model [4, 5]. After the object of interest is detected, the goal is to analyze their behavior that sometimes can be done by tracking them. Usually, particle filters and Kalman filters are employed for the purpose of tracking objects in IVS systems [6, 7, 8, 9, 10].

IVS systems have been applied for different kinds of purposes. Ref. [11] presents an architecture for a perimeter security system dedicated to critical transport infrastructure protection. Ref. [12] addresses a framework for event decision of vision-based intelligent surveillance system based on the fuzzy model. A shape-perceived algorithm using the building block-based matching method is presented in [13] for object tracking of intelligent surveillance applications. A way of unification of flame and smoke detection algorithms by merging the common steps into a single processing flow is proposed in [14] for IVS systems.

The aim of this study is to design a IVS system based on the particle filter. The designed IVS system can gather the information of the number of persons being in the area, the number of persons having been in the area, and hot spots (places of more than usual interest, activity, or popularity) of the area. The Gaussian mixture background model is utilized to detect moving objects by background subtraction in the designed IVS system. The moving object appearing in the margin of the video frame is considered as a new unit (person). When a new person is detected, a new particle filter is established and assigned to track the new person. For saving the computational load, the particle filter is terminated when the corresponding tracked person leaves the video frame. Moreover, the Kalman filters is utilized to estimate the position of the person when the person is occluded. Information of the number of persons in the area (having been in the area) and hot spots is gathered by tracking persons in the video frame. Finally, a user interface is designed to feedback the gathered information to users of the IVS system. By applying the proposed IVS system, the load of security guards can be reduced. Moreover, by hot spot analysis, the business operator can understand customer habits to plan the traffic flow and adjust the product placement for improving customer experience.

2. Preliminary

In this section, three well-known techniques, i.e. adaptive Gaussian mixture model, particle filter, and Kalman filter, are presented for constructing the IVS system.

2.1. Adaptive Gaussian mixture model

The recent history of each pixel, X1X2Xt, is modeled by a mixture of k Gaussian distributions. The probability of observing the current pixel value is given as Eq. [4]:

PXt=i=1kωi,tηXtμi,tΣi,tE1

where k is the number of distributions, ωi,tis an estimate of the weight of the ith Gaussian distribution in the mixture at time t, μi,tis the mean value of the ith Gaussian distribution in the mixture at time t, Σi,tis the covariance matrix of the ith Gaussian distribution in the mixture at time t, and ηis a Gaussian probability density function:

ηXtμΣ=12πΣ12e12XtμTΣ1Xtμ.E2

The updating rules for the parameters of the adaptive Gaussian mixture model can be found in [5]. After the Gaussian mixture model is established, the foreground pixels (representing the moving objects) can be obtained by applying the Mahalanobis distance:

DiXt=Xtμi,tTΣi,tXtμi,t.E3

The adaptive Gaussian mixture model has been applied for different kinds of applications, such as automatic speech emotion recognition [15], tracking targets on long-range radar systems [16], fast sampling-based motion planning [17], etc.

2.2. Particle filter

The key idea of particle filtering is to approximate the probability distribution by a weighted sample set [18]:

S=snπnn=1N.E4

Each sample consists of an element swhich represents the hypothetical state of the object and a corresponding discrete sampling probability πwhere n=1Nπn=1. The evolution of the sample set is calculated by propagating each sample according to a system model. Each element of the set is then weighted in terms of the observations, and N samples are drawn with replacement. The mean state of the object is estimated at each time step by

ES=n=1Nπnsn.E5

Particle filter provides a robust tracking framework.

The particle filter has been successfully applied to many applications. An algorithm to track the vehicle with the adaptively changed scale based on particle filter is propose in [19]. The vehicle guidance with control action computed by a Rao-Blackwellized particle filter is proposed in [20]. The localization of indoor robot based on particle filter with EKF proposal distribution is proposed in [21].

2.3. Kalman filter

The Kalman filter [22] addresses the general problem of trying to estimate the state of a discrete-time controlled process that is governed by the linear stochastic difference equation:

xt=Fxt1+wtE6

with measurement equation

φt=Hxt+vtE7

where xtis the state vector, Fis the transition matrix, φtis the measurement output, His the output matrix, wtis the process noise, and vtis the measurement noise. The process noise and measurement noise are assumed to be independent of each other, white, and with normal probability distributions:

pωN0QE8
pvN0RE9

where Qis the process noise covariance matrix and Ris measurement noise covariance matrix. Moreover, x̂tis defined to be the a priori state estimate at step t, and x̂tis defined to be the a posteriori state estimate. Then, etxtx̂tis defined to be a priori estimate error, and etxtx̂tis defined to be the a posteriori estimate error. The a priori estimate error covariance is then

Pt=EetetTE10

and the a posteriori estimate error covariance is

Pt=EetetT.E11

The equations for the Kalman filter fall into two groups: time update (predictor) equations and measurement update (corrector) equations. The time update equations are given as

x̂t=Fx̂t1E12
Pt=FPt1FT+Q.E13

The measurement update equations are given as

Kt=PtHTHPtHT+R1E14
x̂t=x̂t+KtφtHx̂tE15
Pt=IKtHPt.E16

Figure 1 shows the operation architecture of Kalman filter.

Figure 1.

The operation architecture of Kalman filter.

The Kalman filter has been widely applied to time series analysis and statistical modeling problems. This study [23] improves the navigation performance, when refraction starlight is used to compute the position and velocity of a satellite in unscented Kalman filter. An anti-spoofing algorithm based on adaptive Kalman filter for high dynamic positioning in global positioning system is proposed in [24]. In this work [25], the robust Kalman filter is applied to the people occupancy estimation problem, and an iterative algorithm is developed to handle the state-dependent model uncertainties.

3. IVS system design

The IVS system design is presented in this section. Figure 2 shows the operation process of the IVS system. Firstly, the adaptive Gaussian mixture model is applied for constructing the background model to detect moving objects in the video image frame. The moving object appearing in the margin of the video image frame is considered as a new unit (person), and then a new particle filter is established and designated to track the new person. Moreover, the Kalman filter is utilized to correct the position obtaining by the particle filter and to estimate the position during occlusion. After that, the information of the number of persons in the area, the number of persons having been in the area, and hot spots are obtained by analyzing the tracking paths. Finally, the information is fed back to the user by the user interface. Each block of the IVS system design shown in Figure 2 will be illustrated in the following subsections.

Figure 2.

The operation process of the IVS system.

3.1. Gaussian mixture model for detecting new units

By applying the adaptive Gaussian mixture model described in Subsection 2.1, the moving objects can be detected by using the Mahalanobis distance of Eq. (3). Figure 3 illustrates the foreground pixels representing the moving objects obtained by the adaptive Gaussian mixture model. Here, we assume that a new unit (person) will appear only from the border of the monitored place. Therefore, for a moving object detected in the margin of the monitoring video frame, we need to determine that it is a new person or not.

Figure 3.

Foreground pixels representing the moving objects obtained by the adaptive Gaussian mixture model: (a) background, (b) moving object, and (c) foreground pixels.

Figure 4 shows the checking process for determining the object detected in the margin of the monitoring video frame as a new person or not. At first we need to check the size and ratio of the detected object to identify that the detected object is a person or not. If the size and ratio of the detected object are identified as a person, then we have to check that the detected person is new or not. In the case that there is no tracked person in the video frame, the detected person in the margin of the video frame is determined to be a new person. In the case that there has (have) been tracked person(s) in the video frame, we need to calculate the distance(s) between the detected person and tracked person(s) to check that the detected person is new or not. If the distance(s) is (are all) longer than a predefined threshold Td, the detected person is considered as a new person. If some distances are shorter than Td, we need to apply Eq. (18), which will be described in the following subsection, to calculate the similarities of color distribution between the detected object and the tracked units with distance shorter than Td. If all similarities between the detected object and the tracked units with distance shorter than Td are lower than a predefined threshold Ts, the detected object is determined as a new person.

Figure 4.

The checking process for determining the object detected in the margin of the monitoring video frame as a new person or not.

It is noted that there are several people detection methods [26, 27]. However, in the designed IVS system, we assume that the only moving objects are persons. Hence, we choose a simple method, which is adaptive Gaussian mixture model, to detect that the persons appear from the border of the monitored place for reducing the computational load.

3.2. Particle filter for tracking units

For a new detected person, a new particle filter is established and designated to track the new person. In the design of the particle filter, the target model of target region (the detected person) is the color distribution which is represented by histograms calculated in the HS (Hue, Saturation) space using 8 × 8 bins. A popular measure between two color distributions is the Bhattacharyya coefficient. Considering discrete densities such as two color histograms

p=puu=1m,q=quu=1mE17

the Bhattacharyya coefficient is defined as

ρpq=u=1mpuqu.E18

The larger ρis, the more similar the two distributions are. For two identical histograms, we obtain ρ=1indicating a perfect match. The target region of the detected person is represented by a rectangle so that a sample is given as

s=xyHxHyTE19

where x and y represent the center location of the rectangle and Hx and Hy are the width and length of the rectangle, respectively, as shown in Figure 5. The sample set is propagated through the application of a dynamic model:

st=Ast1+wt1E20

Figure 5.

The target region of the detected person represented by a rectangle.

where A is an 4×4 identity matrix and wt1is a random vector drawn from the noise distribution of the system. Assuming that the target histogram is qand the histogram of the sample snis psn, the observation probability of each sample is given as

πn=ρpsnq.E21

The tracking result can be calculated by Eq. (5). During filtering, samples with a high weight may be chosen several times, leading to identical copies, while others with relatively low weights may not be chosen at all. Figure 6 illustrates persons tracked by particle filter.

Figure 6.

Persons tracked particle filter.

Although the particle filter is a robust method for tracking objects, it cannot deal with some special cases. Since we use color distribution for the target model of the particle filter, it may lose tracking when the color of the background is similar to the color of the tracking object. Moreover, if the tracking object is occluded, still the particle filter will lose tracking.

3.3. Kalman filter for correcting and estimating positions

In the IVS system design, the Kalman filter is utilized to correct the position obtained by the particle filter and to estimate the position during occlusion. Here, the uniform linear movement is considered. Hence, the linear stochastic difference equation is given as

x̂tŷtx̂̇tŷ̇t=1001100100001001x̂t1ŷt1x̂̇t1ŷ̇t1+wt1E22

and the measurement equation is given as

x̂tŷt=10010000x̂tŷtx̂̇tŷ̇t+vtE23

where x̂ŷis the estimating position and x̂̇and ŷ̇represent the estimating speeds on xand ydirections, respectively. In the measurement update equation of Eq. (15), the measurement φt=xtytTis obtained from the tracking result of the particle filter.

For improving the tracking results of particle filter, after propagation by Eq. (20), the estimating speeds x̂̇t1and ŷ̇t1of the Kalman filter are added to the position of each sample of the particle filter such that

stn=xtn+x̂̇t1,ytn+ŷ̇t1,Hx,Hy.E24

When the tracked object is occluded, the Kalman filter is applied to estimate the position of the occluded object. Therefore, for the case that all the similarities between samples of particle filter and the target are lower than lower than a predefined threshold Tp, the Kalman filter doesn’t use the measurement correction and only takes the filter prediction as object position. Moreover, all samples of the particle filter are uniformly distributed around the estimating position such that the particle filter can retrieve tracking after the object recovering from occluded. However, if the occluded object is stayed in the back of obstacle without moving, the Kalman filter will still lose the tracking.

3.4. User interface for feeding back information

Finally, by analyzing the tracking paths, the information of the tracking result, the number of persons in the area, the number of persons having been in the area, and hot spots are obtained and then fed back to the user through the user interface as shown in Figure 7. Furthermore, several parameters can be adjusted through the user interface for adapting different environments.

Figure 7.

User interface of the IVS system.

4. Experiment results

The experiment is done in the San Shia Campus of the National Taipei University, Taipei, Taiwan. Figure 8 illustrates the operation process of the IVS system. In Figure 8, the up-left frame is the original image; the up-right frame is the background subtraction binary image applying Gaussian mixture model; the left-down frame shows the tracking result by applying particle filter (green rectangle) and the result corrected by Kalman filter (white rectangle), and the right-down frame is the final tracking result. Figure 9 makes a comparison to show the utility of adding the estimating speeds of the Kalman filter to the position of each sample of the particle filter, i.e., to show the utility of Eq. (24). It can be seen that a fast-moving bicycle can be tracked by the particle filter with Eq. (24) as shown in Figure 9(a). However, the fast-moving bicycle cannot be tracked by the particle filter without Eq. (24) as shown in Figure 9(b). Figure 10 makes a comparison to show the utility of applying Kalman filter to estimate the position of the occluded object. It can be seen that the particle filter can retrieve tracking after occlusion with applying Kalman filter to estimate the position of the occluded object as shown in Figure 10(a). However, it can be seen that the particle filter cannot retrieve tracking after occlusion without applying Kalman filter to estimate the position of the occluded object as shown in Figure 10(b). Moreover, it can also be seen in Figure 10(a) that all samples of the particle filter are uniformly distributed around the estimating position such that the particle filter can retrieve tracking after the object recovering from occluded.

Figure 8.

Illustration of the IVS operation process.

Figure 9.

A fast-moving bicycle tracked by the particle filter (a) with Eq. (24) and (b) without Eq. (24).

Figure 10.

(a) Applying Kalman filter to estimate the position of the occluded object and (b) without applying Kalman filter to estimate the position of the occluded object.

5. Conclusion

Based on the particle filter, an IVS system has been designed in this study. Utilizing the Gaussian mixture background model, the moving objects appearing in the margin of the video frame can be detected and considered as a new person. Then, a new particle filter is established and designated to track the new considered person. Moreover, the Kalman filter is applied to correct the tracking result and estimate the position when the tracked person is occluded. By analyzing the tracking paths, the information of the number of persons in the area, the number of persons having been in the area, and hot spots can be obtained. Finally, the information is fed back to the user through the user interface.

How to cite and reference

Link to this chapter Copy to clipboard

Cite this chapter Copy to clipboard

Ying-Jen Chen (November 5th 2018). Particle-Filter-Based Intelligent Video Surveillance System, Intelligent Video Surveillance, António J. R. Neves, IntechOpen, DOI: 10.5772/intechopen.76960. Available from:

chapter statistics

200total chapter downloads

More statistics for editors and authors

Login to your personal dashboard for more detailed statistics on your publications.

Access personal reporting

Related Content

This Book

Next chapter

Video Surveillance-Based Intelligent Traffic Management in Smart Cities

By Fozia Mehboob, Muhammad Abbas, Abdul Rauf, Shoab A. Khan and Richard Jiang

Related Book

First chapter

A Personal Robot as an Improvement to the Customers’ In- Store Experience

By Joana Santos, Daniel Campos, Fábio Duarte, Filipe Pereira, Inês Domingues, Joana Santos, João Leão, José Xavier, Luís de Matos, Manuel Camarneiro, Marcelo Penas, Maria Miranda, Ricardo Morais, Ricardo Silva and Tiago Esteves

We are IntechOpen, the world's leading publisher of Open Access books. Built by scientists, for scientists. Our readership spans scientists, professors, researchers, librarians, and students, as well as business professionals. We share our knowledge and peer-reveiwed research papers with libraries, scientific and engineering societies, and also work with corporate R&D departments and government entities.

More About Us