Detecting, Tracking, and Identifying Airborne Threats with Netted Sensor Fence

1. Introduction

Today’s technological advances allow for the development of unmanned aerial systems, fixed-wing aircraft, that are small enough and fly low enough to elude conventional radar detection. Such aircraft could carry out chemical, biological, or nuclear attacks, or they could be employed to smuggle drugs or illegal immigrants across the border.

This chapter describes a low cost, low power (potentially disposable) methodology for performing key 24/7 sentry functions to protect critical civilian and military infrastructure from airborne threats. The methodology is based on joint multi-sensor exploitation technology by designing and developing a forward-based fence that contains a mix of various low cost, low power, netted sensors including a simple radar, acoustic microphones and optical (Infrared and visible) cameras to detect, track and discriminate potential airborne targets. An in-depth understanding of candidate target signature phenomenologies is developed through theoretical, numerical assessments and proof-of-concept field experiments. An integrated (over sensor modality) detection, tracking and discrimination process is developed which forms the basis of the fence’s friend/foe sentry capability and ability to provide accurate/timely intercept information. An experimental prototype end-to-end proof of concept system with deployable software, hardware and connectivity has also been developed to perform the field demonstration.

2. System concept and design

The primary detection component in the system is a radar fence. The radar fence is designed to detect approaching targets and provide a cue to the acoustic and infrared sensors that perform the discrimination task. The radar fence consists of multiple, low power (10 Watts), non-scanning (for low cost and complexity), UHF, pulse-Doppler radars (to estimate target speed, range and eliminate birds and ground clutter), with a radar-to-radar separation of approximately 5 km (Figure 1). Each radar operates with a different carrier frequency (to avoid crosstalk between radars) and has a beamwidth that is broad in both azimuth (so that the number of radars can be kept small) and elevation (to detect both high and low-flying targets). The radars measure target range and radial speed five times per second and report these values to a central processing station that cues the acoustic and infrared sensors (if a target report is issued), and then fuses the reports from all sensors (radars, acoustic, etc.) to form a target track and alert rear-area weapons systems or potential interceptors so that defensive action can be taken. A complete description of the radar parameters and detection characteristics is contained in Tables 1.

Figure 1.

A conceptual illustration of the netted sensor fence

Acoustic microphone arrays are used as the second sensor modality in the system to detect broadband acoustic emissions from approaching targets. Acoustic sensors are non-line-of-sight, passive, low-cost, and portable sensors that can be effectively deployed in wide areas. Primary objectives of acoustic sensors in this sensor fence system are: a) to provide target direction of arrival (DOA) estimates that will then be fused with radar measurements to form a target track; b) to provide a means for target identification and classification; and c) to mitigate false alarms. The system is designed to contain several equally spaced, diagonally-arranged microphone arrays.

The third sensor modality in the fence is an optical system which is cued by the radar and/or acoustic sensors and slews in angle to acquire track and identify the potential airborne threat. The system is designed to contain an uncooled infrared detector sensitive to the 8-12 um waveband to provide day and night time operation. The uncooled IR detector array uses only several Watts of power. A boresighted visible camera is also used for improved target resolution during the daytime. Visible cameras are inexpensive and have improved resolution compared to the infrared detector array.

3. Detection

3.1. Radar detection

Assessments of radar detectability and detection range are accomplished via numerical simulation. In Table 1, the range at which the probability of detection equals 0.9 was calculated assuming a non-fluctuating target (this is why the UHF frequency band was chosen), with the signal-to-noise ratio at target range R calculated from,

$$\frac{E}{N_o}=\frac{P{G}^2\sigma {\lambda }^2\tau }{{(4\pi )}^3{R}^{4}k{T}_{o}FL}$$E1

for ranges where the target is completely uneclipsed (eclipsing occurs if part or all of the return from the target is received while the transmit pulse is still on), which implies R > cT/2, where c = speed of light and T = duration of the uncompressed radar pulse. Also, P = average transmitted power, G = antenna gain at the target (this varies with target location), = target radar cross section, = coherent integration time, = wavelength, k = Boltzmann’s constant, T0 = 290K, L = loss, and F = receiver noise figure. When the target is partially eclipsed, the signal-to-noise ratio in Equation (1) is reduced by (2R/cT). The pulse repetition frequency was chosen so that the radar is unambiguous (unambiguous range = c/2PRF, unambiguous speed = PRF/2) in both target range and speed for all targets of interest. Plots of E/N0 (including antenna patterns) for a target with a 3 m² radar cross section flying at 100 m altitude and crossing the fence directly above radar (X = 0) and at 2 km from radar are shown in Figure 2. These values were used with the standard curves (Skolnik, 1990) of probability of detection per look versus signal-to-noise ratio to calculate the probability, Pd(n), that the target is detected at the end of the coherent integration interval at a time when the target is at a range Rn. Finally, the cumulative probability that the target has been detected by the time it reaches range Rm is

$$P_c(m)=1-\underset{n=0}{\overset{m}{\Pi }}\left[1-P_d(m)\right]$$E2

where R₀ (n = 0) corresponds to the range at which the target comes over the horizon. The range Rm where Pc(m) = 0.9, for the case when X = 2 km, is the value cited in Table 1.

Figure 2.

SNR for Low-Flying Target with a Radar Cross Section = 3 m²

Type	Pulse/Doppler
Frequency	UHF
Scan	None (Radar Stares Forward)
Antenna	Broadbeam, Gain ~ 3 dB
Polarization	Circular
Average Power	10 W
Duty Factor	10%
Pulse Repetition Frequency	10 kHz
Coherent Integration Time	0.2 sec. (5 Hz Update Rate)
Unambiguous Range	75 km
Unambiguous Speed	750 m/sec
Range Resolution	150 m
Speed Resolution	1.9 m/sec
Minimum Detection Range	25 km
Range for Probability Detection = 0.9	7 km (Small Target, 15 km (Large Aircraft)
Noise-Induced False Alarm Rate	1 Per Month
Clutter/Bird Cancellation Approach	Blank Lowest few Doppler Bins
Quantities Measured	Target Range and Speed (Azimuth and Height Estimated)
Target Discrimination	Based on Speed Only. Need Other Sensor Types

Table 1.

Individual Radar Properties

3.2. Acoustic detection

Acoustic detection assessments are primarily accomplished by performing several field measurements of different types of aircraft to obtain information on target acoustic detection and signature characteristics. Although many aircraft acoustic data are available in literature (Ferguson & Lo 2000; Pham & Srour 2004), for the purpose of developing and testing multi-modal sensor fence detection, tracking and classification algorithms, it is critical to simultaneously obtain data measurements from all the fence sensors at the same time. A typical experiment layout and the 8-element acoustic microphone array (with equal element spacing of 0.5m) are shown in Figure 3. The sensor suite which includes acoustic array and IR/visible cameras are positioned near the end of an airport runway. The test aircraft are flying at a flight test matrix with multiple combinations of altitude and engine RPM. GPS data recording systems are mounted on the aircraft so the ground truth information can be retrieved and later can be used for target validation after the flight.

Figure 4 shows a spectrogram of a measured acoustic data showing a rich array of features corresponding to two crop dusters flying above the sensor array. Strong Doppler shifted harmonic structures caused by aircraft’ engine noises are observed. The parabolic energy distortion (in the time period between 60-80 seconds) in the spectrum is caused by the multipath ground reflection interference when the target is passing directly above the sensor array. The spreading width of the parabolas is directly related to the speed of the targets.

Figure 3.

Field experiment set up (right) and the acoustic microphone array (left)

Figure 4.

Spectrogram of two crop dusters flying above the sensor array

3.3. IR detection

The development of the optical subsystem started with an evaluation of IR detectors, wavebands and target signatures in order to calculate the effective range of these IR systems, design and optimize the optical sensing system performance. This work is followed by designing an experimental set-up and experiments to collect images of airborne targets.

Figure 5.

Diagram of the Optical System and the Target Environment

Detector received power for an optical system (Figure 5) whose diameter is D, pixel angular field-of-view is δϕ, target range is R and target spectral radiance is J(λ) W/Sr/μm is calculated below. The backgrounds are clouds and atmospheric aerosol scattering. The magnitude of these backgrounds can be calculated by MODTRAN code runs for any specific defined atmospheric condition

The signal power received from the source by the detector is given by

$$P_S={\displaystyle {\int }_{{\lambda }_1}^{{\lambda }_2}{T}_{optics}{T}_{atmosphere}(\lambda )J(\lambda )\ast \frac{\pi D^2}{4R^2}d\lambda }=\frac{\pi D^2}{4R^2}{T}_{optics}{\displaystyle {\int }_{{\lambda }_1}^{{\lambda }_2}{T}_{atmosphere}(\lambda )J(\lambda )\ast d\lambda }$$E3

Where T_optics is the transmittance (or transmission coefficient) of the optical system and T_atmosphere is the transmission of the atmosphere along the path from the source to the sensor. Generally, with proper optics design, the transmission of the optics is a constant independent of wavelength, but the transmission of the atmosphere is a function of wavelength. The integration is performed over the spectral bandpass of the optical system. This derivation assumes that the source is incident on a single pixel. The expression can be approximately corrected for multiple pixels on the target by dividing by the number of pixels on the target, n.

The power on each detector pixel from the background scene in the field-of-view of each pixel (see figure) is given by

$$P_B={(IFOV)}^2\frac{\pi D^2}{4}{T}_{optics}{\displaystyle {\int }_{{\lambda }_1}^{{\lambda }_2}{T}_{atmosphere}(\lambda )N_{background}(\lambda )d\lambda }$$E4

where, IFOV is the instantaneous angular field of view of each square pixel (radians) and T_atmosphere * N_background is the background spectral radiance W/(Sr cm2 μm) at the aperture. The term T_atmosphere * N_background can be obtained from the MODTRAN atmospheric modeling code and already contains the effects of atmospheric transmission between the noise sources (background scene and/or aerosol scattering) and the receiver aperture. Thus optic transmission term can be dropped from the expression. Note: R and D have units of centimeters, and λ has units of μm.

The circuit power is proportional to the square of the current so, the power signal to noise ratio is given by [(Kingdton, 1978)

$${\left(\frac{S}{N}\right)}_P=\frac{i_S^2}{{\left(i_N^2\right)}_{Mean}}=\frac{{\eta }^2{e}^2{P}_S^2}{{\left(h\nu \right)}^2}\ast \frac{h\nu }{2\eta e^2\left(P_S+P_{B\text{'}}\right)}=\frac{\eta P_S^2}{2h\nu B\left(P_S+P_{B\text{'}}\right)}$$E5

For the typical infrared case of the signal power being much less than the background power P_s << P_B’, (background limited detection) then

$${\left(\frac{S}{N}\right)}_P=\frac{\eta P_S^2}{2h\nu B{P}_{B\text{'}}}$$E6

[Note: the background power, P_B’, is due to all non-signal sources, including (a) nearby detector cryogenic Dewar radiation, (b) optics radiation, (c) radiation from the field-of-view limiting shrouds and (d) other nearby sources]. We can invoke the “pure detection” criterion (Howe, 1996) that if the signal to noise ratio with the target in a pixel minus the signal to noise ratio with the target not in the pixel (SNR= SNR_target-SNR_no-target)\i) is greater than 5, the target will be detected against a cluttered background. The range of detection of an airborne target at various altitudes can then be calculated and the results are shown in Figure 6.

Figure 6.

Attitude and ground range for the detection of airborne targets of various values of target radiance

This calculation is for an uncooled 8-12 μm detector with a noise equivalent temperature difference (NETD) of 0.1 degrees and shows the altitude and ground range for the detection of airborne targets of various values of target radiance. Typical aircraft have spectral radiance of about 100 W/Sr/μm so targets can be detected at ranges in excess of 5 km even if they are at low altitudes. (Note a NEI of 0.1 degrees corresponds to a NEI of about 4.76x10-12 W/cm2 for a 1.9 milliradian IFOV.)

The experimental set-up to collect visible and IR field data is shown in Figure 7. The visible and IR cameras are boresighted with each other to take simultaneous visible and IR images of the target. A frame grabber and computer is used to collect the data and display the images side-by-side. The IR camera is a BAE 320x280 micro-bolometer array which is sensitive over the wavelength range of 7 to 14 μm. The FOV of the array is 7.7x10.3 degrees and the IFOV of each pixel is about 0.56 milliradians. It has a NETD of aboutabut 0.08 Celsius and consumes only 6 Watts of power.

Typical images at short range are shown in Figure 8 at the lower left for a Cessna and the middle for a crop duster. The detection of a twin engine Cessna is at 6.1 km shown at the lower right.

Figure 7.

IR/visible data collection set up.

Figure 8.

The detection of single engine Cessna (left), a crop duster (middle) and a twin engine Cessna at 6.1 km (right)

4. Tracking

A major challenge of the project has been the design and development of sensor fusion techniques which combine data from multi-modality and multi-sensor nodes to achieve improved accuracy in target detection, tracking, and classification. The system employs a Kalman filter (Anderson & Moore, 1979; Grewal & Andrews, 2001) to establish multi-target tracks using radar and acoustic measurements as input. The tracker is designed to handle multiple targets and false reports. It is also designed to have data input flexibilities such as allowing input data measurements from each sensor (radar and acoustic) that are not time coordinated. The tracker must allow that some tracks it creates may be based on false reports, and therefore these tracks must be dropped if they behave erratically or do not associate with further detections at later times. Tracks can be promoted or demoted by evaluating cumulative properties of a score that was originally assigned to the initial tracks. Constant velocity with additive white noise acceleration is introduced in the plant noise assumption. Since the range of radar measurements is on the order of ~10 km, it is adequate to use a flat earth model when calculating tracker updates. The objective of the tracker is to fuse asynchronous radar and acoustic data to predict kinematic properties such as the location, the speed, the heading, and the flight trajectory of the target. This prediction is then used to automatically aim a camera to the predicated point and photograph the target.

4.1. Tracker requirements, functional capability, and restrictions

For an operational system the tracker can be designed to sit at a central location (a/k central node). The remote nodes transmit detections of an aircraft target (time, range, azimuth, elevation) to the central node. Each transmitted signal packet contains all the detections (there may be none) accumulated since the previous transmission. The data is time-stamped at each remote site with the time at which the signal arrives to the sensor. For such multi-modal sensor system the tracker must be designed to accommodate asynchronous data streams from multiple remote sites. The tracker is also required to allow for an arbitrary number of remote sites, remote site dropout during a run, false reports in the data, the correction for propagation time delay of the acoustic signals, and the prediction of the target location in the future. The tracker must also be operating in real time.

These requirements in turn imply that the tracker must have the following functional capabilities:

• Track initialization, i.e., the capability to start a track from the data

• Rejection gates to eliminate false reports

• Data association, i.e., the capability to associate a new report with a track for track updating

• Kalman filter remote site data fusion for minimum variance state estimate

• Dynamic status tables identifying which sites are active or inactive

• Time management and coordination logic to allow for asynchronous data streams and propagation time delays of acoustic data.

The development of such a complex tracker involves considerable effort. In order to keep the tracker development effort within manageable bounds, several key restrictions and simplifications were imposed on the tracker capability. These include a single target track, and a flat earth model assumption

4.2. Tracker processing and time delay correction

The multi-modal kinematic tracker employs a Kalman filter to update the data measurement. At each cycle, the tracker corrects the acoustic data timestamp for propagation delay, attempts to initialize a track if none exists, performs the association function, discards false reports, and performs the Kalman filter update of the time sequenced data.

Assume at time t the state vector of the track is a six dimensional vector of target position and velocity $$(\dot{x}(t),\dot{y}(t),\dot{z}(t))$$ in Cartesian coordinates relative to an east-north-up topocentric coordinate system with origin at the central node,$$x\left(t\right)={\left[\begin{array}{cccccc}x\left(t\right)& y\left(t\right)& z\left(t\right)& \dot{x}\left(t\right)& \dot{y}\left(t\right)& \dot{z}\left(t\right)\end{array}\right]}^T$$. Let $$Z_i\left(t\right)$$ be the measurement vector from remote node i at time t and $$R_i$$ the corresponding covariance matrix. For the radar range measurement, $$Z_i\left(t\right)=\left[r_i\left(t\right)\right],R_i=\left[{\sigma }_R^2\right]$$, where

$$r_i\left(t\right)$$is the range measurement value from remote node i, and $${\sigma }_R$$ is the standard deviation of the covariance matrix. In parallel, for the acoustic angle measurement$$Z_i\left(t\right)=\left[\begin{array}{c}A{z}_i\left(t\right)\\ E{l}_i\left(t\right)\end{array}\right],R_i=\left[\begin{array}{cc}{\sigma }_{Az}^2& 0\\ 0& {\sigma }_{El}^0\end{array}\right]$$. Where $$A{z}_i\left(t\right)$$ is the azimuth angle and $$E{l}_i\left(t\right)$$ is the elevation angle. Due to significant propagation time differences between the radar and the acoustic data (e.g. at a range of 5 km the acoustic sensor data corresponds to a point on the flight path that is 5000/340 = 14.7 sec in the past, while the radar data is virtually instantaneous), it is necessary to correct the timestamp of the acoustic sensor data to correspond to the time the signal left the target.

Assume at time t, $$r=\left[\begin{array}{ccc}x& y& z\end{array}\right]$$is the position and $$\dot{r}=\left[\begin{array}{ccc}\dot{x}& \dot{y}& \dot{z}\end{array}\right]$$ is the velocity of the target. The acoustic measurement received at time t should correspond to an earlier time $$t_0$$at which the position of the target is$$r_0$$, as shown in Figure 9. For a target traveling at a constant speed v (0<v<c, where c isthe spend of sound in the air) between locations of $$r_0$$and$$r$$, through simple geometrical derivations, it is easy to obtain,

$$r_0=\frac{c\sqrt{a^2+\left(c^2-v^2\right)r^2}-ac}{c^2-v^2},0<v<c$$E7

Where $$v=\left|\dot{r}\right|=\sqrt{\dot{r}{\dot{r}}^T}=\sqrt{{\dot{x}}^2+{\dot{y}}^2+{\dot{z}}^2}=\left|{\dot{r}}_0\right|=v_0$$ and $$a=r{\dot{r}}^T=x\dot{x}+y\dot{y}+z\dot{z}$$

Figure 9.

Diagram of acoustic travel time correction

4.3. Tracker initialization

The Kalman filter is a recursive algorithm which starts with an estimated state vector based on past data and updates it with new data. In order to start the algorithm it is necessary to have an initial estimate of the state. This is done by generating an initial guess of a constant-velocity path via a least-squares fit to the batch of collected data. This method allows us to separate a reasonable collection of false alarms from real data without relying on prior knowledge of the target’s position. The least-square fit minimizes the following objective function for a measurement Z_i that has an expected value of E_i,

$$Q={\displaystyle \sum _i\frac{{(E_i-Z_i)}^2}{{\sigma }_i{}^2}}$$E8

This formula is modified slightly for acoustic measurements in that it combines azimuth and elevation measurements into a single term using the law of cosines to calculate the great circle distance on a unit circle:

$$E_i-Z_i=\arccos (\sin (E{l}_i)\sin (E_{El})+\cos (E{l}_i)\cos (E_{El})\cos (E_{Az}-A{z}_i))$$E9

While minimizing Q is the core function of the initialization tracker, it is not nearly adequate for consistently producing reliable tracks. Steps must be taken to both minimize the number of false tracks and maximize the chances of generating acceptable tracks when real targets are present. These steps include windowing, setting a minimum number of data points, placing bounds on target speed for acceptable tracks, discarding outliers, selecting the best of multiple independently generated tracks, and setting maximum values for Q such that a track is still valid.

4.4. Data association

Data arrives to the tracker from various remote nodes. The data from each node consists of either radar range data alone, acoustic sensor azimuth and elevation data alone, or both radar and acoustic sensor data. At each time point the acoustic sensor outputs only one detection (or possibly none) and this probably corresponds to the loudest source. By contrast the radar can output any number of detections at each time point, as many as cross the detection threshold. The tracker must allow that many of these detections could be false alarms, arising from random noise and clutter. They may also be detections of other real targets within the range of the sensors. The association process is an attempt to weed out the irrelevant detections so as not to corrupt the updating of the track.

This association is performed by comparing each new measurement to some previously generated expectation of the target’s location. This previous expectation is generally the result of the most recent tracker output. The comparison between the measurement and the expected target location begins by creating an expected state vector for time t where t is the time of the measurement, and then convert the expected state vector to the measurement vector. The differences between the measurement and the expected value is denoted as, $$\epsilon =Z\left(t\right)-\overline{Z}\left(t\right)$$, where $$Z\left(t\right)$$ is the measurement at time t, and $$\overline{Z}\left(t\right)$$ is the expected measurement converted from the expected state vector. A set of typical range, azimuth and elevation gate$$G_R$$, $$G_{ang}$$can be defined based on sensor properties. The error $$\epsilon$$must be within those gates in order for a data measurement to associate.

4.5. Kalman filter data fusion

Kalman filter is employed for data update and predictions. It is assumed that the flight path is a constant velocity plus a Gaussian white noise acceleration term (plant noise). This implies that the state obeys the linear difference equation

$$x\left(t+\tau \right)=\Phi \left(\tau \right)x\left(t\right)+w\left(t,t+\tau \right)$$E10

Where $$\Phi \left(\tau \right)$$ is the transition matrix,

$$\Phi \left(\tau \right)=\left[\begin{array}{cccccc}1& 0& 0& \tau & 0& 0\\ 0& 1& 0& 0& \tau & 0\\ 0& 0& 1& 0& 0& \tau \\ 0& 0& 0& 1& 0& 0\\ 0& 0& 0& 0& 1& 0\\ 0& 0& 0& 0& 0& 1\end{array}\right]$$E11

and $$w\left(t,t+\tau \right)$$ is a random zero mean Gaussian plant noise process with covariance matrix,

$$Q\left(\tau \right)=\mathrm{cov}w\left(t,t+\tau \right)=\left[\begin{array}{cccccc}{q}_x^2{\tau }^3/3& 0& 0& q_x^2{\tau }^2/2& 0& 0\\ 0& q_y^2{\tau }^3/3& 0& 0& q_y^2{\tau }^2/2& 0\\ 0& 0& q_z^2{\tau }^3/3& 0& 0& q_z^2{\tau }^2/2\\ q_x^2{\tau }^2/2& 0& 0& q_x^2\tau & 0& 0\\ 0& q_y^2{\tau }^2/2& 0& 0& q_y^2\tau & 0\\ 0& 0& q_z^2{\tau }^2/2& 0& 0& q_z^2\tau \end{array}\right]$$E12

Where $$q_x,q_y,q_z$$ are plant noise intensities in x, y, z directions.

Let $$\widehat{x}\left(t|t_0\right)$$ be the Kalman filter optimum estimate of the state $$x\left(t\right)$$ at time t based on data taken up to and including time$$t_0$$, $$P\left(t|t_0\right)$$is the covariance matrix of the errors in this estimate of the state, the recursive Kalman filter estimate of the updated tracker from time $$t_0$$ to time $$t=t_0+\tau$$ is thus written,

$$\begin{array}{c}\widehat{x}\left(t|t_0\right)=\Phi \left(t-t_0\right)\widehat{x}\left(t_0\right)\\ P\left(t|t_0\right)=\Phi \left(t-t_0\right)P\left(t_0\right){\Phi }^T\left(t-t_0\right)+Q\left(t-t_0\right)\end{array}$$E13

Transform the predicted track $$\widehat{x}\left(t|t_0\right)$$ to the measurement the measurement variables$${\widehat{Z}}_i\left(t|t_0\right)$$, Form the innovation, which is the difference between the actual measurement $$Z_i\left(t\right)$$ and the predicted measurement$${\widehat{Z}}_i\left(t|t_0\right)$$,$${\tilde{Z}}_i\left(t|t_0\right)=Z_i\left(t\right)-{\widehat{Z}}_i\left(t|t_0\right)$$.

The covariance matrix of the innovation is

$$S_i\left(t|t_0\right)=H_i\left(t\right)P\left(t|t_0\right)H_i^T\left(t\right)+R_i$$E14

Where $$H_i\left(t\right)$$ is the matrix of partial derivatives of the measurement variables of node i with respect to the state variables at time t. Thus for the radar measurement

$$H_i\left(t\right)=\left[\begin{array}{cccccc}\frac{\partial R_i\left(t\right)}{\partial x}& \frac{\partial R_i\left(t\right)}{\partial y}& \frac{\partial R_i\left(t\right)}{\partial z}& 0& 0& 0\end{array}\right]$$E15

and for the acoustic measurement

$$H_i\left(t\right)=\left[\begin{array}{cccccc}\frac{\partial A{z}_i\left(t\right)}{\partial x}& \frac{\partial A{z}_i\left(t\right)}{\partial y}& \frac{\partial A{z}_i\left(t\right)}{\partial z}& 0& 0& 0\\ \frac{\partial E{l}_i\left(t\right)}{\partial x}& \frac{\partial E{l}_i\left(t\right)}{\partial y}& \frac{\partial E{l}_i\left(t\right)}{\partial z}& 0& 0& 0\end{array}\right]$$E16

At this point the tracker makes a test to assure that the innovation is consistent with its covariance matrix. The Mahalonobis distance d between the measurement and its predicted value is

$$d={\tilde{Z}}_i^T\left(t|t_0\right)S_i^{-1}\left(t|t_0\right){\tilde{Z}}_i\left(t|t_0\right)$$E17

Where d is a chi-squared distributed variable with $$n=1$$ (radar) or $$n=2$$ (acoustic sensor) degrees of freedom. If$$d\le T$$, for a given threshold T, then the data is accepted. If $$d>T$$ then the Mahalonobis distance is too large and the data is rejected. The threshold has been chosen at the 10% level, i.e., the probability that d is larger than T is one in ten. The Kalman filter then updates the state and the associated covariance matrix,

$$\begin{array}{l}\widehat{x}\left(t\right)=\widehat{x}\left(t|t_0\right)+K_i\left(t|t_0\right){\tilde{Z}}_i\left(t|t_0\right)\\ P\left(t\right)=\left(I-K_i\left(t|t_0\right)H_i\left(t\right)\right)P\left(t|t_0\right)\end{array}$$E18

Where $$K_i\left(t|t_0\right)$$is the Kalman gain matrix given by,

$$K_i\left(t|t_0\right)=P\left(t|t_0\right)H_i^T\left(t\right)S_i^{-1}\left(t|t_0\right)$$E19

5. Classification

Target classification is performed as part of the sensor fusion. Once the target track is established from fusing the radar range detections and the acoustic angle detections, the kinematic properties of the approaching targets such as target velocity, range and location can be extracted from the tracker to give an initial classification of target types. This tracker result is also used to automatically aim a camera to the predicated point and photograph the target. Acoustic measurements can be used to further divide the target groups based on Harmonic Line Association (HLA) method by extracting a set of feature vectors from acoustic spectrograms and comparing them against the acoustic target database. Therefore targets such as typical false alarms (e.g., birds, ducks, etc.), propeller driven aircraft (civilian small aircraft), helicopters, and jets can be classified.

5.1. Acoustic feature extraction and classification

As mentioned earlier, the primary targets of interest in this study are small, low-flying aircraft. Such small aircraft tend to emit strong harmonic lines produced by propeller or profane noise. This suggests that a target classification algorithm can be developed based on the Harmonic Line Association (HLA) method.

Given an acoustic time sequence and the corresponding sampling rate, an FFT spectrum is computed at each buffered data frame. A noise spectrum is calculated using a two-pass notched moving average approach with a single-sided window width and a given detection threshold estimated from past experimental data. Spectra peaks, defined as a sequence of 3 FFT bins where a local max occurs are then detected, and the frequencies at which the peaks are detected are accurately determined by doing a parabolic curve fitting to the peak profiles. Using the most significant peak as an anchor, those harmonically related frequency peaks are grouped together to form a hypothetical harmonic feature vector set. This process is then repeated until all the harmonic feature vector sets are extracted for each data frame from all the frames available. It has been found from the field experiments that the most informative aircraft harmonic signatures for small civilian aircraft usually exist within the frequency range from 20-2000 Hz. Given a typical fundamental frequency of small civilian aircraft that are on the order of 50 Hz during normal flight, the first 40 harmonics are selected to form a 40-component feature vector which will be used for the classification. In order to minimize the sound propagation effect and make the feature vector essentially distance invariant, the magnitude of each component is normalized relative to the sum of the magnitudes of the two highest harmonics in the set. Finally the derived feature vectors from each data frame are statistically averaged to form a feature vector template which distinctively represents the aircraft target. The above workflow is summarized in Figure 10. The final classification is performed using a Nearest Neighbor classifier

Figure 10.

Acoustic feature extraction processes

5.2. IR feature extraction and recognition

Aircraft recognition from IR images is done based on the Moment Invariants method. The Moment Invariants method has been frequently used as a feature extraction technique for image processing, remote sensing, shape recognition and classification (Keyes & Winstanley, 2000) The method extracts a set of numerical attributes - the moment feature vectors which uniquely characterize the shape of an object and yet have the desired property of invariance under image translation and rotation. The method was first applied to aircraft shape identification from binary television images by Dudani, etc. (Dudani, etc. 1977) and was shown to be quick and reliable.

The mathematical foundation of Moment Invariants for two-dimensional shape recognition was first introduced by Hu (Hu 1962) in which a set of shape descriptor values were computed from central moments through order three that are independent to object translation, scale and orientation. Translation invariance is achieved by computing moments that are normalized with respect to the centre of gravity so that the centre of mass of the distribution is at the origin (central moments). Size invariant moments are derived from introducing a simple size normalization factor. From the second and third order values of the normalized central moments a set of invariant moments can be computed which are independent of rotation.

In this paper six invariant moment functions that appear to be suitable for the present problem are selected with their mathematical expressions given below,

$$\begin{array}{c}{M}_1=({(}^{{\mu }_{20}}+4{\mu }_{11}{}^2)/r^4\\ M_2=({(}^{{\mu }_{30}}+{(}^3)/r^6\\ M_3=({(}^{{\mu }_{30}}+{(}^{{\mu }_{21}})/r^6\\ M_4=(({\mu }_{30}-\mu u_{12})({\mu }_{30}+{\mu }_{12})\cdot [{(}^{{\mu }_{30}}-3{(}^{{\mu }_{21}}]\\ +(3{\mu }_{21}-{\mu }_{03})({\mu }_{21}+{\mu }_{03})\cdot [3{(}^{{\mu }_{30}}-{(}^{{\mu }_{21}}])/r^{12}\\ M_5=(({\mu }_{20}-{\mu }_{02})\cdot [{(}^{{\mu }_{30}}-{(}^{{\mu }_{21}}]+4{\mu }_{11}({\mu }_{30}+{\mu }_{12})({\mu }_{21}+{\mu }_{03}))/r^8\\ M_6=((3{\mu }_{21}-u_{03})({\mu }_{30}+{\mu }_{12})\cdot [{(}^{{\mu }_{30}}-3{(}^{{\mu }_{21}}]\\ +({\mu }_{301}-3{\mu }_{12})({\mu }_{21}+{\mu }_{03})\cdot [3{(}^{{\mu }_{30}}-{(}^{{\mu }_{21}}])/r^{12}\end{array}$$E20

Where $${\mu }_{pq}=\frac{1}{N}{\displaystyle {\sum }_{i=1}^N(u_i}-\overline{u}{)}^p{(v_i-\overline{v})}^q$$are the central moments, $$u$$and $$v$$ are the image coordinates, and $$r=\sqrt{({\mu }_{20}+{\mu }_{02})}$$ the gyration factor which is used to normalize the moment functions in order to obtain the desired size invariance.

A preprocessing of IR images is performed before the final recognition process. After detection, the area that contains the potential target is first cropped from the original image. Then a binary image is formed by a simple threshold circuit. The aircraft silhouette is next extracted from the resulting binary image and its coordinates are used for the invariance moments feature vector exaction. Figure (11) illustrates the above workflow.

Figure 11.

IR image preprocessing workflow for target classification

To perform an initial classification test, a collection of numerically generated three-dimensional models representing classes of targets of interest (small civilian aircraft, military helicopters, large jets, missile) were chosen to be compared against the IR image collections from the field test. The three-dimensional models, as shown in Figure (12), consist of a Cessna 172, a Black Hawk helicopter, a Lear jet 35, and a missile. These models are constructed based on scaled drawings of the geometric models of each type.

Figure 12.

Three-dimensional models used in IR image classification

Figure 13.

A measured IR image (center) and the projected 2D images of the selected aircraft (left plot), and the extracted invariant moment vectors (right plot)

In order to compare the candidate target templates to the observed IR images, the corresponding three-dimensional models must be projected into a two-dimensional image plane with the appropriate azimuth ϕ and elevation θ angles corresponding to the camera viewing angles. Taking advantage of the multi-modal sensor character, the azimuth and the elevation angles can be effectively determined from the output of the kinematic tracker derived from the corresponding radar and acoustic measurements. Since the tracker provides the kinematic parameters of the target which include the range, the angles of arrival, and the flight trajectory, the perspective azimuth and elevation angles (ϕ, θ) of the target can be derived from a simple geometrical translation. Figure (13) shows an example of a measured IR image and the projected 2D images centered on the observed azimuth and elevation (ϕ=110, θ=60)) with variations of ±10 degrees on both angles. The corresponded invariant moments are extracted from these images and the results are also shown in the figure.

6. Prototype system and field demonstration

An experimental prototype consisting of three remote sensor nodes and a central processing node has been developed and built using COTs components. Figure 14 show the system hardware diagram, respectively. Each remote sensor node contains a low cost, low-power range only radar sensor (NobelTec IR2 X-band marine radar); an equally spaced, 4-element rectangularly-arranged acoustic array (B&K microphones), and a mini-computer (Slim Pro PC) that performs target range and angle detection and reports those results to the central node. A first order classification based on target acoustic signature is also performed at each remote node and the result is reported back to the central node. The central node contains an IR camera (uncooled BAE Micro IR sensitive to the 8-12 um waveband), a Pelco pan and tilt controller device mounted on a small tower, and the central computer that performs data fusion and final target classification. The connectivity is provided through a simple point-to-point 802.11 wireless communications network consisting of signal boosters and omni-directional antenna located at the remote node and a Yagi-type directional antenna located at the central node. This modular and compact system allows for rapid and inexpensive production of nodes and rapid deployment of the netted sensor fence system.

Figure 14.

System hardware diagram

Field tests of the netted sensor fence have been conducted at Nashua Municipal Airport, Nashua, New Hampshire. Typical experiment layout and sensor array positions are shown in Figure 15. The sensor suite is positioned near the end of the runway. The test aircraft are flying at a flight test matrix with multiple combinations of altitude and engine RPM. GPS data recording systems are mounted on the aircraft so the ground truth information can be transmitted in real time to the central node for target validation. The target aircraft used in one of the most recent tests was a Beech BE-76 Duchess. The remote sensor nodes node 1 & 2 were placed at the end of the runway so that planes taking off and those flying parallel to the runway would cross the fence. Due to space constraints, the remote nodes were placed in a T configuration with spacing of approximately 200 meters. The central node was collocated with one of the remote nodes (remote node 3) at the base of the T. Figure 16 shows plots depicting the tracker performance via comparisons of the tracker results with the ground truth recorded by an on board GPS. In general the tracker results show good agreement with the GPS ground truth data.

Figure 15.

Sensor configuration and layout in the field test.

A nearest neighbor classifier is then applied to train the extracted features and the final classification results can then be obtained. Euclidean distances between the moment vectors extracted from the observed images and those from the suspected 3D numerical models can be used to measure the confidence level of the classification results.. An image classification example is shown in Figure 17. In this case, a twin engine Beech BE-76 Duchess was correctly identified.

Figure 16.

Comparisons of tracker results with the ground truth GPS recording as a function of time including plots of: the target position (upper left, note: GPS recordings are denoted in circle dots, whereas the tracker results are denoted by lines ); the range (upper right); the azimuth (lower left), and the elevation (lower right)

Figure 17.

An image classification example shows a pair of collected IR image frames (upper), the extracted target silhouette (lower left) and the classification result (lower right).

7. Conclusion

Small, low-flying airborne vehicles may pose an imminent threat to homeland security and border integrity. Using a forward-based fence that contains a mix of low cost, low power radar, acoustic and optical (Infrared and visible) sensors by appropriate sensor fusion methodologies it is feasible to detect, track and discriminate small, low flying airborne targets and provide 24/7 sentry functions to protect critical civilian and military infrastructure. We have demonstrated the technical feasibility of the netted sensor fence approach. A proof-of-concept initial experimental prototype has been built and tested using COTs components. The technology is highly modular by modality, and adaptable to potential customer needs and requirements.

Acknowledgments

The authors thank Greg Crawford, Ron Fante, Chris Bas, Jeff Atwood, Bryan George, Garry Jacyna, Mike Jeffris, Walter Kuklinski, Tim Nadeau, Michael Otero, Dennis Reeves, Lucien Teig, and Stephen Theophanis for their contributions to this project.