Fusion of Color-Based Multi-Dimensional Scaling Maps For Saliency Estimation

Abstract

This work presents an original energy-based model, using a pixel pair modeling combined with a fusion procedure, to the saliency map estimation problem. More precisely, we formulate the saliency map segmentation issue as the solution of an energy-based model involving pixel pairwise constraints, in terms of color features, to which are then added constraints of higher levels of abstraction given by a preliminary over-segmentation whose location of regions but also contour information are exploited. Finally, this segmentation-driven saliency measure solution is then expressed in different color spaces which are combined together in order to take into account the specific properties of each of these color models with a outlier rejection scheme. Experimental results show that the proposed algorithm is both simple, efficient by performing favorably against state-of-the-art methods and also perfectible.

1. Introduction

Saliency detection (SD) is generally defined as the detection and segmentation of the most important or interesting objects or visual elements in the image scene which immediately and naturally attract and hold the attention of the viewer.

This algorithm basically attempts to imitate the natural eye-fixations process of human visual system (HVS) which has the amazing capability to rapidly detect and precisely localize the most noticeably visible and informative object or region within a (potentially very) cluttered image. In fast, this mechanism, which has developed during the evolutionary process, allows the human (and most mammals) to quickly and efficiently analyze the scene and focus its attention, on the important objects or regions of this one, with a minimal allocated processing (visual) resources.

SD is a low-level image processing task which is important in a variety of vision or image processing applications, especially for those where there is a need to reduce the information overload. This kind of problems can be commonly encountered in content-based image/video retrieval, summarization, categorization, compression, or browsing, automatic resizing, image cropping, advertising design or adaptive image display on small devices to name a few.

A significant number of methods have used (with some variations) the same principle proposed by the pioneer work of Itti and Koch [1] based on local contrast concepts and modeling the fact that a salient object/region presents a strong contrast with respect to its existing spatial neighborhood [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12]. Another class of algorithmic procedures is rather based on the hypothesis that the salient element is remarkably distinct; i.e., it exhibits a (or a mixture) of unique and discriminating colors (eventually combined with textural features) compared to the rest of the image [6, 10, 13, 14, 15, 16, 17, 18, 19, 20].

Nevertheless, local contrast-based algorithms tend to produce over-estimated saliency values close to the edges and contours of objects while producing too low saliency values inside objects instead of highlighting them uniformly and are also affected by high-frequency spatial signals in the image, while techniques based on global contrast can not differentiate regions precisely and are often altered by the presence of cluttered background [21, 22].

That is why some SD models propose to combine local and global contrast-based techniques either by combining local and global features [23] or by combining local contrast cues via a multi-layer or multi-scale approach [19, 21], or a tree structure [18] or have based their saliency estimation on a multi-level segmentation [17].

Other approaches are modeled within the conceptual framework of quantum mechanics [22], bag-of-features [24], matrix decomposition [25] or have been developed in the regression (or equivalently energy-based) framework [17, 21, 26] sometimes guided by the Gestalt-laws [27]. Other approaches are based on graph theory [28] or image statistics possibly obtained in advance from a database of natural images [29] or modeled within the Bayes statistical framework and only based on the statistics of the input image [30, 31] or eventually by using conditional random field (CRF) [23, 32, 33, 34, 35, 36], Bayesian model [37] or Markov random field (MRF) [38] generally by formulating the saliency estimation as a random walk problem [4, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49] or via an absorbing Markov chain [50] possibly guided by depth information [51].

Contrary to the above-mentioned strategies, our saliency model is based on the combination of three contributions:

• The first one relies on a generic energy-based model involving pixel pairwise constraints or using an original modeling by pair of pixels, giving a first and rough saliency map (based on the color cue) whose solution is then efficiently and quickly obtained with a linear computational complexity (in terms of number of pixels in the image).

• The second one is a set of original and efficient constraints based on a preliminary over-segmentation whose location of regions but also contour information are exploited as constraints to improve the previous saliency map result.

• Finally, the third contribution is the use of a simple but efficient fusion procedure combining the saliency result expressed in several color systems with a robust outlier rejection scheme.

The remainder of this paper is organized as follows: Section 2 describes the proposed model. Section 3 reports and evaluates the performances achieved by our model by comparing them with state-of-the-art methods and then discuss the reliability, strengths and interest of the proposed approach and the quantification (in terms of classification gain) of each component of the proposed model. Finally, Section 4 concludes the paper.

2. Proposed model for the saliency map estimation problem

2.1 Color-based multidimensional scaling saliency map

Our saliency model is first based on the hypothesis that the salient region or element to be detected is remarkably distinct; that is to say that it has discriminating colors compared to its surrounding spatial neighborhood. To this end and in order to provide a rough but meaningful first soft saliency measure map $\widehat{x}$ (with values varying from 0 to 1 representing the saliency probability of each pixel in the sense of a given criterion), we can search the solution map $\widehat{x}$ that will translate the distance between the mean color existing between each pair of pixels in the input image by an identical gray-scale distance between each pair of pixels in the resulting saliency map to be estimated (or otherwise said by estimating $\widehat{x}$ that preserves, as much as possible, the color or gray-scale distance between respectively each pair of pixels of the input image and the soft saliency map). Mathematically speaking, this can be efficiently done by searching the map $\widehat{x}$ that minimizes the following energy-based or cost function (which is also commonly called the stress function):

$$\widehat{x}=arg\underset{x}{min}\sum _{i\ne j=1,\dots ,N}{\left({\left(\parallel y_i^{\square }-y_j^{\square }{\parallel }_2-\parallel x_i-x_j{\parallel }_2\right)}^2\right)}^{1/2}$$E1

where $N$ is the number of pixels of the input and the saliency images and the summation ${\sum }_{i\ne j=1,\dots ,N}$ is on all pixel pairs (i.e., for all pixels $i$ and for all pairs of pixels involving $i$) being within the image. $x_i$ corresponds to the gray level at pixel location $i$ and $y_i^{\square }$ is a three-dimensional (3D) vector coding the three mean color channel values computed within a local squared region of size $N_c\times N_c$ centered on the pixel at location $i$ (of the input color image $y$). The notation $\parallel .{\parallel }_2$ refers to the Euclidean distance.

This stress function given in Eq. (1) is often minimized using procedures called stress majorization. This class of techniques act as an efficient global minimizer of this loss function consisting of residual sum of squares by integrating all the pairwise constraints. Conceptually, this kind of optimizers treats each distance or constraints between a pair of pixels (in terms of gray level difference in our application) like a rubber band between the couple of pixels, and searched to reorganize the gray-scale value of each pixel of $\widehat{x}$ to minimize the tension or the stress of the rubber bands (hence the name of stress measure) or in an equivalent way, to fulfill all constraints in a least-squares (LSQ) problem formulation. Multidimensional scaling (MDS) is the most known stress majorization procedure allowing to minimize such a stress function with pairwise constraints and this optimizer has proven to be particularly effective in its ability to solve certain image processing problems requiring a modeling by pair of pixels (e.g., such as color image segmentation [52, 53], hyperspectral compression [54, 55], asymmetry detection [56], human action recognition [57], fusion procedures [58], histogram specification algorithms [59] high dynamic range compression (HDR) [60], multimodal change detection [61] and database browsing and visualization [62] to name a few).

However, the MDS algorithm which is initially proposed (and named metric MDS [63, 64]) is not ideally suited to our application (as well as for all large scale applications) due to its computational and memory load. Indeed, it requires a $O\left(N^2\right)$ computational complexity and an entire $N\times N$ distance matrix to be stored in memory. Instead, we have resorted to a faster (and near-optimal) variant here, named FastMap [65] whose major benefit lies in its linear computational complexity (through an efficient Nyström [66, 67] approximation¹). In addition, it worths mentioning that the efficiency of the FastMap is all the more important as the dimension of the final mapping is low [67]. That’s also why this optimizer is particularly well suited to our application since the final (saliency) mapping has to be estimated at very low dimensions (i.e., one dimension since we finally want to obtain a 1-channel gray-scale saliency map²).

Let us now recall that a configuration $\widehat{x}$ that minimizes Eq. (1) will give us a gray-scale map in which the pixel pairs that are close in terms of mean color in the input image $y$ will also give us pixel pairs, in the saliency map $\widehat{x}$, with gray-scales that are faithfully close (in the LSQ sense). Nevertheless, the configuration, given by the FastMap technique, may optionally generate an image with an inverted gray-scale scale (negative of an image or/positive inversion) (i.e., with values of gray levels all the lower as the regions are estimated as salient and vice versa). In order to rearrange this gray-scale mapping in the appropriate direction (with correspondingly higher gray-scale values for the most salient regions), we simply assume that a salient element/region is a priori more likely to appear in the image center [17, 21, 26, 29] (or conversely unlikely at the edges of the square image frame). To do this, we calculate the Pearson correlation coefficient between the $\widehat{x}$ saliency map estimated by the FastMap and a rectangle, with maximum intensity value and about half the image size, and located in the center of the image, more precisely, a rectangle beginning and ending respectively at the row $\mathit{\text{lgth}}/4$ and $\mathit{\text{lgth}}-\mathit{\text{lgth}}/4$ and respectively at the column $\mathit{\text{wdth}}/4$ and $\mathit{\text{wdth}}-\mathit{\text{wdth}}/4$ where $\mathit{\text{lgth}}$ and $\mathit{\text{wdth}}$ represents the length and the width of the image. If the correlation coefficient is negative (anti-correlation), we must then associate to each pixel (of $\widehat{x}$) its complementary gray value.

2.2 Segmentation-based adaptive central location constraint

In order to improve the saliency map result $\widehat{x}$, we decide to incorporate an adaptive location constraint favoring the fact that salient objects are more likely to appear in the segmented regions locating in the center of the image. To this end, we first exploit an over-segmentation of the input image and more precisely the concept of super-pixel which has already been successfully used, in different ways, in several saliency SD models [17, 18, 26, 30, 68, 69, 70]. An over-segmented region or super-pixel brings together a group of pixels (forming a perceptually meaningful atomic region allowing to preserve the important structures in the image) which can be exploited to replace the rigid pixel grid structure in images. Herein, we use the super-pixel technique introduced by Felzenswal [71] with the settings (default values) proposed by the authors (see Figure 1c).

Figure 1.

Refinement of the MDS-based saliency map $\widehat{x}$ [see Eq. (1)] with the two segmentation-based constraints favoring the fact that salient objects are 1—more likely to appear both in the segmented regions locating in the center of the image and 2—with a strong contour (see Section 2.2) in the RGB color space. (a) Input ECSSD image (b) Ground truth binary salient mask (c) Segmentation into regions given by the FH algorithm (d) Potential $c$ favoring the central regions of the image (e) Potential $c$ favoring the central regions of the image (f) Relative strength (in terms of average gradient magnitude) of the boundary of each region (g) Potential $c$ combined with the segmentation into regions and weighted by the relative strength of the boundary of each region (h) Initial saliency map (i) Saliency map combined with the map obtained at step (g).

In addition to the location of the different regions, this segmentation allows us to estimate the strength of the contour of each region relatively to the other regions. In our application, this visual cue is simply expressed as the average magnitude of the gradient per pixel of the contour of the concerned region divided by the average gradient (per pixel) of all the contour points given by the segmentation (thus giving a positive number $\xi$ all the greater as the region is prominent in the sense of the strength of its outline). This visual cue allows us to incorporate in our model the fact that a saliency region is more likely to appear with a strong contour (see Figure 1f).

Finally, in our model, the set of pixels belonging to each segmented region $R_i$ is assigned the mean the initial saliency measure (within this region) firstly estimated by $\widehat{x}$ and the mean of the potential favoring saliency regions in the center of the image (see Figure 1e) and finally weighted by ${\xi }_{R_i}$, the relative strength of the boundary of $R_i$ (Figure 1f):

$$\text{saliency}\left(R_i\right)={\xi }_{R_i}\cdot \frac{1}{\mid R_i\mid }\sum _{s\in R_i}\left({\widehat{x}}_s\cdot c_s\right)$$E2

with $c_s$ the potential at site $s$ of coordinates (row,col) favoring saliency regions in the center of the image (and disfavoring them on the edges of the image) defined in the following way:

$$c_{\mathrm{row},\mathrm{col}}=2-2max\left(\left|\frac{\mathrm{row}-\text{lgth}/2}{\text{lgth}/2}\right|,\left|\frac{\mathrm{col}-\text{wdth}/2}{\text{wdth}/2}\right|\right)$$E3

where lgth and wdth represents the length and the width of the image and $c_{\mathrm{row},\mathrm{col}}$ is the pixel gray-scale value of the map $c$ at coordinates (row, col). The refinement of the MDS-based saliency map $\widehat{x}$ with these two constraints (see Figure 1i) is noted in the following ${\widehat{x}}_c$. One of the undeniable advantages of these two types of constraints is that they both do not depend on any internal parameter to be adjusted or tuned afterwards (thanks to their inherent adaptation to each image). The effectiveness of these two constraints will be quantified individually in Section 3.3.Algorithm 1. FastMap.

Algorithm 1: FastMap

Input:

$k$: Target space dimension

$N_p$: Object (vector) number in database $\mathcal{O}$

Output:

$X_{\left[N_p,k\right]}$: Array of objects in target space

Initialization:

$d\leftarrow 0$

FASTMAP ALGORITHM (k, D(),$\mathcal{O}$)

• if $k\le 0$ then return $X$

• $d\leftarrow d+1$

• Choose pivot objects $O_a$ and $O_b$ so that the metric $D\left(O_a,O_b\right)$ is maximal

foreach object $i$ from $\mathcal{O}$ do

• Project $O_i$ on the line $\left(O_a,O_b\right)$

     Compute: $X_{\left[i,d\right]}=x_i$

     $x_i=\frac{D^2\left(O_a,O_i\right)+D^2\left(O_a,O_b\right)-D^2\left(O_b,O_i\right)}{2D^2\left(O_a,O_b\right)}$

end

foreach object $i$ from $\mathcal{O}$ do

• Project $O_i$ on an hyper-plane perpendicular to the line $\left(O_a,O_b\right)$

    ${\left(D^{\prime }\right)}^2\left(O_i^{\prime },O_j^{\prime }\right)=D^2\left(O_i.O_j\right)-{\left(x_i-x_j\right)}^2$

end

call FASTMAP($k-1,D^{\prime }\left(\right),\mathcal{O}$)

2.3 Fusion in different color spaces

In order to further improve our final saliency estimation, the segmentation-driven saliency map obtained by the two previous steps (cf. Sections 2.1 and 2.2) is repeated for different and non-linearly related color spaces which are then combined with a fusion method. Let us note that this strategy has already proven its efficiency to improve the accuracy of an image segmentation with different fusion methods optimal in a particular criterion sense [58, 72, 73]. Let us recall that a color space is just an arbitrary model that combines three numbers (or tristimulus values) to describe a sensation of color that our vision system can possibly perceive. In a fusion model combining several color systems, each color space can be in fact regarded as different image channels provided by different sensors or as different complementary filters. In our application, we use $N_s$ saliency maps provided by the $N_s=8$ following non-linearly related color spaces; $\mathcal{C}=\left\{\mathrm{LUV},\mathrm{HSV},\mathrm{LAB},\mathrm{RGB},\mathrm{YIQ},\mathrm{XYZ},\mathrm{HSL},\mathrm{TSL}\right\}$. Each color space has a specific and interesting property³ [74, 75, 76, 77, 78, 79], which can efficiently be taken into account and makes the fusion process very efficient (and whose effectiveness will be quantified in Section 3.3).

In our application, the fusion model is simply performed by averaging the $N_s-1$ more reliable saliency maps achieved in these different color spaces (after their individual normalization in order that all values are between 0 and 255). In order to eliminate the less reliable saliency map which is identified in our fusion model as an isolated outlier, we just calculate the one that is least correlated with all the others in the Pearson’s correlation sense (see Figure 2). This fusion strategy will be quantified in terms of F-measure gain in Section 3.3.

Figure 2.

Saliency map ${\widehat{x}}_c$ obtained by the two previous steps (cf. Sections 2.1 and 2.2) estimated respectively in the (a) RGB (b) HSV (c) LAB (d) LUV (e) YIQ (f) XYZ (g) HSL and (h) TSL color spaces and (i) final saliency map obtained by our fusion model [averaging of the seven more reliable saliency maps. For this ECSSD image, the outlier which is automatically eliminated is the saliency map estimated in the TSL (h) color space].

3. Experimental results

3.1 Set-up and dataset description

First, to limit the computational load, we decided to resize all the images so that the maximum (height, width) of the image is 200 pixels. In the following, all parameters are defined relative to this new base image size.

The only internal parameter of our model is the size of the squared window $N_c$ (introduced in the initial MDS-based saliency map, see Eq. 1). In our experiment, we have tuned this internal parameter in order to maximize the F measure on a subset of 10 images randomly extracted from the ECSSD dataset. This was done by applying a fixed step-size grid search method to select the best parameter (over the space of parameters) and in possible ranges of parameter values (namely $N_c\in \left[3-25\right]$ [step-size: 2]). We have found $N_c=19$ and a quiet relative insensitivity of our model if this size parameter is held within plus or minus $30\%$ of this value. We recall that the internal parameters of the super-pixel algorithm used in our model is set with the default values suggested by the authors [71]. In addition, it is worth also recalling that the two segmentation-based constraints applied on the initial MDS-based saliency map do not depend on any internal parameter (since these two constraints adapt to each image (see Section 2.2).

We have validated our algorithm on the extended complex scene saliency dataset (ECSSD) (see Figures 3 and 4) which was built by Yan et al. [21] and Shi et al. [80]. This database is composed of 1000 images containing various categories (natural or man-made objects) and different backgrounds (sometimes non uniform with possible change in illumination intensity and/or composed of several parts or including small-scale structures). In addition, multiple salient objects possibly exist in one image, while part of or all of them are sometimes transparent (or do not have a sharply clear boundary and/or obvious difference with the background) and are regarded, all the same, as salient decided by an expert. The binary mask or ground truths have been synthetically generated for each image with a well-defined and reliable protocol (more precisely, for each image, several experts have manually drawn a saliency map which was finally combined in one ground truth binary mask that minimizes the inter-subject label consistency in the majority vote sense. (see [80] for additional details). We have also tested our method on the 5000 images of the commonly used MSRA-5000 (or MSRA-5K) saliency dataset [23]. Images of MSRA-5000 are relatively less complex (with less variability) and therefore are therefore rightly considered as less challenging to process than those of the ECSSD image database [80]. Examples of visual comparison between our model and the state-of-the art (SOTA) CHS saliency model presented in [80] (and also the designer/constructor of the ECSSD dataset) on the first 8 images of this image database is shown in Figure 4.

Figure 3.

From lexicographic order, distribution of respectively the $F_{\beta }$, $F$ and mean absolute error (MAE) measures given by our MDSSME (multidimensional scaling map-based saliency map estimation) model on the 1000 images of the ECSSD dataset.

Figure 4.

Visual comparison between our model and the SOTA CHS saliency model presented in [80] on the first eight images of the ECSSD dataset. From left to right, ECSSD image, ground truth salient mask, CHS saliency map [80], our MDSSME saliency result.

3.2 Quantitative measure

Our quantitative assessments and experiments follow the framework presented in [10, 14, 68, 80]. First, the precision-recall curve, associated with the set of saliency measures achieved by our model is plotted. and then compared with other methods (see Figure 5a). In addition, since in many applications, high precision is preferable to high recall, we thus also estimate the $F_{\beta }$-measure proposed in [68, 80], as a function of each possible threshold:

Figure 5.

Qualitative comparison on ECSSD. From top to bottom. Precision-recall curve for each possible threshold (within range $\left[\mathrm{0,255}\right]$) and $F_{\beta }$-measure (see Eq. (4) and [80]) with ${\beta }^2=0.3$ as a function of the threshold.

$$F_{\beta }=\frac{\left(1+{\beta }^2\right)\cdot \text{precision}\cdot \text{recall}}{{\beta }^2\cdot \text{precision}+\text{recall}}$$E4

in which thresholding (within range $\left[\mathrm{0,255}\right]$) are applied, ${\beta }^2$ is set to 0.3 as suggested in [10, 68, 80]⁴ and the obtained $F_{\beta }$-measure is shown and compared to other methods (see Figure 5b).

Besides, we studied as a performance metric, the mean absolute error (MAE) [68, 80] which actually measures the quality of the weighted continuous saliency map (which could turn out to be, for some applications, more important information than the binary mask itself). Mathematically, the MAE measures the MAE between the soft saliency measurement map $x$ and the binary ground truth $x_G$ (both normalized in the interval $\left[0,1\right]$). The MAE metric is given by:

$$\mathrm{MAE}=\frac{1}{\left(\text{hgth}\times \text{wdth}\right)}\sum _{i=1}^{\text{hgth}}\sum _{j=1}^{\text{wdth}}\mid x\left(i,j\right)-x_G\left(i,j\right)\mid$$E5

with $i$ and $j$ designating respectively the row and column coordinates of $x$ or $x_G$.

The precision-recall curve along with the $F_{\beta }$-measure as a function of the threshold and the MAE distance are complementary measures to each other and help interpreting the global or the local behavior (for a given or optimal threshold) of a SD model.

3.3 Results and discussion

First of all, we have tested (see Figure 6) the efficiency (in terms of precision-recall curves) of the different main characteristics of our MDSSME model on ECSSD dataset. More precisely, Figure 5 shows the plot of the precision recall curve obtained by our model for respectively, for the first four curves; (1) a fusion process (see Section 2.3) involving one, two, four or eight space colors; thus quantifying the efficiency of our fusion strategy through different color spaces (while the other characteristics of our model being optimal and remain unchanged), (2) for the last three curves; our model without the a priori constraint that a saliency region is more likely to appear with a strong contour (gradient prior), or without the a priori constraint of the central location (see Section 2.2) or without any constraint, i.e. without the two previous constraint and the over-segmentation process (just the MDS estimation expressed and averaged over the eight color spaces).

Figure 6.

Precision-recall curves (and optimal $F$ and $F_{\beta }$-measures [see Eq. (4)]) for different variations of our MDSSME model on ECSSD [80] dataset.

We can notice that the fusion process through different color spaces is efficient and allows an interesting gain of approximately 12% (on average) on the results but beyond four color spaces, the improvement gain is negligible. Let us recall that the color spaces must be chosen different and as uncorrelated as possible; this guarantees both, an over-segmentation (and a set of constraints based on it), which are different but complementary. We also notice that the gradient-based constraint brings a slight 3% gain compared to the full model. Nevertheless, this constraint (combined with the over-segmentation constraint) remains interesting when one does not consider the commonly used central location constraint.

We have first evaluated our SD model on the ECSSD dataset and compare our results with local methods; IT [2], GB [4], AC [8], and global methods; LC [13], SR [6], FT [10], HC [14], RC [14], CA [15], LR [16], RCC [19], and CHS [80]. Figure 5 shows the plot of the precision recall curves and the $F_{\beta }$ curve [see Eq. (4)] as a function of threshold, and Table 1 lists the MAE distances achieved by these different SD methods compared to our model, on the ECSSD dataset.

AC [8]	CA [15]	FT [10]	GB [4]	HC [14]	IT [2]	LC [13]	LR [16]	SR [6]	RC [14]	RCC [19]	HS [80]	CHS [80]	MDSSME
0.264	0.310	0.270	0.282	0.326	0.290	0.294	0.267	0.264	0.301	0.187	0.224	0.227	0.265
0.228	0.250	0.230	0.243	0.239	0.248	0.245	0.215	0.225	0.264	0.140	0.153	0.150	0.218

Table 1.

Quantitative comparison with [2, 4, 6, 8, 10, 13, 14, 14, 15, 16, 19, 80] in terms of MAE measurement on the ECSSD [80] (first column) and MSRA-5000 (second column) databases.

For the MSRA-5K dataset, the optimal $F_{\beta }$ measure we can achieve is $F_{\beta }=0.791$, (for the particular threshold 136) and $F=0.786$ ($\beta =1$ for the threshold 106) and MAE=$0.218$ (see Table 1).

Overall, we obtain a competitive precision-recall or $F_{\beta }$ measure curve with a correct MAE when compared with existing state-of-the-art algorithms. Nevertheless, in terms of performance, the strong point of our method remains the very competitive optimal $F_{\beta }$-measure (for a given optimal threshold) which is reflected graphically by the fact that the $F_{\beta }$ curve associated to our model reaches a maximum above all other curves (see Figure 6b) with a plateau wide enough for an estimation of the optimal threshold to be possible (this plateau is also centered around the mean value of the gray scale range, i.e. between [100–170]).

We have also compared our model with two very recent salient detection energy-based models, very recently published in the literature [20, 38]. These two energy-based models are based, for the first, on LTP (local ternary patterns) texture based features [20] and, for the second, on pairwise pixel based features [38] to capture the notion of saliency in a scene (and these two energy-based models operate both of them within a single specific color space). We have compared our model on the ECSSD and MSRA-5K datasets in terms of the optimal F-beta and MAE efficiency measures (see Table 2) which shows that our model is very competitive, in terms of these two highly complementary efficiency measures.

	PPMRF [38]	LTPSD [20]	MDSSME
ECSSD	$F_{\beta }=0.727$ MAE = $0.150$	$F_{\beta }=0.729$ MAE = $0.257$	$F_{\beta }=0.734$ MAE = $0.265$
bf MSRA	$F_{\beta }=0.790$ MAE = $0.108$	$F_{\beta }=0.781$ MAE = $0.215$	$F_{\beta }=0.791$ MAE = $0.218$

Table 2.

Quantitative comparison with two recent salient detection energy-based models [20, 38] in terms of MAE and $F_{\beta }$ measurements on the ECSSD (first line) and MSRA (second line) databases.

Our current implementation takes on average 0.6 seconds to process one image with resolution $400\times 300$ on a 8 GB and 3.33 GHz Intel i7 CPU (6675.25 bogomips) with an un-optimized C++ code running under Linux (or about 10 minutes for the processing of the entire ECSSD dataset composed of 1000 images). In the proposed algorithm, each MDS maps is calculated very quickly, in approximately 0.05 second due to the linear computational complexity (in terms of the number of pixels) of the FastMap algorithm and this, we recall, despite the fact that this algorithm will model and take into account a quadratic number of pairs of pixels [see Eq. (1)]. Let us also mention that our SD model can be easily parallelized by using an OpenMP implementation on several CPU cores (each MDS map estimated on each CPU core) or even better, by using a GPU implementation of the FastMap algorithm as proposed in [81], in order to reduce the computation time by a factor of 1000 (on a mid-range GPU card) and to allow the processing of the entire ECSSD dataset in less than 1 second.

Let us note that, in addition to being perfectible (by adding more color spaces) and possibly faster (by using GPU programming), our unsupervised energy-based model also has the property of depending on very few parameters. This defines a low complexity model, ensuring no over-fitting and good generalization properties and consequently, this allows our algorithm to have the advantage of performing well on other datasets.

In addition, and this also complements what has been stated previously, since our algorithm exploits the notion of colors in different (non-linear between them) color spaces, our algorithm is perfectible by combining (or fusing) our model with those using as features; the LTP texture [20] or the notion of pixel pairs [38] (or possibly different other features).

4. Conclusion

In this paper, we have proposed a novel and simple saliency measure and segmentation method. The proposed method is based on an energy-based model, involving a pixel pair modeling using color features and segmentation-based constraints finally combined with a fusion procedure taking into account the specific property and the complementarity of several color space models. The proposed model has very few internal parameters and is designed to be highly adaptive to the image content. Qualitative and quantitative results show that the proposed method is effective and performs particularly well against state-of-the-art methods, especially in terms of optimal F-measure (for a given optimal threshold). In addition to its efficiency, the proposed model achieves a good compromise between simplicity and accuracy while being fast enough, easily parallelizable (in different ways), and also perfectible if more color spaces or more constraints are added to the fusion procedure.

Acknowledgments

This research was funded by individual discovery grant RGPIN-2022-03654. The data supporting the results of this study are freely available at:

HTTP://WWW.IRO.UMONTREAL.CA/MIGNOTTE/RESEARCHMATERIAL/INDEX.HTML

The author wishes to acknowledge the NSERC (Natural Sciences and Engineering Research Council of Canada) for having funded this study under the individual discovery grant program (RGPIN-2022-03654). The author certifies no potential conflicts of interest influencing this study.

Data Availability Statement

The data (C++ code under Linux OS, makefile, image databases, etc.) supporting the results of this article are freely accessible at:

http://www.iro.umontreal.ca/∼mignotte/ResearchMaterial/index.html.