Thresholding Image Techniques for Plant Segmentation

Abstract

There are challenges in the image-based research to obtain information from the objects in the scene. Moreover, an image is a set of data points that can be processed as an object in similarity way. In addition, the research fields can be merged to generate a method for information extraction and pixel classification. A complete method is proposed to extract information from the data and generate a classification model capable to isolate those pixels that are plant from others are not. Some quantitative and qualitative results are shown to compare methods to extract information and create the best model. Classical and threshold-based state-of-art methods are grouped in the present work for reference and application in image segmentation, obtaining acceptable results in the plant isolation.

1. Introduction

There are three fields for the image-based research: Image processing, Computer Vision and Computer Graphics [1]. In a graphical way, this is shown in the Figure 1.

Figure 1.

Fields in image-based research. Source: [1].

Image processing takes an image as input and realize a set of operations to create a new image that improves the interest feature visualization. It is not limited to; the image processing can isolate those features have not meaningful information to be removed from the scene. For Cancer Aided Diagnostic in dermoscopy, shown in Figure 2, the lesion must be bounded but there are meaningless elements as hair and air bubbles. Here, there is a need to improve the lesion visualization removing all those elements. An approach for the processing chain begins with color space transformation, continues with hair detection and finishes with image inpainting [2].

Figure 2.

Hair Inpainting in dermoscopic images.

Computer Vision starts with an image and provides features of the object in the scene. These features allow a quantitative description for object interpretation. The Figure 3 takes the region of interest (ROI), hand for this case, and the 7 Hu’s moments are calculated to describe the hand sign as a feature vector, simplifying the classification process [3].

Figure 3.

Static sign recognition. Source: [3].

Computer graphics generates a visual representation of mathematical model behavior. The models can be visualized from lines to a video that shows time evolve behaviors. Figure 4 shows a finite differences description to represent a coil with ferromagnetic core.

Figure 4.

Numerical analysis for non-destructive electromagnetic test. Source: [4].

The applications are not exclusive from each field of study, these fields could be merged to generate a hybrid system that improves the description and composition to solve a specific problem. To hair removal in the Figure 2, a new image with the hair, described by a statistical operator and a thresholding rule, is generated. The inpainting takes the original and generated images to process only the pixels that do not describe the lesion, this process minimizes the classification error. The solution employees three research fields in image-based systems.

Color indexes. For the primary color image encoding the RGB color space is used. This allows the storage and image representation in Red, Green and Blue colors [5, 6, 7]. Transform the color space brings other image description to simplify the processing or improve visualization.

In matrix representation, the color transformation has the following form,

$$I=T_S\bullet p+k$$E1

where $I$ is the color space with $N$ components, $T_S$ is the $N\times 3$ transformation matrix, $p$ is the RGB vector representation of the pixel, and $k$ is a constant vector. This representation allows a generalized transformation for $N$ Channels with $N$ transformation equations.

Suppose the RGB to YUV color space transformation [5],

$$Y=0.299R+0.587G+0.114B$$

$$U=0.492\left(B-Y\right)$$

$$V=0.877\left(R-Y\right)$$E2

Substituting and expanding Eq. (2),

$$Y=0.299R+0.587G+0.114B$$

$$U=-0.147R-0.289G+0.436B$$

$$V=0.615R-0.515G-0.100B$$E3

For this case, $k$ is a zeros vector. This allows represent RGB to YUV as matrices with Eq. (4)

$$\left(\begin{array}{c}Y\\ U\\ V\end{array}\right)=\left(\begin{array}{ccc}0.299& 0.587& 0.114\\ -0.147& -0.289& 0.436\\ 0.615& -0.515& -0.100\end{array}\right)\left(\begin{array}{c}R\\ G\\ B\end{array}\right)$$E4

Because color transformations do not consider neighborhood pixels, these are known as point-to-point operations. The color spaces are described for break up color and lighting. Moreover, there are other methods to transform color spaces and simplify feature extraction in images, introducing the color index concept.

A color index can take the color space information and generates a new channel, this improves the feature visualization according to the requirements. For plant images, Yuan et al. estimates the Nitrogen content in rice from plant in the digital image [8]. Authors get a measure called GMR, see a representation in Figure 5, subtracting the red channel from green channel responses respectively. After, they apply a fixed threshold for the plant segmentation.

Figure 5.

Colored GMR response of a leaf in an image acquired with polarized light.

The Color Index Vegetation Extraction (CIVE) is used to break up plants and soil. This allows a grow evaluation in the crops. Furthermore, the CIVE shows good response in outdoor environments [9, 10]. If the color is processed in the GMR and CIVE, the color transformation is defined as,

$$\left(\begin{array}{c}\mathit{GMR}\\ \mathit{\text{CIVE}}\end{array}\right)=\left(\begin{array}{ccc}-1& 1& 0\\ 0.441& -0.811& 0.385\end{array}\right)\left(\begin{array}{c}R\\ G\\ B\end{array}\right)+\left(\begin{array}{c}0\\ 18.78745\end{array}\right)$$E5

Similarity Measure. Minkowski distance is a generalized way to similarity measure [11, 12, 13, 14] defined as,

$$d_n\left(X,Y\right)={\left(\sum _{i=1}^n{\left(\left|x_i-y_i\right|\right)}^n\right)}^{\raisebox{1ex}{$1$}\!\left/ \!\raisebox{-1ex}{$n$}\right.}$$E6

where $X=\left\{x_1,x_2,\cdots ,x_n\right\},Y=\left\{y_1,y_2,\cdots ,y_n\right\}\in {\mathcal{R}}^n$ are data points which the algorithm seeks minimum distance. If $n=2$ then Euclidean distance is measured. Substituting in Eq. (6) the following expression is obtained,

$$d_2\left(X,Y\right)=\sqrt{{\left(X-Y\right)}^{⊺}\left(X-Y\right)}$$E7

Other way to measure similarity is by Mahalanobis distance [11, 15], this is calculated by Eq. (8),

$$d_M\left(x,y\right)=\sqrt{{\left(X-Y\right)}^{⊺}{\mathrm{\Sigma }}^{-1}\left(X-Y\right)}$$E8

where $\mathrm{\Sigma }$ is the covariance matrix.

$$\mathrm{\Sigma }=\left[\begin{array}{cccc}{\sigma }_{11}^2& {\sigma }_{12}^2& \cdots & {\sigma }_{1n}^2\\ {\sigma }_{21}^2& {\sigma }_{22}^2& \cdots & {\sigma }_{2n}^2\\ ⋮& ⋮& \ddots & ⋮\\ {\sigma }_{n1}& {\sigma }_{n2}^2& \cdots & {\sigma }_{\mathit{nn}}^2\end{array}\right]$$E9

If $\mathrm{\Sigma }=I$ then Eq. (8) brings the Euclidean distance. Thus, a Weighted Euclidean distance can be calculated as

$$d_{\mathrm{\Omega }}\left(x,y\right)=\sqrt{{\left(X-Y\right)}^{⊺}{\mathrm{\Omega }}^{-1}\left(X-Y\right)}$$E10

where $\mathrm{\Omega }$ is a $n\times n$ weight matrix. If each component is independent form others, then is possible define a weight matrix $W=\mathrm{\Sigma I}.$

$$W=\left[\begin{array}{cccc}{\sigma }_{11}^2& 0& \cdots & 0\\ 0& {\sigma }_{22}^2& \cdots & 0\\ ⋮& ⋮& \ddots & ⋮\\ 0& 0& \cdots & {\sigma }_{\mathit{nn}}^2\end{array}\right]$$E11

With the analysis above, there are 3 cases for the weight matrix:

1. $\mathrm{\Omega }=\mathrm{\Sigma }$: Mahalanobis distance is calculated

2. $\mathrm{\Omega }=I$: Euclidean distance is calculated

3. $\mathrm{\Omega }=W$: Weighted Euclidean distance is calculated.

Find features that describe the object of interest is required to calculate the similarity for each pixel in the image. The distance measure brings the similarity between the pixel and plants, background, fruit and more.

Thresholding. Thresholding is based on a simple rule to cluster data in groups $k_0$ and $k_1$ from a threshold $T$. The groups have not a meaning in their clustering. Moreover, it is possible use this rule as supervised method (classification) [16, 17, 18, 19]. The rule $R$ to identify a data point is described below.

$$R=\left\{\begin{array}{cc}{k}_0,& \epsilon <T\\ k_1,& \mathit{\text{otherwise}}\end{array}\right.$$E12

where $\epsilon$ is a measure according to analysis.

For digital images, an example for pixel identification is shown in Figure 6. From this figure, $T$ is selected with the mean of the data and the pixels are assigned to a group according to it.

Figure 6.

Detected ROI by mean thresholding. a) Original image (Green Channel). b) Histogram (blue) and mean value (red). c) Grouped pixels.

There are two challenges using this rule: which parameters define $\epsilon$ and what conditions bring $T$. According to the analysis, $\epsilon$ could be an independent component. This means that the rule can use only one-color feature. Furthermore, there is a possibility to make a feature composition, improving results. In other perspective, $T$ can be defined by a measure that describes data features for $\epsilon$ and allow an acceptable division between groups.

In state-of-art methods for threshold calculation, the Otsu method is reached [20]. This method brings thresholds that have the better separability over the data. Some indexes and separability obtained are shown in Figure 7.

Figure 7.

CVPPP color index pixel generated dataset break up. a) B Channel from RGB space: There is no way to separate data; b) S Channel from HSV space: There is a medium separability; c) a channel from lab space: There is an acceptable break up. Green: Plant, red: Background, histogram: 64 levels.

In order to measure the quality in the data isolation, the information gain is used. The information gain, good for decision tree generation, is a data homogeneity indicator [21, 22, 23]. This allows a class probability measure for a dataset distribution, indicating that all classes have the same probability when entropy is equal to one.

Information gain is calculated by Eq. (13)

$$G\left(X|T_n\right)=H\left(X\right)-\sum _{v\in T}\frac{\left|X|T_v\right|}{\left|X\right|}\bullet H\left(X|T_v\right)$$E13

where $v$ is each possible value in the random variable for the analyzed color index $T_n$, $X\mid T_v$ is the sub generated dataset for $T_n$,$\left|\bullet \right|$ is the dataset cardinality and $H\left(X\right)$ is dataset entropy.

For threshold case,

$$G\left(X|T_n\right)=H\left(X\right)-\left(\frac{\left|X|T_{<}\right|}{\left|X\right|}\bullet H\left(X|T_{<}\right)+\frac{\left|X|T_{\ge }\right|}{\left|X\right|}\bullet H\left(X|T_{\ge }\right)\right)$$E14

Entropy is a random variable uncertainty measure. This is the most utilized information measure in this kind of processes [24, 25, 26]. In a normalized way, the Shannon entropy is calculated as follows

$$H\left(X\right)=-\sum _{k=1}^C{P}_k\bullet {\mathit{log}}_C\left(P_k\right)$$E15

where $C$ is the available classes in the dataset and $P_k$ is the corresponding probability for each class.

Considering $M$ pixels and each class has $M/C$ elements, Eq. (15) changes to,

$$H\left(X\right)=-\sum _{k=1}^C\frac{1}{C}\bullet {\mathit{log}}_C\left(\frac{1}{C}\right)$$E16

For the binary case, $C$ is replaced by 2, then

$$H\left(X\right)=-\sum _{k=1}^2\frac{1}{2}\bullet {\mathit{log}}_2\left(\frac{1}{2}\right)=1$$E17

If $H\left(X\right)=1,C>1$ then all classes have the same probability. This means that the dataset distribution for feature selection is uniform.

If the conditional entropy $H\left(X|T_{<}\right)=H\left(X|T_{\ge }\right)=0$ then each sub dataset has elements that belongs to only one class $k$, this is a total separability indicator.

2. Methodology

A proposed method consists in two main sections: Feature selection and classification. Feature selection takes the better features that break up those pixels in foreground and background. Classification utilizes a similarity measure to compare pixels and plant description to assign a specific class. Both processes are described below.

To select features that belong to plants in the best way, the method requires a similarity measure. This measure allows to compare and identify which can provide an acceptable separation between data. Considering a dataset with the same quantity samples of each class, the dataset has the form described in Figure 8.

Figure 8.

Pixel dataset representation. Where $i$ es the feature vector that describes the object under study. The dataset contains $N$ features, and $\mathit{\text{Class}}$ describes what is the meaning of the feature vector, in this case, foreground or background.

The feature $i_n$ is considered as continuous random variable. In order to simplify and use simple tools in the classification model generation, the random variable is discretized to a binary variable. Thus, the dataset is presented in Figure 9.

Figure 9.

Discretized variable dataset representation. Where $T_n$ is a threshold overcome indicator.

For data classification, the minimum distance is wanted between the pixel and the object of interest. This analysis requires those features that describes the object to be compared with those features that describes the pixel. For plant image processing, some features are the color indexes as NCIVE, MNGRDI, GMR, etc. The feature selection is dependent to those maximizes data separation.

When the features are defined, independent statistical features are calculated. This information allows an orientation correction and distance threshold magnitude computation. According to this analysis, all components are considered independent from others, thus, the distance measure takes the third scenario. This weights those features that have more variability and brings an eccentricity adjust. An orientation correction should be applied to data because Euclidean distance is rotation variant.

For threshold calculation the standard deviations ${\sigma }_{11},{\sigma }_{22},\cdots {\sigma }_{\mathit{nn}}\in {\mathcal{R}}^n$ are considered as a component vector. The magnitude is calculated with the Weighted Euclidean Distance and is assigned to the threshold. The standard deviation is considered because variance cannot be expressed in the same plane. This thought is expressed as follows,

$$\mathit{Th}=d_{\mathrm{\Omega }}\left(0,\sigma \right)\mid \mathrm{\Omega }=W$$E18

Before calculating distances, an orientation correction is needed with the angle described by the data. Finally, the thresholding defines if a pixel is plant or not with the following rule

$${\mathit{\text{Plant}}}_{\left(r,c\right)}=\left\{\begin{array}{cc}\mathit{\text{True}}& d_W\left(p_{\left(r,c\right)},R\right)<\mathit{Th}\\ \mathit{\text{False}}& \mathit{\text{otherwise}}\end{array}\right.$$E19

where ${\mathit{\text{Plant}}}_{\left(r,c\right)}$ is the classification as plant for pixel $p_{\left(r,c\right)}$ and $R$ is the feature vector defined by data in plant class. As result, the classification model with this method is shown in Figure 10.

Figure 10.

Pixel classification map. a) Data classification with decision boundary in Th; b) normalized classification map with decision boundary in [Th, 2Th, 3Th]. Green: Plant, red: Background, black: Boundary decision. Th: Computed threshold.

3. Results

In the following processes the MATLAB 2020b was used in a Laptop with Xeon E3-1505 M processor with32GB RAM. In addition, only the matrix operations were used for data processing and data image representation.

According to the study in [27], the optimal color index for data break up is the L and a channels in the Lab color space. Moreover, some experiments are done according the dataset variability from the CVPPP dataset [28], which is the test dataset. In one hand, comparing both results is observable that a channel matches with [27] as best index to isolate the plant pixels. In the other hand, there are other indexes that provide an acceptable separability, NCIVE and MNGRDI, shown in Tables 1 and 2. In a comparative way, the Otsu method thresholding has a separability less than the supervised entropy.

i	Threshold	Information Gain
a	0.3843	0.8421
NCIVE	0.4196	0.8036
MNGRDI	0.5294	0.7537
G	0.3882	0.6913
b	0.6705	0.6729
L	0.3843	0.6601
V	0.4078	0.6387
S	0.5294	0.6014
R	0.3137	0.4325
H	0.5098	0.1597
B	0.247	0.0074

Table 1.

Indexes gain information. Otsu method.

i	Threshold	Information Gain
a	0.3968	0.8471
NCIVE	0.4444	0.8428
MNGRDI	0.5238	0.7467
G	0.3174	0.7265
V	0.4078	0.6894
b	0.6507	0.6786
L	0.3333	0.6601
S	0.492	0.6055
R	0.2857	0.4489
H	0.1904	0.3629
B	0.063	0.0748

Table 2.

Indexes gain information. Supervised entropy.

Other observation is the similarity in the order of separability quality. Moreover, the supervised entropy knows the data meaning, consequently, the quality measure is better in most of the cases.

In Figure 11 some examples of the index threshold data separability are shown. From here, is observed twice best cases, with an acceptable separability, and twice worst cases, with no apparent separability way.

Figure 11.

Thresholds and information gains for some color indexes. Left: Supervised entropy method, right: Otsu method.

Finally, Classification map for plant description pixels based on a-NCIVE indexes is shown in the Figure 12.

Figure 12.

Classification map for plant pixel segmentation. Th: Computed threshold.

According to this model, the pixels can be classified as plant or not. Some visual results are shown in the Figure 13.

Figure 13.

Visual results in plant image segmentation for CVPPP dataset.

4. Conclusions

The thresholding methods are effective when the problem data is defined and the superposition between groups is minimum. Furthermore, they are simple methods that provide acceptable results in the segmentation problem. The combination of methods is possible to rise the quality in the models.

The entropy, in a supervised way, can improve the data separability. Because Otsu methods minimizes the variance between groups, the quality in the results using supervised entropy is improved in consequence of consider data meaning. The supervised methods know the expected response, breaking up data in corresponding classes. Otsu only allow clustering data that have not a well-defined meaning yet.

The distance measures refer to reach a similarity in the compared data. It only needs a reference description of the studied object. In this case, all pixels are defined as plant, in a statistical way, a reference with the new pixels going to compared and classified. Thus, the segmentation is possible on those pixels that belong to plant and discard those are not.

From Tables 1 and 2, the best indexes have the same order in both results. Furthermore, the calculated threshold improves the data separability for the supervised entropy case. This allows development of classification maps like in Figure 12 to consider those indexes that achieve the best pixels break up. Figure 11 shown some separation scenarios where pixels that are plant (Green distribution) and are not (Red distribution). Finally, the classification map, shown in Figure 12, illustrates the best classifier obtained with the method to select pixels that belong to plant class.

Acknowledgments

To M. Minervi et al. for providing their dataset. This work is supported by Instituto Politecnico Nacional de Mexico (IPN) (Grant ID: 20201681) and Consejo Nacional de Ciencia y Tecnologia (CONACyT), project 240820.

Conflict of interest

The authors declare no conflict of interest.