The Use of Machine Vision in the Diagnosis of Ripening Strawberries

Abstract

The problem of image recognition using methods based on statistical solutions can be divided into two classes: parametric and nonparametric. In the first case there is a set of data on the selection of image criteria, in the second such criteria must be found. Taking into account, the possibilities of obtaining initial data for the identification of agricultural objects, the first option is considered in the article. Possibilities of diagnostics of strawberry growth at a stage of its cultivation in hothouse conditions are presented. Several image classifiers that can be used to solve such problems are investigated. An experiment was performed, which has shown the accuracy of the applied algorithms for diagnosis and recognition of berries, which allowed to set thresholds for the detection of ripe berries. A number of berry recognition methods were compared in order to study the accuracy of berry recognition and response to berry growth.

1. Introduction

Image processing is an effective tool in the process of studying crops, the care of which largely depends on the observations that are made at all stages of the plant life cycle. However, when part of the crop is in the process of ripening and part is already ripe, especially in greenhouses, there is a need to control this process. Remontant strawberries, which can yield continuously throughout the year, are among the types of crops that need to be identified for ripening. Therefore, you can use machine vision to control the cultivation of such a crop. The use of such systems in agricultural work has a huge potential for accounting automation and harvesting control. Therefore, an important aspect of such tasks is the development of algorithms for recognizing the characteristics of plants and their classification, as well as the choice of existing methods.

Research in this area shows that the most common methods of machine learning for image classification include neural networks and other methods of machine learning [1, 2, 3]. However, due to the high demand for computing power, machine vision systems require both energy and time consumption for image processing. Therefore, depending on the tasks, the identification algorithm should be maximally adapted, because the criteria for selecting the quality of image recognition differ depending on the applicability to different subject areas.

Thus, to solve the narrow problems of identifying the number of ripe berries, approaches such as identifying an object by color, contour, and shape, which require only matrix calculations, can give a quite acceptable result. Taking into account that, obstacles such as brightness and image scale also play an important role, stationary camcorder installation, and artificial lighting can alleviate these problems.

The main stages, in this case, can be considered as follows:

• defining separate image contours which can be further investigated as a result of special filters application;

• determining coordinates of the selected arrays on the established color;

• determining the area of selected objects;

• setting the number of selected revolutions in order to ratio the appropriate number after a certain period of time.

Increasing the number of selected objects will be an indicator of the beginning of the harvest.

Thus, it is necessary:

• to set the number of strawberries and their area in the image for each individual area of plantings;

• to identify a separate area where video recording will take place;

• enter the number of identified ripe berries and their area together with the area identifier of individual plantations in the database in order to further verify their growth.

• video recording to implement on the basis of a moving video camera (Figure 1).

Figure 1.

Movable platform for video recording of plantings (created using Google SketchUp).

As a result, it is possible to diagnose the right time for harvesting, which will allow farmers to harvest ripe berries in time, without losses. Popular algorithms are used for image analysis: SURF; ORB; FAST; SIFT. In order to store data for further comparison, you can use a simple Sqlite database.

2. Literature analysis and problem statement

The image recognition algorithms in the field of agriculture have become very popular, which have differences and certain classifications [4], a large part of which is implemented using a number of methods of machine learning and neural networks. However, the use of neural networks requires huge arrays of data for training, which cannot always be obtained. In addition, the significant capacity for training and the time spent on processing such arrays is quite significant. It is difficult to determine the logic of the distribution of the weights rate. All this and many other requirements for big data processing impose certain restrictions on the choice of computing tools, obtaining training data, and the difficulty of understanding how the algorithm itself works. However, depending on the image identification tasks, the algorithm for determining the desired image can be simplified in some way. In particular, if you use color detectors in combination with contour detectors, for example, which allow under certain conditions to obtain image descriptors, you can identify the object with high accuracy.

There is a considerable amount of work devoted to the identification of objects in agriculture by color and geometric shapes. Thus, the recognition is performed [5] on the basis of statistical characteristics and signs of the set of obtained ranges of descriptor values with the help of the minimum distance classifier. The results of experiments on a database containing about 2635 fruits, made it possible to establish 15 different classes, which has confirmed the effectiveness of the proposed approach.

A similar study to that proposed in the article aims to analyze automated harvesting methods based on fruit categorization. It in turn offers a new approach that separates the fruit in the foreground from the background, which in addition to color spectrum characteristics also takes into account the identification of texture images. The classification model is built using reference vectors (SVM) and learns using function descriptors taken from the training data set. The proposed approach allows the use of small computing power and can run on single-board computers, such as RaspberryPi [6].

Some works describe the possibility of identifying fruits and vegetables by such criteria as the characteristics of the plant color [7], as well as the structure of their leaves [8]. Thus, the combination of these two approaches gives certain results. However, simple ways to identify images have a number of disadvantages, in particular noise generated by shadows or changes in light. Due to the use of a number of filters, the use of neural networks in this case gives a more accurate result [9]. Therefore, such filters are carefully selected. For example, the Otsu method is often used to reduce noise by low-pass filters [10].

Median filters are used to smooth inconspicuous spots [11]. Spectral analysis is used to obtain a clear identification by color. For example, researchers at the University of Oklahoma studied the use of hyper-spectral methods of image analysis to assess the quality, safety, and classification of fruits and vegetables, which resulted in determining the most effective approaches to estimating the color spectrum in the image [12]. Typically, to highlight RGB color saturation, the image classification is converted to HSV or YCbCr. To ensure image clarity, anti-aliasing filters are used, such as the Gaussian filter [13]. The step-by-step selection of individual image contours can be represented using the OpenCV image recognition library, which contains the possibilities of software implementation of the respective stages. Thus, to identify strawberries, by means of this library, you can select separate pixels by color. In other words, you can create a mask that characterizes only the berries of strawberries. Then you can reduce the noise by smoothing with the help of the Gaussian filter [13].

To determine the area of a selected object, you can count its pixel IDs, which characterize such a selection. To do this, you can use the threshold method of the OpenCV library, which turns the image, where all the pixels that are darker (less) than 127 are replaced by 0, and everything that is brighter (more) than 127 - by 255.

In this case, it is possible to set criteria for strawberry growth by comparing the specific gravity of the selected color spectrum at different times. To do this, strict conditions must be met. Photo fixation should be performed each time in the same position of the photo recorder and in the same lighting.

In this case, it is possible to observe changes in the size of the strawberry and changes in its color, which during ripening begins to correspond to the established range of identification. In Figure 2 the corresponding range in the HSV format was set within the following limits: lower threshold—(0,50,20)–(5,255,255); upper threshold—(175,50,20) –(180,255,255).

Figure 2.

The process of identifying strawberries by color.

Taking into account that the photos were taken in different conditions if error and noise in the images a3-d3 are minimal, this approach can be used to assess the growth of strawberries. In addition, if one of the most complex image processing processes is the Gaussian filter, the corresponding tasks can be solved using single-board computers such as RaspberryPi [14].

The stages of identification presented in Figure 2 are highlighted in the following sequence:

• obtaining the original raster image from the photo-capture (.jpg or other similar format);

• color filtering;

• smoothing with a Gaussian filter and converting the selected area to black and white;

• counting the pixels that characterize the selection of strawberries (white color);

• counting other pixels (black color).

The OpenCV library was used to solve them. In the Python programming language, it looks like this:

import cv2

import os

im = cv2.imread('image.jpg')

im = cv2.resize(im, (400, 300))

img_hsv = cv2.cvtColor(im, cv2.COLOR_BGR2HSV)

m1 = cv2.inRange(img_hsv, (0,50,20), (5,255,255))

m2 = cv2.inRange(img_hsv, (175,50,20), (180,255,255))

mask = cv2.bitwise_or(m1, m2 )

cr = cv2.bitwise_and(im, im, mask=mask)

m1 = cv2.GaussianBlur(mask, (7,7), 3)

cr1 = cv2.GaussianBlur(cr, (3,3), 0)

print (len(cr1[cr1==0]), 'black')

print (len(cr1[cr1>0]), 'white')

cv2.imshow("msk", m1)

cv2.imshow("cr", cr1)

cv2.waitKey()

Thus, the ability to assess the growth of strawberries using machine vision, without using complex algorithms for deep learning and neural networks, puts forward the admissibility of our hypothesis. Is it possible to create a strawberry harvest indicator with a low-power computer and simple image operations? The main idea of this hypothesis is, taking into account the studied approaches, to obtain a growth curve of strawberries, based on a simple comparative method. Such a comparison can characterize the dynamics of changes in the area of ​ selected objects in the images, as relative indicators of the specific weight of selected objects in the image. For example, for images a3-d3, presented in Figure 2, this ratio will have the following characteristics (Figure 3).

Figure 3.

Percentage of color-coded objects in images a1-d1 Figure 2.

However, the difficulty lies in the video recording itself, as it must be carried out under certain conditions of image periodicity and with the same angle and lighting, which requires further adjustment (namely, data acquisition, storage, and subsequent processing).

3. Ways to identify the growth of strawberries based on singular point detectors

In order to timely diagnose the yield of strawberries, in addition to identifiers by color, you should pay attention to other methods of pattern recognition. In the framework of the task, detecting the increase in strawberries, you can consider detectors of special points in the image, the number of which can also serve as an indicator of growth. The most common are SIFT and SURF.

The use of the SIFT detector is more efficient but requires a significant amount of computation. Its advantages are that it is possible to identify special points in the image regardless of scale and offset, as well as changes in brightness. However, the invariance of the last three transformations cannot be fully obtained. Basis for his work is the construction of the Gaussian pyramid and the differences of the Gaussians (Difference of Gaussian, DoG). Gaussian is an image that is blurred by a Gaussian filter.

$$\mathrm{L}\left(\mathrm{x},\mathrm{y},\sigma \right)=\mathrm{L}\left(\mathrm{x},\mathrm{y},\sigma \right)\ast \mathrm{L}\left(\mathrm{x},\mathrm{y}\right),$$E1

where:

L - the Gaussian value at a point with coordinates (x, y);

σ - blur radius;

G - Gaussian core;

I - the value of the original image,

* - convolution operation.

In this case, the Gaussian difference is the image obtained by subtracting each pixel of one Gaussian source image from Gaussian with a different blur radius.

$$\begin{array}{c}\mathrm{D}\left(\mathrm{x},\mathrm{y},\sigma \right)=\left(\mathrm{G}\left(\mathrm{x},\mathrm{y},\mathrm{k}\sigma \right)-\mathrm{G}\left(\mathrm{x},\mathrm{y},\sigma \right)\right)\ast I\left(x,y\right)=\\ =\mathrm{L}\left(\mathrm{x},\mathrm{y},\mathrm{k}\sigma \right)-\mathrm{L}\left(\mathrm{x},\mathrm{y},\sigma \right),\end{array}$$E2

To determine a singular point, together with the construction of the Gaussians pyramid, a pyramid of Gaussians differences is constructed, consisting of the differences of adjacent images in the Gaussians pyramid. Accordingly, the number of images in this pyramid will be N + 1.

Figure 4 shows the Gaussians pyramid, which displays their differences, formed on the basis of two neighboring Gaussians. The number of such differences is always one less than the number of Gaussians, because when you move to the next octave, the size of the images is halved. The point that is the local extremum of the Gaussian difference is then determined. If the value of the Gaussians difference at a certain point is bigger or less than all the values ​​of the neighboring Gaussians differences, such a point is considered an extremum point. In Figure 5 this point is represented in shaded.

Figure 4.

Pyramid of Gaussians and their differences.

Figure 5.

Defining a singular point among the differences between Gaussians.

Such points are refined using an approximation of the Gaussian difference function using a second-order Taylor polynomial, which is taken at the point of a definite extremum.

Then, based on the neighborhood of the singular point, a descriptor is constructed, which is presented in the form of a gradient matrix in accordance with the pixels that are around the singular point.

Then a histogram of the descriptor is created, which is used for further correlation of the image with others. Therefore, SIFT detectors need to increase the computational speed, because the image scaling process takes some time. For greater speed, SURF detectors are used which, in order to find singular points, use the Hesse matrix, the determinant of which reaches the extremum at the points of maximum change of the brightness gradient. Thus, if the original image is given by the intensity matrix I, then the current pixel, which is analyzed for changes in color intensity, can be denoted by X = (x, y) and the scale of the filter σ. In this case, the Hesse matrix will look like this:

$$\mathrm{H}\left(\mathrm{x},\sigma \right)=\left[\begin{array}{cc}{L}_{\mathit{xx}}\left(X,\sigma \right)& L_{\mathit{xy}}\left(X,\sigma \right)\\ L_{\mathit{xy}}\left(X,\sigma \right)& L_{\mathit{yy}}\left(X,\sigma \right)\end{array}\right],$$E3

where: $L_{\mathit{xx}}\left(x,\sigma \right),L_{\mathit{xy}}\left(x,\sigma \right),L_{\mathit{yy}}\left(x,\sigma \right)$ - approximation convolution of the second derivative of the Gaussian cores.

The determinant of the Hesse matrix reaches its extremum at the points of maximum change of the brightness gradient. In turn, the SURF method uses a filter with a Gaussian core throughout the image, while finding singular points at which the maximum value of the determinant of the Hesse matrix is achieved. Thanks to this search, both dark spots on a white background and vice versa stand out. Unlike SIFT, the SURF method does not have a mechanism for refining points, but immediately forms descriptors. Like SIFT, SURF is invariant to rotation and scale. Using the capabilities of these detectors, you can use them to establish singular points on the structure of the image of strawberries and with the help of their number observe the yield. This statement is due to the fact that the shape of the strawberry has some common features (Figure 6), the identification of which with these detectors can lead to a positive result.

Figure 6.

Common features in the structure of the shape of strawberries.

In order to do this, we will determine the training image, which will help to obtain descriptors of singular points (Figure 6B). We will also compare the number of singular points for a number of test images, which will establish which method to identify ripening strawberries is more effective, identify color changes, or identify changes in singular points. The following assumption is based on the fact that during the strawberries growth, their structure will expand, which has singular points.

Comparing these two approaches, it should be noted that the use of the SIFT detector will not allow to assess the ripening by color change, but only by size, so it has some limitations.

According to the results of testing the training sample of descriptors of singular points, which were obtained from the image (Figure 6b), a small number of matches were identified on the original images using SIFT and SURF detectors (Figure 7). In addition, as you can see in Figure 6, there is a fairly large error, which indicates the inexpediency of using such detectors in the process of strawberries ripening identification.

Figure 7.

Visualizations and comparison of the received singular points with the training (Figure 6b) and test images.

It is presented in Figure 7:

1. Original test images.

2. Visual characteristics of the definition of singular points on the training image.

3. Number of singular point descriptors that coincided with the training image using the SIFT detector.

4. The total number of singular points using the SUFR detector.

5. Number of singular point descriptors that matched the training image using the SUFR detector.

6. The total number of singular points using the SUFR detector (Figure 8).

Figure 8.

Comparison of the effectiveness of strawberry recognition methods using color filters and SIFT and SURF detectors on samples of test images (Figure 7a1–d1), (%).

4. Identification of strawberry ripening processes by color

Detection of changes in strawberry images can be implemented on the basis of a mobile platform, which is controlled by a single-board computer RaspberryPi. Infrared proximity sensors can be used to identify shooting locations, which can be used to obtain a signal to stop a moving platform at the necessary location for photo fixing of strawberries. Standard infrared distance sensors allow you to respond to approximations at a distance of up to 30 cm, which is enough to identify the stopping point of the platform. The placement of such sensors can be carried out according to the scheme in Figure 9.

Figure 9.

Position of the infrared distance sensor (created using Google SketchUp).

After that, at the set time, the platform will start moving again for the next video recording.

The implementation of the software algorithm can be performed on the basis of Python interpreter and libraries: OpenCV, image processing, sqlite3, database, RPi, RaspberryPi COM port, which will be used to provide a control signal to drive the transport platform. Thus, the algorithm will be as follows (Figure 9).

By implementing this algorithm in Python [15] and using it for a test sample of strawberry images, you can get the following result (Figure 10). Thus, you can see how strawberries grow in Figure 10.

Figure 10.

Algorithm of mobile platform operation for photo-fixation of the strawberry ripening process.

The sequence of sampling images in the order of ripening ($a_1,b_1,c_1,d_1,a_3,b_3,c_3,d_3$) and the number of obtained points, respectively ($a_2,b_2,c_2,d_2,a_4,b_4,c_4,d_4$). The growth dynamics of strawberries can be obtained in the form of the ratio of identified points by color to unidentified ones (Figure 12). However, a small percentage of errors can be observed in Figures 11 and 12, which are caused by the concentration of light on strawberries.

Figure 11.

Identification of strawberries by color during ripening.

Figure 12.

The dynamics of ripening strawberries is obtained as a result of the ratio of selected points by color to the total quantity for a conditional period of time.

5. Conclusion

The most important criterion for strawberry growth is its color and size. Recognition of changes in these criteria in the images can be obtained using conventional color filters, with the specified parameters of the color range. This approach is easy to implement in practice because it does not use complex computational algorithms, which is quite important for use in agriculture.

To identify the ripening of strawberries, it is enough to use an inexpensive single-chamber computer and a mobile platform with a photo recorder, which at a given time can capture changes in the image of strawberries.

Taking into account the slight error due to the formation of the light concentration on strawberries, the experiment resulted in growth dynamics, which characterizes such an approach on the positive side. However, a number of conditions must be met for its application. In particular, detailed images require the use of means of moving the photo-fixator and the invariability of the angle of photo-fixation. This method, in contrast to the detection of singular points in the image using SIFT and SUFR detectors, has shown higher efficiency, because the most important criterion for ripening strawberries is the change of its color.

Acknowledgments

The authors thank our colleagues from School of Engineering, Aerospace Engineering and Aviation team at RMIT University (Australia) and Department of Economic Cybernetics at National Aviation University (Ukraine), who provided insight and expertise that greatly assisted the research, although they may not agree with all the interpretations and conclusions of this research.