Image acquisition setups and their typical usage.
Color and appearance are perhaps the first attributes that attract us to a fruit or vegetable. Since the appearance of the product generally determines whether a product is accepted or rejected, measuring the color characteristics becomes an important task. To carry out the analysis of this key attribute for agriculture, it is recommended to use an artificial vision system to capture the images of the samples and then to process them by applying colorimetric routines to extract color parameters in an efficient and nondestructive manner, which makes it a suitable tool for a wide range of applications. The purpose of this chapter is to give an overview on recent development of image processing applied to color analysis from horticultural products, more specifically the practical usage of color image analysis in agriculture. As an example, quantitative values of color are extracted from Habanero Chili Peppers using image processing; the images from the samples were obtained using a desktop configuration of machine vision system. The material presented should be useful for students starting on the field, as well as for researchers looking for state-of-the-art studies and practical applications.
- color analysis
- color evolution
- feature extraction
- image processing
- Habanero chili
- computer vision
The color assessment of fruits and vegetables in the food industry and agriculture using machine vision and image processing has become a trend in the recent years [1–3]. The color features are one of the key parameters to define the quality of an agricultural product [4, 5]. The color is probably the first factor that consumers use to determine the appearance of a product ; appearance is a subjective factor that leads the consumer to accept or reject a food product . This significantly affects the sales and profits of the industry. Therefore, a considerable effort has been made in the area of automation to improve the quality of agricultural products in the food industry in order to decrease losses.
Building machines with the ability to see color as it does the human being has been a complex task for the scientific community and industry in recent years [8–12]. Among the many challenges to be addressed, we can include appropriate image acquisition systems, lighting problems, color space definitions, mathematical issues, and the development of specific algorithms and synchronization tasks . However, improvements in semiconductors, electronics, and software eventually brought the opportunity to implement image processing and colorimetric projects for various applications [11, 14, 15].
A machine vision system for horticulture products requires the ability to capture process and analyze color images, where algorithms are suitable to detect, extract, and quantify the attribute of color as much as a customer does. Furthermore, other parameters (size, texture, external blemishes, and diseases) are important to determine the appearance, and hence the quality, of an agricultural product [16–18]. This variety of applications is possible due to the interaction of light in the range of visible spectrum (400–700 nm) with the matter that the light can be reflected, transmitted, or absorbed by an object. The light wavelengths received by our eyes are then interpreted by our brain as color.
Perception of color, in humans, is a psychophysical phenomenon that involves three elements: the illuminant, an object, and the observer . The illuminant has the function to irradiate the object with light in the visible spectrum. The object absorbs, transmits, and reflects the light received from the illuminant. The observer perceives the reflected light from the object in the retina and responds to that stimulus generating an electric signal in the optic nerve toward the brain . As this phenomenon takes place in the brain, the process is an important challenge, because machine vision systems need to emulate these three elements, in a proper way, without including the real observer, the eye, and the natural illuminant, the sun .
A machine vision system acquires, processes, and analyzes images, and the proper operation of the system requires adequate capture conditions for the specific application. Basic machine vision components are: (a) image sensor (charge-couple device (CCD) or complementary metal oxide semiconductor (CMOS)), (b) illuminant (D-65, fluorescent lamps, among others), (c) the background (with high contrast for the object of interest), (d) trigger device (used to start image acquisition), and (e) a frame grabber (to capture the actual image). The capture procedure can be divided into the following steps: the illuminator irradiates the object, the image sensor receives the light reflected from the object and the background, and when the trigger is activated, the system extracts the color characteristics and converts them to electrical signals and then the grabber frame stores the image taken [22, 23]. Once the image is on the computer or other processing device, image processing algorithms are used to analyze the data. The basic steps can be listed as: (a) segmentation of images (the background is separated from the region of interest), (b) extraction of characteristics (select pixels of interest); for this stage, several functions and algorithms have been developed in order to get an adequate image processing of the scene. Next, we analyze the data generated from the previous process, the system extracts the color attribute of the objects in the image using colorimetric techniques, as well as others as morphological and texture parameters [24–26], and finally the analysis results are presented.
Colorimetry is the scientific area that measures, quantifies, and represents color . It is very useful in different areas because it provides the ability to turn color into an objective factor rather than a subjective one. Complementing this scientific area with the technology of computer vision systems, it is possible to see the color characteristics in a digital and standard way, in addition to performing the color evaluation with a noninvasive and noncontact procedure, which makes it a suitable technology for application in agriculture and the food industry for quality assessment. Due to their versatility, many other production systems can benefit from the features offered by artificial vision systems, image processing, and colorimetry.
The aim of this study is to evaluate noninvasive and noncontact techniques of image processing and color analysis; for example, the estimation of color from postharvest Habanero chili at different maturity stages is shown. Moreover, the capability of color assessment as a region, instead of a punctual area as is done with a typical colorimeter, identifies threshold values in these stages.
The goal is to identify and quantify color attributes of Habanero chili fruits using the following stages:
Image acquisition system (setup).
Image processing (threshold technique and algorithms).
Feature extraction (pixels of interest and color calculations).
Color analysis (attribute identification, quantification and classification).
2. Image processing
The techniques of image processing are related to algorithms that manipulate the numerical representation of the images to obtain useful information . In particular, segmentation subdivides the digital image into multiple regions or objects that have common characteristics . This action involves several steps that must be taken, before the images can provide valuable data. First, region of interest (ROI) must be identified, i.e., regions in the image that have pixels that matter to the application and which must be separated from the background. In order to develop algorithms that perform this work, it is necessary to define whether the image is black and white or colored, and then a criterion must be established for the segmentation, which must focus on morphological, texture, or colorimetric parameters, before establishing any flow chart as solution. It should be mentioned that algorithms may be suitable for many applications, but none of them is generally applicable to all images, and therefore, suitable algorithms for each particular application should be used [30, 31].
2.1. Image acquisition system and color image representation
It is essential to use an image acquisition system that suits the application properly. It must be defined what type of objects or products will be placed in the system, as well as set the scene location and choose the appropriate settings to obtain acceptable images. The issue of lighting (natural or artificial) depends on the scenes required by the application. Once the capture conditions are adequate, a vision sensor (CCD or CMOS) must be selected, which will suit the entire system for proper performance. A digital camera receives light variations corresponding to images onto a CCD device. The CCD contains capacitors that are stimulated by visible radiation and three filters adjusted for three basic colors: red (R), green (G), and blue (B). Theoretically, every color can be reproduced by the combination of three primary colors.
A color image are represented as a M × N × 3 (components) array of color pixels, where each color pixel is a triplet corresponding to the RGB components of and image at specific spatial location, as shown in Figure 1. By convention, the three images formed and RGB color image are referred to as the R, G, B component images. The data class of the component from the image determines their ranges of values, for example [0–255] or [0–65,535] for RGB images of class uint8 or uint16, respectively . The RGB color space is an additive color model that uses transmitted light to display colors. It is used for television and other devices screens; so this model is device dependent (Its appearance depends on the display.)
A summary of machine vision systems, image processing, and color analysis frequently implemented in the agriculture is presented in Table 1. First column corresponds to the type of application, and four general and typical areas are included. The type of tasks performed by these applications is shown in the second column, including the corresponding references. Third column shows the type of setup employed for each application, and the last column shows the locations where these systems are typically deployed.
|Application||Task||Type of setup||Location|
Aerial inspection [37–42]
|Mobile robot||On field|
|Static machine (placed on a production line)||Packer station|
2.2. Color image segmentation
Algorithms for color image segmentation based on the technique of threshold segmentation should be executed in triplicate, due to the structure of the color image [55, 56]. A typical segmentation process fills in with ones and zeros of the image matrix locations, corresponding to the selected regions in each of the color channels of an image, as shown in Figure 2.
3. Color calculations
The Commission Internationale de l’Eclairage (CIE) determines regulations, standards, and recommendations for color measurements. The CIELAB color space is an international standard developed by the CIE in 1976. Within CIELAB, a psychometric index of lightness (L*) and two color coordinates (a* and b*) are defined. L* is a qualitative attribute of relative luminosity, which is the property according to which each color can be considered as equivalent to a member of the gray scale, ranging between black (L* = 0) and white (L* = 100). Negative values from a* correspond to greenish and the positives to the reddish ones, whereas the yellowish colors takes negatives values from b* and the positive for bluish ones .
It is well known in the food industry that the CIELAB color space is used to analyze color changes in a qualitative way [58–62]. Color and appearance are closely related to the sensory properties and chemical composition of food. Color is usually measured by tristimulus colorimetry. The color stimulus is composed of three different parameters, giving the color a three-dimensional nature.
The color attributes are described in CIELAB as:
Lightness: This feature indicates if a color is lighter or darker. It is a relative measure of the reflected light against the absorbed. Value 0 corresponds to black and value 100 is assigned to white.
Chroma: It determines for each hue, the color difference taking as reference the gray level with same lightness. It can take positive values from zero.
Hue: It is the main attribute. It is a qualitative property, which allows classifying colors as red, yellow, etc. It is related to differences in absorbance of radiant energy at different wavelengths. Hue is specified as an angle.
These attributes are often expressed as L*, Cab*, and hab°, respectively; according to the CIELAB color space, it can be represented as Cartesian coordinates of polar coordinates Cab* and hab°. It can be used on a variety of instruments, such as colorimeters, spectrophotometers, and spectroradiometers. However, these instruments require homogenous samples to achieve a uniform color, which becomes a tedious and complicated task to measure the color of heterogeneous or small objects, such as grape berries and grape seeds. In these cases, the use of digital images for the extraction of color characteristics is advantageous. Digital image analysis appears as a suitable complement since it is possible to extract not only color characteristics but also other characteristics such as shape, texture, and homogeneity.
On the other hand, due to the nature of the CIELAB color space, the calculation of the Euclidean distance can be applied between neighboring samples, in order to obtain a relation between a quantitative and qualitative value on the variation of the color appearance of an object.
4. Application of image processing and color analysis of Habanero chili pepper
Since consumers buy with their eyes, color is considered one of the most important quality parameters of food products. Normally, this is determined by human inspection, or measured using a colorimeter or a spectrophotometer. The first process is subjective and susceptible to fatigue. The second is limited to measure just a small area of the food product, making it difficult to obtain a clear view of the color of the complete sample . In order to overcome these limitations, a system of artificial vision, image processing, and color analysis has been applied to measure the postharvest color of the Habanero chili (Capsicum chinense Jacq.) at different stages of ripening.
4.1. Genus capsicum
The Habanero chili belongs to the family Solanaceae and genus Capsicum. This genus consists of 27 species, five of which have been domesticated and are used worldwide as vegetables, spices, and condiments: Capsicum annum L., Capsicum frutescens L., Capsicum chinense J., Capsicum pubescens R. & P., and Capsicum baccatum L., where nonpungent cultivars of C. annum are the most consumed and are the main objective of most breeding programs .
Chili peppers (Capsicum spp.) are well known for their ability to cause an intense organoleptic sensation of heat when consumed (pungency). Capsaicin and its analogues, collectively called capsaicinoids (a group of alkaloids), are responsible for giving pungency or heat to the fruit; the pungent feature of peppers is only present in the members of the genus Capsicum [65–67].
The Capsicum chinense Jacq. is a very aromatic pepper and also is one of the hottest peppers in the world . These fruits are commonly used to give a pungent or hot sensation to many different meals and food products all around the world. During the past decade, it has been reported that the consumption of certain foods and spices such as pepper may have a positive effect on health. Genus Capsicum shows an incredible diversity and is consumed by a large section of population throughout the world because of its impressive health beneficial chemical compounds such as capsaicinoids, carotenoids (provitamin A), flavonoids, vitamins (vitamins C and E), minerals, essential oils, and aroma of the fruits. These compounds have been shown to possess anticancer, anti-inflammatory, antimicrobial, and antioxidant properties . They are important from the economic point of view for countries like Mexico, not only for the preparation of regional foods that have made Mexican gastronomy famous all over the world, but also because of the great amount and genetic variability that occurs in its territory, especially spicy species [68, 70–73].
The methodology used is described as follows: (1) specimens of Habanero chili here collected and arranged in three groups depending on their maturity stage by visual inspection. (2) The samples are moved to the laboratory to be placed in the machine vision system and the acquisition process is setup. (3) The acquisition process starts with a 24 sampling rates and finishes after 15 days, capturing a single image from each specimen. (4) Once the images were acquired, a database with a total of 900 images was generated. (5) The algorithms of image processing and color analysis were applied to this dataset. 6) Finally, the algorithms generate results with color segmentation, colorimetric measurements in CIELAB color space, color analysis, and statistical analysis of the dataset.
For this case of study samples of Habanero chili were harvested from an aquaponic greenhouse culture. The selection of the samples was carried out by a specialized technician. The expert harvested a representative group of Habanero chili, which showed different stages of maturity. A color categorization was performed by visual inspection. The samples were separated into three groups of colors (green, yellowish and orange), with 20 specimens per group.
4.2.2. Setup of machine vision system
The artificial vision system used a color CCD camera as image capture device. The lighting system contained fluorescent lamps mounted on top to avoid shadows. In order to control the camera and to download the images, we used a standard PC with MATLAB. The system design allowed the camera to move the samples above to keep them stable. Because of the protrusions and cavities presented by the Habanero chili, it is difficult to keep the surface of interest in exactly the same position along the experiment. As shown in Figure 3, the samples were placed in the confined space of the machine vision system to acquire the corresponding images.
4.2.3. Image processing algorithms
To accomplish the image processing tasks, the flow chart was followed as showed in Figure 4. The algorithms are executed using suitable functions in order to get reliable information from digital images. At the beginning, the image from sample is captured using the image acquisition system. Then, image processing routines carry out the required operations to separate the ROI, as well as to convert the image from RGB color space to CIELAB. The calculations to obtain the CIELAB components (L*, a*, b*, Chroma, Hue angle) using tristimulus colorimetry are performed to generate the corresponding array. Finally, the color analysis from the Habanero chili in postharvest conditions can be accomplished.
The original and processed images, by the algorithms from the dataset, are shown in Figure 5. One image, aleatory selected, from each categorized group (green, yellowish, and orange) is presented. At the top, you can identify the categorization of groups, and below is the corresponding label of the sample. Then, the original images, just as the vision system acquired them, are presented. In the middle, the label that indicates the exact day of the acquisition is located. At the bottom, the processed images are shown, where you can clearly observe the segmentation process and how the algorithms filled the background with zeros, and only the information from the region of interest is processed (ROI).
4.2.4. Color and statistical analysis
Habanero chili is a climacteric fruit, which means that once it is cut, it begins to ripen. Depending on the variety of Capsicum chinense Jacq., the color of the Habanero chili changes during maturation. In general, changes occur from the dark green, in its initial stages, and then go through the yellowish-green until reaching an orange color, in the final stage of maturity.
Color evolution can be represented as hue angles (hab°) in a polar graph. Typically, Habanero chili initiates maturity with green stage (under the threshold of hab = 120° for green) and then it moves through the yellowish stage (between the threshold hab = 120 and 60° for yellowish) to achieve orange colors (crossing the threshold of hab = 60° for orange) in the final stage of maturity, as time passes. The color information from the dataset showed this behavior, and the statistical analysis presented in Figure 6 demonstrates that image processing and colorimetry are capable of extracting reliable values from acquired images and detect color changes from these agriculture products. Therefore, this methodology described, as noncontact technique, can be considered a suitable option to analyze the color of Habanero chili.
In Figure 6, chart (a) shows the variation from the green group. In average values of hue angle, a high color change is presented, due to the gradual transition from green colors to oranges passing through the yellowish ones during their maturity process. In chart (b), a descendant gradual color change can be appreciated, even when crossing the threshold of hab = 60°. Instead, the third group (c), corresponding to the orange ones, remains after the threshold of hab = 60° with a slow progressive color changes.
A one-way ANOVA was conducted to evaluate the relationship between color changes in 20 Habanero chili and 15 days of sampling. Table 2 displays the summary for the one-way ANOVA for each group of samples. With the null hypothesis (H0), it was showed that all the color values are equal during 15 days. In the analysis, it was shown that in green, yellowish, and the orange groups, the ANOVA was significant: F(14, 285) = 8.17, F(0.05) = 1.692, F(14, 285) = 100.21, F(0.05) = 1.692, and F(14, 285) =8.17, F(0.05) = 1.692, respectively. The ANOVA results allowed to reject the null hypothesis and supported the conclusion that there is a statistically significant color change during days for the green, yellowish, and orange groups.
|Source||SS||DF||MS||F > F0.05|
|Days||91722.7||14||6551.62||8.17 > > 1.692|
|Days||49264.8||14||3518.91||100.21 > > 1.692|
|Days||1915.77||14||136.841||14.6 > > 1.692|
Artificial vision systems combined with image processing and color analysis are a reliable and affordable option when specific applications require the use of noninvasive and noncontact techniques. Similar characteristics of the samples are extracted from their images and grouped for further analysis using image processing techniques, which helps to obtain consistent and reliable separations of elements. The CIELAB color space provides the parameters needed to analyze and calculate important characteristics of a color image. Color differences can be detected more directly using the CIELAB color space, and it is important to mention that a color difference magnitude can be imperceptible to the naked eye, but it is a basic operation for the vision systems. However, the context in the color analysis should be considered as an important factor, for the proper interpretation of the data generated from the previous process. For example, the color attribute of the Habanero chili is a fundamental parameter for the appearance of the genus capsicum, which can be evaluated by image processing and colorimetry to detect color changes with adequate and reliable results in postharvest analysis. Trend in applications of color analysis and image processing for agriculture will continue to increase in the near future, due to the great variety of colors and shapes of the products, in particular, the interest to obtain the best quality.
The authors wish to thank the National Council for Science and Technology (CONACyT) for providing the funding to carry out this research, as well as to the people involved in this research at the State University of Sonora, the Autonomous University of Baja California, Program for Strengthening Educational Quality (PFCE) and the Center for Scientific Research and Higher Education at Ensenada.