Machine Vision to Determine Agricultural Crop Maturity

Agriculture in Malaysia is still one of the biggest enterprise and the most important sector in Malaysia’s economy. In the last 35 years, Malaysia has emerged as the number one producer of palm oil. In year 2010, Malaysia accounts for about 55% of world palm oil production and about 62% of world exports. The Malaysian oil palm industry continues to contribute significantly to the country’s economic development and foreign exchange earnings. However, the Malaysian oil palm industry needs more emphasis on its research and development to meet the world challenge and to maintain Malaysia as the top world producer of palm oil. The major problem faced by the oil palm plantation is shortage of labor, especially in harvesting of oil palm FFB (MPOIP, 2007).


Introduction
Agriculture in Malaysia is still one of the biggest enterprise and the most important sector in Malaysia's economy.In the last 35 years, Malaysia has emerged as the number one producer of palm oil.In year 2010, Malaysia accounts for about 55% of world palm oil production and about 62% of world exports.The Malaysian oil palm industry continues to contribute significantly to the country's economic development and foreign exchange earnings.However, the Malaysian oil palm industry needs more emphasis on its research and development to meet the world challenge and to maintain Malaysia as the top world producer of palm oil.The major problem faced by the oil palm plantation is shortage of labor, especially in harvesting of oil palm FFB (MPOIP, 2007).
In 1994, the Palm Oil Research Institute of Malaysia (PORIM) established four classes of ordinary oil palm FFB belonging to Elaeis Guineensis species.The classes of FFB arranged in the ascending degree of ripeness are the unripe, under-ripe, ripe and the over-ripe categories.The unripe bunch has purplish black colored fruits, covering more than 90% of the bunch surface.Meanwhile, fruits belonging to underripe and ripe bunches appear reddish orange or purplish red and reddish orange respectively.Finally, the bunch belonging to the over-ripe class with more than 80% of the fruits in the bunch appears darkish red.In year 2000 the authors used camera vision to categories six oil palm FFB.Camera vision was used to investigate the relationship between oil content in oil palm fruit and its surface color distribution.Colorimeter was used to determine the correlation between mesocarp oil content and color values.The above research indicate that there is a direct relationship between the color of the oil palm fruits and its grade and quality (Wan Ishak, et al., 2000).The advantage of predicting the oil palm fruit maturity in real time oil palm plantation will help harvesters to immediately harvest the oil palm FFB at the correct maturity stage.Harvesting oil palm fresh fruit bunches (FFB) at the right stage of ripeness is critical to ensure optimum quality and quantity of oil production, and thus profitability to the industry.Camera vision techniques can be implemented to automate the assessment and increase its consistency.replace the human eye for matured FFB recognition.In real plantation environment, the variations of the daylight caused changing the light intensity that effect on automatic recognition process.Colour is a property of waves (light waves) and the ability to see light as colour is determined by changes in the frequency or wavelength of light.The eyes interpret the incoming light into three colours and finally the three colours are received by the brain and re-interpreted as the complete spectrum in the form of a colour circle.Human perception of colour is a function of the response of three types of colour, that are blue, green and red (Gonzalez and Woods, 2002).Colour is considered a fundamental physical property of agriculture products and foods.There are common indicators used to recognize ripeness of the agricultural products and thus determine the best time to harvest the products.With the aid of modern technology, automated device with intelligent computing functions has been proposed to be used to replace human naked eye in deciding fruit maturity for harvesting time.

Machine vision system
Vision is the most powerful sense.It provides us with a remarkable amount of information of our surroundings and enables us to interact intelligently with the environment.Machine vision applications for automated inspection and sorting of fruits and vegetables have been studied and reported.Machine vision is a technology that employs a computer and camera to analyse and interpret images in a manner resembling human vision.The camera is equivalent to the human eye and the computer is equivalent to the human brain.Vision system is a new field of research in the agricultural sector.In agricultural applications, especially for fruit handling, we cannot detect fruit quality just by its shape or pattern.This is because a fruit may have a different shape and pattern but of the same level of quality.To solve this problem, the vision system should be able to analyse the colour of the object or fruit.Colour vision systems have been found more effective in colour inspection.A colour camera output can be decoded into three images to represent the red, green and blue (RGB) components of the full image.The three components of the colour image can be recombined in software or hardware to produce intensity, saturation and hue images, which can be more convenient for subsequent processing.Besides imaging objects in the visible (VIS) color region, some machine vision systems are also able to inspect these objects in light invisible to humans, such as ultraviolet (UV), near-infrared (NIR) and infrared (IR).The information received from objects in invisible light regions can be very useful in determining preharvest fruit maturity, disease or stress states and very useful in determining plant and vegetable variety.It is also useful in detecting postharvest quality and safety, such as defects, composition, functional properties, diseases and contamination of plants, grains and nuts, vegetables and fruits, and animal products

Oil Palm FFB ripeness
Figure 1 shows images of oil palm fresh fruit bunches of different degrees of ripeness from the five years old tree of Oil Palm Pasifera Variety.The top left is unripe, top right is under ripe, bottom right is ripe and bottom left is overripe.It can be seen from these images that the colour of each group is highly non-uniform which can be cluster to determine the maturity stages of fruit.The Pasifera (D X P) variety contains more oil in mesocarp layer with thin kernel compared to dura and tenera varieties.Optimum ripeness of oil palm fresh fruit bunches (FFB) is very crucial to the oil palm purchasing centers and palm oil mills.Presently the oil palm FFB are graded manually using visual human judgement of fruit bunch grader.The definition of matured colour oil palm FFB differs between planters and sellers due to the fact that human eyes perceive colours differently.The right stage of ripeness is critical to ensure optimum quality and quantity of oil production.The critical issue in harvesting the oil palm FFB is to harvest at the right stage of ripeness for ensuring that the maximum quantity and quality of palm oil is present for extraction of oil at the palm oil mills.If the oil palm FFB is harvested too early, the bunch will be too young and may not reach optimum oil content when processed while if it is harvested at overripe stage, the oil will contain high acid which will decrease the oil quality.Over ripe oil palm FFB is indicated by the number of loose fruits being detached from the bunches.The number of loose fruit detached from the bunches will increase as the oil palm FFB getting older or overipe.Uncollected loose fruits account 3-5% of bunch weight which resulted on further reduction in the oil extraction rate and profits.The time of harvest for the matured oil palm FFB is determined by the colour of the fruits by the naked eyes.The skin color of the fruit bunch will change from black to reddish and orange during ripening process.The change in the mesocarp color is due to the accumulation of the carotene pigments, which also corresponds to the oil content of the mesocarp when analysed.A fruit bunch normally takes 20 to 22 weeks to ripen and those less than 17 weeks old could be classified as unripe or black bunches (Kaida and Zulkifli, 1992).The current practice of harvesting the oil palm FFB is when about 10 loose fruits detached from the bunch when matured and found fallen on the ground.The Malaysian Palm Oil Board (MPOB) manual on FFB grading defines a ripe bunch when the mesocarp colour is reddish orange and the bunch has 10 or more empty sockets of detached fruitlets; an under ripe bunch has yellowish orange with less than 10 empty sockets; and an unripe bunch has yellow mesocarp with no empty sockets.Thus, the conventional method of determining on the 10 number of loose fruits fallen on the ground before harvesting the FFB is not practical and uneconomical.Thus, it is ideal to have a machine or device with artificial vision system replacing human eye to recognize level of oil palm fruits ripeness in order to determine the right harvest time.This will prevent the lost of loose fruits being detached from the FFB before harvest.

Development of crop colour model
Application of machine vision systems in outdoor fields addressed special difficulties of an unstructured work environment.Variability in light source temperature causes changes in the colors of objects in the image, making incorrect digital image value.Most light sources are not 100% pure white but have a certain color temperature, expressed in Kelvin.For instance, the midday sunlight will be much closer to white than the more yellow early morning or late afternoon sunlight.The image sensor sensitivity (ISO) was set at normal 100 which was experimentally suitable for capturing the FFB image under oil palm canopy in plantation.This is where the concept of white balance comes in.If we can tell the camera which object in the room is white and supposed to come out white in the picture, the camera can calculate the difference between the current color temperature of that object and the correct color temperature of a white object and then shift all colors by that difference.The white balances of the cameras were to be set in this experiment.The standard white calibration CR-A74 plate was used to set the white balance.
The shutter speeds determine how long the film or sensor is exposed to light.Normally this is achieved by a mechanical shutter between the lens and the film or sensor which opens and closes for a time period determined by the shutter speed.Slow shutter speeds let more light strike in the image sensor, so an image is lighter, while fast shutter speeds let less light strike it, so an image is darker .The FFB images were monitored and tested on different shutter speeds in actual oil palm plantation.Shutter speed of 0.125 s gave a good view for an image in all variability of lighting intensity on the farm.
Extech instrument datalogging light meter 401036 was used to capture the lighting intensity that affected the image captured during data collection process.When record button was activated, the intensity value (in foot candles unit) was recorded automatically into datalogger.For this study, the intensity was recorded under the canopy of oil palm tree.
The visual basic programming language version 6 was the platform to develop the software analysis.It contains the various components of functions to make an Excel worksheet programming software and populate it with sample scientific data.The API (Application Programming Interface) called GDI32, which stands for Graphic Device Interface: 32-bit version was applied and the hue pixel value was determined by the total value (0-255) assigned for each colour bands of red, green and blue as calculated here: where R is red component pixel value, G is green component pixel value and B is blue component pixel value.
The embedded histogram application in the software analysis is a function to determine the maximum R,G and B pixel value of captured FFB images.These R, G and B values will then be converted to hue value which is assigned as an average value for overall FFB images.
Nikon coolpix 4500 digital camera (Nikon, Japan) was used to capture the fruit image and store it digitally.It was equipped with a lens type Nikon 3x Tele Converter TC-E3ED.Minolta MPOB colorimeter CR 10 (Minolta, Japan) was used to determine the ripeness of oil palm fruit based on mesocarp surface color.This colorimeter equipment was used to validate the ripeness of the oil palm fruits after determining the mesocarp oil content using Soxhlet extraction process.
Firstly, the image was captured using Nikon digital camera and the pictures were automatically saved in its local memory.The images of fruits were captured with highest optical zoom with 155 mm focus length.The size of image is 640 x 480 pixels screen to suit with the size of image analysis.At the same time, the lighting intensity of environment was recorded.The steps were repeated with days interval from immature fruits until the fruits were overripe.The distances between cameras and lightmeter with fruits were fixed where the camera covered under palm tree canopy, within 2-3 m from oil palm tree.

Determination of oil content
The samples of fruits from different maturity stages were harvested from 5, 16 and 20 years old oil palm trees.The samples were weighed and chopped.The samples were dried at 70°C for a day to remove the water in the fruits.The dry nuts and mesocarp were weighed and blended before packed into filter papers (Whatman Cat No. 1001 150).The oil was extracted in Soxhlet extractor using chemical solvent, namely hexane.The remnant fibre and thimble were dried under 70°C for a day to remove the remaining hexane.The samples were weighed.All reading was inserted to automatic embedded calculation for oil to dry mesocarp ratio.Figure 3 shows the bunch analysis procedure to get oil extraction rate (OER).To determine the mesocarp oil content, the fruits were collected instead of whole bunch.

Fig. 3. Bunch Analysis Step Procedure for Oil Extraction
The hue pixel values of different fruits maturity were captured and analysed based on outdoor light condition of environment.The relationship of mesocarp oil content with optical properties for fruit color of hue was illustrated using trendline analysis of polynomial second order method.The results showed that the hue value of FFB image was highly significant in determining the oil content of oil palm fruit: y = -0.0116x 2 + 5.2376x -514.88;R 2 = 0.884 where y = mesocarp oil content, x = hue value, R 2 = coefficient of determination.The value of R 2 = 0.884 was acceptable and showed highly correlation between hue value and the oil content of FFB.The oil content of fruits increased with increasing hue value of fruit skin image.The mesocarp oil content will be maximum at an average of 75% which is similar to

www.intechopen.com
Machine Vision to Determine Agricultural Crop Maturity 121 earlier findings (Mohd Hudzari, 2010).Figure 4 shows the graph of the relationship between the mesocarp oil content and their respective hue digital values.For individual fruitlet, the hue value increases at overripe stage when it exhibits almost entirely reddish orange coloration.At overripe stage, the mesocarp oil content decreased or was stagnant due to reaction of free fatty acids (FFA) on FFB, which affected the quality and quantity of oil.Similar findings were reported on chemical changes on overripe olive oil which affect the quality and quantity of an oil.

Prediction of harvesting day
Figure 5 illustrates the graph to determine the number days of harvesting the oil palm FFB.The graph was developed through the experiments as described above.
The suitable days for harvesting were calculated based on the above equation of y = -0.0116x 2 + 5.2376x -514.88.Let say the hue value from the camera is 200, so the oil content found using the above equation is: y = -0.0116(2002 ) + 5.2376(200) -514.88 = 68.64.The mesocarp oil content for hue of fruit color at 200 is 68.64%.
From the above graph of Figure 5, the matured fruit was found to have hue of 224 with 74.22% oil content which indicated the day for harvest.The unripe fruit with measured hue of 158 has an oil content of 24.44% and indicates a 63 days before the fruit began ripening.Triangulation method was used to determine the harvesting day for FFB which is illustrated by the equation below.
The day estimation to harvest the FFB model is described as below: where J is the initial mesocarp oil content %, K mesocarp oil content %, L matured mesocarp oil content %, J and L are obtained by the chemical analysis in the laboratory, M is the initial day of observation, N is the day calculated and O is the day to harvest the FFB, which was set to 0. The suitable harvesting day is when the mesocarp oil content of FFB is at matured stage or ripe.
Using triangulation method, from 68.64%, the number of days when fruit needs to be harvested will be calculated as:

Real-time oil palm FFB maturity prediction model
The computer-implemented method for recognizing ripeness level of oil palm fruits uses optical properties of the oil palm fruit in determining ripeness with good accuracy.The method is computer-implemented and less likely to introduce any perceptive biases in contrast to human eye.This method provides a non-destructive approach to determine oil content of the oil palm fruits.The computer-implemented method of predicting oil content of oil palm fruits comprising the steps of providing a digital image of the oil palm fruits; acquiring average pixel value of RGB of the digital image; deriving average hue value of the digital image from the acquired average value of RGB; fitting the derived average hue value into an established model to obtain a predicted oil content of the oil palm fruits; or matching the derived average hue value to a database containing a range of preset hue value to obtain the predicted oil content; wherein each of the preset hue value is tagged with respective predetermined oil content.Subsequently, the digital image is processed to generate average pixel value of RGB.Particularly, a colour histogram preferably used as the approach to determine the maximum RGB and average RGB value as well as generate a RGB-colourspace.The histograms are used as the tool for graphing the frequency of intensity of red, blue, green and luminance of the image.Further, the RGB-colourspace is then transformed into a HSI colourspace which describes colour of the image into hue (average wavelength), saturation (amount of white), and intensity.The hue value acquired from the conversion to the HIS system is subjected to less influence of the light intensity of the images as intensity and hue are two separated information in the HIS system.This feature allows the image taken by any common available camera system can be straightly analyzed to acquire the predicted oil content without adjusting the light intensity before or after capturing of the images.
The above model of y = -0.01x 2 + 5.2x -514.8;where y = oil content of the oil palm fruits, and x = hue value.Through substituting the derived hue value, the model give a predicted oil content of the oil palm fruits captured in the image.More preferably, the average hue value is of HIS coordinate system by conversion of the average RGB value.
To avoid potential error in the produced result due to lack of significant lighting, the system may further comprise a light sensing mean to detect light intensity of surrounding environment when acquiring the digital image of the oil palm fruits and prompt the user of insufficient light intensity upon detecting the light intensity of below 500cd.More preferably, the light intensity is in the range of 500 to 8000 cd.Any light intensity further below or higher may lower accuracy of the method in predicting the oil content.The above mentioned light intensity range of 500 to 8000 cd is the practical light intensity for any image taken under the palm tree canopy.More specifically, the derived average hue value of the related digital image is placed into a plotted graph of average hue value against oil content that the graph is pre-established based on data collected from a series of experiments.Further, the location of the derived average hue value of the related digital image on the pre-established graph can be used to refer corresponding oil content of the mesocarp on the graph.
Apart from that, the method may include a step of acquiring predicted number of days left for optimum harvesting of the oil palm fruits by associating the predicted oil content to another model of relative ripeness of oil palm fruits correlated to oil content.To determine the best harvesting time, the method may use a triangulation method which can be deemed as the model of relative ripeness of oil palm fruits correlated to oil content.The second model is summarized as the following, N = M (J-L/K-L), where N is the predicted day left, K is the predicted oil content in percentage based on derived average hue value, L and J are prefixed values for optimal oil content in percentage and initial oil content in percentage www.intechopen.comTrends in Vital Food and Control Engineering 124 respectively.Entering the predicted oil content from the above mentioned two approaches into this model, the number of days left for optimum harvesting is predicted.The graph approach is used to generate the predicted oil content.More specifically, data collected from experiments are used to plot a graph of oil content against relative ripeness.Upon locating the position of the predicted oil content generated from the entered digital image on the plotted graph, the predicted relative ripeness can be referred from the graph, while the difference in between the predicted relative ripeness and the ripeness of optimal oil content in the graph is used to infer the days left for optimal harvest.With the generated information, users are able to harvest the oil palm fruits with optimal oil content thus obtaining greater profits.Furthermore, the predicted number of days left for optimal harvest is associated to the digital image.The method may have another step of categorizing the oil palm fruits into a group of unripe, ripe or overripe based on the derived average hue value and/or the relative ripeness acquired.

Conclusions
These experiments were conducted to determine and model an equation between the hue optical properties of the oil palm fruits at different stages of maturity after considering the affecting outdoor environment intensity in oil palm plantation.The maturity stages were confirmed by determining its mesocarp oil content.This study was carried out on selected immature fruits and monitored with days interval until one loose fruit found which indicated matured FFB.The factor of lighting intensity under canopy of oil palm plantation was not affected for image value which remains considerable using hue colorspace with setting camera parameters.The radiation from the sunlight was intercepted by the oil palm canopy by inter cropping systems for growth resources of solar radiation and caused the intensity decrease.The procedure in monitoring the image pixel value of different maturity stages for real oil palm fruit contributed.Developed system can be applied for maturity detection on other fruits in Malaysia.However, different fruits will have different characteristics for their properties.

Figure 2
Figure 2 shows the Developed Software For Colour Modelling to predict the Luminance and RGB pixel value of the images.When histogram analyses were performed, the maximum reading from graph will indicate the luminance value of the image as shown on the right side of the figure.The Luminance pixel value was determined by the total value (0-255) assigned for each colour bands of red, green and blue.

Fig. 4 .
Fig. 4. The relationship of optical properties for fruit color versus mesocarp oil content.

Fig. 5 .
Fig. 5.The graph to determine the day of harvesting the FFB.