Open Access is an initiative that aims to make scientific research freely available to all. To date our community has made over 100 million downloads. It’s based on principles of collaboration, unobstructed discovery, and, most importantly, scientific progression. As PhD students, we found it difficult to access the research we needed, so we decided to create a new Open Access publisher that levels the playing field for scientists across the world. How? By making research easy to access, and puts the academic needs of the researchers before the business interests of publishers.
We are a community of more than 103,000 authors and editors from 3,291 institutions spanning 160 countries, including Nobel Prize winners and some of the world’s most-cited researchers. Publishing on IntechOpen allows authors to earn citations and find new collaborators, meaning more people see your work not only from your own field of study, but from other related fields too.
In modern agriculture, there is a high demand to move from tedious manual harvesting to a continuously automated operation. This chapter reports on designing a simulation and control platform in V-REP, ROS, and MATLAB for experimenting with sensors and manipulators in robotic harvesting of sweet pepper. The objective was to provide a completely simulated environment for improvement of visual servoing task through easy testing and debugging of control algorithms with zero damage risk to the real robot and to the actual equipment. A simulated workspace, including an exact replica of different robot manipulators, sensing mechanisms, and sweet pepper plant, and fruit system was created in V-REP. Image moment method visual servoing with eye-in-hand configuration was implemented in MATLAB, and was tested on four robotic platforms including Fanuc LR Mate 200iD, NOVABOT, multiple linear actuators, and multiple SCARA arms. Data from simulation experiments were used as inputs of the control algorithm in MATLAB, whose outputs were sent back to the simulated workspace and to the actual robots. ROS was used for exchanging data between the simulated environment and the real workspace via its publish-and-subscribe architecture. Results provided a framework for experimenting with different sensing and acting scenarios, and verified the performance functionality of the simulator.
visual servo control
chapter and author info
Redmond R. Shamshiri
Department of ICT and Natural Sciences, Faculty of Information Technology and Electrical Engineering, NTNU, Norway
Leibniz Institute for Agricultural Engineering and Bioeconomy, Germany
Ibrahim A. Hameed*
Department of ICT and Natural Sciences, Faculty of Information Technology and Electrical Engineering, NTNU, Norway
Department of Biological Systems Engineering, Center for Precision and Automated Agricultural Systems, IAREC, Washington State University, United States
Leibniz Institute for Agricultural Engineering and Bioeconomy, Germany
Traditional harvesting of fruiting vegetables for fresh market is a labor-intensive task that demands shifting from tedious manual operation to a continuously automated harvesting. In spite of the advances in agricultural robotics, million tons of fruits and vegetables are still hand-picked every year in open-fields and greenhouses (Figure 1). Other than the high labor cost, the availability of the skilled workforce that accepts repetitive tasks in the harsh field conditions impose uncertainties and timeliness costs. For robotic harvesting to be cost-effective, fruit yield needs to be maximized to compensate the additional automation costs. This leads to growing plants at higher densities which make it even harder for an autonomous robot to simultaneously detect the fruit, localize, and harvest it. In the case of sweet pepper fruit, with an estimated yield of 1.9 million tons/year in Europe, reports indicate that while an average time of 6 s per fruit is required for automated harvesting, the available technology has only achieved a success rate of 33% with an average picking time of 94 s per fruit . For cucumber harvesting, a cycle time of 10 s was proven to be economically feasible . Only in Washington State, 15–18 billion apple fruits are harvested manually every year. An estimated 3 million tons of apples is reported to have been produced in Poland in 2015 , out of which one-third are delicate fruits and are less resistant to bruising from mass harvester machines. Also in Florida, where the current marketable yield of sweet pepper fruits in open-field cultivation is 1.6–3.0 with potential yield of 4 lb/ft2 in passive ventilated greenhouses , manual harvesting is still the only solution. Therefore, development of an automated robotic harvesting should be considered as an alternative method to address the associated labor shortage costs and timeliness.
Research and development in agricultural robotics date back to 1980s, with Japan, the Netherlands, and the USA as the pioneer countries. The first studies used simple monochrome cameras for apple detection inside the canopy . Advances in the sensor technology and imaging devices have led to the employment of more sophisticated devices such as infrared , thermal  and hyperspectral cameras , or combination of multi-sensors  that are adopted with novel vision-based techniques for extracting spatial information from the images for fruit recognition, localization, and tracking. Examples of some of the recent achievements include automatic fruit recognition based on the fusion of color and 3D feature , multi-template matching algorithm , and automatic fruit recognition from multiple images . Unlike the industrial case, an agriculture robot has to deal with different arrangement of plantings size and shapes, stems, branches, leaves, fruit color, texture, and different location of fruits and plants with respect to each other. Significant contributions have been made by different research groups to address these challenges; however, there is currently no report of a commercial robotic harvesting for fresh fruit market , mainly due to the extremely variable heterogeneous working condition and the complex and unpredicted tasks involved with each scenario. Some of the questions to be addressed in designing of a complete robotic harvesting are the simultaneous localization of fruit and environment mapping, path planning algorithms, and the number of detectable and harvestable fruits in different plant density conditions. The function of a robot can be separated into three main sections as sensing (i.e., fruit recognition), planning (i.e., hand-and-eye coordination), and acting (i.e., end-effector mechanism for fruit grasping) . A common approach in fruit detection is by using a single view point, as in the case of a cucumber harvesting robot , or multiple viewpoints with additional sensing from one or few external vision sensors that are not located on the robot . Other than the issues with frame transformation, this solution is not promising if the fruit is heavily occluded by the high density plant leaves . Obviously, the final robot prototype needs to be relatively quicker for mass-harvest, with an affordable cost for greenhouse growers. Swarms of simple robots with multiple low-cost camera and grippers, or human-robot collaboration are the research topics to solve the facing challenges in robotic harvesting that current technology cannot overcome. These approaches can significantly improve the processing time of multiple fruit detection in the high-density plants, and provide ground truth results over time for machine learning algorithms based on human-operators experience. Research on agricultural robotics with a focus on automated harvesting of fruiting and vegetable are huge. See for example the works carried out on sweet pepper [1, 18, 19, 20], oil palm , cucumber [15, 22, 23, 24], apple , strawberry [26, 27], cherry fruit , citrus , and tomato . Most of these works have used eye-in-hand look-and-move configuration in their visual servo control (Figure 2). Other researches are concentrated on the end-effector design , analysis of the robot performance in the dense obstacle environments using stability tests , motion planning algorithms , and orchard architecture design for optimal harvesting robot . In addition, several software frameworks have been developed for agricultural robotics. An example includes the work of , in which a generic high-level functionality was provided for easier and faster development of agricultural robots. Some of the most recent advances in sensing for robotic harvesting include the works of [29, 35] which address the problem of detecting fruits and obstacles in dense foliage. Moreover,  and  have extensively explored the use of combined color distance, or RGB-D, data on apples and on sweet-peppers, respectively, while  present a study devoted to symmetry analysis in three-dimensional shapes for products detection on the plant.
Improvement of robotic harvesting requires experimenting with different sensors and algorithms for fruit detection and localization, and a strategy for finding the collision-free paths to grasp the fruits with minimum control effort. Experiments with the actual hardware setup for this purpose are not always feasible due to time constraints, unavailability of equipment (i.e., sensors, cameras, and the robot manipulator), and the operation costs. In the other hand, some hardware setups may result in actuator saturation, or create unsafe situation to the operators and/or plants system. Simulation offers a reliable approach to bridge the gap between innovative ideas and the laboratory trials, and therefore can accelerate the design of a robust robotic fruit harvesting platform for efficient, cost-effective and bruise-free fruit picking. This research was motivated based on the sensing task in robotic harvesting, which requires delivering a robust pragmatic computer vision package to localize mature pepper fruits and its surrounding obstacles. The main objective was to create a completely simulated environment for improvement of plant/fruit scanning and visual servoing task through an easy testing and debugging of control algorithms with zero damage risk to the real robot and to the actual equipment. The research was carried out in two main phases: (i) the creation of the simulated workspace in the virtual robot experimentation platform (V-REP), and (ii) the development of communication and control architecture using the robot operating system (ROS) and MATLAB (The MathWorks Inc., Natick, MA, USA). The simulated workspace included an exact replica of the Fanuc LR Mate 200iD robot manipulator with six degrees of freedom (Fanuc America Corporation, Rochester Hills, MI), models of sweet pepper fruit and plant system, and different vision sensors were created in (V-REP). A simulated color camera attached to the end-effector of the robot was used as fruit localization sensor. ROS was used for exchanging data between the simulated environment and the real workspace via its publish-and-subscribe architecture. This provides a tool for validating the simulated results with those from experimenting with a real robot. V-REP and MATLAB were also interfaced to create two-way communication architecture for exchanging sensors and robot control messages. Data from the simulated manipulator and sensors in V-REP were used as inputs of a visual servo control algorithm in MATLAB. Results provided a flexible platform that saves in cost and time for experimenting with different control strategies, sensing instrumentation, and algorithms in automated harvesting of sweet pepper.
2. Overview of the simulation environment
Computer simulation of a complete robotic harvesting task requires: (i) CAD file setup including good replications of the plants-and-fruit scene and the robot manipulators, (ii) simulation environment and calculation modules for the manipulator candidates and platforms (i.e., inverse kinematics and path planning), (iii) different sensors setup, and more importantly (iv) algorithms for control tasks such as visual servoing and gripper control mechanism. The main simulation environment, scene objects, and calculation modules were built in the latest version of V-REP Pro Edu V3.4.0 for Linux 64 (available atwww.coppeliarobotics.com), and ROS installed on Ubuntu 14.04.3 LTS. Some of the used terminal commands are summarized in Table 1.
Show the current date and time
Formatted listing with hidden files
Show this month’s calendar
Change directory to dir_name
Show current uptime
Change to home
Display who is online
Show current directory
Who you are logged in as
Create a directory dir_name
Display information about user
Show kernel information
rm -r dir_name
Delete directory dir_name
rm -f file_name
Force remove file
rm -rf dir_name
Force remove directory dir_name
Show the manual for command
cp file_name_1 file_name_2
Copy file1 to file2
Show disk usage
cp -r dir_name1 dir_name2
Copy dir1 to dir2;
Show directory space usage
mv file_name_1 file_name_2
Rename or move file_name_1 to file_name_2
Working with compressed files
tar xf file.tar
Extract the files from file.tar
Ctrl + Alt + T
Opens a new terminal window:
tar czf file.tar.gz files
Create a tar with gzip compression
Shift + Ctrl + T
Opens a new terminal tab:
tar xzf file.tar.gz
Extract a tar using gzip
Halts the current command
tar cjf file.tar.bz2
Create a tar with bzip2 compression
Stops the current command,
tar xjf file.tar.bz2
Extract a tar using bzip2
Log out of current session, exit
Compresses file and renames it to file.gz
Erases one word in the current line
gzip -d file.gz
Decompresses file.gz back to file
Erases the whole line
Install from source
Type to bring up a recent command
(For example, ./vrep.sh will run v-rep)
Repeats the last command
dpkg -i pkg.deb
Install a package (debian)
Log out of current session
rpm -Uvh pkg.rpm
Install a package (rpm)
List of the most used Ubuntu terminal commands used for navigating in the simulation environment.
ROS Indigo was used to provide a bidirectional communication (information exchange) between simulated robot and cameras with the real world. Experimental packages for Fanuc manipulators within ROS-Industrial (available athttp://wiki.ros.org/fanuc_experimental) were used for controlling the manipulator. This design allows reading information from the simulation scene (i.e., robot joints velocity, position, sensors, etc.) and publishes them across ROS network for further process. Results can be used by the simulation, and/or by the real robots and controllers. The image-based visual servo control was carried out in V-REP and MATLAB. For the sake of this chapter, we only provide a brief description of ROS and V-REP.
ROS is a collection of software frameworks for robot software development. It was originally developed in 2007 by the Stanford Artificial Intelligence Laboratory, and with the support of the Stanford AI Robot project. It provides a solution to specific set of problems encountered in the developing large-scale service robots, with philosophical goals summarized as: (i) peer-to-peer, (ii) tools-based, (iii) multi-lingual, (iv) thin, and (v) free and open-source . From 2008 until 2013, development was performed primarily at Willow Garage, a robotics research institute/incubator. During that time, researchers at more than 20 institutions collaborated with Willow Garage engineers in a federated development model. Since 2010, ROS has released several versions, including Box Turtle (March, 2010), C Turtle (August, 2010), Diamondback (March, 2011), Electric Emys (August, 2011), Fuerte Turtle (April, 2012), Groovy Galapagos (December, 2012), Hydro (September, 2013), Indigo (July, 2014), and Jade (May, 2015). The open-source ROS makes it possible to develop code and applications that can be shared and used in other robotic system with minimum effort. It also offers standard operating system features such as hardware abstraction, low-level device control, implementation of commonly used functionalities, message passing between processes, and package management. ROS Packages are files and folders that are built to create minimal collections of code for easy reuse. A ROS package usually includes the followings folders and files: bin, msg, scripts, src, srv, CMakeLists.txt, manifest.xml (Figure 3).
Fundamental concepts of the ROS are: Nodes, Messages, Topics, and Services. ROS works based on a “publish-and-subscribe” architecture where processes (called nodes) publish and/or subscribe to specific topic on which information is exchanged in the form of messages (Figure 3). A Node is an executable file that uses ROS to communicate with other Nodes. A Message is ROS data type that is used when subscribing or publishing to a topic. Nodes can publish messages to a Topic as well as subscribe to a Topic to receive messages. Service helps Nodes find each other. ROS nodes use a ROS client library to communicate with other nodes. Nodes can also provide or use a Service. With this architecture, each node in ROS is able to respond to input and activate other nodes, allowing participation of a sequence of nodes to complete complicated robot mission tasks. Installation details and basic configuration of ROS environment, as well as installation and configuration of packages such as V-REP/ROS bridge, and the details of the Fanuc manipulator package are not in the concept of this chapter. A more detailed discussion can be found in .
V-REP is like a Swiss knife in robotic simulation community. Its first public release was in March 2010, and its latest version (V3.4.0 v1) was released on April 16, 2017. It possesses various relatively independent functions, features, or more elaborate APIs, that can be enabled or disabled as desired. Compared to gazebo, V-REP is very stable and easy to set up and running. For example, the vision sensors are reasonably well simulated and if the scene is not too complex, the run times of the simulations are generally good as well. If the project requires building a custom robot in the simulator (i.e., NOVABOT or Fanuc LR Mate 200iD manipulator), the setups for links, joints, and calculation modules such as inverse kinematics necessitates some practice, however, that is the case in any robot simulation software. Another big advantage is its true cross-platform, which means it can be run in Windows or Linux. By default, the V-REP distribution for Linux should be automatically ROS enabled based on ROS Indigo and Catkin. Each object/model in V-REP scene can be individually controlled via an embedded script, a plugin, a ROS node, a remote API client, or a custom solution. Controllers can be written in C/C++, Python, Java, Lua, Matlab, and Octaveor Urbi. The three main elements of V-REP simulator are scene object (i.e., joints, shape, sensors, path, etc.), calculation modules (i.e., inverse kinematics, collision detection, etc.), and control mechanism (i.e., scripts, plugin, sockets, etc.). In addition, V-REP inverse kinematics supports four different dynamic engines: The Bullet, ODE, Newton, and the Vortex Dynamics Engine. An overview of V-REP framework architecture is shown in Figure 4.
3. Image processing, publishing and subscription
Quantification of fruits to estimate the time required for robotic harvesting is an intensive labor task that is either ignored in high density greenhouses or is carried out manually by the use of hand pickers. We proposed a low-cost robust sweet pepper fruit recognition and tracking system using stream RGB images. Main hardware and software components of the system included a laptop computer (Lenovo Intel(R) Core(TM) i5-6200 U CPU@2.30GHz, RAM 8.00GB, 64-bit OS Windows 10), a Logitech camera (C920 HD Pro USB 1080p), supplementary halogen lamps, Adafruit Ultimate GPS breakout module 66 channel w/10 Hz (NY, USA), and Arduino Uno Microcontroller board. The image processing algorithm was implemented in MATLAB and applies median filter and image segmentation method to remove color noise from the RGB images of pepper fruits taken in the lab experiments at different angles, positions, and light conditions disturbances (varying illumination and overlapping). Figure 5 shows: (A) original image, (B–D) red, green, and blue bands, (E) mask of only red object, (F) regions filled, (G) masked-red image, (H) extracting red component from the masked red image and applying median filter to filter out the noise, (I) convert the resulting grayscale image into a binary image and removing all pixels with a gray level value less than 3000, (J) masked image showing only red-detected object, (K) blob analysis, bounding the red objects in rectangular box and showing centroid. The image processing algorithm was validated using 78 images obtained from lab experiments and internet sources, with a recognition success rate of 94% and average recognition time of less than 2 s per image. Results of the image processing were sent from MATLAB to V-REP for simulation of visual servo control. For the actual experiment, color images of sweet pepper were acquired under natural daylight condition in different greenhouse environment in the presence of the halogen lamps. Each band of the RGB image was transferred as a 24-bit, 640 by 480 pixels matrix and was processed in real time by the custom built MATLAB application on the laptop computer. ROS was used for exchanging data between the simulated environment and the real workspace via its publish-and-subscribe architecture. Another 57 images were obtained for experimenting with different fruit and plant position scenarios. In addition, internet searched images of sweet pepper taken at different greenhouse environments were used to verify the reliability and to improve the accuracy of the algorithm. The image subscription and publishing was performed by having V-REP ROS enabled based on ROS Indigo and Catkin build. The general ROS functionality in V-REP is supported via a generic plugin “libv_repExtRos.so” or “libv_repExtRos.dylib.” It should be noted that plugins are loaded when V-REP is launched, and the ROS plugin will be successfully loaded and initialized only if “roscore” is running at that time. The plugin is open source and can be modified as much as needed in order to support a specific feature or to extend its functionality. Three of the main ROS package folders in the V-REP, (located in programming/ros_packages) are the “vrep_common,” “vrep_plugin,” and “vrep_joy” as shown in the left side of Figure 6.
The first package was used to generate the services and stream messages that were needed to implement the V-REP API functions, while the second is the actual plugin that was compiled to a “.so” file used by V-REP. The “vrep_joy” package enables interaction with a joystick. Having the services and stream messages in a separate package allows for other application to use them in order to communicate with V-REP via ROS in a convenient way. These packages were copied to the catkin_ws/src folder. The command “$ roscd” was then used to check whether ROS is aware of these packages (e.g., $ roscd vrep_plugin). After navigating to the catkin_ws, the command “$ catkin_make” was used to build the packages and to generate the plugins. The created plugins were then copied to the V-REP installation folder to be used for image subscription and publishing. A new terminal was opened in Ubuntu for staring the ROS master using the command “$ roscore.” Another terminal was opened and was navigated to the V-REP installation folder to launch the V-REP simulator in Ubuntu by typing the command “$. /vrep.sh.” The entire procedure is summarized as these steps: (i) installing ROS Indigo on Ubuntu and setting up the workspace folder, (ii) copying “ros_packages” in V-REP into the “catkin_ws/src” folder, (iii) source “setup.bash” file, (iv) run “roscore” and “. /vrep.sh.” The two available nodes, “/rosout” and “/vrep” and the three topics “/rosout,” “/rosout_agg,” “/vrep/info” were checked using “$ rosnode list” and “$ rostopic list” commands, respectively. In addition, the command “$ rosservice list” was used to advertise all the services. It should be noted that the only V-REP topic that was advertised was “info” publisher that started as soon as the plugin was launched. All other V-REP topics for publishing and subscribing images and sensors were individually enabled using Lua commands: “simExtROS_enablePublisher” and “simExtROS_enableSubscriber.” Moreover, to visualize the vision sensor stream images and data, the “$ rosrun image_view image_view image:=/vrep/visionSensorData” and “$ rostopic echo/vrep/visionSensorData” were used, respectively. Snapshot of the simulation environment is shown in the right side of Figure 6.
4. Simulation scene and objects
Simulation scene in V-REP contains several elemental objects that are assembled in a tree-like hierarchy and operate in conjunction with each other to achieve an objective. In addition, V-REP has several calculation modules that can directly operate on one or several scene objects. Major scene objects and modules used in the simulation scene include (i) sensors, (ii) CAD models of the plant and robot manipulator, (iii) inverse kinematics, (iv) minimum distance calculation, (v) collision detection, (vi) path planning, and (vii) visual servo control. Other objects that were used as basic building blocks are: dummies, joints, shapes, graphs, paths, lights, and cameras (Figure 7). In this section, we provide description for the sensors and CAD models, and assign the next section to the calculation modules.
V-REP supports different vision sensors (orthographic and perspective type) and proximity sensors (Ray-type, pyramid-type, cylinder-type, disk-type, and cone- or randomized ray-type proximity sensors). It is possible to model almost any proximity sensor subtype, from ultrasonic to infrared. In addition it has built-in CAD models of several available commercial sensors such as Microsoft Kinekt, 2D and 3D laser scanners, blob detection camera, Hokuyo URG 04LX UG01, SICK S300, and TimM10 sensors. Other models can be built similarly based on combinations of different vision and proximity sensors. The V-REP model of each sensors used for this simulation is shown below its actual images in Figure 8 which include: Fish-eye RGB Axis 212 PTZ sensor (Figure 8A), Infrared Proximity Sensor Long Range-Sharp GP2Y0A02YK0F (Figure 8B), SICK TiM310 fast laser measurement scanner (Figure 8C), Fast Hokuyo URG-04LX-UG01 scanning Laser Rangefinder (Figure 8D), and Microsoft Kinect (Figure 8E).
The fish-eye RGB camera was added for fruit detection, tracking, and for visual servo control with a custom set of filters that were designed for the image processing algorithm in MATLAB and V-REP. Two color cameras were also added for tracking the scene and the position of the robot end-effector with respect to the fruit and plant in order to provide a wider view of the vision sensor. The V-REP model of the Microsoft Kinect sensor includes RGB and depth vision sensors, and was used in the scene to calculate the time needed for the laser signal to hit an object and bounce back to its source, creating in this way a three-dimensional representation of the object. Five different proximity sensors with different shapes were also experimented in the simulation, including: laser ray, pyramid, cylinder, disk, and randomized ray-type. The laser-scanner rangefinder was considered in the simulation to measure distance between an observer object (i.e., the robot gripper or the end-effector camera) and a target (i.e., fruit, plant, or obstacles). Typical range finders work based on time-of-flight (TOF) and frequency phase-shift technologies. The TOF method utilizes laser by sending a pulse in a narrow beam toward the object and measuring the time taken by the pulse to be reflected off and return to the sensor. The frequency-phase shift method measures the phase of multiple frequencies on reflection together with performing simultaneous math calculations to deliver the final measure. Rangefinders are available in V-REP in the form of vision-sensors and proximity sensors. For example, the Hokuyo URG-04LX-UG01 and the 3D laser scanner range finder use a ray-type laser proximity sensor. The V-REP model for Fast-3D laser scanner uses vision sensor with the filters as illustrated in Figure 9. It should be noted that vision-sensors-based rangefinders have high calculation speed but lower precision, while proximity-sensors-based rangefinders have higher prevision in calculating the geometric distance with relatively lower calculation speed.
4.2. CAD models
The CAD models of the sweet pepper plant, including stem system, leaves, and pepper fruits, as well as the single and multiple arms robot manipulators that were used in the simulation are shown in Figures 10–13. The Fanuc LR Mate 200iD robot manipulator shown in Figure 11 is a compact six-axis robot with the approximate size and reach of a human arm. It combines best-in-class robot weight-load capacity with standard IP67 protection and outstanding FANUC quality. This makes the Fanuc LR Mate 200iD the best and most reliable mini robot for process automation in many industries. The maximum load capacity at wrist = 7 kg, repeatability = 0.02 mm, mechanical weight = 25 kg, and reach = 717 mm. The joints motion range and maximum speed are summarized in the operator manual . As alternative innovative solutions, simple robots, including a platform with multiple linear actuators (Figure 12), and multiple SCARA robot arms (Figure 13) with multiple lower-cost cameras and grippers were also designed for simulation.
5. Calculation modules
In order to setup the robot manipulator for different experiment, several calculation modules, including minimum distance calculation, collision detection, path planning, inverse kinematics, and different control mechanism were used in V-REP. Snapshot of the calculation modules is provided in Figure 14. V-REP control mechanism are divided into (i) local interfaces, including Embedded scripts, Plugins, Add-ons, and (ii) remote interfaces, including remote API clients, custom solutions, and ROS nodes, as shown in Figure 14A. It should be noted that different V-REP control mechanisms can be used simultaneously in one scene, or even work in conjunction with each other, which provides a multipurpose and accessible framework for the purpose of more complex robotic simulation. Scripting in V-REP is in the Lua language which is a fast scripting language designed to support procedural programming. Scripts in V-REP are the main control mechanism for a simulation. For the sake of this book chapter, we only provide brief illustration of the inverse kinematic task for the NOVABOT manipulator and the visual servo control.
5.1. Inverse kinematics
The inverse kinematic (IK) task in V-REP requires three things: (i) CAD data of the manipulator links (ii) joints, (iii) kinematic chain, (iv) tip and target dummies, and (iv) IK task. The CAD file was imported to the scene from [Menu bar --> File --> Import --> Mesh]. It should be noted that depending on how the original CAD data was generated in the original CAD software, the imported mesh file could be at a different scale, different location, or even subdivided into several shapes. The assigned color of imported shapes is random. V-REP also provides basic tools and options for creating model of a new robot if the CAD file is not available from external sources. Upon importing the CAD file, a single simple shape is located in the middle of the scene and appears in the scene hierarchy on the left hand side of the main window. For the IK task, the single CAD shape was divided by selecting [Menu bar --> Edit --> Grouping/Merging --> Divide selected shapes]. This divided the original shape into several sub-shapes that were grouped manually for a same rigid entity using [Menu bar --> Edit --> Grouping/Merging --> Group selected shapes]. For example, all shapes that were related to the robot base were grouped together and renamed as robot_base in the scene hierarchy. It is usually easier to change the color of each shape for a better visual appearance and for selecting the shapes that belong to one group. In the case, when all shapes that were meant to be grouped shared the same visual attributes, they were merged together instead using [Menu bar --> Edit --> Grouping/Merging --> Merge selected shapes]. After the shapes were grouped in a compound shape, the robot joints that logically belong to a shape (robot link) were added into the scene using [Menu bar --> Add --> Joint --> Revolute] with their correct position and orientation specified. All joints were then set to the IK mode and were placed at the correct position. In case, when the exact joint positions were not known, they were extracted manually based on the position of the relevant shapes. It is often helpful to refer to the robot design manual for a better understanding of links and joints functionality for building the kinematic chain, going from tip to base. The IK task requires specification of the kinematic chain described with a “tip” dummy and a “base” object, and a “target” dummy that the “tip” dummy will be constrained to follow as shown in Figure 15. After all elements for the definition of the IK task were ready, the “target dummy” was selected as the linked dummy to the “tip dummy,” and the IK task was registered as an IK group with proper selection of calculation method (DLS or pseudo inverse), damping, and constraints (x, y, z, alpha-beta, gamma).
5.2. Visual servo control algorithm
A robot can be controlled in V-REP simulation through several ways such as child script, writing plugins, ROS nodes, external client applications that relies on the remote API, or writing an external application that communicates with V-REP plugin or script via pipes, sockets, or serial port. V-REP supports seven supported languages: C/C++, Python, Java, Matlab, Octave, Lua, and Urbi. In this research, we used MATLAB as the remote API because it provides a very convenient and easy way to write, modify and run image based visual servoing control codes. This also allows controlling a simulation or a model (e.g., a virtual robot) with the exact same code as the one that runs the real robot. The remote API functionality relies on the remote API plugin (on the server side), and the remote API code on the client side. Both programs/projects are open source (i.e., can be easily extended or translated for support of other languages) and can be found in the ‘programming’ directory of V-REP’s installation. Visual servo control scheme with eye-in-hand configuration, as shown in Figure 16, was implemented in MATLAB based on image moment method. For the case of the multiple linear actuators and the SCARA arms, we divided the camera view into multiple camera views to enhance the accuracy of the fruit detection algorithm and also to accelerate the image processing time (Figure 17). Details of the visual servo control algorithm are considered intellectual property of authors’ research group and are beyond the content of this chapter.
6. Results and discussions
Results provided a simulated environment for improvement of plant/fruit scanning and visual servoing task through easy testing and debugging of control algorithms with zero damage risk to the real robot and to the actual equipment. It also contributed to experimenting new ideas in robotic harvesting of sweet pepper, as well as testing different sensing instrumentation and control strategies on the currently used manipulators. Three groups of experiments, with separated V-REP scenes were designed for investigating different algorithms, robot manipulator, and sensors setup. They are summarized as experimenting with: (i) fruit detection and tracking algorithm using different camera views (Figures 18 and 19), (ii) manual and automated plant/fruit scanning in x-y, x-z, and y-z plane, and x-y-z space (Figures 20 and 21), (iii) fruit/plant scanning using Kinect, Hokuyo, fast 3D Laser, proximity 3D Laser scanner, and proximity Hokuyo URG04LXUG01 sensors (Figure 22), and (iv) visual servoing control law on single (Figure 23) and multiple (Figure 24) robot manipulator. Depending on the objectives of each scenario, sensors were placed in fixed spots, or on a moving link of the robot such as the end-effector. For example, the RGB vision sensor for fruit detection and tracking was used as eye-in-hand configuration with end-point closed-loop control. For the manual fruit/plant scan experiment with RGB sensors, the robot joints were controlled via sliders or by directly entering the desired joint angles in each label box as shown in Figure 20. This enabled sensing from the gripper from multiple viewpoints. In order to provide an interface with real workspace, two 2-axis analog Joysticks with push button were then used with Arduino Uno microcontroller to manually control angular positions for the joints. The automated fruit/plant scan experiments with RGB sensor were also carried out in different x, y, and z direction. The objective from this experiment was to simulate various camera pose and views for the best fruit attack and harvest. For Scanning in x-y plane, a 360° scan configuration of the fruit in the horizontal x-y plane is shown in Figure 20, with 30° increment snapshots of the simulated fruit. A similar scanning has been employed by . For scanning in x-y-z space, two scan configurations in x-y-z space were used with snapshots of the resulting camera view shown in Figure 21. In this setup, the RGB sensor mounted on the robot tip is moved on the horizontal plane x-y to find the best view of the fruit. Moreover, the manipulator is “twisted” to provide different viewpoints for the end-effector camera.
The “3D Laser Scanner Fast” sensor model in V-REP is based on vision-sensor with a perspective angle equal to 45°, a resolution of 512 by 512 and minimum and maximum 0.05 and 5 m distance of operation. Snapshot of the experiment with this sensor is shown in Figure 22. The “Fast Hokuyo URG-04LX-UG01” model in V-REP also works in perspective mode with an operability angle equal to 120°, and a resolution that was set at 512 by 1 which means it scans along a line shown on the floating view. It has a minimum and maximum distance of operability, respectively, equal to 0.04 and 5. The image processing in this case is similar to the 3D laser sensor except that the intensity map scale component is omitted. This sensor in fact does not come with any floating view by default. Two floating views were added for the two vision sensors of the “Fast Hokuyo URG-04LX-UG01” model. The black line inside the floating view of each sensor indicates presence of object (i.e., fruit, leaf, robot, and plant). First row images in Figure 22 are snapshot of the scene with Kinect depth sensor in action for fruit/plant scan, and the bottom row images are, respectively, from left to right are: snapshot of the scene with vision sensors, showing the “3D Laser scanner Fast,” the “Fast Hokuyo URG-04LX-UG01,” snapshot of the scene with proximity sensors showing the “3D-Laser scanner,” and the “Hokuyo URG-04LX-UG01” scanning scene.
For the visual servo control task, a robot end-mounted camera was used to position the robot arm in a plane orthogonal to the axis, such that the fruit to be harvested is centered in the camera’s field of view. The system had no trajectory generator, instead a feedback loop closed visually, was used to control the robots arm position. The sensor and robot was programmed for visual servoing task in such a way that the end-effector tracks the largest detected fruit until maximum possible view of that fruit is provided. Two different control approaches was designed and tested, one based on joint velocity control and the other based on joint position control. In both design a PID control law was implemented to minimize the offset error between image position of a detected fruit and center of the camera frame. Results showed that the robot could adjust itself in such a way that its tip RGB sensor shows maximum possible view of the largest detected fruit and become stable in a short time. It should be noted that both control algorithms were designed and tuned based on the experiments and statistical results from fruit/plant scan. Video demonstration of the entire experiments can be accessed from the links listed in Table 2.
Video demo link
Simulation of NOVABOT innovative manipulator
Simulation of multiple SCARA arms
Simulation of multiple linear actuators
Robotic Harvesting with Fanuc LR Mate 200iD
Simulation of Environment mapping and scanning
Detailed demonstration of fruit and plant scan
Detailed demonstration of visual servo control
Testing Visual Servo Control Algorithm
Environment Setup: Ubuntu, V-REP, ROS
Real-time, robust and rapid red-pepper fruit detection
Links to the video demonstrations.
As the final discussion, we would like to highlight that agricultural robotic is a part of the big picture in the future production of vegetable and crops, i.e., growing plants in space. An example includes space research for development of Mars greenhouses to produce vegetables during a mission to Mars or at Antarctica. The trend in food production is toward urban farming techniques, compact Agri-cubes, and automated systems with minimum human interface. The idea is that even people with limited experience/knowledge in vegetable cultivation can operate these units. While this integration might seem too ambitious, it can serve as a prophetic awareness for a perceptive outlook in the farming system. For example, the conventional arrangements of citrus groves, orchards, and the trees shapes in Florida had to be reconsidered for the mechanical harvesting machines to operate successfully. It is likely that the greenhouse planting systems for sweet pepper will also be re-shaped to match with a customized robotic platform. Two of the key challenges to be solved during the design of robotic harvesting framework are addressed by  as (i) detection of a target location of the fruit, and (ii) moving the end-effector toward that location with precision for harvesting task. We argue that these two challenges have not been stated accurately. First of all, it is not always necessary to detect the target location of the fruit, especially in the case of a mass harvesting platform with shaking grippers. Second, moving the end-effector toward the target fruit is not a scientifically sound statement, e.g., considering the strategy case in which the plant system is moved toward a fixed end-effector. To avoid this divergence, the task should have been stated as minimizing the error between the location of the end-effector and the target fruit. In fact, a promising solution to the robotic harvesting is not through a single robot manipulator. We provided a quick review of the previous works, and used simulation approach to reveal that single arm robots for harvesting are still far beyond realization, and have failed mainly due to the sensing and moving action in high vegetation density. In this approach, even if the fruit localization is accurate, and the robot control calculates an optimum trajectory to reach the fruit without receiving additional sensing feedback from the camera, the moment it enters into the dense plant canopy it disrupts the exact location of the target fruit.
Research and development for the use of robotics in agriculture that can work successively have grown significantly in the past decade; however, to this date, a commercial robotic harvesting is still unavailable for fresh fruit market. With the decrease of greenhouse workforce and the increase of production cost, research areas on robotic harvesting have received more and more attention in recent years. For the success of robotic harvesting, the identification of mature fruit and obstacle is the priority task. This chapter reported on a simulation and control platform for designing, testing, and calibration of visual servoing tasks in robotic harvesting of sweet-pepper. Creation of a virtual environment was carried out as a response to the improvement of fruit detection rate. We provided a documented guideline for a reliable, cheap and safe experiment platform with a faster approach for development, testing, and validating control strategies and algorithms to be used with different robot candidates and gripper mechanism in automated harvesting of fruiting vegetables. Results of the image processing confirmed that the presented approach can quantify and track mature red sweet pepper fruits from its surrounding obstacles in the real-time. It can be concluded that development of an affordable and efficient harvesting robot requires collaboration in areas of horticultural engineering, machine vision, sensing, robotics, control, intelligent systems, software architecture, system integration, and greenhouse crop management. In addition, practicing other cultivation systems in the greenhouse, such as single row, might be necessary for overcoming the problems of fruit visibility and accessibility. It can also be concluded that human-robot collaboration might be necessary to solve the challenges in robotic harvesting that cannot yet be automated. In a collaborative harvesting with human-robot interface, any fruit that is missed by the robot vision is identified by the human on the touch screen, or the entire robot actions are controlled manually in a virtual environment. Nevertheless, robotic harvesting must be economically viable which means it must sense fast, calculate fast, and move fast to pick a large number of fruits every hour that are bruise free.
Redmond R. Shamshiri, Ibrahim A. Hameed, Manoj Karkee and Cornelia Weltzien (March 14th 2018). Robotic Harvesting of Fruiting Vegetables: A Simulation Approach in V-REP, ROS and MATLAB, Automation in Agriculture - Securing Food Supplies for Future Generations, Stephan Hussmann, IntechOpen, DOI: 10.5772/intechopen.73861. Available from:
Frontiers in Guided Wave Optics and Optoelectronics
By Bishnu Pal
We are IntechOpen, the world's leading publisher of Open Access books. Built by scientists, for scientists. Our readership spans scientists, professors, researchers, librarians, and students, as well as business professionals. We share our knowledge and peer-reveiwed research papers with libraries, scientific and engineering societies, and also work with corporate R&D departments and government entities.