Robot Perception Based on Vision and Haptic Feedback for Fighting the COVID-19 Pandemic

Abstract

The robot perception can be enhanced further through visual and haptic to give more impression. This chapter aims to combine vision and haptic for the robot navigation during tracing their movement. The pandemic has striven humans to do direct contact; therefore, an alternative using the robot as delivery tools is assumed to be one of solution. As the initial experiment has been shown in the previous section, the deviation of angle is quite low and the success rate of arriving at the destination is also quite high around 76%. Future work can be enhanced by improving the success rate by monitoring the robot track closely.

1. Introduction

The pandemic of COVID-19 has affected human life in general. Many prevention actions are used to prevent the virus from the spread, including maintaining social distance and wearing a facemask. To ensure the safety of people, robots are used for controlling the COVID-19 patient bed using the Arduino robot [1]. It is designed to make less contact with patients so reduced the chance of infection. The Coronavirus spreads via the saliva droplets or nose liquid when the sick person is coughing or sneezing [2]. Due to the high rate of infection, mobile robots can be an alternative solution to reduce contact with patients [2]. Robots for support service is one of the solutions for maintaining society awareness toward the virus spread. The mobile robot’s utility during the pandemic might vary, such as delivery service, population awareness, and disinfection facilities [2]. This chapter aims to provide new robot insight through deep learning vision and haptic that can augment the robot’s response toward their environment.

2. Related works

Artificial Intelligence (AI) and robotics are valuable resources for helping the patient treatment, doctors, nurses, and other front-line staff. Intelligent robots can perform good service for a particular task when it is planned and designed better [3, 4]. However, due to the high cost and complexity of the technology, not all country affords to adapt this approach [2]. Furthermore, another researcher tried to utilize a nursing robot for patient monitoring and medicine consumption according to the medication schedule. The other robot known as Lio-A is a robot with a multi-functional arm that has the capability of human-robot interaction and personal care assistant (Figures 1 and 2) [5].

Figure 1.

Robot consultation [2].

Figure 2.

Lio-A robot for a personal assistant [5].

The Lio-A robot has a visual and audio sensor for receiving the command, while laser and ultrasound for navigation and surrounding monitoring. It also has a mechanical sensor for handling the task given to them. Lio-A has the capability for autonomous action by having automatic navigation and recharging [5]. The other researcher also uses robots and realistic virtual reality to enhance interaction between humans and machines. The interaction can be in a gaming-based system, Brain-computer interface, or 3D simulation [6, 7, 8, 9, 10, 11, 12, 13, 14]. Virtual navigation using augmented reality or sensors is also helpful for the robot to achieve the desired direction according to the path that set up for them [15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28]. Machine learning algorithms, such as reinforcement learning also involve managing the crowd behavior of pedestrians, so it will be beneficial for robotic navigation later [29, 30, 31, 32, 33]. In addition, the Kinect camera with its capability for gesture tracking is also beneficial for medical applications along with robotics for helping COVID-19 patients, doctors, and nurses [34, 35, 36, 37, 38, 39, 40, 41].

3. Methodology

The visual perception of the robot can be enhanced through a special camera, such as a Kinect depth camera, that can provide a depth image stream as input. This image will be useful for the navigation of the robot due to its capability of providing a 3D image of the object.

4. Materials

There is three main hardware that involved during experiments:

• Wheel robot using Arduino board Uno3, refer to Figure 3.

• Kinect camera attached to the robot body.

• Laptop for processing.

Figure 3.

System setup.

The Kinect camera will stream the object in front of the robot and send it to the pc, once pc received the image. It will be continued by processing the image whether the image is one of its goals or obstacles. If obstacle then the robot needs to avoid the object, while if it is a goal, the robot needs to grasp the object using the gripper, a detailed methodology is shown in Figure 4.

Figure 4.

System methodology.

5. Result and discussion

The initial testing using the Kinect camera is tracking the user skeleton. It can track the user’s movement, especially hand gestures that can be interpreted as command control. The tracked skeleton is shown as a fragmented line that is imposed on the human body, as shown in Figure 5. The parts of the body, such as the arm, body, and head skeleton, are tracked in real time. Later this body part will be used as reference control to manipulate the robot.

Figure 5.

Skeleton tracking.

Figure 6 shows how the depth image stream of the object streamed into pc for further process. While the depth image with a recognized person is shown in Figure 7.

Figure 6.

Depth image stream.

Figure 7.

Person recognition.

The process of person recognition with pre-train model is quite interesting; the robot can recognize the person by receiving the information from the main process inside the PC. The haptic device is used for helping the robot on avoiding obstacle by sending the vibration as an alert signal. So, the robot can turn left or right to avoid the obstacle. We did several experiments with robotic movement, as shown in Figure 8.

Figure 8.

Robot movement tracks.

6. Conclusions

Visual and haptic enhancement of robotic perception is very important for the success rate of the robot task. In this chapter, we present a combination of haptic with the vision to enhance the robot navigation during performing the delivery task to the user. Robot delivery is one of the essential keys during the pandemic of COVID-19 to avoid direct contact between humans. As the initial experiment has been shown in the previous section, the deviation of angle is quite low and the success rate of arriving at the destination is also quite high around 76%. Future work can be enhanced by improving the success rate by monitoring the robot track closely.

Acknowledgments

This work was supported by the Deanship of Scientific Research (DSR), King Abdulaziz University, Jeddah Saudi Arabia. The authors, therefore, gratefully acknowledge the DSR technical and financial support.