Integration of Electrotactile and Force Displays for Telexistence

1. Introduction

Telexistence (or telepresence) enable us to interact with another human or object in a remote or a virtual place through a robotic system (Tachi & Yasuda, 1994). This technology spreads across the world because of a desire to extend a person’s sensing and interacting capability to remote places. In telexistence technologies, a robotic system called haptic display that provides haptic feedback to our hand is essential to touch the remote human or object (Shimoga, 1993a; 1993b). When we communicate or perform a task, a lack of haptic sensation reduces the realism and interactivity. Therefore, there is increasing requirement for haptic display presently.

The haptic feedback can be divided into two types based on the receptor that acquires the sensory information. One type is tactile (cutaneous) feedback, which is acquired by mechanoreceptors that exist at a depth of several millimetres from the skin surface. The other type is force (or kinesthetic) feedback, which is acquired by the proprioceptors that exist in the muscle, tendon, and joint. Based on the characteristics of human perception, it would be appropriate to provide both types of haptic feedback. In particular, a spatially distributed tactile feedback is necessary for dexterous manipulation. The spatially distributed tactile feedback and force feedback help us to perceive the position of the object and improve the stability of hand movements, respectively. For example, while holding a pen, we can pinch it with our fingertips and feel the reactive force; the position of the pen can be determined by tactile sensations.

Thus far, several haptic interfaces have been developed. However, these are not suitable for dexterous manipulation because of inadequate tactile feedback. The tactile display on conventional interfaces provides only a symbolic “contact” sensation of an object. Therefore, we cannot feel the object on our fingertips. It is believed that handling small objects such as pens is difficult without position information. Recently, some systems that can provide spatially distributed tactile sensation of an object have been proposed (Kim, et al., 2006; Methil, et al., 2006; Wagner, et al., 2005). Unfortunately, the systems proposed in these studies are too large for use in dexterous manipulation. A large system limits the workspace, i.e., the movement range of our finger required to manipulate an object. This limitation of workspace complicates manipulations such as pinching.

On the basis of results of conventional studies, we aimed to develop a haptic display for dexterous manipulation. First, we will summarize the requirements for the tactile feedback display intended for dexterous manipulations; the requirements are as follows:

1. The display should provide a highly realistic and intuitive touch sensation, i.e., it should provide not only the contact sensation but also the spatially distributed tactile sensation that humans perceive.

2. The display should be a compact body that does not invade the workspace of our fingers. Compact displays have several advantages over bulky ones during display implementation. Compact body size will help simplify the integration of this display with the force feedback display.

To fulfill these requirements, we used an electrotactile display as the tactile feedback display. We mounted the display on a force display with a wide workspace. This integration provided a haptic display that was suitable for dexterous manipulations.

In this chapter, we introduce a haptic display that integrates a spatially distributed electrotactile feedback and force feedback for telexistence. By integrating the electrotactile and force displays, we can use robotic system to dexterously manipulate an object (Fig. 1). Human interaction with a remote object through the robotic system can be dramatically improved by applying this concept. In section 2, we describe the concept of electrotactile and force integration. In section 3, we show the efficiency of the electrotactile feedback by a shape recognition experiment. In section 4, we describe the construction of a one-fingered haptic display and evaluate the effectiveness of the electrotactile integration. Finally, in section 5, we introduce a multi-fingered robotic hand system that involves the integration of electrotactile and force display for telexistence.

Figure 1.

Conceptual diagram of integration of electrotactile and force displays for dexterous manipulation.

2. Integration of electrotactile and force displays

2.1. Electrotactile display

The electrotactile display that we have developed (Kajimoto, et. al, 2004) can present spatially distributed tactile sensations. It comprises a pin electrode matrix. It directly activates nerve fibers under the skin by passing an electrical current from the surface electrodes (Fig. 2). The electrical currents flow from an electrode to adjacent electrodes through the skin. This display can selectively stimulate each type of receptor and produce vibratory and pressure sensations at an arbitrary frequency. By periodically changing the pin used for stimulation, we can produce the electrotactile stimulus at any points. Therefore, the electrotactile display allows us to perceive touch sensation which help determine position and exact shape of the object. In addition, the electrode plate of this display is small and lightweight. Therefore, it does not affect the workspace. Further, we can easily mount this display on all types of force displays.

Figure 2.

Electrodes of electrotactile display and method of electrical stimulus.

2.2. Force display

The force display presents the reactive and friction force on object surfaces. It can improve the stability of our hand movements when we manipulate an object. Currently, several types of force displays are used (Bar-Cohen, et al., 2000). In this study, we consider a small-sized display that has multiple degrees of freedom (DOFs) such as PHANToM (SensAble Tec.) and CyberGrasp (Immersion Tec.). Some of these force displays provide a wide workspace and sufficient force feedback to our hand.

2.3. Integration of the displays

When a user touches objects in a remote or virtual environment using our integrated system, he/she can perceive the spatially distributed tactile sensation and reactive force of objects. From these sensations, the user can easily identify the position of the object, its posture, and shape, i.e., he/she can easily recognize the object that he/she touches. For example, from the force sensation of a rounded surface and the tactile sensation of concave-convex surfaces, we can recognize that we are touching a gear (Fig. 3). We believe that this haptic information will also help the user to manipulate objects dexterously.

Figure 3.

Touch sensation by integration of electrotactile and force displays.

3. Electrotactile feedback for shape recognition

The electrotactile display may help perceive the shape of an object. Before implementing the integrated haptic display, we evaluated the efficiency of an electrotactile feedback when it is integrated with a force feedback (Sato, et al., 2007a; 2007c).

3.1. Efficiency of electrotactile feedback

First, we evaluated the efficiency of electrotactile feedback for shape recognition. Figure 4 shows the experimental setup. The participants wore a plastic finger case on their fingertip when they touched the object. The electrode plate used for electrotactile feedback was in the finger case. The electrotactile display that we used was the same as that shown in Fig. 2. In this setup, a “real” force sensation was generated by actual contact, and tactile sensation was generated by using the virtual model of the object in a PC. This condition is simulates a “mixed reality” situation.

We prepared three objects with the following characteristics: a flat surface, a curved face, and an edge (Fig. 5). We considered two modes of touching, namely, pushing and tracing (or sliding) as shown in Fig. 5. Experiments were conducted under six conditions as follows:

1. C1. Pushing with electrotactile feedback

2. C2. Pushing with force feedback

3. C3. Pushing with electrotactile and force feedbacks

4. C4. Tracing with electrotactile feedback

5. C5. Tracing with force feedback

6. C6. Tracing with electrotactile and force feedbacks

Under these conditions, we evaluated the accuracy and time taken for shape recognition.

Figure 6 shows the experimental results for all participants. From the results, we confirmed that the correct answer ratio when electrotactile feedback was present was higher than that when it was absent; moreover, the recognition time when electrotactile feedback was present was shorter than that when it was absent. Further, this result was independent of the participant and mode of touching. Therefore, we inferred that the electrotactile feedback improves the efficiency of shape recognition.

Figure 4.

Experimental environment.

Figure 5.

Objects that participants touched and two mode of touching.

3.2. Importance of electrotactile feedback

For shape recognition, electrotactile feedback is more important than force sensation; a number of shape sensations are generated by the electrotactile stimulus. For example, when the force display generates the sensation of an “object with an edge” while the electrotactile display generates the sensation of a “curved object,” a human being would perceive the latter. We investigated the responses of the participants to the force or electrotactile sensations.

Figure 6.

Results of the shape recognition experiment. The horizontal and vertical axes represent the abovementioned experimental conditions and the correct answer ratio or recognition time, respectively. (Sato, et al., 2007c).

The participants traced the object surface in the manner shown in Fig. 5. The objects they touched were an edge and a curve (Fig. 5). Two stimulation modes were tested for electrical stimulation. The first mode stimulated a “curvature”; the second, an “edge”. The experimental conditions were as follows.

1. C1. Touching curved face with electrotactile feedback of curved face

2. C2. Touching curved face with electrotactile feedback of edge

3. C3. Touching edge with electrotactile feedback of curved face

4. C4. Touching edge with electrotactile feedback of edge

The average response ratio of the “curve” is shown in Fig. 7. In this experiment, the participants tended to respond to an object on the basis of the electrotactile feedback. This result supports the hypothesis that the electrotactile sensation is more important than the force sensation in shape recognition. Therefore, it is suggested that the electrotactile stimulus is efficient in generating the shape sensation. In addition, we suggest that any touch sensation related to a typical object shape can be generated by integrating an electrotactile display with force display.

Figure 7.

Experimental result. The horizontal and vertical axes represent the experimental conditions and the response ratio of the “curve,” respectively. (Sato, et al., 2007a; 2007c).

4. One-fingered system

We constructed the one-fingered system of the electrotactile and force integration. Then, we evaluated the performance of the integrated system and the efficiency of the integration of electrotactile and force displays for a particular task (Sato, et. al., 2007b; 2007e).

4.1. Integration of electrotactile display with PHANToM

Figure 8 shows the configuration of the one-fingered system. In this system, we used PHANToM Omni (SensAble Tec.) as a force display. It provides a wide workspace and generates sufficient force for one finger. We mounted the electrotactile display on the end-effector of the PHANToM. The users placed the tip of their index finger on the electrotactile display and moved the end-effector of the PHANToM. They could control the cursor in the virtual environment using their fingertips. The fingertip was fixed on the end-effector by rubber bands. The electrotactile display that we used is same as shown in Fig. 2.

Figure 8.

Overview of the single-fingered system and electrotactile display on the end-effector of PHANToM.

The position data of the user’s index finger is captured by the PHANToM and translated to the PC. Then, the position of the cursor in the virtual environment is updated. On the basis of the cursor position, the reflection force and the electric current at the electrode pin are calculated. The reflection force is calculated by using the spring-damper model. Current is passed through the electrodes on the basis of the position of the contact field between the cursor and the virtual object. This implies that the electrostimulus is provided by the electrodes at the position corresponding to the contact position of a finger pad and an object. For example, when the finger pad is in contact with the face of a cube, all electrodes send a current to the finger. When the center of the finger pad touches the edge of the cube, the electrodes located in a line send the current.

4.2. Basic performance of the one-fingered system

We used the constructed system to examine the space resolution of the electrotactile feedback by distance and width discrimination. Subsequently, we evaluated the strength resolution of the electrical stimulus by strength discrimination.

We chose three experimental conditions: 2-line, width, and strength conditions. In each condition, there was a floor, a cursor, and two lines (a standard line and a comparison line) in the virtual environment. We specified two modes of touching the lines—pushing and sliding (Fig. 9).

Figure 9.

Two modes of touching lines. (Note that participants were not able to view lines during experiments).

We conducted each experiments by method of constant stimuli. The experimental results for each setting are shown in Fig. 10. From the results, the effect of the touching modes on the resolution seems to be small.

From the results of the 2-line discrimination, the threshold is observed to be approximately 9.5 mm. On the electrotactile display, the electrical current flows from the electrode only to the adjacent electrodes. Therefore, the discrimination threshold should be around 5.0 to 7.5 mm. However, under practical conditions, the electrical current leaks to the surrounding electrodes. This leakage current results in a wide area of contact sensation. Therefore, we believe that the leakage current will cause complications in identifying whether the lines are identical or not.

The width discrimination threshold for the 7.5 mm line is approximately 2.0 mm. On the basis of the distance between the centers of the electrodes, the width discrimination threshold is considered to range from 0.0 to 2.5 mm. This result is in accordance with the theoretical value. Therefore, we conclude that the abovementioned leakage current does not affect width discrimination.

In the case of strength discrimination, the upper and lower thresholds are approximately 0.12 and 0.06 mA, respectively. These thresholds are considered to be small as compared to the range of the strength of the electrical stimuli that the participants could feel comfortably (1.5 mA). Therefore, we believe that the electrotactile display has a high strength resolution. On the basis of this result, it is possible to implement the presentation of magnitude of the pressures by means of the strength of the electrotactile stimulus.

4.3. Tracing task efficiency

Using the one-fingered system, we evaluated the manipulation efficiency in track tracing task. The participants controlled the cursor and traced a circular path in a virtual environment using the constructed system (Fig. 11). The experiment was conducted under the following four feedback conditions:

Figure 10.

Results of experiments on 2-line, width, and strength discriminations. The horizontal and verrtical axes represent the reference value of each experiment and represents the response ratio of participants, respectively. (Sato, et al., 2007e).

Figure 11.

Overview of tracing a circular path in a virtual environment.

1. C1. Integration 1: reflection force and position sensation

2. C2. Integration 2: reflection force and contact sensation

3. C3. Force: reflection force

4. C4. Electrotactile: position sensation

The position and contact sensation were generated by the electrotactile display. In C1, a two-dimensional contact position sensation was generated by each electrode of the electrotactile display. This shows the participant’s finger tip where the cursor touches the circular path. In C2, the contact sensation was generated by all the electrodes of the electrotactile display.

Figure 12 shows the result of the evaluation of the track-tracing task. In order to evaluate the accuracy of the tracing task, we assumed the trajectory that traces the center of the path to be the optimal trajectory. Then, we compared the avarage error between the optimal trajectory and the measured trajectory.

The error in C1 is the smallest for all participants. Therefore, we can confirm that the electrotactile and force integration is effective in the case of the track-tracing task. When we compare the errors in C1, C3, and C4, we find that the error in the case in C4 is the largest. This shows that the force feedback is more important than the electrotactile feedback in for stablity in operation. When we compare the errors in C2 and C3, the error in C2 is larger than that in C3 even though more haptic information is generated in C2. This may mean that tonly contact sensation cannot improve the task efficiency. This result confirms the importance of the proposed spatially distributed tactile feedback.

Figure 12.

Result of the evaluation of the track-tracing task. The horizontal and vertical axes represent the haptic condition and the trajectory error, respectively. (Sato, et al., 2007b).

5. Multi-fingered robotic hand system: Haptic Telexistence

By integrating electrotactile and force displays, we constructed a multi-fingered robotic hand master-slave system named Haptic Telexistence.

5.1. Configuration

Our system consists of four devices, namely, a multi-fingered slave hand, a finger-shaped haptic sensor for the slave hand, an exoskeleton encounter-type master hand, and electrotactile display (Fig. 13).

We mounted the electrotactile display on a multi-fingered master hand (Nakagawara, et al., 2005). This hand has two features. One is a compact exoskeleton mechanism called “circuitous joint,” which covers the wide workspace of an operator’s finger. The other is the encounter-type force feedback. These features help avoid unnecessary contact sensation and enable the unconstrained motion of the operator’s fingers. We set the electrotactile display on the tips of each finger mechanism.

Figure 13.

Configuration of Haptic Telexistence system.

The multi-fingered slave hand (Hoshino & Kawabuchi, 2005) has the following futures. This hand has 15 DOFs — five DOFs for the thumb, one for abduction of other fingers, three for the index finger, and two for the remaining fingers. Each fingertip has an independent DOF, and the index finger and the thumb can be moved in opposite directions. Therefore, a pinching operation by the fingertip is possible. In addition, we developed a finger-shaped haptic sensor (Sato, et al., 2008) using the GelForce technology (Kamiyama, et al., 2005) for this robotic hand. GelForce is a haptic sensor that measures the distribution of both the magnitude and the direction of force.

The master-slave manipulation is realized by bilateral position control of the multi-fingered slave hand and the encounter-type master hand. This control is exercised from the position of the master and slave fingers. The position is calculated using the angle of each finger joint. The refresh rate of the control is 1 kHz. Therefore, we can operate the multi-fingered slave hand smoothly and perceive sufficient force sensation.

When the slave hand touches an object, the finger-shaped GelForce mounted on the slave hand acquires haptic information such as the distribution of the magnitude and the direction of force. Then, this information is transmitted to the master system. The electrotactile display provides a tactile sensation on the basis of this information. Information regarding the distribution of the force is obtained from the pin location which provides electrostimulus. Subsequently, information regarding the magnitude of the force at each position is obtained form the strength of electrostimulus. As a result, we can feel the field, edge, peak, and the movement of an object. By integrating these force and tactile sensations, we can perceive the exact shape and stiffness of the object. This enables highly realistic interactions with remote objects.

5.2. Exhibition of Haptic Telexistence

Figure 14 represents the Haptic Telexistence system designed by us. We exhibited this system in some conferences such as ACM SIGGRAPH 2007 (Sato, et al., 2007d). During the exhibitions, approximately one thousand participants used this system. The participants could feel an object being touched with the finger of slave hand due to the electrotactile and force feedbacks. In addition, many participants pointed out that the Haptic Telexistence system is a useful technology for tele-communication and tele-manipulation in fields such as relesurgery.

In the future, we will evaluate the haptic telexistence system from the viewpoint of efficiency of transmission of haptic information and tele-manipulation.

Figure 14.

Haptic Telexistence system and its exhibition at a conference. (Sato, et al., 2007d).

6. Conclusion

In this chapter, we described a robotic system that enables us to interact with a remote human or object. We proposed the integration of electrotactile and force feedback for dexterous tele-manipulation. The electrotactile feedback can provide spatially distributed tactile sensation; therefore, we consider that the integration of electrotactile and force feedback is effective in perceiving the shape of an object and in manipulating it. We have confirmed the effectiveness of the electrotactile feedback and constructed a multi-fingered telexistence system named Haptic Telexistence.

In the future, we plan to provide more object properties such as texture and temperature. Not only will we be able to shake hands with people at remote locations but we will be able to feel the warmth of their hands. In the case of internet shopping, we will be able to check the texture of an article before purchase. We expect that the Haptic Telexistence system will dramatically improve the human interaction with a remote object.