Kinesthetic Illusion of Being Pulled Sensation Enables Haptic Navigation for Broad Social Applications

Abstract

Many handheld force-feedback devices have been proposed to provide a rich experience with mobile devices. However, previously reported devices have been unable to generate both constant and translational force. They can only generate transient rotational force since they use a change in angular momentum. Here, we exploit the nonlinearity of human perception to generate both constant and translational force. Specifically, a strong acceleration is generated for a very brief period in the desired direction, while a weaker acceleration is generated over a longer period in the opposite direction. The internal human haptic sensors do not detect the weaker acceleration, so the original position of the mass is \\\"washed out\\\". The result is that the user is tricked into perceiving a unidirectional force. This force can be made continuous by repeating the motions. This chapter describes the pseudoattraction force technique, which is a new force feedback technique that enables mobile devices to create a the sensation of two-dimensional force. A prototype was fabricated in which four slider-crank mechanism pairs were arranged in a cross shape and embedded in a force feedback display. Each slider-crank mechanism generates a force vector. By using the sum of the generated vectors, which are linearly independent, the force feedback display can create a force sensation in any arbitrary direction on a two-dimensional plane. We also introduce an interactive application with the force feedback display, an interactive robot, and a vision-based positioning system.

1. Introduction

Haptic interfaces in virtual environments allow users to touch and feel virtual objects. Significant research activities over 20 years have led to the commercialization of a large number of sophisticated haptic interfaces including PHANToM and SPIDAR. However, most of these interfaces have to use some type of mechanical linkage to establish a fulcrum relative the ground (Massie & Salisbury, 1994; Sato, 2002), use huge air compressors (Suzuki et al., 2002; Gurocak et al., 2003), or require that a heavy device be worn (Hirose et al., 2001), thus preventing these mobile devices from employing haptic feedback.

Although haptic feedback provides many potential benefits as regards the use of small portable hand-held devices (Ullmer & Ishii 2000; Luk et al., 2006), the haptic feedback in mobile devices consists exclusively of vibrotactile stimuli generated by vibrators (MacLean et al., 2002). This is because it is difficult for mobile devices to produce a kinesthetic sensation. Moreover, the application of low-frequency forces to a user requires a fixed mechanical ground that mobile haptic devices lack. To make force-feedback devices available in mobile devices, ungrounded haptic feedback devices have been developed since they are more mobile and can operate over larger workspaces than grounded devices (Burdea, 1996). The performance of ungrounded haptic feedback devices is less accurate than that of grounded devices in contact tasks. However, ungrounded haptic feedback devices can provide comparable results in boundary detection tests (Richard & Cutkosky, 1997). Unfortunately, typical ungrounded devices based on the gyro effect (Yano et al., 2003) or angular momentum change (Tanaka et al., 2001) are incapable of generating both constant and directional forces; they can generate only a transient rotational force (torque) sensation. In addition, Kunzler and Runde pointed out that gyro moment displays are proportional to the mass, diameter, and angular velocity of the flywheel (Kunzler & Runde, 2005).

There are methods for generating sustained translational force without grounding, such as propulsive force or electromagnetic force. Recently, there have been a number of proposals for generating both constant and directional forces without an external fulcrum. These includeusing two oblique motors whose velocity and phase are controlled (Nakamura & Fukui, 2007), simulating kinesthetic inertia by shifting the center-of-mass of a device dynamically when the device is held with both hands (Swindells et al., 2003), and producing a pressure field with airborne ultrasound (Iwamoto et al., 2008).

2. Pseudo-Attraction Force Technique

2.1. Haptic interface using sensory illusions

To generate a sustained translational force without grounding, we focused on the characteristic of human perception, which until now has been neglected or inadequately implemented in haptic devices. Although human beings always interact with the world through human sensors and effectors, the perceived world is not identical to the physical world (Figure 1). For instance, when we watch television, the images (a combination of RGB colors) we see are different from physical images (a composition of all wavelengths of light), and TV animation actually consists of a series of still pictures. Such phenomena are usually interpreted by converting them to subjectively equivalent phenomena. These distortions of human perception, including systematic errors or illusions, have been exploited when designing human interfaces. Moreover, some illusions have the potential to enable the development of new haptic interfaces (Hayward 2008). Therefore, the study of haptic illusions can provide valuable insights into not only human perceptual mechanisms but also the design of new haptic interfaces.

Figure 1.

Difference between perceived world and physical world.

2.2. Principle

The method, which is called the pseudo-attraction force technique, exploits the characteristics of human perception to generate a force sensation, using different acceleration patterns for two directions to create a perceived force imbalance, and thereby produce the sensation of directional pushing or pulling. Specifically, a strong acceleration is generated for a very brief time in the desired direction, while a weaker acceleration is generated over a longer period in the opposite direction. The weaker acceleration is not detected by the internal human haptic sensors, so the original position of the mass is "washed out". The result is that the user is tricked into perceiving a unidirectional force. This force can be made continuous by repeating the motions. Figure 2 shows a model of the nonlinear relationship between physical and psychophysical quantities. If the acceleration patterns are well designed, a kinesthetic illusion of being pulled can be created because of this nonlinearity.

Figure 2.

Nonlinear relationship between physical and psychophysical quantities.

2.3. Implementation

To generate the asymmetric back-and-forth motion of a small, constrained mass, we have adopted a swinging slider-crank mechanism as a quick motion mechanism (Figure 3). In the mechanism, the rotation of a crank (OB) makes the weight slide backwards and forwards with asymmetric acceleration. The force display is composed of a single degree of freedom (DOF) mechanism. The force vector of the asymmetric oscillation is

$$\text{F}(t)=m\frac{d^{2}x(t)}{d{t}^2}$$ E1

where m is the weight. The acceleration is given by the second derivative with respect to time of the motion of the weight x, which is given by

$$x(t)=l_{\text{1}}\text{cos}\omega t+\mu (d-l_{\text{1}}\text{cos}\omega t)+\sqrt{l_{\text{3}}-{\{l_{\text{1}}(\mu -1)\sin \omega t\}}^2}$$ E2

where

$$\mu =\frac{l_2}{\sqrt{l_1^2+d^2-2l_{1}d\cos \omega t}}$$ E3

x(t) = OD, d = OA, l ₁ = OB, l ₂ = BC, l ₃ = CD, and ωt = AOB in Figure 3. ω is the constant angular velocity, and t is time.

Figure 3.

Overview of the swinging slider-crank mechanism for generating asymmetric oscillation. The slider (weight) slides backwards and forwards as the crank (OB) rotates. Point A causes the slide to turn about the same point. Since the relative link lengths (AB:AC) are changed according to the rotation of the crank, the slider (weight) moves with asymmetric acceleration.

We fabricated a prototype of the force display. In the prototype, d = 28 mm, l ₁ = 15 mm, l ₂ = 60 mm, and l ₃ = 70 mm. The actual acceleration values of the prototype were measured with a laser sensor (Keyence Inc., LK-G150, sampling 20 kHz) employing a seventh order LPF Butterworth filter with a cut-off frequency of 100 Hz (Figure 4).

Figure 4.

Actual asymmetric acceleration value with the LPF (blue solid line) vs. the calculated value (black dotted line). Humans perceive a unidirectional force by holding the haptic device. This is because the strong and weak acceleration periods yield different sensations, although the device physically generates a bidirectional force.

3. Requirements for perceiving pseudo-attraction force

There are still many aspects of the perception of the pseudo-attraction force that are not well understood, but knowledge has been accumulating. In this section, we introduce various parameters for eliciting the pseudo-attraction force through experimental results.

3.1. Acceleration profile

First, we determined whether oscillations with asymmetric acceleration elicit the perception of a pseudo-attraction force. Two oscillations with different acceleration profiles were compared as haptic stimuli: asymmetric acceleration (shown in Figure 4) and symmetric acceleration (control). For the asymmetric acceleration stimuli, the average percentage correct scores (i.e., how often the perceived force direction matched the crank-to-slider direction in Figure 3) for all subjects were approximately 100% at frequencies below 10 cycles per second when we used a binary judgment task (forward or backward). For the symmetric acceleration stimuli, the scores were between 25% and 75%, which is the chance level. These results show that the symmetric acceleration could not generate a directed force sensation. We performed a binomial test for the average percent correct scores, which showed no significant effect of the control stimuli for any of the frequencies. This means that symmetric acceleration does not elicit the perception of a pseudo-attraction force. Again, no directional force was felt if the mass were merely moved back and forth, but different acceleration patterns for the two directions to create a perceived force imbalance produced the perception of a pseudo-attraction force (Amemiya & Maeda, 2009).

3.2. Frequency of acceleration

Frequency of acceleration plays an important role in eliciting the perception of a pseudo-attraction force. Oscillations with high frequency might create a continuous force sensation, but previous experimental results showed that the performance decreased steadily at frequencies over ten cycles per second (Amemiya et al., 2008). However, low frequency oscillation tends to be perceived as a knocking sensation. If we wish to create a sustained rather than a transient force sensation such as the sensation of being pulled continuously, the frequency should be in the 5 to 10 cycles per second range. In addition, those who experienced the stimuli strongly perceived the sensation at 5 cycles per second independent of other parameters (Amemiya & Maeda, 2009).

3.3. Gross weight of force display

Changes in the gross weight and the weight of the reciprocating mass affects the perceived force sensation. Experimental results have shown that lighter gross weights and a heavier reciprocating mass yield higher percent-correct scores in binary judgment tasks for all frequencies (Amemiya & Maeda, 2009). Considering the Weber fraction of force perception, the differential threshold of force perception is thought to increase as the gross weight increases. In addition, the increase in the gross weight may work as a mass damper, which would reduce the gain of the effective pulse acceleration. The threshold of the ratio of the gross weight and the weight of the reciprocating mass was 16 %, which is a rough standard for effective force perception in the developed prototype.

3.4. Angular resolution

The azimuth accuracy of the perceived force direction versus the stimulated direction generated by an asymmetric acceleration has been examined (Amemiya et al., 2006). The orientation of the force vector was altered from 0 to 360 on the horizontal plane in 15 steps (24 vectors). The subjects were required to reply with one of 360 degrees in a static posture. The results showed that the root mean square of the angular errors between response and stimulus was approximately 20 degrees. When users move or rotate their bodies, i.e., dynamically explore the force vector, their angular resolution would be higher than that in a static posture.

3.5. Cancellation of orthogonal oscillation

If asymmetric oscillation was generated by rotational mechanism such as the slider-crank mechanism, a force perpendicular to the directional one were created because of the motion of the linkages. The side-to-side force prevents the user from sensing the desired direction. The side-to-side force should be cancelled out completely, for instance, by using two identical but mirror-reversed mechanisms (Amemiya et al., 2008).

4. Application

4.1. Overview

For broad social use, we designed an interactive application of haptic navigation based on a pseudo-attraction force display. The scenario was as follows. A waiter (user) in a cafe wants to deliver a drink ordered by a customer (target). The waiter does not know where the customer is sitting. However, his “smart tray” creates an attraction force centered on the customer and guides the waiter to him/her. Since the guidance is based on force sensation, the guidance information is useful regardless of the waiter’s age or language ability. Moreover, since the guidance directions are transmitted via touch, the other senses remain available to the waiter, making it easier for him to move through even the most crowded areas. Finally, the instructions remain entirely private; no one else can discover that the waiter is receiving instructions.

4.2. System configuration

The system consists of a tray held by the user (waiter), a small bag containing a battery and a control device, and a position and direction identification system based on infrared LEDs and a wide-angle camera (Figure 5). The force display and infrared LEDs are embedded in the tray. The user's position and posture are detected by placing three super-high luminance infrared LEDs (OD-100, OPTO Diode Corp., peak wavelength 880 nm, beam angle 120 degrees), at the corners of a right-angled isosceles triangle (side length = 100 mm) on the tray. The infrared rays are captured by a ceiling-mounted IEEE1394 black and white CMOS camera (Firefly MV, FFMV-03MTM; Point Grey Research Inc.) with a wide-angle lens (field angle 175 degrees). The positions and orientations of each IR-LED are obtained by binarizing the brightness value from the acquired camera image with OpenCV library, and calculating the position and orientation from the relationship with a right-angled isosceles triangle formed by three dots (Figure 6).

Figure 5.

System configuration.

Figure 6.

Vision-based position and posture identification system. The white dots in the camera image are the infrared LEDs.

The user must hold the tray horizontally because of the drink being carried on it. Therefore, the user’s posture can be presumed by detecting three IR-LEDs. The image capture rate is about 60 fps. The camera height is about 3.0 m and the camera faces the ground. When three points can be acquired, the position measurement error does not exceed 100 mm. This is sufficient for our demonstration since the distance to the targets is about 1,000 mm.

There are two ways to generate a two-dimensional force vector (Figure 7), and we fabricate each prototype. A turntable-based force display is one module based on a slider-crank mechanism with a turntable (Figure 8). The direction of the force display module is controlled with a stepper motor (bipolar, step angle 1.8 degrees, 1/4 micro step drive; KH42HM2-851; Japanese Servo Ltd.). engaged by a belt with a belt pulley installed in the turntable (Amemiya et al., 2009).

A vector-summation-based force display is designed to generate a force sensation in at least eight cardinal directions by the summation of linearly independent force vectors. Four slider-crank mechanism pairs are embedded in the force display in the shape of a crosshair. By combining force vectors generated by each slider-crank mechanism, the force display can create a virtual force in eight cardinal directions on a two-dimensional plane.

The target is several bear-shaped robots (RobotPhone; Iwaya Inc.). As the customer speaks, he also moves his head and hands to communicate with gestures.

4. 3. Demonstration procedure

The user moved towards the target following the direction indicated by the perceived force sensation. The force direction was controlled so that it faced the target (customer) based on the posture detection system. Control instructions were sent from the computer to the microcomputer via a wireless link (XBee-PRO (60 mW) ZigBee module; MaxStream) when required. The user chose one customer by stopping in front of the target. If this choice was correct, the customer (bear-shaped robot) said, ‘‘thank you’’; otherwise, the customer said, ‘‘I did not order this’’ while moving his head and hands to communicate with gestures.

Figure 7.

Two-dimensional force vector. (a) Turntable approach: one module with rotational mechanism. (b) Vector summation approach: modules of linearly independent vectors.

Figure 8.

Overview of the turntable-based force display

Figure 9.

Overview of vector-summation-based force display

4. 4. Discussion

We demonstrated the above application at an international conference and exhibition. The average rate of correct delivery to the target exceeded 75 % (note that none of the participants received any initial training), indicating that the navigation support provided is effective and appropriate. The results show the usefulness of the proposed technique. The few delivery failures appear to be due to tracking errors in the camera system or a delay between the rotation of the stepper motor and the user’s perception of the change. Moreover, we believe the force’s amplitude to be attenuated by the connection of the device to the tray, and this attenuation also influenced delivery failure. We sometimes observed that not all the LEDs could be detected since some were occasionally obscured by the participant. System robustness could be improved by adopting a different position and posture identification system. This haptic navigation could be also applied to a navigation system for the visually impaired (Amemiya & Sugiyama 2009).

5. Conclusion and future potential

The developed haptic display based on a pseudo-attraction force technique conveyed a kinesthetic illusion of being pulled or pushed. The ability of the haptic display to realize a wide-area social support system was discussed. Future work will include extending the reach by using a global positioning system to allow outdoor use.

Acknowledgements

We thank Dr. Ichiro Kawabuchi for his assistance. This study was supported by Nippon Telegraph and Telephone Corporation and was also partially supported by the sponsorship of CREST, Japan Science and Technology Agency.