Sensory Properties in Fusion of Visual/Haptic Stimuli Using Mixed Reality

1. Introduction

This chapter introduces our experimental investigation about sensory properties in the fusion of visual/haptic stimuli by using Mixed-Reality (MR) technique. Especially, we focus on the discrepancy between the two stimuli.

When making a purchase from TV or online, we are sometimes disappointed by a product whose actual scale and material differ from our image, even though its appearance originally impressed us. This case indicates the use of integrated multiple sensory cues that include not only the visual but also the auditory and haptic to extract the properties of objects. However, the fusion of multiple sensory cues by interaction with each other has not been well examined. We introduce our developed system that can independently control the sensibility parameters of visual and haptic cues to study the effect of these cues on sensory properties.

MR techniques have been applied in factories to assist assembly, inspection and maintenance operations (Ohta & Tamura, 1999) (Wiedenmaier, et al, 2001) (Friedrich, 2002) (Fiorentino et al, 2002) (Nolle & Klinker, 2006). Recently, designing operations for industrial products are gathering attention as the next generation of MR applications (Navab, 2003) (Lee & Park, 2005) (Sandor et al, 2007). In ordinary designing operations, first, designers develop a broad plot of shape, appearance, and inner structure using a Computer Aided Design (CAD) system. After that, a mock-up, which is a dummy model of the designing product, is generated to examine such detailed information as a sense of touch and surface texture that is difficult to represent by CAD. However, generating a mock-up is very expensive, especially since it uses temporal resources. Thus, it is not practical to regenerate a mock-up whenever a design receives a minor change.

The display system shown in Figure 1 might create a new style of product design to reduce operation processes. For example, in ordinary product design, when a designer wants to evaluate different impressions caused by subtle changes of surface material, many similar preproduction samples must be generated that correspond to each change. However, design variations are usually limited because they are too expensive. On the other hand, using our proposed system, evaluation is possible by superimposing computer graphics (CG) textures of various appearances onto a design mock-up that solves not only cost problems but also the limitations of trial design variations.

To realize such a design support system, it is important to investigate how visual and haptic sensory sources are fused and affect each other.

Figure 1.

Example of Designing Operation for Industrial Products by Using Mixed Reality. By using this system, evaluation is possible by superimposing computer graphics (CG) textures of various appearances onto a design mock-up that solves not only cost problems but also the limitations of trial design variations.

2. Fusion of Visual/Haptic Cues Using Mixed Reality

Visual cues seem to affect to estimate environmental properties. As a result, some research reports that haptic cues are affected by visual cues. Several studies have addressed this issue in the real world (Biocca et al, 2001) (Adams et al, 2001) (Rock & Harris, 1967) (Rock & Victor, 1964) (Lederman, & Abbott, 1981) (Hillis et al, 2002). In these researches, however, since subjects could not see an object, they had to imagine that they were grasping what they were looking at. The evaluating saturation is nowhere near the touching/holding operation in daily life, on the other hand, looking at and touching an object is important for evaluation. By focusing on such inconveniences, we developed a system that provides various impressions of an observed object by showing different visual information from the actual shape and material using an MR technique (Nakahara et al, 2007). Wang et al. also developed a MR system that fuses visual and haptic information (Wang et al, 2000). However, when the system developed, the quality of CG technology was not powerful to express subtle difference of appearance caused by changes of surface material. Then, the purpose of the investigation about sensory properties in fusion of visual/haptic stimuli is different from ours. Figure 2 shows an overview of our proposed system. A user perceives a tactile sensation by touching a real object. By displaying the object’s appearance, the tactile sensation is merged with the visual sensation in the user’s perception.

As a procedure to analyze sensory properties, we focus on two features of objects. One is the impression of texture that is intimately involved in the impression of products. The other is the sharpness of a cube’s edge, which is strongly affected by both visual and haptic senses. Below we introduce two experiments that evaluate the impression of texture and the sensation of sharpness by using our MR system.

Figure 2.

Fusion of Visual/Haptic Cues Using Mixed Reality. A user perceives a tactile sensation by touching a real object. By displaying the object’s appearance, the tactile sensation is merged with the visual sensation in the user’s perception.

3. Subjective Evaluation of Texture Impression

We assume the impression of texture consists on two kinds: haptic and visual. Haptic texture is a stimulus given by the tactile sensation of touching an object, and visual texture is a stimulus given by its appearance. When touching glass material with closed eyes, we experience a tactile sensation that resembles the impres-sion of a “glass haptic texture.” When looking at a glass material without touching, we experience a visual sensation that resembles an impression of a “glass visual texture.” This section introduces our experimental evaluation that investigates whether it is possible to control the impression of a haptic texture by changing visual textures.

3.1. Experimental Environment

As shown in Figure 3, the experimental system can overlap various visual textures generated by computer graphics onto a real object. In this system, real objects are several plates made from different materials. We choose stone, cork, unglazed tile, steel, and wood as the materials of the plates. The stone has a rough surface, but it is not polished. When a subject strongly pushes the cork with a finger tip, its shape is deformed. The unglazed tile has a rough surface, but it is not cover coated. The surfaces of the steel and wood are smoothed by filing. An example of a subject’s view is shown in Figure 4. An overlapping texture covers the entire real object.

Figure 5 shows an experimental scene. First, after we measure the temperature of a subject’s hand, he/she puts on a thin latex glove. From pilot studies, we learned that haptic stimulus affects the texture impression more strongly than the visual stimulus. So we use a glove that deadens the haptic sensation to maintain balance.

Since humans can identify material by sensing inherent specific heat of each object, we maintained the temperature of the real objects at the temperature of a subject’s hand by using a thermos-tatically-controlled electric carpet.

To maintain photometric consistency between real and virtual appearances, the system maps actual images captured by a high-end single-lens reflex camera at an identical position as a subject’s viewpoint. We use a high-definition HMD that can display a 1280×1024 image to increase the evaluation reality as much as possible. If a subject’s viewpoint differs from the viewpoints of other subjects, geometric and photometric inconsistencies are created between the visual cues. Subjects were instructed to fix their jaw at a prescribed position while they looked at a CG texture in the middle of an HMD, as shown in Figure 4.

These considerations described above allow evaluation of the impression of texture by only deriving visual and haptic stimuli.

Figure 3.

Subjective Evaluation of Texture Impression. The experimental system can overlap various visual textures generated by computer graphics onto a real object.

Figure 4.

Example of a Subject’s View. An overlapping texture covers the entire real object.

Figure 5.

An Experimental Scene of Evaluation. We use a glove that deadens the haptic sensation to maintain balance of haptic and visual stimuli.

3.2. Elimination of Occlusion by Image Matting

As shown in the picture on the left of Figure 6, when a subject moves a hand over a real object, the hand is occluded by the CG texture. As a result, subjects have difficulty feeling as they are actually touching a real object. We solve this problem by a skin color matting technique (Itoh et al, 2003) that utilizes a property that clusters the skin region in the chroma space. We defined a skin color model in chroma space in advance and segment the skin color region from captured images using the model. As shown in the picture on the right of Figure 6, generating an image is possible that does not spoil the appearance of the user’s hand.

Figure 6.

Example of Occlusion: (Left) subject’s finger region is occluded by CG; (Right) occlusion problem in finger region is solved.

3.3. Procedure of Subjective Evaluation

We evaluated the amount of mistaken sensations caused by the combinations of haptic and visual textures. Since each texture has five types of materials (stone, cork, unglazed tile, steel, and wood), there are 25 combinations. We analyzed texture impressions examining evaluation score which are answered by all subjects for all combinations. In all trials, subjects were permitted to take as much time as needed. The displayed textures are randomly chosen to control for order effects.

The subjective evaluations were conducted by ten male subjects in their 20s who were presented a randomly selected combination of haptic and virtual textures and then were answered whether their impressions matched the material they saw. A five-level rating scale was used. Scale 1 means “completely different impression from what I see.” Scale 2 means a “different impression.” Scale 3 means “No difference” Scale 4 means “almost identical impression what I see.” Scale 5 means “completely identical.” When the subject gives a high score with observing an MR object which has inconsistent visual and haptic texture, it shows that the visual cue has stronger influence than the haptic one for the impression of the object’s texture.

3.4. Results and Discussion

Figure 7 shows the evaluation results. The horizontal axis represents the kinds of materials, and the vertical axis represents the mean evaluating rate for each material. The line with rhombus nodes indicates the result of displaying the visual texture of stone. The line with box nodes indicates the result of displaying the visual texture of cork. The line with triangle nodes indicates the result of displaying the visual texture of unglazed tile. The line with X nodes indicates the result of displaying the visual texture of steel. The line with asterisk nodes indicates the result of displaying the visual texture of unglazed wood.

When we used a cork plate as a real object, few subjects had a different impression from the real material when the visual texture is changed. Based on the questionnaire data, subjects identified the material by its surface softness. When we used a wood plate as a real object, some subjects commented that they could feel the surface softness. As a result, it was difficult to give an impression as if touching other materials.

Note that the evaluation rates of the haptic-stone/visual-steel and haptic-steel/visual-stone combinations are high, even though the haptic textures of their surfaces are quite different. In the questionnaire data, when touching a stone plate with a rough surface while looking at a visual texture of steel, one subject had the impression of touching a steel plate that was not well polished. On the other hand, when touching a steel-plate with a smooth surface while looking at the visual texture of stone, one subject had the impression of touching a well polished stone plate, such as granite or marble. From this result, we assume that if we already had some different impressions (e.g., smooth or rough) about touching materials from past experience, controlling the impression of the real object between the impressions is possible by changing the visual texture. The assumption can be assessed by comparing the rough surfaces of stone and unglazed tile. Because we rarely have impression that the surface of unglazed tile is smooth in daily life, when we use a smooth steel plate as a real object and overlap an unglazed tile texture onto the plate, few subjects had the impression that they were touching an unglazed tile.

Figure 7.

A Result of Subjective Evaluations for Texture Impression. The line with rhombus nodes indicates the result of displaying the visual texture of stone. The line with box nodes indicates the result of cork texture. The line with triangle nodes indicates the result of unglazed tile texture. The line with X nodes indicates the result of steel texture. The line with asterisk nodes indicates the result of unglazed wood texture.

4. Subjective Evaluation of Sharpness

We conducted an experimental survey to investigate whether an edge is perceived sharper than its actual curvature (4 mm) by overlapping a sharper appearance (1 mm) on the surface. Figure 8 shows an overview of this experiment. In this experiment, we quantify the haptic stimulus by curvature radius of the edges of cubes.

Figure 8.

Overview of Subjective Evaluation of Sharpness Sensation. We conducted an experimental survey to investigate whether an edge is perceived sharper than its actual curvature (4 mm) by overlapping a sharper appearance (1 mm) on the surface.

4.1. Experimental Environment

Figure 9 shows a picture of an experimental scene. The subject maintains his viewpoint while putting his jaw at a predefined position to preserve geometric and photometric consistency between the real and virtual information. As illustrated in Figure 10, the distance between the subject’s viewpoint and a target object is about 30 cm and the tilt angle of the subject’s line of sight is 45 .

Figure 9.

Experimental of Subjective Evaluation of Sharpness. The subject maintains his viewpoint while putting his jaw at a predefined position to preserve geometric and photometric consistency between the real and virtual information.

Figure 10.

Positional Relationship between a Real Object and a Subject. The distance between the subject’s viewpoint and a target object is about 30 cm and the tilt angle of the subject’s line of sight is 45 .

We prepared three cubes with 10 cm sides to control haptic stimuli. One cube has 12 edges of curvature radius from 0.0 to 1.1 mm, one has edges from 1.2 to 2.3 mm, and another has edges from 2.4 to 3.5 mm. A cube is shown in Figure 11. It is possible to generate various CG appearances of the cube by controlling the curvature radii of the CG edges.

Figure 11.

Cube with 10 cm Sides to Control Haptic Stimuli. We prepared three cubes with 10 cm sides to control haptic stimuli. One cube has 12 edges of curvature radius from 0.0 to 1.1 mm, one has edges from 1.2 to 2.3 mm, and another has edges from 2.4 to 3.5 mm.

Figure 12.

Angle Between Horizontal Plane and Virtual Light is 60 . This angle is almost the same as the environment light of the expe-rimental room.

Twelve male subjects in their 20s evaluated sharpness sensations by touching a CG overlapped cube that contained a discrepancy between vision and haptic stimuli.

All subjects have visual acuity of nearsightedness over 1.0 to discriminate two lines with 0.14 mm interval which is generated by the sharpest edge in this research on an HMD.

A displayed virtual cube is illuminated by a point-source light. The angle between the horizontal plane and the light is 60 , as illustrated in Figure 12. This angle is almost the same as the environment light of the experimental room. By using ARToolKit (Kato & Billinghurst, 1999), the system overlaps the virtual cube onto a real cube. To prevent subjects from determining an edge’s sharpness by referring to a cut plane of the edge, the system does not display the cut plane, as shown in Figure 13. To solve the CG appearance problem, which involves overlapping of the subject’s hand in MR scenes, subject hands were extracted by using the same procedure described in Section 3.2.

Figure 13.

Example of Displayed Image. To prevent subjects from determining an edge’s sharpness by referring to a cut plane of the edge, the system does not display the cut plane.

4.2. Quantization of Curvature Radii Scale

In the matching process, it is better that each stimulus value can be clearly defined by subjects. In other words, when we used too roughly quantized stimuli in the evaluation, evaluation precision suffers. On the other hand, when we used too finely quantized stimuli, subjects cannot define the differences. Therefore we quantize the scale of the curvature radii as a unit to present to subjects.

Discrimination Threshold

As illustrated in Figure 14 physical stimulus differs from mental stimulus. We examined the discrimination thresholds of subjects by a psychology and statistics experiment. A discrimination threshold is the minimum distance at which subjects can define differences of stimuli.

A method of limits detects a discrimination threshold by examining if a subject can detect differences between two different stimuli while changing the gap step by step. In this experiment, we examined a discrimination threshold (human’s sensitivity) of sharpness by comparing reference stimulus R with standard stimulus R0. Reference stimulus R, which is the curvature radius of an edge, is changed at 0.1-mm intervals.

As illustrated in Figure 15 when a subject felt reference stimulus R was sharper than standard stimulus R0, he recorded “-”. When a subject felt reference stimulus R was duller than standard stimulus R0, he recorded “+”. When discrimination was difficult, he recorded “?”. It is important to set an initial reference stimulus either large or small enough to easily define the difference between standard stimuli. We repeated the question-and-answer process until the subject recorded “?”. A discrimination threshold is defined as a region within marked “?” by operating the process both upward and downward. We conducted evaluations with twelve male subjects in their 20s, all of whom have visual acuity of nearsightedness over 1.0.

Figure 14.

Difference between physical and mental stimulus

Figure 15.

Definition for Mental Discrimination Threshold.

Results

The estimated haptic and visual discrimination thresholds are shown in Table 1 and 2. Visual discrimination thresholds are defined at 0.4-mm interval, but haptic discrimination thresholds are not. In order to investigate how visual/haptic stimuli interact with each other, common criteria are necessary. We define the common criteria as the minimum perceivable seven steps for both of visual/haptic discrimination thresholds, 0.2, 0.6, 1.0, 1.4, 1.8, 2.2, and 2.6 mm. Although, humans can discriminate differences less than 0.1 mm, due to the limitations of the accuracy processing machinery, it is not possible to estimate haptic discrimination thresholds less than 0.1 mm.

By considering different thresholds of visual and haptic information, we quantize the scale of the curvature radii into seven identical parts: 0.2, 0.6, 1.0, 1.4, 1.8, 2.2, and 2.6 mm.

Standard Stimulus (mm)	Discrimination Threshold (mm)	Standard Deviation
0.2	0.332	0.057
0.6	0.382	0.078
1.0	0.395	0.062
1.4	0.322	0.052
1.8	0.338	0.045

Table 1.

Visual Discrimination Threshold

Standard Stimulus (mm)	Discrimination Threshold (mm)	Standard Deviation
0.0	0.000	0.000
0.1	0.000	0.000
0.2	0.130	0.053
0.4	0.270	0.078
0.7	0.238	0.049
1.0	0.237	0.069
1.3	0.300	0.063
1.6	0.418	0.095

Table 2.

Haptic Discrimination Threshold

4.3. Procedure of Subjective Evaluation

First, as a reference of the matching process, subjects were presented a standard stimulus in three ways: only haptic, only visual, and both. When subjects are indicated to observe the object by using only haptic, subjects close the eyes, and then the experimenter leads their hand onto the object. When subjects are indicated to observe the object by only vision, subjects watch the object with putting their hands on the experimental table. When subjects are indicated to observe the object by using haptic and vision, subjects can watch and touch the object without any physical constraints.

After observing a standard stimulus, subjects are required to determine a corresponding stimulus by using only haptic, only vision, and both haptic and vision together. In all trials, subjects are permitted to take as much time as needed. The displayed stimuli for match-up are randomly chosen to control for order effects. When subjects require observing the next stimulus for match-up, the stimulus will be displayed after 15 seconds interval.

There are nine combinations of three displaying ways for standard stimulus and three displaying ways for mach-up stimuli. By executing multiple classification analysis to the result derived by the all combinations, we investigate whether human perception is affected by fusing visual and haptic cues.

If we conduct the experiment by using all perceivable seven steps for both of visual/haptic discrimination thresholds, huge amount of time and labor is needed. On the other hand, it is difficult to extract significant evidence to show that human perception is affected by fusing visual and haptic cues in the most of the trials, when the one stimulus is too week to affect the other stimulus. Thus, we conduct a preliminary experiment to choose combinations of visual and haptic stimuli that can easily introduce the influence caused by the fusion. As the result, a combination {visual: 2.2 mm / haptic 1.4 mm} is selected.

4.4. Result and Discussion

Results are illustrated in Figure 16 and Figure 17. The horizontal axis represents the types of matching procedures, and the vertical axis represents the mean evaluating value of the radii of edges. The line with rhombus nodes is the mean matching response when standard stimuli are presented only by haptic, the line with triangle nodes is only using vision, and the line with box nodes is using haptic and vision together.

In the first evaluation, subjects were given a 1.4 mm haptic curvature radius as a haptic stimulus and a 2.2 mm vision curvature radius as a visual stimulus. The result is shown in Figure 16.

When subjects received a standard stimulus as a 1.4 mm haptic curvature radius and determined a corresponding stimulus by only using haptic (the left rhombus node), they sensed it as 1.40±0.0 mm. On the other hand, when subjects received a standard stimulus as a 1.4 mm haptic curvature radius and a 2.2 mm vision one and determined a corresponding stimulus by only using haptic (the left box node), they sensed it as 1.64±0.2 mm by perceiving that the edge was blunter than the previous result. This result was derived by presenting a 2.2 mm vision stimulus as the standard stimulus.

When subjects received a standard stimulus as a 2.2 mm vision curvature radius and determined a corresponding stimulus by only using vision (the right triangle node), they sensed it as 2.20±0.0 mm. On the other hand, when subjects received a standard stimulus as a 1.4 mm haptic curvature radius and a 2.2 mm vision one and determined a corresponding stimulus by only using vision (the right box node), they sensed it as 2.12±0.4 mm by perceiving that the edge was sharper than the previous result. This result was derived by presenting a 1.4 mm haptic stimulus as the standard stimulus.

When subjects received a standard stimulus as a 1.4 mm haptic curvature radius and a 2.2 mm vision one and determined a corresponding stimulus by using both haptic and vision (the middle box node), they sensed it as 1.84±0.1 mm. This experiment shows that the haptic stimulus seems to be affected by visual stimulus when discrepancy exists between vision and haptic stimuli.

By applying the Student's t-test to our evaluation data, significance differences were found in effectiveness, caused by presenting a standard stimulus in three ways (F(2.18) = 26.694, p<0.05).

Figure 16.

Mean grit sizes selected as matches for visual/haptic, and visual/haptic standards; subjects touched an object with a 1.4 mm haptic curvature radius and a 2.2 mm vision one.

In the second evaluation, we switch the value of visual/haptic stimuli to control the order effect. Thus, a subject is given a 2.2 mm haptic curvature radius as a haptic stimulus and a 1.4 mm vision curvature radius as a visual stimulus. The result is shown in Figure 17.

When subjects received a standard stimulus as a 2.2 mm haptic curvature radius and determined a corresponding stimulus by only using haptic (the left rhombus node), they sensed it as 2.20±0.0 mm. On the other hand, when subjects received a standard stimulus as a 2.2 mm haptic curvature radius and a 1.4 mm vision one and determined a corresponding stimulus by only using haptic (the left box node), they sensed it as 2.16±0.2 mm by perceiving that the edge was sharper than the previous result. This result is derived by presenting a 1.4 mm vision stimulus as the standard stimulus.

When subjects received a standard stimulus as a 2.2 mm haptic curvature radius and a 1.4 mm vision one and determined a corresponding stimulus by using both haptic and vision (the middle box node), they sensed it as 2.04±0.2 mm. This experiment shows that the haptic stimulus seems to be affected by visual stimulus when discrepancy exists between vision and haptic stimuli.

By applying the Student's t-test to our evaluation data, significance differences were found in effectiveness, caused by presenting a standard stimulus in three ways, (F(2.18)=36.394, p<0.05).

Figure 17.

Mean grit sizes selected as matches for visual/haptic, and visual/haptic standards; subjects touched an object with a 2.2 mm haptic curvature radius and a 1.4 mm vision one

These results of subjective evaluations for the sharpness of a cube’s edge show that users perceive an edge to be controllable by presenting a duller or sharper CG edge.

We calculated the occupancy rate of haptic and vision stimuli for the evaluations by using the method introduced in Lederman’s paper (Lederman & Abbott, 1981). Haptic and visual influences are calculated by the following equations:

$$\text{Haptic influence}=\frac{\text{Mean( Vision }+\text{ Vision standard)-Mean(Vision standard)}}{\text{Mean( Vision standard)-Mean(Touch standard)}}$$E1

$$\text{Visual influence}=\frac{\text{Mean(Touch}+\text{Vision standard)-Mean(Touch standard)}}{\text{Mean( Vision standard)-Mean(Touch standard)}}$$E2

In these equations, Mean (Touch+Vision standard) is the mean evaluating value of the radius of an edge calculated from all subject evaluations that were presented standard haptic and vision stimuli. Mean (Vision standard) is the mean evaluating value of the radius of an edge calculated from all subject evaluations that were presented a standard vision stimulus. Mean (Touch standard) is the mean evaluating value of the radius of an edge calculated from all evaluations that were presented a standard haptic stimulus.

In the first evaluation, the occupancy rate of the vision stimulus is 57.1% and the haptic stimulus is 42.9%. In the second evaluation, the occupancy rate of the vision stimulus is 77.8% and the haptic stimulus is 22.2%. These results show that when a curvature radius becomes larger, the haptic sensation becomes duller. As a result, the occupancy rate of the vision stimulus increases.

5. Conclusion

This chapter introduced a system that can present visual/haptic sensory fusion using mixed reality. We investigated whether visual cues affect haptic cues. As a procedure to analyze sensory properties, we focused on two features of objects. One is the impression of texture that is intimately involved in the impression of products. The other is the sharpness of edge, which is strongly affected by both visual and haptic senses. From the result of the subjective evaluation on the impression of visual/haptic texture, we can derive an interesting assumption as follows; if we have learned from past experience that a material may sometimes have different haptic impressions (e.g., smooth and rough), we can control the haptic impression of a real object with the material by changing the visual texture overlaid on the object. Preliminary results of subjective evaluations on the sharpness of edge show that users perceive an edge to be duller or sharper than a real one when presented with an overlaid CG edge with a duller/sharper curvature.