Realistic Haptics Interaction in Complex Virtual Environments

1. Introduction

Simulating interactive behavior of objects such as soft tissues in surgical simulation or in control engine of VR applications has been extensively studied. Adding haptic information such as vibration, tactile array, force feedback simulation enhances the sense of presence in virtual environments. Recreating the realistic contact forces the user would perceive when interacting the virtual objects is in general difficult. Current haptic technology only effectively simulates interactive forces for simple cases, but rather limited when considering complex virtual scenes. Desirable properties in realistic haptics interaction include: the stable and smooth reflection forces at a high refreshing rate around 1KHz, and the smooth exhibition of object deformations and physical-realistic behaviors. Physically based haptic deformations and simulation are mostly computationally intensive and not suitable for interactive virtual-scene applications. Even so, integrating non-physical methods with haptic feedbacks is not natural, and cannot provide realistic physical simulation of deformable objects. VR applications strive to simulate real or imaginary scenes with which users can interact and perceive the realistic effects of their action in real time.

People investigate haptic interfaces to perform interactive tasks mainly in two aspects: to explore part of the environment and thus achieve tactile identification of objects, positions and orientations; to actively utilize force sensations to manipulate/deform objects in the touch-enabled immersive tasks. Researches on haptic display are currently focused on tool-object haptic interaction, during which the users feel and interact with the simulated environment through the tool of a given shape including the hand or fingers. The force feedbacks are generated based on the spring/damping linked to the dynamic haptic interface, but the proper values for those material properties are not always easy to derive from measured approaches. Touch-based surgical simulation is not only made to improve realism of virtual environments, but also to provide important diagnostic information through the sense of touch. Palpation is important to sense with hands during a physical examination, in which the doctor presses on the surface of the body to feel the organs or tissues underneath. We propose the novel body-based haptic interaction approach (Chen & Sun, 2006), which models the intrinsic properties of the tool and virtual objects during the touch-enabled palpation in medical simulation. During the virtual palpation force-sensing, the contact/frictional forces are evaluated based on Hertz’s contact theory, and the press distribution with the contact is specified accordingly. Compliant contact models require surface geometry, material properties, and efficient interactive forces simulation. The non-linear viscoelastic behavior of typical tissues is mimicked using a volumetric tetrahedral mess-spring system, and high performance in computing the touch deformation is acquired to exhibit the typical human tissues in the real world.

More information channels have been provided recently to augment visual techniques with haptic methods, giving the modality for visualization and active data exploration. Most haptic devices utilize point interactions, resulting in a conflict between the low information bandwidth and further complication of data exploration. Unlike our sense of vision, haptic manipulation involves direct interaction with objects being explored, providing the most intuitive way of applying 3D manipulation in virtual scenes. Utilizing multi-resolution methods in haptic display provides a possibility to balance the conflict in the complex virtual scene. These combine the elegance of a recursive hierarchical spatial or geometric decomposition of the scene with the ability to provide simplified drawable representations for groups of related subobjects. We study the multi-resolution haptic interaction scheme of hybrid virtual models in the complex scene, in which the hierarchical imposter representations of surface/volume models are constructed. The optimal performance based on haptic-scene perceptual evaluation at run time is employed to meet both the visual and tangible interaction qualities, providing more information of objects transferred by detailed perceptions (spatial & physical) guided by the sense of touch.

2. Background

Users expected to manipulate the virtual worlds through the tool-based haptic interfaces in cyberspace as if in the real world. Many research efforts have been dedicated to this area (Salisbury et al., 2004), during which the users feel and interact with the simulated environment through the tool of a given shape including the hand or fingers. Most studies simplified the haptic tool-object interaction paradigm into multiple point contacts (Colgate et al., 1995), which provide a convenient simplification because the system needs only render forces resulting from contact between the tool’s avatar and objects in the environment. The force feedbacks are generated based on the spring/damping linked to the dynamic haptic interface, and the proper values for those spring/damping constants are not always easy to derive from measured material properties. Lin and Manocha’s groups worked on 6DOF haptic interactions (Gregory et al., 2000, Kim et al., 2002, Kim et al., 2003) that simulated the interaction between two 3D objects through convex decomposing of polygonal models to accelerate collision detection, and finally relayed a combination of penalty-based restoring forces of cluster contacts to the user.

Modeling of soft tissue deformation in tangible medical simulations is of great importance. The goal is to allow virtual tissues responding to user’s manipulations in a physically realistic manner, as if possessing the true mechanical behavior of real tissues. Thus, tangible surgical simulators may become a safe and feasible alternative for enhancing traditional surgical training. Usually the complex surgical tools were modeled out of simple basic components with points and lines in medical simulation to achieve realistic and fast simulations. Basdogan et al. (Basdogan et al., 2001, Basdogan et al., 2004) have overviewed the research on distributed, collaborative haptic systems for laparoscopic surgery, where surgical tools of long, thin, straight probes and articulated tools for pulling, clamping, gripping, and cutting soft tissues were applied. Bielser et al. (Bielser et al., 1999, Bielser & Gross, 2000, Bielser & Gross, 2002) presented interactive open surgery scenarios applied with surgical hooks and scalpels.

Our sense of touch is spatially focused and has a far lower bandwidth in comparison with visual sense that has the largest bandwidth. Coupling interactive haptic rendering in complex virtual environment is important in tangbile scene navigation and exploration. Multi-resolution descriptions of the scene can provide a solution to the conflict between this low information bandwidth and the complexity of the virtual scene. El-Sana & Varshney (El-Sana & Varshney, 2000) introduced a Continuously-Adaptive Haptic rendering approach to reduce the complexity of the rendered dataset. Asghar et al. (Asghar et al., 2001) implemented multi-resolution descriptions of the scene in a haptic environment based on the affine median filter, providing users with view of varying resolution scene. Zhang et al. (Zhang et al., 2002) applied haptic rendering in different detail levels of soft object by subdividing the area of interest on a relatively coarse mesh model and evaluated the spring constants after haptic subdivision. Otaduy & Lin (Otaduy & Lin, 2003) provided a sensation preserving simplification algorithm based on multi-resolution hierarchies of object geometry for faster time-critical collision queries between polyhedral objects in haptic rendering.

Interactive rendering of complex virtual environments demands the desirable properties of tool-based haptic interaction, as the following:

• The interactive forces should reflect the characteristics between the touch tool and soft-tissue objects at an intuitive and stable way;

• The physical-realistic behaviours such as virtual palpation should be simulated smoothly in real-time;

• Multi-resolution tangible scheme should be established to maximize the perceptual information with larger force applied.

Here, the above properties realized in the recent researches are presented. In tangible simulation, the novel body-based haptic interaction approach that simulates the intrinsic properties of the tool and soft-tissue objects during the virtual palpation is presented. Further, the multi-resolution haptic interactive scheme of hybrid models is constructed to provide more detailed perceptions guided by our sense of touch in complex virtual scene.

3. Dynamic Tangible-active Palpation

Palpation is the essential diagnosis technique, commonly used in cancer diagnosis to find the size, consistency, texture, location, and tenderness of abnormal tissues. Currently, most simulated palpation forces were reduced to point-based interaction model with spring-damper linkage to simulate the contact between one or more fingertips and the virtual object. Some special haptics device was created and applied in breast palpation simulation.

The contact problem between two elastic solids that are pressed by the applied force was first solved by Hertz in 1882 (Landau & Lifshitz, 1986), under assumptions: the contact area is elliptical, each body is approximated by an elastic half-space loaded over an elliptical contact area, and the dimensions of the contact area must be small relative to the dimensions of each body. Hertz’s theory yields stresses, deformations, and the shape of the interface formed at the two contacting bodies. These quantities depend on the elastic properties of the two bodies, the geometric shape at the contact, and the force used to push them together. Pawluk and Howe (Pawluk & Howe, 1999a, Pawluk & Howe, 1999b) investigated the dynamic force and distributed pressure response of the human finger-pad based on Hertz’s theory, and developed a quasilinear viscoelastic model successfully explaining the observed measurements. Barbagli et al. (Barbagli et al., 2004) compared four physical models for rotational friction of soft finger contact and extended the god-object algorithm to simulate contact between one or more fingertips and a virtual object.

3.1. Haptic body-based palpation

Instead of point contact with mass only, we describe each contact as the body-based contact with elliptic paraboloid spheres, during the haptic tool-object interaction. Physical properties described by mass, volume, Poisson’s Ratio and Young’s Modulus with contacted objects are computed to reflect the intrinsic physical properties of the objects. Based on them, the contact/frictional palpation force-sensing between finger and virtual objects is simulated and the press distribution of the finger pad is specified accordingly. Hertz’s contact theory based on solid bodies in contact is investigated to simulate the contact forces during the haptic tool-object interactions.

Basic force model

The virtual index finger is modelled as the single-layer sphere bodies bounding the surface of the forefinger in advance. Each sphere body $$s_i(m_i,r_i,v_i,E_i)$$ in the layer has four attributes: $$m_i$$ is the mass of sphere body $$s_i$$ , $$r_i$$ is it’s radius, $$v_i$$ is the Poisson’s Ratio and $$E_i$$ is it’s Young’s Modulus (fundamental elastic constants reflecting the stiffness of the forefinger). In Figure 1, the left figure models show the simulation with one sphere attached to the finger tip, and the one simulated with 8-spheres to construct the volume of sphere bodies. The virtual index finger is modeled with a triangle-mesh surface object. The basic palpation force model between each sphere body and the object is constructed, shown in the righ part of Figure 1, as follows:

$$\stackrel{\rightharpoonup }{F}={\stackrel{\rightharpoonup }{F}}_c+{\stackrel{\rightharpoonup }{F}}_{\mu }+{\stackrel{\rightharpoonup }{F}}_a$$ E1

where $${\overrightarrow{F}}_c$$ is the contact force between two solids based on Hertz’s contact theory specified in equation (2); $${\overrightarrow{F}}_a$$ is the ambient force in relation to the virtual finger, for example, the gravity $${\overrightarrow{F}}_g$$ of the virtual finger and other compensation forces to balance the downward force of stylus tip of the haptic device; $${\overrightarrow{F}}_{\mu }$$ is the frictional force caused by the roughness of the tissue surface in relation to the contact force and the gravity; and $$\overrightarrow{F}$$ is the integrated palpation force applied to the haptic interface.

When multi contacts are detected between the virtual index finger and the interacted tissue, each palpation force is evaluated using equation (1), and the final compound force is applied to the user through the haptic interface (Chen et al., 2007).

Figure 1.

Virtual finger simulation and body-based basic force model.

Palpation force-sensing

Hertz’s contact theory yields stresses, deformation and the shape of the interface formed at two contacting bodies. These quantities depend on elastic properties, the object shape, the relative position of the two bodies at the point of contact and the force pushing them together. Although original Hertz contact load is based on the smooth (frictionless) contact, it can be developed to account for rough (frictional) surfaces. In virtual tangible palpation, the produced biomechanical behavior of the human tissues is evaluated. The virtual index finger is simulated with the properties of silicon rubber. The four typical human tissue categories, including skin, muscle, ligament, and bone, are simulated based on the physical properties specified. Figure 2 shows the tested environment of the virtual index finger and human tissue models, including the cuboid tetrahedral volume interacting via the virtual finger, the ligment, the cortical bone and the upper leg of human body respectively.

Figure 2.

Simulated human tissue models.

Normal contact force: Assuming h << R, and using the inverse of Hertz’s contact theory based on solid bodies in contact, the contact force $${\overrightarrow{F}}_c$$ exerted on the tool by the elastic deformation of the object is expressed below,

$${\overrightarrow{F}}_c=h^{\frac{3}{2}}\frac{4}{3}\frac{1}{l_1+l_2}{\left(\frac{R_1{R}_2}{R_1+R_2}\right)}^{\frac{1}{2}}{l}_1=\frac{1-v_1^2}{E_1}{l}_2=\frac{1-v_2^2}{E_2}$$ E2

where h is the penetration depth between two contact bodies, v_i (i=1,2) is the Poisson’s Ratio and E_i (i=1, 2) is the Young’s Modulus that describe the elastic properties of two contact bodies respectively.

Pressure distribution: The distribution of the pressure over the contact area is given by the radius of the contact circle and the expression of the pressure, which is exerted on point ξ in the contact area. They are defined as follows:

$$a={\overrightarrow{F}}_c{}^{\frac{1}{3}}{\left(\frac{3}{4}(l_1+l_2)\frac{R_1{R}_2}{R_1+R_2}\right)}^{\frac{1}{3}}\overrightarrow{P}(\overrightarrow{\xi })=-\frac{3{\overrightarrow{F}}_c}{2\pi a^2}{(1-\frac{\left|\stackrel{\rightharpoonup }{\xi }\right|}{a^2})}^{\frac{1}{2}}{\overrightarrow{P}}_0=-\frac{3{\overrightarrow{F}}_c}{2\pi a^2}$$ E3

where ξ is measured from the center of the contact region, and a is the radius of the contact area. $${\overrightarrow{P}}_0$$ specifies the contact pressure at the center of the contact area.

Firctional force: The frictional force on the contact area is determined by the contact force and the gravity force, as follows:

$$\begin{array}{l}{\overrightarrow{F}}_{\mu g}=\mu {\overrightarrow{F}}_g=\mu mg{\overrightarrow{F}}_{\mu c}(\xi )=\mu \overrightarrow{P}(\overrightarrow{\xi })\\ {\overrightarrow{F}}_{\mu c}={\displaystyle \underset{\xi a}{\iint }{\overrightarrow{F}}_{\mu c}(\xi )}d\sigma {\overrightarrow{F}}_{\mu }={\overrightarrow{F}}_{\mu g}+{\overrightarrow{F}}_{\mu c}\end{array}$$ E4

where µ is the friction coefficient depending on the roughness of the object, $${\overrightarrow{F}}_{\mu g}$$ is the frictional force in relation to the gravity of each body, $${\overrightarrow{F}}_{\mu c}$$ is the frictional force caused by the contact force. $${\overrightarrow{F}}_{\mu c}(\xi )$$ describes the frictional force of the unit area to the locally exerted pressure. The integration over the entire contact area is superposed to the final $${\overrightarrow{F}}_{\mu c}$$ . Finally $${\overrightarrow{F}}_{\mu }$$ is the integrated dynamic frictional force between the virtual index finger and the tissue.

Figure 3 records the palpation force simulated between the virtual index finger and the touched tissues of bone/skin/ligment/muscle/upper leg. The force is depicted as a regression via a natural logarithm function. Here, the left graph shows the palpation force in relation to the indentation, and the right graph presents the force computed in relation to the local geometry of the contacted tissues. The simulated nonlinear force curves are similar to the typical load-deformation curves exhibited by corresponding human tissues.

Figure 3.

Force evaluation between virtual finger and tissues.

3.2. Dynamic curvature estimation

An important step in our body-based haptic interaction model is constructing the equivalent sphere representing the shape of the object at the contact area dynamically. The mean curvature, describing insight to the degree of flatness of the surface, is applied to estimate the radius of sphere of the object at the contact area. Similar to normal vector voting (Page et al, 2001; Page et al., 2002) of curvature estimation on piecewise smooth surfaces, the mean curvature of contacted area is estimated dynamically during the haptic tool-object interaction.

Discrete approximation

Taubin (Taubin, 1995) showed that the symmetric matrix M_p at point p on a smooth surface,

$$M_p=\frac{1}{2\pi }{\displaystyle {\int }_{-\pi }^{\pi }{k}_p(T_{\theta })T_{\theta }{T}_{\theta }^{t}d}\theta k_p^1=3m_p^1-m_p^2{k}_p^2=3m_p^2-m_p^1$$ E5

has the equivalent eigenvectors to the principal directions { $$T_1,T_2$$ } and the eigenvalues { $m_p^1,m_p^2$ }, where { $m_p^1,m_p^2$ } are related to the principal curvatures { $$k_p^1,k_p^2$$ } through a fixed homogenous linear transformation. As a result, the mean curvature can be acquired by $$k_p=(k_p^1+k_p^2)/2$$ . An approximation of the matrix on a discrete mesh is given by Taubin,

$${\tilde{M}}_p=\frac{1}{2\pi }{\displaystyle \sum {\omega }_i{k}_i{T}_i{T}_i^t}$$ E6

where $${\tilde{M}}_p$$ denotes the approximation of $$M_p$$ at vertex p through the combination of a finite set of directions $$T_i$$ and curvatures $$k_i$$ . $${\omega }_i$$ is a discrete weight version of the integration step and has the constraint $${\displaystyle \sum {\omega }_i=2\pi }$$ . The two principal curvatures can be acquired by the eigen analysis of matrix $${\tilde{M}}_p$$

Curvature estimation

The estimation of the mean curvature at the contact point p is transformed into the curvature voting of the vertices within q-rings’ adjacent neighbourhood $$Adj(p)$$ shown in Figure 4 (where $$Adj(p)=\left\{v\right|Dist(p,v)\le q\}$$ , $$Dist(p,v)$$ is the shortest path connecting point p with point v). All triangles within the neighborhood will vote to estimate the normal at the point, then the vertices within the same neighborhood will vote to estimate the curvature with the normal estimated. All voted normal N_i are collected through covariance matrix. The surface normal N_p at point p with surface patch saliency is evaluated as the corresponding eigenvector with the largest eigenvalue of matrix V_p. Each vertex $$v_i\in Adj(p)$$ has the curvature k_i, along the direction T_i with estimated normal N_p on point p,

$$k_i=\frac{\Delta {\vartheta }_i}{\Delta s_i},T_i=\frac{\overrightarrow{t_i}}{\Vert \overrightarrow{t_i}\Vert },\overrightarrow{t_i}=\overrightarrow{p{v}_i}-(N_p^t\overrightarrow{p{v}_i})N_p$$ E7

where $$\Delta {\vartheta }_i$$ is the change in the angle, and $$\Delta s_i$$ is the shortest arc length fitting from v_i to p. And $$\Delta {\vartheta }_i$$ is obtained by the following,

$$\cos (\Delta {\vartheta }_i)=\frac{N_p^t\overrightarrow{n_i}}{\Vert \overrightarrow{n_i}\Vert }\overrightarrow{n_i}=N_{vi}-(P_i^t{N}_{v_i})P_i{P}_i=N_p\times T_i$$ E8

where $$N_{v_i}$$ is the normal at vertex $$v_i$$ , and $$\overrightarrow{n_i}$$ is its projection to the plane П_v defined at point p with the normal N_p. Through collecting all voted curvatures, the discrete matrix $${\tilde{M}}_p$$ in equation (6) is obtained. Thus two principle curvatures can be computed, and the sign of k_i is the same as the sign of $$T_i^t{\overrightarrow{n}}_i$$ . The mean curvature radius $$1/k_p$$ is evaluated as the simulated radius in equation (2) at the contact area of the soft-tissue during virtual palpation.

Figure 4.

Curvature voting from the vertices in neighborhood.

3.3. Soft tissue behavior simulation

The study of biomechanics shows that soft tissues are non-linear, time-dependent and history-dependent viscoelastic materials. It is difficult to precisely express the complex behavior of soft tissues in computer-generated organs. Here, human organs are simulated with viscoelastic behavior by volumetric tetrahedral mesh-spring systems with attention focused on small deformation restricted into a local area. It is convenient to extract multiple iso-surfaces among the different tissues. The tetrahedral element can support modeling of 3D organs with arbitrary shape.

The volumetric tetrahedral mass-spring system consists of mass points and connected springs along the edges. The Viogt rheological model is used to depict the time-dependent viscoelastic behavior of tissues. The linear springs obey the Hook’s law, whereas the viscous dampers generate a resistance force proportional to the velocity. The dynamics of points are governed by the Newton’s Second Law of motion. The nodal displacement of the ith point (u_i R³) due to an external force F_i is given by the following,

$$m_i{\ddot{u}}_i+d_i{\dot{u}}_i+{\displaystyle \sum _j\frac{{\sigma }_{ij}(|{\stackrel{\rightharpoonup }{r}}_{ij}|-l_{ij})}{\left|{\stackrel{\rightharpoonup }{r}}_{ij}\right|}{\stackrel{\rightharpoonup }{r}}_{ij}=}{F}_i$$ E9

where m_i is the mass of the point i, d_i is the damping constant of the same point, $${\stackrel{\rightharpoonup }{r}}_{ij}$$ is the vector distance between point i and point j, l_ij is the rest length,and σ_ij is the stiffness of the spring connecting two mass points. The right-hand term F_i is the sum of other external forces.

The motion equations for the entire system are assembled through concatenating the position vectors of the N individual mass points into a single 3N-dimensional position vector U. Then the Lagrange’s dynamics equation is satisfied,

$$M\ddot{U}+D\dot{U}+KU=F$$ E10

where M, D and K are the 3N3N mass, damping and stiffness matrices respectively. M and D are diagonal matrices and K is banded because it encodes spring forces which are functions of distances between neighboring mass points only. The vector F is a 3N-dimensonal vector representing the total external forces acting on the mass control points. We can higly reduce the order of the dynamic computing by approximately fixing the vertices far from the acting forces.

In virtual palpations, the virtual index finger is modelled with a triangle mesh of 179 triangles, and the tested models are counted by the nodes and tetrahedrals (cube: 140/432; head: 456/1470; ligament: 601/1900; upper-leg: 728/2550). For dynamic finger-tissue collision detection, a ColDet 3D library is modified to detect multi contacts simultaneously. The working rate of this library is sufficient in considering the complexity of our experiments. The cost of force evaluation is mainly contributed by the curvature evaluation of the contacted tissues. Through restricting the deformation of the object within a local area, the refreshing rate of dynamic deformation during virtual palpation is fast. Our proposed dynamic tangible palpation model can guarantee a high refreshing rate of haptic interaction and meet the requirements of our visual update. Figure 5 illustrates the time-dependent viscoelastic mechanical properties of the simulated soft human tissues. In the left plot, the time step used is 0.01 to simulate the skin and the ligament tissues, and in the right plot, the time step used is 0.05 to simulate the muscle and a portion of the upper leg. The left part of the curve shows that the simulated creep is an increase in strain under constant stress; the right part of the curve shows that relaxation is a decrease in stress under constant strain. With higher elastic parameters in the simulated skin and ligament, the relaxation process is prolonged as it is more difficult to calm down to rest stage.

Figure 5.

Simulated soft-tissue palpation responses.

4. Multi-resolution Haptic Interactions

Developing a flexible multi-rsolution force evaluation scheme of tool-based haptic interactions within a complex virtual environment is of large demand. Compared with visual sense that has the largest bandwidth (Kokher, 1987), our sense of touch is spatially focused and has a far lower bandwidth. Thus, the rate at which information exchanged through touch is far less than that achieved through visual perception. Recent studies in haptic technologies provide more information channels that allow visual techniques to be augmented with haptic methods, giving an additional modality for visualization and active data exploration. Most haptic devices utilize point interactions, resulting in a conflict between the low information bandwidth and further complication of data exploration. Multi-resolution descriptions of hybrid models can provide a solution to the conflict between this low information bandwidth and the complexity of the virtual scene.

To effectively manage scene complexity, the recursive hierarchical descriptions are investigated with the ability to provide simplified drawable representations for groups of related subobjects. Shade et al. (Shade et al.,1996) accelerated walkthroughs of complex environments with hierarchical image caching. Funkhouser and Sequin (Funkhouser & Sequin, 1993) noted that predictive level-of-detail selection could be used for active frame rate control. Mason and Blake (Mason & Blake, 1997) represented extensions of the predictive, but essentially non-hierarchical optimization approach to settle the optimization problem in hierarchical level-of-detail scene descriptions. Mason and Blake (Mason & Blake, 2001) further provided a graphical representation of the state spaces of hierarchical scene descriptions. Chen and Sun (Chen & Sun, 2004) established the body-based haptic interaction model in a unified multi-resolution framework.

Figure 6.

Multi-resolution representations (graphics, haptics).

4.1. Multi-resolution tangible exploration

A detailed scene description of haptic interactions is complicated because it involves tracking continuously changing surfaces of the contacted objects being manipulated; evaluating the penetration depth that identifies the intersection volumes; resulting stress distribution in the materials. To simplify these complications, we employ the hierarchical description of hybrid models for both the visual display and haptic perception to balance the tangible-scene rendering requirements in interactive exploration.

Hierarchical Impostors

Hybrid virtual models, such as surfaces and volumes, are represented as multi-level information for both graphics display and haptic perception in our multi-resolution framework. Here, hierarchical imposters for graphics rendering and haptics perception have been constructed in advance, and selected to meet the real-time tangible-scene interactions. Hybrid objects in the virtual scene are rendered using different levels of graphics and haptics impostors. When the haptic input interacts and navigates the hybrid models, weight factors related with graphics impostors and haptics impostors are evaluated dynamically, to detect when and which of the objects should be traced up or down to a new level of displays. For instance, multi-resolution graphics imposters for triangle mesh surface data is obtained from Loop surfaces through subdivision (Sun et al., 2007), shown in upper rectangle pictures in Figure 6. Moreover, haptics imposters is derived from data model that represents the highest resolution, different curvature maps including mean curvature, Poisson’s ratio and Young’s modulus related with different parts or materials constructed with different adjacent neighborhood applied, shown in lower circle pictures in Figure 6.

The rendering of the interactive tangible scene is the optimal selection of graphics imposters and haptics imposters of the objects in the scene collectively. The pseudo code for controlling the force trigger logic with the haptic imposters is outlined in the following, thus the updated precision of tool-based haptic interface can lead to more or less detailed accuracy of interactive force perceptions.

Figure 7.

4.1.2. Tangible-scene interface

Haptic interactions in the virtual scene involve composing the hybrid objects, and exploring detailed graphics and haptics descriptive properties of the objects. Figure 7 is exampled system interface for hybrid tangible-scene exploration, during which two rendering servo loops, graphics and haptics, are involved. In the graphics rendering, object trees with selected graphics impostors are retrieved and taken into effect in each cycle. The cost for graphics display is the time for each cycle, which is reciprocal of graphics refreshing rate. In the haptics rendering, selected impostors are inserted into the haptic scene as touchable objects within the haptic-input influence range. Similar to graphics display, the cost of haptics is reciprocal of refreshing rate of the haptic system.

In our tangible-scene interface, the left side constructed a tree of control components, including virtual world, mouse control, display views, PHANToM haptic interaction, and navigation tools. Virtual world specifies the space and its center in the virtual scene. Mouse control traces the position and different actions (displacement, zoom, and rotate) related to each button. Display views setup the display attributes of the scene, including the display window, view point and direction, number of lights and location & property with each light. Haptic scene triggers the force feedback associated with the haptic interaction. PHANTOM specifies the coordinate of world, local movement and surface contacted point. Navigation tools specify the objects attached to the 6-DOF PHANToM input, which can be freely moved or rotated during the interactive haptic-scene navigation. Through the hybrid impostor hierarchy, different drawable representations and the haptic properties of virtual models such as stiffness, damping, and spring can be dynamically retrieved during the real-time haptic interactions.

Figure 7.

Interactive haptic-tool interface.

4.2. Interactive scene evaluation

To convey the maximal perception qualities of the objects within the targeted tangible-scene cost, a heuristic value of each impostor computed in advance is defined as the benefit ratio of an object to its cost. The impostors of the objects are selected according to the descending order of their values until the targeted scene cost is met. If two impostors of the same object are both within the cost range, only one with the higher benefit is retained. When any selective update of impostor representations is triggered, the scene cost in graphics and haptics rendering is evaluated in order to meet the optimal performance and achieve the desired perception qualities of the haptic scene.

Two important scene criteria, cost and benefit of graphics and haptics impostors, have to be evaluated in interactive haptics exploration. The scene cost estimates time required to render the objects with different resolution in graphics/haptics processing, involving the coordinate transformations, collision detection, and evaluation of intersection point and normal. The upper limit of scene cost is reciprocal of rendering refreshing rate for graphics and haptics respectively. The benefit evaluates the contribution of impostors to model perception in the virtual environments. Ideally, it predicts the amount and accuracy of information conveyed to the user related with different rendering algorithm and object resolution. It is in general difficult to accurately model human perception and performance. In similarity, the size of perceptible area of an object after rendering is measured. Objects that appear larger to observer contribute more to vision, and the closer objects within the haptic-tool influence region contribute more touch-enabled details to haptics. An influenced area of the contacted force is computed as the constrained region to improve the working rate. During the haptic-scene interactions, our multi-resolution framework selects the imposters that are encapsulated for the optimal scene perfromance, acquiring both continuous graphics display and improved haptic perception qualities.

Figure 8.

Haptics-scene interactions and navigation.

The interactive haptic scene of hybrid models is constructed in Figure 8, with several surface datasets and volume objects placed on table, chair and sofa in the scene. During the haptic-tool interactions and navigation, different impostors of the hybrid objects are selected for the optimal graphics/haptics displays in real time. When haptic tool moves to and interacts with the kettle object, the most detailed kettle data on the table is exhibited in graphics and haptics rendering. The volume data of teapot in the middle of the table is shifted to a higher impostor level, within the influenced region of haptic input. Other objects such as F-shape and vase on the round table, dragon on the chair and bunny on the sofa remain in the same impostor representation, as far away from the haptic input. Figure 8 shows the impostor update of the hybrid tangible objects, while the haptic input passes teapot and interacts with vase on the round table within the influence region. The F-shape and vase in the scene go up to a higher impostor level, when closer to the moving haptic tool.

5. Conclusion

Most studies generally assumed that the contacts are point-based ones, and line/surface contacts are approximated by two or more point contacts. Based on the spring and damping linked to the dynamic haptic interface, the proper values for those dynamic parameters are not always easy to derive from measured material properties. Interactive rendering of complex virtual environments is important in scientific visualization and interactive scene navigation. Studies of haptic rendering of such complex-scene interaction for hybrid virtual models are scare. The focus of our work is to develop the novel body-based haptic interaction model that truly simulates the intrinsic physical properties of virtual objects interacting with the haptic tool. The contact forces are generated using our body-based force evaluation method based on Hert’s contact theory. Physical properties of different object materials are computed to simulate the realistic touch perception between the body-based tools and interacting objects. We have applied the body-based interaction model in virtual palpation simulator, which is more suitable to enhance the touch sensation and exhibit the typical soft-tissue responsive behaviors. The non-linear viscoelastic behavior of human tissues is simulated via a volumetric tetrahedral mass-spring system, and high performance in computing is acquired by limiting the contact influence to reduce the order of dynamic systems. Further, we develop multi-resolution graphics/haptics rendering framework of tool-based haptic interactions and navigation in virtual environments. The hierarchical imposter representations of surface and volumetric models are constructed, and the optimal tangible-scene performance is evaluated at run-time to meet both the visual and haptic perceptual qualities.

In comparison with the other work studied haptic interaction in a special local area of the object (Asghar & Barner, 2001, Zhang et al., 2002), our proposed framework established a valuable attempt to integrate the multi-resolution haptic interactions and graphics rendering in complex virtual environments, providing more detailed perceptions (spatial and physical) guided by the sense of touch. Our future work will continue on investigating the constrained deformation of interested objects with the haptic interaction. We will work on more rigorous finite element model of soft-tissues behavior, and the interactive haptic interafce to effectively distinguish tissues by dynamic features, such as varying geometry shape, haptic textures, and touch-sensitive tenderness. The outmost of our goal is to incorporate more realistic haptic modes in tangible tasks, to realize more realistic real-world properties in the hybrid virtual environments.