Perceptual Issues Improve Haptic Systems Performance

1. Introduction

Since its introduction in the early 50's, teleoperation systems have expanded their reach, to address micro and macro manipulation, interaction with virtual worlds and the general field of haptic interaction. From its beginnings, as a mean to handle radioactive materials and to reduce human presence in dangerous areas, teleoperation and haptics have also become an interaction modality with computer generated objects and environments.

One of the main goals of teleoperation is to achieve transparency, i.e. the complete perception by the human operator of the virtual or remote environment with which he/she is interacting (Lawrence, 1993). The ability of a teleoperation system to provide transparency depends upon the performance of the master and the slave, and of its control system. Ideally, the master should be able to emulate any environment, real or simulated, from free-space to infinitely stiff obstacles.

The design of a transparent haptic interface is a quite challenging engineering task, since motion and sensing capabilities of the human hand/arm system are difficult to match. Furthermore, recent studies are providing more and more evidence that transparency is not only achieved by a good engineering design, but also by a combination of perceptual and cognitive factors that affect the operator ability to actually perceive the stimuli provided.

The current knowledge on operator models reflects two separate groups of results. On one hand, there are guidelines for the design of an effective interface, from a human factors points of view, which include perceptual issues related to the cognitive and information processing of the human operators (see Subsection 2.4). On the other hand, there are several operator models related to biomechanical, bandwidth and reaction time issues (see Subsection 2.5).

In this work we survey the main human factors that concur to the effectiveness of a haptic interface, and we present a series of psychophysical experiments, which can enrich performance in haptic systems, by measuring the mechanical effectiveness of the interface, providing a measure of the perception of a human operator. In addition the experiments are useful to represent the complex behavior of the human perception capabilities, and to propose new ways for enhancing the transparency of the virtual environment, by proposing suitable force scaling functions. In addition, our experience with psychophysics procedures highlights the needs of non-classical approaches to the problem, but the design of this type of experiments is not trivial, thus the need of a dedicated software tool or library arises.

The study of the perceptual capabilities is relevant for the design of virtual reality simulators, and for the specification of haptics applications that overcomes current users limitations. Their study is important for improving telepresence in tele-manipulation system. There is a growing need to not only continue to improve hardware platforms and rendering algorithms, but also to evaluate human performance with haptic interfaces.

In a kinesthetic interaction, since the user lacks direct tactile information, the probe of the haptic device has to firmly penetrate the virtual surface before the user, via force feedback, is able to make use of kinesthetic cues and deduce the features of the body. It is necessary to achieve a compromise between accuracy in tissue discrimination, governed by the magnitude of force feedback, and the temporal and displacement extent of surface penetration, which is tightly related to the probability of damaging the tissue.

In the state of the art, we review earlier results on human bandwidth and biomechanical models of an operator involved in manual control, and the methods to identify the range and threshold of human haptic perception. These earlier results point out the lack of analysis of the interplay between different sensing modes of the human perception system, and the need to move from experiments testing a single factor to experiments involving several interplaying factors.

These considerations motivated the development of a series of experiments, carried out in the Altair Laboratory in Verona, combining multiple biomechanical and human factors. In particular, we evaluate those human factors most relevant to surface contact task in low stiffness environments. We address the study of human factors, among over force detection threshold, reaction time, contact velocity, and minimal penetration depth in a contact task with pliable surfaces.

Psychophysical experiments are conducted using different haptic devices. In the first two experiments we measure the absolute force discrimination threshold of the human hand when grasping, a haptic device, both for onset and for sinusoidal force stimuli. We develop a set of compensation rules, capable of granting higher overall accuracy in perceiving haptic virtual environments, by directly involving the results collected in the previous perceptual experiments. The overall goal is accurate rendering of haptic interactions between a tool and any pliable body.

The third experiment combines the just-mentioned factors in a single task of surface detection, in which the subjects are instructed to halt exploration as quickly as they feel the force due to the contact with a virtual object. We measure the penetration depth that can be used to reliably perceive the contact surface, and the enhancement due to the proposed perceptual based scaling method.

The Chapter is organized as following. In the follow up of this Section we present some relevant findings on biomechanical properties of the human arms and the models and guidelines that have been derived. In Section III a series of psychophysical experiments are presented to consider the relevance of perceptual findings for teleoperation systems; in Section IV we justify the needs for a circular approach between perceptual issues and teleoperation system. An overview of a new library for perceptual experiments is presented in Section V. In Section VI conclusions and future works are pointed out.

2. Relevant Findings from Human Factors

In this section, we summarize some of the main findings relevant to the quantitative measures of the human perceptual capabilities. We start from the bandwidth measurements of the human haptic perception. Then, we describe some of the haptics parameters analyzed using one-factor psychophysical methodology. That is, we review the human capabilities on length, angle, force, and stiffness detection and discrimination; we also consider the perception of the peri-personal space and some measures of human performance in bilateral teleoperation. Furthermore, we advance the guidelines that arise from these findings for teleoperation system. Finally, we describe the human models, mostly biomechanical, that these tests have produced.

2.1. Human Response Characteristics

In manual control, the human perception system does not have one single bandwidth (Burdea, 1996). Human bandwidth is a function of the mode in which he/she is operating. Sensation of mechanical vibration of the skin has been reported as high as 10 Khz, but the ability to discriminate one signal from another declines above 320 Hz. In general, the human hand can sense compressive stress (about 10 Hz), skin motion stimulus (30 Hz), vibration (50-400 Hz) and skin stretch (low frequency).

With respect to specific aspects of teleoperation, there are a number of important sensory inputs to the human operator: tactile, proprioceptive, and kinesthetic ones. Because the bandwidth of the muscular actuation is limited at about 10 Hz, Brooks (1990) argues that a hand controller should be asymmetrical in data flow. In fact, a good hand controller must track hand motions up to 5-10Hz, and must be able to feedback to the operator signals as high as 30 Hz for proprioceptive/kinesthetic sensing and possibly up to 320 Hz low-amplitude vibrational information.

2.2. Experiments on Thresholds Detection

To refine the analysis of the above bandwidth characteristics, measures related to the Just Noticeable Difference (JND), also called the Weber fraction, are used. This measure is the minimal difference between two intensities of stimulation (I vs. I + ∆I) that leads to a change in the perceptual experience. The JND is an increasing function of the base level of input, generally defined as a percentage value by:

$$JND\%=\frac{(I+\Delta I)-I}{I}\times 100$$ E1

In haptics, the perceptual experience is investigated considering several independent factors. That is, the perception of length, angle, and parallelism, the perception of force vectors, and surface stiffness, the relevance of

the peri-personal space, the numerosity judgments are investigated with classical psychophysical methods. Besides, several haptic perceptual illusions and performance in haptic tasks are considered. Several examples of measurements methods and relevant findings are the following.

2.2.1. Length

Length measures are addressed by Durlach et al (1989). They observe that the JND in length measured in discrimination experiments is roughly 1 mm for reference lengths of 10 to 20 mm. It increases monotonically with reference length but violates Weber's law. Similar results are reported by Tan et al. (1992): using a Constant Stimuli paradigm, they find that the JND is not linearly proportional to the reference length L: it is 8.1% for L=10 mm, 4.6% for L=40 mm and 2.8% for L=80 mm.

2.2.2. Angle

The threshold for detecting changes in angle is determined as about 15%, using the magnitude estimation and magnitude production (Newberry et al., 2007). It is argued that both cutaneous and proprioceptive feedbacks are relevant for haptic angle discrimination (Levy et al. 2007).

2.2.3. Force

The JND percentage value for pinching motions between finger and thumb is found to be ranged between 5% and 10% of the reference force (Pan et al., 1991). In a force matching experiment about the elbow flexor muscles, Jones (1989) observes a JND ranging between 5% and 9%. JND is relatively constant over a range of different base force values between 2.5 and 10 N (Allin et al., 2002). Tan et al. (1992) conclude that the force JND is essentially independent from reference force and displacement. It is argued that sensorimotor predictions, visual object information and prior experience influence force perception (Kording et al., 2004).

Detection of force vibration has been studied extensively. The large number of studies results from the large number of variables that can affect the vibratory perception: frequency, duration, direction, contact geometry, contact area, contact force, state of adaptation, skin site, skin temperature, age, and pathology.

In haptics research, the detection thresholds for the somatosensory system have been well characterized in terms of the smallest perceivable amplitude of sinusoidal movements over the frequency range 0.4 and 600 Hz.

The results of these studies are accounted for well by the hypothesis that vibratory detection depends on some critical level of activity in Pacinian afferents at high frequencies (at about 300 Hz, or more) and in a separate set of afferents at low frequencies (at about 40 Hz, or less) which have been identified as Meissner afferents (Johnson et al., 2000). It is observed a decreasing trend in the relationship between threshold and frequency, especially in the range 2 and 300 Hz (Yang et al., 2004). The relationship between threshold and frequency is the traditional U-shaped function within the range 40 and 300 Hz (Brisben et al., 1999).

2.2.4. Stiffness

Using a contralateral limb-matching procedure, Jones and Hunter (1990) calculate the psychometrical function for stiffness. The JDN for stiffness is 0.23, which is three times that reported for elbow flexion forces and forearm displacement. These findings indicate that perceptions are based on sensory signals conveying force and movement information. Shon and McMains (2004), considering stiffness values over 1000N/m, compute the Weber fraction of 0.67 of the base stimulus, and find an overshoot error from 3 to 13mm for different stiffness values, decreasing when values of stiffness increase.

2.2.5. Velocity

In a force control task, using a reference speed factorially varied in the range 1 and 30 mm/s, Wu et al. (2005) determine the upper bound of human force control ability which occurs at or below a velocity of 20 mm/s. Moreover, they find that performance decreases as the velocity of hand motion increases.

2.3. Perception of the peri-personal space

To address more complex stimuli, researchers developed other testing paradigms that include also the effects of other sensory channels, such as visual and tactile sensing, on the peri-personal space.

2.3.1. Force directions

Barbagli et al. (2006) find discrimination thresholds for force directions to be in the range between 18.4, and 31.9%. The results show that the congruency of visual information significantly affects haptic discrimination of force directions and that the force-direction discrimination thresholds do not seem to depend on the reference force direction. In a similar task, Elhajj et al. (2006) find that humans perceive force direction more accurately between the 60 and 120 degree region than the other regions. The perceptual capabilities are also related to training effects, and fatigue.

2.3.2. Objects spatial properties

These results support the statement that the haptic perception of objects spatial properties is systematically distorted (Fasse et al., 2000). That is, what humans haptically perceive as parallel is often far from physically parallel. These deviations from parallelity are highly significant and very systematic. There exists accumulating evidence, both psychophysical and neuro-physiological, that what is haptically parallel is decided in a frame of reference

intermediate to an allocentric and an egocentric one. The results of Kappers and Viergever (2006) show that deviation size varies strongly with condition, exactly in the way predicted by the influence of a hand-centered egocentric frame of reference.

2.3.3. Haptic Illusion

Several haptic illusion are evaluated with psychophysics paradigm, in order to better understand the role of the internal sensorimotor predictions, from both a neural and cognitive prospective, and to create new perceptual experiences. For example, Diedrichsen et al. (2007) report a novel force illusion, in which constant force acting on a hand is perceived to increase when the apparent reason for that force is removed. This illusion arises because of a violation of an internal prediction. When an object is removed from a supporting hand, we expect that the load force generated by the object will be eliminated.

Starting from the peculiarities of the human perceptual system, and, in particular, the sensory saltation illusion, Tan et al. (2000) develop a new haptic interface, capable of presenting haptic information in an intuitive and effective manner. This perceptual phenomenon is a haptic spatiotemporal illusion that, with the appropriate spatial and timing parameters, evokes a powerful perception of directional lines. Their findings show that the saltatory signals share unique and consistent interpretations of directional lines among the group of observers tested.

2.4. Guidelines from Human Factors

Starting from the previous findings in human perception, several works identify the conditions under which haptic interaction displays can enhance the human capabilities, both in accuracy and performance. The results provide a set of guidelines for the effective design of a haptic interface that minimizes perceptual and cognitive aspects of the interface.

Earlier research addresses the teleoperating multifingered robotic hands in which Shimoga (1992) analyzes some specific requirement on the design of dexterous master devices regarding constructional and functional aspects. The constructional issues consist of the isomorphism, portability, motion range capability and accommodation for human hand size variability. The functional issues consist of the bandwidth compatibility with the human hand, which has asymmetric input/output characteristics, the proprioceptive (force limit) compatibility and the consideration of the psychophysics stability of the human hand in sensing force magnitudes and variations.

In their works, Stanney et al. (1998) state that integrating haptic interactions in multimodal systems requires understanding user's sensory, perceptual, and cognitive abilities and limitations. They argue that haptic interaction relates to all the aspects of touch and body movement. This involves not only sensation and perception, but also the motor and cognitive aspects of active movement (that is, self-initiated movement) for which detailed motor plans are created, stored in memory, and compared to receptor feedback from the muscles, joints, and skin. They review several significant human factors issues that could stand in the way of virtual reality realizing its full potential. These issues involve maximizing human performance efficiency in virtual environments, minimizing health and safety issues and circumventing potential social issues through proactive assessment.

Hale and Stanney (2004) indicate a set of guidelines for the design of the kinesthetic (body motion and position) interaction based on a psychophysical motivated approach. (i) To ensure more accurate limb position, use active rather than passive movement. (ii) Avoid minute, precise joint rotations, particularly at distal segments. (iii) Minimize fatigue by avoiding static positions at or near the end range of motion. (iv) Surface stiffness of 400N/m should effectively promote haptic information transfer. (v) End-point forces of 4N should effectively promote haptic information transfer. (vi) Add kinesthetic information to enhance objects spatial location. (vii) Gestures should be intuitive and simple. (viii) Minimize fatigue by avoiding frequent, awkward, or precise gestures. (ix) Avoid precise motion gestures, as making accurate or repeatable gestures with no tactile feedback is difficult.

To the best of our knowledge, the findings on human capabilities in haptics, of which few examples have been summarized above, led to identify guidelines and proposal for the design of new haptic devices. However, complete models of the human operator inclusive of several psychophysics characteristics have been proposed in the literature, to aid in the design of teleoperation systems. The most common models are based on biomechanical concepts, as described next.

2.5. Human Operator Models in Teleoperation

Based on the above findings, a number of biomechanical models have been developed, which are then used to validate manual control as well as perception capabilities of human operators during control task.

One the best known models is the one of McRuer and Krendal (1959), used especially to simulate tracking task of moving targets. The operator's responses, mainly defined as a time delay, will depend upon at least the following factors: the dynamic characteristics of the controlled elements; the type of input or forcing function driving the system; the individual reaction delays, and thresholds during the particular operation; the motivation, attention, previous training, and general psychological and physiological condition of the human at the time of the operation. This model shows a phase shift proportional to the frequency without any associated amplitude change. This human reaction delay of about 150 msec manifests itself as a linear increase in phase with frequency (Sheridan & Ferrell, 1974).

More recently, Townsend and Salisbury (1989), pointed out the difficulties of defining the bandwidth of a force display in terms of a force input-output transfer function, since it depends strongly on the boundary conditions encountered by the device. The performance of a haptic interface is often reported in terms of the dynamic range of impedances it may represent. At the low end, the range is typically limited by inherent dynamics of the interface device, such as inertia and friction. At the high end, the range is typically limited by system stability. A benchmark problem of considerable importance is the implementation of a stiff "wall". A theoretical analysis of stiff wall implementation is presented by Colgate et al. (1993), who develops a criterion for the passivity of a virtual wall in terms of two non-dimensional parameters.

Moreyra and Hannaford (1998) defined a novel method of analysis of haptic perception, the Structural Deformation Ratio (SDR), which makes it relatively easy to quantify some aspects of the high frequency performance of force displays. This model addresses the difficulty of defining the bandwidth of a force display, as pointed out by Townsend and Salisbury (1989).

The inclusion of cognitive effects has been proposed by Corker (1999) using a model of human performance in large scale and complex systems called Man-Machine Integrated Design and Analysis System. This model has long served to engineers in prediction of system performance, and it has also been used to identify performance shortfalls in the human-machine system under a range of anticipated scenarios. This model assumes a functional architecture about the underlying process of human behavior. The human operator model functions as a closed-loop control model with inputs coming from the world and action being taken in the worlds. However, the perceptual module discussed here is composed only by the vision and audition micro-modules, thus lacking a haptic (tactile, kinesthetic, proprioceptive) one.

As summarized in the previous Section, the current knowledge on operator models applicable to force reflecting systems reflects two separate groups of results from haptics and teleoperation. There is the need to merge the two body of knowledge with experiments that would allow to test appropriate combinations of biomechanical and perceptual factors, and to propose methods and instruments which can be relevant for either of them.

3. Current Research

Our research focus is on teleoperation with force feedback. In this area of application there are highly variable forces. For example, in fine manipulation or high dexterity operations very low forces are often involved. Furthermore, these forces are difficult to perceive but very important to accomplish the task. By means of a bilateral teleoperation system we would like to overcome this limitation and provide a perception enhancement that would lead to an increased transparency.

We investigate the haptic perception using psychophysics methods (in particular Threshold detection and Magnitude Estimation). These methods grant precision and thoroughness of results using a high number of trials in an extremely accurate and well-defined setup.

We account for the haptic perception of force, both as onset force as well as sinusoidal waves, in order to quantify how well a user detects changes in force magnitude, and how well one can perceive unappreciable forces.

With the aim of making low force feedback suitable and significant, we incorporate the human perception abilities into the necessary conditions for designing a new scaling function for master-slave systems. That is, after the identification of the JND related to kinesthetic perception, we describe a perceptual-based scaling method that relies on these results and permits to correctly recognize low stimuli generated on the salve side. The validation of the method is made designing a further experiment.

Figure 1.

The experimental setup with (a) NASA-JPL Force Reflecting Hand Controller

haptic device, and (b) MPB’s Freedom 7S device.

3.1. Experimental Setup

3.1.1. Involved Haptic Devices

Psychophysical experiments on force perception are conducted using different haptic devices. In the prior experiment, we use a Force Reflecting Hand Controller (FRHC), developed at NASA's Jet Propulsion Laboratory (Bejczy & Salisbury, 1980). It has been refurbished recently, in particular with the addition of a custom designed controller implemented mostly on a NI PCI-7831R FPGA board (National Instruments, Austin, TX), that drives the device motors (Galvan et al., 2006). This device has 6 Degrees-of-Freedom consisting of one translational and five rotational joints (see Figure 1/a). With this structure, an operator can work with full dexterity in a cubic workspace of 303030 cm³.

In the latter experiments, in order to render the virtual surfaces, we involve a high performance Freedom 7S force-feedback device (MPB Technologies Inc., Montreal, Quebec, see Figure 1/b). The pen-hold grasping configuration is involved by concurrently using the thumb, index, and middle fingers. The range of motion of this device allows for hand movements that pivot at the wrist. In translation, the position resolution is 2 m; the resolution in force rendering is 40 mN. The update and log rate is > 2 kHz.

3.1.2. Psychophysics Procedure for Threshold Measurement

The Green’s (1993) Maximum-Likelihood adaptive procedure of psychophysics measurement, in the 2 alternative forced-choice paradigm, is involved to measure the absolute threshold; this procedure promises highly efficient trial placement and threshold estimation, minimizes the number of trials, and then the session duration.

During each trial, different controllable forces are applied in successive intervals to a single joint, generated by the force-feedback device. Subjects were given a choice of 2 alternatives, and they had to select the one containing the greater one. That is, the subjects know that exactly one alternative contains the stimulus s_i, and that the other has the comparison stimulus s_c.

After each stimuli presentation, a set of candidate logistic psychometric functions is fitted to all the data collected up to that point, and the likelihood associated with each function is computed. Starting from these parameters, a further comparison stimulus is calculated. The final estimate of threshold was extracted from the most likely psychometric function after a number of trials.

3.1.3. Statistical Analysis

Statistical analyses are conducted separately for each subject and for aggregate data. Every group analysis includes a factor for individual subjects so that differences between subjects are not counted as random variation; this makes each analysis more sensitive to the stimulus parameter being varied. Unless otherwise indicated, all statements referring to statistical significance are based on the criterion of probability level p< 0.001.

3.2. Force and torque magnitude discrimination thresholds

Our aim is to identify whether force intensities and orientation are associated with different values of JND%. This finding could let us discover a force threshold felt by the human operator and distinguish the most sensitive directions for the arm, and thus allowing us to identify one or more suitable scaling functions for force-feedback in haptic environments.

The purpose of the experiments described here following is to explore the differences in ability of a person to discriminate a wide range of force intensities applied along the axes of a reference frame positioned at the hand-grip.

3.2.1. Experimental Setup

For each joint of the FRHC haptic device, 10 reference forces or torques s_i were identified among the physical stimulus domain. The forces ranged from 0.4 to 9.0 N and the torques from 0.02 to 0.50 Nm. The comparison stimulus presented on the next trial was the sweet-point of the most likely psychometric function (Green, 1993). Each stimulus was applied for 1,200 ms; the interval between different stimuli was 300 ms. No feedback was given during the experiment. The participants had to take a break among runs at about every 7 minutes and whenever needed.

Direction		Reference Stimuli
		s 1	s 2	s 3	s 4	s 5	s 6	s 7	s 8	s 9	s 10
X	Reference [N]	-5.99	-3.28	-1.80	-0.87	-0.54	0.50	0.83	1.79	3.26	5.98
	JND%	15.3	16.3	18.0	18.1	25.7	31.6	22.7	15.7	17.0	9.6
Y	Ref. [N]	-7.21	-3.87	-2.06	-0.80	-0.51	0.41	1.07	2.29	4.07	7.35
	JND%	11.62	12.56	19.29	36.93	25.33	29.58	32.66	24.87	13.25	12.16
Z	Ref. [N]	-8.77	-4.71	-2.52	-1.01	-0.60	0.62	1.01	2.30	4.42	8.33
	JND%	15.73	16.95	19.92	56.47	57.06	48.57	32.47	20.31	12.96	11.25
R	Ref. [Nm×~10 ^-2]	-32.93	-16.87	-8.17	-4.50	-2.78	2.59	4.78	10.25	18.22	32.94
	JND%	12.65	23.25	45.06	35.88	41.99	63.02	56.46	31.48	23.85	13.79
P	Ref. [Nm×~10 ^-2]	-21.24	-12.02	-9.40	-6.50	-3.60	1.95	6.87	11.37	15.43	29.72
	JND%	11.13	16.20	18.56	17.56	66.73	"/ 100	46.14	31.61	28.19	14.65
Y	Ref. [Nm ×~10 ^-2]	-41.22	-20.86	-15.07	-8.67	-3.50	4.84	11.86	18.86	25.20	47.48
	JND%	12.11	19.30	21.16	24.02	"/ 100	46.04	30.86	27.48	20.49	12.21

Table 1.

Mean values of the threshold and slope distributions estimated for the reference forces s_i for each joint.

Figure 2.

JND versus reference force and torque. Each point is the participants' threshold. The blue line maps the median values; the dashed grey lines map the lower and the upper hinge.

3.2.2. Results

Data were analyzed in order to estimate the perceptual threshold for the reference stimuli. Results are reported in Table 1. For each joint, a between-subjects ANOVA was conducted to determine if there were significant differences among the perceptual thresholds due to the different reference values with respect for the different subjects' thresholds. Significant differences in the JND along the stimulus continuum were observed for all the translational and rotational axes.

In Figure 2 the force JND is plotted versus the reference force for all the joints. We observed a non-linear relationship between the reference stimuli and the JND. Considering the lower reference stimuli, the force stimulus and the JND appeared to be inversely associated: the lower the force or torque applied, the higher the perceived JND value.

A direct result of this experiment is the identification of force thresholds for each hand direction and many indirect results are coming out from intra- and post-experimental observations. The observation of the asymmetries in the JND plots leads us to an in depth investigation of the human workspace, not considering only the reachability but also the accuracy of each region of the workspace.

The perceptual thresholds for intense stimuli were higher than in the foregoing experiments: our findings provide evidence that the perceptual threshold or torques can be treated as linear for intense values, with an average value of about 15% of the reference force. But, low forces (i.e. below 0.5 N) cannot be perceived adequately with the current setup. Only very large differences between different stimuli are perceived and recognized by the neuro-musculo-skeletal system. Besides, the mechanical capabilities of the involved device do not allow considering these force range. Thus, we do not know exactly how these restrictions are due to the specific 6 DoFs haptic device or to human peculiarities.

3.3. Perception of harmonic forces

To the best of our knowledge, there is only few data on the perception of very low forces, and no available data on the relation between force discrimination threshold and frequencies signals. Therefore, it is necessary to first estimate the absolute detection thresholds for force signal (both presented as onset force, and as a sinusoidal wave), in the form of a force-magnitude versus temporal-frequency function, involving a high precision haptic display.

With this experiment, fully presented in Vicentini & Botturi (2009), we measure the absolute detection threshold (i.e. the smallest amount of stimulus energy necessary to produce a subjective sensation) for the hand in a pen-hold configuration (typical for using haptic devices), for both low force signal and for sinusoidal wave signals. We hypothesize the lowest detection threshold for wave signals. With the validation of our hypothesis, we can assert that there is an intensity range in which low forces are not perceivable, but wave signals, characterized by the same intensity, are. From these results it is possible to identify a particular signal manipulation, which can allow enhancing the perception of sub-threshold force intensities.

Figure 3.

Prototypical representation of the force signal: (a) onset force, with no frequency components, (b) sinusoidal wave.

3.3.1 Experimental Setup

The experimental design for absolute force detection is defined by the factor force signal, defined by five levels, generated according to (a) an onset force, with no frequency components, or to (b) sinusoidal waves at 40, 100, 300, and 600 Hz (a prototypical representations is depicted in Figure 3). Subjects repeated each session 4 times. The presentation order was fully randomized.

Figure 4.

Median values in absolute force estimation for all the subjects (strong line, with standard error bars), and for each subjects (dotted lines).

3.3.2. Results

Figure 4 reports the median values in absolute force estimation for all the subjects, and for each subjects. It shows the critical values for the minimal force detectable by a subject concurrently using the first three fingers, like grasping a pen. It is evident that the force thresholds decrease in detecting high frequency force signals.

With our experimental setup, when detecting onset force, the discrimination threshold is 39.28 mN (Interquartile Range (IR) from 24.40 to 46.68 mN), for sinusoidal 40 Hz wave is 24.49 mN (IR 16.99 to 32.96 mN), for 100 Hz is 19.65 mN (IR 12.71 to 25.75 mN), for 300 Hz is 17.30 mN (IR 12.22 to 22.76 mN), and for 600 Hz is 8.37 mN (IR 6.17 to 20.55 mN).

An ANOVA (force signals × subjects) was conducted to verify whether the significance of the differences in force thresholds. There are clear differences in absolute force thresholds (F_4,106=25.62, p < 0.001). The Tukey's Honestly Significant Difference post-hoc test (HSD) reveals that the onset force threshold is always greater than the wave signals ones. There is also a significant difference between the 40 Hz force wave and the 300 Hz one: no improvements in force detection are observed for higher values.

As shown in Figure 4, we observe relevant differences among subjects’ thresholds. Two subjects report an outlier behavior in force detection of high frequency waves: they show a constant trend in force thresholds for force waves = 300 Hz. That is, while there is a clear difference between onset force and wave signals perception, it is not observable the well-known decreasing trend from 40 to 300 Hz (Brisben et al. 1999).

Starting from these results, we can now assert that there is an intensity range in which low forces are not perceivable, but wave signals, characterized by the same intensity, are. We argue that this perceptual range can be successfully involved to enhance the presence experience, by letting a user to accurately feel the contact with a virtual surface, without involving dedicated display.

Figure 5.

Force signal manipulation according to Eq. (2) for over-thresholds forces, and to Eq. (3) for intensities under the onset force threshold

4. Signal Manipulation Based on Force Perceptual Thresholds

In this section we show how to build a perceptual-based force signal manipulation that can be embedded in a bilateral teleoperation system in order to improve performance in terms of the amount of force absolute detection that can be perceived by the user. A suitable signal processing is needed to improve the force feedback and moreover the operation performance, without compromising the stability of the teleoperation system.

Our proposal is aimed at augmenting the perception, for a human being, by superimposing, on undetectable force values, frequencies components generated by a common haptic device, without using any specific tactile display, to carry out a teleoperation task.

This is a new approach to force augmentation in teleoperation, since the acquainted concept of signal manipulation in this field concerns mainly constant amplification or variable force scaling. As originally proposed by Okamura et al. (1998), we use stimuli in frequency domain for very low forces. But, now, this concept is used in a standard teleoperation scenario to present only the sub-threshold, not-perceivable forces. Meanwhile, the use of common haptic devices can cover all the force range needed for a contact task with pliable surface.

With the following experiment setup we are aimed at accounting for the effectiveness of the proposed perceptual-based signal manipulation. We define a surface contact task characterized by a low stiffness (pliable) virtual surface; we manipulate the force signal, which is proportional to the penetration in the virtual wall, by substituting the sub-threshold not-perceivable forces with wave force signals, without changing the force intensity. A vibration waveform is superimposed onto the force signal sent to the actuator motors. With this approach, rather than amplifying the sub-threshold forces, we enhance the contact with a pliable surface minimizing penetration depth, according to the “perceptual based” capabilities in high-frequency detection (see Sect. 3.3).

4.1. Experimental setup

The haptic stimulus consists in a virtual surface, placed in at l = 8010 mm far from the initial position of the red circle. The virtual surface is not visually rendered, so as not to bias the subject with respect to the location of the surface contact point.

Subjects are asked to a) move the probe with the Freedom haptic device along a designed direction until they feel the surface; b) once contact is perceived, subjects are instructed to instantly move backwards.

As depicted in Figure 5, when the tip penetrates the virtual object, force F_p is rendered according to

$$F_p=-k\cdot D_p$$ E2

where k is the stiffness value, D_p is the penetration inside the virtual surface. When signal manipulation is applied, if F_p is smaller than the absolute force threshold for onset force, force F_p is rendered according to

$$F_p=-k\cdot D_p\cdot (1+\sin (2\pi \varpi t))/2$$ E3

where is the sinusoidal wave frequency, t is the time.

Two experimental factors (stiffness and signal manipulation) are combined in a 33 factorial design. Stiffness values k have been chosen amongst commonly reported values of stiffness for the human body: 300, for a surface similar to the human fat, 700, and 1100 N/m, for a harder surface similar to the human skin or muscle tissue. The further experimental factor is signal manipulation. We test whether there are differences in penetration depth for the following three conditions: a) no signal manipulation, forces are rendered according to Eq. (2); b) not perceivable forces are rendered according to Eq. (3) and = 40 Hz; c) not perceivable forces are rendered according to Eq. (3) and = 300 Hz.

4.2. Results

We evaluate the relevance of the position of the virtual surface and of the effective contact velocity on the penetration depth. A robust regression is conducted over the collected data. No significative relevance of the surface position (t-value = -1.045, p = 0.296), nor of the effective contact velocity (t-value = -1.766, p= 0.059) is observed.

The penetration depth collected within the factorial design is plotted in Figure 6. Data shows that penetration depth inversely decreases as surface stiffness increases. Furthermore, we observe that there is an improvement in the task performance given by the signal manipulation. That is, the penetration inside the virtual surface is always greater when the surface is rendered according to the lone Eq. (2) than when forces are rendered with high frequencies sinusoidal waves, as expressed in Eq. (3). In these latter conditions, we observe an average decreasing in penetration depth equal to 10.33 % with respect to the previously mentioned one.

We employ a two-way ANOVA (stiffness × signal manipulation) to measure the significance of these differences in penetration depth values. The second-order interaction between stiffness and signal manipulation is not significant (F_4,1595 = 1.33, p = 0.257), while the first order factors are. The relevance of the factor stiffness is confirmed (F_2,1595 = 160.47, p < 0.001) with an high performance device, and with a less accurate device (Vicentini & Botturi, 2008).

Figure 6.

Penetration depth is plotted against stiffness values for pooled data. The solid lines refer to the no signal manipulation condition, expressed by Eq. (2); the dotted lines refer to the signal manipulation at 40 and 300 Hz, as expressed in Eq. (3).

The lower penetration depth found in this study indicates that the perception of surface contact is impaired with the resolution and the inertial mass of the haptic device.

As hypothesized, the factor signal manipulation is relevant (F_2,1595 = 6.44, p< 0.002) for the task. There are clear differences in maximal penetration depth due to the involved force signal processing. The HSD test reports significant differences between the onset force and 40 Hz conditions (p = 0.0018), the onset force and 300 Hz conditions (p = 0.0165), and no significant differences between the 40 Hz and the 300 Hz conditions (p = 0.776).

These results provide evidences on the effectiveness of the proposed perceptual based signal manipulation, especially when a 40 Hz force sinusoidal wave is involved.

The approach here presented is intended to complete the panorama of methods used to improve the human perception in case of environment interaction through devices. Methods based on tactile display (Kontarinis & Howe, 1995; Goethals et al., 2008) are limited to the specific device, which is not able to supply the complete force range of an interaction task. The force scaling method proposed in Vicentini et al. (2007) is useful in case of wide range of forces and in different directions. In this work the device has to support the frequencies signal manipulation but at the same time has to undertake a complete contact task. Therefore usability of those methods is for different purpose, in one case touching mainly virtual surface, in the other contact in classical teleoperation, in our case we propose an interaction with pliable surface.

5. PsychoGear: a Psychophysics Library for Haptics Experiments

Only few works in the literature bring together physiological, perceptual and cognitive factors, such as velocity, force, reaction time, mental workload, EMG measurement, and transmission delay. The main reason is that experiments testing a number of factors are not trivial, and the design implementation has to carefully take over the synchronization and the interface between different stimuli.

We have experienced the difficulties to implement a multi-factorial experimental design, even harder when the subject has to interact with the environment where there are noisy sensors, delayed stimuli and complex devices. The experiments just mentioned show the needs of multi-factorial design, in which several cognitive factors have to be evaluated in order to model the human perception in haptic system.

We are developing PsychoGear, a new psychophysics library to control all the aspects of the experimental setup, maintaining the rigor of the psychophysics methodology within a haptic system. The library will be able to control the experiments by governing stimuli generation and synchronization, data acquisition from different sensorial channels and data logging.

5.1. Features

Our approach follows the research line described in Anderson (2001), that is, to combine multiple psychophysics measurements within a full factorial design. We are working on a modular approach that arises from the decomposition of an experiment in a series of psychophysics methods, each one composed by a variable number of trials. For each trial we consider the presentation of a stimulus, the hardware source, the psychophysics procedure type (classical or adaptive) and the logging of the subject's response.

The focus of the approach described is on the infrastructure, not on a specific method. We want to provide a framework in which including a new experimental method is quick and easy. The user can define the design and choose the parameters without dealing with the underlying code because we have decoupled the programming phase from the utilization one. Moreover, we pursue the code reusability and all the implementations such as trials, stopping rules, generation of the next stimulus, and data collection are stored in the framework.

With this library the user can concentrate on the design of the experiment putting together methods he/she needs, the programmer can rely on the framework to implement new details, without even know the psychophysics purpose. Finally, we point out in this library the possibility to have multi-factorial design and easy device integration.

As stated above arranging experiments with factorial design is a complex task, in which several cognitive factors have to be evaluated. The design is even harder when the subject has to interact with a real environment. This is a possible scenario when dealing with human perception in a real teleoperation task, taking into account noisy sensors, delayed stimuli and complex devices.

In order to control all the aspects of such a complicated setup maintaining the rigor of the psychophysical methodology and the requests from a teleoperation system, we are working on the development of a new psychophysics library.

Starting from the idea of the Psychophysics Toolbox extensions (Brainard, 1997) developed for MatLab, and the PsychoPy-Psychophysics software in Phython (Peirce, 2007), we are looking at providing C++ classes and methods to cover several aspects, from stimulus presentation and response collection using classical and adaptive procedures, to simple data analysis, such as psychometric function fitting.

We are working on a modular approach that arises from the decomposition of an experiment in a series of psychophysical methods, each one composed of a variable number of trials. For each trial we consider the presentation of a stimulus that may deal with the hardware, both with classical and adaptive procedures, and the log of the subject's response.

The novelty of the approach is that we want to provide a framework in which the addition of a new method is easy and immediately integrates with the overall design of the experiment. We do not focus on a specific method but we provide the infrastructure. The aim is to relief the programmer from redefining the experiment when a new psychophysical method is needed and to let the final user to concentrate on the right choice of parameters without dealing with the underlying code.

Figure 7.

PsychoGear: Experimental Psychophysics Library UML Class Diagram.

5.2. Class Structure

In Figure 7 we show the object hierarchy: an Experiment contains an arbitrary number of methods, each Method contains a State class, which contains all the parameters needed for a correct execution of all the sub-component, a Tracking class, that controls the stimuli intensities, a Stopping class, that checks the end of the method, and a Trial class, that manages the stimulus presentation, Stimulus, the response acquisition, Response, and logs the data, Logger.

Each class has a simple and easy to use interface. The main component of this framework is the Method class that supervises all the different control aspects of the experiment procedure. We have to present a sequence of stimulus intensities, that are known a priori or settled at run time, and for this purpose the Tracking class gives basic commands to control the evolution of the stimuli in the procedure.

A very important aspect is the procedure ending management. The Stopping subclass checks the end of the procedure and also allows easily change between different stopping rules. Moreover following the psychophysics definition, Trial class is implemented with a Stymulus subclass, that manages the stimulus presentation, and a Response subclass, which controls the responses acquisition. In addition, a Logger subclass is implemented to save all the meaningful parameters that describe user activities data.

Besides the psychophysics procedure uses some statistical tools, for example probability distribution and random number generator therefore the Mathtool class is implemented to contain all the auxiliary mathematical functions.

It is common to design a real experiment that involves more than a single psychophysics procedure (for example, two staircase procedures with random stimulus presentation). So we need to control which procedure has to be executed, how many trials have to be presented, when the entire experiment finishes. To manage this type of design, the Experiment class manages different methods defining a global end criteria and a switching rule between methods in every step of stimulus presentation.

The specific parameters of the classes involved in the experiment are stored in the State class. This class contains all the parameters in the same place, so it is easy for every sub-class to access useful sharable information.

When an experiment is clearly designed, it may be seen as a set of psychophysics procedures and parameters (i.e. stimulus intensities, starting values, stopping rule, presentation time). Especially during the pilot testing, the user needs to tune some parameters to fit the current design to the goal of her/his study.

To make this operation easier, we set a unique XML configuration file, where the user can easily choose the components and set the parameters of the experiment. The ExperimentBuilder class reads the configuration file and initializes the experiment with the correct sub-classes and parameters.

5.3. Developing new experiments

The proposed psychophysics library takes advantage from the experience of other implementations and the needs of the haptic perception experiments. The novelty of this library is the native management of the psychophysics procedures and of the haptic system; moreover some problems from the existing libraries are solved in this implementation ensuing the software engineer suggestions (i.e. object oriented code organization, haptic device support).

The library architecture is not only due to software engineering, but is meaningful for the organization and the development of every experimental design, in haptic research is even more important, given the complexity of the experiments designs. We are extending the implementation of the basic components, adding, for example, the support for all the fundamental adaptive methods, as discussed in Leek (2001). Once the base code allows a wide set of example and a fast building process we will use them as an “how to” for our library, to allow an easy and fast startup time for new users.

We are also widening the hardware device support and to improve their integration with other source of stimulus devices, i.e. audio/video, with a unified synchronization protocol. We are introducing support for OpenGL to improve the rendering of visual stimuli, and we are working on support for OpenCL to obtain faster parallel computation (i.e. in psychophysics functions fitting evaluation).

Moreover a graphical user interface is being improved to drag and drop the components during the creation procedure and to manage the experiment execution, broadening the PsychoGear library use both for researchers and for students.

6. Conclusion

In this work we present how the results of psychophysics experiments in human perception can improve the telepresence of a haptic system.

In the past, perceptual experiments were carried out to obtain quantitative measurements of human perceptual capabilities. From there, information about peri-personal space and fusion with others sensory channels was extracted and biomechanical models were developed. At the same time, a lot of work was spent about the concept of transparency of teleoperated systems from a pure engineer point of view, i.e. control and mechanical design.

We consider these two aspects as the sides of the same coin. We introduce perceptual experiments to support our vista where the two points of view converge and to provide a methodology of research. We introduce two experiments about force perceptual thresholds, both for high and low intensities. We analyze the results and formulate a scaling model based on the perceptual threshold: in a teleoperation system we use stimuli in frequency domain to let the user perceive sub-threshold, not-perceivable forces generated by the interaction with pliable surfaces. That is, rather than amplifying the under threshold forces, we enhance the feeling of contact with any pliable body by minimizing penetration depth, according to the “perceptual based” capabilities in high-frequency detection.

The second step is concerned with validating this scaling method in a task of pliable surface contact. Hence, we analyze the conditions under which haptic displays can enhance human accuracy and performance in a teleoperation task, by presenting an innovative force scaling for bilateral teleoperated system. We focused our attention on human perception capabilities and, we exploit the human ability to perceive low forces as onset or sinusoidal stimuli. We have used acquired data in order to obtain the proper modifications needed to enhance the information contained in the force signal coming back from the environment by defining a function that maps the not-perceivable forces to let the user completely interpret the interaction with the environment.

Data collected from perceptual experiments can provide guidelines for the design of better haptic devices, but we think that these results are also useful in order to obtain one or several models of the human operator from a psychophysical point of view. These models can go beyond the simple mechanical and muscular representation of the human/machine interaction since they can catch the way human reacts to haptic stimuli and how the sensor fusion strategies works, thus permitting an useful enrichment of force information with visual and acoustic clues.

The main goal refers to the identification of the haptic human space in order to exploit its peculiarities. The measurement of perceptual abilities and the possibility to find relations among perceptual and cognitive factors permit to broaden the concept of transparency and telepresence providing the values to assign to specific parameters in order to maximize the perception with respect to specific tasks.