Method for Objectively Evaluating Psychological Stress Resulting when Humans Interact with Robots

1. Introduction

Most of us have seen robots in movies, animations and comic book stories, so the word “robot” tends to conjure up images of fictional robots rather than the real thing. The robots in Japanese cartoons such as Astro Boy and Doraemon have human-like social skills, and their physical abilities make it possible for them to live alongside humans without any difficulties. In reality, robots are quite different from these fictional creations. At least, the robots of the early 21^st century are still unable to interact smoothly with humans (Norman, 2007). Due to the large disparity between the fictional image of robots and their actual appearance, people sometimes feel stressed when confronted with robots. To facilitate smoother interactions between humans and robots, we must not only to improve the intelligence and physical ability of robots, but also find some way of evaluating the psychological stress felt by humans when they have to interact with robots. To develop robots that can interact smoothly with humans, we need to be able to ascertain the psychological and physiological characteristics of humans by evaluating and analyzing the stress they experience in everyday activities, design robots based on human characteristics, and evaluate and study these robots. In short, stress evaluation is a key requirement for the realization of smooth interactions between robots and humans.

In this chapter, we discuss methods for objectively evaluating and investigating the psychological stress that people experience when interacting with robots. For the evaluation of stress, we used acceleration pulse waveforms and the saliva constituents which are biochemical stress markers. These were used to evaluate the psychological stress of a surgeon using a surgical assistant robot.

A surgical assistant robot is a robot that interacts with a surgeon and is situated in contact with the patients to provide support for surgical operations. Interaction with humans is of greater importance for surgical assistant robots than for any other type of robot. A laparoscope robot is one robot of this type that is put to practical use and is a typical example of a robot where interaction with humans is important. This is a robot that is used instead of a human camera assistant in order to hold the laparoscope in position during laparoscopic surgery (Jaspers et al., 2004). Laparoscopic surgery is a technique where surgical tools and a laparoscope are inserted into the patient’s body through small holes in the abdomen, and the surgeon carries out the surgery while viewing the images from the laparoscope on a TV monitor. Laparoscopic surgery has grown rapidly in popularity in recent years, not only because it is less invasive and produces less visible scarring, but also because of its benefits in terms of healthcare economy, such as shorter patient stays. The most important characteristic of this technique is that the surgeon performs the operation while watching the video image from the laparoscope on a monitor instead of looking directly at the site of the operation. Thus, an important factor affecting the safety and smoothness of the operation is the way in which the video images are presented in a field of view suitable for the surgical operation. Manipulation of the laparoscope is not only needed for orienting the laparoscope towards the parts requiring surgery, but also for making fine adjustments to ensure that the field of view, viewing distance and so on are suitable for the surgical operation being performed. A camera assistant operates the laparoscope according to the surgeon’s instructions, but must also make independent decisions on how to operate the laparoscope in line with the surgeon’s intentions as the surgery progresses. Consequently even the camera assistant that operates the laparoscope must have the same level of experience in laparoscopic surgery as the surgeon. However, not many surgeons are skilled in the special techniques of laparoscopic surgery. It is therefore not uncommon for camera assistants to be inexperienced and unable to maintain a suitable field of view, thus hindering the progress of the operation. To address this problem, a laparoscope robot was developed to hold and position the laparoscope instead of a human camera assistant. Figure 1(a) shows how laparoscopic surgery is conventionally performed with a human camera assistant operating the laparoscope, and Figure 1(b) shows how laparoscopic surgery is performed using a laparoscope robot. When using a laparoscope robot, the laparoscope is held and positioned by the manipulator part of the laparoscope robot which is situated beside the surgeon and is operated by a human-machine interface based on speech recognition or the like.

Figure 1.

a) Conventional laparoscopic surgery where the laparoscope is operated by a human camera assistant. (b) Robot-assisted surgery where the laparoscope is operated by a laparoscope robot.

Laparoscope robots have already been made commercially available and are in widespread use. These include Hitachi’s Naviot™ (Kobayashi et al., 1999; Tanoue et al., 2006), the AESOP™ made in the US by Computer Motion (now known as Intuitive Surgical Inc.) (Sackier & Wang, 1994), and EndoAssist™ made by Prosurgics (Finlay, 2001). These commercial products all move according to the surgeon’s instructions. Meanwhile, although still at the research stage, there are other systems in which the surgeon’s movements are autonomously determined by the robot which positions the laparoscope automatically. A typical example is the laparoscope positioning system developed by Nishikawa et al. (Sekimoto et al., 2009; Nishikawa et al., 2008; Nishikawa et al., 2006).

Laparoscope robots are generally evaluated by measuring work efficiency, precision and error rates, and by using interviews and questionnaires to gather the opinions of surgeons. In cases where the interaction between laparoscope robots and the surgeons operating them resulted in bad feelings, the result was that this drawback worsened the overall performance of the system even if the robot performed excellently in all other aspects. It is therefore necessary to evaluate stress by using interviews, questionnaires and the like. However, interviews and questionnaires produce subjective results that tend to be rather vague, and it is also possible that the results are affected by the human relationship between the examiner and examinee. For the objective measurement of stress, there is growing interest in methods that use biological stress responses.

The concept of biological stress responses was defined by the physiologist Hans Selye as “the nonspecific response of the body to any demand upon it” (Selye, 1936; Selye, 1974). Since stress appears to originate from very complex mechanisms, not only do different people respond differently to stimuli, but even the same person can exhibit a range of different responses to the depending on whether the stress is comfortable or uncomfortable, psychological or physical, and so on.

In the field of physiology, biological stress responses to psychological stress stimuli take place in the autonomic nervous system and endocrine system. In biological stress responses of the autonomic nervous system, sympathetic nerves produce a very fast biological response in which the activity of sympathetic nerves takes priority, and a biophylactic mechanism acts to resist the stress stimulus. In biological stress responses of the endocrine system, processes such as hormone secretion from the adrenal cortex causes a biological response that changes the organism’s internal environment so as to keep it in a suitable state.

Methods for the evaluation of biological stress responses include biochemical methods that measure stress-related substances in biological samples of blood, saliva or the like, and methods that involve performing a statistical dynamic analysis of physiological markers such as blood pressure and heart rate.

In the following section, as a typical stress evaluation technique, we describe the evaluation of stress based on biochemical markers and acceleration pulse waveforms.

2. Evaluation of stress with biochemical markers (saliva, urine)

Stress responses can be generally distinguished by two systems — the hypothalamus – sympathetic nerves – adrenal medulla system (sympathetic-adrenal-medullary axis: SAM) and the hypothalamus – pituitary – adrenal cortex system (hypothalamic-pituitary-adrenal axis: HPA). When an excessive stress is loaded, this is reflected as changes in biochemical markers in blood, urine and saliva (Figure 2).

Figure 2.

Physiological reaction to stress loading

The SAM system corresponds to the response of the autonomic nervous system, where the stimulus of stress load is transmitted to the cerebral cortex and causes the catecholamines (epinephrine, norepinephrine, etc.) to be released via the hypothalamus, either directly from the autonomic nervous system or indirectly via the adrenal medulla. These catecholamines and related substances can be useful as stress markers. On the other hand, the HPA system corresponds to the response of the endocrine system, where the stress stimulus is transmitted to the cerebral cortex and causes corticotropin releasing factor (CRF) to be released from the hypothalamus, promoting the release of adrenocorticotropic hormone (ACTH) from the pituitary gland and the secretion of glucocorticoids such as cortisol from the adrenal cortex. These pituitary and adrenal cortex hormones can be also useful as stress markers.

In the case of evaluating the stress when people use robots or work together with robots, it is not recommended to use biochemical markers in blood because an invasive medical practice is accompanied to obtain blood samples. Therefore urinary and salivary markers are more suitable because of obtaining the samples by non-invasive means. In this section we discuss especially important and useful stress markers in saliva and urine.

As mentioned above, the largest merit of using urinary and salivary markers is to obtain samples by non-invasive means, but the data often have larger variation than these of blood samples with depending on the condition of the samples, so it is necessary to select suitable collecting and sampling methods for the markers being measured. Especially in the case of saliva, it is necessary to select different collecting methods according to which salivary gland the target substances are mainly secreted from (submandibular, parotid, sublingual, etc.). A suitable collecting apparatus must be selected for the markers being measured [e.g. test tube for collecting saliva samples (Salivett^® Sarstedt AG & Co.), a short straw, etc].

As possible urinary markers for the stress response of the SAM system, vanillylmanderic acid (VMA) and homovanillic acid (HVA) are recommended, which are metabolites of catecholamines, individually norepinephrine and dopamine (Frankenhaeuser et al., 1986). Norepinephrine and dopamine in blood are a direct reflection of sympathetic nerve activity, so it has been suggested that these markers make it possible to detect changes in autonomic nerve balance induced by stress loads. However, it is not easy to identify the time point at which measuring the blood concentrations of these substances, moreover the concentrations depend on the clearance from the blood (Esler et al., 1984), so catecholamines in blood have been found to be unsuitable for use as stress markers, besides the sample collection needs invasive clinical practice. Therefore it is recommended to use the urinary concentrations of VMA and HVA as stress markers. Urinary VMA and HVA have long been used as clinical markers of neuroblastomas in infancy, and measurement methods using high performance liquid chromatography (HPLC) have been established. In human studies psychological stress load (having to perform calculations and operate a PC) is given for 4 hours, the level of VMA in urine is found to increase compared with that of unstressed condition. Also, in the case of physical stress load (ergometer exercise) for 4 hours, the urinary VMA and HVA levels are found to be higher for 4 hours after the load is given. Thus in the last few years, urinary VMA and HVA have attracted attention as markers for evaluating the effect of stress-reducing foods and medicines. More recently, they have also been used to evaluate electrical appliances for reducing stress. In one report, it was confirmed that stress-related increases in urinary HVA could be suppressed by controlling the airflow of cooling air conditioners, thus confirming the use of urinary HVA. These reports suggest that urinary VMA and HVA levels are thought to be promising stress markers for surgeons using robots, and it is expected that they will lead to the creation of robots that reduce stress.

Possible markers in saliva include α-amylase and chromogranin A as stress responses to the SAM system, and cortisol as a stress response to the HPA system.

Salivary α-amylase is mainly secreted by the parotid salivary glands, and the control of these secretions is known to be regulated by sympathetic nerves (Nater et al., 2006). When a stress load is given, this can be detected as an increase in salivary α-amylase activity, but this mechanism is thought to involve two pathways — one where the autonomic nervous system acts directly on the salivary glands, and another which is mediated by the secretion of norepinephrine from the adrenal medulla. This stress response generally occurs within 10 minutes. Salivary α-amylase activity is known to have circadian rhythm, increasing from the morning until midday and decreasing at night (Nater et al., 2007). Therefore it is no problem when evaluating acute phase stress, but when evaluating sub-acute or chronic stress for several hours or longer, the control sample must be obtained at the same time of another day. Salivary α-amylase activity is confirmed to change by both physical and psychological stress load. In the clinical study for the evaluation of electrical appliances, it has been reported that under 8-hour psychological stress loading conditions, an airbag-type automated massage chair (medical appliance) can inhibit the increase in salivary α-amylase activity. Salivary α-amylase activity can be measured by using the Caraway method, which is established as a method for the clinical examination of α-amylase in blood and urine that is a highly reliable measurement system. It has also been used to evaluate stress in surgeons using laparoscope robots.

Chromogranin A is an acid glycoprotein with a molecular weight of approximately 49,000 which is separated from adrenal medulla chromaffin cells. It is known to be widely distributed the endocrine and nervous systems, and is mostly found in the adrenal medulla and pituitary gland (Winkler & Fischer-Colibrie, 1992). A characteristic of this protein is that it coexists and is co-released with catecholamine which contributes to the stress response of the SAM system, so the blood level of chromogranin A reflects the sympathetic nerve activity. Chromogranin A is also present in the ducts of the submandibular glands, and is known to be released in the saliva as a result of stress loading (Saruta et al., 2005). Salivary chromogranin A is therefore used as a stress marker. Interestingly, it has been reported that specific changes only occur for a psychological stress load (Kanamaru et al., 2006), and in our studies we also observed changes for psychological stress loads but not for physical stress loads. The ELISA method was established for the measurement of salivary chromogranin A concentrations. Although it has not yet been demonstrated to be useful for stress evaluation electrical appliances, it is very interesting to see how salivary chromogranin A chages when using a robot.

Cortisol is released from the adrenal cortex when the pituitary is stimulated by ACTH as a stress response of the HPA system, and has been studied for a very long time as a stress marker (Levine, 1993). Since cortisol also affects the immune system and central nervous system, it is an important hormone that reflects not only stress levels but also physiological condition. Hitherto it has been used together with ACTH as a stress marker in blood. In recent years, a method has been developed for the measurement of salivary cortisol concentrations with ELISA, and this has come to be widely used as a stress marker. Salivary cortisol concentrations are of the order of a few percent compared to that in blood, but have been found to have a very strong correlation with stress. Cortisol level generally increases from 20 to 30 minutes after the application of stress load. The response time depends on the types of load, which is a slower response than the SAM system. Also, like salivary α-amylase, the salivary cortisol is known to have circadian rhythm, with a high concentration in the morning which decreases rapidly by midday, so it is essential to perform evaluations by comparing the results with a control sample. Salivary cortisol responds to both physical and psychological stress (Nozaki et al., 2009), and it has been shown that the abovementioned massage chair reduced cortisol concentrations caused by psychological stress loading. Furthermore, as introduced in this section, it is also used to evaluate the stress of surgeons when using a laparoscope robot.

3. Evaluation of stress with accelerated plethysmography

The stress response of the SAM system can be detected as a change in autonomic nerve functions by using a physiological marker. Changes in autonomic nerve function can be evaluated in various ways such as nerve impulses, electroencephalograms and electrocardiograms. Acceleration pulse waveforms are especially useful because they can be measured quickly and easily by accelerated plethysmography (Figure 3). The acceleration pulse waveform is a secondary differentiation of plethysmogram readings based on measurements of the optical absorbency of hemoglobin in peripheral blood vessels of a fingertip or other region. These waveforms have been generally used to evaluate arteriosclerosis. The a-a interval of the acceleration pulse waveform is strongly correlative to the R-R interval in an electrocardiogram in physiological aspect. The electrocardiogram R-R interval can be used to evaluate autonomic nerve functions by the coefficient of variation and by the frequency analysis of time-series data with maximum entropy method or fast Fourier transform method (Akselrod et al., 1985). Even in the a-a interval of the acceleration pulse waveform, when the coefficient of variation reflects the activity of parasympathetic nerves or by the analysis of time-series data, it is shown that the low-frequency component (LF: 0.02–0.15 Hz) mainly reflects the sympathetic nerve activity, while the high-frequency component (HF: 0.15–0.5 Hz) reflects the parasympathetic nerve activity, and it is known that the LF/HF ratio indicates the autonomic nerve functions and that LF/HF increases in stress states (when sympathetic nerves become predominant). When a physical stress load is given, it has been reported that in comparing before with after the stress load, the coefficient of variation of the a-a interval decreases and the LF/HF increases. These markers are often used to evaluate the stress-reducing effects of foods (Nukui et al., 2008). Recently, it has also been applied to evaluating the stress-reducing effects of electrical appliances.

It has also been found that LF/HF in the frequency analysis is related to fatigue as well as stress. The acceleration pulse waveform is useful for not only the evaluation of stress and fatigue when using electrical appliances, but also the detection of the worker’s fatigue level before the start of work, it is possible to detect the worker’s health condition before operating a robot.

Figure 3.

Evaluation of stress based on autonomic nervous system functions

4. Objective evaluation of psychological stress by analyzing biochemical markers and acceleration pulse waveforms

In this section we describe a method for objectively evaluating psychological stress in examinees by analyzing acceleration pulse waveforms and the examinee’s biochemical markers measured before and after performing a task. Saliva was used as the biochemical marker. For the acceleration pulse waveform data, we used the LF/HF ratio.

The duration of the task was set to 25 minutes. Immediately before and after the test, the examinee’s saliva was sampled and acceleration pulse waveform measurements were performed.

The saliva samples were obtained by having the examinee chew the cotton swab from a saliva collection test tube (Salivette^®, made by Sarstedt AG & Co.) for three minutes with the back teeth on one side of the mouth. If necessary, the saliva was stored by freezing after collection. Since the saliva constituents have circadian rhythm, in cases where multiple measurements were made on the same examinee, the saliva samples were obtained on the same day of the week and at the same time. The test subjects were also asked to chew the cotton swab with the same teeth on each occasion. The measurement of acceleration pulse waveforms was performed by attaching an infrared acceleration pulse waveform meter to the index finger and taking readings under resting conditions. The same finger was used for all measurements. The examinees were required to rest for approximately 30 minutes before starting the task. The cortisol in saliva samples was measured using a method such as ELISA. Also, the salivary α-amylase was measured using a method such as the Caraway method. The results of the salivary cortisol and α-amylase measurements are shown in Figure 4(a) and (b). Here, the subscripts “Before” and “After” indicate the results of measurements made immediately before and after performing the task. The numbers shown above the bar graphs are the measurement results or the average of multiple measurements. The results of measuring the acceleration pulse waveforms were used to calculate the LF/HF ratios, and the change before and after the task is shown in Figure 4(c) in the same way as in Figure 4(a) and (b).

Salivary cortisol, salivary α-amylase and the LF/HF ratio each have different reaction times to stress. Salivary α-amylase increases (activates) within about 10 minutes of applying a stress stimulus, whereas salivary cortisol increases (activates) roughly 20–30 minutes after applying a stress stimulus. The LF/HF ratio increases instantaneously when stress is given. By using these differences in reaction time, it is possible to estimate the stress before, during and after the task from the saliva constituents and acceleration pulse waveforms measured before and after the task lasting approximately 25 minutes as shown in Figure 5. In this Figure, the results of salivary cortisol measurements made immediately before the task (COR_Before) represent the stress levels 20–30 minutes before the start of the task, the results of salivary α-amylase measurements made immediately before the task (AMY_Before) represent the stress levels up to 10 minutes before the start of the task, the results of acceleration pulse measurements made immediately before the task (LF/HF_Before) represent the stress levels immediately before the start of the task, the results of salivary cortisol measurements made at the end of the task (COR_After) represent the stress levels in the first half of the task (20–30 minutes before the end of the task), the results of salivary α-amylase measurements made at the end of the task (AMY_After) represent the stress levels in the second half of the task (up to 10 minutes before the end of the task), and the results of acceleration pulse measurements made at the end of the task (LF/HF_After) represent the stress levels at the end of the task. By exploiting the time lags to the stress responses of each factor in this way, it is possible to estimate the stress variation over a wide period of time by making just a few measurements.

Figure 4.

Examples of measurement results

Figure 5.

Stress distribution obtained by exploiting the different stress response times of salivary constituents and acceleration pulse waveforms

Figure 6.

Format of stress variation diagram

Next, from the results of measuring the salivary constituents and acceleration pulse waveforms, we will discuss a method for plotting a stress variation diagram depicting the temporal variation in stress. Figure 6 shows the format of a stress variation diagram. The vertical axis shows the variation of stress, with larger numbers representing higher levels of stress and smaller numbers representing lower levels of stress. Since this diagram is more concerned with changes in stress levels, the absolute values are of no great significance. The horizontal axis represents time. The task starts at point D and ends at point F. Saliva and acceleration pulse waveform data are acquired at points D and F. The stress quantities for COR_Before, COR_After, AMY_Before, AMY_After, LF/HF_Before and LF/HF_After are plotted along axes A, B, C, E, D and F respectively, and are connected by lines. Here, t _T is the task duration (25 minutes), t _C is the salivary cortisol reaction time, and t _A is the salivary α-amylase reaction time. The acceleration pulse waveform is assumed to respond instantaneously. The stress variation diagram is drawn by following the four steps shown below.

Plot the salivary cortisol data

With regard to the salivary cortisol values measured before and after the task, COR_Before represents the stress state 20 to 30 minutes before the task (axis A), and COR_After represents the stress state 20 to 30 minutes before the end of the task (first half of the task) (axis B).

In this stress variation diagram, the COR_Before value is taken as a reference point (100%) as a basis for expressing subsequent stress values. First, the value of COR_Before is plotted at the 100% point 1 on axis A, and is denoted by γ ₀ = 100%. Using Equation (1), the value of COR_After is converted to a percentage taking that value of COR_Before as 100%. This converted value γ is plotted at point 2 on axis B. A line is then drawn between points 1 and 2.

$$\gamma =\frac{CO{R}_{After}}{CO{R}_{Before}}{\gamma }_0$$ E1

Example: From Figure 4(a), the salivary cortisol value is 0.075 µg/dl before the operation and 0.090 µg/dl after the operation. From Equation (1), this corresponds to γ = 120% (a 20% increase), so the stress variation diagram starts out as shown in Figure 7.

Figure 7.

Plotting the data from salivary cortisol measurements

Plot the salivary -amylase and LF/HF data obtained before surgery

Before the task, the examinees were assumed to be in a relaxed state with a small stress amplitude, so the point where the line drawn in step 1 intersects with axis C is assumed to correspond to AMY_Before and is called intersection point 3. Similarly, the point where the line drawn in step 1 intersects with axis D is assumed to correspond to LF/HF_Before and is called intersection point 4. In this way, intersection points 3 and 4 are points that are automatically determined from the salivary cortisol data of step 1 and the positions of axes C and D. Therefore, the value α ₀ at intersection point 3 is given by Equation (2), and the value β ₀ at intersection point 4 is given by Equation (3).

$${\alpha }_0={\gamma }_{Before}+\frac{t_C\text{-}{t}_A}{t_T}\left(\gamma \text{-}{\gamma }_0\right)$$ E2

$${\beta }_0={\gamma }_{Before}+\frac{t_C}{t_T}\left(\gamma \text{-}{\gamma }_0\right)$$ E3

Here, the values of α ₀ and β ₀ are liable to be affected by the stress state before the task, so it is important that a relaxed state is maintained before the task.

Figure 8.

Plotting the α-amylase and LF/HF data before surgery

Example: When the pre-surgery salivary α-amylase data AMY_Before and the pre-surgery LF/HF data LF/HF_Before are plotted, the stress variation diagram appears as shown in Figure 8. Here, γ ₀ = 100%, γ = 120%, t _T (task duration) = 25 minutes, t _C (salivary cortisol reaction time) = 20 minutes, t _A (salivary α-amylase reaction time) = 5 minutes. Based on these values, the value α _Before at intersection point 3 from Equation (2) is 112.0%, and the value β _Before at intersection point 4 from Equation (3) is 116.0%.

Plot the salivary -amylase data obtained after surgery

The salivary α-amylase data AMY_After obtained after surgery represents the stress at point E within 10 minutes before the end of the task. This stress represents a stress quantity relative to the salivary α-amylase data AMY_Before obtained before surgery, so we can use Equation (4) to convert this into an increase or decrease α with respect to the base point 1. This value α is entered on axis E as intersection point 5. A line is drawn between intersection points 2 and 5.

$$\alpha =\frac{AM{Y}_{After}}{AM{Y}_{Before}}{\alpha }_0$$ E4

Example: From Figure 4(b), the pre- and post-surgery salivary α-amylase data AMY_Before and AMY_After have values of 45 and 71 KU/L respectively, and the value of α ₀ is 112.0%. Thus from Equation (4), the value of α is 177.0%, and the resulting stress variation diagram is as shown in Figure 9.

Figure 9.

Plotting the salivary α-amylase after surgery

Plot the LF/HF data after surgery

The post-surgery LF/HF data LF/HF_After represents the stress level at the end of the task (point F).

This stress level represents an amount of stress relative to the pre-surgery LF/HF data LF/HF_Before, so Equation (5) is used to transform LF/HF_After into an increase or decrease β relative to the base point 1. This value β is entered on axis F as intersection point 6. A line is drawn between intersection points 5 and 6.

$$\beta =\frac{{\text{LF/HF}}_{\text{After}}}{{\text{LF/HF}}_{\text{Before}}}{\beta }_0$$ E5

Example: From Figure 4(c), the pre-surgery LF/HF data LF/HF_Before has a value of 10, the post-surgery acceleration pulse waveform data LF/HF_After has a value of 8, and the value of β ₀ is 116.0%. Thus according to Equation (5), β is equal to 92.8%, and the stress variation diagram appears as shown in Figure 10.

Figure 10.

Plotting the LF/HF data after surgery

By following the above procedure of steps 1 through 4, it is possible to draw a stress variation diagram.

The axes A, C and D in the stress variation diagram represent the stress values before the start of the task, axes D, B and E represent the intermediate stress levels after the start of the task, and axes E and F represent the stress levels in the second half of the task.

In the example shown in Figure 10, there is a gentle increase in stress before the start of the task, and a clear increase in stress from the beginning through to the middle of the task, but this stress is eliminated in the second half of the task.

5. A practical example of psychological stress evaluation

In this section, to illustrate how the stress variation diagrams described in section 4 can be used in practice, we show how this technique can be used in the evaluation of a laparoscopic robot. In this example, surgeons performed in-vitro laparoscopic cholecystectomy simulations using pig livers (which have an anatomically similar structure to that of human organs). These operations were performed with a laparoscope operated by a laparoscope robot, and with a laparoscope operated by a human assistant. By analyzing the surgeons’ LF/HF ratio and salivary cortisol and α-amylase levels before and after each surgery, we conducted a multilateral and objective evaluation of their biological stress responses.

5.1. Laparoscope robot

For the laparoscope robot, we used the automatic laparoscope positioning system proposed by Nishikawa et al. (Nishikawa et al., 2006), which includes the ability to plan the workspace before the operation begins. This laparoscope robot is a fully autonomous system that uses a robot to hold and automatically position the laparoscope instead of a human camera assistant. The position of the laparoscope and the image zoom factor to be used during surgery are set up just before the surgery by preoperative planning whereby the surgeon selects several working area at the operation site, while at the same time determining the best image zoom factor (i.e., the distance from the working area to the laparoscope tip) for working at this position, and stores this information on a PC. Once the operation has started, the robot tracks the surgical instrument in three dimensions so that the tip of the surgical instrument remains in the center of the laparoscope image. When the tip of the surgical instrument has been positioned at the working area determined during preoperative planning, the zoom factor of the laparoscope image is automatically adjusted according to the preoperative planning. Figure 11 shows the hardware configuration of the laparoscope robot, and Figure 12 shows the control flow. The laparoscope robot consists of a manipulator, an optical three-dimensional position-measuring device (Polaris Accedo^®, made by NDI Corporation), a control PC (Linux-based), a scan converter and a television monitor. The manipulator has a parallel link mechanism that uses three motors to perform positioning with three degrees of freedom. When the field of view moves to the left or right and up or down, the longitudinal position of the laparoscope camera can be adjusted to enlarge or reduce the field of view.

Figure 11.

Hardware configuration of laparoscope robot

Figure 12.

Control flow of laparoscope robot

5.2. In-vitro tests

Surgeons were asked to perform in-vitro laparoscopic cholecystectomy simulations on pig livers, using either a human camera assistant or a laparoscope robot to operate the laparoscope. Before and after each operation, the surgeon’s saliva was sampled and the acceleration pulse waveform was measured. The salivary cortisol and salivary α-amylase constituents of the saliva were measured. The salivary cortisol was measured by the ELISA method using reagents made by Salimetrics, and the salivary α-amylase was measured by the Caraway method using reagents made by Wako Pure Chemical Industries Ltd. The acceleration pulse waveform was measured using an Artett C acceleration pulse waveform meter made by U - Medica Inc.

For the in-vitro laparoscopic cholecystectomy simulations performed using pig livers, a fresh pig liver was placed inside a test box to represent the abdomen, and the surgeon performed a mock cholecystectomy (Figure 13). This operation is performed by the following procedure: (1) move the field of view to Calot’s triangle, (2) expose and cut the cystic duct, (3) detach the gallbladder from the liver (Figure 14).

The examinees were two right-handed clinicians with extensive experience in laparoscopic cholecystectomy simulations (examinees A and B). The examinees had no previous experience in the use of laparoscope robots. In total, they performed the operation 14 times over a period of four days. The surgeon and laparoscope operator in each operation were as follows:

Figure 13.

Set-up of tests conducted with a laparoscope robot

Figure 14.

Laparoscope view. (a) Moving the field of view to Calot’s triangle and exposing/cutting the cystic duct. (b) Detaching the gallbladder from the liver

Day 1

1. (1) Surgeon: examinee A, laparoscope operator: laparoscope robot

2. (2) Surgeon: examinee A, laparoscope operator: examinee B

3. (3) Surgeon: examinee B, laparoscope operator: laparoscope robot

4. (4) Surgeon: examinee B, laparoscope operator: examinee A

In operations (1) and (3), we sampled the surgeon’s saliva before and after the operation, and in operations (2) and (4) we sampled the saliva of both the surgeon and camera assistant. Acceleration pulse waveform measurements were not performed in operation (1).

Day 2

1. (5) Surgeon: examinee A, laparoscope operator: examinee B

2. (6) Surgeon: examinee B, laparoscope operator: laparoscope robot

3. (7) Surgeon: examinee B, laparoscope operator: examinee A

4. (8) Surgeon: examinee A, laparoscope operator: laparoscope robot

In each operation, saliva samples and acceleration pulse waveform measurements were taken from the surgeon.

Day 3

1. (9) Surgeon: examinee B, laparoscope operator: examinee A

2. (10) Surgeon: examinee A, laparoscope operator: laparoscope robot

3. (11) Surgeon: examinee A, laparoscope operator: examinee B

4. (12) Surgeon: examinee B, laparoscope operator: laparoscope robot

In operations (9) and (11) we obtained saliva samples and acceleration pulse waveform measurements from the camera assistant, and in operations (10) and (9) we obtained saliva samples and acceleration pulse waveform measurements from the surgeon.

Day 4

1. (13) Surgeon: examinee B, laparoscope operator: examinee A

2. (14) Surgeon: examinee A, laparoscope operator: examinee B

3. In the operations performed on day 4, we obtained saliva samples and acceleration pulse waveform measurements from the surgeon.

The above test schedule was planned to take into consideration the circadian rhythm in the substances used to evaluate psychological stress. By scheduling operations (1), (5) and (9) at the same time of day, it was possible to acquire data at the same time of day for examinee A performing the operation with a laparoscope robot and with a human camera assistant, so when making a comparative study of the data from each operation, there was no need to take into consideration the effects of circadian rhythm in the substances used to evaluate psychological stress. Similarly, operations (3), (7) and (11) were performed at the same time of day by examinee B, operations (2), (6), (10) and (13) were performed at the same time of day by both examinees, and operations (4), (8), (12) and (14) were performed at the same time of day by both examinees so that data could be collected in the same way.

The results of salivary cortisol measurements on examinees A and B before and after surgery are shown in Figure 15 and 18, and the results of salivary α-amylase measurements are shown in Figure 16 and 19. Since the results of measurements of salivary constituents were obtained by taking circadian rhythm of stress evaluation substances into consideration, the data was all processed together. Figure 17 and 20 show the results of LF/HF measurements from examinees A and B before and after the operations. Note that Figure 15 through 20 only show the data for the surgeon. The duration of the operations performed by examinees A and B are shown in Table 1 as supplementary material.

Figure 15.

Salivary cortisol levels (Examinee A)

Figure 16.

Salivary α-amylase activity levels (Examinee A)

Figure 17.

LF/HF ratios (Examinee A)

Figure 18.

Salivary cortisol levels (Examinee B)

Figure 19.

Salivary α-amylase activity levels (Examinee B)

Figure 20.

LF/HF ratios (Examinee B)

		With Surgical	With Camera
		Assistant System	Assistant
Examinee A	Ave.	13	47
	S.D.	20	5
	Max.	37	13
	Min.	57	5
Examinee B	Ave.	48	5
	S.D.	35	20
	Max.	53	25
	Min.	6	0

Table 1.

Operating times

We plotted stress variation diagrams based on the results of measuring saliva constituents and acceleration pulse waveforms shown in Figure 15 through 20 (Figure 21 and 22). Here, the task duration t _T was set to 25 minutes, the salivary cortisol reaction time was set to 20

Figure 21.

Stress variation diagram for examinee A

Figure 22.

Stress variation diagram for examinee B

minutes, and the salivary α-amylase reaction time t _A was set to 5 minutes. The task duration t _T can be taken as the average duration for each task as shown in Table 1. Figure 21 shows the stress variation diagrams for examinee A. This Figure shows the stress variation measured when using the laparoscope robot and when using a human camera assistant to operate the laparoscope. Figure 22 shows the corresponding results for examinee B. The line graphs shown in these Figures allow the comparative evaluation to be made between surgery with a laparoscope robot and surgery with a human camera assistant.

First of all we will consider the results for examinee A (Figure 21). Examinee A was not stressed before the operation or during the middle stages of the operation, but became stressed at the end of the operation. Examinee A was also slightly more stressed when performing the operation with a camera assistant than when performing the operation with a robot. The same can also be said of the raw data shown in Figure 15 through 17. From the salivary cortisol and salivary α-amylase results for examinee A (Figure 15 and 16), examinee A had no pronounced stress reaction in either operation. Next, from the LF/HF results (Figure 17), examinee A had a greater predominance of sympathetic nerve activity in the second half of the operation than in the first half, and tended to be slightly more stressed.

Next we will consider the results for examinee B. As Figure 22 shows, examinee B felt stressed before the operation and during the first half of the operation when using the laparoscope robot, but this stress reduced during the second half of the operation. On the other hand, when performing the operation with a human camera assistant, examinee B was not stressed before the operation, but the stress level increased as the operation began and there was no reduction of stress during the operation. Looking at the data of Figure 18 through 20 separately, in the salivary cortisol results for examinee B (Figure 18), a stress reaction occurred before the operation when using the laparoscope robot. Also, from the salivary α-amylase results (Figure 19), there was a slight stress reaction during all the operations, and the largest stress reactions were observed in operations where the laparoscope was operated by a camera assistant. When using the laparoscope robot, according to the LF/HF results (Figure 20), the sympathetic nerves are predominant around the start of the operation and suppressed at the end of the operation. On the other hand, when performing surgery using a camera assistant, the sympathetic nerves are more predominant at the end of the surgery. In other words, examinee B tended to be more stressed (tense or agitated) at the end of the operation than at the start of the operation when using a camera assistant, but tended to be more stressed at the start of the operation when using a laparoscope robot.

From the operation times shown in supplementary Table 1, the style of operation was found to cause no difference in operation times, and we found it impossible to evaluate stress in terms of how long the operation took to perform.

Thus by analyzing saliva constituents and acceleration pulse waveforms, we were able to objectively evaluate the stress experienced by surgeons when performing laparoscopic surgery with a laparoscope operated by a human camera assistant and with a laparoscope operated by a laparoscope robot.

6. Conclusion

We have described a method for objectively evaluating the psychological stress experienced by people performing a task with a robot for about 25 minutes by analyzing their saliva constituents and acceleration pulse waveforms before and after the task. In particular, in this study we investigated an example where highly skilled examinees (surgeons) engaged in high-level interaction with a functionally enhanced robot (laparoscope robot) to perform a particular task (laparoscopic surgery) in a particular environment (operating theatre). A laparoscope robot is a good example of where humans and robots can interact successfully. Methods for objectively evaluating the psychological stress of humans due to interactions with robots will become increasingly important as robots become more commonplace in society. Further research will be needed to investigate stress evaluation methods that are simpler, less invasive and cheaper to implement. In the future, we plan to investigate a method for using the human herpes virus (HHV6) to evaluate long-term and chronic fatigue in surgeons, and to study an integrated stress evaluation method that combines subjective and objective stress evaluation methods.

Acknowledgments

This research was supported in part by “Special Coordination Funds for Promoting Science and Technology: Yuragi Project” of the Ministry of Education, Culture, Sports, Science and Technology, Japan, Grant-in-Aid for Scientific Research (A) (No. 19206047) of the Japan Society for the Promotion of Science.