InTech uses cookies to offer you the best online experience. By continuing to use our site, you agree to our Privacy Policy.

Computer and Information Science » Human-Computer Interaction » "Augmented Reality", book edited by Soha Maad, ISBN 978-953-7619-69-5, Published: January 1, 2010 under CC BY-NC-SA 3.0 license. © The Author(s).

Chapter 8

Human Factor Guideline for Applying AR-based Manuals in Industry

By Miwa Nakanishi
DOI: 10.5772/7131

Article top

Overview

Plugs
Figure 1. Plugs
Operational panel
Figure 2. Operational panel
Manual
Figure 3. Manual
Overlay of AR-based manual on panel
Figure 4. Overlay of AR-based manual on panel
Experimental environment
Figure 5. Experimental environment
Process of inserting a plug
Figure 6. Process of inserting a plug
Classification of errors during the experiment
Table 1. Classification of errors during the experiment
Average time for the task
Figure 7. Average time for the task
Average frequency of errors
Figure 8. Average frequency of errors
STD
Figure 9. STD
RSD
Figure 10. RSD
Position of a subject
Figure 11. Position of a subject
Time per task
Table 2. Time per task
Position of a subject
Figure 12. Position of a subject
Average time per task
Figure 13. Average time per task
Scores for the five questions in each condition
Figure 14. Scores for the five questions in each condition
Error rate per task
Figure 15. Error rate per task
Average time per task
Figure 16. Average time per task
Experimental procedure
Figure 17. Experimental procedure
Shift of average time per task as terms progress
Figure 18. Shift of average time per task as terms progress
Shift of CFF as terms progress
Figure 19. Shift of CFF as terms progress
Shift of W.R. as terms progress
Figure 20. Shift of W.R. as terms progress
Shift of self-evaluation score as terms progress
Figure 21. Shift of self-evaluation score as terms progress
Positions of a subject and the LED boards: view from the top (left) and view from the side (right)
Figure 22. Positions of a subject and the LED boards: view from the top (left) and view from the side (right)
Detection rate of each LED board when subjects used the paper-based manual
Figure 23. Detection rate of each LED board when subjects used the paper-based manual
Detection rate of each LED board when subjects used STDs
Figure 24. Detection rate of each LED board when subjects used STDs
Detection rate of each LED board when subjects used RSDs
Figure 25. Detection rate of each LED board when subjects used RSDs
A pattern of the operational panel
Figure 26. A pattern of the operational panel
A block on the operational panel
Figure 27. A block on the operational panel
A pattern of the AR-based manual
Figure 28. A pattern of the AR-based manual
An overlay of the operational panel and AR manual
Figure 29. An overlay of the operational panel and AR manual
Operational panel (m = 4)
Figure 30. Operational panel (m = 4)
Operational panel (m = 6)
Figure 31. Operational panel (m = 6)
Operational panel (m = 8)
Figure 32. Operational panel (m = 8)
Operational panel (m = 10)
Figure 33. Operational panel (m = 10)
Operational panel (m = 12)
Figure 34. Operational panel (m = 12)
Whole indication (m = 8)
Figure 35. Whole indication (m = 8)
One-line indication (m = 8)
Figure 36. One-line indication (m = 8)
Individual indication (m = 8)
Figure 37. Individual indication (m = 8)
Position of the PC monitor and a subject
Figure 38. Position of the PC monitor and a subject
Error rate per operation unit (m = 4)
Figure 39. Error rate per operation unit (m = 4)
Error rate per operation unit (m = 6)
Figure 40. Error rate per operation unit (m = 6)
Error rate per operation unit (m = 8)
Figure 41. Error rate per operation unit (m = 8)
Error rate per operation unit (m = 10)
Figure 42. Error rate per operation unit (m = 10)
Error rate per operation unit (m = 12)
Figure 43. Error rate per operation unit (m = 12)
Operation time(m = 4)
Figure 44. Operation time(m = 4)
Operation time(m = 6)
Figure 45. Operation time(m = 6)
Operation time(m = 8)
Figure 46. Operation time(m = 8)
Operation time (m = 10)
Figure 47. Operation time (m = 10)
Operation time (m = 12)
Figure 48. Operation time (m = 12)
A pattern of the operational panel
Figure 49. A pattern of the operational panel
A pattern of the AR manual
Figure 50. A pattern of the AR manual
An overlay of the operational panel and the AR-based manual
Figure 51. An overlay of the operational panel and the AR-based manual
Operational panel (m = 4)
Figure 52. Operational panel (m = 4)
Operational panel (m = 8)
Figure 53. Operational panel (m = 8)
Operational panel (m = 12)
Figure 54. Operational panel (m = 12)
indication (m = 8)
Figure 55. indication (m = 8)
indication (m = 8)
Figure 56. indication (m = 8)
indication (m = 8)
Figure 57. indication (m = 8)
indication (m = 8)
Figure 58. indication (m = 8)
Error rate per operation unit (m = 4)
Figure 59. Error rate per operation unit (m = 4)
Error rate per operation unit (m = 8)
Figure 60. Error rate per operation unit (m = 8)
Error rate per operation unit (m = 12)
Figure 61. Error rate per operation unit (m = 12)
Rate of misoperation (m = 4)
Figure 62. Rate of misoperation (m = 4)
Rate of misoperation (m = 8)
Figure 63. Rate of misoperation (m = 8)
Rate of misoperation (m = 12)
Figure 64. Rate of misoperation (m = 12)
An example of effective complexity (m = 4)
Figure 65. An example of effective complexity (m = 4)
An example of effective complexity (m = 6)
Figure 66. An example of effective complexity (m = 6)
An example of effective complexity (m = 8)
Figure 67. An example of effective complexity (m = 8)
An example of effective complexity (m = 10)
Figure 68. An example of effective complexity (m = 10)
An example of effective complexity (m = 12)
Figure 69. An example of effective complexity (m = 12)
An example of effective complexity (m = 4)
Figure 70. An example of effective complexity (m = 4)
An example of effective complexity (m = 8)
Figure 71. An example of effective complexity (m = 8)
An example of effective complexity (m = 12)
Figure 72. An example of effective complexity (m = 12)

Human Factor Guideline for ApplyingAR-based Manuals in Industry

Miwa Nakanishi1

1. Introduction

Augmented reality (AR)-based manuals, which replace the paper-based manuals used in the past, is a promising application of AR in industrial settings such as manufacturing, maintenance and so on. In our previous research, we conducted experiments to compare the case where workers referred to a traditional paper-based manual with that where they referred to an AR-based manual in the tasks of assembling, wiring and inspection (Nakanishi & Okada, 2006, Nakanishi et al., 2007). We confirmed that, in the latter case, they completed each type of task in around 15% less time and made fewer errors than in the former case. Further, we found that psychological stress in referring to the manual was mitigated in the latter case. This is possibly because the AR-based manual enabled workers to view task-related information superimposed on real objects and made it easy to cognitively link the objects and information.

For these reasons, AR-based manuals are expected to serve as effective tools for increasing efficiency, preventing human errors and promoting comfort in industrial settings. However, before practical use of such manuals, it is necessary to clarify human factor requirements concerning workers, the environment and the information presented.

Thus, we have been examining basic human factor requirements through a series of experiments, particularly under the assumption that workers see an AR-based manual using monocular see-through head-mounted displays (HMDs). In this chapter, we review how the performance of workers using AR-based manuals is changed by differences in the workers themselves, the work environment and the information presented by HMDs, based on behavioural, physiological and psychological data. Furthermore, we summarise a human factor guideline for applying AR-based manuals. The guideline can give suggestions for smooth and effective introduction of AR technology to industries.

2. Data on improvement of workers’ performance using AR-based manuals

First, we introduce an experimental result that shows that workers’ performance is improved by using an AR-based manual instead of a traditional paper-based manual.

2.1. Method

The experimental task was to insert plugs into a panel according to information in the manual. Five colours of cables (red, green, blue, white and yellow) with plugs on both ends were prepared (Fig. 1). The panel had 37 holes on the left side and 43 on the right (Fig. 2). Subjects inserted one end of each cable into a left-side hole and the other into a right-side hole following the manual shown in Fig. 3. In this manual, two numbers and a character appear inside each circle corresponding to each hole on the left side. The upper-left-hand number indicate the order in which one end should be inserted, the upper-right-hand character indicate the plug colour, and the lower number indicate the hole on the right side into which the other end should be inserted. Subjects inserted both ends of 37 plugs into the panel in a task. Then they performed the task lying on their back (Fig. 5), as seen in some actual situations such as maintenance or inspection.

In this experiment, basic and comparative conditions were prepared. In the basic condition, subjects performed the task using the paper-based manual. In the comparative condition, they wore HMDs (NOMAD, made by Microvision, Inc.) and used an AR-based manual, so that they could see the manual superimposed on the panel, as shown in Fig. 4. Subjects (six students with good vision) performed the task using different patterns in the manual in each condition.

media/image1.png

Figure 1.

Plugs

media/image2.png

Figure 2.

Operational panel

media/image3.png

Figure 3.

Manual

media/image4.png

Figure 4.

Overlay of AR-based manual on panel

media/image5.png

Figure 5.

Experimental environment

2.2. Results

Figure 6 illustrates the process flow for inserting a plug. Based on this process, errors that occurred during the task were classified into four types, as shown in Table 1. Below, the time for a task and the frequency of each error are compared for the two conditions.

media/image6.png

Figure 6.

Process of inserting a plug

media/image7.png

Table 1.

Classification of errors during the experiment

(I)Time for a task

Fig. 7 shows the average time for the task in each condition. It is clear that the time was about 15% less in the comparative condition than in the basic condition. One reason is that subjects’ eye movement was reduced because they could see both the manual and the panel at the same time when they used an AR-based manual, whereas they had to see them one after the other when they used a paper-based manual. Another reason is that they could plug into the panel more easily in the comparative condition, which allowed hands-free operation, than in the basic condition, which required them to hold the paper-based manual.

media/image8.png

Figure 7.

Average time for the task

(II) Frequency of errors

Fig. 8 shows the frequencies of four types of errors in each condition. First, the error in remembering the procedures was less frequent in the comparative condition than in the basic condition. As described above, subjects could perform the task more smoothly when they used an AR-based manual. Further, it prevented them from forgetting what number hole they should next plug into. Second, the frequency of errors in recognising the positions was smaller in the comparative condition than in the basic condition. When subjects used HMDs, they could see the manual superimposed on the panel so that they could easily check both of them. Third, the error in recognising the objects seldom occurred in either condition. Fourth, the frequency of error in action in the comparative condition was about a quarter of that in the basic condition. This is probably because an AR-based manual made it less difficult to compare the manual and the panel, reducing such careless errors.

media/image9.png

Figure 8.

Average frequency of errors

In addition, we have reported that an AR-based manual can support both skilled and unskilled workers (Nakanishi & Okada, 2006). An AR manual is especially effective for rule-based tasks of the three types described in the SRK task model (Nakanishi, et al., 2009a), and it makes it easy to foresee future situations in dynamic control tasks (Nakanishi & Okada, 2003, Nakanishi & Okada, 2004; Nakanishi & Okada, 2005).

3. Examination of human factor requirements for applying AR-based manuals

As shown in the previous section, AR-based manuals are promising as a next-generation tool for both preventing errors and enhancing task efficiency. Based on this prospect, it is necessary and important to clarify the basic human factor requirements before they are put into practical use.

In this section, we describe the conditions required to effectively enhance the performance of the workers who use an AR-based manual. In particular, we discuss what should be considered when they work with HMDs, and also how much information should be provided by HMDs depending on the real view, through experiments (Nakanishi et al., 2007, Nakanishi et al., 2008a).

3.1. Points to be examined

The main characteristics of using an AR-based manual are wearing HMDs and superimposing a digital image on the real view. Thus, we focus on the following eight significant points.

(I) Effect of wearing HMDs

When an AR-based manual is used in assembling or inspecting tasks, it is of primary importance that workers can see real objects clearly enough. Accordingly, the use of a monocular transparent HMD is recommended. Thus, we examine the following six points that should be considered when workers wear HMDs.

  1. Difficulty in preparing to use AR-based manuals: Does it take more time to prepare to use AR-based manuals with HMDs than paper-based manuals?

  2. Effect of eyesight correction: Although it is currently difficult for the workers wearing glasses to also wear HMDs, is it possible to apply AR-based manuals with HMDs to the workers wearing contact lenses?

  3. Effect of eye dominance: Is it better for the workers to wear monocular HMDs on their dominant or non-dominant eye?

  4. Effect of surrounding illumination: Is the workers’ view affected by surrounding illumination when they use HMDs?

  5. Workload: Is the workers’ workload heavier when they use AR-based manuals with HMDs than when they use traditional paper-based manuals?

  6. Attention to surroundings: Is it more difficult to recognise changes in surroundings when they use AR-based manuals with HMDs than when they use paper-based manuals?

(II) Effect of superimposing information on the real view

An AR-based manual allows workers to see an overlay of task-related information and real objects to be manipulated. This characteristic shows a great potential to reduce human errors and enhance task efficiency in actual work situations. However, if too much information is given by an AR-based manual, it may interfere with the real view. Thus, we further examine the following two points for clues to the proper design of AR-based manuals.

  1. Effect of information presented by HMDs (static background): How does the workers’ performance change depending on the balance between the complexities of digital information and the static real background?

  2. Effect of information presented by HMDs (dynamic background): How does the workers’ performance change depending on the balance between the complexities of digital information and the dynamic real background?

3.2. Method

To answer the above questions, we considered an experimental approach. To examine points (1) to (6), which concern the use of an HMD, we conducted experiments in which subjects performed the same task described in section 2. The detailed settings of the experiment were different for each examined point (described in the following sections).

Further, to examine points (7) and (8), which concern superimposition of digital images on the real view, we prepared another experimental task in which subjects experienced various grades of complexity of the AR-based manual and the real view. Its details are also given in later sections.

3.3. Human interface

Current technology offers two easily available human interfaces for AR-based manuals: a see-through display (STD) and a retinal-scanning display (RSD). An STD (Fig. 9) is an HMD that lets a user see images projected from a PC onto a half-mirror in front of his/her eye, and an RSD (Fig. 10) is an HMD that lets a user see images by shining a low-power laser beam directly into his/her retina and scanning it at high speed.

In the experiments for examining points (1) to (6), we asked subjects to perform the task using the AR-based manual with STDs (Dataglass2A, made by SHIMADZU) and RSDs (NOMAD, made by Microvision, Inc). Further, if it was necessary for comparison, we asked them to perform the task while holding the paper-based manual, an A4 paper showing the same image as Fig. 3, in their hand. When they used the AR-based manual, they could see the superimposed image of the manual and the panel, as shown in Fig. 4. The image size was set to the same in every case. However the transparency was different between STDs (approximately 15%) and RSDs (almost 100%). Moreover, in the experiment for examining points (7) and (8), subjects used RSDs whose transparency was higher.

media/image10.png

Figure 9.

STD

media/image11.png

Figure 10.

RSD

In all the experiments, they practiced well in advance of data collection.

In the following sections, the detailed settings of each experiment and their results are described.

3.4. Setting details and results

3.4.1. Difficulty in preparing to use AR-based manuals

(I) Setting details

Subjects were nine students whose right eyes were dominant. Each of them sat as shown in Fig. 11 with an STD or RSD in his/her hand. At a signal, they started putting on the STD or RSD on their dominant eye and then adjusted their own position so that they could see the superimposed image of the panel and the manual presented by the STD or RSD. They were asked to say “Wore” when they finished putting on the STDs or RSDs and “Saw” when they could see the superimposed image; the time they said these words were recorded. Each of them repeated the procedure three times.

media/image12.png

Figure 11.

Position of a subject

(II) Result

The average time until they finished putting on the STDs was 19.7 (s), and that until they finished putting on the RSDs was 19.2 (s). The average time until they could see the perfectly superimposed image was 5.4 (s) when they used STDs and 6.0 (s) when they used RSDs.

From these results, we can say that workers easily prepare to use either STDs or RSDs. Moreover it is expected that in future these HMDs will become simpler, just like glasses, so workers can more easily wear them.

3.4.2. Effect of eyesight correction

(I) Setting details

Subjects were nine students with normal eyesight of 1.0 or better (normal group) and nine students with eyesight of 1.0 or better corrected by contact lens (corrected group). Their eyesight was tested with the Landolt C test. Each subject’s dominant eye was the right.

Their task was to insert 25 plugs into holes on the panel following the manual as quickly as possible. Two experimental conditions were prepared: a) wearing an STD on the non-dominant eye and b) wearing an RSD on the non-dominant eye. They performed the task once in each condition. The position of a subject and the panel was the same as in Fig. 11.

(II) Result

The error rate per task for both groups was below 1.0 in both conditions. The average time per task for each group is given in Table 2.

media/image13.png

Table 2.

Time per task

These results show no significant difference between the groups in either condition. Thus, we can say that workers whose eyes are corrected by contact lenses as well as workers with good eyesight can use AR-based manuals with either STDs or RSDs.

3.4.3. Effect of eye dominance

(I) Setting details

Subjects were 14 students whose right eyes were dominant and 6 students whose left eye was dominant. Their dominant eye was determined by the Rosenbach test.

Their task was to insert 20 plugs into holes on the panel according to the manual as quickly as possible. Four experimental conditions were prepared: a) wearing an STD on the dominant eye, b) wearing an STD on the non-dominant eye, c) wearing an RSD on the dominant eye and d) wearing an RSD on the non-dominant eye. They performed the task once in each condition and after each task five questions were answered by giving scores of −5 to +5. The position of a subject and the panel is shown in Fig. 12.

(II) Result

The error rate per task was below 1.0 in all conditions. The average time per task for each condition is given in Fig. 13.

This chart shows that if they used either STDs or RSDs, the time tended to be shorter when they wore it on their non-dominant eye than when they wore it on their dominant eye. The scores for the five questions are given in Fig. 14. The score for the question “Easy to see the panel or not?” means that they felt it easier to see the panel when they wore STDs or RSDs on their non-dominant eye, while the score for the question “Easy to see the manual or not?” means that they felt it easier to see the manual when they wore STDs or RSDs on their dominant eye. This result is natural, because they can more clearly see an object with their dominant eye. However, focusing on the questions asking about factors that could

media/image14.png

Figure 12.

Position of a subject

media/image15.png

Figure 13.

Average time per task

media/image16.jpeg

Figure 14.

Scores for the five questions in each condition

potentially influence their overall performance, such as “Feel much eye-fatigue or not?” or “Easy to insert plugs or not?”, the scores were found to be more favourable when they wore STDs or RSDs on their non-dominant eye. This seems to be because it was primarily important for them to clearly see the real panel and plugs in this task.

From the above examination, it is recommended that if either STDs or RSDs are used for AR-based manuals in tasks such as assembly or inspection, they should be worn on the non-dominant eye. Accordingly, design of the manuals shown by STDs or RSDs should be simple enough for the workers to easily read them with their non-dominant eye.

3.4.4. Effect of surrounding illumination

(I) Setting details

Subjects were 10 students whose right eyes were dominant.

Their task was to insert 25 plugs into holes on the panel according to the manual as quickly as possible. Two experimental conditions were prepared: a) wearing an STD on the non-dominant eye and b) wearing an RSD on the non-dominant eye. The position of a subject and the panel is the same as in Fig. 12.

Japanese Industrial Standard (JIS) Z 9110 specifies that 300 (lx) to 3000 (lx) is recommended as the standard of illumination in assembling or inspecting tasks (Japan Standards Association, 1979). However it does not consider the use of STDs or RSDs, and workers sometimes have to work under low or high illumination. So we set the experimental environment as follows: six grades of indoor illumination [100 (lx), 400 (lx), 800 (lx), 1200 (lx), 1600 (lx)] and outdoor illumination (under the direct rays of the morning sun). Each subject performed the task once each using an STD or RSD under each of seven lighting conditions.

(II) Result

The outdoor illumination level was 68,333 (lx) on average. The error rate per task is given in Fig. 15. The error rate in cases of indoor illumination was below 1.0 when subjects used either STDs or RSDs. However, it was comparatively high when they used STDs under outdoor illumination. The average time per task is given in Fig. 16. We see that the time was more than 150 (s) longer only when they used STDs under outdoor illumination compared with the other cases. This is possibly because an STD sends images to a user’s retina via a half-mirror and it is easily affected by surrounding illumination, whereas an RSD sends images directly to the retina and is scarcely affected by surrounding illumination.

media/image17.png

Figure 15.

Error rate per task

media/image18.png

Figure 16.

Average time per task

From the above examination, we can say that there is no evident problem in using either STDs or RSDs under indoor illumination if it is within the range of the standard level. However, when workers use STDs under extremely high illumination, it may be difficult to read the information they present. In such cases, the problem can be solved by using RSDs instead of STDs.

3.4.5. Workload

(I) Setting details

Subjects were 10 students whose right eyes were dominant.

Their task was to insert 30 plugs into holes on the panel according to the manual as quickly as possible, and three repetitions of this task were defined as a term. In this experiment, each of them was given three terms, including breaks of two minutes. Before the first term and after each term, an electrocardiogram (ECG) was recorded using EP-202 (made by Parama-Tech), and the critical flicker frequency (CFF) was recorded using 501BTKK (made by Takei Kiki Kogyo). They also answered 22 questions concerning workload such as “heavy head”, “stiff shoulders” by giving scores of 1 (light) to 5 (heavy) (Institute for Science of Labour, 1970). Fig. 17 shows the experimental procedure. The experimental conditions were as follows: a) using the paper-based manual, b) wearing an STD on the non-dominant eye and c) wearing an RSD on the non-dominant eye. Subjects experienced each experimental condition on different days following the above procedures. The position of a subject and the panel is the same as in Fig. 11.

(II) Result

Generally, it has been said that workload should be examined using multiple indicators (Hayashi et al., 1981). We adopted the following four indicators to evaluate workload (Ito, 1988).

i. Time per task: As workload is increased, performance drops, and the time should increase.

ii. CFF: As workload is increased, the CFF should decrease. We calculated the average value of top-down CFF and bottom-up CFF for each measurement and focused on the difference from the first value.

media/image19.png

Figure 17.

Experimental procedure

iii. Cardiac wave ratio (W.R.): This is the ratio of waves to the cardiac frequency. As workload is increased, W.R. should increase. We focused on change against the first W.R. W.R. is given by the following expressions.

WAVE=i=2N1Y(i),i2
(2)
Y(i)=1.:if{X(i)X(i+1)}and{X(i)X(i+1)}orif{X(i)X(i+1)}and{X(i)X(i1)}Y(i)=0.:elseX(i)X(i):TimesequentialdataofRRintervals(i=1,2,...,N)N:Cardiacfrequency
(3)

iv. Self-evaluation score: This is the total score for all questions. As workload is increased, the score should increase. We focused on the difference from the first score.

First, the result for indicator (i) is given in Fig. 18. We cannot find any significant change in this chart, so it seems that their performance hardly decreased. Second, the result of indicator (ii) is given in Fig. 19. This chart shows that the CFF gradually decreased when the paper-based manual was used; however, it did not do so when the AR-based manual with an STD or RSD was used. Thus, it seems that their workload did not increase when they used the AR-based manual as much as when they used the paper-based manual. Third, the result of indicator (iii) is given in Fig. 20. This chart shows a similar tendency to Fig. 19. That is, the W.R. in the condition of the paper-based manual increased monotonously, but the W.R. in the conditions of the AR-based manual went up once and back down. Fourth, the result of indicator (iv) is given in Fig. 21. In this chart, we can see that the score in any condition increased as the terms progressed. However, there was some difference in the scores after the third term between the conditions.

From these results, we can understand that the AR-based manual does not add more workload to workers than the paper-based manual. In fact, the AR-based manual saves time for these tasks, so it can reduce their workload compared with the traditional way.

media/image23.png

Figure 18.

Shift of average time per task as terms progress

media/image24.png

Figure 19.

Shift of CFF as terms progress

3.4.6. Attention to surroundings

(I) Setting details

The subjects were 14 students whose right eyes were dominant. The position of a subject and the panel is the same as in Fig. 12. The panel was surrounded by 45 small LED boards (NT-16, made by EK JAPAN), which were positioned in a horizontal range of −90 (deg) to +90 (deg) and a vertical range of −60 (deg) to +60 (deg), as shown in Fig. 22. On each board, 10 LEDs were lined up. During the experiment, the LED boards flashed at random.

media/image25.png

Figure 20.

Shift of W.R. as terms progress

media/image26.png

Figure 21.

Shift of self-evaluation score as terms progress

media/image27.png

Figure 22.

Positions of a subject and the LED boards: view from the top (left) and view from the side (right)

Their task was to insert 20 plugs into holes on the panel according to the manual as quickly as possible. They were told to concentrate on this task, but if they noticed one of the LED boards flashing, they should say “Found” immediately. The experimental conditions were as follows: a) using the paper-based manual, b) wearing an STD on the non-dominant eye and c) wearing an RSD on the non-dominant eye.

(II) Result

The detection rate of each LED board in the condition of using the paper-based manual is given in Fig. 23, that in the condition of using STDs is given in Fig. 24 and that in the condition of using RSDs is given in Fig. 25. In these charts, the rate can be distinguished by colour. Darker cells represent higher rates. By comparing these charts, we can see that Figs. 24 and 25 include more dark cells than Fig. 23, in particular in the horizontal range of −60 (deg) to +60 (deg) and the vertical range of −45 (deg) to 0 (deg). This is probably because the subjects could keep viewing the panel through the task when they used the AR-based manual. However, we can also observe that when they used RSDs, they could not notice any flash positioned in the vertical range over 55 (deg). This is possibly because their view to the upper front was blocked by the RSD’s frame (Fig. 10). Summarising the above results, we can infer that it was easier for workers to notice a change in their surroundings when they used STDs than when they used the paper-based manual. Moreover, when they used RSDs, they could easily notice changes occurring in the vertical range under 50 (deg), but they

media/image28.png

Figure 23.

Detection rate of each LED board when subjects used the paper-based manual

media/image29.png

Figure 24.

Detection rate of each LED board when subjects used STDs

media/image30.png

Figure 25.

Detection rate of each LED board when subjects used RSDs

could not notice those in the vertical range over 55 (deg). However, it is expected that RSDs will become smaller in the near future, so the problem of blocked sight will be improved.

3.4.7. Effect of information presented by HMDs (static background)

(I) Setting details

For the experiment, a simple operational panel was displayed on a PC monitor (PCG-241N, made by Sony). A pattern of the operational panel is shown in Fig. 26, and a pattern of the AR-based manual is shown in Fig. 28. In addition, an overlay image of the operational panel and the AR-based manual is shown in Fig. 29. On the operational panel, blocks including buttons numbered from 1 to 6 (Fig. 27) were arranged in a matrix of m m. In the AR-based manual, numbers within the range of 1 to 6 were arranged in the same size matrix as on the operational panel. The matrix size of the operational panel and the number of numbers presented at the same time by the AR-based manual were changed depending on the experimental conditions explained in the following paragraph. Clicking a button that

media/image31.png

Figure 26.

A pattern of the operational panel

media/image32.png

Figure 27.

A block on the operational panel

media/image33.png

Figure 28.

A pattern of the AR-based manual

media/image34.png

Figure 29.

An overlay of the operational panel and AR manual

had the same number as that given by the AR-based manual was defined as a unit operation for a block. Subjects’ task was to complete operation for all blocks correctly as quickly as possible.

We set five levels of complexity for the operational panel by changing the matrix size (m m): m = 4 (Fig. 30), m = 6 (Fig. 31), m = 8 (Fig. 32), m = 10 (Fig. 33) and m = 12 (Fig. 34). Further, we set three levels of complexity for the AR-based manual by changing the number of numbers appearing at the same time: the whole indication (m m) (Fig. 35), one-line indication (1 m) (Fig. 36) and individual indication (1 1) (Fig. 37). In one-line and individual indication, subjects had to switch images of the AR-based manual from one line to the next or from one number to the next with a keystroke.

media/image35.jpeg

Figure 30.

Operational panel (m = 4)

media/image36.jpeg

Figure 31.

Operational panel (m = 6)

media/image37.jpeg

Figure 32.

Operational panel (m = 8)

media/image38.jpeg

Figure 33.

Operational panel (m = 10)

media/image39.jpeg

Figure 34.

Operational panel (m = 12)

media/image40.jpeg

Figure 35.

Whole indication (m = 8)

media/image41.jpeg

Figure 36.

One-line indication (m = 8)

media/image42.jpeg

Figure 37.

Individual indication (m = 8)

Fifteen conditions were prepared: five levels of complexity of the operational panel three levels of complexity of the AR-based manual. Subjects were six students whose right eyes were dominant. They wore RSDs on their non-dominant eye, and performed the task five times in each of the fifteen conditions. The numbers presented by the AR-based manual were random. We recorded the operation logs during the task in time sequence. The position of a subject and a PC monitor displaying the operational panel is shown in Fig. 38.

media/image43.png

Figure 38.

Position of the PC monitor and a subject

(II) Result

In the present situations, the complexity of the real view is given. Thus, we focused on how subjects’ performance changed depending on the conditions of the AR-based manual under each condition of the operational panel.

First, we counted the case where a subject clicked on a different numerical button from the number presented by the AR-based manual as an error and calculated the error rate per unit operation. Figs. 39, 40, 41, 42 and 44 show the error rates in each condition of the operational panel (m = 4, 6, 8, 10, 12). The vertical axis is scaled individually for each graph so that the error rates can be compared between the conditions of the AR-based manual under different conditions of the operational panel. When the matrix size of the operational panel was 4 and 6 (m = 4, 6), the error rates were comparatively low when the AR-based manual presented the whole indication. This is possibly because in this case subjects did not have to switch images of the AR-based manual and could pay attention to the operational panel throughout the task. On the other hand, when the matrix size was 8, 10 and 12 (m = 8, 10, 12), the error rates were comparatively high when the AR-based manual presented the whole indication. This is possibly because, when many blocks appear on the operational panel, and furthermore numbers were overlaid on each of those blocks by the AR-based manual, too much information was given to subjects at once. Then they tended to click on wrong buttons or mistake operated blocks for unoperated ones.

media/image44.jpeg

Figure 39.

Error rate per operation unit (m = 4)

media/image45.jpeg

Figure 40.

Error rate per operation unit (m = 6)

media/image46.jpeg

Figure 41.

Error rate per operation unit (m = 8)

media/image47.jpeg

Figure 42.

Error rate per operation unit (m = 10)

media/image48.jpeg

Figure 43.

Error rate per operation unit (m = 12)

Second, we analysed time per unit operation. Figs. 44, 45, 46, 47-4 and 48 show the average time per unit operation in each condition of the operational panel (m = 4, 6, 8, 10, 12). When the matrix size of the operational panel was 4 and 6 (m = 4, 6), the operation time tended to be longer when the AR-based manual presented the individual indication. On the other hand, when the matrix size was 8, 10 and 12 (m = 8, 10, 12), the operation time was shorter in the case of the one-line indication than in the other cases. Moreover, when the matrix size was 12 (m = 12), the operation time was remarkably longer in the case of the whole indication. We can understand these results as follows. When a small number of blocks appeared on the operational panel, the efficiency of operation was affected more by switching images of the AR-based manual than by viewing much information. Conversely, when many blocks appeared on the operational panel, the efficiency of operation was affected more by viewing much information than by switching images of the AR-based manual. When these two factors were balanced, the efficiency of operation was highest.

The above examination yields the following suggestions for designing AR-based manuals. If the real view is not very complex, giving a large amount of information at a time saves workers the trouble of switching images in an AR-based manual. However, if the real view is rather complex, giving too much information reduces workers’ performance, so giving information part by part is recommended. Moreover, in situations where human errors are to be strictly avoided, it will be better to give information one after the other.

media/image49.jpeg

Figure 44.

Operation time(m = 4)

media/image50.jpeg

Figure 45.

Operation time(m = 6)

media/image51.jpeg

Figure 46.

Operation time(m = 8)

media/image52.jpeg

Figure 47.

Operation time (m = 10)

media/image53.jpeg

Figure 48.

Operation time (m = 12)

As an extra challenge, we attempted to build a model that describes the most effective design of AR-based manuals according to real-world conditions (Akasaka et al., 2007 , Tamamushi et al., 2008, Nakanishi et al., 2008b; Nakanishi et al., 2009). First, we considered two aspects of task performance, accuracy (lack of errors) and efficiency (speed). Assuming that both accuracy and efficiency were equally necessary and important, we defined damage to task performance (DP) using both error rates (E) and unit operation time (T).

DP=0.5S(E)+0.5S(T)S(E): Standardised ES(T): Standardised T
(4)

Second, we quantified the conditions of visual information based on the idea of complexity. In general, the more crowded the items are, the more complex the information looks. Thus, we defined complexity (C) as the number of items to be attended to (n) divided by their dispersion (M). M was defined as the standard deviation of the distance from each item to the centre of the items (di: i = 1, 2,..., n) divided by the mean of the distances ( ), so that C did not depend on measurement of di.

C=n/M=nd¯/i=1n(did¯)2/(n1)
(5)

We examined the experimental data and found that the relationship between the complexity of the real view (CR), the complexity of the AR-based manual (CA) and DP could be expressed by the following equation.

DP¯=(2.57×104CR+8.71×101)(6.63×103CA8.76×101)+(1(2.57×104CR+8.71×101)(1.00×10+2CA+6.22)3.00)
(6)

Moreover, we suggested that when CR was given, CA that minimised DP could be determined by the following equation.

min.

C^A=1.00×10+2(1(2.57×104CR+8.71×101))6.63×103(2.57×104CR+8.71×101))6.22
(8)

Equation (6) provides the effective complexity of AR-based manuals according to the complexity of the real view. Accordingly, it can be regarded as the basic model that describes effective design of AR-based manuals using the number and dispersion of information items.

3.4.8. Effect of information presented by HMDs (dynamic background)

(I) Setting details

In this experiment, the blocks displayed on the PC monitor moved left to right at a constant speed (Fig. 49). The numbers on the AR-based manual were synchronised with the blocks (Fig. 50). Thus, the subjects could see the correspondence between them (Fig. 51). A unit operation for a block was defined as clicking a button that had the same number as that given by the AR-based manual. The subjects’ task was to operate each block as correctly and quickly as possible. The task was continued for three minutes.

media/image59.png

Figure 49.

A pattern of the operational panel

media/image60.png

Figure 50.

A pattern of the AR manual

media/image61.png

Figure 51.

An overlay of the operational panel and the AR-based manual

We set three levels of complexity for the operational panel by changing the matrix size (m m): m = 4 (Fig. 52), m = 8 (Fig. 53) and m = 12 (Fig. 54). Furthermore, we set four levels of complexity for the AR-based manual by changing the number of numbers appearing at the same time: 1-indication (1 1) (Fig. 55), 6-indication (3 2) (Fig. 56), 8-indication (4 2) (Fig. 57) and 16-indication (4 4) (Fig. 58). Twelve conditions were prepared: three levels of complexity of the operational panel four levels of complexity of the AR-based manual. During the task, if subjects operated the blocks indicated by the AR-based manual, they switched the manual images to refer to the next indication with a keystroke.

media/image62.jpeg

Figure 52.

Operational panel (m = 4)

media/image63.jpeg

Figure 53.

Operational panel (m = 8)

media/image64.jpeg

Figure 54.

Operational panel (m = 12)

media/image65.jpeg

Figure 55.

indication (m = 8)

media/image66.jpeg

Figure 56.

indication (m = 8)

media/image67.jpeg

Figure 57.

indication (m = 8)

media/image68.jpeg

Figure 58.

indication (m = 8)

Subjects were 15 students whose right eyes were dominant. They wore RSDs on their non-dominant eye and performed the task once in each of the twelve conditions.

(II) Result

First, we counted the case where a subject clicked on a different numerical button from the number presented by the AR-based manual as an error and calculated the error rate per unit operation. Figs. 59, 60 and 61 show the error rates for each condition of the operational panel (m = 4, 8, 12). The vertical axis is scaled individually for each graph so that the error rates can be compared between the conditions of the AR-based manual under a condition of the operational panel. For any matrix size on the operational panel, the error rate was high when the AR-based manual presented 1-indication. Moreover, as a large number of blocks appeared on the panel, the error rate when the AR-based manual presented 16-indication increased.

Second, we counted the number of blocks that left the PC monitor without any operation as a misoperation and calculated the rate per unit operation. Figs. 62, 63 and 64 show the misoperation rate. These results show a tendency roughly similar to that of the error rate (Fig. 591, 60 and 61).

Because 1-indication required switching the manual images after every operation, it took more time than the other indication patterns. Moreover, because subjects had to adjust the superimposition every time in this case, errors tended to occur. On the other hand, when many blocks existed on the panel, the 16-indication made the distinction between each block more difficult. Further, in particular, when the matrix size was 12 (m = 12) and when the AR-based manual presented 16-indication, the blocks, which were quite close to each other, moved continuously, such that it was not easy for subjects to correctly operate on the block that they intended.

media/image69.jpeg

Figure 59.

Error rate per operation unit (m = 4)

media/image70.jpeg

Figure 60.

Error rate per operation unit (m = 8)

media/image71.jpeg

Figure 61.

Error rate per operation unit (m = 12)

media/image72.jpeg

Figure 62.

Rate of misoperation (m = 4)

media/image73.jpeg

Figure 63.

Rate of misoperation (m = 8)

media/image74.jpeg

Figure 64.

Rate of misoperation (m = 12)

From the above examination, we can understand the relationship between AR manual design and task performance as follows. When the items flow dynamically in real-world conditions, the complexity of visual information depends on the number of items, their dispersion and the distance between neighbouring items. Task performance is damaged if the complexity of the real view and that of an AR-based manual is unbalanced.

As an extra challenge, we tried to build a model that gives the most effective design of AR-based manuals according to real-world conditions (Tamamushi et al., 2009a, Tamamushi et al., 2009b). First, assuming that both aspects of task performance, accuracy (lack of errors) and efficiency (speed), were equally necessary and important, we defined damage to task performance (DP) using both error rates (E) and misoperation rates (L).

DP=0.5S(E)+0.5S(L)S(E): Standardised ES(L): Standardised L
(9)

Second, we quantified the complexity (C) of visual information that was dynamically flowing using the number of items to be attended to (n), their dispersion (M) and the separation between neighbouring items (s). M was defined as the standard deviation of the distance from each item to the centre of the items (di: i = 1, 2,..., n).

C=log(en/M)=log(ena/i=1n(did¯)2/(n1))
(10)

We examined the experimental data and found that the relationship between the complexity of the real view (CR), the complexity of the AR-based manual (CA) and DP could be expressed by the following equation.

DP¯=(8.66×101CR8.02)(1.67CA6.40)+(4.40×101CA6.91×101)+3.42)(1.38×10+2CA+1.38×10)2.14×103)
(11)

Moreover, we suggested that when CR was given, CA that minimised DP could be determined by the following equation.

min.

C^A=1.38×10+2(4.00×101/(CR+6.91×101)+3.42))1.67(8.66×101CR8.02)1.38×10
(13)

Equation (10) provides the effective complexity of AR-based manuals according to the complexity of the real view. Accordingly, it can be regarded as the basic model, which gives effective design of AR-based manuals when the items are dynamically flowing in real-world conditions.

4. Human factor guideline for applying AR-based manuals

We clarified experimentally the conditions favourable for the effective use of AR-based manuals in actual work situations, particularly considering that AR manuals require users to wear HMDs and to view digital images superimposed on the real view. Based on the results, a human factor guideline for effective use of AR manuals is proposed as follows.

1. Difficulty in preparing to use AR-based manuals:

It is easy to prepare to use either STDs or RSDs. Thus, there is no disadvantage compared to traditional paper-based manuals. Simple and compact HMDs just like glasses are expected to be developed in the near future, allowing workers to more easily use AR-based manuals.

2. Effect of eyesight correction:

Workers with contact lens can use AR-based manuals with either STDs or RSDs as well as workers with good eyesight. Although it is currently difficult for workers wearing glasses to also wear HMDs, this is simply a technical problem. Depending on their needs, it is fully possible to redesign HMDs for workers wearing glasses.

3. Effect of eye dominance:

Wearing STDs or RSDs on the non-dominant eye is recommended when these are used for presenting AR-based manuals in assembling or inspecting tasks, because it is of primary importance for workers to clearly see real-world objects. Design of the AR-based manuals should be simple enough for the workers to easily read them with their non-dominant eye.

4. Effect of surrounding illumination:

There is almost no problem in using AR-based manuals under standard illumination. However, under extremely high or low illumination, RSDs should be better. If workers have to perform a difficult task under morning sunshine or a delicate task in a dark laboratory, they may not clearly read a paper-based manual; however, they can clearly read an AR-based manual presented by an RSD.

5. Workload:

AR-based manuals do not add more workload than paper-based manuals do. In fact, AR-based manuals enable workers to refer to task-related information without using their hands, increasing task efficiency, so they can reduce the workload compared with the traditional way.

6. Attention to surroundings:

Workers can easily notice changes in their surroundings even when they wear STDs or RSDs. In current situations, a problem remains because the frame of these HMDs blocks part of a worker’s view; however, the design is fully expected to be improved in the near future.

7. Effect of information presented by HMDs (static background):

When the real background is static, the effective complexity of AR-based manuals can be determined according to the complexity of the real view, which is described by the following formula.

C^A=1.00×10+2(1(2.57×104CR+8.71×101))6.63×103(2.57×104CR+8.71×101))6.22
(14)

Figs. 65, 66, 67, 68 and 69 show rough examples of the effective complexity when the background real view is static.

8. Effect of information presented by HMDs (dynamic background):

When the real background is dynamically flowing, as products move on a conveyer belt, the effective complexity of AR-based manuals can be determined according to the complexity of the real view, which is described by the following formula.

C^A=1.38×10+2(4.00×101/(CR+6.91×101)+3.42))1.67(8.66×101CR8.02)1.38×10
(15)

Figs. 65, 66 and 67 show rough examples of the effective complexity when the real background view is dynamic.

media/image82.jpeg

Figure 65.

An example of effective complexity (m = 4)

media/image83.jpeg

Figure 66.

An example of effective complexity (m = 6)

media/image84.jpeg

Figure 67.

An example of effective complexity (m = 8)

media/image85.jpeg

Figure 68.

An example of effective complexity (m = 10)

media/image86.jpeg

Figure 69.

An example of effective complexity (m = 12)

media/image87.jpeg

Figure 70.

An example of effective complexity (m = 4)

media/image88.jpeg

Figure 71.

An example of effective complexity (m = 8)

media/image89.jpeg

Figure 72.

An example of effective complexity (m = 12)

5. Conclusion

In this chapter, first, we introduced experimental data showing that an AR-based manual helped reduce errors and enhanced task performance compared to the traditional paper-based manual, and demonstrated that AR manuals show promise for practical use. Second, considering that the main characteristics of using an AR-based manual are wearing HMDs and superimposing a digital image on the real view, we showed the results of detailed experiments conducted to clarify the human factor requirements that should be examined before practical use. Furthermore, we summarised these results and provided a guideline for effective use of an AR-based manual by workers.

Although some imperfect points remain in the usability of an HMD, which plays a major role in an AR-based manual, in current situations these can be addressed by rising demand and continued efforts of developers. In fact, a certain Japanese manufacturing corporation has developed a compact and light HMD that looks like glasses, and some industries are interested in introducing it into their operations.

New application will be expanded into other fields by the mutual advance of both users’ needs and technical development. In the case of an AR-based manual, although the technical aspects are maturing rapidly, information concerning the influence of its introduction on workers or the restrictions in using it has not been sufficiently provided to field workers yet. Accordingly, workers have not been able to discuss and judge, based on data, whether they should choose an AR-based manual instead of the paper-based manual used in the past. We are sure that the contents of this chapter complement such information and we hope that it helps workers understand AR-based manuals and triggers dramatic improvement in industrial settings.

References

1 - T. Akasaka, M. Nakanishi, Y. Okada, 2007 Appropriate Complexity of Image Displayed on Head-Mounted Displays in Augmented Reality, Proceedings of the 12th International Conference on HCI (Human Computer Interaction), CD-ROM
2 - Y. Hayashi, K. Ogi, S. Nakamura, Y. Yazu, T. Yukimachi, 1981 Ergonomics (in Japanese), Japanese Standards Association, 152 167
3 - Y. Ito, 1988 The relationship between the changes of biophysiological reactions and subjective mental workload in the typing tasks under time pressures, Japan Ergonomics Society, 24 4 253 260
4 - JIS Z 9110-1979. 1979 Recommended Levels of Illumination, Japan Standards Association
5 - Institute for Science of Labour. 1970 Questionnaires on Self-Conscious Stress (in Japanese), Digest of Science of Labour, 25 6 12 33
6 - M. Nakanishi, Y. Okada, 2003 A Study on Application of a Monocular See-Through Head-Mounted Display for Visual Tracking, Proceedings of IEA (International Ergonomics Association) XVth Triennial Congress, CD-ROM
7 - M. Nakanishi, Y. Okada, 2004 A Study on Application of a Monocular See-Through Head Mounted Display for Air Traffic Control System, Proceedings of the 7th International Conference on WWCS (Work with Computing Systems), CD-ROM
8 - M. Nakanishi, Y. Okada, 2005 Information Sharing by Using Monocular See-Through Head-Mounted Displays for Cooperative Work in Air Traffic Control Systems, Proceedings of the 11th International Conference on HCI (Human Computer Interaction), CD-ROM
9 - M. Nakanishi, Y. Okada, 2006a Practicability of using active guidance with retinal scanning display in sequential operations, Proceedings of IEA (International Ergonomics Association) 2006 Triennial Congress, CD-ROM
10 - M. Nakanishi, Y. Okada, 2006b Development of an Instruction System with Augmented Reality Technology-for supporting both skilled and unskilled workers-, Proceedings of INSCIT(International Conference on Multidisciplinary Information Sciences and Technologies), 112 116
11 - M. Nakanishi, T. Miura, Y. Okada, 2007 How does the Digital Manual with Controllability and Perfect-Transparency Effect on Workers’ Cognitive Processes?-Application of Augmented Reality Technology to Manufacturing Work-, Proceedings of the 4th International Conference on CITSA(Cybernetics and Information Technologies, Systems and Applications), CD-ROM
12 - M. Nakanishi, M. Ozeki, T. Akasaka, Y. Okada, 2007 Human Factor Requirements for Applying Augmented Reality to Manuals in Actual Work Situations, IEEE Systems Man & Cybernetics 2007, CD-ROM
13 - M. Nakanishi, T. Akasaka, Y. Okada, 2008 Modeling Approach to "Effective Augmentation" for Designing Manuals with Augmented Reality, Proceedings of the 2nd International Conference on AHFE (Applied Human Factors and Ergonomics), CD-ROM
14 - M. Nakanishi, M. Ozeki, T. Akasaka, Y. Okada, 2008 What Conditions are Required to Effectively Use Augmented Reality for Manuals in Actual Work, Journal of Multimedia, 3 3 34 43
15 - M. Nakanishi, K. Taguchi, Y. Okada, 2009a How can visual instruction with see-through HMD be effectively used in safety-critical fields?, Proceedings of the Asia Pacific Symposium on Safety 2009, CD-ROM
16 - M. Nakanishi, S. Tamamushi, Y. Okada, 2009b Study for Establishing Design Guidelines for Manuals Using Augmented Reality Technology-Verification and Expansion of the Basic Model Describing “Effective Complexity”-, Enterprise Information Systems, 11 16 ,
17 - S. Tamamushi, M. Nakanishi, Y. Okada, 2008a How to Design of Task-related Information Overlaid by AR Technology-Focusing on Task Characteristics-Proceedings of the 2nd International Conference on AHFE (Applied Human Factors and Ergonomics), CD-ROM
18 - S. Tamamushi, M. Nakanishi, Y. Okada, 2008b Mathematical modelling for determining effective information quantity given by Augmented Reality Manual-Overlay on Dynamic Background-, Proceedings of Asia Pacific Symposium on Safety 2009, CD-ROM
19 - S. Tamamushi, M. Nakanishi, Y. Okada, 2009 Suitable Amount of Information for an Augmented Reality Manual-Superimposing on Dynamic Actual Field of Vision-, Proceedings of the 13th International Conference on HCI (Human Computer Interaction), CD-ROM