Wearable Technology in Automotive Industry: from Training to Real Production

1. Introduction

This paper describes two different research activities carried out in the context of the wearIT@work project to analyse the benefits and drawbacks of using wearable technology to support blue-collar workers in real situations. To this aim we describe the UCD process followed, the tests performed and the results obtained in several experiments performed at the SKODA production facilities in Czech Republic.

User Centred Design approach has been followed in the two scenarios identified: training and quality control activities.

Compared to stationary computer systems, mobile- and wearable computing technology have seriously caught up in performance, functionality, scalability. This makes training solutions based on mobile- and wearable computing an attractive consideration for industrial organisations. In this sense, the objective was to supplement the training procedures at Skoda with a context-sensitive wearable computing solution. The idea was that the trainees gained mobile access to the information to carry out their assembly tasks. In fact, the wearable system was used to recognize the context of performed work, and as a result to provide the trainee with the required information to adequately perform individual assembly tasks. The wearable solution was able to track and analyse the trainee’s actions, while providing the end-user with means for error handling. As a result, semi-autonomous training of trainees in automotive production was possible.

In the second scenario we moved to real production environment, specifically to the CP8: check point 8, the place where cars are visually and manually inspected before being delivered to customers. There, two worker teams work in parallel in inspection tasks: examining lights and bumpers misalignment, identifying dents and scratches, checking the spaces between doors and windows, and any other kind of faults.

Each time an error is found, the worker reports in one of the three check-list forms they handle. The sheets have a matrix structure identifying the different parts in the car and the possible types of faults.

The objectives of using wearable technology in this scenario are multiple, among others: making the worker activity easier and more efficient, allowing a paperless and hands-free inspection, to guarantee that all verifications have been performed (avoid oversights-mistakes), allow permanent documentation access and to facilitate workers interaction.

2. Summary of activities carried out

The ‘wearit@work: empowering the Mobile Worker by wearable computing’ project aimed to study wearable technologies and their application in the workplace to improve the conditions in which workers carry out their activities.

To this end we created the multidisciplinary consortium with 35 partners from 15 countries that have worked together for the four and half years that the project was officially programmed for.

The technological developments have been based on User Centred Design (UCD) and have been developed in 4 different scenarios:

• Maintenance. Led by Giunti Labs and EADS. The target market was made up of workers carrying out maintenance tasks in the aeronautical sector.

• Emergencies. Led by the Paris fire service and Fraunhofer FIT. The study focused on fire-fighters in emergency situations.

• Health. Led by GESPAG and SAP. This study focused on health workers on their daily ward rounds in hospitals.

• Production. Led by TEKNIKER and SKODA. This is the scenario that Tekniker has been most involved in and which will be described in more detail below.

• The following common viewpoint was reached after internal discussions about the wearable computing concept compared with mobile computing:

Figure 1.

1. Mobile vs Wearable

“Wearable computing is working with instead of on the computer, meaning: Using the computer is not the main activity but it is used while the activity is being carried out”

In the production scenario, TEKNIKER focused on designing prototypes and experimenting with end users in the two SKODA plants where the developments and pilot studies were validated.

• Pilot Case Study: Learning in the Vrchlabí plant

• Pilot Case Study: Quality Control in the Mladá Boleslav plant

3. Pilot Case Study: Learning

It took place at the SKODA plant in Vrchlabí. The facilities have what they call an ‘E-factory’, this being the certification process that must be passed by the workers before they start working on the production line. During this training period they are taught the basics of the work related to a particular task by means of a PowerPoint presentation and small questionnaires. After passing the test they then have a prototype car chassis on which they have to carry out the assembly task in question within a time limit while being supervised by an instructor. When they achieve the established production parameters they can join the assembly line.

Figure 2.

2. The learning island, the foreman and the trainee

This pilot case study was designed to create an environment that facilitated the worker training programme so that:

• The training period could be shortened

• It could be carried out by the trainees alone, without needing to have an instructor

When designing the prototype, different factors were taken into consideration to ensure that the wearable system did not interfere with the assembly work and that it could be accessed as naturally as possible.

The development process involved working directly with the plant workers and carrying out a usability experiment in the plant.

The prototype consisted of the following components:

• Hardware: Xybernaut V computer, Microoptical SV6 Head Mounted Display and a standard headset.

• Software. A programme that presented the 14 steps needed to carry out the chosen task in the proper sequence and allowed different forms of interaction. Each of the tasks was documented with the use of text, photo and video resources that could be selected by users according to their preferences.

Figure 3.

3. UI of the application

The operators had to use the application to ‘learn’ and practice assembling the right-hand front headlight.

As certain functionalities were not yet implemented, the Wizard of Oz (Kelley J.F., 1983) technique was used to simulate them. For example:

• We did not have an Automatic Speech Recognizer (ASR) for voice-based interaction, so we used a Czech speaker who understood the user and ‘simulated’ the interaction using a remote control application.

• To simulate task recognition, the ‘Wizard’ pressed the ‘Next’ button every time the operator completed a task, so that the information about the following task was displayed on screen.

• We wanted to test the validity of a virtual keyboard, so we created keyboard strips that were attached to the operators. They pressed the simulated keys and the ‘Wizard’ interacted with the application.

Figure 4.

4. Virtual fabric keyboard layout

The system recorded the most important process information: time used and mistakes.

The operators had to complete an evaluation questionnaire after carrying out the process.

The analysis of the experiment results allowed us to conclude that the system was well accepted in general terms, highlighting the nil use of the videos and the difficulty of accessing the information using the HMD.

As a result of the experiment, we decided to carry out more intensive experiments at TEKNIKER, in which a large number of individuals could take part without the limitations found in the Czech Republic caused by the lack of a common language for the participants and the TEKNIKER researchers.

3.1 Experiment: Validation of Wearable Technology for Learning Tasks

In order to carry out the experiments, we built a metal structure on which those taking part in the tests had to carry out the allocated assembly tasks. The following was taken into account when designing the structure:

• It should allow different manual tasks to be carried out

• All tasks should be of similar difficulty

Figure 5.

5. Platform for experiments at Tekniker

The task to be carried out by the operators consisted of:

• Assembling a set of metal shapes (three shapes) on a metal plate with 100 holes with different size threads. The assembly had to be carried out using Allen head screws and an Allen key.

• Make 3 pairs of connections using three wires on a panel with 16 BNC connectors.

The experiment was designed to:

• To validate the initial results of the experiment at Vrchlabí

• To measure user acceptance

• To analyse the improvement in learning in terms of time

40 workers divided into two groups took part in the experiment:

• With the first group, the aim was to measure and compare their efficiency in assembling when they accessed to information on paper compared to using wearable technology. To this end, the participants in the experiment had to carry out a single assembly as quickly as possible.

• The second group had to learn (memorise) the whole assembly process as quickly as possible, this being the parameter to be measured. In this case, the operators had to carry out the assembly as many times as was needed with the support of information until they managed to learn it and carry it out with no help at all. The information could be accessed on paper or by using wearable technology. In order to take into account the short-term memory effect, the assembly had to be carried out once again the following day.

In both cases, the operators had to perform the experiment twice: once using information on paper and once using wearable technology. In the second case, each operator was allocated a different mode of interaction:

• Access always involved using an Head Mounted Display (HMD)

• Interaction was in one of the following modes: textile keyboard on the sleeve of the working clothes, by voice or implicit interaction, this involving simulating that the system had detected completion of a task to present the information about the following task automatically.

The following parameters were measured during the experiment:

• Time

• Mistakes made

• Users acceptance, measured through open and closed questionnaires

• Measurement of mental load. We used the NASA TLX test (Hart & Lowell,1988), a subjective tool for evaluating the mental load of the operators when they carry out activities using machines

• Influence of learning styles. We used the VARK questionnaire (Fleming, 2001) to provide a learning preferences profile in accordance with four categories: Visual (learn through seeing), audio (learn through listening), Reading/writing (learn through reading) and kinaesthetic (learn through doing)

The workers used the following infrastructure for the wearable technology test:

• Microoptical VI head mounted display

• OQO computer

As at Skoda, we used the Wizard of Oz technique to simulate the three types of interaction (textile keyboard, voice and automatic recognition of activities).

The following fundamental conclusions were drawn from the experiments:

• Users improved their efficiency when they used recognition of activities as their source of automatic interaction. The operations were performed faster and with fewer mistakes: using implicit interaction (automatic recognition of activities) they used an average of 67 seconds less than when accessing the information on paper, the second best option.

• Users did not learn faster using wearable technology. They only achieved similar learning times to learning using paper when they used implicit interaction.

• Curiously, during the test carried out the following day, learning using paper gave the best results and implicit interaction the worst.

• Interaction using the voice was the option preferred by the users.

• Information in images was better accepted than that presented as text.

• In general terms, the workers rated the wearable technology based system as very useful for carrying out complex tasks, allowing them to work with their hands free and avoiding movements to access information

3.2 Experiment: Usability of Visual Information Access Devices

The second experiment was designed to compare the benefits of using Head Mounted Displays (HMDs) with using a large monitor near the workstation to access the information.

20 workers took part in the experiments and they had to perform four complete assembly procedures using the above-described infrastructure.

The devices compared were:

• A large monitor near the workstation

• HMD Carl Zeiss look-around binocular

• HMD, Carl Zeiss see-through HMD

• Microoptical VI monocular look-around.

The time used by each operator for assembly was measured and a usability questionnaire was used to measure the satisfaction and subjective opinions of the participants.

The experiment allowed us to conclude that:

The best response in terms of times was obtained when using the big monitor

The worst response was obtained using the binocular model HMD. However, we should point out that this access model obtained the best ratings in the satisfaction survey.

Figure 6.

6. Zeiss HMD used in the experiment

3.3 Wearable System Prototype

On the basis of the results obtained from the experiments and trials at the Skoda plant and at Tekniker, the first ‘real’ prototype was defined and built in collaboration with ETHZ, Univ. Passau and HP.

The prototype required the inclusion of different kinds of sensors:

1. 5 FSR (force sensitive resistor) and 4 reed switches on the chassis of the car used for the learning process. These sensors detected the termination of certain assembly operations: screwing certain parts, fitting the headlight and checking its correct alignment with the bodywork.

The prototype also consisted of the following elements:

• An RFID reader on the back of the user’s glove. This allowed the user to verify that the right tools are used for the fixing operations.

• The same glove was fitted with a triaxial accelerometer and a gyroscope to detect the movement caused when the preset torque was reached in certain fixing operations.

• All this information was sent via bluetooth to a central P.C. where the activity detection data was processed. Here we used the Context Recognition Toolbox, a Framework containing a set of frequently used data processing algorithms.

The wearable computing prototype used was that shown in the Figure and consisted of:

1. For the wearable computer we used an OQO Model 01+ attached to the operator’s belt.

2. Microoptical VI monocular HMD

3. A Bluetooth Sony Erickson HBH-300 headset

4. A sensorised glove with the RFID reader, gyroscope, accelerometer and a set of FSR sensors to detect muscle activity

Figure 7.

7. Hardware prototype

The software architecture used is shown in the figure below:

Figure 8.

8. Diagram showing the software architecture used for the prototype

The application was written in Java and used the Open Wearable Computing Framework (OWCF) developed in the project, more exactly the following components:

• Wearable User Interface (WUI)

• Context component (JContextAPI)

The application was modelled as a state machine in which each of the sub-tasks making up the assembly process were represented. The transitions were triggered by user actions, both explicit (e.g. by voice) and implicit (automatically detected by the system).

The application could monitor the actions of the user and inform him/her if there was a mistake detected.

Figure 9.

9. Example of the UI, showing the verification positions

4. Pilot Case Study: Quality Control

After the initial phase, where we focused on the operator training process, we transferred our attention to the production line. After in-depth analysis of the possibilities of applying wearable technology, we opted to work on the last of the operations carried out on the cars just before they are sent to the dealers. Manual and visual inspection of the finish of the cars is carried out in this phase.

The process takes place on an inspection line, where a team of operators (two parallel lines of ten operators each) verifies different parts of the cars and, where necessary, repair small defects and misalignments. Basically, they detect impacts, marks on the paintwork, correct door opening and closing, headlight misalignment, etc. Hands and eyes are the tools used as inspection tools. During the inspection process they have to enter in the vehicle, stand, sit, crouch and walk around the cars.

During the process they carry a set of sheets with a matrix on them, on which they have to note down the errors detected. The procedure should not be regarded as a check-list. On the contrary, the operators use their own criteria to carry out the inspection and only mark the form when they find something wrong. This procedure can clearly lead to possible mistakes.

Once again, we followed the user-based design process to design the system aimed at helping operators to carry out their wok in a more usable fashion. To this end, we carried out two on-site studies that gave us an in-depth understanding of the process and gather the opinions of the potential users. This process took place in late 2006 and early 2007 and included video recordings and interviews with operators and line managers.

4.1 Social factors

Over the two days of our visit to the Mladá Boleslav plant, we used the following techniques to gain an understanding of the process from the social point of view:

• Observation – of the workers performing their normal tasks in the workplace, the interactions between them and the supervisors, time pressure, speed of the activity, rotations, atmosphere at work, etc.

• Interviews – semi-structured, based on a questionnaire and aimed at identifying the expectations of the workers as regards wearable computing, their attitude to new technologies and factors related to their normal activities in the plant. 8 people took part in the interviews (4 men and 4 women)

The study led to the following conclusions:

• The atmosphere at work was calm and relaxed. The environment was clean, tidy and well lit. There are common areas for relaxation and meetings. The staff is friendly and polite.

• The workers were continually interacting and helping each other. They rotated throughout every eight-hour shift to avoid the monotony of repetitive work. One person in each team acted as a “joker” to allow other members of the team to leave the line in case of need.

• At inspection point 8, the subject of the final pilot case study, the working atmosphere was even more relaxed.

• Communication – All the workers interviewed thought that wearable computing could be an interesting support tool. They didn’t see it as a threat that would limit their capacity for interaction, but they didn’t think it was either viable or desirable to use it permanently.

• Privacy – Although this was one of our major concerns, the interviewees didn’t rate it negatively. In fact, they said that the presence of cameras in the plant was already more ‘threatening’ than the solution we were proposing.

• Responsibility – There were contradictory ratings regarding the possibility that wearable computing might affect the responsibilities assumed at the workstation.

• Experience with the technologies – except for one person, all the others were habitual users of computers. Obviously, none of them had experience with wearable computing.

In general terms, the workers were willing to use wearable computing because it could be of benefit to them in the following ways:

• To avoid errors and oversights

• To perform the work more quickly and efficiently

• To improve communication mechanisms

4.2 Activity recognition

During this stage of the study, we acquired data to create a prototype that would allow us to identify the activities performed by the workers.

The following sensors were used to capture the movements of the trunk and upper extremities:

7 Xsens MTx inertial sensors to detect body posture

• FSR (force sensing resistor) on the arms: 8 FSR of 4.7 x 4.7 cm. Fitted to each arm to detect gestures and activities

• A set of Ubisense sensors allowed us to locate the position of the operator of the car being inspected. In practice, they gave calibration problems

A worker was equipped with this equipment and the data was captured during an inspection process to allow later calibration of the algorithms for automatic recognition of activities.

Given the low quality and insufficient data obtained, the process was repeated in the laboratory with 8 individuals and 10 hours of data capture. See (Stiefmeier, 2008) for more details.

4.3 Prototype

The prototype for inspection post number 8 was designed to:

• Facilitate and make the activities of the operators efficient

• Allow paper-free inspection

• Guarantee verification of all points by avoiding oversights

• Provide permanent and easy access to the documentation

• Enhance interaction between the workers

On the basis of these goals, a prototype was created with the following functionalities:

• The worker and the car arriving at the inspection post are identified automatically

• Compliance with all programmed inspections is recorded

• Every time a worker is located in a different area around the car, he/she is presented with a list of possible tasks

• Voice is used as the interaction mechanism: using natural language or identifying the columns and rows that represent the verification document (check-list type)

• The OQO cursor is used for interaction whenever necessary

• The operator can consult the list of faults, related documents and establish a VoIP connection with other operators or the shift manager

Figure 10.

10. Inspection using the wearable prototype

The system consisted of the following Hardware:

• Zeiss HMD binoculars

• OQO

• Headset

• The above-mentioned system for recognising gestures and activities

• A specially-designed jacket to carry the hardware (Bo et al., 2006)

4.4 Test & Results

The prototype created was used with 8 Skoda workers to evaluate different factors:

• The validity of paper-free inspection

• Remote support

• Access to documentation

• Voice interaction

• Recognition of activities

• Usability of the HMD and the jacket

A document explaining the experiment was drafted and the workers were asked to collaborate voluntarily. Finally, 8 of them took part in the experiment.

The experiment took place without interrupting normal work processes. The process was as follows:

• They were told how the system worked (in Czech) and they were helped to adjust the hardware

• They carried out their work with the prototype as a support element

• They answered two questionnaires (open and closed questions with a 7 point Liekert scale)

The most significant results of the study were as follows (in brackets the mean value of their answers on a scale from 1 to 7, where 1 means completely disagree and 7 completely agree):

• The system is easy to use (3.7)

• The users didn’t have long to familiarise themselves with the system. A longer adaptation session would improve this perception

• I’ll be able to carry out my work faster (4.8)

• An interesting result that could be improved as they become more used to the system

• The system is comfortable (3.0)

• Heat and the size of the jacket (one size only) could be the reason for this result

• The system is easy to learn (5.0)

• It’s clear that it’s easy to learn, but they need to become more familiar with its use

• The information is effective for completing the work (4.8)

• The font size is right (4.6)

• A specific trial should be carried out to optimise this size

• I like the interface (5.5)

• Positive response

• The system has the expected functionalities (5.3)

• It responds to initial expectations. Continued use could lead to new expectations

• In general, I am satisfied with the system (4.4)

• Positive response

• I can contact the supervisor easily (5.4)

• Relevant

• I always knew where I was in the application (5.7)

• Means that the UI was well designed to avoid operators getting lost between windows

• The commands are obvious (5.8)

• They knew what the result of each command was going to be

• The system responds quickly to the commands (4.8)

• This response is more positive than it may seem, as the Wizard of Oz was used for certain functionalities, something that creates an additional response delay

• I would recommend the system to my fellow workers (4.4)

• Positive and could be improved by resolving certain aspects.

• The glasses are heavy (3.0)

• The glasses used (Binocular HMD from Carl Zeiss) offer the best quality vision, but they are heavy and their use in longer sessions should be analysed

• The jacket makes my work more difficult (4.3)

• Interesting, despite the problems inherent with using a single size for all

• The system is heavy (3.9)

• Includes the weight of the complete unit: jacket + OQO + Cable + HMD battery

• I would use the current system if it were optional (2.7)

• Very negative. Taking other answers into account, we believe that re-engineering the system would improve this rating.

• The on-screen image is large enough (6.8)

• Corroborates the results of the experiment at Tekniker.

• I will make less mistakes in my work (4.0)

• Positive but with room for improvement. The limited time of the experiment did not allow them to evaluate the benefits of the system when real oversights occurred.

• The voice interaction is simple (4.5)

• Positive but with room for improvement through training

• The jacket is comfortable (3.3)

• The size and stiffness of the OQO-HMD cable may lie behind this response.

• Paper-free inspection is easier than using paper (4.3)

• The clothing is very hot (5.9)

• It is easier to access the information on paper (5.0)

• Negative response. The fact that we used a PDF document with no specific formatting may lie behind this.

• I prefer to access the information using the glasses than by using a large screen (3.8)

• No clear conclusion, as they didn’t evaluate access to such a screen from different positions in and around the car.

• I felt tense at times (5.3)

• Not relevant. The presence in the surroundings of 5 members of the research team during the experiment may explain this response.

• The glasses made my work more difficult (4.7)

• They sometimes forgot the fact that this model has an option allowing them to partially remove the glasses.

• I like the on-line supervisor option (5.9)

• Positive.

• I felt controlled (3.8)

• By the system or by the 5 members of the research team?

• I wouldn’t like to use the system all day long (4.2)

• Resolving certain aspects could improve this score.

• I felt that the system distracted me (4.9)

• It was the first time and everything was new: from the hardware to the functionality

• The menus and the information were well organised (6.8)

• This means that the UI was well designed.

• I felt stupid wearing the glasses (5.7)

• Although negative, if everyone used the same system they wouldn’t feel that way.

• It’s easy to get used to the glasses (3.6)

• They didn’t have enough time to become familiar with them.

• The jacket made it difficult to get into the car (5.4)

• The size and stiffness of the OQO-HMD cable may lie behind this response.

• I was afraid of breaking the system (3.3)

• Positive, it was the first time and the system was not specifically designed not to be damaged or not to cause damage.

• I was afraid of damaging the car with the system (3.4)

• Positive, it was the first time and the system was not specifically designed not to be damaged or not to cause damage.

To sum up, the evaluation was positive, although we detected certain aspects with room for improvement.

5. Final conclusions and acknowledgement

The wearIT project has highlighted the possibilities of wearable computing when it comes to improving the working conditions of certain groups of workers. It’s obvious that there is no single valid option for all situations, but each case needs to be analysed with the help of the user groups to design a tailored solution.

In more general terms, we need to make more effort to design the hardware elements needed to exploit these benefits: lighter HMD (wireless where possible), more robust computing elements (the original QBIC solution continues to be an attractive option, due to its shape, although it has connection and computing power limitations), longer life batteries, more usable headphones, systems with better heat dissipation, etc.

We would like to thank the European Union for the funding that made it possible to carry out this research as part of the Sixth Framework Programme.

We would also like to thank all the Partners who collaborated on the project, especially Skoda, ETHZ, Univ. Passau, Univ. Cork, HP-Italy, Edna Pasher, Carl Zeiss, BIBA, Unity and TZI-Univ. Bremen