To maintain a competitive edge in global market, manufacturing enterprises must leverage their digital information assets, which include a tremendous amount of engineering data: CAD models, FEM analysis, tolerances, annotations, etc.. An emerging technology called Digital Master embeds all the engineering knowledge about a product into its CAD model. Tools and process to efficiently manage, distribute, and modify this information are essential. The digital master, which virtually eliminates the stacks of technical drawings, is not yet completely adopted in industrial practice, mainly because engineering software do not support at best all phases of the virtual product development process. At present time, Engineering Data Management (EDM) tools use standard desktop-based GUI, which demonstrated the following limits: scarce usability for not CAD professionals, low cooperation level, limited management for different expertises and low integration between real and virtual models.
As far as regards usability, current Graphical User Interfaces (GUI) in EDM are mainly based on part features tree view and spreadsheets operations. This GUI may not be user friendly to navigate and perceive complex CAD models or to be used in manufacturing environments. Moreover desktop GUI does not facilitate team working, because of the physical barrier of the computer screens (called “immediate individual environment”). Effective EDM tools should provide a better digital cooperative workspace similar to a meeting table and should improve communication among experts in different fields.
Another issue to face in current EDM systems is the information overload. A new generation of software tools is needed to simplify the visualization and database access according to users’ roles and expertises.
An additional drawback of current EDM software is the lack of a unique workspace where virtual models, technical drawings and physical products can coexist for discussion. This scenario is common in industry because a virtual shape (i.e. CAD model) often needs to be compared with physical mock-ups.
According to the authors’ vision, the ideal collaborative drawing-less workspace for EDM could benefit from Augmented Reality (AR) technology (Figure 1).
Born in military and aerospace applications to augment information-dense environments with digital data, AR can provide an ideal solution for Digital Master exploration. In fact, Dunston demonstrated with engineering user tests, that the perception of 3D design can be improved using AR (Dunston et al., 2002).
Moreover AR system can provide different levels of model perception according to the type of display and the viewpoint control. Lee categorized them in three main configurations (Lee et al., 2005). In mobile configurations the viewpoint is coincident to the user’s eyes. This can be achieved by see-through displays or by using projectors (spatial augmented reality). The users are free to change their viewpoint while the digital content is overlaid onto the real scene. In fixed configurations, the real scene is acquired by a fixed camera positioned accurately to cover the entire workspace. The augmented scene is visualized on a display, usually a large screen, decoupled from the real world. The user is not compelled to wear or hold any device. Finally the tele-mobile configuration is similar to the fixed one, but with the user moving the camera.
Several different industrial showcases of AR are reported in the ARVIKA project (www.arvika.de). They demonstrated that one of the most important aspect of AR is the possibility to use tangible interfaces. A Tangible User Interface (TUI), is a user interface in which a person interacts with digital information through the physical environment. TUIs have emerged as an alternative paradigm to the more conventional Graphical User Interfaces (GUI) letting users manipulate objects in virtual space using real, thus “tangible”, objects (Ullmer et al., 2000). The early developments of TUIs date back to the early 1980s, when Frazer explored different approaches to parallel physical and digital interactions with his 3D data input devices (Frazer, J., 1995).
The ARIEL system (Mackay et al., 2000) was one of the first examples in literature of TUI applied to technical augmented reality environments. ARIEL is based on a graphics tablet for capturing user gestures and on a projector to overlay 2D multimedia annotations onto real technical drawings. Their experiments proved the benefits of a tangible medium (i.e. the paper drawings) when augmented with digital content.
Terry presented JUMP, a tangible interface for augmented paper-based architectural documents (Terry ad al., 2007). He presented a novel set of tangible tools for the navigation and interaction using 2D augmented technical drawings. In JUMP, filter tokens can be placed on the paper to modify the visualization of electrical, mechanical, and structural information. They developed also a physical rectangular selection tool by framing a portion of the drawing with physical brackets.
Some researchers have proved by user tests that gesturing, navigation, annotation and viewing are the four primary interactions with design artefacts in technical meeting (Tory et al., 2008). According to their studies, the form of the design information (2D vs. 3D, digital vs. physical) has minimal impact on gesture interactions, although navigation varies significantly with different representations. They spotted bottlenecks in the collaborative design process when meeting participants attempted to navigate digital information, interact with wall displays, and access information individually and as a group.
However, most researches in literature are proof-of-concept prototypes which hardly progress beyond the lab-phase. For this reason, the attention of commercial EDM software producers is more and more focused on user-friendly interfaces in order to deal with the growing database complexity. Dassault Systemes, for example, in its latest EDM commercial products, lets users intuitively browse, zoom, select and inspect the product definition dynamically using virtual 3DLive turntables. 3DLive turntable is an innovative interface, specifically developed for 3D model understanding and tangible rotation using a touch-screen.
But TUI cannot replace completely the common GUI. Precise selection, text and numerical input, very common in EDM system are more efficient using a common GUI based (keyboard and mouse) interaction.
By means tangible interfaces and AR we developed a novel and efficient interaction paradigm with the digital master for better perceive, understand and add contents to EDM knowledge.
2. Tangible interfaces for engineering
Tangible interfaces are a novel approach to access EDM contents efficiently and user friendly. In our idea, the user can introduce a tangible element with a unique visual ID in the workspace (office desk, meeting table, production workbench, etc.). This visual ID can be a binary coded image, the marker, working as spatial reference for the AR visualization overlay. These markers can be printed or attached as stickers on paper, i.e., technical drawings. We decided to use these tangible interfaces, for three main reasons: (i) because drawings are currently used in any EDM stage, (ii) because technical users are used to handle, store and sort sheets of paper (iii) because of simplicity and the low costs of implementation. We implemented such interfaces in our AR engineering framework (Fiorentino et at, 2009a).
EDM model and data are displayed in real time as 3D info spatially referenced to the tangible interface. Data navigation (pan and zoom) is easily achieved by physically handling the tangible marked drawings (Figure 3).
Accessing directly to the digital master has three main advantages compared to traditional paper drawing: always up-to-date information, access to multimedia content and web 2.0 technology, and custom\role oriented interface.
Collaboration can be fostered by sharing the digital workspace using personal TUI or by sharing a common TUI. In the first case all the users of the team are able to see and annotate the digital master with their data in their personal AR environments. In the second case, exchanging ideas in a common physical workspace is promoted using face to face dialogue and supported by augmented visualization and annotations.
Another TUI can be also a real mock-up, augmented by sticking a set of identification markers. For example, in design review a real product can be augmented with simulation data (Figure 4).
The only TUI does not support at best all EDM activities. Therefore we adopted an hybrid user interface with a combination of GUI/TUI to benefit from the advantages of each approach. The user may exploit TUI for intuitive model navigation and GUI for precise control (i.e. through touchscreen, digital pen, mouse, keyboard).
3. Engineering scenarios
Product development brings together a large number of activities and a single workspace does not satisfy all the user requirements. In this section we present a non-exhaustive list of possible scenarios for tangible interface applications in EDM. For each scenario we outline the following aspects: hardware configuration, viewpoint type, TUI\GUI interaction level, application field, physical collaboration support and remote co-working capabilities.
3.1. Augmented user
The user wears see-through AR glasses connected to a wearable PC. See-through displays allow the user to be aware of the real industrial environment (Fig. 5a). This configuration allows maximum mobility for the user letting him\her work in a large workspace with free hands. EDM database is accessed via wireless network. The interaction is achieved mostly by TUI with none or limited GUI. Suggested applications for this setup are: inspection, training, etc. Disadvantages may include the display resolution, the limited field of view and the optical tracking robustness in hostile manufacturing environments (e.g. dust, electrical noises, bad lighting, etc.).
In another setup the user holds a handheld (flashlight-like) camera and a wearable PC connected to the network. The user is free to move in the industrial environment and to
teleconference with other users remotely logged. The difference compared to previous setup is the viewpoint mobility. The user can move the camera in the industrial environment, reaching potentially every location under wireless coverage. Local tracking is provided by markers (in future may be RFID active markers) and broadcasted to the system. This scenario is particularly important in maintenance, where remote experts can guide and assist the user. The user loads his\her customized visualization of the model and broadcasts it remotely. The main advantage of this configuration is the maximum mobility for point of view. This may also lead to an unsteady point of view due to the fact that the user must hold the camera. TUI and GUI interaction is also rather limited.
3.2. Mobile window
The user holds a tablet PC with a camera on the back side (Fig. 5b). Tablet displays allow the user to be fully aware of the real industrial environment. This configuration allows a good mobility for the user letting him/her work in a large workspace but it requires that at least one hand holds the tablet. EDM database is accessed via wireless network. The interaction is achieved mostly by GUI with the tablet pen. Suggested applications for this setup are: design review, inspection, etc. Disadvantages may include the weight of the tablet and the single-handed interaction limitation.
3.3. Augmented desktop
The user works on a desktop workstation with a camera pointing on a free area on the desk which will be the augmented workspace (Fig. 6a). The AR workspace is limited to the user’s desktop and the model interaction is achieved by moving the TUI (augmented technical drawings) and by the traditional desktop GUI with a mouse and a keyboard. In normal use, the TUI is just a support to the ordinary GUI. For this reason, this scenario is suggested for all EDM tasks which involve an heavy use of keyboard entry of numerical or text data: e.g. detailed design, engineering, numerical analysis, etc. The main advantage of this setup is the similarity with the traditional working environment, allowing an easy access even for a non technical user. Users, in fact, find much easier and intuitive the navigation of 3D models using a tangible metaphor. A limiting factor is that it must be implemented in an office-like environment.
3.4. Augmented workshop
This scenario is similar to the augmented desktop as regards the hardware setup, but it is designed for a production stage environment (Fig. 6b) instead of a clean office desk. The user is on a workbench on the production line where no keyboard or mouse is present. The user can interact by touch screen on industrial monitor and by tangible augmented drawings. An industrial buttonbox can also be used. The main advantages are: both hands free for the user, possibility to display high resolution rendering of the 3D model and EDM data, comfortable working environment, similar to a non augmented one. Ideal applications may be quality check or guided assembly.
3.5. Augmented collaborative table
This scenario supports collaborative workspace at best. It consists of a meeting table with the function of shared augmented area and of a large screen. The screen can be vertical or horizontal and eventually have stereographic or holographic display. The configuration is depicted in Fig. 7a. All users can access to the augmented shared area with their tokens and they can annotate the model using their own PC laptop for precise GUI input. Remotely located users can join the group and participate with virtual meeting tools. The system will take care of the synchronization of the digital master data including annotations, chat and history. The main applications of this scenario are marketing and design review: the shared workspace can contain virtual CAD models, real pre-production mock ups, on-line technical content and simulation results for collaborative discussion. The main advantages of this scenario are the high collaboration support, the coexistence of real and virtual products and the social contact of real meetings.
3.6. Augmented presentation
This scenario considers a speaker who wants to present a solution to a large audience. A large screen is the main visualization device. The data management is achieved mainly by TUI in form of digital drawing or mock-up placed on the speaker’s stand (Fig. 7b). The audience can access to the same digital data with personal visualization devices and can add annotations which are updated in real time for all the members of the discussion.
The characteristics of each scenario are summarized in Table 1.
4. Engineering digital contents
The standards (ASME Y14.41-2003 and ISO 16792:2006) define three levels of product data documentation:
Drawings only: paper drawings are supported to define a product, with the addition of Geometric Dimensioning and Tolerancing (GD&T) symbols, axonometric views as dimensionable views, and supplemental geometry;
Models and Drawing (or reduced content drawings): an engineering drawing is available, but does not contain all the necessary information (i.e. free forms sections) which must be retrieved from the CAD model;
Models Only: the practices, requirements, and interpretation of the CAD data when there is no engineering drawing.
By defining a models only documentation (Figure 2), these standards formalize practices already in use in aerospace and automotive industry, with the main advantage of paving the way to the full digital model integration. One important missing point is that standards do not regulate interactive interfaces. For example, 3D annotations must be placed above fixed datum planes defined once by the user.
Another significant standard for technical publications is the S1000D (www.s1000d.org). Originally developed by the aerospace and defence industries for military aircraft maintenance and operations, the S1000D provides the guidance and rules for the presentation and use of technical information. It defines two main types of documentation: page-oriented presentation on paper or screen and interactive electronic technical publications presentation on screen. This second section provides guidance for software developers of interactive digital technical documentation. It supports desktop based GUIs (mouse, keyboard, touch screen, etc.) and defines in detail screen layout, title\menus bar, dialog boxes, etc. S1000D standard is rather limited in the definition of 3D CAD model navigation and web based tools. In fact it rules 3D visualization in terms of 2D static or animated graphical figures (prospective or orthographic), fly-through (3D model navigation) and helper applications (e.g. Adobe Acrobat) or plug-ins (e.g. Arbortext IsoView). User annotations are supported in the form of: action complete indicator, personal data annotation, and redlining. Callouts are supported to show where a component is located. Due to their novel implementations and the scarce knowledge about their deployment, actual standardization do not support all means of “digital master” navigation, considering recent interfaces such augmented reality or tangible interface.
In next sections we present a technical ontology related to EDM data which can be displayed on the model.
4.1. Ontology of engineering data associated to the model
The digital master is a CAD model which contains two different kinds of information: nominal geometry and product data. The first type contains the mathematical representation of the ideal (i.e. no production faults) shape. Product data are additional technical information such as: material, simulation constraints and boundary conditions, surface finish, tolerances, quality control, simulation parameters, etc. These data are strictly connected to the CAD model features like faces, holes, rounds, chamfers, etc. Technical annotations can be divided in three main functional types: dimensions, geometric tolerances and general annotations.
Spatial dimensions of the model are fundamental elements in design, production and inspection phases. Examples of dimensions are: distance between, angle between, coordinate dimension, angular coordinate, symmetry. Dimensions can be derived from CAD exact geometry or by a formula or by a variable defined in the EDM database.
Dimensions items use, as reference, model features such as: edges, key points, sketches, construction surfaces, axes and coordinate systems. If dimensions are functional, they contain tolerances values, which provide information about the acceptable variation in the size or location of a feature on a part, according to standards (ISO 286-2). While paper drawings represent dimensions by distributing and organizing them among different views (one instance for each dimension is allowed), a unique 3D CAD model representation requires an optimal visualization in order to avoid confusion and overlapping. Dimensioning needs differ also during product lifecycle according to the application. For example in design dimensions must be related to structural stress, while in inspection only the checked features need to be displayed.
4.3. Geometric tolerance
Geometric tolerance is a technical annotation used to establish the relationships among features or datum of parts according to the standards (ASME Y14.5-1994 and ISO 1101:2004).
In 2D drawing, standards require that geometric tolerance must be indicated using two or more rectangular boxes containing the following information: geometric characteristic symbol, datum reference, tolerance zone symbol, tolerance value, material condition symbol.
4.5. Annotations and linking
An large part of the manufacturing process involves different users (experts from design to inspection) to provide digital master with a variety of EDM information in form of text, notes, graphics, sketches, reference, audio\video content, etc.
EDM data annotations can be arrangement on the base of the application: (i) manufacturing (surface finish, welding notations), (ii) inspection (key locations points), (iii) assembly instructions (iv), product information (materials, suppliers, part numbers), etc.
Annotations can be associative or non-associative. An associative annotation is semantically attached to model faces, surfaces, curves, edges, and sketch elements, and even to existing dimensions and annotations. Annotations with leaders have the following components: leader line, break line, terminator, and annotation. An annotation can have more than one leader. The terminator end must move with the element it is connected. Most used associative annotations are Leader, Balloon, Callout, Surface Texture Symbol, Weld Symbol, Edge Condition, Feature Control Frame, Datum Frame, Datum Target. Differently, non-associative annotations are usually located in free space.
|Graphic Element||Dimension||Geometric Tolerance||Annotation|
Annotations are often related to linking functionality. An example is the connection of the model geometry to data as cross-references tables, simulations data, images, 2D view and even external links to web pages or to pdf based digital documentation.
4.6. Data filtering
A three-dimensional digital master requires an optimal visualization in order to avoid confusion and overlapping. "Information overload" is a term coined by Alvin Toffler which refers to an excess amount of information being provided, making the perception of information very difficult for the individual because sometimes the user can not see the validity behind the information. A correct dynamic data filtering is essential for each phase of product lifecycle which requires different dimensioning and annotations. For example, the design phase needs dimensions for assembly, while the inspection phase needs only the dimensions under control.
5. Issues and solutions
5.1. Real time tangible engineering simulation
We decided to implement and evaluate the novel idea of “touch and see” real time FEM analysis (Fiorentino et at, 2009b). The main goal was to allow the user to modify the simulation parameters via a tangible interface and immediately visualize the results overlaid on the real object. Augmented reality visualization techniques display the results as a video overlay “attached” to the real model which can be handled naturally by the user to explore the data.
In our implementation a specific module extracts the data from an engineering simulation software (COMSOL Multiphysics ®) and uploads them to the visualization system.
The user can modify the simulation parameters via a tangible interface and instantly visualize the results overlaid on the real object. The benchmarks demonstrated that the application is ‘real time’ (simulation refresh rate > 6Hz; visualization refresh rate > 30Hz) and the user is not aware of the simulation latency.
5.2. Tracking integration
We decided to manage the tracking system as a plug-in which handles 3D position, orientation and ID of tangible interfaces. We developed a tracking plug-in for the ARToolkitPlus marked based tracking system (Wagner & Schmalstieg, 2007). Other tracking systems (e.g. markerless) may be supported in our framework by simply adding new plug-ins. We implemented a macro in the EDM system which generates and manages the markers ID for each cad part. For example during the 2D draft creation, markers are automatically embedded in the Title Block. When a known marker enters the AR working area, the system automatically retrieves the EDM data, visualizing the 3D geometry and the associated technical annotations. The markers are used to track the location (pose estimation) of the tangible interface. We found that better results can be achieved with multi-marker tracking. The pose estimated from a single marker is not always stable enough, moreover, a single marker can exit the view area. Using multi-marker tracking, the pose estimation stability and precision are dramatically improved by merging information extracted from several pre-calibrated markers. We developed and tested a drawing template with four markers.
5.3. Contents management
An important issue is related to the visualization of technical content associated to the model. These contents can be added as floating 2D labels overlaid on the scene. In our implementation, the rendering of 2D data is based on a standard web browser technology which is stable, widely-documented and popular. We embed the Mozilla FireFox engine into our labels using a library called “NaviLibrary”.
An important benefit of this solution is the integration of web 2.0 based applications into our AR workspace. Chat, forum, web browsing, document viewers, online technical manuals can be accessed with TUI.
Our AR workspace (Uva et al, 2009) uses a server/client architecture to exploit collaboration. Each user may have a different AR configuration while the digital master is hosted on a server and shared/updated among the clients (even geographically distributed).
The data on the server side are received by a PHP front-end, that serializes it to a back-end MySQL database. At any data change, the server broadcasts the modified data to the logged clients.
To improve the broadcast of modifications in EDM we presented a novel approach we called product feed (Fiorentino et at, 2009a). This idea extends to EDM the concept of web feed, widely adopted for notification of web contents. A web feed is a document which contains data (full or summarized text, plus metadata such as publishing dates and authorship) with web links to update versions. This technology is undervalued in the EDM community and in enterprises. Today’s web feed is used for non technical content (blog, news, podcasting, etc.), but feeds could be enhanced by CAD or EDM data. In our implementation when a user subscribes to (with the appropriate permissions) a set of product feed, it is visualized using an aggregator. A feed aggregator, also known as a feed reader, is a client module which aggregates syndicated EDM contents in a single location for easy viewing. For any change, e.g. a price change for a supplied part, the EDM database syndicates the updates to the authorized and interested users.
5.4. Augmented label placement
The placement of 2D labels on 3D objects is a complex task because of occlusion and readability problems. Several studies (Azuma et al., 2003; Götzelmann et al., 2007) have been presented in literature but only a few for technical annotations. This issue is even more critical in AR because the user’s point of view and the background can vary continuously. A three-dimensional model and, specifically, an industrial prototype can assume rather complex structures and can present a considerable amount of information. Showing all labels together would result in overlapping and overloading the scene with information and in losing desirable clearness and effectiveness.
We implemented a dynamic labelling management with view-driven filtering and placement (Figure 9) in a specific plug-in. We considered the label placement as a cost minimization problem. The cost function is a linear combination of several aspects of layout quality.
This algorithm handles the continuous variability of both the virtual scene and background that occurs in AR applications when using tangible interfaces. In order to avoid information overload for complex models, our algorithm does not visualize the labels whose anchor point is occluded in the current view. This adaptive filtering helps the user to perceive clearly and univocally the label association to model features.
5.5. Augmented badge
In a EDM system, for security and competence reasons, data access must be granted according to the user profile. Users should be allowed to view/assign/modify data with a different level of access according to:
Expertise profile (engineer, design, manager, etc.)
Group: R&D, Engineering, Operations, Manufacturing, Quality
Skill Level (advanced, medium, essential)
Security & digital signatures (internal, staff, guest, etc.)
We implemented a plug-in for user management and logging via the augmented badge. The permission to view or to modify data is granted by showing the badge in the AR environment. The augmented badge loads also user’s preferences (layers, GUI settings, personal notepad, etc.) and personal information and 3d avatar. This is extremely useful in a distributed collaborative environment because remote users can visualize information about the person logged. Security can be enforced by additional systems (RFID, smart card, etc.)
All the personal annotations are included in EDM database and synchronized in real time among all the participants of the workgroup using the product feed.
5.6. Tangible layers
To manage the visualization of the massive amount of data in EDM, we implemented a layering system with tangible interface. Our layers are EDM feed aggregators. We manage the activation/deactivation of layers in three tangible modes: (i) layer chip tokens, (ii) layer tabs, (iii) augmented badge.
Layer chip tokens look like gambling chips with a marker associated to a specific layer. By dropping a chip in the AR working area, the associated layer is visualized in AR (Figure 10). The layer can be switched off by simply flipping the chip (hiding the marker) on the table.
Layer Tabs, with small markers, can be embedded in technical drawings. By simply folding or unfolding tabs the user can hide/show the associated layers.
In the third modality, each user, instead of physical managing multiple tokens, can select a set of preferred layers. His/her company badge has a marker (augmented badge) and works as a token to activate the user’s layer preferences.
In last decade, digital data in engineering problems has grown in complexity and computer assisted tools demonstrated their limits: low usability for non CAD experts, low cooperation support, limited understanding of 3D geometries and low integration between real and virtual models.
We presented a novel approach in Engineering Data Management based on two key technologies: Augmented Reality and Tangible Interfaces. The first allows to add information directly on the physical model or on a physical contexts as technical drawings. The second provides a direct EDM data navigation and visualization using intuitive and collaborative metaphors such as rotating or moving a real object on a meeting table. We explored new tangible approaches using marker-based recognition. The users can access, navigate and annotate the 3D CAD model, can define structural stress\constraint, can manage layers and explore the simulation results in a highly collaborative workspace without the limit of mouse or keyboard. Tangible interfaces cannot replace completely the common GUI. Precise selection, text and numerical input, very common in EDM system are more efficient using a common GUI based (keyboard and mouse) interaction. To assist different industrial activities a flexible framework which can combine graphical (i.e. forms) and tangible interfaces is required. We developed a novel and efficient interaction paradigm with digital master for better perceive, understand and add contents to EDM knowledge.
We implemented and tested different industrial scenarios according to specific engineering activities: collaborative technical discussion, interactive FEM analysis and remote maintenance. The marker-based tracking allows a simple and low-cost integration in the industrial lifecycle process. The novel interface required to address new challenging issues. We presented our solutions to: real time simulation, tracking integration, content management, labels placement and EDM data filtering and access. We tested the system in a manufacturing enterprise with a highly configurable products. We supported product variability in three main phases: early design, marketing and assembly. The main benefits experienced have been the significant reduction of paper drawings and the increased speed of the revision phases. Limiting factors with current hardware systems are the marker-based tracking which may be not robust enough for manufacturing environment and displays issues (i.e. limited resolution and field of view). Emerging technologies like marker-less tracking, RFID, pico-projectors, may solve these issues in the next future.