Coordinated and Multiple Data Views in Augmented Reality Environment

1. Introduction

The advances in software and hardware technologies have enabled, in an increasing rate, electronic storage of great amounts of data in several formats and fields of knowledge, such as medicine, engineering, biology, financial market, and many others. A correct and fast analysis of these data is essential to guarantee the differential in competitive marketing or progress in investigative sciences.

Information visualization tools are one of the most used computational resources for a good and fast analysis of data and the associated relationships (Spence, 2001)(Chen, 1999). These tools provide users visual and interactive representations of the data (Card et al, 1999). Currently, the use of multiple views of the data is appreciated in information visualization, for it enables the creation of better-manageable visualizations of the data, i.e., less complex visualizations (Baldonado et al., 2005) (North & Shneiderman, 2000), and improves the perception of the user through diverse perspectives on the data.

It is important to remark that the great concern is the cognitive overload that the user may suffer when manipulating and analyzing the data by using multiple views. In order to reduce this problem, the user interactions that refer to data manipulation must be coordinated among all views, updating their visual representations coherently, improving the user’s perception of the data and facilitating the discovery of nontrivial relationships.

The usage of multiple coordinate views brings some current challenges, such as: development of easy interaction mechanisms for coordination, configuration and organization of layouts among views. One of the objectives of the Augmented Reality (AR) as a research area is to provide more natural and intuitive interfaces for the interaction with computational systems (Bimber & Raskar, 2005) (Azuma, 2001).

Moreover, AR enriches the real environment with virtual information. This allows the user to use objects or to collaborate with other people in the real environment while he or she simultaneously visualizes and interacts with virtual information. Finally, AR provides much more natural and ample environments to organize the information that will be visualized when compared to desktop environments. Thus, AR presents alternatives of solutions for the current challenges on multiple coordinate views applied to information visualization. This chapter presents a prototype that implements coordinated multiple views in information visualization for augmented reality environments. The applied information visualization technique was the 3D scatter plot for each data view, and a modified version of ARToolKit (Kato, 2005) has been used for the visualization of the augmented environment. The prototype was developed based on recommendations for a good information visualization tool (Shneiderman, 1996)(Carr, 1999) and multiple coordinated views (Baldonado et al, 2005), with coordinated characteristics of views, configuration, dynamic filters, selection and details on demand. Finally, this paper presents initial usability tests results after the application of some tasks proposed by (Pillat et al. 2005).

2. Related work

This section presents some tools that apply multiple coordinated views or augmented reality to different fields in information visualization.

(Maple et al. 2004) uses multiple coordinated views in three-dimensional virtual environments to assist navigation and orientation in these environments.

(Slay et al. 2001) uses augmented reality with the main objective of visualizing a graph information technique. The interface for configuration and generation of view is bidimensional.

(Meiguins et al. 2006) developed a prototype in augmented reality for visualization and interaction of data by using the 3D scatter plot technique.

3. Multiple views in information visualization

3.1. Information visualization (IV)

Information visualization is an area that studies transformation of abstract data into images that can be visualized and easily understood by human beings (Spence, 2001)(Chen, 1999). Information visualization tools are computational tools that implement data interaction and presentation mechanisms. The tools must offer the user a fast and easy manipulation and visual reorganization of the multidimensional data to assist tasks such as data query or analysis.

According to Carr’s work (Carr, 1999), a good visualization tool should present characteristics according to possible user tasks. Among them, some can be remarked: general view, zoom, filter and details on demand.

Systems of multiple views use two or more distinct views to assist the investigation process of a single conceptual entity (Baldonado & Kuchinsky, 2000).

In order to develop information visualization systems with multiple coordinated views, the most frequent recommendations are (Baldonado & Kuchinsky, 2000):

• When there is a diversity of attributes, models, user profiles, abstraction levels;

• When the different views point out correlations or disparities;

• When there is a need to reduce the complexity of the data set, by using simpler multiple views;

• Use multiple views minimally; justify the use of multiple views in relation to the cost for the user and visualization space.

(Pillat et al. 2005) stands out the main possibilities of coordination in multiple views:

• Selection: data items selected in a view are pointed out in other views;

• Filter: to reduce dataset for analysis in all views;

• Color, Transparency and Size: visual characteristics to represent the variation of values of attributes in all views;

• Sort: values of an attribute define the order of the visual representations of the data;

• Label: it determines what content the labels will present for each data item of the views;

• Manipulation of Attributes: it allows the user to add/remove attributes off the data views.

4. Augmented reality

Augmented reality is a system that supplements the real world with computer-generated virtual objects, which seem to coexist in the same space and present the following properties (Bimber, 2005)(Azuma et al., 2001):

• It combines real and virtual objects in real environment;

• It executes interactively in real time;

• It lines up real and virtual objects;

• It is applied to all senses of the user.

The augmented environment was based on the ARToolKit library (http://jerry.c-lab.de/jartoolkit), developed by the HIT Lab in C Language and distributed as open source, which allows the programmers to develop applications in AR (Kato et al., 2005).

ARToolKit uses computational view techniques for identification of predefined symbols inserted in the real scene. Once a symbol or a marker is identified, the virtual object is inserted in the real scene in the same position of the identified object. The final scene presented to the user is the visual combination of the real world with virtual objects.

The construction of the objects that are combined with the real world can be made through applications in OpenGL and VRML. There is also an ARToolKit version written in Java (JARToolKit) (Kato et al., 2005) with which JAVA3D can be used (Walsh & Gehringer, 2002).

5. Prototype

The prototype uses augmented reality to implement multiple views of 3D scatter plot technique in a coordinated way. The main points of its conception were:

• An environment of easy interaction;

• Work with several database types;

• Implement the 3D scatter plot technique;

• Develop mechanisms of dynamic filters in the augmented environment;

• Develop coordination mechanisms among data views, such as: selection, filters, details on demand;

• Develop auxiliary graphics, such as pie and bar, also coordinated with data views;

• Conception of software architecture that facilitates the inclusion of new information visualization techniques.

5.1. Architecture

ARToolKit has three basic modules: Scene Capturer, Augmented Reality (AR) and Augmented Image Generator (Kato et al., 2005). The Scene Capturer module is a set of video routines that captures input frames sent by webcam or any other video device. The Augmented Reality module is responsible for identifying the markers in the scene, tracking the captured markers and associating virtual objects with them. Finally, the Augmented Image Generator module is responsible for generating the augmented image (real scene and virtual objects), and is a set of graphical routines based on OpenGL and GLUT.

The modifications made in the ARToolKit in order to implement multiple coordinated views can be seen in Figure 1, and are concentrated in the AR module. The creation of several modules was taken into effect in order to help the maintainability, extensibility, efficiency and reutilization of the code (Figure 1).

Figure 1.

Summary of the prototype architecture

A brief description of the implemented modules is presented below:

1. Identification of Interaction Module identifies the type of interaction with markers performed by the user: insertion, occlusion or leaving the scene. It sends a message either to the control module in order to change the view data, or, when interaction is performed only visually and there is no modification in the visible data set, to the coordinated view module;

2. Control Module is responsible for managing the communication between the coordinated view module and the data module, providing transparency when these module exchange messages;

3. Data Module is responsible for data access in text files, XML or relational databases;

4. Coordinated View Module is responsible for managing what each data view must present, and thus assure coordination among all views;

5.2. Augmented interface

The augmented interface is formed by the 3D scatter plot view and other virtual objects, as well as the interaction controls based on markers and real objects. The prototype builds two data views by using the 3D scatter plot technique to represent the elements in a dataset. The main view configurations are axis X, Y and Z, 3D Shape, Color and Size. Figure 2 shows an example of the prototype during its execution, pointing out the simultaneous presence of real and virtual objects.

Figure 2.

Example of the prototype in executing mode

Most of the user interaction is directly performed by the occlusion of the markers. The occlusion-based interaction consists of blocking the capture of the marker’s symbol by the video device. This may be performed with his or her own hands. It is possible to apply transformations based on translations, rotations and scale in data views or other virtual objects in the scene just by moving or interacting with the markers. The markers are grouped according to their functionality (Figure 3), and can be freely manipulated in the real environment, enabling an infinite array of layouts to visualize the analyzed dataset.

An important characteristic of the prototype is its ability to set a fixed position to any generated virtual object in the scene, as a 3D scatter plot, just by occluding the object’s marker (Figure 4). The prototype stores the last register of the transformation matrix in order to place the virtual object in the fixed position in the scene. This is important because it avoids any unintentional interaction of the user with markers.

Figure 3.

Markers set according to functionality

Figure 4.

Virtual object fixed in the scene

5.3. Coordinated views

Some of the coordinative characteristics of the prototype should be remarked:

• Data: It uses a single dataset for all views;

• Layout flexibility: the user may analyze or query data with individual or simultaneous views (Figure 4);

• Coordination: It is classified as static, i.e., the coordination between pairs of views is predefined. It may be either strongly coordinated, as color (Figure 2) – once defined, the same color is applied to all views, or loosely coordinated, as semantic zoom or Axis values, which can be used in any data view, but have to be manually configured for each view in the augmented interface.

The coordinated actions are:

• Strongly Coordinated Actions: filters, environment configuration for color, shape and size attributes, and for selection of objects. They affect directly all views, even if they are not present in the scene (unreachable by the video device);

• Loosely Coordinated Actions: axis configuration, semantic zoom (Figure 6) and navigation (translation and rotation). They only affect a view which is present in the scene

5.4. Filters

Concepts of dynamic queries have been applied (Shneiderman, 1994) for categorical and continuous values. This type of action allows the user to check databases without needing to use command lines, manipulating only graphic components of interface (Figure 5 and 6).

Figure 5.

Augmented representation of the categorical attribute filter

In the prototype, any filter can perform the following actions:

• Hiding: Take off the scene a determined item or data items which have a previously selected characteristic;

• Isolating: Leave only items which have a previously selected characteristic in the scene;

• Restoring: Undo the filtering processes previously performed on data items.

• The categorical attribute filters work on configurable characteristics of a data view, such as: color, shape and size. Figure 5 illustrates the filter control for categorical attributes.

• The continuous attribute filters specify ranges of values to isolate and hide data items from views (Figure 6).

Figure 6.

Augmented representation of the continuous attribute filter.

5.5. Semantic zoom and auxiliary chart

Semantic Zoom allows the user to visualize the data space more precisely and with additional details as his or her perspective gets closer to the virtual objects (Figure 7). The zoom marker has two faces, one to zoom in and the other to zoom out.

5.6. Environment configuration

The prototype allows the user, though an environment configuration control, to freely change the attributes of X, Y and Z Axes and of Shape, Color and Size for each data view. Figure 8 shows a situation in which the user changes the color and shape configuration to another categorical attribute, and the changes are presented in the multiple coordinated views.

5.7. Details on demand and help

A resource called “Virtual Pointer” has been developed to select virtual objects and analyze their hidden information. The selected objects in one view are pointed out in the other view (Figure 9).

This item helps to detail the activities of each marker, in case there is a doubt about its functionality. Its use is simple: while the help marker is visible, all groups of markers present in the scene provide information related to its use mode.

Figure 7.

Example of Semantic Zoom

Figure 8.

Changing the color and shape attributes in views

Figure 9.

Selection of items performed in a strongly coordinated way

The prototype presents auxiliary bidimensional pie and bar graphics that provide additional information on the visualized data (Figure 10).

Figure 10.

Example of auxiliary charts (Pie and Bar)

6. Usability tests

Usability tests were performed to evaluate the use of information visualization techniques in an augmented reality environment with multiple coordinated views. The users were asked to perform a set of tasks previously defined by (Pillat et al. 2005) that demand different actions such as: view configuration, data correlation, and range specification, among others.

1. Task 1: Are the 4-cylinder Japanese cars usually lighter than the 6-cylinder American cars?

2. Answer 1: No.

3. Task 2: Analyze the data and describe the main characteristics of the American cars;

4. Answer 2: Acceleration is between 8 and 22.2, mainly between 11 and 19. Most of the cars have 8 cylinders. Weight is uniformly distributed. MPG values are also uniformly distributed. Horsepower is concentrated between 88 and 155.

5. Answer 3: Acceleration between 12.2 and 24.8, and light weight. The horsepower kept stable until 1977 when it rose just to reduce again the next year. There are few 5 or 6-cylinder cars but most of them are 4-cylinder. MPG was between 10 and 31 from 1970 to 1976 and considerably rose since them.

The used dataset contains information about American, Japanese and European cars from 1970 to 1982. (Pillat et al. 2005) There are 8 attributes: 3 categorical and 5 continuous.

After a 20-minute training in augmented environment and interactive markers, task one was used as a practical example in order to build confidence and improve the users skills. The comparative tests were restricted to tasks 2 and 3 that are similar but with increasing level of difficulty. The tests involved 5 users, all 21-32 male with good computer skills. None of the users had previously interacted with augmented reality environments with markers. All users had previous knowledge on information visualization techniques.

Each item of user answers was analyzed. For example, for task 2 what was the answer to attribute1, attribute2, and so on. The accuracy rate is based on the total number of correct answers to each item of each task. Figure 11 and Figure 12 present the results in terms of accuracy rate and task execution time, respectively.

Figure 11.

Accuracy rate in user answers

The accuracy rate plot indicates that 80% of the users had a better or similar accuracy for task 3 that is considered complex (Pillat et al. 2005). All the users spent less time performing task 3 than task 2. So even with a reduced number of tests it is possible to infer that once the user has experience and confidence in the environment he tends to achieve precise and prompt results.

Figure 12.

Task Execution Times

7. Final remarks

This chapter presented a prototype that implements multiple coordinate views in augmented reality environments. The main supported views are based on the 3D scatter plot technique. The augmented interface provides coordinated actions control among views, such as dynamic filters for continuous and categorical attributes; details on demand (object selection); environment configuration; auxiliary pie and bar graphics; semantic zoom; and free navigation.

Initial usability tests were performed in order to evaluate the proposed approach to information visualization. During the execution of the tests it was possible to observe the efficient use of the developed controls and the coordinated views to solve the assigned tasks. Usability tests also revealed the high adaptability of the users to the augmented environment. Only one user had major problems performing task one, taking 11 minutes to adapt to the environment and to the marker-based interaction. Other pointed out difficulties were related to the excessive use of markers and video capture or identification problems that made the interface unstable. The following remarks were made by the users on the interview following the tests.

• A first experience in AR environments: users highlighted the main advantages, such as the easy adaptation and learning, the more immersive, free and sometimes fun environment, the freedom to move and manipulate virtual and real objects simultaneously and a larger workspace. As disadvantages the users pointed out the lack of precision of some movements because of marker detection failures; the excessive need to repeat interactions; the excessive use of markers.

• Use of information visualization techniques in AR: the users pointed out the freedom to manipulate data views and the free workspace to work with virtual and real objects, and the collaborative aspect as the main advantages. The main disadvantage was the need for more appropriate equipment like augmented reality glasses that would, according to the users, significantly improve precision and performance.

• The use of a multiple views coordinated environment in AR: the configuration of graphics axis on views and the different information perspectives allowed better and faster data comparisons and analysis.