InTech uses cookies to offer you the best online experience. By continuing to use our site, you agree to our Privacy Policy.

Computer and Information Science » Human-Computer Interaction » "Human-Computer Interaction", book edited by Inaki Maurtua, ISBN 978-953-307-022-3, Published: December 1, 2009 under CC BY-NC-SA 3.0 license. © The Author(s).

Chapter 17

Understanding Group Work in Virtual Environments: Performance, Creativity, and Presence

By Ilona Heldal and Lars Brathe
DOI: 10.5772/7729

Article top

Understanding Group Work in VirtualEnvironments: Performance, Creativity, and Presence

Ilona Heldal1 and Lars Bråthe2

1. Introduction

Virtual environments (VEs) have the following dimensions: content, geometry and dynamics. Experiences in VEs refer largely to the sensory experience produced by a computer technology, which often is a ‘sense of physical reality... a construction from the symbolic, geometric, and dynamic [computer generated] information directly presented to our senses’ Ellis 1995, p. 16. In a VE the following aspects are also important: user-involvement, direct interactivity, the possibility to elaborate a sense of being in the three-dimensional computer-generated space, and to receive direct responses from the environment so that the participants experience a control on their actions.

By visualizing three-dimensional spaces and objects, computers can support collaboration. People at the same place or at distributed locations can share similar images and use them for work or leisure. The computer systems that allow these visualization possibilities are termed collaborative virtual environments (CVEs). By providing services such as distance conferencing, shared industrial applications, distance learning, maintaining online groups, and games, CVEs offer several advantages: to overcome travel problems, to avoid ambiguities during the different phases of the collaboration by visualizing the same processes, and being dynamically changeable, which can be time- and cost-effective in comparison with using real models. By visualizing a model around the users at actual scale, it is possible to experience buildings, ships, airplanes, or roads before they are built in order to explore them and select the most suitable alternatives. Other benefits are e.g. to visualize safety-critical situations, to aid rehabilitation and to train when learning, and in general for scenarios that may be hard to explore in reality. Firemen can train by using VEs to rescue, patients can do their rehabilitation exercises or overcome their fears, pilots can train to fly airplanes, and so on. These systems also bring together people with the same interests, who are distributed over space, to interact remotely.

The overall objective of this chapter is to identify issues that contribute to creative, effective and enjoyable group collaboration in CVEs. For this purpose we review theories on supporting performance, creativity, and presence. We examine studies on the contribution of performance, creativity and presence in the use of computer mediated collaboration (see Section 4), and we also use some previous research on creativity support in CVEs Roberts et al., 2007, Heldal et al., 2007. Many of the already identified supporting factors depend on the quality of how technologies allow seamless interaction and how they support social aspects. This study extends the previous work and exemplifies that a more usable and intuitive CVE does not necessarily support group work better, and shows that the notion of presence is not necessarily unambiguous for determining high quality work in CVEs.

The goal is to find approaches supporting collaborative experiences. By reviewing research on differences between individual work and group work and how it is influenced by presence, creativity and performance, the findings will contribute to better understanding of networked collaboration.

2. Motivation and structure

To produce better systems in general, the benefits of defining suitable evaluation methods and guidelines have already been shown. Research has defined methods and identified many different concepts and factors that contribute to developing advanced VEs. Such factors are presence, creativity, performance, enjoyment, interaction, immersiveness, input and output devices etc. However, the factors that contribute to efficient and enjoyable work for single users are not necessarily the same or equivalent in impact on group collaboration.

CVE technologies are complex, diverse, and in many cases without established standards and with ‘homemade’ adjustments. Their use and impact are increasing, since they offer a number of benefits. Allowing users to interact with each other – or each other’s representations – and the same or similar computer-generated graphical images, to see what the others are doing, and to communicate with each other also through visual and auditory channels are clear benefits. A user can write, describe by voice, or show to her remote partners what she means, and whether or how she follows the group activities. Thus, the interaction can be multifaceted. The amount of technical and social factors influencing the collaborative results just blossoms out. To answer the questions ‘Under which conditions does collaboration have added value over individual work when using VE technologies?’, ‘Does a technology supporting creative individuals supports creativity for groups?’, and ‘What is the relation between the most important factors influencing work in CVEs that have the greatest impact on the outcome?’ is the broad motivation behind this study.

A factor that influences an individual in one way does not necessarily influence another in the same way, even if the members of the group use similar technologies. For example, a person can experience high presence, while another finds it low in the same virtual room. If these two people work together, both the ability to experience higher presence for the first person and the collaborative outcome can be lowered, e.g. by unnecessary discussions, technical interruptions, or social misunderstanding Heldal et al., 2006. For a superior outcome it is not necessarily enough to allow advanced technologies and provide successful networking. How the individuals handle these, how the group is organized, what the main technical characteristics and the social differences are, also have to be taken into account.

Object-focused collaboration is common in creative tasks and is well researched in CVEs, and typically involves problem-solving Wolff et al., 2006. We use object-focused interaction as the basis for discussion of group work in CVEs.

The paper is structured as follows. Section 3 briefly presents single-user work versus collaboration. In Section 4, performance, creativity, presence and other factors and concepts are examined that are considered to have a great impact on evaluations and outcomes. Section 5 briefly examines relations between these factors, with focus on presence and creativity. Section 6 includes a pragmatic discussion about treating performance as a main goal for examining collaborative work. This is followed by Section 7 with the findings and discussion, and suggestions for future directions. The last section, Section 8, presents the conclusions.

To acquire an overall view of performance for collaboration when people use CVEs, one must examine the following, according to HeldalHeldal 2004:

  1. 1)how people can work, with special focus on problem-solving,

  2. 2)how technologies can support this,

  3. 3)what they experience.

Thus, performance in CVEs is described in terms of 1), 2) and 3). We use these types of activities in order to examine creativity and presence contributing to group experiences and outcome for object-focused work.

Since many studies, especially those investigating collaborative creativity, are for networked computer technologies in general, even though the focus is on collaborative work in CVEs, we do not limit this study to examining exclusively VEs, but also consider research from computer-mediated communication (CMC) or computer-supported collaborative work (CSCW). Some relations, e.g. between creativity and group creativity or social creativity, are exploited more for general communication technologies.

3. Single-user work versus collaboration

3.1. Understanding computer-mediated collaborative outcomes

Collaboration is not always easy, yet it is often a necessity in order to reach one’s goals in society, organizations, or relations between individuals. In trying to collaborate efficiently, one can experience difficulties caused by, for example, distance, differences regarding motivation, language, culture, lack of agreement upon a set of definitions, or different mental models. To avoid these difficulties, more support for collaboration and better suited equipment is needed.

To find out how a group can be more than the sum of individuals has preoccupied a number of researchers for several decades. By considering various technologies that support work, many old questions become actual again. In general, it is difficult to explore the benefits of group work in terms of individual contribution, task and context, and to obtain measurable and consistent results through several projects. Research, though, has acknowledged that individuals contribute with at most 70% of their optimal individual performance for a well-working collaboration. The results often depend on task, time, group, chosen evaluation method or applied theories Brown 2000. Brown’s research focused on face-to-face collaboration. For the CMC, the CSCW and the CVE studies too, most evaluations treat face-to-face collaboration in real environments as a reference for obtaining superior communication and collaboration results. Even if computer-mediated work can have unique advantages prior to work in physical environments Walther 1996, it is worth considering the potential for working with physical objects rather than technology-mediated work, where applicable, since naturalness still has unique value Wolff et al., 2006.

Among the earliest, probably most promising distributed applications are those that support meetings and conferencing. For these, ScottScott 1999 summarized several group collaboration models in an input, process and output framework that also includes task characteristics for the input, grouped according to McGrath’s decision typology: create a plan, choose, negotiate, and execute McGrath 1884. Neal et al. also argue for the importance of treating, in CSCW evaluation methods, the second-level social system effects such as coupling of work, joint awareness and coordination, which can be more easily approximated by laboratory studies Neale et al. 2004. Knowing more about normative behaviors and main tendencies can yield benefits for field-work evaluations. Understanding such behaviors is important since including social interaction in the evaluation of collaborative work is crucial.

Much applied research on using VEs concerns handling prototypes and models. Group work for these applications, too, can be supported by knowing more about social interaction Heldal et al. 2006. The establishment of common grounds, conventions, awareness, trust, naturalness and human proximity (Hinds et al. 2002), time and distance Poltrock et al. 2003, and knowing the prerequisites for single users or good collaboration Gutwin et al. 1998 are important roles for these applications as well.

3.2. Towards collaboration in CVEs

Several studies have identified some factors such as presence, performance, intuitiveness, interaction, and leadership as important for VEs Tromp 2001, Slater et al. 2000, Schroeder et al. 2006). Certain of these may be more closely associated with one specific application or type of VE than with another. For example, desktop systems can be more effective than immersive environments for problem-solving when visualizing large molecules, while immersive systems can be more effective for visual modeling of object-focused collaboration with a few objects Heldal 2004.

There are also differences when varying some technical factors of an environment. For example, varying latency, field of view, different rendering usage, interaction styles, or varying perspectives for navigation can result in different performance and presence measures Tromp 2001, Steed et al. 2003, Polys et al. 2004, Steed et al. 2005, Schroeder et al. 2006. For CVEs there are still many problems remaining, for example, the influence of technical interaction, chosen interaction style, the flow of interaction, devices and user interfaces, communication modalities and considering the influence of social interaction (Wilson, 2003). The social interaction and the technical interaction often take place in parallel, or they are interconnected in a non-deterministic way due to the nature of social interaction. Possibly because of the novelty and complexity of the technology, or because it supports applications from several different areas, there are only a few works on approaching e.g. overall outcome or overall usability for CVEs Schroeder et al. 2006) To approach collaboration in CVEs that can be focused or unfocused, Tromp 2001 has identified three main stages of the collaboration that are embedded in a ‘meta-collaboration’ context. She divided the temporal structure of the interactions into the beginning of the interaction, proper collaboration, and ending the collaboration Tromp et al. 2003.

Understanding the task, problem-solving in relation to available or spent time in the environments, and choosing the right strategies are important for increased performance and presence Schroeder et al. 2006. This can mean quicker plan creation, less time spent for negotiation, and quicker decision-making McGrath 1984.

3.3. Towards creative collaboration

Creativity is helped by an uncluttered state of consciousness but while creative tools promote this through intuitive and natural interfaces, this naturalness of interaction does not yet extend to distance collaboration. For group creativity to be considered collaborative it must have some common focus Roberts et al. 2007. Allowing creativity is considered to augment the quality of work Fencott 1999, Waterworth et al. 2001.

Numerous studies associate creativity with the multiple dimensions of experienced flow defined by Csíkszentmihályi. These are Csíkszentmihályi, 1996:

  • 1. Clear goals – this means having clearly defined objectives and, during the whole work, immediate feedback so that one knows instantly how well one is doing.

  • 2. To immediately understand one's perceived ability to act. This also means that the task should be suitable to individual skills.

  • 3. Action and awareness should merge.

  • 4. Supporting concentration on the task. This also means that irrelevant stimuli must disappear from consciousness; one should not worry or be concerned about irrelevant issues.

  • 5. The individual should feel a sense of potential control.

  • 6. To feel, however, a loss of self-consciousness, and a sense of growth and of ‘being part of some greater entity’.

  • 7. Altered sense of time. Time should be experienced to pass faster.

  • 8. The experience should become autotelic, worth doing for its own sake.

Experiencing creativity as individuals may differ from experiencing group creativity. It is debated whether individual creativity can contribute to group creativity at all Fischer et al. 2005. Moreover, by using technologies one adds a further level of possible disturbances: experiencing creativity for groups while using technologies Shneiderman et al. 2007.

Aiming to obtain creative work in CVEs thus raises a lot of questions. How do the application, technology and social factors such as ‘joint thinking, passionate conversations, and shared struggles among different people‘ Fischer et al. 2005, p. 4 impact upon this? And how is this connected with supporting presence, collaborative presence and social presence?

Handling innovations in a company may also differ from focusing on the need for supporting creativity from the beginning. The starting points for innovation work usually consider identification of the users’ and customers’ needs, identification of a new market possibility (the work forward to successful business plans), considering new technological possibilities, or simply the conviction of an individual or small group that it is possible to do something in a completely new or simpler way. This often refers to efficiency in the first instance. Thus, the starting point for creative work might be quite different and takes more time and often, unfortunately, is more diffuse. Hence, the composition of the group may also differ; e.g. the members might have been working close together before, or they might be new to each other and coming from different cultures. Thus, they may need different support to perform tasks in a creative way.

4. Factors influencing work

4.1. Performance

One of the major factors that influence performance is the perceived and actual interaction in these technical environments. Interaction can be defined by the following components: affective and emotional, cognitive or intellectual, non-verbal, content or message, and social or personal network segments McLeish et al. 1973. It is almost impossible to divide and define exactly the different components of the interaction. They are embedded in the content and in the network of relationships. However, if the aim is to improve collaboration in a technical environment, we need to separate these, since both the origins, and the effects of social interaction and interaction mediated via technologies are different. Thus, a separation – at least in these two main categories – contributes to better understanding of issues supporting or disturbing work.

There are differences in overall performance, depending on the time the users have spent in the environment and on aspects of their expertise Nielsen 1993, Nilsson et al. 2002. The design of a new system does not incorporate either how the variation in the surrounding physical environments will influence the use, or how the social interaction will do so via the technology. Research has demonstrated that time does have effects on the use of media for groups. Ad hoc groups must sometimes handle decisions in organizations through a medium, but this can have negative consequences on their agreements. The long-term use of a certain medium supporting group collaboration has the result that media effects may disappear Walther 2002.

Sonnenwald showed statistically significant differences with respect to performance and adoption, in the sense that the users developed work-around to cope with the perceived disadvantages of collaborating remotely with scientific systems Sonnenwald, et al. 2002. It is well known that people, if they can do so, use systems in different ways Caroll 1995. Disruptive technologies, and the way that they allow seamful interaction may also support experiences Höök et al. 2008. Webster et al.Webster et al. 2001 show that people interact differently in different social contexts. We can also mention here that differences in the attitude towards use exist between members of different national groups Vöhringer-Kuhnt 2002.

Lampton et al.Lampton et al. 2002 pointed to the value of considering proper performance measurements in evaluating VEs. This should support meaningful comparisons and contribute to cost-effectiveness and safety, among many other reasons. They based their view on a list of problems regarding measurements in evaluations from the American National Standards Institute American National Standards Institute 1993. Some of the problems were a lack of general theories to guide human performance measurement, difficulties in handling the inverse relationship between operational control and realism, the multiple dimension of behavior, how to measure cognitive tasks etc. Based on ten categories defined by the same institute (time, accuracy, amount achieved, frequency of occurrence, behavior categorization, consumption, workload, probability, space/distance, errors), they discussed performance measurements for VEs Lampton et al. 2002.

Bystrom and BarfieldBystrom and Barfield 1999 have previously analyzed collaborative task performance in a VE. They found that task performance is affected by the presence of others and by the level of control, even though the sense of presence was not correlated with the collaborative experience. They suggest that experiences in VEs should be grouped according to three factors: presence, the quality of VEs, and task difficulty. Another study, by Slater et al., also argued that efficient performance could be a consequence of the VE experience. Subjects in a more realistic environment performed better Slater et al. 2001, Wolff et al. 2006). Time and efficient workflow also influence effectiveness Heldal 2004.

Social studies treat technology-mediated collaboration in such a way that the technical factor is often implicitly embedded in the process of social interaction. Yet the technology often matters through its presence, or when a problem occurs during its use. By developing technology that supports mediated collaboration, the nuances of social interaction are not considered. But at a certain stage – and VEs have passed this stage – the technology is mature enough to be evaluated for groups and improved to provide what the users need for collaboration.

Putting the arguments together, these all means that one can not possibly examine and understand interaction (and performance) in VEs without separating and examining social interaction and interaction via technologies. However, how one factor influences the other and the overall experiences during the time of use should also be considered.

4.2. Creativity

As we described previously, creativity has been linked to a state of mind known as flow, considered a precondition for creative experience in numerous works. Flow is defined by CsikszentmihályiCsikszentmihályi 1996 via eight distinct dimensions. Some of these dimensions, however are clear social interactions that can be hard to implement in technologies. Table 1 examines how the certain dimensions can be treated for technology mediated interaction. On the basis of important issues influencing individual, and technology-supported group creativity, we will examine how these issues are considered in different research studies.

As pointed out earlier, to examine creativity (and later presence) for performance in general, we focus on creativity (and later presence) support for problem-solving, experiences, and technical support. Therefore, we shall briefly overview relevant research on: creative problem-solving, creative experiences, and creativity-supporting tools. Many of the studies examining creativity for these three areas originate in considering the dimensions of flow, described above. Table 2 summarizes these and present studies offering possibilities for support it.

Vass et al. defined a framework to support creativity especially for problem-solving based on the dimensions defined for flow. The authors emphasized the importance of considering an appropriate balance between challenges and skills, and of the immediate feedback and the clarity of goals and problem-solving to the sixth dimension, which also means ‘No worry of failure’. They also emphasized the value of differentiating time-dependent relations for their workflow model Vass et al. 2002.

DimensionsChallenges for technologiesChallenges for g roup s using technologies
Main success f actors – important for social interaction (SI) :Main success f actors – important for interaction via technical devices (IT) :
clear objectivesW ell - defined TAO ften assumed to be already knownU sability T, TA, IT.
immediate feedbackUsable, intuitive T and clear TAMissing feedback (SI) often cause s confusionDepends on how ‘ transparent ’ a T is , IT
skills suited to challengesI, T, TAEasiest for homogeneous GH omogeneous use , symmetrical T , usable settings , IT
action and awareness mergeI, T, TASize G, transparent T, and allowing I and SG communicationC ompletely intuitive T. I and SG communication should also be supported
allowing high concentra - tionI, the usability and intuitiveness of T Depends on the size of G, SI. Secondly on how T becomes transparent and how it supports I and SGThe T should be completely intuitive to support this
sense of controlI, TA and the usability of TConcerns what to do next, comfort with ‘ the social group ’ , I, SG, G, SI, IT , T Easy to use T, ‘ enough ’ challenging TA, IT
loss of self-conscious - nessIntuitive T, IDepends on I, the size of the group G and how I and SG are supported. TA, TUsable and intuitive T, IT
altered sense of timeMainly I, TAI, SG, G, SI – if T and IT workCompletely intuitive T and IT
autotelic experienceMainly I, but also T and TAI, SI – but also G, SGCompletely intuitive T and IT

Table 1.

From individual creativity to technology-supported group creativity. The second column shows main challenges for technology supporting the corresponding dimension for single users. The third and fourth columns are for collaboration in terms of social interaction (SI) and interaction via technical devices (IT). The success factors in terms of individual abilities (I), the capability of technologies (T), tasks (TA), groups (G) and subgroups (SG).

However, the model defined by Vass et al.Vass et al. 2002 cannot necessarily be extended for group work, since it does not consider enhanced experiences for problem-solving and different dimensions for social aspects. This work is important since it explains more granulated influential dimensions of problem-solving and relates to a generic framework aimed to support human needs by using technologies defined by ShneidermanShneiderman 2002. By considering the dimensions identified for flow, this work suggests certain tasks that have to be considered for designing tools that support creativity: (1) Searching and browsing, (2) Visualizing relationships, (3) Intellectual and emotional support, (4) Allowing free associations, (5) Exploring solutions, (6) Composing artefacts and performances, (7) Reviewing and replaying sessions, (8) Disseminating results Shneiderman 2002. This framework gives a greater understanding of the connection between specified functionalities needed for creativity Shneiderman 2007. In general, to understand how certain social aspects can be mirrored and implemented in technologies is important in order to develop better creativity-supporting tools.

What user(s) doMainly Internal vs. external A ctivitiesSuggestions for supporting individual creativitySuggestions for supporting group collaboration
Solve problemsInternalBy considering time-dependent relations, workflow ( e.g. Vass et al. 2002) and focus of attention (Heldal, 2004) .Support for integrating individual work in group work (e.g. Fischer et al. 2005) . Managing to handle easier interpretations and transparency of others’ activities (Heldal et al. 2006) , and allowing personal space.
ExperienceInternal – high-tech might attractImplement environments prepared to handle experiences, (e.g. Fencott 1999) , and allow seamless interaction (e.g. Wolff et al. 2005) .Common targets, clear objectives, feedbacks, rewards. Separate activities where group awareness stimulates group members.
Handle technologiesExternal – hard to understand sometimes Define tasks that should be considered for designing creativity-supporting tools, (e.g. Shneiderman 2002) .Symmetry helps. Otherwise make the group aware of each other’s possibilities (Tromp et al. 2003) .

Table 2.

Examples of studies enhancing how to support creativity for a single user, or for a group.

Creativity-supporting tools in distributed scientific communities need to support flexibility in granularity of planning Farooq et al. 2005. Roberts later demonstrated characteristics of immersive collaborative environments that can easily provide a seamless workflow through transitions between content and detail in planning Roberts et al. 2007. Experiencing flow can be impacted by the way in which people handle different events, and by how they build a conceptual map in VEs. According to Fencott, to support creativity, knowledge about social and technical contexts should be considered already in designing VEs, to support VE users in handling sureties, surprises, and shocks Fencott 1999.

Another way to understand creativity-supporting tools is the one that examines the origins and context of creative activities. A great pioneer in this work is Fisher with the work done by his group(s). Fischer differentiates between two levels of creativity, viz. historical creativity associated with fundamentally novel ideas and discoveries, and psychological creativity associated with ideas and discovery from everyday work practice. Accordingly, historical creativity can be more easily associated with individual work, while psychological creativity incorporates prerequisites from groups which also can offer prerequisites that support social creativity. Beside functionality, it has been necessary to consider factors, such as cultural diversity, the context of the experiment, individual versus group support, allowing reflection on minority conflicts, and supporting flexibility in granularity of planning. Fischer states that individual creativity is usually integrated in social creativity. The social structure and mindset contribute better-formulated problem areas and stable environments. The collaboration is examined through several applications, e.g. Creation (collaborative drawing art) and Linux; it often has non-simultaneous characteristics, where the contributors take turns at work. During his or her turn, one can

4.3. Presence

Short, William and Christie introduced measurements of the quality of distributed work in 1976, such as social presence as the users’ subjective sense of being present in technology in a social setting with another person Short et al. 1976. Later, presence was identified as a main factor that contributes to an improved experience for VEs. It refers to experiencing being in a place other than where one is physically present, which is possible by using computer graphics. If this sense refers to being there together with one’s partner, then we often speak of copresence Slater et al. 2000.

There are many studies that try to define relations between presence and other factors that are representative for VEs. The level of presence has often been related to task performance Witmer et al. 1998, Slater et al. 1994. At first Slater et al. defined external and internal determinants of the sense of presence. The external determinants include display quality, consistency of presentation across displays, ability to interact, the anthropomorphism of the user representation, and the clarity of causal relationships between user actions and reactions. The internal ones are the representation system and perceptual position. These were defined with the help of neurolinguistic programming (NLP) and three key representations: the visual, the auditory, and the kinesthetic. They examined the effects of various display modalities, interaction techniques, and system algorithms on the reported level of presence. Presence is measured in several ways, by objective and subjective measurements, using questionnaires or external observations Slater 2003, Heldal 2004. Usually there is a measurement type that fits and is used in more specific areas to be studied. Later, Maya Garau summarized the determinants that are recognized by different researchers. These are: (1) the extent of fidelity and sensory information, (2) the match between sensors and displays, (3) content, and (4) user characteristics (Garau, 2003).

During collaboration an important aspect is that the partners should be aware of each other and each other’s presence. Therefore, an important question here is how awareness relates to presence and to the collaborative experience. Intuitively, it seems that there should be a correlation between presence and copresence, and such that more immersive VEs should provide an enhanced sense of copresence. However, in CVEs this is not necessarily the case. Higher presence does not necessarily result in higher copresence. The collaborative experience of ‘being there together’ has to do with the real-time information on the others – who the others are, how they are represented and what they are doing. Except for the co-located environments, this information can be transmitted by the different technologies, i.e. it can be externalized Polanyi 1966. The influence on experiences, though, can differ from one technology to another.

What user(s) doMainly Internal vs. external ActivitiesSuggestions for supporting individual presenceSuggestions for supporting group collaboration
Solve problemsInternal – although an external observer can see it.Fidelity and sensory information, match between sensors and displays (Garau 2003) . Clear interaction, focus of attention (Heldal 2004).Support for integrating individual work in group work, e.g. creativity (Fischer et al. 2005). Manage to handle easier interpretations, have awareness transparency of the others’ activities (Heldal et al. 2006), seamless communication, show intentions and emotions (Slater et al. 2000) . S upport decision - making etc.
Handle technologiesExternal – Disturbances Internal – Intuitive TechnologiesScreen size, immersiveness, less breaks in presence (Brogni et al. 2003) Seamless interaction (Shneiderman 2002; Heldal 2004).Consider the dimensions defined by Garau (2003). Symmetry helps (Heldal 2004); otherwise make the group aware of each other’s possibilities. Clear object laws (Roberts et al. 2007).
ExperienceBoth internal and external – Immersive technology, design may benefitT racking, real time (Waterworth et al. 2001). Implement challenges (Heldal 2004), seamless interaction (Wolff et al. 2006); for certain applications photorealism helps (Heldal 2004).To support group awareness, quick feedback, common objectives, allowing rewards, problem-free communication, using rich technologies that transmit human cues, important movements, naturalness (Heldal et al. 2006). Consider cultural differences, leadership, emotion, etc.
Table 3. Possibilities to support presence for a single user or for a group .

Table 3.

Possibilities to support presence for a single user or for a group

Another potential explanation of eventual trade-off between potential presence and copresence is that it may be difficult to sustain high sense of presence and also high sense of copresence over time. In either case, it seems that the situations of trade-offs between presence and copresence can be explained by focus of attention: in certain situations it is not possible to focus on both the space and the other at the same time. It can be hard to explain true relationships since presence and copresence are often evaluated after the task performance. People sometimes look after each other, point to the other person, or ask questions in the environment, without necessarily having such intentions to strengthen copresence. If the presence is evaluated by questionnaires afterwards, they need to remember these intentions. Sometimes they intuitively use the technology, or use it wrongly, e.g. point with non-tracked hands in immersive environments, in such a way that the other person does not necessarily observe these activities. If, for example, one points unintentionally with the wrong hand, this does not disturb one’s presence. The copresence may be disturbed since the activity is not transmitted, but none of the subjects can possibly report it afterwards. Table 3 lists the possibilities to support presence in relation to what people or group do in VEs.

While collaborative and social presence does make a great contribution to user experiences, the way in which this influences overall performance, usability, and particularly effectiveness is not necessarily known Schroeder et al. 2006. As shown in Table 3, the relation between presence and overall outcome is hard to know. Presence for certain tasks and for certain settings is valuable, but to implement it in technologies is difficult; this would need several individually adjustable features for each person and task for each time.

4.4. Other factors influencing the workflow in CVEs

Time: People’s behavior, including the way in which they interact socially and with objects, changes over time when immersed in a virtual environment Tromp et al. 2003. At the beginning and end people focus on social interaction, while interaction via technical interfaces and the virtual representations plays a more important role for task-focused collaboration. Considering flow in the wider context, we can postulate that time is likely to impact on many of the previously defined dimensions. Time is a dimension of interaction with the environment – for example, immediate feedback and control. It is also a dimension of the individuals themselves: for example, it takes time for action and awareness to merge, and to lose the sense of time Heldal et al. 2007. As we pointed out earlier, research has demonstrated that time does have effects on the use of media for groups. Ad hoc groups must sometimes handle decisions in organizations through a medium, but this can have negative consequences on their agreements. The long-term use of certain media supporting group collaboration has the result that media effects may disappear (Hinds 2002, Heldal 2004. Embodiment: To consider social issues when designing new technologies and new applications is relatively new, especially in the context of developing networked VE technologies, but has a great impact on the general outcomes. This work did not treat, for instance, embodiment – even if embodiment and using avatars has a great impact on social issues influencing collaboration flow, for example by reflecting the orientation of the user within the environment and in some cases gestures that can be directed towards the focus of interest (Heldal et al. 2006). Emotions: Another example that should be studied, especially in connection with creativity and presence and the collaborative outcome, is emotion. Experiencing emotions is an important social resource in managing creative collaborations Slater 1999.

Furthermore, there are several facilitators that may contribute to enjoyable and effective communication in VEs. Some examples are: transparent intention, visualizing and knowing about attention, motivation, awareness Roberts et al. 2007, praise, recognition and rewards Roberts 2009. Common ground, team spirit and a sense of belonging are needed in order to build trust and consensus Culnan 1987 – and these qualities are hard to support through communication media. DourishDourish 2001 explored the meaning behind possible interactions in context, by exploring common trends from tangible and social computing. He argues for considering technologies as ‘embedded’ in their context of use, in order to support meaningful interaction.

5. Relation between the factors influencing work

There is a huge difference when people solve problems alone instead of together. If the individual can contribute less than 70% of his capability for the group outcome (as we presented before, stated by BrownBrown 2000), it is important to know whether this contribution is best for the group in terms of creativity and performance. Problem-solving requires mental work which is to a great extent individual Brown 2000. The fact that individual problem-solving also requires high presence is obvious. Following Fischer’s argumentation for supporting creativity for individuals and for groups, we note the need of private space for the individual to make it possible to give her best in collaboration (Fischer et al. 2005).

Collaborative problem-solving, as we showed in the previous sections, requires intuitive technology. Via this, the group can interact more easily and support peripheral collaboration that also requires social presence. Studies show that symmetrical settings help Heldal 2004 and technology can be neglected and used more intuitively Roberts et al. 2007, however, the group still needs to cope with differences between the group members Schroeder et al. 2006. Interruptions caused by nonintuitive devices, bad design, or needless social interaction can disturb a member, as well as a group (Hinds et al 2002). On the other hand, as Höök Höök 2008 presented, sometimes disruptive technologies are required for well being. For these cases, however, the interests of individuals (their social need to be aware of things and happenings) steers the requirements. People need fresh air, food, motion and awareness of their social relationships in order to be well-functioning. See Table 4 for examining requirements for individuals and group in terms of presence and creativity for superior results.

Social behavior in a group can support peripheral communication, strengthening group awareness of the members and vice versa. High social presence is required for maintaining peripheral awareness in networked group activities, allowing coordination, supporting decision-making processes, negotiations and choosing strategies. This can be better supported by using different communication modalities, transmitting the right social cues. To have high social presence is important, for example, for applications exploring museums, cities and collaborative games Stones 2002. Probably high social presence is required to attain superb experiences. High social presence in turn can allow increased social creativity.

What user(s) D oIndividualCollaborative or social group
Solve problemsBoth presence and creativity need to be supported. High presence is needed for individual creativity. High presence often contributes to high performance (Slater’s earlier work).Differs for proper collaboration versus peripheral collaboration. Also depends on application type, e.g. object-focused task- solving versus learning (Heldal 2004) . Maintaining and sustaining group activities needs social skills, high copresence, and social presence. Group creativity requires awareness of the members’ activity, i.e. social creativity is favored by high social presence (Fischer et al. 2005) .
Handle technologiesTo reach high presence, intuitive technologies and natural interactions are needed (Wolff et al 2006) . This is a precondition for supporting individual creativity.One can help the others in networked situations to handle technologies and eventual problems (Heldal 2004) . Helping requires different communication modalities. Symmetry or information on asymmetry helps ( Schroeder et al. 2006) . Naturalistic technologies may support peripheral collaboration, copresence and communicative activities that are required for group creativity (Roberts et al. 2007) .
ExperienceHigh individual presence is required for experiencing VEs. Creativity is less important.Group maintenance, collaborative activities and awareness are most important for a good outcome. Naturalness, high social presence can help . Time may be influential (Tromp 2001).

Table 4.

How problem-solving, handling technical devices and experiences, is influenced by presence and creativity for a single user or for collaborative settings.

Continuous workflow can also contribute to increased presence and creativity in CVEs. This can be maintained by avoiding or decreasing disturbances from the surroundings, which cause breaks in presence and interrupt concentration. Flow can be impacted by the way in which people handle unexpected events. Presence requires the building of a conceptual map that is grounded in the experience of sureties and even of plausible surprises, but is weakened by shocks Frencott 1999.

Creativity must be combined with knowledge of social and technical context in the design of content, for VEs to provide challenges in handling sureties, interest in handling surprises and support in handling shocks. We postulate, however, that flow requires the transition between handling these disruptions to be seamless. Interacting naturally and intuitively, for example, can also be disruptive or seamful interaction Höök et al. 2008 – for the aim of certain applications and for specific and expected handlings. In general, hiding the technology so that people can interact naturally with the simulation significantly increases not only performance, but also engagement, motivation, enjoyment, and creativity Wolff et al., 2006. Since temporal issues are crucial in work and many people work for long periods on computers, it is essential to include the effect of time in using CVEs. For many of the studies performed in the fields studying the usage of VEs, the users do not spend more than 15-30 minutes in the environments; this also influences earlier results. As we showed, time plays a very important role for all factors. The users adapt to using technologies and settings, and avoid hindrances, if they spend longer time in the environments. They also develop work-around to cope with the perceived disadvantages Heldal et al. 2006.

6. A pragmatic approach

In contrast to using the hard-to-interpret concept of presence, it is also possible to give some general advice with a basis in the group members and their human need to be recognized for their achievements. Based on the background from sections 3.2 and 3.3 that collaboration can be divided in three main stages, we consider the following.

The initial stage of collaboration can mean: getting to know each other, defining a task to solve or discuss, determining the rules of the group, and building (or acknowledging) trust. All of these activities often require a formal or an informal leader or leader group. In our experience, the emergent leader group often consists of a few people who already know each other quite well and trust each other. The values with this leader in the groups are often well developed. However, there are cases, e.g. for Internet-based communities, where there are no formal leaders Weisband 2007. Listening, and trying to understand and to explain, are catalytic actions by the leader(s). It would be advantageous Roberts, 2009 to have high resolution CVEs available at this stage when building trust, as much of this is related also to body language. Probably these VEs would require stronger governance during the first stage. If successful, the group would form a kind of society, a separate partition of the participant’s ordinary life, in which the innovative action would take place. This society with rather determined/fixed rules should in some respects be comparable to other virtual worlds. In this partition, the participants would have a feeling of belonging, meaning and security.

The minimum requirements for getting into the second stage of work and expecting results are: clear targets (what is to be achieved?), clear rules for collaboration (what is the reward?) and knowledge of who takes part in the group and their interests.

With the information available, the participant should experience trust and be able to answer the question: ‘What's in it for me?’. The work in the second stage would typically follow the funnel model: create many ideas and nurture them, and then gradually sort forward the best ideas. In this stage it would be supportive to have e.g. homework between meetings in CVEs. The group member would still require stimulation, response and energy from the rest of the group at regular meetings. Here the role of the leader still would be listening, trying to understand, and explaining. Some innovative groups use designers to visualise the ideas that are generated, and this seemingly stimulating interaction between the innovators and the people who visualize appears to be quite fruitful. For that case an efficient sketching tool would be required, a tool that everybody can use, and it is as intuitive as using paper and pen. During this stage it would be desirable to be able to transfer energy between participants and to have a system that allows jokes and other rather impulsive actions to keep spirits up.

The third stage, finally, with evaluation and praise and rewards, would require displays that make it possible also to read the body language. The possibility to come back and use the group for other work later would very much depend on a fair and well-organized conclusion of the present work.

7. Discussion and future directions

As was shown in the first sections, considerable work has been done regarding evaluation for VEs, but there is much less research on evaluation for CVEs. The work was more concerned with how technology can be connected at all, how it allows sharing of resources without bottlenecks, and how a previous type of interaction can be transmitted Heldal et al. 2006. By addressing these issues, the technological precondition for collaboration is mostly fulfilled. But even if this is a necessary precondition, there are several other aspects regarding the group and the context of application that have a major impact on the entire process of collaboration.

According to many studies the role of social interaction and social space becomes more important for collaboration Schroeder 2007, Tromp et al. 2003. The drawback of asymmetrical setups can be overcome by letting people trade places and learn about each other’s different capabilities. To evaluate CVEs, some of the problems are inherited from the lack of general theories to guide human performance measurement, difficulties in handling the inverse relationship between operational control and realism, the multiple dimension of behavior, how to measure cognitive tasks and how creativity is supported, etc.

We showed that there are at least three stages in sustainable constructive/creative work. The first stage is getting the group together, defining rules and roles, and establishing common ground, letting members be aware of each other, and formulating tasks and problems. During this peripheral communication, the socio-cultural context plays an important role; questions in relation to organizing work and group activities have to be settled.

The second stage is performing the creative or constructive work – the actual collaboration. In this stage, there are two possibilities. If the problem or tasks are intended to carry out rather moderate changes/innovations, so-called psychological creativity Fischer et al. 2005, one part of the aim is to get the entire group to arrive at and accept the relatively foreseeable solutions. In this case it is still important to have simultaneous participation of the entire group. Here, supporting individual presence and thus creativity, and seamlessly integrating it into the group work, are very important. Disturbances from the technical devices and from the group can lower both individual and group results. For well-structured problems with clear goals, the personal motivations to contribute in the group are important. For problems and tasks that aim to support learning, exploration, or general social interaction, maintaining both collaborative and individual presence is important. Even though ‘isolating’ a member can be disturbing for the group, it may still contribute to overall efficiency.

If the purpose is to find a really innovative and so far unknown idea, which Fischer called historical invention Fischer et al. 2005, the surroundings that support creativity with tools and tasks are important. The environment needs to allow evaluation of alternative choices, using different strategies, easy backtracking and searching. Even tough creativity has the highest importance; the social presence helps to support this. Once the context is set, the group is needed only to keep up the pace and to evaluate the proposed solutions. With this kind of formulation, it is possible and even desired to allow space and time for the group individuals to reflect and come up with ideas. The third stage is concluding the work, or certain work steps, making the solution and its evaluation known and giving rewards (of whatever type they may be). In this stage the group must be collective. Many of the preconditions for starting the work have to be considered for this stage too.

Another aspect that has to be reevaluated for supporting high copresence and creativity is consideration of interruptions. There are many interruptions that disturb individuals endeavoring to contribute their own creativity for working with physical objects in face-to-face conditions in real environments. These interruptions have many old characteristics inherited from face-to-face work, but they can also differ. Problems with devices, user interfaces, networking, who is talking to whom and why, turn-taking, or when the user interface does not transmit personal cues, or wrongly or non-naturalistically transmits certain actions such as pointing, looking, etc., can disturb the personal space needed for individual creativity. Furthermore, we showed that external observations play a great role in evaluating collaborative activities and group work. The observations can distinguish certain critical sequences and activities that the users went through quickly during their work and did not pay attention to.

For the study of CVEs, multiple users play an important role. To see CVEs as extensions of users that support seamless collaboration and creativity as in face-to-face conditions may be too complex and resource-demanding. For this, laboratory studies still are important. As we have seen, making VR systems usable for the average user requires generalization but also tailor-made solutions for specific purposes and technologies. However, one cannot expect that end-users will be able to describe their needs in technical terms and with an overall understanding of the complexity of VEs. Throughout the previous sections we have shown that future work has to consider developing seamlessly distributed work that allows: creative problem-solving, creativity-supporting tools, and allowing creative experiences. This often requires high presence and knowing more about the influence of presence. However, implementing ‘presence’ may not be necessary, especially for seamful technologies Höök et al. 2008. The recent literature still lacks a consistent theoretical approach to guide experiential research and technological development towards collaborative applications that support group creativity and allow high experiences. In our opinion, for the first step we have to identify the main dependencies between technology, application types, users, and time Heldal 2004.

8. Conclusions

By reviewing research on differences between individual work and group work, and how they are influenced by presence and creativity, this chapter has sought a better understanding of networked collaboration and how to support creative, collaborative activities in CVEs. As we have shown, in many cases presence may be either a precondition or a desired prerequisite for creativity; however, it is almost impossible to determine exactly the preconditions for individual presence for certain technologies, applications and groups. A suggestion here is that user experiences should be approached instead of presence.

Allowing creativity is considered to contribute to higher-quality work in general. However, to support group creativity, the peripheral collaboration should be seamless. Results on copresence show that maintaining and seamlessly integrating individual creativity, and allowing subgroup activities that contribute to group results, would be beneficial here. It is found, according to FischerFischer 2005, that one of the most important prerequisites for successful outcomes is providing enough time and space for individual creativity contributions. This may also be a requirement for supporting small groups within the larger groups. However, future research is needed in order to determine how to resolve certain social issues in technologies for better supporting individuals, subgroups and groups.

Overall, work was examined in terms of three main activities: how people approach problem-solving, how they handle technical devices, and how they experience virtual representations. The analytical distinction has helped us to review the creativity literature for these three areas and to connect it with available presence results, which are essential for evaluating VEs. In relation to methodology we have shown the importance of observations and of examining normative behaviors via laboratory studies.

9. Acknowledgements

The authors are grateful to David Roberts, Robin Wolff, and Jon van Leuven for their help.


1 - American National Standard Institute. 1993 Guide to Human Performance Measurements. American Institute of Aeronautics, Washington DC.
2 - A. Brogni, M. Slater, A. Steed, 2003 More Breaks Less Presence. Proc. Presence 2003, The 6th Annual International Workshop on Presence, Denmark.
3 - R. Brown, 2000 Group processes: dynamics within and between groups. Oxford: Blackwell Publishers.
4 - K. E. Bystrom, W. Barfield, 1999 Collaborative Task Performance for Learning Using a Virtual Environment.. Presence, MIT Press, 8(4), 435 448 .
5 - J. M. Carroll, 1995 The Scenario Perspective on System Development. In Carroll, J. M. (ed.), Scenario-Based Design: Envisioning Work and Technology in Systems.
6 - M. J. Culnan, 1987 Mapping the Intellectual Structure of MIS 1980-1985: A Co-citation Analysis. MIS Quarterly, 11( 3), 341 353 .
7 - M. Csikszentmihályi, 1996 Creativity: Flow and the Psychology of Discovery and Invention. New York: Harper Perennial.
8 - P. Dourish, 2001 Where the Action Is: The Foundations of Embodied Interaction. MIT Press, Cambridge, Massachusetts.
9 - S. R. Ellis, 1995 Origins and Elements of virtual Environments. In Barfield, W. (ed.), Virtual Environments and Advanced Interface Design, Oxford University Press, 14 57 .
10 - U. Farooq, J. M. Carroll, C. H. Ganoe, 2005 Supporting Creativity in Distributed Scientific Communities. Proceedings of the International GROUP Conference, 217 226 .
11 - C. Fencott, 1999 Content and Creativity in Virtual Environment Design. Proc. of VSMM’99.
12 - G. Fischer, E. Giaccardi, et al. 2005 Beyond Binary Choices: Integrating Individual and Social Creativity. Human-Computer Studies (IJHCS). 63(4-5), 482 512 .
13 - C. Gutwin, S. Greenberg, 1998 Design for Individuals, Design for Groups: Tradeoffs between Power and Workspace Awareness. Proc. CSCW’98, ACM Press, Seattle, 207 216 .
14 - I. Heldal, 2004 The Usability of Collaborative Virtual Environments: Towards an Evaluation Framework. Ph.D. Thesis, Dept. of Technology and Society, Chalmers, Sweden.
15 - I. Heldal, L. Bråthe, et al. 2006 Analyzing Fragments of Collaboration in Distributed Immersive Virtual Environments. In Schroeder, R. Axelsson, A. (eds.), Avatars at work and play. Springer Verlag, 97 130 .
16 - I. Heldal, D. Roberts, L. Bråthe, R. Wolff, 2007 Presence, Creativity and Collaborative Work in Virtual Environments. In Lecture Notes in Computer Science, Human-Computer Interaction, Springer Berlin/Heidelberg, 802 811 .
17 - K. Höök, A. Ståhl, et al. 2008 Interactional empowerment. Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems. ACM Press.
18 - D. R. Lampton, J. P. Bliss, C. S. Morris, 2002 Human Performance Measurement in Virtual Environments. In K.M. Stanney (ed.), Handbook of Virtual Environments. Design, Implementation, and Applications. Lawrence Erlbaum Assoc., Mahwah, 710 720 .
19 - J. M. W. Mc Leish, J. Park, 1973 The Psychology of Learning Group. Hutchinson, London.
20 - J. E. Mc Grath, 1984 Groups: Interaction and Performance. Englewood Cliffs, NJ: Prentice Hall.
21 - J. Nielsen, 1993 Usability engineering. AP Professional, Boston.
22 - D. C. Neale, J. M. Carroll, M. B. Rosson, 2004 Evaluating Computer Supported Cooperative Work Models and Frameworks. Proceedings of CSCW’04, 112 121 .
23 - A. Nilsson, I. Heldal, et al. 2002 The Long-Term Uses of Shared Virtual Environments: An Exploratory Study. In Schroeder, R. (ed.), The Social Life of Avatars. Springer, London, 112 126 .
24 - M. Polanyi, 1966 The Tacit Dimension. Garden City, NY: Doubleday.
25 - N. F. Polys, D. A. Bowman, 2004 Design and Display of Enhancing Information in Desktop Information Rich Virtual Environments. Virtual Reality, 8, 41 54 .
26 - S. Poltrock, J. Grudin, 2003 Collaboration Technology in Teams, Organizations, and Communities. Proceedings CHI’03, Tutorial.
27 - D. Roberts, I. Heldal, et al. 2007 Factors influencing Flow of Object Focused Collaboration in Collaborative Virtual Environments. Virtual Reality, 10 2 119 33 .
28 - D. Roberts, 2009 Reproducing face-to-face interaction. Presentation in Gothenburg, Chalmers, Sweden, 27 April 2009.
29 - R. Schroeder, I. Heldal, J. Tromp, 2006 The Usability of Collaborative Virtual Environments and Methods for the Analysis of Interaction.. Presence 15(6), 655 667 .
30 - R. Schroeder, 2007 Rethinking Science, Technology and Social Change. Stanford Univ. Press.
31 - C. R. Scott, 1999 Communication Technology and Group Communication. In Frey, L.R. (ed.), The Handbook of Group Communication Theory and Research. Sage, 431 472 .
32 - B. Shneiderman, 2002 Establishing Framework of Activities for Creative Work. Creativity Support Tools, Communication of the ACM, 45(10), 116 120 .
33 - B. Shneiderman, G. Fischer, et al. 2007 Proceedings of the 6th Conference on Creativity & Cognition, Washington, DC, USA, June 13 15 .
34 - J. Short, E. Williams, B. Christie, 1976 The Social Psychology of Telecommunications. London: John Wiley.
35 - M. Slater, M. Usoh, A. Steed, 1994 Depth of Presence in Virtual Environments. Presence 3(2), 130 144 .
36 - M. Slater, V. Linakis, et al. 1996 Immersion, Presence, and Performance in Virtual Environments. Proc. ACM Virtual Reality Software and Technology, 163 172 .
37 - M. Slater, 1996 Co-presence as an Amplifier of Emotion: Available http://wwwais.
38 - M. Slater, A. Sadagic, et al. 2000 Small-group behavior in a virtual and real environment: A comparative study. Presence. 9(1), 37 51 .
39 - M. Slater, 2004 How colourful was your day? Why Questionnaires Cannot Assess Presence in Virtual Environments. Presence 13(4), 484 493 .
40 - D. H. Sonnenwald, P. Solomon, et al. 2002 Collaboration in the large: Using Video Conferencing to Facilitate Large Group Interaction. In Khalil (ed.), Knowledge and Information Technology in 21st Century Organizations, Idea Publishing Co., 136 155 .
41 - A. Steed, C. Parker, 2005 Evaluating Effectiveness of Interaction Techniques across Immersive Virtual Environment Systems. Presence 14(5), 511 527 .
42 - J. S. Stone, 2002 Applications of Virtual Environments: An Overview. In K.M. Stanney (ed.), Handbook of Virtual Environments. Design Implementation, and Appl., 827 856 .
43 - J. Tromp, 2001 Systematic Usability Design and Evaluation for Collaborative Virtual Environments. PhD Thesis, University of Nottingham, Nottingham, UK.
44 - J. Tromp, A. Steed, J. Wilson, 2003 Systematic Usability Evaluation and Design Issues for Collaborative Virtual Environments.. Presence, 10(3), 241 267 .
45 - J. B. Walther, 1996 Computer-mediated communication: Impersonal, interpersonal, and hyperpersonal interaction.. Communication Research, 23, 3 43 .
46 - J. B. Walther, M. Parks, 2002 Cues filtered out, cues filtered in. Handbook of Interpersonal Communication. Thousand Oaks, CA: Sage Publications.
47 - M. Vass, J. M. Carroll, C. A. Shaffer, 2002 Supporting Creativity in Problem Solving Environments, Proceedings of the Fourth Creativity & Cognition Conference, 31 37 .
48 - J. A. Waterworth, E. L. Waterworth, 2001 Affective Creative Spaces: the Interactive Tent and the Illusion of Being. Proc. of the 1st Int. Conf. on Affective Human Factors Design.
49 - C. M. Webster, L. C. Freeman, C. G. Aufdemberg, 2001 The Impact of Social Context on Interaction Patterns. Journal of Social Structure, 2(1).
50 - S. P. Weisband, 2007 Leadership at a distance. Research in Technologically-Supported Work. Psychology Press.
51 - J. R. Wilson, 2003 If VR Has Changed, then Have its Human Factors? In Waard, D. et al. (eds.), Human Factors in the Age of Virtual Reality, Shaker Publishing, 9 30 .
52 - B. G. Witmer, M. J. Singer, 1998 Measuring Presence in Virtual Environments: A Presence Questionnaire.. Presence, 7(3), 225 240 .
53 - R. Wolff, D. J. Roberts, et al. 2006 A Review of Tele-collaboration Technologies with Respect to Closely Coupled Collaboration. International Journal of Computer Applications in Technology (IJCAT), 11 26 .
54 - T. Vöhringer-Kuhnt, 2002 The Influence of Culture on Usability. Freie Universität, Berlin.