QoS Adaptation for Realizing Interaction Between Virtual and Real Worlds Through Wireless LAN

Some of these existing studies realize scalability on sharing virtual space between many users using P2P technologies. However, they allow sharing virtual space and objects among only virtual users. On the other hand, the existing MR and AR technologies, the criteria (1) to (3) can be satisﬁed. However, they require special devices, servers and networks, and thus satisfying the criteria (4) and (5) is difﬁcult.

In this chapter, we propose a framework named FAIRVIEW (Yamamoto et al. (2007)) which realizes smooth cooperation and interaction between real and virtual users satisfying all the criteria (1) to (5) using inexpensive devices off the shelf. To satisfy the criteria (1) to (3), FAIRVEIEW produces a hybrid shared space by overlapping a virtual space and a real space, and provides a mechanism for allowing the virtual and real users to observe each other. To satisfy the criterion (4), we suppose that virtual users use ordinary PCs with an internet connection, and that real users use wearable computers with HMDs (head mount display) or smartphones, with internet connection via wireless LAN. In FAIRVIEW, the information regarding to orientations and positions of real objects (called AR information, hereafter) are measured at short intervals using an existing AR measurement tool. The information is exchanged among user terminals, and objects are displayed as 3D graphics on the display of the virtual user's terminal. To achieve the criterion (5), we propose a mechanism for delivering AR information as well as action/reaction to the objects in real time (which we call AR event delivery mechanism, hereafter) to realize smooth cooperation among real and virtual users. Here, information of action/reaction consists of information of attribute (the color or the form, etc), action type (push, pull, etc), action strength and action direction. In addition, we define the AR event that consists of AR information and the information of action/reaction. The AR event delivery mechanism includes a QoS adaptation mechanism for controlling the intervals of transferring AR events between users so that the total transmission rates will not exceed the limit of available bandwidth. The adaptation mechanism decides the transfer intervals according to importance of each object for each user, which is determined automatically according to the distance and position of the object in the user's view.
To evaluate the proposed mechanism, we analyzed the required bandwidths and investigated the effectiveness of our QoS adaptation mechanism under some configurations with different numbers of objects. In addition, in the real space, users may not get sufficient efficacy of QoS adaptation because users share wireless bandwidth in a single AP coverage area. However, this problem can be improved by introducing multiple APs with appropriate AP selection. Therefore, we conducted computer simulation for measuring improvement of QoS adaptation with strategic AP selection when there are multiple APs in the real space. Furthermore, in the real-time interaction, users often do the sudden action, such as turn around. Therefore, to evaluate tolerance of our QoS adaptation mechanism to the sudden action of users, we measured the delay until reflecting update frequency of objects when the user's view changes, by using a prototype of FAIRVIEW. Simultaneously, to evaluate whether FAIRVIEW can achieve sufficiently short delays for real-time interaction, we measured the end-to-end AR event delivery delays through the AR event delivery mechanism in LAN. As a result, we confirmed that the proposed mechanism realizes smooth interaction involving a large number of real and virtual users/objects on an ordinary wireless LAN and internet environment.

Related work
Studies regarding to NVE and DVE, can be roughly categorized into two types: one is aiming at reducing traffic of game information and keeping scalability by efficiently managing Area of Interest (AOI) of users, and another is aiming at efficiently using computation and network resources.
In Hu & Liao (2004);Vleeschauwer et al. (2005);Yu & Vuong (2005), efficient AOI (Area of Interest) management methods are proposed. In Hu & Liao (2004), game space is dynamically divided based on Voronoi diagram for direct communication among players in a same fragment. In Vleeschauwer et al. (2005), the space is divided into small areas called micro cells. To distribute the load for processing events among multiple servers, the regions managed by each server are dynamically changed. In Yu & Vuong (2005), the shared virtual space is divided into honeycomb regions, and a mechanism based on Pastry (Rowstron & Druschel (2001)) is used to allow each user to receive information for players and objects in the player's AOI.
Although these existing NVEs enable efficient information exchange among virtual users, it is difficult to apply them to the mixed space of real and virtual worlds which requires a large amount of information exchange in real time on resource-limited wireless network, which is the target environment of FAIRVIEW.
In Chertov & Fahmy (2006); Eraslan et al. (2003), a load distribution and a QoS adaptation mechanisms for DVEs are proposed, respectively. In Chertov & Fahmy (2006), in order to cope with so called area boundary problem (inconsistency caused by neighboring areas managed by different servers), the whole shared space is managed by each of game servers, and game processing tasks are flexibly distributed among the servers. Unfortunately, this method supposes high performance servers and high-speed networks and treats only virtual users. In Eraslan et al. (2003), an IPv6-based network architecture called VESIR-6 is proposed for realizing a large-scale DVE where users can share a 3D virtual space and objects. Aiming at efficient utilization of network resources, VESIR-6 uses multicast for delivering object state updates to users, anycast for load distribution among servers, and IntServ/DiffServ-based QoS adaptation mechanism for regulating per-flow transmission rate. However, VESIR-6 does not suppose wireless network environment which is necessary for interaction between real and virtual users. Also, delivery of object state updates is managed only by joining/leaving the corresponding multicast group. Therefore, VESIR-6 cannot provide fine-grain QoS adaptation like the proposed method.
The most related study to our work is tele-immersion which captures the whole environment and reproduces it at geographically distant location. TEEVE (Yang et al. (2010)) displays 3D live visuals from each user's view in real-time, and constructs an environment for cooperative working. However, expensive and specialized devices and infrastructures such as 3D cameras and broadband network environments are needed to construct tele-immersion environment.

Overview of FAIRVIEW
This section overviews the functions of FAIRVIEW and presents example applications, then describes the target environment and the basic ideas for implementation.

Functions of FAIRVIEW
FAIRVIEW overlaps real space and virtual spaces, and provides the environment where users in real space (called real users) and users in virtual spaces (called virtual users) can interact as if they were in the same space. We call the overlapped space hybrid space 1 in Fig. 1. The main functions of FAIRVIEW are as follows: Function (1) Providing the same view to real/virtual users: Virtual users can change their positions in virtual space using keyboards and mice Object sharing: Real objects and virtual objects can be registered in (and also unregistered from) the hybrid space. The registered objects can be seen by both virtual and real users. Users themselves are also objects 2 . Shared objects can be seen by all users whose views include the objects. We assume that 3D geometry data for the registered objects are prepared beforehand. Function (4) Moving and bonding objects: Users can move neighboring objects. Virtual objects can be moved by both real and virtual users, while real objects can be moved only by real users. Movement of objects can be observed by other users. Two or more objects can be bonded by specifying their relative positions. Bonded objects basically move together, but if a user moves a virtual part of bonded objects made of real and virtual objects, the bonded object is disengaged.

Applications of FAIRVIEW
We describe two example applications which are enriched by using the functions of FAIRVIEW described above.
Virtual flea market In this application, real and virtual users trade goods as if they were in a real market. Virtual users go shopping to a flea market in hybrid space. Real sellers register their goods to hybrid space. Virtual buyers see what kinds of goods are sold from distance. The sellers and buyers interact with each other via gestures and voice. The sellers show and explain details of their goods to virtual users by rotating and moving the goods. This kind of application includes exhibition, trade fair, and shopping center.
Multiplayer game In this application, virtual users participate in a paintball wargame held in real space. In paintball wargames, players possess airguns and targets, and shoot opponent groups' targets. Players whose targets are shot lose the game. Real players only need to register airgun and target. The virtual users cooperate with other real or virtual users. Virtual characters like a huge dinosaur can participate in the game. This kind of application includes attractions in theme parks, events on a street corner. Table 1 shows the list of necessary equipments for the users. Real users use wireless small computing devices such as smartphones, and HMD with which the real view can be seen as the background of virtual view. These devices should not hinder real users' movements. The equipments also include positioning devices such as GPS receiver, sensors to detect position and direction, and audio input/output devices. We assume that inexpensive devices are available for these purposes. We use webcams and dedicated software like ARtoolkit (ARtoolkit) for positioning and detecting orientations of objects.

Basic ideas to implement FAIRVIEW
To realize an application using AR technology, we considered how the views seen by users are rendered inexpensively. In TEEVE (Yang et al. (2010)), images captured by 3D multi-cameras are processed and transferred through Internet2. Since we are aiming at inexpensively implementing the functions, we decided to measure AR information using inexpensive sensors. To realize this method, we have to resolve the following three problems: (i) measuring AR information accurately; (ii) transferring the measured AR information in real time; and (iii) rendering the objects.
For resolving the problem (i), we use GPS receiver if a user is in outdoor space. If a user is in indoor space, we use an indoor positioning method such as a method using wireless LAN access points (Kitasuka et al. (2003)), a method using speakers and microphones (Scott & Dragovic (2005)), Place Lab (Lab.), and Weavy (Kourogi & Kurata (2003)). The orientation of an object can be measured using rotational and translational acceleration sensors, methods based on image processing such as ARtoolkit (ARtoolkit) or the method in Fudono et al. (2005). To use the methods based on image processing, stickers with special patterns printed are put on objects, and the orientation and position of the objects can be obtained instantaneously from an image captured by a web camera. This kind of methods have cost advantage. In FAIRVIEW, AR information can be measured by combining the methods described above.
For the problem (iii), we assume that users have terminals capable of rendering 3D graphics. The problem (ii) is the main problem which we treat in this chapter. To resolve this problem, we need a delivery mechanism with which AR information can be exchanged among wireless and wired users in real time. We call this mechanism AR event delivery mechanism.I n applications such as flea market or paintball wargame, hybrid space may have many objects and bandwidth shortage can occur as the amount of exchanged AR information becomes large. Therefore, we have to lower the frequency of delivering AR information. However, if the frequency becomes too low, the user experience of interaction may be ruined. Therefore, we propose a QoS adaptation mechanism which maximizes users' satisfaction by prioritizing transfer of AR information among objects based on relative position and orientation between users and objects. As a solution, we present the AR event delivery mechanism in Sect. 4, and the QoS adaptation mechanism in Sect. 5. This section describes details of the AR event delivery mechanism.

Notation
Let R and V be the target real space and the corresponding virtual space, respectively. Let H =(R, V) be the hybrid space produced by overlapping R and V. We suppose that H is an axis-aligned rectangle on x-y plane of 3D coordinate system. Let hvec =( 1, 0, 0) and vvec = (0, 0, 1) be the horizontal standard vector and the vertical standard vector of H, respectively.
Let RO = {ro 1 , ..., ro n }, VO = {vo 1 , ..., vo m }, RU = {ru 1 , ..., ru l } and VU = {vu 1 , ..., vu k } be the set of real objects in R, the set of virtual objects in V, the set of real users (i.e., mobile users) and the set of virtual users (i.e., remote PC users), respectively. In addition, let AO be the set of all objects including users, where each member object is denoted by ao ∈ AO. divide hybrid space M sub hybrid spaces Note that RU ⊆ RO and VU ⊆ VO.L e t node(u) be the user terminal of u for each user u ∈ RU ∪ VU. RN = {node(u)|u ∈ RU} and VN = {node(u)|u ∈ VU} are the real user terminals and the virtual user terminals, respectively.
For each object ao ∈ AO,l e tAR(ao)=( pos, angleH, angleV) be the AR information of ao, where pos, angleH,andangleV are its position on or beyond H, the horizontal angle to hvec, and the vertical angle to vvec, respectively. Each item of AR information is referred to by, e.g., ao.pos. In addition, let ARe(ao)=( AR(ao), attribute, action) be the AE Event of ao, where AR(ao) is the AR information of ao, attribute = color, form, ... is attribute information such as color and so forth, and action = type, direction, strength, ... is action to other objects, characterized by type ∈{push, pull, ...}.
We suppose that each real user terminal in RN can measure its user's AR information at a sufficiently high frame rate, e.g., 60 times per second, with equipment explained in Sect. 3. We also suppose that AR information of each real object ro except for users can be measured by a user terminal in ro's proximity with ARtoolkit and webcam.

Assumption on user communication
We suppose that the whole real space R is covered by only one AP connected to the Internet. Therefore, each real user terminal in RN can communicate with any virtual user terminal in VN.
Let BW AP be the available bandwidth between real user terminals and AP. Note that all real user terminals share the bandwidth. For each vn ∈ VN,l e tbw AP (vn) be the available bandwidth between vn and AP. For each vn ∈ VN and rn ∈ RN,letbw(vn, rn) be the available bandwidth between vn and rn.Not et h a tbw(vn, rn)=Min(BW AP , bw AP (vn)).Forea c hp a i r of virtual user nodes vn1, vn2 ∈ VN,l e tbw(vn1, vn2) be the available bandwidth between vn1andvn2.

AR event delivery mechanism
We choose some of the user terminals as server nodes to manage efficient AR events exchange. In order to reduce processing and traffic load of each server node based on user's AOI, we

Fig. 3. Overlay Network for AR Event Delivery
divide the whole shared space H into small rectangular sub-areas, as shown in Fig. 2, and assign a server node called area node to each sub-area. This is a similar approach to existing P2P-based MMOG gaming architectures such as Yamamoto et al. (2005). Let an A be the area node assigned to sub-area A.
The sub-area an A receives AR events of objects in A, and delivers the AR events to users (in A and neighboring sub-areas) who are watching the objects.
In FAIRVIEW, since the network resource is tight due to wireless communication and calculating the reaction of an object as a consequence of an action is heavy for a real user terminal, we introduce new server node called bandwidth controller node (bwc-node).
Onebwc-nodeispreparedforeachvirtualuserterminalvn ∈ VN satisfying that bw(vn, an A ) is less than necessary traffic for AR event transmission or for the set of all real user terminals 3 . Let bn(u) be the bwc-node for user u ∈ RU ∪ VU.T h e nbn(u) applies QoS adaptation to the AR event stream between an A and node(u), by monitoring bw(bn(u), node(u)).
Consequently, we construct the overlay network per sub-area, consisting of an area node, N object nodes, M user nodes, and M bwc-nodes, as shown in Fig. 3, where N and M are the numbers of objects and users in the subarea, respectively. We will explain how these nodes exchange AR events below.
User Node In FAIRVIEW, (1) a user u ∈ RU ∪ VU can watch other objects in its view, (2) u can be watched by other users since u is also an object, and (3) u can take an action to other objects.
For (1), node(u) measures u's AR information continuously and if the information differs from the last measurement, node(u) sends the information as AR event to the area node an A (Fig. 3 (a)). For (2), node(u) receives the AR events of the objects in u's view and draws the latest appearance of objects on display of node(u). To receive AR event, node(u) uses publish/subscribe model (Tanenbaum & Steen (2002)). Once node(u) sends its AR event to the area node an A , an A automatically identifies the objects in u's view, and forwards the AR events of the objects to node(u) via the bwc-node (Fig. 3 (b)(b')). As shown in Fig. 3 (b'), the real user terminals receive the AR events from the bwc-node assigned for real space R via wireless AP by broadcast. For (3), when u takes an action to virtual object o, node(u) sends an action event containing power and direction, to an A (Fig. 3 (c)).
Area Node For each sub-area A of the hybrid space H, a virtual user node is selected and assigned to the area node. A virtual node with sufficient network and computation resources is selected, e.g., by the lobby server when the application starts 4 .
The area node an A manages the positions of the objects as well as the users' view in the sub-area A.W h e nan A receives the AR events of all objects in A from the corresponding user nodes and object nodes (Fig. 3 (c)), an A identifies the users who are watching part of other sub-areas neighboring A, based on their views calculated from their AR events (e.g., pos and angleH), and sends the AR events to the neighboring area nodes if needed. The sub-area an A also receives the AR events of such users in other sub-areas from the neighboring area nodes. Finally, an A sends the AR events of the objects in A to the user nodes via the corresponding bwc-nodes.
Bandwidth Controller Node One bwc-node is assigned to each user node u, although one node may serve as the bwc-nodes of multiple users. The bwc-node bn(u) monitors available bandwidth to its associated user node node(u), and regulates transmission rates of AR event streams.

The overall flow
The overall flow is as follows.
Step(1) User Node node(u) sends AR event ARe(u) to Area Node an. Then, when User Node node(u) has the managerial responsibility of an object o, it acquires the AR information of o, makes AR event ARe(o), and sends ARe(o) to an.
Step(2) Area Node an adds an importance to AR event ARe(ao) using the QoS adaptation mechanism described in Sect. 5.
Step(3) Area Node an sends to Bandwidth Control Node bn,A Re v e n tARe(ao) with the importance.
Step(4) Bandwidth Control Node bn regularly redistributes bandwidth with the QoS adaptation mechanism explained in Sect. 5 and decides update frequency of each AR event ARe(ao).
Step(5) Bandwidth Control Node bn sends the set of AR events to User Node node(u) with the update frequency defined in Step(4).
Step(6) User Node node(u) updates data of Hybrid Space H with the set of received AR events.
Step(7) User Node node(u) draws Hybrid Space H with the updated data.

View-oriented QoS adaptation
The basic ideas of our QoS adaptation mechanism are as follows: (1) we decide relative importance value of each object according to how important the object is for a user; and (2) for each user, we regulate transmission rates of AR events of observable objects based on their importance values so that the sum of transmission rates are less than the available bandwidth.

Decision of importance value
Let Watcher ( Fig. 4. Here, the objects located on zone a have the highest importance, and the importance of objects on zones b, c, d and e decreases in this order.

Decision of framerates of AR events
The transmission rate of AR events for each object towards user u is decided based on the ratio of its importance value to the sum of importance values of all objects in u's view. The available bandwidth is distributed among the objects, and the framerate of AR events for each object is decided from the assigned transmission rate and the size of each AR event.
As mentioned in Sect. 4, AR events of each object are delivered to each user node through the corresponding bwc-node (see Fig. 3). The bwc-node drops packets of the AR events so that the transmission rate does not exceed the assigned bandwidth.
We describe our view-oriented QoS adaptation mechanism using examples. The QoS adaptation for AR event streams differs depending on the receiver type (i.e., virtual user or real user). Thus, we give two examples in the following subsections.

QoS adaptation for virtual user
Suppose that a virtual user v ∈ VU is watching three objects o 1 , o 2 and o 3 in his view. In this case, node(v) receives the AR events of those three objects via the bwc-node bn v ∈ VN (see Fig. 3).
We assume that the available bandwidth between bn v and node(v) is 1Mbps. We also assume that transmission rates for AR event streams of objects o 1 , o 2 and o 3 are 0.5 Mbps respectively, that is, the sum of the streams is 1.5 Mbps. In this case, the available bandwidth (1 Mbps = 0.3Mbps are assigned to the AR event streams, respectively. Based on this result, the bwc-node bn v controls transmission of the AR event stream of each object by dropping some of the received packets.

QoS adaptation for real user
Suppose that a real user r ∈ RU is watching three virtual objects o 1 , o 2 and o 3 in his view. In this case, node(r) receives the AR events of those three objects via wireless AP and the bwc-node bn W ∈ VN (see Fig. 3).
In this case, we define the importance value Imp r (o) of an virtual object o ∈ VO as the sum of the importance rated by the users observing the object o, i.e., Imp r (o)= ∑ u∈Watcher(o) (Imp(o, u)). The framerate of AR events of each object is decided similarly to the case in Sect. 5.2.1, and the bwc-node bn W applies the QoS adaptation to the AR event streams to real users.

Experiments
In order to evaluate the proposed method, we conducted experiments on five items. First, we measured the required bandwidth for application of users and objects (Sect.6.2). Second, we measured the framerates at which users can watch the objects for two cases with and without our QoS adaptation mechanism (Sect.6.3). Thirdly, we measured the improvement of QoS adaptation with emulation in which users choose AP according to a strategy when there are multiple APs (Sect.6.4). Fourth, we measured the delay necessary for changing update frequency by QoS adaptation when the user's view changes, under some configurations with different numbers of objects using a prototype of FAIRVIEW (Sect.6.5). Finally, we measured the end-to-end delay through AR event delivery mechanism using a prototype of FAIRVIEW (Sect.6.5). Note that the experiments in Sect.6.2, 6.3 and 6.4 were emulations. The experiment in Sect.6.5 was conducted on a testbed using a prototype of FAIRVIEW with LAN. We show the experimental configurations in Table 3. The sizes of virtual space V and real space R are both 50 × 50 m. As user's view in Fig. 4, we set the radius of a half circle to be 15m and divided the half circle equally so that the angle and the radius of each zone are 1/3 (i.e., π/3 and 5m), respectively. In this configuration, the average number of objects in each user's view is around 1-2 when there are 50 objects in the whole space. In Fig. 4, zone c exceeds in the number of objects compared with the zones a, b because zone c is the largest in the gross floor area. Therefore, in zones a, b, there is not much advantage than in zone c when we give 2 times importance sequentially backward from zone e. Hence, we give zones a, b importance of 4 times sequentially from c backward. Thus, we assign the importance 64, 16, 4, 2, 1 to the zones a, b, c, d, e, respectively.

Required bandwidth for user terminals
The experimental configurations are shown in Table 3. The other setting is as follows. There is one wireless AP whose radio range covers the whole real space R. The available wireless bandwidth BW AP is shared among all real users. One area node and bwc-nodes are allocated on PCs on fixed wired network. 100 real users, 100 virtual users, and n virtual objects are placed on the positions decided at random in the hybrid shared space, changing n from 50 to 5000. The directions of the users are also set randomly. Each user terminal sends 60 packets of AR events every second, and the size of each packet is 32 bytes. The result is shown in Fig. 5. This result is average of 100 trials.
We measured the required network bandwidth between a user node and the corresponding bwc-node under the configurations in Sect. 6.1 for both cases with and without our QoS adaptation mechanism. The average results of 100 simulations are shown in Fig. 5.
Without the QoS adaptation mechanism, the required bandwidths for each real user terminal ("RU without QoS" in Fig. 5) and each virtual user terminal ("VU without QoS" in Fig. 5) exceeded the capacity (i.e., 10Mbps and 5Mbps ) when the number of objects in the whole space becomes more than 250 and 500, respectively. With our QoS adaptation mechanism, the required bandwidth is regulated below the capacity even if the number of objects increases to 5000 ("RU with QoS" and "VU with QoS" in Fig. 5).

Impact of QoS adaptation
With our QoS adaptation method, AR events of more important objects are transmitted at higher transmission rates (i.e., framerates) than other objects, within the available bandwidth.
To examine the effect of the QoS adaptation, we measured the framerates of AR events for objects in view zones a to e under the configurations in Sect. 6.2. The results are shown in Fig.  6 and Fig. 7.
In Fig. 6 and Fig. 7, the lines with labels a, b, c, d,ande show the average framerates for objects on the corresponding zones, and the line with label uniform shows the average framerate when the bandwidth is uniformly distributed to objects. Fig. 6 shows that each virtual user can watch important objects in zones a and b at higher framerates than uniform. Especially, framerate of objects in zone a keeps more than 50 frames/sec while the number of objects is less than 4500. The framerates of less important objects on zones c, d and e are reduced below uniform. Fig. 7 shows that each real user can also watch important objects in zones a at better framerates than uniform. However, the effect is smaller than the case for virtual users. The framerates of objects in other zones are reduced below uniform. This is because each object's importance value is decided as the maximum value among its watchers as explained in Sect. 5.1 and large portion of objects are regarded as important objects when the number of objects is large. However, the framerates of the objects in zone a are improved to a great extent when the number of objects is less than 2000.

Improvement of QoS adaptation control by AP selection
The real users share AP and receive AR events broadcast from AP. In the real space, each object's overall importance value is the sum of the values rated by all users sharing AP. Therefore, it cannot sufficiently reflect the importance of each user (shown as Sect. 5.2.2 and Sect. 6.3). If multiple APs are installed in real space, and users can choose AP which can reflect one's importance effectively, then we can improve the satisfaction of users. Therefore, we measured the improvement of QoS adaptation with simulation in which users choose AP according to a strategy when there are multiple APs. In this experiment, we installed 2 to 4 APs. In case the number of AP is 2, we equally divide real space in x-axis into two areas and placed one AP in each area. In case the number of AP is 4, we placed AP in each of the two by two matrix regions formed by equally sectoring a long x-axis and y-axis. In this experiment, to focus on the AP selection strategy, we supposed that a radio range is the whole area. In this experiment, the strategies that users are supposed to use are as follows.
1. UNIFORM: the user connects to AP with ID equal to his/her ID mod the number o f APs.
2. POSITION: the user connects to the nearest AP from his/her position.
3. VIEW: the user connects to the nearest AP from his/her position taking into account his/her view. Thus, comparing the his/her position and the end point of his/her view (e.g. distance = 15m and angle = 0 • ), it employes the point whose distance to the AP is greater.
The results are shown in Fig. 8. This figure is typical boxplot that show how much frame-rate was assigned according to importance with strategic AP selection and the number of the AP. In this figure, the bar graph is Max-Min range and the box graph is the inter-quartile range. For example, in importance 64, 2 APs, "UNIFORM" in Fig. 8, we can see that there are the users for whom only 50 frames/sec was assigned to objects of importance 64, while many users can watch objects of importance 64 in 60 frames/sec. In this result, we can see how efficiently bandwidth allotment reflects the importance of the object for each user. In "UNIFORM" in Fig.  8, when several APs are available, the improvement is seen in comparison with the result in case the number of objects equals 2500 of Fig. 7. The value of framerates to be given disperse, because this strategy is random selection. In "POSITION" or "VIEW" in Fig. 8, dispersion of provided framerates for the importance of the user becomes small. In these results, "VIEW" was approximately equal to "POSITION". "VIEW" strategy removes users facing the other APs than the AP concerned. In spite of that, it is seen in Fig. 8, that bandwidth assignment in "VIEW" strategy seems more dispersive than "POSITION" strategy. The reason is that in "VIEW" strategy AP accepts far-off users looking towards itself as compensation does not accept far-off users looking outwards. As a result, the AP selection in consideration of the orientation of the user is effective in the QoS adaptation control for the real users. In QoS adaptation control for the real space, when there are some APs are available, we can get big benefits with strategic AP selection.

Processing delay of QoS adaptation
We measured the the coordination time of update frequency under some configurations when moving user's view by prototype system with different numbers of objects in LAN.  Table 3. The other setting is as follows. The user is placed in the center of the space. He/She walks at 1 m/sec towards the destination decided at random. When he/she arrives at the destination, the next destination is decided at random again. In this experiment, we place n virtual objects every 3 meters from the user. Then, the space has just only enough room for the all objects. Updates of the data of each object by the AR event, the transmission of packets, reception of packets, changes of the importance, and QoS adaptation control are preformed by parallel computation at about 30 times per second in the prototype for an experiment. Then, each process is asynchronous processed in an individual timeslot. The change of the importance is executed in a bwc-node and the QoS adaptation control is executed in an area node. We used TCP for communication protocol.
We measured, using the timestamp, the processing delay necessary for the recalculation of the importance and the recomputation of QoS adaptation control. In addition, we measured the end-to-end delay by the difference of timestamp from the time when user node sends AR event to the time when user node receives and updates data. These data do not include the delay of thread generation. The results are shown in Figs 9 is a graph of the processing delay of recomputation of the importance to change along the position of the avatar ("importance" in Fig.9) and recomputation by QoS adaptation control ("QoS Adaptation" in Fig.9). As a result, in the recalculation of the importance, a processing delay considerably grows as the number of objects increases because computational complexity increases according to the number of objects. The main cause of this processing delay is calculation of distance and angle of the avatar vs. the object. Therefore, when there are many objects, the importance is difficult to calculate in short time (e.g., as for every frame). As a solution, we can increase the number of area nodes or lower the frequency of the recalculation. The delay of QoS adaptation control changes by a timing of the processing. However, this delay stay lower than 25ms and does not depend on the number of objects. Furthermore, by the total of the recalculation of the importance and recomputation of QoS adaptation control, we can see the tolerance of user's sudden action (such as turns around), how much delay the influence is reflected on QoS adaptation when the importance of each object largely changes.
In Fig. 10, we show the end-to-end delay through AR event delivery mechanism including a result of Fig. 9 (Step(1)- (7) in Sect. 4.4). The end-to-end delay is quite satisfactorily and less than 200ms. The delay can be factored as follows: the recalculation of importance, the recomputation of QoS adaptation control, the wait time of the asynchronous processing. In Fig. 9, we see that the sum of the recomputation of QoS adaptation control and recalculation of importance is less than 50ms. Therefore, we show a main cause is wait time of the asynchronous processing. In addition, in Figs. 9 and 10, when there are objects more than 1,500, the total calculation delay is about 30ms and the influence is reflected within 200ms. If a human turns around and considers movement to adjust a focus to an object , we can permit this delay in practical uses. As a result, our framework can sufficiently cope with games requiring quick response.

Conclusion
In this chapter, we proposed a framework for interaction between real and virtual users in hybrid shared space called FAIRVIEW. In FAIRVIEW, the users share the information about moving objects including users themselves with Publish/Subscribe method on divided sharing space. In addition, in order to use FAIRVIEW in the ordinary environments, we proposed a QoS adaptation mechanism that can be implemented in a network with bandwidth limitation.
Through experiments, we confirmed that our method can handle hundreds of real and virtual users and thousands of objects with sufficiently fast framerate in an ordinary wireless LAN and internet environment. In addition, we confirmed FAIRVIEW could effectively offer QoS adaptation for real users by introducing strategic AP selection among a number of APs, compared with the case in which QoS adaptation is made through a single AP.
The future work is as follows. In the wireless communication environment, the delay tends to greatly fluctuate in comparison with the Internet. To such environments, we need to devise methods for realizing a short latency. In addition, e.g. in game applications, targets and obstacles are treated equally because our QoS adaptation control method treats all objects based on their values of importance. We will examine adjustment of the importance according