Communications of CONSORTS-S.
To recognize a context of a user, it is important to know a physical condition of the user and a status of his/her surrounding environment. For example, to maintain a healthcare condition of an elderly person, his/her care workers require monitoring not only his/her physiological conditions but also physical statuses of his/her surroundings, such as room temperatures. However, most approaches to the context recognition have been focusing on either environmental-centric recognition or user-centric recognition.
In ubiquitous computing researches, most approaches have focused on environment-centric recognition mechanism; they specifically examine how to recognize user’s activities by sensors embedded in an environment (Garlan et al., 2002; Roman et al., 2001). The recognition includes location of the users. On the other hand, in research fields of wearable mobile computing, most approaches have focused on user-centric recognition mechanism; they specifically examine activity recognition by using sensors worn by the users, such as wristwatch-type accelerometers (Van Laerhoven et al., 2008; Clarkson et al., 2000). Hence, context-aware systems that use both wearable body sensors and environmental sensors have been few.
In this study, we specifically address context-aware services by using both wearable body sensors and surrounding wireless sensor networks, and propose a concept of mobile sensor fusion platform. Mobile sensor fusion platform is a new concept of a mobile service platform coordinating mobile phones and wireless sensor devices, such as wearable body sensors and environmental sensor networks. It will provide the following facilities to recognize user’s contexts: 1) communicating with wearable body sensors and surrounding wireless sensor devices via mobile phone, 2) analyzing and fusing the sensed data by cooperating with sensor middleware, which manages sensed data on a remote server and 3) providing context-aware services for users by using mobile phones, such as remote monitoring of physiological status. In this chapter, we propose our vision of mobile sensor fusion platform. First, we describe services, requirements, and two types of the fusion architecture: “mobile sensing architecture” and “stable sensing architecture.” Then, we propose a prototype platform, called CONSORTS-S ( Sashima et al., 2008 ), of the mobile sensing architecture. As an exemplar service of the platform, we show a mobile healthcare service being aware of user’s physical conditions such as heartbeat, posture, and movement based on analysis of physiological signals (e.g., electrocardiograph, thermometer, and 3-axis accelerometer). The service can be also aware of environmental status (e.g., temperature). Then, an indoor temperature monitoring system, an example of the sensor fusion and analysis based on stable sensing architecture, is described. Finally, we discuss possibilities and future work of the platform.
2. Related work
In ubiquitous computing studies, a few works specifically address the context-awareness of mobile users. For example, Solar (Chen & Kotz, 2002) is a platform for context-aware mobile applications. Although the work provides a sophisticated programming model to access local resources seamlessly, no application using sensor devices has been shown.
Some services for mobile users have been studied in mobile computing for many years. However, studies examining the use of everyday mobile phones with sensor technologies are few. Several studies in the field of ubiquitous computing have been undertaken to detect a user’s contexts using sensor devices embedded in mobile phones (Lester et al., 2006; Kawahara et al., 2007). In addition, various kinds of services on top of the mobile phone infrastructure have drawn attention from researches and applications. A typical service is providing healthcare service by using mobile phones (Leijdekkers & Gay, 2006; Oliver & Flores-Mangas, 2006). Contrasted against these studies, our work specifically examines the cooperative architecture that comprises independent components, such as wireless sensors, mobile sensor routers, mobile phones, and sensor middleware.
Recently, the concepts of people-centric sensing (Campbell et al., 2006; Miluzzo et al., 2008) and community sensing (Krause et al., 2008) were proposed. People-centric sensing is designed to realize large-scale, general-purpose sensor networks for the general public (e.g. consumers). Community sensing offers mechanisms for sharing data from privately held sensors, such as GPS devices. Although our platform shares similar concepts related to personal information devices (e.g. mobile phone) as sensor nodes, we specifically address supporting users to access the sensed data derived from surrounding wireless sensor networks and provide personalize services based on the data. To motivate such a concept, we present an implementation of healthcare services using commercially available mobile phones.
3. Mobile sensor fusion platform for context-aware services
3.1. Service scenarios
In this study, we have been developing applications to maintain and promote one’s health condition by monitoring both environmental information (e.g., room temperature) and physiological information (e.g., heartbeat, posture and skin temperature) from surrounding sensors. As such services of mobile sensor fusion platform, we have considered the following application scenarios.
Mobile and remote healthcare service for elderly people: In this scenario, the service system detects and prevents health troubles of elderly person caused by improper room temperature, such as heatstroke. The users wear physiological sensors and carry mobile phones. The service system always monitors a user’s health conditions (e.g., heartbeat, posture, and locomotion) based on the sensed data. In addition, the mobile phone communicates with environmental sensors to monitor environmental conditions (e.g., room temperature). If it detects the conditions are harmful for users, it notifies them.
Self-monitoring service on physical exercise: When a person is doing physical exercise, such as aerobics, running, and sports, it is important to monitor his/her physical conditions for effective and safe trainings. In this scenario, service users wear physiological sensors and carry mobile phones. The service system analyzes a user’s health conditions (e.g., heartbeat and locomotion) based on the sensed data. In addition, the mobile phone communicates with environmental sensors to monitor environmental conditions (e.g., air pollution level and temperature). If it detects the conditions are harmful for users, it notifies them.
Remote health monitoring service in emergency situation: When an emergency, such as a fire emergency, occurs in an indoor environment, it is important both to monitor a change of the emergency situation and to know physical conditions of residents in the environment. In this scenario, service users (e.g., residents) wear physiological sensors and carry mobile phones. The service system analyzes a user’s health conditions (e.g., heartbeat and posture) based on the sensed data. The mobile phone communicates with environmental sensors to monitor environmental conditions (e.g., air pollution level and temperature). If it detects the conditions are harmful for users, it notifies a rescue team of present user’s situations.
To realize the above service scenarios, we assume that a mobile sensor fusion platform is required to suffice for the following requirements.
Providing mobile services on user’s cellular phone: Because user’s everyday mobile phone is used for providing context-aware services, the platform is implemented on top of the mobile phones popularly used by ordinal people. It uses communication facilities of the phone to provide monitoring and notification services. It also uses a graphical user interface of the phone to shows analysis results, such as air pollution levels.
Wireless communication with surrounding sensors: The platform can collect and aggregate sensing data by using wireless communication. It communicates with wearable body sensors and surrounding sensor networks. It is required to speak common communication protocols to connect to various sensor devices in an ad-hoc manner.
Distributed Analysis of sensed data: The platform fuses collected sensor data and analyzes them to recognize users’ contexts, and provides services based on the analytic results. Because complex analysis, such as motion recognition, consumes much computation resources which may not be available at a mobile phone, the complex analysis can be performed at a remote server on the Internet.
Management of sensed data: The platform manages the sensor data received from the sensor networks. Because some applications are required to analyze sensed data of the past hours/days, the service can store the sensed data on storage of the mobile phones, or on remote database systems.
Considering mobile sensor fusion platform that suffices for the above requirements, we propose the following two types of architecture. We require both types of the architecture to realize our vision, and have been developing them.
Mobile sensing architecture: The architecture is shown in Fig. 1. This architecture uses a mobile phone carried by a user to communicate with surrounding environmental sensors. A mobile router attached to the mobile phone communicates with wireless sensors in ad-hoc manner. Although the router and phone are required to perform ad-hoc communication mechanisms, interaction styles of the sensors are intuitive for the user; the proximity based communication architecture is not required to manage user’s locations. Under the vision of the architecture, we have been developing CONSORTS-S, a mobile sensing platform ( Sashima et al., 2008 ). The CONSORTS-S platform is designed to confirm the validity of this architecture based on mobile phone popularly used in Japan and its network architecture (e.g., 3G network). Healthcare services for mobile users by accessing surrounding wireless sensors are implemented on the platform. The outline of the platform is described at section 4.
Stable sensing architecture: The architecture is shown in Fig. 2. In this architecture, sensor middleware always communicates with predetermined environmental sensors, collects the sensed data, and stores them with locations where the data are sensed. To provide a service using the sensed data from user’s surrounding sensors, sensor middleware must know user’s location to retrieve the data. Hence, sensor middleware is required to manage user’s location by using some location systems like GPS. Because the architecture does not use the ad-hoc connection to the wireless environmental sensors, communication mechanism can be simple. Under the vision of the architecture, sensor middleware, SENSORD ( Sashima et al., 2008 ) has been developed to confirm the validity of this spatial sensor data management and analysis, namely spatial sensor fusion. An indoor temperature monitoring system has been implemented with the middleware ( Sashima et al., 2008 ). Currently, sensing information of wearable sensors is not integrated into the system. Outline of the system is described at section 6.
4. Mobile sensing platform: CONSORTS-S
The CONSORTS-S platform uses two additional components that work with mobile phones: mobile sensor router and sensor middleware. Fig. 3 shows the architecture of the platform. A mobile sensor router is attached to a mobile phone for the communication. Sensor middleware on a remote server performs management and fusion and analysis of sensed data coordinating with the software in the mobile phone.
4.1. Wireless sensors
We assume wireless sensor networks of two types communicating with CONSORTS-S: embedded type sensors stayed in an environment and mobile type sensors worn or carried by a user. The sensor networks are assumed to communicate with other nodes in a peer-to-peer manner. Sensors of several types are assumed: thermometers, hygrometers, vision systems, microphone array systems, and so on. Because the vision systems and microphone array systems might produce enormous raw sensor data continuously, we assume that the mobile phones receive not raw sensor data from the sensor systems but rather binary information, such as exists/not-exists and passed/not-passed, which is pre-processed at the sensor systems.
Wireless sensor units communicate with CONSORTS-S platform via an original 2.4 GHz ISM band communication. Currently, a wireless sensor unit - (see Fig. 4) is ready for such communication. The unit is a small wireless sensor unit that includes built-in sensors of three kinds: an electrocardiograph (potentiometer), thermometer, and 3-axis accelerometer. It is just a transmitter of sensed data; it has no function to receive communication signals. Maximum sensing rates of sensors are 204 Hz; the maximum wireless communication rate is about 1 Mbps. Its communication range is about 15 m.
The sensor unit is mainly designed for medical uses. However, it can be used as embedded type sensor (e.g., thermometer) stayed in an environment. We use the sensor for monitoring biological information and environmental information.
4.2. Mobile sensor router
We have developed a board for communicating with wireless sensor networks, called a mobile sensor router. The router is a wireless communication board attached to a serial port (UART) of the mobile phone (see Fig. 5).
A mobile sensor router has the following functions:
Communicating with sensor networks The router must understand a communication protocol of the sensor networks to collect sensor data. In the current implementation, we used an original 2.4 GHz ISM-band communication.
Reducing the number of sensed data Although the router can receive full rates of the sensor data stream from the wireless biosensor described before, the mobile phone cannot receive such a data stream. Therefore, to compensate the limited computational ability of the mobile phone, the router sends not raw data received from the sensors but their reduced data at the router.
Communicating with a mobile phone The router is connected to a serial port of the mobile phone for communicated through the wire. The communication rate is 115000 bps. We have developed a content level communication protocol between them, called CONSORTS-S-Binary Protocol. It is a lightweight protocol for controlling the router function and obtaining sensed data.
4.3. Mobile phone
Mobile phones used popularly in Japan have a runtime environment in which a program downloaded from Web sites can be executed. We use a mobile phone - that can execute a Java program (J2ME CLDC edition with extensional APIs for accessing native functions). A CONSORTS-S program running at the runtime environment, and communicates to a mobile sensor router. It collects sensed data and sends the data to the remote web server. The program also provides a user interface of CONSORTS-S that controls its behaviors: starting (or stopping), setting up parameters, and so on. The interface shows contents received from the remote web server.
The phone continuously sends reduced sensor data to sensor middleware on a remote server for analyzing them. The phone sends data using W-CDMA wireless communication (Max 384 kpbs). A mobile phone of CONSORTS-S must make connections to the remote server to send sensed data to a sensor middleware on a remote server. We set up a web/application server for downloading the programs and providing proxy services of the sensor middleware. We adopt Tomcat 5.5 - as the web/application server.
We have developed a communication protocol between mobile phone and web server, called CONSORTS-S-iHTTP Protocol. It is a lightweight protocol implemented on HTTP protocol. Because HTTP protocol is a connectionless protocol, each message of the CONSORTS-S-iHTTP protocol requires including some identification signals of users, their mobile phones, and sessions. In the CONSORTS-S-iHTTP protocol, a mobile phone is first authorized with login name, password, and subscriber ID, and receives a session cookie. Consequently, the mobile phone sends reduced sensor data with the subscriber ID and the session cookie to be identified by the server. Using this scheme, multiple mobile phones can access the same web server simultaneously. Table 1 summarizes communications of CONSORTS-S.
|Sensor <="/ Router||Wireless (2.4 GHz)||Original||bps (Max.)|
|Router <="/ Mobile Phone||Serial (UART)||CONSORTS-S Binary||115K bps (Max)|
|Mobile Phone <="/ Web Server||W-CDMA + IP||CONSORTS-S iHTTP||7.2M bps (Max)|
The phone shows analytic results of the sensed data and messages (e.g., recommendations) to users. The graphical user interface depends on implementation of the application and services; it does not depend on CONSORTS-S. Therefore, we describe details of the user interface with an example of mobile sensing services in section 5.
4.4. Sensor middleware: SENSORD
We adopt sensor event-driven service coordination middleware (SENSORD) (Sashima et al., 2006) as sensor middleware of the platform. A servlet program on the application server includes a SENSORD core module. It processes sensed data received on HTTP protocol. It analyses them by calling SENSORD API, such as FFT API. A SENSORD module stores sensed data into an in-memory data container to achieve fast and complex analysis (see Fig. 6). The analysis results of the module are sent back to the mobile phone as a HTTP response. The response is represented as a mobile-phone-friendly format, such as a HTML document. Because this transaction is a synchronous process, the HTTP communication process of the mobile phone waits to receive the results. The sensed data is stored in a relational database -.
Sensor middleware of CONSORTS-S performs several signal processing and classification processes using machine learning techniques, such as Fast Fourier Transform (FFT), support vector machine (SVM), and nearest neighbor learning. Using such algorithms, the sensed data are analyzed statistically or classified into some qualitative categories. The analysis results are based on remote sensing data from users’ surroundings.
The module can store and retrieve spatial information (or geometric map), specifically locations of objects, humans, and sensor devices. Sensed data of CONSORTS-S is stored with spatial information: locations of the sensor devices. The locations are represented as using a widely used coordinate system, WGS84 latitude and longitude. The WGS84 coordinate system is used by GPS devices in mobile phones. Consequently, users’ locations and their sensor data are managed easily on the CONSORTS-S platform.
5. Mobile healthcare services
In this study, we propose a services scenario to maintain and promote one’s health condition by monitoring both environmental information (e.g. room temperature) and biological information (e.g., heartbeat, posture and skin temperature) from wireless sensors. We have implemented a prototype of the healthcare service considering the following scenarios ( Sashima et al., 2008 ).
Particularly, elderly people need preparation for unexpected health troubles. In this scenario, the service detects and prevents health troubles of older people, such as falling down and heart failure. Service users wear wearable biological sensors and carry mobile phones. The service system analyzes a user’s health conditions (e.g., heartbeat, posture, and locomotion) based on the sensed data. Then it notifies the user. Doctors and family members can monitor the conditions remotely. In addition, the mobile phone communicates with environmental sensors to monitor whether environmental conditions (e.g., room temperature) are not harmful for users. The service supports a user’s health based on both environmental information and biological information obtained from wireless sensors.
Fig. 7 depicts an outline of a prototype service that we have implemented to address the vision implied by the healthcare scenarios described above. It monitors a user’s health conditions and room temperatures using popular 3G phones and wireless sensors.
A sensor to monitor physiological signals must be attached to a user’s chest by sticking electrodes of the sensor on tightly with a peel-off adhesive seal. Once it is attached to a user’s chest, it can detect the inclination and movement of the upper half of a user’s body using a 3-axis accelerometer. On the other hand, the sensor to monitor room temperature is set in a room. The temperature sensor is assumed to be set in each room where users will stay. The user can know the room temperature by communicating with the sensor.
The monitoring service has the following modes: graph mode, multi sensor mode, and remote monitor mode. Screen images of the modes are shown in Fig. 8.
Sensed physiological signals are sent to the mobile phone through the mobile sensor router described in section 4.2. In graph mode, the mobile phone shows a graph of the physiological signals, such as electrocardiography (see the left image of Fig. 8).
In multi sensor mode, environmental information (e.g. room temperature) is shown (see the middle image of Fig. 8). The temperature information is obtained using the environmental sensors that can be communicated with the mobile phones. Locations and unique IDs of the sensors are also shown (e.g. room (abee0811)). Because the system communicates with one sensor at a time, it periodically switches from a communication channel for a sensor to other. In the current implementation, the system has a list of sensor devices (device id and location) that might be communicated with. This process interrupts data processing of the graph modes. Therefore, physiological signals are not shown when this mode is working.
In remote monitor mode, the monitoring software on a mobile phone of a remote user periodically accesses the sensor middleware and receives current analysis results of cared-person’s physiological signals, such as the posture (see the right image of Fig. 8). In this mode, it analyzes sensing signals of electrocardiograph and calculates cared-person’s heart rates as beat per minute (BPM). The heart-rates are calculated by Fast Fourier Transform (FFT) of finite length time series of sensed signals. In the current implementation, the length of the time series is about 16 seconds (128 samples).
Moreover, it shows user’s the movement recognition, which classifies the signals into some qualitative statuses, for example, running, walking, standing-still, etc. In the movement recognition, the steps are recognized by calculating FFT of finite length time series of sensed data of the y-axis accelerometer. In the current implementation, the length of the time series is 8 seconds (64 samples). It also classifies user’s statuses into three categories: staying, walking, running based on the results of FFT. When user’s status is “walking” or “running,” it also shows an average speed of user’s movements as Steps Per Minute (SPM).
To detect cared-person’s posture (e.g., standing-still, facing downward, and facing upward), it calculates the inclination of the upper half of user’s body based on the sensed signals of the 3-axis accelerometer. The classification uses simple rule-based inference on the inclination.
In this section, we have described a prototype system of the mobile healthcare service by using mobile phones with surrounding wireless sensor networks. The demonstration has shown a first step of a mobile sensor fusion that enables mobile users to collect and publish their favorite sensing information anywhere and anytime. Although we have developed CONSORTS-S and confirmed a basic idea of mobile sensor fusion, fusion and analysis of sensed data have not been shown. In next section, we will show an example of such analysis by using SENSORD, a key component of CONSORTS-S. Although, the service does not use wireless sensors, analysis of collected sensor data is performed in an analogous fashion.
6. Indoor temperature monitoring system
An indoor temperature monitoring system has been developed under the vision of stable sensing architecture ( Sashima et al., 2008 ). The experimental area of the floor is about 700 square meters. It analyzes sensed data of thermometers - placed in a floor of our laboratory building, and graphically shows distributions of temperature. Distributions of temperature are analyzed and monitored 24 hours a day.
Fig. 9 displays an overview of the configuration of 20 thermometers used for the system. The thermometer has a wired network interface (TCP/IP) to send sensed data to the middleware. A black box with a number shows a location of the sensor. A yellow area shows a monitoring area by analyzing sensor data.
Because the thermometers are irregularly placed on the floor, it is required to apply various types of interpolation functions in analyzing sensed data to achieve an appropriate 2- dimensional distribution of temperature. In addition, the interpolation algorithms use complex, numerical computing techniques so that the implementations are not easy. Thus, we have developed the system based on SENSORD/Stat (Sashima, Ikeda, Inoue & Kurumatani, 2008). SENSORD/Stat is an extension of SENSORD. In the SENSORD/Stat system, SENSORD cooperatively works with a statistical and numerical analysis environment, called R (Ihaka & Gentleman, 1996). Because the R environment provides various statistical analysis packages, users can try to apply them in the sensor data analyses. By using the SENSORD/Stat system, application developers can access to both statistical and numerical analyses of R and spatiotemporal sensor data management of SENSORD. Hence, spatial sensor fusion analysis can be performed on the system. Fig. 10 shows an outline of the analysis.
In the analyses, we have selected an interpolation function package, named akima  -. It is a package for a linear or cubic spline interpolation for irregularly gridded data (Akima, 1996). Applying various add-on packages of R, we have easily developed the program that calculates appropriate 2-dimensional distributions of temperature. Fig. 11 shows an example of the distributions.
7. Discussion and future work
In current implementation of the healthcare service based on mobile sensing architecture, sensor discovery and communication protocols are predefined. How it can communicate with environmental sensors in an ad-hoc manner is an important issue. Users do not stay in their homes. They visit various places, such as commercial facilities. How can users access environmental sensors in such public spaces? A mechanism to discover and communicate with the environmental sensors is necessary to realize our service scenario. Developing the mobile sensor router on top of other standardized communication stacks, such as ZigBee -, might be a solution for this issue.
In current implementation of the indoor temperature monitoring system based on stable sensing architecture, sensing information obtained by mobile users is not integrated into the system. To integrate the information into the monitoring system based on users’ locations, we plan to use indoor human location tracking technologies, such as RFID, with the system. However, location tracking of humans is apt to make a violation of privacy. Hence, secure and privacy preserving data analysis is a next challenge of mobile sensor fusion.
Combination of environmental information and biological information obtained by wireless sensors is useful for non-healthcare applications. For example, energy management of home appliances according to a user’s physiological information might be an interesting application. In the current implementation, the mobile sensor router does not communicate with multiple sensors simultaneously. Hence, the monitoring process of heartbeat can not work with the monitoring process of room temperatures. We plan to develop a new mobile sensor router that can communicate with multiple sensors to realize our scenario.
In this chapter, a vision of mobile sensor fusion platform has been proposed. First, two types of the fusion architecture: “mobile sensing architecture” and “stable sensing architecture” have been described. Then, a prototype platform of the mobile sensing architecture has been proposed. As an exemplar service of the prototype, a mobile healthcare service being aware of user’s physical conditions and environmental status has been shown. An indoor temperature monitoring system as an example of the sensor fusion and analysis base on stable sensing architecture also has been shown.
- MESI RF-ECG http://www.mmdevice.co.jp/product_all.html#p1 (in Japanese)
- NTT-docomo FOMA N903i http://www.nttdocomo.co.jp/support/utilization/ product/n903i/index.html (in Japanese)
- T&D TR72W: http://www.tandd.com/product/tr7w/index.html