Cooperative Human-Centric Sensing Connectivity

Abstract

Human-centric sensing (HCS) is a new concept relevant to Internet of Things (IoT). HCS connectivity, referred to as “smart connectivity,” enables applications that are highly personalized and often time-critical. In a typical HCS scenario, there may be many hundreds of sensor stream connections, centered around the human, who would be the determining factor for the number, the purpose, the direction, and the frequency of the sensor streams. This chapter examines the concepts of HCS communications, outlines the challenges, and defines a roadmap for solutions for realizing HCS networks. This chapter is organized as follows. Section 1 introduces the concept of cooperation in information and communications technologies (ICT), and in the context of IoT. Section 2 discusses cooperation in the context of the personal and extra-personal user space and identifies the remaining open challenges and requirements for realizing the benefits of this approach to enabling more resources and services in a hyper-connected society. Section 3 defines a roadmap toward realizing simple, efficient, and trustable systems based on advanced technologies combining security, cloud, and IoT/big data technologies and outlines the challenges related to this vision. Section 4 concludes the chapter.

1. Introduction

Information and communications technologies (ICT) have progressed rapidly in this millennium for people to communicate and exchange information using multimedia (speech, video/image, text), and the same has extended to Internet of Things (IoT) and machine-to-machine (M2M) and machine-to-human communication (M2H). Propelled by the explosive growth in IoT, this trend is only going to accelerate in the years to come with new inventions in the area of human-IoT interaction technologies to deliver powerful engaging and intuitive experiences. The cyberspace of the future would rely on multisensory communications for a virtually rich personalized communication experience involving human-to-human (H2H), human-to-machine (H2M), and M2M interactions. Concepts like artificial intelligence (AI), machine learning, crowd and cloud computing, virtualization, and quantum computing (QC) will involve the human as a critical computing and interpreting node in a distributed computing architecture, pushing the concept of IoT toward a vision of Internet of Beings (IoB).

Cooperation between the human nodes is a powerful generator of streams of data that with the advances in multisensory communication will evolve to become complex and diverse in terms of the type of content, size, context, value, purpose, and so forth.

1.1. The concept of cooperation

Cooperation in communication networks is not a new concept, and its applicability and essence have evolved jointly with advances in ICT. Cooperation has been widely researched in the context of cellular and cognitive communications [1, 2, 3, 4, 5, 6] for the purpose of optimizing network performance, planning, and deployment, as a way to enhance network capacity, release extra resources, and enable the flexible use of the frequency spectrum. Cooperation in the context of sensor network management has been studied as a way for energy-efficient data transmission [7], for improving the tolerance and dependability of the network [8], for optimal path discovery [9], and for enabling trustworthy node relationships [10]. Most recently, cooperation has been explored to enable that diverse IoT applications share resources among each other in order to enhance their functionalities and improve the level of services they deliver [11, 12]. There are an ever-growing number of sensors that automatically communicate to the Internet without human intervention, and these are essential in allowing people to acquire knowledge about their environment and determine how to use it for their needs. The data generated by human-wearable/mobile devices is another key source of information about the person, enabling the personalization of the IoT applications. However, there are still the challenges of the limited spectrum availability and energy limitations to deploy practical applications fully. With the increased complexity of data, the requirements for more processing power and longer battery life have stimulated the research in the direction of energy harvesting based on cooperative transmission as a way to increase the reliability and battery power of wireless devices [13, 14].

Figure 1 shows an example of interconnected devices in a home environment cooperating for the generation of data that would be streamed in support of eHealth applications [15]. All the devices are connected to the home PC for collecting and processing the home signals to extract the care recipient’s context. The primary user interface (UI) device is a large touch screen. An optional smartphone is the secondary UI device, also facilitating the collection of data from consumer devices, such as a wearable activity/sleep monitor (i.e., wearable device) and a sleep monitor. Two other consumer devices may form a Zigbee mesh network for socket sensing and controlling, and a gateway and lamps would be used for lighting controlling. Audiovisual sensing in the living room may be facilitated by the motion sensing input devices as in Figure 1. Medical sensors would be used for monitoring vital body signs, which are connected via Bluetooth in the scenario shown in Figure 1. These are meant to provide data to the patient at given times every day or during exercising at home. The environmental monitoring sensors would communicate via Zigbee radio with custom-packaged sensing modules.

Figure 1.

Cooperation approach to sensing within a home environment.

The cooperation scenario shown in Figure 1 has been implemented as part of an eHealth platform developed in [15], and the data collected by the devices shown would be quite diverse requiring different approaches to their processing. On the one hand, there are the data extracted from the environmental signals to model the expected sensor values and understand deviations from those. Much more elaborate processing would be needed to distinguish the heart and lung signals obtained through the stethoscope and to measure the metadata related to these two organs. On the other hand, there are data associated with the voice commands (handled by the Kinect software) and face tracking and analytics. An enhanced face tracking system based on Kalman filters would also boost the processing speed performance, regardless of the visual complexity of the scenario, which could be critical in an eHealth-related scenario [16].

All the sensors in the scenario of Figure 1 are cooperating in order to transfer their data to the cloud. Because of the bulkiness of some of the collected data, they would be processed locally by different algorithms implemented in a device gateway [17]. The resulting metadata would be indexed in a local database, from which they would be streamed via a remote proxy to the cloud. This process is shown in Figure 2.

Figure 2.

Cooperation of sensing devices for information processing.

In order to deliver personalized applications, cooperation is required between the IoT sensing environment and the data management environment (i.e., the cloud) to guarantee the quality, security, capacity, and reliability of information exchange, which would be sent via proper interfaces to the cloud environment. Within the cloud, algorithms will cooperate in order to extract meaningful information from the collected data and relate it to a particular user and use case. An example of a key requirement to a successful cooperation in the above context relates to trust when collecting and storing the data.

1.2. Cooperation and interoperability

An interconnected IoT world implies diversity and innovation, both in terms of devices, applications, technologies, and user needs. Different from the single-purpose wireless system standards, the wireless technologies delivering human-centric services have the hard task of operating an ever-growing number of heterogeneous networked devices that can communicate with each other or with people or robots for satisfying very dynamic and high-level user expectations. There are still no interoperability standards in place, although international standardization bodies, such as the International Telecommunication Union (ITU) [17] and the European Telecommunications Standards Institute (ETSI) [18], have dedicated an effort within the areas of the various study and working groups. Due to the multitude of stakeholders within an IoT scenario, to reach the potential for advancements in the area would only be possible through standards that facilitate interoperability among systems and devices, provide unqualified privacy and security, address the unique needs of the developing world, and leverage existing ubiquitous technologies such as social media applications and mobile devices.

Establishing ways to put in place cooperative behavior in a heterogeneous IoT landscape opens up a new road to deal with the challenge of interoperability.

2. Cooperation in a human-centric context

Cooperation in a human-centric context provides the unique opportunity to exploit the dynamics of dependencies between the individual users, which are heterogeneous and time-varying and thus create possibilities to address multiple scenarios simultaneously.

2.1. The human node capabilities

In a smart connectivity scenario, the human becomes a critical computing and interpreting node represented by the human-centric sensing network (HCS-N) that can enable the required scalability. The HCS-N is built around the user to enable information about the personal and extra-personal space and, thus, not only provides access to resources, services, and applications [19] but also promotes binding of the person to these two spaces, thus releasing an additional information to support many tasks of daily life. Such awareness gives the possibilities for the extraction of data on demand for the purpose of releasing additional resources or creating a trustworthy network for a particular application need. The HCS-N would be able to cooperate with other human nodes (i.e., HCS-Ns), within itself and/or with the environment depending on the need. The rapid formation of the HCS-N around the human sensing node would eventually evolve into a smart ubiquitous wireless network, relying mostly on short-range technologies (e.g., Zigbee, Z-Wave, IEEE 802.22 standard, etc.) and low-consumption node devices to enable multi-hop connections between self-configurable nodes.

Many approaches have been proposed for modeling the relationships between nodes in various contexts.

In [10], a social mathematical model was developed to find suitable node partners in large-scale wireless environments of dynamic topologies and resource-constrained nodes. Building the model, a set of parameters may be designed, taking into account the characteristics of each node, alongside with its capabilities. Thus, making it possible for the nodes to present themselves, to share their specifications and services, and evaluate to the benefits of forming a temporary ad hoc network. This concept may be easily extended to model the ad hoc networks composed of human nodes with the strict requirement of complying with minimum levels of security, privacy, and trust. In addition to choosing nodes with the right capabilities and functionalities, those nodes should also be reliable and trustworthy.

Game theory has been studied extensively as an approach to enabling cooperation. Craciunescu et al. [6] proposed a novel set of functions to model the node selection process in a scenario of cooperative wireless communications. A utility function would reflect the behavior and influence that a selected node may have on the quality of the cooperation to be established. The utility function could be adjusted to reflect on the parameters defining the cooperation scenario with the overall goal to maximize the overall network performance. This approach has a strong potential for HCS-N cooperation because of the ability to also assess security and reliability of the selection. The general block chart of establishing a cooperation in an HCS-N context is shown in Figure 3.

Figure 3.

General steps during HCS-N cooperation establishment.

Reliability in the context of HCS becomes a multidimensional concept when we consider the evolving complexity of data content to be delivered through such cooperation. Immersive multisensory communications have been gaining momentum because of their strong potential to improve the quality of our life more than ever before. However, the existing technology still lacks solutions allowing for the detection, sensory analysis and evaluation methodology, coding/decoding, synchronization, transmission, and reconstruction over the ICT infrastructure of complex data associated with the olfactory, gustatory, and tactile experiences of a user. Regardless, it is an enabling trigger for ongoing groundbreaking research in preventive medical diagnostics (e.g., certain types of cancer, diabetes, etc., can be diagnosed early by means of smell) and in industrial process surveillance, waste reduction, and prognosis of critical process states. Enhancing the sense of presence/immersion, is of great benefit to various military and first responders training applications, can boost the efficacy of tele-surgery and numerous active ambient assisted living applications, and is well aligned with the cutting-edge research in augmented reality, smart mobile personal devices, sensors, and wearable devices. Replicating the sense of touch is essential in robotic technologies where certain forms of surgical operations may be needed. Touch has already become an essential feature of the success of a number of everyday devices from mobile phones to PC and tablets. The integration of the physical and chemical senses will enhance user experience and open to a large variety of application contexts ranging from education to environmental monitoring and from gaming and entertainments to healthcare. The applications would range from reliable detection of hazardous materials and low-cost and efficient environmental monitoring and food control up to advanced and noninvasive medical examinations and easily deployable threat detection systems.

Such an immersive scenario defines very strict requirements for the levels of reliability, privacy, and security that are current approaches to cooperation lack and should be performed in parallel with research on cybersecurity.

The following key requirements for cooperation in a human-centric context can be summarized:

• Self-organization at the network level

  • Security, neighbor discovery, path optimization, authentication

  • Edge management and processing

• HCS-N aware data and service management

• Automatic, controlled establishment of HCS-N cooperation

• HCS-N-aware context management

• User satisfaction

2.2. Cooperation in the intrapersonal space

A good example of an intrapersonal space cooperation is the smart body area network (S-BAN), introduced as the smallest unit of the HCS-N in [19]. An S-BAN can be built by placing sensors on the human body and/or implantable devices within the human body as part of very advanced health monitoring and stimulating systems. A communication link between an implanted transmitter inside the brain with different body parts can be established via a brain-computer interface (BCI). The BCI provides a real-time artificial communication channel between the brain and external devices such as smart phones and wearable devices.

The scenario is shown in Figure 4. Multiple transmitters are implanted within the brain matter, and the signals are transmitted wirelessly to a receiver placed externally. By using multiple implantable transmitters in the brain, one would enhance the bandwidth though the transmission data rate is reduced [20]. Multiple tags would increase the quality and quantity of the electrocorticographic (EcoG) signals, but the probability of collisions would increase; therefore, special anti-collision algorithms should also be applied. Multiple transmitters can be used for implementing many channels responsible for moving or connecting many different body parts. Various access control schemes may be applied, but TDMA- or FDMA-based approach generally is preferred. Implanted transmitters do not transmit data all the time but respond to the neural signal generations and the distribution of the signals. Therefore, an enhanced MAC protocol considering the low power consumption of transmitters, bandwidth utilization, throughput enhancement, and minimizing the transmission delay is recommended to BCI applications.

Figure 4.

Intrapersonal cooperation by BCI.

BCI application transmissions need to be wireless, low power, and energy-efficient. Data to be perceived are usually about 200 Mbps to be within the constraints of human safety and tolerance. The signal should be received and interpreted in exactly the same way as it has been transmitted, and the full understanding of the radio signal transmission through the human tissues, blood, and other matters, which is still an open challenge, is essential to guarantee the patient’s safety and signal quality. Both UHF passive RFID and UWB system could efficiently and accurately transmit up to millions of independent signals, accommodating all possible future BCI needs.

There have been substantial efforts for developing various signal propagation models for radio frequency waves from wearable and implantable devices for both narrowband and wideband communication systems. However, knowledge of the biological channel characteristics such as path loss, received signal strength, channel capacity, impulse response, and delay spread for networks of implantable devices transmitting data are very much open issues at the time of writing of this chapter despite the extensive research results available on the radio communication channel models in mobile communication networks. The general channel model for body area networks described by IEEE 802.15.6 working group has no focus on the particular part of the human body or human tissue [2]. Recently, ETSI has launched a study group on SmartBan that defined and specified a low-power physical and medium access control layers for SmartBans and studied the related coexistence issues of the radio environment in this scenario. In April 2015, ETSI TC SmartBan released its first two standard publications, i.e., technical specification (TS) 103,326 for an ultralow power PHY [21] and TS 103325 for a low complexity MAC [22]. Despite that the specifications are considered externally placed on the human body devices rather than implantable ones, it was concluded that more research is required on robustness in high-interference environment [23].

Due to the ethical difficulties arising with implantable devices, research in this area has focused on experimenting with simulated environments. An example of such an experimental framework based on the use of a wireless identification and sensing platform (WISP), which has been developed within the Wireless Laboratory at the Department of Electrical and Computer Engineering at San Diego State University is shown in Figure 5.

Figure 5.

Simulated laboratory environment for signal transmission for BCI applications.

As it is not conventionally plausible to provide power to implantable sensors by batteries for longer duration, the setup used programmable passive RFID implantable tag (WISP), which is battery-free and uses power transfer mechanism to excite its circuitry, in contrast to active RFIDs which contain batteries. Separated Ziploc bag has been used for each WISP and placed close to each other over a chemical solution of a glycerin and saltwater (emulating the human tissue and blood) barely touching it, mimicking implantable electrodes. To replicate real-world scenarios, the WISP would be implanted inside the human brain surrounded by blood and tissue fluid, and the RFID antenna is sitting below the beaker as the RFID antenna is placed outside the brain located on top of the skull. Figure 5 also shows how the RFID antenna is connected to the Impinj RFID reader through cable and how the controller (laptop) is connected to the RFID reader through an Ethernet cable. The plastic beaker with various titivations of glycerin and salt water from 1 to 4 cm allows for performing the analysis of the depth requirement of the sensor implantation. UHF RFID signals from multiple tags from various implant depths would be captured and analyzed in MATLAB in terms of received signal strength (RSSI) and signal-to-noise ratio, channel capacity, and path loss.

Important requirements to be considered during such scenario setup are the size of the sensor devices, which should be as small as possible, lightweight, and low maintenance. The BCI technology has a great potential in helping patients and hospitals by monitoring critical physiological signals and also for implantable camera-based diagnosis. Patient comfort can be greatly increased if the implantable and wearable biomedical devices are small, wireless, and with batteries that last long. The UWB and RFID radio technology can also be used in wireless endoscopy with higher-resolution images transmitted at lower power by the UHF passive RFID and UWB transmitter, also, for the diagnosis of Crohn’s disease, celiac disease, benign and cancerous tumors, ulcerative colitis, gastrointestinal diseases, Barrett’s esophagus, and several others.

3. Simple, efficient, and trustworthy convergence

Future development of applications and technologies is greatly pushed by the explosive growth of data and the growing number of interconnected devices. Because of the scale and complexity of the data to be generated, the concept of hyper-data (i.e., big) has been introduced.

In order to capitalize on the enormous business potential presented by such rapid digitalization, new interoperable architectural and platform solutions are required. The value created by collecting, communicating, coordinating, and leveraging the data from connected devices depends on evolving the key IoT technologies that relate to identification, sensors, localization, wireless and information exchange protocols, data storage and security, and their seamless convergence with advanced cloud computing concepts.

In a scenario of hyper-connectivity, supporting networks would need to meet the challenge of the generation of massive sets of streaming digital data, defined by volume, variety, veracity, velocity, and value [24]. This would require distributed application logic cloud infrastructures, able to handle the diversity of data sources and formats and to support the continuous nature of the data acquisition. Security becomes more difficult to address as it is difficult to develop a generic security strategy or model [25], also in view of the emerging “openness” of the networks. Streaming data, in addition, demands ultrafast response times from security and privacy solutions [26]. To realize a comprehensive hyper-data platform, a set of sophisticated and scalable analytic functions should be implemented at infrastructure, platform, and software level, while some of the key requirements to consider relate to response time, reliability, accuracy, and so on.

The following functionalities are minimum requirements for the support of the reliable gathering, exchange, and processing of hyper-data in order to take an intelligent real-time decision-making in relation to a given application:

• Support of 24/7 continuous collection of data from various sources and environments by means of an elastic wireless system based on a converged low power consumption of local area mesh-based communication network (e.g., based on IEEE 802.14.5) complemented by a wide-area mesh-based communication network (based on IEEE 802.22 or 802.16.n network) [27]

• Optimized single APIs capable to expose the collected data to a sophisticated in terms of prediction accuracy, sensitivity, and speed of response data processing platform

• Transformation of the exchanged data into an active decision related to a personalized user application by means of novel parallel/distributed data mining algorithms able to handle multidimensional datasets

• Protection of the collected, exchanged, and transformed data by means of on-the-fly deployments/positioning of security/privacy functions that may be enabled by a combination of software-defined networking (SDN) and network function virtualization (NFV)

• Trustworthiness of decision-making enabled by distributed ledger technology (DLT)

• Protection of data based on privacy-by-design approach

A conceptual vision for a platform able to handle the real-time analysis of large diverse and unstructured datasets acquired in a continuous manner is shown in Figure 6.

Figure 6.

Conceptual vision of a converged IoT cloud platform for hyper-data.

At the user equipment (UE) level, data would be collected from multiple (in the order of thousands and more) nodes located indoors at the user’s home, office, car, or public spaces. These nodes can be battery-operated sensor devices or Internet-enabled personal devices. Functionalities enabling lower layer security to the IoT device or as protection of the physical infrastructure should be put into place at the UE level. The collected data may be processed at the edge level for real-time decision-making while keeping the data mining components at the remote level and distinct locations (e.g., at the cloud). This approach allows for isolation of the individual microservices, and this simplifies the security-related design. The edge component may implement distributed ledger technology (DLT) to enable trustworthiness between the various platform entities. The edge component will also implement some storage capabilities and will expose the API to the collected metastreams of macro-data. Other functionalities at the edge level are the processing and indexing of the micro-data, configurations, and exchange of control data, and decisions that would trigger alarms. The API should support both message-based and pull and push communication.

By introducing an edge level, one can enable certain tasks, such as giving a fast response to a query sent by a user or securing the data transmissions between sensors and concentrators by defining a zero trust security at the entering points. Security can be implemented as a service by means of distributed networking solutions based on technologies such as software-defined networking (SDN), network function virtualization (NFV), network slicing, and evolved cloud computing paradigms. Blockchain and other distributed ledger technologies like Tangle and Hashgraph have recently drawn much attention due to their distributed nature and the possibility of not needing a Central Authority (CA) for establishing trust. Such technology can enable to implement a distributed ledger SDN that can reduce attack window times by allowing IoT forwarding devices to quickly check and download the latest flow-table rules.

The degree to enable privacy and protection of the data is expressed through the capability of a system to anonymize and pseudonymize data, which may be enabled by the privacy-by-design approach currently under standardization within ETSI and within the European Union Agency for Network and Information Security (ENISA).

An emerging trend is the increasing complexity of the data to be collected, prompted by research on how to involve all five human senses for an immersive experience. Such data can be enabled by new types of digital sensors able to replicate the human touch, smell, and taste and their synchronous integration with traditional sensors, which requires also a novel approach to extracting the contextual information carried by the multisensory data and technologies for reproducing the experience at the receiver’s end.

There are a number of open critical challenges in order to be able to deploy the above technological advances as a converged and operational platform. The main difficulty of enabling convergence comes forth from the conflicting properties of the IoT and cloud environments. The IoT environment resides at the UE level and is typically resource constrained and location specific. The remote-level functionalities have plentiful of resources and are typically location independent.

A key number of open challenges have been summarized below:

• Collecting, exchanging, and processing of hyper-data without distorting the quality of the collected data and without compromising the personal aspects of the processed data

• Processing of hyper-data in a user- and application-centered manner, without overwhelming the user with affluence of information, which requires sophisticated data mining algorithms

• Supporting real-time processes

• Enabling resources from the cloud for the IoT applications and data residing at the physical infrastructure level, which requires novel functionalities and capabilities to be deployed at the edge level

• Enabling a unison of performance response/trade-offs between the wireless and cloud infrastructure, which requires flexible software solutions

• Interoperability of solutions

Standards, regulation, and open-platform solutions are key to the deployment and commercialization of any of the above research solutions toward achieving the vision of HCS-N cooperation and realizing its social potential. Common standards are essential to achieving interoperability, which in turn is essential for reliable and smooth operation of technological solutions across various deployment scenarios and for stimulating further innovation.

Currently, there is an effort within the ITU-T to standardize Blockchain and DLT, which is also closely linked to the standardization effort on ITU-T SG16 (multimedia), ITU-T SG17 (security), and ITU-T SG20 (IoT, smart cities, and communities)​. Another standardization effort is within the IEEE with the objective to evolve the mesh function in wireless networks as 802.16n, 802.15.4g, and 802.15.4e, as an integral part of a converged IoT/cloud/big data scenario.

4. Conclusions

This chapter introduced and explored the concept of HCS-N cooperation on various levels and in the context of a converged IoT-cloud-hyper-data scenario. This vision is of global importance for releasing the potential of robust interoperable technologies to deliver business and/or societal applications. A scenario of hyper-connectivity requires a supporting platform that is open in nature and allows for its deployment under any type of legislative framework to realize in full the visions of smart cities and digital single market. The role of the human user, an ability to deal with an ever-increasing amount of sensors, smart objects and data, enhanced security and privacy, and trust are some of the key open research challenges to be resolved.

The emerging hyper-connectivity trend implies pervasive and exponentially increasing complex type of data traffic that pushes against the boundary of the power and design of current communication and information processing networks. The intensely high streams of wireless traffic and immersive data necessitate scalably to their continued growth of wireless and data management architectures, offering the required efficient processing techniques and capacity without an additional infrastructure expansion.

Acknowledgments

Some of the research work on BCI presented in this chapter was supported by award number EEC-1028725 from the National Science Foundation (NSF). Authors would like to thank Dr. Joshua R. Smith and Dr. Matt Reynolds, professors of electrical engineering at the University of Washington, Seattle, for their continuous support and assistance in the research.

Acronyms and abbreviations

ICT	Information and communications technologies
IoT	Internet of Things
M2M	machine to machine
M2H	machine to human
H2H	human to human
SDN	software-defined networking
NVF	network virtualization function
HCS-N	human-centric sensing network
BCI	brain computer interfaces