States of growth.
Internet of things (IoT) artifacts form systems where touchscreen and speech interaction is the norm. As IoT systems are inherently open (artifacts can be added or removed, software can be updated), we observe that the natural state of an IoT system is changed, “growth.” This chapter describes a designerly experiment exploring how to design for embodied and rich interaction in these “growing” IoT systems. We present four design cases showcasing four approaches to the design challenge: a hybrid, a modular, a shape changing, and a service approach. We describe and appraise the four approaches and discuss insights from the designerly experiment. We conclude that it is indeed possible to design for embodied and rich interaction in “growing” IoT systems and see our work as a first step toward diversifying IoT interaction styles.
- rich interaction
- embodied interaction
- growing systems
- internet of things
The design of socio-technical systems receives an increasing amount of attention. Systems concepts that have been explored in literature over the past 2 decades (e.g., ubiquitous computing , pervasive computing , ambient intelligence ) are now brought to the market as the “internet of things” (IoT) .
The primary interest of this chapter is the human-product interaction within IoT systems in home and we feel inspired by research areas like tangible interaction , embodied interaction , or rich interaction . The academic community gives a range of arguments for the value of tangible, embodied, and rich interaction: Ullmer  argues that we have a familiarity with the physical world around us that we can capitalize on when making interfaces tangible. The field of embodied cognition adds the argument that we make sense of the world and its complexity through the physicality of it and the situatedness of our actions , for the latter also see . Hummels and van der Helm  makes the argument for resonant interaction and argues that different people prefer (resonate with) different interaction styles, including tangible interaction styles. Finally, van Campenhout  argues that the value of tangible, embodied, and rich interaction must be sought in esthetics (the esthetics of the third stand). This esthetic cannot be either in the physical or in the digital alone; it only exists in the coupling of the two. Making physicality an important prerequisite for esthetics of interaction.
Despite this compelling rationale, these interaction styles are not landing in industry; neither in the interactive products on the market, nor in home IoT that is dominated by (touch)screen and speech interaction (e.g., Philips Hue , Reality Editor , IFTTT , Amazon Echo ).
Where academics and industry are currently operating in their own world, we feel that home IoT can be crucial in bridging the gap between them. IoT revolves around “things”: artifacts that stand in the tradition of the objects that we used before the world was interactive (e.g., coffee mugs, kitchen appliances, lamps, or stereo equipment). These artifacts inherit interaction styles from this tradition; interaction styles that are familiar, situated, and that resonate with specific people because of their esthetics in interaction. As electronic “intelligence” pervades our living rooms through home IoT, we can capitalize on these qualities and explore how tangible, embodied, and rich interaction can be made to fit these IoT systems. At the same time, we consider home IoT to be an inherently complex phenomenon and see opportunity to help people make sense of it by adopting a more embodied and rich interaction style.
This leads us to explore what it takes to design for embodied and rich interaction for home IoT. In what follows, we introduce a designerly experiment and present a student design challenge that was setup to explore embodied and rich interaction in home IoT. We present four different design approaches to solve the design challenge. After discussing the approaches, the chapter concludes with a brief look into future work.
2. A designerly experiment
In this section, we introduce our designerly experiment and its theoretical backdrop.
2.1. Internet of things as a growing system
At present, the internet of things  is promising us a connected future, where IoT artifacts produced by different manufacturers form networks of products in the home, IoT systems. There are ongoing efforts both in academia (e.g., Semantic connections , Reality Editor ) and on the market (e.g., Home kit , IFTTT ) to truly make the connected future in home IoT, a reality, but these are not yet fully adopted.
We see IoT systems as inherently open (IoT artifacts can be added or taken away and software can be updated) and we observe that the natural state of an IoT system is a change (we label this as “growth” indicating that home IoT grows to match the preference of its user). This means that the functionality in IoT systems is not stable. The consequence of adding or updating IoT artifacts means that on a system level, functionality can emerge in unpredictable ways because of how the IoT artifacts are combined by their users. We feel that when IoT systems become truly connected, forming actual “growing” systems, emergent functionality becomes one of its more fascinating features. At the same time, emergent functionality will be a challenge to design for. This has consequences for the design process [19, 20] and the interaction solutions in IoT.
2.2. Embodied and rich interaction
We position the design challenge in embodied and rich interaction. Embodied interaction is defined as: “the creation, manipulation, and sharing of meaning through engaged interaction with artifacts” [6,p. 126]. Embodied interaction is a perspective on interaction that emphasizes that one can meaningfully interact with the world through doing rather than through knowing, resonating strongly with Gibson’s work on ecological perception , and the research area of tangible interaction (e.g., ). Rich interaction  shares this theoretical basis and can be regarded as a product centric exponent of embodied interaction.
The rich interaction framework is aimed at designing for meaningful interaction by respecting all human skills (i.e., perceptual-motor, cognitive, and emotional skills ) and by designing for a unity of form, interaction, and function. This results in strong-specific  interactive products that express in their form what can be done with them, both from an action point of view and from a functional point of view: they give feed-forward. Below we give two examples.
2.2.1. Rich actions camera
The rich actions camera features rich action possibilities that express their functionality in form and interaction (Figure 1). For example, to take a picture with this camera, the user pushes the “trigger” at the side of the screen. The form of the “trigger” invites the thumb to push. The trigger is also shaped such that it keeps the screen in place. When the “trigger” is pushed, it releases the screen. The screen flips away from the lens on a hinge and a photo is taken (also see ).
2.2.2. Third stand payment terminal
This payment terminal (Figure 2) is designed to bring digital monetary transactions back to the physical. The vendor (right hand side) enters the price of a purchase and pushes the “traveler” (i.e., the construction with the round screen) toward the customer side. In turn, the customer places a payment token (not shown) on the physical drawer and pushes it into the terminal, simultaneously pushing some of his (digital) money toward the vendor. The traveler accepts the digital money and (physically) moves back to the vendor side [25, 26]. The payment terminal expresses in its form and behavior how the transaction plays out and invites for different actions during the process of payment.
2.3. Research question and research aim
When we look at embodied and rich interaction through the lens of “growing” IoT systems, they seem to be incompatible at first glance. As already mentioned, there are preciously few (if any) examples of IoT artifacts that offer an embodied and rich interaction style. Arguably, redesigning the IoT artifacts could easily solve this. But this is not where the incompatibility and hence the complexity lies. As argued above, we consider truly connected IoT systems to be “growing” systems where functionality is inherently dynamic and emergent. The result of this is that functionality is undetermined at the time of designing, the connected artifacts that live within IoT systems. This is an ill fit with the rich interaction paradigm that aims for meaningful interaction by expressing functionality in form and interaction; it is problematic to express in form and interaction that what is not known yet. To understand this better, we formulated two research questions. Our first research question approaches this “looking forward” and asks: “how to design for embodied and rich interaction in ‘growing’ home IoT systems.” Our second question “looks back” and asks: “how does the concept of rich interaction needs to change when applied to ‘growing’ IoT systems.”
This research aims to inform interaction design within a home IoT context by example and reflection. We intent to find approaches to this design challenge but also learn from these approaches how rich interaction itself changes.
2.4. Design challenge
We chose an IoT media center as the context for our design challenge (Figure 3). This media center was a small media server with a range of input and output devices in a home context. At present, each (software or hardware) component that is added comes with its own dedicated remote control. These remote controls were to be replaced by a new “growing” remote control offering an embodied and rich interaction style. The “growing” media center is easily recognized as an IoT system and has a clearly visible way of how it can “grow”: by adding new (software or hardware) components. Which components are added was up to the students. Finally, the challenge sets the stage for functionality to emerge as new components combine with existing parts of the media center to create functionality that is neither present in the existing components nor in the new components alone.
3. Four approaches to design for growth
Starting in 2010 and ending in 2014, we gave our students the challenge to design a “growing” embodied and rich interface for a media center. The semester long design challenge was open to industrial design students doing their final bachelor project, first year master project and final master project. The challenge yielded over 20 cases that we have analyzed for differences and similarities. We found four patterns with clear differences regarding approach.
In what follows, we present these patterns by means of four design cases. We chose to present recent cases that demonstrated the patterns best.
3.1. Hybrid approach
Hybrid solutions are perhaps the simplest route to success in this design challenge. It comprises combinations of screen-based interaction with rich action possibilities. Typical for this approach is that it employs a screen to deal with the aspects of “growth” and change and that it makes use of rich action possibilities for the aspects of the interface that are not subject to change.
3.1.1. Ball remote
Joep Elderman showed a design for a remote control for a media center that features a ball-shaped token that is placed on top of a horizontally placed screen (Figure 4a). The position of the token is tracked. The user can move the ball token, but it can also move autonomously. On top of the screen, a template is mounted that has two indents and a round track where the ball can be placed to access the functionality. The ball can be loaded with multi-media content and will playback this content when placed in the circular track. Playhead control is achieved by manually moving the ball in the track (after which it moves autonomously) or by stopping the ball (Figure 4d). When the ball is taken from the circular track, the whole surface of the remote control shows GUI elements, by placing the ball on these elements a menu structure is entered to access more complex or emergent functionality (Figure 4b, c).
3.1.2. Appraising the hybrid approach
The benefit of this approach is that it offloads the complexity of dealing with emergent functionality to the screen and to conventional menu structures. These are good at adapting to new content or incorporating new functionality . On the other hand, the potential of embodied and rich interaction to offer a more direct and less mediated interaction style that emphasizes man as a whole rather than just his cognitive skills is only partially met. The promise of a physical interaction style literally giving handles on the complexity of systems is not completely fulfilled. In this approach, it is crucial that the coupling between the physical action possibilities and that what happens on the screen is designed to fit each other specifically. Generic menu structures need to be avoided.
3.2. Modular approach
An approach that is very prominent in the work of our students is the use of modularity. By creating inter-connectable interactive modules, each offering dedicated rich interaction, remote controls can be “composed” that “grow” together with the system that they live in. This approach takes cues from early work on modular interfaces like DataTiles  or more recent work on generic and modular tangibles [29, 30].
3.2.1. Tiled remote control
Jordy Rooijakkers created a series of interactive tiles (Figure 5a, b) that can be used to layout a specific multi-media system and through which multi-media content can be moved by means of an interactive viewer/selector tile that sits on top of the other tiles (Figure 5c). The interactive viewer/selector tile has different behavior depending on which tile it rests. It offers a rotary control at the side (Figure 5d) that can be used to make a media selection. If desired, this selection can be “grasped” and brought to a different tile by squeezing it (Figure 5b). Adding new tiles to the remote control can open up new media center components.
3.2.2. Appraising the modular approach
The modular approach is capable of responding to the “growth” in “growing” IoT systems: when the media center gets a new component, this can be matched by adding a new component to the remote control and it can do this while offering embodied and rich interaction where each component of the remote control can be designed to express its function in its form and interaction. Still, there is one challenge that it does not solve: the challenge of dealing with emergent functionality. Interactive modules open up dedicated functionality of specific components. Emergent functionality is in the combination of components and not in a specific component and that makes it difficult to grasp emergent functionality by means of a modular approach.
3.3. Shape changing approach
A promising approach is that of shape change. Where the modular approach “grows” through addition, the shape change approach is self-contained and changes shape under computational control: an interactive node could present new, rich action possibilities in response to “growth” of the systems. That our students are not alone in seeing this is clear from the abundance of literature on the subject (e.g., [31–33]).
3.3.1. Adaptive remote control
Paul van Beek designed an adaptive remote control (Figure 6a) for controlling a video on demand system. His remote control offered basic interactions for navigating a screen-based menu structure in its default shape (Figure 6b). When more specific controls were needed, the remote control responded by sliding open and offering more (physical) controls (Figure 6c–e), which controls and the amount of controls visible on the slider could be varied as a response to the activity of the user by sliding the remote further open or closed.
3.3.2. Appraising the shape changing approach
If we share Ishii’s vision on Perfect Red , a programmable material that can take any shape, shape change potentially solves the “growing” IoT systems challenge. Yet, at present technology has not advanced to the point that matter is truly under computational command. Till that moment, the shape changing approach relies on mechanical solutions. These mechanical solutions toward “growing” IoT systems share a problem with the modular approach in how to deal with emergent functionality: it is difficult to design for shape change if it is not known what the desired changed shape needs to be. That is not to say that in constrained situations, this approach cannot be of value: shape changing controls can be designed that do offer changing forms to express changing functionality and accommodate changing interactions.
3.4. Service approach
A final approach to deal with rich interaction in “growing” systems that we present is a service approach. This approach responds to the challenge by means of updating the interactive nodes by replacing its interaction surfaces in short cycles as a service, creating opportunities for “hyper-personalization” of embodied and rich interfaces, somewhat similar to commercial approaches like NikeID  and such. Strictly speaking, it is a variation of the modular approach, but it offers much more integrated solutions and hence is singled out as a separate approach.
3.4.1. Generated remote control
Tom Fejèr presented a remote control for the “growing” media center that revolved around personal media use (Figure 7a, b). He proposed a design that would follow a user’s preferences in media, giving direct access to his favorite songs or series. The rationale was that as media use changed, the remote control would need to change in shape and action possibilities (Figure 7c). This would be done by printing new, parametrically generated shapes on a regular basis. His remote control featured a touch sensitive core where the 3D–printed shells were placed over. While the example shows a faceted touch surface, the premise was that also rich interfaces could be algorithmically generated.
3.4.2. Appraising the service approach
When interactive nodes can be updated by replacing its interaction surfaces with newly designed interaction surface, the “growing” IoT systems challenge is solved as a unity of form, interaction, and function can be guaranteed. The service approach has a solution for the changing of form and interaction to open new functionality but the challenge of designing these interaction surfaces remains. The example shows a parametrical implementation and these have similar problems as the shape change approach: the software needs to be designed to generate expressive geometry without knowledge of what sort of expressivity is needed. If the service approach follows the template of the example (i.e., a high-tech core with customizable interaction surfaces), the question also remains how to design the “interface” between the technology and the interaction surfaces flexible enough that the remote as a whole can deal with the dynamics of “growing” IoT systems.
Here, we take the time to look back at the designerly experiment and reflect on our research questions: (1) “how to design for embodied and rich interaction in ‘growing’ home IoT systems” and (2)“how does the concept of rich interaction needs to change when applied to ‘growing’ IoT systems.”
4.1. Designing for embodied and rich interaction in growing home IoT systems
In answering our first research question, we first look back at the design processes of our students. Our students particularly stumbled over the “openness” of the challenge that is caused by the requirement of having “growing” interfaces. It seemed that the complexity of the challenge paralyzed them and made them try and out-think the challenge rather than to tackle it through designerly exploration. A successful design strategy proved to be to artificially constrain the challenge and harness the openness by starting to design “loci of interaction” for three or four pre-defined states of “growth” (Table 1). In this manner, grip on the phenomenon of “growth” and emergent functionality could be acquired by studying the state transitions in a controlled manner to generalize a strategy to deal with “growth” when the artificial constraints were lifted.
|State 1||streaming video|
|State 2||streaming video + hard-disc recording|
|State 3||streaming video + hard-disc recording + distributed audio|
Next, we look at the benefits of the four approaches toward solving the challenge that we presented. We feel that there is not one approach that can be singled out as the ultimate solution to the challenge, let alone that a “recipe” can be formulated, more research is necessary. At the same time, we feel confident in saying that it is not impossible to design for embodied and rich interaction in a home IoT context as the four approaches show promising directions. The four approaches act as “templates” (to be used singularly or combined) to inform the design process of designing for embodied and rich interaction in home IoT (growing systems) and provide more grip on the “openness” of the design challenge.
4.2. Reflecting on the changes in embodied and rich interaction
It is clear that the openness (caused by “growth” and emergent functionality) in the design challenge goes against the grain of the designed unity of form, interaction, and function that is one of the defining features of rich interaction (as a product centric operationalization of embodied interaction). As a result, this designed unity needs to be reconsidered; the unity should be released but not left. The hybrid approach proposes to have a standard set of rich and specific controls, designed as a unity but releases the unity through screens that offer more generalized controls. The modular approach offers rich and specific modules that each can be designed as a unity but the unity is compartmentalized in modules. The shape changing approach potentially offers a unity but necessarily releases it in the design process due to the openness of the challenge. Finally, the service approach can potentially offer rich and specific controls that are designed as a unity, but this unity is likely released due to the compromises that need to be made in mating the rich and specific interface to the generic technological parts (this is particularly true in the design example that is given).
5. Concluding remarks
Lastly, we discuss the contribution of this chapter and come back to the value of embodied and rich interaction for “growing” IoT systems. We see value in the exploration of alternative interaction styles in the context of home IoT as we give interaction designers the tools and exemplars to design IoT interfaces appropriate for their functionality, context of use, and fitting the preference of its user (s). This diversify the interaction styles in IoT but also implies the promise of multi-specific IoT artifacts (amplifying the notion of strong-specific products ) that can be tuned to different tasks and that stay relevant and meaningful while the IoT systems “grow.” Next to this, we see value in the framing of IoT systems as “growing” systems and the consequences this has on the design process of embodied and rich interactive IoT artifacts in four approaches.
6. Future work
The next step is to further investigate the four design approaches. The approaches themselves, with all of their idiosyncrasies, need to be better understood. Next to this it is well possible that more approaches to accommodate “growth” in IoT systems can be formulated. We imagine combinations of the existing approaches but we are also searching for new approaches.
Next to this, we are particularly interested in exploring distributed approaches toward designing for interaction in “growing” IoT systems. While the centralized approach has had its use in this research driven design challenge by offering constraints to make the approaches comparable, it is not to say that it should be copied to any IoT systems design challenge. In fact, the argument could be made that particularly embodied and rich interaction (informed by tangible interaction) takes a spatially distributed approach, necessitating the consideration of distributed schemes or mixed schemes.
We thank the master students and faculty who have kindly provided photos of their work: Lukas van Campenhout, Joep Elderman, Jordy Rooijakkers, Paul van Beek, and Tom Fejèr.