As intelligent systems permeate the world, our everyday lives are made easier and less tedious. However, there exist too many “intelligent” systems whose lack of communication or low intelligibility frustrate users. In this study, we present a tangible interface aimed to bridge human-system interaction. It expresses behaviors through shape-change, and its body movements indicate system status and are responsive and rapid enough for perceptual crossing. Based on preliminary results of a user study conducted with 16 participants, the prototype’s implicit interactions show promise in establishing a basic dialog and point to goals and challenges in designing technology that feels truly “smart.”
- shape-changing interfaces
- intelligent systems
- implicit interaction
In a technology-driven market, a main area of focus is designing “smart” products to improve people’s daily lives. It is estimated that by 2020, people will have more than 20 smart devices on their body or in their immediate surroundings . Those intelligent agents will continuously sense and proactively suggest changes with goals including: better energy efficiency, improved productivity, and greater entertainment. To reduce the efforts of controlling increasingly numerous, complex, and capable technologies, many systems will also be able to learn. User preferences will be computed along with outside factors to automatically adapt devices and the environment. However, there is a problem in that an excess of automation often leads to user frustrations . Lack of user control and machines’ failure to effectively communicate with users are two important challenges surrounding interactions with intelligent systems [3, 4].
Several studies, for example [2, 5, 6], suggest that users want to at least feel a degree of control over an intelligent system’s decisions. One possible strategy is to communicate system reasoning to the user . However, a notable study  highlighted two challenges in improving users’ mental models (understanding) of a system. First, users’ knowledge and skills might be insufficient for understanding the complex reasoning of their machines. Second, users often lack the interest or time to invest in learning how a system works. To address those problems, this study aimed to explore interactive designs that could provide incidental intelligibility from the interactions with intelligent machines or systems. We investigated the means by which machines might express their reasoning, their willingness to cooperate, and their ability to negotiate conflicts. With an intelligent lighting system as an application area, we developed a tangible interface that utilizes movement to acknowledge the user’s approach, invite their interactions, convey its learning, and show its subjectivity in an implicit way. Our focus is to learn to foster successful social relationships between humans and intelligent systems so that they may coordinate and perform tasks together smoothly and pleasurably.
2. Theoretical background
As intelligent systems enter everyday lives, people often encounter very basic problems in communication . Users move through interactive fields often without knowing which objects or spaces to interact with because too many systems are “faceless” and not revealing themselves to be “smart” until the user produces a correct interaction cue through the medium(s) that the system anticipates . Such a lack of a prompt, or feedforward, represents a total decoupling of actions and functions. Deckers et al.  proposed the concept of perceptual crossing to show the system’s “face” and let users know their approach is acknowledged: a reciprocal interplay of perceiving while being perceived. With perceptual crossing, users can not only recognize the possibility to initiate interactions with machines but also engage in a more continuous way with something akin to an artificial living creature.
The notion of “calm computing” proposed by Weiser  is a pattern for intelligent systems in which designers use implicit communication for informing without annoying. Relatedly, a tendency is that as systems develop their perceptional capabilities and intelligence, they require less of an explicit command and control relationship with humans . Implicit interactions can take us far in managing attention, controlling expectations, and minimizing cognitive load. These are helpful factors in applying our research to the successful control of an environment .
In implicit human-to-human interactions, body language is a medium through which information is transmitted easily, intuitively, and both continuously and subconsciously. The physical body given to the prototype (as described in the following section) aims to use body language to similarly evoke and even convey emotions (statuses) on this behavioral level. The use of body language also left us with a satisfying amount of ambiguity, allowing for interpretation which along with perception is a crucial pillar for implicit interaction . Ambiguity, in this case, was also a design resource to encourage close engagement with the artifact, an approach detailed by Gaver et al. .
3. Anthox: a physical hypothesis
The name of our prototype, Anthox, is an abbreviation for “Anthropomorphism Box.” None of the preceding research directly focused on the design theories of anthropomorphic products. However, it is apparent that the topics of perceptual crossing, implicit interaction, and body language use human interactions and qualities as a starting point for study and analysis. As such, anthropomorphist qualities were an intuitive goal to aim for in the overall characterization of our design.
With the objective of testing reactions to the prototype’s interaction styles as well as their intelligibility level, an experiment (Section 4) was designed in conjunction with Anthox. As to what would be communicated, the plot chosen was a machine-learning scenario in which Anthox represented system change over time. In such a scenario, the system would need an amount of training data to learn to serve its users. At first inexperienced, Anthox would need to elicit interactions from the user; it might be “needy” or even “insecure” at its lack of knowledge. Later on, a more “self-assured” Anthox might try to communicate its confidence in what it has learned and even offer resistance to a user’s input; the message might then be interpreted as a gentle assertion of the intelligent system’s competence or superiority. With this evolution in mind, a vocabulary of movements for Anthox was designed. Overall, the expectation was that its implicit and tangible methods of interaction would not only enrich the expressiveness of intelligent systems but also be more intelligible and accepted by users.
3.1. The intelligent system
The system in this case is a speculative, intelligent lighting system deployed within an office environment. The exact capabilities of this imagined lighting system were left open ended. It would have some autonomy and be more than a reactive setup, where, for example, lights turn on when you enter the room. Instead, it would incorporate information gathered from sensors in its physical context and other data such as the weather forecast or the office’s calendar and agenda. It might compute employees’ levels of fatigue by tracking sleep patterns, caffeine intakes, eye movements, or any other related parameters. Emotions could also be tracked, as today it is possible to read these wirelessly and with astonishing accuracy . With all of these data, the automatic control of light (color temperature and brightness) could be optimized to be energizing and to enhance comfort and efficiency . Said benefits are measurable, and it is well-documented that light affects humans on psychological, physiological, and emotional levels . Although this system remains mostly speculative for now, it will soon be possible for our lighting environments to be automatically improved in a way that would be infeasible with conventional, manual controls.
3.2. The design
Anthox serves as the physical face and locus of interaction for an otherwise largely intangible lighting system. As presented in Figure 1, Anthox is a white cube with a circular opening on the top surface. This is where interaction happens. Under a layer of mesh fabric, there is a circular control surface consisting of a graphic mapping of light color temperature and brightness (Figure 2). This control plate is translucent and backlit. The light enables the graphic to be read through the stretchy fabric mesh above it. Single-touch inputs are received on the control plate (through the fabric) as in Figure 1. Users are given functional feedback through connected Philips Hue lights which change according to their inputs.
The circular control plate is the place for both the input and the output on the Anthox. The control plate is capable of moving up and down relative to the top surface of the cube; it can rise above the rest of the box, be flush with the top surface, and also sink down (Figure 3). The fabric is attached to the box around the circumference of its (stationary) top circular opening and also in the middle of the rising and falling control plate. Therefore, when the control plate sinks below the top of the box, the fabric is pulled down in the middle, creating a cone shape pointing down (Figure 4). This middle point on the control plate to which the fabric is attached is also capable of rotating. The rotation produces a twisted, wrinkled spiral in the fabric. This resulting spiral can be created while the circular control plate is at any height, be it protruding over the box or sunk down inside of it, as diagrammed in Figure 3.
These two parameters of body movement constitute Anthox’s potential for expressivity and natural interaction . The level change and spiral motions work to change the controller’s affordances. When the control plate rises or is flush, the colored graphic (Figure 2) is legible and highly accessible to touch. When the control plate falls, the graphic becomes less visible because the fabric becomes separated from the plate, and the fabric itself becomes a soft barrier between the user’s hand and the control surface (Figure 4). The spiral which can be formed by the fabric similarly serves to partially conceal the control plate and to make the surface uneven and less receptive to touching. Through shape-change, Anthox alters its affordances to present dynamic relationships to the users in an implicit way.
4. User study
A lab format user study was conducted in order to understand whether Anthox could facilitate users’ perceptions and interactions with an invisible intelligent system. The study sought to learn if these interaction styles could successfully establish a feeling of communication and if so, to what degree it was intelligible. This mainly involved testing for perceived evolution or change over time in the system. Finally, the study sought to learn about the relationship established with the artifact on an emotional level.
As mentioned, the design of Anthox was done in conjunction with the design of the experiment. For said testing, two behaviors were developed: Scenario A and Scenario B. The latter behavior, Scenario B, was designed to match the designers’ narrative of machine learning. Detailed below, it gradually removes affordances and thereby becomes less accessible to the user, demonstrating its “confidence” and independence from user input. In testing for intelligibility, half of the participants were shown this sequence without any prior prompts about machine learning. If they detected a change over time and correctly attributed it to an evolution in system status, then the system might be judged as intelligible and successful in one of its goals.
By contrast, Scenario A was designed as a completely inverse behavior. The purpose of testing a completely opposite sequence (where affordances were gradually added, not removed) was to avoid confirmation bias. By also not adhering exclusively to our own interpretations of Anthox’s implicit interactions, more room was left for other users’ interpretations. Additionally, Scenario A was a point of comparison to Scenario B when it came to analyzing results.
The test began with a short introduction to the topics of artificial intelligence (AI), automated lighting systems, and highly-capable LEDs. Machine-learning, or evolution in that AI, was designedly not mentioned. The prototype was then introduced as a controller for a ‘smart lighting system in an office’, but the exact capabilities of said system (i.e. amount and types of sensors, data) were left undefined. Participants were first able to interact with Anthox freely, with no particular tasks given. After becoming briefly acquainted with the control of light on the immobile prototype, participants were put through five hypothetical scenarios of use over time, during which the prototype then became animated.
The Anthox was controlled in a “Wizard of Oz” method by the evaluator via a hidden set of controls. This method was favored over preprogrammed sequences in order to maintain flexibility in the responses. The goal was to increase the likelihood of users feeling that they experienced a perceptual crossing or engaged in a dialog with the Anthox. As far as the connected lights that the system would be controlling, we used a Philips Hue color bulb (placed in a table lamp next to the user) and a Hue Lightstrip (above the user, near the ceiling). Both capable of displaying 16 million colors, they are an example of the highly capable technologies that AI may help us use to fuller potential in the future.
Figure 5 details the procedure used. The graphic in said figure represents five sectional views of the Anthox over time. Here one can see that Scenario A is an inverse of Scenario B in terms of movement of the control plate. That is, Scenario A sees the control plate rise over the course of the five hypothetical scenarios that the users were put through, while Scenario B sees it falling over the same scenarios. The left column (labeled Scen.) shows that progression with reference to the labeled graphic above.
The middle column in Figure 5 (labeled Narrative) is what the participant heard during each of the scenarios. Here are the actual scenarios verbalized by the evaluator during the test as a directive of what users should imagine and respond to. They mention the passage of time but make no mention of change over time by the AI itself. Again, this and any other judgments are up to the user to interpret from the Anthox’s movements.
The column on the right in Figure 5 (labeled controls) is what the evaluator used as rules for the behavior of the movements of the Anthox. These are the movements the evaluator executed through the prototype. These rules are described in terms of the two parameters possible: up and down of the control plate and spinning of the fabric. Note that UP/DN is relative to the position of the control plate at the given scenario; the starting point for any UP/DN motions follows the progression of low to high (Scenario A) or high to low (Scenario B). Meanwhile, the SPIN category describes movement of the fabric above the control plate in degrees: 0° being the default and 180° being the fully twisted position. The order of movements is crucial. Apart from the gradual removal of affordances, latter scenarios require multiple inputs before the system “confirms” a command. This is meant to reinforce the notion of the system becoming independent.
Below the ‘UP/DN’ and ‘SPIN’ rules are the lighting controls executed in each scenario. The Philips Hue app was also used to covertly control the connected lights in the room. To aid in response times and consistency, preset “scenes” that Philips includes with the Hue app were used by the evaluator to respond to the users’ touch inputs on the prototype. The values for brightness specified in the table can also be found and manipulated through the app for these same “scenes”.
4.2. Participants and evaluation methods
A total of 16 Master’s students (mean aged 24 years, 8 males and 8 females) from the authors’ department participated in this study. They all are familiar with topics of AI, ubiquitous computing, connected lighting systems, etc. They, therefore, were capable of understanding and responding to queries on a high level. They were tested in a between-group design, participants being randomly assigned to Scenario A or Scenario B, with an equal split in gender.
During the evaluation, participants were asked to think out loud, and their interactions with Anthox were video recorded. They also filled in an affect grid  to help better communicate the resulting feelings or impressions. After the interactions, we used audio-recorded open-ended interviews and discussed topics including: general opinions of Anthox, interpretations of movements, nature of the relationship, perceived intelligence, and change over time. This was also an opportunity for the designers to discuss their opinions over the usage of implicit interactions over explicit ones. Further discussion on these and other topics are presented in the following sections.
The result of the affect grid survey is shown in Figure 6. An overwhelming 81.25% of participants engaged the interaction with high levels of physiological arousal. More than half felt it was pleasant to use the prototype. Meanwhile, the movement of Scenario B (from high to low) was thought to be pleasant by twice as many participants than that of Scenario A.
A popular topic for remarks was that of the dynamic affordances, especially when the control plate sank to its lowest position. This setting elicited the most engagement, as users had to more closely inspect and probe the prototype to execute their commands. Although cited by half of the participants as the point where they doubted if they had control over the system, only one participant reported losing total control here. All others were confident in their ability to override the system, and this decreased affordance was seen more like an increased threshold and not an absolute barrier.
In over half of the open-ended interviews, participants mentioned anthropomorphic and zoomorphic adjectives as part of their descriptions of Anthox and its behaviors. Of eight participants who used anthropomorphic adjectives to describe Anthox, three also reported feeling that they were not in absolute control, indicating some sort of power struggle. However, in all cases of users regarding Anthox as anthromorphic or zoomorphic, responses were positive in regard to their relationship with the system.
There were only four participants who correctly interpreted the overall change in level of the control plate as a visualization of a machine-learning process. Scenario B, in which the plate sinks down over time, was understood by more participants (3) than Scenario A (only 1). The sole Scenario A participant who correctly interpreted machine learning even went so far as to propose a redesign of the sequence which they experienced (low to high) to match the reverse: the order of Scenario B.
This is yet another point in favor of Scenario B, which overall yielded slightly more favorable reviews in the open-ended interviews. Five of eight Scenario A participants had negative comments about Anthox, while only one Scenario B participant expressed any serious criticism. This is also apparent in Figure 6, where cyan inputs representing Scenario B lean slightly more toward the pleasant (right) side of the matrix than their counterparts from Scenario A.
Interesting takeaways came from participants’ descriptions of the human-computer relationship they felt was established. At the very least, participants felt they interacted with some sort of subordinate, often anthropomorphized (like a child or an animal). Only one participant felt that they reached a sense of negotiation with the Anthox. Four others reported feeling close to a negotiation, but it became clear that Anthox needs a way of offering explicit suggestions to make negotiation possible.
Overall, all but three participants felt that they reached some understanding of the “language” or signals being exchanged in the interactions, and most of them stated that an even better understanding could be developed with time. We can therefore suggest that implicit interactions were successful in establishing at least a basic dialog, and that certainly there is potential in making improvements toward this goal.
6. Limitations and future work
There are three limitations in the study presented in this manuscript. Firstly, Anthox helped us to investigate participants’ opinions in interacting with a shape-changing system through the “Wizard of Oz” approach. However, further work is needed to investigate how a human user will interact with a system that is able to express its own intelligence. Secondly, in a machine-learning scenario, it would take a training period for the intelligent system to learn the human users’ behaviors and preferences, and vice versa. Subtler aspects of the user experience might not have been revealed in the short period of time users participated in our experiment. Finally, although more than half of our participants found the simple movements to be pleasant and easy to understand, shape-changing forms could be further explored to express alternative semantics.
Based on the results of this study, we are currently working to give the prototype simple machine learning functions. We plan to deploy the system in an office environment to investigate how people perceive its intelligence and react to its dynamic affordances. The goal of our research is to understand how to design the interactions with human-like characteristics in order to improve the understanding between user and system. With longitudinal testing, we would be able to contribute much more valuable knowledge in designing for intelligent systems.
To address problems of technologies’ intelligibility and the associated frustrations, this study applied implicit interactions through shape-change to attempt to bridge interactions between humans and AIs. With the two simple movements it is capable of, Anthox was able to implicitly communicate a variety of messages. In comparison to more explicit forms of signaling, our data also suggest that users might be more willing to encounter dissent from an interface with a more, “playful” interaction style or appreciable “personality.” For now, only two participants imagined that over time the Anthox’s interaction style could become tedious or annoying. While not definitive, this encourages further exploration of this paradigm.
The prototype is named for its dependence on natural interaction styles anthropomorphic or zoomorphic in nature. A risk in the investment toward this approach was the potential of creating a sense of conflict between system and user; certainly to perceive something as anthropomorphic does not equate to feeling favorably toward that object. This is especially relevant in control relationships, where a power struggle with an entity perceived as somehow sentient could become very unpleasant. However, as touched on in the previous section, all participants characterizing Anthox as anthropomorphic felt positively toward it, and only one participant ever felt they lost control completely. Favoring simple or playful interaction styles seemed in this case key to maintaining these positive relationships.
When anthro/zoomorphic adjectives began to be used by participants, there seemed to be an associated recognition of perceptual crossings; this is a point where the artifact started being imagined as sentient and more aware. When this happened, users also attributed more complex mental models to the Anthox. For example, one participant from Scenario A noted, “it is like a baby, you always have to guess at what it wants.” Choosing instead to see Anthox as a being of another species, one user from Scenario B stated, “you never have to think about what you are going to say to your dog […] but somehow the interactions with them (dogs) are always pretty successful.”
Participant preferences for Scenario B supported our own hypothesis in designing the removal of affordances. Through comparisons of qualitative data between the two, we found that indeed Scenario B was more understandable and more pleasant. With our own intuitions confirmed in this regard, future work should look toward testing and understanding more complex behaviors and distinct messages.
The most promising contributors to our experiments were the concepts of natural and implicit interaction styles. The increase in complexity of our changing technological context (has and) will be unmanageable for the human attention span and cognition. Information overload will have to be managed by artificial intelligence and diluted down to less formal and explicit communication channels, where perhaps implicit interaction will be the primary way for us to navigate through it all.
The authors developed this project within the Interactive Lighting Squad of the Industrial Design Department at the Eindhoven University of Technology, and therefore inspired and aided by the students and staff within the squad, as well as their work. Inspiring also was previous work at the same university and department in the realm of shape-changing interfaces, notably the GHOST (Generic and Highly Organic Shape-changing inTerface) module led by Miguel Bruns Alonso and Matthijs Kwak.