This paper presents a research-through-design exploring interaction as a dialog between the user and the product that is in contact for sharing something, typically quantitative and/or qualitative information (e.g., data, points of view, feelings, and so on). This exchange is made possible by the implementation of a given kind of common language. Traditionally, human-computer interaction relies on an explicit, codified language, as for example when designers use icons, text, or pictures to convey a message. In contrast, we define empirical sensory language as those sensory stimuli coming from an artifact, processed most often unconsciously, which play a constructive role in generating a meaningful interactive experience, yet do not require any explicit exchange of information messages. Our investigation aimed at exploring potentialities and limits of applying a sensory language to arouse meaningful interactions leading to a desired change in routine behaviors. We thus designed two product prototypes intended to lead users to decrease water consumption. Our approach opens up a new space for design that is currently not covered by explicit, codified forms of interaction. We discuss implications for a product designer to design for a sensory language and the results of an exploratory user evaluation.
- sensory language
- material interaction
- user-product dialog
- water saving
In recent years, microchips have become so small that they can be embedded in traditional materials such as wood, glass, polymers, fabrics and even more, making such materials “smart” or, as Vallgårda and Redström define them , “computational.” Thanks to computational materials, products can now change their sensory features (i.e., shape, color, texture, etc.) proactively and in a reversible manner, according to a specific situation. Such changes can be designed with an informative intent, i.e., a device that changes shape to communicate the temperature inside the home. This means product designers have currently new material opportunities to work on; and the industrial design field can be enriched with new forms of material interaction and novel ways to convey meanings: indeed, products are able to establish dialogs with users through their sensory features, defining a novel “sensory language.” Previous studies demonstrated that the use of a sensory language instead of the traditional alphanumerical one, conveyed through digital screens, could be a fruitful strategy to engage users during the interaction with products . The positive engagement and the sensorial richness offered by the materiality of artifacts might support the behavioral changes during the interaction with products.
In order to explore both limits and potentialities of applying a sensory language to the user-product interaction, we focused on the issue of water savings in the household environment. Indeed, this appeared to be a promising field of application: previous studies [3–6] stressed the importance to establish a dialog with the final user (mainly through digital screen) in order to change his/her behavior toward more sustainable consumptions.
2. Interaction as a dialog
Technological evolution makes products dynamic and interactive. Products can be designed to be responsive and able to adapt to the surroundings (environment, users, and context in general). Thus, “a domain which was once considered pure industrial design is faced with many interaction design challenges” .
“An interaction is a transaction between two entities, typically an exchange of information” . Indeed, in the Cambridge dictionary, an interaction is defined as “an occasion when two or more people or things communicate with or react to each other”1. Thus, the main aspect of an interaction is that both the involved subjects have to be reactive and responsive to each other. For this reason, we assume that users and products should be related to each other in a circle of influences: the object with its (changing) material features (e.g., shape, weight, color, etc.) affects the user’s behaviors and thoughts, and vice versa.
The idea of a transaction between two entities in an interactive relationship and the idea of an exchange of information over time is also at the core of the definition of “dialog”: “A conversation between two or more people”2. To converse means to “talk between two or more people in which thoughts, feelings, and ideas are expressed, questions are asked and answered, or news and information is exchanged”3.
In these definitions, it is possible to find several correspondences between an “interaction” and a “dialogue.” Both require the involvement of two actors who are in contact for sharing something. Typically, they share information but it is also possible to enrich this relationship with emotional factors (like feelings) and personal point of view (like thoughts and ideas). This sharing creates a cycle of correspondences between actions and responses that is inherent to the human use of artifacts .
2.1. Language as a design matter
“Nowadays, design becomes a matter of using the right language to generate a dialogue about the functionality, intended use of the object, and to generate thoughts and meanings in the user’s mind” .
Dialog, as a way for sharing ideas and information, requires a common basis, the language. In the interaction design field, Moggridge  categorized four languages according to their “dimensions” (Figure 1):
The 1D language: words and poetry
The 2D languages: painting, typography, diagrams, and icons
The 3D languages: physical and sculptural forms
The 4D languages: music, cinema, and animation.
The limit of this categorization is that languages are considered as separated. However, they can be used together to enrich the interaction between a system (products or digital interface) and a user. We indeed believe that any of these languages may be enriched transferring some features of one to another. Particularly, this research is based on a number of philosophical studies of the use of words (i.e., 1D language), which suggest two different ways of using them: as a codified language and as an empirical language [12–14].
Starting from the phenomenological analyses of Merleau-Ponty , we define codified language as a form of expression based on conventions/symbols/codes shared by a group of people. This language is useful to be applied when the contents of the information have to be clear and not misunderstood. For example, the traffic light informs users about their possibility to cross the street applying three colors with conventional meanings: green, red, and yellow. One important aspect of codified language is that there is an arbitrary link between the code (e.g., the shape of a letter) and what it signifies (an element in a linguistic utterance). Furthermore, when a designer uses these codes to convey a message, the meaning of each of the codes is predefined: it does not depend on the situational details of any particular interaction in which the codes figure. So, for example, it does not matter on the emotional state or the previous series of events a person experienced for determining the meaning of the “black coffee” symbol on the coffee machine: it means black coffee. It could be that the person is lead to misunderstand the correct meaning, for example when the symbol looks a bit like cappuccino and the person is tired and not paying attention. But this kind of variability in meaning would be considered an error and certainly not an intended aspect of the design.
On the other side, we defined empirical language a form of expression in which the contents of the dialog are not completely established a priori, but meanings are to some extent subjectively created by the user during the interaction itself. In other words, the contribution of both the designed artifact and the person determine what meaning is experienced, rather than the artifact or the person “transferring” certain predefined meanings over a codified communication channel. Empirical language is often used to elicit or catalyze meaningful experiences, which the person will have courtesy of being engaged in. Paradigmatic examples can be found in the artistic and poetic fields.
Historically, industrial designers are familiar with the physical forms of artifacts. For this reason, this research focuses on 3D languages, interpreting them not only as pure physical shapes but, more in general, as all that can be perceived by the human sensory apparatus. In other words, this research intends “sensory language” as a language based on any kind of stimuli that can be perceived by one of the five senses (view, smell, touch, taste, and hearing). The concept “Scent of Time” (Figure 2) is a good example: it is a clock that releases a specific smell in the environment to denote a specific time of the day.
3. Research-through-design question
This research aimed at exploring the design qualities of a sensory language in establishing meaningful dialogs between users and interactive products, used in mundane, practical settings (i.e., not as an art exhibition or critical provocation). To this end, we delve into the differences of applying a codified versus an empirical language, also attempting to create a “cross-fertilization” between features that traditionally apply to well-separated kind of languages: the 1D language (based on words) and the 3D language (based on materiality as defined above).
As already said, nowadays, in applying a sensory language to products, any designer can exploit the new opportunities coming from the development of smart materials and new technologies.
We applied a research-through-design approach: two design activities were carried out in order to develop and test design proposals based on insights coming from the theoretical investigation.
The first design activity explored the sensory language as a codified language, and the second activity explored the sensory language as an empirical language. As a result, two functioning prototypes were developed: “glass of water” and “feelings and experiences for an embodied learning” (F.E.E.L.).
The two prototypes were tested with users during three focus groups, organized with the aim of exploring the interests, feelings, and engagement of the users as well as their understanding of the information conveyed.
5. Design activities
5.1. Glass of water
The first design activity focused on the exploration of a codified language based on the correspondence between a signifier and significant. In the field of semiotics, this correspondence is called “code” . Specifically, Lachman et al.  defined a code as: “a set of specific rules or transformations whereby messages, signals, or states of the world are converted from one representation to another, one medium of energy to another, one physical state to another” (p. 68). In brief, codes specify how information is to be converted from one form to another .
5.1.1. How to transform alphanumerical information into sensorial stimuli
A literature review was performed in order to define a set of specific communication parameters that can be useful for designers to translate information related to water consumption into a sensory language. The objective of this investigation was to find new ways to inspire designers in the concept generation phase. The result is a list of opportunities/limits for the application of the sensory language to convey certain kinds of information that are usually intended to be quantitative (e.g., average water/energy consumption per day/month/year).
Even if the general focus was on the exploration of tangible aspects of products, in this phase, we enlarged the research also to the field of digital interfaces, persuasive technologies, and interaction design. Indeed, in these fields, several studies have already investigated the importance of giving feedbacks on energy consumption, and some of these studies tried to give suggestions to designers facing this matter. Only studies reporting results on users’ investigation were considered, since our intent was to explore the user understanding of the contents of the dialog. As a result, 12 studies [5, 6, 17, 19–28] were selected and analyzed in deep, and three main communication parameters emerged :
Metrics (related to the unit of measure). It has been observed that it is better to transmit qualitative information instead of quantitative ones (e.g., “Today your consumptions are good,” instead of “today, you have consumed 120 KW of electricity.”)
Frequency, related to when and how often it is necessary to give information to users. It has been observed that users need continuous information instead of having information only when an event occurs.
Representation, related to the “form” of the data. Some studies underlined the importance of visualizing information through metaphors or analogy (e.g., recalling the effects on the barrier reef of positive/negative consumption behaviors). Moreover, from these studies emerged the importance of giving positive information in order to support the motivation of users.
Any of this parameter has to be communicated through a sensorial language (i.e., through the products material features), to make the product itself a media for conveying information about the user’s consumption.
5.1.2. Concept description
Glass of water is a tentative of applying the three parameters coming from the literature review (i.e., the first and the third one). It consists of a set of little spheres suitable for all faucets of the house. The spheres aim at reminding the user that the quantity of water at their disposal is not unlimited. To this end, they are conceived as small meters able to give information about the water consumption of the connected faucet. By means of a blue light, they tell users the correct amount of water they should use for their domestic activities (according to Gleick , around 50 L per person per day).
Any time the user turns the faucet on, the connected sphere gradually lose its brightness to show that the amount of water suitable for the domestic activities is decreasing until ending, when all the lights of all the spheres turn off. These changes are intended to recall the idea of a glass of water getting empty (Figure 3).
A prototype was developed, based on an Arduino board that controls the brightness of the sphere’s light (Figure 4).
When the faucet is not used, the light is off (Figure 5); as soon as the water is open, the light gives (qualitative) information about the amount of water that can be still consumed. When the consumption of water exceeds 50 L per day, the light does not react anymore until the following day.
This raw prototype aims to be representative of the language. Thus, its functionality is simplified.
5.2. Feelings and experiences for an embodied learning (F.E.E.L.)
The second design activity was focused on the empirical language, based on the idea that meanings are created by the user during the interaction with the product.
During the development of the concept, we retained useful to apply the principles of embodied interaction. According to embodied interaction theories, there is a circle of influences among the physical properties of the products, the user’s actions on/with the product, and the creation of meanings in the user’s mind . In phenomenological terms, material properties of the world are taken up as elements sustaining stabilities in action-perception loops, which govern a persons’ skilled, routine-like dealing with the world [14, 31, 32]. The sense-making process is not just based on decoding information conveyed in the products features, but it is made possible by the creation of a personal dialog between the user and the product itself. Such dialog encourages reflection-on-action and the creation of new meanings.
Due to the complexity of the process, it was decided to focus on the water consumed into the shower, and it was decided to concentrate on the tactile stimuli provided by the shower tray. The aim of this second design activity, thus, was to create meaningful tactile experiences provided by an interactive shower tray able to affect the users’ behaviors, influencing them “in the situation.”
5.2.1. Preliminary research
With the aim of exploring the users’ behaviors and feelings when taking a shower, users’ observations were performed. Particularly, three short tests with users were organized in order to gain some insights into: (i) users’ behaviors, (ii) tactile feelings, and (iii) users’ emotional experiences. Video recording was the tool for exploring the users’ feet behavior in the shower and the tactile interaction with the shower tray.
Moreover, testers were asked to fill a 1-week diary in order to keep track of their experiences and feelings during the shower. Both videos and diaries gave us insights into the context of use through pictures and words . At the end of this phase, we collected four videos and four diaries.
5.2.2. Concept description
Thanks to the preliminary research on users’ habits, useful insights were gained referring to: (i) the shape of the shower tray, (ii) the materials’ selection, and (iii) the design of new behaviors and scenarios. These outcomes were useful for the development of the concept feelings and experiences for an embodied learning (F.E.E.L.). F.E.E.L. is a squared shower tray able to change its shape in order to create a new routine into the shower. It is composed by an external case and a number of soft “pins.” These pins pop up randomly according to different rhythms, creating each time a novel tactile experience as a sort of feet massage (Figure 6).
The data collected by the 1-week diary show that users take a shower with two different aims:
as a (short) refresh activity
to relax and pamper themselves
According to these two scenarios, F.E.E.L. is designed to change its shape in a fast and more marked way in the initial minutes of the shower, for the average amount of time that users usually spend for a short shower. Then, it decreases the speed of its movements over time until stopping at the achievement of the maximum average time usually spend under the shower, that is, the “natural” moment at which people experience that they are now finished and it is time to get out. Over time, F.E.E.L. creates a dynamic coupling between the user’s action and the temporal choreography played out in the responsive shower floor. Once this coupling is in place, in order to lead users to decrease the time spent under the shower, the system will over time decrease the durations of the temporal pattern. The size of the decrease would be small and not consciously noticeable. Indeed, it is important that the user feels as he/she has always the same routine during the shower. As a result, the user will after several weeks reach the “natural” feeling of “being finished” minutes earlier than before, thereby saving water, without feeling forced or without feeling as if one needs to make a conscious, ethical decision to “do the right thing.”
We created a raw prototype in wood, steel, and soft materials. It aims to be representative of the language, and its functionality is simplified (Figure 7).
6. Focus groups
Three focus groups with users were organized in order to collect qualitative feedbacks about the two concepts. It is important to stress that they were not intended as traditional user-evaluation focus groups, asking users whether the products would actually reduce water consumption. Rather, they were intended to extract some insights from the user’s own sense making of our designs in order to answer our theoretical questions concerning the potentialities of applying a sensory language in designed interactions and the difference between codified and empirical languages. To this end, the first focus group presented to users “glass of water,” that is, an example of codified language. The second focus group presented F.E.E.L that applies an empirical language. In a third focus group, both the concepts were presented in order to let users compare the two kinds of languages.
Each focus group involved four people (two men and two women, aged 25–36). Our choice was to involve people with different backgrounds. None of the tester was a designer, since the aim of this activity was to gain feedbacks from nonexpert people. Moreover, we decided to involve people who lives alone and who already manage his/her consumptions.
All of them had the same organization: they started with a brief presentation aimed to introduce the concept/s to be discussed. During the session, videos and pictures were shown in order to describe the concepts in the real context. At the end of the presentation, participants were invited to discuss together. The discussion was guided by five open questions aimed to investigate the following aspects: (i) the engagement that they perceived with the concept, (ii) potentialities and limits of the concept, and (iii) how the concept could fit in their daily life.
As a result, testers focused on two main aspects of the proposed concept: (i) the novelty factor and (ii) the clarity of the communicative intent. Both the projects were evaluated as novel, original, and unusual (compared to the objects that testers have in their home). However, participants evaluate the project F.E.E.L. as more innovative: “It proposes an unusual experience”; “This project is more cool!”; and “I have never seen something like that!”
Glass of water was perceived as less surprising and more similar to digital interfaces. During the focus groups, it was observed that the novelty factor influences the enthusiasm of the users to have a trial with the prototype in the real context.
Comparing the two projects, it emerged that the communicative intent of glass of water is more evident and immediate to users. Meanwhile, the communicative intent of F.E.E.L. required extra explanations by the facilitator. However, the concept F.E.E.L. was perceived as more able to support and motivate the changing of the users’ behaviors, since the learning process was judged as subtle and more linked to personal awareness.
On the contrary, the learning process provided by glass of water was judged as cognitive and intentional: “Glass of water is telling me how much sustainable I am. In a way, it is pushing me to turn off the faucet to save water. Meanwhile, F.E.E.L. helps me to relax and enjoy my shower. Water consumptions are up to me.”
Several studies on smart meters providing users with alphanumerical feedbacks on their consumptions have shown that they have a limited influence on users’ behavior [4, 24, 34]. More detailed investigations show that the situation is complex: numerical feedbacks may fail either in taking into account the realities of users’ life [20, 21, 26] or their understanding of units and quantities . Moreover, they do not help users in having a wider understanding of their behaviors . Another limit of information provided through a display is that they require users to look at them regularly, checking the consumption trends. To overcome such limits, studies carried out in different domains (interaction design, esthetics of interaction, ambient display, and visualization of data) have explored more sensorial ways (change in color, in light, in form) to give information about the amount of consumed resources.
Our results show that designing for a codified (sensory) language or for an empirical (sensory) language have different implications both for the design process and for the users’ experience.
From the users’ point of view, it was observed that establishing a material dialog through a codified code gives prominence to the message and to the communicative intent of the product (as underlined by the focus group discussion). Thus, the designer can apply a codified language when he/she wants the user to be conscious of the informative content, i.e., when the aim of the product is to give users’ information on what would be a wise thing to do, to give them useful methods for achieving a certain goal, or to give them explicit reward for good behavior.
On the other side, applying an empirical sensory language means to focus on the embodied experience of interaction with the product . This experience typically does not lead to a conscious reflection in the mind of the user about what to do. We strongly believe that the sensory empirical language opens up the possibilities to design for behavioral change other than through passing on information that the user must process and make a conscious decision about. As the example of F.E.E.L. shows, we may be able to gradually change the experience of the user in such a way that the user finds himself wanting to get out of the shower. This tactic has been applied in similar fashion by Bruns et al. , who demonstrated how fiddling with a pen that gives haptic feedback would lead to a reduced level of stress even when the user was not conscious of this.
It is now important to briefly discuss what makes up for an empirical sensory language, and what would just be a natural consequence of interacting with the physical world. As a first idea, one might say that an empirical sensory language comprises anything sensed by a person that does not include explicit coding, but does help to generate a meaningful experience. This would, for example, include how the physical properties of a tennis racket in interaction with a student would help the student master the game of tennis through experiential learning. We do feel that our design examples are of a different kind. While the racket is indeed a designed artifact, it is not designed to generate a learning experience: it is designed to play tennis. The potential for experiential learning is in some sense a by-product that you get for free. More importantly, you would not be able to change such physical aspects of the product without changing the entire product.
On the other side, we consider what happens when you sit in a bath. The water slowly gets cold. This change in the baths’ “behavior” leads to a temporal dynamic in sensory signals that may very well give rise to a quite similar experience as the one intended by the designed choreography of the shower tray. Processed largely unconsciously, the falling temperature will at some point make one think: I should get out now. But this is not the result of a designed experience. In contrast, in the shower floor, we were able to design the system in an adaptive way so as to first align the choreography with the natural routine of user and then gradually shorten the pattern in order to elicit an earlier onset of the experience of “being finished.”
To conclude, in a sensory language, rather than considering just the sensory signals coming from the physical properties of any part of the artifact more generally, there always remains to some degree an arbitrariness, or space for variation, for creating a coupling between the designed sensory signal that a person receives, and the meaning that will arise from it. This space for variation is precisely what gives designers the freedom they need to design the interaction. We say this space of variation exists “to some degree,” because a complete freedom on part of the designer to determine the couplings between each sensory signal and the meaning it should generate in the user experience would of course turn the system into a codified system once more.
With our research, we aimed to contribute to achieve a “crossfertilization” between features that traditionally apply to the alphanumeric language and the sensory language. Indeed, the sensory language can nowadays exploit new opportunities coming from the development of smart materials and new technologies.
In our research-through-design case study, we applied two different kinds of sensory language (a codified and an empirical one) to the design of two prototypes intended to lead users decrease the water consumption. In general, the results show it is possible to design for behavior change using a sensory language.
Entering into the differences between a codified and an empirical language, we have noted that establishing a material dialog through a codified code gives prominence to the message and to the communicative intent of the product. On the other side, an empirical sensory language addresses our embodied experience more directly, which opens up possibilities to design for behavior change in ways other than through explicit message passing or pure physical enforcing.
One limitation of this study is that this research does not explore the users’ perception of sensorial codified and empirical languages overtime. In this perspective, future investigations should be set in order to test the two languages capacity to affect the user behaviors over a prolonged period of time.