Calm technology for biofeedback : why and how ?

We discuss several possibilities and fundamental difficulties when designing biofeedback systems based on calm technology. As a carrier for the discussion we develop a novel biofeedback installation based on heart rate variability. The system is built-in to an elegant table and gives visual feedforward or feedback for relaxation based on breathing. When in feed-forward mode, the system will show a sine wave of about 7 cycles per second, close to the well-known resonant breathing frequency. Alternatively, the amplitude of the movement can give feedback on the heart rate variability level, which is known to be directly associated with a reduced level of mental stress. The demonstrator has a pulseplethysmography sensor which measures the beat-to-beat intervals of successive heart beats. The mechanical design of the actuator is designed to operate completely noise-less. Both the adaptive algorithm and the actuator are new to the best of our knowledge. Still new fundamental questions arise.


Introduction
There are many relaxation techniques and meditation techniques in which breathing plays an important role.Burn-out and depression are contemporary conditions with high prevalence [1,2] and there is increasing evidence that certain types of breathing-based biofeedback are helpful to support therapies [3,4].Key notions in these feedback approaches are breathing and heart rate variability (HRV).
The term "calm technology" has been proposed by Mark Weiser and John Seely Brown in their first essay in 1995 [5] and it has been very influential in the sense that ambitious research programs have been based on it such as the Ambient Intelligence program described by Emile Aarts and Stefano Marzano in 2003 [6].As Weiser and Brown argue, "Information technology is more often the enemy of calm.Pagers, cellphones, newservices, the World-Wide-Web, email, TV, and radio bombard us frenetically."Despite the original enthusiasm for calm technology, we are still bombarded frenetically, by these media, and at the same time by a growing variety of new social media, geographic information systems and large public displays.One would expect that the application domain of biofeedback for relaxation would embrace calm technology but, with a few exceptions, that is not what happened.Instead, different developments can be observed: gamification of biofeedback and the rise of mindfulness as an alternative approach for coping with stress.But the topic of calm biofeedback technology is still studied in Stanford University's Calming Technology Lab [7] and at TU/e.Also, there exist several design proposals where the biofeedback is rendered more ambient, not in a handheld device or a smart phone.Examples are the ExoPranayama system by Stuart Moran and others [8] and the Living Surface by Bin Yu in cooperation with Alissa and Nienke [9].
Traditional breathing techniques fall into two categories: (1) breathing in a prescribed rhythm, and (2) observing one's own breath mindfully, without any restrictions.Pranayama yoga belongs to the first category, Anapanasati (mindfulness of breathing) to the second.Modern technology-based techniques can be categorized into two somewhat related categories: feedforward systems and feedback systems.Paced breathing software such as EZ-air belongs to the first category, Stress-eraser and Cardio Sense Trainer to the second category.But there are differences between the moderntechnology-based techniques and the traditional breathing techniques, based on a new understanding of our internal mechanisms connecting breathing, heart rate and stress.We mention the following insights [10].


Heart rate variability (HRV) tends to go up under conditions of relaxation and it goes down under stress.The HRV signal is rich and complicated and can be analyzed in many ways, for example distinguishing low-and high-frequency components.Visualizing HRV allows performing biofeedback training to reduce stress levels and supports reducing anxiety, burnout and depression.


Breathing in, heart rate goes up, if we breathe out, heart rate goes down.The phenomenon is called respiratory sinus arrhythmia (RSA), and it allows to artificially increase heart rate variability.The effect is most strong at a particular frequency near 0.1Hz which is called resonant breathing.
In this article we present and demonstrate a novel biofeedback installation based on heart rate variability.We discuss the design choices, some of the alternatives, and some of the fundamental challenges we encountered.Both the information display and the model-based biofeedback software are new to the best of our knowledge.Whereas today's trend is to use the smart phone as the main platform for displaying information, we believe there is a need for a real calm technology, which keeps a distance from the overloaded screen, the multi-functionality and the busy social-media life that are associated with smart phones.Therefore we decided to try turning the biofeedback system into a minimalist modernist furniture element with a novel shape changing interface as information display.To be in contact again with one's heart and indirectly with one's unconscious emotion regulation system is an important activity which deserves a special place and attention to the biofeedback training ritual.The biofeedback system is designed to support that.We aim at an information display which is a subtly shape changing interface, which is completely flat at rest, having certain aesthetic qualities of its own.
The idea of calmness is also essential to the design of the software implementing the translation from the sensor to the actuator.Existing feedback systems fall into various overlapping categories: 1.The system calculates a variety of plots such as frequency distributions, tachograms, numeric representations of heart rate and heart rate variability (which has many different ways of being quantified).2. The system calculates a performance indicator related to heart rate variability (HRV) and uses that in an abstract or playful setting to motivate the user and try to raise HRV levels and do that during a longer period of time or more sessions.3. The system presents a direct visualization of the beat-to-beat intervals, which is close to the raw data of the sensor, for example a tachogram (a plot of successive beat-to-beat intervals), 4. The system highlights the differences between successive beat-to-beat intervals so the user has immediate and fast feedback.
The first category is a rich cocktail of 2., 3. and 4.But is certainly not adopted here because it is not calm.On the contrary, it adds to the information overload which is perhaps one of the causes of stress in the first place.The second category (labelled I, for integrating in our 2010 paper [11]) has great potential for being calm or playful, but the integrative nature of the calculation has the effect that the improvements are only apparent after a significant number of heart beats or even after several breaths.The delay between cause and effect is likely to hamper learning.The third and fourth intervention, called proportional (P) and differential (D) in [11], are more immediate, but they are another source of unrest themselves: as each heart beat appears, the pumping irregular rhythm of the heart is visible in a dominant way.Bin Yu tested auditory variations of the fourth category [12] and found that they were difficult to understand and not optimal for relaxation.Of course it is possible to filter the tachogram, smoothing the plot so the individual beats do longer appear.
Regretfully, the classical filter techniques unavoidably cause delays of at least several seconds (several beats).This is a fundamental difficulty which we set out to resolve in Section 4. We developed a novel filter which combines smoothing with predictive filtering.Section 2 (usage scenario) and 3 (shape changing information-display) provide a context for the more technical work in Sections 4 and 5.

Usage scenario
The installation will be in the living room or private library of the user.Next to the dedicated table, which includes the shape changing information display and the embedded software, there is a comfortable chair.When the user decides to do a relaxation session, she sits in the comfortable chair, attaches the sensor to one finger and relaxes.She watches the shape changing information display, which is the top of the table, and observes the regular rhythm of the subtly moving surface.
Alternatively she can put a hand on the top of the table and feel the moving surface.She breathes in the very same rhythm indicated by the moving surface.During the first five minutes, that is what she does.It is a relaxing experience.Then the system gradually and automatically moves into feedback mode.The user has to train in a goal-oriented fashion now.The goal is to maximize the amplitude of the movement of the surface.Larger amplitude means higher heart rate variability, which means higher level of relaxation.The system does not provide instructions explicitly, the goal to maximize amplitude is told to the user beforehand when she obtains or buys the system.After a total of ten or fifteen minutes the session can be stopped, as the user wishes.The user feels more relaxed and is ready for other daily tasks.The user can do one or two relaxation sessions per day, according to personal preferences or needs.

Design of a shape changing information display
The side table was designed by Sander Lucas and produced in studio LUCAS&LUCAS in Eindhoven.We had asked Lucas to re-use the design language of furniture for the Mind the Step exhibition, which was also designed by Lucas.Mind the Step is an annual design and technology exhibit which is part of the Dutch Design Week (DDW).We appreciate the modernist and minimalist form of this furniture.The table is made of wood, painted white, very compatible with the calm design aim.On top is the shape changing interface: a surface which moves up and down, in accordance with the breath feedforward or feedback.The surface is made of polyoxymethylene (POM), which has high stiffness and low friction.We experimented with polymethylacrylate (PMA) too, but it is too brittle.The plate material is turned into a flexure by laser-cutting.The cut is a double-threaded spiral with a particular pattern with angles of (close to) 90 degrees forming an interlocking tessellation.The basic tiles are connected by thin strips, which give rise to an extremely flexible surface.One sees not only the lasercut spiral, but also, as a Gestalt, emerging spiral lines like in a pine cone or a pineapple (phyllotaxis).
When the surface is flat, the latter spirals are apparent, the cut spiral is harder to see.When the surface deforms from flat to non-flat, the vertical cuts appear very clearly with a subtle and beautiful effect.
Figure 1: The double-threaded tessellated spiral used for laser-cutting.
There is significant freedom in the design of the flexure.The spiral was considered a good solution for the shape changing surface because it is a kind of labyrinth, the archetype of the journey to one's self.For the tessellation we adopted a specific fashionable and intriguing pattern, mathematically explored by the first author in earlier studies published in Bridges.The spiral generator is written in Processing, and uses Oogway, a turtle-graphics library designed by Jun Hu in 2013 [13].Each spiral is written, starting at the inner loop, with the turtle spiraling outward.The outer demarcation line of one basic tile consist of twenty commands forward(s) in turtle graphics, eight commands left(90), and nine right(90 + dα).So each right-turning angle has a slight deviation dα and the accumulated effect is that after T tiles there is a total deviation of T×9×dα, which must be 360 degrees.We arbitrarily chose T = 21, solving dα = 1.90476190 degrees.After each tile, the basic step-size s is increased by a factor 1.0148.So after each full turn, the tessellated tiles are a factor (1.0148) 21 = 1.3614 larger than before.After nine full turns the outermost tile is (1.3614) 9 = 16 times larger than the innermost tile.We were surprised by the extreme flexibility of the spiral, first cut in a wooden version.Just holding it in our hands, we got the serendipity moment to see that such spiral would be a good solution for the design challenge of a shape-changing interface to be combined with the biofeedback algorithms already under development.We then optimized the design by choosing size, material and more technical elements such as a voice coil to obtain the visible effects for the biofeedback installation.We considered servo-motors but they are noisy and they wear out, so we rejected them and began exploring voice coils.The voice coil is precisely the same as the voice coils used in loudspeakers and a current of .5 Ampère is enough to move 5mm (free) or 3mm (loaded with the spiral).Although it is very difficult to produce very low frequency soundwaves with voice coils, it is not hard to produce mechanical movement (the difficulty would be in transferring the movement to the free air).To drive the voice coil we developed a simple direct current (DC) amplifier with a LM324 op-amp and an MJE3055 transistor in emitter-follower configuration (traditional audio amplifiers will not work).During further explorations we found that the actuator does not serve well as a haptic actuator, yet the visible effect is precisely what we need.The connection between the voice coil and the spiral went through many rounds of exploration and testing.In the end we found the best solution to be that the spiral is supported by flat surface underneath and is pushed up by the voice coil at one or a few off-center points only.The effect is that the spiral lies completely flat when not in use but works as an information display during feedforward and feedback sessions.In the terminology of Skakoon [14] the surface is flat by force closure where the weight of the spirals act as the nesting force (in the flat state).We found that the actuator works completely silently (no noticeable voice-coil sounds, no noticeable mechanical friction sounds).

Design of the smoothing predictive filter
The main characteristic of the heart signal is that it consists of discrete events appearing at intervals which are slightly irregular.The heart beat events can be detected using a variety of techniques.The most reliable method is by peak detection in the electro cardiogram (ECG), which requires a fairly obtrusive sensor (something with electrodes).Each peak is called an R event (referring to the QRS complex discovered by Einthoven).We used a pulse-plethysmography (PPG) sensor, which works with a small clip on the finger or the ear.PPG is a reasonable compromise between ease of use and reliability.The sensor, amplifier, detection circuit and Arduino interrupt routine are described by Langereis [15].Note that most smart watches and face-color based camera detector systems can deliver heart rate (HR), but are not yet able to detect each individual beat and hence cannot (yet) reliably calculate beat-to-beat intervals, which is essential for heart rate variability (HRV).Now we describe why we aim at a novel filter which combines smoothing with predictive filtering.To clarify the fundamental challenge we refer to Figure 3. Consider a sequence of heart beats arising at times t(i-5), t(i-4), … , t(i-1), t(i), where i is the index of the last beat detected (a).In a tachogram, the beat-to-beat intervals RR(k) = t(k) -t(k-1) are plotted vertically against a horizontal time axis, which is a typical way to visualize heart rate variability on a screen (as in Stress-eraser).For a shape-changing interface, we could translate them directly into a Voltage V(t) proportional to RR(i) and use that to drive the voice coil during the time interval t(i) < t < t(i+1).But the heart beats appear as staircases in the tachogram and as jumps in V(t) and hence as jumps by the shape-changing interface (b).It is technically possible to get rid of the staircases by analog or digital filter techniques such as resistorcapacitor filters or finite impulse response (digital) filters, but the filters always introduce a delay.So an increase in RR interval appears in the filtered tachogram or on the voice-coil's Voltage only several beats after the increase happened (c).Even if we would interpolate the vertical plots linearly (d, first four lines), we would still not know what to plot or what Voltage to provide during the interval after the last beat, t(i) < t < t(i+1)?But if we would have a prediction RR*(t+1) for RR(t+1), "looking into the future", we could plot a continuous line (d, including dashed line).Next we could explore all kinds of spline techniques or smoothing techniques to eliminate the staircases and triangular forms.Therefore we decided to develop an experimental predictive filter.
Predictive filters are used in areas such as motion detection of cameras or path prediction of airplanes (for military purposes, regretfully).The most used technique is Kalman filtering, which deploys a given dynamic model of the object's behavior (typically based on Newtonian or Lagrangian equations of motion).The time constants and eigen-frequencies of the dynamic behavior are built-in to the model.In our case, we do not know the breathing frequency beforehand so we need a special model, not precisely Kalman.We work in a transformed RR space where we can estimate the momentary breathing rate, denoted as E(BR).The algorithm is remotely related to the minima and maxima-counting method of Schaefer and Kratky [16], who claim that it allows not only determining average values over the investigated time interval, but also to define an instantaneous respiratory rate.A Fourier transform is not very helpful because that would find a spectrum after many breaths only and cannot adapt quickly (and is not accurate either, as shown by McMullen et al. [17]).Further details of the special model are outside the scope of the present paper.We take the previous RR intervals RR(i), RR(i-1), RR(i-2) etc. as inputs, but using exponential weighing (so the very old values do not matter anymore).From RR(i) and RR*(i+1) we can find plot values or Voltages for at least one (estimated) interval duration after the last detected beat.We explored Catmull Rom splines, based on the polynomial V(τ ) = ½(-τ+2τ 2 -τ 3 )RR(i-1) + ½(2-5τ 2 +3τ 3 )RR(i) + ½ (τ+4τ 2 -3τ 3 )RR*(i+1) + ½(τ 2 +τ 3 )RR*(i+2) with τ = t -t(i) for t > t(i), which has the advantage that the lines are continuous and their slopes are continuous too.Schaefer and Kratky [16] work with splines as well, yet not for t > t(i).After multiple trial versions we decided for linear interpolation with jump smoothing of slope discontinuities, which we found to be more robust against outliers and poor predictions.The diagram is in Figure 4.The filter is realized in software, first developed in Processing (for easy testing) and then ported to Arduino as an embedded system within the installation.The system has no buttons, remote control or data collection options, in accordance with the design goal to create calm technology.Plugged into the mains, it works.There are several alternative options for extracting meaningful data from the sensor data.We could work with mixed linear models (good in case of missing data) or machine learning (taking advantage of large data sets).We should also mention that we see new technologies at the horizon for getting the heart beat or respiratory data even better, that is, less intrusively.Ballisto-cardiographic or face image-based methods (vascular mapping, blood perfusion modeling) could allow us to get rid of the PPG sensor.

Mechanical design of the actuator
The mechanical design of the actuator is designed to operate completely noise-less.Although specific users might like the sound of servo-motors [8], the requirement of calm design suggests that it is much better to keep the mechanism silent.A voice coil meets this requirement very well.Although this is the very same mechanism used in loudspeakers to generate high volume sound, a voice coil can operate silently at very low frequencies -as in this design.The basic idea of the mechanism is described in Figure 5.At present there are two driving pins, but it could be any other number.The force of gravity makes sure that the spiral flexure lays completely flat, except at certain positions where a driving pin pushes it upward.The force of gravity works as a nesting force [14], yet when the driving pin is low, the spiral flexure cannot sink into the (circular) holes.

Conclusions and outlook
The system can work both in feedforward mode and in feedback mode.When in feedforward mode, the system will show a sine wave of about 7 cycles per second, close to the well-known resonant breathing frequency [10].This is an easy way for the user to get started.Then after five minutes, the system gradually goes into feedback mode.When the deep and regular breathing continues, the system will respond with movements which are of the same frequency and approximate phase as the user's breathing.In this state, the distinction between feedback and feedforward has disappeared (with respect to frequency).User and system are synchronized.The amplitude of the actuator's movement still provides feedback on HRV amplitude.
As a limitation we mention that the present set-up is not yet comfortable enough for sessions longer than ten or 20 minutes.However, for relaxation that should be enough (typical audio tapes of progressive muscle relaxation are also not longer than 10 minutes).
The design goal of creating calm technology has been met in several ways.We found that for the form-giving, and the display the aim of "calm" can be implemented satisfactorily: the chosen houndstooth labyrinth is just one of many possibilities for the flexure pattern; the voice coil works silently.The PPG sensor is not very comfortable, but we leave it for now (in a few years, camera-based pulse detection technology could be available).The interaction by breathing works well in feedforward mode and also in feedback mode when breathing is regular indeed.Still there is a lot of work to be done as we found that irregular breathing breaks the synchronization, and we are still exploring options to provide useful feedback in such conditions.We also found an open question regarding HRV: how to interpret the HRV components which are not caused by RSA?If breathing is shallow and other, non-breath HRV components dominate, the combined smoothing and predictive filtering algorithm does not work well.In a different setting McCullen et al. [17] discuss how RSA peak frequency and breathing frequency do not coincide.We do not know what precisely causes the nonbreathing related HRV component?Does it allow one-beat ahead prediction?How does it correlate to thoughts or absence thereof?There is literature on the various sources of vagal cardiac control, but so far we got lost in literature, which is either on vagal control brain nuclei studied in rats, or on statistical long-term HRV parameters -which are not helpful for short-term prediction.We leave the matter as an (extremely interesting) option for future research.
Finally let us mention how to conduct experiments for fine-tuning the feedforward and feedback subsystems.The formal evaluation of the feedforward could be done along the lines of Bin Yu's research who has been testing soundscapes and their effect on subjective and objective relaxation [18]: for subjective relaxation there exist instruments such as the Relaxation Rating Scale (RSS).An alternative is the State-Trait Anxiety Inventory (STAI).These can be combined with open interviews.Objective relaxation can be assessed using HRV.For testing the feedback subsystem it would make sense to benchmark the breathing extraction algorithm against other ways of deriving the respiratory influences on the HRV, for example using so-called Orthogonal Subspace Projection (OSP) [19].We leave these as options for future research.

Figure 2 :
Figure 2: The furniture of the biofeedback installation.

Figure 3 :
Figure 3: heart beats appearing at irregular intervals (a), tachogram with intervals plotted vertically (b), RC-filtered signal (c) and continuous smooth interpolation with predication for last segment (d).

Figure 4 :
Figure 4: Overview of predictive and smoothing system.