Open access peer-reviewed conference paper

Sensory Augmentation through Tissue Conduction

Written By

Peter Lennox and Ian McKenzie

Submitted: 11 September 2017 Published: 18 October 2017

DOI: 10.5772/intechopen.71126

Chapter metrics overview

1,513 Chapter Downloads

View Full Metrics

Abstract

One hundred volunteers have undergone short (5 min) listening tests in a novel multi-transducer bone-and-tissue conduction apparatus for spatial audio. The subjects subsequently described their experiences in an unstructured qualitative elicitation exercise. Their responses were aggregated to identify key themes and differences. Emergent themes are: enjoyable, informative, spatial and strange. Tactile supplementation of spatial audio display was noted in a positive light. We note that some spatial attributes are more perceptible than others. The implications for perceptual augmentation are discussed, particularly in relation to conductive hearing deficits. We conclude that the technique has potential for development and discusses future research directions.

Keywords

  • bone conduction
  • tissue conduction
  • multimodal perception
  • spatial audio
  • augmented perception
  • vibrotactile

1. Introduction

Hearing impairment is a sensory deprivation that constrains the information bandwidth available to the individual. Consequences include poor speech discernment (especially in environments with high background noise), poor auditory spatial performance and lack of pleasurable access to music. Hearing impairment can be due to sensorineural or conductive inadequacies, or both. Amelioration strategies include assistive technologies to augment individuals’ residual sensory capacities (for example: hearing aids) or to substitute alternative information pathways where one stage of auditory processing is defunct (for example: cochlea implants). Performance is generally better for communication problems than for spatial and pleasurable-listening problems. There is some evidence to indicate that age-related hearing deficits may play some causal role in the onset and progression of dementia, in part due to social disengagement because of increasing difficulty in disambiguating complex auditory scenes, and in a feedback effect, because neural pathways that receive little stimulus become less efficient [1]. Given that prevalence of hearing loss doubles with each age-decade [2], hearing rehabilitation techniques may be expected to become increasingly important as the average life-expectancy increases. There is a quality-of-life (QoL) issue here; as the auditory information-channel gradually falls into disuse, access to entertainment and intellectual stimulus, in the forms of conversation, music listening and television, becomes scarce.

There is a general trend towards heightened spatial competence in artificial audio, offering increased involvement and informativeness. We are investigating whether bone and tissue conduction techniques can be developed to provide increased enjoyment and informativeness through extended spatial impressions. We have developed a prototype 5-transducer vibrotactile tissue conduction system to display multichannel spatial sound recordings; our objective is to identify, and subsequently parameterise available qualia in this context. One hundred short listening demonstrations were conducted and responses were aggregated and examined for frequency of occurrence of adjectives and synonyms; these formed the basis of an initial set of themes. The cohort was then re-analysed to identify co-occurring themes.

Advertisement

2. Tissue conduction of sound

Techniques for utilising vibration to produce auditory percepts have been known of for centuries; 16th century Girolamo Capivaccio struck an iron rod held against the teeth to assess ear pathology; Ludwig van Beethoven used a wooden rod, with one end held between his teeth and the other resting on the piano he was able to continue his work even though considered profoundly deaf. Late 18th century saw the development of early bone conduction devices, the Fonifero in 1876 by Giovanni Paledino, and the Audiphone in 1879 by Richard Rhodes used mechanical transduction of sound to assist hearing.

These are putatively termed ‘bone conduction’ techniques, though this is a slight misnomer, as skin, fluid and the soft-tissue contents of the cranium also contribute transmission pathways to various extents. We prefer the term ‘tissue conduction’ (TC) as a more comprehensive description. There is general agreement that multiple transmission pathways are in use. Several sources record the pathways to be frequency dependant; inertial forces acting on the ossicular structure and cochlear fluids at low frequencies relative to skull vibration; high frequencies causing distortion of the temporal bone and cochlear shell. Sound generated in the occluded ear canal via Osseo-tympanic transmission provides increased sensitivity at low frequencies; contents of the skull and fluid pathways induce sensitivity at high frequencies [36]. The prominence of each pathways contribution remain in question; however, the resultant wave motion in the basilar membrane as a summed contribution of all pathways appears to be the same as for air-conducted sound; cancellation experiments between AC, BC and TC show this to be the case [3, 7, 8].

Cochlear stimulation through TC is commonly elicited using a vibrotactile transducer in contact with the skull; various body locations have also been shown effective [6]. Monaural and binaural presentation in contact with the mastoid, condyle or forehead have provided common TC stimulation sites; many additional locations on the skull have featured in research, all using singular or dual transducer presentation [3, 5, 9]. Until recently, TC conveyed audio signal but not spatial information, and so the experience was not equivalent (in this respect) to real-world hearing. Latterly, researchers have shown that a considerable of degree of lateralisation, in some case approaching that of normal binaural hearing, is feasible [1012]. Nevertheless, the results lack equivalence in terms of overall spatial performance, significantly lacking spatial attributes such as externalisation, spaciousness, range perception and elevation perception.

Advertisement

3. Multi-transducer listening tests

3.1. Equipment

The prototype array uses five BCT-1 8 Ω 90 dB 1 W/1 m tactile transducers held in a tensioned framework exerting contact force with skull through a hemi-spherical plastic medium on each transducer. For reference, the transducer locations are numbered left to right: 1—left mastoid, 2—left temporal region above the zygomatic arch, 3—forehead, 4—right temporal region above the zygomatic arch, and 5—right mastoid. Principal design considerations were that of transducer location, contact force and surface area that would work with considerable variations in head size and shape. Signal sets were processed using Reaper DAW on mac, interfaced through a Focusrite PRO 26i/o and sent discretely to each transducer through individual 1 W amplifiers; the array has a frequency range of 200 Hz–16 kHz. A set of banded style 3M Ear Plugs were available for listeners to use and compare the experience with the plugs in vs. out (Figure 1).

Figure 1.

Prototype 5 transducer array, amplifiers, interface and DAW.

Figure 2.

Emergent themes from 100 participants’ comments.

3.2. Signals

Signals were processed using Reaper DAW and spatially encoded using WigWare 1st order ambisonic panning; FX Plugins were used to construct early and late reflections and then decoded through a WigWare 1st order periphonic ambisonic decoder patched to the transducer array.

A 1st order ambisonic recording of a country park captured using a Soundfield™ microphone provided the ambient background; stereo recordings of bird sounds, voices, a steam train and music alongside mono FX clips were used to create the soundscape in which 1st order ambisonic recordings of a motorbike and aeroplane were placed.

3.3. Method

In this study, we used 100 naïve (i.e. inexperienced in TC listening) and untutored (subjects receive no instructions on target attributes) listeners, who were then invited to offer observations and comment on the experience; of the 100 listeners non-reported previous experience of tissue conducted sound. 24 female and 76 male participants took part; each was asked to record their age, sex, occupation and whether or not they were a musician alongside their comments on the experience. 24 female’s age range 16–61 years, 14 musicians and 10 non-musicians, 76 male’s age range 16–66 years, 48 musicians and 28 non-musicians; occupations were recorded for use in future analysis. For discussions of elicitation problems and techniques, see [1316]. This open-ended approach does not presuppose noteworthy attributes but is used to elicit them.

The listening tests took place across three days under non-ideal conditions at PLASA London, as part of the Exploratorium exhibit we shared the space with four other exhibitors. The Exploratorium was located on the upper level of the large exhibition hall, a large footfall and other exhibitors using amplified sound produced a considerable noise floor (see Section 5.1).

Participants were self-selecting; when any interest was shown they were invited to take in the listening test before any discussion could take place. Once seated, the headset was placed on the participants head and a short piece of music played while they were shown how to increase the overall amplitude to a comfortable level; banded ear plugs were cleaned and given to the participant to use at their discretion. Each audition lasted five minutes and the volunteers were invited to record brief details and their observations on prepared forms immediately afterward. The method of recording responses proved to be suboptimal, as many volunteers went on to describe the experience in greater detail verbally during post-test discussion than subsequently on paper.

Advertisement

4. Responses and analysis

After the auditions, the data were collated into a spreadsheet for analysis; the transcribed responses were examined for frequently occurring descriptive terms and related synonyms. This resulted in a collection of themes, the dominant theme was ‘positive’ at 78% and this was then correlated with other descriptive themes to elicit accompanying qualia that might contribute to the overall impression of ‘positive’.

4.1. Participant comment samples

  1. Male, 23, DJ, Musician

    ‘Very interesting, new experience of sound. Vibrations feel slightly unusual but also add a new dimension to the sound experience. Very cool’ (positive, interesting, vibrations, weird).

  2. Male, 28, Theatre Tech, Non-Musician

    ‘Loved the vibrations of the aeroplane flying over and in general how the sound felt all around’ (positive, spatial, surround, vibrations, external).

  3. Male, 37, Equipment Sales, Musician

    ‘I enjoyed the vibrations on the pressure points. Sound has remarkable stereo/surround perception. With ear plugs in, it felt like listening to headphones. A pleasant experience’ (positive, surround, vibrations, feel, external, headphones).

  4. Female, 18, Student, Musician

    ‘Occasionally you could feel the deeper sounds as physical vibrations especially in the front central point. The higher sounds like bird song were easier to pick up what direction it was coming from. The music sounded better than it would through regular headphones as you felt surrounded by the sound as you would in a realistic setting of an orchestra’ (positive, spatial, surround, clarity, vibrations, feel, external, headphones).

4.2. Emergent themes

ThemeDescriptors in class
PositiveNice, incredible, amazing, awesome, excellent, loved, good, enjoyed, cool, wonderful, extraordinary, impressive, effective
NegativeMuddy, muffled, lacking, limited, quiet, dull, distortion
Hearing lossHearing loss
SpatialSpatial, surround, 3D, virtual reality, image location, movement, image positioning, 360 sound-field, external
ClarityClarity, clear, crisp, pure
InterestingInteresting, fascinating
WeirdWeird, unusual, surreal, strange, uncanny, ethereal, eerie, bizarre
VibrationsVibrations, tickling, tickling
FeelFeel, felt, feeling, natural, sensorial
ExternalDistant, immersive, overhead, above, around, spacious, outside
HeadphonesHeadphones

Table 1.

Emergent themes and descriptors used for each.

4.3. Co-occurring themes

Themes were cross-correlated to elicit what impressions might contribute to overall positivity (or not) of the experience. So, for instance, ‘interesting’ mapped significantly to ‘positive’; Figure 3 shows themes mapped against positive and Figure 4 shows themes mapped against vibrations and positive combined as this forms an area of future interest.

Figure 3.

Emergent themes mapped against positive.

Figure 4.

Themes present when mapped against vibrations + positive.

Advertisement

5. Discussion

The high incidence of ‘positive’, ‘spatial’ and ‘interesting’ descriptors indicates the technique is worth investigating further, while the significant correlations between ‘spatial’ and other categories suggest that the potential informativeness may extend beyond that for single or twin displays.

Some listeners (19%) specifically commented in terms of ‘clarity’, which was interesting in the context of the background noise in the listening venue.

It is plausible that the experience is unfamiliar and not completely understandable in the short time frame, 58% (3% overlap) of listeners comments contain reference to ‘weird’ or ‘interesting’. Another, commensurate explanation is that this constitutes a different kind of experience, an artificial exaptation [17].

In respect of reports of spatial impressions, we are currently reviewing the question of whether we are actually presenting signals that are physically equivalent (to the spatial signal set one would normally apprehend via air conduction), or whether we are presenting something which achieves a degree of perceptual equivalence through more abstract relationships. In the first case, what would be implied is that we are, inadvertently, providing equivalents to those aspects of the head-related transfer functions (HRTFs) that would bear strong relationships with perceptions of externalisation and elevation. Because of the multiplicity of signal paths inside the cranium, which vary with frequency and transducer locations, differences in frequency component arrival times within grouped and segregated sensory data [36, 18] may induce interaural time difference fluctuations and hence elicit some sense of spacious envelopment [19]. We are not yet able to model the complex signal arriving at the cochleae, and in any event, it is exceedingly unlikely that we are precisely mimicking the individualised HRTFs of all listeners.

If, on the other hand, more abstract perceptual equivalence is in evidence, it is puzzling that spatial impressions are elicited in such short exposures. Informal listening tests of prolonged and repeated exposures do seem to indicate that spatial judgements improve; small sample size and uncontrolled test circumstances constrain conclusions.

An intriguing alternative possibility is that we are actually generating non-audible cues that are perceptually interpreted as auditory spatial cues. A significant (and unanticipated) contributor to the experience is that of vibration, which, in 24% of comments appears in a neutral or positive context, and in 12% is positively associated with comments on spatial impression. The vibrations are due to listening-circumstance inadequacies; the ambient noise floor was high, the transducers have limited dynamic range and modest frequency range. We assumed that vibration would be strongly associated with negative terms, but this only proves to be the case in 3% of responses. The tentative inference is that coherent (i.e. covariant with modulated auditory input) tactile input is potentially perceptually assimilable; this would exemplify multimodal perception.

Multimodality of perception has received increasing interest in the last four decades. The ubiquity of multisensory neurons (that can receive inputs from two or more sensory domains) in the brain indicates that multisensory integration is not limited to ‘higher’ cognitive processes but can occur at more fundamental levels. For instance, neurons in the primary visual cortex receive inputs from the primary auditory cortex [20]. Multimodality can be discussed in terms of cross-modal effects (where stimulus input to one modality alters the perceptual conclusions in another), for examples, the motion bounce illusion, see [21] and the McGurk effect [22]. It can also be discussed in terms of super-additive effects, where application of concurrent multimodal stimuli produces more disproportionately more robust perceptual conclusions than for unimodal stimuli [23]. For discussion of multisensory interplay, see [24].

The key observation in the question of unimodal and multimodal perception is that, while brain-region specialisation is well documented and unimodal perception is known to occur, perception in one modality can be affected by input to another and further, if stimuli to one modality are impoverished, input via another can be effectively cognitively utilised [25].

5.1. Limitations

The transducers have restricted frequency response of 200 Hz to 16 KHz and component matching is problematic.

The generic headset could not be calibrated for consistency of transducer location and contact force for each individual in such a large cohort, leading to inconsistencies in the sensory experience across the cohort.

High ambient noise levels in the listening area probably interfered with subtlety of detail in the programme material.

Although listeners were unsolicited, some arguably had prior knowledge and possible expectations borne from previous listeners’ comments. The ‘interesting’ category should be considered with caution, since, by definition, volunteers were interested enough to come forward.

Respondents were generally more fluent in their verbal descriptions than their written responses; only the written responses were recorded for analysis.

Variations in descriptions were noted; for instance, while some commented on a startling degree of clarity, others observed the opposite. Such variation may stem from variations in ‘degree of fit’ of the prototype apparatus, variations in physiology (skull thickness, for instance) or variations in biomechanical and/or neurological auditory processing.

An important limitation is that we did not categorise responses in terms of degree of emphasis, due to intrinsic uncertainties of use of language; for example, ‘very spacious’ and ‘spacious’ were categorised similarly.

Advertisement

6. Conclusions and further work

The initial qualitative investigation indicates that the use of multiple transducers, decoded to so as to display spatial and musical information, is worth further exploration. The areas of interest are: externalisation (i.e. not ‘in the head’), control of spaciousness, range perception and tactile augmentation.

In terms of informational bandwidth, there seem to be several justifications for using multiple transducers: dynamic range and frequency response of the apparatus is improved simply because more moving mass (in the transducers) and power are deployed. A general improvement in sense of spatiality is indicated, though the spatial impressions evinced are not precisely the same as for real environments, or other artificial means of depicting spatial sound. This is uncontroversial, since spatial qualia for headphone, in-earphone and loudspeaker presentations also differ. Headphones can give impressions of ‘in the head’ intimate sound fields but are correspondingly poor in producing the impression of externalisation and range-perception, while the reverse is true for loudspeaker presentations. The tissue-conducted sound fields do, reportedly, convey some sense of spaciousness, envelopment or immersiveness, indicating externalisation, though it is unclear whether range perception can be coherently controlled. Directional localisation of sources appears to be imprecise; while some remarks suggest impressions of elevation, this requires more precise investigation.

The question of whether (and how) tactile stimuli interact with auditory stimuli requires elucidation as this has valuable implications for normal and hearing-impaired listeners; auditory spatial perception might actually be augmented with coherent tactile input. There is evidence that tactile stimuli can affect auditory conclusions [26, 27] and visual perceptions [28]. To investigate this, we shall have to improve the transduction of audio signals to an extent where spurious vibrations are minimised, while additionally utilising transducers specifically manage tactile input.

In the present context, multimodality has this implication: in conditions of suboptimal conditions such as hearing deficits, background noise and display limitations, it might be possible to utilise cross-modal and super-additive effects to enhance auditory perception. Enhancements could include improved source segregation and intelligibility, along with more holistic qualia such as immersiveness and enjoyability.

References

  1. 1. Lin FR, Metter EJ, O'Brien RJ, Resnick SM, Zonderman AB, Ferrucci L. Hearing loss and incident dementia. Archives of Neurology. 2011;68(2):214-220. DOI: 10.1001/archneurol.2010.362
  2. 2. Lin FR, Niparko JK, Ferrucci L. Hearing loss prevalence in the United States. Archives of Internal Medicine. 2011;171(20):1851-1852. DOI: 10.1001/archinternmed.2011.506 171/20/1851 [pii]
  3. 3. Stenfelt S, Goode RL. Transmission properties of bone conducted sound: Measurements in cadaver heads. The Journal of the Acoustical Society of America. 2005;118:2373-2391. DOI: 10.1121/1.2005847
  4. 4. Sohmer H. Reflections on the role of a traveling wave along the basilar membrane in view of clinical and experimental findings. European Archives of Oto-Rhino-Laryngology. 2015;272:531. DOI: 10.1007/s00405-014-3045-z
  5. 5. Jun KC. Objective measurements of skull vibration during bone conduction audiometry [thesis]. Medical Faculty: Zürich University; 2009
  6. 6. Lenhardt ML, Shulman A, Goldstein BA. Bone-conduction propagation in the human body: Implications for high-frequency therapy. The International Tinnitus Journal. 2007;13(2):81-86
  7. 7. Dietz AJ, May BS, Knaus DA, Greeley HP. Hearing protection for bone-conducted sound. In: New Directions for Improving Audio Effectiveness; Meeting Proceedings RTO-MP-HFM-123, Paper 14. Neuilly-sur-Seine, France: RTO; 2005; p. 1–18
  8. 8. Chordekar S, Kriksunov L, Kishon-Rabin L, Adelman C, Sohmer H. Mutual cancellation between tones presented by air conduction, by bone conduction and by non-osseous (soft tissue) bone conduction. Hearing Research. 2012;283(1–2):180-184. DOI: 10.1016/j.heares.2011.10.004
  9. 9. McBride M, Letowski TR, Tran PK. Bone Conduction Head Sensitivity Mapping: Bone Vibrator. ARL-TR-3556. MD: U.S. Army Research Laboratory: Aberdeen Proving Ground; 2005
  10. 10. Stanley R, Walker BN. Lateralization of sounds using bone-conduction headsets. In: Proceedings of the Annual Meeting of the Human Factors and Ergonomics Society (HFES 2006); 16-20 October 2006; San Francisco, CA. p. 1571-1575
  11. 11. MacDonald JA, Henry PP, Letowski TR. Spatial audio through a bone conduction interface. International Journal of Audiology. 2006;45:595-599
  12. 12. Stenfelt S, Zeitooni M. Binaural hearing ability with mastoid applied bilateral bone conduction stimulation in normal hearing subjects. The Journal of Acoustical Society of America. 2013;134(1):481-493. DOI: 10.1121/1.4807637
  13. 13. Berg J, Rumsey F. Spatial attribute identification and scaling by repertory grid technique and other methods. In: Proceedings of the Audio Engineering Society International Conference on Spatial Sound Reproduction (AES16); 10-12 April 1999; Rovaniemi, Finland.
  14. 14. Zacharov N, Koivuniemi K. Audio descriptive analysis and mapping of spatial sound displays. In: Proceedings of the 7th International Conference on Auditory Display. 29 July–1st August 2001; Helsinki, Finland. p. 95–104
  15. 15. Guski R. Psychological methods for evaluating sound quality and assessing acoustic information. Acustica. 1997;83:765-774
  16. 16. N. Ford, F. Rumsey, B. de Bruyn, Graphical elicitation techniques for subjective assessment of the spatial attributes of loudspeaker reproduction—A pilot investigation In: Proc. Audio Engineering Society (AES) 110th Convention, Amsterdam, 2001
  17. 17. Gould SJ, Vrba S. Exaptation—a missing term in the science of form. Paleobiology. 1982;8:4-15
  18. 18. Bregman AS. Auditory Scene Analysis. MIT Press: Cambridge, MA; 1990
  19. 19. Rumsey F, Mason R. Interaural time difference fluctuations: their measurement, subjective perceptual effect, and application in sound reproduction. In: Proceedings of the Audio Engineering Society Conference: 19th International Conference: Surround Sound-Techniques, Technology, and Perception (AES19); 21-24 Jun. 2001; Schloss Elmau, Germany
  20. 20. Murray MM, Thelen A, Thut G, Romei V, Martuzzi R, Matusz PJ. The multisensory function of primary visual cortex in humans. Neuropsychologia. 2015, 2015; PMID: 26275965
  21. 21. Sekuler R, Sekuler AB, Lau R. Sound alters visual motion perception. Nature. 1997;385:308
  22. 22. McGurk H, Macdonald J. Hearing lips and seeing voices. Nature. 1976;264:746-748
  23. 23. Sumby WH, Pollack I. Visual contribution to speech intelligibility in noise. Journal of the Acoustical Society of America. 1954;26:212-215
  24. 24. Driver J, Noesselt T. Multisensory interplay reveals crossmodal influences on ‘sensory-specific’ brain regions, neural responses, and judgments. Neuron. 2008;57(1):11-23 https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2427054/
  25. 25. Stein BE, Meredith MA. The Merging of the Senses. Cambridge, MA: MIT Press; 1993
  26. 26. Tajadura-Jiménez A, Väljamäe A, Kitagawa N, Ho H. Whole-Body Vibration Influences Sound Localisation in the Median Plane. In: Proceedings of the 10th Annual International Workshop on Presence. Barcelona; Spain; October 2007
  27. 27. Caclin A, Soto-Faraco S, Kingstone A, Spence C. Tactile “capture” of audition. Perception & Psychophysics. 2002;64(4):616-630
  28. 28. Violentyev A, Shimojo S, Shams L. Touch-induced visual illusion. NeuroReport. 2005;16(10):1107-1110

Written By

Peter Lennox and Ian McKenzie

Submitted: 11 September 2017 Published: 18 October 2017