Semiotic Principles in Cognitive Neuroscience

Abstract

Cognitive neuroscientific approaches to language(s) and brain in the twenty-first century have made an important contribution to understanding the importance of the relationship of invariance and variation of language mappings across individuals, the dynamic nature of neurological processing of languages throughout the life cycle, and more ecologically valid modeling of cognitive processing that focus on the interactive nature of linguistic perception and production in the cultural context. Beginning with Ojemann’s unique contribution to language mappings through cortical stimulation mapping (CSM) and continuing through to fMRI studies, contemporary neuroscience research paradigms have moved toward analyzing neural networks and connectivity, the relevance of embodied cognition, and the complex nature of signification and meaning-generation. This chapter presents important interactions between recent cognitive neuroscience studies of language and brain using proficiency with semiotic principles and semiotic theory as given in Peirce, Eco, Lotman, and Sebeok.

“For an event to become linguistic…a great many brains must play in unison” – D. Bolinger ([1], p. 233)

“Nevertheless, it deserves more emphasis that functional localization is an intermediate goal, or the ‘homework problem’ that requires figuring out where things are, before scrutiny turns to the harder and deeper question of how things work.” D. Poeppel ([2], p. 4)

1. Fundamental concepts and principles

One of the most important principles recognized by the contemporary cognitive neurosciences is the commitment to conducting ecologically valid research and experimentation [3, 4]. Depending on the particular subfield of cognitive neuroscience, there will be different ways to achieve ecological validity. For example, in the field of cognitive neurolinguistics, a straightforward approach to ensuring ecological validity is by constructing protocols that are designed around dynamic categories that are found in normative speech, including speech acts ([5], p. 196). In addition to this guiding principle, it is imperative to include robust empirical methods at all levels of analysis, whether the object of study is based in experimentation with human subjects (healthy and lesion-deficit, behavioral and neuroimaging, laboratory and in situ) or textual studies. The questions discussed here focus on identifying the key areas where semiotic theory has impacted the cognitive neurosciences, which include approaches to understanding sensory-motor mappings in the human brain, multimodality as opposed to modularity, and embodied cognition. The contribution of semiotic principles to reliable scientific research in cognitive neuroscience is discussed in the conclusion.

2. What is semiotics?

It is often the case that in research that identifies itself as “semiotic,” there is not always a clear explanation of what it is and what it means. In the present work, we will synthesize the epistemological approach to semiotics and semiosis found in Peirce, Sebeok, Lotman, Jakobson, and Eco, where semiotics as an epistemology is articulated originally by Locke as a “doctrine” and evolves into what Sebeok calls “a peculiarly human form of inquiry that is in its essence a theory of perception” ([6], p. 12). For all of these approaches, signification is primary and the basis for communication and text generation.¹

Umberto Eco points out that semiotics in the United States and France was deeply associated with structuralism in the context of research on cultural and linguistic phenomena in the 1960s ([9], pp. ix-x). It is also at this time that Lotman, working in Estonia, publishes a fundamental work on semiotic methods in the study of literature, in which he calls for the elimination of the opposition between exact science and humanities, it presents semiotic systems as modeling systems that both construct and explain the world in which humans live, all cultural spaces are based on multiple linguo-cultural codes, and it considers semiotics to be a cognitive science.² These modeling systems are not ahistorical and are embedded in complex systems of “non-hereditary collective memory” [11].

Peircean semiotics is a clear departure from the structuralist tradition and has strong ties to the philosophy of science and provides a framework for categorizing triadic sign types in a fundamentally non-binary modeling system. Peirce’s most important contribution to semiotics is found in his theory of signs (minimally triadic, never binary) and types of inference (Peirce 2.246, 2.250-2, [12, 13]). For Peirce, the triadic sign always involves (1) its relation to itself (sign-signrelationship), (2) relation to its object (sign-dynamic object relationship), and (3) its relation to its interpretant (sign-final interpretant) (ibid.).

It was Jakobson who introduced Peirce’s icon/index/symbol triad (part of the sign-object relationship) into modern linguistic theory. Through this triad, Peircean semiotic theory became widely known in linguistics and was applied in works that demonstrated the significance of the non-arbitrariness of the linguistic sign [1]. Non-arbitrariness must be understood as a relative phenomenon, not an absolute, and is particularly powerful at the phonological, morphological, and morphophonemic levels in languages. Much later, sociolinguistics and identity studies borrowed the terminology as well but with changes (cf. [14]).

3. The semiotic approach and semiosis: no more binaries

The semiotic perspective requires a re-evaluation of research techniques and applications. Such a re-evaluation brings into question the very object of study, as well as the instrumentation used to evaluate the object. Once it is recognized that the focal point of Peircean semiotic theory is not the sign, but semiosis (the sign in action, the prerequisite process of the exchange of information [signification]), the next step is to take a closer look at the interface between functioning signs as they are used [6, 10].

Semiotic theory that focuses on signs in action provides an epistemological framework where any linguistic speech act includes the users of the language, and codes and messages are embodied in the users themselves. This idea can be extended to neuroscience and can provide the basis for proposing that the sensory-motor neural image of a given linguistic phenomenon, be it grammatical or lexical, is multisensory and involves cross-modal effects—a notion that is central to all of the current neuroscience research on embodied cognition (including [15, 16]). Another outcome of a theory of dynamic signs requires treating referential meaning as a type of meaning—not the only kind of meaning and not even the most important kind in all instances. In other words, the outcome of this framework is that linguistic meaning is never solely determined or determinable by reference alone.³ This outcome has a profound impact on the modeling of multimodality as an organizing principle of human cognition.

3.1 Defining icon/index/symbol: redefining non-arbitrariness

Following Peirce (Peirce 8.368, [13] pp. 4–23;[21]), the icon is defined as a qualitative likeness between a sign and its object. The quality upon which the similarity is based belongs to the sign, whether or not its object exists. The index is a real connection between the sign and its object, and the relationship is given via the dynamic object of the sign. The symbol is a general rule between the sign and object, and the object is related to the sign via the interpretant. [Jakobson’s rendering of these terms is reanalyzed back into the dyadic terms signifier/signans and signified/signatum, similarity/contiguity (i.e., paradigmatic/syntagmatic axes), and the notions of factual (effectual) and habitual [22, 23]] Each of these 3 sign-object relationships are embedded in each other such that none of these categories may ever occur alone, the icon is embedded in the index and symbol, and the index is embedded in the symbol (Peirce 2.306, 4.447 [21]). Note the following diagram with Jakobson’s terminology: Figure 1

Figure 1.

Peirce’s sign-object triadic relation.

The principle of iconicity, as based on Peirce’s definitions, is critical to the reimagining of the non-arbitrariness of the linguistic sign as a relative phenomenon. Bolinger ([1], pp. 18–23), in his work on morphemes, presents interesting examples from English of phonological iconicity at the syllabic and lexical levels, including “hard of hearing” (alliterative/iconic phonological properties, [h_r/h_r], and “short circuit,” [rt/r_t]), not “hard of seeing/smelling” and “long circuit.” Bolinger also exposes important iconic relationships in prosody in later works and fine-tunes his approach to relative non-arbitrariness ([24], pp. 92–93).⁴

3.2 The importance of the interpretant: how to talk about meaning

Peirce posits three basic types of interpretants in the sign complex—immediate, dynamic, and final (Peirce 8. 315, 8.372, [13]). The interpretant is the level of the triadic sign where the meaning of the sign is realized:

“…the meaning of a sign is the sign it has to be translated into;” (4.132)

“creates in the mind…an equivalent sign, or perhaps a move developed sign;” (2.228)

“the proper significate outcome of the sign” (4.127)

It is interesting that while linguistic applications of Peirce are rich in the use of the icon/index/symbol triad, this is not the case with interpretants.⁵ However, the one fundamental point that has persisted is the certainty of translation and the recognition that all translations change meanings. Lotman ([28], p. 15) does not ever use Peirce’s term but makes an important argument about the inevitability of tensions that arise in any speech/communication act. In so doing, Lotman articulates a new perspective—the most valuable information derived from speech acts is a result of the translation of that which is seemingly untranslatable.

4. Cognitive neuroscience studies of languages

“How does fMRI create images of neuronal activity? The short answer is that it does not. Instead, fMRI creates images of physiological activity that are correlated with neuronal activity” ([21], pp. 127-8).

4.1 Understanding activations: defining the relationship between the generation of meaning(s) and activations

Imaging techniques that involve the hemodynamic response (PET, fMRI) only measure correlates of neuronal activity and processes and not actual neuronal firings. Electrophysiological techniques (including EEG, MEG) can measure electrical activity in ensembles of neurons and changes in magnetic fields related to neuronal activity.

Invasive direct single-neuron mappings, as found in cortical stimulation mappings (CSM), are only conducted during surgery. George Ojemann, who developed the technique, conducted over 1100 surgeries that included CSM mappings to identify areas important to production and comprehension of language(s). This technique was developed in order to identify areas related to important functions of production (motor) and comprehension (sensory) in language(s) and has proven successful in preserving these functions in surgeries that require removal of tissue. Dr. Ojemann has published over 170 papers of CSM data with important conclusions about neural mappings of language and languages [29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41].

Understanding activations found in BOLD fMRI studies presents a significant challenge.⁶ When considering the results of fMRI studies of language(s), one finds a much broader range of activations across both hemispheres in areas not represented in the traditional Broca/Wernicke targets from the “classical model”; these results are in keeping with research findings in fMRI language studies and represent a consensus among the neuroscience community (cf. [4, 5, 45, 46, 47, 48, 49, 50, 51]). There are challenges in analyzing neural activations, and Raichle reminds us that it is:

“…impossible to distinguish inhibitory from excitatory cellular activity on the basis of changes in either blood flow or metabolism. Thus, on this view a local increase in inhibitory activity would be as likely to increase blood flow and the fMRI BOLD signal as would a local increase in excitatory activity” ([48], p. 12).

Other challenges in using fMRI for cognitive studies include the timing delay in the hemodynamic response that lags behind neuronal activity. It is essential to remember that there can be no one-to-one relationship between neural activations acquired during imaging and the subject’s knowledge or ability.

Interpreting the activations recorded from ROIs in fMRI language studies is strengthened when significance can be found with behavioral data (e.g., empirical proficiency testing data) using multivariate analysis of covariance (MANCOVA). One such example is a longitudinal study of second language acquisition that also includes Common European Framework proficiency testing data [5, 45]. The use of empirical language proficiency data is particularly important for bi- and multilingual fMRI studies, where it is not atypical that a subject may demonstrate higher levels of activation in a language that they do not know well (or at all) than in a language that is the L1 or highly proficient L2. Abutalebi et al. [52] also includes language proficiency and show that it contributes explanatory power in understanding language switching and cognitive control in bi-/multilinguals. Birdsong notes that brain imaging studies demonstrate that second language (L2) proficiency, not age of acquisition, is “the strongest predictor of degree of similarity between late learners & monolingual natives” ([53, 54] pp. 24-5).

The importance of developing ecologically valid protocols for imaging experiments is another factor that can strengthen validity in interpreting activations. Other considerations in protocol development include protocol design that lends itself to the subtractive method that is often applied in fMRI experiments and minimizing “confounding factors” ([55, 56], p. 290). Activation levels resulting from task-based fMRI, by themselves, may not be interpretable (activation levels do not necessarily correlate with knowledge). Thus, the importance of including other statistical models and empirical measurements (e.g., proficiency) becomes critical for strengthening conclusions.

Raichle [57, 58, 59, 60] proposes the default mode network (DMN) as one approach to understanding the changes in activations where specific brain regions decrease their activity during a task condition. Gusnard and Raichle ([61], p. 689) also suggest a way to characterize “tonically active areas” by distinguishing between “functionally active” and “activated.” For Raichle, the DMN is one of the most important of the hierarchical networks, and it plays a central role in coordinating among brain systems and their interactions across system boundaries.⁷ The DMN and other such networks point toward the heightened interest in the application and inclusion of resting state fMRI in protocols and connectivity models [62, 63]. Functional connectivity modeling and analysis in recent neuroimaging studies are an important move away from older approaches that focus on modularity and localization.

4.2 Embodied cognition and languages

Sensory-motor interactive modeling of language and brain systems has become an important part of cognitive neuroscientific discourse over the past 15 years. The debates concerning embodied cognition are central to acquiring and developing new methods in order to understand the neurological interface of human languages, the relevance of multimodality and functional connectivity, and a rejection of modularity for modeling the processing of language(s) in the brain. Gallese and Lakoff [15] emphasize the importance of multimodal modeling of language and brain, because this perspective not only (1) takes into account its evolutionary trajectory but (2) characterizes these linguistic structures as part of the sensory-motor system at the neuronal level. Multimodal modeling, if acknowledged, requires a rejection of the modular view of language mappings in the brain.

The central arguments presented by Gallese and Lakoff stress the importance of multimodality, including mirror, premotor and parietal neurons, and the realization of multimodalities through functional clusters ([15, 64], p. 458). Approaches based on principles of embodied cognition will play an important role in developing robust theories of brain and language(s).

As I have noted earlier, “the kind of sensory-motor alignment that Gallese and Lakoff present is but one type of the significant multimodal aspects of human language and the brain, and the cautionary statements given in Mahon and Caramazza [16] are important to keep in mind” (2013, p. 136). Mahon and Caramazza, while rejecting a disembodied cognition hypothesis, argue that it is necessary to understand “whether the motor system is activated due to ‘leakage’ of (or cascading) activation from an ‘abstract’ conceptual level’, or occurs in parallel to (or independently of) activation of the ‘abstract’ conceptual level?” ([16], p. 60). While they suggest that some concepts might include sensory-motor information, they are not willing to concede the case with abstract concepts ([16], p. 60):

“For abstract concepts there is no sensory or motor information that could correspond in any reliable or direct way to their ‘meanings’.”

While Gallese and Lakoff provide one important form of the alignment (e.g., English lexeme grasp ([15], p. 457)), one can identify a more pervasive sensory-motor synthesis in three ways:

1. The realizations of specific embodied forms of grammatical and lexical meaning as produced/articulated and perceived (cf. Bolinger’s/sl/and/gr/lexemes in English—slippery, sleezy, slimy, slinky, etc.; grasp, grip, grab, grub, greed, etc.)

2. The specific gestures that accompany language-specific lexical categories and sound-based alternative systems of auditory perception (lyric and music)

3. Visual meanings given in written language (e.g., to/two/too, sea/see/C] or ideograms) that are not given in the sound forms (see also Andrews, 2013, pp. 128-137)

As noted in Andrews [45], all forms of linguistic meanings are negotiated in context via speech acts, and these speech acts “are always multiples and are embedded in… speech communities and communities of practice,” which are, in the end, what one could call “the inalienable context” ([45], pp. 196, 198). And while sensory-motor systems are internally determined, there is never “language in the one” ([45], p. 198):

“Language is a consequence of humans interacting in cultural space…We are always multifaceted users of language; we play the roles of speakers, hearers and observers (sometimes simultaneously), and we as users are defined by the multiple and variegated…speech communities and communities of practice in which we language” (ibid.)

4.3 Empiricism and interpretation of results

“What is important in a model is not its accord with experiment, but, on the contrary, its ‘ontological range,’ in which it states the manner in which the phenomena take place and in which it describes their underlying mechanisms.” Thom ([65], p. 111)

Strong empirical methods that yield repeatable results are an important component of achieving reliable conclusions in the analysis of neuroimaging data. Standard software programs typically used in fMRI analysis (e.g., FSL, SPM) are an important component of the statistical methods used in analysis, and these systems continue to be expanded and improved upon. Collecting behavioral data that can be used to correlate with imaging data is another important way to strengthen confidence in the results of the analysis. Below are four major points that describe the advantages of empirical data and analysis that includes behavioral “can do” data in opposition to more traditional experiments that put more emphasis on static, essentialist characteristics of the experimental subjects:

• Study participants demonstrate ability (i.e., dynamic) using empirical measurement tools relevant to establishing baselines.

• Usage of internationally recognized proficiency measurements provides quantitative data that can be understood across studies.

• Behavioral study data enhances the CONTEXT for framing the analysis and interpretation of empirical data.

• Given the challenges of understanding and interpreting the hemodynamic response and activations acquired in fMRI, MANCOVA and other statistical methods strengthen the validity of the analysis and emerging conclusions.

• Essentialist categories are static, do not provide empirical data for analysis, do not reflect the importance of learning, and are not supported by neurological models of brain development and cognition.

5. Biosemiotics, markedness, and autopoiesis: precursors to understanding the importance of multimodalities and embodied cognition

…experience is moored to our structure in a binding way. We do not see the “space” of the world; we live our field of vision….we invariably find that we cannot separate our history of actions – biological and social – from how this world appears to us. ([66], p. 23)

The impact of Maturana and Varela’s formulation and examination of autopoiesis (a neologism) on linguistics, semiotics, and later on the cognitive sciences is quite significant. The term itself was the result of a search for a word without a history that could adequately define what they considered to be the essence of the organization of the living system—autonomy, self-reference, and “circular organization” ([66], p. xvii). In his introduction to Biology of Cognition ([66, 67], p. xv), Maturana describes how his work on vision and visual perception led him to two specific conclusions—one being the “nervous system [is] a closed network of interaction neurons” and the second being the realization that the study of cognition is “a legitimate biological problem” (ibid.). As a result, a new set of research questions emerged: how do organisms obtain information about their environments, and how is it that organisms have structures that allow them to operate in their existing environments ([66], p. xvi)?⁸

One of the significant outcomes of the epistemological approach of autopoiesis for semiotics and the cognitive sciences is the reinforcement of semiotic principles that were articulated in a different context by Jakob von Uexküll [68, 69] and Yury Lotman [28, 70], which include Uexküll’s Umwelt and functional circle and Lotman’s semiosphere. In Beer’s preface to Maturana and Varela’s book ([66], pp. 63–72), he clearly articulates the importance of the new approach as a metasystemic synthesis that is NOT interdisciplinary; rather, this new approach transcends disciplines ([66], p. 65). This is very much in keeping with the modeling proposed by Lotman (see Umberto Eco’s introduction to Shukman’s translation of Lotman’s Universe of the Mind, 1990).⁹

While we cannot provide here a full articulation of Maturana and Varela’s important work, there is one other major point that must be noted in this context—their definition of the observer. One of the two guiding principles given in their 1990 and 1992 books is “everything said is said by an observer” (1990, pp. xxii, 8; 1992, p. 26). For Maturana and Varela, the observer is defined as a living system who can only be accounted for when cognition is seen as a biological phenomenon (1990, p. 9). This statement, along with their definition of four consequences of autopoietic organization (1990, pp. 80-81),¹⁰ takes us to the brink of the twenty-first century cognitive neuroscience methods and analysis as defined by perspectives that are more multimodality-oriented than modular and that give a prominent place to sensory-motor systems in explaining language processing in the brain.

6. By way of conclusion: frames of interaction and understanding functional connectivity and neural networks (DMN and others)

In the end, it is clear that cognitive neuroscience and neuroimaging continue to incorporate important ideas, even if it is more often than not in a nonconscious and unarticulated manner, from a range of anthroposemiotic and biosemiotic models described in the previous sections. One of the most powerful and recent outcomes of this important interaction of research methods and concepts is seen in the move away from focusing imaging analysis within regions of interest to understanding the networks and functional connectivity in operation during not only task-based fMRI but resting state fMRI. The first network that was identified is known as the default mode network (DMN) [57, 59, 60]. Current research demonstrates that there are multiple such interactive networks and the details of these networks are clearest in studying intrinsic functional connectivity in individuals, not groups [72]. The recognition of neural functional connectivity and networks is in keeping with the autopoietic approach given by Maturana and Varela, where they foresee such an outcome:

The fact that we can divide physical autopoietic machines into parts does not reveal the nature of the domain of interactions that they define as concrete entities operating in a physical universe (1992, p. 82).

As fMRI imaging methods shift to include individual and group data analysis and protocols expand from task-based to include resting state, there will be more evidence contributing to our understanding of the default network as “the apex transmodal association network” [72].

The interaction of semiotics and the cognitive neurosciences has not been generally noted at the level of discrete analyses or individual experiments but becomes more obvious and prescient when the philosophical and epistemological underpinnings of the two disciplines are made explicit. That is one of the goals of the present analysis. The other is to draw attention to the importance of multidisciplinarity and the obligatory re-evaluation of disciplinary boundaries that has emerged within semiotics and the cognitive neurosciences. The major frames that support both of these efforts are found in the importance of ecological validity in research, recognition of connectivity modeling and multimodal perspectives, the re-evaluation of arbitrariness and the relative nature of non-arbitrariness, and the inalienable role of context both in neurological networks and signification systems.