Cognitive Semiotics: An Overview

Abstract

This chapter revises evolving theories on cognition in relation to semiotics, the transdisciplinary study and doctrine of sign systems, and meaning-making. Cognition entails very complex networks of biological processes and actions that encompass perception, attention, manipulation of objects, memory mechanisms, and the formation of knowledge by means of direct experience as well as by learning from others, for which forms of communication and comprehension are also necessary. In view of this complexity, many different disciplines are involved in the study of cognition. These include neuroscience, anthropology, psychology, sociology, philosophy, semiotics, linguistics, and more recently, computational intelligence, information processing, and neural networks used in machine learning, to name but a few. The chapter opens with an introduction to the field of cognitive semiotics and continues with a brief presentation of the interdisciplinary evolution of the 4Es. It also includes an in-depth discussion of Peircean semiotics in relation to the approaches known as wide cognition.

1. Introduction

Jordan Zlatev defined cognitive semiotics ‘CogSem’ as a transdisciplinary field that integrates “methods and theories developed in the disciplines of cognitive science with methods and theories developed in semiotics and the humanities, with the ultimate aim of providing new insights into the realm of human signification and its manifestation in cultural practices” [1]. Zlatev explains that as a transdisciplinary pursuit, the study of semiotics is concerned with “the overarching unity of knowledge” [2].

CogSem has been mainly associated with the Center for Semiotics (CfS) established in 1995 in Aarhus, Denmark by Per Åage Brandt, with researchers such as Frederik Stjernfelt, Peer Bundgaard, Svend Østergaard or Riccardo Fusaroli, among others. The Centre for Cognitive Semiotics (CCS) at Lund University (Sweden) was created in 2009, with members such as Göran Sonesson and Zlatev himself. An international journal, Journal of Cognitive Semiotics (JCS), is running since 2007. The International Association for Cognitive Semiotics (IACS) was established in 2011 in Lund. Research by the so-called Grupo μ at Liège, Belgium, is also worth noting.

Semiotics occupies a transdisciplinary area that can contribute to build bridges between various disciplines; in this case cognition and its material forms of instantiation. Defined as ‘the action of signs’ [3], semiotics has recognized the inter-actions and intra-actions of anything acting as a sign; present also within the framework of the 4Es (embodied, embedded, enacted, extended). At a systemic scale, semiotic interactions include specific media channels, with the term ‘medium’ understood in its broadest sense, including bio-entities along with artefacts and technologies that can be material, and thus physically perceived by humans and animals, but also digital, which can lie beyond the scope of perception. The lines that follow attempt to advance towards a clearer formulation in this regard and point out some of the potentials of semiotic studies, focusing in particular on Charles S. Peirce’s contribution.

2. Embodied cognition

Research on ‘Embodied Cognition’ was popularized in the 1980s [4, 5]. This approach is mostly human-centred and contemplates biological factors and bodily experience, notably body symmetry, perception, and motor interaction involving the physical manipulation of objects, as affecting the formation of abstract mental structures known as ‘image schemas’ [6]. There is still no consensus about the specific nature of image schemas (the term ‘model’ is preferred by cognitive anthropologists such as David Kronenfeld). For instance, from the perspective of cognitive linguistics, a noun instantiates a schema related to ‘thing’ whereas a verb instantiates the schema related to ‘process’ [7]. Propositions are then combined into larger-scale mental and discourse structures by way of metaphorical and metonymic conceptual mappings, explained by shifts from one conceptual domain (the target domain) to another conceptual domain (the source).

Schemas could be simplified pre-conceptual experiences turned into abstract mental structures. They define how humans make sense of the world within specific perceptual and cognitive domains. Since they have functional biological bases, some schemas are basic in the sense of being irreducible to anything more fundamental. For instance, the temporal duration and spatial perception are realms of potential experience within which conceptions emerge through analogic structures that are in relation to perceptual and motor-experience (dynamic inter-actions are subject to physical constraints such as the pull of gravity) and human bodily orientation.

Thus, embodied cognition considers that abstract and high-level cognition is explained in terms of physical experiences (body as a container, based on symmetry, balance, and centre-periphery experiences; action explained as source-path-goal schema, and so on). Schemas are imported from these pre-conceptual structures by way of metaphorical and metonymic conceptual mappings. Even the language of emotions (i.e. ‘you broke my heart’) largely reflects culturally mediated conceptualisations of feelings in terms of body parts, transferred across domains through conceptual metaphors and metonymies.

Ultimately, schemas might be based on the human ability to detect and recognize recurrent patterns and establish mappings or conceptual correspondences from the source, generally more abstract, to the target, more grounded on the physical world [8]. Neuroscientist Antonio Damasio has identified that fundamental mappings taking place in our brain are those that register the internal state and condition of the body, as well as those that map in relation to the external environment [9].

However, sociocultural aspects are also important because humans share an inborn basis for social interaction. This is manifested for instance in their capacity to follow someone’s gaze, to read intentions, face recognition, and so on. Collectively, these factors provide a universal neuro-physical basis for cognitive development, also grounded on sociocultural interaction. Applying the framework proposed by Merlin Donald, Jordan Zlatev explored the re-enactment or representation via imitation as a fundamental prerequisite of shared communication. Donald defined ‘mimesis’ as “the ability to produce conscious, self-initiated, representational acts that are intentional but not linguistic” [10] and found it to be rooted in movements, involving a range of aspects from facial expressions, eye movement, gestural signs, and tones of voice, including the communication of emotional states. He recognized the dependence of mimesis on joint attentional frames, that is, deictic markers such as child and mother gaze interactions. Donald observed that this ability operates based on the metaphorical principle of perceptual resemblance, enhanced by repetition [11]. While he emphasized metaphor as a cognitive tool, recent research also notes the importance of metonymy, because an enactment evokes the imagery of a background scene and the elements within it (in Peirce’s terms, ‘indexical proximity’).

Paul Friedrich aligns image schemas to Peirce’s notion of ‘Firstness’, explaining that they depend on qualities (Peirce’s ‘qualisign’ refers to a quality, a feeling or a possibility of functioning as a sign) that are primary or irreducible, leaning towards physicality, and dependent on emotional content. Peirce wrote that a quality “cannot act as a sign until it is embodied; but the embodiment has nothing to do with its character as a sign” [12, 13]. Even habits, routines, and general laws (that Peirce integrates in ‘Thirdness’) are embodied in actual things or events (themselves manifesting a range of possible qualities). Thus, ultimately, the fact that something is interpreted as a sign depends on material phenomenological aspects. Peirce’s well-known classification of signs as icons, indexes, and symbols is based on how signs are related to their objects. An icon signifies its object by virtue of shared qualities, an index by virtue of a causal relation, and a symbol by virtue of an action ruled by a norm or habit [14] (see the section below for more details).

‘Secondness’ initiates relations between domains and is essentially dyadic. According to Friedrich, the modal forms that accompany some verbal processes include expressions of mood that run from emphatic assertion to passivity and emotional content such as outrage, fury, joy and sadness, sarcasm, threat, and irony. These can interact and combine with each other introducing a ‘sinsign’, a sign usually consisting in a reaction/resistance, or an actual singular thing, occurrence, or fact. Peirce also held that an index can be a general thing (not only singular; the etymology of ‘seme’ points in this direction). For instance, a symptom of a disease, a label, a diagram (which can be both iconic and indexical), a proper name, a pronoun, etc. Indexes or pointers make connections through spatiotemporal proximity or contiguity, crucially bound up with the situation or context. If interpreted as linguistic signs in triadic symbolic relations, pointers become personal pronouns (I, you, we, they, etc.), deictic adverbs (here/there, now/then), demonstratives (this/that), and grammatical categories of tense and aspect, all of which that are situationally contingent.

Thus, starting at the lowest level, embodiment creates the potential for schemas to emerge, as conceptual blending theory has shown [8]. Schemas are associated with experientially based forms of behaviour, specific to certain situations. The cognitive operations originating in bodily experience pass through processes of metaphorical and metonymic projections based on recognition of patterns presented in experience. Once corresponding qualities between material and mental spaces have been mapped, their integrative projections yield symbolic signs. As structures become more symbolic, their connection with bodily experience turns more indirect, and cultural particularities emerge. Particularities are filled-in with salient ‘situated’ cultural content within specific population groups. Conceptual integration includes out-of-awareness forms of cognition, such as tacit knowledge of what is possible, permissible, and acceptable within a community [15].

3. Embedded cognition

The debate on embedded cognition (sometimes known as ‘sociocultural ‘situatedness’) began to take place within ethnographic anthropology (i.e. Franz Boas) early in the 20th century. It argued against previous ideas on the universal basis of human language and defined ways in which human cognitive processes are shaped by social interactions and cultural structures and practices. In terms of linguistic studies, it was found that different languages classify experiences differently and that linguistic categories are in close relation to thought patterns. For instance, greeting gestures and speech acts of salutation are very different across cultures. These studies also formed the basis of the hypothesis of linguistic relativism (Sapir-Whorf hypothesis), that is, the idea that the structure of a language affects its speakers’ cognition and worldview. It was also present in connectionism, in which mental phenomena are seen as interconnected neural networks of units [16, 17, 18] (for a recent overview of the semiotic of gestures [19]).

Another example of how contextual factors, alongside particular language uses, shape the way metaphors are employed and add nuances to the purely cognitive account of conceptual metaphor/metonymy as transferring inferences across domains is the research pursued by Lionel Wee. He notes the shifting of conceptual models from the correspondence model [4] to the class-inclusion model [20]. The first assumes consistent horizontal or lateral relationships between source and target so that particular relations between objects and their properties are preserved. The second model operates vertically including metonymic displacements; so that the source domain is merely treated as a prototypical instantiation of a newly created superordinate category, which is then seen to encompass both the source and target. In the correspondence model a metaphor like LIFE IS A JOURNEY understands goals in life as destinations, and difficulties as impediments in the motion of progress. The class-inclusion model tries to explain expressions like MY JOB IS A JAIL, where both job and jail are understood in terms of a superordinate category defined as including unpleasant situations, confining, etc. [20]. Wee points out that in post-capitalist late modernity experiences are commodified and contemplated as functional resources, determining how metaphors are constructed. According to him, it is important to pay attention to indexes that, in the case of EXPERIENCES ARE RESOURCES might point to particular skills that serve as indicators of value within experiences [21]. The metaphor EXPERIENCES ARE RESOURCES also highlights the increasing influence of ‘small stories’ and under-represented tellings in late modernity, evidencing in language the reframing of experiences from one context to another.

4. Enactivism

In the 20th century, the impact of cybernetics upon human theoretical models saw the growth of systemic forms of explanation. A system can be defined as a non-linear dynamic set of actors, relations, objects and things, and all their intra- and inter-connections. Systems can be biological, for instance, an ecosystem, but also cultural, situated in a particular environment, place, and time. Systems can be open or closed to their surroundings. Closed systems have boundaries or walls, often defined artificially, like territorial borders. Although finite to a certain extent and with degrees of closeness, different systems are interconnected. For instance, in the human body, the digestive system functions in relation to the respiratory system, circulatory, and all other bodily systems. Operating in a sort of network, the distribution of system components can vary, which means that a given system can acquire different states in a short time-span while remaining the same in a longer time span. Thus, a state of a system is understood as a momentary position in space and time. This position depends on physical properties (space-context) as well as on the distribution of these properties in a particular time. To have a complete understanding of how a system works, we need to contemplate it from an integrated approach that looks at the full spectrum of scales, networks, states, and multiple spatiotemporal dimensions, considering the intra- and inter-actions of all system components.

A systemic approach in semiotics is evident, for instance, in Actor-Network Theory (ANT) which, in the 1980s, stressed that the social is constituted by systemic networks and relationships created among people through the use of artefacts. These relations are simultaneously material (physical exchanges) and semiotic (conceptual exchanges). ANT was mainly occupied with the relationality among ‘actants’, a term inspired by the modéle actantiel of semiotician Algirdas Greimas. ANT did not consider the individual act-ant as an agent. Rather, agency—that is, what makes things happen—was seen as distributed throughout the entire network of people, artefacts and instruments, all of which constitute a given assemblage.

A similar approach to ANT has been taken in enactivist approaches, where agency is the result of relationships among actors, not their property. However, in enactivism, the network is not a self-contained closed system. It is an open disorganized meshwork, rather than a network [22]. Action is not so much the result of an agency that is distributed around the network, but emerges from the interplay of relations in the meshwork, characterized in terms of patterns emerging from the relationships (material and semiotic) in the environment surrounding humans and nonhumans.

In the 21st century, systemic visions have moved even closer to eco-criticism and environmental concerns. The condition of openness in systemic relations has also expanded beyond artefacts and technologies. The ‘nonhuman’, a category first used for computer programs and robot-like devices with human-like characteristics, is being used for animals as well as other material forms. The concept of ‘vibrant matter’ [23] or that of ‘transcorporeality’ [24] suggests that even inanimate bio-entities, like rocks or the sands and dunes of deserts, are forms of materiality open to their environment and in constant systemic interaction [25], while being free of semantic notions of intentionality. Hutto & Myin’s concept of ‘teleosemiotics’ suggests that cognition is essentially extensive, not merely contingently extended [26]. Natural and geological forces, processes of decomposition, bio-deterioration, and disintegration [27], as well as episodes of climate change, glacial flows, and the evolution of the oceans, all exhibit various forms of nonhuman agency, influencing human life in various ways. In this scenario, nonhuman entities are understood as performing actions in the world, even if this agency is different from human agency. Thus, agency has come to be defined as the capacity to influence a given environment, and can be contemplated not as an individual trait but as an emergent state emanating from systemic inter- and intra-actions, as well as sensorimotor contingencies. Unfortunately, vulnerability to the agency of nonhuman entities has become evident during the Covid-19 pandemic.

The publication of Dan Dennett was a landmark on the discussion about agency, intentionality, and consciousness [28], with is ‘false belief test’ to find out if intelligent animals, chimpanzees in this case, were able to recognize the intentions of a human actor [29]. A long debate ensued exploring how humans develop a theory of mind, trying to understand how an observer can differentiate agentive capacities in others, and whether these capacities are species-specific and if they intentional or not, that is, if they imply a feeling of being in control. It was concluded that humans do not develop an understanding about other people’s minds from their actions until they acquire consciousness and self-awareness [9]. However, the growth of so-called intelligent agents in cybernetics complicated the understanding of agency since some forms of AI can learn from their actions, being designed as ‘autonomous’ entities capable of functioning in the absence of human intervention and able to direct their activity towards a given purpose or goal.

The concept of ‘affordance’ (in Peircean terms, set of stimuli that an object provides an interpretant so that it acquires the character of a sign representamen, that is, it stands for something in some respect or capacity) was initially developed by psychologist James J. Gibson (1977), who argued that humans can modify affordances in their environment to their benefit. Learning to perceive affordances emerges through direct object manipulation and sensorimotor processes as well as by learning from the experiences of others. In the 1980s, affordance became synonymous with ‘action possibilities’ and was applied to human-computer interaction and design [30, 31]. From this perspective, cognition arises through a dynamic interaction between acting organisms and their environment. Gradually, enactivism has become part of a cluster of related theories known as the 4Es, which include embodiment, embedding, enaction, and, more recently, the extended mind.

In the 1990s, an approach in cognitive science known as ‘Distributed Cognition’ or DCog began to gain ground. It originated in the work of Russian psychologist Lev Vygotsky through research by Edwin Hutchins who explained that cultural schemas emerge from changing patterns of interaction among members of cultural groups and which are constantly negotiated and renegotiated across time and space [32]. Hutchins explained that this perspective aspired to “rebuild cognitive science from the outside in, beginning with the social and material setting of cognitive activity, so that culture, context, and history can be linked with the core concepts of cognition” [33]. DCog explores the ways in which cognition involves coordinated ‘enaction’, including artefacts and technological means in specific environments. Like ‘sociocultural ‘situatedness’, DCog came to be assimilated to ‘embedding’, as part of the 4Es.

As aforementioned, these changes were mainly influenced by the explosion of the waves of cybernetics [34]. The term ‘enaction’ appeared during the third wave in Humberto Maturana and Francisco Varela’s ground-breaking work [35]. To these authors, ‘enact’ means ‘bring forth’, a notion connected to their theory of ‘autopoiesis’ [36]. Biological systems are ‘autopoietic’, meaning that they are complex, proactive, and adaptive (self-organizing and self-regulating) in particular spatiotemporal spans. Enactivism considers that bodies and minds interact and respond to things in the world, creating meaning from environmental cues, rather than representing reality. Living beings and their environments stand in relation to codetermination [37].

Similar ideas were contemplated in the then emerging field of bio-semiotics. Jesper Hoffmeyer termed ‘emergence’ the process through which all kinds of things come together in the world and their encounter and settling down, at least in short-term equilibrium before dynamically engaging again, they can creatively produce new kinds of organisations that are greater than the sum of their parts [38, 39].

Thus, enactivism relies on a model of cognition wherein new thoughts emerge through a dynamical engagement between the human mind and the material world. It foregrounds the differences between material things functioning as lower-order signs and higher-level cognitive activities. In this regard, Shaun Gallagher brings forth the distinction between body schema and body image. The first includes unconscious body awareness and automatic sensorimotor functions. Body images, however, are conscious self-aware representations of experiences encompassing some sensorimotor functions that serve intentional action, as well as other mental states (i.e. desires, beliefs, etc.). Drawing on previous work [28, 40] and the European phenomenological tradition (i.e. Husserl, Dewey, Merleau-Ponty [41]), Gallagher considers the building of conscious mental narrative structures and their relation to the sense of self and intentional action. According to him, complex animals and some forms of autonomous AI experience self-consciousness as immediate, punctual, and not extended in time; in other words, as signs that may contain non-conceptual content, only events. He terms this the ‘minimal self’, which might have a sense of self-agency but not self-ownership for actions. On the other hand, the ‘narrative self’ involves personal identity and continuity in time; a more or less coherent self-image constituted including the present, past, and future orientations. This temporal continuity is achieved by means of human language acquisition and the ability to make the kind of cause-effect semiotic connections present in the human telling. The development of a self-image coexists with the ‘narrative self’, involving narrative competency and the capacity for self-narrative and explanation of one’s actions. This distinction amounts to modulation of agency, since the ‘minimal self’ might be aware of self-agency but not have the sense of self-ownership for actions, a continuity only achieved through the development of the ability to make the kind of cause-effect connections present in human language [42, 43].

Gallagher also moves beyond Vittorio Gallese’s notion of automatic resonance systems built into human motor experiences and their replications through mirror-neuron structures to a more complex understanding of the relationship between intersubjective experiences, the building of empathy, and the ‘narrative self’ [44, 45]. Along similar lines, the “Interactive Brain Hypothesis” [46, 47] has argued that narratives modulate intersubjective experiences through affordances and complementarity between a given environment and human social cooperation, trying to demonstrate that even less obvious interactive situations, like reading and writing, have interactive origins.

Additionally, Gallagher describes enactivism as ‘philosophy of nature’ [45], situating mind and behaviour in a holistic pragmatic perspective, a Life-Mind [47] already present in Peirce’s theories.

5. The semiotics of Charles S. Peirce ‘Synechism’ and the life-mind continuum

In the next section of this chapter, I will be speaking about the theories known as ‘wide cognition’. Before, a little introduction to Charles S. Peirce’s triadic model of semiosis becomes necessary. In it, an object determines a sign (‘representamen’) in a process called ‘Firstness’, which in turn determines another sign or ‘interpretant’ in ‘Secondness’. The ‘interpretant’, fulfilling its function as a sign of the object, determines a further ‘interpretant sign’ in ‘Thirdness’ [48]. The sign, or representamen, stands for something, its object, in some respect or capacity, not in all respects. Peirce calls this the ground of the representamen [49].

The distinction related to ‘ground’ in Peirce’s definition is crucial because it recognizes that the sign perceived is relevant to its semiotic object only in a particular respect or capacity. The concept is also important from the perspective of evolutionary anthropology, for instance, in that it emphasizes that what is cognised is a thematic aspect of what preceded it (whether a physical thing or a previous thought). In other words, while some signs are readily perceived, others require prior familiarity with their sign function, often established as habits/laws in social communities (see below).

Peirce’s classification of sign interactions moves from monadic relations, expressing quality, to dyadic, expressing reaction, and sometimes resistance, to triadic (symbolic) relations involved in representation and mediation. Three fundamental relations occur between the representamen and its object: iconicity, indexicality, and symbolicity. These relations are based on fundamental cognitive operations.

Iconicity is related to varying grades of semblance/similarity with what is perceived. An icon is a sign that denotes its object by virtue of a quality that resembles or imitates its object. Iconic signs do not possess the properties of the object but reproduce some conditions of common perception. Depending on material aspects, Peirce established three types: (a) the image, which depends on a simple quality; (b) the diagram, whose internal relations, mainly dyadic or so taken, represent by analogy; and (c) the metaphor, which represents by drawing parallelism to something else; for instance, an abstraction represented by physical resemblance. In the early 1970s, a debate ensued trying to clarify the cultural aspects of icons, extending the notion of ‘quality’ beyond phenomenological analogous relationships, which, according to some scholars prevented from analysing the iconic sign as a social product, and therefore as an object of convention, including its possible ideological depths [50]. Endorsing perceptual aspects of contiguity (i.e. proximity) and factuality (i.e. metonymic part/whole relations, since experience can occur in terms of parts and totalities) indexicality compels attention without conveying information about its object [49]. Finally, symbolic signs are ruled by habits, as we shall see below.

It is important to emphasize that, in Peirce’s view, the action of signs only ‘enacts’ some aspects in a particular space-time within the continuum of experience. For Peirce, cognitive semiotic functions are simultaneously materialized in the brain and the material artefacts used in meaning-making. Emphasizing ‘the action of signs’, which can both generated and generative, Peirce explained that potentially anything can acquire the function of sign, rooted in the continuities that come about between internal representation and external reality [51]. Thus, Peirce developed a form of phenomenology that he described as ‘synechism’, from the Greek synechismos, wherein ‘all that exists is continuous’ [52]. In Peirce’s view, the action of signs only enacts some aspects in a particular space-time (‘ground’) within the continuum of experience, a continuum that problematizes the relationship between interiority and exteriority [49]. Peirce explains that language does not only reside in the brain [53], and sustains that “consciousness has a bodily and social dimension, the latter originating outside the individual self” [54].

Winfried Nöth has noted that, in giving the well-known example of the inkstand, in which Peirce claims that the faculty of language resides both in his brain and in his inkstand [55], Peirce’s purpose is to illustrate the role of efficient causality in creative semiosis. However, Peirce’s example also provides insights into how this efficient cause may evolve to become a factor acting as a final agency. Efficient causes, according to Peirce, are the causes by which machines function, insofar as they are determined by mechanical forces operating ‘in a perfectly determinate way’ [56, 57]. Peirce asserts that “Final causality cannot be imagined without efficient causality” [58]. Nöth notes that the inkstand is a metonymic sign, an index pointing to the medium of writing which ink makes possible. He explains that authorship also depends on the technical medium of writing so that there is a situation of co-agency [59].

In his previous 2009 work, Nöth points out an important consideration: “the agent in the process of semiosis in which the sign creates an interpretant, is the sign, not the addresser, and the agency of the sign is one of final causality: it is the purpose of the sign to create an interpretant” [60]. Signs are not mere instruments but semiotic agents acting with a semiotic autonomy of their own. They mediate relations between things in the world and “operate by final causality, even though they cannot do without efficient causes to convey their messages. Final causality involves triadic interaction; it is the long-term causality of purposes, intentions, ideas, signs, and general laws, all of which belong to the Peircean category of thirdness” [60].

This argument serves Nöth to formulate his hypothesis that complex media machines, like AI, are “co-agents in the process of media semiosis to the degree that they determine the availability and choice of signs, partially restricting, partially increasing the creative potential of their users and thus transforming the impact of their messages” [61]. Thus, in the case of certain technologies that may be incorporated to the human body, the distinction of devices external to the human body must be reconsidered. Noting that the term ‘organ’, which refers to bodily parts, comes from the Greek form órganon, meaning ‘tool’, he insists that “the object and the interpretant of a machine qua sign are the ways in which the machine has been produced and used and in which it may be used in the culture to which it belongs” [62].

Thus, objects/tools have practical functions when used to transform directly the environment, but they also have semiotic functions in the subject’s indirect interaction with the environment by means of them, “serving a practical purpose thus does not preclude an object from serving semiotic purposes at the same time […] since signs have a semiotic agency of their own” [63]. Therefore, semiotic activity is not only an agency of a sign creator; it is inherent in the sign itself. In Peirce’s conception, intentions are not the causes of all sign processes so “it is not altogether surprising if final causality presupposes efficient causality in all cases” [64].

Peirce contended that it is “a widespread error to think that a ‘final cause’ is necessarily a purpose. A purpose is merely that form of final cause which is most familiar to our experience” [65]. Thus, Peirce set out to clarify the distinction between cause and explanation, concluding that life is an ongoing process where concrete moments are not substances but only momentary states part of a continuum [66]: “We ought to suppose a continuity between the characters of mind and matter” [67]. The transitory nature of these states or events can only be expressed in the form of abstracted forms of explanation formulated by means of ‘narrative’ propositions (by means of symbols) also called ‘facts’. Menno Hulswit clarifies the distinction between causality (a relationship between facts), which might require a ‘narrative self’, and causation (purely a matter of events, that might be cognized as non-symbolic schemata, relying on Firstness and Secondness) [68].

Peirce is aware that his hypothesis might be called materialistic since it attributes to the mind one of the recognized properties of matter, extension. He also notes that it attributes to all matter a certain excessively low degree of feeling, together with a certain power of taking habits [67]. In other words, in Peirce’s view, signs become semiotic habits or cognitive routines. A perceptual embodied experience is associated with a schema of activity embodied non-discursively (icon) which connects to an action-reaction salient cue (index) and builds up a habit that, only in the case of the ‘narrative self’, comes to represent propositional content (symbols). As already mentioned, Firstness or monadic relations reflect possibilities (quality) [48], Secondness or dyadic relations stand for actualities (action-reaction) [69], and Thirdness or necessity/potentiality (law-habit), which allows Interpretants to transcend external reality through habits [70]. Peirce insists that this situation happens in all things. It is a generalizing tendency that constitutes a regularity, continually on the increase, and it is also capable of similar generalizations; and thus it is self-generative [71].

Peirce’s graduated continuum of semiotic functioning brings together the anticipated experiences of an agent organism which, influenced by activity in the present adjusts towards the future [72], thus providing the basis of Peircean Life-Mind continuity [73]. ‘Symbols’ (signs resulting of Thirdness) evince a more complex degree of semiotic mediation because they are thoroughly bound up in conventional (law/habit) relationships. Nonetheless, they incorporate ‘indices’ (signs resulting of Secondness) to point to objects of signification. In turn, indices require ‘icons’ (signs in Firstness) to make evident the character of objects [74].

In spite of all these forms of triadic organization, Peirce also recognized the complexity of the natural world and explained that laws are not merely mechanical but probabilistic; springing from diversity and spontaneous occurrences, rather than following deterministic patterns [75]. Moving from the material world (Firstness) to the world of abstractions (Thirdness) reduces the number of dimensions within the ‘ground’ of each undefined ‘First’ (thing), which carries potential semiosis. The number of dimensions is reduced as the ‘First’ relates to a ‘Second’ becoming a named ‘object’, later interpreted as ‘Third’ carrier of ‘significant’ information. In the case of human Interpretants, the ideas and pre-conceptions to which one links a ‘thing’ over-determine, to some extent, its interpretation as an ‘object’ in the mind. In other words, prior knowledge may over-determine semiosis, as seen in the different ways of interacting with the world that children and adults exhibit. The reduction in dimensions from ‘thing’ to ‘object’ in human cognition is achieved by establishing neural patterns (symmetries, similarities, regularities, repetitions) between different observations. The following quotation from Peirce makes this evident:

Doctor X: I should think that so passionate a lover of doubt would make a clean sweep of his beliefs.

Pragmaticist: You naturally would, holding the infant’s mind to be a tabula rasa and the adult’s a school state on which doubts are written with a soapstone pencil to be cleaned off with the dab of a wet sponge. But if they are marked with talc on man’s ‘glassy essence,’ they may disappear for a long time only to be revived by a breath [76].

In terms of agency modulation, we might distinguish between ‘routines’, which look for symmetries to define a problem; ‘non-reflexive actions’, which are sometimes experiences of an intuitive static moment in finding a solution; and ‘intentional purposive actions’ [77]. Peirce also spoke of “a modification of a person’s tendencies toward action, resulting from previous experiences or from previous exertions of his will or acts, or from a complexus of both kinds of cause” [78] and “degrees of self-control” that lead humans to “outgrow the applicability of instinct” [79]. As experience and learning merge, embedded in particular institutional and cultural contexts, it becomes almost impossible to establish a vertical hierarchy of influences. The entire exchange occurs in a continuum that involves the materiality of things: “Time and space are continuous because they embody conditions of possibility, and the possible is general, and continuity and generality are two names for the same absence of distinction of individuals” [67, 80].

6. Wide cognition

Peirce’s concept of ‘synechism’, “the tendency to regard everything as continuous” [81] has been discussed as anticipating ‘wide cognition’ approaches [77, 82]. Clark and Chalmers’ ‘Parity Principle’ in their ‘Extended Mind’ hypothesis claims that if a part of the world functions as a process which, were it done in the head, we would easily recognize as part of a cognitive process, then that part of the world is part of a cognitive routine [83, 84]. Their hypothesis asserts that certain forms of adaptive behaviour arise from perceptual dynamical ‘couplings’ between the nervous and the peripheral sensorimotor systems in a sort of multidirectional process. These couplings between an organism’s perceptions and the objects/artefacts in its environment play a functional role when filtered in sensorimotor activity and propagated across the cognitive system. However, it is not the mere presence of a coupling that matters, but its effect; that is, the way it poises (or fails to poise) information for a certain kind of use within a specific kind of problem-solving routine [85].

The consideration of ‘extended mind’ activity is, in Clark’s words, a modest insight into cognition, not a grandiose theory of everything, and presupposes a view of agency connected to semiotic activity. In Peirce’s definition of the sign relation to its Object and Interpretant, the latter is “that which the Sign produces in the Quasi-mind that is the Interpreter by determining the latter to a feeling, to an exertion, or to a Sign, which determination is the Interpretant” [86]. It is clear that the Interpretant is not a person but the result of sign semiosis. It does not presume human consciousness. The Interpretant approximates the object-sign relationship through a representation that is informed by the object and directly brought about by the sign [69, 78, 87, 88]. Thus, a signing action can be itself a sign for the next, since signs have their own agentive properties, becoming a “more developed sign” [49]. According to Peirce, a sign is not a sign unless it translates itself into another sign in which it is more fully developed [89].

A ‘quasi-mind’, in Peirce’s terms, can be any organism capable of cognition, whether human or nonhuman: “What we call matter is not completely dead, but is merely mind hidebound with habits” [90]. Thus, Peirce seems to imply a certain scaling of ‘agency attribution’ and a potential differentiation in the agentive qualities between humans and nonhumans, animate and inanimate. He explains that thought is not necessarily connected to a brain and that it appears in the work of bees, crystals, and throughout the physical world; it develops in the world through being embodied. Without embodiment, he writes, there would be no signs [91].

Peirce’s assertions are being discussed in the context of the problems that the cognitive sciences face today. (1) How to scale-up the ‘couplings’ between organisms, entities, and their environments; (2) whether mental models, schemata, and internal planning procedures are part of representational structures in the brain, or if they are temporally emergent and dynamic products of situated activity; and (3) how to measure agency attribution, particularly in nonhumans.

Material Engagement Theory (MET) emphasizes material agency from a non-anthropocentric approach, opening the way to posthuman conceptions: “For MET’s proponents then, the world is not an external realm that transmits information to an internal processor, but an emergent product of the organism’s coupling with the environment” [92]. A key figure in MET, Lambros Malafouris, views artefacts as integral parts of the thinking process and, like Peirce, speaks of a continuum “dynamic co-evolutionary process of deep enculturation and material engagement” [93].

MET draws also upon enactive sensorimotor contingency theory [41, 43] to support the idea that active engagement with material things/signs brings forth meaning-making, that is, semiotic activity. In the case of symbols, following Peirce’s categorization of signs, Malafouris indicates that engagement takes the form of a sort of visual code or language and thus invites reading. This suggestion seems to be aligned with Gallagher’s claims about the ‘narrative self’. Overall, MET agrees with the Extended Mind hypothesis but tries to go beyond, claiming that Chalmers and Clark’s theory is simply an expansion of the ontological boundaries of the res cogitans rather than the dissolution of those boundaries altogether. Malafouris claims that the functional anatomy of the human mind includes the whole organism, brain and body, and adds that it is also socially embedded in everyday experiences which are often constituted by the use of material objects. Thus, he believes that all these aspects should be seen as continuous, integral, and active parts of the human cognitive architecture [94].

As to agency, rather than seeing it as the result of prior intention, Malafouris sees it as the emergent product of semiotic activity: “meaning is not the product of representation but the product of a process of conceptual integration between conceptual and material domains” [95]. In its attempt to decouple agency from human consciousness, MET affirms that “While agency and intentionality may not be properties of things, they are not properties of humans either; they are the properties of material engagement, that is, of the grey zone where brain, body and culture conflate” [96]. Furthermore, some materials, such as clay, afford a flow of noetic activity beyond skin and skull that enhances neural plasticity. Malafouris speaks of a symmetric relationship between potter and clay: “trying to separate cause from effect inside the loop of pottery making is like trying to construct a pot trying to keep your hands clean from the mud” [97]. He explains that although it is the potter who makes the decisions, external factors like the texture of the clay, its physical properties etc., may determine some parts of the actions performed by the potter. The potter’s wheel, for instance, “shapes the field of action and has a share and saying on our will and intentions” [98].

Accordingly, Malafouris argues that agency needs to be de-coupled from subject-object distinctions and dissociated from intentionality as unique human property. Appealing to Searle’s distinction between ‘prior intention’ in premeditated or deliberate action, and ‘intention in action’, where no intentional state is formed in advance of the action, Malafouris concludes that in ‘intention in action’ the internal intentional state and the external movement become indistinguishable, but still have a pragmatic effect in the world. This shows that agency is an emergent product of mediated activity in material engagement, not an innate and fixed attribute of the human condition: “The ultimate cause of action in this chain of micro and macro events is none of the supposed agents, humans or non-humans; it is the flow of activity itself” [99].

7. Posthuman agency

Until recently, agency was considered only in relation to human consciousness and in connection to intentional action. This view originated in Cartesian dualism, which posited self-awareness and purposefulness as essential components of the human mind. Peirce described dualism as “the philosophy which performs its analyses with an axe, leaving as the ultimate elements, unrelated chunks of being” [100]. The Cartesian view also highlighted anthropocentrism and justified the use of the natural world to satisfy the needs of humans as superior beings. During the 20th century, however, the increasing engagement with digital machines gave rise to an inquiry into non-anthropocentric considerations of agency and human relationships with nonhumans, from complex machines to bio-entities (animal, plants, and the environment in general) [101].

Several disciplines, including semiotics, are exploring new theories on agency to recognize the active role of nonhumans. Quantum physicist Karen Barad has explored ‘agential realism’, and the concept of ‘intra-action’, which modulates the concept of agency, not exclusively tied to human subjectivity. For Barad, the agency is “a matter of intra-acting; it is an enactment, not something that someone or something has” [102].

In the context of global warming, climate change, and the unexpected impact of environmental aspects upon human life, Jane Bennett speaks of ‘thing-power’ [23] and Stacy Alaimo of ‘transcorporeality’ [24]. Like proponents of enactivism, these scholars emphasize that agency and intentionality are not “properties of things, they are not properties of humans either; they are the properties of material engagement” [102]. Exploring nonhuman agency in trees, Owain Jones and Paul Cloke (2008) speak of several forms of agency: ‘Agency as routine action’, associated with the ongoing process of life existence. ‘Agency as transformative action’, involving natural fields of relations often bound up with geo-transformation. ‘Agency as purposive action’, beyond human intentionality, for nonhumans can influence courses of action through the encoded blueprint present in their DNA. Finally, ‘agency as non-reflexive action’, recognizing that nonhumans can engender affective and emotional responses from humans [103].

Like Peirce, these authors consider that only final causation, which involves complex semiosis, yields the ‘experience of agency’, which relies on self-consciousness, and is different from simple ‘agency’. In Gallagher’s enactive interaction theory, complex interpersonal understanding aligns with an elaborate understanding of others’ motives and goals, due to a shared familiarity with self-narratives, and understanding that resembles Peirce’s distinction between having a mind and having the experience of mind [55].

There are, however, detractors to these ideas. Mendoza-Collazos and Sonesson (2021), for instance, consider that relationships between human and nonhuman actors are nonsymmetrical. According to these authors, agency is the capability to act based on the agent’s intrinsic intentionally. This implies that agents must be living beings [104, 105]. Aligned with humanist and internalist approaches, the authors attribute the “capability to plan, imagine, and improve artefacts, by means of the intentional shaping and assembly of materials” as manifest expression of the uniqueness of human agency, distinct to that of other species, including primates [106]. For Andy Clark, the problem lies in that consciousness may be internalist, even if the mind is extended.

The discussion above is of interest not only to conceptions of the posthuman related to the environment but also in relation to AI. As indicate before, complex machines that convey information via digital artefacts connected to analogic instruments are strong candidates for extended cognition. For instance, an optical microscope extends human visibility range through lenses. However, an Atomic Force Microscope (AFM) can produce data, in the place of an optical visual process. Complex machines might not increase the power of human observation by delivering immediate sensory data. Instead, they offer access to nonobservable data, even if this process does not resemble human perception. Furthermore, digital information can now be stored in biological tissue and DNA (see work by Mark Bathe at the Broad institute MIT & Harvard). Facial recognition technologies, and even wearable devices, are also activated from physiological parameters, which are then transformed into digital data. The concept of the DNA of Things (DoT) is already merging biological and digital information [107], an integration that creates additional concerns regarding agency, since there is often a long causal chain of mixed human and machine interactions.

There are not only many (human) hands; there are also what one could call ‘many things’: many different technologies. In AI process and history, various software is involved but also more literally various things, material technological artefacts: things that are relevant since they causally contribute to the technological action, and that may have some degree of agency [108].

8. Conclusions

This chapter highlighted the relationship between cognitive semiotics and other cognitive sciences. The chapter has focused on the increasing presence of wide cognition theories and on the need to explain material engagement and the modulation of human and nonhuman co-agency, a fundamental discussion that involves both environmental and technological concerns. The chapter has shown how Peircean semiotics anticipates some of these issues.

Article title footnote

This is an extended version of “Cognitive Semiotics and Agency in the Anthropocene,” published in Biglari A., editor, 2022, Open Semiotics. L’Harmattan. For a more comprehensive treatment of agency see Biglari’s edition.