Rethinking ‘Affordance’, ‘Agency’ and ‘User’ from a Semiotic Technologies Perspective: The Emergence of a Typology of Signs-as-Agents

Abstract

This chapter shows how educationalists might (re)conceptualise digital tool use within teaching and learning scenarios as a social practice. This is achieved by focusing on the development of a theoretical and analytical framework for identifying signs-as-agents that was shown to be applicable to other educational tools and scenarios. The chapter illustrates how a Semiotic Technologies approach, that forms the main pillar of the framework, can unite a number of well-established perspectives, from research in Human-Computer Interaction, Social Cognition Theory, Critical digital literacy, Distributed Cognition Theory and be applied to different educational scenarios. This is in order to support a more holistic and critical understanding of agency and agents – both ‘human’ and ‘digital’ – which is relevant for education researchers and practitioners working with STEAM and STEM. Three examples, from various educational, digital scenarios are used to illustrate how the theoretical and analytical framework can be applied. The analysis involves the identification of screen-based signs-as-agents through a focus on initiation and response turns, primarily with the screen. Analysis of the artefacts related to the screen are initially analysed separately from how they are being used in the social practice, following a Semiotic Technologies analytical model. The central contribution of this chapter is a typology of signs-as-agents that has been conceived in order to expand the Critical Digital Literacies teaching and research agenda, specific to pedagogy. The study is aimed at supporting critiques of how digital tools can shape how teachers and learners act, not only how they can think.

1. Introduction

In this chapter I want to show how educationalists might (re)conceptualise digital tools and also reconceptualise teaching and learning as a social practice with digital tools. I do this by focusing on the development of a theoretical and analytical framework for identifying signs-as-agents that was shown to be applicable to other educational tools and scenarios. In themselves, the proposals may not be very novel, since they include or build on well-established traditions in various fields, though perhaps less so in the field of Education. However, I wish to suggest that a Semiotic Technologies approach, focused on signs-as-agents, applied to educational scenarios, can unite a number of well-established perspectives on digital tool use. Such an approach can also be very relevant for researchers and practitioners working with STEAM and STEM.

Through the presentation of three examples, from various educational, digital scenarios, I will show how the theoretical and analytical framework, involving the identification of initiation and response turns, understood as a multimodal exchange structure, can be used to identify screen-based signs-as-agents. In addition, I show how through the establishment of such multimodal exchange structures, signs can be understood as being agentalised. Throughout this process, I highlight how theoretical and contributions from research in Human-Computer Interaction [1]; [2, 3, 4], Social Cognition Theory [5], Critical digital literacy [6, 7, 8], Distributed Cognition Theory [9] as well as theoretical and methodological contributions from a Semiotic Technologies perspective [10, 11], can support a more holistic and critical understanding of agency and agents – both ‘human’ and ‘digital’ – in educational scenarios. The central contribution of this chapter is a typology of signs-as-agents that has been conceived in order to expand the Critical Digital Literacies teaching and research agenda specific to pedagogy. It is aimed at supporting critiques of how digital tools can shape how teachers and learners act, rather than how we think. The latter has been the traditional domain of critical digital literacies.

The term Semiotic Technologies refers to “the whole array of technologies people use in order to make meaning as part of specific social practices, such as writing and drawing on a blackboard with chalk in mathematic lessons in schools, photographing beaches and mountains for image bank stock photos, or taking a selfie and uploading it on Instagram for sharing with friends and acquaintances” (p. 596) [12]. A Social Semiotic Technologies perspective is a growing subfield of social semiotics [12]. An analytic model for analysing semiotic technologies has been developed within this perspective [10, 11], that according to Djonov and Van Leeuwen [13] involves an analysis of the semiotic practices and artefacts (e.g., a written text or illustration) that warrant critical attention, as well as an analysis of the technologies that support them. Such a critical multimodal study of software, that includes the evaluation of software design, its use, and its relationship to broader semiotic and cultural practices would conceivably offer a holistic analytical approach [13] to digital technologies and their use in educational scenarios.

While a small number of studies have focused on digital tools in education from a social semiotic perspective (e.g., [14, 15, 16, 17]) the model for analytic separation of artefact and practices developed by (Zhao et al. [10] and Zhao and Van Leeuwen [11], has only more recently been applied to other digital tools in education (e.g., [10, 11, 18]). This approach to the study of the relationship between semiotic technology and social practices is highly relevant and applicable for the study of technologies in digital-based teaching and learning practices because teachers may not be the only actors (or agents) participating in what appears on the interface pages that students face [18].

I will argue that this approach is fruitful for understanding agency in digital scenarios because, following the emphasis on analytical separation, it conceivably allows us to look at how agency and agents may present themselves in relation to the digital tool itself, as well as in the social practices by teachers and/or learners with the tool. Another relevant contribution to the study of agents from a Semiotic Technologies perspective is the notion of “exchange structures” that can be realised multimodally through “user-user interactions” and “system-user interactions” and must at least contain an initiating move and a response [19]. Such an analytical focus conceivably would give insight into the intentions of human or digital agents as exchanges take place between the teacher or learner and system as well and may also give insight into how these exchanges can shape their semiotic work.

My arguments for a Social Semiotic Technologies approach are based on a critique of three premises that often underly much discourse in studies surrounding digital tool use in educational scenarios. The first premise is that digital tools should be evaluated and researched for their ‘affordances’ [20] as potential uses for supporting human intentions. While countless studies within the field of Education attest to this premise, it is important to highlight that a digital tool with a screen does not typically have a singular design intention, unlike a chair (which the notion of ‘affordance’ was initially applied to). But rather, digital tools are increasingly more complex, and are increasingly seen as having unlimited potentials, including new roles like instructing or advising humans (e.g., service bots with Artificial Intelligence) [18]. An interface that learners may face, for example, may have been designed by multiple people including interface designers and teachers and may have also passed through the hands of translation or legal teams for final touches with logos or orthographic corrections. Furthermore, a screen might present the user with many digital tools at once, so that grammar checkers, for example, can be installed ‘within’ other tools so that the affordances across multiple tools can be customised to each user. Designers may be multiple and the final product or digital tool(s), often incorporating a screen, may be the result of many months or years of design input by each one: each with their own intentions imbued in the presence of their signs.

The second premise is that digital tool use is favourable in supporting individual learner empowerment. Many studies in the field of Education also attest to this. Within a university environment, for example, e-Learning tools can create ‘leaner-centric’ and ‘collaborative-learning environments’ where learners are empowered to self-control their learning processes [21]. However, this perspective, which can be understood as part of the ‘empowerment paradigm’ [22] is often typically focused on learners and their agency rather than the potential agency that digital resources on the screen may ‘enact’ through learners’ use of them. This enactment of ‘digital’ agency, or agency enacted through digital means, can be traced back to studies in Human-Computer Interaction from the 1980’s with the creation and the identification of pedagogical agents as ‘passive’ or ‘active’ [1, 4]. A passive agent waits until it receives a request to act, while an active agent takes action on its own when it sees an opportunity to do so. These studies have highlighted the technological intentions to communicate and/or shape human action, that originate in the design of human others. More recent studies in Social Semiotics (e.g., [10, 11]) and linguistics (e.g., [23]) highlight the ability of digital technologies to shape and influence human semiotic work. This shaping potential does not refer to the shaping potential of ideas or ideology communicated typically through text and image, but rather its shaping potential is carried out through ‘exchange structures’. ‘Exchange structures’ is a notion that was originally drawn from conversational analysis and functional linguistics. Following this understanding of exchange structures, exchanges must at least contain an initiating move and a response, which can be realised multimodally. Jovanovic and Van Leeuwen [19] also mapped out the ways that digitally-mediated interactions occur across various forms of social media, including “user-user interactions” and “system-user interactions”, in what the authors describe as “a multimodal realisation of an exchange structure”. This notion has also been reflected in educational research as ‘semiotic initiations or responses’ [24] that can be understood as part of ‘multimodal turn-taking’ [25]. An example of this is a pop-up that ‘requests’ a user to accept cookies and then the user clicks the screen-based icon as the ‘acceptance’. These exchange structures have attributes of design, intention and enactment that can be assigned to constructs of agency. Therefore, the learner as a user may be only one social agent that is being empowered when digital tool use is being enacted: the social interests of each screen-based resource and how they may be empowered through turn-taking with them, should conceivably also be considered when learners carry out their semiotic work. This is especially the case in educational scenarios because, as Djonov and Van Leeuwen [26] propose, there are social interests that permeate the design and use of software.

The third premise is that a screen is solely for learners as ‘end’ users. However, unlike a page, whatever is communicated by the user through the screen, is connected to a network. The information that is passed through a screen, either entered by humans through typed text or through navigation and touch, is at least a two-way information flow. Therefore, resources on a screen can conceivably be considered more than just signs on an interface in which meaning is communicated to the user. Indeed, it could be argued that there is a relationship between humans and the screen, not only in terms of content (e.g., ideas communicated through text that is read or watched by users) but also in terms of a relationship made up of a human-digital system that is carried out through “exchange structures” [19]. With this altered conceptualisation, the screen is not just for the human users at the ‘end’ but also for digital and human ‘users’ at the other ‘end’ of the exchange which teachers and/or learners do not see. The screen is the point at which, however, potentially part of the exchange structures between the human and digital ‘system’ can be made visible to teachers and/or learners.

The three premises that I have briefly described above can be critiqued within a Social Semiotic Technologies perspective. I propose that it is therefore a potentially valuable perspective in advancing the development of Critical Digital Literacies [6, 7, 8] within educational scenarios which Pangrazio [27] has proposed is in need of expanding. Such an expansion could involve a focus on identifying signs-as-agents. While many definitions of Critical Digital Literacies exist, Luke [28] refers to it as processes of “naming and renaming the world, seeing its patterns, designs and complexities, and developing the capacity to redesign and reshape it” [28] (p. 29).

Such a critique, broadly focusing on ‘patterns’, ‘designs’ and ‘shaping’ [28] signals a number of shifts in how educationalists might (re)conceptualise digital tools and their use through a critical lens.

Firstly, this reconceptualisation is a shift in emphasis from the notion of a digital tool with a singular design intention, to a tool with multiple design intentions and designers and designs with various signs that may change meaning, patterns of use and use for different purposes over time. Secondly, while many studies in the field of Education attest to digital tool-use as being linked with learners’ individual and/or social empowerment, if learners’ creations can be shaped in the process of tool use, students’ digital semiotic work might be conceivably re-conceptualised more accurately as a meaning negotiation process rather than (solely) as an empowering, creative one. That is to say, ‘negotiating’ what semiotic work from pre-built designs in digital tools by other designers is accepted/rejected, included/excluded and/or suggested/left out by teachers and/or learners. This conceptualisation highlights that digital tool use in (social educational) practices may be a site of struggle between all potential agents in the creation, negotiation, use or re-use of a semiotic piece of work. This in turn, signals the need for a more critical stance towards power relations with digital tools while in use.

Thirdly, because a Social Semiotic Technologies methodological approach involves analysis of digital tool use and the tool itself, including design intentions of the tool, this approach conceivably also supports an analytical as well as a theoretical shift. This shift may be from tool-use and its affordances towards tool-use for different social practices. I propose that this approach can facilitate a space for identifying and analysing the various ‘voices’ of the designers and/or creators that may be ‘at play’ at any point in its use/re-use.

In the next section (Section 2), I will focus on the theoretical and analytical framework for identifying signs-as-agents, how this can be applied to other educational tools and scenarios and finish with some conclusions.

Section 2 focuses on the theoretical framework for identifying signs-as-agents. It highlights the framework’s compatibility with theoretical contributions from Human Computer Interaction [1, 2, 3, 4], Social Semiotics [29, 30, 31], Social Cognition Theory [5], Critical digital literacy [6, 7, 8] and Distributed Cognition Theory (see Hutchins [9]).

Section 3, illustrates how signs (or screen-based resources) can be identified and agentalised through use, thus becoming agents. For this, an exploratory study of ‘hidden’ digital agents in an online language scenario with a digital App called Tandem [18] is summarised in order to highlight key findings and demonstrates how an analytical tool for identifying signs-as-agents emerged.

Sections 4 and 5 focus on how the theoretical and analytical insights from the exploratory study, outlined in Section 3, can be applied to other digital tools and social practices that are commonly used in educational contexts: namely two webpages for Webquests and two ‘Google for Education’ tools for communication purposes between teachers and students.

Finally, I will conclude with some arguments as to why it is necessary for educationalists involved in educational, digital scenarios such as STEM and STEAM to critically rethink the three premises of ‘affordance’, ‘agency’ and ‘user’ and how this might be carried out.

2. Signs-as-agents

Sign is a key concept in traditional semiotics. “Signs are elements in which the signified (‘meaning’) and signifier (‘form’) have been brought together. Social semiotics holds that the process of sign-making is subject to the interest of sign-makers, their availability of semiotic resources and the aptness of those resources to the meanings which they wish to realise” (p.3) [31]. That is to say, the relation between ‘form’ and ‘meaning’ is not arbitrary but motivated [29]. While the notion of ‘sign’ emphasised available resources as part of a system, the notion of ‘semiotic resource’ that also evolved from social semiotics, focused on how the context of communication and the sign maker shaped signs and meaning. It was focused on people’s situated choice of resources rather than a system (p. 3) [31]. This is relevant to our understanding of digital tools and their use because whereas the screen may offer learners signs, that in the designers’ intentions have a ‘logic’ which connects the signs through layout and colour for example, learners can also bring other resources from their own environment. Such resources can include the choice of language that they use while using the tool or their decision about what to include in the background if they are making a video call. Van Leeuwen [30] describes semiotic resource as follows:

Semiotic resources are the actions, materials and artefacts we use for communicative purposes, whether produced physiologically – for example, with our vocal apparatus, the muscles we use to make facial expressions and gestures – or technologically – for example, with pen and ink, or computer hardware and software – together with the ways in which these resources can be organised [30]. Van Leeuwen [30] also noted that the notion of ‘resource’ began to replace the notion of ‘sign’.

The distinction between ‘sign’ and ‘resource’ is important in understanding a digital tool in use because it allows teachers and researchers to choose whose perspective we are looking at a screen-based resource from, at any given time. That is to say, the intentions of the pre-embedded signs on the screen by an interface designer for example, or alternatively, the intentions of the learner while using the signs on the screen, or indeed, the incorporation of their own semiotic resources made available through the screen. A Semiotic Technologies approach allows for both perspectives and analysis of intentions because it attends to a) the materiality of the digital tool and also b) how that tool is used in a social practice. Intentions are an important aspect of the construct of agency because human goals, whether related to expression of identity or carrying out and completing a task, are not achieved without intentionality.

To understand ‘signs’ as potential agents that can take part in exchanges with humans, we can draw from studies in Human Computer Interaction (henceforth HCI). Knight, Dooly and Barberà [18] highlighted how the screen-based signs, were considered as pedagogical agents in HCI studies in the following way:

Pedagogical (non-human agents have been classified into two different categories: animated pedagogical agents [3] and reactive pedagogical agents [2]. Animated agents simulate human behaviour, such as facial expression, body movement and gesture whereas “reactive agents” respond to events in the environment, for instance displaying messages when certain threshold values have been reached. According to Jondahl and Mørch [4], reactive agents can be further classified as passive or active: a passive agent waits until it receives a request to act; an active agent takes action on its own when it sees an opportunity for doing so.

From the field of psychology, Social Cognition Theory, developed from Social Learning Theory (SLT) by Albert Bandura in the 1960s, also contributed significantly to theory on human agency. There are two notions that are relevant to researching signs-as-agents from Social Cognition Theory. The first is Bandura’s notion of “proxy agent” [5] and the second is a systems-based understanding of agency, all be it human systems.

The role of a “proxy-agent” [5] is where a (human) agent can be enacted or represented by others (e.g., a parent completing a task on behalf of a child). To extend this notion, signs on a screen can be understood as proxy agents that act on behalf of their designers in that they are designed to communicate an idea or act on the designer’s/designers’ behalf.

Bandura’s [5] human, systems-based understanding of human agency highlights the motor, sensory and cognitive systems in carrying out of intentional actions. This notion connects with important notions from the field of Social Semiotics, namely ‘signs’, that can be made up of ‘modes’, such as image, text and speech. The motor and sensory systems of humans are conceivably co-reliant on such modes in order to be put to use by humans to carry out their intentional actions. If humans initiate or respond to signs on the screen through their sensory or motor systems, the interaction conceivably becomes a shared human-computer one. This interaction can be characterised as “user-user interactions” and “system-user interactions” [19]: the screen facilitates oral/visual interaction between humans and users are faced by a number of signs on the screen to which they can respond to in various ways in what Jovanovic and Van Leeuwen describe as “a multimodal realisation of an exchange structure” [19]. In addition to this, a “user-system” initiated exchange [18] can also be considered a structure, given that users can initiate ‘turns’ with the digital system, through touch for navigation or voice activation for example. This human computer exchange foregrounds the notion of a systems-based notion of agency: a recognition of human systems to carry out intentional actions and also of digital systems to establish or carry out intentional actions. This understanding of agency recognises digital tools and established human experience as forming “unified ecologies, with agency distributed throughout the system [as] artefacts, context, and humans together create particular morphologies of action” (p. 9) [32]. This notion stems from Distributed Cognition Theory, a theoretical framework that was originally introduced by Edwin Hutchins and his colleagues in the mid-1980s (see Hutchins [9] for the principles of Distributed Cognition).

Following on from the theoretical framework, Section 3 illustrates how signs (or screen-based resources) were identified and agentalised through use, which led to the key concept presented in this section (Section 2) of ‘signs-as-agents’. Section 3 revisits the data from an exploratory study, spanning over 5 years, that focused initially on learner agency in an online language learning scenario to develop speaking skills [23, 25, 33] and evolved into a focus on ‘hidden’ digital agents from a more critical perspective [18].

3. Signs-as-agents in the Tandem tool: The exploratory study

In order to illustrate how educationalists and researchers might research and identify signs-as-agents, rather than just affordances of digital tools, it is pertinent to revisit the first data set from an exploratory study and its relevant findings. The signs-as-agents in the Tandem tool were initially identified in a study by Knight, Dooly and Barbera [18] but the term ‘signs-as-agents’ has not been used in relation to the exploratory study until this section.

The following screenshots and analysis relate to a language learning App called Tandem which is an audioconferencing tool that facilitates oral interaction between students in a synchronous mode. Students cannot see each other. Instead, they are faced with various textual instructions, navigation buttons and texts in order to support the student interaction.

As in keeping with a Semiotic Technology approach, the materiality of the tool, in this case the screen-based resources or signs, were analysed separately from their use. However, also in keeping within a Semiotic Technology approach, the tool in-use by students was also analysed in the form of the audio recordings of what students were saying during the process of interacting with the screen as well as each other. This gave insight into intentions and actions taking place during the learning process from the student perspective.

Pop-ups (Figures 1 and 2) appeared on the screen at different moments, “inviting” students to “respond” by clicking on the screen ‘button’ provided, namely ‘Start’ (Figure 1) and ‘Close’ (Figure 2). These were identified and later labelled as ‘active agents’ according to the classification of Pedagogical agents from studies in HCI, shown in Figure 3.

Figure 1.

Pop-up on Interface page before students start conversing orally.

Figure 2.

Pop-up on Interface page (when time is up for task 1).

Figure 3.

Classification of pedagogical agents in HCI studies (cf. Fischer et al., 1985; Müller, 1998; Johnson et al., 2000; Jondahl and Mørch, 2002).

Navigation resources such as ‘Next Task’ (Figure 4) and ‘See solution’ (Figure 5), were identified and later labelled as ‘passive agents’. Passive agents were available on the screen and when students wanted to use them to move forward (or back) through the task sequence, they clicked on them.

Figure 4.

Navigational button on Interface page for students to navigate to the ‘next task’.

Figure 5.

Navigational button on Interface page for students to check their answers ‘see solution’.

This identification highlighted the ‘semiotic initiation and responses’ [24] or ‘multimodal turn-taking’ [25] that was carried out between student and the screen, also while students were interacting orally. Consequently, these screen-based resources or signs were considered to be “direct discourse agents” [18] because they could carry out turns with humans as ‘agentive turn takers’ [23]. When initiations, either human or digital, met with a response (e.g., from the screen ‘moving on’ to another page or from the learner, clicking) the signs were considered to be “agentalised”, a notion from Van Leuwen [34]. This “multimodal experience revealed how peer-to-peer talk can occasionally resemble a multi-party encounter whereby some resources can act as or be oriented towards as participants in the interaction” [23].

This dual-focused analysis, particularly in the analysis of learners’ oral turns from the audio-recordings, also revealed another type of human-computer ‘relationship’ in the form of screen-based resources that acted as “shaping agents” [18]. The resources that acted as “shaping agents” were invoked or made relevant in the context of learners’ oral turns, thereby they became part of the oral messages in the human, user-user, oral interaction. Specifically, the screen-based resources became: 1) embedded or modified in oral turns (as in Figure 6); 2) resources to initiate and support oral turns (also in Figure 6) or 3) diverse topics of talk (as in Figure 7).

Figure 6.

Transcript of student-student oral interaction whereby screen-based resources became embedded or modified in oral turns or resources to initiate and support oral turns.

Figure 7.

The screen-based resources become topics of talk.

Regarding shaping agents, The ‘?’ sign, with suggested answers next to it (Figure 6), was used by learners as a resource to initiate and support oral turns. Also in Figure 6, a textual lexical item on the screen, ‘Where do you live?’ becomes embedded in learners’ oral turns.

In Figure 7, the visual textual signs are used as topics of talk, namely about whether the students are going to ‘close’ the pop-up or not, a word that also appears on the pop-up.

Signs were also considered turn-takers in the interaction, as shown in Figure 7 where the pop-ups initiates an action and the students refer to the pop-up as “the computer” or “it” (Figure 7).

These examples and results from the exploratory study on learner agency in an online language scenario contributed theoretically as a first step towards a typology of signs-as-agents: Signs can shape human (oral) turns [23] and can be understood as ‘shaping agents’ [18]. In addition, signs that can establish an exchange structure with humans (either ‘system-user’ [19] or ‘user-system’ [18]) can be understood as either ‘active’ or ‘passive’ ‘direct discourse agents’ [18]. This emerging typology is illustrated in Figure 8. It reflects the classification of agents from HCI studies but also expands on how the concept of agents relates to the notion of signs from the field of Social Semiotics. The notion of “shaping agents” is not in Figure 3 of the classification of pedagogical agents in HCI studies because social practices, or use of the digital tool or signs was not explicitly considered in such HCI studies. However, I incorporate it into the typology of signs-as agents now in Figure 8. Figure 8 also includes the notion related to “shaping agents”, that signs are agentalised when the screen-based signs shape the human (oral) turn in some way.

Figure 8.

Emerging typology: Version A.

The following sections, Sections 4 and 5, illustrate how this emerging typology can be applied to two other educational scenarios, involving digital tools, and can be developed even further.

4. Signs-as-agents in WebQuests

A WebQuest is an inquiry-oriented lesson format in which most or all the information that learners work with comes from one or more websites [35]. The following screenshots are from webpages that were incorporated into pre-service teachers’ designs for WebQuests. Webquests typically involve teachers creating questions and choosing internet-based links so that students can follow the links and collect specific information on a topic.

The following examples illustrate how the typology for signs-as-agents, developed in the first analysis from the Tandem tool, can be applied to websites incorporated into WebQuests. WebQuests are particularly interesting to analyse, because depending on the WebQuest designer, the web sites included in the quest may include a wide range of internet pages with diverse modes (e.g., static, moving images, videos), functionalities (e.g., ability to ‘like’, ‘dislike’, leave textual comments on the site) as well as ‘housing’ different intentions which may be designed for educational purposes or not.

The following screenshots (Figures 9–12) were taken from a screencast video recording of a pre-service teacher trying out various webpages as part of WebQuests designed by other pre-service teachers. It is a common educational (social) practice for teachers to try out teaching and learning activities to check whether they are appropriate for the students that they are going to teach. They need to check whether the activities will develop the particular competences that they are trying to develop in their students. Additional screenshots (Figures 13–16) of the webpages were also taken by the researcher, while navigating to and within the web pages. The two websites in the analysis pertain to ‘Cool Kids Facts on Photosynthesis’ and ‘Kids Britannica’ focused on the topic of ‘culture’.

Figure 9.

A screenshot of a screencast video recording of a pre-service teacher trying out the ‘Cool Kids Facts’ webpage.

Figure 10.

A screenshot of a screencast video recording of a pre-service teacher trying out the ‘Cool Kids Facts’ webpage.

Figure 11.

Pop-up translation tool offering a translation of the English website to ‘Espanyol’ (Spanish). Offered by Google translate.

Figure 12.

A Britannica kids banner with ‘7 day free trial’ in another colour.

Figure 13.

Animated advertisement for furniture company ‘Rochebobois’ (right) and hyperlink for a ‘quiz’ (bottom).

Figure 14.

Animated advertisement for an technology company ‘IBM’ (right) and animated advertisement for furniture compant ‘Rochebobois’ (bottom).

Figure 15.

Pop-up with ‘subscribe today’ (bottom) and an invite to know about cookies (top).

Figure 16.

Tabs with different options – ‘Fundamentals’, ‘kids’, ‘students’, ‘scholars’ that can be clicked (top) and 6 square images representing ‘articles and videos’.

As in keeping with a Semiotic Technology approach, the signs on the screen, captured from a pre-service teacher’s video of her/him “checking out” different webpages that form part of other pre-service teachers’ WebQuests, were analysed separately from their use. During the researcher’s follow up navigation, the researcher used the opportunity to check if the signs, were ‘passive’ or ‘active’ by clicking various signs, as the pre-service teacher did not do this because he/she had his/her own navigational path and intentions. The screenshots were chosen, to illustrate how the developing typology, outlined in Section 3, might be applied to different educational, digital resources and scenarios apart from the Tandem App.

Table 1 illustrates how the typology was used to analyse web pages in order to identify signs-as agents. While the analysis is not systematic¹ in the sense that not every sign was analysed in relation to the different semiotic modes (e.g., layout, texture, colour, sound, etc.), which is a common feature of studies from a Social Semiotics perspective, there is enough visual, textual and navigational information from the two web pages (Cool Kids Facts Photosynthesis and Kids Britannica) to tentatively apply the typology for the purposes of this chapter.

Webpage	Signs-as-agents
	Active (REACTIVE)	Passive (REACTIVE)	Animated
Cool Kids Facts Photosythesis	Pop-up about consent for using Data and Cookies. Requiring to be clicked by the user (Figure 9)	The word ‘quiz’ is a hyperlink, that when clicked leads to a ‘Photosythesis Facts Quiz’ (Figure 13)	Adverts on the left change and are animated with moving arrows or the start of a video which then stops (Figures 13 and 14)
			Advert banners at the bottom whereby the companies change (Roche Bobois/Just Eat/Western Union) and some of the text in the banner changes as well (Figure 14)
Kids Britannica	Pop-up translation tool offering a translation of the English website to ‘Espanyol’ (Spanish) offered by Google Translate. It can be removed by clicking the ‘x’ (Figure 11)	Banner at the top of the page giving information about Cookies with an ‘x’ that symbolises ability to close this banner (Figure 15)	None
		A Britannica Kids Banner with ‘7 Day Free Trial’ in another colour (Top of Figure 12)
		A ‘Subscribe Today’ text in a coloured box (Figure 15)
		Tabs with different options – ‘Fundamentals’, ‘Kids’, ‘Students’, ‘Scholars’ that can be clicked (Figure 16)
		6 Square images representing ‘Articles and videos’

Table 1.

Analysis of two webpages in order to identify signs-as-agents.

4.1 Pop-ups as active signs-as-agents

The pop-ups that appeared when navigating to both webpages, ‘Cool Kids Facts’, and the ‘Kids Britannica’, can be conceptualised as signs as potential agents because they “invite” the user to click on them. The pop-up consent for using Data and Cookies in Figure 9 requires the user to “Aceptar” (Accept) or “Cambiar Las Preferencias” (Change the preferences). When the user clicks in response, the exchange structure can be considered to be established and the sign is ‘agentalised’.

Similarly, when navigating to the Kids Britannica webpage, a pop-up in the top right corner occurs before the webpage fully appears (Figure 11). The text on the pop-up says “Traducir siempre de Ingles” (Always tanslate from English) which can be considered to be an invite to change the language preferences when navigating to a page in another language. The pop-up requires the user to click “Espanyol” (Spanish) to translate to or close the pop-up indicated by an ‘x’. In this case, the pop-up was closed. This pop-up was not generated by the webpage Kids Britannica but rather “Google Translate” (Figure 11). Importantly, this highlights a need to recognise distributed agents and agency across the “layers” [36] “of message, interface, communication flow, organisation of information, the platform” [36] in order to unveil power relationships more fully. Designers (or signmakers) can be involved not only in webpage design but also navigation design between pages, namely browser design, in this case from Google.

4.2 Hypertext and visual icons as passive signs-as-agents

Both websites had what can be considered to be ‘passive agents’ on the pages. These were in the form of hypertext, indicated by a different colour than the body copy (Figures 12 and 13) and visual icons that could be made up of colour, text and shape such as the ‘Free 7 day Trial’ (Figure 12) and ‘Subscribe Today’ icon (Figure 15). These could be clicked, or “initiated”, at any moment by the user, depending on user intentions. The “response” of the system would be to move the user on to another page. Therefore, they are considered to be passive signs as potential agents that can be “agentalised” when clicked. Similarly, visual/textual icons including ‘tabs’ (4 Tabs in Figure 16) and visual icons including images could also be clicked (6 square images in Figure 16).

During the analysis, another type of agent was present that did not emerge in the results of the exploratory study with the Tandem tool, namely ‘Animated agents’. Animated agents simulate human behaviour, such as facial expression, body movement and gesture. In the case of webpages for example, certain signs flickered on and off and people in embedded videos moved, all of which humans could click through navigation. Such animated movement can be understood as mirroring human behaviour.

4.3 Animated signs-as-agents

Finally, with respect to animated signs, only the Cool Kids Facts webpage housed these. These were located on the right hand side of the page (Figures 13 and 14), taking up a third of the page space. In addition, an animated sign banner was also present at the bottom of the screen saying ‘Nueva Tienda’ (New shop) for the furniture company Roche Borbois (Figure 14). This banner changed adverstisements from different companies including the companies ‘Just Eat’ and ‘Western Union’ while the researcher was observing. Clicking on this banner led the user to the various company web pages.

The key point to highlight before applying the typology to another digital tool, is to note the presence of animated signs that were highlighted in the HCI classification of agents (Figure 3. Classification of pedagogical agents in HCI studies), as well as passive and active signs/agents. These are signs/agents that can be agentalised through the completion of either a computer generated or human initiation and response. These turns are multimodal in nature and through analysing the webpage, as well as its use, agents can be identified.

The analysis of the webpages for WebQuests shows how the typology outlined in Section 3, can be applied to another digital tool and practice within Education. Unlike the Tandem tool, where the social interests were pedagogical in nature, the webpages house social interests that are not purely educational in nature, but rather, have legal and commercial interests. Legal interests are present in the Cookies ‘agreement’ as the pop-up seeks acceptance to use users’ data. Commercial interests can be seen in the ‘7 Day Free Trial’ for access to Britannia Kids Encyclopedia, which requires later payment. Although this commercial aspect does relate to the area of education, commercial interests that are unrelated to education are also present. Notably, these interests can take up approximately half of the screen space, underscoring how the co-existence of different social interests can be embued in the diferent design intentions of signs on one ‘educational’ webpage. While these signs/agents can be considered as ‘direct discourse agents’ because users can click on them as a response or initiation, they can also be understood as attempts to shape users’ tool use. The adverts (Figures 13 and 14) have the potential to shape a user’s attention, and therefore the user’s interests, away from the educational content on the webpage on ‘photosynthesis’, towards commercial interests. Similarly, the navigation choices on the Cookies pop-up are limited to ‘accept’ and continue, to change preferences or to click away from the site to another. The use of a solid block of colour used to highlight ‘accept’ can be understood as an attempt to shape the user’s navigation pattern towards ‘accepting’ a legal interest before the continuation to the educational content for the user is made navigationally possible.

5. Two digital tools from ‘Google workspace for education’ fundamentals

The final example in this study relates to two tools within the ‘Google Workspace for Education’ suite of tools for educators. Google Workspace for Education is used by many educational institutions including universities and schools. It is a cloud-based tool that according to Google, provides “a free suite of easy-to-use tools that provide a flexible and secure foundation for learning, collaboration, and communication” (Google, 2020 https://edu.google.com/products/workspace-for-education/education-fundamentals/). The range of tools include Gmail (email tool), Calendar, Meet (videoconferencing tool), Docs, Sheets, Slides, Forms, Classroom, Assignments, Sites, Groups, Drive, and the Administrator Dashboard. The suite of tools can be shared with other users so that they can collaborate synchronously or asynchronously.

Screenshots of the tools in use were taken by the researcher. The screenshots are from my own social practice, in my role as a teacher who has to manage a wide variety of social actions related to teaching and learning. These roles include contacting students and communicating with colleagues by email (Gmail) as well as creating educational resources by ‘writing’ on a document in Google Drive.

Figure 17 is a screenshot of an email from an (anonymized) student to me. In the email, the student has sent me work, attached as a document. At the bottom of the screenshot there are three phrases: ‘Great’, Thanks for letting me know’, ‘Great!’ and ‘Great. Thanks!’. One of the affordances of these phrases for the user is being able to click these phrases for intentional use as a response to the sender, that may save time writing. It is also useful for making sure that the phrases used by the writer of the email are grammatically and orthographically correct. This is conceivably important if English is not your first language. However, using the typology in Figure 8 of signs-as-agents, I propose that these phrases, that can be clicked on by users as “off the shelf” pre-made responses, can be understood as passive agents. The design intention of the signs (‘Great. Thanks for letting me know.’, ‘Great!’ and ‘Great. Thanks!’) is presented to the user in order for the user to initiate (click) instead of making a human typed response. In this action, the initiation serves to replace a human written/typed turn.

Figure 17.

Screenshot of an email from an (anonymized) student to me.

At this point it is useful to compare this finding with the findings from the Tandem tool, specifically how learners’ oral turns can be shaped in different ways. Whereas Knight, Dooly and Barberà [23] highlighted how screen-based resources could shape oral turns by lexical/visual items becoming embedded or becoming topics of talk, results of the analysis in this study illustrates how screen-based resources can shape written (typed) turns of humans in a different way. This is illustrated in Figure 17 whereby if the user intentionally clicks on a sign/screen-based resource, the chosen sign can then replace the user’s (typed) turn completely. We can understand these signs as other potential shaping agents.

In Figure 18, I turn to another tool in the Google Suite, namely a Google Document in Google Drive. The affordance of using Google documents is that they can be shared across time, space and also be used by a number of users, synchronously or unsynchronously.

Figure 18.

Screenshot of a Google Document that is being used for planning and educational course.

In the Google Document, in my role as a teacher, I have started to use the document for course planning purposes within an educational setting. As I am typing, the system ‘suggests’ how I might finish the word that I have started to type (‘said’) as well as suggesting complete words that I might type next (‘and done’): acting as a predictor. To ‘accept’ the suggestion, I can press the right-arrow key. To ‘reject’ the suggestion, I can keep typing. Following the emerging typology, we can understand this predictive text to be a shaping agent that is passive and that can become embedded in my typed turn (understood as the completion of the word). Also following the typolgy, I can agentalise it by ‘accepting’ using the right-arrow key. This choice of ‘accepting’ and ‘rejecting’ can be understood as my response turn. Furthermore, this process conceivably resembles a negotiating process with the digital tool, rather than a purely creative one, as I negotiate my responses through touch, with the screen-based resources.

Before concluding, a second version of the typology is presented in Figure 19, encompassing all the signs-as-agents identified across the three examples analysed.

Figure 19.

Emerging typology: Version B.

6. Conclusions

This study aimed to highlight how educationalists might (re)conceptualise digital tools and their use through a critical lens. It achieved this by specifically focusing on the development of a theoretical and analytical framework that formed the basis for a typology for identifying signs-as-agents. This typology was shown to be applicable to other educational tools and scenarios. It can be expanded further, taking into account different digital technologies in use and different social practices it is being applied to. The study has highlighted, through analysis with examples, why educationalists should rethink the concepts of ‘affordances’ of tools and ‘agency’ because digital agents are also present as human users/agents carry out their semiotic work. This focus highlights a shift in emphasis from a human centric, individual agency towards a systems-based understanding of agency in which the human systems (e.g., motor system) together with the digital system, form a part. Furthermore, the focus highlighted why the notion of ‘designers’ and ‘design intentions’, rather than end ‘user’ is perhaps a more relevant concept for a more holistic and critical understanding of how semiotic work is produced/created in educational social practices. Importantly, the study has shown how and why this process is a negotiated process with other designers’ intentions, not a purely productive or creative endeavour of a human user.

This paper, as a critique, broadly focused on ‘patterns’, ‘designs’ and ‘shaping’ [28] which are key notions in Critical Digital Literacies. The patterns under focus were exchange structures (or interactional patterns), understood as turn-taking. This focus included an analysis of the ‘designs’ of potentially many designers and users of others’ designs across time and with different social interests or intentions. This highlighted the social interests that permeate the designs and use of digital tools [26]. Furthermore, signs-as-agents can be understood as an extension of their human designers (as agents) highlighting the role of signs as ‘proxy agents’ [5] for designers. Signs understood as extensions of their designers also underscores the notion of designers’ interests or intentions being ‘distributed’, echoing Distributed Cognition Theory [9]. Furthermore, the ability of signs-as-agents to act on and be acted on by teachers and learners highlights the distributed agency that both human and digitally manifested intentions are capable of.

In addition, the results highlighted how the ‘shaping’ of both learners’ and teachers’ semiotic work can occur, as well as the use of/shaping of pre-designed digital tools and their accompanying signs-as-agents. The ways of ‘shaping’ human semiotic work were shown to be extremely diverse, understood through an analysis of human oral and typed turns.

The chapter sought to add to pre-existing theoretical and analytical tools for educationalists to understand and approach digital tool use from a more critical stance. In doing so, the author proposes that the process of identifying and uncovering signs-as-agents, can contribute to the development of Critical Digital Literacies for Education. This could include looking beyond notions of creation and empowerment to notions that are centred on awareness, design intentions and teachers’ and learners’ abilities to identify and negotiate with those intentions.

Finally, the central contribution of this chapter is a typology of signs-as-agents. Specifically, it has emerged and been developed in order to critique how digital tools shape how we act in educational social practices, rather than how we think. The typology and underpinning theoretical framework potentially supports ‘expanding’ the critical digital literacy research agenda’ [27]. This typology can serve a critical literacies agenda towards digital tools in education that Knight, Dooly and Barberà [18] call ‘Critical Digital Literacy Pedagogies’ (CDLP). CDLP was a proposal conceived to expand Critical Digital Literacies specifically to pedagogy. CDLP can be understood to include, amongst other things, identifying fake news (e.g. The National Literacy Trust, UK’s teaching and learning resources, https://literacytrust.org.uk/research-services/research-reports/fake-news-and-critical-literacy-final-report/) as well as uncovering ideologies or undemocratic discourses in digital content (e.g. see the DISCO project – Embedding a Democratic Culture Dimension in Teacher Education Programmes (EDCD-TEP) focused on embedding the Competences for Democratic Culture in Primary Teacher Education Programmes, 2021 [23], https://pjp-eu.coe.int/en/web/charter-edc-hre-pilot-projects/embedding-a-democratic-culture-dimension-in-teacher-education-programmes-edcd-tep). The proposal for CDLP could conceivably unite analytical tools that aim not only to expose power relations in relation to content and how people think but also in relation to power relations with other digital signs-as-agents and how people (inter)act and negotiate with them.

Regarding future research, the typology could be applied in order to support a much wider analysis of digital tools by teachers and researchers. In doing so, the typology can be added to and developed further. Furthermore, while this chapter has focused on tools and scenarios that involve human responses to signs-as agents while speaking, reading and/or typing as social practices, greater attention could be made to human initiations. For example, the use of QR codes in Mobile Assisted Learning as well as the use of human voice activation with digital tools which would encompass more modes and signs than exist in the current version of the typology.

Acknowledgments

I would like to thank Leo van Lier, who is no longer with us, but through his chapter ‘From Input to Affordance’ (2000), helped to inspire this paper.