The Psychology of Trust from Relational Messages

Abstract

A fundamental underpinning of all social relationships is trust. Trust can be established through implicit forms of communication called relational messages. A multidisciplinary, multi-university, cross-cultural investigation addressed how these message themes are expressed and whether they are moderated by culture and veracity. A multi-round decision-making game with 695 international participants assessed the nonverbal and verbal behaviors that express such meanings as affection, dominance, and composure, from which people ultimately determine who can be trusted and who not. Analysis of subjective judgments showed that trust was most predicted by dominance, then affection, and lastly, composure. Behaviorally, several nonverbal and verbal behaviors associated with these message themes were combined to predict trust. Results were similar across cultures but moderated by veracity. Methodologically, automated software extracted facial features, vocal features, and linguistic metrics associated with these message themes. A new attentional computer vision method retrospectively identified specific meaningful segments where relational messages were expressed. The new software tools and attentional model hold promise for identifying nuanced, implicit meanings that together predict trust and that can, in combination, serve as proxies for trust.

1. Introduction to relational communication

In today’s world, where volatile interactions abound, a critical question that arises is how trust can and should be fostered. A fundamental underpinning of all social relationships is trust, and interpersonal communication is the mechanism through which trust is often accomplished. A multidisciplinary, multi-university, cross-cultural investigation was undertaken to address this question as well as to further explore how trust is established through implicit forms of communication. Employing a decision-making game with multiple rounds and 695 international participants, the University of Arizona, University of California Santa Barbara, Rutgers University, Stanford University, University of Maryland and Dartmouth University investigated the manner in which nonverbal relational messages, comprised of nonverbal and verbal communication, might secure trust [1]. Applying a spiral model of trust, we formulated predictions of how people utilize implicit, relational messages to define their interpersonal relationships and from those exchanges, ultimately arrive at a determination of who can be trusted and who, not. We examined how relational messages of affection, dominance and composure signal and elicit trust, either universally across cultures or not, and how those messages are moderated by deception.

1.1 Relational communication and trust

An integral part of human communication is the exchange of what are called relational messages. These are implicit messages that enable people to assess how they relate to one another and how they regard their interpersonal relationship. For example, at the most basic level, people must determine who is friend and who is foe, who they like and who they dislike, and whether the relationship is superficial or one of depth. These messages more often than not are expressed through nonverbal behaviors, which are the focus of this report. Although they also can be expressed verbally, for example, telling another that you trust them and find the relationship to be a deep and abiding one, the preponderate share of relational communication is managed nonverbally. In this way, verbal and nonverbal communication accomplish a division of labor, with the verbal aspects of communication handling substantive matters and the nonverbal aspects of communication handling much of the relationship work.

For example, a discussion in a classroom devoted to the topic of the election may be transacted through words, while nonverbally the students and teacher signal what the power relationship is—whether the instructor is in charge and the students are acquiescent to her or his authority, or the instructor is intending to instill a communication environment of equality; whether the instructor and students like one another or harbor some hidden animosity; whether they are engaged in the topic or are disinterested and detached from it; and so on. These various messages combine to build a foundation of mutual trust and goodwill such that the instructor presents what he/she believes to be the most current and valid material and the students enter the exchange accepting that the instructor is knowledgeable and credible, or the students are distrustful of the instructor’s motivations, material and credibility and reject it.

The topoi of relational communication are generic continua of message exchange by which we can characterize all human interactions [2]. Burgoon and Hale [3, 4], after reviewing analyses of human relationships from such disciplines as anthropology, ethology, psychology, psychiatry, sociology and communication, identified up to 12 dimensions along which communication is transacted. To the extent that these are universal, these themes should arise in all cultures, though possibly to different degrees. How they are expressed, and in particular how the central ones relate to trust, are the major objective of this current research project.

Our theory of relational communication is that relational message themes are universal, interdependent, and together, through their dynamic exchange, become the cornerstones of trust. One avenue of our work examined self-reports of the communication behaviors people use and observe. This self-report work examined how relational themes are shown in various, disparate countries; the extent to which those countries are similar or different in the emergence of relational messages; and what behaviors contribute to perceptions of trust. A second avenue of work examined macro- and micro-level kinesic, vocalic and linguistic behaviors indicative of the major relational themes of dominance, liking, and composure and ultimately, how they contribute to trust. Three open-source software tools, OpenFace, OpenSmile, and SPLICE, were employed to investigate what nonverbal and verbal behaviors predict relational messages of dominance, composure and liking (see [5]) and whether the same behaviors could be used to develop a predictive model of trust. The nonverbal and verbal communication behaviors were also examined across six countries. A third avenue drilled deeper into the interpretive micro-level behavioral aspects of relational themes using computer vision techniques. Together the lines of investigation explored how trust spiraled dynamically over the course of group decision making and what relational message themes showed the most change.

1.2 Topoi of relational communication

In human relationships, an intrinsic theme of relational communication is dominance-submission, which reflects the vertical dimension of primate relationships in social settings. People must know what the power structure is, whether there is a discernible status hierarchy, and who sits atop the pecking order and who is at the bottom. One’s relative position in the hierarchy is typically negotiated through nonverbal kinesic, vocalic, proxemic, haptic, physical appearance, artifactual and chronemic signals. These signals are arrayed as continua. One can variously be highly, moderately or not at all dominant in relation to another. In dyads, groups, families, organizations and the like, people can be arrayed from most to least powerful, highest status to lowest status, most acquiescent to not at all. According to Burgoon and Dunbar [6], dominance is dynamic and situationally contingent. It is an actual action that recruits a submissive, acquiescent response from another. Whereas power may reflect a potential to act, dominance is the actual expression of that potentiality. If a dominant overture fails to elicit a submissive response, it is not dominance, but merely domineeringness. Thus, dominance requires both an action by Person A and a complementary, coupled response by Person B.

A second dimension is variously called affection-disaffection, love-hate, or liking-dislike. It reflects the valence dimension of relational communication that ranges from highly positive to highly negative. It is orthogonal to the vertical dimension. People may feel affection toward another and express it through a host of nonverbal signals. Conversely, they may dislike another and express that sentiment through nonverbal signals as well, although social mores inhibit sending highly visible or vocal expressions of dislike.

Along with dominance, affection is one of most prominent relational message themes. These two themes are central ones around which the other relational communication themes are arrayed. Three additional nonorthogonal topoi include composure-nervousness, involvement-detachment and similarity-dissimilarity. A person may express a sense of poise and composure in the presence of another or may appear nervous, anxious and uncomposed. In other words, one’s demeanor is altered in relation to the other person. It does not reflect a general demeanor around others but rather, a person-specific nervousness or composure. A person may also show high or low involvement with another, that engagement being behavioral, cognitive and emotional. Yet another of the topoi is similarity-dissimilarity. Ongoing interactions with unfamiliar others require trying to assess the degree of similarity that exists between them. Such similarity is a starting place for communication. When homophily between individuals is high, communication is likely to be the most successful [7].

1.3 Spiral model of trust

All of these topoi are interrelated to the theme of trust. As explained in the spiral model of trust [7], trust is an interactive and iterative process that derives from multiple factors. It flows from, is sustained by, and modified through dynamic communication patterns. These patterns include the dominance relationships the parties bring to an interaction—such as the father being the head of a family and wife and children showing obedience to the father’s strictures in a traditional paternal family structure, or the members of egalitarian LGBTQ couples showing similar degrees of dominance while negotiating decisions. The degree of positive affection that members of a social unit feel toward one another is communicated through the kinds of kinesic, vocalic and haptic patterns measured in our study. In families, for example, how loving the siblings feel toward one another and their parents dictates how they express that affection dimension through nonverbal messages. A third dimension reflected here is composure or nervousness—how at ease or tense members of the relationship feel in the presence of one another. If a person feels uneasy in the presence of others in a group, they may display that uneasiness through nonverbal messages of discomfort.

These dimensions combine to spiral into greater or lesser trust, and that spiral can change over time, becoming more intensely trustful or more suspicious and less trusting. Trust is a moving target. It is modified by the situation in which people find themselves and the relational messages they receive from others. In a group setting, for instance, members who wish to promote others’ trust in themselves may attempt to temper their demeanor initially by being nondominant, but over time to bolster their persuasiveness by increasing signals of dominance. To promote liking and composure, they may send to others positively toned messages of liking and being at ease in hopes that those sentiments will be reciprocated. The important points to draw from the spiral model are that trust is the product of many different relational messages, as illustrated in Figure 1, and are in a state of flux, depending on the current context. The relational messages are comprised of various nonverbal and verbal signals in various strengths and combinations, the net result of which is the expression of trust and receipt of messages of trust. These messages may be communicated in similar ways across cultures, but to the extent they are communicated dissimilarly, culture must be taken into account.

Figure 1.

The relationship of relational messages to trust.

The context for this study of relational communication and trust is situated within a multi-national study of cultural differences in conducting and detecting deception. In potentially adversarial situations, messages may be moderated by deception, which adds a toxic element to the exchanges. Thus, deception is also a centrally important moderator.

1.4 The relationship of culture to relational communication and trust

Trust places people in a state of vulnerability to deception by others. Trust is often equated with a truth-bias, in other words, expecting that others are truthful, not deceptive. Defined formally, truth-bias is an overestimate of another’s truthfulness independent of their actual honesty [8]. Thus, to understand trust is to understand deception and vice-versa. Trust and deception are intricately interrelated. Eliciting another’s trust is accomplished by showing that one is not deceptive, by conveying authentic or apparent honesty. Likewise, assessing another’s trustworthiness may be based on spontaneous impressions from another’s nonverbal demeanor, which can lead to dangerous decisions when that demeanor is false [10].

Unfortunately, most deception research pertaining to trust has been done in a “cultural vacuum” [11]. Moreover, the vast majority of studies on verbal and nonverbal cues to deception or deception detection skill have been done in English-speaking, western cultures. The work involving culture has focused largely on whether people who are from the same culture can detect deception within an interaction episode better or worse than people who are from two different cultures (see [12] for a review). Very few studies have analyzed cultural differences in displays associated with deception or in the detection of deception (i.e., comparing norms and behaviors of people who are situated in different cultures, such as cues used during deception by people in Japan versus by people in the U.S.). This leaves questions about cultural-level variations in decision-making concerning trust of an interaction partner unanswered.

There are two main theoretical perspectives on how to detect deception across cultures and hence, whether to trust an interaction partner. The first is the universal cues hypothesis [13]. The central premise of this perspective is that due to the evolutionary benefits of successful deception and deception detection being similar for all humans, the cues emitted by deceivers are unlikely to vary from one culture or society to the next. Moreover, for the same reason, the universal cues hypothesis says detectors of deception will experience and interpret those cues similarly in all cultures. In other words, the universal cues hypothesis expects deceivers should act similarly and deception detectors should have evolved similarly to spot deceptive behavior across cultures. Supporting this view, the Global Deception Research Team’s study [14] found that deception has vast similarities across 75 countries. Another investigation across 5 countries [15] proposed and supported a pan-cultural typology of 10 motives for deceiving. In sum, the universal cues hypothesis predicts minimal cultural differences in deception detection, and thus by extension, how trust decisions are formed between interaction partners.

The second view is the specific discrimination perspective [12, 16]. This perspective takes the position that people rely on learned, culturally-determined norms and expectations to guide both their behavior and sensemaking during an interaction. As such, lying is conditioned by culture because cultures differ in their nonverbal behavior norms and displays, the value attached to honesty, frequency of lying, conditions for interpersonal trust, and responses to others’ lies. Consequently, the specific discrimination perspective posits that deception and its detection are specific to communication patterns that vary across cultures. This helps to explain findings that people can better identify a liar from their own culture than a liar from a different culture by noticing deviations from their own learned cultural code (e.g., [12]). Applying this perspective to the decision-making process regarding trust of an interaction partner, the same forces lead to the prediction that trust and trust decision-making should vary in different cultures.

A few studies have tested these competing hypotheses about the influence of culture in deception detection. For example, George and colleagues [17] studied deception and its detection in three countries: America, India, and Spain. Participants evaluated 32 snippets of recorded interviews involving the three cultural groups across two languages. Within each stimulus set, half of the snippets were honest and the other half were dishonest. The researchers measured the cues that judges in each country cited as important to their decision about a person’s honesty. Twenty-three cues were identified (e.g., nervousness, logical structure, talk time, voice pitch, etc.). The authors found that judges across the three cultural/language groups relied on similar cues. For example, nine cues including lack of eye contact, fidgeting, tone or pitch, pauses, stuttering, vague reply, repetitive answers, contradicting oneself, and bragging accounted for a large majority of the deception cues used by judges in all three countries. The cues included kinesic, vocalic and verbal indicators. George and colleagues concluded that their results tend to support the universal cues hypothesis (see also [12, 13, 16]). Also supporting the universal cues hypothesis, others find that the frequency, motives for lying, and skill in deception detection are also similar across cultures (see [18] for a review). One of the few investigations of actual behavior and trust, and a model for the current experiments described in this chapter, is [19].

The consistency in findings helps to offset the paucity of culture-based experiments. Nevertheless, more controlled investigations rather than anecdotal reports are needed to confirm a universal cues hypothesis. Our investigation begins to fill the void by conducting the study in multiple countries ranging across four different continents and observing actual behavior rather than relying on self-reports of behavior, thereby allowing for the potential for variability that supports the specific discrimination perspective.

1.5 Behavioral indicators of relational messages

Much research looks at people’s perceptions rather than actual behavior. We were interested in going beyond perceptions to look at actual behaviors people display that foster trust and that signal they trust another person. We looked at nonverbal signals from the voice (known as vocalics), the face and body (known as kinesics) and linguistic (verbal) indicators. We hypothesized that trust would be communicated by those signals associated with messages of liking, moderately high dominance and moderate composure, and that it would be a combination of these signals that would evince and elicit trust. In other words, the more a person showed that they liked another individual and felt reasonably relaxed and composed around that individual, the more they were likely to trust that person. Similarly, people would be trusted the more they showed they liked others, exhibited many of the signals associated with moderate dominance and conveyed that they were moderately relaxed, not nervous.

2. The experiments

2.1 Sample

Participants were recruited from nine universities in six countries: Zambia, Israel, Singapore, Fiji, Hong Kong, and the U.S. (which included three of the universities, all with a diversity of international students). Country selection was a function of finding universities with a willing local host and a federal government that would consent to the 60-page Institutional Review Board requirements. All participants were current students (mean age = 22 years). In total, 695 people participated in the experiments and 95 games were played.

2.2 Method

We devised a method for analyzing trust using an interactive social game created by Don Eskridge called The Resistance (variations of the game are sometimes known as Mafia or Werewolf) played in groups comprised of five to eight strangers. A detailed description of the game is found in [24]. It began with an ice-breaker activity designed to establish a baseline for perceptions of dominance, liking, nervousness and trust. During the ice-breaker, players introduced themselves to the group, told an interesting fact about themselves, and then another player probed with a question to elicit more information. We measured their perceptions of the other players on self-report scales immediately following the ice-breaker.

Following the ice-breaker, Resistance players were randomly and secretly assigned to play one of two roles: deceivers (called “Spies”), or truth-tellers (called “Villagers”). There were two to three Spies per game, depending on the size of the group, but Villagers always outnumbered the Spies. The Spies were aware of who the other Spies were, but the Villagers did not know anyone else’s role. Villagers attempted to deduce the other players’ identities within the game. The players engaged in a practice round before the game play actually began to ensure they understood their goals in the game. Spies’ purpose was to win the game by remaining hidden and infiltrating the Villagers’ groups while the Villagers’ goal was to uncover who the Spies were to avoid infiltration. So, finding out who to trust and who not to trust was especially crucial for the Villagers but important to the Spies as well.

Players completed a series of hypothetical “missions” by forming teams of five to eight members. At the beginning of each round, players elected a leader, who then chose other players for these missions based on who they thought would help them win the game. All players voted to approve or reject the team leader and then voted on the leader’s proposed team. Again, trust played an important role because players from both sides needed to trust that the leader was picking the team that would ensure they won the round. Players took both a public and private vote, thus introducing the potential for distrust when the two votes did not match. Those who were chosen by the leader to go on the mission team secretly voted for the mission to succeed or fail. Villagers won rounds by figuring out who the spies were and excluding them from the mission teams to ensure mission success. Spies won rounds by causing mission failures. The ultimate winners of the game (Spies or Villagers) were determined by which team won the most rounds (up to a maximum of 8 rounds). Additionally, players won monetary rewards by being voted as a leader or winning the game. (See [24] for more details).

2.3 Measures

Following Cho and colleagues’ recommendation [20], culture was measured both at the macro level in terms of each player’s country of residence and also at the individual level in terms of self-construals on the cultural orientations of vertical and horizontal individualism and collectivism, as well as positive and negative face [20, 21, 22]. Before the game began, participants completed a set of self-report measures used by [19] to gauge individuals’ cultural orientations along dimensions of individualism–collectivism, horizontal-vertical status, and positive–negative face [22, 23]. In total, six measures were taken: vertical collectivism, vertical individualism, horizontal collectivism, horizontal individualism, positive face, and negative face.

Vertical individualists place value on independent individual achievement and tend to be competitive with others, while vertical collectivists accept inequality in the social structure, value self-sacrifice for group goals and collaborate with others. Horizontal collectivists value cooperation and caring among group members and strive for group harmony. Horizontal individualists value both equality and uniqueness in a way that respects individual decision-making. People high in positive face have a desire to protect their self-esteem by making positive impressions on others, as positive face reflects a felt need for social approval. In contrast, people high in negative face feel a need for interpersonal distance to protect their autonomy and value privacy for both self and others.

During the game and at its close, participants completed self-report measures to gauge their perceptions of liking, dominance, composure and trust of each other player in their game after an ice-breaker activity and then again after rounds 2, 4, 6, and 8, if the game lasted that many rounds. Because participants responded to these items three to five times about each of the other players, we used single items to avoid fatigue.

2.4 Behavioral indicators

Many of the same behaviors were featured in the various relational messages because they were expected to be highly correlated. In [5], the verbal, kinesic, and vocalic indicators were summarized. Among those that were tested, the significant predictors of trust are noted below with an *.

Behaviorally, we looked for the following indicators of liking:

• proximity, direct body orientation

• Backchannel cues (nods, uh-huhs)

• Postural mirroring

• Relaxed laughter

• Vocal pitch variety

• Rapid turn-switches

• First-person plurals

• Positive affect language

Dislike was expected to be signaled primarily by the opposites of these, such as indirect body orientation, lack of backchanneling and mirroring, and absence of relaxed laughter.

For dominance, we looked for these signals:

• High immediacy or nonimmediacy (combination of proximity, gaze, body orientation, forward lean, touch)

• More head movement, pitch, roll and yaw*

• More facial expressiveness*

• Verbal nonimmediacy

• Frequent initiation of conversation

• High visual dominance ratio (more looking while talking than looking while listening)

• Deeper and more variable pitch*

• More interruptions

• More and longer turns at talk*

• More first-person pronouns

• Louder amplitude*

Nondominance or deference would be conveyed by the opposites, such as physical rigidity, passive facial expressions, and low visual dominance ratio, higher pitch, few interruptions, shorter turns at talk, and more first-person plural pronouns or third-person pronouns. More of these indicators were vocalic or verbal.

For nervousness, we looked for the following:

• Softer amplitude*

• Higher pitch

• More nonfluencies

• Shorter and fewer turns at talk*

• More gaze aversion

• More indirect body orientation or facing

• More facial rigidity (FAU02)*

• More postural, head, and vocal rigidity*

We hypothesized that trust would be communicated by those signals associated with messages of liking, moderately high dominance and moderately low nervousness, and that it would be a combination of these signals that would evince and elicit trust.

3. Results

3.1 Descriptives

The number of participants and games played in each country was uneven but contributed to acquiring a diverse sample: U.S. (30 games, 209 players), Singapore (12 games, 84 players), Fiji (14 games, 106 players), Israel (9 games, 64 players), Zambia (15 games, 117 players), and Hong Kong (15 games, 115 players). Player cultural background was quite diverse. Players reported being from 42 different nationalities and over 60 different ethnicities. Participants self-classified as either Asian (38%), white/European (18%), black/African (17%), Fijian/Pacific Islander (15%), Latinx (3%), Middle Eastern (1%), mixed (3%), or other (5%).

Despite this diversity, and interestingly, mean scores across participants on the cultural orientations were more similar than expected. Figure 2 shows the means for each of the cultural orientations measured, in each of the six countries. Scores in the US, Israel (IL), Hong Kong (HK), and Singapore (SP) were very similar across all of the orientations. Participants in Zambia (ZM) and Fiji (FJ) reported the highest scores on horizontal and vertical collectivism, horizontal individualism, and positive face, but the lowest scores on vertical individualism compared to participants in the other countries. We did not find individualism and collectivism to align with expectations about participants from eastern versus western cultures, nor did we find those two cultural orientations to be orthogonal as originally conceived by Hofstede [21]. Instead, most participants in all six locations reported roughly equal levels of individualism and collectivism. Others have found similar results, sparking debates about whether Hofstede’s original conceptualization of the cultural differences needs updating, especially in light of greater cross-cultural communication and globalization in recent decades (e.g., [25, 26].

Figure 2.

Mean scores for each cultural orientation by country.

Games were played in English. 39% of the sample were native English speakers. Among those who were not native speakers, their average self-reported English-speaking fluency was quite high (5.82 on a 7-point scale, with higher scores indicating greater fluency.) The average age at which the non-native speakers began to speak English was 6 years. This is not surprising because in all of the countries except Israel where data for this study were collected, English is an official language, and in Israel, English is required as a second language in schools. Thus, individuals in the sample possessed a high level of English language proficiency. Based on this, we feel confident that the results would be very similar as if all players had played the game in their primary language. Future research, however, is needed to understand if differences in the language used for communication influences both the expression and perception of trust.

Experience with similar deception-detection games varied across the locations: 50% or more of the players in Singapore, the U.S., and Hong Kong reported playing a similar game in the past. The number of rounds played within the allotted one-hour period also differed significantly by country, with the most rounds played on average in the U.S. (6.6) and the fewest in Israel (3.6) and Zambia (3.6).

3.2 Cultural impact

Our first analysis was whether culture makes a difference. If results differ by culture, culture must be included as a variable in other analyses or each culture should be measured separately.

Villagers in the six countries varied in their trust of other players. Zambian Villagers reported the lowest trust of all other players in their games with no differentiation between deceivers (Spies) and fellow truth-tellers; whereas Villagers in Singapore reported significantly lower trust of Spies than fellow truth-tellers. Only a little evidence was found to support a relationship between trust and the cultural orientations. Deception detectors higher on vertical individualism (i.e., who are competitive and prefer to work alone to defeat an enemy) reported less trust of the deceivers in their games (r = −.11, p < .05), indicating that vertical individualism sensitizes people to deception cues perhaps via the competitiveness aspect of this cultural orientation. That said, the correlation is weak and none of the other cultural orientations were significantly correlated with trust.

We also looked at trust dynamically over the course of the game. The same pattern was found in all countries such that villagers’ trust decreased for both truth-tellers (other Villagers) and deceivers (Spies) in the early rounds of game play, but then rebounded for trust of truth-tellers (see Figure 3a) while continuing to decline for deceivers (see Figure 3b).

Figure 3.

Trust of (a) truth-tellers and (b) deceivers over the course of the game by location.

Despite some variability, the results in the aggregate show limited cultural differences across judgments and behavioral patterns relating to trust. The similarities better warrant a conclusion in favor of the universal cue hypothesis than the specific discrimination perspective when it comes to trust. That said, cultural differences remain a persistent point of interest that warrant continued examination in the future.

3.3 Subjective judgments

For subjective judgments of relational messages, we hypothesized that Spies who display more signals of liking, dominance and composure are trusted more by Villagers.

First, the rating of liking at the end of the game was highly correlated with the post-game ratings of dominance and composure. The more players expressed dominance and composure, the more they were liked. In turn, as predicted by the spiral model, such players were trusted more.

A regression model showed that the Villagers’ judgment of Spies’ dominance was a significant predictor of trust (R² = .059). This was true for all the countries sampled, especially Singapore and the U.S. The exception was Israel, where more dominance was associated with less trust, possibly because Israelis already scored high on dominance and might have found extreme dominance to be excessive and suspect.

As another indicator of trust, Villagers identified who they thought were Spies among all the players in their game, and the actual spies were less often judged to be one (R² = .078).

In detailed results reported by Dunbar et al. [27], Villagers’ ratings of Spies on dominance decreased over time, whereas it increased for Villagers as they came to their final game round. Dominance was correlated with trustworthiness. Dominant players may have been more confident in their abilities or perhaps had more charisma and extroverted styles that led them to appear more trustworthy. Despite their increased dominance, Villagers were not more likely to win (the win ratio was nearly 50/50 for Villagers and Spies). Thus, the link between dominance and trustworthiness did not seem to result in outcomes that are beneficial to the players. Ratings of apparent nervousness were only mildly affected by the players’ game role: Villagers were less nervous over time than Spies but only slightly so. Villagers became more relaxed, while Spies remained somewhat nervous.

Figure 4a and b show the pattern of relational messages predicting trust by role, in this case illustrated with the dominance relational message. Although ratings declined initially, ratings of Villagers remained higher than Spies and showed an upswing over time. In other words, the more Villagers communicated moderately high dominance, the more others saw them as trustworthy; Spies expressing dominance were also seen as more trustworthy. The correlation between dominance and trust ranged from .23 in the baseline to .37 at game’s end, indicating that moderately high dominance contributed to more trust.

Figure 4.

Relationship of dominance to trust, by veracity.

3.4 Behavioral relational messages

Facial expressions can convey a lot of information about one’s physical and emotional state. People rely on facial expressions to “collect” both intentional and unintentional meaning during interactions. The Facial Action Coding System (FACS) [28] was developed as a systematic way to code facial motion by segmenting the face into separate regions (forehead and eyebrows, eyes and cheeks, mouth and chin). Each of the motions, such as an eyebrow raise or an open-mouth smile, is a Facial Action Unit (FAU).

Results from a dissertation showed that dominance was predicted from nonverbal kinesic signals, head movement and vocal signals. As illustration, Table 1 shows all the facial action units, both means and standard deviations, with significant relationships with dominance [29]. There are several facial muscles involved with various emotional expressions, but the most noticeable effect is high standard deviations, meaning there is a lot of variability or expressiveness. Composure had several relationships, although fewer than dominance, but trust had only two FAUs that emerged as significant. AU23 appeared most frequently. (Liking was not examined.)

Dependent Variable: Dominance (Low/Medium/High)
Control Variables: Gender, Game Experience, English Proficiency
FAU	Description	Significant Means	Significant Standard Deviation Coefficients
AU1	inner brow raiser		*
AU2	outer brow raiser		*
AU5	upper lid raiser	*	*
AU6	cheek raiser		*
AU9	nose wrinkler	*	*
AU10	upper lip raiser		*
AU12	lip corner puller		*
AU14	dimpler	*	*
AU15	lip corner depressor	*	*
AU17	chin raiser	*	*
AU20	lip stretcher		*
AU23	lip tightener	*
AU25	lips part	*	*
AU26	jaw drop	*	*
AU45	blink		*

Table 1.

Linear mixed-model analysis of facial action units related to dominance.

Among the most important behavioral vocalic signals in these models were utterance length, harmonic noise ratio (a quality measure), pitch, loudness and shimmer. Longer utterances, more voice quality, deeper pitch, louder, more variability in loudness and more shimmer were more indicative of dominance. These signals were most evident around critical decision points in the game, such as choosing team members and voting on leaders for the mission team.

Liking was most predicted linguistically by the number of sentences. Liking was higher, the more a person spoke multiple sentences. Vocally, it was most predicted by measures of voice quality (harmonic noise ratio standard deviation, jitter standard deviation, and shimmer). These measures of voice quality indicate liking was higher the less the voice HNR and jitter varied and the lower the presence of shimmer. These are indicative of a consistent, unvarying voice.

Nervousness was associated with softer amplitude, more jitter, less dominant language but surprisingly, longer turns at talk. Composure would be the opposites of these, i.e., louder, less jitter, more dominant language, and shorter speaking turns.

Linguistically, trust was most predicted by more turns at talk, more words, and a higher readability score (i.e., more articulate speech). More talk, with a more educated voice, elicited trust.

Does the smallish number of predictors of trust mean there are few nonverbal and verbal signals of it? No. The reason is the interdependence of these variables. They are correlated with one another, so the statistical models using multiple regression identify the signals that account for the most shared and unique variance (have the biggest impact). Although included as possible predictors are all the variables identified above in the 2.4 Behavioral Indicators section, because so many of these variables are highly correlated with another, the statistical models will only retain the most significant variables (i.e., the ones accounting for the most variance). These are the best predictors, but doubtless several other indicators combine with them, or substitute for one another to convey a given meaning. For example, affection can be expressed by smiles, or frequent eye gaze, or touch, or direct facing, or a combination of these, as well as plural first-person pronouns and more intimate language.

What is apparent is that all the communication channels—verbal, kinesic, vocalic, and linguistic--play a role in the trust process and together can convey trust in a very substantial manner. Classification analysis for specificity (i.e., identifying truth-tellers) showed 74 - 79% accuracy in spotting truth tellers, a significant level of discrimination.

3.5 Retrospective attention mechanism through computer vision

One of the unique contributions of the current investigation was the development and application of a computer vision method for retrospectively finding the most meaningful segments in a video. Here we describe the framework and its application to dominance, liking and trust in videos using robust facial features. We create a mechanism to compute the attention of the detection model in the time domain, identifying key frames. We use those key frames to draw conclusions about the kind of micro-expressions that emerge as important during the attentional periods of the model.

The Facial Action Coding System (FACS) [9] was developed as a systematic way to code facial motion with respect to non-overlapping facial muscle actions called Facial Action Units (FAUs).

With so much communicated by the facial expressions, we opted to incorporate facial cues into a system to investigate whether the presence and intensity of some specific facial expressions correlate with how dominant, likable, and trusted a person is perceived by others. For the technically minded, our approach has the following pipeline. A morphable model is superimposed to a subject’s face and, with the help of a feature extractor, for each frame of the input video the intensities of 17 Facial Action Units (FAUs) are computed. These are normalized with the parameters of the morphable model, resulting in 17 identity-agnostic FAU intensities. Also, gaze angles of the subject are tracked for each frame of the input video.

The 19 1-dimensional signals (17 FAUs and 2 gaze signals) were concatenated, channel-wise, and this signal was fed as input to a model for video classification. The model used was the Temporal Convolutional Network (or TCN) [30, 31]. We used 250 videos and trained our model to regress dominance, liking and trust using an MSE loss function. Given a trained model, we predicted dominance, liking and trust on a test set and retained 25 subjects with the lowest error for further analysis. By using the gradients of the model, we identified the key frames in the input video and performed retrospective analysis on the facial features that are most prevalent in the model’s prediction. The overall framework can be seen in Figure 5. In Figure 6, we can see the attention visualization of the model.

Figure 5.

Illustration of the proposed framework. FAU intensities and gaze angles are extracted from video sequences which are considered as 1D normalized channel-wise concatenated signals to train a predictor model. Model attention is computed to enable retrospective analysis of dominance, liking and trust.

Figure 6.

(left) screenshot of frames from original videos; (right) FAU waveforms and attention visualizations of the predictor model. We can see that the model trained for liking identifies as key timesteps, the frames that the subject smiles; lip corner puller (FAU 12) and upper lip raiser are maximally activated at those frames.

Analysis of the players’ facial behaviors revealed that some facial action units including lip stretching, blinking, and fake smiling occurred more frequently during deceptive acts. These might be expected with inauthentic trust. Further analysis of the players’ facial muscles suggests that subjects who were more dominant, likable and trusted had more intense facial expressions. Speculatively, it seems that those subjects were more involved in the game and as a result gained the trust of their peers. Furthermore, specific facial expressions, such as smiling and eyebrow raising, emerged more than others.

There were no noticeable differences when examining the FAUs across subjects from different countries, further supporting our intuition that expressions of trust are culture-invariant.

4. Discussion and implications

As part of a cross-cultural, multi-purpose investigation, this project investigated whether trust is signaled by kinesic and vocalic (nonverbal) features and linguistic (verbal) features. Results show that all these features convey relational messages of dominance, liking and composure, which in turn combine to signal trust or distrust. Although the relational messages are moderately correlated with one another, different types of signals are present in each relational message. Whereas both kinesic and vocalic signals play a role in conveying dominance or nondominance, vocalic signals are more prominently featured in composure or nervousness, and facial expressions are especially salient in signaling liking and disliking.

The various indicators, or their perceptual representation, spiral together to form an amalgam of trust. The verbal and nonverbal signals are dynamic, so that their meaning is in flux. Rather than judgments being made anew during each round, it appears that trust is forged from an accretion of meaning built up by the interaction context. For example, in the current task, over the course of several rounds of decision-making, players had the benefit of results from prior rounds to inform their current judgments and build up impressions of other players’ trustworthiness. In other words, judgments were cumulative rather than transitory. First impressions may also have set an interpretation frame that colored all that followed. If, for instance, someone had a reputation for dishonesty, nonverbal and verbal signals by that individual might be attached to that initial expectation and help build an impression of someone who should not be trusted. This is often the case when members of law enforcement quickly narrow an investigation to a single suspect and in a hypothesis-confirming manner, interpreting all relational messages to fit their first impression. Scam artists hoping to swindle elders out of their social security checks rely on this principle to create a favorable first impression and continue to build upon it.

Other relational messages may fill out skeletal first impressions, adding, for instance, messages related to involvement, emotional arousal, formality or informality, similarity of dissimilarity, inclusion or exclusion, task or social orientation and so on. The context may dictate which messages may be salient and have a potential connection to trust. The key is to understand that these implicit messages that are exchanged are part of the interaction spiral that forms trust or distrust. It is the communication exchanges and the resultant relational message interpretations that become the psychological template of trust.

Future research could explore expressivity from other kinesic indicators in the trunk, limbs and hands. Which behaviors generate a sense of energy and engagement that promotes trust? In contrast, the behavioral opposites of inexpressiveness and rigidity may generate suspicion. The suspicion and distrust aspect of developing trust is understudied yet quite important to probing relationships in adversarial relationships and relationships in which trust is eroding. Facial impassivity and wooden postures can be a potent clue that someone is being deceptive. Onlookers may develop early suspicions from such behavior apart from anything that is said. People in intimate personal relationships may begin to develop distrust of their partner from such nonverbal behaviors before other actions begin to undermine trust.

An interesting result of our work is that culture did not appear to be a major driver of trust. While we found a few differences in trust patterns across the six countries, and that the cultural orientation of vertical individualism negatively correlated with trust, on balance the results generally support the universal cues hypothesis more than the specific discrimination perspective. These results are in line with some other recent studies of culture and deception (e.g., [17]).

The current investigation, beyond contributing to an understanding of the psychology of trust, presents a number of methodological advances that have, or could be made. Many insights come from the dissertation of Walls [29] in using artificial intelligence to transform behavior into actionable insights. Walls observes that before significant accuracy can be achieved in creating a set of classifiers for predicting trust, measurement decisions must be made about the length and duration of units of observation to be used. Whether analysis is at the level of individual turns at talk, interact exchanges between pairs or clique groups, phases of decision-making, or entire rounds of a game can alter estimates of accuracy. Also, eliminating periods of silence in videotaped recordings and narrowing judgments to meaningful segments such as voting periods can also alter and improve predictive accuracy. The parameters of the classification models can be learned in a data-driven approach using machine learning techniques.

This research successfully demonstrates that automatic action unit extractions and feature creation for facial analysis, combined with the latest in computer vision techniques, represent an unbiased analysis of videos that brings an understanding of trust. Model building is accomplished using feature creation algorithms, machine learning techniques, and analysis. This project has demonstrated the utility of this approach by using the same analysis to predict team selection, leader elections and game wins. That is, the design has replications built into it. Such within-subject designs have the benefit of controlling other sources of “noise” such as personality or gender because those sources of variance remain constant across the replications. Thus, the system design is general in that it can discover connections to any set of cues.

Another innovation of the research reported here is the use of the attention mechanism to locate sparsely exhibited behaviors and identify the key frames in the video that may be especially consequential in understanding trust. Just as interpersonal relationships have trajectories that change at turning points, key frames may signal those important turning points that signpost the positive or negative development of trust. For example, disclosure of highly intimate details about one’s past may encourage reciprocal disclosure from the partner and escalate the relationship’s mutual trust to the next level. In videos consisting of thousands of frames, the ability to locate the points at which turning points occur can be very useful.

The use of the attention mechanism can alleviate the need to label every timestep of importance in the video. We store only meta-data for each video, such as the subject’s role, and use the attention mechanism to identify the important time-steps in the video. Those vary depending on the task that the model is trained for. In that way, we can perform retrospective analysis of those frames, while keeping the data collection protocol simple.

A further next step in research would be to use the key frames identified by the attention mechanism to re-train the model. Re-training the model can be useful for large input videos, since the models can ignore the majority of the input frames, which are irrelevant for the modeling task.

Future collaborations with others investigating behavioral networks of linguistics, vocalics, and kinesics is sure to bring new discriminating features to the machine learning techniques. This is easy to do with the current developed methods since they can be used with different modalities. Only one step to transform every modality to a canonical input form is required. Obtaining new features can be accomplished by having new experimental participants watch the game videos and record their perceptions of trust.

The psychology of trust is a rich construct for investigation. Examination of how it develops through implicit relational messages promises to bring greater understanding of the construct. New software and automated computer vision tools can accelerate and amplify the progress of those investigations.

Acknowledgments

This research was sponsored by the Army Research Office and was accomplished under Grant Number W911NF-16-1-0342. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the Army Research Office or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation herein.

Conflict of interest

Judee Burgoon and Jay Nunamaker are principals in Discern Science International, a for-profit entity conducting research and systems development for credibility assessment. The remaining authors declare no conflict of interest.

Notes/thanks/other declarations

The authors wish to thank Bradley Dorn, Rebecca (Xinran) Wang, Xunyu Chen, Steven Pentland, Lee Spitzley, Tina Ge, Matt Giles, Mohammed Hansia, Chris Otmar, Yibei Chen, Becky Ford, Ligong Han, and Lezi Wang for their contributions to conducting this research.