Sentiment Analysis of Social Media Using Artificial Intelligence

Abstract

Social media refers to the development and sharing of sentiment, information, and interests, as well as other forms of opinion via virtual communities and networks. Nowadays, social networking and micro blogging websites are considered reliable sources of information since users may openly express their opinions in these forums. An investigation of the sentiment on social media could assist decision-makers in learning how consumers feel about their services, products, or policies. Extracting emotion from social media messages is a difficult task due to the difficulty of Natural Language Processing (NLP). These messages frequently use a combination of graphics, emoticons, text, etc. to convey the sentiment or opinion of the general people. These claims, known as eWOM (Electronic Word of Mouth), are quite common in public forums where people may express their opinions. A classification issue arises when categorizing the sentiment of eWOM as positive, negative, or neutral. We could not use standard NLP tools to examine social media sentiment. In this chapter, we will study the role of Artificial Intelligence in identifying the sentiment polarity of social media. We will apply ML(Machine Learning) methods to resolve this classification issue without diving into the difficulty of eWOM parsing.

1. Introduction

Sentiment polarity has a context-sensitive meaning in sentiment analysis. Based on the sum of the positive and negative opinions stated about an event, automatic methods calculate the sentiment polarity. Generally, daily sentiment scores are often calculated by measuring the number of positive as well as negative words in a sentence. A sentence with more negative words (reflecting violence, anger, sadness) than positive (displaying happiness, celebration, joy) is deemed negative.

Definition: Sentiment polarity of textual data is the result of analysis expressed in terms of a numerical value obtained by the algebraic sum of opinion contained in each entity of the sentence, document, or message. The sentiment polarity can be determined as positive, negative, or neutral.

Sentiment polarity score ($S_p$) of a message is the linear combination of all polarities. This is in turn converted as a ratio to the sum to get a rational value between −1 and 1. Hence,

$$S_p=\left(W_p‐W_n\right)/\left(W_p+W_n\right)$$E1

where $W_p$ denotes the number of positive words in the and $W_n$ represents a number of negative words in the message.

Several decision-makers, including business organizations and governmental authorities, may learn more about public opinion by using social media analytics. Short messages are used by users to share their opinions on social networking platforms. Sentiment analysis of social media evaluates emotions and opinions [1]. These social media messages reflecting the opinion, sentiment, and emotion of the public via a combination of text, images, and emoticons are sometimes referred to as eWOM.

Definition: eWOM refers to messages that are written to be exposed via internet-mediated online communication, especially towards a brand, product, or organization.

The field of social media analysis has grown up fast in this decade to answer the question ‘What do people feel about a certain event or topic?’. Analyzing the sentiments, opinions, and emotions of people has its importance. For instance, we could assess a community’s well-being; we can stop suicides, etc. Additionally, by examining client feedback, businesses may gain a great deal of insight into the level of consumer satisfaction. However, the non-standard format of these messages makes it challenging to understand the language of eWOM. For example, the message ‘Enjoying my lazy Sunday

!!’ is a mix of words and emoji representing positive sentiment about Sunday. An eWOM like this is difficult for parsing because it contains special and emoticons symbols. The context of the sentence could not be interpreted unless parsers are also familiar with the meaning of the emoji

. Sentiment analysis would not be correct until these messages are translated into plain text while retaining their context and emotions.

Social media users always express their sentiments over a product or a public event. A user-generated message containing sentiment can be defined as a quadruple:

$$\mathrm{u}=\left(\mathrm{o},\mathrm{f},\mathrm{s},\mathrm{h}\right)$$E2

here

o indicates a target object.

f represents a feature of the object.

s denotes the sentiment value of the opinion (+ve, −ve, or neutral).

h signifies opinion holder.

For example, the following is a user review of mobile phones.

Although the battery life of my new phone is not long, that is ok for me.

When we parse this message, we get the quadruple as follows:

o new phone.

f battery life.

s not long (-ve sentiment).

h ok for me.

Sentiment analysis involves major sub-tasks namely pre-processing, transformation, and classification. Though pre-processing and transformation can be done using text manipulation, classification is a complex task. It involves recognizing the sentiment along with the context. The system should interpret the meaning and analyze the sentiment similar to human intelligence. A systematic methodology is needed as shown below.

• Cleansing and extracting textual content from eWOM.

• Identifying the sentiment polarity of the extracted text.

Artificial Intelligence (AI) is a powerful source for the crowd’s wisdom to filter out non-textual information and our study focuses on how artificial intelligence can effectively be used to identify the sentiment of eWOM.

2. Architecture of social sentiment analysis system

After extracting plain text from social media messages, we can use computer algorithms to automatically classify the sentiment polarity. The following two categories might be used to broadly classify sentiment analysis algorithms:

1. Lexicon based: This method uses a pre-built repository of emotional words to match the message. A knowledge base having textual units annotated with sentiment labels is called an emotion lexicon. They depend on lexical resources such as lexicon, word banks, or ontology.

2. AI-based: AI-based approaches use ML methods to identify the sentiment. The machine learning approach uses algorithms that may learn from data by making use of document similarity between text messages.

The subjectivity, polarity, or object of any opinion is frequently determined by a lexicon-based system using a set of criteria created by people. These rules may take into account a variety of NLP methods developed in computational linguistics, like part-of-speech tagging, tokenization, stemming, and parsing.

Contrary to lexicon-based systems, AI-based methods are on the basis of ML algorithms instead of hand-crafted rules. Typically, the process of identifying the polarity of sentiment is defined as a classification problem, in which a classifier is provided a text and outputs a polarity label, like positive, negative, or neutral.

Lexicon-based systems do not perform well due to the non-standard language of social media users. Results are more accurate, which is one major advantage of AI-based systems. They resemble a human scoring system while classifying the sentiment polarity by taking the contextual information into account. Machine Learning algorithms are widely used to solve complex real-world problems. It is a promising strategy that has been widely used in AI disciplines including NLP, semantic parsing, transfer learning, computer vision, and many others. Social media sentiment analysis is a complicated task because of the following difficulties:

• People do not have a formal writing style.

• People use colloquial and personal elements in their language.

• People use a mixture of text, emojis, abbreviations, images, and symbols.

• Hashtags are also often used to highlight their importance.

• Millions of messages are generated every second.

Automatic systems try to extract the sentiment polarity using computer algorithms and techniques. The colloquial language makes it hard for automatic systems to interpret the context and sentiment being expressed. The sentiment analysis methodology we show here employs machine learning methods to classify sentiment at the sentence level and acts directly at that level (Figure 1).

Figure 1.

Architecture of sentiment analysis system.

The first step is to translate the eWOM into plain English text using pre-defined mapping for symbols and emojis. The text is then converted to a feature vector representing the features. The Ml method creates a model from pairs of feature vectors and labels (such as positive and negative), which are input into the process. The well-trained model predicts sentiment polarity for new incoming input.

3. Feature engineering of messages

Social media messages are not well-formed and unstructured. The system for analyzing sentiment needs to be tailored to handle the style and specifics of this informal writing style. For example, the message “Enjoying my lazy Sunday

” signifies a positive sentiment. It comprises one word and one emoji representing happiness. Feature engineering is the method of selecting and transforming unprocessed data into features that could be utilized in ML. The inclusion of symbols and emojis plays a significant role in detecting hidden sentiments. The first step in processing the eWOM is feature engineering. We need to derive meaningful sentences from eWOM so that the message is informative, non-redundant and supports the next learning and generalization steps [2]. Feature engineering (FE) works as follows.

• Emojis are replaced with Common Locale Data Repository meaning if a mapping is available. They are eliminated otherwise.

• Punctuation such as brackets, periods, and commas are removed.

• Stop words (a, an, and, etc.,) are removed.

• Words with special letters and digits are eliminated, as are words that don’t only include alphabetical characters.

• Lowercase letters are used for all words.

Translation of emojis is a crucial part of the feature selection. Popular emojis used in sentiment analysis can be translated to plain text using their corresponding Unicode Common Locale Data Repository (CLDR) meaning. The Unicode chart offers a list of the emoji character, codes, and meanings. For example,

has the code U+1F604 and means smiling face. Similarly,

has the code U+0270C which means victory hand. A complete list of Unicode for emojis is available at “http://unicode.org/emoji/charts/full-emoji-list.html” for reference. Translation of emojis helps us to capture the sentiment without losing the original context. The following diagram indicates the steps involved in the feature engineering of eWOM (Figure 2 and Table 1).

Figure 2.

Feature engineering of eWOM.

Table 1.

Sentences extracted from eWOM.

The converted plain text may not be a meaningful English sentence but it captures the original sentiment and context. This could be applied as an input vector to ML algorithms. Following are examples of a few plain texts extracted from eWOM.

The extracted plain text is given as input to an automatic system that uses a machine learning algorithm for classification. The automatic system is defined as follows.

Sentiment polarity classification system M is a quadruple

$$\mathrm{M}=\left\{\mathrm{\alpha },\mathrm{\lambda },\mathrm{\delta },\mathrm{\phi }\right\}$$E3

where

$\mathrm{\alpha }=\left\{e_1,e_2,\dots e_n\right\}$ is a set of messages from social media,

$\mathrm{\lambda }=\mid e_i-e_j\mid$ is a function that calculates the similarity score of messages,

$\mathrm{\delta }\to$an algorithm to classify the sentiment polarity, and

$\mathrm{\phi }=\left\{S_p,S_n,S_o\right\}$ is a set of output labels: positive, negative, and neutral.

For a given input in α, the algorithm δ produces an output label in φ with the help of the function the following diagram shows the overview of the sentiment classification machine (Figure 3).

Figure 3.

Sentiment polarity classification system.

The sentiment polarity of social media messages can be efficiently predicted by an artificial intelligence system if we identify the right choice for

1. Function to calculate the similarity score of two messages.

2. A machine learning algorithm for classification.

We will discuss the selection of algorithms for these two tasks in detail in the following sections.

4. Similarity score of messages

Machine learning algorithms mainly work on numerical input vectors. It is essential to convert the pre-processed text to a numerical vector for algorithms to predict the sentiment polarity correctly. The ML model is trained with labeled data sets for positive, negative, and neutral messages. A well-trained model predicts the sentiment polarity of incoming messages by comparing it with members of the training data set. Comparing two messages and giving a numerical score on how similar they are being important for high accuracy. TF-IDF (“Term Frequency-Inverse Document Frequency”) is a vectorization approach used widely in text processing. This approach examines the relative frequency of terms in a text via an inverse percentage of the phrase throughout the full corpus of documents [3, 4]. It works well for text categorization or making it possible for machines to interpret input words represented as numbers. In TF-IDF, the same texts must result in a closer vector. TF-IDF is the multiplicative product of the Term Frequency and Inverse Document Frequency scores of the word.

TF represents the “number of times the word appears in the doc / Total number of words in the document.

$$\mathrm{IDF}=ln\left(\text{Number of documents}/\text{Number of documents the word appears in}\right).$$

$$\mathrm{TF}‐\mathrm{IDF}={\mathrm{TF}}^{\ast }\mathrm{IDF}$$E4

The phrase is rarer and vice versa, depending on the TF-IDF score. A smaller score between the documents indicates that they are highly similar to each other. A score of 0 indicates that the papers are entirely equal. Following is an illustration of TF-IDF score calculation using two simple sentences.

1. Document 1: It is going to rain today.

2. Document 2: Today I am not going outside.

Step 1: Tokenize the words and count their frequency (Table 2).

Word	Count
going	2
today	2
rain	1
outside	1
i	1
am	1
is	1
it	1
to	1
not	1

Table 2.

Word count.

Step 2: Find Term Frequency “(TF).

$$\mathrm{TF}=\text{(Number of times the word appears in the document)}/\left(\text{Total}\mathrm{no}.\text{of words in}\mathrm{a}\text{document}\right)$$E5

(See Table 3).

Word	Document 1	Document 2
going	0.17	0.17
today	0.17	0.17
rain	0.17	0
outside	0	0.17
i	0	0.17
am	0	0.17
is	0.17	0
it	0.17	0
to	0.17	0
not	0	0.17

Table 3.

Term frequency.

Step 3: Find IDF for documents.

$$\mathrm{IDF}=\mathrm{Log}\left[\left(\text{Number of documents}\right)/\left(\text{Number of documents containing the word}\right)\right]$$E6

(See Table 4).

Word	IDF value	IDF (in decimal)
going	${log}_2\left(2/2\right)$	0
today	${log}_2\left(2/2\right)$	0
rain	${log}_2\left(2/1\right)$	1
outside	${log}_2\left(2/1\right)$	1
i	${log}_2\left(2/1\right)$	1
am	${log}_2\left(2/1\right)$	1
is	${log}_2\left(2/1\right)$	1
it	${log}_2\left(2/1\right)$	1
to	${log}_2\left(2/1\right)$	1
not	${log}_2\left(2/1\right)$	1

Table 4.

Inverse document frequency.

Step 4: Calculate TF-IDF for each word.

$$\mathrm{TF}‐\mathrm{IDF}={\mathrm{TF}}^{\ast }\mathrm{IDF}$$E7

(See Table 5).

Word	Document 1	Document 2
going	0	0
today	0	0
rain	0.17	0
outside	0	0.17
i	0	0.17
am	0	0.17
is	0.17	0
it	0.17	0
to	0.17	0
not	0	0.17

Table 5.

TF-IDF for two documents.

To measure the similarity between these 2 documents, we need a distance metric. Hamming distance is suitable to measure the distance between column vectors.

Hamming distance is used usually with boolean or string vectors, detecting the points where the vectors do not match. While comparing two vectors of equal length, it is the number of positions in which the values are different. It is also known as the overlap metric.

Now let’s use hamming distance to measure the distance between the documents (Table 6).

Word	Document 1	Document 2
going	0	0
today	0	0
rain	0.17	0
outside	0	0.17
i	0	0.17
am	0	0.17
is	0.17	0
it	0.17	0
to	0.17	0
not	0	0.17

Table 6.

Hamming distance between Document 1 and 2.

$$\mathrm{\lambda }\left(\mathrm{doc}\mathit{1},\mathrm{doc}\mathit{2}\right)=8$$E8

We observe that these two documents have only two words in common. Hamming distance of TF-IDF is eight. The similarity score generated by our algorithm is a good metric to measure the distance between documents. Two similar documents will have a score of zero. A non-zero score indicates their distance. Having converted the input to a numerical vector and identified a distance metric, we can now use a machine-learning algorithm for sentiment polarity classification.

5. Sentiment classification using machine learning

An identification task for sentiment polarity is usually defined as a classification problem, where a classifier is fed a text and outputs a label, like “positive, negative, or neutral”. In general, supervised and unsupervised learning techniques may be used to classify text using a machine learning methodology. Many tagged documents are used in the supervised approaches. Unsupervised approaches are employed when it is challenging to find these labeled training documents. Since thousands of messages of emitted every minute on social media, we can manually label a few thousand and use them for supervised learning. A training data set consisting of three labels namely, positive, negative, and neutral is prepared by picking messages from social media sites like Twitter, Facebook, etc. Only the training documents that are most similar to the incoming document are used by the machine learning algorithm to label it. Popular text classification algorithms include K-Nearest Neighbor (KNN). With the help of supervised learning, this method divides objects into one of the predetermined categories of a sample group. Here we will see how KNN can be used to classify sentiment polarity. If we represent the numerical vectors as points in a diagram, then the training data set will look like this (Figure 4).

Figure 4.

KNN classification with k = 9.

It is trained with a set of labeled data sets. The label of incoming input is predicted based on the majority voting of its neighbors. The algorithm can be described below.

Algorithm: KNN classification

1. Calculate the distance from the query data item to the labeled data items.

2. Order the labeled samples by increasing distance.

3. Using accuracy, determine the k nearest neighbours that are the heuristically optimal number. Cross-validation is used to accomplish this.

4. Assign a class label to the query example based on majority voting (neighbors).

5. A confusion matrix may be used as a tool to check the KNN classification algorithm’s accuracy.

We have three clusters for the labels namely positive, negative, and neutral. During training, labeled data items form the clusters based on the distance from their neighbors. A new data point is assigned a label by picking up nine neighbors and their majority voting.

5.1 Popular algorithms for classification

We have seen how KNN supervised learning method could be applied used for sentiment polarity classification. However, there are few other machine learning algorithms. Researchers can experiment with the following algorithms and choose the right choice based on performance.

XG boost: Extreme Gradient Boosting is referred to as XGBoost. The GBDT (“Gradient-Boosted Decision Tree”) ML approach is scalable and distributed. It offers parallel tree boosting and is the most effective ML technique for classification and regression issues.

Support vector machines (SVM): SVM is a supervised method in which the learning method examines the data and finds patterns. We display the data as points in an “n-dimensional” space. The value of every attribute is then connected to a specific coordinate, which facilitates categorization.

Naive bayes(NB): NB is depending on Bayes' Theorem, an approach for determining conditional probability on the basis of previous information and the naive belief that each attribute is independent of the others. The greatest advantage of Naive Bayes is that it works quite well even with small amounts of training data, whereas the majority of ML algorithms rely on huge amounts of training data.

Convolutional neural networks (CNN): Deep learning is gaining popularity due to enhanced chip processing capabilities (GPU units), much cheaper hardware costs, and major advancements in ML methods. Deep neural network design was first used by researchers to assess document similarity. A series of word embeddings constructed from data sets used as inputs to train a CNN-based representation learning model may also be utilized to classify the sentiment polarity.

5.2 Popular performance metrics for classification

The performance of a machine learning algorithm needs to be evaluated before selecting the model for real-world applications. Our sentiment analysis is a classification problem. The results of any classification algorithm can be evaluated by creating a confusion matrix and popular metrics. The following table shows the confusion matrix for a “binary classification” model (Table 7).

	Predicted values
Actual values	True Positive; (TP)	False Negative; (FN)
	False Positive; (FP)	True Negative; (TN)

Table 7.

Confusion matrix for classification model.

Based on the elements of the confusion matrix, a set of metrics is generally calculated for assessing the performance of the classification model.

5.2.1 Accuracy and error rate

A classification model’s quality may be assessed using these key metrics. A “true positive, a false positive, a true negative, and a false negative”, respectively, are denoted as TP, FP, TN, and FN. Following are definitions for the terms Accuracy and Error Rate in classification.

$$\text{Accuracy}=\frac{\left(\mathrm{TP}+\mathrm{TN}\right)}{\mathrm{N}},\text{Error Rate}=\frac{\left(\mathrm{FP}+\mathrm{FN}\right)}{\mathrm{N}}$$E9

where N indicates the total number of samples. Clearly, we have Error Rate = 1 Accuracy.

5.2.2 Recall, F1 score, and precision

These are also the main metrics for unbalanced test sets, and they are applied more frequently than error rate or accuracy. For binary classification, precision, as well as recall, are specified below. The harmonic mean of recall and accuracy is the F1 score. The F1 score is best when it is 1 (perfect recall and precision), and it is worse when it is 0.

$$\text{Precision}=\frac{\mathrm{TP}}{\left(\mathrm{TP}+\mathrm{FP}\right)},\text{Recall}=\frac{\mathrm{TP}}{\left(\mathrm{TP}+\mathrm{FN}\right)},\mathrm{F}1‐\text{Score}=\frac{2^{\ast }{\text{Prec}}^{\ast }\mathrm{Rec}}{\left(\text{Prec}+\mathrm{Rec}\right)}$$E10

We may always calculate recall and precision for every class label in multi-class classification problems, assess each class label’s performance individually, or simply average the numbers to obtain the overall recall and precision. The average for the three classifications positive, negative, and neutral in our situation may be determined.

6. Applications of social media sentiment analysis

Traditional polling may be replaced by AI-based social media analysis, which is also a more affordable way for decision-makers to comprehend the situation and address any emerging crises. Social media is used by the public proactively to express their sentiment and opinion. People post millions of messages every minute on social media. If these messages are analyzed and opinion is extracted, it will help decision-makers to quickly respond to any crisis. Following are examples of sentiments captured by an automatic system from Twitter (Table 8).

Table 8.

Sentiment classification by automatic system.

Results are convincing in the possibility to replace manual sentiment classification with automatic systems. The experimental results show promising output and can be used by online marketing companies, and government agencies for decision-making [5]. Online advertisement agencies can use this study for effectively targeted marketing campaigns. On the other hand, Government organizations could know how the public is affected by a policy or decision and then decide how to respond to public opinion. It has many applications in the real world. The above diagram shows the areas where social media analytics is finding its applications (Figure 5).

Figure 5.

Real world applications of social media analytics.

It allows business organizations to determine how consumers feel about their brands and products, identify how they feel about advertising efforts, and generally track which way the wind is blowing. Social media analytics helps commercial organizations in the following areas:

• Derive customer sentiment towards products and services.

• Understand conversations - what is being told and how it is being received.

• Spot trends related to brands and service offerings.

• Find the high-value components of a service or product.

• Assess the reaction to messaging on social media and other channels.

• Find out what the competition is saying and its efficacy.

• Learn how third-party partners and channels may impact performance.

7. Conclusion

The AI strategy produced greater results when it came to categorizing eWOMs' sentiments based on polarity. The two-step process, namely Feature Extractor and Machine Learning, eliminates the main difficulty in employing NLP tools to comprehend social media communications. Commercial enterprises may increase accuracy and acquire greater insights when assessing customer comments and complaints by employing a centralized sentiment analysis system. The following are some general advantages of AI-based sentiment analysis:

1. Sorting data at scale: It is difficult and time-consuming to manually review thousands of tweets, customer service discussions, or survey responses. AI-based sentiment analysis enables businesses to analyze massive amounts of data economically and efficiently.

2. Real-time analysis: Organizations may immediately detect dangerous circumstances on a real-time basis with the use of social media analysis and take action before consumers start to leave. Text sentiment labeling is highly subjective and is affected by personal experiences, viewpoints, and opinions. Words such as extremely, quiet, most, etc. are examples of intensifiers. These are the terms that affect how the adjacent non-neutral terms feel. They may be broken down into 2 categories: those that raise the intensity of feeling (very, very much) and those that tone it down (little). Through a rule-based method, determining the strength of an emotion might not be straightforward. The AI-based model may still be improved to determine the level of emotion intensity.