Anomaly Detection in IoT: Recent Advances, AI and ML Perspectives and Applications

Abstract

IoT comprises sensors and other small devices interconnected locally and via the Internet. Typical IoT devices collect data from the environment through sensors, analyze it and act back on the physical world through actuators. We can find them integrated into home appliances, Healthcare, Control systems, and wearables. This chapter presents a variety of applications where IoT devices are used for anomaly detection and correction. We review recent advancements in Machine/Deep Learning Models and Techniques for Anomaly Detection in IoT networks. We describe significant in-depth applications in various domains, Anomaly Detection for IoT Time-Series Data, Cybersecurity, Healthcare, Smart city, and more. The number of connected devices is increasing daily; by 2025, there will be approximately 85 billion IoT devices, spreading everywhere in Manufacturing (40%), Medical (30%), Retail, and Security (20%). This significant shift toward the Internet of Things (IoT) has created opportunities for future IoT applications. The chapter examines the security issues of IoT standards, protocols, and practical operations and identifies the hazards associated with the existing IoT model. It analyzes new security protocols and solutions to moderate these challenges. This chapter’s outcome can benefit the research community by encapsulating the Information related to IoT and proposing innovative solutions.

1. Introduction

The wide variety of IoT devices lacking any standard creates connectivity issues and increases the security vulnerability of IoT local networks and the entire Internet. Machine Learning techniques are already used in ECG, X-ray, pattern recognition, cancer detection, brain signal modeling, and IoT services on electrical impedance planes to discover defects. Extending ML and DL technologies to detect anomalies where it is already operating is a natural and effective transition. Anomalies are events or patterns that deviate significantly from predictable behavior. Detection methods are expected to identify anomaly occurrences and their probable cause promptly. To comply with this chapter topic, we focus on these applications incorporating Machine Learning and Deep learning methods. Chatterjee & Ahmed [1] provide a comprehensive survey on Anomaly Detection in IoT and propose four measurements for evaluating IoT Anomaly Detection methods: how they approach the problem, how they are applied, the type of method, and the algorithm latency. Anomaly detection using deep learning is described by Chalapathy and Chawla [2], and Yassine et al. [3] provide a review of the methodologies, situations, and computation platforms used for anomaly detection in the energy industry. Talagala et al. [4] propose a distributional unsupervised for anomaly detection in high-dimensional data. Yin et al. [5] extract unique temporal features from a given temporal data file using a combination of CNN and LSTM and continue in [5] to detect anomalies involving CNN, LSTM, and Deep neural network (DNN).

They [1] also define 18 application types of anomaly detection processes. The following are examples of various application types. Sobhani et al. [6] demonstrate that the accuracy of final load projections is improved when eliminating observations from the original input using local load information. T. Asakura et al. [7] detect damage to industrial rotating equipment by calculating the feature vectors of the anomaly vibration data extracted from sensors’ vibration signal features to construct a monitoring system for machinery equipment. Huang et al. [8] proposed anomalies detection in Manufacturing using density peak weighted fuzzy C-means (WFCM). Yasaei et al. [9] detect unexpected event changes in sensor signals using an adaptive data-driven monitoring method. Zekry et al. [10] use a convolutional LSTM model for anomaly detection in the context of connected vehicles. Wang et al. [11] log anomalies in IoT systems using a natural language processing approach, extracting the relevance between words and vectoring them. The method trains supervised models to detect anomalies reducing computational time. Xu et al. [12] used I-LSTM and Deep learning in smart-city data for multi-classification anomaly detection to improve smart homes’ service quality. Tripathi et al. [13] proposed reliable and transparent city connectivity using IoT, MEC, and Blockchain consensus. Ullah et al. [14] presented a timely identification of abnormal incidents in surveillance networks, incorporating LSTM with CNN, where CNN features are collected from successive frames. LSTM is used to distinguish between normal and abnormal values. The in-depth features and multi-layer BD-LSTM provide high-level training and validation data to real-world IoT surveillance networks. The DeL-IoT framework [15] detects IoT abnormalities by observing flow-level traffic instances that pass through switches. The IoT anomaly identification and prediction framework uses a Deep Learning technique to identify anomalies. Mirsky et al. [16] proposed a Blockchain-based distributed anomaly detection algorithm using the Markov chain (MC) to simulate sequences efficiently. Y. An et al. [17] proposed anomaly detection capable of relieving network congestion and CPUs from the computing pressures of centralized servers, unlocking the potential of edge intelligence in IoT. Shen et al. [18] propose a privacy-preserving SVM training strategy using encrypted IoT data. Data providers encrypt their data locally using their private keys and then record the encrypted data on the Blockchain.

The rest of the chapter comprises as follows: The next section outlines security issues unique to the IoT environment. Section 3 presents a generic two-stage Anomaly Detection approach. In the first stage, a process builds the envelope around the weighted average, and the comparison is done in the second stage. In Section 4, Anomaly Detection using Random Forest Machine Learning is presented, and it concludes in Section 5.

2. IoT security issues

We see a considerable rise in the use of IoT applications in our day-to-day lives. The IoT enhances web-based applications by enabling connections via the Internet between people and their equipment/devices in a real-world or virtual environment. IoT improves Web-enabled applications by allowing links between “everyone” and “everything” in a real-world and virtual environment [19]. Utilizing IoT applications and services is now easier than ever because of the exponential expansion of smart devices. As the asset value of the data kept, processed, and conveyed increases along with scale, so do the attacks against them. These predictions show that there will be a rise in the number and level of threats and attacks against IoT devices, necessitating more robust security measures. This section aims to investigate recent IoT cybersecurity solutions.

Artificial intelligence, Machine/Deep Learning, and networking have become the current area of IoT-related research. Adopting ultra-lightweight protocols for security and core functionality is a significant development in the IoT.

IoT security is constantly evolving, with new risks always being found. The focus of IoT security discussions is ACL techniques, interim encryption techniques, hardware-specific security solutions, and SQL-related attack measures. Identifying IoT-related cybersecurity risks, providing classifications, and looking for prevailing solutions to address them. The following questions are addressed:

1. What architecture can be used considering various criteria?

2. What are the IoT standards and protocols currently in use?

3. What are the IoT cyber security threats?

2.1 Literature review

The recent industrial trends include embedded networking in the wireless segment, where IoT is the major player. The demand for smart applications and systems grew, leading to the rise of IoT in commercial segments [20]. Due to the immense increase in the retail segments, the usage of smart applications has spiked up, increasing their dependability, which further leads to high risks. IoT devices have emerged as the spot for intrusion activities because of the lightweight protocols and standards that are currently present on these devices [21, 22], and the entities that make up these devices have easier access to servers [23] because the security is not fully resolved. The problem with the traditional model is the lack of low-powered device algorithms and the incompatibility of security tools due to differences in policy and implementation methods [24]. A variety of hardware-based techniques and unique solutions have been suggested in recent research to address traditional security challenges.

Xin Zhang and Fengtong Wen [25] proposed an authentication for IoT, where two algorithmic models have been built to ensure valid authentication. The scope of the security solution offered in this work is constrained to protect only lightweight sensor devices from the standard network layer and physical layer-based attacks. M. Dahman Alshehri and Farookh K. Hussain [26] proposed a cluster-based fuzzy architecture and a secured communications model for IoT nodes. This study effectively provides a detection technique against the network’s malicious nodes but does not cater to the threats posed by the audit attack surface. This study does not adequately address the performance analysis of operational communication and computing costs. Chen et al. [27] offered a unique Low scale Denial of-Service attack detection approach that incorporates trust evaluation with Hilbert-Huang Transformation in Zigbee WSN to address the security risks considering a large number of devices with low energy which is susceptible to attacks. This work’s signal and anomaly detection technique helps reduce the attack level. It has an extensible design because it supports cloud and edge computing, but higher storage overheads persist as a problem. In traditional network security, IDS is entrusted with identifying and keeping track of threat behaviors. Hence, such models do not expressly target the IoT environment.

2.2 Security architecture and communication

This section discusses the IoT security architecture. Use-cases for IoT range from single node devices to cross platform deployments of technology and real-time cloud systems [28]. IoT operations consist of three main tasks: transmitting, retrieving, and data processing. Application Layer: Embedded interface modules enable devices to communicate with the underlying architecture. The device Management Plane identifies the data’s source and destination to maintain the device’s input–output operations. For instance, the Aggregator aggregates the given device data assets into a fixed set. A communication Layer is an intermediary layer with network components that establish various communication protocols and standards. This layer comprises stacks of current protocols and criteria for controlling traffic throughout the system. Standard protocols enable proper communication among IoT devices. Such systems need a defined set of simple rules to initialize and share data information. Figure 1 depicts the multi-layer architecture of IoT.

Figure 1.

IoT multi-layer architecture.

The IoT’s communication protocols include:

1. Z-wave – The protocol facilitates device communication in a closed network. It implies that the Z-wave regulating code is not publicly accessible. It prohibits anyone from changing the code and suggests that each Z-wave device has a unique ID granting access to all remote controls. This architecture ensures effective interoperability and security, the Z-wave protocol’s core.

2. BLE –Bluetooth low energy is a widely used protocol. Due to its propensity to consume less energy, it works well with low-energy devices. Based on Generic Attributes, this protocol uses Services and Characteristics to carry out its operations.

3. MQ Telemetry Transport (MQTT) is a protocol for small Internet of Things (IoT) devices that allow data transmission and some reception between the sensor nodes.

4. Advanced Message Queueing Protocol (AMQP): includes efficiency, portability, multichannel support, and security, a TCP-dependent binary protocol that ensures authentication using SASL or TLS.

5. Limited Application Protocol (CoAP) is a protocol for constrained-based environments. Significant traits of this protocol include its REST API-based foundation, design for system applications, effective congestion control, cross-protocol interoperability, and many others.

6. The Data Distribution Service protocol is an Internet of Things protocol for M2M communications. Like the MQTT and CoAP protocols, data can be exchanged using the publish-subscribe approach; the significant distinction is that this architecture does not require a broker, unlike the latter two. DDS employs multicasting to provide apps with high QoS.

7. 6LoWPAN is the 6th version of the Low-power Wireless Personal Area Network. It is a standard protocol for implementing IPv6 on wireless networks comprising low-power wireless modules.

8. DTLS: Datagram Transport Layer Security is a security protocol for the Internet of Things and is intended to safeguard data transmission between apps that use datagrams. It offers the same level of security and is majorly focused on the Transport layer security protocol.

Heterogeneous physical components such as switches, actuators, gateways, sensor nodes, and other embedded devices make up this unstable environment. A significant impact on networking principles is made by the intelligent device’s engineering process, which is the backbone of the whole concept. Gadgets with self-configuring capabilities of the M2M communication paradigm are IoT innovations. Through algorithms and auxiliary technology, this configuration gives nodes the intelligence they need to make decisions for themselves under any circumstance [29, 30]. It is helpful during rescue operations and other emergencies where configuring the network for a specific area is complex, and there is no support for damaged nodes. However, as machines are not failsafe, it becomes susceptible if it depends too heavily on them. Particularly in the present, adversaries use weak authentication, unpatched firmware, and online credential vulnerabilities [31].

Following are some of the IoT security issues:

1. Heterogeneous devices: the paradigm most sensitive to access requests, detecting third-party indulgence, and limited scalability compliance with security management. Several security issues with IoT today relate to traditional network architecture, including IoT devices that interact with the physical environment differently than conventional network devices did in the past. IoT devices’ heterogeneous nature ramifies other components as they operate. NIST stressed that IoT-specific privacy regulations [32] and cyber controls must consider the consequences that impact physical systems [33], ultimately affecting the physical world.

2. Regulations & Policies: No global IoT security standard applies to all IoT industry segments. Although some regulations are in the process (such as the EU’s General Data Protection Regulation and the US IoT Cybersecurity Improvement Act), they are relatively fragmented and do not address issues with IoT. IoT devices are used globally by many servers, whether they are in a business/in a person’s workspace. Such devices can be monitored/managed using a different rule engine, and the security policy varies depending on the system’s devices. Therefore, regularization requires updating every device, which is time-consuming and challenging for any company. Problems include an un-uniform pace of updating, new switches leaving some devices behind that are not updated, or inadequately configured nodes since it takes time to maintain track of millions of nodes.

3. Additional Plugins and Security: Since providing security measures for IoT was never modeled, further security controls are added to the IoT’s security architecture. Unlike the traditional networking paradigm, the effectiveness of security characteristics relies on the IoT architecture’s ability to function with additional resources. The efficiency of the IoT’s security is also influenced by client behaviors, such as how they choose among the various security solutions.

4. Lack of compliance: The lack of compliance among manufacturers is always a cause for concern. A device should generally satisfy the following requirements: Operational Compliance, Security Compliance, and Manufacturing Compliance.

5. An IoT network may be in danger if operational compliance is not maintained. A city’s power distribution login system could be part of the network. The network on which they operate is at risk due to legacy operating systems which delayed security patches and other issues. Few makers of IoT devices utilize open-source code. When these IoT devices join a network, the entire system’s integrity may be compromised. A lack of security compliance only makes IoT security issues more difficult. Many IoT device producers need to create patchable IoT products.

6. Exposure Threats: IoT endpoints, such as sensors and IP cameras which are in public spaces, are the threat points that are easiest for the enemy to access. As a result, the user’s integrity and authentication are threatened by physical-based and proximity threats [34]. Our changes to the protocol method to safeguard devices from adversaries are the biggest security difficulties in this area.

2.3 Classification of IoT attacks

Several commercial businesses have made significant financial investments to secure their IoT-based networks in recent years. IoT attacks are split into two modules:

2.3.1 Protocol-based attacks

Protocol-based attacks utilize known published protocols to serve their benefits, affecting the communication channel. It is divided into two types:

1. Communications protocols attacks: (a) Attacks on communication protocols—several types of exploitation occur when nodes transition, such as sniffer attacks, flooding attacks, and key preshredding attacks. (b) Network protocol attacks where connection establishment is exploited include Wormhole attacks, selective forward attacks, and sniffing attacks.

2.3.2 Transmitted data attacks

Threats on initial packets and messages moving across communication nodes. Some of its most severely affected security exploitations are data leakage, malicious node VM formation, hash collision, and denial of service. Active and passive attacks compromise the system’s security—the effectiveness of the network is less affected by passive attack protection systems, which are restricted to monitoring techniques. Modern, responsive security techniques are needed to counter active attacks to reduce risk and affect network performance.

1. Distributed Denial of Service attack — DDoS [35] impacts a network security parameter’s availability. Botnets enable DDoS threats on sensor nodes. Affected packets from various sources get access via these points, travel down network data routes, and end up clogging the entire link architecture, making servers unusable.

2. Sniffing attack [36] falls under data collection, a threat vector in which vital system data is collected and used for attacks. With the use of sophisticated tools, information assets are examined. Most devices available on the market need to be sufficiently clever to counteract and are mainly targeted by them.

3. Replay Attack – A replay attack consists of the following steps: “eavesdropping on the communication link between IoT devices or the gateway; intercepting the acknowledgments or connection-establishing components; and deceitfully delaying or rerouting the message.”

4. Masquerade attack [37] – This attack impersonates a valid access identification procedure to get access to target node information. Devices that have poor authorization procedures are highly vulnerable. Such attacks use stolen passwords and user credentials by exploiting program gaps or developing workarounds for the current authentication procedure.

5. Port Scanning - Synchronize requests, target ports, sources, firewalls, packets, open nodes [38], and listening nodes. Synchronize scans are a frequently used technique that creates a partial connection to the target node on the target port by sending a synchronized packet to test the host system’s initial response.

2.4 IoT security solutions

In contrast to traditional security, which is tool-centric, the most recent cybersecurity solutions focus on software-oriented techniques [39, 40]. The security characteristics that current systems address are authentication, trust, and integrity. Even in its current state, the Internet of Things (IoT) cannot support powerful devices and is not adaptable enough to keep up. Table 1 summarizes the IoT protocols, emphasizing their characteristics and security concerns. According to the findings, protocol-based security solutions protect against most IoT attack surfaces [41]. Using secure techniques performed over the Data Link and Transport layers, protocols like COAP and DDS enable efficient immunity against well-known attacks like DDoS attacks and botnet attacks. In Sigfox and EnOcean, new methodologies prevent new threat issues like asynchronous code definition and poor payload encryption. The lightweight protocols MQTT and BLE have also emerged as a viable defense against dangers posed by malicious nodes and Man in Middle attacks. Divided security management is beneficial for more straightforward management of security measures and increases the efficacy of the most suggested solutions.

S.No	Protocol used	Features	Cyber Security issues
1	Z-wave	Z-Wave is a low-power RF technology that can control up to 230 devices at once and builds a wireless mesh network by delivering signals in the sub-1GHz frequency.	An attacker within Z-Wave radio range could control weak devices, deny service, force devices to fail, deplete batteries, intercept, observe, and replay traffic.
		Minimal interference, reliable connectivity, high security through encryption, and fewer disconnections will be the main advantages of using this IoT Data Protocol.
2	BLE	Offers a similar range to traditional Bluetooth.	Susceptible to cyberattacks and interception when sending and receiving data.
		Has a mesh networking structure.
		Designed for low-energy gadgets.
3	MQTT	Power usage is comparatively low.	Internet-based MQTT servers that have been exposed, and malicious third-party MQTT message subscriptions.
		Malicious sinkhole and wormhole attacks against nodes and Distributed Denial of Service (DDoS) assaults.
		Provides a simple protocol for TCP data exchange between machines.
4	AMQP	Deliveries of messages with reliability, messages delivered quickly, and acknowledgments in messages.	Security of message broker is affected.
		Most corporate messaging uses AMQP.
5	CoAP	Designed for the limited network device environment.	In a DDoS attack, a third party simultaneously sends forged IP packets during CoAP reflection and amplification.
		Specialized application for the homogeneous community of restricted devices.
		Consists of a variety of end node devices, constrained small networks over the Internet connection.
6	DDS	Has a communication protocol that varies from machine to machine.	Because of the expandability feature, poorly implemented and managed devices might result in Man in the Middle or DDoS attacks.
		High performance
7	NFC	Make sure the two-way connection is safe; Usage of smartphones as the end nodes.	Malicious wormhole attack based on nodes.
8	SigFox	With low-power consumption, it makes the most of both the cellular and WiFi networks.	Poor payload encryption.
		Supports star network topology and dense node networks.
		Has restricted endpoint access control and cloud access.
9	EnOcean	Self-powered wireless sensor network that is user-driven and gathers data.	Optional blocking, preshared security keys, and undefinable re-synchronization of rolling codes are frequently overlooked.
		Key features include less idle current.
10.	DTLS	A retransmission timer is used by DTLS to address the packet loss problem. The client retransmits the data if the timer expires before it receives the server’s confirmation message.	DDos Attacks.
		By assigning a unique sequence number to each message, the reordering problem is resolved. This aids in assessing whether or not the subsequent message to be received is in sequence. If it is out of order, it is placed in a queue and dealt with when the appropriate number in the sequence is reached.
		DTLS is used in applications where data loss is significantly less essential latency.

Table 1.

Summary list of security protocols for IoT.

2.5 Summary

This section discussed IoT’s current cyber security trends by researching various protocols, standards, and threats. The research findings on the cyber security risks convey that the traditional methods must be more efficient against attacks in heterogeneous IoT environments. Our study further reveals that most cyber security solutions include encryption techniques with low energy use, which also is successful in securing channel attacks in IoT. IoT security increased after integrating with various technologies.

The complications of the IoT system have increased, and security features’ openness has decreased. Even though the previously discussed issues have been attempted to address the evolution of communication technologies and protocols, there is always room for research.

3. Anomaly detection using an optimized envelope

IoT systems collect vast amounts of data to track and analyze the structure of future recorded data. However, this data cannot be stored as is due to limited storage but must be reduced to allow future data analysis based on past data that will not be compromised. We propose a parameterized method of sampling the data optimally. Our approach has three parameters– an averaging process for constructing an average data cycle from past observations, an envelope method for defining an interval around the average data cycle, and an entropy method for comparing new data cycles to the constructed envelope enabling identifying anomalies and predicting future cycle behavior. This section concentrates on finding the optimal envelope using entropy methods.

We often have sequential data collected by sensors, and computational power and bandwidth resources prohibit us from collecting large-scale data. Sampling preserves the most critical information from the original data and reduces the complexity of the subsequent knowledge discovery task to a traceable version without compromising performance. Dictionary learning [42] helps extract patterns hidden in data. We can apply dictionary learning to sequential data for natural language processing, video analysis, and nonsequential data tasks [43]. Given the IoT data collected sequentially, we can find a method that maintains a basis where we have enough elements to describe the sequential patterns of the data. It helps to extract a set of common sequential patterns from the sequential telematics data. In a smart home system, we may collect the most frequent activity trajectories for home members to use for member authentication. We aim to find an optimal sampling method given a set of time-series records, where we collect information before and after the sampling reduction process regarding the data’s purpose in the context of the relevant application. Many known data reduction techniques enable restoring the original data set from the reduced one. Among these are compression and compaction routines and dictionary methods. Given the sequential data, we may apply Classification and Prediction. Classification defines whether a series of daily temperatures represent an El-Niño year or whether the data points to suspected intrusion.

3.1 Related work

Vlachos et al. [44] proposed a procedure for getting the best practical estimated gap between two extreme measurements related to any data sequence. Sakurada and Yairi [45] use auto-encoders with nonlinear dimensionality reduction for the anomaly detection task. Reeves et al. [46] generate domain representations using scaleable layers. Chilimbi and Hirzel [47] implement an iterative scheme that uses temporal data to construct a profile. Then, they identify repeated data sequences with the same order, prefetches them, and let the program continue executing the prefetched instructions. Lane and Brodley [48] use instance-based learning (IBL) for boundary determination by good user behavior and heuristics. Kasiviswanathan et al. [49] detect and cluster user content for optimization. Mairal et al. [42] create a dictionary and adapt it to specific data using data vectors proposing an optimization algorithm for dictionary learning based on stochastic approximations. Aldroubi et al. [50] claim that a collection of subspaces gives the best sparse representation providing an optimized sampling in subspaces union. Rubinstein et al. [51] survey the various options up to the most recent contributions and structures. Cherian et al. [52] propose learning over-complete dictionary models where the signal can have both Gaussian and (sparse) Laplacian noise. Dictionary teaching in this setting leads to a complex nonconvex optimization problem, further exacerbated by large input datasets. Duarte-Carvajalino and Sapiro [53] introduce a framework for the joint design and optimization of the nonparametric dictionary and the sensing matrix. They demonstrate the use of random sensing matrices and those optimized independently of the learning of the dictionary. They complement the classical image datasets, maximizing the size of the sampling data to keep the balance between the sampling data and the information extracted from it. Our problem statement focuses on extracting concepts, methods, rules, and measurements so that, at the end of the process, the original sampling data becomes redundant and need no longer be stored. However, we incorporate an ongoing learning process to keep improving and adjusting the extracted artifacts to natural changes in the sampled mechanism’s behavior. Our study concentrates on time-dependent streaming sampling data divided by fixed periods to repeat the analysis process for each period/cycle. We propose a condensed and adjustable representation of the data. Reeves et al. [46] offer an alternative to the subject.

3.2 Introducing the envelope approach

Assuming periodic data sampling and extraction of logical artifacts at the period level, we analyze the data collected over several periods. We divide the period into time units. For example, we divide it into daily time units for a year. We average the samples collected during each time unit and extract one value representing it. We repeat this process for the period and get a graph illustrating the average values for an intermediate and typical period. We then calculate the envelope around this average. The generated envelope represents the standard range of values such that unanalyzed periods are compared to this envelope. This period is normal if its graph value is entirely within the envelope. If it is totally out of the envelope, it is an exception. If just sections of the graph are within the envelope, we use an entropy measure to calculate the “distance” of the given period from the standard envelope. Assuming an existing entropy threshold, we can decide whether the period is typical. We apply the same concept at the unit level and determine whether a specific time unit in a period is within the standard. This particular check is relevant to anomaly detection of IoT behavior. Figure 2 depicts the main blocks of the envelope construction process.

Figure 2.

The process of constructing the optimal envelope.

The process has three key elements: an average measure per time unit, the boundaries around the middle chart, and an entropy value representing the distance of an actual chart from the envelope. We propose an optimal intensity of each component to generate a balanced and reliable anomaly detection method. We start by analyzing typical data collected from several time-dependent cycles, determining the average value per time unit, and drawing the boundaries around the average to get the envelope, as described in detail in Figure 3.

Figure 3.

The process of Constructing the Optimal Envelope.

Figure 4 describes the anomaly detection process by summing–up the number of cases in the examined chart that exceeds the envelope boundaries and in what direction.

Figure 4.

Classifying an unclassified Cycle.

This envelope method is generic and may be used for any application for anomaly detection, such as IoT sensors. In high variations, it can detect damaged or attacked sensors or support automatic instant corrections where abnormal behavior is seen. We may run a backtracking process for ongoing calibration of system parameters. This idea may be used to construct a multi-dimension envelope to comply with dependency among several columns within the same record.

3.3 Experiment

We accepted detailed Meteorological data about El-Niño (EN) and NonEl-Niño years (NEN) from 1980 to 1998. We took data from the El-Niño years 1982, 1983, 1987, 1988, 1991, and 1992 for the positive envelopes. All other years in the range were Non-El-Niño years. We tested three methods for generating envelopes: (1) minimum over all cycles and maximum over all cycles, (2) average cycle ± standard deviation, and (3) confidence interval (CI). Figure 5 visually confirms that 1995 is a regular year concerning its temperature spread. The Red and Blue charts represent the envelope’s upper and lower borders, respectively, while the Green chart represents the temperature in 1995. We realize that most temperatures are within the envelope upper/lower boundaries, generating a relatively low Entropy, 0.3631, beneath the selected threshold, concluding that 1995 is indeed a NEN year. However, referring to the 1992 and 1988 years, we got 0.4266 and 0.3857 Entropy values above the threshold; hence they are classified as EN years. However, we did not get a precise classification when we applied the ± standard deviation and the confidence interval (CI) methods.

Figure 5.

Min–max envelope for 1995 NEN.

3.4 Summary

Classification methods have recently gained attention due to rising IoT security issues and threats. In this section, we proposed an envelope construction to classify streams of time-dependent events within a defined data cycle. We discussed three envelope construction options: min–max, standard deviation, and confidence interval (CI). We described an Entropy calculation and a Threshold determination to classify whether a given steam data cycle is abnormal. We used Meteorological data streams to demonstrate our proposal technology’s correct classification of daily temperature streams for a year cycle. Several extensions to our proposal include discovering early trends of behavior changes, determining the number of data cycles required for constructing the optimal envelope, exploring the possibility of dividing one cycle into segments associating different envelopes to each segment, and defining rules for anomaly discovery.

4. Anomaly detection using random forest machine learning

The total transmitted data over the various sensors is growing accordingly. Sensors typically are low in storage, memory, and processing power resources. Data security and privacy are part of this ever-increasing domain’s significant concerns and drawbacks. A penetration discovery tool is recommended to predict possible attacks. Machine Training data leads to the definition of good and bad patterns for generating a Lightweight and activation framework comprised of Machine learning rule discovery, threat modeling, and timely reaction to rule violations. The model discovers exceptions and immediately updates the system. Random Forest (RF) is used for anomaly detection and rules generation. We converge IoT groups’ resource sharing to build an efficient IoT security framework. IoT networks collect and exchange vast data raising major security issues. To cope with it, we propose a decentralized, layered, distributed, and parallel processing model embedded in the.

The IoT network utilizes the remaining resources to execute the RF method to detect abnormalities. The model supports continued use and is decentralized over time.

The system identifies repeated patterns, while the Machine Learning algorithms discover the geometric, arithmetic, and additive. The patterns are translated into rules to be executed in violation cases. Anan adaptive extension is used to detect changes in generating data and adapt the decision mod to manage suspected situations sel.

The aim is to have a framework with training data collection analyzing it to detect patterns, proportions, etc., and converting it to rules. Combining the collected rules and RF trees is deployed in the IoT devices and network. The rules are executed when data is received from or transmitted to an IoT device. The corresponding action is triggered to cope with the situation if the result is positive or negative.

4.1 Literature review

Eghbal et al. [54] propose analyzing numerical data and generating fuzzy rules. The algorithm uses some rule-and-data-dependent parameters and a function that modifies the rule evaluation measures to assess the candidate rules effectively. Ref. [55] uses Sugeno integrals. They are qualitative criteria aggregations where it is possible to assign weights to criteria groups. It shows how to extract if-then rules expressing the selection of good situations based on local regulations and evaluations to detect bad conditions [56]. Dealing with converting data into the appropriate layout requires a significant investment in manual reformatting. The paper introduces a synthesis engine to extract structured relational data. It uses examples to synthesize a program utilizing an extraction language that extends regular expressions with geometric constructs. Ref. [57] proposes a fast and compact decision rules algorithm. It works online to learn rule sets. It presents a technique to detect local drifts relying on the rule set modularity. Each rule monitors the evolution of performance metrics to detect concept drift. It provides valuable information about the dynamics of the process generating data, faster adaptation to changes, and generates more compact rule sets [58, 59]. It uses averaging techniques to propose a method in which a previous algorithm for association rules mining specifies the minimum support automatically. It uses fuzzy logic to distribute data in different clusters and then tries to introduce to the user the most appropriate threshold automatically [60]. Suggests a two-stage hybrid model for data classification and rule extraction. The first stage uses a Fuzzy ARTMAP classifier with Q-learning and Genetic Algorithm for rule extraction from QFAM. Given a new data sample, the model can provide a prediction about the target class of the data sample and give a fuzzy if-then rule to explain the forecast. Q-values are applied to reduce the number of prototypes generated by QFAM [61]. Proposes a granular-rules extraction method to simplify a data set into a granular-rule set with unique granular rules [62]. It describes a QAR (Quick Access Recorder) anomaly detection algorithm. The method retains the time characteristics data and strengthens the relationship between the condition and decision attributes [63]. Describes an approach of data mining with Excel using the XLMiner add-on. It presents an example of mining association rules to illustrate this approach’s steps [64]. Introduces an algorithm for choosing which instances to request next in a setting where the learner can access a pool of unlabelled samples and request some labels [65]. It focuses on understanding the stochastic process’s role and how it defines a distribution over functions. It presents the simple equations for incorporating training data and examines how to learn the hyper-parameters using the marginal likelihood [66]. Proposes an active learning algorithm that balances such exploration with refining the decision boundary by dynamically adjusting the investigation probability at each step [67]. Offers a multiclass learning model that optimizes informative training compounds to support learning progress. Random Forest (RF) is used to predict quantitative compound activities. The global prediction is made by aggregating the predictions of the ensemble. Y. Brostaux [68] investigates the impact of noise in training data on the RF learning curve.

The reviewed literature focuses on improvements to known rule discovery mechanisms to transform them to become lightweight and able to be executed in a limited resource setting. In most cases, the proposed solution remains general purpose but can run with fewer required resources. Our proposal exploits the unique IoT attributes utilizing it to build a combined comprehensive framework for IoT security.

4.2 Rules generation and deployment process

The process consists of seven stages. Stage 1 composes training data from the IoT network; Stage 2 uses discovery techniques to extract essential measurements and patterns. Stage 3 consists of generating for each measure and pattern a rule. Stage 4 evaluates the effectiveness of each law against a set of training data. Stage 5 checks the generated rule set’s completeness and integrity. Stage 6 simulates the same training data expecting all the designated rules to be executed. Stage 7 deploys the developed regulations set. The system is ready to accept the IoT traffic data in real-time and automatically check it against the rules set. Figure 6 depicts The seven stages Anomaly Detection Process.

Figure 6.

The anomaly detection process.

4.3 Extracting simple rules from training data

Sensor record layout includes record ID, timestamp, and values per attribute. Simple rules, such as if-then, max, min, etc., are extracted directly from the record and its associated workflows.

4.4 Compound and multi-stage rules extraction

IoT rule engines assume real-time data streaming, instant reasoning, and actuators using Machine Learning extraction of compound rules from the continuous data records. The outcome contains thresholds, measurements, and decision trees that keep expanding, consuming vast storage, memory, storage, and runtime when analyzing the decision tree for the specific rule and tracing the tree path to understand its logic. Complex Event Processing (CEP) engines support matching time-series data patterns from different sources but have downsides in IoT since the logic requires high processing power and much time. We cope with these drawbacks by reducing the number of decision trees and improving the search navigation scope to a reasonable search time. IoT attributes and functionality are used to optimize tree navigation and process sharing. We use the bootstrap aggregation technique, counting the majority vote in the case of decision trees. Many trees reduce the depth and width of each tree and eventually save pruning and analysis time. The algorithm accepts the number of trees, K, and the number of features, F, randomly sampled features for building a decision tree. For extensive and high-dimensional data, a large K is used. Estimating the performance of Random Forest for one core is based on the following parameters: # trees [K], # features [F], # rows [R], and maximum depth [D]. The estimated runtime formula is K*F²*R*2^D. Hence, keeping just the most critical features, lowering the number of records, and keeping the maximum depth low will improve the overall Random Forest performance.

4.5 Experiment and summary

We use Excel functions and macros to generate compound rules such as pattern recognition for practical purposes. We also ran the Excel Machine Learning extension to create additional rules. We loaded the spreadsheet with 8 years of training data. All IoT devices are interconnected. In each device, we installed RF searching executable and deployed the generated simple rules and the RF trees in each device. We loaded the data by streaming it to the testing environment. Some generated rules do not require real-time reaction, consume processing power and memory space beyond the capacity of a typical sensor, and are executed at cloud processes. To have meaningful testing data, we intentionally added to the El-Niño file abnormal extreme values (e.g., over the maximum or lower than the minimum), wrong correlations, and classification interrupts. We loaded the data by streaming it to the testing environment. The corresponding rules and RF trees instantly detected all anomalies. We did not notice any data flowing interruptions or delays.

This section demonstrates the ability to build a lightweight, simple, and handy framework for anomaly detection, rules extraction, and rules execution given enough training data. We then described accuracy and performance improvements. Based on the accuracy and performance results, the feasibility and effectiveness of the proposed framework have been empirically proven.

5. Specific examples and case studies of successful anomaly detection

This section outlines practical and successful anomaly detection examples in various application domains. Most modern hospitals have automated laboratories, such as Chemistry, where all the blood tests are executed by dedicated machinery, which is frequently calibrated at every time interval. The calibration is done according to the manufacturer’s instructions. However, some laboratory managers run ongoing anomaly detection demons to ensure real-time control. We got a request to develop an ongoing anomaly detection process that also considers actual historical testing results and incorporates an anomaly detection check that considers the history of the specific population who visited the lab in the past. We collected 3 years of lab results per machine. We ran our envelope construction process and provided a very compressed envelope considering many parameters. As a result, any machinery problem is detected in near real-time, preventing any escape of exceptional results.

Another example is detecting abnormal data streaming sequencing, timing, and frequency from a permanent external resource using a sensor for each sampled attribute. The system listens to the communication line for a while when receiving transmissions from the designated source. The method constructs a multi-dimensional envelope corresponding to each feature based on the collected features, such as timing, interval length, and frequency. The multi-dimensional envelope and a weighted compound entropy measurement provide comprehensive communications anomaly detection.

6. Limitations and practical considerations related to IoT anomaly detection mechanism

Anomaly detection systems include a preprocessing stage for defining the normal value range where any value within the specified range is designated normal. In contrast, any other value is an exception. For a time-dependent data stream, the standard value range may vary depending on the repeatable cycle, such as season or different repeatable time ranges. Therefore, the correct determination of the repeatable cycle is crucial to the accuracy of the anomaly detection process. Thus, the following vital limitations and vulnerabilities are essential to mention:

1. Identifying the repeatable cycle length is the most critical step in IoT data analysis. A wrong cycle length leads to wrong detected anomalies.

2. Detecting abnormalities at the beginning and the end of each cycle is more complex because the difference between a normal state and an abnormal state is minor; therefore, the chance of making a mistake is more significant.

3. To maintain accuracy in the standard indices, we must continuously examine the correctness of the envelope values and their adaptation to the cycle we have defined and predict natural and justified changes in the cycle and its corresponding values used to check the anomalies over time.

7. Conclusion

This chapter deals with current and future trends in Anomaly detection concepts and technologies for the IoT context. We started with an overview of various IoT applications spread over most functional domains, such as Industry machinery, Health, Smart home, and smart city. Most of the new developments in IoT focus on solutions to the severe security breach caused by interconnecting numerous IoT devices to the Internet. These solutions provide tools for detecting/identifying operations anomalies. Therefore, we allocated Section 2 to cover IoT operation and communications security aspects. Then we elaborated on generating an envelope for anomaly detection for temporal transactions, which are the nature of IoT activity and networks. We finally elaborate on advanced technology for anomaly detection using Random Forest distributed over a network of IoT devices.

IoT keeps evolving and spreading fast everywhere in all functional domains in the modern world. Thus, new developments and recent trends will continue growing, so new chapters will follow.