End-to-End Benchmarking of Chiplet-Based In-Memory Computing

Abstract

In-memory computing (IMC)-based hardware reduces latency and energy consumption for compute-intensive machine learning (ML) applications. Several SRAM/RRAM-based IMC hardware architectures to accelerate ML applications have been proposed in the literature. However, crossbar-based IMC hardware poses several design challenges. We first discuss the different ML algorithms recently adopted in the literature. We then discuss the hardware implications of ML algorithms. Next, we elucidate the need for IMC architecture and the different components within a conventional IMC architecture. After that, we introduce the need for 2.5D or chiplet-based architectures. We then discuss the different benchmarking simulators proposed for monolithic IMC architectures. Finally, we describe an end-to-end chiplet-based IMC benchmarking simulator, SIAM.

1. Introduction

1.1 Modern-day AI algorithms

Machine learning (ML) and artificial intelligence (AI) significantly impacted society. AI algorithms, such as deep neural networks (DNNs), achieved accuracy that exceeded human-level perception for various applications, including medical imaging, natural language processing, and computer vision [1, 2, 3]. The popularity of AI algorithms was mainly driven by the availability of big datasets (for various applications like image classification and object detection) [1, 4, 5], as well as the increased computing power provided by the next-generation ML hardware accelerators and general-purpose computing platforms. Figure 1 illustrates the taxonomy of ML algorithms, which could be broadly categorized into supervised and unsupervised learning. Unsupervised learning involved extracting features from a distribution without data annotation, including selecting samples, denoising data, and clustering data into groups. The unsupervised learning algorithm aimed to find the optimal representation of the data that preserved maximum information about the input data x while ensuring the representation was simpler than the data itself, utilizing constraints such as lower-dimensional, sparse, or independent representation [6, 7]. Some popular unsupervised learning techniques were clustering, principal component analysis (PCA), autoencoders, and Gaussian potential functions.

Figure 1.

Taxonomy showing different ML with different learning techniques.

Supervised learning involves training the ML model using a set of labeled training data and then testing it using a labeled testing set. There are two types of supervised learning: classical approaches and deep learning. Classical approaches use conventional techniques that employ a probabilistic model to determine the next state based on a set of parameters. Some popular classical techniques are decision trees, support vector machines (SVM), Markov chains, and maximum likelihood estimation (MLE) [8, 9, 10, 11]. However, classical techniques had limitations such as difficulty in scaling, lack of generalization, and the need for significant data engineering for each algorithm. Classical techniques serve as the foundation for deep learning algorithms, which overcome their limitations. This chapter focuses on deep learning techniques used in supervised learning. Convolutional neural networks (CNNs) stand out for their superior performance in various machine-learning tasks, including computer vision, object detection, and object segmentation. Additionally, recurrent neural networks (RNNs) excel in processing temporal data, while graph convolutional networks (GCNs) combine graphs and neural networks for various applications. The chapter covers recent advancements in CNNs, RNNs, and GCNs, discussing their structures, training methods, and execution efficiency for training and inference operations.

Conventional CNNs consist of layers connected either sequentially or with skip connections. Besides convolutional layers, ReLU, pooling, and batch-normalization methods are commonly used to enhance performance. Figure 2 illustrates the typical structure of a convolutional and fully connected (FC) layer. Sequential layers usually involve a stack of convolution (conv) layers that extract features from the input. Convolution layer kernels may include 7 × 7, 5 × 5, 3 × 3, and 1 × 1. Additionally, as proposed in MobileNet [12], depthwise convolutions divide a given N × N convolution into two parts: first, a N × 1 convolution is performed, followed by a 1 × N convolution. Depth-wise convolution yields better accuracy and lower hardware complexity. Pooling layers are periodically utilized to reduce the feature map size and eliminate noisy input. In order to perform classification on extracted features, a set of FC layers or classifier layers are utilized. These layers, along with the Conv layers, have a set of weights that are trained to achieve the best accuracy. Popular CNN structures, such as AlexNet [1], GoogleNet [13], ResNet [14], DenseNet [15], MobileNet [12], and SqueezeNet [16], utilize a combination of convolutional, pooling, and FC layers to achieve high accuracy in a variety of ML tasks. Additionally, CNNs like DenseNet and ResNet utilize skip connections from previous layers to create a highly branched structure, which aims to improve the feature extraction process. These skip connections are present within the conv layers only. In contrast, conventional CNNs face several limitations, such as the vanishing gradient problem, higher hardware costs during training and inference, and over-parameterization [17, 18, 19]. To address these issues, network architecture search (NAS) was introduced to automatically find the optimal neural network architecture based on the specific design point for the target application. Different design points, such as higher accuracy, better generalization, higher hardware efficiency, and lower memory access, have been proposed for NAS. Popular techniques like NasNet [20], FBNet [21], AmoebaNet [22], PNAS [23], ECONas [24], and MNasNet [25] have been developed to address these issues.

Figure 2.

(a) Convolution layer consisting of the input feature map (IFM), kernel, and the output feature map (OFM), (b) FC layer in a CNN.

RNN is a commonly used deep learning technique that offers an effective solution for modeling data with sequential or temporal structures and variable length inputs and outputs in various applications [26, 27, 28]. It processes sequential data one element at a time, using a connectionist model that selectively passes information. This enables RNNs to model input and/or output data consisting of a sequence of dependent elements. Additionally, RNNs can simultaneously model both sequential and time dependencies at different scales. They employ a feedforward network that includes edges connecting adjacent time steps, which introduces time to the model. While conventional edges do not have cycles, recurrent edges that connect adjacent time steps can form cycles. Modern RNN architectures can be categorized into two main types. The first is long-short-term memory (LSTM), which includes a memory cell, a computation unit that replaces traditional nodes in the hidden layer of a network [29]. The second variant of RNNs is bi-directional RNNs (BRNNs), as proposed in [30].

As more applications rely on graphs to represent data, using CNNs and RNNs to capture hidden patterns within Euclidean data becomes limited. For instance, in e-commerce, a graph-based learning system can use interactions between users and products to recommend highly accurate products. However, graphs’ complexity and irregularities pose significant challenges to existing DNNs. To address this issue, Graph Neural Networks (GNNs) were introduced and categorized into three types: Recurrent GNNs (RecGNNs) [31, 32, 33], Convolutional GNNs (CGNNs) [34, 35, 36], and Graph Autoencoders (GAE) [37, 38, 39]. RecGNNs use recurrent neural architectures to learn node representations, while CGNNs generalize convolution operations to graph data by aggregating features from neighboring nodes. GAEs map nodes into a latent feature space, preserving the node’s topological information with a low-dimensional vector. Finally, GAEs learn network embeddings using an encoder and a decoder to enforce the preservation of graph topological information.

1.2 Hardware implications of DNNs

State-of-the-art DNNs, such as CNNs, RNNs, and GCNs, have diverse structures that lead to significant demands on compute and memory resources. Achieving higher accuracy with these machine-learning models requires increased computational complexity and model size, which, in turn, require more memory to store both the weights and activations. The increased model size and complexity also lead to a larger volume of on-chip data movement. For instance, the ImageNet dataset’s [1] ResNet-50 [14] requires 50 MB of memory and 4 GFLOPs per inference, while DenseNet-121 [15] requires 110 MB of memory and 8 GFLOPs per inference. Conventional architectures that separate memory and computation lead to a considerable number of external memory accesses, which reduce energy efficiency and performance. The average cost of an external memory access is 1000 times higher than the energy required for computations [40]. When considering the total energy spent on performing inference for VGG-16 and ResNet-50 using conventional von Neumann architectures, a floating-point 32-bit (FP-32) multiplication results in 3.2pJ, while an FP-32 add requires 0.9pJ in the 45 nm technology node [41]. Therefore, performing inference for one image consumes 65 mJ of energy using the VGG-16 CNN, while ResNet-50 consumes 16 mJ. Scaling the computation energy up, for 1000 inference performs, VGG-16 takes 65 J while ResNet-50 consumes 16 J of energy. In conclusion, the higher accuracy achieved by DNNs results in higher computational complexity, increased memory requirements, more off-chip memory access, and lower energy efficiency.

This chapter delves into in-memory computing (IMC) as an alternative to traditional von-Neumann architecture, which offers improved energy efficiency, better performance, and reduced off-chip memory access. IMC has emerged as a promising solution to tackle the memory access, energy efficiency, and performance bottlenecks encountered by DNN applications. Hardware architectures for IMC, such as those based on SRAM and nanoscale nonvolatile memory (e.g., resistive RAM or RRAM), provide a dense and parallel structure to achieve high performance and energy efficiency [42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57]. Additionally, the chapter introduces chiplet-based IMC architectures, as well as a benchmarking simulator for this architecture. The SIAM simulator and the associated architecture enabled through SIAM are described in detail. Overall, this chapter explores various benchmarking simulators for IMC architectures.

2. IMC architectures

In the preceding section, we examined the hardware implications of modern DNNs, specifically the memory and computation complexity associated with von Neumann architectures. For instance, dense structures such as DenseNet require roughly $2.7\times 10^7$ off-chip memory accesses to process an image frame [15]. This elevated number of off-chip memory access negatively impacts the energy efficiency of the overall system. IMC architectures provide a promising alternative to traditional von Neumann architectures. Figure 3 depicts a generic block diagram of an IMC architecture with RRAM/SRAM memory cells. IMC uses analog- or digital-domain computation to carry out multiply-and-accumulate (MAC) operations. The crossbar-based IMC structure efficiently combines memory access and analog-domain computation into a single unit, resulting in faster execution of DNN workloads. The superior energy efficiency is primarily due to a full-custom design, higher density, and higher memory bandwidth [44, 45, 58]. Consequently, IMC-based systems are becoming increasingly popular for implementing compute- and memory-intensive AI applications. This section will examine various IMC architectures in depth using both SRAM and RRAM memory cells.

Figure 3.

Generic block diagram of an IMC architecture for DNN acceleration. It consists of an array of IMC tiles connected by an NoC with each tile consisting of a number of IMC arrays [44, 45].

2.1 RRAM/SRAM-based IMC architectures

2.1.1 RRAM Device

RRAM-based IMC architectures consist of an RRAM memory cell at each crosspoint within the IMC crossbar array. RRAM is a two-terminal device with programmable resistance representing the neural network’s weights. It has high integration density, fast read speed, high memory accessing bandwidth, and good compatibility with CMOS fabrication technology. For example, the RRAM device stack can include a TiN bottom electrode, ${\mathrm{HfO}}_2$ mem-resistive switching layer, a PVD Ti oxygen exchange layer (OEL), and ∼ 40 nm TiN top electrode [59, 60]. This specific stack is implemented between the M1 and M2 metallization layers using a FEOL-compatible process flow.

Each RRAM cell can be characterized by the number of resistance levels accessed within them. Broadly, RRAM can be classified into single-level cells (SLC) and multi-level cells (MLC). SLC only has two resistance levels, that is, they can store only binary data. On the other hand, MLC cells have multiple resistance levels that represent higher precision data. The number of available resistance levels is governed by the ratio of the off resistance $\left({\mathrm{R}}_{\mathrm{off}}\right)$ to the on resistance $\left({\mathrm{R}}_{\mathrm{on}}\right)$ ratio [61]. The ratio provides the range of resistances accessible for the given RRAM device. The overall resistance range can be divided into two main states, a low resistance state (LRS) and a high resistance state (HRS). LRS deals with the lower spectrum within the resistance band, while the HRS deals with the upper band of resistance of the RRAM device.

To program the RRAM device, a series of steps need to be followed [62]. First, the RRAM device is formed by applying a large voltage across the two terminals. This process breaks the barrier and allows for electron flow across the terminals. Next, the RRAM is programmed to the required resistance by passing a specific current (compliance current) through the two electrodes. The RRAM can be programmed at different resistances depending on the compliance current. Furthermore, different resistance levels can be achieved depending on the RRAM device (SLC or MLC). Finally, once the RRAM device is programmed, we can perform a read by applying a voltage across the device electrodes. For the RRAM device proposed in [59, 60], a read voltage of up to 0.4 V can be sustained by the RRAM device. Applying a higher voltage damages the device or goes into the write state, changing the programmed resistance level.

2.1.2 IMC Architecture

Studies involving crossbar architectures have demonstrated that a 100× to 1000× improvement in energy efficiency is achieved as compared to traditional CPU and GPU architectures [44, 45, 47, 49, 50, 52, 63, 64, 65, 66, 67]. Figure 3 shows the block diagram of a IMC architecture with an RRAM/SRAM memory cell. The architecture consists of an array of IMC tiles connected by a network-on-chip (NoC). The architecture also consists of a global pooling unit, nonlinear activation unit, accumulator, and input/output buffers. A global control logic performs the architecture’s overall handling of the blocks.

Each tile consists of an array of processing elements (PEs), where each PE is an IMC crossbar array with either an SRAM or an RRAM cell. Each IMC crossbar array consists of a set of peripheral circuits that enable the MAC computations.

Figure 4 shows the generic block diagram for a single RRAM-based IMC crossbar array. In the case of RRAM IMC, a transistor connects the gate to the wordline (WL) of the IMC crossbar array [60]. The WL connects to the access transistors for the SRAM-based IMC with a conventional 6 T structure. The IMC crossbar arrays consist of a wordline (WL) decoder, WL driver, a column multiplexer, analog-to-digital converter (ADC) or a sense amplifier, shifter and add circuit, control logic, and input/output buffers. The WL decoder turns on and off the WL for the IMC crossbar array. Meanwhile, the WL driver and level shifter ensure the driver can turn on the memory cell. Next, for an N × N IMC crossbar array, M columns are shared across the read-out circuit. The read-out circuit consists of the ADC, shift and add circuit, and the precharge circuit for the read operation. To enable the sharing of M columns, a column multiplexer is used. Finally, a custom control logic is utilized to drive the control signals during the operation of the IMC crossbar array. We will now go over the operation for both the SRAM and RRAM-based IMC architectures. First, we will detail the working of the RRAM-based IMC architecture. Figure 4 shows the generic block diagram for a single RRAM-based IMC crossbar array. The RRAM devices are programmed by connecting the two terminals to a given voltage. To facilitate this, the terminals are connected to the bitline (BL) and the selectline (SL). By applying a voltage across the BL and the SL, forming, programming, and reading operations are performed cell-by-cell. Each cell is chosen during the write state of the IMC, and then the write is performed. During the compute state, the RRAM undergoes the read operation. Two kinds of read-out are performed, parallel and serial. During the parallel read-out, all/multiple WLs are turned on simultaneously, and the output is accumulated across the BL. Two kinds of input schemes are employed for single and multi-bit inputs. The first method uses a digital-to-analog converter (DAC) to convert the input vector to an analog voltage and performs the computation in the charge domain [44]. The second method is to perform bit-serial computing, where each bit in the input vector is computed one at a time. Each input vector’s bit significance is handled using a shift and add circuit [45, 49, 63].

Figure 4.

Block diagram of a RRAM-based IMC crossbar array. An array of RRAM cells form the IMC crossbar array. Peripheral circuits such as bitline (BL)/select line (SL)/column multiplexer (MUX), precharge circuit, wordline (WL) decoder and driver, buffers, level shifters, ADC, and shift and add circuit complete the RRAM-based IMC.

Depending on the resistance stored in the RRAM, an output current/charge is generated by the product of the voltage and resistance (conductance). This operation is analogous to the multiply with the MAC. This current/charge is then accumulated across all rows for a given column to perform the addition in the MAC. In the case of the serial read-out, row-wise access of the IMC array is performed for MAC computations. Overall, the final MAC output is generated by accumulating across all rows of the IMC crossbar array.

Figure 5 shows the generic block diagram for a single SRAM-based IMC crossbar array. Next, we will discuss the operation for an SRAM-based IMC architecture [48, 49, 51, 52, 68, 69, 70]. Depending on the SRAM bit-cell type and the degree of parallelism, the IMC design can be largely divided into three categories [71]: 6 T bit-cell with parallel computing, 6 T bit-cell with local computing, and (6 T + extra-T) bit-cell with parallel computing. Initially, SRAM-based IMC architecture employed the 6 T bit-cell with a parallel computation [72, 73]. The parallel computation was achieved by turning on all the WL together to perform the MAC operations. The WL is driven by the input vector where a one means it turns on that cell, while a zero means the cell is turned off. Next, a 6 T bit-cell with a local compute structure is utilized where a special compute engine is designed to perform the MAC operation [70]. Here, the MAC operation is performed row-by-row, similar to the serial read-out in RRAM-based IMC. Finally, in addition to the 6 T cell, extra transistors are added in each bit-cell to perform parallel compute [51, 68, 69]. In addition to the bit-cell structure, peripheral circuits such as precharge circuits, ADCs, write drivers, column multiplexers, row decoders, and row drivers are used.

Figure 5.

Block diagram of a SRAM-based IMC crossbar array. An array of SRAM cells 6 T or 6 T + additional circuit) form the IMC crossbar array. Peripheral circuits such as bitline (BL) and bitline bar (BLB) precharge and conditioning circuits, row decoder and WL driver, column multiplexers, buffers, write drivers, ADC, and shift and add circuit complete the SRAM-based IMC.

2.1.3 Challenges with IMC Architectures

IMC architectures are known for their improved energy efficiency and throughput, but they have some drawbacks. One such drawback is the limited precision of the IMC crossbar array, particularly in the memory cell and ADC, which can affect the accuracy of DNN inference [74, 75]. Additionally, noise within analog computation can also harm DNN inference accuracy. The challenges associated with an RRAM-based IMC architecture are discussed next. RRAM devices suffer from several non-idealities, including limited resistance levels, device-to-device write variations, stuck-at-faults, and limited ${\mathrm{R}}_{\mathrm{off}}/{\mathrm{R}}_{\mathrm{on}}$ ratio, which make it difficult to design reliable RRAM-based IMC architectures [60, 61, 76, 77, 78, 79, 80, 81, 82, 83, 84]. These non-idealities can cause programmed weight values (resistance value) to deviate, significantly reducing post-mapping accuracy for DNNs. Moreover, the limited array size of the IMC crossbar structure necessitates the splitting of large convolution (conv) or FC layers into partial operations, which can introduce additional errors due to the limited precision of the peripheral circuits (ADC and shift and add) of the RRAM-based IMC crossbar.

Previous research has proposed several methods to address the post-mapping accuracy loss associated with RRAM-based in-memory acceleration of DNNs. Two methods that have been proposed are Closed-Loop-on-Device (CLD) and Open-Loop-off-Device (OLD), which involve iterative read-verify-write (R-V-W) operations at the RRAM device until the resistance converges to the desired value [85, 86]. Other methods, such as in References [78, 87], utilize variation-aware-training (VAT) based on known device variation (σ) characterized from RRAM devices. In [76], VAT is combined with dynamic precision quantization to mitigate the post-mapping accuracy loss. Another approach proposed in [75] involves injecting RRAM macro measurement results that include variability and noise during the DNN training process to improve the DNN accuracy of the RRAM IMC hardware. Mohanty et al. [88] proposes post-mapping training by selecting a random subset of weights and mapping them to an on-chip memory to recover the accuracy. Charan et al. [79] utilizes knowledge distillation and online adaptation for accuracy mitigation, using an SRAM-based IMC as the parallel network, while [88] proposes to use a register file and a randomization circuit. Finally, [77, 80] propose a custom unary mapping scheme by mapping the most significant bit (MSB) and least significant bit (LSB) of the weights to RRAM devices based on individual cell variations and bit significance.

Next, we discuss the challenges associated with SRAM-based IMC architectures. A compromise between parallelism and reliability is employed for best performance. In a conventional 6 T SRAM IMC architecture, parallel computation is achieved by turning on all or multiple rows. The higher parallelism raises the critical issue of read disturbance, resulting in the WL voltage being driven with a lower voltage [72, 73, 89]. To mitigate this, reduced parallelism is employed by exploiting the local compute engine [70]. The reduced parallelism results in reduced throughput for DNN inference. Yin et al. [51, 68, 69] proposes to utilize additional transistors that isolate the bit-cell and employ parallel computation. Such a solution comes at the cost of increased area overhead, thus limiting the density of the SRAM-based IMC architecture. The additional transistor solution is typically implemented using a resistance or a capacitance. The resistive IMC method implements a multi-bit MAC operation by utilizing a resistive pull-up/down using transistors [51, 68, 69]. The pull-up/down characteristics of the transistors exhibit a non-linear behavior for the read bitline (RBL) transfer curve across different voltage ranges, thus reducing reliability. At the same time, the capacitive SRAM-based IMC utilizes a capacitor per bit-cell and charge sharing and capacitive coupling to perform the MAC operations [52]. The capacitive SRAM IMC exhibits a more linear transfer characteristic on the RBL but at the cost of a capacitor per bit-cell. Finally, the limited precision of the ADC and the noise on the bitline (BL) requires careful algorithm design to achieve the best inference accuracy [89].

2.2 2.5D/Chiplet-based AI accelerators

The area of monolithic hardware accelerators increases with the increasing number of parameters of AI algorithms. The higher silicon area of a single monolithic system reduces the yield, increasing fabrication cost [46]. The chiplet-based system solves the issue of higher fabrication cost by integrating multiple small chips (known as chiplets) on a single die. Since the area of each chiplet in the system is considerably lower than a monolithic chip (for the same AI algorithm), the yield of the chiplet-based system increases, which reduces the fabrication cost. The communication between chiplets is performed through network-on-package (NoP), as shown in Figure 6. Several works in the literature propose NoP for chiplet-based systems considering different performance objectives (e.g., latency, energy) [42, 46, 90, 91, 92].

Figure 6.

Chiplet-based IMC architecture [46, 64] that includes an NoP for on-package communication, NoC for on-chip communication within each chiplet, and a point-to-point network like H-Tree for within tile communication.

Kite is a family of NoP proposed in [90], mainly targeted for general-purpose processors. In this work, three topologies are proposed—Kite-Small, Kite-Medium, and Kite-Large. First, an objective function is constructed with a combination of the average delay between the source and destination, diameter, and bisection bandwidth of the NoP. Experimental evaluations on synthetic traffic show that the proposed Kite topologies reduce latency by 7% and improve the peak throughput by 17% with respect to other well-known interconnect topologies. A chiplet-based system with a 96-core processor, INTACT, is proposed in [91]. The chiplets are connected through a generic chiplet-interposer interface (called 3D-plugs in the paper). 3D-plugs consist of micro-bump arrays. However, both Kite and INTACT are not specific to AI workloads.

Shao et al. designed and fabricated a 36-chiplet system called SIMBA for deep learning inference [92]. The chiplets in the system are connected through a mesh NoP. Ground-referenced signaling (GRS) is used for intra-package communication. The NoP follows a hybrid wormhole/cut-through flow control. The NoP bandwidth is 100 GBps/chiplet, and the latency for one hop is 20 ns. Extensive evaluation on the fabricated chip shows up to 16% speed up compared to baseline layer mapping for ResNet-50. A simulator for the chiplet-based systems, SIAM, is proposed in [46], targeting AI workloads. In this simulator, a mesh topology is considered for NoP. It is shown that up to 85% of the total system area is contributed by NoP. In this work, multiple studies were performed by varying NoP parameters. For example, it is shown that increasing the NoP channel width increases the energy-delay product of the NoP for ResNet-110. This phenomenon is demonstrated for systems with 25 and 36 chiplets. However, none of the prior works considered any workload-aware optimization for the NoP. Therefore, there is ample opportunity for future research considering NoP optimization for AI accelerators.

3. Benchmarking simulators

3.1 Monolithic AI benchmarking simulators

Numerous researchers have put forth assessment frameworks for evaluating the performance of IMC-based AI accelerators. One such framework is NeuroSim [93], the first simulator designed to evaluate the performance of IMC-based AI accelerators. It includes various performance evaluation metrics such as area, latency, and power consumption of an IMC system under a given workload of DNN. NeuroSim is highly flexible and allows users to assess the performance of IMC-based AI accelerators under different system specifications, considering both traditional CMOS-based memory technology (such as SRAM) and emerging nonvolatile memory technologies (like ReRAM and STTMRAM) for the in-memory compute elements. NeuroSim assumes a tile-based architecture [44], consisting of multiple tiles where processing elements (PEs) and IMC-based crossbar arrays reside. Predictive Technology Model (PTM) [94] is used to simulate lower-level components, such as buffers, ADC, and multiplexers, and verified against circuit simulation (e.g., SPICE), reaching more than 90% accuracy. The interface between NeuroSim and popular ML frameworks such as PyTorch and TensorFlow has also been created to make it more user-friendly [95]. However, one major drawback of NeuroSim is that it assumes H-Tree based bus interconnect for inter-tile communication, which can consume up to 90% of the total energy consumption of DNN inference [96]. To overcome this issue, Krishnan et al. [97] (Figure 7) proposed an evaluation framework for IMC-based AI accelerators that considers cycle-accurate network-on-chip (NoC) simulation [98]. Similarly, MNSIM [99] also evaluates the performance of IMC-based systems to execute AI applications like NeuroSim. In addition, MNSIM integrates software-hardware co-design techniques in the evaluation framework. GeneiX, proposed by Chakraborty et al. [78], is another evaluation framework for crossbar-based IMC accelerators considering the non-idealities in the memory elements.

Figure 7.

Block diagram on an IMC benchmarking simulator proposed in [97]. The simulator consists of a circuit part and an interconnect part that perform system-level benchmarking of IMC architectures.

While it is essential to evaluate hardware performance under AI workloads, it is also important to assess the accuracy of the AI workload when implemented on-chip. Memory imperfections can lower the accuracy of DNNs. RxNN [100] and SpikeSim [66] are frameworks that evaluate the accuracy of DNN workloads in the presence of memory imperfections. These techniques focus on evaluating the performance of IMC systems executing DNN inference. However, emerging edge devices require online learning, which involves training the DNN. Therefore, evaluating the performance of AI accelerators while executing DNN inference alone is insufficient. An evaluation framework for IMC-based AI accelerators with on-chip training is introduced in [101]. The authors incorporate non-linearity, asymmetry, device-to-device, and cycle-to-cycle variation of weight update into the Python wrapper and peripheral circuits for error/weight gradient computation in NeuroSim core for a given AI workload. The training framework is based on the authors’ prior work [102], where they proposed an SRAM-based transposable function. Since SRAM-based arrays can perform write operations quickly while consuming minimal energy, the weight-gradient computation function is implemented through SRAM-based arrays rather than other nonvolatile memory technologies.

3.2 SIAM: Chiplet-based AI benchmarking simulator

This section introduces SIAM [46], a benchmarking simulator for IMC architectures based on chiplets. SIAM supports both generic or homogeneous and custom chiplet-based IMC architectures. A homogeneous architecture has a fixed number of chiplets that the user determines. On the other hand, a custom architecture comprises a specific number of chiplets required to map the DNN under consideration. In both cases, the chiplet structure contains a fixed number of user-defined IMC crossbar arrays. Figure 6 illustrates a homogeneous chiplet-based IMC architecture SIAM uses.

Figure 8 illustrates how the comprehensive framework provided by SIAM can be used to benchmark the performance of chiplet-based in-memory compute (IMC) architectures. SIAM generates a chiplet-based IMC architecture based on user-defined inputs. It assesses the corresponding hardware performance by computing various performance metrics, including area, energy, latency, energy efficiency, power, leakage energy, and IMC utilization. The SIAM framework is developed using Python and C++ programming languages and has a top-level Python wrapper that integrates the different components of the simulator. Additionally, SIAM is designed to interface with widely used deep learning frameworks, such as PyTorch and TensorFlow, and supports a range of network structures in current literature. Therefore, SIAM can be used to explore neural architecture search (NAS) techniques. Table 1 describes the user inputs and their associated parameters for the SIAM benchmarking tool. SIAM consists of four engines:

• Partition and mapping engine (Python)

• Circuit and NoC engine (C++)

• NoP engine (Python and C++)

• DRAM engine (Python and C++)

Figure 8.

Block diagram of the chiplet-based IMC architecture simulator SIAM [46].

User input	Description	User input	Description
Intra-chiplet architecture		Inter-chiplet architecture
ADC resolution	Bit-precision of flash ADC	Chiplet size	Number of IMC tiles within each chiplet
Read-out method	Sequential or parallel	Total chiplet count	Fixed count or DNN specific custom count
Crossbar size	IMC crossbar array size	Chip mode	Monolithic or chiplet-based IMC architecture
Buffer type	SRAM or register file	Chiplet structure	Homogeneous or custom chiplet structure
NoC width	Number of channels in the NoC	NoP frequency	Frequency of the NoP driver and interconnect
NoC topology	Mesh or tree	Global accumulator size	Size of global accumulator
Frequency	Frequency of operation	NoP channel width	Number of parallel links for TX and RX
DNN algorithm		Device and technology
Data precision	Weights and activation precision	Memory cell	RRAM or SRAM
Network structure	DNN network structure	Tech node	Technology node for fabrication
Sparsity	DNN layer-wise sparsity	Bits/cell	Number of levels in RRAM

Table 1.

Definition of the user inputs to SIAM.

The individual engines within SIAM operate independently on various subsets of user inputs and communicate with each other through the top-level Python wrapper. Figure 8 provides an overview of the simulation flow used by SIAM to provide a better understanding of the framework. Firstly, the partition and mapping engine is used to perform layer partition and mapping onto the chiplets and IMC crossbars, which generates the IMC architecture structure, the number of required chiplets and IMC tiles per layer, the IMC architecture utilization, the volume of intra-chiplet and inter-chiplet data movement, and the number of global accumulator accesses. Next, the circuit and NoC engine evaluate the intra-chiplet and global circuit performance, respectively, providing hardware performance metrics such as area, energy, and latency. Meanwhile, the NoP engine evaluates the cost of chiplet-to-chiplet data movement. Lastly, the DRAM engine assesses the memory access cost, providing energy and latency performance metrics. Except for the partition and mapping engine, all engines operate concurrently, resulting in shorter simulation times. Additionally, SIAM can be used to benchmark traditional monolithic IMC architectures. In the following sections, we provide detailed information on the four engines that comprise SIAM’s core functionality.

3.3 Partition and mapping engine

The operational steps of the partition and mapping engine are explained in Algorithm 1. The engine is responsible for partitioning the DNN layers and mapping them onto the IMC chiplets and crossbar arrays. The partitioning and mapping are carried out layer by layer for the entire DNN. The engine takes various user inputs such as DNN structure, the precision of DNN weights, the mapping scheme for IMC chiplets, the size of IMC chiplets, and the size of IMC crossbars.

To begin with, we will describe the IMC mapping approach that is employed in SIAM. Suppose we have a layer i with weight matrix ${\mathrm{W}}_i$ represented by ${\mathrm{Kx}}_i\times {\mathrm{Ky}}_i\times {\mathrm{Nif}}_i\times {\mathrm{Nof}}_i,$ where Kx and Ky indicate the kernel size, Nif refers to the number of input features, and Nof denotes the number of output features. SIAM uses the same mapping scheme presented in [44, 45]:

$$N_i^r=⌈\frac{{\mathit{Kx}}_i\times {\mathit{Ky}}_i\times {\mathit{Nif}}_i}{\left({\mathit{PE}}_x\right)}⌉;N_i^c=⌈\frac{{\mathit{Nof}}_i\times N_{\mathit{\text{bits}}}}{\left({\mathit{PE}}_y\right)}⌉$$E1

The equation given above calculates the required number of rows and columns for the IMC crossbars to map layer i of the DNN. ${\mathrm{N}}_i^r$ and ${\mathrm{N}}_i^c$ represent the number of rows and columns required, while ${\mathrm{N}}_{\mathit{\text{bits}}},{\mathrm{PE}}_x,$ and ${\mathrm{PE}}_y$ denote the DNN weight precision and the number of rows and columns in the IMC crossbar array. To find the total number of required IMC crossbar arrays to map layer i of the DNN, one can multiply ${\mathrm{N}}_i^r$ and ${\mathrm{N}}_i^c$ together, which gives ${\mathrm{N}}_i^{\mathit{\text{Total}}}$ (line 7 of Algorithm 1).

SIAM can create homogeneous and custom chiplet-based IMC architectures using two types of chiplet partitions. The partition and mapping engine generates architectures based on the assumption that each DNN layer cannot be divided across multiple chiplets and that each chiplet can support multiple layers for optimal chiplet utilization. To map an entire layer of the DNN, multiple chiplets with IMC crossbar arrays are required due to numerous multi-bit weights in each layer. Dividing a layer across multiple chiplets increases overhead in control logic required for routing inputs, higher chiplet-to-chiplet communication energy and latency, and greater inter-chiplet data communication volume. The engine uniformly divides the layer across multiple chiplets during partitioning to prevent workload imbalance issues. The engine determines the number of chiplets required to map layer i of the DNN based on the total number of required IMC crossbar arrays, $N_i^{\mathit{\text{Total}}},$ using the equation $N_i^{\mathit{\text{Chiplet}}}=⌈\frac{N_i^{\mathit{\text{Total}}}}{S}⌉$, where S represents the total number of IMC crossbar arrays within a chiplet (the chiplet size). The resulting architectures generated by the partition and mapping engine can be seen in Figure 9.

Figure 9.

The figure illustrates two chiplet-based IMC architectures, namely homogeneous (left) and custom (right), generated by SIAM for the same DNN. The homogeneous architecture is a generic architecture, whereas the custom architecture is tailored for the specific DNN. The NoP router is denoted by R in both architectures.

After determining the number of required chiplets to map a layer of the DNN, the next step is to determine the total number of chiplets in the architecture. This is done at line 9 of Algorithm 1. In the homogeneous chiplet partition scheme, the user inputs a fixed number of chiplets to map the DNN. The engine compares the total number of chiplets required to map the DNN $\left({\mathrm{N}}^{\mathit{\text{Chiplet}}}\right)$ with the maximum available chiplets in the architecture (C). If the number of required chiplets exceeds the maximum available chiplets, the engine throws an error and requests an increase in the number of available chiplets. On the other hand, if the number of required chiplets is less than or equal to the maximum available chiplets, the engine continues with the partition and mapping process for the subsequent layers in the DNN.

The custom partition scheme in SIAM creates a chiplet-based IMC architecture tailored to the specific DNN being considered without any upper limit on the number of chiplets used. Each chiplet in this scheme has a consistent structure with a fixed number of IMC tiles containing IMC crossbar arrays and peripheral circuitry. On the other hand, the homogeneous partition scheme uses a fixed number of chiplets (user input) to map the DNN in a generic manner. SIAM allows for the comparison of both architectures on a single platform. After partitioning and mapping the layers onto the IMC chiplets, the engine calculates the total amount of data communicated within and across the chiplets, taking into account instances where a layer is partitioned across multiple chiplets. The global accumulator is used to generate the layer output in such cases. The engine also determines the number of additions performed by the global accumulator and the number of global buffer accesses required. The engine output includes the layer partition across chiplets, the necessary number of chiplets and IMC crossbars, IMC crossbar utilization, intra- and inter-chiplet data movement volume, and the number of the global accumulator and buffer accesses. This information is then used by the circuit, NoC, and NoP engines to evaluate the performance of the chiplet-based IMC architecture.

3.4 Circuit and NoC engine

After the partitioning and mapping of the DNN, the next step in SIAM is to arrange the inter- and intra-chiplet components in a floorplan and place them. This process leads to the final design of the chiplet-based IMC architecture. Once the architecture is determined, the circuit and NoC engine estimate the performance of the hardware, as illustrated in Figure 10. The engine uses a model-based estimator to evaluate the circuit aspect, while it employs a trace-based estimator for the interconnect portion.

Figure 10.

Block diagram of the circuit and NoC engine within SIAM. The engine utilizes a separate circuit and NoC simulators that perform the overall hardware performance estimation.

3.4.1 Circuit estimator

The circuit estimator evaluates the hardware performance of the chiplets, global accumulator, and global buffer in the entire chiplet-based IMC architecture. To perform this evaluation, the engine considers a range of inputs such as the placement of components within and across chiplets, the number of chiplets and IMC crossbars per layer, the IMC utilization per layer, the technology node, the operating frequency, the type of IMC cell, the number of bits per cell, and the ADC precision. The intra-chiplet circuits include the IMC crossbar array, buffer, accumulator, activation unit, and pooling unit. In contrast, the peripheral circuits consist of the ADC, multiplexer circuit, shift and add circuit, and decoders. The circuit estimator is calibrated using NeuroSim [95].

The circuit estimator evaluates the performance of the entire chiplet-based IMC architecture, where each layer of the DNN is considered separately, and each chiplet is responsible for computations for a subset of layers. The partition and mapping engine provides the chiplet count, IMC crossbar count, and IMC utilization values for each layer. The estimator estimates the area, energy, and latency from the device level to the circuit and architecture levels, using user inputs such as technology node, IMC cell type, IMC crossbar size, ADC precision, and read-out mode. For each IMC crossbar within the chiplet, the cost of a single crossbar and its peripheral circuits are evaluated to obtain the total area, energy, and latency of the IMC chiplet, which includes the buffer cost, shift and adder circuitry, accumulator, pooling, and activation units. The global accumulator and global buffer accumulate the partial sum of a layer across chiplets at the chiplet-level. The estimator utilizes the number of additions performed, data volume from each chiplet, and the accumulator size to determine the global accumulator and buffer’s area, energy, and latency. Finally, the estimator repeats the estimation for all chiplets required for a given layer of the DNN to determine the overall hardware performance.

3.4.2 NoC Estimator

Effective communication is essential for achieving optimal hardware performance in DNN accelerators [63]. In [103], communication-intensive DNN accelerators are extensively discussed. Every layer of a DNN transmits a significant amount of data to other layers. Studies have demonstrated that communication can account for up to 90% of the total inference latency in DNNs [45]. Therefore, it is vital to develop an efficient communication protocol for DNNs. To achieve this goal, we incorporate the cost of communication between multiple layers within a chiplet. As NoC is the standard interconnect fabric used in the SoC-domain [104], we consider using an NoC for intra-chiplet communication. We customize a cycle-accurate NoC simulator, BookSim [98], to evaluate NoC performance. First, we generate a trace file for each chiplet following Algorithm 2. The algorithm considers the number of tiles, input activations, chiplets, layer-to-chiplet mapping, quantization bit-precision, and bus width. We then determine the source and destination tile information for each layer in each chiplet. Next, we calculate the number of packets for each source-destination pair and generate a trace file as a tuple consisting of the source tile ID, destination tile ID, and timestamp. Finally, we simulate the trace file using BookSim to obtain the area, energy, and latency for on-chip communication within each chiplet.

3.5 NoP engine

Using specialized signaling techniques and driver circuits, the NoP handles on-chip data movement between different chiplets. This is achieved through a silicon interposer or organic substrate, as shown in [105, 106]. Figure 11 (left) shows the cross-sectional image of a 2.5D integration with chiplets and an interposer. However, modeling the NoP’s performance can be challenging due to its complex interconnect structure, specialized driver architectures, and corresponding signaling techniques. To accurately estimate performance, the NoP engine models each component of the NoP. Figure 11 (right) shows different NoP implementations with the corresponding energy-per-bit $\left({\mathrm{E}}_{\mathit{bit}}\right)$ proposed in prior works. The NoP performance evaluation comprises two main components: (1) NoP latency estimation and (2) NoP area and power estimation. To estimate NoP latency, the engine uses a cycle-accurate simulator to evaluate the interconnect. It generates the NoP trace using Algorithm 2, similar to the one used for NoCs, based on the chiplet-to-chiplet data volume generated by the partition and mapping engine. The generated traces are simulated using a cycle-accurate simulator or the NoP estimator to determine the latency of the NoP interconnect. To estimate NoP area and power consumption, the engine obtains interconnect parameters such as wire length, pitch, width, and stack-up. The engine then determines the interconnect capacitance and resistance using the PTM interconnect models [107]. Based on these parameters, it generates timing parameters for the interconnect and compares them to the target bandwidth. If the timing parameters do not meet the bandwidth requirements, the NoP engine chooses the maximum allowable bandwidth.

Figure 11.

(Left) Cross-sectional image of the NoP interconnect. The NoP is routed within the interposer connecting different chiplets across the architecture. Μ bumps connect the chiplets to the interposer, (Right) Energy per bit for different NoP driver circuit and signaling techniques proposed in prior works.

The NoP engine evaluates the NoP transmitter/receiver (TX/RX) circuits, which includes clocking circuitry, based on the energy per bit (Ebit), number of TX/RX channels, bandwidth, chiplet-to-chiplet data volume, and operating frequency to estimate the energy and latency cost of the TX/RX circuits. The energy calculation for the NoP driver is provided in Algorithm 3. The NoP engine first calculates the number of bits being transferred between chiplets. It then retrieves the energy per bit $\left({\mathrm{E}}_{\mathit{bit}}\right)$ from previous research, which is illustrated in Figure 11 (right). The total energy for the TX/RX channel is computed by multiplying the number of bits and the energy per bit, as indicated in line 9 of Algorithm 3. Afterward, the NoP engine determines the NoP driver area using the NoP driver area cost from previous implementations (Figure 11). Finally, the engine integrates the interconnect and driver performance metrics to determine the overall NoP performance.

The functional flow of the NoP engine is summarized below:

• NoP trace generation based on the chiplet placement, inter-chiplet data transfer volume, and inter-chiplet layer partition,

• NoP interconnect evaluation using a cycle-accurate simulator to generate energy, latency, and area metrics,

• NoP router modeling and TX/RX driver based on real measurements. Finally, the NoP engine combines the NoP driver and the interconnect metrics to generate the overall NoP performance.

3.6 DRAM engine

The architecture of the IMC based on chiplets includes a DRAM chiplet that serves as the external memory for the IMC chiplets. The DRAM engine estimates the external memory access required for this architecture. The assumption is that the DRAM will transfer the entire set of weights to the chiplet only once before performing the inference task. This assumption remains constant across different architectural configurations and inference runs for a specific DNN model. The DRAM engine includes a DRAM request generator, RAMULATOR [108] for estimating the latency for DRAM transactions, and VAMPIRE [109] to estimate the DRAM transaction power. The DRAM choice depends on user input and currently supports DDR3 and DDR4 [110, 111]. For a given DNN model, the DRAM engine generates the required traces and memory requests with timestamps, which include the location within the DRAM memory and the operation. SIAM uses a customized version of the cycle-accurate simulator RAMULATOR and the model-based power analysis tool VAMPIRE. To reduce simulation time, the DRAM engine estimates smaller sets of instructions, which are then multiplied by the total number of sets required to represent all the weights in the DNN. An experiment was performed to calibrate the method (Figure 12(a) and (b)), which showed that a reduction of 50% of DRAM instructions to the engine results in less than 2% EDP accuracy degradation than that at 100% instructions. The overall EDP for different networks across different datasets for DDR4 shows an exponential increase in EDP with the increase in the model size of the DNN. The DRAM engine provides a fast and accurate estimation of external memory access for the entire range of DNNs through this method. To summarize, the DRAM engine’s execution involves the following essential steps:

• Generate DRAM requests based on the data precision and DNN model size,

• Calculate the latency cost of DRAM transactions using a customized version of RAMULATOR, and calculate the power consumption by utilizing a customized version of VAMPIRE,

• Combine the results to produce the total cost of DRAM access.

Figure 12.

(a) The accuracy of EDP prediction for different numbers of instructions processed to represent 3000 DRAM instructions. Reduction in the number of instructions to half results in less than 2% EDP accuracy degradation for half the simulation time. (b) EDP of DRAM transactions (DDR4) for different DNNs. There is an exponential increase in DRAM cost with an increase in DNN model size.

4. SIAM dataflow

This section outlines the default dataflow of the SIAM chiplet-based IMC architecture. The computation dataflow is illustrated in Figure 13. Before starting the inference task, the weights are obtained from the DRAM and allocated to the IMC chiplets according to the partition and mapping engine described in Section III-C. There are two possible scenarios based on this: either no layer is spread across several chiplets, or a layer is distributed across multiple chiplets. Let us consider the case where layer N of the DNN is assigned to the first chiplet, as depicted in Figure 13(a). During computation, the entire layer is processed within one chiplet, generating the computed output activations for layer N. The global accumulator and buffer are not utilized during this process and are turned off. When the computation is done, the output activations are sent to the chiplets that execute layer N + 1. If two chiplets are required to map the weights for layer N + 1, the NoP transfers the output activation from layer N to both chiplets, as shown in Figure 13(a). Figure 13(b) presents the computation flow for layer N + 1, where both chiplets execute computations in parallel. The mapping ensures that an equal number of weights are assigned to each chiplet, avoiding any workload imbalances. After the computation is completed, the generated partial sums are aggregated using the global accumulator and buffer. Then, the accumulated outputs from layer N + 1 are transferred to the chiplets that hold the weights for layer N + 2. This process is repeated until all layers are processed and the final output is generated. The algorithmic implementation of the dataflow used in the SIAM IMC chiplet architecture is explained in Algorithm 4.

Figure 13.

Computation dataflow within the chiplet-based IMC architecture in SIAM. Two cases arise: (a) no layer is partitioned across two or more chiplets and (b) a layer is partitioned across two or more chiplets.

5. Conclusion

In this chapter, we discussed benchmarking chiplet-based IMC-based AI accelerators. We discuss various IMC architectures proposed in the literature. CMOS- (e.g., SRAM) and memristor- (e.g., RRAM) based IMC architectures are discussed. Although IMC improves the energy efficiency of computing elements, it increases on-chip communication volume. To address this, we discuss chiplet-based in-memory architectures. We also discuss different benchmarking simulators for monolithic and chiplet-based IMC architectures in detail. Finally, we dive deeply into SIAM, a chiplet-based IMC benchmarking simulator. SIAM provides a unified framework for performance benchmarking of chiplet-based IMC architectures. SIAM supports both homogeneous (generic) and custom chiplet-based IMC architectures. Finally, SIAM interfaces with popular deep learning frameworks such as PyTorch and TensorFlow and can be integrated with modern NAS techniques.