Scaling Subspace-Driven Approaches Using Information Fusion

Abstract

In this work, we seek to exploit the deep structure of multi-modal data to robustly exploit the group subspace distribution of the information using the Convolutional Neural Networks (CNNs) formalism. Upon unfolding the set of subspaces constituting each data modality, and learning their corresponding encoders, an optimized integration of the generated inherent information is carried out to yield a characterization of various classes. Referred to as deep Multimodal Robust Group Subspace Clustering (DRoGSuRe), this approach is compared against the independently developed state-of-the-art approach named Deep Multimodal Subspace Clustering (DMSC). Experiments on different multimodal datasets show that our approach is competitive and more robust in the presence of noise.

1. Introduction

Unsupervised learning is a very challenging topic in Machine Learning (ML) and involves the discovery of hidden patterns in data for inference with no prior given labels. Reliable clustering techniques will save time and effort required for classifying/labeling large datasets that might have thousands of observations. Multi-modal data, increasingly in need for complex application problems, have become more accessible with recent advances in sensor technology, and of pervasive use in practice. The plurality of sensing modalities in our applications of interest, provides diverse and complementary information, necessary to capture the salient characteristics of data and secure their unique signature. A principled combination of the information contained in the different sensors and at different scales is henceforth pursued to enhance understanding of the distinct structure of the various classes of data. The objective of this work is to develop a principled multi-modal framework for object clustering in an unsupervised learning scenario. We extract key class-distinct features-signatures from each data modality using a CNNs encoder, and we subsequently non-linearly combine those features to generate a discriminative characteristic feature. In so doing, we work on the hypothesis that each data modality is approximated by a Union of low dimensional Subspaces which highlights underlying hidden features. The UoS structure is unveiled by pursuing sparse self-representation of the given data modality. The subsequent aggregation of the multi-modal subspace structures yields a jointly unified characteristic subspace for each class.

1.1 Related work

Subspace clustering has been introduced as an efficient way for unfolding union of low-dimensional subspaces underlying high dimensional data. Subspace clustering has been extensively studied in computer vision due to the vast availability of visual data as in [1, 2, 3, 4]. This paradigm has broadly been adopted in many applications such as image segmentation [5], image compression [6], and object clustering [7].

Uncovering the principles and laying out the fundamentals for multi-modal data has become an important topic in research in light of many applications in diverse fields including image fusion [8], target recognition [9, 10, 11, 12], speaker recognition [13], and handwriting analysis [14]. Convolutional neural networks have been widely used on multi-modal data as in [15, 16, 17, 18, 19]. A multi-modal subspace clustering-inspired approach was also proposed in [20]. The emphasis of our formulation results in a different optimization problem, as the multi-modal sensing seeks to not only account for the private information which provides the complementarity of the sensors, but also the common and hidden information. This yields, as an end result, a different network structure than that of [20] with a different application space inspiration. In addition, the robustness of fusing multi-modal sensor data each with its distinct intrinsic structure, is addressed along with a potential scaling for viability. A thorough comparison of our results to the multimodal fusion network in [21] is carried out, with a demonstration of resilient fusion under a variety of limiting scenarios including limited sensing modalities (sensor failures). In [22], the authors proposed a deep multi-view subspace clustering approach that combined global and local structures to help achieve a small distance between samples of the same cluster and make samples in different clusters of different views farther. To that end, they used a discriminative constraint between different views. The discriminative constraint is based on the Hadamard product between the features extracted by the convolutional auto-encoder for the different views. In contrast, our approach is based on the minimizing the group norm, which we proved with a derivation in earlier work [23] and entails a smaller angle between the different subspaces across all modalities, thus promoting the goal of obtaining a common latent space. Moreover, minimizing the group norm also provides as well as group sparse solution along data modalities. Sun et al. [24] proposed a deep trainable multi-view subspace clustering method, named self-supervised deep multi-view subspace clustering (S2DMVSC) that learns the common latent subspace using two losses: spectral clustering loss and classification loss in order to denoise the imperfect correlations among data points.

In this paper, we prove that our formulation, which is based on the group norm of the self-representation matrices and the commutation loss between them, provides a natural way to fuse multi-modal data by employing the self-representation matrix as an embedding for each data modality, making our approach robust under different types of potential limitations. It is good to note that our proposed approach secures the individual sensor data-points relations resulting in more flexibility for each data modality.

1.2 Contributions

Building on the work of Deep Subspace Clustering (DSC) [25], we propose a new and principled multi-modal fusion approach which accounts for a sensor’ capacity to house private and unique information about some observed data as well as that information which is likely also captured and hence common to other sensors. This is accounted for in our robust fusion formulation for multi-modal sensor data. Unveiling the complex UoS of multi-modal data also requires us to account for scaling in our proposed formulation and solution, which in turn invokes the learning of multiple/deep scale Convolutional Neural Networks. Our proposed Multi-modal fusion approach, by virtue of each sensor information structure (i.e., private plus shared) seeks to enhance and robustify the subspace approximation of shared information for each of the sensors, thus yielding a parallel bank of UoS for each of the sensors. The robust Deep structure effectively achieves scaling while securing structured representation for unsupervised inference. We compare our approach to a well-known deep multimodal network [21] which was also based on [25].

In our proposed approach, we thus define the latent space in a way that safeguards the individual sensor private information which hence dedicates more degrees of freedom to each of the sensors. In contrast to the approach in [21]. In our evaluation, we use two recently released data sets each of which we partition into learning and validation subsets. The learned UoS structure for each of the data sets is then utilized to classify new observed data points, which illustrates the generalization power of the proposed approach. Different scenarios with corresponding additive noise to either the training set or the testing set, or both, were used to thoroughly investigate the robustness, and resilience of the clustering approach performance. Experimental results confirm a significant improvement for our Deep Robust Group Subspace Recovery network (DRoGSuRe) under numerous limiting scenarios and demonstrate robustness under these conditions.

The balance of the paper is organized as follows, in Section 2, we provide the problem formulation, background along with the derivation for our proposed approach, Deep Robust Group Subspace Recovery (DRoGSuRe). In Section 3, we describe the attributes of the proposed approach and contrast it to Deep Multimodal Subspace Clustering algorithm (DMSC). In Sections 4 and 5, we present a substantiative validation along with experimental results of our approach, while Section 6 provides concluding remarks.

2. Deep robust group subspace clustering

2.1 Problem formulation

We assume having a set of data observations, each represented as a $\mathit{m}$dimensional vector ${\mathit{x}}_{\mathit{k}}\left(\mathit{t}\right)\in {\mathbb{R}}^{\mathit{m}}$, where $\mathit{k}\mathit{=}\mathbf{1}\mathit{,}\mathbf{2}\mathit{,}\dots \mathit{,}\mathit{n}$. Moreover, we consider having $\mathit{T}$ data modalities, indexed by $\mathit{t}\mathit{=}\mathbf{1}\mathit{,}\mathbf{2}\mathit{,}\mathbf{3}\mathit{,}\dots \mathit{,}\mathit{T}$. Each data modality can then be described as ${\mathit{x}}_{\mathit{k}}\left(\mathit{t}\right)\in {\mathbb{R}}^{\mathit{m}}$, where $\mathit{X}\left(\mathit{t}\right)\mathit{=}\left[{\mathit{x}}_{\mathbf{1}}\left(\mathit{t}\right){\mathit{x}}_{\mathbf{2}}\left(\mathit{t}\right)\dots {\mathit{x}}_{\mathit{n}}\left(\mathit{t}\right)\right]\mathit{.}$ Our objective is to assign each set of data observations into clusters that can be efficiently represented by a low-dimensional subspace. This is equivalent to finding a partitioning $\left\{{\mathit{X}}^{\mathbf{1}}\left(\mathit{t}\right),{\mathit{X}}^{\mathbf{2}}\left(\mathit{t}\right),\dots ,{\mathit{X}}^{\mathit{P}}\left(\mathit{t}\right)\right\}$ of $\left[\mathit{n}\right]$ observations, where $\mathit{P}$ is the total number of clusters underlying each data modality indexed by $\mathit{p}$. Furthermore, each linear subspace can be described as ${\mathit{S}}^{\mathit{p}}\left(\mathit{t}\right)\subset {\mathbb{R}}^{\mathit{m}}$ with dim ${\mathit{S}}^{\mathit{p}}\left(\mathit{t}\right)\ll \mathit{m}$.

We will exploit the self-expressive property presented in [1, 26], which entails that each data observation ${\mathit{x}}_{\mathit{i}}\left(\mathit{t}\right)$ can be represented as a linear combination of all features from the same subspace $\mathit{S}\left({\mathit{x}}_{\mathit{i}}\left(\mathit{t}\right)\right)$ as follows,

$${\mathit{x}}_{\mathit{i}}\left(\mathit{t}\right)=\sum _{i\ne j,{\mathit{x}}_{\mathit{j}}\left(t\right)\in S\left({\mathit{x}}_{\mathit{i}}\left(t\right)\right)}{\mathit{w}}_{\mathit{ij}}\left(t\right){\mathit{x}}_{\mathit{j}}\left(t\right).$$E1

If we stack all the data points ${\mathit{x}}_{\mathit{i}}\left(\mathit{t}\right)$ into columns of the data matrix $\mathit{X}\left(\mathit{t}\right)\mathit{.}$ The self-expressive property can be written in a matrix form as follows,

$$\mathit{X}\left(\mathit{t}\right)=\mathit{X}\left(\mathit{t}\right)\mathit{W}\left(\mathit{t}\right)\mathit{s}.\mathit{t}.{\mathit{W}}_{\mathit{ii}}=\mathbf{0}.$$E2

The important information about the relations among data samples is then recorded in the self-representation coefficient matrix $\mathit{W}\left(\mathit{t}\right)$. Under a suitable arrangement/permutation of the data realizations, the sparse coefficient matrix $\mathit{W}\left(\mathit{t}\right)$ is an $\mathit{n}\mathit{\times }\mathit{n}$ block-diagonal matrix with zero diagonals provided that each sample is represented by other samples only from the same subspace. More precisely, ${\mathit{W}}_{\mathit{ij}}\left(\mathit{t}\right)\mathit{=}\mathbf{0}$ whenever the indexes $\mathit{i}\mathit{,}\mathit{j}$ correspond to samples from different subspaces. As a result, the majority of the elements in $\mathit{W}$ are equal to zero. A diagram showing our algorithm is depicted in Figure 1.

Figure 1.

Deep robust group subspace clustering diagram.

Our algorithm consists of three main stages; the first stage is the encoder which encodes the input modalities into a latent space. The encoder consists of $\mathit{T}$parallel CNNs, where $\mathit{T}$ is the number of data modalities. Each modality data is fed into one network, and the output of each network represents the modality data projection into its corresponding hidden/latent space. The second component of the auto encoder is $\mathit{T}$ self-expressive layers, the goal of which is to enforce the self-expressive property among the data observations of each data modality. Each self-expressive layer is a fully connected layer which independently operates on the output of each encoder. The last stage is the decoder which reconstructs input data from the self-expressive layers’ output. The objective function sought through this approximation network is reflected in Eq. (5). The group sparsity introduced in [23] requires the minimization of the group norm of matrices $\mathit{W}\left(\mathit{t}\right)$, which in turn, entails a smaller angle between the different spaces across all modalities, thus promoting the goal of obtaining a common latent space. Note that minimizing group norm provides a group sparse solution along data modalities. If we in addition, constrain the coefficient matrices corresponding to each data modality to commute, therefore, we ensure their sharing the same eigen vectors. The idea of commutation has been used in [27, 28, 29]. We define $\mathit{\Omega }$ = ${\left\{\mathit{W}\left(\mathit{t}\right)\right\}}_{\mathit{t}=\mathbf{1}}^{\mathit{T}}$, where $\mathit{W}\left(\mathit{t}\right)$ = ${\left[{\mathit{w}}_{\mathit{kj}}\left(\mathit{t}\right)\right]}_{\mathit{k},\mathit{j}}$ and the group l-norm ${‖\mathbf{\Omega }‖}_{\mathbf{1},\mathbf{2}}$ as:

$${‖\Omega ‖}_{1,2}=\sum _{k,j}\sqrt{\sum _{t=1}^T{w}_{k,j}^2\left(t\right)}.$$E3

We also define [$\mathit{W}\left({\mathit{t}}_{\mathbf{1}}\right),\mathit{W}\left({\mathit{t}}_{\mathbf{2}}\right)]$ as,

$$\left[\mathit{W}\left(t_1\right),\mathit{W}\left(t_2\right)\right]=\mathit{W}\left(t_1\right)\mathit{W}\left(t_2\right)-\mathit{W}\left(t_2\right),\mathit{W}\left(t_1\right)=0$$E4

The loss function is then rewritten as,

$$\begin{array}{l}{\mathit{min}}_{\mathit{W}\left(t\right)/w_{\mathit{kk}}\left(t\right)=0}\sum _{t_1,t_2}^T{‖\left[\mathit{W}\left(t_1\right),\mathit{W}\left(t_2\right)\right]‖}^2+{‖\mathit{\Omega }‖}_{1,2}\\ +\frac{\gamma }{2}\sum _{t=1}^T{‖\mathit{X}\left(t\right)-{\mathit{X}}_{\mathit{r}}\left(t\right)‖}_F^2\\ +\rho \sum _{t=1}^T{‖\mathit{W}\left(t\right)‖}_1+\frac{\mu }{2}\sum _{t=1}^T{‖\mathit{L}\left(t\right)-\mathit{L}\left(t\right)\mathit{W}\left(t\right)‖}_F^2\end{array}$$E5

where ${\mathit{X}}_{\mathit{r}}\left(\mathit{t}\right)$ represent the reconstructed data corresponding to modality $\mathit{t}$, and $\mathit{L}\left(\mathit{t}\right)$ is the output of the ${\mathit{t}}^{\mathit{th}}$ encoder with input $\mathit{X}\left(\mathit{t}\right)$. $\mathit{W}\left(\mathit{t}\right)$ is the sparse weight function that ties the data observation for modality $\mathit{t}$. Solving DRoGSuRe in Tensorflow and using the adaptive momentum based gradient descent method (ADAM) [30] results in minimizing the loss function. For each data modality, the weights of the encoder, the self-expressive layer and the decoder are individually calculated, however, fine-tuning the weights is based on the loss function, which is a function of the group norm and the pairwise product difference between sparse coefficient matrices. ${‖‖}_{\mathbf{1}}$ denotes the ${\mathit{l}}_{\mathbf{1}}$ norm, i.e., the sum of absolute values of the argument. The Lagrangian objective functional may be rewritten as,

$$\begin{array}{l}L\left(\mathit{W}\left(t\right)\right)=\sum _{t_1,t_2}^T{‖\left[\mathit{W}\left(t_1\right),\mathit{W}\left(t_2\right)\right]‖}^2+{‖\mathit{\Omega }‖}_{1,2}\\ +\rho \sum _{t=1}^T{‖\mathit{W}\left(t\right)‖}_1+\frac{\gamma }{2}\sum _{t=1}^T\Vert \mathit{X}\left(t\right){-{\mathit{X}}_{\mathit{r}}\left(t\right)\Vert }_F^2\\ +\sum _{t=1}^T\frac{\mu }{2}{‖\mathit{L}\left(t\right)\mathit{W}\left(t\right)-\mathit{L}\left(t\right)‖}_F^2\\ +\sum _{t=1}^T<\mathit{L}\left(t\right)\mathit{W}\left(t\right)-\mathit{L}\left(t\right),\mathit{Y}\left(t\right)>\end{array}$$E6

Assume $\widehat{\mathit{W}}\left(\mathit{t}\right)=\mathit{I}-\mathit{W}\left(\mathit{t}\right)$, we update $\mathit{W}\left(\mathit{t}\right)$ as follows,

$$\begin{array}{l}{\mathit{W}}_{\mathit{k}+\mathbf{1}}\left(t\right)=\mathit{arg}{\mathit{min}}_{\mathit{W}\left(t\right)}\sum _{t_1,t_2}^T{‖\left[\mathit{W}\left(t_1\right),\mathit{W}\left(t_2\right)\right]‖}^2+{‖\mathit{\Omega }‖}_{1,2}+\rho {‖\mathit{W}\left(t\right)‖}_1+\\ <{\mathit{L}}_{\mathit{k}+\mathbf{1}}\left(t\right)\mathit{W}\left(t\right)-{\mathit{L}}_{\mathit{k}+\mathbf{1}}\left(t\right),{\mathit{Y}}_{\mathit{k}}\left(t\right)\\ >+\frac{{\mu }_k}{2}{‖{\mathit{L}}_{\mathit{k}+\mathbf{1}}\left(t\right)\mathit{W}\left(t\right)-{\mathit{L}}_{\mathit{k}+\mathbf{1}}\left(t\right)‖}_F^2\end{array}$$E7

Similar to [4], we utilize linearized ADMM [31] to approximate the minimum of Eq. (7) since the algorithmic solution is complicated and yields a non-convex optimization functional. It has been shown that linearized ADMM is very effective for ${\mathit{l}}_{\mathbf{1}}$ minimization problems and the augmented Lagrange multiplier (ALM) method can take care of the non-convexity of the problem [32, 33]. Therefore, utilizing an appropriate augmented Lagrange multiplier ${\mathit{\mu }}_{\mathit{k}}$, we can compute the global optimizer by solving the dual problem. The solution to Eq. (7) can be approximated, using linearized soft thresholding, as follows,

$${\mathit{W}}_{\mathit{k}}^{+}\left(t\right)={\mathit{\text{prox}}}_{\frac{\rho }{\mu {\eta }_1}}({\mathit{W}}_{\mathit{k}}\left(t\right)+\frac{{\mathit{L}}_{\mathit{k}+\mathbf{1}}^{\mathit{T}}\left({\mathit{L}}_{\mathit{k}+\mathbf{1}}{\widehat{\mathit{W}}}_{\mathit{k}}\left(t\right)-\frac{{\mathit{Y}}_{\mathit{k}}\left(t\right)}{{\mu }_k}\right)}{{\eta }_1}+\sum _{t_1,t_2=1,t_1\ne t_2}^T\left\{\left({\mathit{W}}_{\mathit{k}}\left(t_1\right){\mathit{W}}_{\mathit{k}}\left(t_2\right)-{\mathit{W}}_{\mathit{k}}\left(t_2\right){\mathit{W}}_{\mathit{k}}\left(t_1\right)\right)W_k^T\left(t_2\right)+W_k\left(m\right)\left({\mathit{W}}_{\mathit{k}}\left(t_1\right){\mathit{W}}_{\mathit{k}}\left(t_2\right)-{\mathit{W}}_{\mathit{k}}\left(t_2\right){\mathit{W}}_{\mathit{k}}\left(t_1\right)\right)\right\}$$E8

$${\mathit{W}}_{\mathit{k}+\mathbf{1}}\left(t\right)={\gamma }_{\frac{\rho }{\mu {\eta }_2}}\left({\mathit{W}}_{\mathit{k}}^{+}\left(t\right)\right)$$E9

where ${\mathit{\eta }}_{\mathbf{1}}\ge {‖\mathit{L}‖}_{\mathbf{2}}^{\mathbf{2}}$. We alternatively update $\mathit{L}\left(\mathit{t}\right)$ as,

$${\mathit{L}}_{\mathit{k}+\mathbf{1}}\left(t\right)={\mathit{L}}_{\mathit{k}}\left(t\right)+{\mu }_k\left({\mathit{L}}_{\mathit{k}}\left(t\right){\widehat{\mathit{W}}}_{k+1}\left(t\right)-\frac{Y_k\left(t\right)}{{\mu }_k}\right){\widehat{W}}_{k+1}^T\left(t\right).$$E10

where ${\mathit{\text{prox}}}_{\mathit{\beta }}\left({\mathit{A}}_{\mathit{i},\mathit{j}}\left(\mathit{t}\right)\right)={\mathit{A}}_{\mathit{i},\mathit{j}}\left(\mathit{t}\right)\ast \frac{max\left\{\left(\sqrt{\sum _{\mathit{t}=\mathbf{1}}^{\mathit{T}}{\mathit{A}}_{\mathit{i},\mathit{j}}{\left(\mathit{t}\right)}^{\mathbf{2}}}-\mathit{\beta }\right),\mathbf{0}\right\}}{\sqrt{\sum _{\mathit{t}=\mathbf{1}}^{\mathit{T}}{\mathit{A}}_{\mathit{i},\mathit{j}}{\left(\mathit{t}\right)}^{\mathbf{2}}}}$ and ${\mathit{\gamma }}_{\mathit{\tau }}\left({\mathit{B}}_{\mathit{i},\mathit{j}}\right)=\mathit{sign}\left({\mathit{B}}_{\mathit{i},\mathit{j}}\right)\ast max\left\{\left(\left|{\mathit{B}}_{\mathit{i},\mathit{j}}\right|-\mathit{\tau }\right),\mathbf{0}\right\}$. The Lagrange multipliers are updated as follows,

$${\mathit{Y}}_{\mathit{k}+\mathbf{1}}\left(t\right)={\mathit{Y}}_{\mathit{k}}\left(t\right)+{\mu }_k\left({\mathit{L}}_{\mathit{k}+\mathbf{1}}\left(t\right){\mathit{W}}_{\mathit{k}+\mathbf{1}}\left(t\right)-{\mathit{L}}_{\mathit{k}+\mathbf{1}}\left(t\right)\right)$$E11

$${\mu }_{k+1}=ϵ{\mu }_k$$E12

After computing the gradient of the loss function, the weights of each multi-layer network, that corresponds to one modality, are updated while other modalities’ networks are fixed. In other words, after constructing the data during the forward pass, the loss function determines the updates that back-propagates through each layer. The encoder of the first modality is updated, afterwards, the self-expressive layer of that modality gets updated and finally the decoder. Since the weights corresponding to each modality are dependent on other modalities, we update each part of the network corresponding to each modality with the assumption that all other networks’ components corresponding to other modalities are fixed. The resulting sparse coefficient matrices $\mathit{W}\left(\mathit{t}\right)$’s, for $\mathit{t}\mathit{=}\mathbf{1}\mathit{,}\mathbf{2}\mathit{,}\dots \mathit{,}\mathit{T}$ are then integrated as follows,

$${\mathit{W}}_{\mathit{\text{Total}}}=\sum _{t=1}^T\mathit{W}\left(t\right)$$E13

Integrating the sparse coefficient matrices helps reinforcing the relation between data points that exist in all data modalities, thus establishing a cross-sensor consistency. Furthermore, adding the sparse coefficient matrices reduces the noise variance introduced by the outliers. A similar approach was introduced in [34] for Social Networks community detection, where an aggregation of multi-layer adjacency matrices was proved to provide a better Signal to Noise ratio, and ultimately better performance. To proceed with distinguishing the various classes in unsupervised manner, we construct the affinity matrix as follows,

$$\mathit{A}={\mathit{W}}_{\mathit{\text{Total}}}+{\mathit{W}}_{\mathit{\text{Total}}}^{\mathit{T}}$$E14

where $\mathit{A}\in {\mathbb{R}}^{\mathit{n}\mathit{\times }\mathit{n}}$. We subsequently use the spectral clustering method [35] to retrieve the clusters in the data using the above affinity matrix as input.

2.2 Theoretical discussion

In order to justify the multiple banks of self-expressive layers, we assume that each modality $\mathit{X}\left(\mathit{t}\right)$ may be expressed as a private information contribution ${\mathit{X}}_{\mathit{p}}\left(\mathit{t}\right)$ and a shared information ${\mathit{X}}_{\mathit{s}}\left(\mathit{t}\right)$ such that,

$$\mathit{X}\left(t\right)={\mathit{X}}_{\mathit{s}}\left(t\right)+{\mathit{X}}_{\mathit{p}}\left(t\right)$$E15

The shared information can be represented as follows,

$${\mathit{X}}_{\mathit{s}}\left(t\right)=\sum _{t=1}^{T}F\left(\mathit{W}\left(t\right)\left({\Pi }_s\mathit{X}\left(t\right)\right)\right)$$E16

where ${\mathit{\Pi }}_{\mathit{s}}={\cap }_{\mathit{t}=\mathbf{1},\dots ,\mathit{T}}$${\mathit{\Pi }}_{\mathit{s}}^{\mathit{t}}$. ${\mathit{X}}_{\mathit{s}}\left(\mathit{t}\right)$ and ${\mathit{X}}_{\mathit{p}}\left(\mathit{t}\right)$ are distinct and will hence lie in different subspaces, which will hence be mapped to different components in $\mathit{W}\left(\mathit{t}\right)$. Similarly for the subspaces spanned by ${\mathit{X}}_{\mathit{p}}\left({\mathit{t}}_{\mathit{i}}\right)$ and ${\mathit{X}}_{\mathit{p}}\left({\mathit{t}}_{\mathit{j}}\right)$, $\mathit{i}\ne \mathit{j}$, the corresponding components of $\mathit{W}\left({\mathit{t}}_{\mathit{i}}\right)$ and $\mathit{W}\left({\mathit{t}}_{\mathit{j}}\right)$ will almost surely not coincide. On the other hand, the components of $\mathit{W}\left({\mathit{t}}_{\mathit{i}}\right)$ and $\mathit{W}\left({\mathit{t}}_{\mathit{j}}\right)$ corresponding to ${\mathit{X}}_{\mathit{s}}\left({\mathit{t}}_{\mathit{i}}\right)$ and ${\mathit{X}}_{\mathit{s}}\left({\mathit{t}}_{\mathit{j}}\right)$ will almost surely coincide, thus justifying the construction of a layered ${\mathit{W}}_{\mathit{\text{Total}}}$, and thereby improving the SNR. In addition, the decoder will help protect and maintain the private information corresponding to each modality ${\mathit{X}}_{\mathit{p}}\left(\mathit{t}\right)$ by ensuring that data can be reconstructed again from the latent space with minimal loss. In the following, we will elaborate more on how aggregating affinity matrices should impact the overall clustering performance. The idea of aggregating affinity matrices is not new, in fact, it has been used extensively in clustering and community detection field. For example, in [36], the authors proposed a method that combines the self-similarity matrices of the eigenvectors after applying a Singular Value Decomposition on clusters. In [37], they proposed merging the information provided by the multiple modalities by combining the characteristics of individual graph layers using tools from subspace analysis on a Grassmann manifold. In [38], they propose a multilayer spectral graph clustering (SGC) framework that performs convex layer aggregation.

Proposition: The persistent differential scaling of $\mathit{m}$-modal Group Robust Subspace Clustering Fusion yields an order $\mathit{m}$-improvement resilience over the singly differential scaling fusion.

The proof of the proposition can be found in Appendix A. We basically show that by perturbing one or more data modalities, our proposed approach introduces less error to the overall affinity matrix as compared to DMSC. Hence, preserving the performance and yielding a graceful degradation of the clustering accuracy as an increasing number of modalities get corrupted by noise.

3. Affinity fusion deep multimodal subspace clustering

For completeness, we provide a brief overview of the Deep Multimodal Subspace Clustering algorithm which was proposed in [4]. As noted earlier for DRoGSuRe and similarly for Affinity Fusion Deep Multimodal Subspace clustering (AFDMSC), the network is composed of three main parts: a multimodal encoder, a self-expressive layer, and a multimodal decoder. The output of the encoder contributes to a common latent space for all modalities. The self-expressiveness property applied through a fully connected layer between the encoder and the decoder results in one common set of weights for all the data sensing modalities. This marks a divergence in defining the latent space with DRoGSuRe. Our proposed approach, as a result, safeguards the private information ${\mathit{X}}_{\mathit{p}}\left(\mathit{t}\right)$; $\mathit{t}\mathit{=}\mathbf{1}\mathit{,}\dots \mathit{,}\mathit{T}$ individually for each of the sensors, i.e., dedicating more degrees of freedom for each of the sensors. This contrasts with AFDMSC. The reconstruction of the input data by the decoder, can yield the following loss function to secure the proper training of the self-expressive network,

$${\mathit{min}}_{W\mid w_{\mathit{kk}}=0}{‖\mathit{W}‖}_2+\frac{\gamma }{2}\sum _{t=1}^T{‖\mathit{X}\left(t\right)-{\mathit{X}}_{\mathit{r}}\left(t\right)‖}_F^2+\frac{\mu }{2}\sum _{t=1}^T{‖\mathit{L}\left(t\right)-\mathit{L}\left(t\right)\mathit{W}‖}_F^2,$$E17

where $\mathit{W}$ represents the parameters of the self expressive layer, $\mathit{X}\left(\mathit{t}\right)$ is the input to the encoder, ${\mathit{X}}_{\mathit{r}}\left(\mathit{t}\right)$ denote the output of the decoder and $\mathit{L}\left(\mathit{t}\right)$ denotes the output of the encoder. $\mathit{\mu }$ and $\mathit{\gamma }$ are regularization parameters. An overview for the DMSC approach is illustrated in Figure 2.

Figure 2.

Deep multimodal subspace clustering diagram.

4. Experimental results

4.1 Dataset description

We will evaluate our approach on two different datasets. The first dataset we will use is the Extended Yale-B dataset [39]. The same dataset has been used extensively in subspace clustering as in [1, 40]. The dataset is composed of 64 frontal images of 38 individuals under different illumination conditions. In this work, we will use the augmented data used in [4], where facial components such as left eye, right eye, nose and mouth have been cropped to represent four additional modalities. Images corresponding to each modality have been cropped to a size of 32$\mathit{\times }$32. A sample image for each modality is shown in Figure 3. The second validation dataset we use is the ARL polarimetric face dataset [41]. This consists of facial images for 60 individuals in the visible domain and in four different polarimetric states. All the images are spatially aligned for each subject. We have also resized the images to 32$\mathit{\times }$32 pixels. Sample images from this dataset are shown in Figure 4.

Figure 3.

Sample images from the augmented extended Yale-B Dataset. (a) Face. (b) Left eye. (c) Right eye. (d) Mouth. (e) Nose.

Figure 4.

Sample images from the ARL polarimetric dataset. (a) Visible. (b) DoLP. (c) S0. (d) S1. (e) S2.

4.2 Network structure

In the following, we will elaborate on how we construct the neural network for each dataset. Similar to [4], we implemented DRoGSuRe with Tensorflow and used the adaptive momentum based gradient descent method (ADAM) [30] to minimize the loss function in Eq. (5) with a learning rate of ${\mathbf{10}}^{\mathit{-}\mathbf{3}}$.

In case of ARL dataset, we have five data modalities and will therefore have 5 different encoders, self-expressive layers and decoders. Each encoder is composed of three neural layers. The first layer consists of 5 convolutional filters of kernel size 3. The second layer has 7 filters of kernel size 1. The last layer has 15 filters with kernel size equals 1.

For EYB dataset, we also have five data modalities, therefore, we have 5 different encoders, self-expressive layers and decoders. Each encoder is composed of three neural layers. The first layer consists of 10 convolutional filters of kernel size 5. The second layer has 20 filters of kernel size 3. The last layer has 30 filters of kernel size 3.

4.3 Noiseless results

In the following, we compare the performance of our approach versus the DMSC approach when learning the union of subspaces structure of noise-free data. First, we divide each dataset into training and validation sets to be able to classify a newly observed dataset, using the structure learned through the current unlabeled data. The ARL expression dataset used for training consists of 2160 images per modality. The validation baseline images include 720 images total per modality. For the EYB, we randomly selected 1520 images per modality for training and 904 images for validation. The sparse solution $\mathit{W}\left(\mathit{t}\right)$ corresponding to each data modality, provides important information about the relations among data points, which may be used to split data into individual clusters residing in a common subspace. Observations from each object can be seen as data points spanning one subspace. Interpreting the subspace-based affinities based on $\mathit{W}\left(\mathit{t}\right)$ as a layered set of networks, we proceed to carry out what amounts to modality fusion. The $\mathit{T}$ sparse matrices are added to produce one sparse matrix for both modalities, ${\mathit{W}}_{\mathit{\text{Total}}}$, thereby improving performance. Observations associated with one object/individual are clustered as one subspace where the contribution of each sensor is embedded in the entries of the ${\mathit{W}}_{\mathit{\text{Total}}}$ matrix. For clustering by ${\mathit{W}}_{\mathit{\text{Total}}}$, we apply spectral clustering.

After learning the structure of the data clusters, we validate our results on the validation set. We extract the principal components (eigen vectors of the covariance matrix) of each cluster in the original (training) dataset, to act as a representative subspace of its corresponding class. We subsequently project each new test point onto the subspace corresponding to each cluster, spanned by its principal components. The ${\mathit{l}}_{\mathbf{2}}$ norm of the projection is then computed, and the class with the largest norm is selected to be the class of this test point. For DRoGSuRe, we use the coefficient matrix ${\mathit{W}}_{\mathit{\text{Total}}}$ in Eq. (13) to cluster the test data points coming from all data modalities. We compare the clustering output labels with the ground truth for each dataset. The results for ARL and EYB datasets are depicted in Tables 1 and 2 respectively. From the results, it is clear that DRoGSuRE technique for the fused data remarkably outperforms DMSC in case of ARL dataset. The reason behind the significant improvement is the layered structure of our proposed approach that constructs the latent space in a way that safeguards the individual sensor private information which hence dedicates more degrees of freedom to each of the sensors. In addition, the ARL dataset structure offers modalities that are different in nature and individually provides new information in contrast to the EYB dataset. However, in case of EYB dataset and in the noiseless case, DMSC performed better than DRoGSuRe.

	Learning	Validation
DMSC	97.59%	98.33%
DRoGSuRe	100%	100%

Table 1.

Performance comparison for ARL dataset.

	Learning	Validation
DMSC	98.82%	98.89%
DRoGSuRe	98.42%	98.76%

Table 2.

Performance comparison for EYB dataset.

4.4 Noise training with single and multiple modalities

In the following, we test the robustness of our approach in the case of noisy learning. We distort one modality at a time by shuffling the pixels of all images in that modality during the training phase. By doing so, we are perturbing the structure of the sparse coefficient matrix associated with that modality, thus impacting the overall W matrix for both DRoGSuRe and DMSC. Testing with clean data, i.e., no distortion, demonstrates the impact of perturbing the training and hence performing an inadequate training, e.g., insufficient data or non-convergence. This can also be considered as augmenting the training data with new information or a new view for one modality which might not necessarily contained in the testing or the validation data. Moreover, we repeat the same experiment with the distortion of two modalities before learning the sparse coefficient matrices for both DMSC and DRoGSuRe. The results for the ARL dataset are depicted in Tables 3 and 4, while results for the EYB dataset are shown in Tables 5 and 6. For ARL dataset, we refer to Visible, S0, S1, S2 and DoLP as Mod 0, 1, 2, 3 and 4 respectively. For EYB Dataset, we refer to Face, left eye, nose, mouth and right eye as mod 0, 1, 2, 3, and 4. We refer to each modality as Mod, where L denotes learning and V denotes validation results. From the results, it is clear that DRoGSuRe is showing a significant improvement in the clustering accuracy as compared to DMSC for both learning and validation set. The reason for that, is again, due to the fact that perturbing one or two modalities would have less impact on the overall performance for DRoGSuRe in comparison to DMSC.

	DMSC L	DMSC V	DRoGSuRe L	DRoGSuRE V
Mod 0	87.17%	86.67%	95.37%	95%
Mod 1	91.67%	90%	98.29%	98.33%
Mod 2	92.77%	92.78%	99.17%	99.44%
Mod 3	90.55%	90.57%	99.31%	99.44%
Mod 4	92.78%	91.11%	96.44%	96.67%

Table 3.

ARL dataset: Distorting one modality.

	DMSC L	DMSC V	DRoGSuRe L	DRoGSuRE V
Mod 0 & 1	82.22%	82.78%	92.27%	94.58%
Mod 1 & 2	91.11%	91.11%	97.22%	97.36%
Mod 0 & 3	85.51%	82.56%	93.01%	95.42%
Mod 1 & 4	91.67%	89.44%	97.22%	97.36%
Mod 2 & 3	90%	89.72%	97.69%	97.78%

Table 4.

ARL dataset: Distorting two modalities.

	DMSC L	DMSC V	DRoGSuRe L	DRoGSuRE V
Mod 0	87.96%	88.5%	93.29%	94.69%
Mod 1	91.84%	91.15%	95.79%	97.46%
Mod 2	89.01%	88.72%	98.03%	97.57%
Mod 3	92.69%	91.81%	95.59%	96.68%
Mod 4	91.45%	91.59%	97.17%	97.35%

Table 5.

EYB dataset: Distorting one modality.

	DMSC L	DMSC V	DRoGSuRe L	DRoGSuRE V
Mod 0 & 2	86.64%	85.18%	96.84%	96.13%
Mod 0 & 4	87.83%	89.16%	94.54%	95.8%
Mod 1 & 4	86.38%	86.06%	94.21%	95.8%
Mod 2 & 3	88.22%	84.96%	91.58%	93.92%
Mod 3 & 4	88.03%	86.28%	94.08%	95.35%

Table 6.

EYB dataset: Distorting two modalities.

4.5 Testing with limited noisy testing data

In the following, we study the effect of using noiseless data for training while validating with noisy and missing data. We add Gaussian noise to one data modality in the validation set and vary the SNR by varying the noise variance. We subsequently assume that we only have one modality available at testing. Then, we keep increasing the number of available noiseless data modalities beside the noisy modality. We average the results considering all different combinations of data modalities for ARL and EYB datasets. The results are depicted in Figures 5 and 6 respectively. For the ARL dataset, we note the increasing gap between DMSC and DRoGSuRe as we augment the sensing capacity with noise-free modalities. On the other hand, for the EYB dataset and at lower SNR, the performance of DRoGSuRe is slightly worse than DMSC which might be explained by the results in Table 2; as the training accuracy for DMSC is slightly better than DRoGSuRe in the case of clean training. However, at higher SNR, the performance of the two approaches is very close.

Figure 5.

ARL noiseless training and validating on limited noisy data.

Figure 6.

EYB noiseless training and validating on limited noisy data.

4.6 Missing modalities during testing

In the following, we evaluate the performance of DRoGSuRe and DMSC in case of missing data modalities during testing. It is not uncommon to have one or more sensors that might be silent during testing, thus justifying this experiment for further assessment. We try different combinations of available modalities during testing, and we average the clustering accuracy for each trial. Results are depicted in Figures 7 and 8 for ARL and EYB data respectively. Again, we notice a significant improvement for DRoGSuRe over DMSC for ARL Dataset. For EYB dataset, there is a slight improvement for DRoGSuRe over DMSC. The reason behind the slight improvement is because our approach introduces less error to the overall affinity matrix as compared to DMSC. Hence, preserving the performance and yielding a graceful degradation of the accuracy although DMSC was the state of the art for the EYB dataset.

Figure 7.

Missing modalities during testing for ARL dataset.

Figure 8.

Missing modalities during testing for EYB dataset.

5. Feature concatenation

Here we propose a rationale along with an alternative solution for enhancing the performance for EYB multi-modal data. Due to the specific structure of the EYB multi-modal data, the concatenation of the features corresponding to each modality is a reasonable alternative. By doing so, we are adjoining together the features representing each part of the face. Since the four modalities correspond to non-overlapping partitions of the face, the feature set corresponding to each partition will solely provide complementing information. A similar idea is proposed in [4] and is referred to as Late concatenation, where the multi-modal data is integrated in the last stage of the encoder. Their resulting decoder structure remains the same for either affinity fusion or late concatenation. This entails de-concatenating the multi-modal data prior to decoding it. Our proposed approach on the other hand, results in a self-expressive layer being driven by the concatenated features from the $\mathit{M}$ encoder branches. Afterwards, we feed the self-expressive layer output to each branch of the decoder. The concatenated information results in a more efficient code for the data, thereby resulting in an overall parsimonious with a sparse structure of the decoder, results in a decoder composed of three neural layers. The first layer consists of 150 filters of kernel size 3. The second layer consists of 20 layers of kernel size 3. The third layer consists of 10 layers of kernel size 5. Our approach is illustrated in Figure 9. We optimize the weights of the auto-encoder as follows,

Figure 9.

CNN Concatenation Network.

$${\mathit{min}}_{W\mid w_{\mathit{kk}}=0}\rho {‖\mathit{W}‖}_1+\frac{\gamma }{2}\sum _{t=1}^T{‖\mathit{X}\left(t\right)-{\mathit{X}}_{\mathit{r}}\left(t\right)‖}_F^2+\frac{\mu }{2}\sum _{t=1}^T{‖\mathit{N}-\mathit{NW}‖}_F^2,$$E18

where $\mathit{N}=\left[\mathit{L}\left(\mathbf{1}\right)‖\mathit{L}\left(\mathbf{2}\right)‖\mathit{L}\left(\mathbf{3}\right)‖\mathit{L}\left(\mathbf{4}\right)‖\mathit{L}\left(\mathbf{5}\right)\right]$.

We compared the performance of our proposed approach against the late concatenation approach in [4] and the results are depicted in Table 7 for the EYB dataset.

	Learning	Validation
DMSC Late Concatenation	95.66%	94.7%
CNN Concatenation Network	99.28%	99.3%

Table 7.

Concatenation performance for EYB dataset.

From the previous table, we can conclude that concatenating the features from the encoder and feeding the concatenated information to each decoder branch achieves a better performance for this type of multi-modal data structure. The reason behind this enhancement is the combination of efficient extraction of the basic features from the whole face and finer features from each part of the face. Promoting more efficiency as noted, this concatenation may also be intuitively viewed as adequate mosaicking, in which different patterns complement each other. In the following, we will show how our proposed approach performs in two cases: missing and noisy test data. The results of the new proposed approach, which we refer to as CNNs concatenation network, is compared to the state-of-the-art DMSC network [4]. We start by training the auto-encoder network using 75% of the data and then we test on the rest of the data. In Figure 10, we show how the performance degrades by decreasing the number of available modalities at testing from five to one. From the results, it is clear how the CNNs concatenation network outperforms the DMSC network. Additionally, we repeated the same experiment we performed in subsection 4.5. We train the network with noiseless data and then add Gaussian noise to one data modality at the testing. Additionally, we vary the number of available modalities at testing from one to four. The results are depicted in Figure 11. From the results, it is clear how the concatenated CNNs is more robust to noise than DMSC.

Figure 10.

Missing modalities during testing for EYB dataset.

Figure 11.

EYB noiseless training and validating on limited noisy data.

In addition, we have utilized the Concatenation network to perform object clustering on the ARL data. We compare the clustering performance of the concatenation network with both DMSC and DRoGSuRe. The results are depicted in Table 8. From the results, we conclude that DRoGSuRe still outperforms the other approaches for the ARL dataset. Although the number of parameters involved in training the DRoGSuRe network is higher than other approaches, since there are multiple self-expressive layers, however, DRoGSuRe is more robust to noise and limited data availability during testing.

	Learning	Validation
DMSC	97.59%	98.33%
DRoGSuRe	100%	100%
CNN Concat	99.44%	99.17

Table 8.

Concatenation performance for ARL dataset.

6. Conclusion

In this paper, we proposed a deep multi-modal approach to fuse data through recovering the underlying subspaces of data observations from data corrupted by noise to scale to complex data scenarios. DRoGSuRe provides a natural way to fuse multi-modal data by employing the self-representation matrix as an embedding for each data modality. Experimental results show a significant improvement for DRoGSuRe over DMSC under different types of potential limitations and provides robustness with limited sensing modalities. We also proposed the concatenated CNNs model, which can work better for different multi-modal data structures.

Acknowledgments

This work was in part supported by DOE-National Nuclear Security Administration through CNEC-NCSU under Award DE-NA0002576. The first author was also in part supported by DTRA.

Conflict of interest

The authors declare no conflict of interest.

A.

To theoretically compare our proposed variational scaling fusion approach DRoGSuRe to DMSC, we proceed by way of a first order perturbation analysis on the parameter set ${\mathit{W}}^{\mathit{i}}$ of respectively either technique $\mathit{i}\mathit{=}\mathbf{1}\mathit{,}\mathbf{2}$. This will, in turn impact the associated affinity matrix ${\mathit{A}}^{\mathit{i}}$, which as we will later elaborate directly impacts the subspace clustering procedure which is central to the inference following the fusion procedure.

Adopting the original formulation for the first persistently differential scaling approach, namely that T modalities are jointly exploited, results in, ${\mathit{X}}^{\mathbf{1}}\left(\mathit{t}\right)=\left[{\mathit{x}}_{\mathbf{1}}^{\mathbf{1}}\left(\mathit{t}\right){\mathit{x}}_{\mathbf{2}}^{\mathbf{1}}\left(\mathit{t}\right)\dots {\mathit{x}}_{\mathit{n}}^{\mathbf{1}}\left(\mathit{t}\right)\right]$, where ${\mathit{x}}_{\mathit{k}}^{\mathbf{1}}\left(\mathit{t}\right)\in {\mathbb{R}}^{\mathit{m}},\mathit{t}=\mathbf{1},\mathbf{2},\dots ,\mathit{T}$ represents the ${\mathit{k}}^{\mathit{th}}$ observation. The second approach only effectively uses only one subspace structure of the fused modalities ${\mathit{X}}^{\mathbf{2}}\left(\mathit{t}\right)=\left[{\mathit{x}}_{\mathbf{1}}^{\mathbf{1}}{\mathit{x}}_{\mathbf{2}}^{\mathbf{1}}\dots {\mathit{x}}_{\mathit{n}}^{\mathbf{1}}\right]$.

A first order perturbation on the data may be due to noise or to a degradation of a given sensor, and results in a perturbation of the UoS parameters,

$${\stackrel{\sim }{\mathit{W}}}_{\mathbf{1}}^{\mathit{i}}={\mathit{W}}_{\mathbf{1}}^{\mathit{i}}+{\delta }^i$$EA1

For the first method, each modality will have an associated subspace cluster parameter set ${\left\{{\mathit{W}}_{\mathit{t}}^{\mathbf{1}}\right\}}_{\mathit{t}=\mathbf{1},\dots ,\mathit{T}}$, with ${\mathit{W}}_{\mathit{t}}^{\mathbf{1}}\in {\mathbb{R}}^{\mathit{n}\times \mathit{n}}$. The overall parameter set for DRoGSuRe can then be written as,

$${\stackrel{\sim }{\mathit{W}}}^{\mathbf{1}}={\stackrel{\sim }{\mathit{W}}}_1^1+{\mathit{W}}_2^1+\dots +{\mathit{W}}_m^1$$EA2

Where the unperturbed overall sparse coefficient matrix is written as follows, ${\mathit{W}}_{\mathit{tot}}^{\mathbf{1}}={\mathit{W}}_{\mathbf{1}}^{\mathbf{1}}+{\mathit{W}}_{\mathbf{2}}^{\mathbf{1}}+\dots +{\mathit{W}}_{\mathit{m}}^{\mathbf{1}}$. A similar development follows for method 2, with the difference that the contributing modalities are fused a priori.

Proof. We first write the affinity matrix associated with DRoGSuRE as,

$${\stackrel{\sim }{\mathit{A}}}^1={\stackrel{\sim }{\mathit{W}}}_{\mathit{tot}}^1+{\left({\stackrel{\sim }{\mathit{W}}}_{\mathit{tot}}^1\right)}^T$$EA3

$${\stackrel{\sim }{\mathit{A}}}^1={\stackrel{\sim }{\mathit{W}}}_{\mathbf{1}}^{\mathbf{1}}+{\mathit{W}}_{\mathbf{2}}^{\mathbf{1}}+\dots +{\mathit{W}}_{\mathit{m}}^{\mathbf{1}}+{\left({\stackrel{\sim }{\mathit{W}}}_{\mathbf{1}}^{\mathbf{1}}+{\mathit{W}}_{\mathbf{2}}^{\mathbf{1}}+\dots +{\mathit{W}}_{\mathit{m}}^{\mathbf{1}}\right)}^T$$EA4

where the superscript $\mathit{T}$ denotes transpose. This is equivalent to,

$${\stackrel{\sim }{\mathit{A}}}^1={\stackrel{\sim }{\mathit{A}}}_1^1+{\sum }_{i=2}^T{\mathit{A}}_i^1$$EA5

Where $\mathbf{0}\le {\stackrel{\sim }{\mathit{A}}}_{\mathbf{1}}^{\mathbf{1}}\left(\mathit{i},\mathit{j}\right)\le \mathbf{1}+{\mathit{\delta }}^{\mathbf{1}}$ . The unperturbed collective affinity matrix ${\mathit{A}}^{\mathbf{1}}$ can be similarly written ${\mathit{A}}^{\mathbf{1}}={\sum }_{\mathit{i}=\mathbf{1}}^{\mathit{T}}{\mathit{A}}_{\mathit{i}}^{\mathbf{1}}$ with the unity constraint on each entry of all matrices. We may also write the magnitude of the difference as,

$$\left|{\mathit{A}}^{\mathbf{1}}\mathit{-}{\stackrel{\sim }{\mathit{A}}}^{\mathbf{1}}\right|={\delta }^1+{\left({\delta }^1\right)}^T$$EA6

Letting $∆={\mathit{\delta }}^{\mathbf{1}}+{\left({\mathit{\delta }}^{\mathbf{1}}\right)}^{\mathit{T}}\in {\mathbb{R}}^{\mathit{n}\times \mathit{n}}$, and assuming $\mathit{ϵ}={\mathit{max}}_{\mathit{i},\mathit{j}}{\left[∆\right]}_{\mathit{i},\mathit{j}}$, we can write,

$${‖\mathit{A}-\stackrel{\sim }{\mathit{A}}‖}_F\le \mathit{nϵ}$$EA7

Given the $∆$ matrix individual entry bounds, we conclude that,

$$0\le ϵ\le \frac{1}{t}$$EA8

Since DMSC assumes having one sparse coefficient matrix $\mathit{W}$ for all data modalities, which is equivalent to only one subspace structure of the fused modalities ${\mathit{X}}^{\mathbf{2}}\left(\mathit{t}\right)=\left[{\mathit{x}}_{\mathbf{1}}^{\mathbf{2}}\dots {\mathit{x}}_{\mathit{n}}^{\mathbf{2}}\right]$. Therefore, the UoS parameters will be perturbed by ${\mathit{\delta }}^{\mathbf{2}}$ as follows,

$${\stackrel{\sim }{\mathit{W}}}^2={\mathit{W}}^{\mathbf{2}}+{\delta }^2$$EA9

The affinity matrix associated with DMSC can be written as follows, ${\stackrel{\sim }{\mathit{A}}}^{\mathbf{2}}={\stackrel{\sim }{\mathit{W}}}^{\mathbf{2}}+{\left({\stackrel{\sim }{\mathit{W}}}^{\mathbf{2}}\right)}^{\mathit{T}}$, which is equivalent to,

$${\stackrel{\sim }{\mathit{A}}}^{\mathbf{2}}={\mathit{W}}^{\mathbf{2}}+{\delta }^2+{\left({\mathit{W}}^{\mathbf{2}}\right)}^T+{\left({\delta }^2\right)}^T$$EA10

Similarly, the unperturbed affinity matrix will be as follows,

$${\mathit{A}}^{\mathbf{2}}={\mathit{W}}^{\mathbf{2}}+{\left({\mathit{W}}^2\right)}^T$$EA11

From Eqs. (A10) and (A11), the magnitude of the difference can be written as follows,

$$\mid {\mathit{A}}^{\mathbf{2}}-{\stackrel{\sim }{\mathit{A}}}^{\mathbf{2}}\mid ={\delta }^2+{\left({\delta }^2\right)}^T$$EA12

Letting $\mathit{\gamma }={\mathit{\delta }}^{\mathbf{2}}+{\left({\mathit{\delta }}^{\mathbf{2}}\right)}^{\mathit{T}}\in {\mathbb{R}}^{\mathit{n}\times \mathit{n}}$, i.e., $\left|{\mathit{A}}^{\mathbf{2}}-{\stackrel{\sim }{\mathit{A}}}^{\mathbf{2}}\right|=\mathit{\gamma }$, and assuming $\mathit{\Psi }={\mathit{max}}_{\mathit{i},\mathit{j}}{\left[\mathit{\gamma }\right]}_{\mathit{i},\mathit{j}}$, we can write ${‖{\mathit{A}}^{\mathbf{2}}-{\stackrel{\sim }{\mathit{A}}}^{\mathbf{2}}‖}_{\mathit{F}}\le \mathit{n\Psi }$. Given the $\mathit{\gamma }$ matrix individual entry bounds, we conclude $\mathbf{0}\le \mathit{\Psi }\le \mathbf{1}$. If we only perturb one modality, knowing that $\mathbf{0}\le \mathit{A}\left(\mathit{i},\mathit{j}\right)\le \mathbf{1}$, therefore the error could lie between $\mathbf{0}\le \mathit{\Psi }\le \mathbf{1}$, which entails either creating a fake relation between two data points or erasing an existing relation. $\mathit{ϵ}$ and $\mathit{\Psi }$ are random variables that do not have to follow a specific distribution, however, in any case $\mathit{E}\left({\mathit{ϵ}}^{\mathbf{2}}\right)\ll \mathit{E}\left({\mathit{\Psi }}^{\mathbf{2}}\right)$ and therefore ${\mathit{SNR}}_{\mathit{\text{DRoGSuRe}}}\gg {\mathit{SNR}}_{\mathit{\text{DMSC}}}\mathit{.}$

In light of the above two bounds, and the results of [42], where it is shown that the spectral clustering dependent on the respective projection operators ${\mathit{P}}_{{\mathit{W}}^{\mathbf{1}}}$ and ${\stackrel{\sim }{\mathit{P}}}_{{\stackrel{\sim }{\mathit{W}}}^{\mathbf{1}}}$ onto the vector subspaces spanned by the principal eigenvectors of ${\mathit{W}}_{\mathit{tot}}^{\mathbf{1}}$ and ${\stackrel{\sim }{\mathit{W}}}_{\mathit{tot}}^{\mathbf{1}}$ of may be written as,

$${‖P_{W^1}-{\stackrel{\sim }{P}}_{{\stackrel{\sim }{W}}^1}‖}_F\le \frac{\sqrt{2}}{{\alpha }^1}{‖{\mathit{A}}^{\mathbf{1}}-{\stackrel{\sim }{\mathit{A}}}^1‖}_F$$EA13

where ${\mathit{\alpha }}^{\mathbf{1}}$ is the spectral gap between the ${\mathit{k}}^{\mathit{th}}$ and ${\left(\mathit{k}+\mathbf{1}\right)}^{\mathit{st}}$ eigen value of ${\mathit{A}}^{\mathbf{1}}$, $\left|{\mathit{\lambda }}_{\mathit{k}}^{\mathbf{1}}-{\mathit{\lambda }}_{\mathit{k}+\mathbf{1}}^{\mathbf{1}}\right|$. Similarly, for DMSC, the bound on the projection operators is,

$${‖P_{W^2}-{\stackrel{\sim }{P}}_{{\stackrel{\sim }{W}}^2}‖}_F\le \frac{\sqrt{2}}{{\alpha }^2}{‖{\mathit{A}}^{\mathbf{2}}\mathit{-}{\stackrel{\sim }{\mathit{A}}}^{\mathbf{2}}‖}_F$$EA14

where ${\mathit{\alpha }}^{\mathbf{2}}=\left|{\mathit{\lambda }}_{\mathit{k}}^{\mathbf{1}}-{\mathit{\lambda }}_{\mathit{k}+\mathbf{1}}^{\mathbf{1}}\right|$. Since ${\mathit{W}}_{\mathbf{1}}^{\mathbf{1}},{\mathit{W}}_{\mathbf{2}}^{\mathbf{1}},\dots ,{\mathit{W}}_{\mathit{T}}^{\mathbf{1}}$ happen to commute and if they happen to be diagonalizable, therefore, they share the same eigenvectors. As a result, the eigenvectors of ${\mathit{W}}_{\mathbf{1}}^{\mathbf{1}}+{\mathit{W}}_{\mathbf{2}}^{\mathbf{1}}+\dots +{\mathit{W}}_{\mathit{T}}^{\mathbf{1}}$ are also the same and the corresponding eigenvalue that is the sum of the corresponding eigenvalues of ${\mathit{W}}_{\mathbf{1}}^{\mathbf{1}},{\mathit{W}}_{\mathbf{2}}^{\mathbf{1}},\dots$and ${\mathit{W}}_{\mathit{T}}^{\mathbf{1}}\mathit{.}$Therefore, ${\mathit{\lambda }}_{\mathit{k}}^{\mathbf{1}}\gg {\mathit{\lambda }}_{\mathit{k}}^{\mathbf{2}}$ From all the above, we can conclude that smaller error yielding to better clustering, hence preserving the performance, yields the improvement by the T-factor noted in the proposition and shown in the two perturbation developments.