Open access peer-reviewed chapter

Bilinear Applications and Tensors

By Rodrigo Garcia Eustaquio

Submitted: October 30th 2019Reviewed: December 19th 2019Published: September 9th 2020

DOI: 10.5772/intechopen.90904

Abstract

In this chapter, a theoretical approach to the vector space of tensor of order 3 and the vector space of bilinear applications will be presented in order to present an isomorphism between these spaces and several properties about tensor and bilinear applications. With this well-defined isomorphism, we will present how to calculate the product between tensor of second derivatives and a vector, where such a product is used in several numerical methods such as Chebyshev-Halley class and others mentioned in the introduction. In addition, concepts on differentiability are presented, allowing a better understanding for the reader about second-order derivatives seen as a tensor.

Keywords

• tensor
• bilinear application
• isomorphism
• second derivative
• inexact tensor-free Chebyshev-Halley class

1. Introduction

Frequently, discretization of mathematical models demands solving a system of equations, which is generally nonlinear. Such mathematical problems might be written as

findxIRnsuch thatFx=0E1

where F:IRnIRn.

There exist iterative methods for solving (1) that have cubic convergence rate, for instance, the methods belonging to the following class of methods named Chebyshev-Halley class, which was introduced by Hernández and Gutiérrez in [1]:

xk+1=xkI+12LxkIαLxk1JFxk1Fxk,E2

for all kIN, where

Lx=JFx1TFxJFx1Fx,E3

and JFxand TFxdenote the first and second derivatives of Fevaluated at x, respectively. The parameter αis a real number and Iis the identity matrix in IRn×n.

Discretized versions of Chebyshev-Halley class have already been considered in [2] in such a way that the tensor of second derivatives of the function Fwas approximated by bilinear operators. A tensor is a multi-way array or multidimensional matrix. A generalization of the Chebyshev-Halley class 2where no second-order derivative information is required but that also has cubic convergence rate, named inexact tensor-free Chebyshev-Halley class, was introduced by Eustaquio, Ribeiro, and Dumett [3]. Other families of iterative methods with cubic convergence rate were extensively described in Traub’s book [4].

Several alternatives exist for the product of the tensor of second derivatives of Fby vectors [5, 6, 7, 8], and this needs to be elucidated.

The aim of this chapter is to present concepts and relationships between tensors of order 3and bilinear applications, in order to relate them to the second derivative of a two-differentiable application. We will see later that given the vectors u,vIRn, the i-th row of the matrix TFxvis defined by vT2fix, where 2fixis the Hessian of the i-th component of Fevaluated at x. The i-th component of the vector TFxvuis defined by vT2fixu.

2. Tensors

Tensors naturally arise in some applications, such as chemometry [9], signal processing [10], and others. According to [8], for many applications involving high-order tensors, the known results of matrix algebra seemed to be insufficient in the twentieth century. There were some workshops and congresses on the study of tensors, such as:

• Workshop on Tensor Decomposition at the American Institute of Mathematics which took place at the Palo Alto, California, 2004, organized by Golub, Kolda, Nagy, and Van Loan. Details in [11];

• Workshop on Tensor Decompositions and Applications, 2005, organized by Comon and De Lathauwer. Details in [12]; and

• Minisymposium on Numerical Multilinear Algebra: A New Beginning, 2007, organized by Golub, Comon, De Lathauwer, and Lim and which took place at the Zurich.

Readers interested in multilinear singular value decomposition, eigenvalues, and eigenvectors may consult as references [5, 6, 7, 8, 13, 14]. In this text, we will focus our attention on tensors of order 3.

Let I1,I2, and I3be three positive integers. A tensor Tof order 3is an three-way array where its elements ti1i2i3are indexed by i1=1,,I1, i2=1,,I2, and i3=1,,I3and the n-th dimension of the tensor is denoted by In, for n=1,2,3. For example, the first, second, and third dimensions of a tensor TIR2×4×3are 2,4,3, respectively.

Obviously, tensors are generalizations of matrices. A matrix can be viewed as a tensor of order 2, while a vector can be viewed as a tensor of order 1.

From an algebraic point of view, a tensor Tof order 3is an element of the vector space IRI1×I2×I3, whereas from the geometric point of view, a tensor Tof order 3can be seen as a parallelepiped [15], with I1rows, I2columns, and I3tubes. Figure 1 illustrates a tensor TIR2×4×3.

In linear algebra, it is common to see a matrix through its columns. If AIRm×n, then Acan be viewed as A=a1an, where ajIRmdenotes the j-th column of the matrix A. In the case of tensor of order 3, we can see them through fibers and slices. Hence follow the definitions.

Definition 1.1.A tensor fiber of a tensor of order 3is a one-dimensional fragment obtained by fixing only two indices.

Definition 1.2.A tensor slice of a tensor of order 3is a two-dimensional section (fragment), obtained by fixing only one index.

Generally in tensors of order 3, a fiber is a vector and a slice is a matrix. We have three types of fibers:

• column fibers (or mode-1fiber), where the indices i2and i3are fixed;

• row fibers (or mode-2fiber), where the indices i1and i3are fixed; and

• tube fibers (or mode-3fiber), where the indices i1and i2are fixed.

We also have three types of slices:

• horizontal slice, where the index i1is fixed;

• lateral slice, where the index i2is fixed; and

• frontal slice, where the index i3is fixed.

For example, consider a tensor TIR2×4×3with i=1,2, j=1,2,3,4, and k=1,2,3. The i-th horizontal slice, denoted by Ti::, is the matrix

Ti::=ti11ti12ti13ti21ti22ti23ti31ti32ti33ti41ti42ti43,

the j-th lateral slice, denoted by T:j:, is the matrix

T:j:=t1j1t1j2t1j3t2j1t2j2t2j3

and the k-th frontal slice, denoted by T::k, is the matrix

T::k=t11kt12kt13kt14kt21kt22kt23kt24k.E4

Figures 2 and 3 illustrate the three types of fibers and slices, respectively, of a tensor TIR2×4×3.

2.1 Tensor operations

The first issue to consider in this subsection is how to calculate the product between tensors and matrices. It is well known from elementary algebra that given matrices AIRm×nand BIRR×m, it is possible to calculate the product BA, because the first dimension (number of rows) of matrix Aagrees with the second dimension (number of columns) of matrix B, and each product element is the result of the inner product between rows of matrix Band columns of matrix A.

The product between tensors of order 3and matrices or vectors is a bit more complicated. In order to obtain an element of the product between a tensor and a matrix, it is necessary to specify what dimension of the tensor will be chosen to agree with the number of columns of the matrix, and each resulting element will be a result of the inner product between the mode-nfibers (column, row, or tube) and the columns of the matrix. We will use the solution adopted by [8], which defines the product mode-nbetween tensors and matrices and the solution adopted by [5] that defines the contracted product mode-nbetween tensors and vectors.

The mode-nproduct is useful when one wants to decompose into singular values a high-order tensor in order to avoid the use of the generalized transpose concept. We refer to [5, 7, 8, 13] for details.

Definition 1.3.(mode-ntensor matrix product) The mode-1product between a tensor TIRm×n×pand a matrix AIRR×mis a tensor

Y=T×1AIRR×n×p

where its elements are defined by

yrjk=i=1mtijkariwherer=1,,R,j=1,,n,andk=1,,p.

The mode-2product between a tensor TIRm×n×pand a matrix AIRR×nis a tensor

Y=T×2AIRm×R×p

where its elements are defined by

yirk=j=1ntijkarjwherei=1,,m,r=1,,Randk=1,,p.

The mode-3product between a tensor TIRm×n×pand a matrix AIRR×pis a tensor

Y=T×3AIRm×n×R

where its elements are defined by

yijr=k=1ptijkarkwherei=1,,m,j=1,,nandr=1,,R.

To understand the mode-nproduct in terms of matrix, consider matrices AIRm×n, BIRk×m, and CIRq×n. By Definition 1.3 we have

A×1B=BAIRk×nandA×2C=ACTIRm×q.

Thus, the singular value decomposition of matrix Acan be written as

UΣVT=Σ×1U×2V=Σ×2V×1U.

The mode-nproduct satisfies the following property [8]:

Property 1.Let Tbe a tensor of order 3and matrices Aand Bof convenient sizes. We have for all r,s=1,2,3

T×rA×sB=T×sB×rA=T×rA×sBforrsandE5
T×rA×rB=T×rBAE6

The idea of Bader and Kolda [5] to calculate the product between tensor and vector is to calculate the inner product of each mode-nfiber (column, row, or tube) with the vector. It is not advantageous to treat an m-dimensional vector as a matrix m×1. For example, if we consider a tensor TIRm×n×pand a vector vIRm×1, with m,n,p1, by Definition 1.3, the product between Tand vis not well defined, but it is possible to calculate T×1vT.

Definition 1.4.(Contracted product mode-n between tensors and vectors) The contracted product mode-1between a tensor TIRm×n×pand a vector vIRmis the matrix

A=T×¯1vIRn×p

where its elements are defined by

ajk=i=1mtijkviwherej=1,,nandk=1,,p

where viis the i-th component of the vector v.

The contracted product mode-2between a tensor TIRm×n×pand a vector vIRnis the matrix

A=T×¯2vIRm×p

where its elements are defined by

aik=j=1ntijkvjwherei=1,,mandk=1,,p

where vjis the j-th component of the vector v.

The contracted product mode-3between a tensor TIRm×n×pand a vector vIRpis the matrix

A=T×¯3vIRm×n

where its elements are defined by

aij=k=1ptijkvkwherei=1,,mandj=1,,n

where vkis the k-th component of the vector v.

A caution must be added when calculating the product between matrices and vectors by considering the definitions 1.3 and 1.4. For example, note that if AIRm×n, uIRn, and vIRm, then A×¯2uand A×2uThave the same elements, but

A×¯2uA×2uT,

because A×¯2uIRm(column vector) and A×2uTIR1×m(row vector). Note that, in relation to the matrix product of elementary algebra, we have

Au=A×¯2uE7
vTA=A×1vTA×¯1v.E8

In particular, given a tensor TIRn×m×mand a vector vIRm, by Definition 1.4 together with (8), it follows that T×¯2vIRn×mand

T×¯2v×¯2v=T×¯2vvIRn.

The contracted product mode-nsatisfies the following property [5]:

Property 2.Given a tensor Tof order 3and vectors uand vof convenient sizes, we have for all r=1,2,3and s=2,3that

T×¯ru×¯s1v=T×¯sv×¯ruforr<s.

For example, consider a tensor TIR2×4×3, and denote the k-th column and the q-th row of matrix Aby colkAand rowqA, respectively. Note that if:

1. 1. xIR2, then T×¯1xIR4×3and

colkT×¯1x=a1ka2ka3ka4k=t11kt21kt12kt22kt13kt23kt14kt24kx1x2=T::kTxandrowjT×¯1x=aj1aj2aj3=x1x2t1j1t1j2t113t2j1t2j2t213=xTT:j:

1. 2. xIR4, then T×¯2xIR2×3and

colkT×¯2x=a1ka2k=t11kt12kt13kt14kt21kt22kt23kt24kx1x2x3x4=T::kxandrowiT×¯2x=ai1ai2ai3=x1x2x3x4ti11ti12ti13ti21ti22ti23ti31ti32ti33ti41ti42ti43=xTTi::

1. 3. xIR3, then T×¯3xIR2×4and

coljT×¯3x=a1ja2j=t1j1t1j2t1j3t2j1t2j2t2j3x1x2x3=T:j:xandrowiT×¯3x=ai1ai2ai3=x1x2x3ti11ti21ti31ti41ti12ti22ti32ti42ti13ti23ti33ti43=xTTi::T

This example can be easily generalized to arbitrary dimensions. In particular, for a tensor TIRm×n×nand a vector xIRn, we have

rowiT×¯2x=xTTi::E9
rowiT×¯3x=xTTi::TE10

Lemma 1.5.Let TIRn×n×nbe a tensor. If Ti::is a symmetric matrix for all i=1,,n, then

T×¯2uv=T×¯2vu

for all u,vIRn.

Proof.By Property 2, it follows that T×¯2uv=T×¯3vu. By (10), (11), and the symmetry of Ti::, we have T×¯3v=T×¯2v.

3. Space of bilinear applications

In this section, we define bilinear applications on finite dimensional vector spaces, in order to relate them to the second derivative of a two-differentiable application, as well as a tensor of order 3.

Definition 1.6.Let U,V,Wbe vector spaces. An application f:U×VWis a bilinear application if:

1. fλu1+u2v=λfu1v+fu2vfor all λIR, u1,u2U, and vV.

2. fuλv1+v2=λfuv1+fuv2for all λIR, uU, and v1,v2V.

In other words, an application f:U×VWis a bilinear application if it is linear in each of the variables when the other variable is fixed. We denote by BU×VWthe set of all bilinear applications of U×Vin W. In particular, if U=Vand W=IRin Definition 1.6, then f:U×UIRis a bilinear form in which we are used to quadratic forms, for example.

A simple example of bilinear application is the function f:U×VIRdefined by

fuv=hugv,E11

with hUand gV, where Udenotes the dual space to U. In fact, we have for all λIR,u1,u2UandvVsuch that

fλu1+u2v=hλu1+u2gv=λhu1+hu2gv=λfu1v+fu2v.

Similarly, it is easy to see that fuλv1+v2=λfuv1+fuv2for all λIR,uUandv1,v2V.

The next theorem ensures that a bilinear application f:U×VWis well defined when the image of fapplied in the bases elements of Uand Vis known.

Theorem 1.7.Let U, V, and Wbe vector spaces; u1umand v1vnbases of the Uand V, respectively; and wiji=1mandj=1na subset of W. Then, there exists an only bilinear application f:U×VWsuch that fuivj=wij.

Proof. Let u=i=1mαiuiand v=j=1nβjvjbe arbitrary elements of Uand V, respectively. We defined an application f:U×VWby

fuv=i=1mj=1nαiβjwij.

It is easy to see that fis a bilinear application and fuivj=wij. Such an application is unique because if gis another bilinear application satisfying guivj=wij, then

guv=gi=1mαiuij=1nβjvj=i=1mj=1nαiβjguivj=E12
=i=1mj=1nαiβjwij=fuv.E13

Therefore g=f.

The following theorem guarantees the isomorphism between space of bilinear applications and space of tensor of order 3.

Theorem 1.8.Let U, V, and Wbe vector spaces with dimensions n, p, and m, respectively. Then, the space BU×VWhas dimension mnp.

Proof. The idea of the proof is to exhibit a basis for space BU×VW. For this, let w1wm, u1un, and v1vpbe bases of the W, U, and V, respectively. For each triple ijk, with i=1,,m, j=1,,n, and k=1,,p, we define a bilinear application fijk:U×VWsuch that

fijkurvs=wiifr=jands=k0ifrjorsk.E14

Theorem 1.7 ensures the existence of the fijk. We will then show that the set

A=fijki=1mj=1nandk=1p

is a basis of the space BU×VW. Let fBU×VW. We note in passing that

furvs=i=1mairswiE15

for all r=1,,nand s=1,,p. Consider the bilinear application

g=i=1mj=1nk=1paijkfijk.

Our goal is to show that g=f. In particular, we have

gurvs=i=1mj=1nk=1paijkfijkurvs=i=1mairswi=furvs

for all r=1,,nand s=1,,p. Therefore g=f. The set Ais linearly independent, because if

i=1mj=1nk=1paijkfijk=0,

then

0=k=1pi=1mj=1naijkfijkurvs=i=1mairswi.

Since w1wmis a basis of W, it follows that airs=0for all i=1,,m, r=1,,n, and k=1,,p.

In particular, if the dimensions of the vector spaces Uand Vare mand n, respectively, then the vector space BU×VIRhas dimension mn. Now, as two vector spaces of the same finite dimension are isomorphic [16], there exists a matrix m×nassociated with each fBU×VIR. By considering B=u1umand C=v1vnbases of Uand V, respectively, and if u=i=1mαiuiand v=j=1nβjvj, then by doing fuivj=aijfor all i=1,,mand j=1,,n, we have

fuv=i=1mj=1nαiaijβj,

which in matrix form is fuv=uBTAvC, where A=aijand vCdenote the vector components vin the basis C. Hence follows the next definition:

Definition 1.9.Let Uand Vbe vector spaces of finite dimension and ordered bases B=u1umUand C=v1vnV. We define, for each fBU×VIR, the matrix A=aijIRm×nof the frelative to the ordered bases Band C, whose elements are given by aij=fuivjwith i=1,,mand j=1,,n.

Consider now the space BIRm×IRnIRpand the canonical bases e1em, e¯1e¯n, ê1êpof the IRm, IRn, and IRp, respectively. Consider fBIRm×IRnIRp. For all uIRmand vIRn, we have

fuv=j=1mk=1nujvkfeje¯k

where ujand vkare the components of the uand vin the canonical bases of IRmand IRn, respectively. Denote the i-th component of the fby fi. Note that fiBIRm×IRnIR. So for each i=1,,p, we have

fiuv=j=1mk=1nujvkfieje¯k.

By Definition 1.9, the matrix of the fiin relation to the canonical bases is the matrix

Ai=tijkIRm×n,

where tijk=fieje¯k. So, we can write

fiuv=uTAiv.

In general, we can define pmatrices m×nas a tensor TIRp×m×n; this means that the pmatrices can be seen as the horizontal slices of the tensor T. We note in passing that we can write fuvas a product between tensor Tand vectors uand v, that is,

fuv=uTA1vuTA2vuTApv=T×¯2uv.E16

Thus, we can generalize Definition 1.9 as follows:

Definition 1.10.Let Uand Vbe finite dimension vector spaces. For fixed bases B=u1umand C=v1vnof the Uand V, respectively, we define, for each fBU×VIRp, the tensor T=tijkIRp×m×nof the frelative to the ordered bases Band C, whose elements are given by tijk=fiujvkwhere fiis the i-th component of the f, that is, fiBU×VIR, with i=1,,p, j=1,,m, and k=1,,n.

4. Differentiability

Let Ube an open subset of IRmand F:UIRmIRna differentiable application throughout Uand aU. Denote LIRmIRnthe set of all linear applications of IRmin IRn. When F:UIRmLIRmIRnis differentiable in aU, we say that the application Fis twice differentiable in aUand then the linear transformation FaL(IRm,LIRmIRn)is the second derivative of Fin aU.

The norm of Fais naturally defined. For any hIRm, it follows that

Fah=supk=1FahkcomkIRm

and then

Fa=suph=1Fah=suph=1supk=1Fahk.

An important observation with respect to Theorem 1.8 is that the spaces L(IRm,LIRmIRn)and BIRm×IRmIRnare isomorphic. This means that Fais a bilinear application belonging to space BIRm×IRmIRn. Such isomorphism can be found in classical analysis books [17, 18]. On the other hand, by the same theorem, the space of bilinear applications BIRm×IRmIRnand space of tensor IRn×m×mare also isomorphic.

In many practical applications, such as algorithm implementations, the second derivative Famay be implemented as a tensor belonging to space IRn×m×m. The question now is how the tensor elements are formed. For this, consider the application A:IRIRn×mand αIR. We have Aαas a matrix with nrows and mcolumns. Its elements are denoted by aijαwhere aijare components functions of Awith i=1,,nand j=1,,m. Case aij:IRIRis differentiable in αfor all i=1,,nand j=1,,m; the derivative of Ain αis the matrix

Aα=aijαIRn×m.E17

The definition of the derivative of Aα(17) is a classical definition. We refer to [19] for details.

In the sense of generalizing (17), consider now A:UIRpIRn×ma differentiable application in uUwith component function aij:IRpIRwith i=1,,nand j=1,,m. When aijis differentiable in ufor all i=1,,nand j=1,,m, we defined the derivative of Ain uas the tensor

Au=aijuIRn×m×p.E18

Note that in fact (18) is a generalization of (17). With fixed iand j, aijuis a tube fiber of the tensor Au, whose elements are

Auijk=aijxkuE19

for all k=1,,p.

For example, consider an application F:UIR2IR3twice differentiable in aUwhere Uis an open set. The Jacobian matrix of Fin ais given by

JFa=f1aTf2aTf3aT=f1x1af1x2af2x1af2x2af3x1af3x2a

and its derivative is, by (18), the tensor

JFa=TFa=fixjaIR3×2×2E20

where, by (19), its elements are described as

tijk=2fixkxja.

With fixed i, it is easy to see that the i-th horizontal slice of the TFais the Hessian matrix 2fiadefined by

2fia=TFai::=2fix1x1a2fix1x2a2fix2x1a2fix2x2a.E21

We note in passing that any column of the matrix 2fixis a row fiber of the i-th horizontal slice.

As mentioned in the introduction, some numerical methods need to calculate the product between tensor TFaand vectors in IR2.

From Definition 1.4, it is possible to calculate the contracted product mode-2and mode-3. As Hessian matrices are symmetrical, given vIR2, by Lemma 1.5 together with (10) and (11), we have

TFa×¯3v=TFa×¯2v=row1TFa×¯2vrow2TFa×¯2vrow3TFa×¯2v=vT2f1avT2f2avT2f3aIR3×2

and consequently it follows that

TFa×¯2vu=vT2f1auvT2f2auvT2f3auIR3E22

for all u,vIR2.

This means that the tensor TFadefined by (20) is the associate tensor to bilinear application Fa, in relation to canonical basis of IR2, by means of Definition 1.10. Without loss of generality, we have

TFa×¯3v=TFa×¯2v=TFav

and by means of Lemma 1.5, it follows that

TFauv=TFavu=TFavu.

To finish, we consider the following particular case. We know that the k-th column of Jacobian JFxis equal to product JFxek, where ekis the k-th canonical vector of IRn. It is worth noting what the slice of the matrix TFxekis. By definition, we have

TFxek=ekT2f1xekT2f2xekT2fnx=rowk2f1xrowk2f2xrowk2fnx

Given that rowk2fixis the k-th tube fiber of i-th horizontal slice, we have TFxekas the k-th lateral slice, or, by symmetry of Hessians, it is the transpose of k-th frontal slice. In short, for the twice differentiable application F:UIRnIRm, we have TFxIRm×n×nwhere the mhorizontal slices are the Hessians 2fix, with i=1,,mand the nlateral and frontal slices obtained by the following product TFxek, with k=1,,n.

5. Conclusions

In this text, we have shown some properties of tensors, in particular those of order 3. In addition, we have approached bilinear applications, and we have shown the isomorphism between space of bilinear applications and of tensor of order 3. As mentioned in the introduction, to solve a nonlinear system, some numerical methods use tensors, either in the iterative scheme or in the proof of theorems. For this reason, we have written a section on differentiability of applications by showing how to calculate the product between tensor of second derivatives and vectors.

More

© 2020 The Author(s). Licensee IntechOpen. This chapter is distributed under the terms of the Creative Commons Attribution 3.0 License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

How to cite and reference

Cite this chapter Copy to clipboard

Rodrigo Garcia Eustaquio (September 9th 2020). Bilinear Applications and Tensors, Advances on Tensor Analysis and their Applications, Francisco Bulnes, IntechOpen, DOI: 10.5772/intechopen.90904. Available from:

Related Content

Advances on Tensor Analysis and their Applications

Edited by Francisco Bulnes

Next chapter

Kinematic-Energy Measurements of the Torsion Tensor in Space-Time

By Francisco Bulnes, Isaías Martínez, Omar Zamudio and Edgar Navarro

Advances in Quantum Communication and Information

Edited by Francisco Bulnes

First chapter

Introductory Chapter: Advanced Communication and Nano-Processing of Quantum Signals

By Francisco Bulnes

We are IntechOpen, the world's leading publisher of Open Access books. Built by scientists, for scientists. Our readership spans scientists, professors, researchers, librarians, and students, as well as business professionals. We share our knowledge and peer-reveiwed research papers with libraries, scientific and engineering societies, and also work with corporate R&D departments and government entities.