InTechOpen uses cookies to offer you the best online experience. By continuing to use our site, you agree to our Privacy Policy.

Mathematics » "Applied Linear Algebra in Action", book edited by Vasilios N. Katsikis, ISBN 978-953-51-2420-7, Print ISBN 978-953-51-2419-1, Published: July 6, 2016 under CC BY 3.0 license. © The Author(s).

# Likelihood Ratio Tests in Multivariate Linear Model

By Yasunori Fujikoshi
DOI: 10.5772/62277

Article top

# Likelihood Ratio Tests in Multivariate Linear Model

Yasunori Fujikoshi
Show details

## Abstract

The aim of this chapter is to review likelihood ratio test procedures in multivariate linear models, focusing on projection matrices. It is noted that the projection matrices to the spaces spanned by mean vectors in hypothesis and alternatives play an important role. Some basic properties are given for projection matrices. The models treated include multivariate regression model, discriminant analysis model, and growth curve model. The hypotheses treated involve a generalized linear hypothesis and no additional information hypothesis, in addition to a usual liner hypothesis. The test statistics are expressed in terms of both projection matrices and sums of squares and products matrices.

Keywords: algebraic approach, additional information hypothesis, generalized linear hypothesis, growth curve model, multivariate linear model, lambda distribution, likelihood ratio criterion (LRC), projection matrix

## 1. Introduction

In this chapter, we review statistical inference, especially likelihood ratio criterion (LRC) in multivariate linear model, focusing on matrix theory. Consider a multivariate linear model with p response variables y1, …, yp and k explanatory or dummy variables x1, …, xk. Suppose that y = (y1, …, yp)′ and x = (x1, …, xk)′ are measured for n subjects, and let the observation of the ith subject be denoted by yi and xi. Then, we have the observation matrices given by

 Y=y1y2…yn′,X=x1x2…xn′. (1.1)

It is assumed that y1, …, yn are independent and have the same covariance matrix Σ. We express the mean of Y as follows:

 EY=η=η1,…,ηp. (1.2)

A multivariate linear model is defined by requiring that

 ηi∈Ω,foralli=1,…,p, (1.3)

where Ω is a given subspace in the n dimensional Euclid space Rn. A typical Ω is given by

 Ω=ℛX=η=Xθ;θ=θ1…θk′,−∞<θi<∞,i=1,…,k. (1.4)

Here, ℛ[X] is the space spanned by the column vectors of X. A general theory for statistical inference on the regression parameter Θ can be seen in texts on multivariate analysis, e.g., see [18]. In this chapter, we discuss with algebraic approach in multivariate linear model.

In Section 2, we consider a multivariate regression model in which xi's are explanatory variables and Ω = ℛ[X]. The maximum likelihood estimator (MLE)s and likelihood ratio criterion (LRC) for Θ2=O are derived by using projection matrices. Here, Θ=Θ1Θ2. The distribution of LRC is discussed by multivariate Cochran theorem. It is pointed out that projection matrices play an important role. In Section 3, we give a summary of projection matrices. In Section 4, we consider to test an additional information hypothesis of y2 in the presence of y1, where y1 = (y1. …, yq)′ and y2 = (yq + 1. …, yp)′. In Section 5, we consider testing problems in discriminant analysis. Section 6 deals with a generalized multivariate linear model which is also called the growth curve model. Some related problems are discussed in Section 7.

## 2. Multivariate regression model

In this section, we consider a multivariate regression model on p response variables and k explanatory variables denoted by y = (y1, …, yp)′ and x = (x1, …, xk)′, respectively. Suppose that we have the observation matrices given by (1.1). A multivariate regression model is given by

 Y=XΘ+E, (2.1)

where Θ is a k × p unknown parameter matrix. It is assumed that the rows of the error matrix E are independently distributed as a p variate normal distribution with mean zero and unknown covariance matrix Σ, i.e., Np0,Σ.

Let L(Θ, Σ) be the density function or the likelihood function. Then, we have

2logLΘ,Σ=nlog|Σ|+trΣ1YXΘYXΘ+nplog2π.

The maximum likelihood estimators (MLE) Θ^ and Σ^ of Θ and Σ are defined by the maximizers of L(Θ, Σ) or equivalently the minimizers of −2log L(Θ, Σ).

Theorem 2.1 Suppose that Y follows the multivariate regression model in (2.1). Then, the MLEs of Θ and Σ are given as

Θ^=XX1XY,Σ^=1nYXΘ^YXΘ^=1nYInPXY,

where PX = X(XX)− 1X′. Further, it holds that

2logLΘ^,Σ^=nlog|Σ^|+nplog2π+1.

Theorem 2.1 can be shown by a linear algebraic method, which is discussed in the next section. Note that PX is the projection matrix on the range space Ω=X. It is symmetric and idempotent, i.e.

PX=PX,PX2=PX.

Next, we consider to test the hypothesis

 H:EY=X1Θ1⇔Θ2=O, (2.2)

against K;Θ2O , where X = (X1X2), X1n × j and Θ=Θ1Θ2,Θ1;j×p. The hypothesis means that the last k − j dimensional variate x2 = (xj + 1, …, xk)′ has no additional information in the presence of the first j variate x1 = (x1, …, xj)′. In general, the likelihood ratio criterion (LRC) is defined by

 λ=maxHLΘ,ΣmaxKLΘ,Σ. (2.3)

Then we can express

2logλ=minH2logLΘ,ΣminK2logLΘ,Σ=minHnlog|Σ|+trYXΘYXΘminKnlog|Σ|+trYXΘYXΘ.

Using Theorem 2.1, we can expressed as

λ2/nΛ=nΣ^ΩnΣ^ω.

Here, Σ^Ω and Σ^ω are the maximum likelihood estimators of Σ under the model (2.1) or K and H, respectively, which are given by

 nΣ^Ω=Y−XΘ^Ω′Y−XΘ^Ω,Θ^Ω=X′X−1X′Y=Y′In−PΩY (2.4)

and

 nΣ^ω=Y−X1Θ^1ω′Y−X1Θ^1ω,Θ^1ω=X1′X1−1X1′Y=Y′In−PωY (2.5)

Summarizing these results, we have the following theorem.

Theorem 2.2 Let λ = Λn/2 be the LRC for testing H in (2.2). Then, Λ is expressed as

 Λ=SeSe+Sh, (2.6)

where

 Se=Σ^Ω,Sh=Σ^ω−Σ^Ω, (2.7)

and SΩ and Sω are given by (2.4) and (2.5), respectively.

The matrices Se and Sh in the testing problem are called the sums of squares and products (SSP) matrices due to the error and the hypothesis, respectively. We consider the distribution of Λ. If a p × p random matrix W is expressed as

W=j=1nzjzj,

where zjNpμj,Σ and z1, …, zn are independent, W is said to have a noncentral Wishart distribution with n degrees of freedom, covariance matrix Σ, and noncentrality matrix Δ=μ1μ1++μnμn. We write that WWpn,Σ;Δ. In the special case Δ=O, W is said to have a Wishart distribution, denoted by WWpn,Σ.

Theorem 2.3 (multivariate Cochran theorem) Let Y=y1yn , where yiNpμi,Σ , i = 1, …, n and y1, …, yn are independent. Let A, A1, and A2 be n × n symmetric matrices. Then:

1. YAYWpk,Σ;ΩA2=A,trA=k,Ω=EYAEY.

2. YA1Y and YA2Y are independentA1A2 = O.

For a proof of multivariate Cochran theorem, see, e.g. [3, 68]. Let B and W be independent random matrices following the Wishart distribution Wpq,Σ and Wpn,Σ respectively, with n ≥ p. Then, the distribution of

Λ=WB+W

is said to be the p-dimensional Lambda distribution with (qn)-degrees of freedom and is denoted by Λp(qn). For distributional results of Λp(qn), see [1, 3].

By using multivariate Cochran’s theorem, we have the following distributional results:

Theorem 2.4 Let Se and Sh be the random matrices in (2.7). Let Λ be the Λ-statistic defined by (2.6). Then,

1. Se and Sh are independently distributed as a Wishart distribution Wpnk,Σ and a noncentral Wishart distribution Wpkj,Σ;Δ respectively, where

 Δ=XΘ′PX−PX1XΘ. (2.8)

2. Under H, the statistic Λ is distributed as a lambda distribution Λp(k − jn − k).

Proof. Note that PΩ = PX = X(XX)− 1X′, Pω=PX1=X1X1X11X, and PΩPω = PωPΩ. By multivariate Cochran’s theorem the first result (1) follows by checking that

InPΩ2=InPΩ,PΩPω2=PΩPω,InPΩPΩPω=O.

The second result (2) follows by showing that Δ0=O, where Δ0 is the Δ under H. This is seen that

Δ0=X1Θ1PΩPωX1Θ1=O,

since PΩX1 = PωX1 = X1.

The matrices Se and Sh in (2.7) are defined in terms of n × n matrices PΩ and Pω. It is important to give expressions useful for their numerical computations. We have the following expressions:

Se=YYYXXX1XY,Sh=YXXX1XYYX1X1X11X1Y.

Suppose that x1 is 1 for all subjects, i.e., x1 is an intercept term. Then, we can express these in terms of the SSP matrix of (y', x')′ defined by

 S=∑i=1nyi−y¯xi−x¯yi−y¯xi−x¯=SyySyxSxySxx, (2.9)

where y¯ and x¯ are the sample mean vectors. Along the partition of x=x1x2′, we partition S as

 S=SyySy1Sy2Sy1S11S12Sy2S21S22. (2.10)

Then,

 Se=Syy⋅x,Sh=Sy2⋅1S22⋅1−1S2y⋅1. (2.11)

Here, we use the notation Syyx=SyySyxSxx1Sxy, Sy21=Sy2Sy1S111S1y , etc. These are derived in the next section by using projection matrices.

## 3. Idempotent matrices and max-mini problems

In the previous section, we have seen that idempotent matrices play an important role on statistical inference in multivariate regression model. In fact, letting EY=η=η1,,η,p consider a model satisfying

 ηi∈Ω=ℛR,foralli=1,…,p, (3.1)

Then the MLE of Θ is Θ^=XX1XY, and hence the MLE of η is denoted by

η^Ω=XΘ^=PΩY.

Here, PΩ = X(XX)− 1X′. Further, the residual sums of squares and products (RSSP) matrix is expressed as

SΩ=Yη^ΩYη^Ω=YInPΩY.

Under the hypothesis (2.2), the spaces ηi’s belong are the same and are given by ω = ℛ[X1]. Similarly, we have

η^ω=XΘ^ω=PωY.Sω=Yη^ωYη^ω=YInPωY,

where Θ^ω=(Θ^1ω'O) and Θ^1ω=X1X11X1Y. The LR criterion is based on the following decomposition of SSP matrices;

Sω=YInPωY=YInPΩY+YPΩPωY=Se+Sh.

The degrees of freedom in the Λ distribution Λp(fhfe) are given by

fe=ndimΩ,fh=kj=dimΩdimω.

In general, an n × n matrix P is called idempotent if P2 = P. A symmetric and idempotent matrix is called projection matrix. Let Rn be the n dimensional Euclid space, and Ω be a subspace in Rn. Then, any n × 1 vector y can be uniquely decomposed into direct sum, i.e.,

 y=u+v,u∈Ω,v∈Ω⊥, (3.2)

where Ω is the orthocomplement space. Using decomposition (3.2), consider a mapping

PΩ:yu,i.e.PΩy=u.

The mapping is linear, and hence it is expressed as a matrix. In this case, u is called the orthogonal projection of y into Ω, and PΩ is also called the orthogonal projection matrix to Ω. Then, we have the following basic properties:

(P1) PΩ is uniquely defined;

(P2) In − PΩ is the projection matrix to Ω;

(P3) PΩ is a symmetric idempotent matrix;

(P4) ℛ[PΩ] = Ω, and dim[Ω] = trPΩ;

Let ω be a subset of Ω. Then, we have the following properties:

(P5) PΩPω = PωPΩ = Pω.

(P6) PΩPω=PωΩ, where ω is the orthocomplement space of ω.

(P7) Let B be a q × n matrix, and let N(B) = {y; By = 0}. If ω = N[B] ∩ Ω, then ω ∩ Ω = R[PΩB '].

For more details, see, e.g. [3, 7, 9, 10].

The MLEs and LRC in multivariate regression model are derived by using the following theorem.

Theorem 3.1

1. Consider a function of f(Σ)=log|Σ|+trΣ1S of p × p positive definite matrix. Then, fΣ takes uniquely the minimum at Σ=S , and the minimum value is given by

minΣ>OfΣ=fS+p.

2. Let Y be an n × p known matrix and X an n × k known matrix of rank k. Consider a function of p × p positive definite matrix Σ and k × p matrix Θ=θij given by

gΘ,Σ=mlog|Σ|+trΣ1YXΘYXΘ,

where m > 0, − ∞ < θij < ∞, for i = 1, …, k; j = 1, …, p. Then, gΘ,Σ takes the minimum at

Θ=Θ^=XX1XY,Σ=Σ^=1mYInPXY,

and the minimum value is given by mlog|Σ|^+mp.

Proof. Let 1, …, p be the characteristic roots of Σ1S. Note that the characteristic roots of Σ1S and Σ1/2SΣ1/2 are the same. The latter matrix is positive definite, and hence we may assume 1 ≥ ⋯ ≥ p > 0. Then

fΣfS=log|ΣS1|+trΣ1Sp=log|Σ1S|+trΣ1Sp=i=1plogi+i10.

The last inequality follows from x − 1 ≥ log x (x > 0). The equality holds if and only if 1 = ⋯ = p = 1 ⇔ Σ=S.

Next, we prove 2. we have

trΣ1YXΘYXΘ=trΣ1YXΘ^YXΘ^+trΣ1XΘ^ΘXΘ^ΘtrΣ1YInPXY.

The first equality follows from that YXΘ=YXΘ^+XΘ^Θ and YXΘ^XΘ^Θ=O. In the last step, the equality holds when Θ=Θ^. The required result is obtained by noting that Θ^ does not depend on Σ and combining this result with the first result 1.

Theorem 3.2 Let X be an n × k matrix of rank k, and let Ω = ℛ[X] which is defined also by the set {y : y = X θ }, where θ is a k × 1 unknown parameter vector. Let C be a c × k matrix of rank c, and define ω by the set {y : y = Xθ , C θ = 0}. Then,

1. PΩ = X(XX)− 1X′.

2. PΩ − Pω = X(XX)− 1C′{C(XX)− 1C}− 1C(XX)− 1X′.

Proof. 1 Let ŷ = X(XX)− 1X′ and consider a decomposition y = ŷ + (y − ŷ). Then, ŷ′(y − ŷ) = 0. Therefore, PΩy = ŷ and hence PΩ = X(XX)− 1X′.

2. Since C θ = C(XX)− 1X′ ⋅ X θ, we can write ω = N[B] ∩ Ω, where B = C(XX)− 1X′. Using (P7),

ωΩ=PΩB=XXX1C.

The final result is obtained by using 1 and (P7).

Consider a special case C = (O Ik − q). Then ω = ℛ[X1], where X = (X1X2), X1 : n × q. We have the following results:

ωΩ=InPX1X2,PωΩ=InPX1X2X2InPX1X21X2InPX1.

The expressions (2.11) for Se and Sh in terms of S can be obtained from projection matrices based on

Ω=X=1n+InP1nX,ωΩ=InP1nPInP1nX1X2.

## 4. General linear hypothesis

In this section, we consider to test a general linear hypothesis

 Hg:CΘD=O, (4.1)

against alternatives Kg : CΘD ≠ O under a multivariate linear model given by (2.1), where C is a c × k given matrix with rank c and D is a p × d given matrix with rank d. When C = (O Ik − j) and D = Ip, the hypothesis Hg becomes H : Θ2 = O.

For the derivation of LR test of (4.1), we can use the following conventional approach: If U=YD, then the rows of U are independent and normally distributed with the identical covariance matrix DΣD, and

 EU=XΞ, (4.2)

where Ξ=ΘD. The hypothesis (4.1) is expressed as

 Hg:CΞ=O. (4.3)

Applying a general theory for testing Hg in (2.1), we have the LRC λ:

 λ2/n=Λ=SeSe+Sh, (4.4)

where

Se=U'InPXU=DYInPAYD,

and

Sh=CXX1XUCXX1C1CXX1XU,=CXX1XYDCXX1C1CXX1XYD.

Theorem 4.1 The statistic Λ in (4.4) is an LR statistic for testing (4.1) under (2.1). Further, under Hg, Λ ∼ Λd(cn − k).

Proof. Let G = (G1G2) be a p × p matrix such that G1 = D, G1G2=O, and |G| ≠ 0. Consider a transformation from Y to UV=YG1G2.

Then the rows of UV are independently normal with the same covariance matrix

Ψ=GΣG=Ψ11Ψ12Ψ21Ψ22,Ψ12:d×pd,

and

EUV=XΘG1G2=XΞΔ,Ξ=ΘG1,Δ=ΘG2.

The conditional of V given U is normal. The rows of V given U are independently normal with the same covariance matrix Ψ112 , and

EV|U=XΔ+UXΞΓ=XΔ+UΓ,

where Δ*=ΔΞΓ and Γ=Ψ111Ψ12. We see that the maximum likelihood of V given U does not depend on the hypothesis. Therefore, an LR statistic is obtained from the marginal distribution of U, which implies the results required.

## 5. Additional information tests for response variables

We consider a multivariate regression model with an intercept term x0 and k explanatory variables x1, …, xk as follows.

 Y=1θ′+XΘ+E, (5.1)

where Y and X are the observation matrices on y = (y1, …, yp)′ and x = (x1, …, xk)′. We assume that the error matrix E has the same property as in (2.1), and rank (1nX) = k + 1. Our interest is to test a hypothesis H2 ⋅ 1 on no additional information of y2 = (yq + 1, …, yp)′ in presence of y1 = (y1, …, yq)′.

Along the partition of y into (y1′, y2′) let Y, θ, Θ, and Σ partition as

Y=Y1Y2,Θ=Θ1Θ2,θ=θ1θ2,Σ=Σ11Σ12Σ21Σ22.

The conditional distribution of Y2 given Y1 is normal with mean

 EY2|Y1=1θ2′+XΘ2+Y1−1nθ1′−XΘ1Σ11−1Σ12=1nθ′˜02+XΘ˜2+Y1Σ11−1Σ12, (5.2)

and the conditional covariance matrix is expressed as

 VarvecY2|Y1=Σ22⋅1⊗In, (5.3)

where Σ221=Σ22Σ21Σ111Σ12 , and

θ˜2=θ2θ1Σ111Σ12,Θ˜2=Θ2Θ1Σ111Σ12.

Here, for an n × p matrix Y=y1yp, vec (Y) means an np-vector y1yp. Now we define the hypothesis H2 ⋅ 1 as

 H2⋅1:Θ2=Θ1Σ11−1Σ12⇔Θ˜2=O. (5.4)

The hypothesis H2 ⋅ 1 means that y2 after removing the effects of y1 does not depend on x. In other words, the relationship between y2 and x can be described by the relationship between y1 and x. In this sense, y2 is redundant in the relationship between y and x.

The LR criterion for testing the hypothesis H2 ⋅ 1 against alternatives K21:Θ˜21O can be obtained through the following steps.

(D1) The density function of Y=Y1Y2 can be expressed as the product of the marginal density function of Y1 and the conditional density function of Y2 given Y1. Note that the density functions of Y1 under H2 ⋅ 1 and K2 ⋅ 1 are the same.

(D2) The spaces spanned by each column of EY2|Y1 are the same, and let the spaces under K2 ⋅ 1 and H2 ⋅ 1 denote by Ω and ω, respectively. Then

Ω=1nY1X,ω=1nY1,

and dim(Ω) = q + k + 1, dim(ω) = k + 1.

(D3) The likelihood ratio criterion λ is expressed as

λ2/n=Λ=SΩSω=SΩSΩ+SωSΩ.

where SΩ=Y2InPΩY2 and Sω=Y2InPωY2.

(D4) Note that EY2|Y1PωPωEY2|Y1=O under H2 ⋅ 1. The conditional distribution of Λ under H2 ⋅ 1 is Λp − q(kn − q − k − 1), and hence the distribution of Λ under H2 ⋅ 1 is Λp − q(kn − q − k − 1).

Note that the Λ statistic is defined through Y2InPΩY2 and Y2PΩPωY2 , which involve n × n matrices. We try to write these statistics in terms of the SSP matrix of (y′, x′)′ defined by

S=i=1nyiy¯xix¯yiy¯xix¯=SyySyxSxySxx,

where y¯ and x¯ are the sample mean vectors. Along the partition of y=y1y2′, we partition S as

S=S11S12S1xS21S22S2xSx1Sx2Sxx.

We can show that

Sω=S221=S22S21S111S12,SΩ=S221x=S22xS21xS11x1S12x.

The first result is obtained by using

ω=1n+InP1nY1.

The second result is obtained by using

Ω=1n+Y˜1X˜=1n+InP0X+InPXInP0Y1,

where Y˜1=InP1nY1 and X˜=InP1nX.

Summarizing the above results, we have the following theorem.

Theorem 5.1 In the multivariate regression model (5.1), consider to test the hypothesis H2 ⋅ 1 in (5.4) against K2 ⋅ 1. Then the LR criterion λ is given by

λ2/n=Λ=S221xS221,

whose null distribution is Λp − q(kn − q − k − 1).

Note that S221 can be decomposed as

S221=S221x+S2x1Sxx11Sx21.

This decomposition is obtained by expressing S221x in terms of S221 , S2x1 , Sxx1 , and Sx21 by using an inverse formula

 H11H12H21H22−1=H11−1OOO+−H11−1H12IH22⋅1−1−H21H11−1I. (5.5)

The decomposition is expressed as

 S22⋅1−S22⋅1x=S2x⋅1Sxx⋅1−1Sx2⋅1. (5.6)

The result may be also obtained by the following algebraic method. We have

S221S221x=Y2PΩPωY2=Y2PωΩY2,

and

Ω=1n+Y˜1X˜,ω=1n+Y˜1.

Therefore,

ωΩ=InP11PY˜1Y˜1X˜=InP11PY˜1X˜,

which gives an expression for PωΩ by using Theorem 3.1 (1). This leads to (5.6).

## 6. Tests in discriminant analysis

We consider q p-variate normal populations with common covariance matrix Σ and the ith population having mean vector θi. Suppose that a sample of size ni is available from the ith population, and let yij be the jth observation from the ith population. The observation matrix for all the observations is expressed as

 Y=y11…y1n1y21…yq1…yqnq′. (6.1)

It is assumed that yij are independent, and

 yij~Nθi,Σ,j=1,…,ni;i=1,…,q, (6.2)

The model is expressed as

 Y=AΘ+E, (6.3)

where

A=1n10001n20001nq,Θ=θ1θ2θq.

Here, the error matrix E has the same property as in (2.1).

First, we consider to test

 H:θ1=…=θq=θ, (6.4)

against alternatives K : θi ≠ θj for some ij. The hypothesis can be expressed as

 H:CΘ=O,C=Iq−1,−1q−1. (6.5)

The tests including LRC are based on three basic statistics, the within-group SSP matrix W, the between-group SSP matrix B, and the total SSP matrix T given by

 W=∑i=1qni−1Si,B=∑i=1qniy¯i−y¯y¯i−y¯′,T=B+W=∑i=1q∑j=1niyij−y¯yij−y¯′, (6.6)

where y¯i and Si are the mean vector and sample covariance matrix of the ith population, and y¯ is the total mean vector defined by 1/ni=1qniy,¯i and n=i=1qni. In general, W and B are independently distributed as a Wishart distribution Wpnq,Σ and a noncentral Wishart distribution Wpq1,Σ;Δ respectively, where

Δ=i=1qniθiθ¯θiθ¯,

where θ¯=1/ni=1q+1niθi . Then, the following theorem is well known.

Theorem 6.1 Let λ = Λn/2 be the LRC for testing H in (6.4). Then, Λ is expressed as

 Λ=WW+B=WT, (6.7)

where W, B , and T are given in (6.6). Further, under H, the statistic Λ is distributed as a lambda distribution Λp(q − 1, n − q).

Now we shall show Theorem 6.1 by an algebraic method. It is easy to see that

Ω=A,ω=NCAA1AΩ=1n.

The last equality is also checked from that under H

EY=A1qθ=1nθ.

We have

T=YInP1nY=YInPAY+YPAP1nY=W+B.

Further, it is easily checked that

1. InPA2=InPA,PAP1n2=PAP1n.

2. PAP1nPAP1n=O.

3. fe = dim[ℛ[A]] = tr(In − PA) = n − q,

fh=dim1nA=trPAP1n=q1.

Related to the test of H, we are interested in whether a subset of variables y1, …, yp is sufficient for discriminant analysis, or the set of remainder variables has no additional information or is redundant. Without loss of generality, we consider the sufficiency of a subvector y1 = (y1, …, yk)′ of y, or redundancy of the remainder vector y2 = (yk + 1, …, yp)′. Consider to test

 H2⋅1:θ1;2⋅1=⋯=θq;2⋅1=θ2⋅1, (6.8)

where

θi=θi;1θi;2,θi;1;k×1,i=1,,q,

and

θi;21=θi;2Σ21Σ111θi;1,i=1,,q.

The testing problem was considered by [11]. The hypothesis can be formulated in terms of Maharanobis distance and discriminant functions. For its details, see [12, 13]. To obtain a likelihood ratio for H2 ⋅ 1, we partition the observation matrix as

Y=Y1Y2,Y1:n×k.

Then the conditional distribution of Y2 given Y1 is normal such that the rows of Y2 are independently distributed with covariance matrix Σ221=Σ22Σ21Σ111Σ12 , and the conditional mean is given by

 EY2|Y1=AΘ2⋅1+Y1Σ11−1Σ12, (6.9)

where Θ21˙=(θ1;21,,θq;21). The LRC for H2 ⋅ 1 can be obtained by use of the conditional distribution, and following the steps (D1)–(D4) in Section 5. In fact, the spaces spanned by each column of EY2|Y1 are the same, and let the spaces under K2 ⋅ 1 and H2 ⋅ 1 denote by Ω and ω, respectively. Then

Ω=AY1,ω=1nY1,

dim(Ω) = q + k, and dim(ω) = q + 1. The likelihood ratio criterion λ can be expressed as

λ2/n=Λ=SΩSω=SΩSΩ+SωSΩ.

where SΩ=Y2InPΩY2 and Sω=Y2InPωY2. We express the LRC in terms of W , B , and T. Let us partition W , B , and T as

 W=W11W12W21W22,B=B11B12B21B22,T=T11T12T21T22, (6.10)

where W12:q×pq , B12:q×pq , and T12:q×pq. Noting that PΩ=PA+PInPAY1 , we have

SΩ=Y2InPAInPAY1Y1InPAY11Y1InPAY2=W22W21W111W12=W221.

Similarly, noting that Pω=P1n+PInP1nY1 , we have

Sω=Y2InP1nInP1nY1Y1InP1nY11Y1InP1nY2=T22T21T111T12=T221.

Theorem 6.2 Suppose that the observation matrix Y in (6.1) is a set of samples from Npθi,Σ i = 1, …, q. Then the likelihood ratio criterion λ for the hypothesis H2 ⋅ 1 in (6.8) is given by

λ=W221T221n/2,

where W and T are given by (6.6). Further, under H2 ⋅ 1,

W221T221Λpkq1,nqk.

Proof. We consider the conditional distributions of W221 and T221 given Y1 by using Theorem 2.3, and see also that they do not depend on Y1. We have seen that

W221=Y2Q1Y2,Q1=InPAPInPAY1.

It is easy to see that Q12=Q1 , rankQ1=trQ1=nqk , Q1A=O , Q1X1 = O, and

EY2|Y1Q1EY2|Y1=O.

This implies that W221|Y1Wpknqk,Σ221 and hence W22 ⋅ 1 ∼ Wp − k(n − q − kΣ22 ⋅ 1). For T221 , we have

T221=Y2Q2Y2,Q2=InP1nPInP1nY1,

and hence

T221W221=Y2Q2Q1Y2.

Similarly, Q2 is idempotent. Using P1nPA=PAP1n=P1n , we have Q1Q2=Q2Q1=Q1 , and hence

Q2Q12=Q2Q1,Q1Q2Q1=Q.

Further, under H2 ⋅ 1,

EX2|X1Q2Q1EX2|X1=O.

## 7. General multivariate linear model

In this section, we consider a general multivariate linear model as follows. Let Y be an n × p observation matrix whose rows are independently distributed as p-variate normal distribution with a common covariance matrix Σ. Suppose that the mean of Y is given as

 EY=AΘX′, (7.1)

where A is an n × k given matrix with rank k, X is a p × q matrix with rank q, and Θ is a k × q unknown parameter matrix. For a motivation of (7.1), consider the case when a single variable y is measured at p time points t1, …, tp (or different conditions) on n subjects chosen at random from a group. Suppose that we denote the variable y at time point tj by yj. Let the observations yi1, …, yip of the ith subject be denoted by

yi=yi1yip,i=1,,n.

If we consider a polynomial regression of degree q − 1 of y on the time variable t, then

Eyi=Xθ,

where

X=1t1t1q11tptpq1,θ=θ1θ2θq.

If there are k different groups and each group has a polynomial regression of degree q − 1 of y, we have a model given by (7.1). From such motivation, the model (7.1) is also called a growth curve model. For its detail, see [14].

Now, let us consider to derive LRC for a general linear hypothesis

 Hg:CΘD=O, (7.2)

against alternatives Kg:CΘDO. Here, C is a c × k given matrix with rank c, and D is a q × d given matrix with rank d. This problem was discussed by [1517]. Here, we obtain LRC by reducing it to the problem of obtaining LRC for a general linear hypothesis in a multivariate linear model. In order to relate the model (7.1) to a multivariate linear model, consider the transformation from Y to UV :

 UV=YG,G=G1G2, (7.3)

where G1 = X(XX)− 1, G2=X˜ , and X˜ are a p × (p − q) matrix satisfying X˜X=O and X˜X˜=Ipq. Then, the rows of UV are independently distributed as p-variate normal distributions with means

EUV=AΘO,

and the common covariance matrix

Ψ=GΣG=G1ΣG1G1ΣG2G2ΣG1G2ΣG2=Ψ11Ψ12Ψ21Ψ22.

This transformation can be regarded as one from y = (y1, …, yp)′ to a q-variate main variable u = (u1, …, uq)′ and a (p − q)-variate auxiliary variable v = (v1, …, vp − q)′. The model (7.1) is equivalent to the following joint model of two components:

1. The conditional distribution of U given V is

 U|V∼Nn×qA*Ξ,Ψ11⋅2. (7.4)

2. The marginal distribution of V is

 V∼Nn×p−qO,Ψ22, (7.5)

where

A*=AV,Ξ=ΘΓ,Γ=Ψ221Ψ21,Ψ112=Ψ11Ψ12Ψ221Ψ21.

Before we obtain LRC, first we consider the MLEs in (7.1). Applying a general theory of multivariate linear model to (7.4) and (7.5), the MLEs of Ξ , Ψ112 , and Ψ22 are given by

 Ξ^=A*′A*−1A*′U,nΨ11⋅2=U′In−PA*U,nΨ22=V′V. (7.6)

Let

S=YInPAY,W=GSG=UVInPAUV,

and partition W as

W=W11W12W21W22,W12:q×pq.

Theorem 7.1 For an n × p observation matrix Y , assume a general multivariate linear model given by (7.1). Then:

1. The MLE Θ^ of Θ is given by

Θ^=AAA1AYS1XXS1X1.

2. The MLE Ψ^112 of Ψ112 is given by

nΨ112=W112=XS1X1.

Proof. The MLE of Ξ is Ξ^=A*A*1A*U. The inverse formula (see (5.5)) gives

Q=A*A*1=AA1OOO+AA1AVIpqVInPAV1AA1AVIpq=Q11Q12Q21Q22.

Therefore, we have

Θ^=Q11A+Q12VU=AA1AYG1AA1AYG2G2SG21G2SG1.

Using

G2G2SG21G2=S1G1G1S1G21G1S1,

we obtain 1. For a derivation of 2, let B=InPAV. Then, using PA*=PA+PB , the first expression of (1) is obtained. Similarly, the second expression of (2) is obtained.

Theorem 7.2 Let λ = Λn/2 be the LRC for testing the hypothesis (7.2) in the generalized multivariate linear model (7.1). Then,

Λ=|Se|/|Se+Sh|,

where

Se=DX'S1X1D,Sh=CΘ^DCRC1CΘ^D

and

R=AA1+AA1AYS1SXXS1X1X×S1YAAA1.

Here Θ^ is given in Theorem 7.1.1. Further, the null distribution is Λd(cn − k − (p − q)).

Proof. The test of Hg in (7.2) against alternatives Kg is equivalent to testing

 Hg:C*ΞD=O (7.7)

under the conditional model (7.4), where C* = (C O). Since the distribution of V does not depend on Hg, the LR test under the conditional model is the LR test under the unconditional model. Using a general result for a general linear hypothesis given in Theorem 4.1, we obtain

Λ=|S˜e|/|S˜e+S˜h|,

where

S˜e=DUInA*A*A*1A*UD,S˜h=CΞ^DC*A*A*1C*1CΞ^D.

By reduction similar to those of MLEs, it is seen that S˜e=Se and S˜h=Sh. This completes the proof.

## 8. Concluding remarks

In this chapter, we discuss LRC in multivariate linear model, focusing on the role of projection matrices. Testing problems considered involve the hypotheses on selection of variables or no additional information of a set of variables, in addition to a typical linear hypothesis. It may be noted that various LRCs and their distributions are obtained by algebraic methods.

We have not discussed with LRCs for the hypothesis of selection of variables in canonical correlation analysis, and for dimensionality in multivariate linear model. Some results for these problems can be found in [3, 18].

In multivariate analysis, there are some other test criteria such as Lawley-Hotelling trace criterion and Bartlett-Nanda-Pillai trace criterion. For the testing problems treated in this chapter, it is possible to propose such criteria as in [12].

The LRCs for tests of no additional information of a set of variables will be useful in selection of variables. For example, it is possible to propose model selection criteria such as AIC (see [19]).

## Acknowledgements

The author wishes to thank Dr. Tetsuro Sakurai for his variable comments for the first draft. The author’s research is partially supported by the Ministry of Education, Science, Sports, and Culture, a Grant-in-Aid for Scientific Research (C), no. 25330038, 2013–2015.

## References

1 - Anderson, T. W. (2003). An Introduction to Multivariate Statistical Analysis, 3rd ed. John Wiley & Sons, Inc., Hoboken, N.J.
2 - Arnold, S. F. (1981). The Theory of Linear Models and Multivariate Analysis, John Wiley & Sons, Inc., Hoboken, N.J.
3 - Fujikoshi, Y., Ulyanov, V. V., and Shimizu, R. (2010). Multivariate Statistics: High-Dimensional and Large-Sample Approximations. John Wiley & Sons, Inc., Hobeken, N.J.
4 - Muirhead, R. J. (1982). Aspect of Multivariate Statistical Theory, John Wiley & Sons, Inc., New York.
5 - Rencher, A. C. (2002). Methods of Multivariate Analysis, 2nd ed. John Wiley & Sons, Inc., New York.
6 - Seber, G. A. F. (1984). Multivariate Observations. John Wiley & Sons, Inc., New York.
7 - Seber, G. A. F. (2008). A Matrix Handbook for Statisticians. 2nd edn. John Wiley & Sons, Inc., Hobeken, N.J.
8 - Siotani, M., Hayakawa, T., and Fujikoshi, Y. (1985). Modern Multivariate Statistical Analysis: A Graduate Course and Handbook, American Sciences Press, Columbus, OH.
9 - Harville, D. A. (1997). Matrix Algebra from a Statistician’s Perspective. Springer-Verlag, New York.
10 - Rao, C. R. (1973). Linear Statistical Inference and Its Applications, 2nd ed. John Wiley & Sons, Inc., New York.
11 - Rao, C. R. (1948). Tests of significance in multivariate analysis. Biometrika. 35, 58–79.
12 - Fujikoshi, Y. (1989). Tests for redundancy of some variables in multivariate analysis. In Recent Developments in Statistical Data Analysis and Inference (Y. Dodge, ed.). Elsevier Science Publishers B.V., Amsterdam, 141–163.
13 - Rao, C. R. (1970). Inference on discriminant function coefficients. In Essays in Probability and Statistics, (R. C. Bose, ed.), University of North Carolina Press, Chapel Hill, NC, 587–602.
14 - Potthoff, R. F. and Roy, S. N. (1964). A generalized multivariate analysis of variance model useful especially for growth curve problems. Biometrika. 51, 313–326.
15 - Gleser, L. J. and Olkin, I. (1970). Linear models in multivariate analysis, In Essays in Probability and Statistics, (R.C. Bose, ed.), University of North Carolina Press, Chapel Hill, NC, 267–292.
16 - Khatri, C. G. (1966). A note on a MANOVA model applied to problems in growth curve. Ann. Inst. Statist. Math. 18, 75–86.
17 - Kshirsagar, A. M. (1995). Growth Curves. Marcel Dekker, Inc.
18 - Fujikoshi, Y. (1982). A test for additional information in canonical correlation analysis, Ann. Inst. Statist. Math., 34, 137–144.
19 - Akaike, H. (1973). Information theory and an extension of the maximum likelihood principle. In 2nd International Symposium on Information Theory (B.N. Petrov and F. Csáki, eds.). Akadémia Kiado, Budapest, Hungary, 267–281.