STAT 340: Applied Regression Methods€¦ · 3 = 0 @0 1 1 A 9 =; forms a basis for R3. The set given by D @= 8

1/21

STAT 340: Applied Regression Methods

Lecture Notes 2:Simple matrix properties

for the study of linear models

2/21

Basis

A basis in a vector space M is a set of linearly independent vectors suchthat every x ∈M is a linear combination of vectors in the set.

3/21

Example

The set of vectors

D =

x1 =

100

, x2 =

010

, x3 =

001

forms a basis for R3. The set given by

D∗ =

x1 =

100

, x2 =

330

, x3 =

222

also forms a basis for R3.

4/21

This example illustrates the fact that bases are not unique. However, therepresentation of a vector as a linear combination of basis elements isunique.

To see this, for x ∈M, suppose we can write x =∑r

i=1 αivi andx =

∑ri=1 βivi where v1, . . . , vr for a basis for M.

Then x − x = 0 =∑r

i=1 αivi −∑r

i=1 βivi =∑r

i=1(αi − βi )vi .

But we know the vi are linearly independent since they form a basis.Therefore, αi − βi = 0, or equivalently αi = βi , for all i .

Note: the coefficients of the vectors are called with the coordinates withrespect to that basis.

5/21

Practice

Define two different bases for R2. For each set, represent the vector

v =

(12

)in terms of the basis elements.

6/21

Span and column space

The span of a set of vectors D, written S(D), is a vector space given bythe set of all possible linear combinations of elements of D.

Notes:

1. a basis for a vector space M is a linearly independent set ofelements of M whose span is M.

2. every basis of a vector space M contains the same number ofelements, called the dimension of M, written dim(M), or the rankof M, denoted r(M).

7/21

Span and column space

Suppose A is an n× p matrix so that each column of A is a vector in Rn,i.e. A = (x1, x2, . . . , xp) where xi ∈ Rn, i = 1, . . . , p. The space spannedby the columns of A, S(A), is called the column space of A, writtenC (A). That is, S(A) = C (A) = {z : Ax = z , x ∈ Rp}.

8/21

Example

Suppose A =

1 01 11 2

. Then

C (A) = S

1

11

,

012

=

α 1

11

+ β

012

: α, β ∈ R1

Note that r(A) = 2 since columns are linearly independent.

9/21

Some important results

Suppose M = Rn, then:

1. dim(M) = r(M) = n.

2. n + 1 vectors in M (n-dimensional space) must be linearlydependent.

3. n vectors in M form a basis for M if and only if they are linearlyindependent (span the space.)

4. D = {x1, . . . , xr}, r < n cannot form a basis for M.

10/21

Practice

Suppose B =

1 21 22 4

.

1. Define the column space of B, denoted C (B).

2. Does C (B) form a basis for R3? Why or why not?

11/21

Orthogonality

Suppose M is a vector space in Rn and x , y ∈M. x and y are said tobe orthogonal, written x ⊥ y , if x ′y = 0.

Two subspace N1 and N2 are said to be orthogonal if for every x ∈ N1

and y ∈ N2, x ′y = 0.

Two vectors (spaces) are orthogonal if the projection of one onto theother results in the null. More on projection operators soon!

12/21

Orthogonality

Suppose N is a subspace of Rn. We say {x1, . . . , xn} is an orthogonalbasis for N if for every i 6= j , x ′i xj = 0.

Furthermore, {x1, . . . , xn} is an orthonormal basis for N if in additionx ′i xi = 1 ∀i .

Note: Two orthogonal vectors are linearly independent.

13/21

Practice

Verify that

{( √3212

),

(− 1

2√32

)}forms an orthonormal basis for R2.

14/21

Orthogonality

Let N be a subspace of a vector space M⊂ Rn. DefineN⊥ = {y ∈M : y ⊥ N} = {y : x ′y = 0, x ∈ N}. N⊥ is called theorthogonal complement of N with respect to M.

A square matrix P is said to be orthogonal if and only if PP′ = P′P = I.That is, P−1 = P′. Note that this is the same as saying that the columnsof P are orthonormal.

15/21

Singularity

Suppose A is an n × n matrix. A is said to be non-singular if there existsa matrix A−1 such A−1A = AA−1 = I. If no such matrix exists, then Ais said to be singular.

For B an n × n, non-singular matrix,tr(A) = tr(ABB−1) = tr(BAB−1) = tr(B−1AB).

16/21

Some important results

Let A be an n × n matrix. The following 5 conditions are equivalent:

1. A is non-singular

2. r(A) = n

3. C (A) is a basis for Rn

4. Columns of A are linearly independent

5. Ax = 0 implies x = 0 for all x ∈ Rn.

17/21

Null space

The set of all x such that Ax = 0 is a vector space and is called the nullspace of A, written N(A).

Notes:

1. Suppose A is n × n. If r(A) = r , then r(N(A)) = n − r .

2. For an n × p matrix A, N(A) = C (A′)⊥.

18/21

Example

Let A =

(1 12 2

). We know that C (A) = S

{(12

)}.

N(A) = {x : Ax = 0} =

{x :

(1 12 2

)(x1x2

)=

(00

)}

=

{x :

(x1 + x2

2x1 + 2x2

)=

(00

)}= {x : x1 = −x2}

= S{(

1−1

)}Note that together the null space of A and the column space of A spanR2.

19/21

Positive definite

A symmetric n × n matrix A is positive definite if x ′Ax > 0 for all x 6= 0,x ∈ Rn.

A symmetric n × n matrix A is positive semi-definite (non-negativedefinite) if x ′Ax ≥ 0 for all x 6= 0, x ∈ Rn.

20/21

Some important results

1. The eigenvalues of a positive definite matrix are all positive and theeigenvalues of a positive sem-definite maxtrix are all non-negative.

2. Covariance matrices are positive definite.

3. An n × n symmetric matrix is positive definite if and and only ifaii > 0 for i = 1, . . . , n where aii is the ith diagonal element of Aand the determinant of every submatrix (from top left, movingdown) of A is positive.

21/21

Example

Let A =

2 −1 1 1−1 4 0 21 0 1 31 2 3 2

. A is positive definite because:

1. aii > 0, i = 1, 2, 3, 4 and

2. det

(2 −1−1 4

)= 8− 1 = 7 > 0, det

2 −1 1−1 4 01 0 1

> 0 and

det(A) > 0.