1
Chapter 8 – Symmetric Matricesand Quadratic Forms
Outline8.1 Symmetric Matrices8.2 Quardratic Forms8.3 Singular Values
2
8.1 Symmetric Matrices• (Spectral theorem) A matrix A is orthogonally diagonalizable
(i.e., there is an orthogonal S such that S-1AS=STAS is diagonal) if and only if A is symmetric (i.e., AT=A).
• Consider a symmetric matrix A. If and are eigenvectors of A with distinct eigenvalues λ1 and λ2, then ; that is, is orthogonal to .
• A symmetric n×n matrix A has n real eigenvalues if they are counted with their algebraic multiplicities.
1v
2v
021 vv
2v
1v
3
Example 1• If A is orthogonally diagonalizable, what is the
relationship between AT and A?• (sol)
– We have S-1AS=D or A=SDS-1=SDST, for an orthogonal S and a diagonal D. Then AT=(SDST)T=SDTST=SDST=A.
– We find that A is symmetric: AT=A.
4
Example 2• For the symmetric matrix find an orthogonal S such
that S-1AS is diagonal.• (sol)
– We will first find an eigenbasis. The eigenvalues of A are 3 and 8, with corresponding eigenvectors respectively.
7224
A
,21
and 1
2
5
Example 2 (II)– Note that the two eigenspaces, E3 and E8, are perpendicular. Therefore,
we can find an orthonormal eigenbasis simply by dividing the given eigenvectors by their lengths:
– If we define the orthogonal matrix
– then S-1AS will be diagonal, namely,
.21
51 ,
12
51
21
vv
,2112
51
||
||
21
vvS
.80031
ASS
6
Example 3• For the symmetric matrix find an orthogonal S such
that S-1AS is diagonal.• (sol)
– The eigenvalues are 0 and 3, with
– Note that the two eigenspaces are indeed perpendicular to one another.– We can construct an orthonormal eigenbasis for A by picking an
orthonormal basis of each eigenspace.
,111111111
A
.111
span and 101
,011
span 30
EE
7
Example 3 (II)– In Figure 3, the vectors form an orthonormal basis of E0, and
is a unit vector in E3. Then, is an orthonormal eigenbasis for A. We can let to diagonalize A orthogonally.
– If we apply the Gram–Schmidt process to the vectors– spanning E0, we find
– The computations are left as an exercise. For E3, we get
– Therefore, the orthogonal matrix
– diagonalizes the matrix A:
21,vv 3v
321 ,, vvv
321 vvvS
101
,011
.211
61 and
011
21
21
vv
111
31
3v
3/16/203/16/12/13/16/12/1
|||
|||
321 vvvS
300000000
1ASS
8
Orthogonal Diagonalization of a Symmetric Matrix A
• Find the eigenvalues of A, and find a basis of each eigenspace.• Using the Gram–Schmidt process, find an orthonormal basis of
each eigenspace.• Form an orthonormal eigenbasis for A by
combining the vectors you found in part (b), and let
S is orthogonal, and S-1AS will be diagonal.
nvvv
,,, 21
.|||
|||
21
nvvvS
9
Example 4• Consider an invertible symmetric 2×2 matrix A. Show that the
linear transformation T(x)=Ax maps the unit circle into an ellipse, and find the lengths of the semimajor and the semiminor axes of this ellipse in terms of the eigenvalues of A. Compare this with Exercise 2.2.50.
• (sol)– The spectral theorem tells us that there is an orthonormal eigenbasis
for T , with associated real eigenvalues λ1 and λ2. Suppose that |λ1| ≥ |λ2|. These eigenvalues will be nonzero, since A is invertible. The unit circle in R2 consists of all vectors of the form
21,vv
.)sin()cos( 21 vtvtv
10
Example 4 (II)– The image of the unit circle consists of the vectors
– an ellipse whose semimajor axis has the length , while the length of the semiminor axis is .
– In the example illustrated in Figure 4, the eigenvalue λ1 is positive, and λ2 is negative.
,)sin()cos()()sin()()cos()(
2211
21
vtvtvTtvTtvT
11v
111 v
222 v
11
8.2 Quardratic Forms• A function q(x1, x2, . . . , xn) from Rn to R is called a quadratic
form if it is a linear combination of functions of the form xixj (where i and j may be equal). A quadratic form can be written as
for a symmetric n × n matrix A.• Consider a quadratic form from Rn to R. Let B be an
orthonormal eigenbasis for A, with associated eigenvalues λ1, . . . , λn. Then
where the ci are the coordinates of with respect to B.
,)( xAxxAxxq T
xAxxq
)(
,)( 2222
211 nncccxq
x
12
Example 1
• Consider the function q(x1, x2) = 8x12-4x1x2+5x2
2 from R2 to R.Determine whether q(0,0)=0 is the global maximum, the global minimum, or neither.Recall that q(0,0) is called the global (or absolute) minimum if q(0,0)≤q(x1, x2) for all real numbers x1, x2; the global maximum is defined analogously.
• (sol)– There are a number of ways to do this problem, some of which
you may have seen in a previous course. Here we present an approach based on matrix techniques. We will first develop some theory, and then do the example.
13
Example 1 (II)– Note that we can write
– More succinctly, we can write
or– The matrix A is symmetric by construction. We find
with associated eigenvalues λ1 = 9 and λ2 = 4.– If we write , we can express the value of the function as
follows:
(Recall that since is an orthonormal basis of R2.)
21
21
2
12221
21
2
1
5228
548xx
xxxx
xxxxxx
q
5228
where,)( AxAxxq
.)( xAxxq T
,21
51 ,
12
51
21
vv
2211 vcvcx
.49)()()( 22
21
222
2112221112211 ccccvcvcvcvcxAxxq
,1 and ,0 ,1 222111 vvvvvv
21,vv
14
Example 1 (III)– The formula shows that for all nonzero ,
because at least one of the terms is positive.– Thus q(0,0)= 0 is the global minimum of the function. The preceding
work shows that the c1–c2 coordinate system defined by an orthonormal eigenbasis for A is “well adjusted” to the function q. The formulais easier to work with than the original formulabecause no term involves c1c2:
22
21 49)( ccxq
0)( xq x
22
21 4 and 9 cc
22
21 49 cc
,548 2221
21 xxxx
22
21
2221
2121 49548),( ccxxxxxxq
15
Example 2• Consider the quadratic form
Find a symmetric matrix A such that for all in R3.• (sol)
– We let
– Therefore,
.642379),,( 32312123
22
21321 xxxxxxxxxxxxq
xAxxq )( x
. if ), oft coefficien( ), oft coefficien( 212 jixxaaxa jijiijiii
332371
219
16
Positive Definite Quadratic Forms• Consider a quadratic form , where A is a symmetric
n×n matrix.We say that A is positive definite if is positive for all nonzero x in Rn, and we call A positive semidefinite if , for all x in Rn. Negative definite and negative semidefinite symmetric matrices are defined analogously. Finally, we call A indefinite if q takes positive as well as negative values.
• A symmetric matrix A is positive definite if (and only if) all of its eigenvalues are positive. The matrix A is positive semidefinite if (and only if) all of its eigenvalues are positive or zero.
• Consider a symmetric n×n matrix A. For m=1,...,n, let A(m) be the m×m matrix obtained by omitting all rows and columns of A past the mth. These matrices A(m) are called the principal submatrices of A. The matrix A is positive definite if (and only if) det(A(m))>0, for all m=1,...,n.
xAxxq )(
)(xq
0)( xq
17
Example 3• Consider an m×n matrix A. Show that the function is a
quadratic form, find its matrix, and determine its definiteness.• (sol)
– We can writeThis shows that q is a quadratic form, with matrix ATA. This quadratic form is positive semidefinite, because for all vectors in Rn.
– Note that if and only if x is in the kernel of A. Therefore, the quadratic form is positive definite if and only if
2)( xAxq
).()()()()()( xAAxxAAxxAxAxAxAxq TTTT
0)( 2 xAxq x
0)( xq
}.0{)ker(
A
18
Example 4• Sketch the curve• (sol)
– We found that we can write this equation aswhere c1, c2 are the coordinates of with respect to the orthonormal eigenbasis for
We sketch this ellipse in Figure 4.– The c1- and the c2-axes are called the principal
axes of the quadratic form– Note that these are the eigenspaces of the
matrix of the quadratic form.
.1548 2221
21 xxxx
,149 22
21 cc
x
,21
51 ,
12
51
21
vv .
5228
A
.548),( 2221
2121 xxxxxxq
.5228
A
19
Principal Axes• Consider a quadratic form , where A is a symmetric
n×n matrix with n distinct eigenvalues. Then the eigenspaces of A are called the principal axes of q. (Note that these eigenspaces will be one-dimensional.)
• Consider the curve C in R2 defined by
Let λ1 and λ2 be the eigenvalues of the matrix of q.both λ1 and λ2 are positive, then C is an ellipse. If there is a positive and a negative eigenvalue, then C is a hyperbola.
xAxxq
)(
.1),( 2221
2121 cxxbxaxxxq
cb
ba2/
2/
20
8.3 Singular Values• Show that if is a linear transformation from R2 to R2,
then there are two orthogonal unit vectors in R2 such that vectors are orthogonal as well (although not necessarily unit vectors).
• (sol)– Following the hint, we first note that matrix ATA is symmetric, since
(ATA)T=AT(AT)T=ATA. The spectral theorem tells us that there is an orthonormal eigenbasis for ATA, with associated eigenvalues λ1, λ2. We can verify that vectors and are orthogonal, as claimed:
xAxL )(
21 and vv
)L( and )( 21 vvL
21,vv
11)( vAvL
22 )( vAvL
.0)()()()()( 212221212121 vvvvvAAvvAvAvAvA TTTT
21
Example 2• Consider the linear transformation where
– Find an orthonormal basis of R2 such that vectors are orthogonal.
– Show that the image of the unit circle under transformation L is an ellipse. Find the lengths of the two semiaxes of this ellipse, in terms of the eigenvalues of matrix ATA.
• (sol)– We will find an orthonormal eigenbasis for matrix ATA:
– The characteristic polynomial of ATA isλ2-125λ+2500=(λ-100)(λ-25),so that the eigenvalues of ATA are λ1=100 and λ2=25.
xAxL )(
6726
A
21,vv )( and )( 21 vLvL
40303085
6726
6276
AAT
.21
span1530
3060ker and
12
span60303015
ker 25100
EE
22
Example 2 (II)– To find an orthonormal basis, we need to multiply these vectors by the
reciprocals of their lengths:
– The unit circle consists of the vectors of the form , and the image of the unit circle consists of the vectors . This image is the ellipse whose semimajor and semiminor axes are . What are the lengths of these axes?Likewise,Thus,
– We can compute the lengths of vectors directly, of course, but the way we did it before is more informative. For example,
so that
21
51,
12
51
21 vv
21 )sin()cos( vtvtx
)()sin()()cos()( 21 vLtvLtxL
)( and )( 21 vLvL
111111111112
1 )()()()( )( vvvvvAAvvAvAvL TTT
22
2 )( vL
.525 )( and 10100 )( 2211 vLvL
)( and )( 21 vLvL
,20
105
11
26726
51)( 11
vAvL
.1020
105
1)( 1
vL
23
Example 2 (III)
24
Singular Values• The singular values of an m×n matrix A are the square roots of the
eigenvalues of the symmetric n× n matrix ATA, listed with their algebraic multiplicities. It is customary to denote the singular values by σ1, σ2,..., σn, and to list them in σ1≥σ2≥···≥σn.
• Let be an invertible linear transformation from R2 to R2. The image of the unit circle under L is an ellipse E. The lengths of the semimajor and the semiminor axes of E are the singular values σ1 and σ2 of A, respectively.
• Let be a linear transformation from Rn to Rm. Then there is an orthonormal basis of Rn such that– Vectors are orthogonal, and– The lengths of vectors are the singular values σ1,
σ2, . . . , σn of matrix A.– To construct , find an orthonormal eigenbasis for matrix ATA.
Make sure that the corresponding eigenvalues λ1, λ2, . . . , λn appear in descending order: λ1≥λ2≥···≥λn.
xAxL
)(
xAxL
)(nvvv
,,, 21
)(,),(),( 21 nvLvLvL
)(,),(),( 21 nvLvLvL
nvvv
,,, 21
25
Example 3• Consider the linear transformation
– Find the singular values of A.– Find orthonormal vectors in R3 such that
are orthogonal.– Sketch and describe the image of the unit sphere under the transformation
L.
• (sol)–
– The eigenvalues are λ1=3, λ2=1, λ3=0. The singular values of A are
.011110
where)(
AxAxL
321 ,, vvv )( and ),(),( 321 vLvLvL
.110121011
011110
011110
AAT
.0 ,1 ,3 332211
26
Example 3 (II)– Find an orthonormal eigenbasis v1, v2, v3 for ATA:
– We compute and check orthogonality:
– We can also check that the length of is σi :
11
1
31,
101
21,
121
61
11
1span)ker(,
101
span,121
span
321
013
vvv
AEEE
321 ,, vAvAvA
.00
,11
21,
33
61
321
vAvAvA
ivA
.0,1,3 332211 vAvAvA
27
Example 3 (III)– The unit sphere in R3 consists of all vectors of the form
– The image of the unit sphere consists of the vectors.1 where 2
322
21332211 cccvcvcvcx
.1 where)()()( 22
212211 ccvLcvLcxL
28
Singular-Value Decomposition (SVD)• If A is an m×n matrix of rank r , then the singular values σ1,..., σr
are nonzero, while σr+1,..., σn are zero.• Any m×n matrix A can be written as A=UΣVT , where U is an
orthogonal m×m matrix; V is an orthogonal n×n matrix; and Σ is an m×n matrix whose first r diagonal entries are the nonzero singular values σ1,... , σr of A, and all other entries are zero (where r = rank(A)). Alternatively, this singular value decomposition can be written as
where the and the are the columns of U and V, respectively.
,111Trrr
T vuvuA
iu
iv
29
Singular-Value Decomposition (SVD) (II)
30
Example 4• Find an SVD for• (sol)
– We find
– The columns of U are defined by
– Finally,
– You can check that A=UΣVT .
.6726
A
2112
51 that so ,
21
51 and
12
51
21 Vvv
21 and uu
.1221
51 thereforeand
,12
511,
21
511
22
211
1
U
vAuvAu
.50010
00
2
1
31
Example 5• Find an SVD for• (sol)
– We find that
– Check that A = UΣVT.
.011110
A
.010003 and
,2/12/12/12/1,
3/12/16/13/106/2
3/12/16/1
UV