linear algebra course part 2

7/28/2019 linear algebra course part 2

1/94

Chapter 2

Solving Linear Equations

Po-Ning Chen, Professor

Department of Computer and Electrical Engineering

National Chiao Tung University

Hsin Chu, Taiwan 30010, R.O.C.


2/94

2.1 Vectors and linear equations 2-1

What is this course Linear Algebra about?

Algebra ()

The part of mathematics in which letters and other general sym-bols are used to represent numbers and quantities in formulas and

equations.

Linear Algebra ()

To combine these algebraic symbols (e.g., vectors) in a linear fashion.

So, we will not combine these algebraic symbols in a nonlinear fashionin this course!

Example of nonlinear equations forx = x1x2: x1x2 = 1

x1/x2 = 1

Example of linear equations forx =

x1x2

:

x1 + x2 = 4

x1 x2 = 1


3/94

2.1 Vectors and linear equations 2-2

The linear equations can always be represented as matrix operation, i.e.,

Ax = b.

Hence, the central problem of linear algorithm is to solve a system of linearequations.

Example of linear equations. x 2y = 1

3x + 2y = 11

1 2

3 2 x

y = 1

11 2 A linear equation problem can also be viewed as a linear combination prob-

lem for column vectors (as referred by the textbook as the column pictureview). In contrast, the original linear equations (the red-color one above) is

referred by the textbook as the row picture view.

Column picture view: x

1

3

+ y

2

2

=

1

11

We want to find the scalar coefficients of column vectors

1

3and

2

2 to form

another vector 1

11

.


4/94

2.1 Matrix equation 2-3

From now on, we will focus on the matrix equation because it can be easilyextended to higher dimension ( 2).

Ax = b.In terminologies,

Ax = b will be referred to as matrix equation.

A will be referred to as coefficient matrix.

x will be referred to as unknown vector.

More terminologies:

The elements of a matrix A as well as its size are indexed first by row, and

then by column.Example.

A23 =

a1,1 a1,2 a1,3a2,1 a2,2 a2,3


5/94

2.1 Other representations of matrix 2-4

In MATLAB:

MATLAB (Matrix Laboratory) is a numerical computing environment,

innovated by Cleve Moler, and developed by MathWorks. It allows matrix

manipulation, plotting of functions and data, implementation of algorithms,

creation of user interfaces, and interfacing with programs in other languages.

A matrix

A2,3 = a1,1 a1,2 a1,3a2,1 a2,2 a2,3 = 1 2 3

2 5 2is represented by

A=[1 2 3 ; 2 5 -2] % Again, row first, and then column.

% Rows are separated by semicolon ;

A vector x = 1 1 1 is inputed asx=[1 ; -1 ; 1]


6/94


The matrix multiplication b = Ax is then given by

b=Ax

Note: The textbook refers to this as dot product (equivalently, innerproduct) in MATLAB, which is not entirely correct. MATLAB treated this as the usual matrix multiplication.

An element of a matrix A can be extracted by row and column indexes.

A(1,2) % This is exactly a1,2 in A2,3 =

a1,1 a1,2 a1,3a2,1 a2,2 a2,3

.

The row vector of a matrix A can be extracted by row index.

A(1,:) % This is exactly a1,1 a1,2 a1,3 in A2,3. The submatrix of a matrix A can be extracted by row and column index

ranges.

A(1:2,1:2) % This is exactly a1,1 a1,2a

2,1a

2,2 in A2,3.


7/94


The matrix multiplication b = Ax can then be rewritten by two forms:

% Inner product with row vectors

b=[A(1,:)*x ; A(2,:)*x]% Linear combination of column vectors

b=A(:,1)*x(1)+A(:,2)*x(2)+A(:,3)*x(3)

The above MATLAB rules for matrix manipulation can be easily extended:

A(:,2)=[] % Eliminate the second column.A(1:2,:)=[] % Eliminate the first two rows.

A=[A; a3,1 a3,2 a3,3] % Add a third row to matrix A.

v=[3:5] % v =

3

4

5


8/94


Some useful MATLAB functions appear in textbook Problems.

A=eye(3) % Generate a 3-by-3 identity matrix A = 1 0 00 1 00 0 1

A=ones(3) % Generate a 3-by-3 all-one matrix A =

1 1 1

1 1 1

1 1 1

A=zeros(3) % Generate a 3-by-3 all-zero matrix A =

0 0 00 0 00 0 0

v=ones(3,1) % Generate a 3-by-1 all-one vector v =

1

1

1

A=zeros(3,1) % Generate a 3-by-1 all-zero vector A =

0

0

0


9/94

2.2 The idea of elimination 2-8

Is there a systematic method to solve the linear equations?

Answer: Forward elimination and back(ward) substitution.

Example.

x 2y = 1

3x + 2y = 11

Step 1: Forward elimination.

Eliminate in a forward sequential order, i.e., x, y, . . ..x 2y = 1 . . . (1)

3x + 2y = 11 . . . (2)

x 2y = 1 . . . (1)(3x 3x) + (2y (6y)) = (11 3) . . . (2) 3 (1)

x 2y = 1 . . . (1)

8y = 8 . . . (2) 3 (1)

Multiplier


10/94

2.2 The idea of elimination 2-9

Step 2: Back(ward) substitution.

Substitute in a backward sequential order, i.e., . . ., y, x.

x 2y = 1 . . . (i)8y = 8 . . . (ii)

x 2y = 1 . . . (i)

y = 1

x 2 = 1 . . . y = 1 substitude (i)y = 1

x = 3

y = 1

After forward elimination, the equations form an upper triangle.x 2y = 1

8y = 8

In terminology, the element (e.g., x in the first equation) that is used to elimi-

nate the same unknown in the remaining equations is called pivot. The pivotsare on the diagonal of the upper triangle after elimination.


11/94

2.2 Failure of forward elimination 2-10

Can the systematic method fail?

Answer: IfAx = b has no solution or has infinitely many solutions, then some

pivots will disappear during the process of forward elimination.Example.

x 2y = 1 . . . (1)

2x 4y = 11 . . . (2)

x 2y = 1 . . . (1)

(2x 2x) + (4y (4y)) = (11 2) . . . (2) 2 (1)

x 2y = 1 . . . (1)

0 = 9 . . . (2) 2 (1) No solution since 0 = 9!

Hence, the next pivot disappears (or was canceled). 2

Multiplier

If obtaining 0 = 0, then there are infinitely many solutions for the linear

equations.


12/94

2.2 Failure of forward elimination 2-11

Can the systematic method fail if the solution exists and is unique?

Answer: Yes, if the sequence for forward elimination is not properly or-

dered.Example. Sequence of choosing pivots in forward elimination {x from (1), yfrom (2), z from (3)}.

x 2y + z = 1 . . . (1)

2x 4y + z = 11 . . . (2)x + y z = 22 . . . (3)

This has a unique solution

(x,y,z) = (12, 1, 9).

x 2y + z = 1 . . . (1)

z = 9 . . . (2) 2 (1)

3y 2z = 21 . . . (3) 1 (1)

No y from (2) can be found!

2

In such case, one can simply switch (2) with any of the remaining equations

(e.g., (3)) of which its y still exists (i.e., has a coefficient other than zero).


13/94

2.3 Forward elimination/back substitution in matrix form2-12

The forward elimination/back substitution can be easily done in matrix form.

Example. The pivots are the diagonal elements. 2 4 24 9 32 3 7

xyz

= 2810

. . . (1) with 2 the first pivot. . . (2). . . (3)

2 4 2

0 1 1

0 1 5

x

y

z =

2

4

12

. . . (1) with 2 the first pivot

. . . (2) 2 (1)

. . . (3) (1) (1)

2 4 20 1 10 1 5

x

y

z

=

2

4

12

. . . (1)

. . . (2) with 1 the second pivot

. . . (3)

2 4 20 1 10 0 4

xyz

= 248

. . . (1). . . (2) with 1 the second pivot. . . (3) 1 (2)

Based on the above, back substitution can be straightforwardly followed. 2

The above procedure transforms Ax = b into Ux = c, where U is an upper

triangular matrix.


14/94


We summarize the algorithmic procedure of forward elimination for n nmatrix A and n unknowns as follows.

Use the first diagonal element as the first pivot to create zeros below it.

If the diagonal element is zero, switch the row with any row below it, forwhich its respective element at the same column is non-zero.

Use the new second diagonal element as the second pivot to create zerosbelow it.

If the diagonal element is zero, switch the row with any row below it, forwhich its respective element at the same column is non-zero.

Use the new third diagonal element as the third pivot to create zeros below

it. If the diagonal element is zero, switch the row with any row below it, for

which its respective element at the same column is non-zero.

. . .

Repeat the above procedure until either an upper-triangular U is resulted orthe next pivot does not exist even with row switching.


15/94


This algorithmic procedure can be graphically illustrated as:

... ... ... . . . ...

0

0 ... ... ... . . . ...

0

0 0 0 ... ... ... . . . ...

0 0

0 0 0 ... ... ... . . . ...

0 0 0

Terminology:

When a set of linear equations has either no solution or infinite many solutions,

it is often referred to as a singular system.


16/94

2.3 Elimination using matrices 2-15

Reminder of terminologies (in English)

Sigma notation: The dot or inner product can be expressed using the

so-called sigma notation.

v w =n

j=1

vjwj.

m by n matrix: The terminology to describe the size of a matrix with m

rows and n columns.

Entry of a matrix: ai,j in the ith row and jth column of matrix A isreferred to as an entry of A.

Component of a vector: vj in a vector v = v1 v2 vn is referredto as a component ofa.


17/94

2.3 The matrix form of forward elimination 2-16

The forward elimination process can be implemented by multiplying the

elimination matrix.

Example. The pivots are the diagonal elements. 2 4 24 9 3

2 3 7

xy

z

=

28

10

. . . (1) with 2 the first pivot. . . (2)

. . . (3)

2 4 20 1 1

0 1 5

xyz

= 2412

. . . (1) with 2 the first pivot. . . (2) 2 (1). . . (3) (1) (1)

2 4 20 1 1

0 1 5 x

y

z = 2

4

12. . . (1)


. . . (3)

2 4 20 1 10 0 4

x

y

z

=

2

4

8

. . . (1)


. . . (3) 1 (2)


1 0 02 1 0

0 0 1

2 4 24 9 3

2 3 7

1 0 00 1 0(1) 0 1

2 4 20 1 1

2 3 7

1 0 00 1 0

0 1 1

2 4 20 1 1

0 1 5


18/94


The forward elimination process can then be implemented by multiplying the elim-

ination matrices E2,1, E3,1 and E3,2 in sequence.

1 0 00 1 00 1 1

E3,2

1 0 00 1 0(1) 0 1

E3,1

1 0 02 1 00 0 1

E2,1

2 4 24 9 32 3 7

x

= 1 0 00 1 00 1 1

E3,2

1 0 00 1 0(1) 0 1

E3,1

1 0 02 1 00 0 1

E2,1

2810

which immediately gives 2 4 20 1 10 0 4

x = 248

.2

For matrix multiplications, associative law is true (e.g., (E3,1E2,1)A =E3,1(E2,1A)) but commutative law is false (e.g., EA = AE).


19/94


We now understand why the forward elimination process can be implemented

by multiplying the elimination matrix.

What if a row exchange is needed (because the next diagonal element iszero) during the process?

Answer: Use the row exchange matrix. Specifically, insert the row ex-

change matrix inbetween the elimination matrices.

Denote by Pi,j the row exchange matrix that exchanges row i androw j by left-multiplying it.

Pi,j is an identity matrix with row i and j reversed. For example,

P2,3 = 1 0 0

0 0 10 1 0 The original row 3. The original row 2.


20/94


Example. The pivots are the diagonal elements.

2 4 24 8 3

2 3 7 x

y

z = 2

8

10. . . (1) with 2 the first pivot. . . (2) (Replace the middle 9 by 8)

. . . (3)

2 4 20 0 1

0 1 5

x

y

z

=

2

4

12

. . . (1) with 2 the first pivot. . . (2) 2 (1)

. . . (3) (1) (1)

2 4 20 1 50 0 1

xyz

= 2124

. . . (1). . . (2) with 1 the second pivot. . . (3)

2 4 2

0 1 50 0 1

xyz =

2

124 . . . (1). . . (2) with 1 the second pivot

. . . (3) 0 (2)


1 0 02 1 00 0 1

2 4 24 8 32 3 7

1 0 00 1 0

(1) 0 1

2 4 20 0 1

2 3 7

1 0 00 0 10 1 0

2 4 20 0 10 1 5

1 0 0

0 1 00 0 1

2 4 2

0 1 50 0 1


21/94


The forward elimination process can then be implemented by multiplying E2,1, E3,1,

P2,3 and E3,2 in sequence.

1 0 00 1 00 0 1

E3,2

1 0 00 0 10 1 0

P2,3

1 0 00 1 0(1) 0 1

E3,1

1 0 02 1 00 0 1

E2,1

2 4 24 8 32 3 7

x

= 1 0 00 1 00 0 1

E3,2

1 0 00 0 10 1 0

P2,3

1 0 00 1 0(1) 0 1

E3,1

1 0 02 1 00 0 1

E2,1

2810

which immediately gives 2 4 20 1 50 0 1

x = 2124

.2


22/94

2.3 The augmented matrix for forward elimination 2-21

Since we need to multiply both A and b by E = E3,2P2,3E3,1E2,1 in order to

solve Ax = b, e.g.,

1 0 00 1 00 0 1

E3,2

1 0 00 0 10 1 0

P2,3

1 0 00 1 0(1) 0 1

E3,1

1 0 02 1 00 0 1

E2,1

2 4 24 8 32 3 7

x

= 1 0 0

0 1 00 0 1

E3,2

1 0 00 0 10 1 0

P2,3

1 0 00 1 0(1) 0 1

E3,1

1 0 02 1 00 0 1

E2,1

2810

we can introduce a so-called augmented matrix A b so that x can besolved by directly computingE

A b

= E

2 4 2 2

4 9 3 8

2 3 7 10

=

2 4 2 20 1 5 12

0 0 1 4

.


23/94

2.3 Appendix 2-22

If we place E2,1 on the right, what would happen then?

Answer: Row operations will become column operations. For example, 1 0 02 1 00 0 1

2 4 24 8 32 3 7

= 2 4 20 0 12 3 7

where

(row 1) = (row 1)

(row 2) = (row 2) 2 (row 1)

(row 3) = (row 3)

becomes

2 4 2

4 8 32 3 7

1 0 0

2 1 00 0 1 =

6 4 2

12 8 34 3 7

where

(column 2) = (column 2)

(column 1) = (column 1) 2 (column 2)


Exchange1 2rowcolumn

%


24/94

2.3 Appendix 2-23

If we place P2,3 on the right, what would happen then?

Answer: Row operations will become column operations. For example,1 0 00 0 10 1 0

2 4 24 8 32 3 7

= 2 4 22 3 74 8 3

where

(row 1) = (row 1)

(row 2) = (row 3)

(row 3) = (row 2)

becomes

2 4 2

4 8 32 3 7

1 0 0

0 0 10 1 0 =

2 2 4

4 3 82 7 3

where




Exchange2 3rowcolumn

%


25/94

2.3 Appendix 2-24

The below definition gives what is required by Problem 8.

(Problem 8, Section 2.3) The determinant of M = a bc d is det(M) =ad bc. Subtract times row 1 from row 2 (i.e., row 2 - row 1) to

produce a new M. Show that det(M) = det(M) for every . When

= c/a, the product of pivots equals the determinant: (a)(d b) equals

ad bc.


26/94

2.3 Appendix 2-25

Definition (Determinant ): A determinant, often denoted as

det(A) or simply detA, is a determinant factor associated with a square

matrix. For example, for a matrix formed by vw, where v and w are

2-dimensional (column) vectors, the determinant is the (signed) area of the

parallelogram formed by v and w, and is given by

detA = detv1 v2w1 w2 = v1 v2w1 w2 = v1w2 v2w1

w

R

k

v

Area of parallelogram = vw sin() ( : from v to w counterclockwisely)

= v

w

1 cos2() = vw1 v wvw2

=

v2w2 (v w)2

Area of parallelogram = vw sin(2 ) ifA = w1 w2v1 v2


27/94

2.3 Appendix 2-26

We can say more about determinant.

The determinant of a matrix formed by v

w

u, where v, w and u are

3-dimensional (column) vectors, the determinant is the (signed) volumn of

the parallelepiped formed by v, w and u.

. . .

(We will introduce the determinant in great detail in Chapter 5.)


28/94

2.3 Appendix 2-27

The below definition gives what is required by Problem 30.

(Problem 30, Section 2.3) Write M = 3 45 7 as a product of many factorsA =

1 0

1 1

and B =

1 1

0 1

.

(a) What matrix E subtracts row 1 from row 2 to make row 2 of EM

smaller?(b) What matrix F subtracts row 2 of EM from row 1 to reduce row 1 of

F EM?

(c) Continue Es and Fs until (many Es and Fs) times (M) is (A or B).

(d) E and F are the inverses of A and B! Moving all Es and Fs to theright side will give you the desired result M = product ofAs and Bs.

This is possible for all matrices M =

a b

c d

> 0 that have ad bc = 1.


29/94

2.3 Appendix 2-28

Definition (Positivity of a matrix): A positive matrix is a matrix in whichall the elements are greater than zero. It is often denoted by A > 0.

Particularly note that a positive matrix is different from a positive definitematrix, which we will introduce in Chapter 6.


30/94

2.4 Rules for matrix operations 2-29 Let us revisit matrix operations and summarize the rules that they obey.

Matrix addition is simply a term-by-term addition.

The definition of matrix multiplication is repeated below.Definition (Product of matrices): The product of two matricesis defined as the inner products ofrow vectors and column vectors respec-

tively from the first matrix and the second matrix, counted from the left.

Specifically as an example,

A23B32 =

a1a2

b1 b2

a1b1 a

1b2

a2b1 a2b2

22

=

a1 b1 a1 b2a2 b1 a2 b2

22

,

where for the two matrices on the extreme left of the above equation, 2

is the number of vectors, and 3 is the dimension of the vectors.Note that the dimension of the vectors shall be the same, otherwise the

inner product operation cannot be performed.


31/94

2.4 Rules for matrix operations 2-30 Rules or laws that matrix operations should obey.

Commutative law for addition: A + B = B + A.

Commutative law for multiplication: AB = BA. Distributive law from the left: A(B + C) = AB + AC.

Distributive law from the right: (A + B)C = AC+ BC.

Distributive law with respect to scaler: (A + B) = A + B.

Associative law for addition: (A + B) + C = A + (B + C).

Associative law for multiplication: (AB)C = A(BC).

Power law:

ApAq = Ap+q, ifp and q are positive integers

ApAq = Ap+q, ifp or q is a negative integer, and A1 exists

A0 = I, ifA is a square matrix,

where I is an identity matrix

Note: We can talk about the product AA

only when A is a square matrix.


32/94

2.4 Rules for matrix operations 2-31 The way to prove these rules may require the below two pictures.

Column picture of matrix product: From the definition of matrix product,

we can see that

A23B32 =a1a2

b1 b2

=a1a2

b1

a1a2

b2

a1b1 a

1b2

a2b1 a2b2

22

or we can simply write

A23B32 = A b1 b2 = Ab1 Ab2 .In summary, the jth column of product AB is given by Abj, where bj is

the jth column ofB.


33/94

2.4 Rules for matrix operations 2-32 Row picture of matrix product: From the definition of matrix product,

we can see that

A23B32 = a1

a2 b1 b2 =

a1 b1 b2a2b1 b2

a1b1 a1b2a2b1 a

2b2

22

or we can simply write

A23B32 = a1

a

2B = a1B

a

2B .In summary, the ith row of product AB is given by aiB, where a

i is the

ith row of A.


34/94

2.4 Block manipulation of matrices 2-33 The above column or row picture indicates that we can manipulate the ma-

trix in blocks. For example, matrix B can be divided into two blocks whencalculating the matrix product.

AB = A b1 b2 = Ab1 Ab2 . Recall that we can solve the linear equations using the augmented matrix

A b

, which consists of two blocks of different sizes.

To generalize the idea, we can represent matrix A in the form

A =

1 0 1 0 1 0

0 1 0 1 0 1

1 0 1 0 1 0

0 1 0 1 0 1

=

I I I

I I I


35/94

2.4 Block manipulation of matrices 2-34 We can treat blocks as number entries in matrix operation, which is usually

simpler.

Example.A1,1 A1,2A2,1 A2,2

+

B1,1 B1,2B2,1 B2,2

=

A1,1 + B1,1 A1,2 + B1,2A2,1 + B2,1 A2,2 + B2,2

and

A1,1 A1,2A2,1 A2,2

B1,1 B1,2B2,1 B2,2

= A1,1B1,1 + A1,2B2,1 A1,1B1,2 + A1,2B2,2A2,1B1,1 + A2,2B2,1 A2,1B1,2 + A2,2B2,2


36/94

2.4 Block manipulation of matrices 2-35 By block manipulation, the product AB does not have to be calculated/represented

in terms of rows of A and columns of B, but can be done in a vice versa

way.

AB =a1a2

b1 b2

=a1b1 a1b2a2b1 a

2b1

Rpresentation based on Inner product

=

c1 c2

d1d2

= c1d

1 + c2d

2 Representation based on Outer product

The representation using inner product is more convenient in the sense that

we can have an entry-wise identification of elements in AmnBnp. Hence, the

(i, j)th entry of AmnBnp is the inner product of row i of A and column j

ofB, i.e.,

aibj = ai bj.

Hence, we need to perform m p inner products to obtain AmnBnp.


37/94

2.4 Block manipulation of matrices 2-36

The representation using outer product can represent AB as sum of all outer

products ofcolumn i ofA and row j ofB, i.e.,

AmnBnp =c1 c2 cn

d1d2...

dn

= ni=1

cidi.

Hence, we need to perform n outer products to obtain AmnBnp.


38/94

2.4 Block manipulation of matrices 2-37 By block manipulation, the forward elimination can be done in a

block-by-block fashion.

Example.

E2,1F = 1 0

ca1 1

a bc d

=a b

0 d ca1b

and

E2,1

F = I 0CA1 I A B

C D =

A B

C CA1A D CA1B

= A B0 D CA1B The matrix D CA1B is important in linear algebra; hence, it gets itself a

terminology Scher complement, named after Issai Scher.


39/94

2.4 Appendix 2-38

What is the (i, j)th entry ofABC?

Solution:

The (i, j)th entry of AD is the inner product of row i of A and column jofD, i.e. aidj.

The column j of D = BC is the matrix multiplication of matrix B andcolumn j ofC, i.e., dj = Bcj.

Therefore, the answer is aiBcj =

ai,1 ai,2

Bc1,jc2,j

...

.

Exercise: What is the (i, j)th entry ofABCD?


40/94

2.4 Appendix 2-39Exercise: Problems 27, 28 and 37 in textbook.

(Problem 27, Section 2.4) Show that the product of upper triangular matrices is

always upper triangular:

AB =

x x x0 x x

0 0 x

x x x0 x x

0 0 x

=

0

0 0

.

Proof using dot products (Row times column) (Row 2 ofA)(column 1 ofB)=0.

Which other dot products gives zeros?

Proof using full matrices (Column times row) Draw xs and 0s in (column 2

ofA) times (row 2 ofB). Also show (column 3 ofA) times (row 3 ofB).

(Problem 28, Section 2.4) Draw the cuts in A (2 by 3) and B (3 by 4) and AB

to show how each of the four multiplications rules is really a block multiplication:(1) Matrix A times columns ofB. Columns of AB(2) Rows ofA times the matrix B. Rows of AB

(3) Rows ofA times columns ofB. Inner products (numbers in AB)(3) Columns ofA times rows ofB. Outer products (matrices add to AB)


41/94

2.4 Appendix 2-40

(Problem 37, Section 2.4) To prove (AB)C = A(BC), use the column vec-

tors b1, . . . , bn of B. First suppose that C has only one column c with entries

c1, . . . , cn:

AB has columns Ab1, . . . , Abn and then (AB)c = c1Ab1 + + cnAbn.Bc has one column c1b1 + +cnbn and then A(Bc) equals A(c1b1 + +cnbn).

Linearity gives equality of those two sums. This proves (AB)c = A(Bc). The

same is true for all other ofC. Therefore (AB)C = A(BC). Apply to

inverses:

IfBA = I and AC = I, prove that the left-inverse B equals the right-inverse C.


42/94

2.5 Inverse matices 2-41

Definition (Inverse matrix): The matrix A is invertible if there exists amatrix A1 such that

A1A = I and AA1 = I.

A is invertible only when A is a square matrix.

A is invertible if, and only if, Ax = b has a unique solution.

This unique solution must be A1b. IfA is invertible, then Ax = 0 cannot have a non-zero solution.

Ann is invertible if, and ony if, forward elimination (with possibly row ex-

change) should give n non-zero pivots. (This test is what you can use now!)

IfA is invertible if, and only if, the determinant ofA (i.e., det(A)) is non-zero.

(We will come back to this in Chapter 5.)

Hence, for a square matrix, either both left and right inverses do not exist

(det(A) = 0) or both left and right inverses exist (det(A) = 0).


43/94

2.5 Inverse matices 2-42 The inverse matrix ofA is unique.

Proof: IfL and R are both the inverse ofA, then

L = LI = L(AR) = (LA)R = IR = R.

2

In fact, the above proof also verifies that for a square matrix A, its leftinverse L always equals its right inverse R.

Again, it is not possible for a square matrix without a left (respectively,right) inverse to have a right (respectively, left) inverse.

Again, it is not possible for a square matrix to have distinct left and right

inverses.


44/94

2.5 Inverse matices 2-43Lemma. Suppose A and B are both square matrices. Then, AB has an inverseif, and only if, both A and B have inverses.

Proof:

1. if-part: If both A and B have inverses, then

(AB)(B1A1) = A(BB1)A1 = AIA1 = AA1 = I.

Hence, AB has an inverse B1A1.

2. only-if-part: Suppose AB has an inverse C. Then, it is not possible that Aor B has no inverse because if A had no inverse (meaning that both left and

right inverses ofA do not exist), then

ABC = I

implies that the square matrix A has a right inverse BC, a contradiction.On the other hand, if B has no inverse (meaning that both left and right

inverses ofB do not exist), then

CAB = I

implies that the square matrix B has a left inverse CA, a contradiction.

2


45/94

2.5 How to find the inverse matrix of A 2-44 How to find A1 if it exists?

Answer: Gauss-Jordan method.

What is Gauss-Jordan method?

Answer: To repeat forward elimination and back substitution n times for

an n-by-n matrix A.

Example. Suppose A is a 3-by-3 matrix.

Find the solutions x1, x2 and x3 for

Ax1 = e1

1

0

0

, Ax2 = e2

0

1

0

and Ax3 = e3

0

0

1

Then,

Ax1 x2 x3

=

Ax1 Ax2 Ax3

=e1 e2 e3

= I,

which means A1 = x1 x2 x3 .


46/94

2.5 How to find the inverse matrix of A 2-45 Gauss said We can solve the three linear equations in one time by using the

augmented matrix.

Ax1 = e1

Ax2 = e2

Ax3 = e3

= A e1 e2 e3 = A I .

Example. Suppose A = 2 1 0

1 2 1

0 1 2 .

A I

=

2 1 0 1 0 0

1 2 1 0 1 0

0 1 2 0 0 1

. . . (1)

. . . (2)

. . . (3)

2 1 0 1 0 00 32 1 12 1 0

0 1 2 0 0 1

. . . (1) . . . (1). . . (2) 12 (1) . . . (2). . . (3) 0 (1) . . . (3)

2 1 0 1 0 0

0 32 112 1 0

0 0 43 13 23 1

. . . (1)

. . . (2)

. . . (3) 23 (2)2


47/94

2.5 How to find the inverse matrix of A 2-46 What is the next step after obtaining

2 1 0 1 0 0

0 32 112 1 0

0 0 43 13 23 1

Answer: Back substitution to solve

x1 = x1,1

x2,1x3,1

, x2 = x1,2

x2,2x3,2

and x3 = x1,3

x2,3x3,3

.Example (Continue).

x1 =

x1,1x2,1

x3,1

=

1(1)x2,1(0)x3,12

1/2(1)x3,13/2

1/34/3

=

34

12

14


48/94

2.5 How to find the inverse matrix of A 2-47and

x2 = x1,2x2,2x3,2

=

0(1)x2,2(0)x3,22

1(1)x3,2

3/22/34/3

=

12

1

12

and

x3 =x1,3x2,3

x3,3

= 0(1)x2,3(0)x3,3

2

0(1)x3,33/2

14/3

= 14

12

34

2


49/94

2.5 How to find the inverse matrix of A 2-48 Jordan said We can also solve

x1 x2 x3

by backward elimination.

Example. (Continue)

2 1 0 1 0 00 32 1 12 1 00 0 43

13

23 1

. . . (1). . . (2). . . (3)

2 1 0 1 0 0

0 3

2

0 3

4

3

2

3

40 0 4313

23 1

. . . (1)

. . . (2) (3/4) (3) = (2)

. . . (3)

2 1 0 1 0 0

0 32

0 34

32

34

0 0 4313

23 1

. . . (1) (0) (3) = (1)

. . . (2)

. . . (3)

2 0 0 32 1 120 32 0 34 32 34

0 0 43

13

23

1

. . . (1) (2/3) (2) = (1). . . (2). . . (3)

Based on this new augmented matrix, we can solve x1, x2 and x3 simply by

dividing the pivot in each row (i.e., to make the pivot equal to 1). The resultantmatrix is said to be in its reduced echelon form.


50/94

2.5 How to find the inverse matrix of A2-49

1 0 0 3412

14

0 1 0 12 112

0 0 114

12

34

. . . (1)/2

. . . (2)/(3/2)

. . . (3)/(4/3)Hence,

A1 =

34

12

14

12 1

12

1

4

1

2

3

4

.

2

Gauss-Jordan method:

Multiply A I by A1 to get I A1 usingforward elimination and backward elimination.


51/94

2.5 Terminologies2-50

Row echelon form: A form of matrices satisfying

Every all-zero row is below non-all-zero rows.

The leading coefficient (the first nonzero number from the left, also calledthe pivot) of a non-all-zero row is always strictly to the right of the leading

coefficient of the row above it.

Example. Upper triangular matrix obtained from forward elimination.

(row) Reduced echelon form: A form of matrices in row echelonform with the leading coefficient being one and also being the only non-zero

entry in the column.

Example.1 1 0 10 0 1 1

0 0 0 0


52/94

2.5 MATLAB revisited2-51

In MATLAB:

The Gauss-Jordan method can be performed as:

I = eye(n); % Define the n-by-n identity matrix.R = rref([A I]); % Obtain the row reduced echelon form.

X = R(:, n+1:n+n); % Pick up the last n columns to form A1.

You can also obtain the inverse matrix directly:

X=inv(A) % A has to be a square matrix.

Question: If inv() has already fulfilled the purpose of finding inverse, why we

need rref()?


53/94

2.5 Gauss-Jordan method revisited2-52

Example. Find the right inverse R32 of a non-square matrix A23 =a1 a2 a3

.

(Note that this cannot be done by inv().)

A

r1,1

r2,1

r3,1

= 10

Ar1,2

r2,2

r3,2

= 01= Form the augumented matrix

A

1

0

0

1

= Find E22 such that EA E1

0 E0

1 = I22 Ea3 E1

0 E0

1 =

r1,1

r2,1

= E

1

0

Ea3r3,1

r1,2

r2,2= E

0

1 Ea3r3,2

The right inverse R is not unique!


54/94


Further suppose A =

2 1 0

1 2 1

. Then,

1 012 1

2 1 0 1 01 2 1 0 1 = 2 1 0 1 00 32 1 12 1 . . . Forward elimination1 230 1

2 1 0 1 0

0 32 112 1

=

2 0 23

43

23

0 32 112 1

. . . Backward elimination

12 00 23 2 0 23

43

23

0 32 1 12 1 = 1 0

13

23

13

0 1 23 13 23 . . . Pivot normalization

Hence,

E =

12 0

0 23

1 230 1

1 012 1

=

23

13

13

23

.


55/94


The problem is reduced to

EA

r1,1

r2,1

r3,1

= E

1

0

EA

r1,2

r2,2

r3,2

= E0

1

=

1 0 13

0 1 23

r1,1

r2,1

r3,1

=

2313

1 0 130 1 2

3

r1,2

r2,2

r3,2

= 132

3We can let r3,1 = r3,2 = 0 and obtain

r1,1

r2,1 = E1

0 = 231

3r1,2

r2,2

= E

0

1

=

1323

So, the right inverse is R = E0 0 .2

Question: Can we choose r3,1 = r3,2 = 1? Hint: R = E Ea3 r3,1r3,2r3,1 r3,2

.


56/94


In MATLAB:

The Gauss-Jordan method can be performed as:

I = eye(2); % Define the 2-by-2 identity matrix.

A = [2 -1 0; -1 2 -1]; % Define A.S = rref([A I]); % Obtain the row reduced echelon form.

% S =

1 0 13

23

13

0 1 2313

23

R = [S(:, 4:5); 0 0 ]; % Pick up the last 2 columns to form R.

Important summary:

In the process of Guass-Jordan method, we solve and obtain directly the rightinverse in the above example.

Using the same technique, we may obtain the left inverse, which is theproduct of the forward elimination matrices, the backward elimination matrices

and the pivot normalization matrix.

In the above example, A23 does not have the left inverse because no matrix

L32 can make L32A23 = I33.


57/94

2.5 Diagonal and tridiagonal matrices2-56

(1) The inverse matrix of a diagonal square matrix (if the inverse exists) is a

diagonal square matrix.

(2) The inverse matrix of a tridiagonal square matrix may however be a dense

square matrix (since a diagonal matrix is also a tridiagonal matrix).

Definition (Diagonal and tridiagonal matrices): A square matrix isa diagonal matrix if all the off-diagonal entries are zero. Likewise, a tridiagonal

matrix is one that all the entries are zero except those in the main diagonal

and two adjacent diagonals (which can be zeros).

Some may define the tridiagonal matrix as one that all entries in the maindiagonal and two adjacent diagonals are non-zeros. In such case, its inverse

is always dense.

It is also worth knowing that antidiagonal entries of a matrix A area1,n, a2,n1, a3,n2, . . . , an,1.

Definition (Dense matrix): A matrix is dense if all the entries are non-

zeros.


58/94

2.5 Triangular matrix and singular matrix2-57

IfL is lower triangular with 1s on the diagonal, so is its inverse.

A square triangular matrix is invertible if, and only if, all its diagonals are

nonzeros.

Definition (Singular matrix): A square matrix is called singular if it has noinverse.


59/94

2.5 Symmetric and determinant2-58

The inverse matrix of a symmetricmatrix (if the inverse exists) is symmetric.

Definition (Symmetric matrix): A square matrix is symmetric if its(i, j)th entry is equal to its (j, i)th entry for every i and j.

Recall that the determinant is the signed area (volume, etc) of the parallelo-

gram (parallelepiped, etc) formed by the column vectors of a matrix A.

The Gauss-Jordan method can also be used to determine the determi-

nant, which is equal to the product of all the pivots before nor-malization.

Example. Suppose A =

2 1 0

1 2 1

0 1 2

.

A I

G.J.Method=

2 0 0 32 1 120 32 0 34 32 34

0 0 4313

23 1

det(A) = 2 3

2

4

3= 4.


60/94

2.6 Elimination = Factorization : A = LU2-59

Definition (Factorization): Factorization is the decomposition of an object(e.g., a number or a polynomial or a matrix) into a product of other objects, named

factors, such that when being multiplied together, one gets the original.

Example.

15 = 3 5

x2 3x + 2 = (x 1)(x 2)

A = LU

2

Triangular factorization: How to find lower triangular L and upper

triangular U such that A = LU?

Answer: Gauss-Jordan method (Actually, just forward elimination.Why? See the below example).

Forward elimination produces a matrix E such that EA = U.

The inverse of lower triangular E is also lower triangular L = E1.

This immediately gives A = E1U = LU .


61/94

2.6 Elimination = Factorization : A = LU2-60

Example.

2 4 2 1 0 04 9 3 0 1 0

2 3 7 0 0 1


. . . (2)

. . . (3)

2 4 2 1 0 00 1 1 2 1 0

0 1 5 1 0 1

. . . (1) with 2 the first pivot. . . (2) 2 (1)

. . . (3) (1) (1)

2 4 2 1 0 00 1 1 2 1 0

0 1 5 1 0 1

. . . (1). . . (2) with 1 the second pivot. . . (3)

2 4 2 1 0 0

0 1 1 2 1 0

0 0 4 3 1 1. . . (1)


. . . (3) 1 (2)

1 0 02 1 0

0 0 1

2 4 24 9 3

2 3 7

1 0 00 1 0(1) 0 1

2 4 20 1 1

2 3 7

1 0 00 1 0

0 1 1

2 4 20 1 1

0 1 5


62/94

2.6 Elimination = Factorization : A = LU 2-61

The forward elimination process can then be implemented by multiplying the elim-

ination matrices E2,1, E3,1 and E3,2 in sequence.

1 0 00 1 00 1 1

E3,2

1 0 00 1 0(1) 0 1

E3,1

1 0 02 1 00 0 1

E2,1

2 4 2 1 0 04 9 3 0 1 02 3 7 0 0 1

= 2 4 2 1 0 0

0 1 1 2 1 00 0 4 3 1 1

Then we have that after forward elimination

E3,2E3,1E2,1 A I

=

EA E

=

U E

where E = E3,2E3,1E2,1.Next, we need to determine L = E1 = E12,1 E

13,1 E

13,2 . This is easy because every

Ei,j is formed by changing just one off-diagonal entry from an identity matrix.


63/94


What we need to do is to negate the only non-zero off-diagonal entry!

1 0 0

0 1 0

0 1 11

E13,2

= 1 0 0

0 1 0

0 1 1 , 1 0 0

0 1 0

(1) 0 11

E13,1

= 1 0 0

0 1 0

(1) 0 1

and 1 0 0

2 1 0

0 0 11

E12,1

= 1 0 0

2 1 0

0 0 1More surprisingly, when they are multiplied in reverse order, the product is simply

to place all these non-zero entries at their respective position!

L = E1 = E12,1 E13,1 E

13,2 =

1 0 02 1 0

1 1 1

2


64/94


Intuitions behind the above example

Why the inverse ofEi,j is simply to negate the multiplier ofEi,j?

Answer: The inverse action of (i) i,j (j) is simply (i) + i,j(j).

Why the product of E12,1 E13,1 E

13,2 is to place all the non-zero entries in their

respective position?

Answer:

E13,1 = only updates row 3 ofE13,2

1 0 0

0

by row 1, i.e., (3) = (3) + 3,1 (1) .

Hence,

(3) =

(3), if (1) = 0 (such as the (1, 2)th and (1, 3)th entries ofE13,2 )

3,1, if (3) = 0 and (1) = 1

Hint of a formal proof:

[I+ (E13,1

I)][I+ (E13,2

I)] = I+ (E13,1

I) + (E13,2

I) + (E13,1

I)(E13,2

I) = I+ (E13,1

I) + (E13,2

I).

Similar interpretation can be done for E12,1 onto E13,1 E

13,2 .

We summarize the observation in the below lemma.


65/94


Lemma: For an invertible matrix A, if no row exchange is necessary duringforward elimination, then its lower triangular factor L and upper triangular

factor U satisfy

A = LU;

L has 1 on all main diagonal entries, and multipliers below the diagonal entries.

When a row of A starts with r zeros, so does the rows ofL (because themultipliers are zeros).

U has (non-normalized) pivots on its main diagonal.

When a column ofA starts with c zeros, so does the columns ofU (because

forward eliminations will not change these zeros).

Example. Suppose A =

a1,1 a1,2 a1,3a2,1 a2,2 a2,3a3,1 a3,2 a3,3

.

A


66/94


Thena1,1 a1,2 a1,3a2,1 a2,2 a2,3a

3,1a

3,2a

3,3

a1,1 a1,2 a1,30 a2,2 (a2,1/a1,1)a1,2 a2,3 (a2,1/a1,1)a1,3

0 a3,2 (a3,1/a1,1)a1,2 a3,3 (a3,1/a1,1)a1,3

. . . (1). . . (2) (a2,1/a1,1) (1)

. . . (3) (a3,1/a1,1) (1)

Then, we know

L =

1 0 0a2,1/a1,1 1 0

a3,1/a1,1 ? 1

and U =

a1,1 a1,2 a1,30 a2,2 (a2,1/a1,1)a1,2 a2,3 (a2,1/a1,1)a1,3

0 0 ?

2

Remember the formula of multiplier: 2,1 = a2,1/a1,1 and 3,1 = a3,1/a1,1.

By LU decomposition, Ax = b can be solved by

Lc = b (forward substitution)Ux = c (back substitution)

2 6 Eli i i F i i A LU


67/94


What if row exchange is needed during forward elimination?

Example.

2 4 2 1 0 04 8 3 0 1 02 3 7 0 0 1

. . . (1) with 2 the first pivot. . . (2) (Replace the middle 9 by 8). . . (3)

2 4 2 1 0 0

0 0 1 2 1 0

0 1 5 1 0 1


. . . (2) 2 (1)

. . . (3) (1) (1)

2 4 2 1 0 00 1 5 1 0 1

0 0 1 2 1 0

. . . (1). . . (2) with 1 the second pivot

. . . (3)

2 4 2 1 0 00 1 5 1 0 1

0 0 1 2 1 0


1 0 02 1 00 0 1

2 4 24 8 32 3 7

1 0 00 1 0

(1) 0 1

2 4 20 0 1

2 3 7

1 0 00 0 10 1 0

2 4 20 0 10 1 5

1 0 00 1 0

0 0 1

2 4 20 1 5

0 0 1

2 6 Eli i ti F t i ti A LU


68/94




1 0 00 1 00 0 1

E3,2

1 0 00 0 10 1 0

P2,3

1 0 00 1 0(1) 0 1

E3,1

1 0 02 1 00 0 1

E2,1

2 4 2 1 0 04 9 3 0 1 02 3 7 0 0 1

= 2 4 2 1 0 00 1 5 1 0 10 0 1 2 1 0

Then we have after forward elimination

E3,2P2,3E3,1E2,1 A I

=

EA E

=

U E

where E = E3,2P2,3E3,1E2,1.Next, we need to determine L = E1 = E12,1 E

13,1 P

12,3 E

13,2 . This is easy because

every Eij is formed by changing just one off-diagonal entry from an identity

matrix, and the inverse ofP2,3 is itself!



69/94



1 0 0

0 1 0

0 1 11

E13,2

= 1 0 0

0 1 0

0 1 1,

1 0 0

0 1 0

(1) 0 11

E13,1

= 1 0 0

0 1 0

(1) 0 1

and

1 0 0

2 1 0

0 0 1

1

E12,1

=

1 0 0

2 1 0

0 0 1

L = E1 = E12,1 E13,1 P

12,3 E

13,2 =

1 0 0

2 0 1

1 1 0 Not a lower triangular matrix!2



70/94


How can we guarantee there is no row exchanges necessary for finding A = LU?

No row exchanges during forward elimination if, and only if, all the upper-left

square sub-matrices ofA are invertible. (See Problem 24 on p. 106.)

(Problem 24, Section 2.6) Which invertible matrices allow A = LU (elimination

without row exchanges)? Good question! Look at each of the square upper left

submatrices ofA.

All upper left k by k submatrices Ak must be invertible (sizes k = 1, . . . , n).

Explain that answer: Ak factors into because LU = Lk 0 Uk 0 . In such case Ak = LkUk, where Ak (similarly, Lk and Uk) consists of the first

k rows and the first k columns ofA.

a1,1 a1,2 a1,3a2,1 a2,2 a2,3a3,1 a3,2 a3,3

= 1 0 02,1 1 0

3,1 3,2 1

u1,1 u1,2 u1,30 u2,2 u2,30 0 u3,3

2 6 Elimination Triple Factorization : A LDU


71/94

2.6 Elimination = Triple Factorization : A = LDU 2-70

Now assume again no row exchange is needed during forward elimination.

Sometimes, we wish to make the diagonal entries of U being all ones. Hence,

a diagonal matrix D is introduced.

Example.

2 4 20 1 1

0 0 4

2 0 00 1 0

0 0 4

D

1 2 10 1 1

0 0 1

U2

The advantage of triple factorization is that for symmetric matrix A = LDU,we have

L = U

,where U is the transpose of matrix U (i.e., the (i, j)th entry ofU is the (j, i)th

entry ofU). Hence,

A = LDL.

2 6 The cost of elimination


72/94

2.6 The cost of elimination 2-71

Question: Again, assume with no row exchanges during elimination. What will bethe complexity of using forward elimination and back substitution to solve Ax = b?

Answer:

i) Forward elimination for matrix A, operations exact approximation

ai,1 0 for i = 2, . . . , n n(n 1) n2

ai,2 0 for i = 3, . . . , n (n 1)(n 2) (n 1)2

... ... ...

ai,n1 0 for i = n 2 1 22

totaln

i=2

i(i 1) =1

3n(n2 1)

1

3n3

Note: For example, n(n 1) = n operations/row (n 1) rows.

Note: The blue-colored numbers are used for step ii) in the next page, which requires

for example, 1 operations/row (n 1) rows.

2 6 The cost of elimination 2 72


73/94


ii) Forward elimination for vector b, we needn

i=2 (i 1) =12n(n1) computations.

iii) Back substitution Ux = c requires

or / operations exact

xn = cn/un,n 1

xn1 = (cn1 un1,nxn)/un1,n1 2... ...

x1 = n

totaln

i=1

i = 12

n(n + 1)

Hence, ii) and iii) require n2 multiplications.

Using MATLAB, the above algorithm requires about 1 second to solve Ax = b

with n = 1000.

Doubling n makes the algorithm 23 = 8 times slower.

Example. n = 100 1000 1003 seconds = 11.574 days. 2



74/94


Question: How about the complexity for calculating A1 (without row exchange)using Gauss-Jordan? Does it require n

13n(n

2 1) + n2

multiplications as we

need to solve Axi = ei for i = 1, . . . , n?

Answer:i) Forward elimination for augmented matrix

A I

,

operations exact

ai,1 0 for i = 2, . . . , n 2n(n 1)

ai,2

0 for i = 3, . . . , n (2n 1)(n 2)... ...

ai,n1 0 for i = n (n + 2) 1

totaln

i=2(n + i)(i 1) =

1

6n(n 1)(5n + 2)



75/94

2.6 The cost of elimination 2-74ii) Backward elimination for augmented matrix

A I

,

operations exact

ai,n 0 for i = n 1, . . . , 1 (n + 1)(n 1)

ai,n1 0 for i = n 2, . . . , 1 (n + 2)(n 2)

... ...

ai,2 0 for i = 1 (2n 1) 1

totaln1

i=1(2n i)i =

1

6n(n 1)(4n + 1)

iii) Pivot normalization: n2 divisions.

Three steps in total, 12n(n 1)(3n + 1) + n2 = 12n(3n

2 1) 32n3.

We can also find the inverse of A by solving Axi = ei for i = 1, . . . , n sep-

arately. In such case, the forward elimination part in slide 2-71 (step i)) canbe shared among all n equations, while steps ii) and iii) in slide 2-72 (steps

ii) and iii)) must be repeated n times. This yields 13n(n2 1) + n n2 43n

3

multiplications.

Backward substitution is more efficient than backward elimination.

2 6 The cost of elimination 2-75


76/94


Again, assume with no row exchanges during elimination. What will be the com-

plexity of using forward elimination and back substitution to solve Ax = b, ifA is

a band matrix with w non-zero diagonals both below and above the main diagonal?

Answer:i) Forward elimination for matrix A,

operations exact approximation

ai,1 0 for i = 2, . . . , w + 1 w w w2

ai,2 0 for i = 3, . . . , w + 2 w w w2... ... ...

ai,nw 0 for i = n w + 1,. . . ,n w w w2

... ... ...

ai,n1 0 for i = n 1 1 w2

total

w1i=1

i2 + w2(n w)

= 16w(6nw 4w2 3w + 1)

nw2

Note: The blue-colored numbers are used for step ii).



77/94

2.6 The cost of elimination 2 76

ii) Forward elimination for vector b, we needw1

i=1 i + (n w)w =12w(2n w 1)

computations.

iii) Back substitution Ux = c requires

operations exact

xn = cn/un,n 1

xn1 = (cn1 un1,nxn)/un1,n1 2... ...

xnw = (cnw unw,nxn )/unw,nw w + 1... ...

x1 = w + 1

total 12(w + 1)(2n w)

Hence, ii) and iii) require n(1 + 2w) w(1 + w) computations.A band-(2w + 1) matrix A (if no row exchange is needed during forward elimination)

satisfies that its factors L and U are both band matrices with bandwidth ofw + 1.

2.7 Transpose and permutation 2-77


78/94

p p

Definition (Matrix transpose): The transpose A of a matrix A is simplyto exchange the rows and columns. Hence, the (i, j)th entry of A is the (j, i)th

entry ofA.

Note: A matrix transpose is sometimes denoted by A or AT. For example,

A is used by MATLAB.

AT is adopted by the textbook. (We will use this notation in the sequel!)

Properties of transpose

Sum Property: (A + B)T = AT + BT

Proof: This can be proved by listing all the matrix entries of both sides. 2



79/94

p p

Product Property: (AB)T = BTAT

Proof: First, prove using Sum Property of Transpose.For a matrix A and vector b,

(Ab)T = a1 a2 bT= (b1a1 + b2a2 + )

T = b1aT1 + b2a

T2 + = b

T

aT1aT2...

= bTAT

Then,

(AB)T = (Ab1 b2

)T

= Ab1 Ab2 T

= bT1A

T

bT2AT

... = bT1bT2

... AT = BTAT.

2



80/94

p p

Inverse Property: (AT)1 = (A1)T

Proof: A1A = I (A1A)T = IT = I;

hence, the Product Property of Transpose gives AT(A1)T = I, which

implies (A1

)T

is the inverse ofAT

.2

2.7 Inner product revisited 2-80


81/94

Recall the difference between inner product and outer product (for real-valued

vectors)

Inner product v w = vTw = v1 v2 w1w2 = v1w1 + v2w2and

Outer product v w = vwT =

v1v2

w1 w2

=

v1w1 v1w2v2w1 v2w2

To memorize it,

for inner product, T is placed inner (inbetween v and w)

for outer product, T is placed outer (outside v and w).



82/94

The dot product we have introduced in Chapter 1 should be only termed as

a product between two vectors, which is denoted by a dot.

It happens to satisfy the three axioms of an inner product.

Definition: A mapping from V V to F, denoted by , , is an innerproduct if for every x,y,z V and a F,

1. Positive-definiteness: x, x 0 with equality only when x = 0

2. Symmetry: x, y = y, x

3. Linearity:ax,y = ax, y

x + y, z = x, z + y, z

Vector dot product is an inner product with V = F = (set of all realnumbers).

It is then possible to parallel linear algebra with calculus along this conception.



83/94

Example. What is the calculus for function transpose operation, such as in

(Ax)Ty = xT(ATy), where A is the difference matrix?

Inner product of two vectors can be expressed in terms of transpose as

x,y = x y = xTy.

Hence, (Ax)Ty = xT(ATy) can be expressed as

Ax,y = x, ATy.

Inner product of two real functions can be defined as

x, y

x(t)y(t)dt.

We then reasonably guess the calculus counterpart of difference matrix A isderivative. I.e., we guess

t

x, y

=

x,

t

Ty

.

Question is what

t

T is?



84/94

Given that

|x(t)|dt < and

|y(t)|dt < , integration by parts

gives us

x, y =

x(t)

t

y(t)dt

= x(t)y(t)|

x(t)y(t)

tdt (Integration by parts)

=

x(t)

y(t)

t

dt = x, y

Hence,

Transpose of differentiation is

t

T=

t.

Verification:

Ax =

1 0 01 1 0

0 1 1

x =

x1x2 x1

x3 x2

and ATy =

1 1 00 1 1

0 0 1

y =

(y2 y1)(y3 y2)

y3

2



85/94

Summary

Inner product is a more general concept, and has much more extensional use

than what we have introduced here.

2.7 Symmetric matrix 2-85


86/94

We can re-define symmetric matrix (different from that given in slide 2-58)using transpose.

Definition (Symmetric matrix): A matrix A is said to be symmetric if

AT = A.

Example. Prove that ATA is a symmetric matrix.

Proof: (ATA)T = ATA; hence, it is symmetric by definition. 2

Remarks. Based on the above definition, we can even introduce skew-symmetric matrix

defined as AT = A .

One advantage of symmetric matrices is that its triple factorization has a

symmetric form. In other words, A = LDLT

ifA is symmetric (and no rowexchange during forward elimination).

This can save half efforts in, e.g., forward elimination (from n3/3 down to

n3/6). (See slide 2-71.)

2.7 Symmetric matrix 2-86


87/94

Question: Again, assume without row exchanges during elimination. What willbe the complexity of forward elimination for a symmetric A?

Answer:

i) Forward elimination for matrix A,

operations exact

ai,1 0 for i = 2, . . . , n 2 + 3 + + (n 1) + n =12(n 1)(n + 2)

ai,2 0 for i = 3, . . . , n 2 + 3 + + (n 1) =12(n 2)(n + 1)... ...

ai,n1 0 for i = n 2 = 121 4

totaln

i=2

1

2(i 1)(i + 2) =

1

6n(n 1)(n + 4)

ii) During the above eliminations, we only need to retain each multiplier (i.e., i,j)

and pivot (i.e., ui,i) to form L and D.(Note that in the above process, we know that A will be reduced to LDLT, which is

of course known after knowing L and D.)

a1,1 a1,2 a1,3a2,1 a2,2 a2,3a3,1 a3,2 a3,3

= 1 0 0

2,1 1 03,1 3,2 1

u1,1 0 u2,2 0 0 u3,3

LT

2.7 Factorization with row exchange: P A = LU 2-87


88/94

What if row exchange is needed during forward elimination?

Example. (This is exactly the same example that appears on slide 2-66!)

2 4 2 1 0 04 8 3 0 1 02 3 7 0 0 1

. . . (1) with 2 the first pivot. . . (2) (Replace the middle 9 by 8). . . (3)

2 4 2 1 0 0

0 0 1 2 1 0

0 1 5 1 0 1


. . . (2) 2 (1)

. . . (3) (1) (1)

2 4 2 1 0 00 1 5 1 0 1

0 0 1 2 1 0

. . . (1). . . (2) with 1 the second pivot

. . . (3)

2 4 2 1 0 00 1 5 1 0 1

0 0 1 2 1 0


1 0 0

2 1 00 0 1

2 4 24 8 3

2 3 7

1 0 00 1 0(1) 0 1

2 4 20 0 1

2 3 7

1 0 00 0 1

0 1 02 4 20 0 1

0 1 5

1 0 00 1 00 0 1

2 4 20 1 50 0 1



89/94



1 0 0

0 1 00 0 1

E3,2

1 0 0

0 0 10 1 0

P2,3

1 0 0

0 1 0(1) 0 1

E3,1

1 0 0

2 1 00 0 1

E2,1

2 4 2 1 0 0

4 9 3 0 1 02 3 7 0 0 1

= 2 4 2 1 0 00 1 5 1 0 1

0 0 1 2 1 0Then we have that after forward elimination

E3,2P2,3E3,1E2,1

A I

=

EA E

=

U E

where E = E3,2P2,3E3,1E2,1.Next, we need to determine L = E1 = E12,1 E

13,1 P

12,3 E

13,2 . This is easy because

every Eij is formed by changing just one off-diagonal entry from an identity

matrix, and the inverse ofP2,3 is itself!



90/94


1 0 0

0 1 0

0 1 1

1

E13,2

=

1 0 0

0 1 0

0 1 1

,

1 0 0

0 1 0

(1) 0 1

1

E13,1

=

1 0 0

0 1 0

(1) 0 1

and

1 0 0

2 1 0

0 0 1

1

E12,1

=

1 0 0

2 1 0

0 0 1

L = E1 = E12,1 E13,1 P

12,3 E

13,2 =

1 0 0

2 0 1

1 1 0Not a lower triangular matrix!

Then, we obtain

A = LU = 1 0 0

2 0 1

1 1 02 4 2

0 1 5

0 0 1



91/94

Hence,

PA =

1 0 0

0 0 1

0 1 0

A = PLU = LU =

1 0 0

1 1 0

2 0 1

2 4 20 1 50 0 1

2

Factorization with row exchange: Find permutation matrix P such that

P A = LU.

The pre-permutation matrix that can force zero row exchange is not unique!

For (another) example, check the below for the above example.

0 1 0

0 0 11 0 0A =

1 0 0

1

2 1 012 0 1

4 8 3

0 1

11

20 0 12



92/94

The MATLAB command for LU-factorization is

[L,U,P]=lu(A)

Note: The MATLAB will make row exchanges such that the next pivot is the

largest! This is the reason why for

A =

2 4 2

4 8 32 3 7

,

the first pivot that MATLAB chooses is 4 !



93/94

Some useful notes

A permutation matrix P (which has the rows of identity matrix I in any order)

satisfies PT = P1.

Proof: Permutation does not change the inner product; hence (Px)T(Py) =

xTy. Since x and y are arbitrary, PTP = I. 2

It is not necessary true that P1 = P as P may not be symmetric.

There are n! permutation matrices of size n n.

The determinant ofP is either 1 (when the number of row changes is even) or1 (when the number of row changes is odd).

We can also use forward elimination to make symmtric of a matrix A, i.e.,A = ES where E is an elementary row operation matrix, and S is symmetirc.

The idea is simply to make the lower triangle entry ai,j the same as aj,i(instead of making ai,j = 0).

Example (Problem 34 on page 119). A =

1 2

4 9

=

1 0

2 1

1 2

2 5

= ES

2



94/94

(Problem 34, Section 2.7) Write A =

1 2

3 4

as the product EH of an

elementary row operation matrix E and a symmetric matrix H.

Definition (Elementary row operation matrix): A matrix Ethat only includes a combination of operations as

1. row exchanges;

2. row multiplications; and

3. row additions.

In fact, we can use triple factorization to symmetricize a matrix.

A = LDU = L(UT)1

E(UTDU)

S= ES.

linear algebra course part 2

Documents