Lecture 6 Matrix Operations and Gaussian Elimination for Solving Linear Systems Shang-Hua Teng.

Lecture 6Matrix Operations

and Gaussian Elimination for Solving Linear Systems

Shang-Hua Teng

Matrix (Uniform Representation for Any Dimension)

• An m by n matrix is a rectangular table of mn numbers

ji

nmmm

n

n

ajiA

aaa

aaa

aaa

A

,

,2,1,

,22,21,2

,12,11,1

),( write weSometime

...

...

...

...

Matrix (Uniform Representation for Any Dimension)

• Can be viewed as m row vectors in n dimensions

nmmm

n

n

aaa

aaa

aaa

A

,2,1,

,22,21,2

,12,11,1

...

...

...

...

Matrix (Uniform Representation for Any Dimension)

• Or can be viewed as n column vectors in m dimensions

nmmm

n

n

aaa

aaa

aaa

A

,2,1,

,22,21,2

,12,11,1

...

...

...

...

Squared Matrix

• An n by n matrix is a squared table of n2 numbers

nnnn

n

n

aaa

aaa

aaa

A

,2,1,

,22,21,2

,12,11,1

...

...

...

...

Some Special Squared Matrices

• All zeros matrix

0...00

...

0...00

0...00

),(0

mn

• Identity matrix

1...00

...

0...10

0...01

),(

nnII

Matrix Operations

• Addition

• Scalar multiplication

• Multiplication

1. Matrix Addition:

mnmnmm

nn

nn

mnmm

n

n

mnmm

n

n

baba

bababa

bababa

BA

bbb

bbb

bbb

B

a aa

a aa

a aa

A

,

,

11

2222222121

1112121111

21

22221

11211

21

22221

11211

Matrices have to have the same dimensionsWhat is the complexity?

2. Scalar Multiplication:

mnmm

n

n

mnmm

n

n

aaa

aaa

aaa

aaa

aaa

aaa

A

21

2 2221

11211

21

22221

11211

What is the complexity?

3. Matrix Multiplication

n

i ipmiimi

n

i ipi

n

i ii

n

i ii

n

i ipi

n

i ii

n

i ii

npnn

p

p

mnmm

n

n

baba

bababa

bababa

BA

bbb

bbb

bbb

B

aaa

aaa

aaa

A

1

n

1=i 1

1 21 221 12

1 11 211 11

21

22221

11211

21

22221

11211

,

,

Two matrices have to be conformalWhat is the complexity?

Matrix Multiplication

B) of j(column A) of i row(

,

,

21

22221

11211

21

22221

11211

BA

bbb

bbb

bbb

B

aaa

aaa

aaa

A

npnn

p

p

mnmm

n

n

Two matrices have to be conformal

The Laws of Matrix Operations

• A + B = B + A (commutative)

• c(A+B) = cA + c+B (distributive)

• A + (B + C) = (A + B) + C (associative)

• C(A+B) = CA + CB (distributive from left)

• (A+B)C = AC+BC (distributive from right)

• A(BC) = (AB)C (associative)

• But in general: BAAB

Counter Example

00

01

01

00

00

10

10

00

00

10

01

00

BA

but

AB

Special Matrices

• Identity matrix I– IA = AI = A

• Square Matrix A

pqqp

qpqp

p

p

AA

AAA

AAAAA

Elimination: Method for Solving Linear Systems

• Linear Systems == System of Linear Equations

• Elimination: – Multiply the LHS and RHS of an equation by a

nonzero constant results the same equations

– Adding the LHSs and RHSs of two equations does not change the solution

0),()(:)()(: xgxfxxgxfx

)()()()();()(:)()();()(: 2121112211 xgxgxfxfxgxfxxgxfxgxfx

Elimination in 2D

• Multiply the first equation by 3 and subtracts from the second equation (to eliminate x)

1123

12

yx

yx

880

12

y

yx

• The two systems have the same solution

• The second system is easy to solve

Geometry of Elimination

1123

12

yx

yx

1123 yx

12 yx

(3,1)8y = 8

880

12

y

yxReduce to a 1-dimensional problem.

Upper Triangular Systems and Back Substitution

• Back substitution– From the second equation y = 1– Substitute the value of y to the first equation to obtain

x-2=1– Solve it we have: x = 3

880

12

y

yx

• So the solution is (3,1)

How Much to Multiply before Subtracting

• Pivot: first nonzero in the row that does the elimination

• Multiplier: (entry to eliminate) divided by (pivot)

1123

12

yx

yx

Multiply: = 3/1

How Much to Multiply before Subtracting

• Pivot: first nonzero in the row that does the elimination

• Multiplier: (entry to eliminate) divided by (pivot)

1123

242

yx

yx

Multiply: = 3/2

880

242

y

yx

The pivots are on the diagonal of the triangle after the elimination

Breakdown of Elimination

• What is the pivot is zero == one can’t divide by zero!!!!

1163

12

yx

yx

Eliminate x:

80

12

y

yx

No Solution!!!!: this system has no second pivot

Geometric Intuition(Row Pictures)

• Two parallel lines never intersect

1163 yx

12 yx

(3,1)8y = 8

Geometric Intuition(Column Picture)

11

1

3

1

6

2

Two column vectors are co-linear!!!!

1163

12

yx

yx

Geometric Intuition

Geometric degeneracy cause failure in elimination!

Failure in Elimination May Indicate Infinitely Many Solutions

• y is free, can be number!

• Geometric Intuition (row picture): The two line are the same

• Geometric Intuition (column picture): all three column vectors are co-linear

363

12

yx

yx00

12

y

yx

Failure in Elimination(Temporary and can be Fixed)

• First pivot position contains zero• Exchange with the second equation

523

420

yx

yx

42

522

y

yx

Can be solved by backward substitution!

Singular Systems versus Non-Singular Systems

• A singular system has no solution or infinitely many solution– Row Picture: two line are parallel or the same– Column Picture: Two column vectors are co-

linear

• A non-singular system has a unique solution– Row Picture: two non-parallel lines– Column Picture: two non-colinear column

vectors

Gaussian Elimination in 3D

• Using the first pivot to eliminate x from the next two equations

10732

8394

2242

zyx

zyx

zyx

Gaussian Elimination in 3D

• Using the second pivot to eliminate y from the third equation

125

4

2242

zy

zy

zyx

Gaussian Elimination in 3D

• Using the second pivot to eliminate y from the third equation

84

4

2242

z

zy

zyx

Now We Have a Triangular System

• From the last equation, we have

84

4

2242

z

zy

zyx

Backward Substitution

• And substitute z to the first two equations

2

4

2242

z

zy

zyx

Backward Substitution

• We can solve y

2

42

2442

z

y

yx

Backward Substitution

• Substitute to the first equation

2

2

2442

z

y

yx

Backward Substitution

• We can solve the first equation

2

2

2482

z

y

x

Backward Substitution

• We can solve the first equation

2

2

1

z

y

x

Generalization

• How to generalize to higher dimensions?

• What is the complexity of the algorithm?

• Answer:

• Express Elimination with Matrices

Step 1Build Augmented Matrix

10732

8394

2242

zyx

zyx

zyx

Ax = b

10732

8394

2242

bA[A b]

Pivot 1: The elimination of column 1

1

2

10732

8394

2242

10732

4110

2242

12510

4110

2242

Pivot 2: The elimination of column 2

1

12510

4110

2242

8400

4110

2242

Upper triangular matrix

Backward Substitution 1: from the last column to the first

8400

4110

2242

Upper triangular matrix

2100

4110

2242

2100

2010

2242

2100

2010

6042

2100

2010

2002

2100

2010

1001

Expressing Elimination by Matrix Multiplication

Elementary or Elimination Matrix

• The elementary or elimination matrix

That subtracts a multiple l of row j from row i can be obtained from the identity entry by adding (-l) in the i,j position

jiE ,

jiE ,

10

010

001

1,3

l

E

Elementary or Elimination Matrix

3,33,12,32,11,31,1

3,22,21,2

3,12,11,1

3,32,31,3

3,22,21,2

3,12,11,1

3,32,31,3

3,22,21,2

3,12,11,1

1,3

10

010

001

alaalaala

aaa

aaa

aaa

aaa

aaa

laaa

aaa

aaa

E

Pivot 1: The elimination of column 1

12510

4110

2242

1

2

Elimination matrix

10732

8394

2242

10732

4110

2242

10732

8394

2242

100

012

001

12510

4110

2242

10732

4110

2242

101

010

001

The Product of Elimination Matrices

101

012

001

100

012

001

101

010

001

111

012

001

101

012

001

110

010

001