Numerical Methods Part: Cholesky and Decomposition .

Numerical Methods

Part: Cholesky and Decomposition

http://numericalmethods.eng.usf.edu

TLDL

For more details on this topic

Go to http://numericalmethods.eng.usf.edu

Click on Keyword

http://numericalmethods.eng.usf.edu/

You are free

to Share – to copy, distribute, display and perform the work

to Remix – to make derivative works

Under the following conditions Attribution — You must attribute the

work in the manner specified by the author or licensor (but not in any way that suggests that they endorse you or your use of the work).

Noncommercial — You may not use this work for commercial purposes.

Share Alike — If you alter, transform, or build upon this work, you may distribute the resulting work only under the same or similar license to this one.

Chapter 04.09: Cholesky and DecompositionMajor: All Engineering Majors

Authors: Duc Nguyen


Numerical Methods for STEM undergraduateshttp://

numericalmethods.eng.usf.edu 504/21/23

Lecture # 1

TLDL



(1)

1

Introduction

6


][]][[ bxA

= known coefficient matrix, with dimension

where

][A nn

][b= known right-hand-side (RHS) 1n vector

][x = unknown 1n vector.

http://numericalmethods.eng.usf.edu7

Symmetrical Positive Definite (SPD) SLE

A matrix can be considered as SPD if either of

(a) If each and every determinant of sub-matrix is positive, or..

the following conditions is satisfied:

for any given vector (b) If

As a quick example, let us make a test a test to see if the given matrix

nnA ][

),...,2,1( niAii ,0AyyT 0][ 1

ny

110

121

012

][A is SPD?


Symmetrical Positive Definite (SPD) SLE

Based on criteria (a):

The given matrix 33 is symmetrical, because

Furthermore,

022det 11 A

03

21

12det 22

A

jiij aa


01

110

121

012

det 33

A

Hence is SPD.][A


Based on criteria (b): For any given vector

0

3

2

1

y

y

y

y , one computes

32

23

22

21

221

3223

2221

21

3

2

1

321

2

2222

110

121

012

yyyyyyy

yyyyyyy

y

y

y

yyy

Ayyscalar T


0232

21

221 yyyyyscalar

hence matrix is SPD][A

12

Step 1: Matrix Factorization phase

][][][ UUA T

33

2322

131211

332313

2212

11

333231

232221

131211

00

00

00

u

uu

uuu

uuu

uu

u

aaa

aaa

aaa

Multiplying two matrices on the right-hand-side (RHS) of Equation (3), one gets the following 6 equations

1111 au 11

1212 u

au

11

1313 u

au

21

2122222 uau

22

13122323 u

uuau

2

1223

2133333 uuau

(2)

(3)

(4)

(5)


2

11

1

2

i

kkiiiii uau

ii

i

kkjkiij

ij u

uuau

1

1

Step 1.1: Compute the numerator of Equation (7),such as

1

1

i

kkjkiij uuaSum

Step 1.2 If is an off-diagonal term (say )iju ji then (See Equation (7)). Else, if is a

iiij u

Sumu iju

diagonal term (that is, ), thenji Sumuii

(6)

(7)

(See Equation (6))


As a quick example, one computes:

55

47453735272517155757 u

uuuuuuuuau

Thus, for computing , one only needs)7,5( jiu

to use the (already factorized) data in columns )5(# iand of , respectively.)7(# j ][U

(8)


k = 1

k = 2

k = 3

k = 4

iiu iju

ju4

ju3

ju2

ju1

iu4

iu3

iu2

iiu

i = 5

Col. # i=5 Col. # j=7

Figure 1: Cholesky Factorization for the term iju


Step 2: Forward Solution phaseSubstituting Equation (2) into Equation (1), one gets:

][]][[][ bxUU T Let us define:

][][][ yxU Then, Equation (9) becomes:

][][][ byU T

3

2

1

3

2

1

332313

2212

11

0

00

b

b

b

y

y

y

uuu

uu

u

(9)

(10)

(11)

(12)


1111 byu

11

11 u

by

From the 2nd row of Equation (12), one gets

2222112 byuyu

22

11222 u

yuby

Similarly

33

22311333 u

yuyuby

(13)

(14)

(15)


In general, from the row of Equation (12), one hasthj

jj

j

iiijj

j u

yuby

1

1 (16)


Step 3: Backward Solution phase

As a quick example, one has (See Equation (10)):

3

2

1

3

2

1

33

2322

131211

00

0

y

y

y

x

x

x

u

uu

uuu

(17)


From the last (or ) row of Equation (17), rdthn 3

one has 3333 yxu

hence

33

33 u

yx

Similarly:

22

32322 u

xuyx

and

11

31321211 u

xuxuyx

(18)

(19)

(20)


In general, one has:

jj

n

jiijij

j u

xuy

x

1 (21)


TLDLA ]][][[][

For example,

100

10

1

00

00

00

1

01

001

32

3121

33

22

11

3231

21

333231

232221

131211

l

ll

d

d

d

ll

l

aaa

aaa

aaa

Multiplying the three matrices on the RHS of Equation (23), one obtains the following formulas for

the “diagonal” , and “lower-triangular” matrices:][D ][L

(22)

(23)


1

1

2j

kkkjkjjjj dlad

jj

j

kjkkkikijij d

ldlal11

1

(24)

(25)


Step1: Factorization phaseTLDLA ]][][[][

Step 2: Forward solution and diagonal scaling phase

Substituting Equation (22) into Equation (1), one gets:

][][]][][[ bxLDL T Let us define:

][][][ yxL T

(22, repeated)

(26)


Also, define: ][]][[ zyD

3

2

1

3

2

1

33

22

11

00

00

00

z

z

z

y

y

y

d

d

d

niford

zy

ii

ii ......,,3,2,1,

Then Equation (26) becomes:

][]][[ bzL

(29)

(30)


3

2

1

3

2

1

3231

21

1

01

001

b

b

b

z

z

z

ll

l

niforzLbzi

kkikii ......,,3,2,1

1

1

(31)

(32)


Step 3: Backward solution phase

3

2

1

3

2

1

32

3121

100

10

1

y

y

y

x

x

x

l

ll

1......,,1,;1

nniforxlyxn

ikkkiii


Numerical Example 1 (Cholesky algorithms)

Solve the following SLE system for the unknown

vector ? x][]][[ bxA

where

110

121

012

A

0

0

1

][b


Solution:

The factorized, Uupper triangular matrix can be computed by either referring to Equations(6-7), or looking at Figure 1, as following:

][1

0414.1

0

7071.0414.1

1

414.1

2

11

1313

11

1212

1111

Uofrow

u

au

u

au

au


][2

8165.0225.1

07071.01225.1

1

225.1

7071.02

2

1312

22

11

123

23

2

2

12

12

2

111

1

22222

Uofrow

uu

U

uuau

u

uau

i

kkjki

i

kki


][3

5774.0

8165.001 22

2

1223

21333

2

121

1

23333

Uofrowuua

uaui

kki

Thus, the factorized matrix

5774.000

8165.0225.10

07071.0414.1

U


The forward solution phase, shown in Equation (11), becomes:

byU T

0

0

1

5774.08165.00

0225.17071.0

00414.1

3

2

1

y

y

y


5774.0

5774.0

4082.08165.07071.000

4082.0

225.1

7071.07071.00

7071.0

414.1

1

33

223113

21

13

3

22

112

11

12

2

11

11

u

yuyu

u

yuby

u

yu

u

yuby

u

by

jj

j

iiij

jj

j

iiij


The backward solution phase, shown in Equation (10), becomes:

yxU

5774.0

4082.0

7071.0

5774.000

8165.0225.10

07071.0414.1

3

2

1

x

x

x


1225.1

18165.04082.0

15774.0

5774.0

22

3232

3

312

33

3

3

u

xuy

u

xuy

x

u

y

u

yx

jj

N

jiijij

jj

j


1414.1

1017071.07071.011

3132121

3

211

u

xuxuy

u

xuy

xjj

N

jiijij

Hence

1

1

1

][x


Numerical Example 2

( Algorithms)TLDL

Using the same data given in Numerical Example 1,

find the unknown vector ][x by algorithms?TLDL

Solution:

The factorized matrices and can be computed

from Equation (24), and Equation (25), respectively.

][D ][L


LandDofmatricesofColumn

d

al

d

a

d

ldlal

alwaysl

a

dlad

jj

j

kjkkkik

j

kkkjk

1

02

0

5.02

1

)!(1

2

11

3131

11

21

01

121

21

11

11

01

1

21111


LandDmatricesofColumn

d

ldlal

alwaysl

dl

dlad

j

k

j

kkkjk

2

6667.05.1

5.0201

)!(1

5.1

25.02

2

22

11

121113132

32

22

2

11221

11

1

22222


LandDmatricesofColumndldl

dladj

kkkjk

3

3333.0

5.16667.0201

122

2223211

231

21

1

23333


Hence

3333.000

05.10

002

D

and

16667.00

015.0

001

L


The forward solution shown in Equation (31), becomes:

bzL

0

0

1

1667.00

015.0

001

3

2

1

z

z

z

or,

1

1

i

kkikii zlbz (32, repeated)


Hence

3333.0

5.06667.0100

5.0

15.00

1

23213133

12122

11

zLzLbz

zLbz

bz


The diagonal scaling phase, shown in Equation (29), becomes

zyD

3333.0

5.0

1

3333.000

05.10

002

3

2

1

y

y

y


or

ii

ii d

zy

Hence

13333.0

3333.0

3333.05.1

5.0

5.02

1

33

33

22

22

11

11

d

zy

d

zy

d

zy


The backward solution phase can be found by referring to Equation (27)

yxL T

1

333.0

5.0

100

667.010

05.01

3

2

1

x

x

x

N

ikkkiii xlyx

1(28, repeated)


Hence

1

1015.05.0

1

16667.03333.0

1

1

33122111

2

33222

33

x

xlxlyx

x

xlyx

yx


Hence

1

1

1

3

2

1

x

x

x

x


Re-ordering Algorithms ForMinimizing Fill-in Terms [1,2].

During the factorization phase (of Cholesky, orAlgorithms ), many “zero” terms in the original/given

matrix will become “non-zero” terms in the factoredmatrix

TLDL

][A

][U. These new non-zero terms are often

called as “fill-in” terms (indicated by the symbol )F It is, therefore, highly desirable to minimize thesefill-in terms , so that both computational time/effortand computer memory requirements can be substantially reduced.


For example, the following matrix and vector

are given:

][A ][b

1100102

0440030

0066040

1008850

03451107

20007112

A

14

47

70

94

129

121

][b

(33)

(34)


The Cholesky factorization matrix , based on the

original matrix (see Equation 33) and

Equations (6-7), or Figure 1, can be symbolically computed as:

00000

0000

000

00

0

000

F

FF

FF

F

U

][U][A

(35)


IPERM (new equation #) = {old equation #}

such as, for this particular example:

1

2

3

4

5

6

6

5

4

3

2

1

IPERM

(36)

(37)


Using the above results (see Equation 37), one will be able to construct the following re-arranged matrices:

11270002

71105430

0588001

0406600

0300440

2010011

*A

and

121

129

94

70

47

14

][ *b

(38)

(39)


Remarks:In the original matrix (shown in Equation 33), the nonzero term (old row 1, old column 2) = 7 will move to new location of the new matrix (new row 6, new column 5) = 7, etc.

The non zero term (old row 3, old column 3) = 88 will move to (new row 4, new column 4) = 88, etc.

The value of (old row 4) = 70 will be moved to (or located at) (new row 3) = 70, etc

b*b

A*A

A*A *A


Now, one would like to solve the following modified system of linear equations (SLE) for ],[ *x

][]][[ *** bxA

rather than to solve the original SLE (see Equation1). The original unknown vector can be easily recovered from and ,shown in Equation (37).

}{x][ *x IPERM

(40)


00000

0000

000

0000

0000

000

*

FU

The factorized matrix can be “symbolically” computed from as (by referring to either Figure 1, or Equations 6-7):

][ *U][ *A

(41)


4. On-Line Chess-Like Game For Reordering/Factorized Phase [4].

Figure 2 A Chess-Like Game For Learning to Solve SLE.


(A)Teaching undergraduate/HS students the process how to use the reordering output IPERM(-), see Equations (36-37) for converting the original/given matrix , see Equation (33), into the new/modified matrix , see Equation (38). This step is reflected in Figure 2, when the “Game Player” decides to swap node (or equation) (say ) with another node (or equation ) , and click the “CONFIRM” icon!

][A][ *A

""i 2i"" j


Since node is currently connected to nodes hence swapping node with the above nodes will “NOT” change the number/pattern of “Fill-in” terms. However, if node is swapped with node then the fill-in terms pattern maychange (for better or worse)!

"2" i;8,7,6,4j 2i"" j

2i,5,3,1 ororj


(B) Helping undergraduate/HS students to understand the “symbolic” factorization” phase, by symbolically utilizing the Cholesky factorized Equations (6-7). This step is illustrated in Figure 2, for which the “game player” will see (and also hear the computer animated sound, and human voice), the non-zero terms (including fill-in terms) of the original matrix to move to the new locations in the new/modified matrix .

][A][ *A


(C) Helping undergraduate/HS students to understand the “numerical factorization” phase, by numerically utilizing the same Cholesky factorized Equations (6-7).

(D) Teaching undergraduate engineering/science students and even high-school (HS) students to “understand existing reordering concepts”, or even to “discover new reordering algorithms”


5. Further Explanation On The Developed Game

1. In the above Chess-Like Game, which is available on-line [4], powerful features of FLASH computer environments [3], such as animated sound, human voice, motions, graphical colors etc… have all been incorporated and programmed into the developed game-software for more appealing to game players/learners.


2. In the developed “Chess-Like Game”, fictitious monetary (or any kind of ‘scoring system”) is rewarded(and broadcasted by computer animated human voice) to game players, based on how he/she swaps the node (or equation) numbers, and consequently based on how many fill-in terms occurred. In general, less fill-in terms introduced will result in more rewards!""F


3. Based on the original/given matrix , and existing re-ordering algorithms (such as the Reverse Cuthill-Mckee, or RCM algorithms [1-2]) the number of fill-in terms can be computed (using RCM algorithms). This internally generated information will be used to judge how good the players/learners are, and/or broadcast “congratulations message” to a particular player who discovers new “chess-like move” (or, swapping node) strategies which are even better than RCM algorithms!

][A

)"("F


4. Initially, the player(s) will select the matrix size ( , or larger is recommended), and the percentage (50%, or larger is suggested) of zero-terms (or sparsity of the matrix). Then, “START Game” icon will be clicked by the player.

88


5. The player will then CLICK one of the selected node (or equation) numbers appearing on the computer screen. The player will see those nodes which are connected to node (based on the given/generated matrix ). The player then has to decide to swap node with one of the possible node

""i"" j

""i][A

""i "" j


After confirming the player’s decision, the outcomes/results will be announced by the computer animated human voice, and the monetary-award will (or will NOT) be given to the players/learners, accordingly. In this software, a maximum of $1,000,000 can be earned by the player, and the “exact dollar amount” will be INVERSELY proportional to the number of fill-in terms occurred (as a consequence of the player’s decision on how to swap node with another node ).

""i"" j


6. The next player will continue to play, with his/her move (meaning to swap the node with the node) based on the current best non-zero terms pattern of the matrix.

thi thj

THE ENDhttp://numericalmethods.eng.usf.edu

This instructional power point brought to you byNumerical Methods for STEM undergraduatehttp://numericalmethods.eng.usf.eduCommitted to bringing numerical methods to the undergraduate

Acknowledgement


For instructional videos on other topics, go to

http://numericalmethods.eng.usf.edu/videos/

This material is based upon work supported by the National Science Foundation under Grant # 0717624. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.

http://numericalmethods.eng.usf.edu/videos/

The End - Really

Numerical Methods Part: Cholesky and Decomposition .

Documents

spd http

term http

computes http

getssimilarly131415

has16 http

rhs of equation

row of equation

given matrix