Numerical Methods Part: Cholesky and Decomposition .
Post on 12-Jan-2016
230 Views
Preview:
Transcript
Numerical Methods
Part: Cholesky and Decomposition
http://numericalmethods.eng.usf.edu
TLDL
For more details on this topic
Go to http://numericalmethods.eng.usf.edu
Click on Keyword
You are free
to Share – to copy, distribute, display and perform the work
to Remix – to make derivative works
Under the following conditions Attribution — You must attribute the
work in the manner specified by the author or licensor (but not in any way that suggests that they endorse you or your use of the work).
Noncommercial — You may not use this work for commercial purposes.
Share Alike — If you alter, transform, or build upon this work, you may distribute the resulting work only under the same or similar license to this one.
Chapter 04.09: Cholesky and DecompositionMajor: All Engineering Majors
Authors: Duc Nguyen
http://numericalmethods.eng.usf.edu
Numerical Methods for STEM undergraduateshttp://
numericalmethods.eng.usf.edu 504/21/23
Lecture # 1
TLDL
(1)
1
Introduction
6
http://numericalmethods.eng.usf.edu
][]][[ bxA
= known coefficient matrix, with dimension
where
][A nn
][b= known right-hand-side (RHS) 1n vector
][x = unknown 1n vector.
http://numericalmethods.eng.usf.edu7
Symmetrical Positive Definite (SPD) SLE
A matrix can be considered as SPD if either of
(a) If each and every determinant of sub-matrix is positive, or..
the following conditions is satisfied:
for any given vector (b) If
As a quick example, let us make a test a test to see if the given matrix
nnA ][
),...,2,1( niAii ,0AyyT 0][ 1
ny
110
121
012
][A is SPD?
http://numericalmethods.eng.usf.edu8
Symmetrical Positive Definite (SPD) SLE
Based on criteria (a):
The given matrix 33 is symmetrical, because
Furthermore,
022det 11 A
03
21
12det 22
A
jiij aa
http://numericalmethods.eng.usf.edu9
01
110
121
012
det 33
A
Hence is SPD.][A
http://numericalmethods.eng.usf.edu10
Based on criteria (b): For any given vector
0
3
2
1
y
y
y
y , one computes
32
23
22
21
221
3223
2221
21
3
2
1
321
2
2222
110
121
012
yyyyyyy
yyyyyyy
y
y
y
yyy
Ayyscalar T
http://numericalmethods.eng.usf.edu11
0232
21
221 yyyyyscalar
hence matrix is SPD][A
12
Step 1: Matrix Factorization phase
][][][ UUA T
33
2322
131211
332313
2212
11
333231
232221
131211
00
00
00
u
uu
uuu
uuu
uu
u
aaa
aaa
aaa
Multiplying two matrices on the right-hand-side (RHS) of Equation (3), one gets the following 6 equations
1111 au 11
1212 u
au
11
1313 u
au
21
2122222 uau
22
13122323 u
uuau
2
1223
2133333 uuau
(2)
(3)
(4)
(5)
http://numericalmethods.eng.usf.edu13
2
11
1
2
i
kkiiiii uau
ii
i
kkjkiij
ij u
uuau
1
1
Step 1.1: Compute the numerator of Equation (7),such as
1
1
i
kkjkiij uuaSum
Step 1.2 If is an off-diagonal term (say )iju ji then (See Equation (7)). Else, if is a
iiij u
Sumu iju
diagonal term (that is, ), thenji Sumuii
(6)
(7)
(See Equation (6))
http://numericalmethods.eng.usf.edu14
As a quick example, one computes:
55
47453735272517155757 u
uuuuuuuuau
Thus, for computing , one only needs)7,5( jiu
to use the (already factorized) data in columns )5(# iand of , respectively.)7(# j ][U
(8)
http://numericalmethods.eng.usf.edu15
k = 1
k = 2
k = 3
k = 4
iiu iju
ju4
ju3
ju2
ju1
iu4
iu3
iu2
iiu
i = 5
Col. # i=5 Col. # j=7
Figure 1: Cholesky Factorization for the term iju
http://numericalmethods.eng.usf.edu16
Step 2: Forward Solution phaseSubstituting Equation (2) into Equation (1), one gets:
][]][[][ bxUU T Let us define:
][][][ yxU Then, Equation (9) becomes:
][][][ byU T
3
2
1
3
2
1
332313
2212
11
0
00
b
b
b
y
y
y
uuu
uu
u
(9)
(10)
(11)
(12)
http://numericalmethods.eng.usf.edu17
1111 byu
11
11 u
by
From the 2nd row of Equation (12), one gets
2222112 byuyu
22
11222 u
yuby
Similarly
33
22311333 u
yuyuby
(13)
(14)
(15)
http://numericalmethods.eng.usf.edu18
In general, from the row of Equation (12), one hasthj
jj
j
iiijj
j u
yuby
1
1 (16)
http://numericalmethods.eng.usf.edu19
Step 3: Backward Solution phase
As a quick example, one has (See Equation (10)):
3
2
1
3
2
1
33
2322
131211
00
0
y
y
y
x
x
x
u
uu
uuu
(17)
http://numericalmethods.eng.usf.edu20
From the last (or ) row of Equation (17), rdthn 3
one has 3333 yxu
hence
33
33 u
yx
Similarly:
22
32322 u
xuyx
and
11
31321211 u
xuxuyx
(18)
(19)
(20)
http://numericalmethods.eng.usf.edu21
In general, one has:
jj
n
jiijij
j u
xuy
x
1 (21)
http://numericalmethods.eng.usf.edu22
TLDLA ]][][[][
For example,
100
10
1
00
00
00
1
01
001
32
3121
33
22
11
3231
21
333231
232221
131211
l
ll
d
d
d
ll
l
aaa
aaa
aaa
Multiplying the three matrices on the RHS of Equation (23), one obtains the following formulas for
the “diagonal” , and “lower-triangular” matrices:][D ][L
(22)
(23)
http://numericalmethods.eng.usf.edu23
1
1
2j
kkkjkjjjj dlad
jj
j
kjkkkikijij d
ldlal11
1
(24)
(25)
http://numericalmethods.eng.usf.edu24
Step1: Factorization phaseTLDLA ]][][[][
Step 2: Forward solution and diagonal scaling phase
Substituting Equation (22) into Equation (1), one gets:
][][]][][[ bxLDL T Let us define:
][][][ yxL T
(22, repeated)
(26)
http://numericalmethods.eng.usf.edu25
Also, define: ][]][[ zyD
3
2
1
3
2
1
33
22
11
00
00
00
z
z
z
y
y
y
d
d
d
niford
zy
ii
ii ......,,3,2,1,
Then Equation (26) becomes:
][]][[ bzL
(29)
(30)
http://numericalmethods.eng.usf.edu26
3
2
1
3
2
1
3231
21
1
01
001
b
b
b
z
z
z
ll
l
niforzLbzi
kkikii ......,,3,2,1
1
1
(31)
(32)
http://numericalmethods.eng.usf.edu27
Step 3: Backward solution phase
3
2
1
3
2
1
32
3121
100
10
1
y
y
y
x
x
x
l
ll
1......,,1,;1
nniforxlyxn
ikkkiii
http://numericalmethods.eng.usf.edu28
Numerical Example 1 (Cholesky algorithms)
Solve the following SLE system for the unknown
vector ? x][]][[ bxA
where
110
121
012
A
0
0
1
][b
http://numericalmethods.eng.usf.edu29
Solution:
The factorized, Uupper triangular matrix can be computed by either referring to Equations(6-7), or looking at Figure 1, as following:
][1
0414.1
0
7071.0414.1
1
414.1
2
11
1313
11
1212
1111
Uofrow
u
au
u
au
au
http://numericalmethods.eng.usf.edu30
][2
8165.0225.1
07071.01225.1
1
225.1
7071.02
2
1312
22
11
123
23
2
2
12
12
2
111
1
22222
Uofrow
uu
U
uuau
u
uau
i
kkjki
i
kki
http://numericalmethods.eng.usf.edu31
][3
5774.0
8165.001 22
2
1223
21333
2
121
1
23333
Uofrowuua
uaui
kki
Thus, the factorized matrix
5774.000
8165.0225.10
07071.0414.1
U
http://numericalmethods.eng.usf.edu32
The forward solution phase, shown in Equation (11), becomes:
byU T
0
0
1
5774.08165.00
0225.17071.0
00414.1
3
2
1
y
y
y
http://numericalmethods.eng.usf.edu33
5774.0
5774.0
4082.08165.07071.000
4082.0
225.1
7071.07071.00
7071.0
414.1
1
33
223113
21
13
3
22
112
11
12
2
11
11
u
yuyu
u
yuby
u
yu
u
yuby
u
by
jj
j
iiij
jj
j
iiij
http://numericalmethods.eng.usf.edu34
The backward solution phase, shown in Equation (10), becomes:
yxU
5774.0
4082.0
7071.0
5774.000
8165.0225.10
07071.0414.1
3
2
1
x
x
x
http://numericalmethods.eng.usf.edu35
1225.1
18165.04082.0
15774.0
5774.0
22
3232
3
312
33
3
3
u
xuy
u
xuy
x
u
y
u
yx
jj
N
jiijij
jj
j
http://numericalmethods.eng.usf.edu36
1414.1
1017071.07071.011
3132121
3
211
u
xuxuy
u
xuy
xjj
N
jiijij
Hence
1
1
1
][x
http://numericalmethods.eng.usf.edu37
Numerical Example 2
( Algorithms)TLDL
Using the same data given in Numerical Example 1,
find the unknown vector ][x by algorithms?TLDL
Solution:
The factorized matrices and can be computed
from Equation (24), and Equation (25), respectively.
][D ][L
http://numericalmethods.eng.usf.edu38
LandDofmatricesofColumn
d
al
d
a
d
ldlal
alwaysl
a
dlad
jj
j
kjkkkik
j
kkkjk
1
02
0
5.02
1
)!(1
2
11
3131
11
21
01
121
21
11
11
01
1
21111
http://numericalmethods.eng.usf.edu39
LandDmatricesofColumn
d
ldlal
alwaysl
dl
dlad
j
k
j
kkkjk
2
6667.05.1
5.0201
)!(1
5.1
25.02
2
22
11
121113132
32
22
2
11221
11
1
22222
http://numericalmethods.eng.usf.edu40
LandDmatricesofColumndldl
dladj
kkkjk
3
3333.0
5.16667.0201
122
2223211
231
21
1
23333
http://numericalmethods.eng.usf.edu41
Hence
3333.000
05.10
002
D
and
16667.00
015.0
001
L
http://numericalmethods.eng.usf.edu42
The forward solution shown in Equation (31), becomes:
bzL
0
0
1
1667.00
015.0
001
3
2
1
z
z
z
or,
1
1
i
kkikii zlbz (32, repeated)
http://numericalmethods.eng.usf.edu43
Hence
3333.0
5.06667.0100
5.0
15.00
1
23213133
12122
11
zLzLbz
zLbz
bz
http://numericalmethods.eng.usf.edu44
The diagonal scaling phase, shown in Equation (29), becomes
zyD
3333.0
5.0
1
3333.000
05.10
002
3
2
1
y
y
y
http://numericalmethods.eng.usf.edu45
or
ii
ii d
zy
Hence
13333.0
3333.0
3333.05.1
5.0
5.02
1
33
33
22
22
11
11
d
zy
d
zy
d
zy
http://numericalmethods.eng.usf.edu46
The backward solution phase can be found by referring to Equation (27)
yxL T
1
333.0
5.0
100
667.010
05.01
3
2
1
x
x
x
N
ikkkiii xlyx
1(28, repeated)
http://numericalmethods.eng.usf.edu47
Hence
1
1015.05.0
1
16667.03333.0
1
1
33122111
2
33222
33
x
xlxlyx
x
xlyx
yx
http://numericalmethods.eng.usf.edu48
Hence
1
1
1
3
2
1
x
x
x
x
http://numericalmethods.eng.usf.edu49
Re-ordering Algorithms ForMinimizing Fill-in Terms [1,2].
During the factorization phase (of Cholesky, orAlgorithms ), many “zero” terms in the original/given
matrix will become “non-zero” terms in the factoredmatrix
TLDL
][A
][U. These new non-zero terms are often
called as “fill-in” terms (indicated by the symbol )F It is, therefore, highly desirable to minimize thesefill-in terms , so that both computational time/effortand computer memory requirements can be substantially reduced.
http://numericalmethods.eng.usf.edu50
For example, the following matrix and vector
are given:
][A ][b
1100102
0440030
0066040
1008850
03451107
20007112
A
14
47
70
94
129
121
][b
(33)
(34)
http://numericalmethods.eng.usf.edu51
The Cholesky factorization matrix , based on the
original matrix (see Equation 33) and
Equations (6-7), or Figure 1, can be symbolically computed as:
00000
0000
000
00
0
000
F
FF
FF
F
U
][U][A
(35)
http://numericalmethods.eng.usf.edu52
IPERM (new equation #) = {old equation #}
such as, for this particular example:
1
2
3
4
5
6
6
5
4
3
2
1
IPERM
(36)
(37)
http://numericalmethods.eng.usf.edu53
Using the above results (see Equation 37), one will be able to construct the following re-arranged matrices:
11270002
71105430
0588001
0406600
0300440
2010011
*A
and
121
129
94
70
47
14
][ *b
(38)
(39)
http://numericalmethods.eng.usf.edu54
Remarks:In the original matrix (shown in Equation 33), the nonzero term (old row 1, old column 2) = 7 will move to new location of the new matrix (new row 6, new column 5) = 7, etc.
The non zero term (old row 3, old column 3) = 88 will move to (new row 4, new column 4) = 88, etc.
The value of (old row 4) = 70 will be moved to (or located at) (new row 3) = 70, etc
b*b
A*A
A*A *A
http://numericalmethods.eng.usf.edu55
Now, one would like to solve the following modified system of linear equations (SLE) for ],[ *x
][]][[ *** bxA
rather than to solve the original SLE (see Equation1). The original unknown vector can be easily recovered from and ,shown in Equation (37).
}{x][ *x IPERM
(40)
http://numericalmethods.eng.usf.edu56
00000
0000
000
0000
0000
000
*
FU
The factorized matrix can be “symbolically” computed from as (by referring to either Figure 1, or Equations 6-7):
][ *U][ *A
(41)
http://numericalmethods.eng.usf.edu57
4. On-Line Chess-Like Game For Reordering/Factorized Phase [4].
Figure 2 A Chess-Like Game For Learning to Solve SLE.
http://numericalmethods.eng.usf.edu58
(A)Teaching undergraduate/HS students the process how to use the reordering output IPERM(-), see Equations (36-37) for converting the original/given matrix , see Equation (33), into the new/modified matrix , see Equation (38). This step is reflected in Figure 2, when the “Game Player” decides to swap node (or equation) (say ) with another node (or equation ) , and click the “CONFIRM” icon!
][A][ *A
""i 2i"" j
http://numericalmethods.eng.usf.edu59
Since node is currently connected to nodes hence swapping node with the above nodes will “NOT” change the number/pattern of “Fill-in” terms. However, if node is swapped with node then the fill-in terms pattern maychange (for better or worse)!
"2" i;8,7,6,4j 2i"" j
2i,5,3,1 ororj
http://numericalmethods.eng.usf.edu60
(B) Helping undergraduate/HS students to understand the “symbolic” factorization” phase, by symbolically utilizing the Cholesky factorized Equations (6-7). This step is illustrated in Figure 2, for which the “game player” will see (and also hear the computer animated sound, and human voice), the non-zero terms (including fill-in terms) of the original matrix to move to the new locations in the new/modified matrix .
][A][ *A
http://numericalmethods.eng.usf.edu61
(C) Helping undergraduate/HS students to understand the “numerical factorization” phase, by numerically utilizing the same Cholesky factorized Equations (6-7).
(D) Teaching undergraduate engineering/science students and even high-school (HS) students to “understand existing reordering concepts”, or even to “discover new reordering algorithms”
http://numericalmethods.eng.usf.edu62
5. Further Explanation On The Developed Game
1. In the above Chess-Like Game, which is available on-line [4], powerful features of FLASH computer environments [3], such as animated sound, human voice, motions, graphical colors etc… have all been incorporated and programmed into the developed game-software for more appealing to game players/learners.
http://numericalmethods.eng.usf.edu63
2. In the developed “Chess-Like Game”, fictitious monetary (or any kind of ‘scoring system”) is rewarded(and broadcasted by computer animated human voice) to game players, based on how he/she swaps the node (or equation) numbers, and consequently based on how many fill-in terms occurred. In general, less fill-in terms introduced will result in more rewards!""F
http://numericalmethods.eng.usf.edu64
3. Based on the original/given matrix , and existing re-ordering algorithms (such as the Reverse Cuthill-Mckee, or RCM algorithms [1-2]) the number of fill-in terms can be computed (using RCM algorithms). This internally generated information will be used to judge how good the players/learners are, and/or broadcast “congratulations message” to a particular player who discovers new “chess-like move” (or, swapping node) strategies which are even better than RCM algorithms!
][A
)"("F
http://numericalmethods.eng.usf.edu65
4. Initially, the player(s) will select the matrix size ( , or larger is recommended), and the percentage (50%, or larger is suggested) of zero-terms (or sparsity of the matrix). Then, “START Game” icon will be clicked by the player.
88
http://numericalmethods.eng.usf.edu66
5. The player will then CLICK one of the selected node (or equation) numbers appearing on the computer screen. The player will see those nodes which are connected to node (based on the given/generated matrix ). The player then has to decide to swap node with one of the possible node
""i"" j
""i][A
""i "" j
http://numericalmethods.eng.usf.edu67
After confirming the player’s decision, the outcomes/results will be announced by the computer animated human voice, and the monetary-award will (or will NOT) be given to the players/learners, accordingly. In this software, a maximum of $1,000,000 can be earned by the player, and the “exact dollar amount” will be INVERSELY proportional to the number of fill-in terms occurred (as a consequence of the player’s decision on how to swap node with another node ).
""i"" j
http://numericalmethods.eng.usf.edu68
6. The next player will continue to play, with his/her move (meaning to swap the node with the node) based on the current best non-zero terms pattern of the matrix.
thi thj
THE ENDhttp://numericalmethods.eng.usf.edu
This instructional power point brought to you byNumerical Methods for STEM undergraduatehttp://numericalmethods.eng.usf.eduCommitted to bringing numerical methods to the undergraduate
Acknowledgement
For instructional videos on other topics, go to
http://numericalmethods.eng.usf.edu/videos/
This material is based upon work supported by the National Science Foundation under Grant # 0717624. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.
The End - Really
top related