Solution of Sparse Linear Systems
Post on 11-Jan-2016
59 Views
Preview:
DESCRIPTION
Transcript
Solution of Sparse Linear Systems
Numerical Simulation
CSE245
Thanks to: Kin Sou, Deepak Ramaswamy, Michal Rewienski, Jacob White, Shihhsien Kuo and Karen Veroy and
especially Luca Daniel
2
Outline of today’s lecture
• Solution of Sparse Linear SystemsSolution of Sparse Linear Systems– Examples of Problems with Sparse Matrices Examples of Problems with Sparse Matrices
• Struts and joints, resistor grids, 3-D heat flowStruts and joints, resistor grids, 3-D heat flow – Tridiagonal Matrix FactorizationTridiagonal Matrix Factorization– General Sparse FactorizationGeneral Sparse Factorization
• Fill-in and ReorderingFill-in and Reordering• Graph Based ApproachGraph Based Approach
– Sparse Matrix Data StructuresSparse Matrix Data Structures• ScatteringScattering
3
X
X
X
X
X
X
X
X
X
1
2
3
4
5
6
7
8
9
2 x 2 block
X =
Sparse MatricesApplications
Space Frame
Space FrameX
X X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
XX
X X
X
X
X X
X
Unknowns : Joint positionsEquations : forces = 0
X
Nodal Matrix
X
X
X
X
X
X
X
XX
X
X
4
1 2 3 4 1m m
2m
1m 2m 3m 2m
( 1) ( 1)m m Unknowns : Node VoltagesEquations : currents = 0
Sparse MatricesApplications
Resistor Grid
Nodal Formulation
Matrix non-zero locations for 100 x 10 Resistor Grid
Sparse MatricesApplications
Resistor Grid
Nodal FormulationSparse MatricesApplications
Temperature in a cube
Temperature known on surface, determine interior temperature
1 2
1m 2m
2 1m 2 2m CircuitModel
7
Outline
• Solution of Dense Linear SystemsSolution of Dense Linear Systems
• Solution of Sparse Linear SystemsSolution of Sparse Linear Systems– LU Factorization Reminder.LU Factorization Reminder.– Example of Problems with Sparse MatricesExample of Problems with Sparse Matrices
• Struts and joints, resistor grids, 3-d heat flowStruts and joints, resistor grids, 3-d heat flow – Tridiagonal Matrix FactorizationTridiagonal Matrix Factorization– General Sparse FactorizationGeneral Sparse Factorization
• Fill-in and ReorderingFill-in and Reordering• Graph Based ApproachGraph Based Approach
– Sparse Matrix Data StructuresSparse Matrix Data Structures• ScatteringScattering
Nodal FormulationSparse MatricesTridiagonal Example
Matrix Form
1 2 3 4 1m m
X X
X X X
X X X
X X X
X X X
X X X
X X X
X X
m
How many operations to do LU factorization?
9
Sparse MatricesGE Algorithm
Tridiagonal Example
For i = 1 to n-1 { “For each Row”
For j = i+1 to n { “For each target Row below the source”
For k = i+1 to n { “For each Row element beyond Pivot”
} }}
Pivotjiji
ii
MM
M
Multiplier
jk jk ji ikM M M M Order n Operations!
10
Outline
• Solution of Dense Linear SystemsSolution of Dense Linear Systems
• Solution of Sparse Linear SystemsSolution of Sparse Linear Systems– LU Factorization Reminder.LU Factorization Reminder.– Example of Problems with Sparse MatricesExample of Problems with Sparse Matrices
• Struts and joints, resistor grids, 3-d heat flowStruts and joints, resistor grids, 3-d heat flow – Tridiagonal Matrix FactorizationTridiagonal Matrix Factorization– General Sparse FactorizationGeneral Sparse Factorization
• Fill-in and ReorderingFill-in and Reordering• Graph Based ApproachGraph Based Approach
– Sparse Matrix Data StructuresSparse Matrix Data Structures• ScatteringScattering
11
1R
5R
3R
Sparse MatricesSparse Matrix Fill-In
4R
2R1V 2V
3V
1Si
1
1
2
3
0
0
S
V
V
iV
1
1
R 2
1
R
2
1
R
2
1
R
2
1
R 3
1
R
4
1
R
4
1
R
4
1
R
4
1
R 5
1
R
SymmetricDiagonally Dominant
Nodal Matrix
Resistor Example
Example
0
12
Sparse MatricesFill-In
Example
X X X
X X 0
X 0 X
X X X
X X 0
X 0 X
X
X
X X
X= Non zero
Matrix Non zero structure Fill Ins
X X
X = fill in where 0 became non-zero
13
Sparse MatricesFill-In
Second Example
X X X X
X X 0 0
0 X X 0
X 0 00
Fill-ins Propagate
XX
X
X
X
X X
X
X X
Fill-ins from Step 1 result in Fill-ins in step 2
14
Sparse MatricesFill-In
Reordering
1V 2V
3V
0
x x x
x x x
x x x
Fill-ins
Node Reordering - Can reduce fill-in - Preserves properties (Symmetry, Diagonal Dominance) - Equivalent to swapping rows and columns
No Fill-ins2V 1V
3V
0
x x 0
x x x
0 x x
15
Sparse MatricesFill-In
Reordering
Where can fill-in occur ?
x
x x
x x
x x
x
x
Multipliers
Already Factored
Possible Fill-inLocations
Fill-in Estimate = (Non zeros in unfactored part of Row -i) (Non zeros in unfactored part of Col -i) MarkowitzMarkowitz product
0 0
x 0
0 x
16
Sparse MatricesFill-In
Reordering
Markowitz Reordering
tor oF 1i n with min Mard kowitz Productiagonan lFi d ij
anSw d rows columns a p j ji i the new row and determine fiFactor ll-insi
End
Greedy Algorithm !
17
Sparse MatricesFill-In
Why only try diagonals ?
• Corresponds to node reordering in Nodal formulation
03
1 2
02
3 1
• Reduces search cost • Preserves Matrix Properties - Diagonal Dominance - Symmetry
Reordering
18
Sparse Matrices Fill-In
Pattern of a Filled-in Matrix
Very Sparse
Very Sparse
Dense
19
Sparse Matrices Fill-In
Unfactored Random Matrix
Symmetric
20
Sparse Matrices Fill-In
Factored Random Matrix
Symmetric
21
Outline
• Solution of Dense Linear SystemsSolution of Dense Linear Systems
• Solution of Sparse Linear SystemsSolution of Sparse Linear Systems– LU Factorization Reminder.LU Factorization Reminder.– Example of Problems with Sparse MatricesExample of Problems with Sparse Matrices
• Struts and joints, resistor grids, 3-d heat flowStruts and joints, resistor grids, 3-d heat flow – Tridiagonal Matrix FactorizationTridiagonal Matrix Factorization– General Sparse FactorizationGeneral Sparse Factorization
• Fill-in and ReorderingFill-in and Reordering• Graph Based ApproachGraph Based Approach
– Sparse Matrix Data StructuresSparse Matrix Data Structures• ScatteringScattering
22
X
X
X
X
X
X
X X
X
XX
X
X
X X 1
2
4
3
5
• One Node Per Matrix Row
• One Edge Per Off-diagonal Pair
X
X
Sparse MatricesMatrix Graphs
Construction
Structurally Symmetric Matrices and Graphs
23
Sparse MatricesMatrix Graphs
Markowitz Products
X
X
X
X
X
X
X X
X
XX
X
X
X X
X
X 1
2
4
3
5
Markowitz Products = (Node Degree)2
11 3 3 9M 2(degree 1) 933
22 2 2 4M 2(deg ree 2) 422
33 3 3 9M 2(deg ree 3) 933
44 2 2 4M 2(degree 4) 422
55 2 2 4M 2(degree 5) 422
24
X
X
X
X
X
X
X X
X
XX
X
X
X X 1
2
4
3
5
• Delete the node associated with pivot row
• “Tie together” the graph edges
X
X
X
X
X
Sparse MatricesMatrix Graphs
Factorization
X
One Step of LU Factorization
25
Sparse MatricesMatrix Graphs
Example
x x x x
x x x
x x x x
x x x x
x x x x
1 2 3 4 5
Graph
Markowitz products( = Node degree)
Col Row
3 3 9
2 2 4
3 3 9
3 3 9
3 3 9
1
2
3
4
5
26
Sparse MatricesMatrix Graphs
Example
x x x
x x 0 x x
x 0 x x x
x x x x
x x x x
Swap 2 with 1
X
X
2 3 4 5
Graph
2 3 4 5
Eliminate 1
112
27
1 2 3 4 1m m
2m
1m 2m 3m 2m
( 1) ( 1)m m Unknowns : Node VoltagesEquations : currents = 0
Sparse MatricesGraphs
Resistor Grid Example
28
Sparse MatricesMatrix Graphs
Grid Example
29
Sparse Matrices For growth control?Or to avoid fill-ins?
What should you pivot for?
A) LU factorization applied to a strictly diagonally dominant matrix will never produce a zero pivot
B) The matrix entries produced by LU factorization applied to a strictly diagonally dominant matrix will never increase by more than a factor 2(n-1)
[which is anyway the best you can do by pivoting for growth control]
Bottom line:if your matrix is strictly diagonally dominantno need for numerical pivot for growth control!
Sparse Factorization Approach
1)1) Assume matrix requires NO numerical pivoting.Assume matrix requires NO numerical pivoting.Diagonally dominant or symmetric positive definite.Diagonally dominant or symmetric positive definite.
2)2) Pre-Process:Pre-Process:• Use Graphs to Determine Matrix OrderingUse Graphs to Determine Matrix Ordering
• Many graph manipulation tricks used.Many graph manipulation tricks used.
• Form Data Structure for Storing Filled-in MatrixForm Data Structure for Storing Filled-in Matrix• Lots of additional non-zero addedLots of additional non-zero added
3)3) Put numerical values in Data Structure and factorPut numerical values in Data Structure and factorComputation must be organized carefully!Computation must be organized carefully!
Summary of Sparse Systems
• LU Factorization and Diagonal Dominance.LU Factorization and Diagonal Dominance.– Factor without numerical pivotingFactor without numerical pivoting
• Sparse Matrices Sparse Matrices – Struts, resistor grids, 3-d heat flow -> O(N) non-zerosStruts, resistor grids, 3-d heat flow -> O(N) non-zeros
• Tridiagonal Matrix FactorizationTridiagonal Matrix Factorization– Factor in O(N) operationsFactor in O(N) operations
• General Sparse FactorizationGeneral Sparse Factorization– Markowitz Reordering to minimize fillMarkowitz Reordering to minimize fill
• Graph Based ApproachGraph Based Approach– Factorization and Fill-inFactorization and Fill-in– Useful for estimating Sparse GE complexityUseful for estimating Sparse GE complexity
32
Outline
• Solution of Sparse Linear SystemsSolution of Sparse Linear Systems– Example of Problems with Sparse MatricesExample of Problems with Sparse Matrices– Tridiagonal Matrix FactorizationTridiagonal Matrix Factorization– General Sparse FactorizationGeneral Sparse Factorization
• Fill-in and ReorderingFill-in and Reordering• Graph Based ApproachGraph Based Approach• Summary of the AlgorithmSummary of the Algorithm
– Sparse Matrix Data StructuresSparse Matrix Data Structures• Scattering and symbolic fill inScattering and symbolic fill in
33
Sparse Matrices Sparse Data Structure
Val 11
Col 11
Val 12
Col 12
Val 1K
Col 1K
Val 21
Col 21
Val 22
Col 22
Val 2L
Col 2L
Val N1
Col N1
Val N2
Col N2
Val Nj
Col Nj
1
N
Arrays of Data in a RowVector of row pointers
Matrix entriesColumn index
Row pointersColumn Indices
Goal:Never store a 0Never multiply by 0
34
Sparse Matrices Sparse Data Structure
Problem of Misses
, 1 , 4 , 5 , 7 , 9 , 12 , 15
1 4 5 7 9 12 15j k j k j k j k j k j k j kM M M M M M M
k k k k k k k
, 1 , 7 , 15
1 7 15i k i k i kM M M
k k k
Eliminating Source Row i from Target row j
Row i
Row j
Must read all the row j entries to find the 3 that match row i
35
Sparse Matrices Sparse Data Structure
Data on Misses
Rows Ops Misses
Res 300 904387 248967
RAM 2806 1017289 3817587
Grid 4356 3180726 3597746
Every Miss is an unneeded Memory Reference!
More missesthanops!
36
Sparse Matrices Sparse Data Structure
Scattering for Miss Avoidance
Row j
, 1j kM , 4j kM , 5j kM , 7j kM , 9j kM , 12j kM , 15j kM
Use target row approach – Row j is the target.1) Read all the elements in Row j, and scatter them in an n-length vector2) Access only the needed elements using array indexing!
, 1 , 4 , 5 , 7 , 9 , 12 , 15
1 4 5 7 9 12 15j k j k j k j k j k j k j kM M M M M M M
k k k k k k k
37
Sparse Matrices – Another Data StructureOrthogonal linked list
But if fill in occurs, pointer structures change. What do we do?
Pre-compute pivoting order and sparse fill in structure symbolically
38
Summary of Sparse Systems
• LU Factorization and Diagonal Dominance.LU Factorization and Diagonal Dominance.– Factor without numerical pivotingFactor without numerical pivoting– hard problems (ill-conditioned)hard problems (ill-conditioned)
• Sparse Matrices Sparse Matrices – Struts, resistor grids, 3-d heat flow -> O(N) nonzerosStruts, resistor grids, 3-d heat flow -> O(N) nonzeros
• Tridiagonal Matrix FactorizationTridiagonal Matrix Factorization– Factor in O(N) operationsFactor in O(N) operations
• General Sparse FactorizationGeneral Sparse Factorization– Markowitz Reordering to minimize fillMarkowitz Reordering to minimize fill
• Graph Based ApproachGraph Based Approach– Factorization and Fill-inFactorization and Fill-in– Useful for estimating Sparse GE complexityUseful for estimating Sparse GE complexity
• Sparse Data StructuresSparse Data Structures– Scattering and symbolic fill inScattering and symbolic fill in
top related