Existence, Uniqueness, and Conditioning Solving Linear Systems Special Types of Linear Systems Software for Linear Systems Scientific Computing: An Introductory Survey Chapter 2 – Systems of Linear Equations Prof. Michael T. Heath Department of Computer Science University of Illinois at Urbana-Champaign Copyright c 2002. Reproduction permitted for noncommercial, educational use only. Michael T. Heath Scientific Computing 1 / 88
88
Embed
Scientific Computing: An Introductory Surveyheath.cs.illinois.edu/scicomp/notes/chap02.pdf · Chapter 2 – Systems of ... (4 2)=3 T is solution for any real number , ... Computing
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Existence, Uniqueness, and ConditioningSolving Linear Systems
Special Types of Linear SystemsSoftware for Linear Systems
Scientific Computing: An Introductory SurveyChapter 2 – Systems of Linear Equations
Prof. Michael T. Heath
Department of Computer ScienceUniversity of Illinois at Urbana-Champaign
Row scaling: premultiplying both sides of system bynonsingular diagonal matrix D, DAx = Db, multiplieseach row of matrix and right-hand side by correspondingdiagonal entry of D, but solution x is unchanged
Column scaling: postmultiplying A by D, ADx = b,multiplies each column of matrix by correspondingdiagonal entry of D, which rescales original solution
x = (AD)−1b = D−1A−1b
Michael T. Heath Scientific Computing 28 / 88
Existence, Uniqueness, and ConditioningSolving Linear Systems
Special Types of Linear SystemsSoftware for Linear Systems
If one equation in system involves only one component ofsolution (i.e., only one entry in that row of matrix isnonzero), then that component can be computed bydivision
If another equation in system involves only one additionalsolution component, then by substituting one knowncomponent into it, we can solve for other component
If this pattern continues, with only one new solutioncomponent per equation, then all components of solutioncan be computed in succession.
System with this property is called triangular
Michael T. Heath Scientific Computing 29 / 88
Existence, Uniqueness, and ConditioningSolving Linear Systems
Special Types of Linear SystemsSoftware for Linear Systems
Having obtained LU factorization, Ax = b becomesLUx = b, and can be solved by forward-substitution inlower triangular system Ly = b, followed byback-substitution in upper triangular system Ux = y
Note that y = Mb is same as transformed right-hand sidein Gaussian elimination
Gaussian elimination and LU factorization are two ways ofexpressing same solution process
Michael T. Heath Scientific Computing 43 / 88
Existence, Uniqueness, and ConditioningSolving Linear Systems
Special Types of Linear SystemsSoftware for Linear Systems
Gaussian elimination breaks down if leading diagonal entryof remaining unreduced matrix is zero at any stageEasy fix: if diagonal entry in column k is zero, theninterchange row k with some subsequent row havingnonzero entry in column k and then proceed as usualIf there is no nonzero on or below diagonal in column k,then there is nothing to do at this stage, so skip to nextcolumnZero on diagonal causes resulting upper triangular matrixU to be singular, but LU factorization can still be completedSubsequent back-substitution will fail, however, as it shouldfor singular matrix
Michael T. Heath Scientific Computing 48 / 88
Existence, Uniqueness, and ConditioningSolving Linear Systems
Special Types of Linear SystemsSoftware for Linear Systems
In principle, any nonzero value will do as pivot, but inpractice pivot should be chosen to minimize errorpropagation
To avoid amplifying previous rounding errors whenmultiplying remaining portion of matrix by elementaryelimination matrix, multipliers should not exceed 1 inmagnitude
This can be accomplished by choosing entry of largestmagnitude on or below diagonal as pivot at each stage
Such partial pivoting is essential in practice for numericallystable implementation of Gaussian elimination for generallinear systems < interactive example >
With partial pivoting, each Mk is preceded by permutationPk to interchange rows to bring entry of largest magnitudeinto diagonal pivot positionStill obtain MA = U , with U upper triangular, but now
M = Mn−1Pn−1 · · ·M1P1
L = M−1 is still triangular in general sense, but notnecessarily lower triangularAlternatively, we can write
PA = LU
where P = Pn−1 · · ·P1 permutes rows of A into orderdetermined by partial pivoting, and now L is lowertriangular
Michael T. Heath Scientific Computing 50 / 88
Existence, Uniqueness, and ConditioningSolving Linear Systems
Special Types of Linear SystemsSoftware for Linear Systems
Complete pivoting is more exhaustive strategy in whichlargest entry in entire remaining unreduced submatrix ispermuted into diagonal pivot positionRequires interchanging columns as well as rows, leadingto factorization
PAQ = LU
with L unit lower triangular, U upper triangular, and P andQ permutationsNumerical stability of complete pivoting is theoreticallysuperior, but pivot search is more expensive than for partialpivotingNumerical stability of partial pivoting is more thanadequate in practice, so it is almost always used in solvinglinear systems by Gaussian elimination
Michael T. Heath Scientific Computing 51 / 88
Existence, Uniqueness, and ConditioningSolving Linear Systems
Special Types of Linear SystemsSoftware for Linear Systems
Using small pivot, and correspondingly large multiplier, hascaused loss of information in transformed matrixIf rows interchanged, then pivot is 1 and multiplier is −ε, so
M =
[1 0−ε 1
], L =
[1 0ε 1
],
U =
[1 10 1− ε
]=
[1 10 1
]in floating-point arithmeticThus,
LU =
[1 0ε 1
] [1 10 1
]=
[1 1ε 1
]which is correct after permutation
Michael T. Heath Scientific Computing 54 / 88
Existence, Uniqueness, and ConditioningSolving Linear Systems
Special Types of Linear SystemsSoftware for Linear Systems
Residual is as small as we can expect using 3-digitarithmetic, but exact solution is
x =[1.00 1.00
]Tso error is almost as large as solutionCause of this phenomenon is that matrix is nearly singular(cond(A) > 4000)Division that determines x2 is between two quantities thatare both on order of rounding error, and hence result isessentially arbitraryWhen arbitrary value for x2 is substituted into firstequation, value for x1 is computed so that first equation issatisfied, yielding small residual, but poor solution
Michael T. Heath Scientific Computing 59 / 88
Existence, Uniqueness, and ConditioningSolving Linear Systems
Special Types of Linear SystemsSoftware for Linear Systems
Gaussian elimination has general form of triple-nested loop
forfor
foraij = aij − (aik/akk)akj
endend
end
Indices i, j, and k of for loops can be taken in any order,for total of 3! = 6 different arrangementsThese variations have different memory access patterns,which may cause their performance to vary widely ondifferent computers
Michael T. Heath Scientific Computing 60 / 88
Existence, Uniqueness, and ConditioningSolving Linear Systems
Special Types of Linear SystemsSoftware for Linear Systems
Despite variations in computing it, LU factorization isunique up to diagonal scaling of factors
Provided row pivot sequence is same, if we have two LUfactorizations PA = LU = LU , then L−1L = UU−1 = Dis both lower and upper triangular, hence diagonal
If both L and L are unit lower triangular, then D must beidentity matrix, so L = L and U = U
Uniqueness is made explicit in LDU factorizationPA = LDU , with L unit lower triangular, U unit uppertriangular, and D diagonal
Michael T. Heath Scientific Computing 61 / 88
Existence, Uniqueness, and ConditioningSolving Linear Systems
Special Types of Linear SystemsSoftware for Linear Systems
Elementary elimination matrices Mk, their inverses Lk,and permutation matrices Pk used in formal description ofLU factorization process are not formed explicitly in actualimplementation
U overwrites upper triangle of A, multipliers in L overwritestrict lower triangle of A, and unit diagonal of L need notbe stored
Row interchanges usually are not done explicitly; auxiliaryinteger vector keeps track of row order in original locations
Michael T. Heath Scientific Computing 62 / 88
Existence, Uniqueness, and ConditioningSolving Linear Systems
Special Types of Linear SystemsSoftware for Linear Systems
LU factorization requires about n3/3 floating-pointmultiplications and similar number of additions
Forward- and back-substitution for single right-hand-sidevector together require about n2 multiplications and similarnumber of additions
Can also solve linear system by matrix inversion:x = A−1b
Computing A−1 is tantamount to solving n linear systems,requiring LU factorization of A followed by n forward- andback-substitutions, one for each column of identity matrix
Operation count for inversion is about n3, three times asexpensive as LU factorization
Michael T. Heath Scientific Computing 63 / 88
Existence, Uniqueness, and ConditioningSolving Linear Systems
Special Types of Linear SystemsSoftware for Linear Systems
Even with many right-hand sides b, inversion neverovercomes higher initial cost, since each matrix-vectormultiplication A−1b requires n2 operations, similar to costof forward- and back-substitutionInversion gives less accurate answer; for example, solving3x = 18 by division gives x = 18/3 = 6, but inversion givesx = 3−1 × 18 = 0.333× 18 = 5.99 using 3-digit arithmeticMatrix inverses often occur as convenient notation informulas, but explicit inverse is rarely required toimplement such formulasFor example, product A−1B should be computed by LUfactorization of A, followed by forward- andback-substitutions using each column of B
Michael T. Heath Scientific Computing 64 / 88
Existence, Uniqueness, and ConditioningSolving Linear Systems
Special Types of Linear SystemsSoftware for Linear Systems
In Gauss-Jordan elimination, matrix is reduced to diagonalrather than triangular formRow combinations are used to annihilate entries above aswell as below diagonalElimination matrix used for given column vector a is of form
Gauss-Jordan elimination requires about n3/2multiplications and similar number of additions, 50% moreexpensive than LU factorization
During elimination phase, same row operations are alsoapplied to right-hand-side vector (or vectors) of system oflinear equations
Once matrix is in diagonal form, components of solutionare computed by dividing each entry of transformedright-hand side by corresponding diagonal entry of matrix
Latter requires only n divisions, but this is not enoughcheaper to offset more costly elimination phase
In principle, solution to linear system is unaffected bydiagonal scaling of matrix and right-hand-side vector
In practice, scaling affects both conditioning of matrix andselection of pivots in Gaussian elimination, which in turnaffect numerical accuracy in finite-precision arithmetic
It is usually best if all entries (or uncertainties in entries) ofmatrix have about same size
Sometimes it may be obvious how to accomplish this bychoice of measurement units for variables, but there is nofoolproof method for doing so in general
Scaling can introduce rounding errors if not done carefully
Michael T. Heath Scientific Computing 72 / 88
Existence, Uniqueness, and ConditioningSolving Linear Systems
Special Types of Linear SystemsSoftware for Linear Systems
Iterative refinement requires double storage, since bothoriginal matrix and its LU factorization are required
Due to cancellation, residual usually must be computedwith higher precision for iterative refinement to producemeaningful improvement
For these reasons, iterative improvement is oftenimpractical to use routinely, but it can still be useful in somecircumstances
For example, iterative refinement can sometimes stabilizeotherwise unstable algorithm
Michael T. Heath Scientific Computing 75 / 88
Existence, Uniqueness, and ConditioningSolving Linear Systems
Special Types of Linear SystemsSoftware for Linear Systems
Symmetric SystemsBanded SystemsIterative Methods
Special Types of Linear Systems
Work and storage can often be saved in solving linearsystem if matrix has special properties
Examples include
Symmetric : A = AT , aij = aji for all i, j
Positive definite : xTAx > 0 for all x 6= 0
Band : aij = 0 for all |i− j| > β, where β is bandwidth of A
Sparse : most entries of A are zero
Michael T. Heath Scientific Computing 76 / 88
Existence, Uniqueness, and ConditioningSolving Linear Systems
Special Types of Linear SystemsSoftware for Linear Systems
Symmetric SystemsBanded SystemsIterative Methods
Symmetric Positive Definite Matrices
If A is symmetric and positive definite, then LUfactorization can be arranged so that U = LT , which givesCholesky factorization
A = LLT
where L is lower triangular with positive diagonal entriesAlgorithm for computing it can be derived by equatingcorresponding entries of A and LLT
In 2× 2 case, for example,[a11 a21a21 a22
]=
[l11 0l21 l22
] [l11 l210 l22
]implies
l11 =√a11, l21 = a21/l11, l22 =
√a22 − l221
Michael T. Heath Scientific Computing 77 / 88
Existence, Uniqueness, and ConditioningSolving Linear Systems
Special Types of Linear SystemsSoftware for Linear Systems
Symmetric SystemsBanded SystemsIterative Methods
Cholesky Factorization
One way to write resulting general algorithm, in whichCholesky factor L overwrites original matrix A, is
for j = 1 to nfor k = 1 to j − 1
for i = j to naij = aij − aik · ajk
endendajj =
√ajj
for k = j + 1 to nakj = akj/ajj
endend
Michael T. Heath Scientific Computing 78 / 88
Existence, Uniqueness, and ConditioningSolving Linear Systems
Special Types of Linear SystemsSoftware for Linear Systems
Symmetric SystemsBanded SystemsIterative Methods
Cholesky Factorization, continued
Features of Cholesky algorithm for symmetric positivedefinite matrices
All n square roots are of positive numbers, so algorithm iswell definedNo pivoting is required to maintain numerical stabilityOnly lower triangle of A is accessed, and hence uppertriangular portion need not be storedOnly n3/6 multiplications and similar number of additionsare required
Thus, Cholesky factorization requires only about half workand half storage compared with LU factorization of generalmatrix by Gaussian elimination, and also avoids need forpivoting
< interactive example >Michael T. Heath Scientific Computing 79 / 88
Existence, Uniqueness, and ConditioningSolving Linear Systems
Special Types of Linear SystemsSoftware for Linear Systems
Symmetric SystemsBanded SystemsIterative Methods
Symmetric Indefinite Systems
For symmetric indefinite A, Cholesky factorization is notapplicable, and some form of pivoting is generally requiredfor numerical stability
Factorization of form
PAP T = LDLT
with L unit lower triangular and D either tridiagonal orblock diagonal with 1× 1 and 2× 2 diagonal blocks, can becomputed stably using symmetric pivoting strategy
In either case, cost is comparable to that of Choleskyfactorization
Michael T. Heath Scientific Computing 80 / 88
Existence, Uniqueness, and ConditioningSolving Linear Systems
Special Types of Linear SystemsSoftware for Linear Systems
Symmetric SystemsBanded SystemsIterative Methods
Band Matrices
Gaussian elimination for band matrices differs little fromgeneral case — only ranges of loops change
Typically matrix is stored in array by diagonals to avoidstoring zero entries
If pivoting is required for numerical stability, bandwidth cangrow (but no more than double)
General purpose solver for arbitrary bandwidth is similar tocode for Gaussian elimination for general matrices
For fixed small bandwidth, band solver can be extremelysimple, especially if pivoting is not required for stability
Michael T. Heath Scientific Computing 81 / 88
Existence, Uniqueness, and ConditioningSolving Linear Systems
Special Types of Linear SystemsSoftware for Linear Systems
Symmetric SystemsBanded SystemsIterative Methods
Tridiagonal Matrices
Consider tridiagonal matrix
A =
b1 c1 0 · · · 0
a2 b2 c2. . .
...
0. . . . . . . . . 0
.... . . an−1 bn−1 cn−1
0 · · · 0 an bn
Gaussian elimination without pivoting reduces tod1 = b1for i = 2 to n
mi = ai/di−1di = bi −mici−1
endMichael T. Heath Scientific Computing 82 / 88
Existence, Uniqueness, and ConditioningSolving Linear Systems
Special Types of Linear SystemsSoftware for Linear Systems
Symmetric SystemsBanded SystemsIterative Methods
Tridiagonal Matrices, continued
LU factorization of A is then given by
L =
1 0 · · · · · · 0
m2 1. . .
...
0. . . . . . . . .
......
. . . mn−1 1 00 · · · 0 mn 1
, U =
d1 c1 0 · · · 0
0 d2 c2. . .
......
. . . . . . . . . 0...
. . . dn−1 cn−10 · · · · · · 0 dn
Michael T. Heath Scientific Computing 83 / 88
Existence, Uniqueness, and ConditioningSolving Linear Systems
Special Types of Linear SystemsSoftware for Linear Systems
Symmetric SystemsBanded SystemsIterative Methods
General Band Matrices
In general, band system of bandwidth β requires O(βn)storage, and its factorization requires O(β2n) work
Compared with full system, savings is substantial if β � n
Michael T. Heath Scientific Computing 84 / 88
Existence, Uniqueness, and ConditioningSolving Linear Systems
Special Types of Linear SystemsSoftware for Linear Systems
Symmetric SystemsBanded SystemsIterative Methods
Iterative Methods for Linear Systems
Gaussian elimination is direct method for solving linearsystem, producing exact solution in finite number of steps(in exact arithmetic)
Iterative methods begin with initial guess for solution andsuccessively improve it until desired accuracy attained
In theory, it might take infinite number of iterations toconverge to exact solution, but in practice iterations areterminated when residual is as small as desired
For some types of problems, iterative methods havesignificant advantages over direct methods
We will study specific iterative methods later when weconsider solution of partial differential equations
Michael T. Heath Scientific Computing 85 / 88
Existence, Uniqueness, and ConditioningSolving Linear Systems
Special Types of Linear SystemsSoftware for Linear Systems
LINPACK and LAPACKBLAS
LINPACK and LAPACK
LINPACK is software package for solving wide variety ofsystems of linear equations, both general dense systemsand special systems, such as symmetric or banded
Solving linear systems of such fundamental importance inscientific computing that LINPACK has become standardbenchmark for comparing performance of computers
LAPACK is more recent replacement for LINPACK featuringhigher performance on modern computer architectures,including some parallel computers
Both LINPACK and LAPACK are available from Netlib
Michael T. Heath Scientific Computing 86 / 88
Existence, Uniqueness, and ConditioningSolving Linear Systems
Special Types of Linear SystemsSoftware for Linear Systems
LINPACK and LAPACKBLAS
Basic Linear Algebra Subprograms
High-level routines in LINPACK and LAPACK are based onlower-level Basic Linear Algebra Subprograms (BLAS)BLAS encapsulate basic operations on vectors andmatrices so they can be optimized for given computerarchitecture while high-level routines that call them remainportableHigher-level BLAS encapsulate matrix-vector andmatrix-matrix operations for better utilization of memoryhierarchies such as cache and virtual memory with pagingGeneric Fortran versions of BLAS are available fromNetlib, and many computer vendors provide customversions optimized for their particular systems
Michael T. Heath Scientific Computing 87 / 88
Existence, Uniqueness, and ConditioningSolving Linear Systems
Special Types of Linear SystemsSoftware for Linear Systems
Level-3 BLAS have more opportunity for data reuse, andhence higher performance, because they perform moreoperations per data item than lower-level BLAS