Solving Linear Systems:Direct Methods - UNC Charlotte · Matrix Condition Number Deﬁned for non-singular matrices Deﬁned in terms of the norm of the matrix, jjAjj Can be derived

Solving Linear Systems:Direct Methods

� A number of engineering problems or models can be formulated interms of systems of equations

� Examples: Electrical Circuit Analysis, Radiative Heat Transfer (orequivalently, global diffuse illumination compuation), etc.

� Need a variety of computational techniques, depending on natureof system (sparse systems, for instance, benefit from iterative solu-tions).

� Systems can be very large, thousands of equations is not uncom-mon.

� Large systems require computational/numerical solutions.

ITCS 4133/5133: Intro. to Numerical Methods 1 Solving Linear Systems

Example: Current in an Electrical Circuit


Solving Linear Systems: Direct Methods

� Basic Gaussian Elimination (GE)

� Gaussian Elimination with Row Pivoting

� Tridiagonal Systems (Gauss-Thomas Method)

� Gauss-Jordan Elimination


Linear Systems of Equations

a11X1 +a12X2 + · · · +a1nXn = C1a21X1 +a22X2 + · · · +a2nXn = C2

... ... ... ... ...an1X1 +an2X2 + · · · +annXn = Cn

where aijs are the coefficients, Xjs are the unknowns, and Cis are theknown constants.

Assumptions

⇒ Number of unknowns equal the number of equations

⇒ Equations are linearly independent


Solution of Small Systems of Equations

For small systems such as

a11X1 + a12X2 = C1

a21X1 + a22X2 = C2

can be solved by substitution

⇒ Complex engineering problems could contain thousands of equa-tions.

⇒ Substitution is not a feasible technique for large systems


Linear System Classification

� With solutions

� Without solutions

� With infinite solutions


Gaussian Elimination

� Forward Pass: convert the system of equations into an upper trian-gular matrix,

� Back Substitution: to solve the system.

Permissible Operations

⇒ Changing the order of the equations does not change the solution

⇒ An equation may be scaled (multiplied/divided) by a non-zero con-stant without changing the solution

⇒ May add 2 equations together and replace either one with the newequation, termed (linear combination)


Gaussian Elimination:Matrix Representation

a11 a12 · · · a1n

a21 a22 · · · a2n... ... ... ...an1 an2 · · · ann

X1X2...Xn

=

V1V2...Vn

or,

a11 a12 · · · a1n C1a21 a22 · · · a2n C2... ... ... ... ...an1 an2 · · · ann Cn


Gaussian Elimination:Forward Pass

Results in the upper triangular matrix:1 d12 d13 · · · d1n e10 1 d23 · · · d2n e20 0 1 · · · d3n e3... ... ... ... ... ...0 0 0 · · · 1 en


Gaussian Elimination:Back Substitution

X1 + d12X2 + d13X3 + · · · + d1,n−1Xn−1 + d1nXn = e1

X2 + d23X3 + · · · + d2,n−1Xn−1 + d2nXn = e2

X3 + · · · + d3,n−1Xn−1 + d3nXn = e3

Xn−2 + dn−2,n−1Xn−1 + dn−2,nXn = en−2

Xn−1 + dn−1,nXn = en−1

Xn = en

ImplementationDiagonalize the matrix, resulting in the solution vector in the last column.


Gaussian Elimination:Algorithm

Forward Pass:

⇒ Loop over each row i, making it the pivot row

⇒ Normalize the pivot row, i (pivot element becomes 1)

aij = aij/aii, j = i + 1, i + 2, ..., (n + 1)

Ci = Ci/aii

aii = 1

⇒ Loop over rows (i + 1) to n and reduce elements in each row.

akj = akj − akiaij, j = i, ..., n

Ck = Ck − akiCi, k = (i + 1), ..., n



Forward Pass (Alternate)

� Uses a multiplier, without normalizing pivot element

Input: aij, bi, nColumn: for k = 1 to n− 1

for i = k + 1 to nMultiplier: mik = aik/akk

Row: for j = k + 1 to naij = aij −mikakj

endbi = bi −mikbk

endend

Output: Upper Triangular Matrix



Back Substitution:

⇒ For the last row, Xn = Cn

⇒ For rows (n− 1) through 1,

Xi = Ci −n∑

j=i+1

aijXj, j = (i + 1), ..., n and i = (n− 1), ..., 1


Gaussian Elimination: Algorithm


Example 1:Electrical Circuit


Roundoff Errors and Pivoting

� Truncation and Roundoff errors can occur when coefficients differ byorders of magnitude.

� To minimize errors, diagonal elements should contain the largestcoefficient

� This can be accomplished by swapping rows during Gaussian elim-ination, a process called partial pivoting

� Zero Pivot Element: rearrange equations to prevent zero coefficientsin the pivot element

� Equations must be linearly independent, else we have a singularcoefficient matrix


GE with Pivoting: Motivation

◦ Consider the following example:


GE with Pivoting: Algorithm


GE with Pivoting: Example


Banded Systems

� Systems of equations with banded structure, with non-zero elementson the diagonal and a small number of off-diagonals.

� Such systems can be solved more efficiently.

� Example: Tridiagonal systemd1x1 a1x2 = r1b2x1 d2x2 a2x3 = r2. . .. . .

bn−1xn−2 dn−1xn−1 an−1xn = rn−1bnxn−1 dnxn = rn

� Both memory and computational savings!


Gauss-Thomas Method

� Representation:Vectors d (diagonal), a (above diagonal elements),b (below diagonal elements)

� Example:


Gauss-Thomas Method (contd)


Gauss-Thomas Method:Algorithm


Gauss-Jordan Elimination

� In Gaussian elimination, we create upper triangular matrix, followedby back-substitution

� In Gauss-Jordan elimination, the input matrix is diagonalized in theforward pass, completing the solution

� More efficient when the same system is used to solve multiple right-side vectors.


Matrix Condition Number

� Defined for non-singular matrices

� Defined in terms of the norm of the matrix, ||A||� Can be derived from vector norms:

||x||1 = |x1| + |x2| + ... + |xn|||x||2 = (|x1|2 + |x2|2 + ... + |xn|2)

1/2

||x||∞ = Max|xi|

� Matrix norms are defined as

||A||1 = max(over j) [sum of column j]||A||∞ = max (over i) [sum of row i]||A||2 = ( Max Eigen value of AHA)1/2


Matrix Condition Number (contd)

� Ill-conditioned: Sensitive to small changes in parameters definingthe problem.

� In general, condition number is defined as

κ(A) = ||A||||A−1||

� Spectral condition number:

κ(A) = ||A2||||A−12 ||

� Relative change in solution : depends on product of condition num-ber and relative change in RHS vector (b).


Analysis: Gaussian Elimination

Why does Gaussian Elimination work?

� Equation forming a linear combination of two equations also passesthrough a point common (intersection) to both equations.

◦ Consider

S1 : a0 + a1x1 + . . . + anxn = 0

T1 : a0 + b1x1 + . . . + bnxn = 0

◦ If r = (r1, r2, . . . , rn) is a solution vector it also satisfies m1S1 +m2T1

◦ Elimination produces a system solvable by back substitution.


Analysis: Gaussian Elimination -Computational Complexity

� Measure Flops.

� Row calculations: multiplier calculation (1) + n multiplications and n ad-ditions + RHS update

� Repeat for (n − 1) rows, resulting in (n+1)(n-1) multiplications and divi-sions, n(n-1) additions

k=n−1∑k=1

(n− k + 1)(n− k) =

k=n−1∑k=1

n2 − 2nk + k2 + n− k

=

k=n−1∑k=1

(n2 + n)−k=n−1∑

k=1

(2n + 1)k +

k=n−1∑k=1

k2

=n3

3− n

3


Analysis: GE with Pivoting

� Some form of row scaling can also be used.

� However, ill-conditioned matrices are not helped by scaling.

� Scaling is difficult to incorporate in general purpose packages.

� Computational complexity is O(n3), so preferable to use more efficientmethods.


Analysis: GT Method (Tridiagonal Systems)

� First equation: 2 divisions (scaling the equation)

� Next (n-2) equations: 2 mults, 2 divisions.

� Last equation: 2 mults, 1 division

� Total (Forward Step): 5 + 4(n− 2)

� Back Substitution: (n-1) multiplies.


Solving Linear Systems:Direct Methods - UNC Charlotte · Matrix Condition Number Deﬁned for non-singular matrices Deﬁned in terms of the norm of the matrix, jjAjj Can be derived

Documents