1 Systems of Linear Equations Direct Methods. 2 Solving Linear Equations Two simultaneous equations – the solution is the intersection of two straight.

1

Systems of Linear Equations

Direct Methods

2

Solving Linear Equations

Two simultaneous equations – the solution is the intersection of two straight lines.

Three simultaneous equations – the solution is the point where three planes intersect.

3

The solution of n simultaneous linear equations with n unknowns

In matrix notation Ax = b, when

nnnnnn

n

b

b

b

x

x

x

aaa

aaa

aaa

2

1

2

1

21

222221

11111

bxA

nnnnnn

nn

nn

bxaxaxa

bxaxaxa

bxaxaxa

2211

22222121

11212111

The linear system is said to be of order n and has a unique solution if det(A) ≠ 0.

4

Two approaches for solving systems of linear equations

A. Direct Methods• Assuming negligible computational round-off errors,

direct methods solve a system of linear equations in a finite number of operations.

• These direct techniques are useful when the number of equations involved is not too large (typically of the order of 40 or fewer equations).

• Examples of direct methods: Gauss Elimination and Gauss-Jordan Elimination.

5

Two approaches for solving systems of linear equations

B. Iterative Methods• These iterative methods are more appropriate

when the number of equations involved is large (typically of the order of 100 or more), or when the matrix is sparse.

• They are more economical in memory requirements.

• Examples of iterative methods: Jacobi method and Gauss Seidel iterative methods

6

Direct Methods

1. Gauss Eliminationa. Naïve Gauss Eliminiation

b. Pivoting Strategies

c. Scaling

2. Gauss-Jordan Elimination

3. LU Decomposition

7

Gauss EliminationLinear systems of equations where the matrix is triangular are particularly easy to solve.

nnnn

nn

nn

nn

bxu

bxuxu

bxuxuxu

bxuxuxuxu

33333

22323222

11313212111

8

Exercise

33

16

86

8423

4

43

432

4321

x

xx

xxx

xxxx

What are the values of x1, x2, x3, and x4?

9

Gauss Elimination Two steps of the Gaussian Elimination

1131321211

'223

'23

'22

"33

"33

"3

'2

1

"33

'23

'22

131211

3

2

1

333231

232221

131211

/)(

/)(

/

axaxabx

axabx

abx

b

b

b

a

aa

aaa

b

b

b

aaa

aaa

aaa

Back Substitution –

To find the values of x's.

Forward Elimination –

To transform the set of equations to an upper triangular system.

10

Naive Gauss EliminationIt is called "Naïve" Gauss Elimination because it does not avoid the problem of division by zero.

"33

"33

'23

'232

'22

1313212111

3333232131

2323222121

1313212111

bxa

bxaxa

bxaxaxa

bxaxaxa

bxaxaxa

bxaxaxa

Forward EliminationPurpose: to reduce the set of equations to an upper triangular system

11

Forward Elimination

'33

'332

'32

'23

'232

'22

1313212111

3333232131

2323222121

1313212111

bxaxa

bxaxa

bxaxaxa

bxaxaxa

bxaxaxa

bxaxaxa

Step 1: Remove x1 from all equations except the first equation (which serves as the pivot equation.)

)3row (

)2row(

)1row (

3333232131

2323222121

1313212111

bxaxaxa

bxaxaxa

bxaxaxa

Let's focus on removing x1 from (row 2) first.

12

Forward Elimination

If a11 ≠ 0, define m21 = a21 / a11.

We can replace (row 2) by {(row 2) – m21 x (row 1)} to yield

1212313212321221221

121231321211121323222121

)()(0

)()(

bmbxamaxamax

bmbxaxaxamxaxaxa

1212'2132123

'23122122

'22 bmbbamaaamaa

Put

We have

)'2row (

)1row ('23

'232

'22

1313212111

bxaxa

bxaxaxa

(row 3') can be obtained in the similar fashion.

Note:a11 is called the pivot coefficient.

13

"33

"33

'23

'232

'22

1313212111

'33

'332

'32

'23

'232

'22

1313212111

bxa

bxaxa

bxaxaxa

bxaxa

bxaxa

bxaxaxa

Forward Elimination

Step 2: Remove x2 from all equations except the first two equations (row 1) and (row 2'). That is

After step 1, we have

)'3row (

)'2row (

)1row (

'33

'332

'32

'23

'232

'22

1313212111

bxaxa

bxaxa

bxaxaxa

(Basically repeat step 1 on the subsystem.)

14

Forward Elimination

If a'22 ≠ 0, define m32 = a'32 / a'22.

We can replace (row 3') by {(row 3') – m32 x (row 2')} to yield

'232

'33

'2332

'332

'232

'33

'232

'22323

'332

'32

)(0

)()(

bmbxamax

bmbxaxamxaxa

'232

'3

"3

'2332

'33

"33 bmbbamaa

Put

We have

"33

"33

'23

'232

'22

1313212111

bxa

bxaxa

bxaxaxa

15

Exercise

What are the values of a'22, a'23, a'32, a'33, b'2, b'3 ?

2

5.1

1

214

321

412

2

5.1

1

24

32

42

321

321

321

xxx

xxx

xxx

'3

'2

'33

'32

'23

'22

'3

'2

3'332

'32

3'232

'22

321 1

0

0

412142

b

b

aa

aa

b

b

xaxa

xaxa

xxx

16

Exercise (continue)

0

1

1

630

15.10

412

0

1

1

63

5.1

42

32

32

321

xx

xx

xxx

"3

"33

"33

"33

32

321

1

1

00

15.10

412

1

1

5.1

42

babxa

xx

xxx

What are the values of a"33 and b"33 ?

17

Exercise (continue)

What are the values of x1, x2, x3?

2

1

1

400

15.10

412

2

1

1

4

5.1

42

3

32

321

x

xx

xxx

18

Forward Elimination

In general, given a linear system of n equations, we repeat the procedure until the system has been transformed into an upper triangular system.

)1()1(

)1(2

)1(23

)1(232

)1(22

)0(1

)0(13

)0(132

)0(121

)0(11

332211

22323222121

11313212111

nnn

nnn

nn

nn

nnnnnnn

nn

nn

bxa

bxaxaxa

bxaxaxaxa

bxaxaxaxa

bxaxaxaxa

bxaxaxaxa

19

Back Substitution

121for )1(

1

)1()1(

)1(

)1(

,,nnia

xabx

a

bx

iii

n

ij ji

iji

i

innn

nn

n

After reducing the system of equations into an upper triangular system,

)1()1(

)2(1

)2(,11

)2(1,1

)1(2

)1(23

)1(232

)1(22

)0(1

)0(13

)0(132

)0(121

)0(11

nnn

nnn

nnn

nnnn

nnn

nn

nn

bxa

bxaxa

bxaxaxa

bxaxaxaxa

we can start solving for x's in backward manner.

20

Pseudocode to performForward Elimination

for k = 1 to n-1

for i = k+1 to n

factor = aik / akk

for j = k+1 to n

aij = aij – factor * akj

bi = bi – factor * bk

m21 = a21 / a11

1212'2 bmbb

132123'23

122122'22

amaa

amaa

21

Pseudocode to performBack Substitution

xn = bn / ann

for i = n-1 downto 1

sum = 0

for j = i+1 to n

sum = sum + aij * xj

xi = (bi – sum) / aii

121for)1(1

)1()1(

,,nnia

xabx

iii

n

ij ji

iji

i

i

)1(

)1(

n

nn

nn

n a

bx

22

Operation Count

The execution time of Gauss Elimination is measured in terms of floating-point operations (FLOPS).

23

for k = 1 to n-1

for i = k+1 to n

factor = aik / akk

for j = k+1 to n

aij = aij – factor * akj


Step k aik / akk aij – factor * akj bi – factor * bk

* and / + and - * and / + and - * and /

1 (n-1) (n-1)2 (n-1)2 (n-1) (n-1)

2 (n-2) (n-2)2 (n-2)2 (n-2) (n-2)

… … … … … …

n-2 2 22 22 2 2

n-1 1 12 12 1 1

Total n(n-1)/2 n(n-1)(2n-1)/6 n(n-1)(2n-1)/6 n(n-1)/2 n(n-1)/2

Operation Count (Forward Elimination)

6

)12)(1(

2

)1(

1

2

1

pppj

ppj

p

j

p

j

Note:

24

Operation Count (Back Substitution)

+ - * /

xn 0 0 0 1

xn-1 (i=n-1)

1 1 1 1

… … … …

x2 (i=2) (n-2) 1 n-2 1

x1 (i=1) n-1 1 n-1 1

Total n(n-1)/2 n-1 n(n-1)/2 n

xn = bn / ann

for i = n-1 downto 1

sum = 0

for j = i+1 to n

sum = sum + aij * xj

xi = (bi – sum) / aii

25

Total Number of FLOPS• Addition/Subtraction

3onsubstitutiBack

12

)1(

neliminatio Forward6

)12)(1(

2

)1( 3nn

nnnnnnn

• Multiplication/Division

33

)13(

onsubstitutiBack 2

)1(

neliminatio Forward2

)1(

2

)1(

6

)12)(1(

32 nnnn

nnnnnnnnnn

26

Total Number of FLOPS

• Total number of FLOPS – O(n3)

• For large n, the operation count for Gauss Elimination is about n3. This means that as n doubled, the cost of solving the linear equations goes up by a factor of 8.

• It is also easy to see that most of the cost of Gauss Elimination is in the elimination step.

27

Direct Methods



c. Scaling


3. LU Decomposition

28

Pitfalls of Elimination Methods

• Division by zero– When the pivot coefficient is equal to or close to zero.

Example:

• Round-off error– Round-off error propagates significantly when we

calculate the solution of a large number of equations.

• Ill-conditioned system– When the determinant is close to zero

562

3764

83

321

321

32

xxx

xxx

xx

29

(a) Singular (no solution)

(b) Singular (infinite solution)

(c) Ill-condition system (Near-singular)

Cases that cause problem

30

Ill-condition SystemWhen there is a small change in one or more of the coefficients in a system,

– Well condition system – results in a similar small change in the solution.

– Ill-condition system – results in large changes in the solution.

4.1021.1

102

21

21

xx

xx

3

4

)1.1(2)2(1)10(1.1)4.10(1

2

)1.1(2)2(1)4.10(2)10(2

1

x

x

An example of an ill-conditioned system and its solution

31

Ill-condition System

)change5%aboutonly(4.108.10)1(2)8(1.1

4.10)1(2)8(05.1

10)1(2)8(

If we change the coefficient a21 from 1.1 to 1.05, we have the equations and

the corresponding solution as

4.10205.1

102

21

21

xx

xx1

8

)05.1(2)2(1)10(1.1)4.10(1

2

)05.1(2)2(1)4.10(2)10(2

1

x

x

Substituting the solutions back to the equations,

A change of 0.05 in one of the coefficients results in a large change in the solution. However, when substituting the solutions back to the equations, we are unable to discover a large deviation.

32

Pivoting Strategies• Pivoting – making the magnitude of the diagonal

elements (pivots) as large as possible by interchanging equations.

• In Gauss Elimination, if pivot coefficient akk(k) = 0 for some

k, the method breaks down.

• Example:

122

22

1

321

321

321

xxx

xxx

xxx

This system is non-singular and has a unique solution

1321 xxx

33

However, after the 1st step in the elimination, we get

x1 + x2 + x3 = 1

x3 = 1

x2 + x3 = 0

Note that a22 = 0, we can't proceed as usual. But by

interchanging Row 2 and Row 3, we have the triangular system

x1 + x2 + x3 = 1

x2 + x3 = 0

x3 = 1

34

Pivoting Strategies

• Partial Pivoting– Interchange equations (rows) only– Choose r, the smallest integer for which

i.e. choose the largest coefficient in the column to become the pivot coefficient.

– interchange rows k and r

nikaa kik

krk )()( max

35

Pivoting Strategies

• Complete Pivoting– Interchange both equations and unknown

elements (pivots)– Choose r and s, the smallest integer for which

i.e. choose the largest coefficient in the sub-matrix to become the pivot coefficient.

– interchange rows k and r, and columns k and s.

njikaa kij

krs ,max )()(

36

Exercise (Partial Pivoting)

Suppose k = 2 (2nd iteration). Which row should be selected as the pivot row?

5

4

3

2

1

100580

2992560

16540

1523300

71521

2

4

3

5

1

1523300

2992560

16540

100580

71521

37

Exercise (Complete Pivoting)

Suppose k = 2 (2nd iteration). Which row should be selected as the pivot row?

5

4

3

2

1

100580

2882560

16540

1523300

719921

5

2

3

4

1

108500

1503320

14560

2625880

729911

38

Example (Without partial pivoting)• Use Gauss elimination to solve

0000.10000.10000.1

0001.20000.30003.0

21

21

xx

xx(Ans: x1 = 1/3, x2 = 2/3)

• Note that the first pivoting element a11 is close to 0.• Forward elimination gives

• Backward substitution gives

66669999

0001.20000.30003.0

2

21

x

xx

0003.0

)3/2(30001.2,

3

212

xx

39

Significant

Figuresx2 x1

x1

|εt| (%)3 0.667 -3.33 1099

4 0.6667 0.0000 100

5 0.66667 0.30000 10

6 0.666667 0.330000 1

7 0.6666667 0.3330000 0.1

The solution is sensitive to the number of significant figures in the computation, since we are subtracting two almost equal numbers.

(Ans: x1 = 1/3, x2 = 2/3)

Error!

40

Example (With partial pivoting)• Reverse order

0001.20000.30003.0

0000.10000.10000.1

21

21

xx

xx(Ans: x1 = 1/3, x2 = 2/3)

• Forward elimination gives

• Backward substitution gives

66669999

0000.10000.10000.1

2

21

x

xx

3

1

1

)3/2(1,

3

212

xx

41

Significant

Figuresx2 x1

x1

|εt| (%)3 0.667 0.333 0.1

4 0.6667 0.3333 0.01

5 0.66667 0.33333 0.001

6 0.666667 0.333333 0.0001

7 0.6666667 0.3333333 0.00001

The solution is less sensitive to the number of significant figures in the computation.

(Ans: x1 = 1/3, x2 = 2/3)

Error!

42

Singularity• Now consider x1 + x2 + x3 = 4

2x1 – x2 – x3 = 7

x1 – 2x2 – 2x3 = 12

x1 + x2 + x3 = 4

– 3x2 – 3x3 = -1

– 3x2 – 3x3 = 8

x1 + x2 + x3 = 4

– 3x2 – 3x3 = -1

0x3 = 9We can detect singularity during the elimination process.

43

Summary of Pivoting

• How to perform partial and complete pivoting

• Reasons for performing pivoting– Avoid division by zero– Minimize the effect of rounding error

44

Direct Methods



c. Scaling


3. LU Decomposition

45

Scaling

Rescale all coefficients in a row to make the largest coefficient equal to 1.

Example:

100002.0000,100000,1002 2121 xxxx

46

Questions

• What is the effect of scaling?

• What do we need to scale?

47

Example of Scaling• Use Gauss Elimination to solve

000,100000,1002

2

21

21

xx

xx

(Ans: x1 = 1.00002, x2 = 0.99998)

48

Example of Scaling (Assume 3 significant digits)

• Solution (a): Without pivoting and scaling

000,100000,1002

2

21

21

xx

xx

x2 = 1.00, x1 = 1.00 (Right)

000,100000,100

2

2

21

x

xx


• Back substitution gives

49

• Solution (b): With partial pivoting but no scaling

2

000,100000,1002

21

21

xx

xx

x2 = 1.00, x1 = 0.00 (Wrong)



000,50000,50

000,100000,1002

2

21

x

xx


50

• Solution (c): With partial pivoting and scaling

100002.0

2

21

21

xx

xx

x2 = 1.00, x1 = 1.00 (Right)



00.1

2

2

21

x

xx


51

Observation• Scaling was useful in determining whether

pivoting was necessary.

• Scaling introduces round-off error. Example,

• It is sometimes suggested that the scaling values should be used to determining the pivoting, but the original coefficient values are retained for the actual elimination and substitution computation.

00000333.000000667.0Scaling

1000,3002

21

21

xx

xx

52

Pseudocode for Gauss Elimination with Partial Pivoting// Assume arrays start with index 1 instead of 0.

// a: Coef. of matrix A; 2-D array// b: Coef. of vector b; 1-D array// n: Dimension of the system of equations// x: Coef. of vector x (to store the solution)// tol: Tolerance; smallest possible scaled pivot allowed.// er: Pass back -1 if matrix is singular. (Reference var.) Gauss(a, b, n, x, tol, er) { Declare s[n] // as an n-element array for storing scaling factors

// si = the largest coef. of row i. for i = 1 to n

si = abs(ai,1) for j = 2 to n

if (abs(ai,j) > si)

si = abs(ai,j)

Eliminate(a, s, n, b, tol, er) // Forward Elimination if (er ≠ -1) // If not singular Substitute(a, n, b, x) // Back Substitution}

53

Pseudocode for Gauss Elimination with Partial Pivoting

// Partial Pivoting

Pivot(a, b, s, n, k) {

p = k // Assume row k is the pivot row

// Find the largest scaled coefficient in column k

big = abs(ak,k / sk)

for i = k+1 to n {

dummy = abs(ai,k / si)

if (dummy > big) {

big = dummy

p = i // Record new pivoting row

}

}

// Next: Swap row p and row k if p != k

// Continue next page

54

Pseudocode for Gauss Elimination with Partial Pivoting if (p != k) {

// Swap row p and row k for j = k to n {

dummy = ap,j

ap,j = ak,j

ak,j = dummy }

// swap bp and bk

dummy = bp

bp = bk

bk = dummy

// swap sp and sk

dummy = sp

sp = bk

sk = dummy }}

55

Pseudocode for Gauss Elimination with Partial PivotingEliminate(a, b, n, x, tol, er) {

for k = 1 to n-1 { Pivot(a, b, s, n, k) // Partial Pivoting

if (abs(ak,k/sk) < tol) { // Check for singularity er = -1; return; }

for i = k+1 to n

factor = ai,k / ak,k

for j = k+1 to n

ai,j = ai,j – factor * ak,j


}

if (abs(an,n/sn) < tol) // Check for singularity er = -1;}

Forward Elimination

56

Pseudocode for Gauss Elimination with Partial Pivoting

// Back Subsitution

Substitute(a, n, b, x) {

xn = bn / an,n

for i = n-1 downto 1 {

sum = 0

for j = i+1 to n

sum = sum + ai,j * xj;

xi = (bi – sum) / ai,i

}

}

57

Question

• Why don't we replace the coefficients by the scaled values in the pseudo code?

• Note: In the pseudo code, the equation are not scaled, but scaled values of the elements are used to determine whether pivoting is to be implemented.

58

Summary of Scaling

• Scaling gives us a "true picture" which element is more dominating so that we can correctly carry out pivoting.

• Why do we need to perform pivoting again?

59

Summary

• Gauss Elimination– Forward Elimination and its complexity– Back substitution and its complexity

• Effect of pivoting and scaling

1 Systems of Linear Equations Direct Methods. 2 Solving Linear Equations Two simultaneous equations – the solution is the intersection of two straight.

Documents