G AUSSIAN E LIMINATION Presented by: Max Le Max Le Justin Lavorgna Data Structures II: Algorithm Design & Analysis Algorithm Design & Analysis.

GGAUSSIANAUSSIAN EELIMINATIONLIMINATION

Presented by:Presented by:

Max LeMax Le

Justin LavorgnaJustin Lavorgna

Data Structures II: Data Structures II:

Algorithm Design & Analysis Algorithm Design & Analysis

Gaussian Gaussian Elimination ItineraryElimination Itinerary

Introduction:Introduction:

Who was Carl Friedrich Gauss ?Who was Carl Friedrich Gauss ?What is Gaussian Elimination ?What is Gaussian Elimination ?Some Basic Terminology ?Some Basic Terminology ?

Main Presentation:Main Presentation:

Backward SubstitutionBackward Substitution method methodLULU method methodCompute Compute DeterminantDeterminant of a matrix of a matrix

Conclusion:Conclusion:

Which Algorithm is more efficient ?Which Algorithm is more efficient ?Which Algorithm is more practical ?Which Algorithm is more practical ?

CARL FRIEDRICH CARL FRIEDRICH GAUSSGAUSS

Who Was He?Who Was He?

Born: April 30Born: April 30thth, 1777 Brunswick, 1777 Brunswick

Died: February 23Died: February 23rdrd,1822 Göttingen,1822 Göttingen

One of the all time great German One of the all time great German Mathematicians. His field of study Mathematicians. His field of study consisted of most every aspect in consisted of most every aspect in mathematics today. mathematics today.

Gauss’s work contributed to a variety of Gauss’s work contributed to a variety of different aspects such as:different aspects such as:

Gaussian Elimination (Linear Algebra) Gaussian Elimination (Linear Algebra) Gaussian Primes & Gauss Sums (Number Gaussian Primes & Gauss Sums (Number

Theory)Theory) Gaussian Distribution (Statistics)Gaussian Distribution (Statistics) Gauss (Electromagnetism)Gauss (Electromagnetism) Gaussian Curvature & Gauss-Bonnet Formula Gaussian Curvature & Gauss-Bonnet Formula

(Differential Geometry)(Differential Geometry) Gaussian Quadrature (Numerical Analysis)Gaussian Quadrature (Numerical Analysis) Gauss’s Identity (Hypergeometric Functions)Gauss’s Identity (Hypergeometric Functions)

GAUSSIAN GAUSSIAN ELIMINATIONELIMINATION

What is it?What is it?

For complex systems of equations where:For complex systems of equations where: The number of equations are equal to The number of equations are equal to nn The number of unknowns are equal to The number of unknowns are equal to nn

We must solve by a process of elimination. We must solve by a process of elimination.

EliminationElimination is implied by reducing is implied by reducing the amount of unknowns and the amount of unknowns and

equations in the system.equations in the system.

1. Subtract multiples of the first 1. Subtract multiples of the first equation from all other equations equation from all other equations

The elimination The elimination process:process:

2.2. The goal is to eliminate the first The goal is to eliminate the first variables in each equation.variables in each equation.

a11 a12 a13a11 a12 a13

a21 a22 a23a21 a22 a23

a31 a32 a33a31 a32 a33

u11 u12 u13u11 u12 u13

0 u22 u230 u22 u23

0 0 u330 0 u33

A=A=

U=U=

TERMINOLOGYTERMINOLOGY

The above conditions are what we call a The above conditions are what we call a linear linear equationequation. If . If h = 0h = 0 then the linear equation is then the linear equation is said to be said to be homogeneoushomogeneous. All linear equations . All linear equations make up a make up a linear systemlinear system while all homogenous while all homogenous linear equations make up a linear equations make up a homogenous homogenous linear systemlinear system..

Equation:Equation: ax + by + cz + dw = hax + by + cz + dw = h where:where: aa,,bb,,cc,,dd,, and and hh are known are known

numbersnumbers

while:while: xx,,yy,,zz,, and and ww are unknown are unknown numbersnumbers

HOW IT WORKSHOW IT WORKS

Given System:Given System:

x + y + z = 0x + y + z = 0

x - 2y + 2z = 4x - 2y + 2z = 4

Equation 1Equation 1

x + 2y - z = 2x + 2y - z = 2 Equation 3Equation 3

Equation 2Equation 2

Make system into a matrix:Make system into a matrix:

x + y + z | 0x + y + z | 0 x - 2y + 2z | 4x - 2y + 2z | 4

Equation 1Equation 1

x + 2y - z | 2x + 2y - z | 2 Equation 3Equation 3

Equation 2Equation 2

Objective:Objective:

Kill ‘ Kill ‘ XX ’ Variable in Equation 2 ’ Variable in Equation 2

x + y + z | 0x + y + z | 0

xx - 2y + 2z | 4 - 2y + 2z | 4

Equation 1Equation 1

x + 2y - z | 2x + 2y - z | 2 Equation 3Equation 3

Equation 2Equation 2

1x +1y + 1z | 01x +1y + 1z | 0

1x - 2y + 2z | 41x - 2y + 2z | 4Multiple: 1Multiple: 11x + 1y - 1z | 01x + 1y - 1z | 0

-- ____________________ - 3y + z | 4- 3y + z | 4 New Equation 2New Equation 2

Equation 1Equation 1

Equation 2Equation 2

MatrixMatrix

1x + 2y - 1z | 2 1x + 2y - 1z | 2 Equation 3Equation 3

Step 1. Observe first column from left of the Step 1. Observe first column from left of the matrix. matrix.

Step 2. If leading term is any number other than 1, Step 2. If leading term is any number other than 1, multiply the row by its reciprocal to obtain a 1. multiply the row by its reciprocal to obtain a 1.

1x1x

Step 3. Subtract first row from all other rows until a Step 3. Subtract first row from all other rows until a zero appears below the leading column.zero appears below the leading column.

‘‘X’ unknown is DEAD…X’ unknown is DEAD…

Equation 1Equation 1

x + y + z | 0x + y + z | 0

xx - 2y + 2z | 4 - 2y + 2z | 4

x + 2y - z | 2x + 2y - z | 2

Objective:Objective:

Kill ‘ Kill ‘ XX ’ Variable in Equation 3 ’ Variable in Equation 3

x + y + z | 0x + y + z | 0

xx - 2y + 2z | 4 - 2y + 2z | 4

Equation 1Equation 1

xx + 2y - z | 2 + 2y - z | 2 Equation 3Equation 3

Equation 2Equation 2

1x +1y + 1z | 01x +1y + 1z | 0

1x + 2y - 1z | 21x + 2y - 1z | 2

Multiple: 1Multiple: 11x + 1y - 1z | 01x + 1y - 1z | 0

-- ____________________ y - 2z | 2y - 2z | 2 New Equation 3New Equation 3

Equation 1Equation 1

Equation 3Equation 3

MatrixMatrix

- 3y + 1z | 4- 3y + 1z | 4 Equation 2Equation 2

Step 4. Observe first column from left of the Step 4. Observe first column from left of the matrix. matrix.

Step 5. If leading term is any number other than 1, Step 5. If leading term is any number other than 1, multiply the row by its reciprocal to obtain a 1. multiply the row by its reciprocal to obtain a 1.

1x1x

Step 6. Subtract first row from all other rows until a Step 6. Subtract first row from all other rows until a zero appears below the leading column.zero appears below the leading column.

‘‘X’ unknown is DEAD…X’ unknown is DEAD…

Equation 1Equation 1 x + y + z | 0x + y + z | 0

xx - 2y + 2z | 4 - 2y + 2z | 4

xx + 2y - z | 2 + 2y - z | 2

Objective:Objective:

Kill ‘ Kill ‘ YY ’ Variable in ’ Variable in EquationEquation 3 3

x + y + z | 0x + y + z | 0

xx - 2y + 2z | 4 - 2y + 2z | 4

Equation 1Equation 1

x + x + 22yy - z | 2 - z | 2 Equation 3Equation 3

Equation 2Equation 2

1y - 2z | 21y - 2z | 2

- 3y + 1z | 4- 3y + 1z | 4Multiple: 3Multiple: 33y - 6z | 63y - 6z | 6

--____________________ - 5z | 10- 5z | 10 New Equation 3New Equation 3

Equation 3Equation 3

Equation 2Equation 2

MatrixMatrix

1x +1y + 1z | 01x +1y + 1z | 0 Equation 1Equation 1

Step 7. Observe first column from left of the Step 7. Observe first column from left of the matrix. matrix.

Step 8. If leading term is any number other than 1, Step 8. If leading term is any number other than 1, multiply the row by its reciprocal to obtain a 1. multiply the row by its reciprocal to obtain a 1.

1y1y

Step 9. Add second row to the third row until a Step 9. Add second row to the third row until a zero appears below the leading column.zero appears below the leading column.

‘‘Y’ unknown is DEAD…Y’ unknown is DEAD…

x + y + z | 0x + y + z | 0

xx - 2y + 2z | 4 - 2y + 2z | 4

x + x + 2y2y - z | 2 - z | 2

Objective:Objective:

Solve for Unknowns using Solve for Unknowns using Backward SubstitutionBackward Substitution

BACKWARDBACKWARD

SUBSTITUTIONSUBSTITUTION

Linear System Solution:Linear System Solution:

Final matrix:Final matrix:

x + y + z | 0x + y + z | 0 - 3y + z | 4- 3y + z | 4

Equation 1Equation 1

- 5z | 10- 5z | 10 Equation 3Equation 3

Equation 2Equation 2 U =U =

Solving for z in equation 3:Solving for z in equation 3:

- 5 10- 5 10 - 5 - 5- 5 - 5___ ______ ___-2-2z =z =

Solving for y in equation 2:Solving for y in equation 2:

- 3 4- 3 4

- 3 - 3- 3 - 3___ _________ ______= -2= -2

y y ( + 2)( + 2)==+ (-2)+ (-2)

Solving for x in equation 1:Solving for x in equation 1:

00

1 11 1___ ____________ _________ = 4= 4 ( + 2 + 2)( + 2 + 2)==+ (-2) + (-2)+ (-2) + (-2)

x x

Objective:Objective:

Solve using LU DecompositionSolve using LU Decomposition

LU DECOMPOSITIONLU DECOMPOSITION

LU DecompositionLU Decomposition

Information on LU MethodInformation on LU MethodExampleExample

LU DecompositionLU Decomposition

Given a system A*x = bGiven a system A*x = b

Solving the system Lp*y=b to solve for y. Solving the system Lp*y=b to solve for y.

We want to find Lp and U matrices such We want to find Lp and U matrices such that Lp*U=Athat Lp*U=A

Then solving the system U*x=y to get xThen solving the system U*x=y to get x

OBJECTIVE:OBJECTIVE:

Given a system A*x = bGiven a system A*x = b

LU DecompositionLU DecompositionConsider the system:Consider the system:

2x + 2y + 1z2x + 2y + 1z 2x + 3y - 2z2x + 3y - 2z

4x + 1y - 2z4x + 1y - 2z

Equation in matrix form for A:Equation in matrix form for A:

2 2 12 2 1

2 3 -22 3 -2

4 1 -24 1 -2

A A

11

11 11

xx

zz yy

XX

==

bb

11

11 11

==

==

==

==

Identity MatrixIdentity Matrix

Identity Matrix is a matrix with all 1 in its Identity Matrix is a matrix with all 1 in its diagonal and zeros elsewherediagonal and zeros elsewhere

11 0 0 0 0

0 0 11 0 0 I =I =

0 0 0 0 11

Find the U matrix and keep track of Elementary Find the U matrix and keep track of Elementary matricesmatrices

First Operation:First Operation:

Elementary operation equivalent:Elementary operation equivalent:

2 2 12 2 1 2 3 -22 3 -2 4 1 -24 1 -2

2 2 12 2 1 0 1 -30 1 -3 4 1 -24 1 -2

Multiply row 1 by -1Multiply row 1 by -1

Add row 1 to row 2Add row 1 to row 2

1 0 01 0 0 0 1 00 1 0 0 0 10 0 1

Multiply row 1 by -1Multiply row 1 by -1

Add row 1 to row 2Add row 1 to row 2

1 0 01 0 0 -1 1 0-1 1 0 0 0 10 0 1

= E1= E1

Find the U matrix and keep track of Elementary Find the U matrix and keep track of Elementary matricesmatrices

Second Operation:Second Operation:

Elementary operation equivalent:Elementary operation equivalent:

2 2 12 2 1 0 1 -30 1 -3 4 1 -24 1 -2

2 2 12 2 1 0 1 -30 1 -3 0 -3 -40 -3 -4

Multiply row 1 by -2Multiply row 1 by -2

Add row 1 to row 3Add row 1 to row 3

1 0 01 0 0 0 1 00 1 0 0 0 10 0 1

Multiply row 1 by -2Multiply row 1 by -2

Add row 1 to row 3Add row 1 to row 3

1 0 01 0 0 0 1 00 1 0 -2 0 1-2 0 1

= E2= E2

Find the U matrix and keep track of Elementary Find the U matrix and keep track of Elementary matricesmatrices

Third Operation:Third Operation:

Elementary operation equivalent:Elementary operation equivalent:

2 2 12 2 1 0 1 -30 1 -3 0 -3 -40 -3 -4

2 2 12 2 1 0 1 -30 1 -3 0 0 -130 0 -13

Multiply row 2 by 3Multiply row 2 by 3

Add row 2 to row 3Add row 2 to row 3

1 0 01 0 0 0 1 00 1 0 0 0 10 0 1

Multiply row 2 by 3Multiply row 2 by 3

Add row 2 to row 3Add row 2 to row 3

1 0 01 0 0 0 1 00 1 0 0 3 10 3 1

= E3= E3

= U= U

1 0 01 0 0 -1 1 0-1 1 0 0 0 10 0 1

E1 =E1 = E2 =E2 = 1 0 01 0 0 0 1 00 1 0 -2 0 1-2 0 1

E2 =E2 = 1 0 01 0 0 0 1 00 1 0 0 3 10 3 1

1 0 01 0 0 -1 1 0-1 1 0 0 0 10 0 1

1 0 01 0 0 0 1 00 1 0 -2 0 1-2 0 1

1 0 01 0 0 0 1 00 1 0 0 3 10 3 1

S =S =

S = E1*E2*E3S = E1*E2*E3

Forming the Lp MatrixForming the Lp Matrix

Forming the Lp MatrixForming the Lp MatrixProduct of the elementary matricesProduct of the elementary matrices

1 0 01 0 0 -1 1 0-1 1 0 -5 3 1-5 3 1

S =S =

Inverse of a S will give LpInverse of a S will give Lp

1 0 01 0 0 1 1 01 1 0 2 -3 12 -3 1

1/S =1/S = = Lp= Lp

OBJECTIVE:OBJECTIVE:We want to find Lp and U We want to find Lp and U

matrices such that Lp*U=Amatrices such that Lp*U=A

LU DecompositionLU Decomposition

We can verify that Lp*U = AWe can verify that Lp*U = A

= A= A 1 0 01 0 0 -1 1 0-1 1 0

2 -3 12 -3 1

2 2 12 2 1 0 1 -30 1 -3 0 0 -130 0 -13

LpLp UU

== 2 2 12 2 1 2 3 -22 3 -2 4 1 -24 1 -2

LU DecompositionLU Decomposition

Recall that Upper Triangular Matrix:Recall that Upper Triangular Matrix:

2 2 12 2 1 0 1 -30 1 -3 0 0 -130 0 -13

U =U =

Lp =Lp =

Recall that Lower Triangular Matrix:Recall that Lower Triangular Matrix:

1 0 01 0 0 1 1 01 1 0 2 -3 12 -3 1

OBJECTIVE:OBJECTIVE:

Solving the system Lp*y=b to Solving the system Lp*y=b to solve for y. solve for y.

Solve Lp*y = bSolve Lp*y = bThe system:The system:

Solve for y1, y2 and y3Solve for y1, y2 and y3

y1 = 1y1 = 1

y1+y2 = 1y1+y2 = 1

2*y1-3*y2 + y3 = 12*y1-3*y2 + y3 = 1

bb

11

11 11

y1y1

y3y3 y2y2

YY ==

==

LpLp

=> => y2 = 0y2 = 0 => => y3 = -1y3 = -1

1 0 01 0 0 1 1 01 1 0 2 -3 12 -3 1

OBJECTIVE:OBJECTIVE:

Then solving the system U*x=y Then solving the system U*x=y to get xto get x

Solve U*x = ySolve U*x = yThe system :The system :

Solve for x1, x2 and x3Solve for x1, x2 and x3

-13*X3 = -1 -13*X3 = -1

X2 – 3*X3 = 1X2 – 3*X3 = 1

2*X1+2*X2 + X3 = 12*X1+2*X2 + X3 = 1

YY

11

-1-1 00

X1X1

X3X3 X2X2

XX ==

== 2 2 12 2 1 0 1 -30 1 -3 0 0 -130 0 -13

UU

=> => X3 = 1/13X3 = 1/13

=> => X2 = 3/13X2 = 3/13

=> => X1 = 3/13X1 = 3/13

Objective:Objective:

Solve for linear system Solve for linear system using Determinant of Matrixusing Determinant of Matrix

DETERMINANT of DETERMINANT of MATRIXMATRIX

Make equations into a matrix:Make equations into a matrix:

1 + 1 + 11 + 1 + 11 - 2 + 21 - 2 + 2 1 + 2 - 11 + 2 - 1

Given Equations:Given Equations:

x + y + z = 0x + y + z = 0x - 2y + 2z = 4x - 2y + 2z = 4

Equation 1Equation 1

x + 2y - z = 2x + 2y - z = 2 Equation 3Equation 3

Equation 2Equation 2

How Determinant works:How Determinant works:

aa1111 a a1212

a a2121 a a2222 ==detdet

aa1111-a-a2222 – a – a1212aa2121

det A =det A = ΣΣ ssjjaa11jj det A det Ajj

aa11 11 det [ adet [ a2222 ] – a ] – a1212 det [ a det [ a2121 ] = a ] = a1111-a-a2222 – a – a1212aa2121

aa11 11 det [ adet [ a2222 ] – a ] – a1212 det [ a det [ a2121 ] = ] =

nn

jj = 1 = 1Formula:Formula:

Matrix:Matrix:

-2-2 22 -1-122

1111 111111 -2 2 -2 2 2 -1 2 -1

== detdet detdet-- detdet++

2 2

-1 -1

1 1

1 1

1 -2 1 -2 1 2 1 2

11 11 11

-2 2 -2 2 2 -1 2 -1detdet detdet-- detdet++

2 2

-1 -1

1 1 1 1

1 -2 1 -2 1 21 2

== 1 ( 2 – 4 ) – 1 ( -1 – 2 ) + 1 (2 – (-2) )1 ( 2 – 4 ) – 1 ( -1 – 2 ) + 1 (2 – (-2) )

== -2 + 3 + 4-2 + 3 + 4

== 55detdet

detdet

detdet

CONCLUSIONCONCLUSION

Time ComplexityTime Complexity Backward substitution O(n^3)Backward substitution O(n^3) LU method O(n^3)LU method O(n^3)

Space ComplexitySpace Complexity Backward substitution O(n^2)Backward substitution O(n^2) LU method O(n^2)LU method O(n^2)

ApplicationApplication

Gaussian Elimination plays a major role Gaussian Elimination plays a major role in solving Linear systems, including:in solving Linear systems, including:- Digital Signal processingDigital Signal processing- Linear system analysisLinear system analysis- Image processing.Image processing.- Etc…Etc…