Numerical Methods of Linear Algebra 1 Introductionkfe.fjfi.cvut.cz/PLAPA/modules/SciComp_B2-linalg.pdfNumerical Methods of Linear Algebra 1 Introduction 1.1 Tasks of Linear Algebra

Numerical Methods of Linear Algebra

1 Introduction

1.1 Tasks of Linear Algebra

1. Solving systems of linear equations A~x = ~b

• Solution of a system with regular (= nonsingular, invertible) squarematrix A of order n×n for one or multiple right-hand sides

• Calculation of the matrix inverse A−1

• Calculation of the determinant

• Solving systems of n equations with m unknowns and singular n×nsystems in some pre-defined sense (one particular solution + thebasis of nullspace, solution with the smallest norm, or in least-squaressense)

2. Calculation of eigenvalues and eigenvectors

• Complete eigenvalue problem

• Partial eigenvalue problem

Standard numerical libraries - LINPACK (special library for linear systems),EISPACK (special library for eigenvalues and eigenvectors), NAG (general),IMSL (general)

1.2 Basic Terms and Notation

Here, by vector we always mean a column vector, superscript T stands forvector or matrix transpose, matrix is denoted using boldface

~x =

x1

x2...xn

= (x1, x2, . . . , xn)T , A =

a11 a12 . . . a1m

a21 a22 . . . a2m...

......

...an1 an2 . . . anm

= (aij) .

1

If not stated otherwise, we suppose real matrices and vectors.

Size of a vector or matrix is measured by its vector normA vector norm must satisfy the following 3 conditions:

a) ‖~x‖ ≥ 0 and ‖~x‖ = 0⇔ ~x = ~0,

b) ‖λ~x‖ = |λ|‖~x‖,

c) ‖~x+ ~y‖ ≤ ‖~x‖+ ‖~y‖.

Examples of vector norms:

• Maximum norm (L∞ norm): ‖x‖I = maxi=1,...,n

|xi|

• L1 norm (taxicab norm, Manhattan norm): ‖x‖II =n∑

i=1

|xi|

• Euclidean norm (L2 norm): ‖x‖III =

√√√√ n∑i=1

x2i

Matrix norm

A vector norm of a matrix is called matrix norm, if for all matrices A,B thefollowing additional condition d) holds:

‖A ·B‖ ≤ ‖A‖ · ‖B‖.

A matrix norm is compatible with a vector norm, if ∀A, ~x holds

‖A~x‖ ≤ ‖A‖ · ‖~x‖

Examples of matrix norms:

• Max row sum: ‖A‖I = maxi=1,...,n

n∑j=1

|aij|

• Max column sum: ‖A‖II = maxj=1,...,n

n∑i=1

|aij|

2

• Frobenius norm (Euclidean norm): ‖A‖III =

√√√√ n∑i=1

n∑j=1

a2ij

Each of the matrix norms above is compatible with the vector norm of thesame name.

1.3 Methods for Solution of Linear Systems

1. Direct methods

2. Iterative methods

3. Gradient methods

3

2 Direct Methods for Solving Linear Systems

Direct methods consist in transformation of the matrix to triangular (or diag-onal) shape (forward run), followed by solution of a system with upper (U)or lower (L) triangular matrix (backsubstitution). Backsubstitution is muchfaster than forward run.

2.1 Solving Systems with Triangular Matrix

Systems with upper triangular matrix U is solved by sequential application ofthe formula in the direction of decreasing index k

xk =1

ukk

bk − n∑j=k+1

ukjxj

.

To calculate any xk we need no more than n inner cycles (1 multiplication +1 addition), thus the number of operations grows with ∼ n2 (more precisely,' 0.5n2 inner cycles).

2.2 Gauss and Gauss-Jordan Elimination

We solve the system of equations A~x = ~b. Suppose that in the first stepa11 6= 0 (which can be always achieved by swapping of the equations). Elementa11, used for modification of equations 2, . . . , n, will be called the pivot elementor simply pivot.

From the ith equation we subtract the 1st equation times the multiplier m(1)i =

−ai1/a11. In the modified system, all elements in the 1st column below thediagonal are now zero. This transformation performed together with the right-hand side corresponds to multiplication of the equation system by matrix

D1 =

1 0 . . . 0

−a21/a11 1 . . . 0...

... . . . ...−an1/a11 0 . . . 1

.

After the first transformation the equation reads D1A~x = D1~b. We denote

A(1) ≡ D1A and ~b(1) ≡ D1.

4

After k − 1 transformations the matrix A(k−1) reads

A(k−1) =

a11 a12 . . . a1,k−1 a1k . . . a1n

0 a(1)22 . . . a

(1)2,k−1 a

(1)2k . . . a

(1)2n

...... . . . ...

......

...

0 0 . . . a(k−2)k−1,k−1 a

(k−2)k−1,k . . . a

(k−2)k−1,n

0 0 . . . 0 a(k−1)kk . . . a

(k−1)k,n

0 0 . . . 0 a(k−1)k+1,k . . . a

(k−1)k+1,n

......

......

......

...

0 0 . . . 0 a(k−1)nk . . . a

(k−1)n,n

,

where the superscript stands for the given element’s number of modifications.If a

(k−1)kk 6= 0, we can select it as a pivot, calculate the multipliers m

(k)i =

−a(k−1)ik /a

(k−1)kk for i = k + 1, . . . , n and modify the corresponding equations.

The pivot in kth step of the transformation is an element that has been modi-fied (by subtraction!) k−1 times before→ loss of precision⇒ pivoting needed.

Direct methods without pivoting are useless for general matrices!

Number of operations

We need to zero 12n(n − 1) elements, each of which costs ≤ n inner cycles.

Total number of inner cycles is ∼ n3 (more precisely ' 1/3 n3), so the com-plexity of this algorithm is n3.

Gauss-Jordan elimination

We modify all off-diagonal elements. The matrix is transformed to identity I,matrix inverse A−1 is directly calculated as a result. More operations areneeded, in particular ' n3 inner cycles.

2.3 Pivoting (Selection of the Pivot Element)

In each step of the forward run we are selecting the pivot.

• Full pivoting: we search the entire yet unprocessed region of the matrix:max |aij|. This is slow.

5

• Partial pivoting: we only search the given column (column pivoting) orrow (row pivoting).

• Implicit pivoting: Faster, improved strategy for column pivoting. Wecompare the size of elements in given column normalized to maximumabsolute values in given row of the original matrix.

With pivoting, direct methods can be used for a majority of matrices.For general big matrices (N > 50), double precision is needed. Even then,difficulties are often encountered for large, ill-conditioned matrices!

Difficulties:

1. Singular matrices

2. Singular matrix produced by loss of accuracy during transformations

3. Loss of accuracy

2.4 LU Method

Any regular matrix A can be decomposed as A = L ·U, where L,U is a lowerleft resp. upper right triangular matrix. The system is then solved by solvingtwo systems with triangular matrices

A~x = ~b ⇒ (LU)~x = ~b ⇒ L (U~x)︸ ︷︷ ︸~y

= ~b ⇒ L~y = ~b, U~x = ~y.

LU decompositiona11 a12 a13 . . . a1n

a21 a22 a23 . . . a2n

a31 a32 a33 . . . a3n...

......

......

an1 an2 an3 . . . ann

=

1 0 0 . . . 0l21 1 0 . . . 0l31 l32 1 . . . 0...

...... . . . ...

ln1 ln2 ln3 . . . 1

u11 u12 u13 . . . u1n

0 u22 u23 . . . u2n

0 0 u33 . . . u3n...

...... . . . ...

0 0 0 . . . unn

6

Matrix multiplication

for i ≤ j : aij = uij +i−1∑k=1

likukj

for i > j : aij = lijujj +

j−1∑k=1

likukj

Crout’s algorithm - sequential calculation e.g. going left by columns and downeach column. First

uij = aij −i−1∑k=1

likukj i = 1, . . . , j

uses l from preceding columns and u from preceding rows, and then

lij =

(aij −

j−1∑k=1

likukj

)/ujj i = j + 1, . . . , n

uses l from preceding columns and u from the part of the current column whichis above the diagonal.

Column pivoting (full pivoting impossible).

Elements aij are used only once, resulting elements of matrices L and U canbe stored the same array.

Properties of the LU method:

• Direct method, the same number of steps as the forward run of Gausselimination.

• Main advantage: the decomposition does not touch (depend on) theright-hand side, thus fast calculation for multiple right-hand sides (e.g. ifthese are dynamically obtained during the calculation)

• The solution can be improved by iteration

2.5 Iterative Improvement of the Solution

We are searching for solution ~x of the linear equation A~x = ~b. First we obtainan inaccurate solution ~x

~x = ~x+ ~δx ⇒ A(~x+ ~δx) = ~b+ ~δb ⇒ A ~δx = ~δb = A~x−~b~x = ~x− ~δx

7

We denote by ~x0 the inaccurate solution from the first step A~x0 ' ~b andperform the iteration

~xi+1 = ~xi + ( ~δx)i, A( ~δx)i = ~b−A~xi.

2.6 Conditioning of Linear System Solution

Due to input and roundoff errors, instead of A~x = ~b we in fact solve

(A + ∆A) (~x+ ∆~x) = ~b+ ∆~b.

• First for the case ∆A = 0:

∆~x = A−1∆~b ⇒ ‖∆~x‖ ≤ ‖A−1‖ · ‖∆~b‖,

A~x = ~b ⇒ ‖~x‖ ≥ ‖~b‖‖A‖

.

Therefore, for the relative error of solution it holds

‖∆~x‖‖~x‖

≤ ‖A‖ · ‖A−1‖ · ‖∆~b‖‖~b‖

.

The value Cp = ‖A‖·‖A−1‖ is called the condition number of the matrix.

• If moreover ∆A 6= 0, then

‖∆~x‖‖~x‖

≤ Cp

‖∆A‖‖A‖ + ‖∆~b‖

‖~b‖

1− Cp‖∆A‖‖A‖

.

For Cp � 1, the system is ill-conditioned, meaning that small input errors orsmall roundoff errors during the calculation lead to a large error of the solution.

2.7 Calculation of the Matrix Inverse and Determinant

Gauss-Jordan elimination calculates the matrix inverse directly. With Gausselimination and LU decomposition we obtain the matrix inverse by solving forn right-hand sides, given by the vectors of standard basis.All three methods are equivalent in accuracy as well as computational cost n3.

8

The determinant should not be computed by classical formulas (uncontrolledgrowth of roundoff error), but rather using the fact that the determinant ofmatrix product equals to the product of determinants. For LU decomposition

det(A) = det(L) · det(U) =n∏

j=1

ujj.

Gauss elimination corresponds to multiplication by transformation matrices andif the row itself is not multiplied (i.e. only multiples of other rows are addedto it), the determinant of each such transformation is Di = 1, and thus thedeterminant of A equals to the product of diagonal elements of the triangularmatrix.If rows are swapped by pivoting, the determinant is multiplied by −1 and wehave to remember the number of such swaps (changes of sign).

2.8 Special Types of Matrices

• A sparse matrix has most of its elements = 0.

To solve systems with sparse matrices, gradient methods are frequentlyused, consisting in minimization of a suitable quadratic function such as‖A~x −~b‖2

III , because for a sparse matrix the number of operations forcalculation of A~x is ∼ n instead of ∼ n2, as it is the case for a densematrix.

• A is a band matrix if aij = 0 for |i− j| > p.For p = 1 we have a tridiagonal matrix, for p = 2 a pentadiagonal matrix.

• Systems with a Tridiagonal Matrix

a1 b1 0 0 . . . 0 0 0c2 a2 b2 0 . . . 0 0 00 c3 a3 b3 . . . 0 0 0...

... . . . . . . . . . ......

......

...... . . . . . . . . . ...

......

......

... . . . . . . . . . ...0 0 0 0 . . . cn−1 an−1 bn−1

0 0 0 0 . . . 0 cn an

·

x1

x2

x3.........

xn−1

xn

=

f1

f2

f3.........

fn−1

fn

The tridiagonal matrix is stored in 3 vectors ~a, ~b, ~c. In practice in mostcases we encounter a tridiagonal matrix where pivoting is not needed

9

(strongly regular matrices).

Solution: We expect the backward run (abbreviated backsubstitution)xk = µkxk+1 + ρk. After insertion we have

ci (µi−1xi + ρi−1) + aixi + bixi+1 = fi,

and further rearrangement yields

xi =−bi

ciµi−1 + aixi+1 +

fi − ciρi−1

ciµi−1 + ai,

so that

µi =−bi

ciµi−1 + ai, ρi =

fi − ciρi−1

ciµi−1 + ai.

Initialization: c1 = 0, bn = 0, {µ0, ρ0, xn+1} arbitrary.

• Block tridiagonal matrix: Ai, Bi, Ci are small matrices ⇒ µi, ρi aresmall matrices

2.9 Systems with No Solution or ∞ Solutions

m equations with n unknowns, linearly dependent n×n systems.

SVD method (singular value decomposition): if A~x = ~b has∞ solutions, SVDwill find the solution with the smallest Euclidean norm and the basis of thenullspace. If the solution does not exist, SVD finds the solution in least-squaressense: the vector ~x minimizing ‖A~x−~b‖III .

SVD: A, U matrices m × n, W, V, I matrices n × n, W diagonal, Iidentity, U, V orthogonal (UT ·U = VT ·V = I).

A = U ·W ·VT ⇒ ~x = V · [diag(1/wj)] ·UT ·~b

If wj = 0 (wj ' 0), we substitute 1wj→ 0 (detects a singular matrix).

10

2.10 Computational Cost of Solving a n×n Linear System byDirect Methods

• Gauss–Jordan elimination:The calculation performs ∼ n3 inner cycles, each cycle contains one mul-tiplication and one addition

• Gauss elimination and LU decomposition:forward run is much more expensive. Both methods need ∼ 1

3 n3 inner

cycles in the forward run. In backsubstitution, Gauss elimination needs∼ 1

2 n(n− 1) cycles. LU decomposition needs two backsubstitutions, butthe forward run is slightly shorter, since the right-hand side is not beingmodified. The total number of operations for Gauss elimination and LUdecomposition are equal. For the calculation of matrix inverse, the costsof Gauss–Jordan elimination, Gauss elimination and LU decompositionare equal.

• A method of order < 3:Strassen proved the existence of a method, where the number of opera-tions ∼ nlog2 7 (log2 7 ' 2.807) and thus with increasing dimension n ofthe matrix it grows slower than in the classical methods (which need ∼ n3

operations). This algorithm however requires a complicated bookkeepingof intermediate results, so for small matrices it is much slower than theclassical methods and thus its advantage takes effect only for matrices oforder n� 1000.

11

3 Gradient Methods

The linear equation A~x = ~b is solved for example by minimization of function

f(~x) =1

2|A~x−~b|2.

In each step, we are searching for λ such that f(~x+ λ~u) is minimized. Thus

λ =−~u∇f|A~u|2

, where ∇f(~x) = AT (A~x−~b).

For sparse matrices, the complexity (computational cost) of multiplying vector~x by matrix A reduces from ∼ n2 to ∼ n.

Note: The most popular gradient method is the conjugate gradient method.

12

4 Iterative Methods for Solving Linear Systems

4.1 Some Types of Matrices

• A n×n matrix A is diagonally dominant, if

|aii| >n∑

j=1,j 6=i

|aij| ∀i = 1, 2, . . . , n.

• A symmetric n×n matrix A is positive definite, if ∀~x 6= ~0 the scalarproduct (~x,A~x) > 0.

4.2 The Iteration Process

The solution ~x of linear equation A~x = ~b is first estimated by vector ~x(0)

(initial guess). The next approximation to exact solution is given by

~x(k+1) = Bk ~x(k) + ~c(k).

The solution ~x has to satisfy

~x = Bk ~x+ ~ck.

This implies that ~x(k+1) − ~x = Bk(~x(k) − ~x) = BkBk−1(~x

(k−1) − ~x) = . . . .Thus the necessary and sufficient condition for convergence of the iterationprocess is

limk→∞

BkBk−1 . . .B0 = 0.

Iterative methods are either stationary, i.e. their matrix Bk is constant (Bk =B), or nonstationary.

4.3 Example of a Nonstationary Iterative Method

An example of a nonstationary iterative method is the ”hand-picked” relaxation.For simplicity we consider matrix A with ones on diagonal (aii = 1).Let after the kth iteration the biggest component of the residuum |A~x − ~b|be the ith component. Therefore we want to zero this ith component by the(k + 1)th iteration

x(k+1)i = bi − ai1x(k)

1 − · · · − aii−1x(k)i−1 − aii+1x

(k)i+1 − · · · − ainx

(k)n .

13

Matrix Bk and vector ~ck are now

Bk =

1. . .

1−ai1 . . . −aii−1 0 −aii+1 . . . −ain

1. . .

1

, ~ck =

0...0bi0...0

.

This method is however not suitable for computers, because in each step itrequires the time-consuming search of equation with the biggest deviation fromthe solution.

4.4 Stationary Iterative Methods

All eigenvalues λ of matrix B (i.e. numbers, for which there exists ~v such thatB~v = λ~v) have to satisfy |λ| < 1.

Theorem: Necessary and sufficient condition for convergence of the methodis, that the spectral radius %(B) satisfies

%(B) = maxi=1,...,n

|λi| < 1.

Theorem: If in some matrix norm

‖B‖ < 1,

then the iteration does converge.

Estimation of error. We want to reach precision ε and thus we iterateuntil

‖~x(k) − ~x‖ ≤ ε.

Expression ‖~x(k) − ~x‖ can be estimated by

‖~x(k) − ~x‖ = ‖~x(k) −A−1~b‖ = ‖A−1(A~x(k) −~b)‖ ≤ ‖A−1‖ ‖A~x(k) −~b‖.

14

4.5 Simple Iteration

(Also referred to as Original Richardson iteration or Fixed point iteration forlinear systems)

The linear equation A~x = ~b is rewritten as

~x = (I−A)~x+~b,

where I is the identity matrix.Simple iteration is then given by iterative formula

~x(k+1) = (I−A)~x(k) +~b.

Denoting B = I−A, we can estimate the accuracy of th kth iteration as

‖~x(k) − ~x‖ ≤ ‖B‖k[‖~x(0)‖+

‖~b‖1− ‖B‖

].

In practice, while simple iteration is sometimes used for nonlinear systems, itsuse for systems of linear equations is very rare.

4.6 Jacobi Iteration

This method assumes that matrix A has nonzero diagonal elements aii 6= 0.Components of the (k + 1)th Jacobi iteration of the solution are

x(k+1)i = −ai1

aiix

(k)1 − · · · −

aii−1

aiix

(k)i−1 −

aii+1

aiix

(k)i+1 − · · · −

ainaiix(k)n +

biaii.

Matrix A can be written in the form

A = D + L + R,

where D is diagonal, L lower triangular and R upper triangular matrix (L andR have zeros on diagonals).

Jacobi iteration can be written as

~x(k+1) = −D−1(L + R)~x(k) + D−1~b.

Theorem: If A is diagonally dominant, then the Jacobi iteration methoddoes converge.

15

Proof: For a diagonally dominant matrix it holds thatn∑

j=1,j 6=i

|aij||aii|

< 1

and thus the max row sum norm of matrix ‖D−1(L + R)‖I < 1.

4.7 Gauss–Seidel Iteration

Gauss–Seidel method is similar to the Jacobi iteration, but unlike it to computecomponents of vector x

(k+1)i it employs the already known components from

(k + 1)th iteration. The iteration is given by relation

x(k+1)i = −ai1

aiix

(k+1)1 − · · · − aii−1

aiix

(k+1)i−1 −

aii+1

aiix

(k)i+1− · · · −

ainaiix(k)n +

biaii,

which can be written in the vector form

~x(k+1) = −(D + L)−1R~x(k) + (D + L)−1~b.

Theorem: The sufficient condition for the convergence of Gauss-Seidel iter-ation is, that at least one of the following statements is true:

1. Matrix A is diagonally dominant

2. Matrix A is positive definite (symmetric with positive eigenvalues).

4.8 Successive Over-relaxation (SOR)

Gauss–Seidel method converges for a wide variety of matrices, however itsconvergence can be very slow in some cases. Let ∆x

(k)i = x

(k+1)i − x(k)

i be thedifference between two consecutive Gauss-Seidel iterations. Then the succes-sive over-relaxation method is given by

x(k+1)i = x

(k)i + ω∆x

(k)i ,

where the relaxation factor ω is between 0 and 2, typically ω ∈ 〈1, 2).Relaxation factor is intended to accelerate the method and its optimal valuecan be calculated as

ωopt =2

1 +√

1− %2(B), B = −(D + L)−1R.

B is the iteration matrix of the Gauss–Seidel method. Gauss–Seidel methodis thus a special case of successive over-relaxation method with ω = 1.

16

5 Calculation of Eigenvalues and Eigenvectors

5.1 Introduction

Let for a number λ exist a vector ~x 6= ~0 such that A~x = λ~x. Then λ is aneigenvalue and ~x is an eigenvector of the matrix A.

Two types of tasks:

1. Complete eigenvalue problem – to find all eigenvalues and if needed alsoall eigenvectors

2. Partial Eigenvalue problem – to find one or a few eigenvalues (typicallythe biggest ones)

The characteristic polynomial of matrix A is the determinant det(A− λI).

Note: If A is a n×n matrix, then its characteristic polynomial is of degree n,and thus has n roots (possibly multiple ones). For each eigenvalue there existsat least one eigenvector. The number l of linearly independent eigenvectors isl≤k, where k is the multiplicity of given eigenvalue.

Note: A defective matrix has < n linearly independent eigenvectors. Forexample:

A =

(1 01 1

), det(A− λI) = (1− λ)2 = 0, and thus λ1,2 =1.

Vector ~x =

(01

)is the only eigenvector of A.

Note: A real matrix can have conjugate complex eigenvalues and eigenvectors.For example:

A =

(1 −11 1

), det(A− λI) = (1− λ)2 + 1 = 0, thus λ1,2 = 1± i.

The eigenvectors are ~x1 =

(1−i

)a ~x2 =

(1i

).

A normal matrix A satisfies ATA = AAT. A normal matrix of order n hasn linearly independent eigenvectors.

17

Note: All eigenvalues of a symmetric matrix (A = AT) are real.

Note: All eigenvalues of a triangular matrix are on its diagonal.

Theorem: Similar matrices A and P−1AP have equal eigenvalues (equalspectra).Proof:

det(P−1AP− λI) = det[P−1(A− λI)P] =

= det(P−1) det(A− λI) det(P) = det(A− λI)

If vector ~x is an eigenvector of matrix A, then vector P−1~x is an eigenvectorof matrix P−1AP.

Theorem: For each matrix there exists a similar matrix in Jordan normalform (Jordan canonical form)

J =

J1 0 . . . 00 J2 0... . . . ...0 0 . . . Js

, where Ji =

λ 0 0 . . . 01 λ 0 00 1 λ 0... . . . ...0 0 0 . . . λ

.

Note: For each normal matrix there exists a similar diagonal matrix.

Note: There is no finite process to transform a matrix to Jordan normal form.

Numerical Methods for Solving the Complete Eignevalue Problem

1. Using a sequence of elementary transformations, we convert the matrixto an approximately diagonal (resp. Jordan normal) form or to an ap-proximately special type (e.g. tridiagonal or Hessenberg matrix).

2. We decompose the matrix A into a product of two matrices A = FL ·FR.Matrix A = FRFL is similar to matrix A.Derivation: FRFL = F−1

L FLFRFL = F−1L AFL.

18

5.2 Jacobi Transformation (Method)

The Jacobi method finds all eigenvalues and eigenvectors of a symmetricmatrix. The task is to successively reduce the size of off-diagonal values.

In each step we find the off-diagonal element with the biggest absolute value|apq| = max

i,j<i|aij| and rotate the axes p, q so, that submatrix [app apq ; aqp aqq]

transforms to a diagonal matrix.

The kth iteration of Jacobi method is

A(k) = TTp,qA

(k−1)Tp,q, where Tp,q =

1 0. . .

cosϕ . . . − sinϕ... 1

...sinϕ . . . cosϕ

. . .

0 1

.

After the kth iteration, the matrix element becomes

a(k)pq = (cos2 ϕ− sin2 ϕ) a(k−1)

pq − cosϕ sinϕ (a(k−1)pp − a(k−1)

qq ).

In order to a(k)pq = 0, the angle ϕ has to satisfy

tg 2ϕ =2apq

app − aqq.

Proof of convergence: In the course of Jacobi transformations, off-diagonalzeros are not permanent. The proof of convergence is based on the conver-gence of the sum of the off-diagonal elements to zero.

Let us denote t(A) =n∑

i,j=1;i6=j

a2ij. Then

t(A(k)) = t(A(k−1))− 2a2pq ∨ a2

pq ≥t(A(k−1))

n(n− 1).

Thus

t(A(k)) ≤(

1− 2

n(n− 1)

)t(A(k−1)) ≤

(1− 2

n(n− 1)

)k

t(A).

19

So the sequence is estimated from above by a geometric sequence with com-mon ratio < 1.

5.3 LU Decomposition for Complete Eigenvalue Problem

This method converges very slowly, the calculation needs a lot of operations.

Matrix A is the zeroth step of the iteration, i.e. A0 ≡ A. In the kth step wedecompose the matrix Ak = LkUk and create matrix Ak+1 = UkLk. Thismatrix is similar to Ak.

If sequence Bk = L0L1 . . .Lk → to a regular matrix, then matrix Ak → to atriangular matrix, which has eigenvalues on the diagonal.

There exist special decompositions of matrices suited for the calculation ofeigenvalues and eigenvectors. A similar process using these decompositionsconverges rapidly.

5.4 Partial Eigenvalue Problem

Let us search for the eigenvalue with the biggest absolute value.We start with choosing an arbitrary vector ~x(0). Further we iterate

~x(k+1) =1

%kA~x(k), where %k = ~eT

1 A~x(k) (resp. %k = ‖A~x(k)‖).

Then it holds that

limk→∞

%k = λ1 ∨ limk→∞

~x(k) = ~x1.

To get the next eigenvalue, we reduce the matrix to order (n − 1). If theeigenvector ~x1 = (u1, . . . , un)T, it holds that

P =

u1 0 . . . 0u2 1 0... . . . ...un 0 . . . 1

, P−1AP =

(λ1 ~qT

~0 B

)

20

and thus we now search for the maximal eigenvalue of matrix B.

Note: The drawback is a gradual loss of accuracy.

Note: The smallest eigenvalue can be found as the biggest eigenvalue ofA−1. To find an eigenvalue in certain region we can shift the matrix, since(A + µI)~x = (λ+ µ)~x.

21