MATHEMATICS OF COMPUTATION, VOLUME 31, NUMBER 137 JANUARY 1977, PAGES 148-162 An Iterative Solution Method for Linear Systems of Which the Coefficient Matrix is a Symmetric M-Matrix By J. A. Meijerink and H. A. van der Vorst Abstract. A particular class of regular splittings of not necessarily symmetric M-matrices is proposed. If the matrix is symmetric, this splitting is combined with the conjugate-gradient method to provide a fast iterative solution algorithm. Compar- isons have been made with other well-known methods. In all test problems the new combination was faster than the other methods. 1. Introduction. A time-consuming part of the numerical solution of partial dif- ferential equations using discretization methods is often the calculation of the solution of large sets of linear equations: (1.1) Ax = b, where A is usually a sparse matrix. In this paper, iterative solution methods will be presented which are restricted to equations where A is a symmetric A/-matrix,* although symmetry is not required in most of the theorems. This type of matrix is often generated, e.g., by discretization of elliptic and parabolic differential equations. For an extensive study on this subject, see [7]. Most of the iterative methods are based on the following idea: If AT is an arbi- trary nonsingular matrix, then A = K - R represents a splitting of the matrix A and associated with this splitting is an iterative method (1.2) Kxn + 1=(K-A)xn+b=Rxn+b or (1.3) Xn+i =Xn +K-1ib-Axn) = xn+Axn. The more K resembles A, the faster the method will converge. On the other hand, we have to solve the equation (1.4) Kùxn=b-Axn, during every iteration so K has to be such that only few calculations and not too much memory storage are required to achieve this. For instance, the choice of K to be the diagonal matrix equal to the diagonal of A leads to the Jacobi iterative method, while the Gauss-Seidel iterative method arises by choosing K to be the lower triangular part of A. For both these choices the solution of (1.4) is straightforward. Received May 5, 1975; revised January 16, 1976. AMS (MOS) subject classifications (1970). Primary 65F10, 65N20. * A matrix A = (a,-.-) is an M-matrix if a,-- < 0 for i ¥=j, A is nonsingular and A > 0. Copyright l1)??. Ameritan Mathematical Society 148 License or copyright restrictions may apply to redistribution; see https://www.ams.org/journal-terms-of-use
15
Embed
An Iterative Solution Method for Linear Systems of Which the
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
JANUARY 1977, PAGES 148-162
Coefficient Matrix is a Symmetric M-Matrix
By J. A. Meijerink and H. A. van der Vorst
Abstract. A particular class of regular splittings of not
necessarily symmetric
M-matrices is proposed. If the matrix is symmetric, this splitting
is combined with
the conjugate-gradient method to provide a fast iterative solution
algorithm. Compar-
isons have been made with other well-known methods. In all test
problems the new
combination was faster than the other methods.
1. Introduction. A time-consuming part of the numerical solution of
partial dif-
ferential equations using discretization methods is often the
calculation of the solution
of large sets of linear equations:
(1.1) Ax = b,
where A is usually a sparse matrix.
In this paper, iterative solution methods will be presented which
are restricted to
equations where A is a symmetric A/-matrix,* although symmetry is
not required in
most of the theorems. This type of matrix is often generated, e.g.,
by discretization
of elliptic and parabolic differential equations. For an extensive
study on this subject,
see [7].
Most of the iterative methods are based on the following idea: If
AT is an arbi-
trary nonsingular matrix, then A = K - R represents a splitting of
the matrix A and
associated with this splitting is an iterative method
(1.2) Kxn + 1=(K-A)xn+b=Rxn+b
or
(1.3) Xn+i =Xn +K-1ib-Axn) = xn+Axn.
The more K resembles A, the faster the method will converge. On the
other hand, we
have to solve the equation
(1.4) Kùxn=b-Axn,
during every iteration so K has to be such that only few
calculations and not too much
memory storage are required to achieve this. For instance, the
choice of K to be the
diagonal matrix equal to the diagonal of A leads to the Jacobi
iterative method, while
the Gauss-Seidel iterative method arises by choosing K to be the
lower triangular part
of A. For both these choices the solution of (1.4) is
straightforward.
Received May 5, 1975; revised January 16, 1976.
AMS (MOS) subject classifications (1970). Primary 65F10,
65N20.
* A matrix A = (a,-.-) is an M-matrix if a,-- < 0 for i ¥= j, A
is nonsingular and A > 0.
Copyright l1)??. Ameritan Mathematical Society
148
AN ITERATIVE SOLUTION METHOD FOR LINEAR SYSTEMS 149
For other choices of À', the direct solution of (1.4) is equivalent
to the £¿/-de-
composition of K and the solution of the equations
(1.5) Lyn=b-Axn
(1.6) UAxn=yn.
The choice of K most ideal for the iteration process is A, since
only one iteration is
needed, but the £ ¿/-decomposition of A requires a large number of
calculations and
much memory storage, since £ and U are usually considerably less
sparse than A. This
suggests we look for matrices K = LU which resemble A, with £ and U
almost as sparse
as A.
In [6], Stone presents a method that is based on this idea.
In Section 2 we shall introduce another class of such matrices K.
We shall call
this class "Incomplete £ ¿/-decompositions of A". It will be proven
that this class is not
empty and that the splitting A = K - R is a regular splitting**
which implies that the
iterative method (1.2) will converge.
In Section 3 we shall discuss the stability of incomplete £
¿/-decompositions.
In Section 4 a successful combination with the conjugate-gradient
method will be
described for symmetric matrices.
In Section 5 two special types of incomplete decompositions are
proposed, while
in Section 6 results are presented, discussed and compared with
results of other familiar
iterative methods.
2. Incomplete LU-Decompositions.
Notation. A lower triangular n x n matrix is denoted by £ = (/,-•),
so l¡- = 0 if
i < /, and an upper triangular n x n matrix by U = (",-.).
As mentioned in the introduction, a matrix K approximating A has to
be con-
structed such that the £ and U belonging to K are sparse. This can
be realized by
making an £ ¿/-decomposition of A, during which elements are
neglected in the £ and
U matrices in appropriate places. That is the reason that we shall
call K = LU an
"incomplete £ ¿/-decomposition of A".
Theorem 2.3 guarantees the existence of incomplete £
¿/-decompositions. In
these £ and U, zeros may occur in arbitrary off-diagonal places,
which can be chosen
in advance. These places (/', /) will be given by the set
P c P„ = {(i, /) \t *j, \<i <n, 1</ < «}.
Note that Pn contains all pairs of indices of off-diagonal matrix
entries. The various
algorithms arise by choosing these places. Some choices for special
matrices will be
described in more detail in Section 5.
In the proof of Theorem 2.3 the incomplete £ ¿/-decomposition is
obtained via
Gauss elimination. The proof requires two theorems about operations
on Ai-matrices.
"For n xn real matrices A, K and R, A = K - R is a regular
splitting of the matrix A if K
is nonsingular, K~ > 0 and R > 0.
License or copyright restrictions may apply to redistribution; see
https://www.ams.org/journal-terms-of-use
150 J. A. MEIJERINK AND H. A. VAN DER VORST
The first theorem shows that the matrix that arises from an
Af-matrix after one ehmina-
tion step is again an A/-matrix.
Theorem 2.1 (KyFan [2, p. 44]). If A = (a¡¡) is an M-matrix, then
A1 = (a?) is
so, where A1 is the matrix that arises by eliminating the first
column of A using the
first row.
The second theorem will be used to be able to omit appropriate
nondiagonal ele-
ments during the construction of the incomplete £ ¿/-decomposition
of A.
Theorem 2.2. Let A = (a/;) be an n x n M-matrix and let the
elements of
B = (b¡¡) satisfy the relations
flf/<6//<0 fori±j
and 0 < a¡¡ < b¡¡. Then B is also an M-matrix.
Proof. The proof is essentially the same as a proof given by Varga
[7, Proof of
Theorem 3.12].
Let DA be a diagonal matrix whose diagonal entries are given by du
= l/aw, and
let DB be defined in the same way. Let QA and QB be defined
by
QA=I-DAA and QB = I - DBB.
Since A is an A/-matrix, the spectral radius PÍQA) of QA
satisfies
PiQA) < 1 (see [?. Theorem 3.10]),
and as from the assumptions it follows that 0 < QB < QA, we
have
PiQB) < PÍQa) < ! t7> Theorem 2.8].
From [7, Theorem 3.10] it follows that B is an A/-matrix. D
Theorem 2.3. If A = (a,-) is an n x n M-matrix, then there exists
for every
P C Pn a lower triangular matrix £ = (/„), with unit diagonal (/((-
= 1), an upper triangu-
lar matrix U = (u¡) and a matrix R = (#•«) with
hj = 0 if(i,f)GP, uif = 0 if(i,f)GP, r„ = 0 ifii,j)$P,
such that the splitting A = LU - R is regular. The factors L and U
are unique.
Proof. The proof of this theorem also gives a way to construct £
and U. The
construction process consists of n - 1 stages. The fcth stage
consists of subtracting
from the current coefficient matrix the elements with indices (fc,
j) and (/', k) £ P and
then reducing the matrix in the usual way. So let us define the
matrices
Ak = (a*), Xk = (aff), Lk = (/*) and Rk = (,*)
by the relations:
Ak = LkAk )
Here the matrix Rk is defined by
r*,. = -a*ri, i{ (kJ) E P,
•* = - akk~1, if (i, k) GP and all other /y are equal to
zero.
License or copyright restrictions may apply to redistribution; see
https://www.ams.org/journal-terms-of-use
AN ITERATIVE SOLUTION METHOD FOR LINEAR SYSTEMS 151
Lk is equal to the unit matrix, except for the fcth column, which
written row-wise, is
as follows
[ak ak ak "I0 0 0 j _a*+ifc _±*±i* _^L U, U, V, . . . , I, £,fc ,
^fe , . . . , ~fc .
flfcfc afck "kfcj
From this it can easily be seen that Ak is the matrix that arises
from Ak by elimin-
ating the lowermost n - k elements in the kth column using the kth
row.
A0 = A is an M-matrix, so R1 > 0. From Theorem 2.2 it follows
that A1 is an
M-matrix. Therefore L1 > 0 and applying Theorem 2.1 we see that
A1 is an M-matrix.
Continuing in this manner, we can prove that
Ak is an M-matrix
Rk>0
LkRm = Rm if k < m<
An-\ =Ln-l£n-l =l»-1^/j-2 + Ln-lRn-l
= L"-1L"-2A"-2 + £"-1Ä"-1 = ••• = £"-1£"-2 ••• LlA°
+ L„-lL„-2 ... LlRl +Ln-lLn-2 ... ¿2^2 + ... + L»-lRn-l
By combining these equations, we find
An-\ =Ln-lLn-2 ... ¿1^ + ^1 + p} + ... +/?""1).
Let us now define U = An~l,L = (£"~1£"-2 ••• L1)'1 and/? = R1 + R2+
••• +
R"-1 then LU = A + R, (£¿/)_1 > 0 and R > 0, so the splitting
A = LU - R is reg-
ular. The uniqueness of the factors £ and ¿/ follows from equating
the elements of A
and LU for (i, j) £ P, and from the fact that £ has a unit
diagonal. D
For the case where A is in addition symmetric and thus positive
definite Theorem
2.4 gives a symmetric variant of the preceding theorem. This states
that a symmetric
incomplete £ ¿/-decomposition can be achieved which contains zeros
in a symmetric
pattern of places indicated in advance.
Theorem 2.4. If A is a symmetric M-matrix, there exists for each P
C Pn
having the property that (i, j) £ P implies (j, i) GP, a uniquely
defined lower triangu-
lar matrix L and a symmetric nonnegative matrix R, with l„ = 0 if
(i, j) £ P and
r¡j = 0 if (i, j) £/*, such that the splitting A = LLT - R is a
regular splitting.
Proof. This theorem follows directly from the fact that Choleski
decomposition
is equivalent to gaussian elimination except for a diagonal matrix.
This extra diagonal
matrix does not affect places which contain zeros. D
From the previous theorems the convergence of the method defined in
(1.2) —
(1.4) follows immediately, this is formulated in Theorem 2.5.
Theorem 2.5. If A, L, U and R are defined as in Theorem 2.3, the
iterative
method
License or copyright restrictions may apply to redistribution; see
https://www.ams.org/journal-terms-of-use
152 J. A. MEIJERINK AND H. A. VAN DER VORST
will converge to the solution of Ax = b for every choice
ofxQ.
Proof. This is an immediate consequence of Theorem 3.13 of Varga
[7]. D
By properly choosing P C Pn, we obtain a number of well-known
methods: P — Pn
results in the point Jacobi method, and P = {(/', j) \i </}
results in the point Gauss-
Seidel method. Also, line and block variants of these two methods
can be obtained by
a proper choice of P. So Jacobi and Gauss-Seidel methods are a
subclass of methods
based on incomplete £ ¿/-decompositions, which are themselves a
subclass of methods
based on regular splittings.
3. Numerical Stability. The question which now arises is whether
the construc-
tion of an incomplete £ ¿/-decomposition is stable. In order to
answer this question,
we need Theorem 3.1. This theorem indicates the effect on the
decomposition process
of replacing off-diagonal elements in the matrix by nonpositive
elements that are
smaller in absolute value, as well as the effect of replacing
diagonal elements by larger
ones.
Theorem 3.1. Let A = (af.) and B = (6f.) be defined as in Theorem
2.2. Let
A1 and Bl be the matrices that arise from A and B by eliminating
the first column
using the first row.
and B1 is an M-matrix.
Proof.
4 = fl./-^ai/^/-^/ = H- fori*U*l-
From these relations it follows that b)¡ < 0, for i ¥= f. Now A1
is an M-matrix (Theo-
rem 2.1), and aj¡ > 0 is a property of M-matrices [7, Theorem
3.10], so from Theorem
2.2 B1 is also an M-matrix. D
The following theorem states that the incomplete £ ¿/-decomposition
process is
more stable than the complete £ ¿/-decomposition process (without
partial pivoting).
Theorem 3.2. If A is an M-matrix, then the construction of an
incomplete
LU-decomposition is at least as stable as the construction of a
complete decomposition
A = LU without any pivoting.
Proof. Let A be the matrix that is obtained by setting some
off-diagonal elements
of A to zero in the first column and in the first row (compare
Theorem 2.3). Let Ll
be the gaussian elimination matrix for the first elimination step
on A, and L1 be the
same for A.
Then it is obvious that the elements of £ t are not larger in
absolute value than
the elements of £j. From Theorem 2.1 and Theorem 2.2 it follows
that A t and Ax
are M-matrices, while Theorem 3.1 states that A1 > A1. From
repeated application of
Theorem 3.1 it follows that the elements of the gaussian
elimination matrices £k, in
each stage of the incomplete £ ¿/-decomposition process, are not
larger in absolute value
than the elements of the gaussian elimination matrices Lk that
arise in the complete de-
composition process. This gives the desired result (see [8], [9]).
D
License or copyright restrictions may apply to redistribution; see
https://www.ams.org/journal-terms-of-use
AN ITERATIVE SOLUTION METHOD FOR LINEAR SYSTEMS 153
Corollary 1. If A is a symmetric M-matrix, then the construction of
an in-
complete LLT-decomposition is at least as stable as Choleski's
process.
Note. It is well known that in general £ ¿/-decomposition without
pivoting is not
a very satisfactory process. Therefore, we consider the practical
situation, where A is
a diagonally dominant M-matrix. It is easy to see that gaussian
elimination preserves
the diagonal dominance of the matrix. Therefore gaussian
elimination, in this case, is
identical with Crout £ ¿/-decomposition with partial pivoting, the
latter being fairly
stable.
4. An Accelerated Method for Symmetric Systems of Equations. If the
n x n
matrix A of the linear system of equations Ax = b is an M-matrix,
and LU is an in-
complete decomposition of A, the iterative process, defined in
Theorem 2.5, generates
a sequence {xn}n>0 that converges to x. From simple analysis it
follows that
(4.1) *' = X° ~ (l)l(Ll/)~lA] (*o -*) + ^MU)-lA]\xQ -*)
--+i-l)i(ii)liLU)-1A]i(x0-x).
If A is a symmetric M-matrix (hence, positive definite), an
upperbound for the error
II*,- - x\\A = (A(xi - x), xi - x) is given by
(4.2) H*,. - x \\A < [max{| 1 - Xmin |, 11 - Xmax Q]2i \\xQ - x
»2a>
with
Xmax the largest eigenvalue.
For this special case of A a symmetric M-matrix, symmetric
incomplete ££r-decompo-
sition can be combined with the method of conjugate gradients. This
leads to a sim-
ilar scheme as (4.1), which is known to be faster. For the
discussion of this combined
method and its main properties, results from [1] and [3] are used.
Let M be a square
nonsingular n x n matrix and let H and K be positive definite
symmetric n x n ma-
trices, N = M*HM and T = KN, then a conjugate-gradient method to
solve the equa-
tion Mx = b is defined by
x0 an arbitrary initial approximation to x,
r0=b-Mx0, g0= M*Hr0, p0 = Kg0,
«,- = (gt, Pi)HpP Np¿) = Cjr,, Kgt)l(pt, Npt)
*«+l =xi + aipi
gi+l =M*Hri+1 =gi-aiNpi
ßi = - iNPi> Kgi+i)liPi> *Pi) = igi+v Kg,+ i)l(gt, Kg,)
Pi+i =Kgi+1 +ß,Pi
1,2,.
License or copyright restrictions may apply to redistribution; see
https://www.ams.org/journal-terms-of-use
154 J. A. MEIJERINK AND H. A. VAN DER VORST
(1) the sequence {x¡}¡>0 converges to the solution x within n
iterations.
(2) the conjugate-gradient method minimizes \\x¡ - x \\N for all i,
among all algo-
rithms of the form
(4.4) xt =x0+Pi_1(T)T(x-x0),
(3)
(4.6)
where c = Ksxi^Kini7^-
From the choice M = A, H = A~l and K = (££T) ! which results in TV
= A
and T = (LLT)~1A, it follows that the iterative method defined in
Theorem 2.5 is of
the form (4.4) and hence from property (2) it follows that the
combined method will
converge at least as fast. Also the two upper bounds for the errors
show a substantial
difference. For this choice the iteration scheme can be written
as:
x0 is an arbitrary initial approximation to x,
rQ = b -Ax0, p0 = (LLT)-lr0,
_(rt,[LLT]-lrt) Ui (Pi,APi)
xi+l =xi + aipi
pi+l = [LLT]-iri+l+ßiPi
Remark. The inequality (4.5) does not take advantage of the
distribution of the
eigenvalues of T, while the conjugate-gradient method does so.
Therefore the upper-
bound (4.5) might be pessimistic. This happens especially when most
of the eigen-
values of T are clustered in small intervals compared to the
interval ^¡„(IT), \¡ax(T)].
5. Two Applications of Incomplete Decomposition. For a special type
of ma-
trix, two different incomplete decompositions will be introduced in
this section. The
matrix-equation arises from five-point discrete approximations to
the second-order
selfadjoint elliptic partial differential equation:
7k 7\ A ?\
(5-1) ~ to A(pC' •V) ax"(*' y) ~ Y ^X' ^ ïy U(X' + C{?C' y^U(pC' y)
= D{pc- •y)
with A(x, y), B(x, y) > 0, C(x, y) > 0, and (x, y) £ R, where
R is a square region,
and with suitable boundary conditions on R. The resulting symmetric
positive definite
diagonally dominant nth order matrix A = (a^) is schematically
shown in Figure 1.
Places of zero entries are given by
(5.2) P* = {(i, /)|li-/l#0, l.in},
License or copyright restrictions may apply to redistribution; see
https://www.ams.org/journal-terms-of-use
AN ITERATIVE SOLUTION METHOD FOR LINEAR SYSTEMS 155
form of matrix a
Figure 1
where m is the half bandwidth of the matrix. For the derivation of
such linear systems
see references [6] and [7].
The elements of the diagonal of A are denoted by a¡, the
upper-diagonal ele-
ments are denoted by b¡ and the elements of the mth upper diagonal
are denoted by
c¡, where i is the index of the row of A in which the respective
elements occur. Theo-
rem 2.4 guarantees the existence of incomplete symmetric
decompositions for A. Our
first application considers the incomplete decomposition that
arises in the decomposi-
tion process when all elements are ignored in those places where A
has zero entries.
This variant is characterized by P*.
In the following it will be convenient to write the incomplete
decomposition in
the form: LDLT, where D is a diagonal matrix. If the elements of D
are denoted by
d¡ and the elements of LT are denoted analogous to A by aí¡, b¡ and
cj, then the fol-
lowing recurrent relations hold for these elements:
&; = *>„ ^. = c,.,
(5.3) ~ ~ ' — ' ' ' ' ' ' a¡ = dj-1 = a,. -bf_xdi_x -
^hm^i-m^
where elements that are not defined should be replaced by
zeros.
License or copyright restrictions may apply to redistribution; see
https://www.ams.org/journal-terms-of-use
156 J. A. MEIJERINK AND H. A. VAN DER VORST
Note that, once we have stored A, we need only compute and store
the diagonal
matrix D. Note also that when this process is programmed for a
two-level store com-
puter, for very large matrices, the diagonal D can be constructed
by taking successive
parts of A into fast core. Equation (1.4) reduces to
LDLTAx( = b- Ax¡,
and it is well known that Ax,- can be determined by solving
successively
Ly, = b - Ax¡, Dz¡ = y,
and
LTAxi = z,..
This first variant of incomplete decompositions for the type of
matrices described here,
in combination with the conjugate gradient method, is referred to
by ICCG(0) further
on (i.e., Jncomplete Choleski & Conjugate Gradients, with 0
extra diagonals).
The second variant is characterized by
P3 = W, i) | if "/I * 0,1, 2, m -2,m - \,m}.
In this case, the symmetric decomposition is written as LLT.
The elements of LT are denoted as follows, where / is counted
row-wise again:
With the above notations, the elements of LT can be recursively
computed as follows:
a, = Va,. - b2_x - df_2 - 7f_m + 2 -ff_m + 1 - clm\
(5 4) *' = (ft'' ~ d'~ lb,~ ' " ^i~m + i-m + l " fi-™+27i-m +2)/«/
'
d¡ = - ci_m+2ei_m+2furí, 7, = - (ci_2di_2 + /,_,&,_
1)/«',,
fi = -ci_lbi_Jai, c^cja, for i = 1, 2, . . . , n.
License or copyright restrictions may apply to redistribution; see
https://www.ams.org/journal-terms-of-use
AN ITERATIVE SOLUTION METHOD FOR LINEAR SYSTEMS 157
Elements not defined should be replaced by zeros. It should be
remarked that it is
also possible to avoid the square root computations by a slight
modification of the
Eqs. (5.4).
This second variant, in combination with the conjugate-gradient
method, will be
referred to as ICCG(3), as it has three more diagonals at each side
than the original
matrix A.
6. Numerical Results and Comparison with Other Methods. In this
section, re-
sults are presented and compared with results of other iterative
methods, for two spe-
cial cases of Eq. (5.1). However, it should be mentioned that
similar results have been
obtained in other cases.
We first discuss briefly the different iterative methods. TV will
denote the order
of the matrix A.
ICCG(O). This variant of incomplete decomposition is discussed in
Section 5. Each
iteration of ICCG(O) needs — 16N multiplications.
ICCG(3). For a discussion see Section 5. Each ICCG(3)-iteration
needs — 22N mul-
tiplications.
if intermediate results are stored.
Conjugate Gradients. See Section 4, where for this case LLT should
be replaced by
the identity matrix. Each iteration needs — ION multiplications. If
the
matrix has 'property A', this can be reduced by a factor 2
[5].
SIP. The Strongly Implicit Procedure has been described in detail
by Stone [6].
Each iteration needs — 22N multiplications.
In interpreting the results of the various methods, it should be
noted that any initial
work, such as the work necessary for the estimation of
iteration-parameters or the
computational work for the decompositions of the ICCG methods, was
neglected. This
did not affect the conclusions seriously, because this initial work
will in general be
negligible compared to the computational work needed for even a
small number of
iterations. The methods are compared on the basis of computational
work, which was
measured, rather arbitrarily, using the total number of
multiplications.
The number of multiplications needed for each iteration is
mentioned above. In
the figures, the number of multiplications required for one single
iteration of ICCG(3),
i.e. 22N multiplications, was chosen as the unit for the
computational work.
Example 1. Equation (5.1) is considered over the square region 0
<x < 1,
0 <y < 1, with A(x, y) = B(x, y) = 1, C(x, y) = Dix, y) = 0
and the boundary con-
ditions du/dx = 0 for x = 0 and x = 1, du/dy = 0 for y = 1 and u =
1 for y = 0. A
uniform rectangular mesh was chosen, with Ax = 1/31 and Ay = 1/31,
which resulted
in a linear system of 992 equations. The solution of (5.1) is known
to be u(x, y) = 1,
and as initial starting vector for the iterative scheme described
in Section 4, a vector
was chosen with all entries random between 0 and 2. This was done
to prevent fast
convergence by coincident. The results are plotted in Figure
2.
Example 2. In order to illustrate the power of the ICCG methods,
also for more
License or copyright restrictions may apply to redistribution; see
https://www.ams.org/journal-terms-of-use
158 J. A. MEIJERINK AND H. A. VAN DER VORST
Residual
_L
1 ICCG(O) 2 ICCG (3) 3 SIP 4 SLOR 5 CONJ. GR.
J_ 40
-L 60
computational work,
Figure 2. Results for Example 1
practical nonuniform situations, a problem suggested by Varga [7,
Appendix B] was
considered. Equation (5.1) holds on R, where R is the square region
0 <x, y < 2.1,
as shown below.
2.0 2.1
On the boundary of R the boundary conditions are du/dn = 0. Further
D(x, y) = 0
over R and the functions A, B and C are defined by
License or copyright restrictions may apply to redistribution; see
https://www.ams.org/journal-terms-of-use
AN ITERATIVE SOLUTION METHOD FOR LINEAR SYSTEMS 159
Region
1
2
3
A(x,y)
1.0
2.0
3.0
0.02
0.03
0.05
A uniform rectangular mesh was chosen with mesh spacing 0.05, so a
system of 1849
equations in 1849 unknowns resulted. The solution of this problem
is known to be
u(x, y) = 0; as starting vector for all iterative methods, a vector
was chosen similar to
the one in Example 1. The iteration results are plotted in Figure
3.
Residual
1 ICCG(O)
2 ICCG (3) 3 SIP 4 S LOR 5 CONJ. GR.
Computational work, expressed in number of iterations ICCG(3)
Figure 3. Results for Example 2
These few examples give some impression of the kind of convergence
that is typ-
ical for the ICCG methods. In order to explain this phenomenon, a
complete Choleski-
decomposition of the type of matrix, introduced in Section 5, is
considered. It is then
observed that the nonzero entries in the full decomposition
decrease rapidly in magni-
tude in the directions pointed out below.
License or copyright restrictions may apply to redistribution; see
https://www.ams.org/journal-terms-of-use
160 J. A. MEIJERINK AND H. A. VAN DER VORST
As is known, Choleski-decomposition is a stable process, therefore
it might be expected
that setting some of the smaller elements to zero, results in an
incomplete Choleski-
decomposition, which will be like the full decomposition. Thus, the
matrix (LLT)~1A,
where LLT is an incomplete decomposition, should resemble the
identity matrix in
some way, or more precisely, (LLT)~1A will have all eigenvalues
close to 1.0. The
fact that conjugate gradients gives fast convergence for matrices
with the latter property
explains to some extent the fast convergence of the ICCG
methods.
In order to give an impression of the eigenvalues of (LLT)~1A for
both the
ICCG(O) and ICCG(3) methods, a smaller version of the matrix
arising in Example 1
has been chosen. In fact, the choice Ax = 1/5 and Ay = 1/6,
resulted in a matrix of
order 36. In Figure 4 all the eigenvalues of A, (LQL^)~1A and
(L3L3r)~ïA are
plotted. The lower index indicates which ICCG method is
considered.
</> 5 LU
• EIGENVALUES OF (L0Lq ) A
x x X
8 8 8 8 8 i*¿° ° o # ^0.446
T°^0.119 _L ?"-0:058
Figure 4. Eigenvalues of A, (L^l) 1A and (L3LJ3)'1A
License or copyright restrictions may apply to redistribution; see
https://www.ams.org/journal-terms-of-use
AN ITERATIVE SOLUTION METHOD FOR LINEAR SYSTEMS 161
100
NUMBER OF LINEAR EQUATIONS
2400
Figure 5. Effect of number of equations on the rate of
convergence
It follows from formula (4.5) that the error \\x¡ -x\\A is
multiplied at each step
by at most r = (yjc - l)/(s/c + 1). This helps explain the fast
convergence; for A,
(LqLq)~1A, and (L3L\r)~lA, respectively, we find r = .84, r0 = .53
and r3 = .23.
Finally, for the linear equations arising in Example 1, the
influence of the order
of the matrix on the number of iterations required to reach a
certain precision was
checked for both ICCG(0) and ICCG(3).
Therefore several uniform rectangular meshes have been chosen, with
mesh spac-
ings varying from ~ 1/10 up to ~ 1/50. This resulted in linear
systems with matrices
of order 100 up to about 2500. In each case it was determined how
many iterations
were necessary, in order that the magnitude of each entry of the
residual vector was
below some fixed small number e, when starting with x0 = 0.
License or copyright restrictions may apply to redistribution; see
https://www.ams.org/journal-terms-of-use
162 J. A. MEIJERINK AND H. A. VAN DER VORST
In Figure 5 the number of iterations are plotted against the order
of the matrices
for e = 10-2, e = 10-6 and e = 10-10. It can be seen that the
number of iterations,
necessary to get the residual vector sufficiently small, increases
only slowly for increas-
ing order of the matrix.
7. Conclusions. In the examples, both ICCG methods appeared to be
far superior
to all the other iterative methods mentioned, except possibly CG
when the matrix has
'property A' [5].
If the solution of the linear system is calculated by complete
Choleski, the total
number of multiplications is given approximately by n(m + l)(w +
2)/2 + 2n(m + 1)
[8], where n is the order of the matrix and 2m the bandwidth. For n
= 900
this amount of work is equivalent to about 25 ICCG(3) iterations
(at this time storage
aspects are not considered).
This implies that both ICCG methods can compete with direct
solution with re-
gard to computational work, if we are satisfied with not too high
an accuracy. From
Figure 5 it can be seen that for larger matrices the ICCG methods
are to be preferred
even more.
These statements also hold if the direct method takes advantage of
the very sparse
structure of the matrices. In this case Price and Coats [4] showed
that the total num-
ber of multiplications for the direct method can be reduced by a
factor 6, compared to
the number mentioned above.
Finally, we would like to observe that the ICCG methods have also
been applied
very successfully in practice, in solving both two- and
three-dimensional problems.
Koninklijke/Shell
Utrecht, The Netherlands
1. J. W. DANIEL, "The conjugate gradient method for linear and
nonlinear operator equa-
tions," SIAM J. Numer. Anal, v. 4, 1967, pp. 10-26. MR 36
#1076.
2. KY FAN, "Note on M-matrices," Quart. J. Math. Oxford Ser. (2),
v. 11, 1960, pp. 43-49.
MR 22 #8024.
3. M. R. HESTENES, The Conjugate-Gradient Method for Solving Linear
Systems, Proc.
Sympos. Appl. Math., vol. VI, Numerical Analysis, McGraw-Hill, New
York, 1956, pp. 83—102.
MR 18, 824.
4. H. S. PRICE & K. H. COATS, "Direct methods in reservoir
simulation," Soc. Petroleum
Engrs. J., v. 14, 1974, pp. 295-308.
5. J. K. REID, "The use of conjugate gradients for systems of
linear equations possessing
'Property A'," SIAM J. Numer. Anal, v. 9, 1972, pp. 325-332. MR 46
#4697.
6. H. L. STONE, "Iterative solution of implicit approximations of
multidimensional partial
differential equations," SIAM J. Numer. Anal, v. 5, 1968, pp.
530-558. MR 38 #6780.
7. R. S. VARGA, Matrix Iterative Analysis, Prentice-Hall, Englewood
Cliffs, N.J., 1962.
MR 28 #1725.
8. J. H. WILKINSON & C. REINSCH, Linear Algebra,
Springer-Verlag, Berlin and New York,
1971.
9. J. H. WILKINSON, The Algebraic Eigenvalue Problem, Clarendon
Press, Oxford, 1965.
MR 32 #1894.