8/9/2019 Lec04 Perturbation
1/39
ERROR AND SENSITIVTY ANALYSIS FOR SYSTEMS
OF LINEAR EQUATIONS
Read parts of sections 2.6 and 3.5.3
Conditioning of linear systems.
Estimating errors for solutions of linear systems
Backward error analysis
Relative element-wise error analysis
4-1 GvL 3.5; Heath 2.3; TB 12 pert
8/9/2019 Lec04 Perturbation
2/39
Perturbation analysis for linear systems (Ax=b)
Question addressed by perturbation analysis: determinethe variation of the solution x when the data, namely
A and b, undergoes small variations. Problem is Ill-
conditioned if small variations in data cause very largevariation in the solution.
4-2 GvL 3.5; Heath 2.3; TB 12 pert
8/9/2019 Lec04 Perturbation
3/39
Analysis I: Asymptotic First Order Analysis
Let E, be an n n matrix and eb be an n-vector.
Perturb A into A() =A +E and b into b+eb.
Note: A+ E is nonsingular for small enough.
Why?
The solution x() of the perturbed system is s.t.(A +E)x() =b+eb.
4-3 GvL 3.5; Heath 2.3; TB 12 pert
8/9/2019 Lec04 Perturbation
4/39
Let () =x() x. Then,
(A+ E)() = (b+eb) (A+E)x= (eb Ex)() = (A+ E)1(eb Ex).
x() is differentiable at = 0 and its derivative is
x(0) = lim0()
=A1 (eb Ex) .
A small variation[E, eb]will cause the solution to vary
by roughly x(0) =A1
(eb Ex).
4-4 GvL 3.5; Heath 2.3; TB 12 pert
8/9/2019 Lec04 Perturbation
5/39
8/9/2019 Lec04 Perturbation
6/39
The quantity (A) = A A1 is called thecondition
numberof the linear system with respect to the norm ..When using the p-norms we write:
p(A) = ApA1p
Note: 2(A) =max(A)/min(A)= ratio of largest to
smallest singular values ofA. Allows to define2(A)whenA is not square.
Determinant *is not* a good indication of sensitivity
Small eigenvalues *do not* always give a good indicationof poor conditioning.
4-6 GvL 3.5; Heath 2.3; TB 12 pert
8/9/2019 Lec04 Perturbation
7/39
8/9/2019 Lec04 Perturbation
8/39
Rigorous norm-based error bounds
Previous bound is valid only when perturbation is small
enough, where small is not precisely defined.
New bound valid within an explicitly given neighborhood.THEOREM 1: Assume that (A + E)y = b + eb and
Ax = b and that A1E < 1. Then A + E is
nonsingular andx y
x
A1 A
1 A1 E
E
A+
eb
b
4-8 GvL 3.5; Heath 2.3; TB 12 pert
8/9/2019 Lec04 Perturbation
9/39
To prove, first need to show that A+E is nonsingular
ifA is nonsingular and E is small. Begin with simple case:
LEMMA: IfE
8/9/2019 Lec04 Perturbation
10/39
d) (I E)1 = limk
ki=0 E
i. We write this as
(I E)1 =
i=0
Ei
e) Finally:
(I E)1 =
limk
k
i=0Ei
= limk
k
i=0Ei
limk
ki=0
Ei limk
ki=0
Ei
1
1 E
4-10 GvL 3.5; Heath 2.3; TB 12 pert
8/9/2019 Lec04 Perturbation
11/39
Can generalize result:
LEMMA: If A is nonsingular and A1
E < 1 thenA +E is non-singular and
(A +E)1 A1
1A1 E
Proof is based on relation A + E=A(I+ A1E) and useof previous lemma.
4-11 GvL 3.5; Heath 2.3; TB 12 pert
8/9/2019 Lec04 Perturbation
12/39
Now we can prove the theorem:
THEOREM 1: Assume that (A + E)y = b + eb andAx = b and that A1E < 1. Then A + E is
nonsingular and
x yx
A1
A1 A1 E
EA
+ ebb
4-12 GvL 3.5; Heath 2.3; TB 12 pert
8/9/2019 Lec04 Perturbation
13/39
Proof: From (A+E)y =b+eb and Ax=b we get
(A+ E)(y x) =eb Ex. Hence:
y x= (A +E)1(eb Ex)
Taking norms y x (A+E)1 [eb + Ex]
Dividing by x and using result of lemmay x
x (A +E)1 [eb/x + E]
A11 A1E
[eb/x + E]
A1A
1 A1E eb
Ax
+E
A
Result follows by using inequality Ax b.... QED
4-13 GvL 3.5; Heath 2.3; TB 12 pert
8/9/2019 Lec04 Perturbation
14/39
8/9/2019 Lec04 Perturbation
15/39
Another common form:
THEOREM 2: Let (A+ A)y = b+ b and Ax = bwhere A E, b eb, and assume that
A1E
8/9/2019 Lec04 Perturbation
16/39
8/9/2019 Lec04 Perturbation
17/39
Normwise backward error in just A or b
Suppose we model entire perturbation in RHS b.
Let r =b Ay be the residual.
Then y satisfies Ay =b+ b with b=r exactly.
The relative perturbation to the RHS is rb
.
Suppose we model entire perturbation in matrix A.
Then y satisfies
A ryT
yTy
y =b
The relative perturbation to the matrix is
ryT
yTy
2
/A2 = r2
Ay2
4-17 GvL 3.5; Heath 2.3; TB 12 pert
8/9/2019 Lec04 Perturbation
18/39
8/9/2019 Lec04 Perturbation
19/39
yis given (a computed solution). Eand ebto be selected
(most likely directions of perturbation for A and b).
Typical choice: E=A, eb =b
Explain why this is not unreasonable
Let r =b Ay. Then we have:
THEOREM 3: E,eb(y) = r
Ey+eb
Normwise backward error is for case E=A, eb =b:A,b(y) =
rAy+b
4-19 GvL 3.5; Heath 2.3; TB 12 pert
8/9/2019 Lec04 Perturbation
20/39
Show how this can be used in practice as a means to
stop some iterative method which computes a sequence of
approximate solutions to Ax=b.
4-20 GvL 3.5; Heath 2.3; TB 12 pert
8/9/2019 Lec04 Perturbation
21/39
8/9/2019 Lec04 Perturbation
22/39
Proof of Theorem 3
Let D Ey + eb and E,eb(y). The theoremstates that = r/D. Proof in 2 steps.
First: Any A,b pair satisfying (1) is such that
r/D. Indeed from (1) we have (recall that r =b Ay)
Ay+ Ay =b+ b r = Ay b
r Ay+b (Ey+eb) r
D
Second: We need to show an instance where the minimum
value ofr/D is reached. Take the pair A, b:
A=rzT; b=r with =Ey
D; =
eb
D
The vector z depends on the norm used - for the 2-norm:z =y/y2. Here: Proof only for 2-norm
8/9/2019 Lec04 Perturbation
23/39
a) We need to verify that first part of (1) is satisfied:
(A+ A)y = Ay+r yT
y2y =b r+r
= b (1 )r =b 1 Ey
Ey + eb r= b
eb
Dr =b+ r
(A+ A)y = b+ b The desired result
b) Finally: Must now verify that A = E and
b =eb. Exercise: Show that uvT2 = u2v2
A = ||y2
ryT = EyD
ryy2
=E
b = ||r =eb
Dr =eb QED
4-23 GvL 3.5; Heath 2.3; TB 12 pert
8/9/2019 Lec04 Perturbation
24/39
8/9/2019 Lec04 Perturbation
25/39
Show: a function which satisfies the first 2 requirements
of vector norms (1. (x) 0 (==0, iff x = 0) and 2.
(x) = ||(x)) satisfies the triangle inequality iff itsunit ball is convex.
(Continued) Use the above to construct a norm in R2
that is *not* absolute.
4-25 GvL 3.5; Heath 2.3; TB 12 pert
8/9/2019 Lec04 Perturbation
26/39
Define absolute *matrix* norms in same way. Which of
the norms A1, A, A2, and AFare absolute?
Recall that for any matrix f l(A) =A+E with |E|
u |A|. For an absolute matrix norm
EA
u
What does this imply?
Component-wise analysis requires that we use normsthat are *absolute*
We will restrict analysis to .
See sec. 2.6.5 of text.
4-26 GvL 3.5; Heath 2.3; TB 12 pert
8/9/2019 Lec04 Perturbation
27/39
Analogue of theorem 2 for case E= |A|, eb = |b|:
THEOREM 4 Let Ax = b and (A+ A)y = b+ bwhere |A| |A| and |b| |b|. Assume that
(A) =
8/9/2019 Lec04 Perturbation
28/39
Componentwise relative condition number :
C(A) |A1| |A|
Redo example seen after Theorem 3, (6 6 Vander-
monde system) using componentwise analysis.
4-28 GvL 3.5; Heath 2.3; TB 12 pert
8/9/2019 Lec04 Perturbation
29/39
8/9/2019 Lec04 Perturbation
30/39
Example of ill-conditioning: The Hilbert Matrix
Notorious example of ill conditioning.
Hn =
1 12
13
1n
1
2
1
3
1
4
1
n+1... ... ... ... ...1n
1n+1
12n1
i.e., hij = 1
i +j 1
Forn= 5 2(Hn) = 4.766.. 105.
Let bn =Hn(1, 1, . . . , 1)T.
Solution ofHnx=b is (1, 1, . . . , 1)T
. Let n= 5 and perturb h5,1 = 0.2 into 0.20001.
New solution: (0.9937, 1.1252, 0.4365, 1.865, 0.5618)T
4-30 GvL 3.5; Heath 2.3; TB 12 pert
8/9/2019 Lec04 Perturbation
31/39
Estimating condition numbers.
Avoid the expense of computing A1 explicitly.
Choose a random or carefully chosen vector v.
Solve Au=v using factorization already computed. Then A1 u / v is an guess-timate of A1.
Estimated condition number is (A) Au / v.
Typical choice for v: choose [ 1 ] with signs
chosen on the fly during back-substitution to maximize the
next entry in the solution, based on the upper triangular
factor from Gaussian Elimination.
4-31 GvL 3.5; Heath 2.3; TB 12 pert
8/9/2019 Lec04 Perturbation
32/39
Condition Number Measures How Close to Singularity
1/ relative distance to nearest singular matrix.
Let A, B be two n n matrices with A nonsingular and
B singular. Then
1
(A)
A B
A
Proof: B singular x = 0 such that Bx= 0.
x = A1Ax A1 Ax = A1(A B)x
A1
A BxDivide both sides by x (A) = xA A1
result. QED.
4-32 GvL 3.5; Heath 2.3; TB 12 pert
8/9/2019 Lec04 Perturbation
33/39
Example:
let A =
1 11 0.99
and B =
1 11 1
Then 1
1(A) 0.01
2 1(A)
20.01
= 200.
It can be shown that (Kahan)
1
(A) = minBA B
A | det(B) = 0
4-33 GvL 3.5; Heath 2.3; TB 12 pert
8/9/2019 Lec04 Perturbation
34/39
Estimating errors from residual norms
Let x an approximate solution to system Ax = b (e.g.,computed from an iterative process). We can compute
the residual norm:
r = b AxQuestion: How to estimate the error xx from r?
One option is to use the inequalityxx
x (A) r
b.
We must have an estimate of(A).
4-34 GvL 3.5; Heath 2.3; TB 12 pert
8/9/2019 Lec04 Perturbation
35/39
Proof of inequality.
First, note that A(x x) =b Ax=r. So:x x = A1r A1 r
Also note that from the relation b=Ax, we get
b = Ax A x x b
A
Therefore,
x x
x
A1 r
b/A= (A)
r
b
Show thatxx
x 1
(A)rb
.
4-35 GvL 3.5; Heath 2.3; TB 12 pert
8/9/2019 Lec04 Perturbation
36/39
S ll E l
8/9/2019 Lec04 Perturbation
37/39
Small Example
Solve Ax =b problem in 3-digit decimal arithmetic:0.641 0.242
0.321 0.121
x1x2
=
0.883
0.442
Solution by standard algorithm is y =
0.708
1.775
residual r =b Ay =
7.12
1.12
104.
(actually used 11 bit arithmetic, printed in decimal.)
4-37 GvL 3.5; Heath 2.3; TB 12 pert
E ti t f d i ll l
8/9/2019 Lec04 Perturbation
38/39
Estimate forward error in small example
Get estimated condition number from(1, 2) A= (0.001, 0.000)
Conclude A11 (1, 2)1
(0.001, 0.000)1
= 3000.
Combine with A1 = 0.962 1.
Get lower bound on Condition Number: 1(A) 3000.
So Theorem 1 simplified gives
x y
x A1 A
eb
b 3000
8.24 104
1.325 = 1.866,
predicting no accuracy!
Keep over 4 decimal digits to get any accuracy at all!
4-38 GvL 3.5; Heath 2.3; TB 12 pert
8/9/2019 Lec04 Perturbation
39/39