Error Analysis in Iterative Methods

Introduction Order of Convergence Bisection Method Fixed-point Iterations Newton’s Method Secant Method

Error Analysis in Iterative MethodsUE201 : Seminar

Punarbasu RoySR Number : 10975

April 15, 2016


Outline

1 IntroductionLinear vs Non-linear equationsIterative Methods

2 Order of ConvergenceDefinitionExample

3 Bisection Method

4 Fixed-point Iterations

5 Newton’s Method

6 Secant Method


Linear vs Non-linear equations

Linear equations (for eg: 3x + 4y = 7; 8x + 9y = 2) can besolved by direct methods such as gaussian elimination.

Non linear equations (for eg: x = 2− e−2x , x = x3 + x2 + 1 )in general don’t have any direct method for solving.

They require iterative methods.


Table of Contents



3 Bisection Method


5 Newton’s Method

6 Secant Method


Iterative Methods

Iterative methods generate a sequence of iterates which areapproximations to the solution.

The approximate solution is refined with each iteration anditerations are run unless certain order of accuracy has beenachieved.

Some of the iterative methods are

Bisection Method.

Fixed Point Method.

Newton’s Method.


Iterative Methods

Iterative methods generate a sequence of iterates which areapproximations to the solution.

The approximate solution is refined with each iteration anditerations are run unless certain order of accuracy has beenachieved.

Some of the iterative methods are

Bisection Method.

Fixed Point Method.

Newton’s Method.

Which is the best?


Things to consider:

Number of function evaluations and operations.

Number of iterations needed for reaching desired accuracy.


Things to consider:

Number of function evaluations and operations.

Number of iterations needed for reaching desired accuracy.


Table of Contents



3 Bisection Method


5 Newton’s Method

6 Secant Method


Order of Convergence

Definition

Let {pn }∞n=0 be the sequence that converges to p with pn 6= p forall n. If positive constants λ and α exist such that,

limn→∞

|pn+1 − p||pn − p|α

= λ

then the sequence {pn }∞n=0 converges to p of order α withasymptotic error constant λ where α ≥ 1 and ∞ > λ > 0.

where error at kth iteration (ek) is |pk − p|In general the sequence with higher order of convergenceconverges more rapidly.

The asymptotic error constant also has a role in deciding thespeed of convergence of a sequence but not to the extent oforder.


When α = 1 , the sequence is linearly convergent.

When α = 2 , the sequence is quadratically convergent.

When α = 3 , the sequence is cubically convergent

and so on....


But we don’t know what the exact solution is.

So we can’t figure out what is ek (k = 0, 1, 2, 3, ...)

Instead we use relative change in successive iterates whichshould converge to zero.

Iterates are the approximation to x where f (x) = 0 so we cancheck whether f (xk) is converging to zero or not.


Table of Contents



3 Bisection Method


5 Newton’s Method

6 Secant Method


Example

What will be the order of convergence of the sequence {xn }∞n=1

where xn = 1nk

and k > 1 ?

limn→∞

|xn+1 − x ||xn − x |α

Clearly the sequence converges to 0.

= limn→∞

|xn+1 − 0||xn − 0|α

= limn→∞

1(n+1)k

( 1nk

)α

= limn→∞

( nα

n+1)k


Example


where xn = 1nk

and k > 1 ?

limn→∞

|xn+1 − x ||xn − x |α


= limn→∞

|xn+1 − 0||xn − 0|α

= limn→∞

1(n+1)k

( 1nk

)α

= limn→∞

( nα

n+1)k


Example


where xn = 1nk

and k > 1 ?

limn→∞

|xn+1 − x ||xn − x |α


= limn→∞

|xn+1 − 0||xn − 0|α

= limn→∞

1(n+1)k

( 1nk

)α

= limn→∞

( nα

n+1)k


Example


where xn = 1nk

and k > 1 ?

limn→∞

|xn+1 − x ||xn − x |α


= limn→∞

|xn+1 − 0||xn − 0|α

= limn→∞

1(n+1)k

( 1nk

)α

= limn→∞

( nα

n+1)k


Example


where xn = 1nk

and k > 1 ?

limn→∞

|xn+1 − x ||xn − x |α


= limn→∞

|xn+1 − 0||xn − 0|α

= limn→∞

1(n+1)k

( 1nk

)α

= limn→∞

( nα

n+1)k


= limn→∞

( nα

n+1)k

For α = 1 limit exists. For α > 1 limit does not exist.

So the sequence has order of convergence = 1 that is thesequence converges linearly.


Table of Contents



3 Bisection Method


5 Newton’s Method

6 Secant Method


Bisection Method

Theorem

If f ∈ C [a, b] and f (a).f (b) < 0 then Bisection method generatesa sequence {pn }∞n=0 approximating a zero p of f with

|pn − p| ≤ b − a

2nwhen n ≥ 1

When the initial interval is [a, b], the interval for kth iterationis [ak , bk ] where bk − ak = 2−k(b − a) .

So the upperbound of the error in each iteration is the intervallength of that step.

After each iteration the the interval length is reduced by halfand so is the upperbound of error.


So error in (n + 1)th and nth iteration can be written as

|en+1| = |pn+1 − p| ≤ 12(bn+1 − an+1)

|en| = |pn − p| ≤ 12(bn − an)

Dividing the (n + 1)th one by nth one we can get

|en+1||en|

≤ 12

here α = 1 and λ = 12

Hence the bisection method converges linearly withasymptotic error constant (λ) 1

2


Table of Contents



3 Bisection Method


5 Newton’s Method

6 Secant Method


Rate of Convergence in Fixed Point Iterations

Theorem

(i) If g ∈ C [a, b] and g(x) ∈ [a, b] for all x ∈ [a, b] then g has atleast one fixed point in [a, b].(ii) If, in addition, g ′(x) exists on (a, b) and a positive constantk < 1 exists with |g ′(x)| ≤ k for all x ∈ (a, b) then there is exactlyone fixed point in [a, b]

We use the formula pn+1 = g(pn) to generate the iterates.

Error at nth iteration; en+1 = pn+1 − pen+1 = g(pn+1)− g(p) = g ′(ξn)(pn − p) = g ′(ξn)(en)where ξn lieas between pn and p. (Mean Value Theorem)

limn→∞

|en+1||en| = lim

n→∞|g ′(ξn)| = |g ′(p)|

Converges linearly (α = 1) with asymptotic error constant(λ)= |g ′(p)|


The order of convergence increases when extra conditions on g aremet. Assume g ′(p) = 0 and g ′′ exists on (a, b) .

en+1 = g(pn)− g(p) = g ′(p)(pn − p) + 12g′′(ξn)(pn − p)2

(Taylor Theorem)

en+1 = 12g′′(ξn)(en)2

Result : Quadratic convergence (α = 2) and λ = 12g′′(p)

This leads to a general result in the next theorem.


Theorem

Let g(x) be a function which is m times differentiable on aninterval [a, b]. If g(x) ∈ [a, b] for x ∈ [a, b] and that |g ′(x) ≤ k | on(a, b) for some k < 1. If the unique fixed point p in [a, b] satisfies

0 = g(p) = g ′(p) = g ′′(p) = ... = gm−1(p)

Then for any intial guess (x0) ∈ [a, b], the Fixed-point Iteration

converges to p with α = m and λ = |g(m)(p)

n!|


Table of Contents



3 Bisection Method


5 Newton’s Method

6 Secant Method


Rate of Convergence in Newton’s Method

With initial guess p0 Newton’s Method generates sequence ofiterates approximating the exact solution p by the formula ,

pn = pn−1 −f (pn−1)

f ′(pn−1)

Error at n + 1th iteration is given by,

en+1 = pn+1 − p

= pn −f (pn)

f ′(pn)− p

= en −f (pn)

f ′(pn)

= en −1

f ′(pn)[f (p)− f ′(pn)(p − pn)− 1

2f ′′(ξk)(pk − p)2]

continued...


en+1 = en −1

f ′(pn)[f (p)− f ′(pn)(p − pn)− 1

2f ′′(ξn)(pn − p)2]

= en −1

f ′(pn)[−f ′(pn)en −

1

2f ′′(ξn)(en)2]

= en − en +f ′′(ξn)

2f ′(pn)e2n

en+1 =f ′′(ξn)

2f ′(pn)e2n

limn→∞

|en+1||en|2

= limn→∞

f ′′(ξn)

2f ′(pn)=

f ′′(p)

2f ′(p)

Result :

Newton’s Method converges quadratically (α = 2) and λ =f ′′(p)

2f ′(p)


Table of Contents



3 Bisection Method


5 Newton’s Method

6 Secant Method


Rate of Convergence in Secant Method

In secant method, the formula which generates the sequence ofiterates approximating the solution is given by

pn = pn−1 −f (pn−1)(pn−1 − pn−2)

f (pn−1)− f (pn−2)

Which has order of convergence (α) = 1.618 1

1The derivation is complicated thus not showed here.


Comparison : Newton vs Secant

Theoritical derivations show that Newton’s method has higherorder of convergence than Secant method.

Newton’s method should take less number of iteration thansecant to converge to the exact solution.

Comparing speed of convergence of Newton’s method vs Secantmethod for the function f (x) = x + e−x

2cos(x)

MATLAB programme iterated both methods until relativeerror of order less than 10−6 was achieved.

logarithm of relative error vs number of iterations were plottedfor both methods.


Practically too, Newton’s method converges faster than SecantMethod (at the price of evaluating derivative of the function eachstep).


Thank You


Bibliography

Numerical Analysis 9th Edition Richard Burden J. D. FairesSection 2.4

banach.millersville.edu/ bob/math375/IterativeError/main.pdf

math.usm.edu/lambers/mat460/fall09/lecture12.pdf

macs.citadel.edu/chenm/343.dir/01.dir/lect2 4.pdf

Error Analysis in Iterative Methods

Science