Part I: Linear Systems Part II: Least Squares & Polynomial Interpolation Part III: Nonlinear Equations Part IV: Numer UNSW Mathematics Society presents MATH2089 Revision Seminar Numerical Methods & Statistics Numerical Methods Presented by Janzen Choi T2, 2020
97
Embed
MATH2089 Revision Seminar - mathsoc.unsw.edu.au · MATH2089 Revision Seminar Numerical Methods & Statistics Numerical Methods T2, 2020 Presented by Janzen Choi. Part I: Linear Systems
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Part I: Linear Systems Part II: Least Squares & Polynomial Interpolation Part III: Nonlinear Equations Part IV: Numerical Differentiation and Integration Part V: Ordinary Differential Equations Part VI: Partial Differential Equations
UNSW Mathematics Society presents
MATH2089 Revision Seminar
Numerical Methods & StatisticsNumerical Methods
Presented by Janzen ChoiT2, 2020
Part I: Linear Systems Part II: Least Squares & Polynomial Interpolation Part III: Nonlinear Equations Part IV: Numerical Differentiation and Integration Part V: Ordinary Differential Equations Part VI: Partial Differential Equations
Table of Contents I1 Part I: Linear Systems
LU factorisationCholesky factorisationSparsity of matricesNorm and condition numberSensitivity of a linear system
2 Part II: Least Squares & Polynomial InterpolationLeast squares
Part I: Linear Systems Part II: Least Squares & Polynomial Interpolation Part III: Nonlinear Equations Part IV: Numerical Differentiation and Integration Part V: Ordinary Differential Equations Part VI: Partial Differential Equations
Table of Contents IINewton’s methodSecant method
4 Part IV: Numerical Differentiation and IntegrationTaylor approximationFinite difference methods
Part I: Linear Systems Part II: Least Squares & Polynomial Interpolation Part III: Nonlinear Equations Part IV: Numerical Differentiation and Integration Part V: Ordinary Differential Equations Part VI: Partial Differential Equations
Table of Contents IIISystem of first order ODEs
2 and 4-stage Runge-Kutta methods
6 Part VI: Partial Differential EquationsFinite difference methods of PDEs
Part I: Linear Systems Part II: Least Squares & Polynomial Interpolation Part III: Nonlinear Equations Part IV: Numerical Differentiation and Integration Part V: Ordinary Differential Equations Part VI: Partial Differential Equations
Part I: Linear Systems Part II: Least Squares & Polynomial Interpolation Part III: Nonlinear Equations Part IV: Numerical Differentiation and Integration Part V: Ordinary Differential Equations Part VI: Partial Differential Equations
Matrix factorisation − LU factorisationGiven an n × n matrix A, if the leading principal sub-matrices Akare non-singular for all k = 1, . . . , n, then there exist n × nmatrices L and U where L is unit lower triangular and U is uppertriangular, such that
A = LU.
If the factorisation exists, it is also unique.Obtain the LU factorisation by applying row operations of the form
Ri ← Ri − LijRj .
Cost of factorisationThe cost of the LU factorisation is
Part I: Linear Systems Part II: Least Squares & Polynomial Interpolation Part III: Nonlinear Equations Part IV: Numerical Differentiation and Integration Part V: Ordinary Differential Equations Part VI: Partial Differential Equations
Effects of the permutation matrix
If A is non-singular, then there exist n × n matrices L,U and Pwith P being a permutation matrix such that
PA = LU.
PA reorders the rows of A but does not change the solution to thelinear system.
AP reorders the columns of A and affects the solution to linearsystem.
Part I: Linear Systems Part II: Least Squares & Polynomial Interpolation Part III: Nonlinear Equations Part IV: Numerical Differentiation and Integration Part V: Ordinary Differential Equations Part VI: Partial Differential Equations
Solving linear systems using LU factorisationWe note that
Ax = b =⇒ (PA)x = Pb =⇒ LUx = Pb.
1 Forward substitution: Solve Ly = Pb = z for y. Then
Part I: Linear Systems Part II: Least Squares & Polynomial Interpolation Part III: Nonlinear Equations Part IV: Numerical Differentiation and Integration Part V: Ordinary Differential Equations Part VI: Partial Differential Equations
Part I: Linear Systems Part II: Least Squares & Polynomial Interpolation Part III: Nonlinear Equations Part IV: Numerical Differentiation and Integration Part V: Ordinary Differential Equations Part VI: Partial Differential Equations
Matrix factorisation − Cholesky factorisationGiven an n × n matrix A, if the Cholesky factorisation exists, thenit is of the form
A = RᵀR
where R is an n × n upper triangular matrix, with Rii > 0 for alli = 1, . . . , n.
A Cholesky factorisation is unique when it exists.The matrix A is positive definite and symmetric.All eigenvalues of A are positive.
Cost of factorisationThe cost of the Cholesky factorisation is
Part I: Linear Systems Part II: Least Squares & Polynomial Interpolation Part III: Nonlinear Equations Part IV: Numerical Differentiation and Integration Part V: Ordinary Differential Equations Part VI: Partial Differential Equations
Solving linear systems using Choleskyfactorisation
Part I: Linear Systems Part II: Least Squares & Polynomial Interpolation Part III: Nonlinear Equations Part IV: Numerical Differentiation and Integration Part V: Ordinary Differential Equations Part VI: Partial Differential Equations
Solving linear systems using Choleskyfactorisation
Part I: Linear Systems Part II: Least Squares & Polynomial Interpolation Part III: Nonlinear Equations Part IV: Numerical Differentiation and Integration Part V: Ordinary Differential Equations Part VI: Partial Differential Equations
Part I: Linear Systems Part II: Least Squares & Polynomial Interpolation Part III: Nonlinear Equations Part IV: Numerical Differentiation and Integration Part V: Ordinary Differential Equations Part VI: Partial Differential Equations
Example: (MATH2089, 2010 Q3h)
You are given that using a row-ordering of the variables c`i ,j
produces the coefficient matrix A whose non-zero entries areillustrated below.
Part I: Linear Systems Part II: Least Squares & Polynomial Interpolation Part III: Nonlinear Equations Part IV: Numerical Differentiation and Integration Part V: Ordinary Differential Equations Part VI: Partial Differential Equations
Recall that the sparsity of a matrix is given by
Sparsity =( non-zero elements of A
total number of elements in A
)%.
The number of nonzero entries in a spy plot is given by thevariable nz. The dimension of the matrix is given by the grid size,which in this case is 9× 4 = 36. Hence the sparsity is
Part I: Linear Systems Part II: Least Squares & Polynomial Interpolation Part III: Nonlinear Equations Part IV: Numerical Differentiation and Integration Part V: Ordinary Differential Equations Part VI: Partial Differential Equations
Vector normProperties of vector normsA vector norm ‖·‖ is an operation on the vector with the followingproperties:
‖x‖ ≥ 0 with ‖x‖ = 0 only if x = 0.
Triangle inequality: ‖x + y‖ ≤ ‖x‖+ ‖y‖.
For any constant α, ‖αx‖ = |α| ‖x‖.
Vector p-normsVector p norms are special types of norms on n × 1 vectors. Bydefinition, for p ≥ 1, the p-norm of an n × 1 vector is given by
Part I: Linear Systems Part II: Least Squares & Polynomial Interpolation Part III: Nonlinear Equations Part IV: Numerical Differentiation and Integration Part V: Ordinary Differential Equations Part VI: Partial Differential Equations
Part I: Linear Systems Part II: Least Squares & Polynomial Interpolation Part III: Nonlinear Equations Part IV: Numerical Differentiation and Integration Part V: Ordinary Differential Equations Part VI: Partial Differential Equations
Matrix norms
Properties of vector normsA matrix norm ‖·‖ is an operation on a matrix with the followingproperties:
‖A‖ ≥ 0 with ‖A‖ = 0 only if A = 0.
Triangle inequality: ‖A + B‖ ≤ ‖A‖+ ‖B‖.
For any constant α, ‖αA‖ = |α| ‖A‖.
Consistent matrix normsA matrix norm is said to be consistent if
Part I: Linear Systems Part II: Least Squares & Polynomial Interpolation Part III: Nonlinear Equations Part IV: Numerical Differentiation and Integration Part V: Ordinary Differential Equations Part VI: Partial Differential Equations
Matrix p-norms
For p ≥ 1, the p-norm of an m × n matrix is given by
Part I: Linear Systems Part II: Least Squares & Polynomial Interpolation Part III: Nonlinear Equations Part IV: Numerical Differentiation and Integration Part V: Ordinary Differential Equations Part VI: Partial Differential Equations
Examples of matrix p-norms
Matrix 1-norm (maximum column sum):
‖A‖1 = max1≤j≤n
∑1≤i≤m
|aij |
.
Matrix ∞-norm (maximum row sum):
‖A‖∞ = max1≤j≤m
∑1≤i≤n
|aij |
.
Matrix 2-norm (square root of the largest eigenvalue of AᵀA):
Part I: Linear Systems Part II: Least Squares & Polynomial Interpolation Part III: Nonlinear Equations Part IV: Numerical Differentiation and Integration Part V: Ordinary Differential Equations Part VI: Partial Differential Equations
Condition number
A square matrix A is non-singular if and only ifdet(A) 6= 0 (invertible).
All eigenvalues of A are non-zero.
Condition numberFor a non-singular matrix A, the condition number is defined as
Part I: Linear Systems Part II: Least Squares & Polynomial Interpolation Part III: Nonlinear Equations Part IV: Numerical Differentiation and Integration Part V: Ordinary Differential Equations Part VI: Partial Differential Equations
Condition number
Properties of condition numbers
κ(A) ≥ 1 for consistent matrix norms.
κ(αI) = 1 for all α 6= 0.
For a real symmetric matrix, the 2-norm condition number is
Part I: Linear Systems Part II: Least Squares & Polynomial Interpolation Part III: Nonlinear Equations Part IV: Numerical Differentiation and Integration Part V: Ordinary Differential Equations Part VI: Partial Differential Equations
Example: (MATH2089, S1 2010, Q1c)The coefficient matrix A and the right-hand-side vector b areknown to 8 significant figures, and
Part I: Linear Systems Part II: Least Squares & Polynomial Interpolation Part III: Nonlinear Equations Part IV: Numerical Differentiation and Integration Part V: Ordinary Differential Equations Part VI: Partial Differential Equations
Example
Let A =
2 −1 2−1 1 −12 −1 3
and A−1 =
2 1 −11 2 0−1 0 1
.
Compute the condition numbers κ∞(A) and κ1(A).
The condition number κ∞(A) is simply just
κ∞(A) = ‖A‖∞‖A−1‖∞.
The sum of the magnitude of the rows of A are simply|2|+ | − 1|+ |2| = 5, | − 1|+ |1|+ | − 1| = 3, and|2|+ | − 1|+ |3| = 6. Hence ‖A‖∞ is just 6. We repeat the sameprocess for A−1 and get ‖A−1‖∞ = 4. So
Part I: Linear Systems Part II: Least Squares & Polynomial Interpolation Part III: Nonlinear Equations Part IV: Numerical Differentiation and Integration Part V: Ordinary Differential Equations Part VI: Partial Differential Equations
Sensitivity of a linear systemLet x be an approximation to x. Then the absolute error of x is‖x− x‖ and the relative error is
ρx = ‖x− x‖‖x‖ .
Let A be an approximation to A. Then the absolute error is‖A− A‖ and the relative error is
ρA = ‖A− A‖‖A‖ .
(Theorem) Sensitivity of a linear systemThe sensitivity of a linear system Ax = b to the error in input dataA and b is given by
Part I: Linear Systems Part II: Least Squares & Polynomial Interpolation Part III: Nonlinear Equations Part IV: Numerical Differentiation and Integration Part V: Ordinary Differential Equations Part VI: Partial Differential Equations
Sensitivity of a linear system
Properties of the errorsIf A or b are known exactly, then the errors ρA and ρx are 0. Thatis, there is no error in precision.
Part I: Linear Systems Part II: Least Squares & Polynomial Interpolation Part III: Nonlinear Equations Part IV: Numerical Differentiation and Integration Part V: Ordinary Differential Equations Part VI: Partial Differential Equations
Example: (MATH2089, 2009 Q1c)The following MATLAB code generates the given output for apre-defined real square array A.chk1 = norm(A - A’, 1)chk1 = 1.4052e-015ev = eig(A);evlim = [min(ev) max(ev)]evlim = 4.5107e-002 9.1213e+004
Is A symmetric?
Is A positive definite?
Calculate the 2-norm condition number κ2(A) of A.
When solving the linear system Ax = b, the elements of A and b areknown to 6 significant decimal digits. Estimate the relative error inthe computed solution x.
Given the Cholesky factorisation A = RᵀR, explain how to solve thelinear system Ax = b.
Part I: Linear Systems Part II: Least Squares & Polynomial Interpolation Part III: Nonlinear Equations Part IV: Numerical Differentiation and Integration Part V: Ordinary Differential Equations Part VI: Partial Differential Equations
Example: (MATH2089, 2009 Q1c)The following MATLAB code generates the given output for apre-defined real square array A.chk1 = norm(A - A’, 1)chk1 = 1.4052e-015
Is A symmetric?
From the MATLAB command chk1 = norm(A - A’, 1), thisimplies that ‖A−Aᵀ‖1 ≈ 1.4× 10−15 ≈ 7ε where ε = 2.2× 10−16.Hence the value is small enough such that
Part I: Linear Systems Part II: Least Squares & Polynomial Interpolation Part III: Nonlinear Equations Part IV: Numerical Differentiation and Integration Part V: Ordinary Differential Equations Part VI: Partial Differential Equations
Example: (MATH2089, 2009 Q1c)The following MATLAB code generates the given output for apre-defined real square array A.ev = eig(A);evlim = [min(ev) max(ev)]evlim = 4.5107e-002 9.1213e+004
Is A positive definite?
Because A is symmetric, then the following statements areequivalent.
A is positive definite.
All of the eigenvalues in A are positive.
From our MATLAB command, we see that the minimum eigenvalue,given by the command min(ev), is 4.5× 10−2 > 0. Hence all ofthe eigenvalues are positive and thus, A is positive definite.
Part I: Linear Systems Part II: Least Squares & Polynomial Interpolation Part III: Nonlinear Equations Part IV: Numerical Differentiation and Integration Part V: Ordinary Differential Equations Part VI: Partial Differential Equations
Example: (MATH2089, 2009 Q1c)The following MATLAB code generates the given output for apre-defined real square array A.ev = eig(A);evlim = [min(ev) max(ev)]evlim = 4.5107e-002 9.1213e+004
Calculate the 2-norm condition number κ2(A) of A.
For a real symmetric matrix, the 2-norm condition number is
Part I: Linear Systems Part II: Least Squares & Polynomial Interpolation Part III: Nonlinear Equations Part IV: Numerical Differentiation and Integration Part V: Ordinary Differential Equations Part VI: Partial Differential Equations
Example: (MATH2089, 2009 Q1c)
When solving the linear system Ax = b, the elements of A and b areknown to 6 significant decimal digits. Estimate the relative error inthe computed solution x.
Since A and b are known to 6 significant decimal digits, then wehave
Part I: Linear Systems Part II: Least Squares & Polynomial Interpolation Part III: Nonlinear Equations Part IV: Numerical Differentiation and Integration Part V: Ordinary Differential Equations Part VI: Partial Differential Equations
Example: (MATH2089, 2009 Q1c)
Given the Cholesky factorisation A = RᵀR, explain how to solve thelinear system Ax = b.
Apply forward substitution and back substitution. Let A = RᵀR sothat
Ax = b =⇒ RᵀRx = b =⇒ Rᵀy = b,
where Rx = y. Solve Rᵀy = b by forward substitution to getRx = y and solve Rx = y by back substitution to get x.
Part I: Linear Systems Part II: Least Squares & Polynomial Interpolation Part III: Nonlinear Equations Part IV: Numerical Differentiation and Integration Part V: Ordinary Differential Equations Part VI: Partial Differential Equations
Part I: Linear Systems Part II: Least Squares & Polynomial Interpolation Part III: Nonlinear Equations Part IV: Numerical Differentiation and Integration Part V: Ordinary Differential Equations Part VI: Partial Differential Equations
Least squares
Given a set of data points, determine the line or curve of best fit.
Methods to finding least squaresFor a given m × n matrix A with m > n, we can apply twomethods to finding least square solutions.
1 Normal equation: AᵀAu = Aᵀy.O(mn2) flops.
2 QR factorisation and back substitution.O(mn2) flops.
Part I: Linear Systems Part II: Least Squares & Polynomial Interpolation Part III: Nonlinear Equations Part IV: Numerical Differentiation and Integration Part V: Ordinary Differential Equations Part VI: Partial Differential Equations
Method 1: Normal equationsAssumptions.
A is an m × n matrix (with m > n) and A has full rank.
Au = y =⇒ (AᵀA) u = Aᵀy.
Define a new matrix B to be the matrix B = AᵀA. B is symmetricand positive definite.
Solve Bu = Aᵀy by applying either Cholesky or LU factorisationwith forward and backward substitutions.
Part I: Linear Systems Part II: Least Squares & Polynomial Interpolation Part III: Nonlinear Equations Part IV: Numerical Differentiation and Integration Part V: Ordinary Differential Equations Part VI: Partial Differential Equations
Method 2: QR factorisation
We can try and write A as a product of two matrices
A = QR,
where Q is an orthogonal matrix and R is an upper triangularmatrix.
Q =[Y Z
], where Y is an m× n matrix and Z is an m× (m− n)
Part I: Linear Systems Part II: Least Squares & Polynomial Interpolation Part III: Nonlinear Equations Part IV: Numerical Differentiation and Integration Part V: Ordinary Differential Equations Part VI: Partial Differential Equations
Polynomial interpolation
Idea: We want to make a new polynomial (interpolation) thatpasses through the data points.
Given data points (x0, y0) and (x1, y1), we solve the simultaneousequation
p(x) = a0 + a1x =⇒{
y0 = a0 + a1x0
y1 = a0 + a1x1
Given n number of data points, an interpolating polynomial willhave degree (n − 1).
Part I: Linear Systems Part II: Least Squares & Polynomial Interpolation Part III: Nonlinear Equations Part IV: Numerical Differentiation and Integration Part V: Ordinary Differential Equations Part VI: Partial Differential Equations
Interpolating polynomials in Lagrange form
Given (n + 1) data points (xj , yj), construct Lagrange polynomialsof degree n of the form
`i (x) =n∏
j=0j 6=i
(x − xjxi − yj
)for i = 0, . . . , n.
Note that `i (xj) = 1 for i = j and `i (xj) = 0 if i 6= j .
Part I: Linear Systems Part II: Least Squares & Polynomial Interpolation Part III: Nonlinear Equations Part IV: Numerical Differentiation and Integration Part V: Ordinary Differential Equations Part VI: Partial Differential Equations
Fora function f , the following data are known:
f (0) = 12.6, f (1) = 6.7, f (2) = 4.3, f (3) = 2.7.
What is the degree of the interpolating polynomial P for these data?
Assume that we want to find P in the form
P(x) = a0 + a1x + · · · .
Write down the system of linear equations you need to solve toobtain a0, a1, . . ..Use MATLAB to set up and solve this linear system.
Write down the Lagrange polynomials `j(x) for j = 0, 1, 2, 3.
Write down the interpolating polynomial P using the Lagrangepolynomials.
Part I: Linear Systems Part II: Least Squares & Polynomial Interpolation Part III: Nonlinear Equations Part IV: Numerical Differentiation and Integration Part V: Ordinary Differential Equations Part VI: Partial Differential Equations
Fora function f , the following data are known:
f (0) = 12.6, f (1) = 6.7, f (2) = 4.3, f (3) = 2.7.
What is the degree of the interpolating polynomial P for these data?
As there are 4 data values and a polynomial of degree n has n + 1coefficients, then the degree of the interpolating polynomial isn = 4− 1 = 3.
Part I: Linear Systems Part II: Least Squares & Polynomial Interpolation Part III: Nonlinear Equations Part IV: Numerical Differentiation and Integration Part V: Ordinary Differential Equations Part VI: Partial Differential Equations
Assume that we want to find P in the form
P(x) = a0 + a1x + · · · .
Write down the system of linear equations you need to solve toobtain a0, a1, . . ..Use MATLAB to set up and solve this linear system.
As the interpolating polynomial is of degree 3, then
P(x) = a0 + a1x + a2x2 + a3x3.
We obtain the following system of linear equations
Part I: Linear Systems Part II: Least Squares & Polynomial Interpolation Part III: Nonlinear Equations Part IV: Numerical Differentiation and Integration Part V: Ordinary Differential Equations Part VI: Partial Differential Equations
Assume that we want to find P in the form
P(x) = a0 + a1x + · · · .
Write down the system of linear equations you need to solve toobtain a0, a1, . . ..Use MATLAB to set up and solve this linear system.
Use the backslash command to solve for a to obtain the followingsolution
Part I: Linear Systems Part II: Least Squares & Polynomial Interpolation Part III: Nonlinear Equations Part IV: Numerical Differentiation and Integration Part V: Ordinary Differential Equations Part VI: Partial Differential Equations
Write down the Lagrange polynomials `j(x) for j = 0, 1, 2, 3.
Recall that the Lagrange polynomials are the degree n polynomials
`j(x) =n∏
k=0k 6=j
x − xkxj − xk
.
For the given data x0 = 0, x1 = 1, x2 = 2, x3 = 3, this gives
Part I: Linear Systems Part II: Least Squares & Polynomial Interpolation Part III: Nonlinear Equations Part IV: Numerical Differentiation and Integration Part V: Ordinary Differential Equations Part VI: Partial Differential Equations
Fora function f , the following data are known:
f (0) = 12.6, f (1) = 6.7, f (2) = 4.3, f (3) = 2.7.
Write down the interpolating polynomial P using the Lagrangepolynomials.
Part I: Linear Systems Part II: Least Squares & Polynomial Interpolation Part III: Nonlinear Equations Part IV: Numerical Differentiation and Integration Part V: Ordinary Differential Equations Part VI: Partial Differential Equations
(Theorem) Interpolating polynomial errorIf f is (n + 1) times continuously differentiable on the interval[a, b], then the error in approximating f (x) by p(x) is
Part I: Linear Systems Part II: Least Squares & Polynomial Interpolation Part III: Nonlinear Equations Part IV: Numerical Differentiation and Integration Part V: Ordinary Differential Equations Part VI: Partial Differential Equations
Chebyshev points
Choose xj to minimise
maxx∈[−1,1]
n∏j=0
(x − xj).
On [−1, 1], set tj = cos[(
2n + 1− 2j2n + 2
)π
]for j = 0, . . . , n.
On [a, b], set xj = a + b2 +
(b − a
2
)tj for j = 0, . . . , n.
Chebyshev nodes are the zeros of the Chebyshev polynomialTn+1(x).
Interpolation error is minimised by choosing Chebyshev nodes!
Part I: Linear Systems Part II: Least Squares & Polynomial Interpolation Part III: Nonlinear Equations Part IV: Numerical Differentiation and Integration Part V: Ordinary Differential Equations Part VI: Partial Differential Equations
Part I: Linear Systems Part II: Least Squares & Polynomial Interpolation Part III: Nonlinear Equations Part IV: Numerical Differentiation and Integration Part V: Ordinary Differential Equations Part VI: Partial Differential Equations
Nonlinear equation in standard form
f (x) = 0, x ∈ R.
We aim to solve for x (ie find the zeros of f )
If necessary, rearrange the equation to have the equation instandard form.
Part I: Linear Systems Part II: Least Squares & Polynomial Interpolation Part III: Nonlinear Equations Part IV: Numerical Differentiation and Integration Part V: Ordinary Differential Equations Part VI: Partial Differential Equations
NotationContinuity and differentiability
If f ∈ Cn([a, b]), then f is continuous on [a, b] and n timesdifferentiable on the interval (a, b).
Results of continuity
(Intermediate Value Theorem)If f ∈ C([a, b]) and f (a)f (b) < 0, then there exists at least onezero of f in the interval on (a, b).
(Strictly monotone)If f ′(x) > 0 OR f ′(x) < 0 for all x ∈ (a, b), then f is strictlymonotone on the interval [a, b].
(Uniqueness of root)If f is continuous AND strictly monotone on the interval [a, b] andf (a)f (b) < 0, then f has a unique root on (a, b).
Part I: Linear Systems Part II: Least Squares & Polynomial Interpolation Part III: Nonlinear Equations Part IV: Numerical Differentiation and Integration Part V: Ordinary Differential Equations Part VI: Partial Differential Equations
Iterative methods for solving equations
IterationsHave an initial guess or starting point x1.
Generate sequences of iterates xk for k = 2, 3, . . . based onapproximations of the problem.
Determine whether the sequence generated converges to the truesolution x∗.
Order of convergenceLargest ν such that
limk→∞
ek+1eνk
= β,
where β is the asymptotic constant and ek = |xk − x∗|.
Part I: Linear Systems Part II: Least Squares & Polynomial Interpolation Part III: Nonlinear Equations Part IV: Numerical Differentiation and Integration Part V: Ordinary Differential Equations Part VI: Partial Differential Equations
Iterative method: Bisection
Suppose that f (a)f (b) < 0 and we have f ∈ C([a, b]).
Take the midpoint xmid = a + b2 as x1.
Choose a new interval depending on the result of f (xmid.If f (a)f (xmid) < 0, then choose new interval to be [a, xmid].If f (xmid)f (b) > 0, then choose [xmid, b].
Iterate using the same process.
Order of convergence
Linear convergence with asymptotic constant β = 12 .
Part I: Linear Systems Part II: Least Squares & Polynomial Interpolation Part III: Nonlinear Equations Part IV: Numerical Differentiation and Integration Part V: Ordinary Differential Equations Part VI: Partial Differential Equations
Iterative method: Fixed point iteration
Given a starting point x1, compute
xk+1 = g(xk), for k = 1, 2, . . .
Order of convergence
If g ∈ C1([a, b]) and there exists a K ∈ (0, 1) such that |g ′(x)| ≤ Kfor all x ∈ (a, b), then the fixed point iteration converges linearlywith asymptotic constant β ≤ K , for any x1 ∈ [a, b].
Part I: Linear Systems Part II: Least Squares & Polynomial Interpolation Part III: Nonlinear Equations Part IV: Numerical Differentiation and Integration Part V: Ordinary Differential Equations Part VI: Partial Differential Equations
Iterative method: Newton’s method
Newton’s approximation
Approximate f (x) by its tangent at the point (xk , f (xk)) to form
f (x) ≈ f (xk) + (x − xk)f ′(xk).
Choose x = xk+1 and set f (x) = 0 to form
xk+1 = xk −f (xk)f ′(xk) , assuming f ′(xk) 6= 0.
Order of convergence
If f ∈ C2([a, b]) and x1 is sufficiently close to a simple rootx∗ ∈ (a, b), then Newton’s method converges quadratically to x∗.
Part I: Linear Systems Part II: Least Squares & Polynomial Interpolation Part III: Nonlinear Equations Part IV: Numerical Differentiation and Integration Part V: Ordinary Differential Equations Part VI: Partial Differential Equations
Example: (MATH2089, S1 2009)To find the root of a real number, computers typically implementNewton’s method. Let a > 1 and consider finding the cube root ofa, that is a1/3.Show that Newton’s method can be written as
Part I: Linear Systems Part II: Least Squares & Polynomial Interpolation Part III: Nonlinear Equations Part IV: Numerical Differentiation and Integration Part V: Ordinary Differential Equations Part VI: Partial Differential Equations
We want to find x = a1/3 =⇒ x3 − a = 0. So set f (x) = x3 − a.We then have
Part I: Linear Systems Part II: Least Squares & Polynomial Interpolation Part III: Nonlinear Equations Part IV: Numerical Differentiation and Integration Part V: Ordinary Differential Equations Part VI: Partial Differential Equations
Example: (MATH2089, S1 2013, Q1d)Consider the function f (x) = ex sin(x)− 100.
You are given that f (x) has a simple zero at x∗ ≈ 6.443. If you usea starting value x1 near x∗, what is the expected order ofconvergence for Newton’s method?
The function behaves well since ex and sin(x) are continuousfunctions. So it passes the theorem! Hence the expected order ofconvergence is 2.
Part I: Linear Systems Part II: Least Squares & Polynomial Interpolation Part III: Nonlinear Equations Part IV: Numerical Differentiation and Integration Part V: Ordinary Differential Equations Part VI: Partial Differential Equations
Iterative method: Secant method
Approximate f (x) by a line through the point (xk , f (xk)) and(xk−1, f (xk−1)) to form
f (x) ≈ f (xk) + (x − xk) f (xk)− f (xk−1)xk − xk−1
.
Choose x = xk+1 and set f (x) = 0 to form
xk+1 = xk −f (xk)(xk − xk−1)f (xk)− f (xk−1) .
Order of convergence
If f ∈ C2([a, b]) and x1, x2 are sufficiently close to x∗, then the
secant method converges superlinearly with order ν = 1 +√
Part I: Linear Systems Part II: Least Squares & Polynomial Interpolation Part III: Nonlinear Equations Part IV: Numerical Differentiation and Integration Part V: Ordinary Differential Equations Part VI: Partial Differential Equations
Part I: Linear Systems Part II: Least Squares & Polynomial Interpolation Part III: Nonlinear Equations Part IV: Numerical Differentiation and Integration Part V: Ordinary Differential Equations Part VI: Partial Differential Equations
Taylor series
Taylor seriesA Taylor series is an infinite polynomial series that approximatesnon-polynomial functions by taking higher order derivatives centredaround a point x0.
Part I: Linear Systems Part II: Least Squares & Polynomial Interpolation Part III: Nonlinear Equations Part IV: Numerical Differentiation and Integration Part V: Ordinary Differential Equations Part VI: Partial Differential Equations
(Theorem)
Let f ∈ Cn+1([a, b]). In other words, let f be continuous on [a, b]and n + 1 times differentiable on (a, b). Then
f (x + h) =n∑
k=0
f (k)(x)k! hk + f (n+1)(ξ)
(n + 1)! hn+1
= f (x) + f ′(x)h + f ′′(x)2! h2 + · · ·+ f (n)(x)
Part I: Linear Systems Part II: Least Squares & Polynomial Interpolation Part III: Nonlinear Equations Part IV: Numerical Differentiation and Integration Part V: Ordinary Differential Equations Part VI: Partial Differential Equations
Finite difference methods
Forward difference approximationLet f ∈ C2([a, b]). That is, let f be twice differentiable in theinterval [a, b]. Then
f ′(x) = f (x + h)− f (x)h +O(h).
The roundoff error is
O( ε
h
)= ε
∣∣∣∣ f (x + h)− f (x)h
∣∣∣∣ .The truncation error is O(h) and the total error is O
Part I: Linear Systems Part II: Least Squares & Polynomial Interpolation Part III: Nonlinear Equations Part IV: Numerical Differentiation and Integration Part V: Ordinary Differential Equations Part VI: Partial Differential Equations
Finite difference methods
Central difference approximationLet f ∈ C4([a, b]|). That is, let f be four times differentiable onthe interval [a, b]. Then
Part I: Linear Systems Part II: Least Squares & Polynomial Interpolation Part III: Nonlinear Equations Part IV: Numerical Differentiation and Integration Part V: Ordinary Differential Equations Part VI: Partial Differential Equations
Quadrature rules
We are approximating integrals using weighted sums of functionsvalues. That is,
Part I: Linear Systems Part II: Least Squares & Polynomial Interpolation Part III: Nonlinear Equations Part IV: Numerical Differentiation and Integration Part V: Ordinary Differential Equations Part VI: Partial Differential Equations
Quadrature rules
We look at three quadrature rules:Trapezoidal rule
Part I: Linear Systems Part II: Least Squares & Polynomial Interpolation Part III: Nonlinear Equations Part IV: Numerical Differentiation and Integration Part V: Ordinary Differential Equations Part VI: Partial Differential Equations
Quadrature rule − Trapezoidal rule
QN(f ) = h( f0
2 + f1 + f2 + · · ·+ fN−1 + fN2
).
Approximate an integral using a bunch of trapeziums and sum upthe area under f (x) using the area of each trapezium.
The height h is fixed: h = b − aN .
Function values are fj = f (xj) for all j = 0, . . . ,N.
Weights are w0 = wN = h2 and wj = h for all h = 1, . . . ,N − 1.
Part I: Linear Systems Part II: Least Squares & Polynomial Interpolation Part III: Nonlinear Equations Part IV: Numerical Differentiation and Integration Part V: Ordinary Differential Equations Part VI: Partial Differential Equations
Quadrature rule − Trapezoidal rule
(Theorem) Error of trapezoidal rule
Let f ∈ C2([a, b]). Then
EN(f ) = −b − a12 h2f ′′(ξ),
for some unknown ξ ∈ [a, b].
Rate of convergence: EN(f ) = O(h2) or EN(f ) = O(N−2).
Part I: Linear Systems Part II: Least Squares & Polynomial Interpolation Part III: Nonlinear Equations Part IV: Numerical Differentiation and Integration Part V: Ordinary Differential Equations Part VI: Partial Differential Equations
Part I: Linear Systems Part II: Least Squares & Polynomial Interpolation Part III: Nonlinear Equations Part IV: Numerical Differentiation and Integration Part V: Ordinary Differential Equations Part VI: Partial Differential Equations
Quadrature rule − Simpson’s rule
(Theorem) Error of Simpson’s rule
Let f ∈ C4([a, b]). Then
EN(f ) = −b − a180 h4f (4)(ξ),
for some unknown ξ ∈ [a, b].
Rate of convergence: EN(f ) = O(h4) or EN(f ) = O(N−4).
Part I: Linear Systems Part II: Least Squares & Polynomial Interpolation Part III: Nonlinear Equations Part IV: Numerical Differentiation and Integration Part V: Ordinary Differential Equations Part VI: Partial Differential Equations
Quadrature rule − Gauss-Legendre rule
∫ 1
−1f (x) dx ≈ QN(f ) =
N∑j=1
wj f (xj).
Nodes xj are the zeros of the Legendre polynomial of degree N on[−1, 1].
Weights wj are given in terms of the Legendre polynomials.
Part I: Linear Systems Part II: Least Squares & Polynomial Interpolation Part III: Nonlinear Equations Part IV: Numerical Differentiation and Integration Part V: Ordinary Differential Equations Part VI: Partial Differential Equations
Quadrature rule − Gauss-Legendre rule
(Theorem) Error of Gauss-Legendre rule
Let f ∈ C2N([−1, 1]). Then
EN(f ) = − eN(2N)! f (2N)(ξ),
where ξ ∈ [−1, 1] and eN is some number that depends on N.
Part I: Linear Systems Part II: Least Squares & Polynomial Interpolation Part III: Nonlinear Equations Part IV: Numerical Differentiation and Integration Part V: Ordinary Differential Equations Part VI: Partial Differential Equations
Quadrature properties
Quadratures assume integrand f is sufficiently smooth on [a, b].Assume that f ∈ C2([a, b]) for trapezoidal rule.Assume that f ∈ C4([a, b]) for Simpson’s rule.Assume that f ∈ C2N([a, b]) for Gauss-Legendre rule.
Part I: Linear Systems Part II: Least Squares & Polynomial Interpolation Part III: Nonlinear Equations Part IV: Numerical Differentiation and Integration Part V: Ordinary Differential Equations Part VI: Partial Differential Equations
Part I: Linear Systems Part II: Least Squares & Polynomial Interpolation Part III: Nonlinear Equations Part IV: Numerical Differentiation and Integration Part V: Ordinary Differential Equations Part VI: Partial Differential Equations
Tips for estimating difficult integrals
Unbounded derivatives: apply a change of variables.
Discontinuity on derivative: split the integral and remove thediscontinuous derivative.
High oscillatory: requires a special analytic method.
Narrow spike: either underestimate or overestimate the spikes.
Part I: Linear Systems Part II: Least Squares & Polynomial Interpolation Part III: Nonlinear Equations Part IV: Numerical Differentiation and Integration Part V: Ordinary Differential Equations Part VI: Partial Differential Equations
Part I: Linear Systems Part II: Least Squares & Polynomial Interpolation Part III: Nonlinear Equations Part IV: Numerical Differentiation and Integration Part V: Ordinary Differential Equations Part VI: Partial Differential Equations
First order Initial Value Problems (IVP)
First order ODEOrdinary differential equations are equations that involve itsderivative. A first order differential equation is one such equationwhere the highest order of derivative is 1. A first order initial valueproblem is simply a first order ODE with initial conditions.
Part I: Linear Systems Part II: Least Squares & Polynomial Interpolation Part III: Nonlinear Equations Part IV: Numerical Differentiation and Integration Part V: Ordinary Differential Equations Part VI: Partial Differential Equations
Existence and uniqueness of solutions
(Theorem)
If f (t, y) and ∂f (t, y)∂y are continuous and bounded for all
t ∈ [t0, tmax] and y ∈ R, then the IVP has a unique solution in thetime interval [t0, tmax].
Part I: Linear Systems Part II: Least Squares & Polynomial Interpolation Part III: Nonlinear Equations Part IV: Numerical Differentiation and Integration Part V: Ordinary Differential Equations Part VI: Partial Differential Equations
Euler’s method
Solve a first order IVP y ′ = f (t, y), t ∈ [t0, tmax], y(t0) = y0by
yn+1 = yn + h · f (tn, yn), n = 0, 1, . . . ,N − 1.
Part I: Linear Systems Part II: Least Squares & Polynomial Interpolation Part III: Nonlinear Equations Part IV: Numerical Differentiation and Integration Part V: Ordinary Differential Equations Part VI: Partial Differential Equations
System of first order ODEs
Often times, we may have a system of many equations involvingderivatives.
Part I: Linear Systems Part II: Least Squares & Polynomial Interpolation Part III: Nonlinear Equations Part IV: Numerical Differentiation and Integration Part V: Ordinary Differential Equations Part VI: Partial Differential Equations
Example: (MATH2089, 2010 Q2b)Consider the initial value problem (IVP)
y ′′′ + 2y ′ − (π2 + 1)y = π(π2 + 1)e−t sin(πt)y(0) = 1, y ′(0) = −1, y ′′(0) = 1− π2.
Reformulate the IVP as a system of first-order differential equations
Part I: Linear Systems Part II: Least Squares & Polynomial Interpolation Part III: Nonlinear Equations Part IV: Numerical Differentiation and Integration Part V: Ordinary Differential Equations Part VI: Partial Differential Equations
Begin by observing that the degree of derivative is 3. Hence, weuse
Part I: Linear Systems Part II: Least Squares & Polynomial Interpolation Part III: Nonlinear Equations Part IV: Numerical Differentiation and Integration Part V: Ordinary Differential Equations Part VI: Partial Differential Equations
So we obtain
x′ = f(t, x) =
x2x3
π(π2 + 1)e−t sin(πt)− 2x2 + (π2 + 1)x1
.To find the appropriate initial conditions, take t = 0 to get
Part I: Linear Systems Part II: Least Squares & Polynomial Interpolation Part III: Nonlinear Equations Part IV: Numerical Differentiation and Integration Part V: Ordinary Differential Equations Part VI: Partial Differential Equations
Part I: Linear Systems Part II: Least Squares & Polynomial Interpolation Part III: Nonlinear Equations Part IV: Numerical Differentiation and Integration Part V: Ordinary Differential Equations Part VI: Partial Differential Equations
ExampleFind the solution of the initial value problem
y ′ = 3y + 3t, y(0) = 1, t = 0.2
,Using Euler’s method with h = 0.2.
Using the fourth-order Runge Kutta method with h = 0.2.
Part I: Linear Systems Part II: Least Squares & Polynomial Interpolation Part III: Nonlinear Equations Part IV: Numerical Differentiation and Integration Part V: Ordinary Differential Equations Part VI: Partial Differential Equations
Using Euler’s method:We observe that, here, f (tn, yn) = 3y + 3t.Next, using our initial value of y(t0) = y0 =⇒ y(0) = 1, we obtaint0 = 0 and y0 = 1.Next, we want to find y0.2 given t = 0.2.
Part I: Linear Systems Part II: Least Squares & Polynomial Interpolation Part III: Nonlinear Equations Part IV: Numerical Differentiation and Integration Part V: Ordinary Differential Equations Part VI: Partial Differential Equations
Using the fourth-order Runge Kutta method:Observe that
y0.2 = y0 + h6 [k1 + 2k2 + 2k3 + k4] .
Calculate the values of k1, k2, k3 and k4 respectively.
Part I: Linear Systems Part II: Least Squares & Polynomial Interpolation Part III: Nonlinear Equations Part IV: Numerical Differentiation and Integration Part V: Ordinary Differential Equations Part VI: Partial Differential Equations
Other useful methods for solving IVPsTaylor method of order 2
yn+1 = yn + hf (tn, yn) + h2
2∂f (t, y)∂t
∣∣∣t=tn,y=tn
Implicit Euler’s method
yn+1 = yn + hf (tn+1, yn+1)
Trapezoidal method
yn+1 = yn + h2 [f (tn, yn) + f (tn+1, yn+1]
Heun’s method
yn+1 = yn + h2 [f (tn, yn) + f (tn+1, yn + hf (tn, yn))]
Part I: Linear Systems Part II: Least Squares & Polynomial Interpolation Part III: Nonlinear Equations Part IV: Numerical Differentiation and Integration Part V: Ordinary Differential Equations Part VI: Partial Differential Equations
Part I: Linear Systems Part II: Least Squares & Polynomial Interpolation Part III: Nonlinear Equations Part IV: Numerical Differentiation and Integration Part V: Ordinary Differential Equations Part VI: Partial Differential Equations
Partial Differential Equations
Partial Differential Equations (PDEs) are functions of more thanone variable defined by equations involving their partial derivatives.
Order of the PDE is the order of the highest derivative present!
Part I: Linear Systems Part II: Least Squares & Polynomial Interpolation Part III: Nonlinear Equations Part IV: Numerical Differentiation and Integration Part V: Ordinary Differential Equations Part VI: Partial Differential Equations
Finite difference methods
Treat them no differently to functions of one variable.
The only difference is changing the variable in the derivative!
Part I: Linear Systems Part II: Least Squares & Polynomial Interpolation Part III: Nonlinear Equations Part IV: Numerical Differentiation and Integration Part V: Ordinary Differential Equations Part VI: Partial Differential Equations
Types of PDEsWe will restrict ourselves to PDEs involving only two variables. Asecond order quasi-linear PDE is of the form
Part I: Linear Systems Part II: Least Squares & Polynomial Interpolation Part III: Nonlinear Equations Part IV: Numerical Differentiation and Integration Part V: Ordinary Differential Equations Part VI: Partial Differential Equations
Elliptic PDEs
Divide x interval [0, Lx ] into m + 1 equal length subintervals suchthat
hx = Lxm + 1 .
Divide y interval [0, Ly ] into n + 1 equal length subintervals suchthat
hy = Lyn + 1 .
Central difference approximation of O(h2) at the grid points xi
Part I: Linear Systems Part II: Least Squares & Polynomial Interpolation Part III: Nonlinear Equations Part IV: Numerical Differentiation and Integration Part V: Ordinary Differential Equations Part VI: Partial Differential Equations
Parabolic PDEs
Method 1 (Explicit method)
Forward difference approximation to the time derivative
∂u∂t (xi , t`) ≈
u`+1i − u`i
ht.
Central difference approximation to the space derivative
Part I: Linear Systems Part II: Least Squares & Polynomial Interpolation Part III: Nonlinear Equations Part IV: Numerical Differentiation and Integration Part V: Ordinary Differential Equations Part VI: Partial Differential Equations
Parabolic PDEs
Method 2 (Implicit method)
Backward difference approximation to the time derivative
∂u∂t (xi , t`+1) ≈ u`+1
i − u`iht
.
Central difference approximation to the space derivative
Part I: Linear Systems Part II: Least Squares & Polynomial Interpolation Part III: Nonlinear Equations Part IV: Numerical Differentiation and Integration Part V: Ordinary Differential Equations Part VI: Partial Differential Equations
Parabolic PDEs
Method 3 (Crank-Nicolson method)
Take the average between the explicit and implicit method to obtain
Part I: Linear Systems Part II: Least Squares & Polynomial Interpolation Part III: Nonlinear Equations Part IV: Numerical Differentiation and Integration Part V: Ordinary Differential Equations Part VI: Partial Differential Equations
Hyperbolic PDEs
Method (Explicit method)
Central difference approximation to the time derivative
∂2u∂t2 (xi , t`) ≈
u`−1i − 2u`i + u`+1
ih2
t.
Central difference approximation to the space derivative
∂2u∂x2 (xi , t`) ≈
u`i−1 − 2u`i + u`i+1h2
x.
Substitute into PDE and multiply through h2t to obtain
Part I: Linear Systems Part II: Least Squares & Polynomial Interpolation Part III: Nonlinear Equations Part IV: Numerical Differentiation and Integration Part V: Ordinary Differential Equations Part VI: Partial Differential Equations
A final problem
The heat conduction equation which models the temperature in aninsulated rod with ends held at constant temperatures can bewritten in the dimensionless form as
∂Θ(x , t)∂t = ∂2Θ(x , t)
∂x2
Write a finite difference approximation of this equation using theForward-Time, Central-Space scheme and rearrange it to be solvedby an explicit method.
Part I: Linear Systems Part II: Least Squares & Polynomial Interpolation Part III: Nonlinear Equations Part IV: Numerical Differentiation and Integration Part V: Ordinary Differential Equations Part VI: Partial Differential Equations