Numerical Methods for Computational Science and Engineering Numerical Methods for Computational Science and Engineering Lecture 1, Sept 19, 2013: Introduction Peter Arbenz Computer Science Department, ETH Z¨ urich E-mail: [email protected]NumCSE, Lecture 1, Sept 19, 2013 1/40
40
Embed
Numerical Methods for Computational Science and Engineering · PDF fileNumerical Methods for Computational Science and Engineering Introduction Outline of today’s lecture I What
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Numerical Methods for Computational Science and Engineering
Numerical Methods for Computational Scienceand Engineering
Lecture 1, Sept 19, 2013: Introduction
Peter ArbenzComputer Science Department, ETH Zurich
Numerical Methods for Computational Science and Engineering
Introduction
Literature (cont.)
W. Dahmen & A. Reusken: Numerik fur Ingenieure undNaturwissenschaftler, Springer, 2006.Good reference for large parts of this course; a lot of simple examples and lucid explanations, but also
rigorous mathematical treatment; Target audience: undergraduate students in science and engineering.
Available through Nebis.
H.-R. Schwarz & N. Kockler: Numerische Mathematik.Teubner, 2006. 6. Auflage.Main reference for large parts of this course; Target audience: undergraduate students in science and
engineering. Available through Nebis.
C. Moler: Numerical Computing with Matlab. SIAM 2004.Good reference for some parts of this course; Target audience: Matlab users and programmers. See
Numerical Methods for Computational Science and Engineering
Introduction
Prerequisites
Essential prerequisite for this course is a solid knowledge in linearalgebra and calculus. Familiarity with the topics covered in thefirst semester courses is taken for granted, see
• K. Nipp and D. Stoffer, Lineare Algebra, vdfHochschulverlag, Zurich, 5 ed., 2002.
• M. Gutknecht, Lineare algebra, lecture notes, SAM, ETHZurich, 2009.http://www.sam.math.ethz.ch/~mhg/unt/LA/HS07/.
• M. Struwe, Analysis fur Informatiker. Lecture notes, ETHZurich, 2009.
Numerical Methods for Computational Science and Engineering
Numerical algorithms and errors
Numerical algorithms and errors
I The most fundamental feature of numerical computing is theinevitable presence of errors.
I The result of any interesting computation (and of manyuninteresting ones) is typically only approximate, and our goalis to ensure that the resulting error is tolerably small.
I Example: How many loop iterations are there in this littleMatlab program?
x = 0; h = 1/10;
while x<1,
x=x+h;
% do something depending on x
end
NumCSE, Lecture 1, Sept 19, 2013 17/40
Numerical Methods for Computational Science and Engineering
Numerical algorithms and errors
How to measure errors
I Can measure errors as absolute or relative, or a combinationof both.
I The absolute error in v approximating given scalar quantity uis |u − v |.
I The relative error (assuming u 6= 0) is|u − v ||u| .
Numerical Methods for Computational Science and Engineering
Numerical algorithms and errors
Types of errors
I Errors in the formulation of the problem to be solved.I Errors in the mathematical model. Simplifications.I Error in input data. Measurements.
I Approximation errorsI Discretization error.I Convergence error in iterative methods.I Discretization/convergence errors may be assessed by an
analysis of the method used.
I Roundoff errorsI Roundoff errors arise everywhere in numerical computation
because of the finite precision arithmetic.I Roundoff errors behave quite erratic.
NumCSE, Lecture 1, Sept 19, 2013 20/40
Numerical Methods for Computational Science and Engineering
Numerical algorithms and errors
Discretization errors in action
Problem: want to approximate the derivative f ′(x0) of a givensmooth function f (x) at the point x = x0.
Example: Let f (x) = sin(x), −∞ < x <∞, and set x0 = 1.2.Thus, f (x0) = sin(1.2) ≈ 0.932 . . ..
Discretization: Function values f (x) are available only at a discretenumber of points, e.g. at grid points xj = x0 + jh, j ∈ Z.
Want to approximate f ′(x0) by values f (xj).
NumCSE, Lecture 1, Sept 19, 2013 21/40
Numerical Methods for Computational Science and Engineering
Numerical algorithms and errors
Discretization errors in action (cont.)Taylor’s series gives us an algorithm to approximate f ′(x0):
f ′(x0) ≈ Dx0,h(f ) =f (x0 + h)− f (x0)
h
NumCSE, Lecture 1, Sept 19, 2013 22/40
Numerical Methods for Computational Science and Engineering
Numerical algorithms and errors
Discretization errors in action (cont.)Expanding f (x) by a Taylor series around x = x0 gives
f (x0 + h)− f (x0)
h= f ′(x0)− h
2f ′′(ξ), x0 < ξ < x0 + h.
So, we expect the error to decrease linearly with h.
∣∣∣∣f ′(x0)− f (x0 + h)− f (x0)
h
∣∣∣∣ =h
2
∣∣f ′′(ξ)∣∣ ≈ h
2
∣∣f ′′(x0)∣∣
Or, using the big-O notation:∣∣f ′(x0)− Dx0,h(f )∣∣ = O(h).
NumCSE, Lecture 1, Sept 19, 2013 23/40
Numerical Methods for Computational Science and Engineering
Numerical algorithms and errors
Results
Try for f (x) = sin(x) at x0 = 1.2.
(So, we are approximating cos(1.2) = 0.362357754476674 . . .)
h Absolute error
0.1 4.71667 · 10−2
0.01 4.666196 · 10−3
0.001 4.660799 · 10−4
10−4 4.660256 · 10−5
10−7 4.619326 · 10−8
These results reflect the discretization error as expected.
Note that f ′′(x0)/2 = − sin(1.2)/2 ≈ −0.466.
NumCSE, Lecture 1, Sept 19, 2013 24/40
Numerical Methods for Computational Science and Engineering
Numerical algorithms and errors
Results for smaller h
The above results indicate that we can compute the derivative asaccurate as we like, provided that we take h small enough.
If we wanted∣∣∣∣cos(1.2)− sin(1.2 + h)− sin(1.2)
h
∣∣∣∣ < 10−10.
We have to set h ≤ 10−10/0.466.
The following numbers and plot are generated by.../Greif/programs/chap01/Example1 3Figure1 3.m
NumCSE, Lecture 1, Sept 19, 2013 25/40
Numerical Methods for Computational Science and Engineering
Numerical algorithms and errors
Results for smaller h
h Absolute error
10−8 4.36105 · 10−10
10−9 5.594726 · 10−8
10−10 1.669696 · 10−7
10−11 7.938531 · 10−6
10−13 6.851746 · 10−4
10−15 8.173146 · 10−2
10−16 3.623578 · 10−1
These results reflect both discretization and roundoff errors.
NumCSE, Lecture 1, Sept 19, 2013 26/40
Numerical Methods for Computational Science and Engineering
Numerical algorithms and errors
Results for all h
The solid curve interpolates the computed values of|f ′(x0)− f (x0+h)−f (x0)
h | for f (x) = sin(x) at x0 = 1.2.
The dash-dotted straight line depicts the discretization errorwithout roundoff error.
NumCSE, Lecture 1, Sept 19, 2013 27/40
Numerical Methods for Computational Science and Engineering
Algorithm properties
Algorithm properties
Performance features that may be expected from a good numericalalgorithm.
I AccuracyRelates to errors. How accurate is the result going to be whena numerical algorithm is run with some particular input data.
I EfficiencyI How fast can we solve a certain problem?
Rate of convergence. Floating point operations (flops).I How much memory space do we need?I These issues may affect each other.
I Robustness(Numerical) software should run under all circumstances.Should yield correct results to within an acceptable error orshould fail gracefully if not successful.
NumCSE, Lecture 1, Sept 19, 2013 28/40
Numerical Methods for Computational Science and Engineering
Algorithm properties
Complexity I
Complexity/computational cost of an algorithm :⇐⇒ number ofelementary operatorsAsymptotic complexity = “leading order term” of complexity w.r.t.large problem size parametersThe usual choice of problem size parameters in numerical linearalgebra is the number of independent real variables needed todescribe the input data (vector length, matrix sizes).
operation description #mul/div #add/sub
inner product (x ∈ Rn, y ∈ Rn) 7→ xHy n n − 1 O(n)outer product (x ∈ Rm, y ∈ Rn) 7→ xyH nm 0 O(mn)tensor product
matrix product (A ∈ Rm×n,B ∈ Rn×k) 7→ AB mnk mk(n − 1) O(mnk)
NumCSE, Lecture 1, Sept 19, 2013 29/40
Numerical Methods for Computational Science and Engineering
Algorithm properties
Big-O and Θ notation
For an error e depending on h we denote
e = O(hq)
if there are two positive constants q and C such that
|e| ≤ C hq ∀h > 0 small enough.
Similarly, for w = w(n) the expression
w = O(n log n)
means that there is a constant C > 0 such that
|w | ≤ Cn log n as n→∞.
NumCSE, Lecture 1, Sept 19, 2013 30/40
Numerical Methods for Computational Science and Engineering
Algorithm properties
Big-O and Θ notation
More abstract:Class O(f ) of functions is defined as
The Θ notation signifies a stronger relation than the O notation:a function φ(h) for small h (resp., φ(n) for large n) is Θ(ψ(h))(resp., Θ(ψ(n))) if φ is asymptotically bounded both above andbelow by ψ.
Example:O(h2) means at least “quadratic convergence” (see later). Θ(h2)is exact quadratic convergence.
NumCSE, Lecture 1, Sept 19, 2013 31/40
Numerical Methods for Computational Science and Engineering
Algorithm properties
Complexity II
To a certain extent, the asymptotic complexity allows to predictthe dependence of the runtime of a particular implementation ofan algorithm on the problem size (for large problems). Forinstance, an algorithm with asymptotic complexity O(n2) is likelyto take 4× as much time when the problem size is doubled.
One may argue that the memory accesses are more decisive for runtimes than floating point operations. In general there is a lineardependence among the two. So, there is no difference in the Onotation.
NumCSE, Lecture 1, Sept 19, 2013 32/40
Numerical Methods for Computational Science and Engineering
Elementary operations
Scaling
Scaling ≡ multiplication with diagonal matrices (with non-zerodiagonal entries) from left and/or right.
It is important to know the different effects of multiplying with adiagonal matrix from left or right:
DA vs. AD with Rn×n 3 A,D = diag(d1, . . . , dn)
Scaling with D = diag(d1, . . . , dn)in Matlab:
y = diag(d)*x;
or
y = d.*x;
NumCSE, Lecture 1, Sept 19, 2013 33/40
Numerical Methods for Computational Science and Engineering
Elementary operations
Elementary matrices
Matrices of the form A = I + αuvT are called elementary.Again we can apply A to a vector x in a straightforward and amore clever way:
Ax = (I + αuvT )x
orAx = x + αu(vTx)
Cf. exercises.
NumCSE, Lecture 1, Sept 19, 2013 34/40
Numerical Methods for Computational Science and Engineering
Elementary operations
Problem conditioning and algorithm stability
Qualitatively speaking:
I The problem is ill-conditioned if a small perturbation in thedata may produce a large difference in the result.The problem is well-conditioned otherwise.
I The algorithm is stable if its output is the exact result of aslightly perturbed input.
NumCSE, Lecture 1, Sept 19, 2013 35/40
Numerical Methods for Computational Science and Engineering
Elementary operations
An unstable algorithm
Ill-conditioned problem of computing output values y from inputvalues x by y = g(x): when x is slightly perturbed to x , the resulty = g(x) is far from y .
NumCSE, Lecture 1, Sept 19, 2013 36/40
Numerical Methods for Computational Science and Engineering
Elementary operations
A stable algorithm
An instance of a stable algorithm for computing y = g(x): theoutput y is the exact result, y = g(x), for a slightly perturbedinput, i.e., x which is close to the input x . Thus, if the algorithm isstable and the problem is well-conditioned, then the computedresult y is close to the exact y .
NumCSE, Lecture 1, Sept 19, 2013 37/40
Numerical Methods for Computational Science and Engineering
Elementary operations
Unstable algorithm
Problem statement: evaluate the integrals
yn =
∫ 1
0
xn
x + 10dx , for n = 0, 1, 2, . . . , 30.
Algorithm development: observe that analytically, for n > 0,
yn + 10yn−1 =
∫ 1
0
xn + 10xn−1
x + 10dx =
∫ 1
0xn−1 dx =
1
n.
Also,
y0 =
∫ 1
0
1
x + 10dx = log(11)− log(10).
Algorithm:
I Evaluate y0 = log(11)− log(10).I For n = 1, 2, . . . , 30, evaluate yn = 1
n − 10yn−1.NumCSE, Lecture 1, Sept 19, 2013 38/40
Numerical Methods for Computational Science and Engineering
Elementary operations
Unstable algorithm (cont.)Run the Matlab program Example1 6.m by Ascher and Greif tosee the catastrophic amplification of roundoff errors.This code is available fromhttp://www.siam.org/books/cs07/programs.zip.