Automatic Differentiation in Chebfun for Solution of ...

PrefaceThe Chebfun system

Functional Newton iterationAutomatic differentiation in Chebfun

Chebops for automatic iterations for BVPs

Automatic Differentiation in Chebfunfor Solution of Nonlinear Boundary-Value Problems

Asgeir [email protected]

Collaboration with Prof. Toby Driscoll, Univ. of Delawareand the Chebfun team

University of Oxford, Mathematical Institute, Numerical Analysis Group

Eleventh European Workshop on Automatic DifferentiationDecember, 2010

Asgeir Birkisson Automatic Frechet Differentiation




To solve the nonlinear BVP

0.05u′′ + u′u + cos u = 0, 0 < x < 2, u(0) = 1, u′(2) = −1

2

for the function u = u(x), we only need to execute the followingfour lines of Chebfun code in Matlab :

phi = @(u) 0.05*diff(u,2) + diff(u).*u + cos(u);

alpha = @(u) u-1; beta = @(u) diff(u)+1/2;

N = chebop(domain(0,2),phi,alpha,beta);

u = N\0;

The code runs in 1.0 seconds on a tri-core 2.5 GHz workstation,and yields an accuracy of 9.7 · 10−10. How do we achieve this?






0.05u′′ + u′u + cos u = 0, 0 < x < 2, u(0) = 1, u′(2) = −1

2





u = N\0;







0.05u′′ + u′u + cos u = 0, 0 < x < 2, u(0) = 1, u′(2) = −1

2





u = N\0;






0 0.5 1 1.5 20.9

1

1.1

1.2

1.3

1.4

1.5Solution

x

u(x)

1 3 5 710

−14

10−12

10−10

10−8

10−6

10−4

10−2

100

Norm of update

Iteration number





Table of Contents

1 Preface

2 The Chebfun system

3 Functional Newton iteration

4 Automatic differentiation in Chebfun

5 Chebops for automatic iterations for BVPs





BackgroundchebfunsLinear operators in Chebfun – linops

Chebfun

From the Chebfun website [5] (first result for a Google search on“chebfun”):

Chebfun is a collection of algorithms, and a software system inobject-oriented Matlab , which extends familiar powerfulmethods of numerical computation involving numbers tocontinuous or piecewise-continuous functions.

It also implements continuous analogues of linear algebranotions like the QR decomposition and the SVD, and solvesordinary differential equations.

The mathematical basis of the system combines tools ofChebyshev expansions, fast Fourier transform, barycentricinterpolation, recursive zerofinding, and automaticdifferentiation.






Chebfun










Chebfun










Chebfun










Chebfun demo

In the Chebfun system, we work with functions defined on aninterval [a, b]; the functions are represented by interpolants insuitably rescaled Chebyshev points {xj}nj=0, such that

xj = cos(jπ/n), 0 ≤ j ≤ n.

By overloading common Matlab functions, (and introducingnew ones), we can carry out computations with chebfuns.

chebfun demo






Chebfun demo


xj = cos(jπ/n), 0 ≤ j ≤ n.


chebfun demo






Chebfun demo


xj = cos(jπ/n), 0 ≤ j ≤ n.


chebfun demo






Linear operators in Chebfun – linops

When working with vectors, we apply matrices to form newvectors.

In the functional settings, these becomes linear operators.

In Chebfun, the class linops represents such operators [4].

Linops can amongst other uses be used to representdifferentiation (diff), integration (cumsum) andmultiplication (diag) operators.

A linop can be “expanded” to form a matrix which is thediscretization of the operator it represents on thecorresponding Chebyshev grid with the same number ofpoints.


















































Linop demo

Linops operate in forward mode (*) and reverse mode (\).

By appending boundary conditions, we can solve linear BVPswith linops.

Linop demo






Linop demo



Linop demo






Linop demo



Linop demo





IntroductionNewton iteration for rootfindingNewton iteration for BVPsFrechet derivatives

1 Preface










Newton iteration

We can use our methods for solving linear problems as a basisfor nonlinear ones.

The method is known as Newton-Kantorovich iteration orNewton iteration in function space.

Based on similar ideas as the well-known Newton iteration forrootfinding.






Newton iteration









Newton iteration









Newton iteration

Recall that we use Newton iteration when we wish to find aroot x∗ of a function f , i.e. x∗ such that f (x∗) = 0.

Starting from an initial guess x (0), we find a sequence ofguesses of the solution. Members of the sequence are definedvia

x (k+1) − x (k) = −[f ′(x (k)

)]−1f(x (k)

), (1)

where f ′(x) is the Jacobian matrix of partial derivatives∂fi/∂xj .

Rigorously justifiable using Taylor series.

For a sufficiently good initial guess, Newton iterationconverges quadratically to a root.






Newton iteration



x (k+1) − x (k) = −[f ′(x (k)

)]−1f(x (k)

), (1)









Newton iteration



x (k+1) − x (k) = −[f ′(x (k)

)]−1f(x (k)

), (1)









Newton iteration



x (k+1) − x (k) = −[f ′(x (k)

)]−1f(x (k)

), (1)









Newton iteration for BVPs

Now assume we want to solve the nonlinear BVP

φ(u) = 0,

α(u)∣∣x=a

= 0, β(u)∣∣x=b

= 0,

where u is a function (i.e. u = u(x)) and φ, α, and β areoperators between suitable normed function spaces;presumably, φ is a differential operator, and α and β may alsobe differential operators of lower order.

Letting N denote a BVP operator, we also refer to thisproblem simply as

N (u) = 0







Now assume we want to solve the nonlinear BVP

φ(u) = 0,

α(u)∣∣x=a

= 0, β(u)∣∣x=b

= 0,

where u is a function (i.e. u = u(x)) and φ, α, and β areoperators between suitable normed function spaces;presumably, φ is a differential operator, and α and β may alsobe differential operators of lower order.

Letting N denote a BVP operator, we also refer to thisproblem simply as

N (u) = 0







We generalize the Newton iteration to produce a sequence offunctions u1, u2, . . . to find a u∗ such that N (u∗) = 0.

The crucial component of this iteration is the linearization ofN about each uk .

This process requires the Frechet derivatives of φ, α, and β(derivatives with respect to functions).

























Definition

Let V and W be Banach spaces and let U be an open subset ofV . Then for φ : U →W , the Frechet derivative of φ at u ∈ U isdefined (when possible) as the unique linear operator A = φ′(u),A : V →W , such that

limv→0

‖φ(u + v)− φ(u)− Av‖W‖v‖V

= 0.

Note in this definition that v is a function, and we require that thelimit exists as v → 0 in any manner.







We obtain the linear boundary-value problem

φ′(uk)v = −φ(uk),[α′(uk)v

]x=a

= −α(uk)∣∣x=a

,[β′(uk)v

]x=b

= −β(uk)∣∣x=b

,

or using the notation introduced above,

N ′(uk)v = −N (uk).







We obtain the linear boundary-value problem

φ′(uk)v = −φ(uk),[α′(uk)v

]x=a

= −α(uk)∣∣x=a

,[β′(uk)v

]x=b

= −β(uk)∣∣x=b

,

or using the notation introduced above,

N ′(uk)v = −N (uk).







Solving the problem

N ′(uk)v = −N (uk).

for the function v to obtain the update vk , we define

uk+1 = uk + vk

and iterate until convergence criteria are satisfied.

Under suitable conditions, the Kantorovich Theoremguarantees quadratic convergence for suitably close initialguesses [3].







Solving the problem

N ′(uk)v = −N (uk).

for the function v to obtain the update vk , we define

uk+1 = uk + vk

and iterate until convergence criteria are satisfied.

Under suitable conditions, the Kantorovich Theoremguarantees quadratic convergence for suitably close initialguesses [3].





The use of ADOur AD implementationCode modificationsEvaluation of Frechet derivativesExample of evaluation

1 Preface










AD in Chebfun

Even though Frechet differentiation and normal differentiationw.r.t. a function are conceptually different, they both rely onsimilar rules from calculus (e.g. chain rule).

We can thus use automatic differentiation to calculate theFrechet derivatives of the operators involved – we chooseforward mode AD.

Since Chebfun already overloads Matlab methods, operatoroverloaded AD implementation was our choice.

Since Chebfun has many other uses from solving ODEs, weput a high emphasis on minimizing computational overheadfor all uses.






AD in Chebfun










AD in Chebfun










AD in Chebfun










AD in Chebfun

Leads to delayed evaluation – derivatives are not evaluateduntil explicitly required.

Obtained using anonymous functions.

Our implementation is recursive – we assign a unique ID toevery chebfun.

We go backwards in the evaluation trace until we get a matchof IDs.

Introduce two new fields to the chebfun objects, .jac and.ID.






AD in Chebfun











AD in Chebfun











AD in Chebfun











AD in Chebfun











Modifications to code

To enable AD in Chebfun, we only need to add a couple oflines to already overloaded methods.

Take for example the overloaded sin method.

The essential lines are

function fout = sin(fin)

fout = comp(fin, @(x) sin(x));

fout.jac = @(u) diag(cos(fin))*diff(fin,u);

fout.ID = newIDnum;

Here, the argument u has the meaning of the independentvariable we will be differentiating with respect to.

A call to diff with two chebfun arguments starts theevaluation process.













fout.ID = newIDnum;















fout.ID = newIDnum;















fout.ID = newIDnum;








Examples

Differentiate g w.r.t. fReturn −Msin(f) + M2xf = M2xf−sin(f)

Differentiate g1 w.r.t. fReturn −Msin(f)

Differentiate g2 w.r.t. fReturn M2xf

Differentiate cos(f) w.r.t. fReturn M− sin(f)I = −Msin(f)

Differentiate xf2 w.r.t. fReturn OMf2 + MxM2f = M2xf

Differentiate f w.r.t. fReturn I

Differentiate x w.r.t. fReturn O

Differentiate f2 w.r.t. fReturn M2fI = M2f

Differentiate f w.r.t. fReturn I

1

Figure: Recursive diff. of g with respect f , where f = sin(x),g1 = cos(f ), g2 = xf 2 and g = g1 + g2. In the figure, I stands for theidentity operator, M for a multiplication operator (such thatMh(x) : k(x) 7→ h(x)k(x)) and O for the zero operator on the domain ofthe functions.






Examples

Live demo





IntroductionExamples

1 Preface










Chebops

Combining the ideas of automatic differentiation, Newtoniteration and object oriented programming leads to thedevelopment of the chebop class.

A chebop is an object which contains all required informationto solve a nonlinear BVP.

Performs damped Newton iteration

uk+1 = uk + λkvk

where uk is the current guess of the solution, vk is theNewton update and λk is a damping parameter.

For discussion on the damping strategy, see [2, 1].






Examples

Live demo






Thank you for your attention!






A. Birkisson and T. Driscoll, Automatic Frechetdifferentiation for the numerical solution of boundary-valueproblems.

Submitted.

A. Birkisson and T. Driscoll, Tech. Rep. 10/15, OxfordUniversity Mathematical Institute, November 2010.

P. Deuflhard, Newton Methods for Nonlinear Problems,Springer-Verlag, Berlin/Heidelberg, 2006.

T. A. Driscoll, F. Bornemann, and L. N. Trefethen, Thechebop system for automatic solution of differential equations, BITNumerical Mathematics, 48 (2008), pp. 701–723.

L. N. Trefethen, N. Hale, R. B. Platte, T. A. Driscoll,and R. Pachon, Chebfun version 3.

Oxford University, 2009.

http://www.maths.ox.ac.uk/chebfun/.Asgeir Birkisson Automatic Frechet Differentiation

Existence in any manner

A limit is said to exist in any manner, if given ε > 0, there existsδ > 0 such that for ‖v‖V < δ

‖φ(u + v)− φ(u)− Av‖W ≤ ε‖v‖V .


Automatic Differentiation in Chebfun for Solution of ...

Documents