Top Banner
330

Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Jul 29, 2015

Download

Documents

Herib Blanco
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Page 1: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Computer Methods for Ordinary Di�erential

Equations and Di�erential�Algebraic

Equations

Uri M� Ascher and Linda R� Petzold

December �� ����

Page 2: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Preface

This book has been developed from course notes that we wrote� havingrepeatedly taught courses on the numerical solution of ordinary di�erentialequations �ODEs� and related problems� We have taught such courses at asenior undergraduate level as well as at the level of a �rst graduate course onnumerical methods for di�erential equations� The audience typically consistsof students from Mathematics� Computer Science and a variety of disciplinesin engineering and sciences such as Mechanical� Electrical and Chemical En�gineering� Physics� Earth Sciences� etc�

The material that this book covers can be viewed as a �rst course on thenumerical solution of di�erential equations� It is designed for people whowant to gain a practical knowledge of the techniques used today� The courseaims to achieve a thorough understanding of the issues and methods involvedand of the reasons for the successes and failures of existing software� On onehand� we avoid an extensive� thorough� theorem�proof type exposition� wetry to get to current methods� issues and software as quickly as possible�On the other hand� this is not a quick recipe book� as we feel that a deeperunderstanding than can usually be gained by a recipe course is required toenable the student or the researcher to use their knowledge to design theirown solution approach for any nonstandard problems they may encounter infuture work� The book covers initial�value and boundary�value problems� aswell as di�erential�algebraic equations �DAEs�� In a one�semester course wehave been typically covering over � of the material it contains�

We wrote this book partially as a result of frustration at not being ableto assign a textbook adequate for the material that we have found ourselvescovering� There is certainly excellent� in�depth literature around� In fact� weare making repeated references to exhaustive texts which� combined� coveralmost all the material in this book� Those books contain the proofs andreferences which we omit� They span thousands of pages� though� and thetime commitment required to study them in adequate depth may be morethan many students and researchers can a�ord to invest� We have tried tostay below a ���page limit and to address all three ODE�related areas men�

ii

Page 3: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

iii

tioned above� A signi�cant amount of additional material is covered in theExercises� Other additional important topics are referred to in brief sectionsof Notes and References� Software is an important and well�developed partof this subject� We have attempted to cover the most fundamental softwareissues in the text� Much of the excellent and publicly�available software isdescribed in the Software sections at the end of the relevant chapters� andavailable codes are cross�referenced in the index� Review material is high�lighted and presented in the text when needed� and it is also cross�referencedin the index�

Traditionally� numerical ODE texts have spent a great deal of time de�veloping families of higher order methods� e�g� Runge�Kutta and linear mul�tistep methods� applied �rst to nonsti� problems and then to sti� problems�Initial value problems and boundary value problems have been treated inseparate texts� although there is much in common� There have been fun�damental di�erences in approach� notation� and even in basic de�nitions�between ODE initial value problems� ODE boundary value problems� andpartial di�erential equations �PDEs��

We have chosen instead to focus on the classes of problems to be solved�mentioning wherever possible applications which can lend insight into therequirements and the potential sources of di culty for numerical solution�We begin by outlining the relevant mathematical properties of each problemclass� then carefully develop the lower�order numerical methods and funda�mental concepts for the numerical analysis� Next we introduce the appropri�ate families of higher�order methods� and �nally we describe in some detailhow these methods are implemented in modern adaptive software� An im�portant feature of this book is that it gives an integrated treatment of ODEinitial value problems� ODE boundary value problems� and DAEs� empha�sizing not only the di�erences between these types of problems but also thefundamental concepts� numerical methods and analysis which they have incommon� This approach is also closer to the typical presentation for PDEs�leading� we hope� to a more natural introduction to that important subject�

Knowledge of signi�cant portions of the material in this book is essentialfor the rapidly emerging �eld of numerical dynamical systems� These are nu�merical methods employed in the study of the long term� qualitative behaviorof various nonlinear ODE systems� We have emphasized and developed inthis work relevant problems� approaches and solutions� But we avoided de�veloping further methods which require deeper� or more speci�c� knowledgeof dynamical systems� which we did not want to assume as a prerequisite�

The plan of the book is as follows� Chapter � is an introduction to thedi�erent types of mathematical models which are addressed in the book�We use simple examples to introduce and illustrate initial� and boundary�value problems for ODEs and DAEs� We then introduce some importantapplications where such problems arise in practice�

Page 4: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

iv

Each of the three parts of the book which follow starts with a chapterwhich summarizes essential theoretical� or analytical issues �i�e� before ap�plying any numerical method�� This is followed by chapters which developand analyze numerical techniques� For initial value ODEs� which compriseroughly half this book� Chapter � summarizes the theory most relevant forcomputer methods� Chapter � introduces all the basic concepts and simplemethods �relevant also for boundary value problems and for DAEs�� Chapter� is devoted to one�step �Runge�Kutta� methods and Chapter discussesmultistep methods�

Chapters ��� are devoted to boundary value problems for ODEs� Chapter� discusses the theory which is essential to understand and to make e�ectiveuse of the numerical methods for these problems� Chapter � brie�y con�siders shooting�type methods and Chapter � is devoted to �nite di�erenceapproximations and related techniques�

The remaining two chapters consider DAEs� This subject has been re�searched and solidi�ed only very recently �in the past � years�� Chapter �is concerned with background material and theory� It is much longer thanChapters � and � because understanding the relationship between ODEs andDAEs� and the questions regarding reformulation of DAEs� is essential andalready suggests a lot regarding computer approaches� Chapter �� discussesnumerical methods for DAEs�

Various courses can be taught using this book� A ���week course can bebased on the �rst chapters� with an addition from either one of the remain�ing two parts� In a ���week course �or shorter in a more advanced graduateclass� it is possible to cover comfortably Chapters �� and either Chapters ���or Chapters ����� with a more super�cial coverage of the remaining material�

The exercises vary in scope and level of di culty� We have provided somehints� or at least warnings� for those exercises that we �or our students� havefound more demanding�

Many people helped us with the tasks of shaping up� correcting� �lteringand re�ning the material in this book� First and foremost there are ourstudents in the various classes we taught on this subject� They made usacutely aware of the di�erence between writing with the desire to explainand writing with the desire to impress� We note� in particular� G� Lakatos�D� Aruliah� P� Ziegler� H� Chin� R� Spiteri� P� Lin� P� Castillo� E� Johnson�D� Clancey and D� Rasmussen� We have bene�ted particularly from ourearlier collaborations on other� related books with K� Brenan� S� Campbell�R� Mattheij and R� Russell� Colleagues who have o�ered much insight� adviceand criticism include E� Biscaia� G� Bock� C� W� Gear� W� Hayes� C� Lubich�V� Murata� D� Pai� J� B� Rosen� L� Shampine and A� Stuart� Larry Shampine�in particular� did an incredibly extensive refereeing job and o�ered manycomments which have helped us to signi�cantly improve this text� We have

Page 5: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

PREFACE v

also bene�ted from comments of numerous anonymous referees�

December �� ���� U� M� AscherL� R� Petzold

Page 6: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

vi PREFACE

Page 7: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Contents

� Ordinary Di�erential Equations �

��� Initial Value Problems � � � � � � � � � � � � � � � � � � � � � � ���� Boundary Value Problems � � � � � � � � � � � � � � � � � � � � ���� Di�erential�Algebraic Equations � � � � � � � � � � � � � � � � � ���� Families of Application Problems � � � � � � � � � � � � � � � � ���� Dynamical Systems � � � � � � � � � � � � � � � � � � � � � � � � ���� Notation � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � ��

� On Problem Stability ��

��� Test Equation and General De�nitions � � � � � � � � � � � � � ����� Linear� Constant Coe cient Systems � � � � � � � � � � � � � � ����� Linear� Variable Coe cient Systems � � � � � � � � � � � � � � � ����� Nonlinear Problems � � � � � � � � � � � � � � � � � � � � � � � � ���� Hamiltonian Systems � � � � � � � � � � � � � � � � � � � � � � � ����� Notes and References � � � � � � � � � � � � � � � � � � � � � � � ����� Exercises � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � ��

� Basic Methods� Basic Concepts ��

��� A Simple Method� Forward Euler � � � � � � � � � � � � � � � � ���� Convergence� Accuracy� Consistency and ��Stability � � � � � � ����� Absolute Stability � � � � � � � � � � � � � � � � � � � � � � � � � ����� Sti�ness� Backward Euler � � � � � � � � � � � � � � � � � � � � ���� A�Stability� Sti� Decay � � � � � � � � � � � � � � � � � � � � � � ���� Symmetry� Trapezoidal Method � � � � � � � � � � � � � � � � � ���� Rough Problems � � � � � � � � � � � � � � � � � � � � � � � � � ����� Software� Notes and References � � � � � � � � � � � � � � � � � ��

����� Notes � � � � � � � � � � � � � � � � � � � � � � � � � � � ������� Software � � � � � � � � � � � � � � � � � � � � � � � � � � �

��� Exercises � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � ��

� One Step Methods ��

��� The First Runge�Kutta Methods � � � � � � � � � � � � � � � � ���� General Formulation of Runge�Kutta Methods � � � � � � � � � ��

vii

Page 8: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

viii CONTENTS

��� Convergence� ��Stability and Order for Runge�Kutta Methods ����� Regions of Absolute Stability for Explicit Runge�Kutta Methods ���� Error Estimation and Control � � � � � � � � � � � � � � � � � � ����� Sensitivity to Data Perturbations � � � � � � � � � � � � � � � � ����� Implicit Runge�Kutta and Collocation Methods � � � � � � � � ���

����� Implicit Runge�Kutta Methods Based on Collocation � �������� Implementation and Diagonally Implicit Methods � � � ������� Order Reduction � � � � � � � � � � � � � � � � � � � � � �������� More on Implementation and SIRK Methods � � � � � � ���

��� Software� Notes and References � � � � � � � � � � � � � � � � � �������� Notes � � � � � � � � � � � � � � � � � � � � � � � � � � � �������� Software � � � � � � � � � � � � � � � � � � � � � � � � � � ���

��� Exercises � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � ���

� Linear Multistep Methods ����� The Most Popular Methods � � � � � � � � � � � � � � � � � � � ���

���� Adams Methods � � � � � � � � � � � � � � � � � � � � � � ������� Backward Di�erentiation Formulae � � � � � � � � � � � ������� Initial Values for Multistep Methods � � � � � � � � � � ���

�� Order� ��Stability and Convergence � � � � � � � � � � � � � � � ������� Order � � � � � � � � � � � � � � � � � � � � � � � � � � � ������� Stability� Di�erence Equations and the Root Condition ������� ��Stability and Convergence � � � � � � � � � � � � � � � ���

�� Absolute Stability � � � � � � � � � � � � � � � � � � � � � � � � � ����� Implementation of Implicit Linear Multistep Methods � � � � � ���

���� Functional Iteration � � � � � � � � � � � � � � � � � � � � ������� Predictor�Corrector Methods � � � � � � � � � � � � � � � ������� Modi�ed Newton Iteration � � � � � � � � � � � � � � � � ���

� Designing Multistep General�Purpose Software � � � � � � � � � ������ Variable Step�Size Formulae � � � � � � � � � � � � � � � ����� Estimating and Controlling the Local Error � � � � � � ����� Approximating the Solution at O��Step Points � � � � � �

�� Software� Notes and References � � � � � � � � � � � � � � � � � ����� Notes � � � � � � � � � � � � � � � � � � � � � � � � � � � ����� Software � � � � � � � � � � � � � � � � � � � � � � � � � � ��

�� Exercises � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � ��

More BVP Theory and Applications ��

��� Linear Boundary Value Problems and Green�s Function � � � � ������ Stability of Boundary Value Problems � � � � � � � � � � � � � � ������ BVP Sti�ness � � � � � � � � � � � � � � � � � � � � � � � � � � � ������ Some Reformulation Tricks � � � � � � � � � � � � � � � � � � � � ����� Notes and References � � � � � � � � � � � � � � � � � � � � � � � ���

Page 9: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

CONTENTS ix

��� Exercises � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � ��

� Shooting ���

��� Shooting� a Simple Method and its Limitations � � � � � � � � ���

����� Di culties � � � � � � � � � � � � � � � � � � � � � � � � � ���

��� Multiple Shooting � � � � � � � � � � � � � � � � � � � � � � � � � ���

��� Software� Notes and References � � � � � � � � � � � � � � � � � ���

����� Notes � � � � � � � � � � � � � � � � � � � � � � � � � � � ���

����� Software � � � � � � � � � � � � � � � � � � � � � � � � � � ���

��� Exercises � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � ���

Finite Di�erence Methods for BVPs ���

��� Midpoint and Trapezoidal Methods � � � � � � � � � � � � � � � ���

����� Solving Nonlinear Problems� Quasilinearization � � � � ���

����� Consistency� ��stability and Convergence � � � � � � � � ���

��� Solving the Linear Equations � � � � � � � � � � � � � � � � � � � ���

��� Higher Order Methods � � � � � � � � � � � � � � � � � � � � � � ���

����� Collocation � � � � � � � � � � � � � � � � � � � � � � � � ���

����� Acceleration Techniques � � � � � � � � � � � � � � � � � ���

��� More on Solving Nonlinear Problems � � � � � � � � � � � � � � ���

����� Damped Newton � � � � � � � � � � � � � � � � � � � � � ���

����� Shooting for Initial Guesses � � � � � � � � � � � � � � � ���

����� Continuation � � � � � � � � � � � � � � � � � � � � � � � ���

�� Error Estimation and Mesh Selection � � � � � � � � � � � � � � ���

��� Very Sti� Problems � � � � � � � � � � � � � � � � � � � � � � � � ��

��� Decoupling � � � � � � � � � � � � � � � � � � � � � � � � � � � � ���

��� Software� Notes and References � � � � � � � � � � � � � � � � � ���

����� Notes � � � � � � � � � � � � � � � � � � � � � � � � � � � ���

����� Software � � � � � � � � � � � � � � � � � � � � � � � � � � ���

��� Exercises � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � ���

� More on Di�erential�Algebraic Equations ���

��� Index and Mathematical Structure � � � � � � � � � � � � � � � ���

����� Special DAE Forms � � � � � � � � � � � � � � � � � � � � ���

����� DAE Stability � � � � � � � � � � � � � � � � � � � � � � � ��

��� Index Reduction and Stabilization� ODE with Invariant � � � � ���

����� Reformulation of Higher�Index DAEs � � � � � � � � � � ���

����� ODEs with Invariants � � � � � � � � � � � � � � � � � � ��

����� State Space Formulation � � � � � � � � � � � � � � � � � ��

��� Modeling with DAEs � � � � � � � � � � � � � � � � � � � � � � � ��

��� Notes and References � � � � � � � � � � � � � � � � � � � � � � � ��

�� Exercises � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � ��

Page 10: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

x CONTENTS

�� Numerical Methods for Di�erential�Algebraic Equations ������ Direct Discretization Methods � � � � � � � � � � � � � � � � � � ���

������ A Simple Method� Backward Euler � � � � � � � � � � � �������� BDF and General Multistep Methods � � � � � � � � � � ��������� Radau Collocation and Implicit Runge�Kutta Methods ��������� Practical Di culties � � � � � � � � � � � � � � � � � � � �������� Specialized Runge�Kutta Methods for Hessenberg Index�

� DAEs � � � � � � � � � � � � � � � � � � � � � � � � � � ������� Methods for ODEs on Manifolds � � � � � � � � � � � � � � � � � ���

������ Stabilization of the Discrete Dynamical System � � � � ��������� Choosing the Stabilization Matrix F � � � � � � � � � � ���

���� Software� Notes and References � � � � � � � � � � � � � � � � � ��������� Notes � � � � � � � � � � � � � � � � � � � � � � � � � � � ��������� Software � � � � � � � � � � � � � � � � � � � � � � � � � � ���

���� Exercises � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � ���

Bibliography ���

Index ���

Page 11: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

List of Tables

��� Maximum errors for Example ���� � � � � � � � � � � � � � � � ����� Maximum errors for long interval integration of y� � �cos t�y � ��

��� Errors and calculated convergence rates for the forward Euler�the explicit midpoint �RK�� and the classical Runge�Kutta�RK�� methods � � � � � � � � � � � � � � � � � � � � � � � � � � ��

�� Coe cients of Adams�Bashforth methods up to order � � � � ����� Coe cients of Adams�Moulton methods up to order � � � � � ����� Coe cients of BDF methods up to order � � � � � � � � � � � ����� Example ��� Errors and calculated convergence rates for Adams�

Bashforth methods� � � � � � � � � � � � � � � � � � � � � � � � ���� Example ��� Errors and calculated convergence rates for Adams�

Moulton methods� � � � � � � � � � � � � � � � � � � � � � � � � ����� Example ��� Errors and calculated convergence rates for BDF

methods� � � � � � � � � � � � � � � � � � � � � � � � � � � � � � ��

��� Maximum errors for Example ��� using the midpoint method�uniform meshes� � � � � � � � � � � � � � � � � � � � � � � � � � ��

��� Maximum errors for Example ��� using the midpoint method�nonuniform meshes� � � � � � � � � � � � � � � � � � � � � � � � ��

��� Maximum errors for Example ��� using collocation at � Gaus�sian points� uniform meshes� � � � � � � � � � � � � � � � � � � ���

��� Maximum errors for Example ��� using collocation at � Gaus�sian points� nonuniform meshes� � � � � � � � � � � � � � � � � ���

���� Errors for Kepler�s problem using various �nd order methods� ������ Maximum drifts for the robot arm� � denotes an error over�ow� ���

xi

Page 12: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

xii LIST OF TABLES

Page 13: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

List of Figures

��� u vs t for u��� � � and various values of u����� � � � � � � � � �

��� Simple pendulum� � � � � � � � � � � � � � � � � � � � � � � � � �

��� Periodic solution forming a cycle in the y� � y� plane� � � � �

��� Method of lines� The shaded strip is the domain on whichthe di�usion PDE is de�ned� The approximations yi�t� arede�ned along the dashed lines� � � � � � � � � � � � � � � � � � �

��� Errors due to perturbations for stable and unstable test equa�tions� The original� unperturbed trajectories are in solid curves�the perturbed in dashed� Note that the y�scales in Figures �a�and �b� are not the same� � � � � � � � � � � � � � � � � � � � � ��

��� The forward Euler method� The exact solution is the curvedsolid line� The numerical values are circled� The broken lineinterpolating them is tangential at the beginning of each stepto the ODE trajectory passing through that point �dashedlines�� � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � ��

��� Absolute stability region for the forward Euler method� � � � ��

��� Approximate solutions for Example ��� using the forward Eu�ler method� with h � ��� and h � ��� � The oscillatory pro�lecorresponds to h � ���� for h � ��� the qualitative behaviorof the exact solution is obtained� � � � � � � � � � � � � � � � � ��

��� Approximate solution and plausible mesh� Example ���� � � � ��

�� Absolute stability region for the backward Euler method� � � �

��� Approximate solution on a coarse uniform mesh for Example���� using backward Euler �the smoother curve� and trape�zoidal methods� � � � � � � � � � � � � � � � � � � � � � � � � � �

��� Sawtooth function for � � ���� � � � � � � � � � � � � � � � � � ��

��� Classes of higher order methods� � � � � � � � � � � � � � � � � ��

��� Approximate area under curve � � � � � � � � � � � � � � � � � ��

��� Midpoint quadrature� � � � � � � � � � � � � � � � � � � � � � � ��

xiii

Page 14: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

xiv LIST OF FIGURES

��� Stability regions for p�stage explicit Runge�Kutta methods oforder p� p � �� �� �� �� The inner circle corresponds to forwardEuler� p � �� The larger p is� the larger the stability region�Note the �ear lobes� of the �th order method protruding intothe right half plane� � � � � � � � � � � � � � � � � � � � � � � � � ��

�� Schematic of a mobile robot � � � � � � � � � � � � � � � � � � � ����� Toy car routes under constant steering� unperturbed �solid

line�� steering perturbed by �� �dash�dot lines�� and corre�sponding trajectories computed by the linear sensitivity anal�ysis �dashed lines�� � � � � � � � � � � � � � � � � � � � � � � � � ���

��� Energy error for the Morse potential using leapfrog with h �������� � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � ���

��� Astronomical orbit using the Runge�Kutta Fehlberg method� ������ Modi�ed Kepler problem� approximate and exact solutions � � ���

�� Adams�Bashforth methods � � � � � � � � � � � � � � � � � � � ����� Adams�Moulton methods � � � � � � � � � � � � � � � � � � � � ����� Zeros of ���� for a ��stable method� � � � � � � � � � � � � � � ����� Zeros of ���� for a strongly stable method� It is possible to

draw a circle contained in the unit circle about each extraneousroot� � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � ���

� Absolute stability regions of Adams methods � � � � � � � � � ����� BDF absolute stability regions� The stability regions are out�

side the shaded area for each method� � � � � � � � � � � � � � ���� Lorenz �butter�y� in the y� � y� plane� � � � � � � � � � � � � ��

��� Two solutions u�t� for the BVP of Example ���� � � � � � � � ����� The function y��t� and its mirror image y��t� � y��b� t�� for

� � ��� b � ��� � � � � � � � � � � � � � � � � � � � � � � � � � ���

��� Exact �solid line� and shooting �dashed line� solutions for Ex�ample ���� � � � � � � � � � � � � � � � � � � � � � � � � � � � � ���

��� Exact �solid line� and shooting �dashed line� solutions for Ex�ample ���� � � � � � � � � � � � � � � � � � � � � � � � � � � � � ���

��� Multiple shooting � � � � � � � � � � � � � � � � � � � � � � � � ���

��� Example ���� Exact and approximate solutions �indistinguish�able� for � � �� using the indicated mesh� � � � � � � � � � � ���

��� Zero�structure of the matrix A� m � �� N � ��� The matrixsize is m�N � �� � ��� � � � � � � � � � � � � � � � � � � � � � � ���

��� Zero�structure of the permutedmatrixA with separated bound�ary conditions� m � �� k � �� N � ��� � � � � � � � � � � � � � ��

��� Classes of higher order methods� � � � � � � � � � � � � � � � � ����� Bifurcation diagram for Example �� � kuk� vs �� � � � � � � � ���

Page 15: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

LIST OF FIGURES xv

��� Solution for Example ��� with � � ����� using an upwinddiscretization with a uniform step size h � ��� �solid line��The �exact� solution is also displayed �dashed line�� � � � � � ���

��� A function and its less smooth derivative� � � � � � � � � � � � ������ Sti� spring pendulum� � � ����� initial conditions q��� �

�� � ����� ��T �v��� � �� � � � � � � � � � � � � � � � � � � � � � � ������ Perturbed �dashed lines� and unperturbed �solid line� solu�

tions for Example ���� � � � � � � � � � � � � � � � � � � � � � � ����� A matrix in Hessenberg form� � � � � � � � � � � � � � � � � � � ��

���� Methods for the direct discretization of DAEs in general form� ������ Maximum errors for the �rst � BDF methods for Example

����� � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � ������� A simple electric circuit� � � � � � � � � � � � � � � � � � � � � � ������� Results for a simple electric circuit� U��t� �solid line� and the

input Ue�t� �dashed line�� � � � � � � � � � � � � � � � � � � � � ������ Two�link planar robotic system � � � � � � � � � � � � � � � � � ������� Constraint path for �x�� y��� � � � � � � � � � � � � � � � � � � � ���

Page 16: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Chapter �

Ordinary Di�erential Equations

Ordinary di�erential equations �ODEs� arise in many instances when us�ing mathematical modeling techniques for describing phenomena in science�engineering� economics� etc� In most cases the model is too complex to al�low �nding an exact solution or even an approximate solution by hand� ane cient� reliable computer simulation is required�Mathematically� and computationally� a �rst cut at classifying ODE prob�

lems is with respect to the additional or side conditions associated with them�To see why� let us look at a simple example� Consider

u���t� � u�t� � �� � � t � b

where t is the independent variable �it is often� but not always� convenientto think of t as �time��� and u � u�t� is the unknown� dependent variable�Throughout this book we use the notation

u� �du

dt� u�� �

d�u

dt��

etc� We shall often omit explicitly writing the dependence of u on t�The general solution of the ODE for u depends on two parameters � and

u�t� � � sin�t� ��

We can therefore impose two side conditions�

� Initial value problem� Given values u��� � c� and u���� � c�� the pairof equations

� sin � u��� � c�

� cos � u���� � c�

can always be solved uniquely for � tan�� c�c�and � � c�

sin� �or � �c�

cos� � at least one of these is well�de�ned�� The initial value problem

Page 17: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

� Chapter �� Ordinary Di�erential Equations

0 0.5 1 1.5 2 2.5 3 3.5−2

−1.5

−1

−0.5

0

0.5

1

1.5

2

2.5

3

t

u

ODE trajectories

Figure ���� u vs t for u��� � � and various values of u�����

has a unique solution for any initial data c � �c�� c��T � Such solutioncurves are plotted for c� � � and di�erent values of c� in Fig� ����

� Boundary value problem� Given values u��� � c� and u�b� � c�� itappears from Fig� ��� that for b � �� say� if c� and c� are chosen carefullythen there is a unique solution curve that passes through them� justlike in the initial value case� However� consider the case where b � �Now di�erent values of u���� yield the same value u�� � �u��� �seeagain Fig� ����� So� if the given value of u�b� � c� � �c� then we havein�nitely many solutions� whereas if c� �� �c� then no solution exists�

This simple illustration already indicates some important general issues�For initial value problems� one starts at the initial point with all the solu�tion information and marches with it �in �time�� � the process is local� Forboundary value problems the entire solution information �for a second orderproblem this consists of u and u�� is not locally known anywhere� and theprocess of constructing a solution is global in t� Thus we may expect manymore �and di�erent� di culties with the latter� and this is re�ected in thenumerical procedures discussed in this book�

Page 18: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Chapter �� Ordinary Di�erential Equations �

��� Initial Value Problems

The general form of an initial value problem �IVP� that we shall discuss is

y� � f�t�y�� � � t � b

y��� � c �given�������

Here y and f are vectors with m components� y � y�t�� and f is in generala nonlinear function of t and y� When f does not depend explicitly on t� wespeak of the autonomous case� When describing general numerical methodswe shall often assume the autonomous case simply in order to carry lessnotation around� The simple example from the beginning of this chapter isin the form ����� with m � �� y � �u� u��T � f � �u���u�T �In ����� we assume� for simplicity of notation� that the starting point for

t is �� An extension to an arbitrary interval of integration �a� b� of everythingwhich follows is obtained without di culty�Before proceeding further� we give three examples which are famous for

being very simple on one hand and for representing important classes ofapplications on the other hand�

Example � � �Simple pendulum� Consider a tiny ball of mass � attachedto the end of a rigid� massless rod of length �� At its other end the rod�sposition is �xed at the origin of a planar coordinate system �see Fig� ����

Θ

Figure ���� Simple pendulum�

Denoting by � the angle between the pendulum and the yaxis� the frictionfree motion is governed by the ODE �cf� Example ��� below

��� � �g sin � �����

where g is the �scaled constant of gravity� This is a simple� nonlinear ODEfor �� The initial position and velocity con�guration translates into values

Page 19: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

� Chapter �� Ordinary Di�erential Equations

for ���� and ������ The linear� trivial example from the beginning of thischapter can be obtained from an approximation of �a rescaled ���� for smalldisplacements �� �

The pendulum problem is posed as a second order scalar ODE� Much ofthe software for initial value problems is written for �rst order systems in theform ������ A scalar ODE of order m�

u�m� � g�t� u� u�� � � � � u�m����

can be rewritten as a �rst�order system by introducing a new variable foreach derivative� with y� � u�

y�� � y�

y�� � y����

y�m�� � ym

y�m � g�t� y�� y�� � � � � ym��

Example � � �Predator�prey model� Following is a basic� simple modelfrom population biology which involves di�erential equations� Consider anecological system consisting of one prey species and one predator species� Theprey population would grow unboundedly if the predator were not present� andthe predator population would perish without the presence of the prey� Denote

� y��t� the prey population at time t

� y��t� the predator population at time t

� � �prey�s birthrate��prey�s natural death rate �� � �

� probability of a prey and a predator to come together

� predator�s natural growth rate �without prey� � �

� � increase factor of growth of predator if prey and predator meet�

Typical values for these constants are � � ��� � ���� � ������ � � ����Writing

y �

��y�y�

�A f �

���y� � y�y�

y� � �y�y�

�A �����

Page 20: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Chapter �� Ordinary Di�erential Equations

15

20

25

30

35

40

70 80 90 100 110 120 130

Figure ���� Periodic solution forming a cycle in the y� � y� plane�

we obtain an ODE in the form ���� with m � � components� describing thetime evolution of these populations�

The qualitative question here is� starting from some initial values y���out of a set of reasonable possibilities� will these two populations surviveor perish in the long run� As it turns out� this model possesses periodicsolutions� starting� say� from y��� � ���� ���T � the solution reaches the samepair of values again after some time period T � i�e� y�T � � y���� Continuingto integrate past T yields a repetition of the same values� y�T � t� � y�t��Thus� the solution forms a cycle in the phase plane �y�� y�� �see Fig� ����Starting from any point on this cycle the solution stays on the cycle for alltime� Other initial values not on this cycle yield other periodic solutions witha generally di�erent period� So� under these circumstances the populations ofthe predator and prey neither explode nor vanish for all future times� althoughtheir number never becomes constant� �

�In other examples� such as the Van der Pol equation ������� the solution forms anattracting limit cycle� starting from any point on the cycle the solution stays on it for alltime� and starting from points nearby the solution tends in time towards the limit cycle�The neutral stability of the cycle in our simple example� in contrast� is one reason why

this predator�prey model is discounted amongmathematical biologists as being too simple�

Page 21: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

� Chapter �� Ordinary Di�erential Equations

Example � � �A di�usion problem� A typical di�usion problem in onespace variable x and time t leads to the partial di�erential equation �PDE

�u

�t�

�x

�p�u

�x

�� g�x� u��

for an unknown function u�t� x� of two independent variables de�ned on astrip � � x � �� t � �� For simplicity� assume that p � � and g is a knownfunction� Typical side conditions which make this problem wellposed are

u��� x� � q�x�� � � x � � Initial conditions

u�t� �� � ��t�� u�t� �� � �t�� t � � Boundary conditions

To solve this problem numerically� consider discretizing in the space variable �rst� For simplicity assume a uniform mesh with spacing �x � ���m���� and let yi�t� approximate u�xi� t�� where xi � i�x� i � �� �� � � � �m � ��Then replacing ��u

�x�by a secondorder central di�erence we obtain

dyidt�

yi�� � �yi � yi���x�

� g�xi� yi�� i � �� � � � �m

with y��t� � ��t� and ym���t� � �t� given� We have obtained an initialvalue ODE problem of the form ���� with the initial data ci � q�xi��This technique of replacing spatial derivatives by �nite di�erence approx

imations and solving an ODE problem in time is referred to as the methodof lines� Fig� ��� illustrates the origin of the name� Its more general form isdiscussed further in Example ��� below� �

We now return to the general initial value problem for ������ Our inten�tion in this book is to keep the number of theorems down to a minimum�the references which we quote have them all in much detail� But we willnonetheless write down those which are of fundamental importance� and theone just below captures the essence of the �relative� simplicity and localityof initial value ODEs� For the notation that is used in this theorem andthroughout the book� we refer to x����Theorem � � Let f�t�y� be continuous for all �t�y� in a region D � f� �t � b� �� � jyj ��g � Moreover� assume Lipschitz continuity in y� thereexists a constant L such that for all �t�y� and �t� �y� in D�

jf�t�y�� f�t� �y�j � Ljy� �yj� �����

Then

�� For any c Rm there exists a unique solution y�t� throughout theinterval ��� b� for the initial value problem ����� This solution is differentiable�

Page 22: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Chapter �� Ordinary Di�erential Equations �

x0 1

t

Figure ���� Method of lines� The shaded strip is the domain on which thedi�usion PDE is de�ned� The approximations yi�t� are de�ned along thedashed lines�

�� The solution y depends continuously on the initial data� if �y also satis�es the ODE �but not the same initial values then

jy�t�� �y�t�j � eLtjy���� �y���j� ����

�� If �y satis�es� more generally� a perturbed ODE

�y� � f�t� �y� � r�t� �y�

where r is bounded on D� krk �M � then

jy�t�� �y�t�j � eLtjy���� �y���j� M

L�eLt � ��� �����

Thus we have solution existence� uniqueness and continuous dependenceon the data� in other words a wellposed problem� provided that the condi�tions of the theorem hold� Let us check these conditions� If f is di�erentiablein y �we shall automatically assume this throughout� then the L can be takenas a bound on the �rst derivatives of f with respect to y� Denote by fy theJacobian matrix�

�fy�ij ��fi�yj

� � � i� j � m�

Page 23: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

� Chapter �� Ordinary Di�erential Equations

We can write

f�t�y�� f�t� �y� �

Z �

d

dsf�t� �y� s�y� �y�� ds

Z �

fy�t� �y� s�y� �y�� �y � �y� ds�

Therefore� we can choose L � sup�t�y��D kfy�t�y�k�In many cases we must restrict D in order to be assured of the existence

of such a ��nite� bound L� For instance� if we restrict D to include boundedy such that jy� cj � � and on this D both the Lipschitz bound ����� holdsand jf�t�y�j � M � then a unique existence of the solution is guaranteed for� � t � min�b� �M�� giving the basic existence result a more local �avor�For further theory and proofs see� for instance� Mattheij Molnaar �����

Reader�s advice� Before continuing our introduction� let us re�mark that a reader who is interested in getting to the numericsof initial value problems as soon as possible may skip the rest ofthis chapter and the next� at least on �rst reading�

��� Boundary Value Problems

The general form of a boundary value problem �BVP� which we consider is anonlinear �rst order system of m ODEs subject to m independent �generallynonlinear� boundary conditions�

y� � f�t�y� ����a�

g�y����y�b�� � �� ����b�

We have already seen in the beginning of the chapter that in those caseswhere solution information is given at both ends of the integration interval�or� more generally� at more than one point in time�� nothing general likeTheorem ��� can be expected to hold� Methods for �nding a solution� bothanalytically and numerically� must be global and the task promises to begenerally harder than for initial value problems� This basic di�erence ismanifested in the current status of software for boundary value problems�which is much less advanced or robust than that for initial value problems�Of course� well�posed boundary value problems do arise on many occa�

sions�

Page 24: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Chapter �� Ordinary Di�erential Equations �

Example � � �Vibrating spring� The small displacement u of a vibratingspring obeys a linear di�erential equation

��p�t�u��� � q�t�u � r�t�

where p�t� � � and q�t� � � for all � � t � b� �Such an equation describesalso many other physical phenomena in one space variable t� If the springis �xed at one end and is left to oscillate freely at the other end then we getthe boundary conditions

u��� � �� u��b� � ��

We can write this problem in the form ���� for y � �u� u��T � Better still�

we can use y �

�� u

pu�

�A� obtaining f �

�� p��y�

qy�� r

�A� g �

��y����y��b�

�A� This

boundary value problem has a unique solution �which gives the minimum forthe energy in the spring� as shown and discussed in many books on �niteelement methods� e�g� Strang � Fix ����� �

Another example of a boundary value problem is provided by the predator�prey system of Example ���� if we wish to �nd the periodic solution �whoseexistence is evident from Fig� ����� We can specify y��� � y�b�� However�note that b is unknown� so the situation is more complex� Further treatmentis deferred to Chapter � and Exercise ��� A complete treatment of �ndingperiodic solutions for ODE systems falls outside the scope of this book�What can be generally said about existence and uniqueness of solutions

to a general boundary value problem �����! We may consider the associatedinitial value problem ����� with the initial values c as a parameter vector tobe found� Denoting the solution for such an IVP y�t� c�� we wish to �nd thesolution�s� for the nonlinear algebraic system of m equations

g�c�y�b� c�� � �� �����

However� in general there may be one� many or no solutions for a system like������ We delay further discussion to Chapter ��

��� Di�erential�Algebraic Equations

Both the prototype IVP ����� and the prototype BVP ����� refer to an explicitODE system

y� � f�t�y�� �����

Page 25: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

�� Chapter �� Ordinary Di�erential Equations

A more general form is an implicit ODE

F�t�y�y�� � � ������

where the Jacobian matrix �F�t�u�v��v

is assumed nonsingular for all argumentvalues in an appropriate domain� In principle it is then often possible to solvefor y� in terms of t and y� obtaining the explicit ODE form ������ However�this transformation may not always be numerically easy or cheap to realize�see Example ��� below�� Also� in general there may be additional questionsof existence and uniqueness� we postpone further treatment until Chapter ��Consider next another extension of the explicit ODE� that of an ODE

with constraints�

x� � f�t�x� z� �����a�

� � g�t�x� z�� �����b�

Here the ODE �����a� for x�t� depends on additional algebraic variablesz�t�� and the solution is forced in addition to satisfy the algebraic constraints�����b�� The system ������ is a semiexplicit system of di�erentialalgebraicequation �DAE�� Obviously� we can cast ������ in the form of an implicit ODE

������ for the unknown vector y �

��xz

�A� however� the obtained Jacobian

matrix

�F�t�u�v�

�v�

��I �

� �

�A

is no longer nonsingular�

Example � � �Simple pendulum revisited� The motion of the simplependulum of Fig� ��� can be expressed in terms of the Cartesian coordinates �x�� x�� of the tiny ball at the end of the rod� With z�t� a Lagrangemultiplier� Newton�s equations of motion give

x��� � �zx�x��� � �zx� � g

and the fact that the rod has a �xed length � gives the additional constraint

x�� � x�� � ��

Page 26: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Chapter �� Ordinary Di�erential Equations ��

After rewriting the two secondorder ODEs as four �rst order ODEs� weobtain a DAE system of the form ����� with four equations in �����a andone in �����b�In this very simple case of a multibody system� the change of variables

x� � sin �� x� � � cos � allows elimination of z by simply multiplying theODE for x� by x� and the ODE for x� by x� and subtracting� This yieldsthe simple ODE ���� of Example ���� Such a simple elimination procedureis usually impossible in more general situations� though� �

The di�erence between an implicit ODE �with a nonsingular Jacobianmatrix� and a DAE is fundamental� Consider the simple example

x� � z

� � x� t�

Clearly� the solution is x � t� z � �� and no initial or boundary conditionsare needed� In fact� if an arbitrary initial condition x��� � c is imposed itmay well be inconsistent with the DAE �unless c � �� in which case thisinitial condition is just super�uous�� We refer to Chapter � for more on this�Another fundamental point to note is that even if consistent initial values aregiven we cannot expect a simple� general existence and uniqueness theoremlike Theorem ��� to hold for ������� The nonlinear equations �����b� alonemay have any number of solutions� Again we refer the reader to Chapter �for more details�

��� Families of Application Problems

Initial�value and boundary�value problems for ODE and DAE systems arisein a wide variety of applications� Often an application generates a family ofproblems which share a particular system structure and"or solution require�ments� Here we brie�y mention three families of problems from importantapplications� The notation we use is typical for these applications� and isnot necessarily consistent with ����� or ������� You don�t need to understandthe details given in this section in order to follow the rest of the text � thismaterial is supplemental�

Example � �Mechanical systems� When attempting to simulate the motion of a vehicle for design or in order to simulate safety tests� or in physically based modeling in computer graphics� or in a variety of instances in

Page 27: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

�� Chapter �� Ordinary Di�erential Equations

robotics� one encounters the need for a fast� reliable simulation of the dynamics of multibody systems� The system considered is an assembly of rigidbodies �e�g� comprising a car suspension system� The kinematics de�ne howthese bodies are allowed to move with respect to one another� Using generalized position coordinates q � �q�� � � � � qn�T for the bodies� with m �socalledholonomic constraints gj�t�q�t�� � �� j � �� � � � �m� the equations of motioncan be written as

d

dt

��L

�q�i

�� �L

�qi� �� i � �� � � � � n

where L � T � U �P�jgj is the Lagrangian� T is the kinetic energy andU is the potential energy� See almost any book on classical mechanics� forexample Arnold ���� or the lighter Marion � Thornton ����� The resultingequations of motion can be written as

q� � v �����a�

M�t�q�v� � f�t�q�v��GT �t�q�� �����b�

� � g�t�q� �����c�

where G � �g�q� M is a positive de�nite generalized mass matrix� f are the

applied forces �other than the constraint forces and v are the generalizedvelocities� The system sizes n and m depend on the chosen coordinates q�Typically� using relative coordinates �describing each body in terms of its nearneighbor results in a smaller but more complicated system� If the topologyof the multibody system �i�e� the connectivity graph obtained by assigning anode to each body and an edge for each connection between bodies does nothave closed loops� then with a minimal set of coordinates one can eliminateall the constraints �i�e� m � � and obtain an implicit ODE in ������ Forinstance� Example ��� uses a minimal set of coordinates� while Example ���does not� for a particular multibody system without loops� If the multibodysystem contains loops �e�g� a robot arm� consisting of two links� with the pathof the �hand� prescribed then the constraints cannot be totally eliminated ingeneral and a DAE must be considered in ����� even if a minimal set ofcoordinates is employed� �

Example � � �Method of lines� The di�usion equation of Example ��� isan instance of a timedependent partial di�erential equation �PDE in onespace dimension�

�u

�t� f

�t� u�

�u

�x���u

�x�

�� ������

Page 28: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Chapter �� Ordinary Di�erential Equations ��

Timedependent PDEs naturally arise also in more than one space dimension� with higher order spatial derivatives� and as systems of PDEs� Theprocess described in Example ��� is general� such a PDE can be transformedinto a large system of ordinary di�erential equations by replacing the spatial derivatives in one or more dimension by a discrete approximation �via�nitedi�erence� �nitevolume or �niteelement methods� see texts on numerical methods for PDEs� e�g� Strikwerda ����� Typically� we obtain an initialvalue problem� This technique of semidiscretizing in space �rst and solvingan initial value ODE problem in time is referred to as the method of lines�It makes sense when two conditions are satis�ed� i The �time� variable t issu�ciently di�erent from the �space� variables to warrant a special treatment�ii There is no sharp front in the solution that moves rapidly as a functionof both space and time� i�e� the rapid moving fronts �if there are any can bereasonably welldecoupled in time and space� Typically� the method of linesis more suitable for parabolic PDEs than for hyperbolic ones�Remaining still with the prototype di�usion problem considered in Exam

ple ���� in some situations the �special� independent variable is not time butone of the spatial variables� This is the case in some interface problems�Another way to convert a PDE to an ODE system is then to replace the timederivative by a di�erence approximation� Replacing the time derivative by asimple backward di�erence approximation using time step �t in the di�usionequation yields

un � un��

�t�

��un

�x�� g�x� un�

and using u� � q�x� and the given boundary conditions yields a boundaryvalue problem in x for each n� This technique� of replacing the time derivativeby a di�erence approximation and solving the boundary value problem inspace� is called the transverse method of lines� �

Example � �Optimal control� A rather large number of applicationsgive rise to optimal control problems� For instance� the problem may be toplan a route for a vehicle traveling between two points �and satisfying equations of motion such that fuel consumption is optimized� or the travel timeis minimized� Another instance is to optimize the performance of a chemicalprocessing plant� Typically� the state variables of the system� y�t�� satisfy anODE system which involves a control function u�t� ��

y� � f�t�y�u�� � � t � b� �����a�

�The dimension of u�t� is generally dierent from that of y�t��

Page 29: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

�� Chapter �� Ordinary Di�erential Equations

This system may be subject to some side conditions� e�g�

y��� � c �����b�

but it is possible that y�b� is prescribed as well� or that there are no sideconditions at all� The control u�t� must be chosen so as to optimize somecriterion �or cost function� say

minimize J � ��y�b�� b� �

Z b

L�t�y�t��u�t��dt �����

subject to ������The necessary conditions for an optimum in this problem are found by

considering the Hamiltonian function

H�t�y�u��� �mXi��

�ifi�t�y�u� � L�t�y�u�

where �i�t� are adjoint variables� i � �� � � � �m� The conditions

y�i ��H

��i� i � �� � � � �m

yield the state equations �����a� and in addition we have ordinary di�erentialequations for the adjoint variables�

��i � ��H

�yi� �

mXj��

�j�fj�yi

� �L

�yi� i � �� � � � �m ������

and

� ��H

�ui� i � �� � � � �mu� ������

This gives a DAE in general� however� often u�t� can be eliminated from����� in terms of y and �� yielding an ODE system� Additional side conditions are required as well�

�i�b� ���

�yi�b�� i � �� � � � �m� ������

The system ����������������������� comprises a boundary value ODE �orDAE�An indirect approach for solving this optimal control problem involves the

numerical solution of the BVP just prescribed� The techniques described inChapters � and � are directly relevant� In contrast� a direct approach involvesthe discretization of ������������ and the subsequent numerical solution of

Page 30: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Chapter �� Ordinary Di�erential Equations �

the resulting large� sparse �but �nite dimensional constrained optimizationproblem� The techniques described in this book are relevant for this approachtoo� although less directly� Each of these two approaches has its advantages�and fans� Note that� even though ����� is an IVP� the direct approachdoes not yield a local process� which would have allowed a simple marchingalgorithm� because a change in the problem anywhere has a global e�ect�necessitating a global solution process �as needed for a BVP�A closely related family of applications involves parameter estimation in

an ODE system� Given a set of solution data in time �usually obtained byexperiment� the problem is to choose the parameters to minimize a measureof the distance between the data and the solution of the ODE �or DAEdepending on the parameters�We note� furthermore� that optimal control applications often require� in

addition to the above model� also inequality �algebraic constraints on thecontrols u�t� and on the state variables y�t� �e�g� a maximum speed or acceleration which must not� or cannot� be exceeded in the vehicle route planningapplication� Such inequality constraints complicate the analysis yielding necessary conditions� but we do not pursue this further� There are many bookson optimal control and parameter estimation� e�g� Bryson � Ho �����

��� Dynamical Systems

Recent years have seen an explosion of interest and e�orts in the study of thelong term� qualitative behavior of various nonlinear ODE systems� Typically�one is interested in the behavior of the �ow of a system y� � f�t�y�� not onlyin one trajectory for a given initial value c� Attention is often focussed thenon limit sets �a limit set is a special case of an invariant set� i�e�� a set ofinitial data that is mapped into itself by the �ow��While most of our book is concerned with the accurate and reliable sim�

ulation of solution trajectories� and the reader is not assumed to necessar�ily possess a background in dynamical systems� the techniques we exploreare essential for numerical dynamical systems� Moreover� various additionalchallenges arise when considering the simulation of such qualitative proper�ties� In some cases these additional challenges can be addressed using simpletricks �e�g� for �nding a periodic solution� or for projecting onto a giveninvariant de�ned by algebraic equations�� while on other occasions the chal�lenge is rather more substantial �e�g� �nding an invariant set in general� ornumerically integrating an ODE over a very long time period��

Page 31: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

�� Chapter �� Ordinary Di�erential Equations

Throughout this book we will pay attention to such additional consider�ations� especially when they extend our investigation in a natural way� Wewill certainly not attempt to do a complete job� rather� we will point outproblems� some solutions and some directions� For much more� we refer thereader to Stuart Humphries �����

��� Notation

Throughout the book� we use the following conventions for notation�

� Scalar variables and constants are denoted by Roman and Greek letters�e�g� t� u� y� K� L� N � �� � etc�

� Vectors are denoted by boldface letters� e�g� f � y� c� etc� The ithcomponent of the vector y is denoted yi� �Distinguish this from the no�tation yn which will be used later on to denote a vector approximatingy at position tn��

� The maximum norm of a vector is denoted just like the absolute valueof a scalar� jyj � max��i�m jyij� Occasionally the Euclidean vectornorm jyj� �

pyTy proves more convenient than the maximum norm

� we may drop the subscript when the precise vector norm used doesnot matter or is obvious�

� Capital Roman letters are used for matrices� The induced norms ofmatrices are denoted by double bars�

kAk � supjxj��

jAxjjxj �

Occasionally� a boldfaced capital Roman letter� e�g� A� is used for largematrices consisting of blocks which are themselves matrices�

� The �sup� norms of functions are denoted as follows

kyk � sup��t�b

jy�t�j�

� Letter from other alphabets� e�g� D� L�Nh� are used to denote domainsand operators� Also� Re and Im denote the real and imaginary partsof a complex scalar� and R is the set of real numbers�

Page 32: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Chapter �� Ordinary Di�erential Equations ��

� For a vector function g�x�� where g has n components and x has kcomponents �g may depend on other variables too� e�g� g � g�t�x�y���we denote the Jacobian matrix� i�e� the n � k matrix of �rst partialderivatives of g with respect to x� by gx or by

�g�x��

�g

�x

�i�j

�gx�i�j � �gi�xj

� � � i � n� � � j � k �

We use the Jacobian matrix notation a lot in this book� and occasionally�nd one of these common notational forms to be clearer than the otherin a particular context� Hence we keep them both�

� The gradient of a scalar function of k variables g�x�� denoted rg�x��is its one�row Jacobian matrix transposed into a vector function�

rg�x� � gTx �

The divergence of a vector function g�x� where g and x both have kcomponents� is the scalar function denoted by rg�x� and given by

rg�x� �kXi��

�gi�xi

Page 33: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

�� Chapter �� Ordinary Di�erential Equations

Page 34: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Chapter �

On Problem Stability

The term stability has been used in the literature for a large variety of di�erentconcepts� The basic� qualitative idea is that a model that produces a solution�output� for given data �input� should possess the property that if the inputis perturbed by a small amount then the output should also be perturbedby only a small amount� But the precise application of this idea to initialvalue ODEs� to boundary value ODEs and to numerical methods has givenrise to a multitude of de�nitions� The reader should therefore be carefulwhen speaking of stability to distinguish between stability of problems andof numerical methods� and between stability of initial and boundary valueproblems�

In this chapter we brie�y discuss the stability of initial value problems�No numerical solutions or methods are discussed yet � that will start onlyin the next chapter� Matrix eigenvalues play a central role here� so we alsoinclude a quick review below�

��

Page 35: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

�� Chapter�� Initial Value Problems

Review� Matrix eigenvalues� Given an m�m real matrix A�an eigenvalue � is a scalar which satis�es

Ax � �x

for some vector x �� �� In general� � is complex� but it is guaran�teed to be real if A is symmetric� The vector x� which is clearlydetermined only up to a scaling factor� is called an eigenvector�Counting multiplicities� A has m eigenvalues�A similarity transformation is de�ned� for any nonsingular matrixT � by

B � T��AT �

The matrixB has the same eigenvalues as A and the two matricesare said to be similar� If B is diagonal� B � diagf��� � � � � �mg�then the displayed �i are the eigenvalues of A� the correspondingeigenvectors are the columns of T � and A is said to be diagonalizable� Any symmetric matrix is diagonalizable� in fact by anorthogonal matrix �i�e� T can be chosen to satisfy T T � T����For a general matrix� however� an orthogonal similarity trans�formation can only bring A to a matrix B in upper triangularform �which� however� still features the eigenvalues on the maindiagonal of B��For a general A there is always a similarity transformation into aJordan canonical form�

B �

�BBBBBBBBB�

#� �

#�

��

� #s

�CCCCCCCCCA� #i �

�BBBBBB�

�i � �

�i �� � � �

� �i

�CCCCCCA� i � �� � � � � s �

Page 36: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Chapter�� On Problem Stability ��

��� Test Equation and General Denitions

Consider at �rst the simple scalar ODE� often referred to later as the testequation

y� � �y �����

where � is a constant� We allow � to be complex� because it represents aneigenvalue of a system�s matrix� The solution for t � � is

y�t� � e�ty����

If y�t� and �y�t� are two solutions of the test equation then their di�erencefor any t depends on their di�erence at the initial time�

jy�t�� �y�t�j � j�y���� �y����e�tj � jy���� �y���jeRe���t�

We may consider y�t� as the �exact� solution sought� and �y�t� as the solutionwhere the initial data has been perturbed� Clearly then� if Re��� � � thisperturbation di�erence remains bounded at all later times� if Re��� � � itdecays in time� and if Re��� � � the di�erence between the two solutionsgrows unboundedly with t� These possibilities correspond to a stable� anasymptotically stable and an unstable solution� respectively�The precise de�nition for a general ODE system

y� � f�t�y� �����

is more technical� but the spirit is the same as for the test equation� Weconsider ����� for all t � � and de�ne a solution �or trajectory� y�t� to be� stable if given any � � � there is a � � � such that any other solution�y�t� satisfying the ODE ����� and

jy���� �y���j � �

also satis�esjy�t�� �y�t�j � � for all t � �

� asymptotically stable if� in addition to being stable�jy�t�� �y�t�j � � as t���

It would be worthwhile for the reader to compare these de�nitions tothe bound ���� of the fundamental Existence and Uniqueness Theorem ����Note that the existence theorem speaks of a �nite� given integration interval�These de�nitions are given with respect to perturbations in the initial

data� What we really need to be considering are perturbations at any latertime and in the right hand side of ����� as well� These correspond to thebound ����� in Theorem ��� and lead to slightly stronger requirements� Butthe spirit is already captured in the simple de�nitions above� and the morecomplete de�nitions are left to ODE texts�

Page 37: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

�� Chapter�� Initial Value Problems

Example � � Suppose we integrate a given IVP exactly for t � �� thenwe perturb this trajectory at a point t� � h by an amount � � ��h� andintegrate the IVP exactly again for t � t�� starting from the perturbed value�This process� which resembles the e�ect of a numerical discretization step�is now repeated a few times� The question is then� how do the perturbationerrors propagate� In particular� how far does the value of the last trajectorycomputed at t � b � h� get from the value of the original trajectory at t � b�For the test equation ����� we can calculate everything precisely� If

y�t�� � c then y�t� � ce��t�t��� So� starting from y��� � �� we calculatethe trajectories

y�t� � yI�t� � e�t

yII�t� � �e�h � ��e��t�h� � e�t � �e��t�h�

yIII�t� � e�t � �e��t�h�� �e��t��h�

���

For each such step we can de�ne the error due to the jth perturbation�

ej�t� � �e��t�jh��

So� after n steps the di�erence between the original trajectory and the lastone computed at t � nh is

e�t� �nX

j��

ej�t��

Apparently from the form of ej�t�� the errors due to perturbations tend todecrease in time for asymptotically stable problems and to increase in timefor unstable problems� This e�ect is clearly demonstrated in Fig� ���� wherewe took h � ���� � � ��� and plotted curves for the values � � ��� �� ��Note that the instability of y� � y can really generate a huge deviation forlarge t �e�g� t � ���

��� Linear Constant Coe�cient Systems

Here we consider the extension of the test equation analysis to a simple ODEsystem�

y� � Ay �����

Page 38: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Chapter�� On Problem Stability ��

0 0.5 1 1.5 2 2.5 30

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

t

y

�a� Error propagation for y� �y

0 0.5 1 1.5 2 2.5 30

2

4

6

8

10

12

14

16

18

20

t

y

�b� Error propagation for y� y

0 0.5 1 1.5 2 2.5 30.8

0.85

0.9

0.95

1

1.05

t

y

�c� Error propagation for y� �

Figure ���� Errors due to perturbations for stable and unstable test equations�The original� unperturbed trajectories are in solid curves� the perturbed indashed� Note that the y�scales in Figures �a� and �b� are not the same�

Page 39: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

�� Chapter�� Initial Value Problems

where A is a constant m�m matrix� The solution for t � � isy�t� � eAty���� �����

Review� The matrix exponential� The matrix exponential is de��ned via power series expansion by

eAt �

�Xn��

tnAn

n$� I � tA�

t�A�

��t�A�

�� � � � �

If A � T#T��� where # is a diagonal matrix� then it is easy to showthat eAt � TetT��� where et � diag�e�it��

Denote the eigenvalues of A by ��� ��� � � � � �m� and let

# � diagf��� ��� � � � � �mgbe the diagonal m � m matrix having these eigenvalues as its diagonal el�ements� If A is diagonalizable then there exists a similarity transformationthat carries it into #� viz�

T��AT � #�

Then the change of variables

w � T��y

yields the ODE for w

w� � #w�

The system for w is decoupled� for each component wi of w we have a testequation w�i � �iwi� Therefore� the stability for w� hence also for y� isdetermined by the eigenvalues� stability is obtained if Re��i� � �� for alli � �� � � � �m� and asymptotic stability holds if the inequalities are all strict�In the more general case� A may not be similar to any diagonal matrix�

Rather� we face a Jordan canonical form�

T��AT �

�BBBBBB�

#� �

��

� #l

�CCCCCCA

Page 40: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Chapter�� On Problem Stability �

where each Jordan block #i has the form

#i �

�BBBBBB�

�i � �

� �� �

� �i

�CCCCCCA�

A little more is required then� A short analysis which we omit establishesthat in general� the solution of the ODE ����� is

� stable i� all eigenvalues � of A satisfy either Re��� � � or Re��� � �and � is simple �i�e� it belongs to a �� � Jordan block�

� asymptotically stable i� all eigenvalues � of A satisfy Re��� � ��

Example � � Consider the second order ODE

�u�� � u � �

obtained by taking p � q � � in the vibrating spring example� Writing as a�rst order system we obtain

y� �

��� �

� �

�Ay�

The eigenvalues of this matrix are �� � �� and �� � �� Hence this initial value problem is unstable� �Note that in Chapter � we considered thisODE in the context of a boundary value problem� With appropriate boundaryconditions the problem can become stable� as we�ll see in Chapter ��Returning to the experiment of Example ���� here we have one source of

growing error and one source of decreasing error for the IVP� Obviously� aftera su�ciently long time the growing perturbation error will dominate� even ifit starts from a very small deviation �� This is why one �bad� eigenvalue ofA is su�cient for the onset of instability� �

Example � � The general homogeneous� scalar ODE with constant coe�cients�

aku� ak��u� � � � �� a�u�k� � � ����

�orPk

j�� ajdk�judtk�j

� � with a� � �� can be converted as we saw in Chapter �to a �rst order ODE system� This gives a special case of ���� with m � k�

Page 41: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

�� Chapter�� Initial Value Problems

y� � u and

A �

�BBBBBBBBB�

� �

� �

� �� �

�ak�a� �ak���a� �a��a�

�CCCCCCCCCA�

It is easy to verify that the eigenvalues of this matrix are the roots of thecharacteristic polynomial

���� �kX

j��

aj�k�j � �����

The solution of the higher order ODE ���� is therefore�

� stable i� all roots � of the characteristic polynomial satisfy either Re��� �� or Re��� � � and � is simple

� asymptotically stable i� all roots � of the characteristic polynomial satisfy Re��� � ��

��� Linear Variable Coe�cient Systems

The general form of a linear ODE system is

y� � A�t�y� q�t� �����

where the m�mmatrix A�t� and the m�vector inhomogeneity q�t� are givenfor each t� � � t � b�We brie�y review elementary ODE theory� The fundamental solution

Y �t� is the m�m matrix function which satis�es

Y ��t� � A�t�Y �t�� � � t � b ����a�

Y ��� � I ����b�

Page 42: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Chapter�� On Problem Stability ��

i�e�� the jth column of Y �t�� often referred to as a mode� satis�es the homoge�neous version of the ODE ����� with the jth unit vector as initial value� Thesolution of ODE ����� subject to given initial values

y��� � c

is then

y�t� � Y �t�

�c�

Z t

Y ���s�q�s�ds�� �����

Turning to stability� it is clear that for a linear problem the di�erencebetween two solutions y�t� and �y�t� can be directly substituted into ����� inplace of y�t� with the corresponding di�erences in data substituted into theright hand side �say c� �c in place of c�� So the question of stability relatesto the boundedness of y�t� for a homogeneous problem �i�e� with q � �� aswe let b��� Then the solution of the ODE is

� stable i� sup��t�� kY �t�k is bounded

� asymptotically stable i� in addition to being stable� kY �t�k � � ast���

We can de�ne the stability constant

� � sup��t��

kY �t�k

in an attempt to get a somewhat more quantitative feeling� But an examina�tion of ����� suggests that a more careful de�nition of the stability constant�taking into account also perturbations in the inhomogeneity� is

� � sup��s�t��

kY �t�Y ���s�k� ������

Example � � The simple ODE

y� � �cos t�y

has the eigenvalue ��t� � cos t and the fundamental solution Y �t� � esin t�This problem is stable� with a moderate stability constant � � e� � �� eventhough the eigenvalue does not always remain below �� �

Page 43: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

�� Chapter�� Initial Value Problems

��� Nonlinear Problems

A full exposition of stability issues for nonlinear problems is well beyond thescope of this book� A fundamental di�erence from the linear case is that thestability depends on the particular solution trajectory considered�For a given� isolated solution y�t� of ������ � a linear analysis can be ap�

plied locally� to consider trends of small perturbations� Thus� if �y�t� satis�esthe same ODE with �y��� � �c not too far from c� then �under certain condi�tions� we can ignore the higher order term r�t�y� �y� in the Taylor expansion

f�t� �y� � f�t�y� ��f

�y��y� y� � r�t�y� �y�

and consider the linear� variational equation

z� � A�t�y�z ������

for z �not y�� with the Jacobian matrix A � �f�y�

Example � � Often� one is interested in steady state solutions� i�e� wheny�t� becomes independent of t� hence y� � � � f�y�� An example is

y� � y��� y�

which obviously has the steady state solutions y � � and y � �� The Jacobianis A � � � �y� hence A � � for the value y � � and A � � for y � �� Weconclude that the steady state solution y � � is unstable whereas the steadystate solution y � � is stable� Thus� even if we begin the integration of theODE from an initial value close to the steady state y � �� � � c � �� thesolution y�t� will be repelled from it and attracted to the stable steady statey � ��

Since the Jacobian matrix depends on the solution trajectory y�t�� itseigenvalues do not necessarily retain the same sign throughout the integrationinterval� It is then possible to have a system with a bounded solution over anarbitrarily long integration interval� which contains time subintervals whosetotal length is also arbitrarily large� where the system behaves unstably�This is already possible for linear problems with variable coe cients� e�g�Example ���� but it is not possible for the constant coe cient problems ofx���� Through the periods of solution growth� perturbation errors grow aswell� Then� unless the system is su ciently simple so that these errors shrinkagain� they may remain bounded through stable periods only to grow even

�i�e�� there is some tube in which y�t� is the only solution of ������ but no globaluniqueness is postulated�

Page 44: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Chapter�� On Problem Stability ��

further when the system becomes unstable again� This generates an e�ect ofunpredictability� where the e�ect of errors in data grows uncontrollably evenif the solution remains bounded�

Reader�s advice� In the following section we give some briefbackground for material of current research interest� But for thosewho operate on a need�to�know basis we note that this materialappears� in later chapters� only in sections on notes and referencesand in selected exercises�

��� Hamiltonian Systems

A lot of attention has been devoted in recent years to Hamiltonian systems�A Hamiltonian system consists of m � �l di�erential equations�

q�i ��H

�pi�����a�

i � �� � � � � l

p�i � ��H

�qi�����b�

or in vector notation �with rpH denoting the gradient of H with respectto p� etc���

q� �rpH�q�p�� p� � �rqH�q�p� �

The scalar function H�q�p�� assumed to have continuous second derivatives�is the Hamiltonian��

Di�erentiating H with respect to time t and substituting ������ we get

H � �rpHTp� �rqH

Tq� � �

so H�q�p� is constant for all t� A typical example to keep in mind is thatof a conservative system of particles� Then the components of q�t� are thegeneralized positions of the particles� and those of p�t� are the generalizedmomenta� The Hamiltonian H in this case is the total energy �the sum ofkinetic and potential energies�� and the constancy of H is a statement ofconservation of energy�Next� consider an autonomous ODE system of order m � ��

y� � f�y�

�In Chapters and �� we use e instead of H to denote the Hamiltonian�

Page 45: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

�� Chapter�� Initial Value Problems

with y��� � �y����� y�����T B� for some set B in the plane� Each initialvalue y��� � c from B spawns a trajectory y�t� � y�t� c�� and we can followthe evolution of the set B under this �ow�

S�t�B � fy�t� c� � c Bg �

We then ask how the area of S�t�B compares to the initial area of B� doesit grow or shrink in time! It is easy to see for linear problems that this areashrinks for asymptotically stable problems and grows for unstable problems�recall Example ����� It is less easy to see� but it can be shown� that the areaof S�t�B remains constant� even for nonlinear problems� if the divergence off vanishes�

rf � �f��y�

��f��y�

� ��

This remains valid for m � � provided that rf � �� with an appropriateextension of the concept of volume in m dimensions�Now� for a Hamiltonian system with l � ��

q� � Hp� p� � �Hq

we have for rfrf � ��H

�p�q� ��H

�q�p� �

hence the Hamiltonian �ow preserves area� In more dimensions� l � ��it turns out that the area of each projection of S�t�B on a qi � pi plane�i � �� � � � � l� is preserved� and this property is referred to as a symplecticmap�Since a Hamiltonian system cannot be asymptotically stable� its stability

�if it is stable� which is true in case that H can be considered a norm ateach t� e�g� if H is the total energy of a friction�free multibody system� isin a sense marginal� The solution trajectories do not simply decay to a reststate and their long�time behavior is therefore of interest� This leads to someserious numerical challenges�We conclude this brief exposition with a simple example�

Example � The simplest Hamiltonian system is the linear harmonic oscillator� The quadratic Hamiltonian

H ��

��p� � q��

yields the linear equations of motion

q� � �p� p� � ��q

Page 46: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Chapter�� On Problem Stability ��

or ��qp

�A�

� �J

��qp

�A � J �

�� � �

�� �

�A �

Here � � � is a known parameter� The general solution is��q�t�p�t�

�A �

�� cos �t sin�t

� sin�t cos�t

�A��q���p���

�A �

Hence� S�t�B is just a rotation of the set B at a constant rate depending on�� Clearly� this keeps the area of B unchanged�Note that the eigenvalues of J are purely imaginary� Thus� a small �push�

�i�e� a perturbation of the system of these eigenvalues towards the positivehalf plane can make the system unstable�

��� Notes and References

There are many books and papers on the subject of this chapter� The booksby Hairer� Norsett Wanner ���� Mattheij Molnaar ���� and Stuart Humphries ���� treat the theory carefully with computations in mind� so werecommend them in particular� See also ���� ��� �� � ��� For Hamiltoniansystems� see ���� ��� ���

��� Exercises

�� For each of the following constant coe cient systems y� � Ay� deter�mine if the system is stable� asymptotically stable or unstable�

�a� A �

���� �

� ����

�A �b� A �

���� ��

� ��

�A

�c� A �

��� �

� �

�A �d� A �

�� � �

�� �

�A

Page 47: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

�� Chapter�� Initial Value Problems

�� �a� Compute the eigenvalues of the matrix

A�t� �

����

� ��� cos �t � � �

� sin �t

��� ��sin �t ��

�� �

�cos �t

�A

�b� Determine whether the variable coe cient system y� � A�t�y isstable� asymptotically stable or unstable�

�You may want to use T �t� �

�� cos t sin t

� sin t cos t

�A ��

�� The Lyapunov function is an important tool for analyzing stabilityof nonlinear problems� The scalar� C��function V �y� is a Lyapunovfunction at %y if

d

dtV �y�t�� � � ������

for all y in a neighborhood of %y� If also V �%y� � � and V �y� � � in theneighborhood then V is a positive de�nite Lyapunov function at %y�

It can be shown that if %y is a steady state solution of ����� then %y isstable if there is a corresponding positive de�nite Lyapunov function�If the inequality in ������ is sharp �except at y � %y� then the steadystate solution is asymptotically stable�

�a� Construct a suitable Lyapunov function to show that %y � � isstable in Example ��� �You should �nd it di cult to construct asimilar function for the other steady state� %y � �� for this exam�ple��

�b� Let U�y� be a smooth� scalar function with a minimum at %y �notethat y is not necessarily scalar�� and consider the system

y� � �ryU � ��U�y

Show that %y is a stable steady state solution of this nonlinearODE system�

�� Consider a nonlinear ODE system ����� which has an invariant setMde�ned by the equations

h�t�y� � � ������

i�e�� assuming that the initial conditions satisfy h���y���� � �� thesolution of the ODE satis�es h�t�y�t�� � � for all later times t � ��

Page 48: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Chapter�� On Problem Stability ��

Let us assume below� to save on notation� that f and h are autonomous�De�ne the Jacobian matrix

H�y� ��h

�y

and assume that it has full row rank for all t �in particular� there areno more equations in ������ than in �������

Next we stabilize the vector �eld� replacing the autonomous ����� by

y� � f�y�� HT �HHT ���h�y�� �����

�a� Show that if h�y���� � � then the solution of ����� coincides withthat of the original y� � f�y��

�b� Show that if there is a constant � such that

jHf�y�j� � �jh�y�j�for all y in the neighborhood of the invariant set M then Mbecomes asymptotically stable� i�e� jh�y�t��j decreases in t fortrajectories of ����� starting nearM� provided that � ��

Page 49: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

�� Chapter�� Initial Value Problems

Page 50: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Chapter �

Basic Methods� Basic Concepts

We begin our discussion of numerical methods for initial value ODEs with anintroduction of the most basic concepts involved� To illustrate these concepts�we use three simple discretization methods� forward Euler� backward Euler�also called implicit Euler�� and trapezoidal� The problem to be solved iswritten� as before� in the general form

y� � f�t�y�� � � t � b �����

with y��� � c given� You can think of this at �rst as a scalar ODE � mostof what we are going to discuss generalizes to systems directly� and we willhighlight occasions where the size of the system is important�We will assume su cient smoothness and boundedness on f�t�y� so as

to guarantee a unique existence of a solution y�t� with as many boundedderivatives as referred to in the sequel� This assumption will be relaxed inx����

��� A Simple Method Forward Euler

To approximate ������ we �rst discretize the interval of integration by a mesh

� � t� � t� � � � � � tN�� � tN � b

and let hn � tn�tn�� be the nth step size� We then construct approximations

y��� c�� y�� � � � �yN���yN

with yn an intended approximation of y�tn��

Page 51: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

�� Chapter �� Initial Value Problems

In the case of an initial value problem we know y� and may proceed tointegrate the ODE in steps� where on each step n �� � n � N� we knowan approximation yn�� at tn�� and we seek yn at tn� Thus� as we progresstowards tn we do not need an advance knowledge of the entire mesh beyondit �or even of N � for that matter�� Let us concentrate on one such step� n�� ���

Review� Order notation� Throughout this book we considervarious computational errors depending on a discretization step�size h � �� and ask how they decrease as h decreases� We denotefor a vector d depending on h

d � O�hp�

if there are two positive constants p and C such that for all h � �small enough�

jdj � Chp�

For example� comparing ����� and ����� we see that in ����� theorder notation involves a constant C which bounds �

�ky��k�

In other instances� such as when estimating the e ciency of aparticular algorithm� we are interested in a bound on the workestimate as a parameterN increases unboundedly �e�g� N � ��h��For instance�

w � O�N logN�

means that there is a constant C such that

w � CN logN

as N � �� It will be easy to �gure out from the context whichof these two meanings is the relevant one�

To construct a discretization method consider Taylor�s expansion

y�tn� � y�tn��� � hny��tn��� �

�h�ny

���tn��� � � � � �����

which we can also write� using the order notation� as

y�tn� � y�tn��� � hny��tn��� �O�h�n�� �����

The forward Euler method can be derived by dropping the rightmost termin this Taylor expansion and replacing y� by f � yielding the scheme

yn � yn�� � hnf�tn���yn���� �����

Page 52: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Chapter �� Basic Methods� Basic Concepts ��

This is a simple� explicit method � starting from y� � c we apply �����iteratively for n � �� �� � � � � N � The e�ect of the approximation is depictedin Fig� ���� The curved lines represent a family of solutions for the ODE

0 0.5 1 1.5 2 2.5 3 3.5 40

1

2

3

4

5

6

7

8

9

10

t

y

Figure ���� The forward Euler method� The exact solution is the curved solidline� The numerical values are circled� The broken line interpolating themis tangential at the beginning of each step to the ODE trajectory passingthrough that point �dashed lines��

with di�erent initial values� At each step� the approximate solution yn�� ison one of these curves at tn��� The forward Euler step amounts to takinga straight line in the tangential direction to the exact trajectory starting at�tn���yn���� continuing until the end of the step� �Recall Example ����� Onehopes that if h is small enough then yn is not too far from y�tn�� Let usassess this hope�

Page 53: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

�� Chapter �� Initial Value Problems

��� Convergence Accuracy Consistency and

��Stability

We now rewrite Euler�s method ����� in a form compatible with the approx�imated ODE�

yn � yn��hn

� f�tn���yn��� � ��

To formalize a bit� let the di�erence operator

Nhu�tn� u�tn�� u�tn���hn

� f�tn���u�tn���� ����

be applied for n � �� �� � � � � N for any function u de�ned at mesh points withu�t�� speci�ed� and consider yh to be a mesh function which takes on thevalue yn at each tn� n � �� �� � � � � N � Then the numerical method is given by

Nhyh�tn� � �

�with y� � c��Much of the study of numerical ODEs is concerned with the errors on each

step that are due to the di�erence approximation� and how they accumulate�One measure of the error made on each step is the local truncation error�It is the residual of the di�erence operator when it is applied to the exactsolution�

dn � Nhy�tn�� �����

The local truncation error measures how closely the di�erence operator ap�proximates the di�erential operator� This de�nition of the local truncationerror applies to other discretization methods as well �they di�er from oneanother in the de�nition of the di�erence operator�� The di�erence methodis said to be consistent �or accurate� of order p if

dn � O�hpn� �����

for a positive integer p�For the forward Euler method ������ the Taylor expansion ����� yields

dn �hn�y���tn� �O�h�n�

so the method is consistent of order ��A straightforward design of di�erence approximations to derivatives nat�

urally leads to consistent approximations to di�erential equations� However�our real goal is not consistency but convergence� Let

h � max��n�N

hn

Page 54: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Chapter �� Basic Methods� Basic Concepts ��

and assume Nh is bounded independent of N � The di�erence method is saidto be convergent of order p if the global error en� where en � yn � y�tn��e� � �� satis�es

en � O�hp� �����

for n � �� �� � � � � N � The positive integer p does not really have to be thesame as the one in ����� for the de�nition to hold� But throughout this bookwe will consider methods where the order of convergence is inherited fromthe order of accuracy� For this we need �stability�The di�erence method is �stable if there are positive constants h� and

K such that for any mesh functions xh and zh with h � h��

jxn � znj � Kfjx� � z�j� max��j�N

jNhxh�tj��Nhzh�tj�jg� � � n � N�

�����

What this bound says in e�ect is that the di�erence operator is invertible� andthat its inverse is bounded by K� Note the resemblance between ����� andthe bound ����� which the di�erential operator satis�es� The bound in �����measures the e�ect on the numerical solution of small perturbations in thedata� The importance of this requirement lies in the following fundamentaltheorem �

Theorem � �

consistency � �stability� convergence

In fact� if the method is consistent of order p and �stable� then it is convergent of order p�

jenj � K maxjjdjj � O�hp�� ������

The proof of this fundamental theorem is immediate� simply let xn �yn and zn � y�tn� in the stability bound ������ and use the de�nitions ofaccuracy and local truncation error� �

Turning to the forward Euler method� by this fundamental convergencetheorem we will obtain convergence of order � �assuming that a bounded y��

exists� if we show that the ��stability bound ����� holds� To see this� denote

sn � xn � zn� � � max��j�N

jNhxh�tj��Nhzh�tj�j�

Then for each n�

� �����sn � sn��

hn� �f�tn���xn���� f�tn��� zn����

����� jsnj

hn�����sn��hn

� �f�tn���xn���� f�tn��� zn����

���� �

Page 55: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

�� Chapter �� Initial Value Problems

Using Lipschitz continuity�����sn��hn� �f�tn���xn���� f�tn��� zn����

���� � jsn��jhn

� Ljsn��j ���

hn� L

�jsn��j

so that�

jsnj � �� � hnL�jsn��j� hn�

� �� � hnL���� � hn��L�jsn��j� hn���� � hn�

� � � �

� �� � h�L� � � � �� � hn��L��� � hnL�js�j� �

nXj��

hj�� � hj��L� � � � �� � hnL�

� eLtnjs�j� �

L�eLtn � ����

The last inequality above is obtained by noting that � � hL � eLh implies

�� � hj��L� � � � �� � hnL� � eL�tn�tj� � � j � n

and also�

nXj��

hjeL�tn�tj� �

nXj��

Z tj

tj��

eL�tn�t�dt � eLtnZ tn

e�Ltdt ��

L�eLtn � ���

The stability bound is therefore satis�ed� with K � max�eLb� �

L�eLb � ��

in ������It is natural to ask next if the error bound ������ is useful in practice� i�e��

if it can be used to reliably estimate the step size h needed to achieve a givenaccuracy� This is a tempting possibility� For instance� letM be an estimatedbound on ky��k� Then the error using forward Euler can be bounded by

jenj � hM

�L�eLtn � ��� � � n � N� ������

However� it turns out that this bound is too pessimistic in many applications�as the following example indicates�

Example � � Consider the scalar problem

y� � �ty� � t� �

t�� y��� � �

for � � t � �� �Note that the starting point of the integration is t � �� nott � � as before� But this is of no signi�cance� just change the independentvariable to � � t� �� The exact solution is y�t� � �

t�

Page 56: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Chapter �� Basic Methods� Basic Concepts ��

To estimate the Lipschitz constant L� note that near the exact solution�

fy � ���ty � ����

Similarly� use the exact solution to estimate M � � � ��t�� Substituting thisinto ����� yields the bound

jenj � h

��e���tn���

so jeN j � h��e���� not a very useful bound at all� �

We will be looking in later chapters into the question of realistic errorestimation�

We close this section by mentioning another� important measure of theerror made at each step� the local error� It is de�ned as the amount by whichthe numerical solution yn at each step di�ers from the solution %y�tn� to theinitial value problem

%y��t� � f�t� %y�t�� ������

%y�tn��� � yn�� �

Thus the local error is given by

ln � %y�tn�� yn� ������

Under normal circumstances� it can be shown that the numerical solutionexists and

jdnj � jNh%y�tn�j�O�hp����

Moreover� it is easy to show� for all of the numerical ODE methods consideredin this book� that �

hnjNh%y�tn�j � jlnj�� �O�hn��� ������

The two local error indicators� hndn and ln� are thus often closely related�

�We caution here that for sti problems� to be discussed in x���� the constant impliedin this O�hn� may be quite large�

Page 57: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

�� Chapter �� Initial Value Problems

��� Absolute Stability

Example ��� may make one wonder about the meaning of the fundamentalconvergence Theorem ���� The theorem is not violated� we still have thatjenj � Ch for some constant C� even if large� so as h� �� jenj � �� However�the theoremmay not give a quantitative indication of what happens when weactually compute with a step size h not very small� �The name ���stability�now becomes more intuitive � this concept deals with the limit of h � ���The basic reason why the constant in this example is so pessimistically largeis that while fy � ���� i�e� the exact solution mode is decaying� the stabilitybound uses the Lipschitz constant L � ��� and consequently is exponentiallyincreasing�For large step�sizes� the di�erence equation should mimic the behavior of

the di�erential equation in the sense that their stability properties should besimilar�What stability requirements arise for h which is not vanishingly small!

Consider the scalar test equation

y� � �y �����

where � is a complex constant �complex because later we will be looking atODE systems� and there � corresponds to an eigenvalue�� If y��� � c �sayc � � for notational convenience�� then the exact solution is

y�tn� � ce�tn

whereas Euler�s method� with a uniform step size hn � h� gives

yn � yn�� � h�yn�� � �� � h��yn�� � � � � � c�� � h��n�

Let us distinguish between three cases �cf� Example �����

� If Re��� � �� then jy�t�j � ceRe���t grows exponentially with t� Thisis an unstable problem� although for eRe���b not too large� one can stillcompute solutions which are meaningful in the relative sense� In thiscase� the error bound ������ is realistic� For unstable problems� thedistance between solution curves increases in time�

� If Re��� � �� the solution is oscillating �unless � � �� and the distancebetween solution curves stays the same�

� If Re��� � �� then jy�t�j decays exponentially� The distance betweensolution curves decreases� The problem is �asymptotically� stable� andwe cannot tolerate growth in jynj� This is usually the interesting case�and it yields an additional absolute stability requirement�

jynj � jyn��j� n � �� �� � � � ������

Page 58: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Chapter �� Basic Methods� Basic Concepts ��

For a given numerical method� the region of absolute stability is that regionof the complex z�plane such that applying the method for the test equation������ with z � h� from within this region� yields an approximate solutionsatisfying the absolute stability requirement �������For the forward Euler method we obtain the condition

j� � h�j � � ������

which yields the region of absolute stability depicted in Fig� ���� For instance�

−4 −3.5 −3 −2.5 −2 −1.5 −1 −0.5 0 0.5 1−2

−1.5

−1

−0.5

0

0.5

1

1.5

2

Re(z)

Im(z

)

Stability regions in the complex z−plane

stable

Figure ���� Absolute stability region for the forward Euler method�

if � is negative� then h must be restricted to satisfy

h � �

���

For Example ��� this gives h � �� � In this case the restriction is notpractically unbearable� To see the e�ect of violating the absolute stabilityrestriction� we plot in Fig� ��� the approximate solutions obtained withuniform step sizes h � ��� and h � ��� � For h � ��� the solution pro�le lookslike the exact one �y � ��t�� The other� oscillatory pro�le� is obtained forh � ���� which is outside the absolute stability region� When computing withh � �� using the same forward Euler method and �oating point arithmeticwith a ���hexadecimal�digit mantissa �this is the standard �double precision�in IEEE FORTRAN� for example�� the computed solution oscillates and thenblows up �i�e�� over�ow is detected� before reaching t � � �

Page 59: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

�� Chapter �� Initial Value Problems

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0 5 10 15 20 25

Figure ���� Approximate solutions for Example ��� using the forward Eulermethod� with h � ��� and h � ��� � The oscillatory pro�le corresponds toh � ���� for h � ��� the qualitative behavior of the exact solution is obtained�

It is important to understand that the absolute stability restriction isindeed a stability� not accuracy� requirement� Consider the initial valuec � ���� for the test equation with Re��� � �� so the exact solution isapproximated very well by the constant �� Such an initial value correspondsto an unavoidable perturbation in the numerical method� due to roundo�errors� Now� the forward Euler solution corresponding to this initial pertur�bation of � remains very close to � for all tn � �� like the exact solution�when using any h from the absolute stability region� but it blows up as nincreases if h is from outside that region� i�e� if j� � h�j � ��

The concept of absolute stability was de�ned with respect to a very sim�ple test equation ������ an ODE whose numerical solution is not a com�putationally challenging problem in itself� Nonetheless� it turns out thatabsolute stability gives useful information� at least qualitatively� in more gen�eral situations� where complicated systems of nonlinear ODEs are integratednumerically�

Page 60: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Chapter �� Basic Methods� Basic Concepts �

Reader�s advice� Those readers who are prepared to trust uson the above statement may wish to skip the rest of this section�at least on �rst reading �especially if your linear algebra needssome dusting��

We now consider the extension of the test equation analysis to a simpleODE system�

y� � Ay ������

where A is a constant� diagonalizable� m�m matrix�Denote the eigenvalues of A by ��� ��� � � � � �m� and let

# � diagf��� ��� � � � � �mgbe the diagonal m � m matrix composed of these eigenvalues� Again� theinteresting case is when ������ is stable� i�e� Re��j� � �� j � �� � � � �m� Thediagonalizability of A means that there is a nonsingular matrix T � consistingof the eigenvectors of A �scaled to have unit Euclidean norm� say�� such that

T��AT � #�

Consider the following change of dependent variables�

w � T��y�

For w�t� we obtain� upon multiplying ������ by T�� and noting that T isconstant in t� the decoupled system

w� � #w� ������

The components of w are separated and for each component we get a scalarODE in the form of the test equation ����� with � � �j � j � �� � � � �m�Moreover� since A and therefore T are constant� we can apply the sametransformation to the discretization� Let wn � T��yn� all n� Then theforward Euler method for �������

yn � yn�� � hnAyn��

transforms into

wn � wn�� � hn#wn��

which is the forward Euler method for ������� The same commutativity ofthe discretization and the w�transformation �in the case that T is constant$�holds for other discretization methods as well�

Page 61: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

�� Chapter �� Initial Value Problems

Now� for the decoupled system ������� where we can look at each scalarODE separately� we obtain that if hn are chosen such that h��� h��� � � � � h�mare all in the absolute stability region of the di�erence method �recall h �maxn hn�� then

jwnj � jwn��j � � � � � jw�jso

jynj � kTkjwnj � � � � � kTkjw�j � kTkkT��kjy�j�Denoting by

cond�T � � kTkkT��k ������

the condition number of the eigenvector matrix T �measured in the norminduced by the vector norm used for jynj�� we obtain the stability bound

jynj � cond�T �jcj� n � �� �� � � � � N ������

�recall y��� � c��Note that in general the stability constant cond�T � is not guaranteed

to be of moderate size� although it is independent of n� and it may oftendepend on the size m of the ODE system� An additional complication ariseswhen A is not diagonalizable� The considerations here are very similar tothose arising in eigenvalue sensitivity analysis in linear algebra� Indeed theessential question is similar too� how representative are the eigenvalues ofthe properties of the matrix A as a whole!But there are important special cases where we encounter more favorable

winds� If A is �real and� symmetric� then not only are its eigenvalues real�also its eigenvectors are orthogonal to one another� We may therefore chooseT to be orthogonal� i�e��

T�� � T T �

In this case it is advantageous to use the Euclidean norm l�� because we get

cond�T � � �

regardless of the size of the system� Thus� if hmin��j�m �j is in the abso�lute stability region of the di�erence method� then ������ yields the stabilitybound in the l� norm�

ynTyn � cTc� � � n � N� ������

The importance of obtaining a bound on cond�T � which is independentof m increases� of course� when m is large� Such is the case for the method of

Page 62: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Chapter �� Basic Methods� Basic Concepts ��

lines �Examples ��� and ����� where the ODE system arises from a spatiallydiscretized time�dependent PDE� In this case m is essentially the number ofspatial grid points� This is worked out further for some instances in Exercises��� and ����

��� Sti�ness Backward Euler

Ideally� the choice of step size hn should be dictated by approximation ac�curacy requirements� But we just saw that when using the forward Eulermethod �and� as it turns out� many other methods too�� hn must be cho�sen su ciently small to obey an additional� absolute stability restriction�as well� Loosely speaking� the initial value problem is referred to as beingsti� if this absolute stability requirement dictates a much smaller step sizethan is needed to satisfy approximation requirements alone� In this case othermethods� which do not have such a restrictive absolute stability requirement�should be considered�To illustrate this� consider a simple example�

Example � � The scalar problem

y� � �����y � sin t�� t � �� y��� � �

has a solution which starts at the given initial value and varies rapidly� Butafter a short while� say for t � ����� y�t� varies much more slowly� satisfying y�t� � sin t� see Fig� ���� For the initial small interval of rapid change�commonly referred to as an initial layer or transient� we expect to use smallstep sizes� so that ���hn � �� say� This is within the absolute stability regionof the forward Euler method� But when y�t� � sin t� accuracy considerationsalone allow a much larger step size� so we want ���hn �� A reasonablemesh is plotted using markers on the t axis in Fig� ���� Obviously� however� the plotted solution in this �gure was not found using the forward Eulermethod �but rather� using another method with this mesh� because the absolute stability restriction of the forward Euler method is severely violated here��

Scientists often describe sti�ness in terms of multiple time scales� If theproblem has widely varying time scales� and the phenomena �or� solutionmodes� that change on fast scales are stable� then the problem is sti�� Forexample� controllers are often designed to bring a system rapidly back to asteady state and are thus a source of sti�ness� In chemically reacting systems�sti�ness often arises from the fact that some chemical reactions occur muchmore rapidly than others�

Page 63: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

�� Chapter �� Initial Value Problems

0

0.2

0.4

0.6

0.8

1

0 0.5 1 1.5 2 2.5 3++++++++++++++++++++++++ + + + + + + + + + + + + + + + + +

Figure ���� Approximate solution and plausible mesh� Example ����

The concept of sti�ness is best understood in qualitative� rather thanquantitative� terms� In general� sti�ness is de�ned in terms of the behaviorof an explicit di�erence method� and the behavior of forward Euler is typicalof such methods�

De�nition � � An IVP is sti� in some interval ��� b� if the step size neededto maintain stability of the forward Euler method is much smaller than thestep size required to represent the solution accurately�

We note that sti�ness depends� in addition to the di�erential equationitself� on the

� accuracy criterion� length of the interval of integration� region of absolute stability of the methodIn Example ���� for a moderate error tolerance� the problem is sti� after

about t � ����� If it were required to solve the problem to great accuracy�then it would not be sti� because the step size would need to be small inorder to attain that accuracy� and hence would not be restricted by stability�For stable� homogeneous� linear systems� sti�ness can be determined by

the system�s eigenvalues� For the test equation ����� on ��� b�� we say that

Page 64: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Chapter �� Basic Methods� Basic Concepts ��

the problem is sti� if

bRe������� ������

Roughly� the general ODE system ����� is sti� in a neighborhood of thesolution y�t� if there exists for some bounded data a component of y whichdecays rapidly on the scale of the interval length b� In the general case�sti�ness can often be related to the eigenvalues �j of the local Jacobianmatrix fy�t�y�t��� generalizing ������ to

bminjRe��j����� ������

Thus� we look for methods which do not violate the absolute stabilityrequirement when applied to the test equation ������ even when hRe������� Such a method is the backward Euler method� It is derived for thegeneral ODE ����� just like the forward Euler method� except that everythingis centered at tn� rather than at tn��� This gives the �rst�order method

yn � yn�� � hnf�tn�yn�� �����

Geometrically� instead of using the tangent at �tn���yn���� as in the for�ward Euler method� the backward Euler method uses the tangent at thefuture point �tn�yn�� thus enhancing the stability� The local truncation errorof this method is similar in magnitude to that of the forward Euler method�and correspondingly the convergence bound ������ is similar too �we leavethe ��stability proof in this case as an exercise�� The two major di�erencesbetween these simple methods are�

� While the forward Euler method is explicit� the backward Euler methodis implicit� the unknown vector yn at each step appears on both sides ofthe equation ������ generally in a nonlinear expression� Consequently�a nonlinear system of algebraic equations has to be �approximately�solved at each step� That�s the bad news for backward Euler�

� The good news is the method�s stability� Applying the backward Eulermethod ����� to the test equation� we obtain

yn � yn�� � h�yn

i�e��

yn � �� � h����yn���

The ampli�cation factor� i�e� what multiplies jyn��j to get jynj in abso�lute value� satis�es

j� � h�j � �

Page 65: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

� Chapter �� Initial Value Problems

for all values of h � � and � satisfying Re��� � �� In particular� thereis no absolute stability prohibition from taking hRe��� � ��� e�g� inExample ����

The region of absolute stability of the backward Euler method is depictedin Fig� ��� It contains� in addition to the entire left half�plane of z � h��

−4 −3 −2 −1 0 1 2 3 4−3

−2

−1

0

1

2

3

Re(z)

Im(z

)

Stability regions in the complex z−plane

stability OUTSIDE

shaded area

Figure ��� Absolute stability region for the backward Euler method�

also a major part of the right half�plane� �The latter is a mixed blessing�though� as will be discussed later on�� For a given sti� problem� the back�ward Euler method needs fewer steps than the forward Euler method� Ingeneral� however� each backward Euler step may be more expensive in termsof computing time� Still� there are many applications where the overall com�putational expense using the implicit method is much less than with theexplicit Euler method�

For an implicit method like backward Euler� a nonlinear system of equa�tions must be solved in each time step� For backward Euler� this nonlinearsystem is

g�yn� � yn � yn�� � hf�tn�yn� � �

�where h � hn for notational simplicity�� There are a number of ways tosolve this nonlinear system� We mention two basic ones�

Page 66: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Chapter �� Basic Methods� Basic Concepts �

Functional iteration

Our �rst impulse might be to solve the nonlinear system by functional iter�ation� This yields

y���n � yn�� � hf�tn�y

�n�� � � �� �� � � �

where we can choose y�n � yn��� for instance� �Note that � is an iteration

counter� not a power��The advantage here is simplicity� However� the convergence of this it�

eration requires hk�f��yk � � in some norm�� For sti� systems� �f��y islarge� so the step size h would need to be restricted and this would defeatthe purpose of using the method�

Example � � Let us generalize the ODE of Example ��� to

y� � ��ty� � t��� � t��� t � �

where � � � is a parameter� With y��� � �� the exact solution is stilly�t� � �

t� The backward Euler method gives a nonlinear equation for yn�

yn � yn�� � hn��tny�n � t��n �� hnt

��n � ������

and functional iteration reads

y���n � yn�� � hn��tn�y

�n�

� � t��n �� hnt��n � � � �� �� � � � � ������

The question is� under what conditions does the iteration ����� convergerapidly�Subtracting ����� from ����� and denoting ��n � yn � y�n� we get

����n � hn�tn�y

�n � �y�n���

� hn�tn�yn � y�n���n � �hn���n� � � �� �� � � � �

This iteration obviously converges i� j����n j � j��nj� and the approximate con

dition for this convergence is therefore

hn ��

�j�j �

The convergence is rapid if hn � ��j�j� Now� if � � �� as in Example ����

then convergence of this nonlinear iteration is obtained with h � ���� andchoosing h � ��� yields rapid convergence �roughly� one additional signi�cantdigit is gained at each iteration� But if � � ��� then we must take h � ����for convergence of the iteration� and this is a harsh restriction� given thesmoothness and slow variation of the exact solution� Functional iteration istherefore seen to be e�ective only in the nonsti� case� �

Functional iteration is often used in combination with implicit methodsfor the solution of nonsti� problems� as we will see later� in Chapter �

�This would yield a contraction mapping� and therefore convergence as � �� to the�xed point yn�

Page 67: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

� Chapter �� Initial Value Problems

Newton iteration

Variants of Newton�s method are used in virtually all modern sti� ODEcodes�

Given the nonlinear system

g�yn� � yn � yn�� � hf�tn�yn� � ��

Newton�s method yields

y���n � y�n �

��g

�y

���g�y�n�

� y�n ��I � h

�f

�y

����y�n � yn�� � hf�tn�y

�n��� � � �� �� � � � �

Page 68: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Chapter �� Basic Methods� Basic Concepts �

Review� Newton�s method� For a nonlinear equation

g�x� � �

we de�ne a sequence of iterates as follows� x� is an initial guess�For a current iterate x�� we write

� � g�x� � g�x�� � g��x���x� x�� � � � � �

Approximating the solution x by neglecting the higher order termsin this Taylor expansion� we de�ne the next iterate x��� by thelinear equation

� � g�x�� � g��x���x��� � x���

We can generalize this directly to a system of m algebraic equa�tions in m unknowns�

g�x� � ��

Everything remains the same� except that the �rst derivative ofg is replaced by the m �m Jacobian matrix �g

�x� We obtain the

iteration

x��� � x� ���g

�x�x��

���g�x��� � � �� �� � � � �

We note that it is not good practice to compute a matrix inverse�Moreover� rather than computing x��� directly� it is better in cer�tain situations �when ill�conditioning is encountered�� and neverworse in general� to solve the linear system for the di�erence �between x��� and x�� and then update� Thus� � is computed �foreach �� by solving the linear system�

�g

�x

�� � �g�x��

where the Jacobian matrix is evaluated at x�� and the next New�ton iterate is obtained by

x��� � x� � � �

The matrix �I �h�f��y� is evaluated at the current iterate y�n� This matrixis called the iteration matrix� and the costs of forming it and solving thelinear system �for � � y���

n � y�n� often dominate the costs of solving the

Page 69: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

� Chapter �� Initial Value Problems

problem� We can take the initial guess

y�n � yn��

although better ones are often available� Newton�s method is iterated untilan estimate of the error due to terminating the iteration is less than a user�speci�ed tolerance� for example

jy���n � y�nj � NTOL�

The tolerance NTOL is related to the local error bound that the user aimsto achieve� and is usually well above roundo� level� Because there is a veryaccurate initial guess� most ODE initial value problems require no more thana few Newton iterations per time step� A strategy which iterates no morethan� say� � times� and if there is no convergence decreases the step sizehn �thus improving the initial guess� and repeats the process� can be easilyconceived� We return to these ideas in the next two chapters�

Newton�s method works well for Example ���� without a severe restrictionon the time step�

Newton�s method requires the evaluation of the Jacobian matrix� �f�y�

This presents no di culty for Example ���� however� in practical applica�tions� specifying these partial derivatives analytically is often a di cult orcumbersome task� A convenient technique is to use di�erence approximations� at y � y�n� evaluate �f � f�tn� �y� and &f � f�tn� &y�� where �y and&y are perturbations of y in one coordinate� �yj � yj � �� &yj � yj � �� and�yl � &yl � yl� l �� j� Then the jth column of �f

�ycan be approximated by

�f

�yj� �

����f � &f��

where � is a small positive parameter�

This very simple trick is very easy to program and it does not a�ect the ac�curacy of the solution yn� It often works very well in practice with the choice� � ���d� if �oating point arithmetic with roughly �d signi�cant digits is be�ing used �e�g� d � ��� The technique is useful also in the context of boundaryvalue problems� see for instance x��� and x������ It does not always work well�though� and moreover� such an approximation of the Jacobian matrix may attimes be relatively expensive� depending on the application� But it gives theuser a simple technique for computing an approximate Jacobian matrix whenit is needed� Most general�purpose codes provide a �nite�di�erence Jacobianas an option� using a somewhat more sophisticated algorithm to select theincrement�

Page 70: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Chapter �� Basic Methods� Basic Concepts

Review� Matrix decompositions� Consider a linear system ofm equations

Ax � b

where A is real� square and nonsingular� b is given and x is asolution vector to be found� The solution is given by

x � A��b�

However� it is usually bad practice to attempt to form A���The well�known algorithm of Gaussian elimination �without piv�oting� is equivalent to forming an LUdecomposition of A�

A � LU

where L is a unit lower triangular matrix �i�e� lij � �� i � j� andlii � �� and U is upper triangular �i�e� uij � �� i � j�� Note thatthis decomposition is independent of the right hand side b� It canbe done without knowing b and it can be used for more than oneright hand side� The LU �decomposition requires �

�m� � O�m��

�ops �i�e�� elementary �oating�point operations��Given a data vector b we can now �nd x by writing

L�Ux� � Ax � b �

Solving Lz � b for z involves forward substitution and costsO�m�� �ops� Subsequently solving Ux � z completes the so�lution process using a back substitution and another O�m�� �ops�The solution process is therefore much cheaper when m is largethan the cost of the decomposition�Not every nonsingular matrix has an LU �decomposition� and evenif there exists such a decomposition the numerical process may be�come unstable� Thus� partial pivoting must be applied �unless thematrix has some special properties� e�g� it is symmetric positivede�nite�� A row�partial pivoting involves permuting rows of A toenhance stability and results in the decomposition

A � PLU

where P is a permutation matrix �i�e� the columns of P are them unit vectors� in some permuted order�� We will refer to an LU �decomposition� assuming that partial pivoting has been appliedas necessary�

Page 71: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

� Chapter �� Initial Value Problems

Review� Matrix decompositions� continued� Another impor�tant matrix decomposition is the QRdecomposition

A � QR

where R is upper triangular �like U� and Q is orthogonal�QTQ � I� This decomposition costs twice as much as the LU �decomposition� but it has somewhat better stability properties�because kQk� � kQ��k� � �� which implies an ideal condition�ing� cond�Q� � � �see �������� This is useful also for �ndingleast squares solutions to over�determined linear systems and asa building block in algorithms for �nding matrix eigenvalues�

If A is large and sparse �i�e� most of its elements are zero� thenthe LU and the QR decompositions may or may not remain suit�able� For instance� if all the nonzero elements of A are containedin a narrow band� i�e� in a few diagonals along the main diagonal�whence A is called banded� then both the LU and the QR algo�rithms can be easily adjusted to not do any work outside the band�For boundary value ODEs this typically leads to a reduction inthe algorithm�s complexity from cubic to linear in the matrix di�mension� But inside the band the sparsity is usually lost� andother� iterative algorithms become more attractive� The latter istypically the case for elliptic PDEs� and is outside the scope ofour book�

��� A�Stability Sti� Decay

Ideally� one would desire that a numerical discretization method mimic allproperties of the di�erential problem to be discretized� for all problems� Thisis not possible� One then lowers expectations� and designs discretizationmethods which capture the essential properties of a class of di�erential prob�lems�A �rst study of absolute stability suggests that� since for all stable test

equations� jy�tn�j � jy�tn���j� a good discretization method for sti� problemsshould do the same� i�e� satisfy jynj � jyn��j�This gives the concept of A�Stability� A di�erence method is Astable if

its region of absolute stability contains the entire left half�plane of z � h��

Page 72: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Chapter �� Basic Methods� Basic Concepts �

A glance at Figs� ��� and �� indicates that the backward Euler method isA�stable� whereas the forward Euler method is not�But a further probe into A�stability reveals two de�ciencies� The �rst is

that it does not distinguish between the cases

Re������and

��� Re��� � �� jIm���j � �

The latter case gives rise to a highly oscillatory exact solution� which doesnot decay much� The di culties arising are of a di�erent type� so whenaddressing sti�ness of the type that we have been studying� it is not essentialto include points near the imaginary axis in the absolute stability region ofthe di�erence method�The second possible weakness of the A�stability de�nition arises from its

exclusive use of absolute stability� In the very�sti� limit� hnRe��� � ���the exact solution of the test equation satis�es jy�tn�j � jy�tn���jehnRe��� �jy�tn���j� The corresponding absolute stability requirement� jynj � jyn��j�seems anemic in comparison� since it does not exclude jynj � jyn��j�Let us generalize the test equation a bit� to include an inhomogeneity�

y� � ��y � g�t�� ������

where g�t� is a bounded� but otherwise arbitrary� function� We can rewrite������ as

�y� � ���y � g�t��

where � � �jRe���j�

�� � ��� and note that the reduced solution� obtained for

� � �� is y�t� � g�t�� This motivates saying that the discretization methodhas sti� decay if for tn � � �xed�

jyn � g�tn�j � � as hnRe������� ������

This is a stronger requirement than absolute stability in the very�sti�limit� and it does not relate to what happens elsewhere in the h��plane� Thebackward Euler method has sti� decay� because when applied to ������ ityields

yn � g�tn� � �� � hn�����yn�� � g�tn���

The forward Euler method of course does not have sti� decay�The practical advantage of methods with sti� decay lies in their ability

to skip �ne�level �i�e� rapidly varying� solution details and still maintain a

Page 73: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

� Chapter �� Initial Value Problems

-0.6

-0.4

-0.2

0

0.2

0.4

0.6

0.8

1

1.2

0 0.5 1 1.5 2 2.5 3 3.5

Figure ���� Approximate solution on a coarse uniform mesh for Example ����using backward Euler �the smoother curve� and trapezoidal methods�

decent description of the solution on a coarse level in the very�sti� �not thehighly oscillatory$� case� For instance� using backward Euler with a �xedstep h � �� to integrate the problem of Example ���� the initial layer is poorlyapproximated� and still the solution is qualitatively recovered where it variesslowly� see Fig� ���� Herein lies a great potential for e cient use� as well asa great danger of misuse� of such discretization methods�

��� Symmetry Trapezoidal Method

The forward Euler method was derived using a Taylor expansion centeredat tn��� The backward Euler method was likewise derived� centered at tninstead� Both methods are �rst order accurate� which is often insu cientfor an e cient computation� Better accuracy is obtained by centering theexpansions at tn���� � tn � �

�hn�

Page 74: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Chapter �� Basic Methods� Basic Concepts �

Writing

y�tn� � y�tn����� �hn�y��tn����� �

h�n�y���tn����� �

h�n��y����tn����� � � � �

y�tn��� � y�tn������ hn�y��tn����� �

h�n�y���tn������ h�n

��y����tn����� � � � �

dividing by hn and subtracting� we obtain

y�tn�� y�tn���hn

� y��tn����� �h�n��y����tn����� �O�h�n�� ������

Furthermore� writing similar expansions for y� instead of y and adding� wereplace y��tn����� by �

��y��tn� � y��tn���� and obtain

y�tn�� y�tn���hn

��

��y��tn� � y��tn����� h�n

��y����tn����� �O�h�n�� ������

The latter equation suggests the trapezoidal method for discretizing our pro�totype ODE system ������

yn � yn�� �hn��f�tn�yn� � f�tn���yn����� ������

The local truncation error can be read o� ������� this method is second�orderaccurate�The trapezoidal method is symmetric� a change of variable � � �t on

�tn��� tn� �i�e� integrating from right to left� leaves it unchanged� � Like thebackward Euler method� it is implicit� the cost per step of these two methodsis similar� But the trapezoidal method is more accurate� so perhaps fewerintegration steps are needed to satisfy a given error tolerance� Before beingable to conclude that� however� we must check stability�Both the trapezoidal method and the backward Euler method are ��stable

�Exercise ����� To check absolute stability� apply the method with a step sizeh to the test equation� This gives

yn �� � h�

�� h�yn���

�Consider for notational simplicity the ODE y� f�y�� A discretization method givenby

yn yn�� � hn��yn��� yn� h�

is symmetric if��u� v� h� ��v� u��h� �

because then� by letting zn � yn��� zn�� � yn and h��h� we get the same method forzn as for yn�

Page 75: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

�� Chapter �� Initial Value Problems

The region of absolute stability is precisely the left half�plane of h�� so thismethod is A�stable� Moreover� the approximate solution is not dampenedwhen Re��� � �� which is qualitatively correct since the exact solution growsin that case�On the other hand� we cannot expect sti� decay with the trapezoidal

method� because its ampli�cation factor satis�es j��h���h� j � � in the very�sti�limit� This is typical for symmetric methods� Precisely� for the trapezoidalmethod�

� � h�

� � h�� �� as hRe�������

The practical implication of this is that any solution details must be resolvedeven if only a coarse picture of the solution is desired� because the fast modecomponents of local errors �for which h is �large�� get propagated� almostundamped� throughout the integration interval ��� b�� This is evident in Fig����� where we contrast integrations using the trapezoidal and the backwardEuler methods for Example ��� with a uniform step size h � ��� To apply thetrapezoidal method intelligently for this example� we must use a small stepsize through the initial layer� as in Fig� ���� Then the step size can becomelarger� The indicated mesh in Fig� ��� yields the solution pro�le shown whenusing the trapezoidal method�

Finally� in Table ��� we display the maximum error at mesh points forExample ���� when using each of the three methods introduced hitherto� withuniform step sizes h � ��� h � ��� and h � ���� Note that the error in the

method h max error

forward Euler �� ���e��

forward Euler �� ���e��

forward Euler ��� ���e��

backward Euler �� ��e��

backward Euler �� ���e��

backward Euler ��� ���e��

trapezoidal �� ���e��

trapezoidal �� ���e��

trapezoidal ��� ��e��

Table ���� Maximum errors for Example ����

Page 76: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Chapter �� Basic Methods� Basic Concepts ��

two Euler methods is not only larger in magnitude than the error obtainedusing the trapezoidal method� it also decreases linearly with h� while thetrapezoidal error decreases at a more favorable� quadratic rate�

��� Rough Problems

In the beginning of this chapter we have assumed that the given ODE �����is �su ciently smooth�� in the sense that all derivatives mentioned in thesequel are bounded by a constant of moderate size� This is often the casein practice� Still� there are many important instances where the problem isnot very smooth� In this section we discuss some such situations and theirimplication on the choice of discretization method�In general� if f�t�y� has k bounded derivatives at the solution y�t��

sup��t�b

j dj

dtjf�t�y�t��j �M� j � �� �� � � � � k

then by ������ y�t� has k � � bounded derivatives� and in particular�

ky�j�k �M� j � �� � � � � k � ��

So� if f is discontinuous but bounded then y has a bounded� discontinuous�rst derivative� But the higher derivatives of y appearing in the Taylorexpansion ����� �and hence in the expression for the local truncation error�are not bounded� so a discretization across such a discontinuity may yieldrather inaccurate approximations�Suppose �rst that there is one point� %t� � � %t � b� where f�%t�y�%t�� is

discontinuous� and everywhere else f is smooth and bounded� Note that theconditions of Theorem ��� do not hold on the interval ��� b�� but they do holdon each of the intervals ��� %t� and �%t� b�� Thus� we may consider integratingthe problem

y� � f�t�y�� � � t � %t� y��� � c

followed by the problem

z� � f�t� z�� %t � t � b� z�%t� � y�%t��

For each of these subproblems we can discretize using one of the methodsdescribed in this chapter and the next one� and expect to realize the fullaccuracy order of the method� Now� the algorithm does not �know� that wehave switched problems at t � %t� The integration can therefore proceed asbefore from � to b� provided that %t coincides with one of the mesh points� or

Page 77: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

�� Chapter �� Initial Value Problems

step ends� t�n� On the other hand� if %t is in the interior of a step� i�e� for some%n�

t�n�� � t � t�n�

then an O�h�n� error results� regardless of the order of the �consistent� dis�cretization method applied�

Example � � Consider the function

f�t� y� � t� j�� j� � t � �j � ���� j � �� �� � � � � J�

where � � � is a parameter� The ODE y� � f is therefore a quadrature

0 0.5 1 1.50

0.05

0.1

0.15

0.2

0.25

t

f

Figure ���� Sawtooth function for � � ����

problem for a sawtooth function� see Fig� ���� With y��� � �� the solution is

y�t� � j� ��� � �t� j� ����� j� � t � �j � ���� j � �� �� � � � � J�

We also calculate that away from the discontinuity points j� �

y���t� � �� y����t� � ��

For the trapezoidal method� the local truncation error therefore vanishes on astep �tn��� tn� that does not contain a point of discontinuity� For this specialcase� the trapezoidal method using a constant step size h reproduces y�tn�exactly if � � lh for some positive integer l� If� on the other hand� � ��l� r�h for some fraction r� then an O�h� � error results� up to J times� fora combined O�h� error� A worse� O��� error� may result if the �teeth aresharper�� e�g� f�t� � t�� � j� j� � t � �j � ��� � and � � O�h�� �

Page 78: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Chapter �� Basic Methods� Basic Concepts ��

If f�t�y� has a discontinuity then it is important to locate it and placea mesh point as close to it as possible� Unlike in the simple Example ����however� the precise location of such a discontinuity may not be known� Inmany examples� the discontinuity is de�ned by a switching function� e�g�

f�t�y� �

fI�t�y�t�� if g�t�y�t�� � ��

fII�t�y�t�� if g�t�y�t�� � ��

This situation often occurs when simulating mechanical systems �Example���� with dry friction� A simple event location algorithm� which automati�cally detects when g changes sign and then solves a local nonlinear algebraicequation �using interpolation of nearby yn�values� to locate the switchingpoint g � � more accurately� proves very useful in practice� An alternative�when using a general�purpose code which features adaptive step�size selec�tion� is to simply rely on the code to select a very small step size near thediscontinuity �because the local error is large there�� But this is usually infe�rior to the event location strategy because the theory on which the step sizeselection is based is typically violated in the presence of a discontinuity� andbecause the code can become quite ine cient when taking such small stepsizes�Note that the �rst order Euler methods utilize y�� in the expression for

the local truncation error� while the second order trapezoidal method utilizesthe higher derivative y���� This is general� a method of order p matches p��terms in a local Taylor expansion such as ������ so the local truncation erroris expected to behave like O�hpny

�p����tn��� Thus� if only the �rst l � �derivatives of y�t� exist and are bounded then� in general� any di�erencemethod will exhibit an order of at most l� As we will see in the next fewchapters� higher order methods cost more per step� so if the problem is rough�i�e� l is low� then lower order methods get the nod�

Example � � The harmonic oscillator

u�� � ��u � �� � � t � b

u��� � �� u���� � �

has the solutionu�t� � cos�t�

If the frequency � is high� � �� then the derivatives grow larger and larger�because

ku�p�k � �p�

The local error of a discretization method of order p is

O�hp���p����

Page 79: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

�� Chapter �� Initial Value Problems

This means that to recover the highly oscillatory solution u�t� accurately� wemust restrict

h � ���

regardless of the order of the method� In fact� for h � ���� increasing theorder of the method as such is useless� �

��� Software Notes and References

����� Notes

The basic questions of a numerical method�s order� consistency� ��stabilityand convergence are discussed� in one form or another� in most books thatare concerned with the numerical solution of di�erential equations� But thereis a surprisingly wide variety of expositions� see� e�g�� ��� ��� ��� �� �� ����� ��� ���� We chose to expose concepts in a way which highlights how theymimic properties of the di�erential equation� One bene�t of this is that theconcepts and the treatment naturally extend to boundary value problemsand to PDEs �which we do not pursue directly in this book�� Omitted istherefore a general derivation of the global error as an accumulation of localerrors� such as what leads to �������For this reason we also chose to de�ne the local truncation error as in

����������� Some other authors have chosen to de�ne this quantity multi�plied by hn� making dn and ln generally of the same order� We also chosenot to use the local error �whose de�nition depends on the existence of theapproximate solution� see� e�g�� Exercise ��� to note that existence of thenumerical solution is not always a foregone conclusion� as a tool in our expo�sition of fundamental concepts� despite its practical usefulness as discussedin the next chapter�Another decision we took which deviates from most texts on numerical

methods for initial value ODEs was to introduce the concept of sti�ness atan early stage� This not only highlights the basic importance of the topic� itis also a natural approach if one has the numerical solution of PDEs in mind�and moreover it allows a natural treatment of the concept of absolute sta�bility� The extension of ODE stability restrictions to time�dependent PDEsis somewhat facilitated in x���� but it remains a nontrivial issue in general�because the ODE system size m is very large for the corresponding methodof lines� Some cases do extend directly� though� see Exercises ��� and ����For more challenging cases see� e�g�� Reddy and Trefethen ���� and referencestherein�

Page 80: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Chapter �� Basic Methods� Basic Concepts �

While the concept of sti�ness is intuitively simple� its precise de�nitionand detection have proved to be elusive� Our discussion has excluded� forinstance� the in�uence of rough forcing terms and of small but positive eigen�values� These issues are treated at length in many texts� e�g� Hairer Wanner ���� Shampine ��� and Butcher �����The concept of absolute stability is due to Dahlquist ����� But the search

for a good stability concept to capture the essence of sti�ness in numericalmethods has brought about an often confusing plethora of de�nitions �foran extensive exposition see� e�g�� ����� We feel that an appropriate conceptshould re�ect what one aims to capture� a simple� intuitive phenomenonwhich is independent of the discretization approach� Hence we have usedthe somewhat less well�known terminology and de�nition of sti� decay� Thisterm was introduced by J� Varah in ����� following Prothero Robinson�����A more reliable� and in some cases also more e cient� means than �nite

di�erences for obtaining a Jacobian matrix without user intervention is touse automatic di�erentiation software ����� This software takes as input auser�provided routine that computes f � and produces another routine whiche ciently computes the Jacobian matrix� At present� this requires an initialtime investment� to install and learn to use the automatic di�erentiationsoftware� However� we expect that simpler interfaces will soon be available�The sti� problems dealt with in Chapters � � have eigenvalues with

large� negative real parts� Another type of �sti�ness� is when the problem has�nearly� purely imaginary large eigenvalues� This yields highly �i�e� rapidly�oscillatory problems� See Examples �� and ���� A recent survey on thenumerical solution of such problems is given by Petzold� Jay Yen �����

����� Software

In each of the chapters following this one which deal with numerical methods�there is a section �just before the Exercises� which brie�y describes someavailable software packages where corresponding methods and techniques areimplemented� The methods described in the current chapter are too basicto be implemented alone in quality general�purpose codes� Here we quicklymention instead some outlets through which such software is available� Thereare three types�

�� A complete environment is made available in which users can do theirown code development interactively� having direct access to varioussoftware tools� These tools include certain ODE packages �and muchmore�� Examples of such programming environments areMatlab andMathematica� We have used Matlab in most of the coding de�veloped for the examples and exercises in this book� and we stronglyrecommend it� An interactive environment such as Matlab does not

Page 81: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

�� Chapter �� Initial Value Problems

replace procedural languages like C or FORTRAN� though� especiallyfor production runs and large scale computing� The tools providedwith Matlab currently do not cover all the problem classes of thisbook� nor are they always the best available for a given application�Matlab does allow interfacing with external software written in C orFORTRAN�

�� Collected programs are available for a fee through software librariessuch as Nag and Imsl� These programs are written in FORTRAN orC� The range of software available in this way in some areas is consider�ably more extensive than what is available as part of the integrated pro�gramming environments� and it is more suitable for production codes�The advantage here� compared to the next alternative� is that there isa measure of quality control and support of the software available inthis way� This occasionally also implies some limitations on the rangeand richness of the software available�

�� A large collection of codes is available electronically through Netlib�The web page is at

http���netlib�belllabs�com�netlib�master�readme �

It is possible also to e�mail

netlib�research�att�com

with a request such as

send codename from ode

which causes the �hypothetical� ODE code codename to be e�mailedback� Netlib is a software repository� it is available for free and comeswith no guarantee�

The codes colsys� dassl and their derivatives� which solve sti� initialvalue problems� boundary value problems and di�erential�algebraic problems�and are distinguished by the fact that one of your authors took part in writingthem� are available through Netlib� as well as through this book�s web page�

Most software for scienti�c computation to date is written in FORTRAN�but software is available to convert to� or interface with� C and C�� pro�grams� The user therefore does not need to be �uent in FORTRAN in orderto use these codes� Such porting programs are available through Netlib andalso through this book�s web page�

Page 82: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Chapter �� Basic Methods� Basic Concepts ��

��� Exercises

�� Show that the backward Euler method and the trapezoidal method are��stable�

�� To draw a circle of radius r on a graphics screen� one may proceedto evaluate pairs of values x � r cos �� y � r sin � for a succession ofvalues �� But this is expensive� A cheaper method may be obtained byconsidering the ODE

'x � �y x��� � r

'y � x y��� � �

where 'x � dxd�� and approximating this using a simple discretization

method� However� care must be taken so as to ensure that the obtainedapproximate solution looks right� i�e� that the approximate curve closesrather than spirals�

For each of the three discretization methods introduced in this chapter�namely� forward Euler� backward Euler and trapezoidal methods� carryout this integration using a uniform step size h � ��� for � � � � ����Determine if the solution spirals in� spirals out� or forms an approximatecircle as desired� Explain the observed results� �Hint� this has to dowith a certain invariant function of x and y� rather than with the orderof the methods��

�� The following ODE system

y�� � � � y� � �y�y�� � y��

y�� � y��� � y�� � y��

where � and are parameters� represents a simpli�ed approximationto a chemical reaction ����� There is a parameter value c �

�� � �

such that for � c solution trajectories decay in amplitude and spiralin phase space into a stable �xed point� whereas for � c trajectoriesoscillate without damping and are attracted to a stable limit cycle��This is called a Hopf bifurcation��

�a� Set � � �� and use any of the discretization methods introducedin this chapter with a �xed step size h � ���� to approximate thesolution starting at y���� � �� y���� � �� for � � t � ��� Do thisfor the parameter values � � and � �� For each case plot y�vs t and y� vs y�� Describe your observations�

Page 83: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

�� Chapter �� Initial Value Problems

�b� Investigate the situation closer to the critical value c � ��� �Youmay have to increase the length of the integration interval b to geta better look��

�� When deriving the trapezoidal method� we proceeded to replace y��tn�����in ������ by an average and then use the ODE ������ If instead we �rstuse the ODE� replacing y��tn����� by f�tn�����y�tn������� and then av�erage y� we obtain the implicit midpoint method�

yn � yn�� � hnf�tn������

��yn � yn����� ������

�a� Show that this method is symmetric� second�order and A�stable�How does it relate to the trapezoidal method for the constantcoe cient ODE ������!

�b� Show that even if we allow � to vary in t� i�e� we consider thescalar ODE

y� � ��t�y

in place of the test equation� what corresponds to A�stabilityholds� namely� using the midpoint method�

jynj � jyn��j if Re��� � ��this property is called ANstability ������ Show that the samecannot be said about the trapezoidal method� the latter is notAN�stable�

� �a� Show that the trapezoidal step ������ can be viewed as half a stepof forward Euler followed by half a step of backward Euler�

�b� Show that the midpoint step ������ can be viewed as half a stepof backward Euler followed by half a step of forward Euler�

�c� Consider an autonomous system y� � f�y� and a �xed step size�hn � h� n � �� � � � � N � Show that the trapezoidal method appliedN times is equivalent to applying �rst half a step of forward Euler�i�e� forward Euler with step size h���� followed by N�� midpointsteps� �nishing o� with half a step of backward Euler�

Conclude that these two symmetricmethods are dynamically equivalent ����� i�e�� for h small enough their performance is very similarindependently of N � even over a very long time� b � Nh ��

�d� However� if h is not small enough �compared to the problem�ssmall parameter� say ���� then these methods do not necessar�ily perform similarly� Construct an example where one of these

Page 84: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Chapter �� Basic Methods� Basic Concepts ��

methods blows up �error � ��� say� while the other yields anerror below ���� �Do not program anything� this is a �nontrivial�pen�and�paper question��

�� Consider the method of lines applied to the simple heat equation inone space dimension�

ut � auxx

with a � � a constant� u � � at x � � and x � � for t � �� andu�x� �� � g�x� is given as well� Formulate the method of lines� as inExample ���� to arrive at a system of the form ������ with A symmetric�Find the eigenvalues of A and show that� when using the forward Eulerdiscretization for the time variable� the resulting method is stable if

h � �

�a�x��

�This is a rather restrictive condition on the time step�� On the otherhand� if we discretize in time using the trapezoidal method �the re�sulting method� second order in both space and time� is called Crank�Nicolson�� or the backward Euler method� then no stability restrictionfor the time step arises� �Hint� to �nd the eigenvalues� try eigenvectorsvk in the form vki � sin�ik�x�� i � �� � � � �m� for � � k � m��

�� Consider the same question as the previous one� but this time the heatequation is in two space variables on a unit square�

ut � a�uxx � uyy�� � � x� y � �� t � ��

The boundary conditions are u � � around the square� and u�x� y� �� �g�x� y� is given as well�

Formulate a system ������ using a uniform grid with spacing �x onthe unit square� Conclude again that no restrictions on the time steparise when using the implicit methods which we have presented for timediscretization� What happens with the forward Euler method! �Hint�don�t try this exercise before you have solved the previous one��

�� Consider the ODE

dy

dt� f�t� y�� � � t � b

where b ��

Page 85: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

�� Chapter �� Initial Value Problems

�a� Apply the stretching transformation t � �b to obtain the equiva�lent ODE

dy

d�� b f��b� y�� � � � � �

�strictly speaking� y in these two ODEs is not quite the samefunction� Rather� it stands in each case for the unknown function��

�b� Show that applying any of the discretization methods in this chap�ter to the ODE in t with step size h � �t is equivalent to applyingthe same method to the ODE in � with step size �� satisfying�t � b�� � In other words� the same stretching transformationcan be equivalently applied to the discretized problem�

�� Write a short program which uses the forward Euler� the backwardEuler and the trapezoidal or midpoint methods to integrate a linear�scalar ODE with a known solution� using a �xed step size h � b�N �and �nds the maximum error� Apply your program to the followingproblem

dy

dt� �cos t�y� � � t � b

y��� � �� The exact solution is

y�t� � esin t�

Verify those entries given in Table ��� and complete the missing ones�Make as many �useful� observations as you can on the results in thecomplete table� Attempt to provide explanations� �Hint� plotting thesesolution curves for b � ��� N � ��b� say� may help��

��� Consider two linear harmonic oscillators �recall Example ����� one fastand one slow� u��� � �����u�� %u�� and u��� � ��u�� %u��� The parameteris small� � � �� �� We write this as a �rst order system

u� �

����� �

� �

�Av

v� � ������ �

� �

�A �u� %u�

where u�t��v�t� and the given constant vector %u each have two com�ponents� It is easy to see that EF �

���v�� � �u� � %u���� and ES �

���v

�� � �u� � %u���� remain constant for all t �see x����

Page 86: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Chapter �� Basic Methods� Basic Concepts ��

b N forward Euler backward Euler trapezoidal midpoint

� �� ��e�� ���e�� ���e�� ���e��

�� ���e�� ���e�� ���e�� ��e��

�� ���

���

��� ���� ���� ���� ���e�� ���e��

����

���� ����

����� ���� ����e��� ���e�� ���e��

�����

������ ���� ����� ���e�� ���e��

Table ���� Maximum errors for long interval integration of y� � �cos t�y

Next� we apply the following time�dependent linear transformation�

u � Qx� v � Qz� Q�t� �

�� cos�t sin�t

� sin�t cos�t

�A � K � 'QTQ �

��� ��� �

�A

where � � � is another parameter� This yields the coupled system

x� � QT

����� �

� �

�AQz� �Kx �����a�

z� � �QT

����� �

� �

�AQ�x� %x� � �Kz � �����b�

where %x � QT %u� We can write the latter system in our usual notationas a system of order ��

y� � A�t�y� q�t� �

�a� Show that the eigenvalues of the matrixA are all purely imaginaryfor all ��

�Hint� show that AT � �A��

Page 87: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

�� Chapter �� Initial Value Problems

�b� Using the values %u � ��� ���T � u��� � %u� v���T � �������p�and b � ��� apply the midpoint method with a constant step sizeh to the system ������ for the following parameter combinations�� � ������ k � ���� ���� ����� and � � �� �� �� �a total of �runs�� Compute the error indicators maxt jEF �t� � EF ���j andmaxt jES�t�� ES���j� Discuss your observations�

�c� Attempt to show that the midpoint method is unstable for thisproblem if h � �

p��� ����� Conclude that A�stability and AN�

stability do not automatically extend to ODE systems�

��� Consider the implicit ODE

M�y�y� � f�t�y�

where M�y� is nonsingular for all y� The need for integrating initialvalue problems of this type typically arises in robotics� When thesystem size m is large� the cost of invertingM may dominate the entiresolution cost� Also� �M

�yis complicated to evaluate� but it is given that

its norm is not large� say O����

�a� Extend the forward Euler and the backward Euler discretizationsfor this case �without invertingM�� Justify�

�b� Propose a method for solving the nonlinear system of equationsresulting at each time step when using backward Euler� for thecase where j�f��yj is very large�

Page 88: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Chapter �

One Step Methods

The basic methods developed in Chapter � can be adequate for computingapproximate solutions of a relatively low accuracy �as we will see� for instance�in Example ����� or if the problem being solved is rough in a certain sense�see x����� But often in practice a quality solution of high accuracy to arelatively smooth problem is sought� and then using a basic� low order methodnecessitates taking very small steps in the discretization� This makes theintegration process ine cient� Much fewer steps are needed when using ahigher order method in such circumstances�

In order to develop e cient� highly accurate approximation algorithms�we therefore design higher order di�erence methods� The higher order meth�ods we consider in this book are of two types� one step and linear multistep�In each of these classes of methods it will be useful to distinguish furtherbetween methods for sti� problems and methods for nonsti� problems� Thelarge picture is depicted in Fig� ����

In this chapter we will explore higher�order onestep methods� These aremethods which do not use any information from previous steps �in contrastto linear multistep methods which will be taken up in the next chapter��Thus� in a typical step of size h � hn � tn � tn��� we seek an approximationyn to y�tn� given the previous step end result� yn���

��

Page 89: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

�� Chapter �� Initial Value Problems

Runge-Kutta linear multistep

non-stiff stiff non-stiff stiff

basic methods

Figure ���� Classes of higher order methods�

Review� Recall from Advanced Calculus that Taylor�s Theorem fora function of several variables gives

F �x� y� � F �

��F

�x�x� �x� � �F

�y�y � �y�

��

�$

���F

�x��x� �x�� � � �

�F

�x�y�x� �x��y � �y� � ��F

�y��y � �y��

� � � �� �

n$�nF � � � �

where the functions on the right hand side are evaluated at ��x� �y� and

�nF �nX

j��

�n

j

���nF

�xj�yn�j��x� �y�

��x� �x�j�y � �y�n�j �

The conceptually simplest approach for achieving higher order is to usethe di�erential equation to construct the Taylor series for the solution� Fora scalar ODE y� � f�t� y�� the Taylor series method is given by replacing thehigher derivatives in a truncated Taylor expansion� yielding the formula

yn � yn�� � hy�n�� �h�

�y��n�� � � � �

hp

p$y�p�n��

Page 90: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Chapter �� One Step Methods �

with f�t� y� and its derivatives evaluated at �tn��� yn����

y�n�� � f

y��n�� � ft � fyf

y���n�� � ftt � �ftyf � fyft � fyyf� � f�

y f �����

etc� The local truncation error is hpy�p����tn���p � ��$ � O�hp���� For asystem of di�erential equations� the derivatives are de�ned similarly�

A problem with this method is that it requires analytic expressions forderivatives which in a practical application can be quite complicated� Onthe other hand� advances in compiler technology have enabled much morerobust programs for symbolic and automatic di�erentiation in recent years�which may make this method more attractive for some applications�

We thus seek one�step methods that achieve a higher accuracy order with�out forming the symbolic derivatives of the Taylor series method� This leadsto RungeKutta methods� to which the rest of this chapter is devoted�

��� The First Runge�Kutta Methods

We stay with a scalar ODE for some of this exposition� The extension toODE systems is straightforward� and will be picked up later�

Many Runge�Kutta �RK� methods are based on quadrature schemes� Infact� the reader may want to quickly review basic quadrature rules at thispoint�

Page 91: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

�� Chapter �� Initial Value Problems

Review� Basic quadrature rules�Given the task of evaluating an integralZ b

a

f�t�dt

for some function f�t� on an interval �a� b�� basic quadrature rulesare derived by replacing f�t� with an interpolating polynomial ��t�and integrating the latter exactly� If there are s distinct interpolationpoints c�� � � � � cs� then we can write the interpolating polynomial ofdegree � s in Lagrange form�

��t� �sX

j��

f�cj�Lj�t�

where

Lj�t� � (si���i ��j

�t� ci�

�cj � ci��

Then Z b

a

f�t�dt �sX

j��

wjf�cj�

where the weights wj are given by

wj �

Z b

a

Lj�t�dt�

The precision of the quadrature rule is p if the rule is exact for allpolynomials of degree � p� i�e�� if for any polynomial f of degree � p�

Z b

a

f�t�dt �

sXj��

wjf�cj��

If b � a � O�h� then the error in a quadrature rule of precision p isO�hp���� Obviously� p � s� but p may be signi�cantly larger than s ifthe points cj are chosen carefully�The midpoint and trapezoidal rules have precision p � �� Simpson�srule has precision p � �� Gaussian quadrature at s points has thehighest precision possible at p � �s�

Let�s reconsider the methods we have already seen in the previous chapter

Page 92: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Chapter �� One Step Methods ��

in the context of quadrature� Writing

y�tn�� y�tn��� �Z tn

tn��

y��t�dt �����

we can approximate the area under the curve y��t� �see Fig� ���� using eitherthe lower sum based on y�tn��� �forward Euler� or the upper sum based ony�tn� �backward Euler�� These are �rst order methods�

y’(t)

ntn-1t

Figure ���� Approximate area under curve

For a better approximation� we can use the height at the midpoint of theinterval� i�e� y��tn����� where tn���� � tn � h��� see Fig� ���� This leads to

y’(t)

ntn-1t

Figure ���� Midpoint quadrature�

the midpoint method �recall �������

yn � yn�� � hf

�tn�����

yn�� � yn�

��

Page 93: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

�� Chapter �� Initial Value Problems

This is an implicit Runge�Kutta method� We can construct an explicitmethod based on the same idea by �rst approximating y�tn����� by the for�ward Euler method� and then substituting into the midpoint method to ob�tain

�yn���� � yn�� �h

�f�tn��� yn��� ����a�

yn � yn�� � hf�tn����� �yn������ ����b�

The obtained explicit midpoint method ����� gives us a �rst real tasteof the original Runge�Kutta idea� a higher order is achieved by repeatedfunction evaluations of f within the interval �tn��� tn�� Note that this methodis not linear in f anymore �substitute ����a� into ����b� to see this�� At �rstglance� it might seem that the order would be limited to one� because the �rststage ����a� uses the forward Euler method� which is �rst order� However�note that the term involving �yn���� enters into ����b� multiplied by h� andtherefore its error becomes less important�Indeed� the local truncation error of ����� is given by

dn �y�tn�� y�tn���

h� f

�tn����� y�tn��� �

h

�f�tn��� y�tn����

� y� �h

�y�� �

h�

�y��� �

�f �

h

��ft � fyf� �

h�

��ftt � �ftyf � fyyf

��

��O�h�� �����

where all quantities on the right hand side are evaluated at �tn��� y�tn�����Using the ODE and its derivatives� all but O�h�� terms cancel� Thus themethod is consistent of order ��The trapezoidal method considered in the previous chapter is obtained in

a similar manner based on applying the trapezoidal quadrature rule to �����

yn � yn�� �h

�f�tn� yn� �

h

�f�tn��� yn����

This is another implicit Runge�Kutta method� To obtain an explicit methodbased on this idea� we can approximate yn in f�tn� yn� by the forward Eulermethod� yielding

�yn � yn�� � hf�tn��� yn��� ���a�

yn � yn�� �h

�f�tn� �yn� �

h

�f�tn��� yn���� ���b�

This is called the explicit trapezoidal method� Like the explicit midpointmethod it is an explicit two�stage Runge�Kutta method of order two�

Page 94: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Chapter �� One Step Methods ��

The famous classical fourth order RungeKutta method is closely relatedto Simpson�s quadrature rule applied to ������

y�tn�� y�tn��� � h

�y��tn��� � �y��tn����� � y��tn�

�� �����

To build an explicit approximation of y��tn����� is not a simple matter any�more� though� The formula is given by

Y� � yn��

Y� � yn�� �h

�f�tn��� Y��

Y� � yn�� �h

�f�tn����� Y�� �����

Y� � yn�� � hf�tn����� Y��

yn � yn�� �h

�f�tn��� Y�� � �f�tn����� Y�� � �f�tn����� Y�� � f�tn� Y��

��

It has order ��

Example � � We compute the solution of the simple Example ���

y� � �ty� � t� �

t�� y��� � �

using three explicit RungeKutta methods� forward Euler� explicit midpointand the classical fourth order method� We use various �xed step sizes tointegrate up to t � � and record the absolute errors at the end of the intervalin Table ��� �the exact solution� to recall� is y�t� � ��t� We also recordfor each method a calculated convergence rate� This �rate� is calculated asfollows� if the error at step n behaves like en�h� � yn � y�tn� � chp for someunknown constant c and rate p then the error with half the step size should

satisfy e�n�h��� � c�h�

�p� Thus p � rate �� log�

en�h�

e�n�h���

��

A number of general observations can be deduced already from this verysimple example�

�� The error for a given step size is much smaller for the higher ordermethods� That is the basic reason for embarking on the search for higherorder methods in this chapter and the next� Of course� the cost of eachstep is also higher for a higher order method� Roughly� if the cost ismeasured simply by the number of evaluations of f �which in complexapplications is usually the determining cost factor then the cost of anRK� step is double that of the explicit midpoint method which in turnis double that of the forward Euler method�

Page 95: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

�� Chapter �� Initial Value Problems

step h Euler error rate RK� error rate RK� error rate

��� ���e�� ���e�� ���e��

��� ��e�� ���� ���e�� ����� ���e�� ����

��� ���e�� ���� ��e�� ���� ���e�� ����

���� ���e�� ���� ���e�� ���� ���e��� ����

���� ��e�� ���� ���e�� ���� ���e��� ����

���� ���e�� ���� ���e�� ���� ���e��� ����

����� ���e�� ���� ���e��� ���� ���e��� ����

Table ���� Errors and calculated convergence rates for the forward Euler� theexplicit midpoint �RK�� and the classical Runge�Kutta �RK�� methods

�� Thus� the choice of method depends on the accuracy requirements� Generally� the smaller the error tolerance and the smoother the problem andits solution� the more advantageous it becomes to use higher order methods �see x���� Here� if the maximum error tolerance is ���� then thebest choice would be forward Euler� But for an error tolerance �����

the fourth order method is best�

�� The error is polluted� as evidenced by the deviations of the computedrates from their predicted values of �� � and �� both for very large andfor very small step sizes� For h � �� an error due to partial violationof absolute stability is observed� For h � ���� the truncation errorin the classical fourth order method is so small that the total errorbegins to be dominated by roundo� error �we have been using �oatingpoint arithmetic with �� hexadecimal digits� Roundo� error generallyincreases as h decreases� because more steps are required to cover theintegration interval� The assumption en�h� � chp presupposes that theroundo� error is dominated for this h by the truncation error� which isoften the case in practice�

Page 96: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Chapter �� One Step Methods ��

��� General Formulation of Runge�Kutta Meth�

ods

In general� an s�stage Runge�Kutta method for the ODE system

y� � f�t�y�

can be written in the form

Yi � yn�� � h

sXj��

aijf�tn�� � cjh�Yj� ����a�

yn � yn�� � h

sXi��

bif�tn�� � cih�Yi�� ����b�

The Yi�s are intermediate approximations to the solution at times tn��� cihwhich may be correct to a lower order of accuracy than the solution yn atthe end of the step� Note that Yi are local to the step from tn�� to tn� andthe only approximation that the next step �sees� is yn� The coe cients ofthe method are chosen in part so that error terms cancel and yn is moreaccurate�

The method can be represented conveniently in a shorthand notation

c� a�� a�� � � � a�s

c� a�� a�� � � � a�s���

������� � �

���

cs as� as� � � � ass

b� b� � � � bs

We will always choose

ci �sX

j��

aij� i � �� � � � � s� �����

The Runge�Kutta method is explicit i� aij � � for j � i� because theneach Yi in ����a� is given in terms of known quantities� Historically� the�rst Runge�Kutta methods were explicit� However� implicit Runge�Kuttamethods are useful for the solution of sti� systems� as well as for boundaryvalue problems �see Chapter ���

Some examples of explicit Runge�Kutta methods are given below�

Page 97: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

�� Chapter �� Initial Value Problems

Forward Euler

� �

Oneparameter family of second order methods

� � �

� � �

� � ���

���

For � � �� we have the explicit trapezoidal method� and for � � ��� it is theexplicit midpoint method�There are three one�parameter families of third order �stage methods� Onesuch family is

� � � �

��

��

� �

��

��� �

�����

��

�� � � �

where � is a parameter�

Finally� the classical fourth order method is written using this notation as

� � � � �

��

��� � �

�� � �

� � �

� � � � �

��

��

��

��

We see that there are s�stage explicit Runge�Kutta methods of orderp � s� at least for p � �� One may wonder if it is possible to obtain orderp � s� and if it is possible to always maintain at least p � s� The answersare both negative� There will be more on this in the next section�The choice of intermediate variables Yi to describe the Runge�Kutta

method ����� is not the only natural one� Sometimes it is more naturalto use intermediate approximations to f rather than y at the interior stages�

Page 98: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Chapter �� One Step Methods ��

We leave it to the reader to verify that the general s�stage Runge�Kuttamethod ����� can be written as

Ki � f

�tn�� � cih�yn�� � h

sXj��

aijKj

������a�

yn � yn�� � h

sXi��

biKi� �����b�

��� Convergence ��Stability and Order for

Runge�Kutta Methods

The basic convergence of one�step methods is essentially automatic� All ofthe methods we have seen so far� and any that we will see� are accurate toat least �rst order� i�e� they are consistent� The fundamental Theorem ���tells us that convergence �to the order of accuracy� as in ������� is achieved�provided only that the method is ��stable� We can write any reasonableone�step method in the form

yn � yn�� � h��tn���yn��� h� ������

where � satis�es a Lipschitz condition in y� �This is obvious for explicitmethods� For implicit methods the Implicit Function Theorem is applied��In the previous chapter we showed� following Theorem ���� that the forwardEuler method is ��stable� Replacing f in that proof by � yields the sameconclusion for the general one�step method ������� We leave the details tothe exercises�We next consider the question of verifying the order of a given Runge�

Kutta method� Order conditions for general Runge�Kutta methods are ob�tained by expanding the numerical solution in Taylor series� as we did forsecond�order methods� For an autonomous ODE system�

y� � f�y�

�Without loss of generality� we can consider systems of autonomous dierential equa�tions only� This is because the ODE y� f�t� y� can be transformed to autonomous formby adding t to the dependent variables as follows

t� �

y� f�t� y��

Page 99: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

�� Chapter �� Initial Value Problems

the exact solution satisfying y�tn��� � yn�� �recall that we are interestedhere in what happens in just one step� has at tn�� the derivatives

y� � f �� f�

y�� � f � � ��f

�y�f �� f�

y��� � ��f�

�y�f �� f�

���

y�k� � ��fk��

�y�f �� fk���

�Note� f j is not the jth power of f�� By Taylor�s expansion at y � yn���

y�tn� � y� hy� �h�

�y�� � � � �� hp��

�p� ��$y�p��� � � � �

� y� hf �h�

�f� � � � � � hp��

�p� ��$fp � � � �

For an s�stage Runge�Kutta method ������ substituting y�t� into the di�er�ence equations gives that in order to obtain a method of order p we musthave

sXi��

bif�Yi� � f �h

�f� � � � �� hp��

p$fp�� �O�hp��

A Taylor expansion of the f�Yi� therefore follows�Although this is conceptually simple� there is an explosion of terms to

match which is severe for higher order methods� An elegant theory involvingtrees for enumerating these terms was developed by J� C� Butcher� in a longseries of papers starting in the mid������s� The details are complex� though�and do not yield a methodology for designing a method of a desirable order�only for checking the order of a given one�Instead� we proceed to derive simple� necessary order conditions� to get

the hang of it� The idea is that the essence of a method�s accuracy is oftencaptured by applying our analysis to very simple equations�Consider the scalar ODE

y� � y � tl��� t � � ������

l a positive integer� with y��� � �� for the �rst step �i�e� tn�� � yn�� � ���Then

Yi � h

sXj��

aij�Yj � �hcj�l����

Page 100: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Chapter �� One Step Methods �

We can write this in matrix form

�I � hA�Y � hlAC l���

where A is the s� s coe cient matrix from the tableau de�ning the Runge�Kutta method� Y � �Y�� � � � � Ys�T � C � diagfc�� � � � � csg is the diagonalmatrix with the coe cients cj on its diagonal and � � ��� �� � � � � ��T � Itfollows that Y � hl�I � hA���AC l��� and

yn � h

sXi��

bi�Yi � �hci�l���

� hlbT �I � hA� � � �� hkAk � � � � �C l���

where bT � �b�� b�� � � � � bs�� Now we compare the two expansions for ynand for y�tn� and equate equal powers of h� For the exact solution y�t� �R t� e

t�ssl��ds� we have

y��� � � � � � y�l������ � �� y�l�j���� � �l� ��$� j � ��This yields that for the method to be of order p the following order conditionsmust be satis�ed�

bTAkC l��� ��l � ��$�l � k�$

��

l�l� �� � � � �l � k�� � � l � k � p � ������

�The indices run as follows� for each l� � � l � p� we have order conditionsfor k � �� �� � � � � p � l��In component form the order conditions ������ read

Xi�j�� �jk

biai�j�aj��j� � � � ajk���jkcl��jk��l � ��$�l � k�$

The vector form is not only more compact� though� it is also easy to program�We next consider two simple subsets of these order conditions� Setting

k � � in ������� we obtain the pure quadrature order conditions

bTC l��� �sX

i��

bicl��i �

l� l � �� �� � � � � p � ������

Note that the coe cients aij of the Runge�Kutta method do not appear in������� Next� setting l � � in ������� and k � k � � in ������� we obtainthat for the method to be of order p the following order conditions must besatis�ed�

bTAk��� ��

k$� k � �� �� � � � � p� �����

Page 101: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

�� Chapter �� Initial Value Problems

These conditions are really additional to ������ only when k � �� becauseA� � c� so Ak��� � Ak��c�The leading term of the local truncation error for the ODE ������ with

l � � is

dn � hp�bTAp�� �

�p � ��$

��

For explicit Runge�Kutta methods� A is strictly lower triangular� hence Aj �� for all j � s� This immediately leads to the conclusions�

�� An explicit Runge�Kutta method can have at most order s� i�e�� p � s�

�� If p � s then the leading term of the local truncation error for thetest equation ������ with l � � cannot be reduced by any choice ofcoe cients of the explicit Runge�Kutta method�

Example � � Consider all explicit �stage RungeKutta methods

� � �

c� c� �

b� b�

For l � �� k � �� condition ����� reads b� � b� � �� For l � �� k � �� wehave b�c� � ���� The condition for l � �� k � �� is the same� Denoting� � c� results in the family of twostage� ordertwo methods displayed in theprevious section� For the choice of the parameter �� we can minimize thelocal truncation error for the quadrature test equation y� � tp� It is

dn � hp

�sX

i��

bicpi �

p � �

��

Trying to achieve b�c�� � ��� gives the choice � � ���� b� � ���� But thischoice does nothing special for the ODE y� � y� for instance� �

The obtained set of order conditions ������ �recall also ������ is certainlynecessary for the method to have order p� These order conditions are notsu�cient in general$ Still� they can be used to �nd a simple upper boundon the order of a given method �Exercise ����� and also for the purpose ofdesigning new Runge�Kutta methods� In fact� often the order is alreadydetermined by the conditions ������ plus ����� alone�

Example � � We can now view the classical RungeKutta method as a result of a methodical design process� The starting point is an attempt to extendSimpson�s quadrature rule� which is fourth order� Although Simpson�s rule

Page 102: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Chapter �� One Step Methods ��

has only � abscissae� �� ��� and �� we know already that a method of order� will not result from only � stages� so we settle for � stages and chooseabscissae ci � �� ���� ���� �� Next� we must have from ����� bTA�� �b�a��a��a�� � ����� In particular� we must choose ai���i �� �� The simplestchoice then is to set the rest of the aij to �� yielding ci�� � ai���i� i � �� �� ��The choice b� � b� � ��� is as in Simpson�s rule and results from the quadrature conditions ����� alone� The �nal choice b� � b� is determined by thecondition ����� with k � �� This completes the de�nition of the method� Itsorder does turn out to be p � �� �

Example � � The simple necessary order conditions ������ ����� give anupper bound on a method�s order which turns out to agree with the order formany if not all the methods in actual use� However� counterexamples wherethese conditions are not su�cient can be constructed� For explicit RungeKutta methods of order p with p stages� these conditions are su�cient for p ��� � and �� One is tempted to conclude that this is always true� as the famousjoke goes� �by induction�� For p � �� however� there are two additionalconditions� b�a��c�� � b��a��c�� � a��c

��� � ���� and b�c�a��c� � b�c��a��c� �

a��c�� � ���� The �rst of these is covered by ������ but the second� which isin the form bTCAC� � ���$� is not� Together with ����� and ����� theseconditions imply in particular that we must choose c� � � �Exercise ���� Butthe conditions ����� alone do not imply this� A particular example wherethese conditions are not su�cient is

� � � � �

��

��� � �

�� � �

� � �

�� � �

��� �

� �� ��

���

Example � � For the sake of completeness� we list below the full set of conditions that must hold for a method to have order at least � in addition tothose conditions already necessary for order �� �The order� conditions arediscussed in the previous example ���� For a higher order we recommendthat the reader get a hold of a more indepth book� The �rst � of these are

Page 103: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

�� Chapter �� Initial Value Problems

included in ������

bTC�� ��

bTA�� ��

���

bTA�C�� ��

��

bTAC�� ��

��

bTC�AC� ��

��

bTCAC�� ��

bTCA�C� ��

��

bTCACAC� ��

��Xi�j�k

biaijcjaikck ��

��

Finally� we return to the question� what is the maximal attainable order pby an explicit s�stage Runge�Kutta method! This question turns out to havea complicated answer� Given ������ the number of coe cients in an explicits�stage method is s�s � ����� but it does not bear a simple relationship tothe number of independent order conditions� Indeed� one often encountersfamilies of �eligible� methods for given order and number of stages in practice�Still� we have the following limitations on the attainable order as a functionof the number of stages�

number of stages � � � � � � � � ��

attainable order � � � � � � � � �

This explains in part why the fourth order explicit Runge�Kutta method isso popular �especially when no adaptive error control is contemplated��

Page 104: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Chapter �� One Step Methods ��

��� Regions of Absolute Stability for Explicit

Runge�Kutta Methods

In this section we investigate the regions of absolute stability for explicitRunge�Kutta methods� To recall� this region is obtained for a given methodby determining for what values of z � h� we get jynj � jyn��j when applyingthe method to the test equation�

y� � �y� ������

This test equation is an obvious generalization of ������ without the in�homogeneity� Repeating the same arguments here� we obtain

yn � �� � zbT �I � zA�����yn�� ������

� �� � zbT �I � zA� � � �� zkAk � � � � ���yn���Substituting ����� into this expression and writing ������ as

yn � R�z�yn��

we get for a Runge�Kutta method of order p

R�z� � � � z �z�

�� � � �� zp

p$�Xj�p

zjbTAj��� � ������

For an s�stage explicit method of order p� since Aj�� � �� j � s� we get

yn �

�� � z �

z�

�� � � �� zp

p$�

sXj�p��

zjbTAj���

�yn�� �

In particular� the region of absolute stability of an explicit pth order RKmethod for s � p � � is given by����� � h��

�h���

�� � � �� �h��

p

p$

���� � �� ������

Thus we note that all p�stage explicit Runge�Kutta methods of order p havethe same region of absolute stability� For an s�stage method with order p � s�the absolute stability region is seen to depend somewhat on the method�scoe cients��

The stability regions for the explicit p�stage pth order RK methods� � �p � �� are shown in Fig� ����

�We must consider �� hence z� to be a complex number� because it represents aneigenvalue of a matrix in general�

�For the fourth �and to a lesser extent the third� order methods depicted in Fig� ���there is a stretch along the imaginary axis of z where jR�z�j � �� This translates todissipativity when such methods are used to construct �nite dierence approximationsto hyperbolic PDEs� and it facilitates using such methods as smoothers when designingmultigrid solvers for certain PDEs� No such eect occurs for lower order discretizations�A full discussion of this is well beyond the scope of this book�

Page 105: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

�� Chapter �� Initial Value Problems

−6 −5 −4 −3 −2 −1 0 1−3

−2

−1

0

1

2

3

Re(z)

Im(z

)Stability regions in the complex z−plane

Figure ���� Stability regions for p�stage explicit Runge�Kutta methods oforder p� p � �� �� �� �� The inner circle corresponds to forward Euler� p � ��The larger p is� the larger the stability region� Note the �ear lobes� of the�th order method protruding into the right half plane�

How do you plot a region of absolute stability! Recall that the numbersof modulus one in the complex plane are represented by ei�� for � � � � ��The stability condition is given by jR�z�j � �� where R�z� is given by �������For explicit Runge�Kutta methods R�z� is a polynomial in z � h� given�e�g�� by the expression whose magnitude appears in ������� Thus� to �nd theboundary of the region of absolute stability� we �nd the roots z��� of

R�z� � ei�

for a sequence of � values� Starting with � � �� for which z � �� we repeatedlyincrease � by a small increment� each time applying a root �nder to �nd thecorresponding z� starting from z of the previous � as a �rst guess�� until thestability boundary curve returns to the origin�It is also possible to compute the region of absolute stability via a brute

force approach� To do this� we �rst form a grid over a large part of thecomplex plane including the origin� Then at each mesh point zij� if jR�zij�j ��� we mark zij as being inside the stability region�

�This is an elementary example of a continuation method�

Page 106: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Chapter �� One Step Methods ��

Finally� we note that no explicit Runge�Kutta method can have an un�bounded region of absolute stability� This is because all Runge�Kutta meth�ods applied to the test equation y� � �y yield

yn � R�z�yn��� z � h�

where R�z� is a polynomial of degree s� Since jR�z�j � � as jzj � �� verylarge negative values z cannot be in the region of absolute stability� In fact�it turns out that all known explicit Runge�Kutta methods are inappropriatefor sti� problems� and we are led to consider implicit Runge�Kutta methodsin x���� Before that� though� we discuss some of the ingredients necessary towrite robust software for initial value ODEs�

��� Error Estimation and Control

In virtually all modern codes for ODEs� the step size is selected automaticallyto achieve both reliability and e ciency� Any discretization method with aconstant step size will perform poorly if the solution varies rapidly in someparts of the integration interval and slowly in other� large parts of the inte�gration interval� and if it is to be resolved well everywhere by the numericalmethod �see Exercise ������ In this section we will investigate several waysto estimate the error and select the next step h � hn � tn � tn��� Since westrive to keep the entire integration process local in time �i�e�� we march intime with all the information locally known� we attempt to control the localerror or the local truncation error� rather than the global error� Basically� byspecifying an error tolerance ETOL a user can require a more accurate �andmore expensive� approximate solution or a less accurate �and cheaper� one�Our step�size selection strategy may attempt to roughly equate the errorsmade at each step� e�g��

jlnj � ETOLwhere ln is the local error� �The vector ln has m components for a systemof m �rst order ODEs�� This makes the step size as large as possible� butto achieve a higher success rate in such step�size predictions� we typicallyuse some fraction of ETOL for safety� The global error also relates to thetolerance in case it can be obtained as a simple sum of local errors�If the components of the solution y are very di�erent in magnitude then

we are better o� to consider an array of tolerances� In fact� for each com�ponent j of y �� � j � m� it may be necessary to specify an absolute error

�Recall from ������ that the local truncation error in the nth time step is related tothe local error by hn�jdnj � O�hp���� jlnj�� � O�hn��� Thus� local error control andstep�size selection are sometimes viewed as controlling the local truncation error�

Page 107: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

�� Chapter �� Initial Value Problems

tolerance ATOLj� in addition to a common relative error tolerance RTOL�One then wants to choose h so that for each j� � � j � m�

j�lj�nj � frac �ATOLj � j�yj�njRTOL��

where frac is a safety fraction �say� frac � ���� Good codes allow the speci��cation of m� � tolerances as well as a default option of specifying only oneor two�Let us next assume again a scalar ODE� for notational simplicity� A basic

problem for estimating the step size in Runge�Kutta methods is that theexpression for the local truncation error is so complicated� For example� thelocal truncation error of the ��stage family of explicit second order methodsderived earlier for a scalar ODE is given by

hdn �h�

��

�fyyf

� � �ftyf � ftt�� y���

��O�h���

Since the whole purpose of Runge�Kutta methods was to eliminate the needto calculate the symbolic partial derivatives explicitly� we will look for meth�ods to estimate the error at each step which do not use the partial derivativesdirectly� For this reason� it is convenient in the case of Runge�Kutta methodsto estimate the local error� rather than the local truncation error�The essential idea of the methods described below is to calculate two

approximate solutions yn and �yn at tn� such that �yn� yn gives an estimate ofthe local error of the less accurate of the two approximate solutions� yn� Wecan then check if j�yn � ynj � ETOL� If this inequality is not satis�ed thenthe step h is rejected and another step &h is selected instead� If the methodfor �nding yn has order p then ln�&h� � c&hp��� so we choose &h to satisfy

�&h

h

�p��

j�yn � ynj � fracETOL

and repeat the process until an acceptable step size is found�� If the stepis accepted then the same formula can be used to predict a larger step sizehn�� � &h for the next time step�

Embedded methods

We are searching� then� for methods which deliver two approximations� ynand �yn� at tn� A pair of Runge�Kutta methods of orders p and p��� respec�tively� will do the job� The key idea of embedded methods is that such a pair

�Another safety factor� ensuring that �h�h is neither too large nor too small� is used inpractice� because we are using a simpli�ed model for the error which does not take intoaccount large h� roundo error and absolute stability eects�

Page 108: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Chapter �� One Step Methods ��

will share stage computations� Thus we seek to derive an s�stage formula oforder p� � such that there is another formula of order p embedded inside it�therefore� using the same function evaluations��If the original method is given by

c A

bT

then the embedded method is given by

c A

�bT

We therefore use a combined notation for an embedded method�

c A

bT

�bT

The simplest example is forward Euler embedded in modi�ed trapezoid�

� � �

� � �

� �

��

��

Probably the most famous embedded formula is the Fehlberg ��� pair�It has � stages and delivers a method of order � with an error estimate �or amethod of order without��

��

��

���

���

����

�������� �����

������������

� ������ �� �� �

�� � �����

�� �

�� � ������

� ����� ���

��

���� � ���

���������� ��

���� � ���

�� �� ������ � �

��

Page 109: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

�� Chapter �� Initial Value Problems

Note that we have omitted obvious ��s in the tableau� The somewhatunintuitive coe cients of the Fehlberg pair arise not only from satisfying theorder conditions but also from an attempt to minimize the local error in yn�A question arises with any error estimate� whether one should add the

estimate to the solution to produce a more accurate method �but now withno close error estimate�� Here this would simply mean using �yn rather thanyn for the start of the next step� Of course� this casts doubt on the quality ofthe error estimation� but users rarely complain when a code provides moreaccuracy than requested� Besides� the quality of ETOL as an actual errorestimate is questionable in any case� because it does not directly relate to theactual� global error� More on this later� This strategy� called local extrapolation� has proven to be successful for some methods for nonsti� problems�and all quality explicit Runge�Kutta codes use it� but it is not so common inthe solution of sti� problems�The methods of Dormand and Prince bite the bullet� They are designed

to minimize the local error in �yn� in anticipation that the latter will be usedfor the next step� The ��� pair given below has � stages� but the last stageis the same as the �rst stage for the next step ��yn � Y�� and in the next stepn � �� n� Y� � yn��� so Y� at the current step and Y� at the next step arethe same�� so this method has the cost of a ��stage method�

���

���

���

��� ��

����

�������� �����

�� ����� ��� ����

���

� ������� ��

����������

����� � ���

� ��

� �� � � ��

��������� ��� �

�� ��� �

������� � ���

���������� � �����

������� �����

���

�� � � ��

��������� ��� �

�� ��� � �

Note� For sti� problems� the stability properties of a method and itsembedded pair should be similar� see Exercise ���

Step doubling

The idea behind step doubling is simple� By subtracting the solution ob�tained with two steps of size hn � h from the solution obtained using onestep of size �h� we obtain an estimate of the local error� Since we know the

Page 110: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Chapter �� One Step Methods �

form of the local error as h� �� we can estimate it well� To make this moreprecise� write the local error �recall �������������� as

ln � ��tn� y�tn��hp�� �O�hp����

The function � is called the principal error function� Now� let yn be thesolution using two steps of size h starting from yn��� and let &yn be the solutiontaking one step of size �h from yn��� Then the two local errors satisfy

�ln�h� � �hp��� �O�hp���

ln��h� � ��h�p��� �O�hp���

where we have assumed that the local error after two steps is twice the localerror after one step� �This is true in the limit as h� ��� Then

j&yn � ynj � �hp����p � ��j��tn� yn�j�O�hp����

Thus�

�ln � �hp����tn� y�tn�� � jy� � y�j���p � ���

Although step doubling gives an accurate local error estimate� it is moreexpensive per step� especially for sti� problems� The embedded error esti�mates are cheaper� especially if the importance of an accurate local errorestimate is discounted� The step doubling procedure is general� though� andworks without inventing special embedded pair formulae�

Global error

A similar step doubling procedure can be applied to estimate the global error�Here� after a sequence of steps for integrating the ODE over a given intervalhas been chosen and the integration carried out �say� by an embedded Runge�Kutta pair�� the integration is repeated with each of these steps halved usingthe same discretization method� The above step�doubling error estimationprocedure is repeated� this time for all n� to obtain a global error estimate�

There are other procedures for estimating the global error� but like theone just described� they are all non�local� and as such are much more cumber�some than the local procedures used above� When solving boundary valueproblems one is naturally dealing with the global error anyway� so typicalpractical procedures estimate it� For initial value problems� one would liketo avoid estimating the global error if possible� for the reasons indicated be�fore� However� there are applications where an estimate of the global erroris required�

Page 111: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

�� Chapter �� Initial Value Problems

Example � While the global error often behaves like jenj � maxj jdjj�there are exceptions� Consider

y� � ��y � sin t� � cos t� t � �with y��� � � and � � �� say� Here the exact solution y�t� � sin t issmooth and nicely bounded� and this is what local errors and locally basedstep size selection relate to� But globally the error accumulates roughly likejenj � e�tnmaxj jdjj� Therefore the actual� global error at t � �� say� willbe much poorer than the local error� and will not relate well to a usergivenlocal error tolerance�Fortunately� examples like this are rare in practice� or so one hopes �see

Exercise ����� �

While local error estimates are more easily obtained and they allow amore robust� dynamical error control and step size selection� satisfying aglobal error tolerance is typically closer to what the user �i�e� the personlooking for a solution to a particular application� may want� But how does auser go about specifying a global error tolerance! And how accurately needthe error tolerance�s� be speci�ed and satis�ed!These questions arise in modeling and depend strongly on the application�

Here we just note that a precise error bound is often unknown and not reallyneeded� When a ball�park value for the global error would do� the stock�valueof a local error tolerance goes up�

��� Sensitivity to Data Perturbations

One important factor in assessing the choice of error tolerance for a givenapplication is the accuracy expected of the exact solution� Real�world modelsoften involve various parameters and initial data whose values are determinedby inaccurate means� The exact solution of the given initial value ODEsystem may therefore be viewed as a sample out of a cluster of trajectories�It makes no sense then �it is a waste of resources� to impose an error toleranceso strict that the computed solution is much closer to the exact solution thanthis exact solution trajectory is to its equally valid neighbor trajectories�So� to assess solution accuracy �and worth� in practice a user often needs

a sensitivity analysis� i�e�� we ask by how much does the exact solution changewhen the data are perturbed! Below we consider the relatively simple caseof small perturbations�To be speci�c� let us consider an IVP depending on parameters

y� � f�t�y�p�� � � t � b ������

y��� � c�

Page 112: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Chapter �� One Step Methods ��

The l parameters p can be functions of t� but for simplicity assume they areall given constants� Denote the exact solution of ������ by y�t�� We nextconsider a perturbation vector �� i�e��

%p � p� � �

Call the resulting solution of ������ �i�e�� with %p replacing p� %y�t�� We seeka bound on ky � %yk in terms of j�j in case that j�j is so small that O�j�j��terms can be considered negligible�Thus we write

%y�t� � %y�t�p� �� � y�t�p� �

��y�t�p�

�p

��

and obtain

j%y�t�� y�t�j � jP �t��j�O�j�j�� � � t � b �����a�

where

P ��y

�p�����b�

is an m� l matrix function� The simplest form of sensitivity analysis there�fore consists of approximately calculating P �t�� Then� given bounds on theparameter variation

j�jj � �Uj � � � j � l

we can determine for each t

�i � max jlX

j��

Pi�j�jj �lX

j��

jPi�jj�Uj �

giving the approximate bound

j%y�t�� y�t�j � ��t� �O�j�U j�� � � t � b � ������

The behavior of the perturbation matrix function P �t� is governed by alinear initial value ODE� Di�erentiating ������ with respect to p and notingthat the initial conditions are assumed independent of p� we obtain

P � � ��f

�y�P �

�f

�p� � � t � b ������

P ��� � � �

For each column of P �corresponding to one parameter in p�� we thereforehave an initial value problem which depends on y�t� but is linear in P � Thus�

Page 113: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

�� Chapter �� Initial Value Problems

in order to estimate the perturbation function in practice we may solve ������with a relatively permissive error tolerance� and compute P by integrating������ along as well� using the same time step sequence� The combined systemcan be solved e ciently� noting that the sensitivity system ������ is linearand shares the iteration matrix of the original system ������� and exploitingthis structure�Before turning to an example� we remark that a similar treatment can

be applied to assess solution sensitivity with respect to perturbations in theinitial data c� This is left to Exercise ����

Example � � A simpli�ed description of the motion of a car in an arena isgiven by the equations corresponding to the ODE in ������

x� � v cos � �����a�

y� � v sin � �����b�

�� � vtan�

L�����c�

v� � a� v �����d�

where x and y are the Cartesian coordinates of the car�s center� � is theangle the car makes with the xaxis �see Fig� ���� and v is the velocity withwhich the car is moving� Denote y � �x� y� �� v�T �

x

y

L

�x� y�

Figure ��� Schematic of a mobile robot �not to scale��

Page 114: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Chapter �� One Step Methods ��

The damping �friction factor and the car�s length L are given� wetake � ��� and L � ��� cm �it�s a toy car� The acceleration a andthe steering angle � that the front wheels make with the car�s body are twofunctions which one normally controls in order to drive a car� Here we takethem� for simplicity� as constant parameters� we set a � ��� cm s��� � � ��We are interested in the sensitivity of the car�s position �x� y� with respect toa constant change in ��

Since we are checking sensitivity with respect to only one parameter� P �t�is a vector of length �� The di�erential equations ����� for P �t� are foundby di�erentiating ����� with respect to the parameter �� to obtain

P �� � �vP� sin � � P� cos �

P �� � vP� cos � � P� sin �

P �� � �P� tan� �v

cos� ���L

P �� � � P� �

Note that P depends on y but y does not depend on P � and that the ODEfor P is linear� given y�

We useMatlab to compute the solutions for y�t� and P �t�� starting withy��� � ���� ��� �� ��T � P ��� � �� We evaluate %y��t� � y�t�� �P �t�� and wealso numerically solve ����� directly for the perturbed problems� where � isreplaced by � � �� The resulting plots for � � ���� and for � � ��� aregiven in Fig� ���� We see that for � � ���� the linear sensitivity analysiscaptures the trajectory perturbation rather well� Also� not surprisingly foranyone who drives� the distance between y�t� and the perturbed trajectoriesincreases with t� As the size of the perturbation is increased to � � ���� thelinear approximation becomes less valid� �

Sensitivity analysis plays an important role in a number of situations� inaddition to assessing the accuracy of a model� In model development andmodel reduction� the sensitivity of the solution with respect to perturbationsin the parameters is often used to help make decisions on which parts of themodel are actively contributing in a given setting� Partial derivative matricessimilar to those occurring in sensitivity analysis also arise in the shooting andmultiple shooting methods for boundary value problems �see Chapter ��� aswell as in parameter estimation� design optimization and optimal control�

Page 115: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

��� Chapter �� Initial Value Problems

−4 −2 0 2 4 6 8 10 1210

12

14

16

18

20

22

x

y

y + 0.01P

y − 0.01P

�a� � ����

−6 −4 −2 0 2 4 6 8 10 126

8

10

12

14

16

18

20

22

24

x

y

y − 0.05P

y + 0.05P

�b� � ����

Figure ���� Toy car routes under constant steering� unperturbed �solid line��steering perturbed by �� �dash�dot lines�� and corresponding trajectoriescomputed by the linear sensitivity analysis �dashed lines��

Page 116: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Chapter �� One Step Methods ���

��� Implicit Runge�Kutta and Collocation Meth�

ods

Compared to explicit Runge�Kutta methods� for implicit Runge�Kutta meth�ods there are many more parameters to choose in ����� or ������� Thus� wemight expect to be able to attain a higher order for a given number of stages�This turns out to be the case� as we have already seen in the implicit midpointmethod� which is a ��stage method of order �� Moreover� the ampli�cationfunction R�z� is no longer a polynomial� This enables the construction ofimplicit Runge�Kutta methods which are appropriate for the solution of sti�systems�Many of the most commonly used implicit Runge�Kutta methods are

based on quadrature methods � that is� the points at which the intermediatestage approximations are taken are the same points used in certain classesof quadrature formulas� There are several classes of these methods� of whichwe give some examples with the �rst two instances for each�Gauss methods � these are the maximum order methods � an s�stage Gaussmethod has order �s�

��

��

�implicit midpoint s � ��p � �

��p��

��

���p���

��p�

����

p�

����

��

��

s � �� p � �

Radau methods � these correspond to quadrature rules where one end of theinterval is included �c� � � or cs � ��� and attain order �s � �� The choicec� � � makes no sense so we consider only the case cs � ��

� �

�backward Euler s � ��p � �

��

��

� ���

� ��

��

��

��

s � �� p � �

Page 117: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

��� Chapter �� Initial Value Problems

Lobatto methods � these correspond to quadrature rules where the function issampled at both ends of the interval� The order of accuracy is �s� �� Thereare three families� One such is�

� � �

� ��

��

��

��

trapezoidal method s � ��p � �

� � � �

��

��

��� �

��

� ��

��

��

��

��

��

s � �� p � �

Note that� in constructing a Runge�Kutta method� common sense shouldprevail� For example� while there is no analytical reason why we shouldchoose � � ci � �� in physical applications it sometimes does not make senseto evaluate the function outside the interval�A Runge�Kutta method with a nonsingular coe cient matrix A which

satis�es asj � bj� j � �� � � � � s� is called sti y accurate� This gives sti� decay�Exercise �����

����� Implicit Runge�Kutta Methods Based on Collo�

cation

Collocation is an idea which runs throughout numerical analysis� The basicidea is to choose a function from a simple space �usually a polynomial�� anda set of collocation points� and require that the function satisfy the givenproblem at the collocation points�Starting with a set of s distinct points � � c� � c� � � � � � cs � ��

and considering for simplicity a scalar ODE y� � f�t� y� at �rst� we seek thepolynomial ��t� of degree at most s which collocates the ODE as follows

��tn��� � yn�����ti� � f�ti� ��ti��� i � �� �� � � � � s

where ti � tn�� � cih are the collocation points� This de�nes ��t� uniquely��

�Note that if we collect the polynomial pieces de�ned in this way on each step interval�tn��� tn� into one function de�ned on ��� b�� then we get a continuous� piecewise polynomialapproximation of the solution y�t��

Page 118: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Chapter �� One Step Methods ���

Now� take

yn � ��tn� �

This gives an s�stage implicit Runge�Kutta method� Why! Observe that�� is a polynomial of degree at most s � � which interpolates s data pointsf�ti� ��ti��� De�ne Ki � ���ti�� Now� write �� as a Lagrange interpolationformula

���tn�� � �h� �sX

j��

Lj�tn�� � �h�Kj

where Lj�tn�� � �h� � (si���i��j

���ci��cj�ci�� �Because �

� is a polynomial of degree� s� it agrees with its s�points interpolant identically�� Integrating �� withrespect to t from tn�� to ti� i � �� �� � � � � s� and from tn�� to tn� we get

��ti�� ��tn��� � h

sXj��

�Z ci

Lj�r�dr

�Kj

��tn�� ��tn��� � h

sXj��

�Z �

Lj�r�dr

�Kj �

�Recall again our brief review of basic quadrature�� Now de�ne

aij �

Z ci

Lj�r�dr

bj �

Z �

Lj�r�dr� �����

Thus� Ki � f�ti� ��ti�� � f�ti� yn�� � hPs

j�� aijKj�� and yn � yn�� �hPs

i�� biKi� The obtained formula is therefore a Runge�Kutta method inthe form ������� Finally� note that for the general ODE system

y� � f�t�y�

precisely the same argument can be repeated� where now we have a vectorof m collocation polynomials� ��t��The Gauss� Radau and Lobatto methods introduced above are collocation

methods� That is� given the quadrature points ci� in each case all the othermethod�s coe cients are determined by ������ We note�

� Runge�Kutta methods which are also collocation methods are easy toderive�

� The order of such a collocation Runge�Kutta method is at least s andis determined only by the quadrature order condition ������ �i�e�� theorder limitation is a result from quadrature theory��

Page 119: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

��� Chapter �� Initial Value Problems

� The maximum order of an s�stage Runge�Kutta method is �s�The last two conclusions require a proof which is left for the exercises� Wenote here that the order is restricted to be at most �s by the quadrature ordercondition ������ and that a simple collocation analysis reveals that this order�s is attained by collocation at Gaussian points�

With regard to absolute stability� we have already seen in ������ that forthe test equation y� � �y a Runge�Kutta method reads

yn � R�z�yn��

where z � h� and

R�z� � � � zbT �I � zA����� ������

The region of absolute stability is given by the set of values z such thatjR�z�j � �� For an explicit method� we saw that R�z� is a polynomial� andhence the method cannot be A�stable� For implicit Runge�Kutta methods� incontrast� R�z� is a rational function� i�e� it is a quotient of two polynomials

R�z� �P �z�

Q�z�

and A�stable methods are abundant� All of the implicit Runge�Kutta meth�ods which we have seen so far turn out to be A�stable�When Re�z� � �� we also would like a method to have sti� decay�

For this we must have R���� � � in ������� which is achieved if P �z� hasa lower degree than Q�z�� Note that by ������� if A is nonsingular thenR���� � � � bTA���� so R���� � � if the last row of A coincides withbT � For a collocation Runge�Kutta method this happens when c� � � andcs � �� In particular�

� The Radau methods� extending backward Euler� have sti� decay�� The Gauss and Lobatto methods� which extend midpoint and trape�zoid� do not have sti� decay� although they are A�stable�

These innocent looking conclusions have in fact far reaching importance�The Gauss and Lobatto methods are families of symmetric methods � this isimportant particularly in the context of boundary value problems� Symmet�ric methods can work for sti� problems� but for very sti� problems they donot approximate the exponential function well� The arguments of the previ�ous chapter extend here directly� The Radau methods� on the other hand�are particularly suitable for the solution of sti� initial value ODEs�

Page 120: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Chapter �� One Step Methods ��

Reader�s advice� The technical level of the rest of x��� is higherthan what we�ve had so far� and although it is of practical interest�skipping it should not impede the reading of the next section norof subsequent chapters�

����� Implementation and Diagonally Implicit Meth�

ods

One of the challenges for implicit Runge�Kutta methods is the developmentof e cient implementations� To see why e ciency could be a problem� weconsider again the general Runge�Kutta method

Yi � yn�� � h

sXj��

aijf�tn�� � cjh�Yj�

yn � yn�� � h

sXi��

bif�tn�� � cih�Yi��

For the �th Newton iterate� let �i � Y���i �Y�

i and ri � Y�i � yn�� �

hPs

j�� aijf�Y�j �� Then the Newton iteration takes the form

�BBBBBB�

I � ha��J� �ha��J� � � � �ha�sJs�ha��J� I � ha��J� � � � �ha�sJs

������

� � ����

�has�J� �has�J� � � � I � hassJs

�CCCCCCA

�BBBBBB�

��

�����

�s

�CCCCCCA� �

�BBBBBB�

r�

r����

rs

�CCCCCCA

where Ji � ��f��y�� evaluated at Y�i � i � �� �� � � � � s� We note that for a

system of m di�erential equations� this is an sm � sm system of equationsto be solved at each time step� This is usually not competitive with themultistep methods to come in Chapter � which require only the solution ofan m�m nonlinear system at each time step� Thus� it is important to lookfor ways to make the iteration process less expensive�

Page 121: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

��� Chapter �� Initial Value Problems

Review� The Kronecker product� or direct product of two matricesA and B is given by�

A�B �

�BBBBBB�

a��B a��B � � � a�sB

a��B a��B � � � a�sB���

���� � �

���

as�B as�B � � � assB

�CCCCCCA

There are two important properties of the Kronecker product that wewill need��� �A�B��C �D� � AC �BD

�� �A�B��� � A�� �B���

First� we simplify the Newton iteration by taking J� � J� � � � � � Js �J � ��f��y�� evaluated at yn��� Using the approximate Jacobian doesnot reduce the method�s accuracy� provided that the Newton iteration canstill converge� Using the Kronecker product notation� the simpli�ed Newtonmethod can now be written as

�I � hA� J�� � �r� ������

Note that while � and r depend on the iteration �� the matrix in ������ isthe same for all iterations� and it depends only on the step counter n� So�at most one LU �decomposition is needed per time step �at most� because wemay hold this matrix �xed over a few time steps��

Unfortunately� collocation tends to yield Runge�Kutta methods where thecoe cient matrix A � �aij�

si�j�� has few zeros� In particular� there are no

zeros in the coe cient matrices of Radau collocation methods� For e ciencyreasons one can therefore consider also non�collocation implicit Runge�Kuttamethods for which A is a lower triangular matrix�

One such family of methods are the Diagonally Implicit Runge�Kutta�DIRK� methods� These are implicit Runge�Kutta methods for which thecoe cient matrix A is lower triangular� with equal coe cients a along the

Page 122: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Chapter �� One Step Methods ���

diagonal� Thus� the stages in the Runge�Kutta method are de�ned by

Yi � haf�tn�� � cih�Yi� � yn�� � h

i��Xj��

ai�jf�tn�� � cjh�Yj��

i � �� � � � � s � ������

Instead of having to solve an sm � sm system of linear equations we nowhave to solve s systems of size m�m each� all with the same matrix I�haJ �Hence the nonlinear system can be solved by block back�substitution� Onlyone evaluation of the local Jacobian J and one LU�decomposition of them�m sub�matrix I � haJ need to be done on each time step�

DIRK methods have become popular for the numerical solution of time�dependent PDEs via the method of lines �recall Examples ��� and ����� Herethe Jacobian J is very large and sparse� and iterative methods are often usedfor the linear algebra� A fully implicit Runge�Kutta method makes thingscumbersome� but DIRKmethods o�er a Runge�Kutta alternative to the BDFmethods of the next chapter in this situation� When iterative methods areused it becomes less important to insist that the diagonal elements of thecoe cient matrix A be all the same� however� it turns out that this extrafreedom in designing the DIRK method does not buy much�

Because so many of the coe cients of DIRK methods have been speci�edby construction to be zero� it is not surprising that the maximum attainableorder is much less than for general implicit Runge�Kutta methods� In fact� ithas been shown that the maximum order of an s�stage DIRK method cannotexceed s � �� Some such methods are the midpoint method �s � �� p � ��and

� � �� �

��� ���

s � �� p � �

with � ��p�

�� This latter method satis�es R���� � � �p� � ��������

Thus R���� � �� a marked improvement over the midpoint method whichhas no attenuation � R���� � ��� at the sti�ness limit z � Re���h � ���If the method is to have sti� decay as well then the order is further

�Strictly speaking� DIRK methods do not require equal coe�cients along the diagonal�However� the subset of DIRK methods with equal diagonal coe�cients� which is calledsingly diagonally implicit Runge�Kutta �SDIRK�� are the methods most commonly usedbecause of the possibility for a more e�cient implementation� So� we refer only to theSDIRK methods� and we call them DIRK to distinguish from the rather dierent SIRKmethods which arise in x������

Page 123: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

��� Chapter �� Initial Value Problems

restricted to s� Examples are backward Euler �s � p � ���

� � �

� �

s � �� p � �

where � ��p�� � and

�������� �������� � �

����������� ����������� �������� �

� ����������� ����������� ��������

����������� ����������� ��������

s � �� p � �

����� Order Reduction

When speaking of the order of accuracy of a method and making statementslike

kek � max��n�N

jenj � O�hp�

regarding the error in a method of order p� we mean that as the maximumstep size h � � �and N � ��� the error also shrinks� so fast that h�pkekremains bounded in the limit� In a �nite world we must of course think of has small but �nite� and we normally think of it as being much smaller thanthe smallest scale of the ODE being approximated�This changes in the very sti� limit� We have considered already in Chap�

ter � problems like

y� � ��y � q�t��� � � t � � ������

where � � ��Re��� � h � �� Here there are two small parameters� one

the method�s step�size and the other the problem�s� We consider the limitprocess in which �

�Re��� � � faster than h� �� In this case our statementsabout the method�s order may have to be revised�Indeed� some Runge�Kutta methods su�er from a reduction in their order

of convergence in the very sti� limit� The essential reason is that thesemethods are based on quadrature� and in the very sti� case the integratione�ect is very weak� at least for some solution components� For example�assume that � is real in ������� Upon dividing ������ by �� we see that

Page 124: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Chapter �� One Step Methods ���

y� is multiplied by a constant which shrinks to � while the right hand sideis scaled to �� so there is almost no integration in determining y�t� fromq�t�� Upon applying a Runge�Kutta method in this case we obtain almost aninterpolation problem at the internal stages of a Runge�Kutta method� Theaccuracy order of this interpolation� and not the quadrature precision� thentakes center stage�The phenomenon of order reduction is important particularly for DAEs�

so we leave a fuller discussion to Chapter ��� Here we note that somemethodsare a�ected by this more than others� Unfortunately� DIRKmethods are only�rst order accurate at the very sti� limit� which suggests that they shouldnot be used for very sti� problems� Fortunately� many time�dependent PDEsare sti� but not very sti�� Collocation at Radau points retains its full usualorder in the very sti� limit� That is one reason why these methods are sopopular in practice despite the more expensive linear algebra necessary fortheir implementation�

����� More on Implementation and SIRK Methods

The DIRK methods require a special zero�structure for the coe cient matrixA� which implies the restrictions discussed above� A clever alternative isto seek implicit RK methods where A can be transformed by a similaritytransformation T into a particularly simple form�

T��AT � S�

It can be shown that� upon transforming the variables

�� � �T�� � I��

and multiplying equations ������ by T���I� the matrix problem to be solvedhas the form

�I � hS � J��� � ��r ������

where �r � �T��� I�r� Thus� any lower triangular matrix S yields the DIRKstructure in ������ for the transformed variables ��� An e cient implemen�tation of Radau collocation can be obtained in this way �or using anothersimple form of S�� We can go further� and require that S be a scalar multi�ple of the identity matrix� i�e� we look for methods in which A has a singles�fold real eigenvalue� a� This yields the SIRK �singly implicit Runge�Kutta�methods� Here� at most one m�m matrix needs be formed and decomposedper step� A good s�stage method of this sort has order s � �� and unlikeDIRK this order does not reduce in the very sti� limit�

Page 125: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

��� Chapter �� Initial Value Problems

��� Software Notes and References

����� Notes

Runge ���� and Kutta ���� did not collaborate to invent the methods bearingtheir names� rather� Runge was �rst� and Kutta gave the general form� Buttogether they were responsible for much of the development of the earlyexplicit RK methods� Chapter II of Hairer� Norsett Wanner ��� gives afull exposition which we do not repeat here�

As noted earlier� the order of an e cient RK method can be a challenge toverify if it is higher than � or �� and our exposition in x��� aims to give a tasteof the issues involved� rather than to cover the most general case� Excellentand elaborate expositions of the general theory for order conditions can befound in Butcher ���� or in ���� These references also discuss order barriers�i�e�� limitations on the attainable order for explicit RK as a function of thenumber of stages�

Error estimation and control is discussed in all modern texts� e�g� ������� �� �� ��� Shampine ��� has many examples and elaborates on pointsof practical concern�

For a comprehensive treatment of implicit RK methods and related collo�cation methods we refer to Hairer Wanner ���� It contains the theorems�proofs and references which we have alluded to in x���� The concepts of sti�accuracy and sti� decay were introduced in �����

A large number of topics have been omitted in our presentation� despitetheir importance� Below we brie�y mention some of these topics� Othershave made their way into the exercises�

The Runge�Kutta family is not the only practical choice for obtaining highorder one step methods� Another family of methods is based on extrapolation�Early e�orts in the ���s are due to W�B� Gragg� to J� Stoer and R� Bulirsch�and to H� Stetter ����� See ��� for a full description� These methods haveperformed well� but overall they do not appear to outperform the familiesof methods discussed in this book� The extrapolation idea is discussed inChapter ��

A great deal of theory relating to stability and convergence for sti� prob�lems has been developed in the past few decades� We have described someof the more accessible work� Many di�erent stability de�nitions have beenproposed over the years� perhaps not all of them have stood the test of time�But we mention in particular the theories of order stars �and rational ap�proximations to the exponential� and Bconvergence� for their elegance andability to explain the behavior of numerical methods ����

A topic of practical importance is dense output� or continuous extension�see� e�g�� ��� or ���� Normally� a discretization method yields approximatesolution values only at discrete mesh points� If every point at which a so�

Page 126: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Chapter �� One Step Methods ���

lution is required is taken to be a mesh point� an ine cient procedure mayresult� A better� obvious idea is to use cubic Hermite interpolation �which isa cubic polynomial interpolant using two function values and two derivativevalues� because at mesh points we have both yn and f�tn�yn� which approx�imates y��tn�� Dense output can often be accomplished more e ciently� andat higher order� using Runge�Kutta methods which have been speci�callydesigned for this purpose�One important application� in addition to plotting solutions� for which

dense output is needed� is for event location� Recall from x��� that if f hasdiscontinuities� determined according to a change of sign of some switch�ing function� then it is desirable to place mesh points at or very near suchswitching points �which are points of discontinuity� in t� Hence the �event�of the discontinuity occurrence requires detection� This is done by solvinga nonlinear equation iteratively and the function values needed are suppliedby the dense output mechanism�Another important instance where dense output may be required is in

solving delay di�erential equations� A simple prototype is

y��t� � f�t�y�t��y�t� � ��

where � is the delay� Delay equations can be very complicated� see� e�g�� ����We do not pursue this except to say that when discretizing in a straightfor�ward way at t � tn� say� also the value of y at tn � � is required� If thatdoes not fall on a past mesh point then the dense output extension must becalled upon� An IVP implementation is described in ���� For possibilities ofconverting delay di�erential equations to standard ODEs� see Exercise ����We have commented in x��� on the dissipativity of explicit Runge Kutta

methods of order � � and the use of these methods as a smoother in PDEsolvers� A reference is Jameson ����A lot of attention has been devoted in recent years to symplectic methods�

Recall from x�� that a Hamiltonian system provides a symplectic map� Asa corollary� considering a set of initial values each spawning a trajectoryof a Hamiltonian system� the volume of this set at a later time remainsconstant under the �ow� Next� we may ask if the property of the symplecticmap is retained by a given numerical method� A numerical discretizationthat preserves symplecticity for a constant step size is called a symplecticmethod� Such methods are particularly desirable for applications involvinglong time integration� i�e� Nh � b �� where h is the step size and Nis the number of steps taken� Examples appear in molecular dynamics andcelestial mechanics simulations� For much more on this topic� see Sanz�Serna Calvo ���� and ���� See also Exercises ����� ���� and ����� As it turns out�there are di culties in constructing general� e cient methods of this sort�and varying the step size is also a challenge� yet there are instances wheresymplectic methods impressively outperform standard methods�

Page 127: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

��� Chapter �� Initial Value Problems

Viewing the discretization of a nonlinear ODE as a continuous dynamicalsystem yields a discrete dynamical system� One may wonder if the dynam�ical properties of the two systems are qualitatively similar� especially if thediscretization step size h is not very small or the number of steps taken Nis very large� We have already indicated above that this is not necessarilyso� e�g� when a non�symplectic method is applied to a Hamiltonian system�Another instance is where the discrete dynamical system has more solutionsthan its continuous counterpart� Viewed as a function of the step size hthere are principal solutions which tend towards the corresponding genuinesolutions� and in addition there may be spurious solutions � The latter wouldtend to � as h� �� but they may be confusingly present for a �nite h� Formore on this see� e�g�� Stuart Humphries �����Many e�orts have been invested since the �����s in the development of

Runge�Kutta methods suitable for the solution of large ODE systems on parallel computer architectures� The book by Burrage ���� covers such methodswell� Basic ideas include the design of Runge�Kutta methods where di�erentstages are su ciently independent from one another that they can be eval�uated in parallel� This is parallelism in time t� Another direction exploitsparallelism also in the large system being integrated� and often leads to morespecialized methods� We mention the closely related multirate methods andwaveform relaxation methods� In the former� di�erent components of the sys�tem that vary more slowly are discretized over larger elements �time steps�than more rapidly varying components� allowing for parallel evaluations ofdi�erent parts of the system at a given stage� �The system being integratedmust be such that a block decoupling of this sort is possible� this is often thecase in VLSI circuit simulation� for example�� In the latter� a global iterativemethod in time such as

�y����� �My��� � f�t�y���My�

is considered� where y��t� is known at the start of iteration �� � � �� �� � � � �This is really considered not for all time but over a time window �such as thelarge step size in a multirate method�� and the matrix M is now chosen toallow parallelism in the evaluation of the iteration�

����� Software

Here we brie�y mention some �certainly not all� general�purpose codes andthe methods on which they are based� All of these codes provide error controland step�size selection� Some of them are available through Netlib�

For nonsti� problems�

� Many Runge�Kutta codes have been based on the Fehlberg ��� em�bedded pair ����� An early in�uential code was rkf�� by Shampine

Page 128: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Chapter �� One Step Methods ���

Watts �����

� The code dopri� presented in the monumental book ��� is based onthe Dormand�Prince ��� formulae ����� which uses local extrapolation�The code ode�� used in Matlab is also based on these formu�lae� a switch from earlier Matlab versions where it was based on theFehlberg pair� This re�ects the current accumulated experience whichsuggests that the Dormand�Prince pair performs better in practice�

� Other codes based on embedded Runge�Kutta pairs are dverk by Hull�Enright Jackson ��� which uses a pair of formulae by Verner �����and rksuite by Brankin� Gladwell Shampine ����� which implementsthree Runge�Kutta pairs � ����� ��� and ���� � and uses local extrap�olation� The latter has an option for global error estimation and it alsoautomatically checks for sti�ness�

For sti� problems�

� The code radau� ��� uses the Radau ��stage formula of order withan implementation of the linear algebra along the lines described inx������

� The code stride by Burrage� Butcher Chipman ��� uses a familyof SIRK methods�

The codes dopri�� ode��� rksuite� radau� and stride all have adense output option� The code ode�� also has a built�in event locationmodule�

��� Exercises

�� Show that a general Runge�Kutta method ����� can be written in theform ������� What is the relationship between Ki of �����a� and Yi of����a�!

�� Show that the explicit Runge�Kutta methods described in x��� can allbe written in the form ������� with � Lipschitz continuous in y if f is�

�� Prove� the onestep method ����� is �stable if � satis�es a Lipschitzcondition in y�

�� Write a computer program that will �nd an upper bound on the orderof a given Runge�Kutta method by checking the order conditions �������

Page 129: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

��� Chapter �� Initial Value Problems

or alternatively� only ������ and ����� �and ������� For a given numberof stages s your program should check these conditions for k � �� �� � � �until one is violated �this will happen before k reaches �s � ��� Note�do not use any matrix�matrix multiplications$

�a� Apply your program to all the embedded methods given in x���both methods of each pair��

�b� Apply your program to the Lobbato methods given in x�����c� What does your program give for the counter�example of Example

��� �x����! What is the actual order of that method!� For a ��stage explicit Runge�Kutta method of order � show�

�a�sX

i��

biaij � bj�� � cj�� j � �� � � � � s

�This is a useful additional design requirement in general�� � Theproof is a bit tricky and is given in �����

�b� Using this result show that we must have c� � � in this case�

�� It has been argued that displaying absolute stability regions as in Fig���� is misleading� since a step of an s�stage explicit method costs es�sentially s times a forward Euler step� its stability region should becompared with what forward Euler can do in s steps� Thus� the scaledstability region of an s�stage explicit method is the stability regionshrunk by a factor s�

For all Runge�Kutta methods with p � s� s � �� �� �� �� plot the scaledstability regions� Observe that forward Euler looks mighty good� noother method�s scaled stability region fully contains the forward Eulercircle ����

�� An RK method with a nonsingular A satisfying

asj � bj� j � �� � � � � s ������

is called sti y accurate �����

�a� Show that a sti)y accurate method has sti� decay�

�b� Show that a collocation method is sti)y accurate i� c� � � andc� � ��

�c� Not all RK methods which have sti� decay satisfy ������� Showthat sti� decay is obtained also if A is nonsingular and its �rstcolumn is b���

Page 130: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Chapter �� One Step Methods ��

�� For a given ODEy� � f�y�

consider the ��method

yn � yn�� � hn��f�yn� � ��� ��f�yn����

for some value �� � � � � ���a� Which methods are obtained for the values �i� � � �� �ii� � � ��

and �iii� � � ���!

�b� Find a range of ��values� i�e� an interval ��� �� such that themethod is A�stable for any � � � � �

�c� For what values of � does the method have sti� decay!

�d� For a given �� � � � � �� let us call a method ��damping if

jynj � �jyn��jfor the test equation y� � �y as �h���� �Thus� if y� � � thenfor any tolerance TOL � �� jynj � TOL after n steps when nexceeds logTOL

log ��

Find the range of ��values such that the ��method is ��damping�

�e� Write the ��method as a general Runge�Kutta method� i�e� specifyA� b and c in the tableau

c A

bT

�f� What is the order of the ��method!

�If you managed to answer the previous question then try to an�swer this one without any Taylor expansions��

�� The solution of the problem y� � f�y�� y��� � c� where

f�y� � ��y�� y��T � c � ��� ��T

satis�esy�� � y�� � �

i�e� it is a circle of radius � centered at the origin� Integrating thisODE numerically� though� does not necessarily satisfy this invariant�and the obtained curve in the y� � y� plane does not necessarily close�

Show that when using collocation based on Gaussian points� the ap�proximate solution does satisfy the invariant� i�e� the obtained approx�imate solution stays on the circle� �See also Exercise ���� where a hintis provided��

Page 131: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

��� Chapter �� Initial Value Problems

��� In molecular dynamics simulations using classical mechanics modeling�one is often faced with a large nonlinear ODE system of the form

Mq�� � f�q�� where f�q� � �rU�q�� ������

Here q are generalized positions of atoms� M is a constant� diagonal�positive mass matrix and U�q� is a scalar potential function� Also�rU�q� � � �U

�q�� � � � � �U

�qm�T � A small �and somewhat nasty� instance

of this is given by the Morse potential ���� where q � q�t� is scalar�U�q� � D���e�S�q�q����� and we use the constantsD � ���������e���S � ������ q� � ���� and M � ������

�a� De�ning the momenta p � Mq�� the corresponding �rst orderODE system for q and p is given by

Mq� � p �����a�

p� � f�q�� �����b�

Show that the Hamiltonian

H�q�p� � pTM��p�� � U�q�

is constant for all t � ��

�b� Use a library nonsti� Runge�Kutta code based on a �� embeddedpair to integrate this problem for the Morse potential on the in�terval � � t � ����� starting from q��� � ����� p��� � ��

� M �

Using a tolerance TOL � ��e � � the code should require a lit�tle more than ���� times steps� Plot the obtained values forH�q�t�� p�t���H�q���� p����� Describe your observations�

��� The system ������ is in partitioned form� It is also a Hamiltonian systemwith a separable Hamiltonian� i�e�� the ODE for q depends only on pand the ODE for p depends only on q� This can be used to designspecial discretizations� Consider a constant step size h�

�a� The symplectic Euler method applies backward Euler to �����a�and forward Euler to �����b�� Show that the resulting method isexplicit and �rst order accurate�

�b� The leapfrog� or Verlet method can be viewed as a staggered mid�point discretization�

M�qn���� � qn����� � h pn

pn � pn�� � h f�qn�����

Page 132: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Chapter �� One Step Methods ���

i�e�� the mesh on which the q�approximations �live� is staggeredby half a step compared to the p�mesh� The method can be kick�started by

q��� � q� � h��M��p��

To evaluate qn at any mesh point� the expression

qn ��

��qn���� � qn�����

can be used�

Show that this method is explicit and �nd order accurate�

�c� Integrate the Morse problem de�ned in the previous exercise using���� uniform steps h� Apply three methods� forward Euler� sym�plectic Euler and leapfrog� Try the values h � �� h � ������and h � ����� and plot in each case the discrepancy in theHamiltonian �which equals � for the exact solution�� The plotfor h � ������ is given in Fig� ����

0 500 1000 1500 2000 2500−0.5

0

0.5

1

1.5

2

2.5x 10

−3

t

H(t

) −

H(0

)

Figure ���� Energy error for the Morse potential using leapfrog with h ��������

What are your observations! �The surprising increase in leapfrogaccuracy from h � ������ to h � ����� relates to a phenomenoncalled resonance instability��

�Both the symplectic Euler and the leapfrog method are symplectic �like the exact ODE they conserve certain volume projections for Hamil�tonian systems �x���� We refer to ���� �� ��� for much more on sym�plectic methods��

Page 133: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

��� Chapter �� Initial Value Problems

��� The following classical example from astronomy gives a strong motiva�tion to integrate initial value ODEs with error control�

Consider two bodies of masses � � ����������� and �� � � � � �earthand sun� in a planar motion� and a third body of negligible mass �moon�moving in the same plane� The motion is governed by the equations

u��� � u� � �u�� � ��

u� � �

D�� �

u� � ��D�

u��� � u� � �u�� � ��u�D�

� �u�D�

D� � ��u� � ��� � u������

D� � ��u� � ���� � u�������

Starting with the initial conditions

u���� � ������ u���� � �� u����� � �� u

����� � ����������������������������

the solution is periodic with period � ����� Note thatD� � � at ���� ��and D� � � at ���� ��� so we need to be careful when the orbit passesnear these singularity points�

The orbit is depicted in Fig� ���� It was obtained using the Fehlberg

−1.5 −1 −0.5 0 0.5 1−1.5

−1

−0.5

0

0.5

1

1.5

u_1

u_2

Figure ���� Astronomical orbit using the Runge�Kutta Fehlberg method�

embedded pair with a local error tolerance ��e � �� This necessitated��� time steps�

Page 134: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Chapter �� One Step Methods ���

Using the classical Runge�Kutta method of order �� integrate this prob�lem on ��� ����� with a uniform step size� using ���� ����� ����� and����� steps� Plot the orbit for each case� How many uniform steps areneeded before the orbit appears to be qualitatively correct!

��� For an s�stage Runge�Kutta method ������ de�ne the s� s matrix Mby

mij � biaij � bjaji � bibj�

The method is called algebraically stable ���� if b � � �componentwise�and M is nonnegative de�nite� Show

�a� Radau collocation is algebraically stable�

�b� Gauss collocation is algebraically stable� In fact� M � � in thiscase�

�c� The trapezoidal method� hence Lobatto collocation� is not alge�braically stable�

�d� Algebraic stability is equivalent to AN �stability� i�e� for the nonau�tonomous test equation

y� � ��t�y

one gets jynj � jyn��j whenever Re� � �� all t�

�This exercise is di cult� The basic idea is to write the expression forjynj�� and substitute yn�� in terms of Yi in it ��

��� A Runge�Kutta method ����� is symmetric if it remains invariant undera change of direction of integration� Thus� letting zn � yn��� zn�� �yn� Zj � Ys���j and h � �h� the same method ����� is obtained forzn�

�a� Let

E �

�BBBBBB�

� �

�� �

�CCCCCCA�

Show that the Runge�Kutta method ����� is symmetric if

c� Ec � �� b � Eb

EAE �A � �bT �

�These conditions are essentially necessary for symmetry as well��

Page 135: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

��� Chapter �� Initial Value Problems

�b� Show that a symmetric Runge�Kutta method is algebraically sta�ble if and only if

M � � �

�� The problem considered in Exercise ��� is a simple instance of a systemwith an invariant ���� More generally� an ODE system y� � f�y� mayhave an invariant de�ned by algebraic equations

h�y� � � ������

meaning that for the exact solution y�t� of the ODE we have h�y�t�� ��� provided the initial values satisfy h�y���� � �� The question is�which numerical discretization of the ODE �if any� satis�es the invari�ant precisely� i�e��

h�yn� � h�yn��� n � �� �� � � � � N �

Denote the Jacobian H � hy and assume it has a full row rank for allrelevant y� We say that we have an integral invariant if

Hf � �� �y��See� e�g�� ������

�a� Show that any Runge�Kutta method preserves linear invariants�

�b� Show that collocation at Gaussian points� and only at Gaus�sian points� preserves quadratic integral invariants� �Hint� Writeh�yn� � h�yn����

R tntn��

h� and use your knowledge of quadrature��

�More generally� for Runge�Kutta methods the needed conditionis M � ���

�c� The non�dimensionalized equations in Cartesian coordinates forthe simple pendulum can be written as

'q� � v�� 'v� � �q��� 'q� � v�� 'v� � �q�� � �q�� � q�� � ��

Di�erentiating the constraint twice and eliminating � yields theODE

'q� � v�� 'v� ��q�

q�� � q���v�� � v�� � q��

'q� � v�� 'v� ��q�

q�� � q���v�� � v�� � q��� �

with the invariantsq�� � q�� � �

Page 136: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Chapter �� One Step Methods ���

q�v� � q�v� � ��

Show that the midpoint method preserves the second of theseinvariants but not the �rst� �You may show this by a numericaldemonstration��

��� This exercise builds on the previous one�

�a� Consider the matrix di�erential system

'U � A�t� U�U � � t � b �����

U��� � I

where A and U are m�m and A is skew�symmetric for all U� t�

AT � �A �

It can be shown that the solution U�t� is then an orthogonal ma�trix for each t�

Show that collocation at Gaussian points �including the midpointmethod� preserves this orthogonality� We note that collocationat Lobatto points �including the trapezoidal method� does notpreserve orthogonality�

�b� A number of interesting applications lead to problems of isospectral �ow ����� where one seeks a matrix function satisfying

'L � AL� LA ������

L��� � L�

for a given initial value matrix L�� where A�L� is again skew�symmetric� The eigenvalues of L�t� are then independent of t�

Verify thatL � UL�U

T

where U�t� is the orthogonal matrix function satisfying

'U � AU� U��� � I

and propose a discretization method that preserves the eigenvaluesof L�

��� This exercise continues the previous two�

Collocation at Gaussian points is an implicit� expensive method� Analternative idea is to use an explicit Runge�Kutta method� orthogonal�izing U at the end of each time step ����� Consider a method of the form������ for which we consider the matrix U�t� written as an m��length

Page 137: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

��� Chapter �� Initial Value Problems

vector of unknowns� Since the result of this step is not necessarily anorthogonal matrix� a step of this method starting with an orthogonalUn�� approximating U�tn��� consists of two phases�

�Un � Un�� � h��tn��� Un��� h��Un � UnRn

where UnRn is a QR�decomposition of �Un� The orthogonal matrix Un

is then the projection of the result of the Runge�Kutta step onto theinvariant manifold� and it is taken as the end result of the step�

Write a program which carries out this algorithm using the classical�th order Runge�Kutta method� �A library routine from LINPACK orMatlab can be used for the decomposition�� Try your program on theproblem

U � � �

�� � �

�� �

�AU

whose exact solution is the re�ection matrix

U�t� �

�� cos�t sin�t

� sin�t cos�t

�A

for various values of �� h and b� What are your conclusions!

�Note that the QR�decomposition of a matrix is only determined up tothe signs of the elements on the main diagonal of the upper triangularmatrix R� You will have to ensure that Un is that orthogonal matrixwhich is close to �Un��

��� If you are a Matlab fan� like we are� then this exercise is for you�

Matlab �version � o�ers the user a simple ODE integrator� calledode��� which is based on the Dormand�Prince embedded pair� Weused this facility to generate the plot of Fig� ��� in less than one person�hour� in fact� In the interest of keeping things simple� the designers ofMatlab kept the interface for this routine on an elementary level� andthe user simply obtains �the solution��

�a� Use Matlab to solve the problem of Example ���� Plot the ob�tained solution� Does it look like the exact one y�t� � sin t!Explain your observations�

�b� It can be argued that the solution thatMatlab produces for thisexample does not look plausible �or �physical��� i�e� we could

Page 138: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Chapter �� One Step Methods ���

guess it�s wrong even without knowing the exact one� Can youconstruct an example that will makeMatlab produce a plausible�looking solution which nonetheless is in ��� error! �This ques�tion is somewhat more di cult��

��� The modi�ed Kepler problem ���� �� is a Hamiltonian system� i�e��

q� � Hp� p� � �Hq �

with the Hamiltonian

H�q�p� �p�� � p���

� �r� �

�r�

where r �pq�� � q�� and we take � � ����� Clearly� H � � Hqq

� �Hpp

� � �� so H�q�t��p�t�� � H�q����p����� �t� We consider simu�lating this system over a long time interval with a relatively coarse�uniform step size h� i�e� bh �� The mere accumulation of local errorsmay then become a problem� For instance� using the explicit midpointmethod with h � ��� and b � ��� the approximate solution for rbecomes larger than the exact one by two orders of magnitude�

But some methods perform better than would normally be expected�In Fig� ��� we plot q� vs� q� ��phase plane portrait�� for �a� the implicitmidpoint method using h � ���� �b� the classical explicit Runge�Kuttamethod of order � using h � ���� and �c� the exact solution �or rather�a su ciently close approximation to it�� The initial conditions are

q���� � �� � q���� � �� p���� � �� p���� �p�� � ����� �

with � ���� Clearly� the midpoint solution with this coarse step sizeoutperforms not only the explicit midpoint method but also the �thorder method� Even though the pointwise error reaches close to ���when t is close to b� the midpoint solution lies on a torus� like the exactsolution� whereas the RK� picture is noisy� Thus� we see yet again thattruncation error is not everything� even in some nonsti� situations� andthe theory in this case must include other aspects�

Integrate these equations using the two methods of Fig� ��� with aconstant step size h � ��� and h � ���� �four runs in total�� moni�toring the maximum deviation jH�q�t��p�t�� � H�q����p����j� �Thisis a simple error indicator which typically underestimates the error inthe solution components� and is of interest in its own right�� What areyour conclusions!

Page 139: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

��� Chapter �� Initial Value Problems

−1.5 −1 −0.5 0 0.5 1 1.5

−1

−0.5

0

0.5

1

q1

q2

q1 vs q2 for Kepler problem

�a� Implicit midpoint� ���� uniform steps

−1.5 −1 −0.5 0 0.5 1 1.5

−1

−0.5

0

0.5

1

q1

q2

q1 vs q2 for Kepler problem

�b� RK�� ���� uniform steps

−1.5 −1 −0.5 0 0.5 1 1.5

−1

−0.5

0

0.5

1

q1

q2

q1 vs q2 for Kepler problem

�c� Exact solution for T���

Figure ���� Modi�ed Kepler problem� approximate and exact solutions

Page 140: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Chapter �

Linear Multistep Methods

In this chapter we consider another group of methods extending the basicmethods of Chapter � to higher order� see Fig� ���� The methods consideredhere use information from previous integration steps to construct higher orderapproximations in a simple fashion� Compared to the Runge�Kutta methodsof the previous chapter the methods here typically require fewer functionevaluations per step� and they allow a simpler� more streamlined methoddesign� at least from the point of view of order and error estimate� On theother hand� the associated overhead is higher as well� e�g� when wanting tochange the step size� and some of the �exibility of one�step methods is lost�For our prototype ODE system

y� � f�t�y�� t � �it is customary to denote

fl � f�tl�yl�

where yl is the approximate solution at t � tl� The general form of a k�steplinear multistep method is given by

kXj��

�jyn�j � h

kXj��

jfn�j

where �j � j are the method�s coe cients� We will assume that �� �� �� andj�kj � jkj �� �� To eliminate arbitrary scaling� we set �� � �� The linearmultistep method is explicit if � � � and implicit otherwise� Note that thepast k integration steps are assumed to be equally spaced�Throughout most of this chapter we again consider a scalar ODE

y� � f�t� y�

to simplify the notation� The extension to ODE systems is straightforwardunless otherwise noted� We also assume� as before� that f has as many

��

Page 141: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

��� Chapter �� Initial Value Problems

bounded derivatives as needed� The general form of the method is rewrittenfor the scalar ODE for later reference�

kXj��

�jyn�j � h

kXj��

jfn�j � ����

The method is called linear because� unlike general Runge�Kutta� theexpression in ���� is linear in f � Make sure you understand that this doesnot mean that f is a linear function of y or t� i�e� it is the method which islinear� not the ODE problem to be solved� A consequence of this linearityis that the local truncation error� to be de�ned later� always has the simpleexpression

dn � Cp��hpy�p����tn� �O�hp��� ����

where p is the method�s order and Cp�� is a computable constant� We willshow this in x���

The most popular linear multistep methods are based on polynomial in�terpolation� and even methods which are not based on interpolation useinterpolation for such purposes as changing the step size� So be sure thatyou�re up on polynomial interpolation in Newton�s form�

Page 142: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Chapter �� Multistep Methods ���

Review� The interpolating polynomial and divided di�erences�Let f�t� be a function to be interpolated at the k distinct pointst�� t�� � � � � tk by the unique polynomial ��t� of degree � k which satis��es the relations

��tl� � f�tl�� l � �� �� � � � � k�

The polynomial can be written down explicitly in Lagrangian form aswe did in Chapter �� Here� though� it is more convenient to write ��t�in Newton�s form�

��t� � f �t���f �t�� t���t�t���� � ��f �t�� t�� � � � � tk��t�t���t�t�� � � � �t�tk���

where the divided di�erences are de�ned recursively by f �tl� � f�tl��

f �tl� � � � � tl�i� �f �tl��� � � � � tl�i�� f �tl� � � � � tl�i���

tl�i � tl�

The interpolation error at any point t is then

f�t�� ��t� � f �t�� � � � � tk� t�(ki���t� ti��

If the points ti and t are all in an interval of size O�h� and f has kbounded derivatives then the interpolation error is O�hk�� If h is smallthen k$f �t�� � � � � tk� t� � f �k��t�� Finally� for the case where the pointstl are equally spaced the expression for divided di�erences obviouslysimpli�es� We de�ne for future reference

r�fl � fl

rifl � ri��fl �ri��fl��� ����

An important property of the divided di�erences of f is that theyapproximate the derivatives� rkf � hkf �k��

��� The Most Popular Methods

Linear multistep methods typically come in families� The most popular fornonsti� problems is the Adams family and the most popular for sti� problemsis the BDF family� In this section we derive these methods via the interpolat�ing polynomial� In the next section we give an alternative derivation which

Page 143: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

��� Chapter �� Initial Value Problems

is applicable for general multistep methods� We note that although the de�rived formulae in this section are for a constant step size h� the derivationsthemselves also suggest how to obtain formulae for a variable step size�

����� Adams Methods

Starting with the di�erential equation

y� � f�t� y��

we can integrate both sides to obtain

y�tn� � y�tn��� �Z tn

tn��

f�t� y�t��dt�

For Adams methods� the integrand f�t� y� is approximated by an interpo�lating polynomial through previously computed values of f�tl� yl�� In thegeneral form ���� we therefore set� for all Adams methods� �� � �� �� � ��and �j � �� j � ��The k�step explicit Adams method is obtained by interpolating f

through the previous points t � tn��� tn��� � � � � tn�k� see Fig� ��� The explicit

t

n-1

y’(t)

fn-k

fn-2f

Figure ��� Adams�Bashforth methods

Adams methods� are the most popular among explicit multistep methods� Asimple exercise in polynomial interpolation yields the formulae

yn � yn�� � h

kXj��

jfn�j �

�called Adams�Bashforth after J� C� Adams� who invented them to solve a problem ofcapillary action in collaboration with F� Bashforth published in �����

Page 144: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Chapter �� Multistep Methods ���

where�

j � ����j��k��Xi�j��

�i

j � �� i

i � ����iZ �

��si

�ds �

This formula is a k�step method because it uses information at the k pointstn��� tn��� � � � � tn�k� It is sometimes also called a �k � ���value method� be�cause the total information per step� which determines storage requirements�involves also yn���The local truncation error turns out to be Cp��h

py�p����tn� � O�hp����where p � k� Note that there is only one function evaluation per step�

Example � � The �rst order AdamsBashforth method is the forward Eulermethod� The second order AdamsBashforth method is given by the aboveformula with k � � and � � �� � � ���� This yields

yn � yn�� � h

�fn�� �

�rfn��

or� equivalently�

yn � yn�� � h

��

�fn�� � �

�fn��

��

Table �� gives the coe cients of the Adams�Bashforth methods for k upto ��The Adams�Bashforth methods are explicit methods with very small re�

gions of absolute stability� This has inspired the implicit versions of Adamsmethods� also called Adams�Moulton�The k�step implicit Adams method is derived similarly to the explicit

method� The di�erence is that for this method� the interpolating polynomialis of degree � k and it interpolates f at the unknown value tn as well� seeFig� ��� This yields an implicit multistep method

yn � yn�� � h

kXj��

jfn�j �

�Recall �s

i

s�s � �� � � � �s � i � ��

i��

�s

� ��

Page 145: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

��� Chapter �� Initial Value Problems

p k j � � � � � � Cp��

� � j � ��

� � �j � �� ��

� � ��j �� ��� �

� � ��j �� �� �� �����

���j ���� ����� ���� ����� �� ��

� � ����j ���� ����� ���� ����� ���� ��� ��� ���� �

Table ��� Coe cients of Adams�Bashforth methods up to order �

t

y’(t)

f

)

n-k

fn-2fn-1

f(t n , yn

Figure ��� Adams�Moulton methods

The order of the k�step Adams�Moulton method is p � k�� �that p � k��follows immediately from the fact that k�� points are used in the underlyingpolynomial interpolation�� An exception is the case for k � � where fn�� isnot used� yielding p � k � �� A straightforward interpolation yields thecoe cients summarized in Table ���

Example � � Here are some examples of AdamsMoulton methods�

� k � � with � � � gives the backward Euler method�

� k � � with � �� � gives the implicit trapezoidal method�� k � � gives yn � yn�� � h

�� �fn � �fn�� � fn����

��

The Adams�Moulton methods have smaller error constants than the Adams�Bashforth methods of the same order� and use one step less for the same order�

Page 146: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Chapter �� Multistep Methods ���

p k j � � � � � � Cp��

� � j � ���

� � �j � � � ���

� � ��j � �� � ���

� � ��j � �� � � � �����

� ���j �� ��� ���� ��� ��� � ����

� ����j �� ���� ���� ��� ���� �� � ����� �

Table ��� Coe cients of Adams�Moulton methods up to order �

They have much larger stability regions than the Adams�Bashforth methods�But they are implicit� Adams�Moulton methods are often used together withAdams�Bashforth methods for the solution of nonsti� ODEs� This type ofimplementation is called predictor�corrector and will be described later� inx�����

����� Backward Dierentiation Formulae

The most popular multistep methods for sti� problems are the backwarddi�erentiation formulae �BDF�� Their distinguishing feature is that f�t� y�is evaluated only at the right end of the current step� �tn� yn�� A mo�tivation behind this is to obtain formulae with the sti� decay property�recall x���� Applying the general linear multistep method ���� to theODE y� � ��y � g�t�� and considering the limit hRe��� � ��� we havePk

j�� j�yn�j � g�tn�j�� � �� To obtain yn � g�tn� � � for an arbitraryfunction g�t� we must therefore set � �� � and j � �� j � �� This leavestreating y� in the di�erential equation y��t� � f�t� y�t��� In contrast to theAdams methods� which were derived by integrating the polynomial which in�terpolates past values of f � the BDF methods are derived by di�erentiatingthe polynomial which interpolates past values of y� and setting the derivativeat tn to f�tn� yn�� This yields the k�step BDF formula� which has order p � k�

kXi��

iriyn � hf�tn� yn��

This can be written in scaled form where �� � ��

kXi��

�iyn�i � h�f�tn� yn��

Page 147: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

��� Chapter �� Initial Value Problems

The BDF formulae are implicit and are usually implemented together witha modi�ed Newton method to solve the nonlinear system at each time step�The �rst � members of this family are listed in Table ��� The �rst� one�stepmethod� is again backward Euler�

p k � �� �� �� �� �� � ��

� � � � ��� � �

�� ��

���

� � ��� � ��

����� � �

��

� � ��� � ��

����

����

��

����� � ����

��������� ����

������� � ��

���

� � �����

� �������

�����

�������

�����

� �����

�����

Table ��� Coe cients of BDF methods up to order �

����� Initial Values for Multistep Methods

For one�step methods we set y� � y���� the given initial value� Nothing elseis needed to start up the iteration in time for n � �� �� � � � �

With a k�step method� in contrast� the method is applied for n � k� k ��� � � � � Thus� k initial values y�� y�� � � � � yk�� are needed to start it up� Theadditional initial values y�� � � � � yk�� must be O�hp� accurate for a methodof order p� if the full convergence order is to be realized �x������ If errorcontrol is used� these additional starting values must be accurate to a givenerror tolerance�

To obtain these additional initial values� an appropriate Runge�Kuttamethod can be used� Another approach� utilized in all modern multisteppackages� is to recursively use a �k � ���step method� As we have seen� lin�ear multistep methods tend to come in families� so a general�purpose codecan be written which implements the �rst methods of such a family� fork � �� �� � � � � p� say� Then the code can at a starting �or a restarting� pointt gradually and adaptively increase the method�s number of steps �and cor�respondingly its order��

Example � � We compute the solution of the simple Example ���

y� � �ty� � t� �

t�� y��� � ��

Page 148: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Chapter �� Multistep Methods ���

The exact solution is y�t� � ��t� We record results parallel to Example ����i�e� for the same constant step sizes� measuring absolute errors and convergence rates at t � �� Results for some AdamsBashforth� AdamsMoultonand BDF methods are displayed in Tables ���� ��� and ���� respectively� Theinitial values for the kstep method are obtained by the values of the �k� ��step method of the same family� The symbol � denotes an �in�nite� error�which occurs when the absolute stability restriction is strongly violated�

step h k � � error rate k � � error rate k � � error rate

��� ���e�� � ���� ��e�� ���� ���e�� ���� ���e�� ���� ���e�� ���� ���e��

���� ���e�� ���� ���e�� ���� ��e��� ����

���� ��e�� ���� ��e��� ���� ���e��� ����

���� ���e�� ���� ���e��� ���� ���e��� ���

����� ���e�� ���� ���e��� ���� ���e��� �����

Table ��� Example ��� Errors and calculated convergence rates for Adams�Bashforth methods�

In Table ��� we can observe the high accuracy that the higher order methods achieve for this very smooth problem� However� these small errors arewiped out by an explosion of the roundo� error if the step size is so large that���hy is not in the absolute stability region of the method� The region ofabsolute stability is seen to be shrinking as the order of the method increases�in contrast to the RungeKutta results of Table ����

In the �rst column k � � of Table ��� we see the results for the backwardEuler method� For h small they are very close to those of the forward Euler method �Table ���� but for the larger values of h they are much better�Newton�s method was used to obtain convergence of the nonlinear iterationfor h � ���� and functional iteration was used for the smaller step sizes�The column of p � � describes the performance of the �nd order trapezoidalmethod� For the �th order method the error reaches roundo� level alreadyfor h � ���� The BDF methods perform similarly to the AdamsMoultonmethods for this nonsti� problem� The order of the methods� before the onsetof roundo� error� is clearly re�ected in the results� The absolute value of theerrors is unusually small�

Page 149: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

��� Chapter �� Initial Value Problems

step h k � �� p � � error rate k � �� p � � error rate k � � p � � error rate

��� ���e� ��e�� ���e���

��� ��e�� ���� ���e�� ���� ���e��� ����

��� ���e�� ���� ���e�� ���� ���e��� ����

���� ���e�� ���� ��e��� ���� ��e�� ����

���� ��e�� ���� ���e��� ���� ���e��� �����

���� ���e�� ���� ���e��� ���� ���e��� ����

����� ���e�� ���� ��e��� ���� ���e��� �����

Table �� Example ��� Errors and calculated convergence rates for Adams�Moulton methods�

��� Order ��Stability and Convergence

As in the two previous chapters� the basic convergence theory requires thata method have a certain �positive� order of accuracy �i�e� consistency� andthat it be ��stable� The emphasis� though� is somewhat di�erent here fromwhat we had for Runge�Kutta methods� whereas there ��stability was trivialand attaining useful methods with a high order of accuracy was tricky� here��stability is not automatic �although it is not di cult to check�� whereasattaining high order is straightforward� provided only that we are preparedto use su ciently many past values and provide su ciently accurate initialvalues� Note also that the restriction to a constant step size� which is notneeded in x���� simpli�es life considerably in this section�

����� Order

The simple derivation below is incredibly general� it will give us a tool notonly for checking a method�s order but also for �nding its leading local trun�cation error term and even for designing linear multistep methods� givensome desired criteria�De�ne the linear operator Lhy�t� by

Lhy�t� �kX

j��

��jy�t� jh�� hjy��t� jh�� ����

where y�t� is an arbitrary continuously di�erentiable function on ��� b�� Thelocal truncation error is naturally de�ned as the defect obtained when plug�

Page 150: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Chapter �� Multistep Methods ��

step h k � � error rate k � � error rate k � � error rate

��� ���e�� ���e�� ���e���

��� ��e�� ���� ��e�� ���� ���e��� ����

��� ���e�� ���� ���e�� ���� ��e��� ����

���� ���e�� ���� ���e�� ���� ���e�� ����

���� ��e�� ���� ��e��� ���� ���e��� �����

���� ���e�� ���� ���e��� ���� ���e��� ����

����� ���e�� ���� ���e��� ���� ���e��� �����

Table ��� Example ��� Errors and calculated convergence rates for BDFmethods�

ging the exact solution into the di�erence equation �which here is ���� di�vided by h� see x����� This can be written as

dn � h��Lhy�tn� ���

where y�t� is the exact solution� In particular� the exact solution satis�esy� � f�t� y�t��� so

Lhy�t� �

kXj��

��jy�t� jh�� hjf�t� jh� y�t� jh����

If we now expand y�t � jh� and y��t � jh� in Taylor series about t andcollect terms� we have

Lhy�t� � C�y�t� � C�hy��t� � � � �� Cqh

qy�q��t� � � � �

where the Cq are computable constants� Recall that the order of the methodis p if dn � O�hp�� Thus�

� The order of the linear multistep method is p i�

C� � C� � � � � � Cp � �� Cp�� �� ��

� The local truncation error is given� as advertised in ����� by

dn � Cp��hpy�p����tn� �O�hp����

Page 151: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

��� Chapter �� Initial Value Problems

From the Taylor series expansions� it can be easily seen that the coe �cients are given by�

C� �kX

j��

�j

Ci � ����i��

i$

kXj��

ji�j ��

�i� ��$kX

j��

ji��j

�� i � �� �� � � � ����

To obtain a method of order p� therefore� the �rst p of these expressions mustbe set to �� The �rst few of these conditions read

� � �� � �� � �� � � � � � �k

� � ��� � ��� � � � �� k�k� � �� � � � � � � � �� k�

� ��

���� � ��� � � � � � k��k� � �� � �� � � � �� kk��

etc� When the order is p� Cp�� is called the error constant of the method�

Example � � For the forward Euler method� �� � ��� � � �� So�C� � �� � � �� C� � � � � � �� C� � ��

�� � �

��

For the �step AdamsBashforth method� �� � ��� � � ��� � � ��

�� So�

C� � �� � � �� C� � � � ����

�� �� C� � ��

���

�� � � ��

C� � �������

��

�� ���

���

Example � � The coe�cients of the methods of the previous section can beobtained by applying their family design criteria to select some method coe�cients and then using the order conditions to choose the remaining coe�cientssuch that the order is maximized�For instance� consider a �step BDF� � �� �� � � � � �� The method

isyn � ��yn�� � ��yn�� � h�fn�

The order conditions give the linear equations

� � �� � �� � �

�� � ��� � � � �

�� � ��� � ��

This system can be easily solved to yield � ���� �� � ��

� � �� ���� as per

Table ���� The coe�cient of the leading term of the local truncation error isC� � ��

���� �

�� ��

� � �

Page 152: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Chapter �� Multistep Methods ���

Given some of the ��s and �s we can obviously use these relations to �ndthe remaining ��s and �s for the method of maximal order �see Exercise ���note� though� that this method may not be optimal� or even usable� due tostability considerations��A linear multistep method is consistent if it has order p � �� Thus� the

method is consistent i�

kXj��

�j � ��

kXj��

j�j �

kXj��

j � ��

Sometimes it is more convenient to express the linear multistep methodin terms of the characteristic polynomials

���� �kX

j��

�j�k�j ���a�

���� �

kXj��

j�k�j � ���b�

In terms of these polynomials� the linear multistep method is consistent i����� � �� ����� � �����

Reader�s advice� The material that follows below is importantfor the fundamental understanding of linear multistep methods�We derive simple conditions on the roots of the characteristicpolynomial ���� which guarantee that a method is ��stable� This�together with consistency� then gives convergence� Recall that���� � � by consistency� so this determines one root� The bottomline of the following discussion is that a usable method must haveall other roots of the polynomial ���� strictly inside the unit cir�cle� A reader who is interested mainly in practical aspects maytherefore skip the next few pages until after Example ��� at leaston �rst reading�

����� Stability Dierence Equations and the Root Con�

dition

One way of looking at a linear multistepmethod is that it is a di�erence equa�tion which approximates the di�erential equation� The stability of the linear

Page 153: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

��� Chapter �� Initial Value Problems

multistep method� and the essential theoretical di�erence between multistepand one�step methods� are given by the stability of the di�erence equation�Before discussing stability for linear multistep methods� we review some

basic facts about linear di�erence equations with constant coe cients� Givensuch a scalar di�erence equation

akyn�k � ak��yn�k�� � � � �� a�yn � qn� n � k� k � �� � � �

if fvng is a particular solution for this equation then the general solution isyn � xn� vn� where xn is the general solution to the homogeneous di�erenceequation� �

akxn�k � ak��xn�k�� � � � �� a�xn � �� n � k� k � �� � � �

There are k linearly independent solutions to the homogeneous equation�To �nd them� we try the educated guess �ansatz� xn � �n� Substituting intothe homogeneous di�erence equation we have

���� �kX

j��

aj�k�j � �� ����

thus � must be a zero of the polynomial ����� If all k roots are distinct thenthe general solution is given by

yn �kXi��

ci�ni � vn�

where the ci� i � �� � � � � k� are arbitrary constants which are determined bythe k initial conditions required for the di�erence equation� If the roots arenot distinct� say �� � �� is a double root� then the solution is given by

yn � c��n� � c�n�

n� �

kXi��

ci�ni � vn�

For a triple root� we have �n� n�n� n�n � ���n as solution modes� etc� Thusthe solutions to the di�erence equation are intimately related to the roots ofthe characteristic polynomial which is associated with it�We can de�ne stability for this di�erence equation similarly to stability for

a di�erential equation �see xx�������� particularly Example ����� Clearly� fora perturbation in the ci not to grow unboundedly with n� we need to boundthe roots �i� We de�ne� in complete analogy to the constant coe cient ODEcase�

�For example� fvng can be the solution of the dierence equation with zero initialconditions� v v� � � � vk�� ��

Page 154: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Chapter �� Multistep Methods ���

� The di�erence equation is stable if all k roots of ���� satisfy j�ij � ��and if j�ij � � then �i is a simple root�

� The di�erence equation is asymptotically stable if all roots satisfy j�ij ���

This completes our review of di�erence equations�For multistep methods applied to the test equation y� � �y� the di�erence

equation is given by

kXj��

��j � h�j�yn�j � �� ����

This is a homogeneous� constant�coe cient di�erence equation� like what wehave just treated� with aj � �j�h�j� A solution to this di�erence equationis f�ni g if �i is a root of the polynomial ���� � ���� � h����� � �� Sincethe solution to the ODE �with y��� � �� is y � e�t � �eh��n� we expectone root to approximate eh� so that yn can approximate y�tn� �i�e�� this is aconsistency requirement�� That root is called the principal root� The otherroots are called extraneous roots�What should strike you in the above review is how closely the solution

procedure for the di�erence equation is related to that of a scalar di�erentialequation of order k� The source of these extraneous roots �also referred toat times as parasitic roots� is the discrepancy between the ODE of order �which should be approximated and the ODE of order k which is approximatedinstead by the multistep method� A good multistep method therefore mustensure that these extraneous roots� which cannot do any good� do not causeany harm either� This is what ��stability �and strong stability� to be de�nedbelow� are about�

����� ��Stability and Convergence

Recall that in the previous two chapters convergence followed from accuracyusing a perturbation bound� i�e� ��stability� Consider an ODE system� y� �f�t�y� on the interval ��� b�� The de�nition ����� of ��stability for one�stepmethods needs to be updated here to read that the linear multistep methodis ��stable if there are positive constants h� and K such that for any meshfunctions xh and zh with h � h��

jxl � zlj � Kfk��Xi��

jxi � zij� maxk�n�N

jh��kX

j��

�j�xn�j � zn�j�

�kX

j��

j�f�tn�j �xn�j�� f�tn�j � zn�j��jg� � � l � N� �����

Page 155: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

��� Chapter �� Initial Value Problems

If we have this bound then convergence follows immediately� In fact� byplugging xn � yn and zn � y�tn� in the stability bound ����� we obtainthat if the k initial values are accurate to order p and the method has orderp then the global error is O�hp��The ��stability bound is cumbersome to check for a given linear multistep

method� Fortunately� it turns out that it is equivalent to a simple conditionon the roots of the characteristic polynomial ���� of ���a�� The completeproof is technical and appears in classical texts� Instead� we bring its essence�As the name implies� ��stability is concerned with what happens in the

limit h � �� In this limit� it is su cient to consider the ODE y� � ��corresponding to the fact that y� is the dominant part of the di�erentialoperator y�� f�t�y��� Now� the ODE y� � � is decoupled� so we can considera scalar component y� � �� For the latter ODE� the method reads

�kyn�k � �k��yn�k�� � � � �� ��yn � ��

This is a di�erence equation of the type considered in the previous subsection�It must be stable for the multistep method to be ��stable� Identifying ����of ���� with ���� of ���a�� we obtain the following theorem�

Theorem � �

� The linear multistep method is �stable i� all roots �i of the characteristic polynomial ���� satisfy

j�ij � ��

and if j�ij � � then �i is a simple root� � � i � k�

� If the root condition is satis�ed� the method is accurate to order p� andthe initial values are accurate to order p� then the method is convergentto order p�

Note that the root condition guaranteeing ��stability relates to the charac�teristic polynomial ���� alone� see Fig� ��� Also� for any consistent methodthe polynomial � has the root �� One�step methods have no other roots�which again highlights the fact that they are automatically ��stable�

Example � Instability is a disaster� Here is an example of an unstablemethod�

yn � ��yn�� � yn�� � �hfn�� � �hfn�� �In terms of the local truncation error� this is the most accurate explicit �stepmethod� However� ���� � �� � �� � � �� � ���� � �� The extraneous rootis �� � � and the root condition is violated�

Page 156: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Chapter �� Multistep Methods ���

-1 1

Figure ��� Zeros of ���� for a ��stable method�

Consider solving y� � � with initial values y� � �� y� � �� Then

y� � ��y� � ���y� � ��y� � y� � ���y� � ��y� � y� � ��������

There is no hope for convergence here� �

Consider again the test equation� and its discretization ����� IfRe��� � �then the exact solution decays and we must prevent any growth in the ap�proximate solution� This is not possible for all such � if there are extraneousroots of the polynomial � with magnitude �� For h � � su ciently small thedi�erence equation ���� must be asymptotically stable in this case� see Fig���� We de�ne a linear multistep method to be

� strongly stable if all roots of ���� � � are inside the unit circle exceptfor the root � � ��

� weakly stable if it is ��stable but not strongly stable�

Example � � Weak stability can be a disaster for some problems� too� Consider Milne�s method�

yn � yn�� ��

�h�fn � �fn�� � fn���

Page 157: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

��� Chapter �� Initial Value Problems

-1 1

Figure ��� Zeros of ���� for a strongly stable method� It is possible to drawa circle contained in the unit circle about each extraneous root�

for y� � �y� The error satis�es the equation

en � en�� ��

�h��en � �en�� � en����

Substituting as before en � �n� we have��� �

�h�

��� � �

�h�� �

�� �

�h�

�� �

������ h����� � ��� Clearly� � � �� � � has a root at ��� and a root at ���The roots of the full polynomial equation are given by

� �

��h� �

q� � �

��h���

��� ��h��

By expanding � into a power series in h�� we �nd that

�� � eh� �O�h��

�� � �e��h�� � �O�h���

For � � �� the extraneous root dominates� so the solution is unstable� �

A practically minded reader must conclude that any useful linear multi�step method must be strongly stable� We shall not be interested henceforthin any other methods� But this restricts the attainable order of accuracy� G�Dahlquist showed that

Page 158: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Chapter �� Multistep Methods ���

� Strongly stable k�step methods can have at most order k � ��

Example � The Adams methods� both explicit and implicit� have the characteristic polynomial

���� � �k � �k�� � �� � ���k��

so the extraneous roots are all �� for any k� These methods are all stronglystable� The implicit methods have the highest order attainable�This explains in part the popularity of Adams methods� �

Example � � The BDF methods were motivated in x��� by the desire toachieve sti� decay� This� however� does not automatically mean that theyare strongly stable� Exercise ��� shows that BDF methods are �stable for� � k � � and unstable for k � �� Thus� only the �rst � members of thisfamily are usable� �

��� Absolute Stability

Recall that the general linear multistep method

kXj��

�jyn�j � h

kXj��

jfn�j

applied to the test equation y� � �y gives ����� i�e��

kXj��

�jyn�j � h�

kXj��

jyn�j �

If we let yn � �n� then � must satisfy

kXj��

�j�k�j � h�

kXj��

j�k�j � �����

or ���� � h������

Now� the method is absolutely stable for those values of z � h� such thatjynj does not grow with n� This corresponds to values for which all roots of����� satisfy j�j � ��

Page 159: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

��� Chapter �� Initial Value Problems

For di�erential equations with positive eigenvalues� it is sometimes con�venient to de�ne an alternate concept� the region of relative stability� Thisis a region where the extraneous roots may be growing� but they are growingmore slowly than the principal root� so that the principal root still dominates�We will not pursue this further�Finding the region of absolute stability is simple for linear multistepmeth�

ods� Just look for the boundary

z ���e���

��e���

and plot �the complex scalar� z for � ranging from � to ��In Fig� � we plot absolute stability regions for the Adams methods� The

�rst two Adams�Moulton methods are missing because they are A�stable�Notice how much larger the stability regions for the Adams�Moulton meth�ods are compared to the Adams�Bashforth methods for the same order �orfor the same number of steps�� interpolation is a more stable process thanextrapolation�

−4 −3.5 −3 −2.5 −2 −1.5 −1 −0.5 0 0.5 1−2

−1.5

−1

−0.5

0

0.5

1

1.5

2

Re(z)

Im(z

)

Stability regions in the complex z−plane

k=1k=2

k=3

k=4

�a� Adams�Bashforth k �� �� �� �

−9 −8 −7 −6 −5 −4 −3 −2 −1 0 1−4

−3

−2

−1

0

1

2

3

4

Re(z)

Im(z

)

Stability regions in the complex z−plane

k=2

k=3k=4

�b� Adams�Moulton k �� �� �

Figure �� Absolute stability regions of Adams methods

Recall the de�nition of A�stability� a numerical method is Astable if itsregion of absolute stability contains the left half plane hRe��� � �� Unfor�tunately� A�stability is very di cult to attain for multistep methods� It canbe shown that�

Page 160: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Chapter �� Multistep Methods ��

� An explicit linear multistep method cannot be A�stable�� The order of an A�stable linear multistep method cannot exceed two�� The second order A�stable implicit linear multistep method with small�est error constant �C� �

���� is the trapezoidal method�

The utility of the trapezoidal method has already been discussed in x���� �

If we want to use linear multistep methods for sti� problems� the A�stability requirement must be relaxed� Moreover� the discussion in Chapter� already reveals that in the very sti� limit hRe��� � ��� the A�stabilitybound may not be su cient and the concept of sti� decay is more useful� TheBDF methods introduced in x�� trade in chunks of absolute stability regionsnear the imaginary axis for sti� decay� The size of these chunks increaseswith the number of steps k� until the methods become unstable for k � ��see Fig� ���

−1 0 1 2 3 4 5 6 7 8 9−4

−3

−2

−1

0

1

2

3

4

Re(z)

Im(z

)

Stability regions in the complex z−plane

stability OUTSIDEshaded areas

k=3

k=2

k=1

�a� BDF k �� �� �

−10 −5 0 5 10 15 20 25 30 35

−15

−10

−5

0

5

10

15

Re(z)

Im(z

)

Stability regions in the complex z−plane

stability OUTSIDEshaded areas

k=6

k=5

k=4

�b� BDF k �� �� �

Figure ��� BDF absolute stability regions� The stability regions are outsidethe shaded area for each method�

�These results were given by Dahlquist in the � ���s� They had a major impact onresearch in this area in the � ���s and � ���s� Today it is still easy to appreciate theirmathematical beauty� and the sophistication that went into the proofs� even though aglance at the methods used in successful implementations makes it clear that A�stabilityis not the property that separates the winners from the also�rans�

Page 161: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

��� Chapter �� Initial Value Problems

��� Implementation of Implicit Linear Multi�

step Methods

When using an implicit� k�step linear multistep method� i�e� � �� � in theformula

kXj��

�jyn�j � h

kXj��

jfn�j �

a system of m nonlinear equations for yn has to be solved at each step �recallx����� We can solve this system by some variant of functional iteration �fornonsti� systems�� or by a modi�ed Newton iteration �for sti� systems�� Forany of these iterative methods we must �guess�� or predict a starting iteratey�n� usually by evaluating an interpolant passing through past values of yand"or f at tn� or via an explicit multistep method�

����� Functional Iteration

The simplest way to solve the nonlinear algebraic system for yn is via func�tional iteration� The iteration is given by

y���n � h�f�tn�y

�n��

kXj��

�jyn�j � h

kXj��

jfn�j � s � �� �� � � � �����

This is a �xed point iteration� It converges to the �xed point yn if it isa contraction� i�e�� if kh� �f�yk � r � �� Hence it is appropriate only fornonsti� problems� The iteration is continued until it has been determinedto have converged as described for the Newton iteration below� Usually� ifconvergence is not attained within two to three iterations� or if the rate ofconvergence is found to be too slow� the current step is rejected and retriedwith a smaller step size �for example� halve the step size��

����� Predictor�Corrector Methods

Often in nonsti� codes� the iteration is not taken to convergence� Instead� a�xed number of iterations is used for each time step� First� an approximationy�n to yn is predicted� usually by an explicit multistep method of the sameorder as the implicit method �for example� by the k�step Adams�Bashforthmethod of order k��

P � y�n � ���yn�� � � � �� ��kyn�k � h� ��fn�� � � � �� �kfn�k��

�Do not confuse the notation y���n for the �� � ��st iterate of yn with the notation forthe �� � ��st power� Which of these is the correct interpretation should be clear from thecontext� We have reserved the superscript � for an iteration counter in this chapter�

Page 162: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Chapter �� Multistep Methods ���

Then the function is evaluated at y�n�

E � f�n � f�tn�y�n�

and inserted into the corrector formula �for example� Adams�Moulton oforder k or k � �� to obtain a new approximation to yn� Setting � � ��

C � y���n � ��yn�� � � � �� �kyn�k � h��f

�n � �fn�� � � � �� kfn�k��

The procedure can be stopped here �this is called a PEC method�� or thefunction can be evaluated at y�

n to give

E � f�n � f�tn�y�n�

�this is called a PECE method�� or the steps E and C can be iterated �times to form a P�EC�� method or a P�EC��E method� The �nal functionevaluation in a P�EC��E method yields a better value for f to be used inthe next time step �i�e� n � n � �� as the new fn��� Although it appearsthat the method might be expensive� the �nal function evaluation is usuallyadvantageous because it yields a signi�cant increase in the region of absolutestability over the corresponding P�EC�� method�It should be noted that because the corrector formula is not iterated

to convergence� the order� error and stability properties of the P�EC��E orP�EC�� methods are not necessarily the same as for the corrector formulaalone� The methods of this subsection are di�erent� in principle� from themethods of the previous subsection x���� for the same implicit formula�Predictor�corrector methods are explicit methods� which are members of aclass of methods called general linear methods� This class contains also linearmultistep methods�

Example � �� Combining the �step AdamsBashforth method �i�e� ��� ���� ��� � �� �� � ���� �� � ���� with the �nd order �step AdamsMoultonmethod �i�e� the trapezoidal method� �� � ��� � � � � ���� we obtain thefollowing method�Given yn��� fn��� fn���

�� y�n � yn�� � h

� ��fn�� � fn���

�� f�n � f�tn�y�n�

�� yn � yn�� � h� �fn�� � f�n�

�Unlike an implicit method� an explicit method evaluates the next yn at each stepprecisely �in the absence of roundo error� in a �nite number of elementary operations�This relates to the fact that no such predictor�corrector formula has an unbounded absolutestability region� even if the implicit corrector formula has one� These predictor�correctormethods are suitable only for nonsti problems�

Page 163: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

��� Chapter �� Initial Value Problems

�� fn � f�tn�yn��

This is an explicit� �nd order method which has the local truncation error

dn � � ���h�y����tn� �O�h���

The most�used variant of predictor�corrector methods is PECE� In thecommon situation where the order of the predictor formula is equal to theorder of the corrector formula� the principal term of the local truncation errorfor the PECE method is the same as that of the corrector�

dn � Cp��hpy�p����tn� �O�hp����

The local error is given by a similar expression �see �������� Roughly speak�ing� the principal terms of the error are the same for the corrector as forthe PECE method because y�

n� which is already accurate to the order of thecorrector� enters into the corrector formula multiplied by h� hence the errorwhich is contributed by this term is O�hp����

����� Modi�ed Newton Iteration

For sti� systems� a variant of Newton�s method is used to solve the nonlin�ear algebraic equations at each time step� For the general linear multistepmethod we write

yn � h�f�tn�yn� � �kX

j��

�jyn�j � h

kXj��

jfn�j

where the right hand side is known� Newton�s iteration yields

y���n � y�n �

�I � h�

�f

�y

��� � kXj��

�jyn�j � h

kXj��

jfn�j

where yn� fn and �f��y are all evaluated at y�n� The initial guess y�n is usually

obtained by evaluating an interpolant passing through past values of y at tn�For a simple implementation� this method does the job� However� it is oftennot the cheapest possible�A modi�ed Newton method is usually employed in sti� ODE packages�

where the Jacobian matrix �f��y and its LU decomposition are evaluated�updated� only when deemed necessary� The matrix may be evaluated when�ever

�� the iteration fails to converge� or

Page 164: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Chapter �� Multistep Methods ���

�� the step size has changed by a signi�cant amount or the order haschanged� or

�� after a certain number of steps have passed�

Since forming and LU�decomposing the matrix in Newton�s iteration areoften the major computational expense in carrying out the next step�s ap�proximation� relatively large savings are realized by the modi�ed Newton�smethod�

The iteration is considered to have converged� e�g�� when

�� �jy���

n � y�nj � NTOL

where the Newton iteration tolerance NTOL is usually taken to be a fractionof ETOL� the user error tolerance� say NTOL � ���ETOL� and � is anindication of the rate of convergence of the iteration� which can be estimatedby

� �

� jy���n � y�njjy�

n � y�nj� �

Reader�s advice� The material that follows deals with some ofthe nuts and bolts for writing general�purpose software based onmultistep methods� Depending on your orientation� you may wishto read it with special care� or to skip it�

��� DesigningMultistep General�Purpose Soft�

ware

The design of an e�ective general�purpose code for solving initial�value prob�lems using multistep methods is a challenging task� It involves decisionsregarding error estimation and control� varying the step size� varying themethod�s order and solving the nonlinear algebraic equations� The latter hasbeen considered already� Here we outline some of the important options forthe resolution of the remaining issues�

Page 165: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

�� Chapter �� Initial Value Problems

����� Variable Step�Size Formulae

We have seen in the previous chapters that in some applications varying thestep size is crucial for the e�ective performance of a discretization method�The general k�step linear methods that we have seen so far

kXj��

�jyn�j � h

kXj��

jfn�j

assume that we know the past values �yn�j � fn�j�� j � �� � � � � k� at a sequenceof equally spaced mesh points de�ned by the step length h� Now� if att � tn�� we want to take a step of size hn which is di�erent from the stepsize hn�� used before� then we need solution values at past times tn��� jhn�� � j � k � �� whereas what we have from previous steps are values attn�� � jhn��� � � j � k � �� To obtain approximations for the missingvalues� there are three main options� We will illustrate them in terms of thesecond order BDF method� Note that for Adams methods the interpolationswill be of past values of f instead of past values of y� and errors are estimatedvia di�erences of f and not y�

Fixed�Coe�cient Strategy

The constant step�size� second order BDF formula �for step size hn� is givenby

�yn � �

�yn�� �

�yn��

�� hnf�tn�yn��

where tn � tn�� � hn� The BDF formula requires values of y at tn�� andtn���hn� The �xed�coe cient method computes these values from the valuesat tn��� tn�� � hn�� and tn�� � hn�� � hn�� by quadratic �more generally�polynomial� interpolation� The interpolated values of y at tn���hn becomethe *new� past values yn��� and are used in the �xed�coe cient BDF formulato advance the step�Fixed�coe cient formulae have the advantage of simplicity� However�

there is an error due to the interpolation� and they are less stable thanvariable�coe cient formulae� Stability of the variable step�size formulae isan important consideration for problems where the step size must be changedfrequently or drastically �i�e�� hn � hn�� or hn hn����

Variable�Coe�cient Strategy

Better stability properties are obtained by deriving directly the formulaewhich are based on unequally�spaced data� Recall that the BDF formulaewere derived by �rst approximating y by an interpolating polynomial� andthen di�erentiating the interpolating polynomial and requiring it to satisfy

Page 166: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Chapter �� Multistep Methods ��

the ODE at tn� The variable�coe cient BDF formulae are derived in ex�actly the same way� using an interpolating polynomial which is based onunequally�spaced data� The Adams methods can also be directly extended�using polynomial interpolation of unequally�spaced f �values�For example� to derive the variable�coe cient form of the second order

BDF method� we can �rst construct the interpolating quadratic polynomial��t� based on unequally�spaced data �here it is written in Newton form��

��t� � yn � �t� tn��yn�yn��� � �t� tn��t� tn����yn�yn���yn����

Next we di�erentiate the interpolating polynomial to obtain�

���tn� � �yn�yn��� � �tn � tn����yn�yn���yn����

Then the variable�coe cient form of the second�order BDF formula is givenby

f�tn�yn� � �yn�yn��� � hn�yn�yn���yn����

Note that on an equally spaced mesh� this formula reduces to the �xed step�size BDF method� The coe cients in this method depend on hn and onhn���The variable�coe cient method has the advantage for problems which

require frequent or drastic changes of step size� However� in the case ofimplicit methods� it can be less e cient than the alternatives� To see this�rewrite the formula in terms of past steps

hnf�tn�yn� � yn � yn�� �h�n

hn � hn��

�yn � yn��

hn� yn�� � yn��

hn��

��

Then the iteration matrix for Newton�s method is given by��� �

hnhn � hn��

�I � hn

�f

�y

��

So the coe cients of the iteration matrix depend not only on the currentstep size� but also on the previous one� and more generally on the sequenceof k � � past steps� For economy� it is advantageous to try to save andreuse the iteration matrix and"or its factorization from one step to the next�However� if the coe cients in the matrix change frequently� then this is notpossible�

�Recall that the divided dierences are de�ned by

�yn� yn

�yn�yn��� � � � �yn�i� �yn�yn��� � � � �yn�i���� �yn���yn��� � � � �yn�i�

tn � tn�i�

Page 167: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

�� Chapter �� Initial Value Problems

This changing Jacobian is a serious shortcoming of the variable�coe cientstrategy� in the case of implicit methods for sti� problems� In the design ofcodes for nonsti� problems� for example in Adams codes� the Jacobian matrixdoes not arise and there is no need to consider the next alternative�

Fixed Leading�Coe�cient Strategy

This is a compromise which incorporates the best features of both previousmethods� We describe it for the k�step BDF� First a polynomial ��t� ofdegree � k� which is sometimes called a predictor polynomial� is constructedsuch that it interpolates yn�i at the last k�� values on the unequally�spacedmesh

��tn�i� � yn�i� i � �� � � � � k � ��

Then the �xed leading�coe cient form of the k�step BDF formula is givenby requiring that a second polynomial ��t� of degree � k� which interpolatesthe predictor polynomial on a �xed mesh tn��� tn�� � hn� � � � � tn�� � khn�satis�es the ODE at tn�

��tn � ihn� � ��tn � ihn�� � � i � k

���tn� � f�tn���tn���

and setting

yn � ��tn��

The �xed leading�coe cient form has stability properties which are interme�diate between the other two forms� but is as e cient as the �xed�coe cientform�Whichever method is chosen to vary the step size� it is clear that the

e�ort is more signi�cant than what is required for Runge�Kutta methods�On the other hand� estimating the local truncation error is easier with linearmultistep methods� as we will see next�

����� Estimating and Controlling the Local Error

As was the case for Runge�Kutta methods� the errors made at each step aremuch easier to estimate than the global error� Thus� even though the globalerror is more meaningful� the local truncation error is the one that general�purpose multistep codes usually estimate in order to control the step size andto decide on the order of the method to be used� We recall from ������ thatthe local truncation error is related to the local error by

hn�jdnj�O�hp���� � jlnj�� �O�hn���

Page 168: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Chapter �� Multistep Methods ��

Thus� to control the local error� multistep codes attempt to estimate andcontrol hndn�In developing estimates below using the local truncation error� we will

pretend that there is no error in previous steps� This is of course not truein general� but it turns out that the errors in previous time steps are oftencorrelated so as to create a higher order contribution� so the expressionsderived by ignoring these past errors do yield the leading term of the currentlocal error� There are more di culties with the theory when the order isvaried�

Estimating the Local Truncation Error

In the case of predictor�corrector methods �x������ the error estimate can beexpressed in terms of the di�erence between the predictor and the corrector�Let the local truncation error of the predictor formula be given by

�dn � �Cp��hpy�p����tn� �O�hp����

Subtracting the predicted from the corrected values� we obtain

yn � y�n � �Cp�� � �Cp���h

py�p����tn� �O�hp����

Hence an estimate for the local truncation error of the corrector formula orof the PECE formula is given in terms of the predictor�corrector di�erenceby

Cp��hpy�p����tn� �O�hp��� �

Cp��

Cp�� � �Cp��

�yn � y�n��

This is called Milne�s estimate� In an Adams predictor�corrector pair a k�step Adams�Bashforth predictor is used together with a �k� ���step Adams�Moulton corrector to obtain a PECE method of order p � k with a localerror estimate� at the cost of two function evaluations per step� See Example����Alternatively� it is also possible to use a predictor of order p� �� This is

an instance of local extrapolation� as de�ned in the previous chapter�The local truncation error for more general multistep methods can be

estimated directly by approximating y�p��� using divided di�erences�� Forexample� for second order BDF� if ��t� is the quadratic interpolating yn�yn�� and yn�� then

f�tn�yn� � ���tn� � �yn�yn��� � hn�yn�yn���yn��� � rn�

�Note that in contrast to the methods used in the Runge�Kutta context to evaluatetwo approximations to y�tn�� here the predictor and the corrector methods have the sameorder�

In the case of Adams methods� y�p��� is approximated via the divided dierence of f �using y�p��� f �p��

Page 169: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

�� Chapter �� Initial Value Problems

wherern � hn�hn � hn����yn�yn���yn���yn����

The principal term of the local truncation error is then given by �rn�The error estimate is used to decide whether to accept the results of the

current step or to redo the step with a smaller step size� The step is acceptedbased on a test

EST � ETOL�where EST is hn times the estimated local truncation error�

Choosing the Step Size and Order for the Next Step

Once the current step has been accepted� the next task is to choose the stepsize and order for the next step� We begin by forming estimates of the errorwhich we expect would be incurred on the next step� if it were taken with amethod of order �p� for several possible orders� for example p��� p��� p andp � �� where p is the current order�There are several philosophies for choosing the next order�

�� Choose the next order so that the step size at that order is the largestpossible� We will show how to compute these step sizes�

�� Raise or lower the order depending on whether

jhp��y�p���j� jhpy�p�j� jhp��y�p���j� jhp��y�p���jform an increasing or decreasing sequence� where h is the current stepsize� The philosophy behind this type of order selection strategy is thatthe Taylor series expansion is behaving as expected for higher ordersonly if the magnitudes of successive higher order terms form a decreas�ing sequence� If the terms fail to form a monotone decreasing sequence�the order is lowered� The e�ect is to bias the formulae towards lowerorders� especially in situations where the higher order formulae are un�stable �thus causing the higher order di�erences to grow��

Given the order �p� the step size for the next step is computed as follows�Because the error for the next step is a highly nonlinear function of the stepsize to be chosen� a simplifying assumption is made� The step size expectedfor a step of order �p is computed as if the last �p � � steps were taken atthe current step size� and the step size is chosen so that the error estimatesatis�es the tolerance� More precisely� the new step size hn�� � rhn ischosen conservatively so that the estimated error is a fraction of the desiredintegration error tolerance ETOL�

jr�p��h�p��n C�p��y

��p���j � fracETOL�

Page 170: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Chapter �� Multistep Methods �

with frac � ���� say� If �EST � jh�p��n C�p��y

��p���j is the error estimate� then

r�p�� �EST � fracETOL�

Thus

r �

�fracETOL

�EST

� �

�p��

����� Approximating the Solution at O�Step Points

In many applications� the approximate solution is needed at intermediatetimes which may not coincide with the mesh points chosen by the code�Generally� it is easy and cheap to construct polynomial interpolants basedon solution values at mesh points� Then� we just evaluate the interpolant atthe o��step points� However� we note that the natural interpolant for BDFis continuous but not di�erentiable� and the natural interpolant for Adamsmethods is not even continuous �although its derivative is�$ Although thenatural interpolants yield the requested accuracy� for applications where moresmoothness is required of the numerical solution� interpolants which matchthe solution with greater continuity have been derived�

��� Software Notes and References

�� �� Notes

The Adams�Bashforth methods date back to ����� J�C� Adams also designedthe implicit formulas known as Adams�Moulton� Both F�R� Moulton andW�E� Milne used these formulae in ���� in predictor�corrector combinations�The BDF methods were introduced in the ����s� if not earlier� but theycame to prominence only much later� due to the work of C�W� Gear ����� See��� for more background and early references�The material in xx����� is standard� although di�erent ways have been

used to prove the basic Stability Theorem ��� It is covered �plus more� in anumber of other texts� e�g� ��� �� ��� ��� ��� Early works of G� Dahlquistand others� reported in ���� laid the foundations of this material�For our presentation we chose a di�erent order than the other texts by

combining the nonsti� and the sti� cases� This re�ects our belief that sti�equations should not be considered advanced material to be taught onlytowards the end of a course� Also� as in Chapter �� we have omitted manystability concepts that have been proposed in the literature in the past ��

Page 171: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

�� Chapter �� Initial Value Problems

years� and have instead concentrated only on the properties of sti� decay andA�stability�

Writing a general�purpose code based on multistep methods is a morecomplicated endeavor than for one�step methods� as x� may already sug�gest� The books by Shampine Gordon ���� and Brenan� Campbell Petzold ���� describe such implementations in detail�

While most recent developments in the numerical ODE area seem to haverelated more to Runge�Kutta methods� you should not conclude that linearmultistep methods may be forgotten� In fact� there are still some seriousholes in the theory behind the practical implementation issues in x� on onehand� and on the other hand these methods are winners for certain �but notall� applications� both sti� and nonsti�� Software exists as well� Note alsothat a number of the features and additional topics described in the previouschapter� including for example global error estimation� dense output andwaveform relaxation� are equally relevant here�

�� �� Software

A variety of excellent and widely used software based on linear multistepmethods is readily available� A few of the codes are described here�

� ode� written by Shampine and described in detail in ����� is basedon variable�coe cient Adams PECE methods� It is useful for nonsti�problems� and has a feature to diagnose sti�ness�

� vode� written by Hindmarsh� Brown and Byrne ����� o�ers �xed leading�coe cient Adams and BDF methods� The implicit formulae are solvedvia functional iteration or modi�ed Newton� depending on the optionselected� Thus� this code has options to deal with both sti� and nonsti�problems�

� difsub� written by Gear ����� solves sti� problems and was a veryin�uential code popularizing the BDF methods�

� vodpk is an extension of vode for large�scale sti� systems� In additionto the direct methods for solving linear systems used in vode� vodpko�ers the option of preconditioned Krylov iterative methods �see� e�g������ ���� the user must write a routine which gives the preconditioner�and this in some applications is a major task��

� dassl and daspk ���� are based on �xed leading�coe cient BDF for�mulae and can accommodate di�erential�algebraic equations as well assti� ODEs �see Chapter ����

Page 172: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Chapter �� Multistep Methods ��

��� Exercises

�� �a� Construct a consistent� unstable multistepmethod of order � �otherthan the one in Example ����

�b� Is it possible to construct a consistent� unstable one�step methodof order �! Why!

�� For the numerical solution of the problem

y� � ��y � sin t� � cos t� y��� � �� � � t � �whose exact solution is y�t� � e�t � sin t� consider using the followingfour ��step methods� with y� � � and y� � y�h� �i�e� using the exactsolution so as not to worry here about y���

�a� Your unstable method from the previous question

�b� The midpoint ��step method

yn � yn�� � �hfn��

�c� Adams�Bashforth

yn � yn�� �h

���fn�� � fn���

�d� BDF

yn ���yn�� � yn���

���h

�fn�

Consider using h � ��� for � � ��� � � ��� and � � ���� Discussthe expected quality of the obtained solutions in these �� calculations�Try to do this without calculating any of these solutions� Then con�rmyour predictions by doing the calculations�

�� Write a program which� given k and the values of some of the coe �cients ��� ��� � � � � �k� �� �� � � � � k of a linear k�step method� will

� �nd the rest of the coe cients� i�e�� determine the method� suchthat the order of the method is maximized�

� �nd the error coe cient Cp�� of the leading local truncation errorterm�

Test your program to verify the �nd and the last rows in each of theTables �� and ���

Now use your program to �nd Cp�� for each of the six BDF methodsin Table ���

Page 173: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

�� Chapter �� Initial Value Problems

�� Write a program which� given a linear multistep method� will testwhether the method is

� ��stable� strongly stable

�Hint� This is a very easy task using Matlab� for example��

Use your program to show that the �rst � BDF methods are stronglystable� but the ��step and ��step BDF methods are unstable� �For thisyou may want to combine your program with the one from the previousexercise��

� The famous Lorenz equations provide a simple example of a chaoticsystem �see� e�g�� ���� ����� They are given by

y� � f�y� �

�BBB�

��y� � y��

ry� � y� � y�y�

y�y� � by�

�CCCA

where �� r� b are positive parameters� Following Lorenz we set � ���� b � ���� r � �� and integrate starting from y��� � ��� �� ��T �Plotting y� vs� y� we obtain the famous �butter�y� depicted in Fig����

�a� Using a software package of your choice� integrate these equationsfor � � t � ��� with an error tolerance ��e� �� and plot y� vs� y��as well as y� as a function of t� What do you observe!

�b� Plot the resulting trajectory in the three dimensional phase space�i�e� the three y�coordinates� if in Matlab� type �help plot����Observe the strange attractor that the trajectory appears to settleinto�

�c� Chaotic solutions are famous for their highly sensitive dependenceon initial data� This leads to unpredictability of the solution �andthe physical phenomena it represents�� When solving numericallywe also expect large errors to result from the numerical discretiza�tion� Recompute your trajectory with the same initial data usingthe same package� changing only the error tolerance to ��e � ��Compare the values of y����� for the two computations� as wellas the plots in phase plane� Discuss�

�� �a� Show that the only k�step method of order k which has the sti�decay property is the k�step BDF method�

Page 174: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Chapter �� Multistep Methods ��

−20 −15 −10 −5 0 5 10 15 200

5

10

15

20

25

30

35

40

45

50

y_1

y_3

Figure ��� Lorenz �butter�y� in the y� � y� plane�

�b� Is it possible to design a strongly stable linear multistep methodof order � which has sti� decay!

�� Explain why it is not a good idea to use an Adams�Bashforth methodto predict the �rst iterate y�

n to start a Newton method for a BDF stepwhen solving a sti� problem�

�� Given an ODE system y� � f�t�y�� y��� � y�� we can calculatey���� � f���y��� The initial derivatives are used in modern BDF codesto estimate the error reliably�

Consider the opposite problem� given y��T � at some t � T � �nd y�T �satisfying the ODE� For example� �nding the ODE solution at a steadystate corresponds to specifying y��T � � ��

�a� What is the condition necessary to �nd y�T �� given y��T �! Howwould you go about �nding y�T � in practice! �Note also the pos�sibility for multiple solutions� in which case we want the conditionfor �nding an isolated solution��

�b� Suppose that the condition for solvability that you have just spec�i�ed does not hold� but it is known that the solution of the IVPsatis�es a set of nonlinear equations at each t�

� � h�t�y��

Page 175: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

��� Chapter �� Initial Value Problems

How would you modify the solvability condition! How would youimplement it! �Hint� Exercise �� provides an example��

�� The following ODE system due to H� Robertson models a chemicalreaction system and has been used extensively as a test problem forsti� solvers ��� ���

y�� � ��y� � y�y�

y�� � �y� � y�y� � y��y�� � y�� �

Here � � ����� � ��e��� and � ��e�� are slow� fast and very fastreaction rates� The starting point is y��� � ��� �� ��T �

�a� It is known that this system reaches a steady state� i�e�� wherey� � �� Show that

P�i�� yi�t� � �� � � t � b� then �nd the steady

state�

�b� Integrate the system using a nonsti� code with a permissive errortolerance �say ��e � �� for the interval length b � �� just to seehow inadequate a nonsti� solver can be� How far is y�b� from thesteady state!

�c� The steady state is reached very slowly for this problem� Use asti� solver to integrate the problem for b � ��e � � and plot thesolution on a semilog scale in t� How far is y�b� from the steadystate!

��� Consider the following ��step method �����

yn � yn�� �h

����fn � �fn�� � fn���� �����

Investigate the properties of this method in comparison with the ��stepand the ��step Adams�Moulton formulae� Does this method have anyadvantage!

�Hint� when hRe������ one must consider the roots of the charac�teristic polynomial ���� of ���b� ��

��� The border of the absolute stability region is the curve in the �h�planewhere jynj � j�nj � j�n��j� for � satisfying ���� � h������ Occasionallyit is interesting to plot the region where the approximate solution fory� � �y is actually dampened by a factor � � �� i�e� j�j � � �recallExercise �����

�a� Show that the boundary of this ��region is given by

h� ����e���

���e����

Page 176: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Chapter �� Multistep Methods ���

�b� Plot the ��curves with � � ��� �� �� for the backward Euler� trape�zoidal and ����� methods� Discuss your observations�

��� Often in practice one has to solve an ODE system of the form

y� � f�t�y� � g�t�y�� t � � �����

where f and g have signi�cantly di�erent characteristics� For instance�f may be nonlinear and the ODE z� � f�t� z� is nonsti�� while g islinear but the ODE z� � g�t� z� is sti�� This suggests mixing an explicitmethod for f with an implicit method� suitable for sti� problems� forg� An implicitexplicit �IMEX ���� k�step method has the form

kXj��

�jyn�j � h

kXj��

jfn�j � h

kXj��

jgn�j � ����

The combination of ��step Adams�Bashforth for f and trapezoidal rulefor g is common in the PDE literature �especially in combination withspectral methods��

Show that�

�a� The method ���� has order p if

kXj��

�j � � �����

i$

kXj��

ji�j � � �

�i� ��$kX

j��

ji��j � � �

�i� ��$kX

j��

ji�� j

for i � �� �� � � � � p� and such a condition does not hold for i � p���

�b� The �p � � constraints ����� are linearly independent� providedthat p � k� thus� there exist k�step IMEX methods of order k�

�c� A k�step IMEX method cannot have order greater that k�

�d� The family of k�step IMEX methods of order k has k parameters�

��� A convectiondi�usion partial di�erential equation in one space variablehas the form �recall Examples ��� and ����

�u

�t� u

�u

�x�

�x�p�x�

�u

�x�� � � x � �� t � �

where p � p�x� � � is a given function which may be small in magnitude�in which case the equation is said to be convection�dominated��

Page 177: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

��� Chapter �� Initial Value Problems

We now apply the method of lines �cf� xx���� ����� Discretizing in x ona mesh � � x� � x� � � � � � xJ � �� �xi � xi � xi��� �x � maxi�xi�let yi�t� be the approximation along the line of u�xi� t�� and obtain theODE system of the form ������

y�i � yi

�yi�� � yi���xi ��xi��

��

�xi ��xi��

�pi����

�xi���yi�� � yi�� pi����

�xi�yi � yi���

�� fi�y� � gi�y�� i � �� � � � � J � �� �����

Here� pi���� � ���p�xi� � p�xi���� or� if p is a rough function� we choose

the harmonic average

pi���� � �xi�Z xi

xi��

p���x�dx���

If p is small then the centered discretization leading to fi�y� is ques�tionable� but we do not pursue this further here�

It is natural to apply an IMEX method to ������ since the nonlinearconvection term typically yields an absolute stability requirement ofthe form h � const�x� which is not di cult to live with� whereas thelinear di�usion term is sti� �unless p is very small�� Moreover� due tothe hyperbolic nature of the convection term and the parabolic natureof the di�usion term� an appropriate test equation to investigate thestability properties of the IMEX method ���� is

y� � �a� �b�y �����

with a� b real constants� a � � �� �p���� and where we identify

f�y� � �by and g�y� � ay in ������ �����

�a� What is the domain of absolute stability for an IMEXmethod withrespect to this test equation! What corresponds to a ��region asin Exercise ���!

�b� Plot ��curves with � � �� ��� �� �� for the following ��step IMEXmethods�

� Adams�Bashforth with trapezoidal methodyn � yn�� �

h

���fn�� � fn�� � gn � gn����

� Adams�Bashforth with �����yn � yn�� �

h

�����fn�� � �fn�� � �gn � �gn�� � gn����

� Semi�explicit BDFyn �

���yn�� � yn��� �

�h

���fn�� � fn�� � gn��

Discuss your observations�

Page 178: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Chapter �

More BVP Theory and

Applications

In this chapter and the next two we will consider an ODE system with mcomponents�

y� � f�t�y�� � � t � b �����

subject to m twopoint boundary conditions

g�y����y�b�� � � � �����

We denote the Jacobian matrices of g�u�v� with respect to its �rst andsecond argument vectors by

B� ��g

�u� Bb �

�g

�v� �����

Often in applications� g is linear� i�e� the boundary conditions can bewritten as

B�y��� �Bby�b� � b �����

for some given data b�� and the m�m matrices B� and Bb are constant�Also� often in applications the boundary conditions are separated� i�e�

each of the components of g is given either at t � � or at t � b� but noneinvolves both ends simultaneously�� In this case for each i� � � i � m� eitherthe ith row of B� or the ith row of Bb are identically zero�

Example � Recall the vibrating spring Example ����

��p�t�u��� � q�t�u � r�t�

�Note that the data vector b and the interval end b are not related� Alas� we seem tobe running out of good notation�

�A notable exception is the case of periodic boundary conditions�

���

Page 179: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

��� Chapter �� Boundary Value Problems

u��� � �� u��b� � �

where p�t� � �� q�t� � � for all � � t � b� In more independent variables� aproblem like this corresponds to an elliptic partial di�erential equation�To convert this into a system we have two popular options�

� The standard option is to set y� � u� y� � u�� resulting in a system ofthe form ���� with m � ��

� Often in practice the function p�t� has discontinuities� In this case it isbetter to de�ne the unknowns y� � u and y� � py�� �this y� is sometimesreferred to as the �ux� This gives an ODE system in the form ����

with f�t�y� �

�� p��y�

qy� � r

�A�

The boundary conditions are separated and are given for both choices of unknowns by

B� �

��� �

� �

�A � Bb �

��� �

� �

�A � b � � �

We have already seen in Chapter � that there is no chance for extendingthe general Existence and Uniqueness Theorem ��� for initial value problemsto the boundary value problem case� In particular� assuming that the con�ditions of that theorem hold for f � for each initial value vector c we have asolution y�t� � y�t� c� for the ODE ����� satisfying y��� c� � c� Substitutinginto ����� we have

g�c�y�b� c�� � ��

This gives a set of m nonlinear algebraic equations for the m unknownsc �unknown� because we are asking what initial conditions would yield asolution that satis�es the boundary conditions�� It is well�known that ingeneral such a system may have many solutions� one� or none at all�

Example � The problem

u�� � eu�� � �

u��� � u��� � �

has two solutions of the form

u�t� � �� lnfcosh��t� ��������cosh�����

g

Page 180: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Chapter �� More BVP Theory and Applications ��

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10

0.5

1

1.5

2

2.5

3

Figure ���� Two solutions u�t� for the BVP of Example ����

where � satis�es� �

p�e cosh����� �

This nonlinear algebraic equation has two solutions for � �Exercise ���� Thecorresponding two solutions of the BVP are plotted in Fig� ���� �

The possibility of having more than one solution does not in itself preventus from expecting to be able to �nd them� The question of existence ofunique solutions for a nonlinear BVP must be considered in a local sense�The important question is whether a BVP solution is isolated� i�e�� if thereis a neighborhood about it in which it is the only solution� For this purposewe look at the variational problem for the BVP ������������ assuming forsimplicity of notation that g is linear� i�e� that the boundary conditions arein the form ������ the variational problem corresponding to linearizing theproblem about an exact solution y�t� is

z� � A�t�y�t��z ����

B�z��� �Bbz�b� � �

where A � �f�yis the Jacobian matrix� Now� if the variational problem has

the unique solution z � � then the solution y�t� of the given nonlinear prob�lem is isolated� or locally unique� We will show this claim following ������below� The uniqueness of the zero solution z means that the linearizationis nonsingular and this gives us a �ghting chance at �nding isolated solu�tions using the numerical methods described in the next two chapters� ForExample ���� it can be veri�ed that both solutions are isolated�

Page 181: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

��� Chapter �� Boundary Value Problems

In order to understand the issues arising in the numerical solution ofBVPs� then� we must get a better idea of the theory of linear BVPs�

��� Linear Boundary Value Problems and Green�s

Function

Consider the linear ODE system of m equations�

y� � A�t�y� q�t�� � � t � b �����

and recall that a fundamental solution Y �t� is the m � m matrix functionsatisfying

Y � � A�t�Y� � � t � b

and Y ��� � I� Using this fundamental solution� the general solution of theODE ����� is

y�t� � Y �t�

�c�

Z t

Y ���s�q�s�ds�� �����

The parameter vector c in ����� is determined by the linear boundary condi�tions ������ Substituting� we get

�B�Y ��� �BbY �b��c � b�BbY �b�

Z b

Y ���s�q�s�ds �

The right hand side in the above expression depends on the given data�Thus we have obtained a basic existence and uniqueness theorem for linearboundary value problems�

Theorem � Let A�t� and q�t� be continuous and de�ne the matrix

Q � B� �BbY �b� �����

�remember� Y ��� � I� Then

� The linear BVP ��������� has a unique solution i� Q is nonsingular�

� If Q is nonsingular then the solution is given by ���� with

c � Q���b�BbY �b�

Z b

Y ���s�q�s�ds��

Page 182: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Chapter �� More BVP Theory and Applications ���

Example � Returning to the �rst example in Chapter ��

u�� � �u� u��� � b�� u�b� � b�

we write this in �rst order form with

A �

�� � �

�� �

�A � B� �

��� �

� �

�A � Bb �

��� �

� �

�A � b �

��b�b�

�A �

It is easy to verify that

Y �t� �

�� cos t sin t

� sin t cos t

�A

so

Q � B� �Bb

�� cos b sin b

� sin b cos b

�A �

�� � �

cos b sin b

�A �

This matrix is singular i� b � j for some integer j� Theorem ��� nowimplies that a unique solution exists if b �� j� for any integer j � see Fig����� �

The fundamental solution Y �t� satis�es Y ��� � I� i�e�� it is scaled for aninitial value problem� A better scaled fundamental solution for the BVP athand is

+�t� � Y �t�Q�� � �����

Note that + satis�es the homogeneous ODE� i�e� it is indeed a fundamentalsolution� We have

+� � A+� � � t � b ������

B�+��� �Bb+�b� � I �

So +�t� plays the same role for the BVP as Y �t� plays for the IVP�We often refer to the columns of the scaled fundamental solution +�t�

�or Y �t� in the IVP case� as solution modes� or just modes for short� Theyindicate the solution sensitivity to perturbation in the initial data �recallChapter ���If we carry out the suggestion in Theorem ��� and substitute the expres�

sion for c into ����� then we get an expression for the solution y�t� in termsof the data b and q�t�� Rearranging� this gives

y�t� � +�t�b�

Z b

G�t� s�q�s�ds ������

Page 183: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

��� Chapter �� Boundary Value Problems

0 1 2 3 4 5 6 7 8 9 100

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

t

y

y y1 2

Figure ���� The function y��t� and its mirror image y��t� � y��b � t�� for� � ��� b � ���

where G�t� s� is the Green�s function

G�t� s� �

��� +�t�B�+���+���s� s � t

�+�t�Bb+�b�+���s� s � t������

Green�s function may be loosely viewed as the inverse of the di�erentialoperator �or� as the solution operator��

��� Stability of Boundary Value Problems

To understand the fundamental issues in stability for BVPs� the reader mustbe familiar with the rudiments of stability of IVPs� Therefore� please makesure that you are familiar with the contents of Chapter ��

Consider the test equation y�� � �y� for � � t � b and regard b as verylarge� The IVP �e�g� y���� � �� is stable if Re��� � �� Now apply a variabletransformation � � b � t� The same problem in � then reads dy�

d�� ��y��

with y��b� � �� i�e� this is a terminal value problem and we are integratingfrom b to �� see Fig� ���� Of course� reversing the direction of time doesnot a�ect the stability� which has to do with the e�ect of small changes inthe data on the solution� so this terminal value problem is stable as well �forRe���� � ��� Putting the two together� we obtain that the following BVP

Page 184: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Chapter �� More BVP Theory and Applications ���

is stable

y� � Ay� A �

��� �

� ��

�A

y���� � �� y��b� � �

although the IVP for the same ODE is unstable when Re��� �� �� Thus�the stability of solutions for a given ODE depends on how �and where� theboundary conditions are speci�ed�For a general linear BVP ������ ������ the sensitivity of the solution to

perturbations in the data is immediately given by introducing bounds in������� because this formula gives the solution in terms of the data� Let thestability constant of the BVP be de�ned by

� � max�k+k�� kGk�� � ������

Then from �������

kyk � max��t�b

jy�t�j � �

�jbj�

Z b

jq�s�jds�� ������

Rather than considering families of problems with b becoming unbounded� weshall say qualitatively that the linear BVP is stable if the stability constant� is of moderate size� Roughly� �moderate size� means not much larger thanthe magnitude of the problem�s coe cients� kA�t�kb�Why is ������ a stability bound! Consider a perturbed problem� �y� �

A�t��y��q�t�� B��y����Bb�y�b� � �b� Thus� the inhomogeneities are perturbedby ��t� � �q�t� � q�t� and � � �b � b� Then the perturbation in the solu�tion� x�t� � �y�t� � y�t�� satis�es the same linear BVP ������ ����� for theperturbation in the data�

x� � A�t�x� ��t�� � � t � b

B�x��� �Bbx�b� � ��

So ������ bounds x in terms of the perturbations in the data�

kxk � �

�j�j�

Z b

j��s�jds�� �����

Now we can further explain the concept of an isolated solution for thenonlinear problem ������ ������ Suppose that y�t� is a non�isolated solution�i�e�� for any arbitrarily small � � � there is another solution �y which satis�es�y� � f�t� �y�� B��y��� � Bb�y�b� � b� k�y � yk � �� Then the di�erence x�t� ��y�t�� y�t� satis�es

x� � f�t� �y�� f�t�y� � A�t�y�t�� x�O����� � � t � b

B�x��� �Bbx�b� � ��

Page 185: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

��� Chapter �� Boundary Value Problems

Note that the variational problem ���� has the unique zero solution i� thecorresponding Q of ����� is nonsingular� But if Q is nonsingular then forsome �nite � we get from ����� that

� � kxk � �O���� �

This inequality cannot hold if � is arbitrarily small and positive� Hence� thenon�singularity of the variational problem implies that y�t� is an isolatedsolution of the nonlinear problem�The stability of the problem essentially means that Green�s function is

nicely bounded� Consider next the case of separated boundary conditions�i�e�� assume that the �rst k rows of Bb and the last m� k rows of B� are allzeros� Then from ������� clearly

B�+��� � P �

��Ik �

� �

�A � Bb+�b� � I � P

where Ik is the k�k identity� so P is an orthogonal projection matrix �mean�ing P � � P � of rank k� In this case we can write Green�s function as

G�t� s� �

��� +�t�P+���s� s � t

�+�t��I � P �+���s� s � t

The BVP is said to have dichotomy if there is a constant K of moderatesize such that

k+�t�P+���s�k � K� s � t �����a�

k+�t��I � P �+���s�k � K� s � t � �����b�

The BVP has exponential dichotomy if there are positive constants �� suchthat

k+�t�P+���s�k � Ke��s�t�� s � t �����a�

k+�t��I � P �+���s�k � Ke��t�s�� s � t � �����b�

Dichotomy and exponential dichotomy correspond to stability and asymp�totic stability� respectively� in IVPs� �Compare ������ with �����a� for k �m�� Dichotomy implies that the �rst k columns of +�t� are non�increasing�actually decreasing in case of exponential dichotomy� as t grows� and thatthe last m � k columns of +�t� are nondecreasing �actually increasing incase of exponential dichotomy� as t grows� The k non�increasing modes are

Page 186: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Chapter �� More BVP Theory and Applications ���

controlled in size by the boundary conditions at �� whereas the m� k non�decreasing modes are controlled in size by the boundary conditions at b�Dichotomy is a necessary and su cient condition for stability of the BVP�The situation for non�separate boundary conditions is much more com�

plicated� although the conclusions remain essentially the same�

Example � For the problem

u�� � u� u��� � b�� u�b� � b�

i�e�� with the ODE di�erent from Example ��� but the boundary conditions

the same� we convert to �rst order form with A �

��� �

� �

�A� The fundamental

solution satisfying Y ��� � I is

Y �t� �

��cosh t sinh t

sinh t cosh t

�A �

Clearly� kY �t�k grows exponentially with t� indicating that the initial valueproblem is unstable� For the boundary value problem� however� we have

Q � B�Y ��� �BbY �b� �

�� � �

cosh b sinh b

�A �

so

+�t� � Y �t�Q�� ��

sinh b

�� sinh�b� t� sinh t

� cosh�b� t� cosh t

�A �

Thus� the �rst column of +�t� �here k � � and m � � is decreasing in t andthe second column of +�t� is increasing� Both of these columns are nicelyscaled�

k+k � �even though Q becomes extremely illconditioned as b grows� We leave itas Exercise ��� to show that this boundary value problem is stable and hasexponential dichotomy� �

��� BVP Sti�ness

In x��� we introduce the notion of sti�ness for IVPs� In the terminology ofthe previous section� a sti� �linear� problem is a stable problem which has

Page 187: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

��� Chapter �� Boundary Value Problems

very fast modes� For an IVP such modes can only be decreasing� But for astable BVP we must entertain the possibility of both rapidly decreasing andrapidly increasing modes being present�Corresponding to ������ in x��� we say that a stable BVP for the test

equationy� � �y � � � t � b

is sti� if

bjRe���j �� ������

In contrast to the IVP case� here we no longer require Re��� � �� Similarlyto ������� this generalizes for a nonlinear system y� � f�t�y� to

bjRe��j�j � ������

where �j are the eigenvalues of the local Jacobian matrix�f�y�t�y�t��� �

This extension of the IVP de�nition makes sense� in light of the discussionof dichotomy in the previous section� The practical understanding of thequalitative notion behind the inequalities in ������ and ������ is that wemust look for numerical methods that work also when hjRe��j�j �� whereh is a typical discretization step size�However� this is easier said than done� There are really no known dis�

cretization methods which have a similar robustness to that in the IVP caseof backward Euler and its higher order extensions �e�g� BDF methods andcollocation at Radau points�� The methods discussed in the next chapter�and other variants which are not discussed there� are not suitable for very sti�BVPs� Symmetric di�erence methods like midpoint� which are our methodsof choice for BVPs and are discussed in Chapter �� often perform well inpractice for sti� BVPs� but their theoretical foundation is somewhat shakyin this case� as discussed further in Chapter �� There are methods �e�g�Riccati� which attempt to decouple rapidly increasing and rapidly decreasingmodes explicitly� and then integrate such modes only in their correspondingstable directions� But these methods appear more suitable for special appli�cations than for general�purpose use for nonlinear problems� To explicitlydecouple modes� especially for nonlinear problems� is no easy task�

��� Some Reformulation Tricks

While general�purpose codes for BVPs usually assume a system of the form����� subject to boundary conditions of the form ����� or� even more fre�

�Of course� �j �j�t� may in general have a large real part in some parts of the intervaland a small �in magnitude� real part in others� but let us assume here� for simplicity ofthe exposition� that this does not happen�

Page 188: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Chapter �� More BVP Theory and Applications ���

quently� separated boundary conditions� the natural way in which boundaryvalue problems arise in applications often does not conform to this standardform� An example that we have already seen is the conversion from a higherorder ODE system to a �rst order system� There are other� less obvioussituations� where a given BVP can be reformulated� Of course all this canbe said of IVPs as well� but there is more diversity in the BVP case� Thereare a number of reformulation �tricks� that can be used to convert a givenproblem to standard form� of which we describe a few basic ones here�In many applications� the ODE system depends on an unknown constant�

a� and this gives rise to an additional boundary condition� One can then addthe ODE

a� � �

to the system� This means that the constant a is viewed as a function overthe interval of integration which is independent of t�

Example � The �ow in a channel can be modeled by the ODE

f ��� �R��f ��� � ff ��� � Ra � �

f��� � f ���� � �� f��� � �� f ���� � � �

The constant R �Reynolds number is known� but the constant a is undetermined� There are � boundary conditions on the potential function f whichdetermine both it and a� To convert to standard form we write y� � f� y� �f �� y� � f ��� y� � a� and obtain

y� � f�y� �

�BBBBBB�

y�

y�

R�y�� � y�y� � y��

�CCCCCCA

The boundary conditions are obviously in separated� standard form as well��

The unknown constant can be the size of the interval of integration� As�suming that the problem is given in the form ����� but with the integrationrange b unknown� we can apply the change of variable

� � t�b

to obtain the ODE system

dy

d�� bf�b��y�� � � � � �

db

d�� � �

Page 189: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

��� Chapter �� Boundary Value Problems

So� the new vector of unknowns is

��y�� �

b

�A and its length is m � �� This

is also the number of independent boundary conditions that should be givenfor this system�

The unknown constants trick can also be used to convert non�separateboundary conditions to separated boundary conditions� at the expense ofincreasing the size of the ODE system� �In the end� representing a constantby an unknown function to be integrated throughout the interval in t is neververy economical� so there has to be a good reason for doing this�� We haveseen in the previous section that the theory is easier and simpler for the caseof separated boundary conditions� This tends also to be re�ected in simplersolution methods for the linear systems arising from a �nite di�erence ormultiple shooting discretization�

Given the boundary conditions g�y����y�b�� � �� let a � y��� be ourunknown constants� Then we can rewrite the system in the form

y� � f�t�y�

a� � ��

with

y��� � a���

g�a�b��y�b�� � ��

��� Notes and References

Chapter � of Ascher� Mattheij Russell ��� contains a much more detailedaccount of the material presented in x��� and x���� which includes the var�ious extensions and proofs mentioned here� Classical references on Green�sfunction and on dichotomy are Stakgold ���� and Coppel ����� respectively�For periodic solutions� see Stuart Humphries ����� Sti�ness and decouplingin the linear case are discussed at length in ���� where more reformulationexamples and references can be found as well�

Page 190: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Chapter �� More BVP Theory and Applications ��

��� Exercises

�� Show that the equation

� �p�e cosh�����

has two solutions ��

�� �a� Show that the problem in Example ��� is stable for all b � � andhas exponential dichotomy� What are its Green�s function andstability constant!

�b� Same question for the periodic boundary conditions

u��� � u�b�� u���� � u��b� �

�� Consider the problem

u��� � �u�� � u� � �u� � � t � b

u���� � �� u�b�� u��b� � ��

u��� � �

with b � ����

�a� Convert the ODE to a �rst order system and �nd its fundamentalsolution satisfying Y ��� � I�

� Hint� another� well scaled fundamental solution is

,�t� �

�BBB�

e�t et�b e��t�b�

�e�t et�b �e��t�b�

e�t et�b �e��t�b�

�CCCA

and recall that Y �t� � ,�t�R for some constant matrix R��

�b� It�s not given whether the last boundary condition is prescribedat � � � or at � � b� But it is known that the BVP is stable�with stability constant � � ���� Determine where this boundarycondition is prescribed�

�� Consider an ODE system of size m

y� � f�t�y� �����a�

where f has bounded �rst and second partial derivatives� subject toinitial conditions

y��� � c �����b�

Page 191: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

��� Chapter �� Boundary Value Problems

or boundary conditions

B�y��� �Bby�b� � b � �����c�

It is often important to determine the sensitivity of the problem withrespect to the data c or b� For instance� if we change cj to cj � �for some j� � � j � m� where j�j � �� and call the solution of theperturbed problem �y�t�� what can be said about j�y�t��y�t�j for t � �!�a� Writing the solution of �����a�����b� as y�t� c�� de�ne the m�m

matrix function

Y �t� ��y�t� c�

�c�

Show that Y satis�es the initial value problem

Y � � A�t�Y

Y ��� � I

where A � �f�y�t�y�t� c���

�b� Let �y�t� satisfy �����a� and

�y��� � c� �d

where jdj � � and j�j � �� Show that

�y�t� � y�t� � �Y �t�d�O���� �

In particular� what can you say about the sensitivity of the prob�lem with respect to the j�th initial value!

�c� Answer questions analogous to �a� and �b� above regarding thesensitivity of the boundary value problem �����a�����c� with re�spect to the boundary values b� How would a bound on k�y �yk� � max��t�b j�y�t�� y�t�j relate to the stability constant � of������!

Page 192: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Chapter �

Shooting

Shooting is a straightforward extension of the initial value techniques thatwe have seen so far in this book to solve boundary value problems� Essen�tially� one �shoots� trajectories of the same ODE with di�erent initial valuesuntil one �hits� the correct given boundary values at the other interval end�The advantages are conceptual simplicity and the ability to make use of theexcellent� widely available� adaptive initial�value ODE software� But thereare fundamental disadvantages as well� mainly in that the algorithm inheritsits stability properties from the stability of the initial value problems that itsolves� not just the stability of the given boundary value problem�

��� Shooting a Simple Method and its Lim�

itations

For a system of ODEs of order m�

y� � f�t�y�� � � t � b �����

subject to m two�point boundary conditions

g�y����y�b�� � � �����

we denote by y�t� � y�t� c� the solution of the ODE ����� satisfying theinitial condition y��� c� � c� Substituting into ����� we have

h�c� g�c�y�b� c�� � � � �����

This gives a set of m nonlinear algebraic equations for the m unknowns c�The simple �or single� shooting method consists of a numerical imple�

mentation of these observations� which we have used in previous chapters for

���

Page 193: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

��� Chapter � Boundary Value Problems

theoretical purposes� Thus� one couples a program module for solving non�linear algebraic equations �such library routines are available� with a modulethat� for a given c� solves the corresponding initial value ODE problem�

Example � � Recall Example ��� which considers a very simple model of achemical reaction

u�� � eu�� � �

u��� � u��� � � �

The two solutions are depicted in Fig� ��� �only the lower one is a physicallystable steady state� Converting to �rst order form for y � �u� u��T � weknow that y���� � � � c�� so only y���� � c� is unknown� The IVP has aunique solution y�t� c� �or u�t� c�� for each value c�� even though it is notguaranteed that this solution will reach t � � for any c�� But� as it turns out�this problem is easy to solve using simple shooting� With a starting �angle�of shooting �for the nonlinear iteration c�� � ��� the lower curve of Fig� ���is obtained after a few Newton iterations to solve ����� and with a starting�angle� of shooting c�� � ��� the high curve of Fig� ��� is easily obtained aswell �Exercise ���� �

Let us consider Newton�s method for the solution of the nonlinear equa�tions ������ The iteration is

c��� � c� ���h

�c

���h�c��

where c� is a starting iterate� �di�erent starting guesses can lead to di�erentsolutions� as in Example ����� To evaluate h�c�� at a given iterate we haveto solve an IVP for y�t� c� �see ������� Moreover� to evaluate

��h�c

�at c � c��

we must di�erentiate the expression in ����� with respect to c� Using thechain rule of di�erentiation and the notation of ������ this gives�

�h

�c

�� B� �BbY �b� � Q

where Y �t� is the m�m fundamental solution matrix satisfying

Y � � A�t�Y� � � t � b

Y ��� � I

with A�t�y�t� c��� � �f�y�see Chapter � � this variational ODE should be

familiar to you at this point��

�Note that the superscript � is an iteration counter� not a power�

Page 194: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Chapter � Shooting ���

We see therefore that using Newton�s method� m � � IVPs are to besolved at each iteration �one for h and m linear ones for the columns ofY �t��� However� the m linear systems are simple and they share the samematrix A�t� which can therefore be evaluated once for all m systems� so thesolution of these IVPs typically costs much less than m�� times the solutionof the IVP for h� �

Once convergence has been obtained� i�e� the appropriate initial valuevector c which solves h�c� � � has been �approximately� found� we integratethe corresponding IVP to evaluate the solution of the BVP at any givenpoints�To summarize� here is the algorithm combining shooting with Newton�s

method for a nonlinear BVP ������������

Algorithm � � Shooting with Newton

� Given�� f � �f

�yfor each t and y�

�� g�u�v�� �g�u� �g�vfor each u and v�

�� An initialvalue solver�

�� An initial guess c�� and

�� A convergence tolerance TOL for the nonlinear iteration�

� For s � �� �� � � � � until jcs�� � csj � TOL�

�� Solve the IVP ���� with y��� � cs� obtaining a mesh and solutionvalues ysn� n � �� � � � Ns�

�� Construct h�cs� � g�cs�ysNs��

�� Integrate the fundamental matrix Yn� n � �� � � �Ns �Y� � I� onthe same mesh� using A�tn� �

�f�y�tn�ysn��

�� Form Q � B� �BbYsNsusing

B� ��g

�u�cs�ysNs

�� Bb ��g

�v�cs�ysNs

� �

and solve the linear system

Q� � h�cs�

for the Newton correction vector ��

�Solving the variational ODE is equivalent to computing the sensitivity of the solutionto the original ODE ����� with respect to variations in the initial conditions� see x����

Page 195: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

��� Chapter � Boundary Value Problems

�� Set

cs�� � cs � � �

� Solve the IVP for ���� for y��� � c� with the values c obtained by theNewton iteration�

We note that to maximize the e ciency of a shooting code� methods otherthan Newton�s �e�g� quasi�Newton� should be used� We do not pursue thisfurther� though�

����� Di�culties

From the above description we also see the potential trouble that the simpleshooting method may run into� the conditioning of each iteration dependson the IVP stability� not only on the BVP stability� The matrix that featuresin the iteration is Q � B��BbY �b�� and this matrix can be extremely poorlyconditioned �recall Example ���� even when the BVP is stable and not verysti�� Finding the solution� once the correct initial values are known� alsoinvolves integrating a potentially unstable IVP�It is not di cult to see that if a method of order p is used for the initial

value integrations �in the sense of the IVP methods studied in Chapters�� � and � then a method of order p is obtained for the boundary valueproblem� This follows directly if we assume that the nonlinear iterationfor ����� converges� and in the absence of roundo� errors� The trouble in�nding c �if there is any� does not arise because of truncation errors� becausefor a stable BVP error growth along unstable modes gets cancelled �recallx����� and this e�ect is reproduced by a consistent� stable IVP discretization�Also� if the BVP is unstable then the shooting method is expected to havedi culties� but these will be shared by other standard methods� the casewhere the simple shooting method is particularly unsatisfactory is when othersimple methods �discussed in the next chapter� work well while this methoddoes not� Such is the case in the following example�

Example � � The following problem

y� � A�t�y� q�t�

A �

�BBB�

� � �

� � �

���� �� ��

�CCCA

y���� � b�� y���� � b�� y���� � b�

Page 196: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Chapter � Shooting ���

has the exact solution

y�t� � �u�t�� u��t�� u���t��T � u�t� �e��t���� e���t��� � e��t

� � e��� cos t

�you can evaluate the expressions and values for q�t� � y��t��A�t�y�t� andthe boundary values b from this exact solution� The problem is in the form���� with

B� �

�BBB�� � �

� � �

� � �

�CCCA � Bb �

�BBB�� � �

� � �

� � �

�CCCA �

For � � ��� say� the BVP is stable but the IVP is not�In Figs� ��� and ��� we display the exact and approximate solutions �solid

and dashed lines� resp� for various values of � ranging from a harmless � toa tough �� We use the classical RungeKutta method of order � with a �xedstep size h � ���� and a ��hexadecimal digits �oating point arithmetic�

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1−0.2

0

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

1.8

t

u

exact and approximate u(t)

�a� � �

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1−1

−0.5

0

0.5

1

1.5

2

t

u

exact and approximate u(t)

�b� � ��

Figure ���� Exact �solid line� and shooting �dashed line� solutions for Exam�ple ����

Note that the disastrous e�ect observed is due to the propagation of errorsin the obtained initial values c by unstable modes �recall Example ���� Theerror in c is unavoidable and is due to roundo�� not truncation errors� Wehave chosen the discretization stepsize so small� in fact� that for the case� � � the errors in the initial values vector are all below ����� as is themaximum error in u in the ensuing integration for the approximate solution

Page 197: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

��� Chapter � Boundary Value Problems

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1−250

−200

−150

−100

−50

0

50

t

u

exact and approximate u(t)

�a� � ��

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1−0.5

0

0.5

1

1.5

2

2.5x 10

32

t

u

exact and approximate u(t)

�b� � ��

Figure ���� Exact �solid line� and shooting �dashed line� solutions for Exam�ple ����

of the BVP� For � � ��� already an O��� error is observed �the maximumerror in u is ���� This case may be regarded as particularly worrisome�because the wrong solution obtained for a moderate value of � may also lookplausible� For � � ��� this maximum error is ����� �although the error in theinitial conditions is only less than ����� and for � � �� the overall error inu is ���e� �� and in the initial conditions it is about �� �

The instability is already extreme for � � ��� a value for which the BVPis not very sti�� �

Another potential di culty with the simple shooting method arises fornonlinear problems� The method assumes that the initial value problemsencountered will have solutions� even for inaccurate initial values� that reachall the way to t � b� For nonlinear problems� however� there is no guaranteethat this would be the case� Initial value solutions with incorrect initial valuesare typically guaranteed to exist locally in t� but not necessarily globally� Foranother potential di culty with the nonlinear iteration see Exercise ����

Page 198: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Chapter � Shooting ���

b0

Figure ���� Multiple shooting

��� Multiple Shooting

Both disadvantages of the simple shooting method become worse for largerintervals of integration of the initial value problems� In fact� a rough boundon the propagation error� which is approximately achieved in Example ����is eLb� where L � maxt kA�t�k� The basic idea of multiple shooting is thento restrict the size of intervals over which IVPs are integrated� De�ning amesh

� � t� � t� � � � � � tN�� � tN � b

we consider approximating the solution of the ODE system y� � f�t�y�by constructing an approximate solution on each subinterval �tn��� tn� andpatching these approximate solutions together to form a global one �see Fig������ Thus� let yn�t� cn��� be the solution of the initial value problem

y�n � f�t�yn�� tn�� � t � tn ����a�

yn�tn��� � cn��� ����b�

for � � n � N �� Assuming for the moment that the initial value problems����� are solved exactly� we then have that the exact solution of the problem����������� satis�es

y�t� � yn�t� cn���� tn�� � t � tn� � � n � N

if

yn�tn� cn��� � cn� � � n � N � � ���a�

g�c��yN�b� cN���� � � � ���b�

The conditions ���a� are patching conditions which ensure that y�t� patchedfrom the di�erent pieces yn�t� cn��� is continuous on the entire interval ��� b��and ���b� is just the resulting expression for the boundary conditions ������

�It is important not to confuse this notation with what is used in the �nite dierencechapters �� �� � and � for a slightly dierent purpose� Here yn is meant to be the exact

solution on a subinterval �tn��� tn�� provided we can �nd the right cn���

Page 199: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

��� Chapter � Boundary Value Problems

The conditions ���� give Nm algebraic equations for the Nm coe cients

c � �cT� � cT� � � � � � c

TN���

T �

We write these equations� as before� as

h�c� � � � �����

Applying Newton�s method to solve the nonlinear equations ����� resultsat each iteration � in a system of linear equations which can be viewed asarising from the same multiple shooting method for the linearized boundaryvalue problem

A�c��� � c�� � �h�c��where A � �h

�c�c�� has a sparse block structure� as in ������ below� An

advantage of Newton�s method here �not shared by quasi�Newton methods�is that this sparse block structure remains intact during the iteration process�

Since the system of linear equations is the same as the one obtained byapplying the same multiple shooting method to the linearized problem� letus consider the latter further� For the linear problem

y� � A�t�y� q�t� �����

B�y��� �Bby�b� � b

we can write

yn�t� cn��� � Yn�t�cn�� � vn�t�

where Yn�t� is the fundamental solution satisfying

Y �n � A�t�Yn� Yn�tn��� � I

�in particular� Y� Y �� and vn�t� is a particular solution satisfying� e�g��

v�n � A�t�vn � q�t�� vn�tn��� � � �

The patching conditions and boundary conditions are then

Icn � Yn�tn�cn�� � vn�tn� � � n � N � � ����a�

B�c� �BbYN �b�cN�� � b�BbvN �b� � ����b�

Writing these conditions as a linear system� we get

Ac � r �����

Page 200: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Chapter � Shooting ��

where

A �

�BBBBBBBBB�

�Y��t�� I

�Y��t�� I� � � � � �

�YN���tN��� I

B� BbYN�b�

�CCCCCCCCCA�

c �

�BBBBBBBBB�

c�

c����

cN��

cN��

�CCCCCCCCCA� r �

�BBBBBBBBB�

v��t��

v��t�����

vN���tN���

b�BbvN �b�

�CCCCCCCCCA� ������

The matrix A is large and sparse when N is large� but there are well�knownvariants of Gaussian elimination which allow the solution of the linear systemof equations ����� in O�N� time� This will be discussed in x���� In fact� givenN parallel processors in a computational model which ignores communicationcosts� the solution time for this linear system can be reduced to O�logN��Note that the blocks Yn�tn� can be constructed in parallel too�� Initial valueintegration is applied for these constructions� as well as for the constructionof the vn�s�Turning to the question of whether the instability of the single shooting

method has been improved upon� note that� assuming that the boundarymatrices are scaled to O����

kAk � const

�max

��n�NfkYn�tn�kg� �

��

It can be also veri�ed directly that A has the inverse

A�� �

�BBB�

G�t�� t�� � � � G�t�� tN��� +�t�����

������

G�tN��� t�� � � � G�tN��� tN��� +�tN���

�CCCA ������

where G and + are de�ned in ������ and ������ resp� Therefore� with � thestability constant of the given boundary value problem �recall ��������

kA��k � N�

�For this reason the multiple shooting method is sometimes referred to as the parallel

shooting method�

Page 201: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

��� Chapter � Boundary Value Problems

so

cond�A� � kAkkA��k � const �N

�max��n�N

fkYn�tn�kg� ��

������

for some moderate constant const�The problem with simple shooting is that kY �b�k can be very large� even

for stable BVPs� and it features prominently in the conditioning of the shoot�ing algorithm �because kAk is very large�� The bound on the condition num�ber in ������ is often much more acceptable� For Example ��� with � � ����� equally spaced multiple shooting points produce an accurate solution �to� digits� using the same discretization for the IVPs�� in contrast to the simpleshooting results shown in Fig� ���� The other disadvantage� resulting from�nite escape time in nonlinear initial value problems� is corrected to a largeextent by multiple shooting as well�However� with the signi�cant improvement of the various de�ciencies� the

conceptual simplicity of simple shooting is also gone� Moreover� for very sti�BVPs the number of shooting points must grow unacceptably large with thesti�ness parameter �e�g�� it is proportional to �� as ���� in Example �����

��� Software Notes and References

����� Notes

A detailed treatment of the techniques covered in this chapter can be foundin Chapter � of Ascher� Mattheij Russell ���� See also Mattheij Molenaar����� Earlier references include Keller ���� Our presentation is deliberatelyshort � we have chosen to concentrate more on �nite di�erence methods inthe next chapter�The simple shooting method applied to a linear BVP� see ������ can be

viewed as a method of superposition where the solution is composed of alinear combination of solution modes �columns of Y �t�� plus a particularsolution of the nonhomogeneous problem ����� subject to �say� homogeneousinitial conditions� There are more e cient� reduced superposition variants aswell� see ��� and references therein�There are other initial value techniques like stabilized march and Ric�

cati methods which possess certain advantages �and disadvantages� over themultiple shooting method presented here� They can be viewed as achieving�for a linear�ized� problem� a decoupling of rapidly increasing modes �whoseforward integration yields stability problems� from the other modes� The

Page 202: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Chapter � Shooting ���

algorithm described in Exercise ��� can be made to be stable then� See x���for more on decoupling� For reasons of space and bias� however� we do notexplore these methods further� The interested reader can consult Chapter �of ����

For use of a multiple shooting method for parameter identi�cation� i�e�attempting to �nd unknown parameters which de�ne the ODE given obser�vations on its solution� see �����

����� Software

Many scientists and engineers seem to implement their own application�dependent shooting codes� making use of the excellent and abundant softwarewhich is available for initial value problems� Shooting handles problems withnon�separated boundary conditions� and extends naturally to handle prob�lems with parameters� Sparse linear algebra is avoided� at least when m isnot large� The Nag library has a simple shooting code written by I� Glad�well� Another shooting code is being developed by L� Shampine� at the timeof this writing� for Matlab� However� we �nd the limited applicability ofthis method somewhat unsettling for general purposes�

A number of multiple shooting codes have been developed in the �����sand �����s� We mention the code mus by Mattheij Staarink ���� �� whichis available from Netlib� Earlier codes include suport by Scott Watts�����

��� Exercises

�� Write a simple shooting code� using available software modules for ini�tial value ODE integration� solution of nonlinear algebraic equations�and solution of linear algebraic equations as you �nd necessary� Applyyour code to the following problems�

�a� Find both solutions of Example ���� What are the correct initialvalues for each of the two solutions!

�b� Use your program �only after verifying that it is correct� on somestable boundary value problem of your choice where it is not sup�posed to work� and explain the observed results�

�� �a� Verify that the expression given in ������ is indeed the inverse ofA given in �������

Page 203: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

��� Chapter � Boundary Value Problems

�b� Estimate cond�A� for Example ��� with � � ��� using �� equallyspaced multiple shooting points�

�c� How many multiple shooting points are needed to obtain a similarbound on cond�A� when � � ���!

�� Consider the car model of Example ��� with the same initial conditionsas employed there� Given that a � ���� the task is to �nd a constantsteering angle � so that the car will pass through the point x�b� � ����y�b� � ��

�a� Formulate this as a BVP �of order �� in standard form�

�b� Solve this BVP numerically� using a package of your choice oryour own home�grown program� Verify that the �nal speed isv�b� � ������� What is the required angle �! How long does ittake the car to get to x�b�� y�b�!

�� Consider the nonlinear problem

v�� ��

tv� � �tv � ��v � �� � � t �� ������

v���� � �� v��� � ��This is a well�behaved problem with a smooth� nontrivial solution� Tosolve it numerically� we replace ����� by a �nite� large interval ��� L�and require

v�L� � � �

For large t the solution is expected to decay exponentially� like e��t�for some � � ��

�a� Find the asymptotic behavior of the solution for large t �i�e� �nd��� �You may assume that v�t� is very �i�e� exponentially� smallwhen t is large��

�b� Show that the simple shooting method is unstable for this prob�lem�

�c� Describe the application of the multiple shooting method for thisproblem� Estimate �roughly� the number and location of theneeded shooting points�

�d� What would you do to obtain convergence of your scheme� avoid�ing convergence to the trivial solution!

� The so�called SH equations� arising when calculating the ground dis�placements caused by a point moment seismic source in a layeredmedium� form a simple ODE system

y� � A�t��� k�y� � � t � b

Page 204: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Chapter � Shooting ���

where

A �

�� � ���

�k� � ��� �

�A �

Here the angular frequency � and the horizontal wave number k areparameters� �� � � � �� � � k � �� The independent variablet corresponds to depth into the earth �which is the medium in thisseismological application�� See ���� �� and references therein for moredetails� although you don�t really need to understand the physics in or�der to solve this exercise� A hefty assumption is made that the earth inthe area under consideration consists of horizontal layers� Thus� thereis no horizontal variation in medium properties� Assume� moreover�that there is a partition

� � t� � t� � � � � � tN � b

such that the S�wave velocity �t�� the density ��t�� and thus also��t� � �� are constant in each layer�

� n� � � �n� � � �n� tn�� � t � tn�

�a� At the earth�s surface t � �� y���� �� � is given� Another boundarycondition is derived from a radiation condition� requiring that onlydown�going waves exist for t � b� Assuming that the propertiesof the medium are constant for t � b� this yields

��y��b� � ���y��b� � �

where�� �

pk� � ������

Derive this boundary condition�

�Hint� the eigenvalues of A are ������b� Describe a multiple shooting method that would yield the exact

solution �except for roundo� errors� for this BVP�

�Note that this problem has to be solved many times� for various valuesof k and �� because the obtained solution is used for integrand eval�uation for a double integral in k and �� It is therefore worthwhile totailor a particularly good method for this simple BVP��

�� Delay di�erential equations arise often in applications� There are somesituations in which a conversion to an ODE system can be useful� Con�sider a problem with a single� constant delay � � ��

z��t� � f�t� z�t�� �A�t�z�t� � �� � � t � b �����a�

B��z�t� � b��t�� �� � t � � �����b�

Bb�z�b� � b� �����c�

Page 205: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

��� Chapter � Boundary Value Problems

where in �����a� there are m equations� B�� is full rank k � m� Bb�

is �m � k� �m� B� �

��B��

�A is m � m� and A�t� can be written as

A�t� � �A�t�B�� We further assume that there is a unique� continuoussolution and that b � �J for some positive integer J �

�a� Show that the functions

yj�s� � z�s� �j � ��� �� j � �� � � � � J

satisfy the ODE system

y�j�s� � f�s � �j � ����yj�s�� �A�s� �j � ��� �yj���s�� j � �� � � � � J

� � s � �

y���s� � f�s�y��s�� � �A�s��b��s� � �

B��y���� � b���� �� Bb�yJ�� � � b�

yj�� � � yj������ j � �� � � � � J � ��

where �b� is just b� extended by m � k zeros� This is a BVP instandard form�

�b� Solve the following problem using your code of Exercise ��� or alibrary BVP package�

u���t� � � ���sinu�t�� �t� ��u�t� �� � t� � � t � �

u�t� � t� ���� �� � t � �u��� � ��

��

�Recall x���� in case you need the boundary conditions in separatedform��

�c� In the case that k � m and B� � I� ������ is an initial value delayODE� Describe a method to convert this to a sequence of initialvalue ODEs of increasing size�

�d� Explain why both conversion tricks to standard BVP and to stan�dard IVP forms lose their appeal when � shrinks� i�e� � � b�

�This is a curious thing� because as � � � the delay ODE �����a�becomes �closer to� an ODE system of size m� For more on thistopic see ��� �� �� and references therein��

�� The well�known Newton�Kantorovich Theorem guarantees convergenceof Newton�s method starting at c� for the nonlinear system of algebraic

Page 206: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Chapter � Shooting ���

equations h�c� � �� With the notation J�c� � �h�c� a su cient condition

for convergence is� � ���

where

kJ�c����h�c��k � �

kJ�c����k �

kJ�c�� J�d�k � kc� dk �

Show that for the simple shooting method for ������ ������ we can merelybound � eLb� where L is the Lipschitz constant of f �

�This bound may be realized in practice� and indicates potential troublein the convergence of Newton�s method� unless c� is very close to theexact solution c so that � is very small� The bound on is improved alot when using multiple shooting with uniformly distributed shootingpoints ������

�� For the multiple shooting method we are faced with the challenge ofsolving the linear equations ������ where the matrix A may be largeand sparse if there are many shooting points� A simple way of doingthis involves viewing the equations ����� as a recursion� Thus� we writefor ����a�

cN�� � YN���tN���cN�� � vN���tN���

� YN���tN����YN���tN���cN�� � vN���tN���� � vN���tN���

� � � �

until we can express cN�� in terms of c�� and this is substituted in����b�� The linear system to be solved is then only m�m and can besolved by usual means� This method is called compacti�cation in ����

�a� Carry out the method just outlined for �nding c� i�e�� �nd theformula�

�b� Show that� unfortunately� this method can degrade the stabilityproperties of the multiple shooting method to those of the simpleshooting method� i�e� this method for solving the linear system����� can be unstable�

�c� Discuss the application of this method to the problem of Exercise�� ����

�Note that� compared to simple shooting the method just outlined doeshave improved convergence properties for nonlinear problems��

Page 207: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

��� Chapter � Boundary Value Problems

�� This exercise is concerned with �nding periodic solutions for given ODEproblems� In each case you are required to plot the obtained solutionin phase space and to �nd the length of the period accurate to digits�so eye�balling or trial�and�error would not work well enough for thepurpose of �nding the period�� You are allowed to use any initial valuesoftware and boundary value software you want �including your ownprogram from Exercise ���

�a� Find the period of the heavenly bodies example of Exercise �����Fig� �����

�b� Find the period of the solution of the Predator�Prey Example ����Fig� ����� The initial value used in that example was ���� ����

�c� Find the attracting limit cycle and the period of the Van der Polequation

u�� � ��� u��u� � u� �����

Page 208: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Chapter

Finite Di�erence Methods for

BVPs

As in the previous chapter� we seek numericalmethods for BVPs based on ourknowledge of methods for IVPs� But unlike the previous chapter� here we willnot integrate IVPs� Rather� we consider the suitability of the discretizationsstudied in Chapters �� � and for BVPs� Consider a system of ODEs oforder m�

y� � f�t�y�� � � t � b �����

subject to m two�point boundary conditions

g�y����y�b�� � � � �����

De�ne a mesh �or a sequence of steps� we refer to the entire mesh as �

� f� � t� � t� � � � � � tN�� � tN � bgwith hn � tn � tn�� the nth step size� and consider solving for

y��y�� � � � �yN���yN

with yn the intended approximation of y�tn�� The following observations arestraightforward�

� For BVPs� no particular yn is entirely known before all other meshvalues for y are known� Hence� no di�erence method can be regardedas explicit� So� using what we called in Chapter � an explicit Runge�Kutta method� for instance� o�ers no advantage over using what wasreferred to in the IVP context as implicit Runge�Kutta methods�

� It makes no sense to use multistep methods either� both because thereare really no �past�� known solution values� and because the sparsity

���

Page 209: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

��� Chapter � Boundary Value Problems

structure of the linear system that results is adversely a�ected� com�pared to one�step methods��

� Symmetric� implicit Runge�Kutta methods are natural� because likeBVPs they are indi�erent to the direction of integration� i�e� they actsimilarly for nondecreasing and for nonincreasing modes�

In the sequel we therefore concentrate� with the exception of x���� on sym�metric� one�step methods� As in Chapter �� we start with the midpoint andthe trapezoidal methods�

��� Midpoint and Trapezoidal Methods

We consider below the midpoint method� and leave the parallel developmentfor the trapezoidal method to Exercise ���� Recall that the midpoint methodfor the ODE system ����� reads

yn � yn��hn

� f

�tn�����

��yn � yn���

�� n � �� � � � � N �����

and require also that the boundary conditions be satis�ed�

g�y��yN� � � � �����

In ����������� we havem�N��� algebraic equations for them�N��� unknownmesh values �including the end values�� These equations are nonlinear if f isnonlinear in y� and there are many such equations � it is not unusual to get�� equations for a small ODE system� Their solution is discussed below�Before this we consider an example�

Example � Consider again Example ���� To recall� this is a linear problem of the form ���� with m � ��

A �

�BBB�

� � �

� � �

���� �� ��

�CCCA �

and the exact solution is

y � �u� u�� u���T � u�t� �e��t��� � e���t���� e��t

� � e��� cos t�

�Note that we are discussing �rst order ODEs� For a second order ODE a naturaldiscretization stencil would involve two steps� see Exercises �� ������

Page 210: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Chapter � Finite Di�erence Methods ��

which determines the inhomogeneity vector q�t�� For boundary conditions�u���� u��� and u���� are prescribed� In Tables ��� and ��� we record maximumerrors in u at the mesh points for � � �� � and �� using uniform meshesand specialized� nonuniform meshes� For the uniform meshes� h � ��N �These results are for the midpoint method� similar results are obtained alsofor the trapezoidal method�

N � error rate � error rate � error rate

�� � ���e�� � �� �� ���

�� ��e�� ��� ��� ��� ��� ���

�� ���e�� ��� ���e�� ��� ��� ���

�� ���e�� ��� ���e�� ��� ��� ��

Table ���� Maximum errors for Example ��� using the midpoint method�uniform meshes�

N � error rate � error rate

�� � ��� �� -

�� ��e�� ��� ���e��

�� ���e�� ��� ���e�� ���

�� ���e�� ��� ���e�� ���

Table ���� Maximum errors for Example ��� using the midpoint method�nonuniform meshes�

Note that for � � � the second order accuracy of the midpoint method isre�ected in the computed results� Given the smoothness of the exact solutionit is also clear that there is room for employing higher order methods �seeTable ���� especially if highly accurate trajectories are desired�

For � � �� and even more so for � � ��� the method is much lessaccurate if we use a uniform mesh� and the convergence order is reduced�The reason has already been discussed in Chapter �� O��� errors which aregenerated in the narrow layer regions near the interval ends propagate almostundamped throughout the interval �recall Fig� ���� To retrieve the potentialaccuracy of the midpoint method in regions where the solution varies slowly�the mesh in layer regions must be dense� The nonuniform meshes used for

Page 211: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

��� Chapter � Boundary Value Problems

Table ��� result from a primitive e�ort to handle the layer regions� They aregiven �for N � �� by

���

����

���

�� ��� �� ��� � � �

�� �� �

�� �� �

��� �

and the re�nements obtained by successively subdividing each of the meshelements into two to obtain the next mesh� For � � �� the errors aremeasured only at mesh points away from the layer�

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1−1

−0.5

0

0.5

1

1.5

t

u

exact and approximate u(t)

Figure ���� Example ���� Exact and approximate solutions �indistinguish�able� for � � �� using the indicated mesh�

Even with these simple nonuniform meshes� a signi�cant improvement inthe quality of the solution is obtained� The exact and approximate solutionsfor � � � are plotted in Fig� ���� together with the mesh that was used togenerate these curves� This mesh corresponds to the last entry of Table ����N � ��� The approximate solution is in agreement with the exact one� asfar as the eye can tell� It turns out that for this type of problem it is possibleto construct more sophisticated meshes on which we obtain good� accuratesolutions for any � � � with N independent of �� This is in contrast tomultiple shooting techniques� where N grows linearly with �� �

For solving the many nonlinear algebraic equations we again considerNewton�s method� because it is basic� it is fast when it works well� and itretains the sparsity structure of the Jacobian� which is important for such

Page 212: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Chapter � Finite Di�erence Methods ���

a large system� As it turns out� Newton�s method applied to the midpointequations ����������� is equivalent to the method of quasilinearization coupledwith the midpoint discretization for linear problems� The latter approach hasthe attraction of being more modular� so we describe it next�

����� Solving Nonlinear Problems Quasilinearization

Newton�s method for algebraic equations is obtained by expanding in Taylorseries and truncating the nonlinear terms at each iteration� The quasilin�earization method does the same for the nonlinear di�erential system� Thus�let y��t� be an initial solution pro�le� �a guess�� and write

�y����� � f�t�y�� ��f

�y�t�y���y��� � y��

� � g�y�������y����b��

� g ��g

�u��y������ � y����� �

�g

�v��y����b�� y��b��

where y� � y��t� is a known function at the �th iteration� and g� B� ��g�u

and Bb ��g�vare evaluated at the known iterate �y�����y��b�� on the right

hand side of the last expression� Letting also

A�t� ��f

�y�t�y��t��

we obtain at the �th iteration that the next iterate y��� � y satis�es thelinear BVP

y� � A�t�y� q�t�� � � t � b

B�y��� �Bby�b� � b ����

where

q � f�t�y��t���A�t�y��t�

b � �g�y�����y��b�� �B�y���� �Bby

��b� �

The coe cients in the linear problem ���� may all depend� in general� onthe current iterate y��t�� The quasilinearization procedure therefore de�nesa sequence of linear BVPs whose solutions hopefully converge to that of thegiven nonlinear BVP� Thus� if we know how to discretize and solve linearBVPs then we obtain a method also for nonlinear BVPs�We proceed by applying the midpoint method for the linear problem�

Note that the iterates y��t� are never really needed anywhere other than at

�Here and below we denote iteration number by a simple superscript� e�g� y� for the�th iterate� This should not be confused with the notation for the �th power�

Page 213: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

��� Chapter � Boundary Value Problems

mesh points� It is also easy to verify that the operations of linearization anddiscretization commute here� we obtain the same linear systems to solve aswe would if we apply Newton�s method directly to ������������

Example � Revisiting Example ���� we write the ODE in the �rst orderform ���� for y�t� � �u�t�� u��t��T �

y� �

�� y�

�ey���

�A � � � t � ��

The boundary conditions are linear and homogeneous� They can be writtenas B�y��� �Bby��� � �� with

B� �

��� �

� �

�A � Bb �

��� �

� �

�A �

The Jacobian matrix is apparently

�f

�y�

�� � �

�ey��� �

�A

so at the �th quasilinearization iteration we de�ne

A�t� �

�� � �

�ey�� �t��� �

�A q�t� �

�� y���t�

�ey�� �t���

�A�A�t�y��t�

and solve the linear system ���� with b � � for y � y����t� �Starting with the initial guess

u��t� � c�t��� t�� � � t � �

and employing the midpoint method with a uniform mesh of size N � ��� weobtain convergence after � Newton iterations to goodquality approximationsof each of the two solutions depicted in Fig� ���� upon setting c� � �� andc� � ��� respectively �cf� Example ���� This problem is very easy to solvenumerically� despite its nonunique solutions� �

Instead of solving in the �th quasilinearization iteration for the next it�erate y��� we can �and we prefer to� solve for the Newton direction at y��

��t� � y����t�� y��t�

Page 214: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Chapter � Finite Di�erence Methods ���

and then lety��� � y� � � �

For � �which depends of course on �� we have the linear problem �Exercise����

�� � A�t�� � q�t�� � � t � b

B����� �Bb��b� � b �����

where A� B� and Bb are as before� in ����� but the data simpli�es to

q�t� � f�t�y��� �y���b � �g�y�����y��b�� � �����

Note that in Example ��� we may no longer automatically set b � � whensolving for ��t� � this depends on the initial guess y��t��

The midpoint method applied to the linear problem ���� yields the linearequations

yn � yn��hn

� A�tn�����yn � yn��

�� q�tn������ n � �� � � � � N

B�y� �BbyN � b � �����

This is a large� sparse linear system of m�N � �� equations�

Ay� � r

with

A �

�BBBBBBBBB�

S� R�

S� R�

� � � � � �

SN RN

B� Bb

�CCCCCCCCCA� �����

y� �

�BBBBBBBBB�

y�

y�

���

yN��

yN

�CCCCCCCCCA� r �

�BBBBBBBBB�

q�t����

q�t�������

q�tN�����

b

�CCCCCCCCCA

Page 215: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

��� Chapter � Boundary Value Problems

where

Sn � ��h��n I �

�A�tn�����

�� Rn �

�h��n I � �

�A�tn�����

��

We see that the structure of A is the same as that of A for the multipleshooting method� In fact� to make it even more similar� we can multiply thenth block row of A by R��n � obtaining block rows in the form

� � � R��n Sn I � � ��

withR��n Sn presumably approximating the fundamental solution matrix value�Yn�tn��To summarize� here is the algorithm combining quasilinearization with

the midpoint discretization for a nonlinear BVP ������������

Algorithm � Quasilinearization with Midpoint

� Given�� f � �f

�yfor each t and y�

�� g�u�v�� �g�u� �g�vfor each u and v�

�� a mesh � � � t� � � � � � tN � b�

�� an initial guess y��t�� or just y�n � y��tn�� n � �� �� � � � � N � and

�� a convergence tolerance NTOL for the nonlinear iteration�

� For � � �� �� � � � � until max��n�N jy���n � y�nj � NTOL�

�� For n � �� � � � � N � form Sn� Rn and rn � q�tn����� using

A�tn����� ��f

�y�tn�����

y�n � y�n���

q�tn����� � f�tn�����y�n � y�n��

��� y�n � y�n��

hn�

�� Form A and r of ���� using

B� ��g

�u�y���y

�N�� Bb �

�g

�v�y���y

�N�� b � �g�y���y�N� �

�� Solve the linear system of equations for y� � ���

�� Sety���� � y�� � �� �

Page 216: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Chapter � Finite Di�erence Methods ���

����� Consistency� ��stability and Convergence

The local truncation error� consistency and accuracy of a di�erence methodare de�ned as in x���� There is essentially no dependence in this regardon what type of side conditions are prescribed �be they initial or boundaryconditions� so long as they are approximated well� The question is still byhow much the exact solution fails to satisfy the di�erence equations�For the midpoint method we de�ne

N�u�tn� u�tn�� u�tn���hn

� f

�tn�����

��u�tn��� � u�tn��

so the numerical method is given by

N�y��tn� � �

�with g�y��yN� � ��� By Taylor�s expansion �see Exercise ���� we obtainthat the local truncation error satis�es

dn � N�y�tn� � O�h�n�

so this is a consistent� second order accurate method�The de�nition of convergence is also exactly as in x���� Let

h � max��n�N

hn �

The method is convergent of order p if

en � O�hp�

for n � �� �� �� � � � � N � where en � yn � y�tn�� We expect �nd order conver�gence for the midpoint method�The vehicle that carries accuracy results into convergence statements is

��stability� For nonlinear problems we must con�ne ourselves to a vicinityof an exact� isolated solution �recall Chapter ��� Consider a �discrete tube�around such an exact solution y�t��

S����y� � fu�� jui � y�ti�j � �� � � i � Ng ������

�the notation is for the particular mesh considered� and � � � is the radiusof the tube around y�t��� The rest of the ��stability de�nition is similar tothe IVP case� The di�erence method is �stable if there are positive constantsh�� � and K such that for any mesh with h � h� and any mesh functionsx� and z� in S����y��

jxn � znj � Kfjg�x��xN�� g�z�� zN �j� max

��j�NjN�x��tj��N�z��tj�jg� � � n � N� ������

Page 217: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

��� Chapter � Boundary Value Problems

Substituting xn � y�tn� and zn � yn into ������ we obtain an extension ofthe Fundamental Theorem ��� to the BVP case�

jenj � K maxjjdjj � O�hp�� � � n � N � ������

In particular� the midpoint method is second order convergent� Note alsothat� as in the IVP case� the bound ������ is useful only if K is of the orderof magnitude of the stability constant of the given di�erential problem�How can we show ��stability! Below we consider the linear case� For

the nonlinear problem we consider a linearization� much in the spirit of thequasilinearization method and the variational problem ����� The di�erenceoperator must satisfy certain smoothness and boundedness requirements� andthen the results extend�For the linear BVP ���� the midpoint method ����� has been cast into

matrix form in ������ Obviously� ��stability is obtained if there is a constantK such that for all meshes with h small enough�

kA��k � K �

Indeed� then we would have for the exact solution y�t�� written at meshpoints as

ye � �y����y�t��� � � � �y�tN����y�b��T �

the estimates

Aye � r�O�h��

A�ye � y�� � O�h��

jye � y�j � K O�h�� �

To show ��stability we call upon the closeness of A to the multiple shoot�ing matrix� It is not di cult to see that

R��n Sn � �Yn�tn� �O�h�n� �

Hence� denoting the multiple shooting matrix

M �

�BBBBBBBBB�

�Y��t�� I

�Y��t�� I� � � � � �

�YN �tN� I

B� Bb

�CCCCCCCCCA

Page 218: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Chapter � Finite Di�erence Methods ���

�which is in a slightly� but not meaningfully� di�erent form from A of �������and de�ning the block�diagonal scaling matrix

D �

�BBBBBBBBB�

R���

R���

� � �

R��N

I

�CCCCCCCCCA

we obtain

DA �M�E

where E has the same zero�structure as A and kEk � O�h���

From this we have

A�� � �M�E���D �

Taking norms� and capitalizing on our knowledge of the exact inverse of M�recall �������� we readily obtain

kA��k � ��O�h� K ������

where � is the stability constant of the problem� de�ned in ������� For hsmall enough� the stability bound is therefore quantitative$ If the BVP isstable and not sti�� and the local truncation error is small� then the globalerror is expected to have the order of the local truncation error times thestability constant of the given BVP�

It is important to understand that the closeness just discovered betweenthe midpoint di�erence method and the multiple shooting method is mainlyuseful for theoretical purposes� The placement of shooting points in the lattermethod is done to reduce IVP instabilities� not to control truncation error�which is controlled by the initial value solver�� Thus� the distance betweenshooting points is not necessarily small� If the number of shooting pointsneeded is as large as what is typical for a di�erence method like midpointthen the multiple shooting method becomes rather ine cient� because whereone simple discretization step would do it �res up a whole IVP solver� Also�for sti� BVPs the midpoint method does not use steps so small that R��n Sncan be said to approximate �Yn�tn� well �and wisely so�� The interpretationof the above result is still valid as h� �� as the name *��stability� indicates�

Page 219: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

��� Chapter � Boundary Value Problems

0 5 10 15 20 25 30

0

5

10

15

20

25

30

nz = 198

Figure ���� Zero�structure of the matrix A� m � �� N � ��� The matrix sizeis m�N � �� � ���

��� Solving the Linear Equations

Having discretized a linear BVP using� say� the midpoint method� we obtaina large� sparse linear system of algebraic equations to solve�

Ay� � r ������

with A having the sparsity structure depicted in ������ It is important thatthe reader imagine the structure of this matrix for� say� m � � and N � ���� it is large and rather sparse �only ���� entries out of ����� are possiblynonzero�� In Fig� ��� we depict this structure for more modest dimensions�where zeros are blanked�Of particular concern is the block B� at the lower left corner of A� If it

was not there then we would have a banded system� i�e� all nonzero entriesare concentrated in a narrow band around the main diagonal� Fortunately�the situation for separated boundary conditions is much better than for thegeneral case� just like in x���� If

B� �

��B��

�A � Bb �

�� �

Bb�

�A

where B�� has k rows and Bb� has m� k rows� then we can simply permutethe matrix A� putting the rows of B�� at the top� The right hand side ris permuted accordingly as well� This also establishes a �time� direction �

Page 220: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Chapter � Finite Di�erence Methods ��

0 5 10 15 20 25 30

0

5

10

15

20

25

30

nz = 189

Figure ���� Zero�structure of the permuted matrix A with separated bound�ary conditions� m � �� k � �� N � ���

the lower is the row in the permuted A� the larger is t to which it refers�In Fig� ��� we depict the permuted A corresponding to Fig� ���� where twoboundary conditions are prescribed at t � � and one at t � b�A number of methods which require O�Nm�� �ops �instead of the usual

O�N�m��� to solve the linear system ������ in the case of separated bound�ary conditions have been proposed in the literature� Here we describe thesimplest and crudest of these� and only comment on other methods�Once permuted� the matrixA can be considered as banded� withm�k��

diagonals below the main diagonal and �m � k � � diagonals above themain diagonal possibly having nonzero entries� Outside this total of �m� �diagonals� all entries of A are �� Gaussian elimination with partial pivotingextends to the banded case in a straightforward fashion� Simply� two of thethree nested loops de�ning the elimination process are shortened in order notto eliminate elements known to be � at the start� The �ll�in is only withinthe banded structure� with the addition of a few diagonals due to the partialpivoting� It is not di cult to write a program to carry out this algorithm�Also� there exists standard software to do this� e�g� in Linpack or Lapack�

If you look at the band containing all nonzero elements in the matrixdepicted in Fig� ��� you will notice that there are triangles of zeros withinthe band for each interior mesh point� These zeros are not taken advantageof in the band method just described� Other� more sophisticated methods forsolving ������ attempt to avoid� or at least minimize� �ll�in of these triangles�thereby achieving an additional savings of up to � in both storage andcomputational e ciency�

Page 221: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

��� Chapter � Boundary Value Problems

The fact that all the nonzeros of A densely populate a narrow band as inFig� ��� is typical for boundary value ODEs� where neighboring elements �i�e�subintervals sharing unknowns� can be ordered consecutively� For boundaryvalue PDEs� on the other hand� the band is necessarily much wider and thematrix is sparse inside the band as well� Variants of Gaussian elimination be�come less e�ective then� and iterative methods like preconditioned conjugategradients and multigrid take center stage�

��� Higher Order Methods

The midpoint and trapezoidal methods may be considered as basic methods�The only problem in using them as they are for many applications is that theyare only second order accurate� There are two types of higher order methodsextending the basic methods� higher order Runge�Kutta and accelerationtechniques� The overview picture is given in Fig� ����

symmetric one-sided

acceleration

deferred correctionextrapolation

one-step

basic methods

Figure ���� Classes of higher order methods�

����� Collocation

One class of extensions to higher order methods is simply higher order im�plicit Runge�Kutta methods� Continuing to prefer symmetric methods� thisleads to collocation methods at Gauss or Lobatto points� We have already

Page 222: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Chapter � Finite Di�erence Methods ���

considered the basic properties of these methods in Chapter �� and every�thing else pertaining to BVPs extends in a very similar way to the treatmentin the previous two sections� We summarize without repeating the proof�

� Collocation at s Gaussian points is ��stable with a numerical stabilityconstant satisfying ������� It converges with the error bound

jenj � K maxjjdjj � O�h�s�� � � n � N� �����

Example � We repeat the computations of Example ��� using collocationat � Gaussian points per mesh element� The results are recorded in Tables��� and ����

N � error rate � error rate � error rate

�� � ���e�� � ��e�� �� ���

�� ���e��� ��� ���e�� ��� �� ��

�� ��e��� ��� ���e�� ��� ��� ���

�� ���e��� �� ���e� � ���e�� ���

Table ���� Maximum errors for Example ��� using collocation at � Gaussianpoints� uniform meshes�

N � error rate � error rate

�� � ��e�� �� ��e��

�� ���e�� ��� ���e�� ���

�� ���e� � ��e�� ���

�� ���e�� ��� ���e�� ���

Table ���� Maximum errors for Example ��� using collocation at � Gaussianpoints� nonuniform meshes�

The errors for � � � in Table ��� re�ect the fact that this method is oforder � and has a nice error constant to boot� For � � � the errors are fairlygood even on the uniform mesh� although they are better on the nonuniformmeshes �whose construction is discussed in Example ���� For � � �� anonuniform mesh is certainly needed� see Table ���� In fact� a better layermesh can become useful as well in order to retrieve the full convergence order�which turns out to be � that this method has outside the layer regions� �

Page 223: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

��� Chapter � Boundary Value Problems

Applying quasilinearization to nonlinear problems and considering collo�cation for the resulting linear ones� we obtain a linear system of equations forthe mesh unknowns as well as the internal stages �see ����� or �������� Since�unlike in the IVP case� the solution is not known anywhere in its entiretyuntil it is known at all mesh points� one approach is to solve for all mesh val�ues and internal stages simultaneously� This alters the structure of the linearsystem ������� but A is still in a block form and is banded independently ofN �Alternatively� we eliminate the internal stages locally� in each mesh subin�

terval n� in terms of the mesh values yn�� and yn� This IVP�style approachis called local elimination� or parameter condensation� in the �nite elementliterature� The remaining global system to be solved� ������� has the almostblockdiagonal form ����� independently of s� which adds an attractive mod�ularity to the process� However� the partial decompositions used for thelocal elimination stage have to be stored from one nonlinear iteration to thenext� so the advantage here is in elegance� not in storage or computationale ciency�

����� Acceleration Techniques

The other possibility for extending basic methods to higher order is to staywith the midpoint or the trapezoidal method as the basic discretizationmethod and to accelerate its convergence by applying it more than once�One way of doing this is extrapolation� where the method is applied on

more than one mesh and the results are combined to kill o� the lower orderterms in the error expansion� For instance� if the global error on a givenmesh has the form

en � y�tn�� yn � ch�n �O�h��

where c may vary �slowly� in t but is independent of h� then subdividingeach mesh subinterval into two and applying the same method again yieldsfor the solution &y�n

y�tn�� &y�n ��

�c h�n �O�h��

so ��y�n�yn�

is a �th order accurate approximate solution� This process can berepeated to obtain even higher order methods �Exercise �����Another possibility is deferred correction� where the discretization on

the same mesh is applied a few times� at each instance using the previousapproximation to correct the right hand side by better approximating thelocal truncation error�Unlike extrapolation� which uses the expansion in powers of h for the

global error� defect correction uses the corresponding expansion for the local

Page 224: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Chapter � Finite Di�erence Methods ���

truncation error� For instance� applying the trapezoidal method to ����� weobtain �Exercise ����

dn �rX

j��

h�jn Tj�y�tn������ �O�h�r��n � ������

where

Tj�z�t�� ���

��j����j � ��$f ��j��t� z�t�� ������

if f has continuous partial derivatives up to order �r � � for some positiveinteger r� Now� let y� � fyngNn�� be the obtained solution on a given meshusing the trapezoidal method to discretize the stable BVP ������������ anddenote fn � f�tn�yn�� as in Chapter � Then we can use these values toapproximate T� up to O�h�n�� e�g�

T��y�tn������ � T��n���� � ������

��

��h�n��fn�� � fn�� � fn � fn���� � � n � N � ��

This can be added to the right hand side of the trapezoidal discretization�i�e� we solve

&yn � &yn��hn

��

��f�tn� &yn� � f�tn��� &yn���� � h�nT��n����� � � n � N

g�&y�� &yN� � � �

The local truncation error is now O�h�n�� because the sum in the expressionin ������ starts from j � �� Hence also the global error is �th order�As in the case of extrapolation� the deferred correction process can be

repeated to obtain higher order approximations� Moreover� all approxima�tions are solved for on the same mesh� For a linearized problem� one matrixA must be decomposed� Then� in the ensuing iterations which graduallyincrease the accuracy� only the right hand side vectors are updated� The cor�responding solution iterates are computed each by a pair of forward�backwardsubstitutions� It may look at this point as if we are getting something fromnothing$ The catch� though� is in having to use more and more cumbersomeand accurate approximations to the T �js� The extrapolation method is moreexpensive but simpler�These acceleration methods are useful and important in practice� They

have useful counterparts for IVPs and DAEs as well� Methods of both col�location and the acceleration types just described have been implemented ingeneral�purpose codes� In a broad�brush comparison of the methods� it seemsthat they share many attributes� Methods of the acceleration type seem to

Page 225: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

��� Chapter � Boundary Value Problems

be faster for simple problems� while methods of higher order collocation atGaussian points do a better job for sti� BVPs�

��� More on Solving Nonlinear Problems

Newton�s method �or quasilinearization� converges very rapidly if the �rstiterate is already a su ciently good approximation of the �isolated� solution�This is the typical case for IVPs� even for sti� problems� where the knownvalue of yn�� is only O�hn� away from the sought value of yn� But for BVPsno such high�quality initial iterate is generally available� and getting thenonlinear iteration to converge is a major practical challenge� This makesfor one of the most important practical di�erences between general�purposeIVP and BVP solvers� Below we brie�y discuss some useful approaches� �

����� Damped Newton

For the nonlinear system

h�y�� � �

Newton�s �or the quasilinearization� method at the �th iteration can be writ�ten as solving the linear system�

�h

�y�y���

��� � �h�y���

and forming

y���� � y�� � ��

�see ����� and Algorithm ����� This can be interpreted as taking a step oflength � in the direction ��� If the model on which Newton�s method isbased �which can be viewed as assuming a local quadratic behavior of anappropriate objective function� is too optimistic then a smaller step in thisdirection may be called for� In the damped Newton method we then let

y���� � y�� � �� ������

�Throughout this section we consider a system of nonlinear algebraic equations� andcall the vector of unknowns y� � The somewhat cumbersome index � is there merely toremind us that we seek a mesh function� approximating the solution of the BVP� However�no special properties of the mesh function as such are utilized�

Page 226: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Chapter � Finite Di�erence Methods ���

where the parameter � � � � �� is chosen to ensure a decrease at eachiteration in the objective function� For example� we can require

jh�y���� �j� � ��� ��jh�y���j� ������

where � ensures some minimum decrease� e�g� � � �����It can be shown theoretically that a sequence f �g can be found under

certain conditions �which include non�singularity of �h�ywith a reasonable

bound on the inverse� such that the damped Newton method converges globally� i�e� from any starting iterate y�

�� No such theorem holds for Newton�smethod without damping� which is assured to converge only locally� i�e� withy�� �close enough� to the sought solution y� �recall� e�g�� Exercise �����In practice this technique is useful on some occasions� but it is not su �

cient for really tough BVPs� Typically� in such tough problems the Newtondirection �� is so polluted that it makes no sense to step in that directionfor any step length� There seems to be no easy substitute for the remedy of�nding better initial iterates�

����� Shooting for Initial Guesses

Often users feel more comfortable supplying only guesses for the initial valuesy����� rather than an entire solution pro�le y��t�� This is all that is requiredto �re up a simple shooting method� But if the stability of the BVP issuch that a shooting method may indeed be used then one can instead usean initial value code to solve the IVP once for the guessed initial values�obtaining an initial solution pro�le y��t�� Then a quasilinearization iterationfor a �nite di�erence method may be started�This idea is a trick of convenience� It has obvious limitations� A more

powerful �and more expensive� approach is to develop an appropriate initialsolution gradually� solving a sequence of BVPs� The latter idea is discussednext�

����� Continuation

This approach is powerful and general� We embed the given problem in afamily of problems

��y�� �� � � �� � � � �� ������

where the problem ��y�� ��� � � is easy to solve and ��y�� ��� � h�y���Under suitable conditions this de�nes a homotopy path from an easy problem

Page 227: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

��� Chapter � Boundary Value Problems

to the given problem� which we traverse numerically� Thus� we solve at eachcontinuation step the problem

��y�� � ���� � �

�call its solution y��t�� � ����� given the solution y��t���� where �� is asu ciently small step�size in �� The simplest use of y��t��� is as a �rstiterate y�

��t������ � y��t���� but it is possible to get fancier�This approach can be very successful in practice� although it can become

expensive and it seems hard to automate for really di cult problems� Thebig question is de�ning the family of problems ������� The homotopy pathhas to somehow parameterize the problem well� and automatic choices suchas a simple interpolation between �� and �� typically do not work well inthis sense� Fortunately� there often exists a natural parameterization andembedding of this sort in applications�

Example � Often a nonlinear BVP results from the need to �nd a steadystate solution of a timedependent partial di�erential equation in one spacevariable� Solving the PDE� starting from some initial solution pro�le� canthen be considered as a continuation method for the steady state problem�For instance� consider the di�usion problem of Example ����

�u

�t�

�x

�p�u

�x

�� g�x� u��

For a steady state solution� setting �u�t� � yields the ODE in x

� � �pu��� � g�x� u��

where prime � � denotes di�erentiation with respect to the independent variable x� This ODE is typically equipped with one boundary condition at eachend of the interval in x�Now� solving this nonlinear BVP numerically can be achieved by discretiz

ing the space variable of the PDE while keeping �u�tin� and then applying the

method of lines in t�The timeembedding continuation method is very natural� but it can be

very slow� One can often solve the steady state problem at a tiny fraction ofthe cost of solving the PDE� But this method has the attraction of generality�a straightforward numerical method is applied� regardless of how di�cult thenonlinearity resolution is� Moreover� playing with di�erent initial values maylead to di�erent steady states �in cases where there is more than one suchsolution� perhaps in an intuitive way� �

The continuation technique opens the door to a variety of interestingtopics such as path following� and constructing bifurcation diagrams� butstepping through that door leads us outside the scope of this book� so wemerely give a simple example here�

Page 228: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Chapter � Finite Di�erence Methods ���

Example � The problem considered in Examples ��� and ��� certainlyrequires no fancy means to �nd its two solutions� once we have an idea thatthere are two such solutions and roughly what they look like� But to �ndthem� and much more� automatically� we embed it in the family of problems

u�� � �eu � � ������

u��� � u��� � �

and consider choosing the continuation parameter � � �� As it turns out�continuation in � starting from � � � �where the solution is u � leads tothe stabler of the two solutions for � � e �which is the value of � consideredin Example ���� Continuing with � further� the problem becomes singular atsome � � �� � ��� What happens is that two solutions approach each otheras � increases and then cease to be isolated�A more general continuation procedure uses arclength along the homotopy

path for the continuation parameter �� This is the preferred procedure fora generalpurpose implementation� but it is again beyond the scope of ourpresentation� However� for this example there is also a simple trick� insteadof doing the continuation with � � � of ������ use � � kuk�� Thus� considerthe embedding

u�� � �eu � �

�� � � ������

w� � u�

u��� � u��� � �� w��� � �� w��� � �� �

Carrying out the continuation process for this system ����� from � � � to� � � yields the bifurcation diagram depicted in Fig� ���� where the computation for each � was carried out using a standard BVP code� �Collocation at �Gaussian points was utilized and the problem was solved for ��� equidistantvalues of �� This does not take long � less than a minute in total on anSGI Indigo� R����� Fig� ��� clearly suggests that for � � �� there are twosolutions� for � � �� there is one and for � � �� there are none� The typeof singularity which occurs in this example at � � �� is called a fold� �

��� Error Estimation and Mesh Selection

Another key ingredient to the success of IVP solvers which is lost in BVPsolvers is the ability to control local errors locally� as the solution process

Page 229: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

��� Chapter � Boundary Value Problems

0 0.5 1 1.5 2 2.5 3 3.5 40

1

2

3

4

5

6

7

8

lambda

|| u

||

bifurcation diagram ||u|| vs lambda

Figure ��� Bifurcation diagram for Example �� � kuk� vs ��

proceeds from t � � to t � b� To capitalize on this� IVP solvers oftenabandon controlling the global error en �although this is usually what theuser may want� and control the local truncation error dn instead�In the BVP case there is no compelling reason� in general� not to estimate

the global error en� Such an estimate is compared against user�speci�edtolerances or used to select a new mesh� The process in overview for a givenBVP is to discretize and solve it on a sequence of meshes� where the error inthe solution on the current mesh is estimated and this information is usedto decide what the next mesh should be� in case there is a deemed need fora next mesh� The �rst mesh is a guess�The error estimation can be achieved using a process similar to the one

described for extrapolation methods in x������ For instance� given a midpointor a trapezoidal solution fyng on a mesh and another one f&yjg on a meshobtained by subdividing each element of into two halves� we have

&y�n � yn ��

�ch� �O�h�� �

So

en � y�tn�� yn � �

��&y�n � yn�

and�

&e�n � y�tn�� &y�n � �

��&y�n � yn� �

�Note that we do not get a good error estimate for the �th order extrapolated solution

Page 230: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Chapter � Finite Di�erence Methods ��

For the deferred correction approach the local truncation error is esti�mated as part of the algorithm� A global error estimate can be directlyobtained as well� by solving with the truncation error as the right hand side�It is also possible to construct some cruder indicators for the error en

on a given mesh without recomputing another solution� This can be doneby taking advantage of the form of the error if it has a local leading term�or by considering arclength or other error�monitoring functions� e�g� �en �hknjy�k��tn�j for some � � k � p� with p the order of accuracy� Such monitorfunctions may be su cient to select a mesh� even if they do not provide agenuine� reliable error estimate�Given such an error estimate or indicator� the next mesh is selected based

on the principle of error equidistribution� where one attempts to pick the meshsuch that the resulting solution will satisfy

j�eij � j�ejj� � � i� j � N �

This essentially minimizes maxn j�enj for a given mesh size� The mesh size Nis further selected so that

maxnj�enj � ETOL

for a user�speci�ed error tolerance�

Reader�s advice� The technical level and expertise required ofthe reader for the next section is a touch higher than what hasbeen required so far in this chapter� and it gets even higher inx���� But these sections are important and are worth the extrae�ort�

��� Very Sti� Problems

As in the IVP case we expect the midpoint �or trapezoidal� method to berobust when large eigenvalues with both negative and positive real parts arepresent� so long as the layers are resolved� We have already seen a demon�stration in Examples ��� and ���� The properties of symmetric methods areessentially similar for the IVP and the BVP cases�

�� ���y�n � yn�� The error estimates are tight only for the lower order approximations�Similarly� using midpoint solutions on � meshes it is possible to obtain a �th order ap�proximation with an error estimate or a �th order approximation without a tight errorestimate�

Page 231: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

��� Chapter � Boundary Value Problems

For sti� initial value problems we prefer methods with sti� decay� suchas BDF or collocation at Radau points� Unfortunately� it is not possible toattain this property automatically in the BVP case� because methods withsti� decay cannot be symmetric�For the IVP test equation

y� � �y� Re��� � �we prefer� when hRe���� ��� to use a method like backward Euler�

yn � yn�� � h�yn �

Similarly� changing the direction of integration for the unstable IVP y� � ��yto � � tn � t� we get

d&y

d�� �&y

and applying backward Euler to the equation in &y readily yields the forwardEuler method for the original�

yn � yn�� � h�yn�� �

For the system

y� �

��� �

� ��

�Ay

it then makes sense to use upwinding�

y��n � y��n�� � h�y��n

y��n � y��n�� � h�y��n���

For a general sti� problem� unfortunately� the increasing and decreas�ing modes are coupled together� and there are also slow solution modes forwhich a higher order symmetric discretization method is perfectly suitable�Consider the general linearized di�erential system related to �����

y� � A�t�y� q�t�

where A�t� � �f�y� and de�ne the transformation for some nonsingular� su �

ciently smooth matrix function T �t��

w � T��y�

�The term upwind originates from computational �uid dynamics� where the directionof stable integration corresponds to the upwind �i�e�� against the wind� direction of the�ow� This type of discretization has also been called upstream� a name naturally arisingfrom applications where what �ows is liquid� See Exercise �����

Page 232: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Chapter � Finite Di�erence Methods ���

Then w�t� satis�es the ODE

w� � �T��AT � T��T ��w � T��q �

Now� if T is such that the transformed matrix A can be written in blockform

T��AT � T��T � �

�BBB�B� � �

� B� �

� � B�

�CCCA

where

� B� is dominated by eigenvalues with large negative real parts�

� B� is dominated by eigenvalues with large positive real parts�

� kB�k is not large

then we can use backward Euler for w�� forward Euler for w� and the trape�zoidal method for w�� where wi corresponds to the equations involving Bi

and wT � �wT� �w

T� �w

T� �� The equations resulting from such a discretization

need not be solved for w� Rather� the back�transformation from w to y isused to transform them into di�erence equations for y�

Example The stable BVP

y� �

�� cos t sin t

� sin t cos t

�A�� � �

�� ��

�A��cos t � sin tsin t cos t

�Ay

y���� � �� y���� � �

is sti� when Re������� Applying forward Euler or backward Euler to thisODE with a step size h yields disastrous results when hRe��� � ��� But forw � T��y� where

T �t� �

�� cos t sin t

� sin t cos t

�A

we obtain the decoupled system

w� �

��� �

� ��

�Aw

Page 233: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

��� Chapter � Boundary Value Problems

and the upwind method described above can be applied� yielding a very stablediscretization method� We write it in matrix form as�

��� h� �

� �

�Awn �

��� �

� � � h�

�Awn�� �

De�ning for each n

yn � Tnwn �

�� cos tn sin tn

� sin tn cos tn

�Awn

the obtained upwind method for y� is���� h� �

� �

�A��cos tn � sin tnsin tn cos tn

�Ayn �

��� �

� �� h�

�A��cos tn�� � sin tn��sin tn�� cos tn��

�Ayn��� � � n � N

with

y��� � �� y��N � ��

In Fig� ��� we display the approximate solution using the upwind method justdescribed and a uniform mesh with � � ������ h � ���� We also display the�exact� solution� obtained using the code colnew� employing collocation at �Gaussian points per subinterval on a nonuniform mesh with �� subintervals��The code selects the mesh automatically� satisfying a global error tolerance of��e�� Note that despite the fact that the boundary layers are totally missed�i�e�� skipped over� the solution values at mesh points are approximated wellby the upwind method� in analogy to the IVP case depicted in Fig� ���� �

The upwind discretizationmethod outlined above� and other similarmeth�ods of higher order� work very well for special classes of problems �e�g�� gasdynamics in PDEs�� Layer details can be skipped� as with the backwardEuler method in Chapter �� But �nding the transformation T in generaland applying it in practice are major obstacles� For linear problems thereare recipes for this� but the general case involves such an additional amountof work that straightforward collocation at Gaussian points is often moree cient� Things are worse for nonlinear problems� where the entire processis di cult because the linearization is based on unknowns and a mistake inthe sign of a fast mode is a disaster akin to simple shooting� �Alternatively�

Page 234: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Chapter � Finite Di�erence Methods ���

0 0.2 0.4 0.6 0.8 1

0

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

1.8

2

t

y_1

Figure ���� Solution for Example ��� with � � ����� using an upwinddiscretization with a uniform step size h � ��� �solid line�� The �exact�solution is also displayed �dashed line��

upon using quasilinearization� stable but entirely wrong linear problems aresolved� so the nonlinear iteration may not converge��For symmetric di�erence methods no explicit decoupling transformation

is usually needed� But� as mentioned earlier� the lack of sti� decay can beevident in computations� For the test equation

y� � �y

the midpoint method yields� to recall�

yn �� � h�

� � h�yn���

So� although jynj � jyn��j precisely whenever the exact solution satis�esjy�tn�j � jy�tn���j� as hj�j � � we get

yn � �yn�� �

Thus� if y��� � � and hRe���� �� then the exact solution satis�es y�h� � ��yet for the numerical solution�

yn � ����n� n � �� � � � � N �

This not only necessitates covering layer regions �where the solution variesrapidly� with dense meshes� as we have already seen in Examples ��� and ����

Page 235: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

��� Chapter � Boundary Value Problems

it also means that in the very sti� case local errors propagate through smoothsolution regions �where h is relatively large� almost undamped� yielding non�local error e�ects�There are some exotic examples where a symmetric method may even

blow up when approximating a stable BVP or even a stable IVP �see Ex�ercise ������ These seem to be rare in practice� though� Moreover� andperhaps more importantly� since there is almost no quadrature e�ect when� � j���j � � in a stable problem of the form

y� � ��y � q�t��

�note that the solution is y�t� � q�t��O�j���j� and the quadrature precisiona�ects only the O�j���j� term�� methods which are based on high�precisionquadrature may experience order reduction away from collocation points� seex����� and Exercise ����� For collocation at s Gaussian points we can get�assuming no layer errors and � � jRe���j�� � h� �� the error bound

y�tn�� yn � O�hs� �

This error estimate improves to

y�tn�� yn � O�hs���

if s is odd and some mesh restrictions apply� But this still falls short of theusual nonsti� order O�h�s� when s � �� For Example ���� the e�ective orderis s� � � � �instead of �s � �� when h� is very large�Despite their limitations� as candidates for constructing a general�purpose

solver symmetric discretizations seem to win� It appears that upwinding tech�niques should be reserved for special problems where the explicit decouplingof modes can proceed with relative ease � see Exercises ���� and �����

��� Decoupling

The concept of decoupling of modes of distinctly di�erent types is fundamen�tal to the understanding of numerical discretizations for di�erential equa�tions� But it seems to be particularly important in the context of boundaryvalue ODEs� so we brie�y discuss it here� This section can be viewed as aroad map for numerical methods for BVPs�As we have seen in x��� a stable linear BVP must have a dichotomy� i�e� a

certain number of its fundamental modes are nonincreasing and the rest arenondecreasing� throughout the interval ��� b� on which the di�erential problem

Page 236: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Chapter � Finite Di�erence Methods ���

is de�ned� Slow modes can be grouped either with the rapidly increasing orwith the rapidly decreasing modes� but this is possible only locally� i�e� amode can change from fast to slow and vice versa in di�erent subintervals of��� b��What must be avoided then is a numerical integration of fast modes in

the direction of their increase� When the modes are decoupled� as e�g� in thesystem

y� �

��� �

� ��

�Ay ������

where Re��� � � and j�j is not much larger than j�j� then the modes can beintegrated in suitable directions � for ������ the second ODE is integratedfrom � to b and then the �rst is integrated from b to �� In the general casewhere the modes are not decoupled� some decoupling must be applied� Thesimple shooting method does not apply any decoupling� hence it is unstablein the presence of fast �not even very fast� increasing and decreasing modes�For the multiple shooting method discussed in x��� and the �nite dif�

ference methods discussed in the early sections of this chapter� stability isproved provided that su ciently many shooting points or mesh points areused� This is the limit where these methods are all similar to one another andbounds like ������ apply� indicating that the discretization follows the con�tinuous system closely� The decoupling of modes is then achieved implicitly�through the solution of the linear system of algebraic equations� Recall fromx��� that this system has an almost block diagonal form �Fig� ���� corre�sponding to the sequential ordering of the mesh points� So� when performingLU �decomposition and forward and backward substitutions in order to solvea system like ������� we are in e�ect sweeping forward and then backwardalong the interval of integration� The LU �decomposition itself can be seento correspond to a decoupling transformation along the lines given by thedichotomy bound �������The great robustness of symmetric di�erence methods arises from the pos�

sibility of achieving the decoupling e�ect implicitly� i�e� without an explicittransformation� even for sti� BVPs� But for very sti� problems a methodlike midpoint also tends to transform fast increasing and decreasing modesinto slower ones� when the step size is not very small�Some unfortunate e�ects may also result when fast and slow modes are

not suitably decoupled by the numerical method� This may occur alreadyfor stable IVPs �Exercise ���� is a case in point� but such problems are rarerin practice and� moreover� the decoupling must be done locally� hence explic�itly� as described in x���� which is not very practical for many applications�Trouble can happen also for DAEs when di�erent solution components arenot properly decoupled� as we will see in the next two chapters�

Page 237: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

��� Chapter � Boundary Value Problems

��� Software Notes and References

����� Notes

Much of the early theoretical development of the theory of numericalmethodsfor boundary value problems was done by H� Keller and appears in ��� aswell as in the more modern reference book ����

A lot of work was done on the numerical solution of second order two�point boundary value problems� Often a single ODE is considered in thiscontext� Many papers on numerical methods for sti� BVPs of this sorthave appeared� probably both because of the relevance to advection�di�usionPDEs where advection dominates and because of the relative tractability ofthese problems compared to the general sti� system case� We have devoteda series of exercises to this �Exercises ��� � ������ and refer for more to ���and the references therein�

But for our main exposition we consider the general ODE system case�which naturally extends our discussion in the previous IVP and shootingchapters� All of the material covered in this chapter� including proofs andreferences which we have omitted here �plus� be warned� much more$� canbe found in ����

The linear system solvers used in all the leading software are more so�phisticated than the band solver that we have described� See Chapter � of ���and references therein� For the parallel solution of such systems� see Wright�����

A thorough treatment of discretization methods and their asymptoticexpansions can be found in ����� See also the early book ����� V� Pereyramade fundamental contributions to the theory of deferred corrections� Animportant work in the early development of collocation at Gaussian pointsis de Boor Swartz ����� although the later treatment in ��� is cleaner�

The earliest uses of the principle of error equidistribution seem to havebeen made in de Boor ���� see also ����� �The apparent addition of theword *equidistribution� to the English language is due to M� Lentini and V�Pereyra��

Major contributions on decoupling in BVPs were made in the �����s byR� Mattheij and appear in ����

In Chapter �� of ��� there is a brief description plus relevant referencesof a number of topics which we have omitted here� except in the occasionalexercise� These include eigenvalue problems� singular BVPs� BVPs on in�niteintervals� singular points� bifurcation and arclength continuation� and highlyoscillatory BVPs�

Finally� while we have treated �nite di�erence methods exclusively in thisbook� there has been much theoretical development on �nite elementmethodsas well �see� e�g�� ���� ����� The power of the latter methods� however� appears

Page 238: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Chapter � Finite Di�erence Methods ���

to be more pronounced in the PDE context�

����� Software

Most general�purpose codes for BVPs which are publicly available use themethods described in this chapter�

� The code colsys by Ascher� Christiansen Russell �� and its newerversion colnew by Bader Ascher ���� use collocation at Gaussianpoints� This code is available from netlib�

� Also available from netlib is the code twpbvp by Cash Wright����� which uses deferred correction in combination with certain non�collocation Runge�Kutta methods called mono�implicit� which we havenot covered in this book�

� The NAG library contains the code pasvar by Lentini Pereyra ����which also uses deferred correction� This code has been in�uential formany years�

� The code auto� by Doedel Kernevez ����� which does bifurcationanalysis and �nds periodic solutions� is based on Gauss collocation�

��� Exercises

�� Show that the formulation of the quasilinearization method using � asde�ned in ����������� is equivalent to the formulation using �����

�� Carry out the development of theory and practice as in x��� for thetrapezoidal method ������ instead of the midpoint method�

�� �a� Write down the quasilinearization �or the linearization� problem����� for the BVP �������

�b� Show that this linearized problem is singular �i�e� it does not havea unique solution� when it is carried out about u �� Concludethat starting the quasilinearization iteration with the initial guessu� � is unwise in this example�

�� It can be shown that the error when applying the trapezoidal methodto a su ciently smooth BVP ����������� has the expansion

en � y�tn�� yn �lX

j��

cjh�jn �O�h�l��� �����

Page 239: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

��� Chapter � Boundary Value Problems

where h � maxn hn on a mesh which satis�es

h�minn

hn � constant�

The functions cj are independent of the mesh � Just how large l isdepends on the smoothness of the problem� and we assume l � ���a� Construct a method of order � using extrapolation� based on the

trapezoidal method�

�b� Apply this extrapolation method to the problem of Examples ���and ���� using the same parameter values and meshes� Comparewith collocation at � Gaussian points �Example ����� What areyour conclusions!

� Use your code from the previous exercise� or any available softwarebased on the methods discussed in this chapter� to solve the followingproblems to about �digit accuracy�

�a� Find a nontrivial solution for the problem ������ of Exercise ����

�b� Find the attracting limit cycle and the period of the Van der Polequation ������

�c� Solve ������ for � � �� What is the corresponding value of �!

�� The injected �uid �ow through a long� vertical channel gives rise to theBVP

u���� � R �u�u�� � uu����

u��� � u���� � �

u��� � �� u���� � �

where u is a potential function and R is a given �constant� Reynoldsnumber�

Use your code from the previous exercise� or any available software �wesuggest that it be based on the methods discussed in this chapter�� tosolve this problem for � values of R� R � ��� ���� ���� and R ���� ���� Observe the increased di culty� due to a boundary layer nearthe left boundary� as R increases�

�� Consider the following particle di�usion and reaction system�

T �� ��

tT � � ���Ce����T

���

C �� ��

tC � � ��Ce����T

���

Page 240: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Chapter � Finite Di�erence Methods ��

where C�t� is the concentration and T �t� is the temperature� Repre�sentative values for the constants are � ��� � ����� � � ������ Theboundary conditions at t � � are

T ���� � C ���� � ��

Use any available software �we suggest that it be based on the methodsdiscussed in this chapter�� to solve this problem for the following setsof additional boundary conditions�

�a� T ���� � C ���� � ��

�b� �T ���� � ��T ��� � ��� �C ���� � ���C��� � ��� with � � � �� ����

�This case may cause you more grief� Note that there is a thinboundary layer near t � ���

�� �a� Show that the error expansion ������������� holds for the trape�zoidal method�

�b� The centered approximation ������ is not good near the boundary�e�g� for n � �� Construct one�sided� �nd order accurate di�erencemethods for near�boundary points�

�c� To what order should T� and T� be approximated in order toachieve a �th order deferred correction method! Explain why youneed the �th order &yn� not just the �nd order yn� to constructsuch higher order approximations for the truncation error terms�

�� Consider the scalar ODE of order �

��a�t�u��� � b�t�u � q�t� ������

u��� � b�� u��� � b�

where a � �� b � ���t� We convert this ODE into a �rst order systemwithout di�erentiating a by

u� � a��v

v� � bu� q�

�a� Show that if we discretize the �rst order system by the midpointmethod we obtain a �diagonal matrix A�

�b� Consider instead a staggered midpoint method� on a uniform mesh�the equation for u is centered at tn���� and the equation for v iscentered at tn� with u� de�ned at mesh points and v� at midpoints�

�un � un����h � a���tn�����vn�����vn���� � vn������h � b�tn�un � q�tn��

Page 241: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

��� Chapter � Boundary Value Problems

Show that by eliminating the v�values we obtain for the meshvalues in u a tridiagonal matrix A� Under what condition are weassured that it is diagonally dominant!

�c� The usual ��point formula for discretizing ������ becomes �rstorder if the mesh is no longer uniform� Generalize the staggeredmidpoint method developed above to obtain a �nd order accurate���point method for u on an arbitrary mesh�

�Hint� You can use quadratic interpolation of three adjacent meshvalues of u without changing the sparsity structure of A��

�d� Try your method on the problem given by a � ��t�� b � �� u�t� �sin�t� �calculate the appropriate q�t� and b required for this exactsolution�� Compute maximum errors on three meshes�

� a uniform mesh with h � ����

� a uniform mesh with h � ����

� a nonuniform mesh with ��� subintervals� The step sizes areto be chosen by a random number generator� scaled and trans�lated to lie between ��� and ����

What are your observations!

��� For the second order ODE system

y�� � f�t�y�

we can consider the linear ��point methods

��yn�� � ��yn � ��yn�� � h���fn�� � �fn � �fn��� ������

where we use the notational convention of Chapter and set �� ��� �Note that these methods are compact� the order of the di�erenceequation is �� just like the order of the ODE� so there are no parasiticroots for the stability polynomial here��

�a� Derive order conditions �as in x��� for ��������b� Show that to obtain a consistent method �with a constant h� we

must set �� � ��� �� � �� as in the usual discretization for y���and

� � � � � � � �

�c� Show that to obtain a second order method we must set in addition

� � � �

In particular� the usual formula with � � � � � and � � � issecond order accurate�

Page 242: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Chapter � Finite Di�erence Methods ���

�d� Show that Cowell�s method

yn�� � �yn � yn�� �h�

���fn�� � ��fn � fn��� ������

is �th order accurate�

�e� Describe in detail an implementation of the method ������ for theDirichlet BVP� where y��� and y�b� are given�

��� Consider the scalar Dirichlet problem

��u�� � au� � q�t�

u��� � b�� u��� � b�

where a �� � is a real constant and � � � � �� Assume for simplicitythat there are no boundary layers �i�e� the values b� and b� agreewith the reduced solution satisfying u� � q�a�� and consider ��pointdiscretizations on a uniform mesh with step size h � �

N���

�nun � nun�� � nun�� � qn� � � n � N

u� � b�� uN�� � b� �

The solution for u� � �u�� � � � � uN�T requires solving a linear tridiago�nal system with the matrix

A �

�BBBBBBBBB�

�� � ��� �� � �

� � � � � � � � �

�N�� �N�� � N���N �N

�CCCCCCCCCA

It is desirable that A be diagonally dominant� A related� importantrequirement is that the method be positive�

�n � �� n � �� n � �� �n��This implies a discrete maximum principle which yields stability��

�a� A symmetric� or centered �nd order discretization is given by

h���un�� � �un � un��� �

a

�h�un�� � un��� � q�tn� �

Show that A is diagonally dominant and the method is positive ifand only if

R �jajh�� ��

�R is called the mesh Reynolds number��

Page 243: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

��� Chapter � Boundary Value Problems

�b� An upwind method is obtained by replacing the discretization ofu� with forward or backward Euler� depending on sign�a��

h���un�� � �un � un��� �

a

h�n � q�tn�

�n �

un�� � un a � �

un � un�� a � �Show that this method is positive and A is diagonally dominantfor all R � �� It is also only �rst order accurate�

�c� Show thata

h�n � a

un�� � un���h

� �n

where �n is the ��point discretization of �hjaju��� The upwindmethod can therefore be viewed as adding an O�h� arti�cial diffusion term to the centered discretization�

��� This exercise continues the previous one�

�a� Extend the de�nitions of the centered and upwind ��point dis�cretizations to the ODE

��u�� � a�t�u� � b�t�u � q�t�� � � t � �

where a�t� varies smoothly and can even change sign on ��� ���and b�t� is a smooth� bounded function� What happens whena�t� � t� �

�and � � �� h� �! What happens when there are

boundary or turning�point layers!

�b� Extend the de�nitions of centered and upwind ��point discretiza�tions to the nonlinear problem

��u�� � uu� � b�t�u � q�t�� � � t � �

u��� � ��� u��� � ��

�c� When R � � the centered method is preferred because of its ac�curacy� When R � � the upwind method has superior stabilityproperties� Design a method which mixes the two and graduallyswitches between them� adding at each mesh point just enougharti�cial di�usion to achieve positivity� for any values of �� h� anda or u�

��� �a� Write down the midpoint method� on an arbitrary mesh � for thescalar ODE

y� � ��y � q�t��� � � t � b

Page 244: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Chapter � Finite Di�erence Methods ���

where q�t� is a smooth� bounded function and �� ��� Considerthe IVP case with �hn � ��� where h is the maximum step size�h � max��n�N hn�� Assume no initial layer� so jy��j and highersolution derivatives are bounded independently of ��

�b� Show that the local truncation error satis�es

dn � �znhn��y���tn� �O�hn�� �O�h�n�

where zn � �hn� and that the global error en � y�tn��yn satis�es

en � ��� zn������� � zn���en�� � ��� zn���

��hndn� � � n � N�

�c� Letting zn � ��� � � n � N � show that the global error satis�es

en �nX

j��

����j h�j

��y���tj� �O�hj���

This is the leading error term for the case zn � ��� � � n � N �

�d� Conclude that �for b � O���� the error for the midpoint methodin the very sti� limit reduces to O�h�� However� if the mesh islocally almost uniform� i�e� the steps can be paired such that foreach odd j

hj�� � hj�� �O�hj��

then the convergence order is restored to O�h���

�This mesh restriction is mild� take any mesh� and double it asfor extrapolation by replacing each element by its two halves� Theresulting mesh is locally almost uniform� Note� on the other hand�that even when the second order accuracy is thus restored� thereis no error expansion of the type utilized in x�������

�e� Can you guess why we have included this exercise here� ratherthan in Chapters � or �!

��� Consider the initial value problem��� �t� �

�Ay� �

���� � � t

��� � t�

�Ay �

�� �

sin t

�A ������

y��� � ��� �T

where the two parameters and � are real and � � �� ��

Page 245: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

��� Chapter � Boundary Value Problems

�a� Apply the transformation��� �t� �

�Ay � w

to show that this problem is stable and to �nd the exact solution�

�b� Let � � ������ There is no initial layer� so consider applying themidpoint method with a uniform step size

h � ���max�jj� ��

to ������� Calculate maximum errors in y� for � �� ���� �����What are your observations!

�c� Attempt to explain the observed results� �This may not be easy��

Page 246: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Chapter

More on Di�erential�Algebraic

Equations

In this chapter and the next we study di�erential�algebraic equations �DAEs��already introduced in x���� Here we consider the mathematical structureof such systems and some essential analytical transformations� Numericalapproaches and discretizations are discussed in the next chapter� But here�too� our motivation remains �nding practical computer solutions� Comparedto Chapters � and � this chapter is unusually long� One reason is that DAEtheory is much more recent than ODE theory� As a result DAE theory ismore in a state of �ux� and good expositions are scarce� More importantly�understanding the principles highlighted here is both essential for� and willget you a long way towards� constructing good numerical algorithms�To get a taste of the similarity and the di�erence between DAEs and

ODEs� consider two functions y�t� and z�t� which are related on some interval��� b� by

y��t� � z�t�� � � t � b �����

and the task of recovering one of these functions from the other� To recoverz from y one needs to di�erentiate y�t� � an automatic process familiar to usfrom a �rst calculus course� To recover y from z one needs to integrate z�t� �a less automatic process necessitating also an additional side condition �suchas the value of y�����This would suggest that di�erentiation is a simpler� more straightforward

process than integration� On the other hand� though� note that y�t� is gen�erally a smoother function than z�t�� For instance� if z�t� is bounded but hasjump discontinuities then y�t� is once di�erentiable � see Fig� ����Thus� integration is a smoothing process while di�erentiation is an anti

smoothing process� The di�erentiation process is in a sense unstable� � al�

�If we add to y�t� a small perturbation cos t� where jj � � and � j��j� then z�t�is perturbed by a large amount jj�

���

Page 247: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

��� Chapter �� Di�erential�Algebraic Equations

0 0.5 1 1.5 2 2.5−0.5

0

0.5

1

1.5

t

y

�a� y�t�

0 0.5 1 1.5 2 2.5−1.5

−1

−0.5

0

0.5

1

1.5

t

z

�b� z�t� y��t�

Figure ���� A function and its less smooth derivative�

though it is often very simple to carry out analytically�A di�erential equation involves integration� hence smoothing� the solution

y�t� of the linear system y� � Ay�q�t� is one derivative smoother than q�t��A DAE� on the other hand� involves both di�erentiations and integrations�The class of DAEs contains all ODEs� as well as the problems in Example��� below� But it also contains problems where both di�erentiations andintegrations are intertwined in a complex manner� and that�s when the funreally starts� simple di�erentiations may no longer be possible� but theire�ect complicates the numerical integration process� potentially well beyondwhat we have seen so far in this book�

��� Index and Mathematical Structure

Since a DAE involves a mixture of di�erentiations and integrations� one mayhope that applying analytical di�erentiations to a given system and eliminat�ing as needed� repeatedly if necessary� will yield an explicit ODE system forall the unknowns� This turns out to be true� unless the problem is singular�The number of di�erentiations needed for this transformation is called theindex of the DAE� Thus� ODEs have index �� We will re�ne this de�nitionlater� but �rst let us consider some simple examples�

Example � � Let q�t� be a given� smooth function� and consider the follow

Page 248: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Chapter �� More on Di�erential�Algebraic Equations ���

ing problems for y�t��

� The scalar equationy � q�t� �����

is a �trivial index� DAE� because it takes one di�erentiation to obtainan ODE for y�

� For the systemy� � q�t�

y� � y�� �����

we di�erentiate the �rst equation to get

y� � y�� � q��t�

and theny�� � y��� � q���t� �

The index is � because two di�erentiations of q�t� were needed�

� A similar treatment for the systemu � q�t�

y� � u�� �����

necessitates � di�erentiations to obtain an ODE for y�� hence the indexis ��

Note that whereas m initial or boundary conditions must be given tospecify the solution of an ODE of size m� for the simple DAEs of Example��� the solution is completely determined by the right hand side� Morecomplicated DAE systems will usually include also some ODE subsystems�Thus� the DAE system will in general have l degrees of freedom� where l isanywhere between � and m�In general it may be di cult� or at least not immediately obvious� to

determine which l pieces of information are needed to determine the DAEsolution� Often the entire initial solution vector is known� Initial or boundaryconditions which are speci�ed for the DAE must be consistent� In otherwords� they must satisfy the constraints of the system� For example� aninitial condition on the index�� system ����� �which is needed if we write itas an ODE� must satisfy y���� � q���� For the index�� system ������ thesituation is somewhat more complicated� Not only must any solution satisfy

Page 249: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

��� Chapter �� Di�erential�Algebraic Equations

the obvious constraint y� � q�t�� there is also a hidden constraint y� � q��t�which the solution must satisfy at any point t� so the only consistent initialconditions are y���� � q���� y���� � q����� This is an important di�erencebetween index�� and higherindex �index greater than �� DAEs� higherindex DAEs include some hidden constraints� These hidden constraints arethe derivatives of the explicitly�stated constraints in the system� Index��systems include hidden constraints which are the �rst derivative of explicitly�stated constraints� Higher�index systems include hidden constraints whichcorrespond to higher�order derivatives� for example� solutions to the index��system ����� must satisfy the hidden constraints u� � q��t� and y� � q���t��The most general form of a DAE is given by

F�t�y�y�� � � ����

where �F��y� may be singular� The rank and structure of this Jacobianmatrix may depend� in general� on the solution y�t�� and for simplicity wewill always assume that it is independent of t� Recall also from x��� theimportant special case of a semiexplicit DAE� or an ODE with constraints�

x� � f�t�x� z� ����a�

� � g�t�x� z�� ����b�

This is a special case of ����� The index is one if �g��z is nonsingular�because then one di�erentiation of ����b� yields z� in principle�� For thesemi�explicit index�� DAE we can distinguish between di�erential variablesx�t� and algebraic variables z�t�� The algebraic variables may be less smooththan the di�erential variables by one derivative �e�g� the algebraic variablesmay be non�di�erentiable��In the general case each component of y may contain a mix of di�erential

and algebraic components� which makes the numerical solution of such high�index problems much harder and riskier� The semi�explicit form is decoupledin this sense� On the other hand� any DAE ���� can be written in the semi�explicit form ����� but with the index increased by one� upon de�ning y� � z�which gives

y� � z ����a�

� � F�t�y� z�� ����b�

Needless to say� this re�writing alone does not make the problem easier tosolve� The converse transformation is also possible� given a semi�explicitindex�� DAE system ������ let w� � z� It is easily shown that the system

x� � f�t�x�w�� ����a�

� � g�t�x�w�� ����b�

�Note that a dierentiation of a vector function counts as one dierentiation�

Page 250: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Chapter �� More on Di�erential�Algebraic Equations ��

is an index�� DAE and yields exactly the same solution for x as ������ Theclasses of fully implicit index�� DAEs of the form ���� and semi�explicitindex�� DAEs of the form ����� are therefore equivalent�It is important to note� as the following example illustrates� that in general

the index depends on the solution and not only on the form of the DAE�This is because the local linearization� hence the partial derivative matrices�depend on the solution�

Example � � Consider the DAE system for y � �y�� y�� y��T �

y�� � y�

� � y���� y��

� � y�y� � y���� y��� t�

The second equation has two solutions y� � � and y� � �� and it isgiven that y��t� does not switch arbitrarily between these two values �e�g�another equation involving y�� and y

�� is prescribed with y���� given� implying

continuity of y��t��

�� Setting y� � �� we get from the third equation y� � t� Then fromthe �rst equation y� � y���� � t���� The system has index � and thesolution is

y�t� � �y���� � t���� �� t�T �

Note that this is an index� system in semiexplicit form�

�� Setting y� � �� the third equation reads y� � t� Then� upon di�erentiating the �rst equation� y� � �� The system has index � and the solutionis

y�t� � �t� �� ��T �

Note that� unlike in the index� case� no initial value is required�

If we replace the algebraic equation involving y� by its derivative andsimplify� we obtain the DAE

y�� � y� �����

y�� � �

� � y�y� � y��� � y��� t �

Now the index depends on the initial conditions� If y���� � � the index is ��and if y���� � � the index equals ��

We are ready to de�ne the index of a DAE�

Page 251: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

��� Chapter �� Di�erential�Algebraic Equations

De�nition � � For general DAE systems ����� the index along a solutiony�t� is the minimum number of di�erentiations of the system which would berequired to solve for y� uniquely in terms of y and t �i�e� to de�ne an ODEfor y� Thus� the index is de�ned in terms of the overdetermined system

F�t�y�y�� � �

dF

dt�t�y�y��y��� � �

���dpF

dtp�t�y�y�� � � � �y�p���� � � ������

to be the smallest integer p so that y� in ����� can be solved for in terms ofy and t�

We note that in practice� di�erentiation of the system as in ������ israrely done in a computation� However� such a de�nition is very useful inunderstanding the underlying mathematical structure of the DAE system�and hence in selecting an appropriate numerical method�

Example � � The computeraided design of electrical networks involves simulations of the behavior of such networks in time� Electric circuits are assembled from basic elements such as resistors� diodes� inductors� capacitorsand sources� Large circuits can lead to large DAE systems�A circuit is characterized by the type of elements it has and by its net

work�s topology� For each element there is a relationship of the voltage dropbetween the nodes of the element to the current� For instance� a linear resistor satis�es� by Ohm�s law�

U � RI

where U is the potential drop� I � Q� is the current �Q is the charge� andR is the resistance� for a linear inductor

U � LI �

where L is the inductance� and for a linear capacitor

I � CU �

where C is the capacitance� There are nonlinear versions of these too� e�g�L � L�I� for a currentcontrolled inductor or C � C�U� for a voltagecontrolled capacitor�The network consists of nodes and branches �it is a directed graph and

its topology can be encoded in an incidence matrix A� The �i� j�th entry of Ais � if current �ows from node i into branch j� �� if current �ows in branch

Page 252: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Chapter �� More on Di�erential�Algebraic Equations ���

j towards node i� and � if node i and branch j are not adjacent� Thus� Ais typically large and very sparse� Let uN be the vector function of all nodepotentials� uB the branch potentials and iB the �branch currents� Kircho��scurrent law states that

AiB � � �����a�

and Kircho��s voltage law states that

uB � ATuN � �����b�

Adding to this the characteristic element equations as described earlier�

��iB�uB� i�B�u

�B� � �� �����c�

we obtain a typically very large� sparse DAE�The sparse tableau approach leading to the DAE ����� is general� and

software can be written to generate the equations from a given functionaldescription of a circuit� but it is not favored in practice because it leads totoo much redundancy in the unknowns� Instead� the modi�ed nodal analysiseliminates uB �via �����b and the currents iB� except for those currentsthrough voltagecontrolled elements �inductors and voltage sources� Thisleads to a large� sparse� but smaller DAE of the form

M�y�y� � f�y� � q�t� ������

where the possibly singular and still quite sparse M describes the dynamicelements� f corresponds to the other elements and q are the independentsources�The index of ����� depends on the type of circuit considered� In practical

applications it often equals � or �� but it may be higher� This index is oftenlower than that of ������ because some constraints are eliminated� Standardsoftware exists which generates ����� from a functional description� However� a further reduction to an explicit ODE in the case that M is singularis not a practical option for most large circuits� because the sparsity of M isdestroyed by the necessary matrix decomposition �such as ����� below�A speci�c instance of a circuit is given in Example ����� �

For initial value ODEs� Theorem ��� guarantees solution existence� unique�ness and continuous dependence on initial data for a large class of problems�No corresponding theorem holds in such generality for boundary value ODEs�see Chapter ��� No corresponding theorem holds for general DAEs either�although there are some weaker results of this type� Boundary value DAEsare of course no less complex than boundary value ODEs� and will not beconsidered further in this chapter�

Page 253: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

��� Chapter �� Di�erential�Algebraic Equations

����� Special DAE Forms

The general DAE system ���� can include problems which are not well�de�ned in a mathematical sense� as well as problems which will result infailure for any direct discretization method �i�e� a method based on dis�cretization of y and y� without �rst reformulating the equations�� Fortu�nately� most of the higher�index problems encountered in practice can beexpressed as a combination of more restrictive structures of ODEs coupledwith constraints� In such systems the algebraic and di�erential variables areexplicitly identi�ed also for higher index DAEs� and the algebraic variablesmay all be eliminated �in principle� using the same number of di�erentiations�These are called Hessenberg forms of the DAE and are given below�

Hessenberg index��

x� � f�t�x� z� �����a�

� � g�t�x� z�� �����b�

Here the Jacobian matrix function gz is assumed to be nonsingular for all t�This is also often referred to as a semiexplicit index� system� Semi�explicitindex�� DAEs are very closely related to implicit ODEs� Using the implicitfunction theorem� we can in principle solve for z in �����b�� Substituting zinto �����a� yields an ODE in x �although no uniqueness is guaranteed� seeExercise ���� For various reasons� this procedure is not always recommendedfor numerical solution�

Hessenberg index��

x� � f�t�x� z� �����a�

� � g�t�x�� �����b�

Here the product of Jacobians gxfz is nonsingular for all t� Note the absenceof the algebraic variables z from the constraints �����b�� This is a pureindex�� DAE and all algebraic variables play the role of index�� variables� �

Example � � A practical example of a pure index� system arises from modeling the �ow of an incompressible �uid by the NavierStokes equations

ut � uux � vuy � px � ��uxx � uyy� � � ����a�

vt � uvx � vvy � py � ��vxx � vyy� � � ����b�

ux � vy � � ����c�

�Whether a DAE is Hessenberg index�� or index�� may depend on the solution �Ex�ample ��� but usually doesn�t in practice�

Page 254: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Chapter �� More on Di�erential�Algebraic Equations ���

where subscripts denote partial derivatives� x� y are spatial variables and t istime� u� v are the velocities in the x and y directions� respectively� p is thescalar pressure� and � is the �known kinematic viscosity� Equations �����a�����b are the momentum equations� and �����c is the incompressibilitycondition� The extension to three spatial variables is straightforward� Aftera careful spatial discretization of ����� with a �nitedi�erence� �nitevolumeor �niteelement method� the vectors u�t� and p�t� approximating �u�t� x� y��v�t� x� y� and p�t� x� y� in the domain of interest satisfy

Mu� � �K �N�u��u � Cp � f �����a�

CTu � �� �����b�

In this DAE the mass matrix M is symmetric positive de�nite� Skippingsome nontrivial details of the spatial discretization� we assume not only thatthe same matrix C appears in �����a and �����b but also that CTM��C isa nonsingular matrix with a bounded inverse� This yields an index� DAE inHessenberg form� The DAE could be made semiexplicit upon multiplyingby M��� but the sparsity of the coe�cient matrices of the DAE would be lost�unless M is blockdiagonal� The forcing function f comes from the �spatialboundary conditions�

It is wellknown that obtaining an accurate solution for the pressure in����� can be problematic� Often this variable is treated in a di�erent way bydiscretization methods� For instance� a staggered grid may be used in space�where the pressure values are considered at midcells and the velocity values�live� on cell edges� Part of the reason for this is that the pressure in �����is an indextwo variable� It has the same order of �temporal smoothness asthe derivative of the velocity� The pressure in ����� is playing the role ofthe indextwo variable z in ������One can consider di�erentiating �����c with respect to time and substi

tuting into �����a�����b to obtain a Poisson equation for p with the righthand side being a function of u and v� This is called the pressurePoissonequation � the matrix CTM��C above can in fact be viewed as a discretizationof the Laplace operator plus suitable boundary conditions � and the obtainedsystem has index �� For the index� system the discretization in space needno longer be staggered� but some di�culties with boundary conditions mayarise�

Another way to look at index�� variables like the pressure in ������ de�rives from the observation that these DAEs are closely related to constrainedoptimization problems� From this point of view� p in ������ plays the role of

Page 255: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

��� Chapter �� Di�erential�Algebraic Equations

a Lagrange multiplier� it forces the velocity u to lie in the constraint manifold de�ned by �����b�� The relationship between higher�index DAEs andconstrained optimization problems is no accident� many of these DAEs� in�cluding the incompressible Navier�Stokes equations� arise from constrainedvariational problems�

Example � � Consider the DAE

y�� � �y� � y� ������

y�� � y�� � ��� � sin� t��y� � y�� � ����y� � y���

� � y� � y� � ��sin t��y� � ��� � y� � y� � ��y� � ���

where � is a parameter and y���� � �� y���� � � are prescribed�This DAE is not in semiexplicit form� We can� however� easily convert

it to that form by the constant� nonsingular transformation

x� � y�� x� ��

��y� � y��� z� �

��y� � y��� z� � y�

yielding

x�� � �x� � z� �����a�

x�� � ��� � sin� t�x� � z�� �����b�

� � z� � �sin t��x� � �� �����c�

� � x� � �x� � ���� �����d�

The DAE is now in the semiexplicit form ����� but it is not in Hessenberg form� In particular� �����c yields z� � z��x�� so z� is an index�algebraic variable� whereas z� cannot be eliminated without di�erentiation� Adi�erentiation of �����d and a substitution into �����a con�rm that� for thegiven initial conditions� z� can be subsequently eliminated� Hence the DAEis index� and z� is an index� algebraic variable�Note that if we further carry out the substitution for z� then the resulting

DAE

x�� � �x� � z� ������

x�� � ��� � sin� t�x� � �sin� t��x� � ���� � x� � �x� � ���

is Hessenberg index���

Page 256: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Chapter �� More on Di�erential�Algebraic Equations ���

Hessenberg index��

x� � f�t�x�y� z� �����a�

y� � g�t�x�y� �����b�

� � h�t�y�� �����c�

Here the product of three matrix functions hygxfz is nonsingular�

Example � The mechanical systems with holonomic constraints describedin Example ��� are Hessenberg index�� This type of DAEs often arises fromsecondorder ODEs subject to constraints�Indeed� the ODEs describe Newton�s second law of motion relating body

accelerations to forces� Since accelerations are second derivatives of positions�constraints imposed on the positions imply that two di�erentiations must beburied in the system of ODEs with constraints� �

The index of a Hessenberg DAE is found� as in the general case� bydi�erentiation� However� here only the constraints need to be di�erentiated�

Example � � To illustrate� we �nd the index of a simple mechanical system� the pendulum in Cartesian coordinates from Example ���� We use thenotation q for the position coordinates and v � q� for the velocities� First�the DAE is written as a �rstorder system

q�� � v� �����a�

q�� � v� �����b�

v�� � ��q� �����c�

v�� � ��q� � g �����d�

� � q�� � q�� � � � �����e�

�Note that � � ��t� is an unknown function and g is the known� scaledconstant of gravity� Then the position constraint �����e is di�erentiatedonce� to obtain

q�q�� � q�q

�� � � �

Substituting for q� from �����a and �����b yields the velocity constraint

qTv � q�v� � q�v� � � � ������

Di�erentiating the velocity constraint ����� and substituting for q� yields

q�v�� � q�v

�� � v�� � v�� � ��

Page 257: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

��� Chapter �� Di�erential�Algebraic Equations

Substituting for v� from �����c and �����d� and simplifying using the position constraint� yields the acceleration constraint

�� � q�g � v�� � v�� � �� ������

This yields �� which can be substituted into �����c and �����d to obtain anODE for q and v� To obtain a di�erential equation for all the unknowns�however� we need to di�erentiate ����� one more time� obtaining an ODE for� as well� In the process of getting to the explicit ODE system� the positionconstraints were di�erentiated three times� Hence� the index of this systemis three� �

The index has proven to be a useful concept for classifying DAEs� in orderto construct and identify appropriate numerical methods� It is often notnecessary to perform the di�erentiations in order to �nd the index� becausemost physical systems can be readily seen to result in systems of Hessenbergstructure or in simple combinations of Hessenberg structures�

Example � Consider a tiny ball of mass � attached to the end of a springof length � at rest with a spring constant ���� � � �� At its other end thespring�s position is �xed at the origin of a planar coordinate system �see Fig���� and imagine the rod in the simple pendulum being replaced by a spring�The sum of kinetic and potential energies in this system is

e�q�v� ��

��vTv� ����r � ���� � gq�

where q � �q�� q��T are the Cartesian coordinates� v � �v�� v��T are thevelocities �which equal the momenta p in our scaled� dimensionless notation�r �

pq�� � q�� � jqj� is the length of the spring at any given time and g is

the scaled constant of gravity� The equations of motion are �recall x���

q� � ev � v

v� � �eq � ���� r � �r

q����g

�A �

This is an ODE� Let us next write the same system as a DAE� De�ning� � ����r � �� we get

q�� � ��rq��

���g

�A

�� � r � ��

Page 258: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Chapter �� More on Di�erential�Algebraic Equations ���

This DAE is semiexplicit index�� It is not really di�erent from the ODE ina meaningful way �although it may suggest controlling the error also in � ina numerical approximation�

Next� consider what happens when the spring is very sti�� almost rigid�i�e� � � �� We then expect the radius r to oscillate rapidly about its restvalue� while the angle � varies slowly� This is depicted in Fig� ����

Provided that the initial conditions yield

r�t� � � �O���

we have ��t� � O��� to balance the constraint equation in the index� formulation� The passage to the limit �� � is simple then� and we obtain theDAE

q�� � �q�����g

�A

� � r � �

which gives the equations for the simple pendulum of Examples ��� and �����

This is an index� DAE in Hessenberg form� Unlike the �dependent ODEsolution� the DAE solution varies slowly!

The simple example above leads to some important observations�

� One rich source of DAEs in practice is as the limit systems of singularperturbation ODE problems� when the small parameter tends to ��The solution then is often referred to as the reduced solution for thesingularly perturbed problem�

� A higher index DAE can often be simpler than� or result as a simpli��cation of� an ODE or a lower index DAE� In Example ��� the index��DAE is much simpler to solve than the original ODE �or the index��DAE� for a small ��

� A DAE can in a sense be very close to another DAE with a di�erentindex� Thus� a more quantitative stability theory involving not onlythe DAE index is necessary for a more complete picture�

�In Fig� �� the initial conditions are such that j�j � �� so that the oscillations in q

can be seen by the naked eye� but the limit DAE turns out to be the same�

Page 259: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

��� Chapter �� Di�erential�Algebraic Equations

0 1 2 3 4 5 6 7 8 9 10−1.5

−1

−0.5

0

0.5

1

1.5

t

q_1

�a� Cartesian coordinate q�

0 1 2 3 4 5 6 7 8 9 10−1.2

−1

−0.8

−0.6

−0.4

−0.2

0

0.2

t

q_2

�b� Cartesian coordinate q�

0 1 2 3 4 5 6 7 8 9 100.8

0.85

0.9

0.95

1

1.05

1.1

1.15

1.2

t

r

�c� Radius r

0 1 2 3 4 5 6 7 8 9 101.4

1.6

1.8

2

2.2

2.4

2.6

2.8

3

3.2

t

thet

a

�d� Angle �

Figure ���� Sti� spring pendulum� � � ����� initial conditions q��� � �� ������ ��T �v��� � ��

Page 260: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Chapter �� More on Di�erential�Algebraic Equations ��

Reader�s advice� Below we continue to discuss properties ofthe index of a DAE� and DAE stability� The conclusion at theend of x����� is practically important� but it is possible to skipthe discussion� at least on �rst reading� and still understand thematerial in x����

����� DAE Stability

Example ��� suggests that the index is a local quantity� to be measured aboutan isolated exact solution� Thus� we next consider perturbations in linearDAEs and their relationship to the index and to stability constants� For anonlinear problem we form the variational problem about an exact solutionand its perturbations� and de�ne the index locally based on the index of thislinear problem� As in x��� we note that for a linear problem the objective isto bound the solution in terms of the data �the inhomogeneities�� The samebound then holds for a perturbation to the solution when the inhomogeneitiesare replaced by their perturbations� �See the exposition following ��������For the linear ODE system

y� � A�t�y� q�t�� � � t � b

subject to homogeneous initial or boundary conditions� we can transform theindependent variable by � � t�b for any large b� Let us assume that this hasbeen done and take b � �� We have seen in x��� and in x��� that the followingstability bound holds�

kyk � max��t��

jy�t�j � �

Z �

jq�s�jds � �kqk�� ������

For the trivial index�� DAE

y � q�t�

we have a slightly weaker bound than ������� namely

kyk � kqk�In � ���� we have de�ned the L� norm� Note that

kqk�

Z �

jq�s�jds � max�t��

jq�t�j kqk �

Page 261: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

��� Chapter �� Di�erential�Algebraic Equations

�weaker because the maximumnorm� rather than the L��norm� must be usedfor q�� For the semi�explicit index�� DAE

x� � Ax�Bz� q��t� ����a�

� � Cx�Dz � q��t� ����b�

where A�B�C�D are bounded functions of t and with D boundedly invert�ible� we get a similar result�

kyk � �kqkwhere yT � �xT � zT �� qT � �qT� �q

T� �� The generic stability constant � in�

volves bounds on D��� as well as the stability constant of the underlyingODE for x� once z given by ����b� has been substituted for in ����a�� Thisbound can actually be re�ned to

kzk � �kqk� kxk � �kqk� �For the general index�� linear DAE �

E�t�y� � A�t�y� q�t� ������

still with homogeneous initial or boundary conditions� we can decomposeE�t� into

E�t� � S�t�

��I �

� �

�AT���t� ������

where T and S are nonsingular matrix functions with uniformly boundedcondition numbers� Then a change of variables�

�xz

�A � T��y

where x has the dimension of the identity block in ������ yields a semi�explicit system ������ Hence we obtain again �assuming of course that theunderlying ODE problem is stable� an estimate

kyk � �kqkwhere now the condition numbers of the transformations are also lumpedinto the stability constant ��In short� for a linear index�� problem� if

�We assume here that the system is strictly index�one� i�e� not tending arbitrarilyclosely to a higher�index or singular system�

Page 262: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Chapter �� More on Di�erential�Algebraic Equations ���

� it can be transformed �without di�erentiations� into a semi�explicit sys�tem� and from there to an ODE by eliminating the algebraic variables�

� the transformations are all suitably well�conditioned�� the obtained ODE problem is stable�

then the index�� DAE problem is also stable in the usual sense� Exercise ���makes this statement precise�For higher index problems we must di�erentiate at least some of the

equations� For an index�p DAE we need p � � di�erentiations to obtain anindex�� DAE� hence all we can hope for is a �stability� bound of the form

kyk � �

pXj��

kq�j���k � ������

Fortunately� for a DAE in Hessenberg form this can be somewhat improvedupon� In particular� for an index�� Hessenberg DAE of the form ����� withD � and CB nonsingular we have

kxk � �kqk ������

kzk � �kq�k �All this suggests that a direct numerical discretization of nontrivial higherindex DAEs other than Hessenberg index�� may encounter serious di culties�We will see in the next chapter that this is indeed true�

��� Index Reduction and Stabilization ODE

with Invariant

Often� the best way to solve a high index DAE problem is to �rst convert it toa lower index system by carrying out di�erentiations analytically� In this sec�tion we describe some of the techniques which are available for reformulationof a higher�index� semi�explicit DAE ������ where di�erentiations are appliedto the constraint equations ����b�� The essential concept here is that theDAE is equivalent to an ODE with an invariant� For an index��p� �� DAEin Hessenberg form with m ODEs and l constraints� recall that we need p dif�ferentiations in order to eliminate the algebraic variables and obtain an ODEsystem of size m in closed form� The equations ����b�� together with their�rst p�� derivatives �with z�t� eliminated�� form an invariant set de�ned bypl algebraic constraints� One can consider using these algebraic constraints

Page 263: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

��� Chapter �� Di�erential�Algebraic Equations

at each t in order to de�ne a smaller set of m�pl unknowns� The di�erentialequations for the smaller set of unknowns then describe the dynamics whileenforcing the constraints� This yields an ODE on a manifold and is furtherdiscussed in x������ Since the dimension of the constraint manifold is pl� thetrue dimension �i�e� the number of degrees of freedom� of the entire systemis m� pl� as discussed in the previous section�In the presentation that follows we use constrained mechanical systems

as a case study for higher index DAEs in Hessenberg form� Problems fromthis important class are often solved in practice using the techniques of thissection� The general principles of reformulation of DAE systems are alsouseful in a wide variety of other applications�

����� Reformulation of Higher�Index DAEs

Recall the mechanical systems from Example ����

q� � v �����a�

M�q�v� � f�q�v��GT �q�� �����b�

� � g�q� �����c�

where q are generalized body positions� v are generalized velocities� � �l

are Lagrange multiplier functions� g�q� �l de�nes the holonomic con�straints� G � gq is assumed to have full row rank at each t� M is a positivede�nite generalized mass matrix and f are the applied forces� Any explicitdependence on t is omitted for notational simplicity� but of course all thequantities above are functions of t� We also denote

x �

��qv

�A �m �

corresponding to the notation in ������We now apply two di�erentiations to the position constraints �����c�� The

�rst yields the constraints on the velocity level

� � Gv �� g�� ������

and the second di�erentiation yields the constraints on the acceleration level

� � Gv� ���Gv�

�qv �� g��� � ������

Next� multiply �����b� by GM�� and substitute from ������ to eliminate ��

��q�v� � �GM��GT ����GM��f �

��Gv�

�qv

�� ������

Page 264: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Chapter �� More on Di�erential�Algebraic Equations ���

Finally� � from ������ can be substituted into �����b� to yield an ODE for x�

q� � v �����a�

Mv� � f �GT �GM��GT ����GM��f �

��Gv�

�qv

�� �����b�

In practice we may want to keep �����b� in the equivalent form �����b��

������ as long as possible and to never evaluate the matrix function ��Gv��q

�i�e�� to evaluate only its product with v��The ODE system ������ has dimension m and is the result of an unstabi

lized index reduction� The constraints on the position and the velocity levels�which are now additional to this ODE� de�ne an invariant set of dimension�l�

h�x� �� g�q�

G�q�v

�A � �� �����

Thus� any solution of the larger ODE system ������ with consistent initialvalues� i�e� with initial values satisfying h�x���� � �� satis�es h�x�t�� � � atall later times� We denote the constraint Jacobian matrix

H � hx ������

and note that for the mechanical system �������

H �

�� G �

��Gv��q

G

�A ������

has full row rank �l� Restricted to the constraint manifold� the ODE hasdimension m� �l� which is the correct dimension of the DAE �������Example � � For the DAE ����� of Example ��� we substitute

�� � q�g � v�� � v��

to obtain the ODE corresponding to ������

q�� � v�

q�� � v�

v�� � ��v�� � v�� � q�g�q�

v�� � ��v�� � v�� � q�g�q� � g

and the invariant equations corresponding to ������

� � q�� � q�� � �� � q�v� � q�v��

Page 265: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

�� Chapter �� Di�erential�Algebraic Equations

����� ODEs with Invariants

Di�erential systems with invariants arise frequently in various applications�not only as a result of index reduction in DAEs� The invariant might repre�sent conservation of energy� momentum or mass in a physical system� TheODE system in Example ��� has the invariant that the energy is constant int� as is typical for Hamiltonian systems� Recall also Exercises ����������

The relationship between DAEs and ODEs with invariants goes bothways� Not only does index reduction of a DAE lead to an ODE with aninvariant� also an ODE with an invariant

x� � �f �x� �����a�

h�x� � � �����b�

is equivalent to the Hessenberg index�� DAE

x� � �f�x��D�x�z �����a�

� � h�x�� �����b�

Here D�x� is any bounded matrix function such that HD� where H � hx� isboundedly invertible for all t� The systems ������ and ������ have the samesolutions for x�t�� The exact solution of ������ gives z�t� �� but this is nolonger true in general for a numerical discretization of this system� Note thatthe DAE ������ is not the same as the original DAE ������ in case that thelatter is the source of the system ������� The choice of the matrix function Din ������ de�nes the direction of the projection onto the constraint manifold�A common choice is D � HT � which yields an orthogonal projection��

Indeed� there are applications where simply integrating the ODE is a per�fectly valid and useful approach� The numerical solution does not preciselysatisfy the constraints then� but it is close to satisfying �����b� within theintegration tolerance� But in other applications the invariant cannot simplybe ignored� This is the case when there are special reasons for insisting thatthe error in �����b� be much smaller than the error in �����a�� or when theproblem is more stable on the manifold than o� it�The latter reason applies in the case of a DAE index reduction� To see

this� imagine a nonsingular transformation of variables

q�����

�A �

��g�q�&g�q�

�A

�Note that in the case of mechanical systems � ���� we would like to avoid the lowerleft block of H if at all possible� see Exercise ����

Page 266: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Chapter �� More on Di�erential�Algebraic Equations ��

such that &gq is orthogonal to GT � Now� the di�erentiations of the constraints�i�e� ������� yield

��� � �

and this equation has a double eigenvalue �� This indicates a mild insta�bility� because if ���� � �� ����� � � and ��� � �� then ��t� � ���t�� i�e�perturbations grow quadratically in time� The instability� known as a drifto� the constraint manifold� is a result of the di�erentiations �i�e�� it is notpresent in the original DAE� hence not in the equivalent ODE restricted tothe manifold��Rather than converting the ODE to a DAE� which carries the penalty of

having to solve the resulting DAE� we can consider stabilizing� or attenuatingthe ODE �����a� with respect to the invariant set M � fx � h�x� � �g�The ODE

x� � �f�x�� F �x�h�x� ������

obviously has the same solutions as �����a� on M �i�e� when h�x� � ���It also has the desired stability behavior if HF is positive de�nite and thepositive parameter is large enough� In fact� we can easily apply a Lyapunov�type argument �see Exercises �������� to obtain

d

dthTh � hTh� � hTH��f � Fh�

� � � � ���hTh

where � is a constant such that� using the Euclidean vector norm�

jH�f �x�j � �jh�x�j ������

for all x near M� and �� is the smallest eigenvalue of the positive de�nitematrix function HF �Thus� asymptotic stability of the constraint manifold results for any �

����� What this means is that any trajectory of ������ starting from someinitial value nearM will tend towards satisfying the constraints� i�e� towardsthe manifold� Moreover� this attenuation is monotonic�

jh�x�t� ���j � jh�x�t��j ������

for any t� � � ��To get a grip on the values of � and ��� note that often � � � in �������

in which case the invariant is called an integral invariant� �Because for anyx�t� near M satisfying �����a� it transpires that d

dth � �� hence h�x�t�� is

constant�� For the mechanical system ������ it can be shown that � � ��Exercise ����� Also� if we choose

F �x� � D�HD���

Page 267: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

�� Chapter �� Di�erential�Algebraic Equations

where D�x� is as before in ������� then HF � I� hence �� � ��If the system is not sti� then ������ can be integrated by an explicit

method from the Runge�Kutta or Adams families� which is often faster thanthe implicit methods of x�����

Example � �� We consider again the simple pendulum in Cartesian coordinates and apply the Matlab standard IVP solver to the ODE of Example���� Starting from q��� � ��� ��T � v��� � �����T � the solver is accurateenough and the problem simple enough� that the unit circle is obtained in theqphase space to at least � signi�cant digits� Then we repeat the calculationsfrom the starting points q��� � ������T and the same v���� The resultingcurves are depicted in Fig� ����a�

−1.5 −1 −0.5 0 0.5 1 1.5

−1

−0.5

0

0.5

1

q_1

q_2

Unstabilized perturbed trajectories

�a� Unstabilized pendulum equations

−1.5 −1 −0.5 0 0.5 1 1.5

−1

−0.5

0

0.5

1

q_1

q_2

Stabilized perturbed trajectories

�b� Stabilized pendulum equations� ��

Figure ���� Perturbed �dashed lines� and unperturbed �solid line� solutionsfor Example ����

Next we modify the ODE according to ������ with

DT � H �

���q� �q� � �

v� v� q� q�

�A

and � ��� and repeat these integrations� The results are depicted in Fig�����b� Of course� for the starting values which do not satisfy jq���j� � ��the exact solution of the stabilized ODE is di�erent from the original� but theFigure clearly indicates how the unit circle becomes attractive for the latterODE system� even when the initial values are signi�cantly perturbed� �

Page 268: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Chapter �� More on Di�erential�Algebraic Equations ��

One of the earliest stabilization methods proposed in the literature wasdue to J� Baumgarte� In this method� the acceleration�level constraints arereplaced by a linear combination of the constraints on the acceleration� ve�locity and position levels�

� � g�� � �g� � �g� ������

The parameters � and � are chosen such that the polynomial equation

x� � �x� � � �

has two negative roots� thus the ODE ������ for g is stable� This stabilizesthe invariant setM� The system �����a�� �����b� and ������ is a semi�explicitindex�� DAE� It can be made into an ODE upon elimination of �� and besubsequently solved numerically by standard methods� But the choice of theparameters has proved tricky in practice� Exercise ���� and x���� elaboratemore on this�

����� State Space Formulation

The di�erentiations of the constraints of the given high�index DAE ������yield an ODE ������ with an in�ated dimension� as we have seen� Eventhough the number of degrees of freedom of the system is m� �l� we have in������ m ODEs� and in ����� an additional �l algebraic equations� Ratherthan stabilizing the invariant� another approach is to use these algebraicequations to de�ne a reduced set of unknowns� obtaining an ODE systemof the minimal size m � �l� The main di culty with this idea arises in thepresence of highly nonlinear terms�Suppose that R is a rectangular� constant matrix such that� together with

the constraint Jacobian G� we obtain a nonsingular matrix with a boundedinverse�

k��RG

�A��

k � K � ������

De�ning the change of variables

u � Rq� w � Gq �����

we get

q �

��RG

�A����u

w

�A �

We can now use the constraints de�ning the invariant set� i�e� g�q� � � andGv � �� to express w as a function of u� and hence q in terms of u� For u we

Page 269: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

�� Chapter �� Di�erential�Algebraic Equations

then obtain� upon multiplying the equations of motion by R� an underlyingODE of size �when converted to a �rst order system� m� �l�

u�� � RM���f �GT�� ������

where � is given by �������There are two popular choices for R� The �rst is such that the unknowns

u form a subset of the original q� i�e�� the columns of R are either unitvectors or �� This has the advantage of simplicity� Note� however� thatwe cannot expect in general that one such choice of R will be good for allt� in the sense that ������ will remain valid with a moderate constant K�This coordinate partitioning has to be monitored and modi�ed as necessary�The other choice is to make R orthogonal to M��GT � This eliminates � in������� but introduces additional complications into the calculation due tothe varying R�The attraction of this approach is the small ODE system that is obtained

and the elimination of any drift o� the constraint manifold� On the negativeside� this approach involves a somewhat messier algebra and is less transpar�ent� The transformation non�singularity ������ must be monitored and anywrinkle in the constraint manifold might have to be fully re�ected here� evenif it could be otherwise ignored�

��� Modeling with DAEs

The closing decades of the ��th century have seen many scientists recognizethat their mathematical models are in fact instances of DAEs� Such a recog�nition has often carried with it the bene�t of a�ording a new� sometimesrevealing� computational look at the old problem�Note� however� that whereas a sensible formulation of a mathematical

model as an initial value ODE is typically followed simply by its numericalsolution using some appropriate code� DAE formulations may require moreuser attention and intervention� combining the processes of problem formu�lation and numerical solution� Since high index DAEs are all unstable� weknow already before advancing to Chapter �� that attempting to discretizethem directly may adversely a�ect the resulting numerical scheme� The re�formulations of the problem discussed in the previous section are done withnumerical implementations in mind� In the extreme� a DAE would be con�verted to an ODE� but bear in mind that this may be cumbersome to carryout and costly to work with�Consider a DAE system and its various index reductions and reformu�

lations studied in x���� The exact solution satis�es all such equations� but

Page 270: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Chapter �� More on Di�erential�Algebraic Equations �

numerical discretizations generally result in nonzero residuals� When a semi�explicit DAE such as ����� is discretized and solved numerically� it is auto�matically assumed that the ODE ����a� will be solved approximately whilethe algebraic constraints will be satis�ed �almost� exactly� The residual in����b� is essentially set to �� while that of ����a� is only kept small �at thelevel of the truncation error�� The relative importance of these residualschanges when index reduction is applied prior to discretization�

The situation is similar for an ODE with an invariant ������� Once aparticular formulation is discretized� a greater importance is placed on theconstraints than on the ODE� in the sense described above�

Satisfying the constraints �including the hidden ones� exactly is in someinstances precisely what one wants� and in most other cases it provides ahelpful �e�g� stabilizing�� or at least harmless� emphasis� State space methods�x������ tacitly assume that this constraint satisfaction is indeed desired� andthey provide no alternative for when this is not the case�

Yet� there are also instances where such an emphasis is at odds with thenatural �ow of the ODE� In such cases one may be better o� not to insist onsatisfying constraints too accurately� Such examples arise when we apply themethod of lines �Example ���� for a PDE� allowing the spatial mesh points tobe functions of time and attempting to move them as the integration in timeproceeds as part of the solution process in order to meet some error equidis�tribution criteria which are formulated as algebraic equations �this is called amoving mesh method�� The emphasis may then be wrongly placed� becauseobtaining an accurate solution to the PDE is more important than satisfy�ing a precise mesh distribution criterion� One is better o� using the DAEto devise other� clever moving mesh schemes� instead of solving it directly�Rather than dwelling on this further� we give another such example�

Example � �� Recall from x��� that the Hamiltonian e�q�v� is constantin a Hamiltonian system given by

q� � rve

v� � �rqe

where e�q�v� does not depend explicitly on time t� So� this ODE system hasthe invariant

e�q�t��v�t��� e�q����v���� � � �t�The system is in the form ������ To enforce the preservation of the invariant�conservation of energy� we can write it as a Hessenberg index� DAE �����

�In this chapter and the next we use e rather than H to denote the Hamiltonian� toavoid a notational clash with H hx�

Page 271: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

�� Chapter �� Di�erential�Algebraic Equations

with D � HT � This gives

q� � rve� �rqe�z

v� � �rqe� �rve�z

e�q�v� � e�q����v���� �

Note that the DAE has one degree of freedom less than the original ODE� It issometimes very helpful to stabilize the solution with respect to this invariant�see� e�g�� Example �����But when the Hamiltonian system is highly oscillatory� e�g� in case of

Example ��� with � � �� �� the projected DAE is poorly balanced� �Roughly�large changes in z are required to produce a noticeable e�ect in the ODE forv� but they then strongly a�ect the ODE for q� The observed numericale�ect is that the best direct numerical discretizations of the DAE �which arenecessarily implicit require that the step size h satisfy �at best h � O�

p���

or else the Newton iteration does not converge� With this stepsize restriction�the explicit leapfrog discretization �Exercise ���� of the ODE is preferred�A complete discussion of this example is beyond the scope of this presen

tation� Let us simply state that there are also other reasons why imposingenergy conservation during a largestep integration of highly oscillatory problems is not necessarily a good idea�

Of course� we do not mean to discourage the reader from using DAEmodels and solvers� what with having spent a quarter of our book on them$Rather� we wish to encourage a careful thought on the problem formulation�whether it is based on an ODE or DAE model�

��� Notes and References

A more detailed development of the DAE theory contained in x��� can befound in the books by Brenan� Campbell Petzold ����� Hairer Wanner���� and Griepentrog M.arz ����� See also the survey paper ����� However�unlike in the previous theory chapters the material in this one is not a strictsubset of any of these references�There is an extensive theory for linear DAEs with constant coe cients

which we have chosen not to develop here� For an introduction and furtherreferences� see ����� Be careful not to confuse constant coe cient DAEs withmore general� linear DAEs�It is interesting that� in contrast to the situation with ODEs� theorems

on existence and uniqueness of solutions of nonlinear DAEs did not appear

Page 272: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Chapter �� More on Di�erential�Algebraic Equations ��

until relatively recently� Most of these results are due to Rabier Rheinboldt��� ���� The theory is based on a di�erential geometric approach� see also���� and references therein�

There have been many de�nitions of index in the literature� most of whichhave been shown to be equivalent or at least closely related� for the classes ofproblems to which they apply� The concept which we have de�ned here is are�nement of the di�erential index� In ���� and ���� a related concept calledthe perturbation index was introduced� which is directly motivated by theloss of smoothness in solutions to higher�index DAEs� as discussed in x������However� we chose to restrict the perturbation analysis to linear�ized� DAEs�see Exercise �����Underlying the index de�nition� and more generally our DAE discussion�

is the assumption that� whatever matrix function which after certain ma�nipulations eventually is nonsingular� has this property independently of t�For example� in the semi�explicit form ������ we have considered either thecase that gz is nonsingular for all t or that it is singular for all t� Thisfundamental assumption breaks down for singular DAEs� where this matrixbecomes singular at some isolated points t� �For example� in �����a� considerthe case where a�t� varies and changes sign at some points�� The situationcan become much more complex� and a variety of phenomena may occur� fornonlinear� singular DAEs� The solution may remain continuous or it maynot ���� ��� See also exercises ��� and ������Some of the material covered in x��� is curiously missing from the usual

DAE books� We refer for more to ��� ��� �� �� ��� ���� Generalized co�ordinate partitioning methods were introduced in ���� and tangent planeparameterization methods were implemented in �����

��� Exercises

�� A square matrix is said to be in �block� upper�� Hessenberg form if ithas the sparsity structure depicted in Fig� ���� Can you guess why�DAEs in Hessenberg form� have been endowed with this name!

�� Consider the two�point boundary value problem

�u�� � au� � b�t�u� q�t� �����a�

u��� � b�� u��� � b� �����b�

where a �� � is a constant and b� q are continuous functions� all O���in magnitude�

Page 273: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

�� Chapter �� Di�erential�Algebraic Equations

0 2 4 6 8 10 12 14

0

2

4

6

8

10

12

14

nz = 103

Figure ���� A matrix in Hessenberg form�

�a� Write the ODE in �rst order form for the variables y� � u andy� � �u� � au�

�b� Letting �� �� show that the limit system is an index�� DAE�

�c� Show that only one of the boundary conditions in ������ is neededto determine the reduced solution �i�e� the solution of the DAE��Which one!

�� Consider the DAE ����

y�� � y�y�� � y�y

��

� � y�

� � y�

with y���� � ��

�a� Show that this DAE has index ��

�b� Show that if we add to the right hand side the perturbation

��t� � ��� � sin�t� � cos�t�T

which is bounded in norm by a small �� the perturbed solution �y�t�satis�es �y�� � ���� which is unbounded as � � �� The stabilitybound is seen to depend on ��� as is typical for index�� rather thanindex�� problems�

�c� Show that if we add a similar perturbation to the linearizationaround the solution y�t� for z � �z�� z�� z��T �

z�� � z�y�� � y�z

�� � z�y

�� � y�z

��

� � z�

� � z�

Page 274: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Chapter �� More on Di�erential�Algebraic Equations ��

then the perturbed �z is bounded in terms of k�k� like an index��DAE solution should be�

�� Construct an example of a DAE which for some initial conditions hasindex � and for others index ��

� Consider the IVP for the implicit ODE

�y��� � y�� y��� � ��

�a� Show that this problem has two solutions�

�b� Write down a corresponding Hessenberg index�� DAE with twosolutions�

�� The following equations describe a chemical reaction ���� ���

C � � K��C� � C��R

T � � K��T� � T � �K�R �K��T � TC�

� � R �K�e�K��TC

where the unknowns are the concentration C�t�� the temperature T �t�and the reaction rate per unit volume R�t�� The constants Ki and thefunctions C� and T� are given�

�a� Assuming that the temperature of the cooling medium TC�t� isalso given� what is the index of this DAE! Is it in Hessenbergform!

�b� Assuming that TC�t� is an additional unknown� to be determinedsuch that an additional equation specifying the desired productconcentration

C � u

for a given u�t� be satis�ed� what is the index of this DAE! Is itin Hessenberg form!

�� Given a general linear DAE ������ with E�t� decomposed as in �������apply the decoupling transformation into semi�explicit form� give acondition for the DAE to have index � and formulate a precise stabilitycondition�

�� �a� Writing the mechanical system ������������ in the notation ��������nd H�f and a bound on � in �������

�b� Show that the velocity constraints ������ alone de�ne an invariantmanifold for ������� What are h� H and H�f then!

Page 275: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

��� Chapter �� Di�erential�Algebraic Equations

�c� Show that the position constraints �����c� alone do not de�ne aninvariant manifold for �������

�� Let r �px�� � x�� and consider the ODE ����

x�� � x� � x��r� � �����r��

x�� � �x� � x��r� � �����r���

�a� Show that

h�x� � r� � � � �de�nes an invariant set for this ODE�

�b� Show that there is no �nite � � � for which ������ holds�

��� Consider the mechanical system with holonomic constraints written asan ODE with invariant �������������

�a� Write down the equivalent Hessenberg index�� DAE ������ with

D �

��GT �

� GT

�A �

�b� This D simpli�es HT of ������ in an obvious manner� Verify thatHD is nonsingular�

�c� Show that by rede�ning � the system you obtained can be writtenas

q� � v�GT�

Mv� � f �GT�

� � g�q�

� � Gv�

This system is called the stabilized index� formulation ����

��� �a� Write down the system resulting from Baumgarte�s ��� stabiliza�tion ������ applied to the index�� mechanical system �������

�b� Consider the index�� mechanical system given by �����a�� �����b��������� This is representative of nonholonomic constraints� wherevelocity�level constraints are not integrable into a form like �����c��

Write down an appropriate Baumgarte stabilization

h� � h � �

Page 276: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Chapter �� More on Di�erential�Algebraic Equations ���

for the index�� mechanical system and show that it is equivalentto stabilization of the invariant ������ with

F �

�� �

M��GT �GM��GT ���

�A �

�c� However� Baumgarte�s technique ������ for the index�� problem isnot equivalent to the stabilization ������� Show that the mono�tonicity property ������ does not hold here�

Page 277: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

��� Chapter �� Di�erential�Algebraic Equations

Page 278: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Chapter ��

Numerical Methods for

Di�erential�Algebraic

Equations

Numerical approaches for the solution of di�erential�algebraic equations �DAEs�can be divided roughly into two classes� �i� direct discretizations of the givensystem and �ii� methods which involve a reformulation �e�g� index reduction��combined with a discretization�The desire for as direct a discretization as possible arises because a refor�

mulation may be costly� it may require more input from the user and it mayinvolve more user intervention� The reason for the popularity of reformula�tion approaches is that� as it turns out� direct discretizations are limited intheir utility essentially to index�� and semi�explicit index�� DAE systems�Fortunately� most DAEs encountered in practical applications are either

index�� or� if higher�index� can be expressed as a simple combination ofHessenberg systems� The worst�case di culties described in x������ belowdo not occur for these classes of problems� On the other hand� the mostrobust direct applications of numerical ODE methods do not always work aswell as one might hope� even for these restricted classes of problems� We willoutline some of the di culties� as well as the success stories� in x�����We will consider two classes of problems�

� Fully�implicit index�� DAEs in the general form� � F�t�y�y��� ������

� Index�� DAEs in pure� or Hessenberg formx� � f�t�x� z� �����a�

� � g�t�x�� �����b�

���

Page 279: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

��� Chapter � � Di�erential�Algebraic Equations

Recall that the class of semi�explicit index�� DAEs

x� � f�t�x� z� �����a�

� � g�t�x� z� �����b�

is equivalent to the class of fully implicit index�� DAEs via the transforma�tions ����� and ������ although the actual conversion of DAEs from one formto another may come with a price of an increased system size� For the DAE������� z are algebraic variables which may be index�� or index��� whereasfor the Hessenberg form the variables in z are all index�� �which is why wesay that the DAE is pure index����Although there are in some cases convergence results available for nu�

merical methods for Hessenberg DAEs of higher index� there are practicaldi culties in the implementation which make it di cult to construct robustcodes for such DAEs� For a DAE of index greater than two it is usuallybest to use one of the index�reduction techniques of the previous chapter torewrite the problem in a lower�index form� The combination of this with asuitable discretization is discussed in x�����

���� Direct Discretization Methods

To motivate the methods in this section� consider the regularization of theDAE ������� where �����b� is replaced by the ODE

�z� � g�t�x� z� � ������

which depends on a small parameter � � � � �� Despite the promisingname� we do not intend to actually carry out this regularization� unless spe�cial circumstances such as for a singular DAE �e�g� Exercise ������ requireit� because the obtained very sti� ODE �����a�� ������ is typically more cum�bersome to solve than the DAE �recall� e�g�� Example ������ But this allowsus to consider suitable ODE methods� Observe that�

� Since the regularized ODE is very sti�� it is natural to consider methodsfor sti� ODEs for the direct discretization of the limit DAE�

� ODE discretizations which have sti� decay are particularly attractive�to recall �x���� any e�ect of an arti�cial initial layer which the regular�ization introduces can be skipped� fast ODE modes are approximated

�For this reason we may also assume that the regularized ODE problem is stable undergiven initial or boundary conditions� Were the regularized problem to be actually solved�the term �z� might have to be replaced by a more general �Bz�� where e�g� B �gz�

Page 280: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Chapter � � Numerical Methods for Di�erential�Algebraic Equations ��

well at mesh points� and so the passage to the limit of �� � in ������is smooth and yields a sensible discretization for the DAE�

The rest of this section is therefore devoted to the direct application ofODE methods to low�index DAEs� All the winning methods have sti� decay�but this property alone is not su cient� For initial value DAEs which arecumbersome to transform� and especially for DAEs whose underlying ODEis sti�� the BDF and Radau collocation methods discussed in this sectionare the overall methods of choice� We thus begin with the simplest methodof this kind� the backward Euler method� and then consider its extension tohigher order via BDF or Radau methods� see Fig� �����

Radau collocation

Backward Euler

BDF

Figure ����� Methods for the direct discretization of DAEs in general form�

������ A Simple Method Backward Euler

Consider the general DAE

� � F�t�y�y�� �

The idea of a direct discretization is simple� approximate y and y� by adiscretization formula likemultistep or Runge�Kutta� Applying the backwardEuler method to this DAE� we obtain

� � F

�tn�yn�

yn � yn��hn

�� �����

This gives� in general� a system of m nonlinear equations for yn at each timestep n�Unfortunately� this simple method does not always work� In the worst

case� there are simple higher�index DAE systems with well�de�ned solutions

Page 281: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

��� Chapter � � Di�erential�Algebraic Equations

for which the backward Euler method� and in fact all other multistep andRunge�Kutta methods� are unstable or not even applicable�

Example �� � Consider the following linear index� DAE which dependson a parameter ���

�� �

� �t

�Ay� �

��� �t

� � � �

�Ay �

��q�t��

�A � ������

The exact solution is y��t� � q�t� � �tq��t�� y��t� � �q��t�� which is wellde�ned for all values of �� The problem is stable for moderate values of ��Yet� if � � ��� we show below that there is no solution of the equationsde�ning yn using the backward Euler discretization� It can be shown �seeExercise ���� that the backward Euler method is unstable when � � ���Let us analyze this problem� To transform to semiexplicit form� de�ne

u � y� � �ty�� v � y�� hence

y �

��� ��t� �

�A��uv

�A �

We readily obtainu� � v � �� u � q�t�

for which the backward Euler method gives

un � q�tn�� vn � �q�tn�� un��h

�note that a forward Euler method makes no sense here� Thus� provided thatwe start with a consistent initial value for u� i�e� u� � q���� we have

vn � �q��tn� �O�h�

which is all that one can expect from a �rst order method for ODEs�This is in marked contrast to what happens when applying backward Euler

directly to ��������� �

� �tn

�A yn � yn��

h�

��� �tn

� � � �

�Ayn �

��q�tn�

�A �

De�ning

��unvn

�A �

��� �tn

� �

�Ayn� we get from this latter discretization

un � q�tn�� �� � ��vn � �q�tn�� q�tn�����h �

Page 282: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Chapter � � Numerical Methods for Di�erential�Algebraic Equations ���

We see that� while un is reproduced exactly� vn is unde�ned when � � ��and has O��� error when � �� ��The transformation to semiexplicit form decouples the solution compo

nents y into di�erential and algebraic variables� The backward Euler discretization works well for the decoupled problem� But in general� a directdiscretization of nondecoupled DAEs of index higher than one is not recommended� �

For the remainder of x���� we thus consider only index�� or semi�explicitindex�� DAEs�For the simplest class of nonlinear DAEs� namely semi�explicit index���

x� � f�t�x� z� �����a�

� � g�t�x� z� �����b�

where gz is nonsingular� it is easy to see that the backward Euler methodretains all of its properties �i�e� order� stability� and convergence� from theODE case� First� we recall that by the implicit function theorem� there existsa function &g such that

z � &g�t�x��

�Let us assume� for simplicity� that there is only one such &g� so what isdepicted in Exercise �� does not happen�� Thus the DAE ������ is equivalentto the ODE

x� � f�t�x� &g�t�x��� ������

Now� consider the backward Euler method applied to �������

xn � xn��hn

� f�tn�xn� zn� �����a�

� � g�tn�xn� zn�� �����b�

Solving for zn in �����b� and substituting into �����a� yields

xn � xn��hn

� f�tn�xn� &g�tn�xn�� �������

which is just the backward Euler discretization of the underlying ODE �������Hence we can conclude from the analysis for the nonsti� case in x��� thatthe backward Euler method is �rst�order accurate� stable and convergent forsemi�explicit index�� DAEs�For fully�implicit index�� DAEs� the convergence analysis is a bit more

complicated� It is possible to show that for an index�� DAE� there existstime �and solution��dependent transformation matrices in a neighborhood of

Page 283: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

��� Chapter � � Di�erential�Algebraic Equations

the solution� which locally decouple the linearized system into di�erentialand algebraic parts� Convergence and �rst�order accuracy of the method onthe di�erential part can be shown via the techniques of x���� The backwardEuler method is exact for the algebraic part� The complications arise mainlydue to the time�dependence of the decoupling transformations� which entersinto the stability analysis� �Recall that for fully�implicit higherindex DAEs�time�dependent coupling between the di�erential and algebraic parts of thesystem can ruin the method�s stability� as demonstrated in Example �����Fortunately� for index�� systems it only complicates the convergences anal�ysis� however� it may a�ect some stability properties of the method�� Seex������ for pointers to further details�The convergence result for backward Euler applied to fully�implicit index�

� DAEs extends to semi�explicit index�� DAEs in an almost trivial way�Making use of the transformation ������ it is easy to see that solving theindex�� system ����� by the backward Euler method gives exactly the samesolution for x as solving the original semi�explicit index�� system ������ by thesame method� A separate argument must be made concerning the accuracyof the algebraic variables z� For starting values which are accurate to O�h��it turns out that the solution for z is accurate to O�h�� after � steps havebeen taken�For nonlinear problems of the form ����� a Newton iteration for yn�

starting from an approximation y�n based on information from previous steps�

yields for the �� � ��st iterate�

y���n � y�n �

��

hn

�F

�y���F

�y

���F

�tn�y

�n�y�n � yn��

hn

�� �������

Note that� in contrast to the ODE case� the iteration matrix is not simplydominated by an h��n I term� We discuss the implication of this in x�������

������ BDF and General Multistep Methods

The constant step�size BDF method applied to a general nonlinear DAE ofthe form ������ is given by

F

�tn� yn�

�h

kXj��

�jyn�j

�� � �������

where � and �j � j � �� �� � � � � k� are the coe cients of the BDF method�Most of the available software based on BDF methods addresses the fully�

implicit index�� problem� Fortunately� many problems from applicationsnaturally arise in this form� There exist convergence results underlying themethods used in these codes which are a straightforward extension of theresults for backward Euler� In particular� the k�step BDF method of �xed

Page 284: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Chapter � � Numerical Methods for Di�erential�Algebraic Equations ���

step size h for k � � converges to O�hk� if all initial values are correctto O�hk�� and if the Newton iteration on each step is solved to accuracyO�hk���� This convergence result has also been extended to variable step�size BDF methods� provided that they are implemented in such a way thatthe method is stable for standard ODEs� See the discussion in x�� Aswith backward Euler� this convergence result extends to semi�explicit index�� DAEs via the transformation ������ A separate argument must be madeconcerning the accuracy of the algebraic variable z� For starting values whichare accurate to O�hk�� it turns out that the solution for z is accurate to O�hk�after k � � steps have been taken�There has been much work on developing convergence results for general

multistep methods� For general index�� DAEs and for Hessenberg index��DAEs� the coe cients of the multistep methods must satisfy a set of orderconditions which is in addition to the order conditions for ODEs� to attainorder greater than �� It turns out that these additional order conditions aresatis�ed by BDF methods�You may wonder if all this additional complication is really necessary�

why not simply write ������ as ������� then consider �����b� as the limit of������! Then apply the known theory for BDF from the ODE case$!The answer is that there is no such a priori known convergence theory

in the ODE case� The basic convergence� accuracy and stability theory ofChapters �� � and applies to the case h� �� whereas here we must alwaysconsider � � h� Indeed� since any DAE of the form ������ can be �treated�this way� regardless of index� we cannot expect much in general in view ofthe negative results in Chapter � for higher index DAEs� For an ODE system�����a�������� whose limit is an index�� DAE ������� convergence results asstated above do apply� But these results are not easier to obtain for theODE� on the contrary� the very sti� ODE case is generally more di cult�

Example �� � To check the convergence and accuracy of BDF methods�consider the simple linear example�

x�� � ��� �

�� t�x� � �� � t��z �

� � t

� � t

x�� ��� �

t� � x� � x� � ��� ��z � �et

� � �t� ��x� � �t� � ��x� � �t� � t� ��et

where � is a parameter� This DAE is in a pure index� form ������ For theinitial conditions x���� � x���� � � we have the exact solution

x� � x� � et� z � � et

�� t�

Recall that we can de�ne y� � z with some initial condition �say� y��� � �to obtain a fully implicit index� DAE for x � �x�� x��T and y� The BDF

Page 285: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

��� Chapter � � Di�erential�Algebraic Equations

discretization remains the same� We select � � �� and integrate this DAE

10−4

10−3

10−2

10−1

10−12

10−10

10−8

10−6

10−4

10−2

100

h

max

|err

or|

�a� Errors in x��t�

10−4

10−3

10−2

10−1

10−12

10−10

10−8

10−6

10−4

10−2

100

h

max

|err

or|

�b� Errors in z�t�

Figure ����� Maximum errors for the �rst � BDF methods for Example �����

from t � � to t � � using the �rst three BDF methods� In Fig� ���� wedisplay maximum errors in x� and in z for di�erent values of h ranging from��� to

����� We use a loglog scale� so the slopes of the curves indicate the

orders of the methods� The results clearly indicate that the convergence orderof the kstep BDF method is indeed k and that in absolute value the errorsare pleasantly small� �

������ Radau Collocation and Implicit Runge�Kutta

Methods

Runge�Kutta Methods and Order Reduction

The s�stage implicit Runge�Kutta method applied to the general nonlinearDAE of the form ������ is de�ned by

� � F�ti�Yi�Ki�� ������a�

ti � tn�� � cih� i � �� �� � � � � s ������b�

Yi � yn�� � h

sXj��

aijKj ������c�

and

yn � yn�� � h

sXi��

biKi � �������

Page 286: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Chapter � � Numerical Methods for Di�erential�Algebraic Equations ���

We assume here that the coe cient matrix A � �aij� is nonsingular�For the semi�explicit problem ������ the formula ������� for the internal

stages reads

Ki � f�ti�Xi�Zi�

Xi � xn�� � h

sXj��

aijKj

� � g�ti�Xi�Zi� �

For the algebraic variables z in this case it is often better to avoid the quadra�ture step implied by �������� because there is no corresponding integrationin the DAE� This gives an advantage to sti)y accurate methods which sat�isfy bj � asj� j � �� � � � � s� because for these methods the constraints areautomatically satis�ed at the �nal stage� Indeed� for such methods we have

yn � Ys

in �������� and ������� is not used� For ������ we then simply set xn � Xs�As was the case for general multistep methods� there are additional order

conditions which the method coe cients must satisfy for the method to attainorder greater than �� for general index�� and Hessenberg index�� DAEs� ForRunge�Kutta methods� there is an additional set of order conditions evenfor semi�explicit index�� DAEs� We are often faced with an order reduction�the causes of which are closely related to the causes of order reduction forRunge�Kutta methods applied to sti� ODEs �recall x����� and x����� Thisis not surprising� given the close relationship between very sti� ODEs andDAEs�

�Reader�s advice� It is possible to skip the remainder of thissubsection� if you are interested mainly in using the methods anddo not require an understanding of the causes of order reduction�

To understand this order reduction� consider �rst the simple scalar ODE

�z� � �z � q�t� �����a�

and its limit DAE

� � �z � q�t� �����b�

to which we apply an s�stage Runge�Kutta method with a nonsingular co�e cient matrix A� Using notation similar to Chapter �� the internal stagesolution values are

Zi � zn�� � h��

sXj��

ai�j�q�tj�� Zj�� i � �� � � � � s �

Page 287: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

��� Chapter � � Di�erential�Algebraic Equations

So� with Z � �Z�� � � � � Zs�T � Q � �q�t��� � � � � q�ts��T � we have

Z � ��h��I �A�����h���zn�� �AQ�

orZ � �h��A���zn�� � �I � �h��A���Q�O���h��� �

Letting �� � we get the exact DAE solution at the internal stages

Zi � q�ti�� i � �� � � � � s � �������

At the end of the current step�

zn � zn�� � h

�bT �Z�Q� � zn�� � bTA����zn�� �Q� �O��h���

and for the DAE �����b� this gives

zn � �� � bTA����zn�� � bTA��Q � �������

The recursion ������� for zn converges if jR����j � j� � bTA���j ��� but the order of approximation of the ODE� which involves quadratureprecision� may be reduced� For instance� for an s�stage collocation methodthe approximate solution on the subinterval �tn��� tn� is a polynomial whichinterpolates q�t� at the collocation points ti of ������b�� The local errorzn � q�tn� �assuming zn�� � q�tn��� for a moment� is therefore strictly aninterpolation error� which is O�hs����The situation is much better if the method has sti� decay� which happens

when bT coincides with the last row of A� In this case cs � � necessarily�and

zn � Zs � q�tn�

is exact� This can also be obtained from ������� upon noting that bTA ���� � � � � �� ��� Thus� while Gauss collocation yields a reduced local error orderO�hs���� down from the usual order O�h�s���� Radau collocation yields theexact solution for �����b� at mesh points tn�Next� consider the system

x� � �x� q��t� ������a�

�z� � �z � x� q��t� ������b�

and the corresponding index�� DAE obtained with � � �� Applying thesame Runge�Kutta discretization to this system and extending the notationin an obvious manner� e�g�

Xi � xn�� � h

sXj��

ai�j��Xj � q��tj��

Zi � zn�� �h

sXj��

ai�j��Zj �Xj � q��tj��

Page 288: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Chapter � � Numerical Methods for Di�erential�Algebraic Equations ���

we obtain for ������b� as �� ��

Zi � Xi � q��ti�� i � �� � � � � s

zn � ��� bTA����zn�� � bTA���Q� �X�

with an obvious extension of vector notation� Thus� the stage accuracy ofthe method� i�e� the local truncation error at each stage� enters the error inzn� unless the method has sti� decay�A Runge�Kutta method is said to have stage order r if r is the minimum

order of the local truncation error over all internal stages� For an s�stagecollocation method� the stage order is s� For an s�stage DIRK the stageorder is �� We see that for ������� the local error in zn has the reduced orderr � �� unless the method has sti� decay� For the latter there is no reductionin order�This result can be extended to general semi�explicit index�� DAEs �������

But it does not extend to fully implicit index�� DAEs or to higher�indexDAEs� In particular� DIRK methods experience a severe order reduction forsemi�explicit index�� DAEs and hence also for fully implicit index�� problems�This is true even for DIRK methods which have sti� decay�The rooted tree theory of Butcher has been extended to yield a complete

set of necessary and su cient order conditions for classes of DAEs such assemi�explicit index��� index��� and Hessenberg index�� and index��� We willnot pursue this further here�

Collocation Methods

By their construction� Runge�Kutta methods which are collocation methodsare not subject to such severe order limitations as DIRK methods in theDAE case� These methods were introduced in x���� For the semi�explicitDAE ������ we approximate x by a continuous piecewise polynomial x��t�of degree � s � � on each subinterval �tn��� tn�� while z is approximated bya piecewise polynomial which may be discontinuous at mesh points tn andhas degree � s on each subinterval �see Exercise ������ The convergenceproperties are summarized below�Consider an s�stage collocation method of �ODE� order p� with all ci �� ��

approximating the fully�implicit index�� DAE ������ which has su cientlysmooth coe cients in a neighborhood of an isolated solution� Let � � �bTA��� and assume j j � �� This method converges and the order satis�es�� The error in yn is at least O�hs��� If j j � � then the error in yn is O�hs����

� If � �� and a mild mesh restriction applies then the error in yn isO�hs����

Page 289: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

��� Chapter � � Di�erential�Algebraic Equations

� If cs � � then the error in yn is O�hp��

For the semi�explicit index�� DAE ������� the error results for the di�er�ential variable x are the same as for the index�� system reported above� Forthe algebraic variable z� the error satis�es�

� The error in zn is at least O�hs����

� If j j � � then the error in zn is O�hs��

� If � �� and a mild mesh restriction applies then the error in zn isO�hs��

� If cs � � then the error in zn is O�hs��

In particular� collocation at Radau points retains the full order p � �s��for the di�erential solution components� and so this family of methods isrecommended as the method of choice for general�purpose use among one�step methods for initial value DAEs and for very sti� ODEs�

Example �� � Fig� ���� is a schematic depiction of a simple circuit containing linear resistors� a capacitor� voltage sources �operating voltage Ub andinitial signal Ue� and two npnbipolar transistors� For the resistors and the

3

R 1

R 3

R 4

C R5

5

R2

U0

U

Ue

b

1

24

Figure ����� A simple electric circuit�

capacitor the current relates directly to the voltage drop along the device �recall Example ���� For the transistors the relationship is nonlinear and is

Page 290: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Chapter � � Numerical Methods for Di�erential�Algebraic Equations ��

characterized by the voltage U � UB � UE between the base and the emitter��The third pole of the transistor is the collector C� We use

IE � f�U� � �eU�UF � ��IC � ��IEIB � �� � ��IE

where UF � ������ � � ����� � ��e� ��Applying Kircho��s current law at the � nodes in sequence� we get

� � �U� � Ue��R� � ��� ��f�U� � U��

C�U �� � U ��� � �UB � U���R� � �U� � U���R� � �f�U� � U��

� � �U� � U���R� � f�U� � U��� f�U� � U��

C�U �� � U ��� � �� � ��f�U� � U��� �U� � U���R�

� � �U� � Ub��R � �f�U� � U���

We use the values U� � � �ground voltage� Ub � � Ue � sin�����t��R� � ���� R� � ����� R� � ���� R� � ����� R � ����� C � ���e � �� �Thepotentials are in Volts� the resistances are in Ohms� t is in seconds�This is a simple index� DAE which� however� has scaling and sensitivity

di�culties due to the exponential in the de�nition of f � We can obviouslymake it semiexplicit for the di�erential variable U� � U�� but we leave thesystem in the fully implicit form and apply the collocation code radau��This code is based� to recall� on collocation at � Radau points� It applies toODEs and DAEs of the form

My� � &f�t�y�

�see x������ and Exercise ������ and here we have such a form with theconstant matrix

M �

�BBBBBBBBB�

� � � � �

� C � �C �

� � � � �

� C � �C �

� � � � �

�CCCCCCCCCA�

For consistent initial conditions� only U�����U���� are free� The rest aredetermined by the � algebraic equations� �How �� Three are apparent� thefourth is obtained upon adding up the two equations containing derivatives�which cancels out the derivative term� A consistent initial vector is given by

y��� � ��� Ub� �� �� Ub�T �

Page 291: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

��� Chapter � � Di�erential�Algebraic Equations

0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5

x 10−3

−70

−60

−50

−40

−30

−20

−10

0

10

time (sec)

volta

ge (

V)

Figure ����� Results for a simple electric circuit� U��t� �solid line� and theinput Ue�t� �dashed line��

A plot of the resulting U� as a function of time� as well as the input signalUe� is given in Fig� ����� It is seen that U��t� has periods where it becomeslarge and negative� The solution is not very smooth� The code used � steps�of which ��� were accepted� for this simulation� The right hand side functionwas evaluated almost ���� times� but the Jacobian only ��� times�

������ Practical Di�culties

Even though there are order and convergence results for the backward Eulermethod �as well as for BDF and collocation at Radau points� applied to fully�implicit index�� and semi�explicit index�� DAEs� some practical di cultiespersist� Fortunately� they are not insurmountable�

Obtaining a consistent set of initial conditions

A major di�erence in practice between the numerical solution of ODEs andDAEs is that the solution of a DAE system must be started with a consistentset of initial conditions� Recall from x��� that this means that the constraints�and possibly some hidden constraints� must be satis�ed at the initial point�There are two basic types of initialization problems� when there is not

enough information for a general�purpose code� and when there is too muchinformation� or not the correct type of information� for the DAE to havea solution� To understand the �rst of these better� consider the simplest

Page 292: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Chapter � � Numerical Methods for Di�erential�Algebraic Equations ���

instance of semi�explicit� index�� DAE ������� Suppose that

x��� � x�

is provided� This is precisely the information needed to specify a solutiontrajectory for this problem� For an ODE� i�e� ������ without z and g� we canuse the di�erential equation to obtain also x���� �denote this by x���� Thisinformation is used by a general�purpose code to obtain an accurate initialguess for the Newton iteration and"or a reliable error estimate for the �rsttime step� A general�purpose DAE solver may require� the value of z�� Thisis for � reasons� to completely specify the solution at t � �� to provide aninitial guess for the variant of Newton�s iteration used to �nd z�� and tocompute x�� from

x�� � f���x�� z���

The solution process for z� consists in this case of solving the nonlinearequations

� � g���x�� z��

given x�� Unlike in later steps� where we have zn�� to guess zn initially�here we must face a �cold start�� This can be done with an o��the�shelfnonlinear equation solver� Also� some of the newer BDF software o�ers thisas an initialization option� The implementation requires little in the wayof additional information from the user� and can exploit structure in theiteration matrix� making use of the same linear algebra methods which areused in subsequent time�stepping�Note that the above does not address the question of �nding all initial

values z�� in case there is more than one isolated solution for these nonlinearalgebraic equations� An extension of this procedure is given in Exercise �����Another consistent initialization problem� that of �nding initial values of thesolution variables such that the system starts in a steady state� is presentalready for ODEs and discussed in Exercise ���Consistent initialization of more general index�� DAEs involves more di �

culties� because the di�erential and the algebraic variables are not separated�Thus� information that should be determined internally by the system maybe speci�ed externally �i�e�� in addition to the DAE system��

Example �� � For the semiexplicit index� DAE

u� � ��u� v��� � q��t�

� � �u� v��� � q��t�

it is clear that a prescribed u� determines v� � u� � �q����� and then u�� ���u� � v���� � q�����

�Note that z is not needed for the exact solution� Moreover� this value is never usedin a simple calculation like for Example �����

Page 293: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

��� Chapter � � Di�erential�Algebraic Equations

But now� let u � y� � y�� v � y� � y�� This yields the DAE

y�� � y�� � y� � q��t�

y� � q��t��

To get an isolated solution to the DAE� we need to specify y���� � y�����But we cannot specify y���� and y���� arbitrarily� because y���� � q���� isalready determined� Specifying y����� it is not possible to solve directly for theremaining initial values as we did in the semiexplicit case� Instead� we canonly �nd y����� � y����� � q����� y����� To �nd y����� and y

����� individually�

we need also the information from the derivative of the constraint� namelyy���t�� � q���t��� �

The situation gets more complex� of course� for higher index problems�Recall that consistent initial conditions for higher�index systems must satisfythe hidden constraints which are derivatives of the original constraints�

Example �� � Consider once more the simple pendulum in Cartesian coordinates� The equations ����� for this index� Hessenberg DAE are givenin Example ����

Note at �rst that q��� cannot be speci�ed arbitrarily� given� e�g�� q�����the value of q���� � �p� � q����� is determined up to a sign� Then� fromthe hidden constraint ����� the speci�cation of one of the components ofv��� also determines the other� In other words� the user�s speci�cation ofq��� and v��� must satisfy the constraints �����e and ������

This then determines q���� by �����a�����b and ���� according to ������Finally� v���� is determined by �����c�����d� although this may be considered less necessary�

To make this task easier for non�Hessenberg DAEs �especially in largeapplications such as electric circuits� see Example ����� methods and softwareare available which use graph theoretic algorithms to determine the minimalset of equations to be di�erentiated in order to solve for the consistent initialvalues� Initialization for general index�� systems and for higher�index systemsis often handled on a case�by�case basis�

Ill�conditioning of iteration matrix

Another di culty� which shows up already in the solution of index�� DAEsbut is more serious for index�� systems� concerns the linear system to besolved at each Newton iteration� For explicit ODEs� as hn � � the iteration

Page 294: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Chapter � � Numerical Methods for Di�erential�Algebraic Equations ���

matrix tends to the identity�� For index�� and Hessenberg DAEs� the con�dition number of the iteration matrix is O�h�pn �� where p is the index� Toillustrate� consider the backward Euler method applied to the semi�explicitindex�� DAE ������� The iteration matrix is�

�h��n I � fx �fz�gx �gz

�A �

It is easy to see that the condition number of this matrix is O�h��n �� Forsmall hn� this can lead to failure of the Newton iteration� However� scalingcan improve the situation� In this case� multiplying the constraints by h��nyields an iteration matrix whose condition number no longer depends on h��nin this way� For Hessenberg index�� systems the conditioning problem can bepartially �xed by scaling of both the algebraic variables and the constraints�see Exercise �����

Error estimation for index�� DAEs

Recall that in modern BDF codes� the errors at each step are estimatedvia a weighted norm of a divided di�erence of the solution variables� ForODEs� this norm is taken over all the solution variables� This type of errorestimate still works for fully�implicit index�� DAEs� but it is not appropriatefor index�� problems� as illustrated by the following example�

Example �� Consider the simple index� DAE

y� � q�t�

y� � y��solved by the backward Euler method to give

y��n � q�tn�

y��n �y��n � y��n��

hn�

q�tn�� q�tn���hn

The truncation error is estimated via the second divided di�erence of thenumerical solution which� for the algebraic variable y�� yields

EST � hn�hn � hn����y��n� y��n��� y��n���

� hn�hn � hn���

� y��n�y��n��hn

� y��n���y��n��hn��

hn � hn��

��������

� hn

�q�tn��q�tn���

hn� q�tn����q�tn���

hn��

hn�

q�tn����q�tn���hn��

� q�tn����q�tn���hn��

hn��

��

�Of course� for very sti ODEs the term h��n I which appears in the iteration matrixdoes not help much� because there are larger terms which dominate� The situation for avery sti ODE is similar to that of the limit DAE�

Page 295: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

��� Chapter � � Di�erential�Algebraic Equations

For an ODE� or even for the di�erential variable y� of this example� EST� � as hn � � �all previous step sizes are �xed� However� ������ yieldsfor the error estimate of the algebraic variable

limhn��

EST � limhn��

q�tn�� q�tn���hn

� q�tn���� q�tn���hn��

� q��tn���� q�tn���� q�tn���hn��

which in general is nonzero� Thus� the error estimate for this variable cannotbe decreased to zero by reducing the step size� This can lead to repeated errortest failures� The approximation to y� is actually much more accurate thanthe error estimate suggests� �

The problem can be �xed by eliminating the algebraic variables �and inparticular the index�� variables� from the error test� In fact� it has beenshown that this strategy is safe in the sense that it does not sacri�ce theaccuracy of the lower�index �or di�erential� variables� which control the time�evolution of the system� We note that the algebraic variables should not beremoved from the Newton convergence test�

Given the di culties encountered for direct DAE discretization methods�and our recommendation not to apply such methods for DAEs beyond semi�explicit index��� we must also emphasize again that� on the other hand� suchdirect discretization methods are important in practice� Index reductionmay be necessary at times� but it is often not a desirable medicine$ Oneclass of applications where this is important is in large circuit simulation�as discussed in Example ���� We saw a small instance in Example �����Other such examples often arise in chemical engineering and in a variety ofapplications involving the method of lines� For large problems� which ariseroutinely in practice� a conversion to explicit ODE form can be a disaster ifas a result the sparsity structure of the matrices involved is lost�

������ Specialized Runge�Kutta Methods for Hessen�

berg Index�� DAEs

The methods discussed in this section apply to Hessenberg index�� problems������ and not to the more general form of ������� The structure of the pureindex�� system is exploited to achieve gains which are not possible for theperturbed ODE �������

Page 296: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Chapter � � Numerical Methods for Di�erential�Algebraic Equations ���

Projected Runge�Kutta Methods

As we have seen in Chapter �� one�sided formulas like Radau collocationwithout upwinding are not well�suited for the solution of general boundaryvalue problems� Since a stable boundary value problem can have solutionmodes which decrease rapidly in both directions� a symmetric method ispreferred� or else such modes must be explicitly decoupled� The Gauss col�location methods have been particularly successful for the solution of ODEboundary value problems� However� these methods do not have sti� decay�and when implemented in a straightforward manner as described in x�������they su�er a severe order reduction for Hessenberg index�� DAEs� In general�the midpoint method is accurate only to O��� for the index�� variable z in������� There are additional di culties for these methods applied to Hes�senberg index�� DAEs� including potential instability and the lack of a nice�local error expansion� Fortunately� all of these problems can be eliminatedby altering the method to include a projection onto the constraint manifoldat the end of each step� Thus� not only z��t� but also x��t�� the piecewisepolynomial approximating x�t�� may become discontinuous at points tn �seeExercises ���� and ������Let xn� zn be the result of one step� starting from xn��� zn��� of an implicit

Runge�Kutta method ������� applied to the Hessenberg index�� DAE �������Rather than accepting xn as the starting value for the next step� the projectedRungeKutta method modi�es xn at the end of each step so as to satisfy

�xn � xn � fz�tn�xn� zn��n ������a�

� � g�tn� �xn� � ������b�

�The extra variables �n are needed for the projection only� They are notsaved�� Then set xn � �xn and advance to the next step�Note that for a method with sti� decay� ������b� is already satis�ed by

xn� so there is no need to project� For collocation the projection gives themethods essentially the same advantages that Radau collocation has withoutthe extra projection� In particular� projected collocation methods achievesuperconvergence order for x at the mesh points� The solution for z can bedetermined from the solution for x� and to the same order of accuracy� via apost�processing step�Projected collocation at Gauss points has order �s and is useful for bound�

ary value DAEs�

Half�Explicit Runge�Kutta Methods

For many applications� a fully�implicit discretization method is not war�ranted� For example� many mechanical systems are essentially nonsti� andcan� with the exception of the constraints� be handled via explicit methods�

Page 297: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

��� Chapter � � Di�erential�Algebraic Equations

One way to accommodate this is via halfexplicit RungeKutta methods� Themethods obtained share many attributes with the methods to be describedin the next section�The half�explicit Runge�Kutta method is de�ned� for a semi�explicit DAE

������� by

Xi � xn�� � h

i��Xj��

aijf�tj�Xj �Zj�

� � g�ti�Xi�Zi�� i � �� � � � � s

xn � xn�� � h

sXi��

bif�ti�Xi�Zi�

� � g�tn�xn� zn�� �������

Thus� at each stage i� Xi is evaluated explicitly and a smaller nonlinearsystem is solved for Zi�For semi�explicit index�� DAEs� the order of accuracy is the same as for

ODEs� In fact� the method is not very di�erent from the correspondingexplicit Runge�Kutta method applied to the ODE x� � f�t�x� z�x��� Forsemi�explicit index�� systems in Hessenberg form� there is in general orderreduction� but higher�order methods of this type have been developed�

���� Methods for ODEs on Manifolds

The numerical solution of di�erential systems where the solution lies on amanifold de�ned explicitly by algebraic equations is a topic with interest inits own right� It also provides a useful approach for solving DAEs�As in x������ consider the nonlinear di�erential system

x� � �f�x� ������a�

and assume for simplicity that for each initial value vector x��� � x� thereis a unique x�t� satisfying ������a�� Suppose in addition that there is aninvariant setM de�ned by the algebraic equations

� � h�x� ������b�

such that if h�x�� � � then h�x�t�� � � for all t� There are various ap�proaches possible for the numerical solution of ��������

�� Solve the stabilized ODE ������ numerically� using one of the discretiza�tion methods described in earlier chapters� The question of choosing

Page 298: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Chapter � � Numerical Methods for Di�erential�Algebraic Equations ���

the stabilization parameter arises� As it turns out� the best choice of typically depends on the step size� though � see Exercise ����� andExample �����

�� Rather than discretizing ������� it turns out to be cheaper and moree�ective to stabilize the discrete dynamical system� i�e� to apply thestabilization at the end of each step� Thus� an ODE method is appliedat each step to ������a�� This step is followed by a poststabilizationor a coordinate projection step to bring the numerical solution closer tosatisfying ������b�� not unlike the projected Runge�Kutta methods ofthe previous section�

�� The �automatic� approach attempts to �nd a discretization for ������a�which automatically satis�es also the equations ������b�� This is pos�sible when the constraints are at most quadratic � see Exercises ���������

Of these approaches� we now concentrate on post�stabilization and coor�dinate projection�

������ Stabilization of the Discrete Dynamical System

If the ODE is not sti� then it is desirable to use an explicit discretizationmethod� but to apply stabilization at the end of the step� This is reminiscentof half�explicit Runge�Kutta methods �������� Suppose we use a one�stepmethod of order p with a step size h for the given ODE �without a sta�bilization term�� Thus� if at time tn�� the approximate solution is xn���application of the method gives

&xn � �fh�xn���

as the approximate solution at tn �e�g� forward Euler� �fh�xn��� � xn�� �

h�f�xn�����The poststabilization approach modi�es &xn at the end of the time step

to produce xn� which better approximates the invariant�s equations�

&xn � �fh�xn��� ������a�

xn � &xn � F �&xn�h�&xn� � ������b�

The stabilization matrix function F was mentioned already in ������ and itsselection is further discussed in x�������Example �� � For the scalar ODE with invariant

x� � ���t�

� � x� ��t�

Page 299: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

��� Chapter � � Di�erential�Algebraic Equations

with x��� � ����� where � is a given� su�ciently di�erentiable function� theexact solution is x � ��t��The poststabilization procedure based� e�g�� on forward Euler�

&xn � xn�� � h���tn���

xn � &xn � �&xn � ��tn��

produces the exact solution for this simple example�Consider� on the other hand� the stabilization ������ Here it gives the

stabilized di�erential equation

x� � ���t�� �x� ��t�� �

For � � the invariant is stable but not asymptotically stable� while for � �M is asymptotically stable� with the monotonicity property ����� holding�But this asymptotic stability does not necessarily guarantee a vanishing

drift� consider forward Euler with step size h applied to the stabilized ODE

xn � xn�� � h����tn���� �xn�� � ��tn����� �

The best choice for is the one which yields no error accumulation� This isobtained for � ��h� giving

xn � ��tn��� � h���tn��� �

�Note that this depends on the discretization step size� So� the drift

zn � ��tn� � �h�

�����tn��� �O�h�� �

although second order in h� may not decrease and may even grow arbitrarilywith h �xed� if ��� grows� Such is the case� for instance� for ��t� � sin t� ast grows� �

For the post�stabilization to be e�ective� we must design F such that

kI �HFk � � � �� �������

where H � hx� It has been shown� assuming �i� su cient smoothness nearthe manifold M and �ii� that either � � O�h� or ������ holds� that for anODE method of �nonsti�� order p�

� The global error satis�esxn � x�tn� � O�hp� ������

�i�e� the stabilization does not change the method�s global order� ingeneral��

Page 300: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Chapter � � Numerical Methods for Di�erential�Algebraic Equations ��

� There is a constantK that depends only on the local solution propertiessuch that

jh�xn�j � K��hp�� � h��p���� � �������

� If HF � I then

jh�xn�j � O�h��p���� � �������

Example �� Recall the modi�ed Kepler problem of Exercise ���� �withthe notation e in place of H for the Hamiltonian there� For the unmodi�edproblem� � � � and the solution has period �� Thus� the error in thesolution can be simply measured at integer multiples of �� In Table ���� werecord results using poststabilization with F � HT �HHT ��� �denoted �pstab�and also using an explicit �nd order RungeKutta method with and withoutpoststabilization �denoted �pstabeRK� and �eRK�� resp�� and the projectedmidpoint method of x������ and Exercise ���� ��projmidpt� applied to theprojected invariant formulation ������ All runs are with uniform time stepsh and � ���� Note that the projected midpoint method has better stabilityproperties and preserves the invariant� but the symmetry of the original ODEis lost�

method h jq����j jq����j jq�����j jq����j khk�midpt ��� ��� ��� ��� ��� ���e��

eRK ��� ��� ��� ��� �� ���e��

pstab�midpt ��� ��e�� ���e�� ��e�� ��� ���e��

pstab�eRK ��� ���e�� ���e�� ���e�� ��� ��e��

proj�midpt ��� ��e�� ���e�� ���e�� ���e�� �

midpt ���� ���e�� ���e�� ���e�� ���e�� ���e��

eRK ���� ��e�� ���e�� ���e�� ���e�� ���e��

pstab�midpt ���� ��e�� ���e�� ��e�� ���e�� ���e���

pstab�eRK ���� ���e�� ���e�� ���e�� ����� ���e���

proj�midpt ���� ���e�� ���e�� ���e�� ���e�� �

Table ����� Errors for Kepler�s problem using various �nd order methods�

We observe the second order accuracy of all methods considered and theinvariant�s accuracy order ��p � �� � � of the poststabilization methods�

Page 301: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

��� Chapter � � Di�erential�Algebraic Equations

The stabilization methods improve the constant of the global error� comparedto their unstabilized counterparts� but not the order� The cheapest methodhere� for the given range of time integration and relative to the quality of theresults� is the poststabilized explicit method� The projected midpoint methodis more expensive than the rest and is not worth its price� despite being mostaccurate for a given step size�Note that the midpoint method loses all signi�cant digits for h � ���

before reaching t � �� The pointwise error does not explode� however� butremains O���� Also� the error in the Hamiltonian remains the same� depending only on the step size h� not on the interval length� Calculations withthe poststabilized midpoint method up to t � ���� yield similar conclusionsregarding the invariant�s error for it as well �but not for the poststabilizedexplicit RungeKutta method� where a smaller step size is found necessary��

A closely related stabilization method is the coordinate projection method�Here� following the same unstabilized ODE integration step as before

&xn � �fh�xn���

we determine xn as the minimizer of jxn � &xnj� such that� � h�xn� �

There is a constrained least squares minimization problem to be solved forxn at each step n� As it turns out� the post�stabilization method ������� withF � HT �HHT ��� �for which obviously HF � I�� coincides with one Newtonstep for this local minimization problem� � An analogue of the relationshipbetween these two stabilization methods would be using a PECE versionof a predictor�corrector as in x����� compared to iterating the corrector toconvergence using a functional iteration� In particular� the two methodsalmost coincide when the step size h is very small�For this reason� there has been a tendency in the trade to view the two

methods of post�stabilization and coordinate projection as minor variantsof each other� There is an advantage in e ciency for the post�stabilizationmethod� though� Note that the choice of F is more �exible for the post�stabilization method� that ������� implies that the �rst Newton iteration ofthe coordinate projection method is already accurate to O�h��p����� and thatno additional iteration at the current time step is needed for maintaining thisaccuracy level of the invariant in later time steps�

�An energy normjxn � �xnj

�A �xn � �xn�

TA�xn � �xn�

for a positive de�nite matrix A can replace the ��norm in this minimization� with a cor�responding modi�cation in F � see Exercise ������ The error bound ������� still holds forthe outcome of one Newton step�

Page 302: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Chapter � � Numerical Methods for Di�erential�Algebraic Equations ���

Exercise ���� provides another example of post�stabilization �which coin�cides with coordinate projection� in action�

������ Choosing the Stabilization Matrix F

The smaller kI �HFk is� the more e�ective the post�stabilization step� Thechoice F � HT �HHT ��� which was used in Example ���� above� or more gen�erally the choice corresponding to one Newton step of coordinate projectionF � D�HD���� achieves the minimumHF � I�However� choices of F satisfying HF � I may be expensive to apply� In

particular� for the Euler�Lagrange equations ������� it is desirable to avoid the

complicated and expensive matrix ��Gv��q� Such considerations are application�

dependent� To demonstrate possibilities� let us continue with the importantclass of Euler�Lagrange equations and set

B �M��GT �

Note that inverting �or rather� decomposing� GB is necessary already toobtain the ODE with invariant ��������If we choose for the index�� problem

F �

��B�GB��� �

� B�GB���

�A �������

�or the sometimes better choice

F �

��GT �GGT ��� �

� GT �GGT ���

�A

which� however� requires an additional cost� then

HF �

��I �

L I

�A � L �

��Gv�

�qB�GB���

so HF �� I�Note� however� that

�I �HF �� � ��

The e�ect ofHF � I can therefore be achieved by applying post�stabilizationwith the cheap F of ������� twice� The decomposition �or �inversion�� neededfor evaluating F is performed once and this is frozen for further applicationat the same time step �possibly a few time steps��The application to multibody systems with holonomic constraints then

reads�

Page 303: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

��� Chapter � � Di�erential�Algebraic Equations

Algorithm �� � Post�stabilization for multibody systems

�� Starting with �qn���vn��� at t � tn��� use a favorite ODE integrationmethod �f

h �e�g� RungeKutta or multistep to advance the system

q� � v

M�q�v� � f�q�v��GT �q��

� � G�q�v� ���Gv�

�qv

by one step� Denote the resulting values at tn by �&qn� &vn��

�� Poststabilize using F of ���������qn�vn

�A �

��&qn&vn

�A� F �&qn� &vn�h�&qn� &vn�

�� Set ��qnvn

�A �

���qn�vn

�A� F �&qn� &vn�h��qn� �vn� �

In case of nonholonomic constraints the DAE is index�� and only oneapplication of F per step is needed�

Example �� � Consider a twolink planar robotic system with a prescribedpath for its end e�ector �the �robot�s hand�� Thus� one end of a rigid rodis �xed at the origin� and the other is connected to another rigid rod withrotations allowed in the x � y plane� Let �� be the angle that the �rst rodmakes with the horizontal axis� and let �� be the angle that the second rodmakes with respect to the �rst rod �see Fig� ����� The masses of the rodsare denoted by mi and their lengths are denoted by li� The coordinates of thelink between the rods are given by

x� � l�c� y� � l�s�

and those of the �free� end are

x� � x� � l�c�� y� � y� � l�s��

where ci � cos �i� si � sin �i� c�� � cos��� � ���� s�� � sin��� � ����Referring to the notation of the EulerLagrange equations ������ we let

q � ���� ���T and obtain

M �

��m�l

���� �m��l�� � l���� � l�l�c�� m��l���� � l�l�c����

m��l���� � l�l�c���� m�l

����

�A

Page 304: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Chapter � � Numerical Methods for Di�erential�Algebraic Equations ���

ll2

1y

(x2 , y2 )

(x1 , y

1

1 )

θ

11

θ2

1x

Figure ���� Two�link planar robotic system

f �

���m�gl�c����m�g�l�c� � l�c�����

�m�gl�c����

�A�

��m�l�l�s���������

�� � ��

���

��

�m�l�l�s���������

�A �

In the following simulation we use the data

m� � m� � ��kg� l� � l� � �m� g � ����m�s�

����� � ��o� ����� � ����o� ������ � ������ � ��

So far we do not have constraints g� Indeed� for a double pendulum theequations of motion form an implicit ODE �or an index� DAE� becausethe topology of this simple mechanical system has no closed loops and weare using relative �minimal coordinates to describe the system� But now weprescribe some path constraint on the position of �x�� y��� and this yields� inturn� also a constraint force GT�� We choose the constraint

y��t� � sin��t���

�for y� expressed in terms of q as described above� The obtained constrainedpath for �x�� y�� is depicted in Fig� ����� In this case the constraint forcesbecome large at a few distinct times�In Table ���� we record the measured drifts� i�e� the error in the path

constraint ��driftposition� and in its derivative ��driftvelocity�� based onruns up to b � ��s using an explicit RungeKutta scheme of order � with a

Page 305: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

��� Chapter � � Di�erential�Algebraic Equations

-2 -1.5 -1 -0.5 0 0.5 1 1.5 20

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1y vs x -- position motion

Figure ����� Constraint path for �x�� y���

constant step size h� We record results using Baumgarte�s technique ������denoting it �Baum� �� ���� and also using various choices of F for poststabilization� �Spos� stands for stabilizing only with respect to the positionconstraints g�t�q� � �� �Svel� stands for stabilizing only with respect to thevelocity constraints g� � �� �Sboth� stands for using F of ������ once� �Sboth�� is the choice recommended in Algorithm ����� and �nally �Sfull� usesF � HT �HHT ����Note that without stabilization the computation blows up for h � �����

The Baumgarte stabilization is not as e�ective as the S stabilizations� especially for the case h � ���� Other parameters � �� �� tried do not yield significantly better results� The choice of Algorithm ���� shows driftconvergenceorder � � ��p � �� and� given that it is much cheaper than Sfull and notmuch more expensive than the other choices for F � we conclude that Sboth�

gives the most bang for the buck here� �

���� Software Notes and References

������ Notes

Of course� scientists encountered the need to numerically solve mathematicalmodels involving di�erential equations with constraints and implicit di�er�ential equations for many decades� if not centuries� But the recognition that

Page 306: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Chapter � � Numerical Methods for Di�erential�Algebraic Equations ���

h stabilization drift�velocity drift�position

��� Baum��� �� � ���� Baum���� ��� �� ��e��

��� S�full ���e� ���e��

��� S�both ���e�� ���e�

��� S�both� ���e�� ���e��

��� S�vel ���e��� ���e��

��� S�pos ��� ���e��

���� Baum��� �� ���e�� ��e��

���� Baum���� ��� ���e�� ���e��

���� S�full ���e��� ��e���

���� S�both ���e�� ���e���

���� S�both� ���e�� ���e��

���� S�vel ���e��� ��e��

���� S�pos ���e�� ���e���

Table ����� Maximum drifts for the robot arm� � denotes an error over�ow�

DAE classes are worth being considered as such� in order to methodicallyderive good numerical methods and software� is relatively recent�

The original idea for discretizing DAEs directly with suitable ODE meth�ods was described in the landmark ���� paper of Gear ����� He used BDFmethods and applied them to problems of the type discussed in Example ����This was followed in the �����s by a deluge of e�orts to design and analyzenumerical methods and to write general�purpose software for DAEs� Thedirect discretization methods described in x���� are covered in more detailin Hairer Wanner ��� and Brenan� Campbell Petzold �����

We have chosen not to discuss convergence results for numerical methodsapplied directly to index�� DAEs� However� there are convergence results forsome numerical methods �both BDF and Runge�Kutta� applied to Hessen�berg DAEs of index greater than two� see ���� ���

We have noted that direct discretization methods are not applicable togeneral� higher�index DAEs� Campbell ���� ��� has developed least�squarestype methods for such problems which may be viewed as automatic index

Page 307: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

��� Chapter � � Di�erential�Algebraic Equations

reduction� The methods require di�erentiation of the original DAE� whichis accomplished by an automatic di�erentiation software package such asadifor ����� Using these and similar ideas� initialization schemes for generalDAE systems have been constructed�The �rst results on order reduction for general multistep methods applied

to higher�index DAEs were given by R� M.arz� see ���� �� for a summary andfor further references�More details and proofs for x������ and x������ can be found in Chin ����

and ���� See also ���� which is the source for Examples ���� and �����Example ���� was taken from ���� Other examples from various applica�

tions in the literature are formulated as exercises below�

������ Software

Excellent and widely�available software exists for the solution of initial�valueproblems and boundary�value problems in DAEs� Here we brie�y outlinesome of the available codes� With the exception of software for mechanicalsystems� they all apply to sti� ODEs as well �and if you read the �rst fewpages of this chapter carefully then you should be able to understand whythis is natural��

Initial Value Problems

� The code dassl by Petzold uses the �xed�leading�coe cient form ofthe BDF formulas to solve general index�� DAEs� see ���� for details�Versions for large scale problems �called daspk� and for sensitivityanalysis are also available�

� The code radau� by Hairer Wanner ��� is based on the ��stageRadau collocation method� It solves DAEs of the form

My� � &f �t�y� �������

where M is a constant� square matrix which may be singular� see Ex�ercise ����� The code is applicable to problems of index �� � or �� butthe user must specify which variables are higher�index �this implies aspecial structure��

� There are many codes� both commercial and publicly available� whichare designed speci�cally for simulating constrained mechanical sys�tems� They use many of the methods mentioned here� including Baum�garte stabilization� post�stabilization and coordinate projection� andvarious coordinate partitioning methods� The code mexx by Lubichet al� ���� is based on a half�explicit extrapolation method which we

Page 308: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Chapter � � Numerical Methods for Di�erential�Algebraic Equations ���

have not covered and implements fast linear algebra techniques for tree�structured mechanical systems�

Boundary Value Problems

� The code coldae by Ascher Spiteri ���� uses projected Gauss collo�cation to extend colnew ���� for boundary value� semi�explicit index��DAEs in the form ������� An additional singular value decompositiondecouples algebraic variables of di�erent index if needed�

���� Exercises

�� Show that the implicit Euler method is unstable for the DAE ������ if� � ���

�� Consider the backward Euler method applied to the Hessenberg index��DAE �������

�a� Show that the condition number of the iteration matrix is O�h����

�b� How should the equations and variables be scaled to reduce thecondition number to O���!

�c� What are the implications of scaling the variables on the accuracyone can expect in these variables from the linear system solver!

�� Set � � � in Example ��� and solve the resulting DAE numerically� Youmay use any �justi�able� means you like� including index reduction anduse of an appropriate software package� Plot the solution and comparewith Fig� ���� Discuss�

�� Consider two linked bars of length li and mass mi� i � �� �� One endof one bar is �xed at the origin� allowing only rotational motion in theplane �as in Fig� ����� The other end of the other bar is constrainedto slide along the x�axis�

The equations of motion form a nonlinear index�� DAE of the form������� Using redundant� absolute coordinates� let ui� vi� �i be the co�ordinates of the center of mass of the ith bar� Then de�ne

q � �u�� v�� ��� u�� v�� ���T

M � diagfm��m��m�l�����m��m��m�l

����g

f � ��������� �� �������� ��T

Page 309: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

��� Chapter � � Di�erential�Algebraic Equations

g �

�BBBBBBBBB�

u� � l��� cos ��

v� � l��� sin ��

u� � �u� � l��� cos ��

v� � �v� � l��� sin ��

l� sin �� � l� sin��

�CCCCCCCCCA

G � gq �

�BBBBBBBBB�

� � l��� sin �� � � �

� � �l��� cos �� � � �

�� � � � � l��� sin ��

� �� � � � �l��� cos ��

� � l� cos �� � � l� cos ��

�CCCCCCCCCA�

�a� Following the lines of Example ����� derive a more compact for�mulation of the slider�crank mechanism in relative coordinates�leading to only two ODEs and one constraint� What are the ad�vantages and disadvantages of each formulation!

�b� Set m� � m� � �� l� � �� ����� ���� and ������ � �� Compute

and plot the solution for b � �� and each of the two cases �i�l� � ��� and �ii� l� � ���� Your simulation method should usethe formulation in absolute coordinates given above� and combineindex reduction and some stabilization with an ODE solver or alower index DAE solver�

Explain the qualitatively di�erent behavior observed for the dif�ferent values of l��

� This exercise continues the previous one� Set the various parametersat the same values as above� except l� � l� � �� Then the last row ofG vanishes� i�e� a singularity occurs� each time the periodic solutioncrosses a point where the two bars are upright� i�e�� ���� ��� � �

�� �

��� ��

�a� Use the same method you have used in the previous exercise tointegrate this problem� despite the singularity� Explain your ob�served results� �What you obtain may depend on the numericalmethod you use and the error tolerance you prescribe� so you areon your own� make sure the program is debugged before attempt�ing to explain the results��

�b� Explain why a stabilization method which stabilizes only withrespect to the velocity constraints Gq� � � would do signi�cantlyworse here than a method which stabilizes also with respect tothe position constraints g � �� �Hint� you should have solvedExercise �����b� before attempting this one��

�� Consider a semi�explicit index�� DAE of the form

f�t�x� z�x�� � �

g�t�x� z� � ��

Page 310: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Chapter � � Numerical Methods for Di�erential�Algebraic Equations ��

where the matrices fx� and gz are square and nonsingular�

�a� Show that to specify a solution trajectory the initial value infor�mation needed is x��� � x��

�b� The initialization problem is to �nd x�� � x���� and z� � z����Describe a solution algorithm�

�� Consider the index�� DAE ������ and the two implicit midpoint meth�ods

xn � xn��h

� f�tn�����xn � xn��

��zn � zn��

�� �������

� � g�tn�����xn � xn��

��zn � zn��

��

and

xn � xn��h

� f�tn�����xn � xn��

�� zn����� �������

� � g�tn�����xn � xn��

�� zn������

In the second method z�t� is approximated by a constant zn���� oneach subinterval �tn��� tn�� so the resulting approximate solution z��t�is discontinuous at mesh points tn�

�a� Find an example to show that ������� has better stability proper�ties than �������� �This may be challenging��

�b� Design an a posteriori process� i�e� a process that starts after thesolution to ������� has been calculated� to improve the approxi�mate values of z to be second order accurate at mesh points� Testthis on your example�

�� Consider the Hessenberg index�� DAE ������ and the midpoint method������� applied to it�

�a� Show that the global error is �nd order on a uniform mesh �i�e�using a constant step size� but only �st order on an arbitrary mesh�

�b� What is the condition on the mesh to achieve �nd order accuracy!

�� �a� Describe the projected midpoint method based on ������� andshow that the obtained solution xn is �nd order accurate�

�b� Consider the following modi�cation of ��������

xn � xn��h

� f�tn�����xn � xn��

�� zn����� �������

� � g�tn�xn�

Page 311: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

��� Chapter � � Di�erential�Algebraic Equations

where g���x�� � � is assumed� Investigate the properties of thismethod and compare it to the projected midpoint method�

�This exercise is signi�cantly more di cult than the previous two��

��� �a� Apply the midpoint method� the projected midpoint method andthe method given by ������� to the problem of Example ���� withthe same data� Describe your observations�

�b� Attempt to explain your observations�

�This may prove di cult� if in distress� see �����

��� Consider the Hessenberg index�� DAE

y� � x

x� � �z � ��t�

� � y � ��t�

where ��t�� ��t� are known� smooth functions�

�a� Formulate this DAE as an ODE with invariant�

�b� Discretize the stabilized ODE ������ with F � HT �HHT ��� usingforward Euler� What is the best choice for !

�c� Formulate the Baumgarte stabilization ������ for this simple prob�lem and discretize it using forward Euler� Try to �gure out a bestchoice for the parameters � and � for this case�

�This latter task should prove somewhat more di cult �����

��� Mechanical systems with holonomic constraints yield index�� DAEs�as we have seen� Mechanical systems with nonholonomic constraintsinvolve constraints �on the velocity level�� such as Gv � �� but whichcannot be integrated into constraints involving generalized positionsq alone� So� mechanical systems with nonholonomic constraints yieldindex�� DAEs�

Now� every budding mechanical engineer doing robotics knows thatsystems with nonholonomic constraints are more complex and di �cult than systems with holonomic constraints� whereas every buddingnumerical analyst knows that index�� DAEs are harder than index��DAEs�

Who is right� the engineers or the numerical analysts! Explain�

��� The coordinate projection method for an ODE with invariant ��������using an energy norm based on a symmetric positive de�nite matrix A�is de�ned as follows�

Page 312: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Chapter � � Numerical Methods for Di�erential�Algebraic Equations ���

� At time step n we use an ODE method for ������a� to advancefrom xn�� to &xn�

� Then xn is determined as the solution of the constrained leastsquares problem

minxn

��xn � &xn�TA�xn � &xn�

s� t� h�xn� � ��

Consider one Newton step linearizing h to solve this nonlinear system�starting from x�

n � &xn�

�a� Show that this step coincides with a post�stabilization step �������with

F � A��HT �HA��HT ����

�b� Assuming that the ODE discretization method has order p� showthat one Newton step brings the solution to within O�h��p���� ofsatisfying the constraints ������b��

�c� For the Euler�Lagrange equations� explain why it may be advan�tageous to choose A �M � where M is the mass matrix�

��� Consider the Hamiltonian system describing the motion of a �sti� re�versed pendulum� �����

q� � p

p� � ��r�q�� r��rqr � ������q�� ���rq�

where q � �q�� q��T � p � �p�� p��T � r � jqj�� rqr � r��q� � �arccos �q��r�� rq� � r����q�� q��T �all functions of t of course�� Setr� � �� �� � ��� q��� � �p

���� ��T � p��� � ��� ��T � Let also eS�t� �

�� ��rrTp�� � �r � r���� and �eS � maxt����� jeS���� eS�t�j��a� Use an initial value ODE solver to integrate this highly oscillatory

IVP for � � ����� ���� and ����� �Try also � � ���� if this is notgetting too expensive�� Record the values of �eS and conjecturethe value of this quantity in the limit �� ��

�b� Consider the DAE obtained as in Example ����

q� � p

p� � ��r�q�� r��rqr � �rq���

� � ��q�� �� �

Solve this DAE numerically subject to the same initial conditionsas above and calculate �eS� Compare to the conjectured limitfrom �a�� Conclude that this DAE is not the correct limit DAE ofthe highly oscillatory ODE problem$ ����

Page 313: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

��� Chapter � � Di�erential�Algebraic Equations

�� Many ODE problems arise in practice in the form ������� where M isa constant� possibly singular matrix� Let

M � U

��/ �

� �

�AV T

denote the singular value decomposition of M � where U and V areorthogonal matrices and / � diag���� � � � � �m�l�� with �� � �� � � � � ��m�l � ��

�a� Show that ������� can be written in the semi�explicit form �������where �

�xz

�A � V Ty

and f and g are de�ned in terms of &f � U � V and /�

�b� Show that� since the transformation V is constant and well�conditioned�any Runge�Kutta or multistep discretization applied to �������corresponds to an equivalent method applied to �������

��� �Parts �c� and �d� of this exercise are di cult and time consuming��

The following equations describe a simple steady state� one�dimensional�unipolar hydrodynamic model for semiconductors in the isentropic case����

�� � �E � �J ������a�

E � � �� � ������b�

� � J��� � � ������c�

���� � ��b� � %�� ������d�

The constants J� �� b and %� � � are given� Although you don�t needto know the physical interpretation to solve the exercise� we note that��t� is the electron density and E�t� is the �negative� electric �eld� Theindependent variable t is a space variable� This model corresponds toa current�driven ����� device� and we concentrate on one ��region�

The �ow is subsonic where � � J and supersonic where � � J � Setting%� � J � the question is if transonic �ows are possible� i�e�� is there aregion �a�� b�� � ��� b� where � � J!

�a� Show that for the subsonic or supersonic cases� ������� is an index��� boundary value DAE�

Page 314: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Chapter � � Numerical Methods for Di�erential�Algebraic Equations ���

�b� Propose a computational approach for this boundary value DAEin the strictly subsonic or the supersonic case� Solve for the valuesJ � ���� � � �� %� � �� b � ����� �You may either write your ownprogram or use appropriate software��

�c� For the transonic case� ��t� � J crosses from positive to negativeand back to positive �at least once�� and the simple DAE modelbreaks down� The solution may be discontinuous at such crossingpoints� The �Rankine�Hugoniot� condition for a shock of this sortto occur at t � t� is that the jump in � vanish across such a point�

Using phase plane analysis� show that such transonic solutionsexist for suitable parameter values$

�This part of the exercise is suitable only for those who really likethe challenge of analysis��

�d� For the numerical solution of the transonic case it seems best toabandon the DAE and regularize the problem� replace ������a�by

�� � �E � �J � ����

�a nontrivial� physically�based choice which allows � to be simplyevaluated� no longer one of the primary unknowns� and appendthe boundary condition

����%�� J������ � %�E��� � �J�

Solve the obtained boundary value ODE for J � �� � � �� %� ��� b � ���� and � � ����� Plot the solution� experiment furtherwith di�erent values of b� and discuss�

�Be warned that this may be challenging� expect interior sharp lay�ers where the DAE solution jumps� We suggest to use some goodsoftware package and to employ a continuation method �x�����starting with � � � and gradually reducing it��

Page 315: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

��� Bibliography

Page 316: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Bibliography

��� V�I� Arnold� Mathematical Methods of Classical Mechanics� Springer�Verlag� �����

��� U� Ascher� Stabilization of invariants of discretized di�erential systems�Numerical Algorithms� �������� �����

��� U� Ascher� H� Chin� L� Petzold� and S� Reich� Stabilization of constrainedmechanical systemswith DAEs and invariant manifolds� J� Mech� Struct�Machines� ��������� ����

��� U� Ascher� H� Chin� and S� Reich� Stabilization of DAEs and invariantmanifolds� Numer� Math�� ����������� �����

�� U� Ascher� J� Christiansen� and R� Russell� Collocation software forboundary value ODE�s� ACM Trans� Math Software� ���������� �����

��� U� Ascher and P� Lin� Sequential regularization methods for nonlinearhigher index DAEs� SIAM J� Scient� Comput�� ����������� �����

��� U� Ascher� P� Markowich� P� Pietra� and C� Schmeiser� A phase planeanalysis of transonic solutions for the hydrodynamic semiconductormodel� Mathematical Models and Methods in Applied Sciences� ���������� �����

��� U� Ascher� R� Mattheij� and R� Russell� Numerical Solution of Boundary Value Problems for Ordinary Di�erential Equations� SIAM� secondedition� ����

��� U� Ascher and L� Petzold� Projected implicit Runge�Kutta methods fordi�erential�algebraic equations� SIAM J� Numer� Anal�� ������������������

���� U� Ascher� S� Ruuth� and B� Wetton� Implicit�explicit methods for time�dependent PDE�s� SIAM J� Numer� Anal�� ����������� ����

���� U� Ascher and R� Spiteri� Collocation software for boundary valuedi�erential�algebraic equations� SIAM J� Scient� Comp�� ��������������

���

Page 317: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

��� Bibliography

���� U�M� Ascher and S� Reich� The midpoint scheme and variants for Hamil�tonian systems� advantages and pitfalls� ����� Manuscript�

���� G� Bader and U� Ascher� A new basis implementation for a mixed orderboundary value ODE solver� SIAM J� Scient� Comput�� ��������� �����

���� R� Barrett� M� Beary� T� Chan� J� Demmel� J� Donald� J� Dongarra�V� Eijkhout� R� Pozo� C� Romaine� and H� Van der Vorst� Templatesfor the Solution of Linear Systems� SIAM� �����

��� J� Baumgarte� Stabilization of constraints and integrals of motion indynamical systems� Comp� Methods Appl� Mech�� ������� �����

���� C� Bischof� A� Carle� G� Corliss� A� Griewank� and P� Hovland� adi�for � generating derivative codes from fortran programs� Scienti�cProgramming� �������� �����

���� G� Bock� Recent advances in parameter identi�cation techniques forODE� In P� Deu�hard and E� Hairer� editors� Numerical treatment ofinverse problems� Boston� ����� Birkhauser�

���� R�W� Brankin� I� Gladwell� and L�F� Shampine� Rksuite� A suite ofRunge�Kutta codes for the initial value problem for ODEs� Report ���s��Dept� Mathematics� SMU� Dallas� Texas� �����

���� K� Brenan� S� Campbell� and L� Petzold� Numerical Solution of InitialValue Problems in Di�erentialAlgebraic Equations� SIAM� second edi�tion� �����

���� F� Brezzi and M� Fortin� Mixed and Hybrid Finite Element Methods�Springer�Verlag� New York� �����

���� P� N� Brown� G� D� Byrne� and A� C� Hindmarsh� vode� a variable�coe cient ODE solver� SIAM J� Sci� Stat� Comput�� ������������ �����

���� A� Bryson and Y�C� Ho� Applied Optimal Control� Ginn and Co��Waltham� MA� �����

���� K� Burrage� Parallel and Sequential Methods for Ordinary Di�erentialEquations� Oxford University Press� ����

���� K� Burrage and J�C� Butcher� Stability criteria for implicit Runge�Kuttamethods� SIAM J� Numer� Anal�� �������� �����

��� K� Burrage� J�C� Butcher� and F� Chipman� An implementation ofsingly�implicit Runge�Kutta methods� BIT� ��������� �����

Page 318: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Bibliography ���

���� J� C� Butcher� The Numerical Analysis of Ordinary Di�erential Equations� Wiley� �����

���� M�P� Calvo� A� Iserles� and A� Zanna� Numerical solution of isospectral�ows� Technical report� DAMTP� Cambridge� ����

���� S� L� Campbell� Least squares completions of nonlinear di�erential�algebraic equations� Numer� Math�� �������� �����

���� S� L� Campbell� Numerical methods for unstructured higher�indexDAEs� Annals of Numer� Math�� ��������� �����

���� J�R� Cash and M�H�Wright� User�s guide for twpbvp� a code for solvingtwo�point boundary value problems� Technical report� on line in netlib������

���� H� Chin� Stabilization methods for simulations of constrained multibodydynamics� PhD thesis� Institute of Applied Mathematics� University ofBritish Columbia� ����

���� W�A� Coppel� Dichotomies in stability theory� Springer�Verlag� �����Lecture Notes in Math� Vol� ����

���� G� Dahlquist� A special stability problem for linear multistep methods�BIT� �������� �����

���� G� Dahlquist� Error analysis for a class of methods for sti� nonlinearinitial value problems� In Numerical Analysis� Dundee� pages �����Springer� ����

��� C� de Boor� Good approximation by splines with variable knots� ii� InSpringer Lecture Notes in Mathematics� ��� �����

���� C� de Boor and B� Swartz� Collocation at Gaussian points� SIAM J�Numer� Anal�� ���������� �����

���� L� Dieci� R�D� Russell� and E�S� Van Vleck� Unitary integrators andapplications to continuous orthonormalization techniques� SIAM J� Numer� Anal�� ����������� �����

���� E� Doedel and J� Kernevez� Software for continuation problems in ordi�nary di�erential equations� SIAM J� Numer� Anal�� ��������� �����

���� J�R� Dormand and P�J� Prince� A family of embedded Runge�Kuttaformulae� J� Comp� Appl� Math�� �������� �����

Page 319: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

��� Bibliography

���� E� Fehlberg� Low order classical Runge�Kutta formulas with step sizecontrol and their application to some heat transfer problems� Computing� �������� �����

���� L� Fox� The Numerical Solution of TwoPoint Boundary Value Problemsin Ordinary Di�erential Equations� Oxford University Press� ����

���� C� W� Gear� The simultaneous numerical solution of di�erential�algebraic equations� IEEE Trans� Circuit Theory� CT��������� �����

���� C� W� Gear� Numerical Initial Value Problems in Ordinary Di�erentialEquations� Prentice�Hall� �����

���� C� W� Gear and J� B� Keiper� The analysis of generalized BDF methodsapplied to Hessenberg form DAEs� SIAM J� Numer� Anal�� ���������������

��� C�W� Gear� G� Gupta� and B� Leimkuhler� Automatic integration of theEuler�Lagrange equations with constraints� J� Comput� Appl� Math��pages ������ ����

���� E� Griepentrog and R� M.arz� Di�erentialAlgebraic Equations and TheirNumerical Treatment� Teubner� �����

���� J� Guckenheimer and P� Holmes� Nonlinear Oscillations� DynamicalSystems� and Bifurcations of Vector Fields� Springer�Verlag� New York������

���� W� Hackbusch� Iterative Solution of Large Sparse Systems of Equations�Springer�Verlag� �����

���� E� Hairer� Ch� Lubich� and M� Roche� The Numerical Solution ofDi�erentialAlgebraic Systems by RungeKutta Methods� volume �����Springer�Verlag� �����

��� E� Hairer� S�P� Norsett� and G� Wanner� Solving Ordinary Di�erentialEquations I� Nonsti� Problems� Springer�Verlag� second edition� �����

��� E� Hairer and D� Sto�er� Reversible long term integration with variablestep sizes� SIAM J� Scient� Comput�� ���������� �����

��� E� Hairer and G� Wanner� Solving Ordinary Di�erential Equations II�Sti� and Di�erentialAlgebraic Problems� Springer�Verlag� �����

��� N�A� Haskell� The dispersion of surface waves in multilayered media�Bull� Seis� Soc� Am�� ��������� ����

Page 320: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Bibliography ��

��� P� Henrici� Discrete Variable Methods in Ordinary Di�erential Equations� John Wiley� �����

�� M� Hirsch� C� Pugh� and M� Shub� Invariant manifolds� volume ���Springer�Verlag� �����

��� T�E� Hull� W�H� Enright� and K�R� Jackson� User�s guide for dverk �a subroutine for solving non�sti� ODEs� Report ���� Dept� ComputerScience� U� Toronto� ����

��� A� Jameson� Computational transonics� Comm� Pure Appl� Math��XLI������� �����

��� W� Kampowsky� P� Rentrop� and W� Schmidt� Classi�cation and nu�merical simulation of electric circuits� Surv� Math� Ind�� ������� �����

��� H� B� Keller� Numerical Solution of Two Point Boundary Value Problems� SIAM� �����

���� B�L�N� Kennett� Seismic wave propagation in strati�ed media� Cam�bridge University Press� �����

���� W� Kutta� Beitrag zur n.aherungsweisen integration totaler di�erential�gleichungen� Zeitschr� f"ur Math u� Phys�� ��������� �����

���� J� D� Lambert� Numerical Methods for Ordinary Di�erential Systems�Wiley� �����

���� M� Lentini and V� Pereyra� An adaptive �nite di�erence solver for non�linear two�point boundary value problems with mild boundary layers�SIAM J� Numer� Anal�� ���������� �����

���� Ch� Lubich� U� Nowak� U� Pohle� and Ch� Engstler� mexx � numer�ical software for the integration of constrained mechanical multibodysystems� Preprint sc ������ ZIB Berlin� �����

��� Jerry B� Marion and Stephen T� Thornton� Classical Dynamics of Particles and Systems� Harcourt Brace Jovanovich� third edition� �����

���� R� M.arz� Numerical methods for di�erential�algebraic equations� ActaNumerica� ���������� �����

���� R�M�M� Mattheij and J� Molnaar� Ordinary Di�erential Equations inTheory and Practice� Wiley� Chichester� �����

���� R�M�M� Mattheij and G�W�M� Staarink� Implementing multiple shoot�ing for nonlinear BVPs� Rana ������ EUT� �����

Page 321: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

��� Bibliography

���� C�C� Pantelides� The consistent initialization of di�erential�algebraicsystems� SIAM J� Scient� Comput�� ���������� �����

���� V� Pereyra and G� Sewell� Mesh selection for discrete solution of bound�ary value problems in ordinary di�erential equations� Numer� Math������������� ����

���� L�R� Petzold� L�O� Jay� and J� Yen� Numerical solution of highly oscil�latory ordinary di�erential equations� Acta Numerica� pages �������������

���� F� Potra and W� Rheinboldt� On the numerical solution of the Euler�Lagrange equations� Mech� Structures Mach�� �� �����

���� A� Prothero and A� Robinson� On the stability and accuracy of one�step methods for solving sti� systems of ordinary di�erential equations�Math� Comp�� ���������� �����

���� P� Rabier and W� Rheinboldt� On the computation of impasse points ofquasilinear di�erential algebraic equations� Math� Comp�� ���������������

��� P� J� Rabier and W� C� Rheinboldt� A general existence and uniquenesstheorem for implicit di�erential algebraic equations� Di�� Int� Eqns���������� �����

���� P� J� Rabier and W� C� Rheinboldt� A geometric treatment of implicitdi�erential�algebraic equations� J� Di�� Eqns�� ������������ �����

���� M� Rao� Ordinary Di�erential Equations Theory and Applications� Ed�ward Arnold� �����

���� S� Reddy and N� Trefethen� Stability of the method of lines� Numer�Math�� ���������� �����

���� W�C� Rheinboldt� Di�erential�algebraic systems as di�erential equationson manifolds� Math� Comp�� ����������� �����

���� H� Rubin and P� Ungar� Motion under a strong constraining force�Comm� Pure Appl� Math�� �������� ����

���� C� Runge� Ueber die numerische au�.osung von di�erentialgleichungen�Math� Ann�� ����������� ����

���� J�M� Sanz�Serna and M�P� Calvo� Numerical Hamiltonian Problems�Chapman and Hall� �����

Page 322: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Index ���

���� T� Schlick� M� Mandziuk� R�D� Skeel� and K� Srinivas� Nonlinear reso�nance artifacts in molecular dynamics simulations� ����� Manuscript�

���� M�R� Scott and H�A� Watts� Computational solution of linear two�pointboundary value problems� SIAM J� Numer� Anal�� ��������� �����

��� L� F� Shampine� Numerical Solution of Ordinary Di�erential Equations�Chapman Hall� �����

���� L� F� Shampine and M� K� Gordon� Computer Solution of OrdinaryDi�erential Equations� W� H� Freeman and Co�� ����

���� L�F� Shampine and H�A� Watts� The art of writing a Runge�Kutta code�part i� In J�R� Rice� editor� Mathematical Software III� pages ������Academic Press� �����

���� I� Stakgold� Green�s functions and boundary value problems� Wiley������

���� H� Stetter� Analysis of Discretization Methods for Ordinary Di�erentialEquations� Springer� �����

���� G� Strang and G� Fix� An Analysis of the Finite Element Method�Prentice�Hall� Englewood Cli�s� NJ� �����

���� J�C� Strikwerda� Finite Di�erence Schemes and Partial Di�erentialEquations� Wadsworth Brooks"Cole� �����

���� S�H� Strogatz� Nonlinear dynamics and chaos� Addison�Wesley� Read�ing� MA� �����

���� A�M� Stuart and A�R� Humphries� Dynamical systems and numericalanalysis� Cambridge University Press� Cambridge� England� �����

���� J�H� Verner� Explicit Runge�Kutta methods with estimates of the localtruncation error� SIAM J� Numer� Anal�� ���������� �����

��� R�A� Wehage and E�J� Haug� Generalized coordinate partitioning fordimension reduction in analysis of constrained dynamic systems� J� ofMechanical Design� ���������� �����

���� R� Weiss� The convergence of shooting methods� BIT� ���������� �����

���� S�J� Wright� Stable parallel algorithms for two�point boundary valueproblems� SIAM J� Scient� Comput�� ����������� �����

Page 323: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Index

Imsl� ��Mathematica� �Matlab� �� ��� ���Nag� ��� ���Netlib� ��� ���� ���

Absolute stability� �����implicit Runge�Kutta methods� ���plotting the region of� ��� ���region of� ��explicit Runge�Kutta methods� �����multistep methods� ������

Accuracy� order of� ��Adams methods� �������

��stability� ���absolute stability� ���Adams�Bashforth �explicit�method� ���Adams�Moulton �implicit� method� ���

Algebraic variables �DAE�� ���Almost block diagonal� ���Arti�cial di�usion� ���Asymptotic stability� see Stability� asymp�

toticAutomatic di�erentiation� �� ���Autonomous� �� ��� ��

B�convergence� ���Backward di�erentiation formulae� see BDFBackward Euler method� ����� ���

DAE� ������region of absolute stability� �solution of nonlinear system� �

BDF methods� ���������stability� ���DAE� �������

Bifurcation diagram� ���

Boundary conditionsDirichlet� ���non�separate� ���periodic� ���� ���� ���� ���separated� ���� ���two�point� ���

Boundary layer� see LayerBoundary value problems �BVP�� �

continuation� ���damped Newton method� ���decoupling� ���� ���deferred correction� ���error estimation� ���extrapolation� ����nite di�erence methods� ���������stability� ���collocation� ���consistency� ���convergence� ���solving the linear equations� ���sti� problems� ��

for PDEs� ���in�nite interval� ���mesh selection� ���midpoint method� ���multiple shooting method� �������Newton�s method� ���reduced superposition� ���Riccati method� ���simple shooting method� �������software� ���stabilized march method� ���superposition� ���trapezoid method� ���

BVP codes

���

Page 324: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Index ���

auto� ���colnew� ���colsys� ���mus� ���pasvar� ���suport� ���twpbvp� ���

Chaos� ��Characteristic polynomial� ��Chemical reaction

BVP example� ���� ���� ���Collocation methods

basic idea� ���for BVPs� ���Gauss formulae� ���Lobatto formulae� ���order for DAEs� ���order of� ���projected� for DAE� ���Radau formulae� ���relation to implicit Runge�Kutta� ���

Compact �nite di�erence methods� ���Compacti�cation

multiple shooting method� ���Condition number

eigenvalue matrix� ��iteration matrix �DAE�� ���orthogonal matrix� �

Conservative system� ��Consistency� ��

BVPs� �nite di�erence methods� ���multistep methods� ���

Constraint manifold� ���Constraints �DAE�� hidden� ���Continuation methods� ��� �������

arclength� ���Continuous extension� ���Contraction mapping� �Convection�di�usion equation �PDE�� ���Convergence� ��

BDF methods for DAEs� ���BVPs� �nite di�erence methods� ���

calculated rate� ��multistep methods� ���of order p� ��Runge�Kutta methods� ��

Coordinate partitioning �DAE�� ��Corrector formula� ���Crank�Nicolson method for PDEs� ��

DAE codescoldae� ���daspk� ���dassl� ���mexx� ���radau�� ���

Damped Newton method� ���Decoupling� ���� ���� ���� ���

BVP� �������Decoupling methods �BVP�� ���Deferred correction method� ���Degrees of freedom �DAE�� ���Delay di�erential equation� ���� ���Dense output� ���Diagonally implicit Runge�Kutta methods

�DIRK�� ���Dichotomy� ���� ���

exponential� ���Di�erence equations� ���Di�erence operator� ��Di�erential variables �DAE�� ���Di�erential�algebraic equations �DAE�� ���

���index reduction and stabilization� ���algebraic variables� ���BDF methods� ���consistent initial conditions� ���� ���constraint stabilization� ��convergence of BDF methods� ���coordinate partitioning� ��di�erential geometric approach� ��di�erential variables� ���direct discretization methods� ���existence and uniqueness� ��fully�implicit index��� ���

Page 325: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

��� Index

Hessenberg form� ���� ��Hessenberg index��� ���Hessenberg index��� ���hidden constraints� ���higher�index� ���index reduction� unstabilized� ���index� de�nition� ��least squares methods� ���multistep methods� ���numerical methods� ���ODE with constraints� ��reformulation of higher�index DAEs� ���regularization� ���semi�explicit� ��� ���semi�explicit index��� ���simple subsystems� ���singular� ��� ���stabilization of the constraint� ��stabilized index�� formulation� ���state space formulation� ��underlying ODE� ���� ��

Di�erentiationautomatic� �symbolic� �

Discontinuitydiscretization across� ��location of� ��

Dissipativity� ���Divergence� ��� ��Divided di�erences� ���Drift o� the constraint �DAE�� ��Dry friction� ��Dynamical system� �

discrete� ���

Eigenvalue� ��Eigenvector� ��Error

constant �multistep methods�� ���equidistribution� ��� ���global� ��local� ��local truncation� ��

toleranceabsolute and relative� ��

Error estimationBVPs� ���embedded Runge�Kutta methods� ��global error� �index�� DAE� ���multistep methods� ��Runge�Kutta methods� ��step doubling� ��

Euler methodbackward �implicit�� �forward �explicit�� �symplectic� ���written as Runge�Kutta� ��

Event location� ��� ���Explicit

method� ��ODE� �

Extraneous roots� ���Extrapolation� ���� ���

Finite element method� ���Fully�implicit index�� DAEs� ���Functional iteration� �

multistep methods� ���Fundamental solution� ��� ���

in shooting method� ���Fundamental theorem� �

di�erence methods� ��

Gauss collocation� ���� ���� ��� ���� �������� ���� ���� ���� ���� ���

Gaussianpoints� ��� ���� ���quadrature� ��

Global error� ��estimates of� �� ���

Gradient� ��� ��Green�s function� ���� ��

Half�explicit Runge�Kutta methods �DAE�����

Hamiltonian� ��

Page 326: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Index ���

Hamiltonian systems� ��� ���� ���� ���invariants� ��preservation of the invariant� �

Hermite interpolation� ���Hessenberg form �DAE�� ���� ��Higher index DAEs� ���Homotopy path� ���Hopf bifurcation� ��

Implicitmethod� ��ODE� ��Runge�Kutta methods� �������implementation� ��� ���

Implicit Euler method� see Backward Eulermethod

Implicit�explicit �IMEX� methods� ���IncompressibleNavier�Stokes equations� ���Index� �������

de�nition� ��di�erential� ��perturbation� ��reductionstabilized index�� formulation� ���unstabilized� ���

Initial conditions� consistent �DAE�� ���Initial layer� see LayerInitial value problem �IVP�� �Instability

DAE� drift o� the constraint� ��Interpolating polynomial and divided dif�

ferencesreview� ���

Invariantintegral� ���� ��ODE with� ���

Invariant set� �� ��� ���� ���� ��� ���Isolated solution �BVP�� ��Isolated solution �IVP�� ��Isospectral �ow� ���Iteration matrix� �

Jacobian matrix� �� ��di�erence approximation� �

Kepler problem� modi�ed� ���Kronecker product

review� ��Krylov space methods� ��

Lagrange multiplier� ��DAEs and constrained optimization� ���

Layerboundary� ��� ���� ��� �������� ���initial� ��� �� ��� ���

Leapfrog �Verlet� method� ���� ��Limit cycle� � ��Limit set� �Linearization� local� ��Lipschitz

constant� �continuity� �� ��

Lobatto collocation� ���� ���� ���� ���Local elimination� ���Local error� ��

control of� in Runge�Kutta methods� ��estimation by step doubling� ��relationship to local truncation error�

��Local extrapolation� ��Local truncation error� ��� ��

BVPs� �nite di�erence methods� ���estimation of �multistep methods�� ��multistep methods� ���principal term �multistepmethods�� ��relation to local error� ��

Long time integration� ���Lyapunov function� ��

Matrixbanded� �sparse� �

Matrix decompositionsLU � QR� �review� �

Matrix eigenvaluesreview� ��

Matrix exponential

Page 327: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

��� Index

review� ��Mechanical systems� ��� ���

generalized coordinate partitioning method���

reformulation of higher�index DAEs� ���Mesh� �

locally almost uniform� ���Mesh function� ��Mesh Reynolds number� ���Mesh selection �BVP�� ���Method of lines� �� ��� ��� ���� ���� ���

heat equation stability restriction� ��transverse� ��

Midpoint method� ��� ���dynamic equivalence to trapezoid method�

��explicit� ��explicit� written as Runge�Kutta� ��staggered� ��

Milne�s estimatelocal truncation error �predictor�corrector

methods�� ��Milne�s method �multistep method�� ���Mode

solution� ��� ���Model reduction� ��Molecular dynamics� ���Moving mesh method �PDEs�� �Multiple shooting method� ���

compacti�cation� ���matrix� ��� ���on parallel processors� ��patching conditions� ���

Multiple time scales� ��Multirate method� ���Multistep codes

daspk� ��dassl� ��difsub� ��ode� ��vode� ��vodpk� ��

Multistep methods� ��

absolute stability� ���Adams methods� ���BDF� ���characteristic polynomials� ���consistency� ���DAE� ���error constant� ���implementation� ���initial values� ���local truncation error� ���order of accuracy� ���predictor�corrector� ���software design� ���variable step�size formulae� ��

Newton iterationbackward Euler method� �DAE� ���di�erence approximation� �implicit Runge�Kutta methods� ��in shooting method� ���

Newton�s methoddamped� ���modi�ed� ���quasi�Newton� ���review� �

Newton�Kantorovich Theorem� ���Nonautonomous ODE

transformation to autonomous form� ��

ODEexplicit� �implicit� ��� ��linear constant�coe cient system� ��on a manifold� ���with constraints� ���with invariant� ���� ��

O��step points �multistep methods�� �One�step methods� ��Optimal control� ��

adjoint variables� ��Hamiltonian function� ��

Order notationreview� ��

Page 328: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Index ���

Order of accuracymultistep methods� ���Runge�Kutta methods� ��� ��� ��� ���Runge�Kutta methods for DAEs� ���

Order reduction� �������DIRK� for DAEs� ���in BVPs� ���Runge�Kutta methods �DAE�� ���

Order selection �multistep methods�� ��Order stars� ���Oscillator� harmonic� ��� ��Oscillatory system� �� ���� �

Parallel methodRunge�Kutta� ���

Parallel shooting method� ��Parameter condensation� ���Parameter estimation� �Parasitic roots� ���Partial di�erential equation �PDE�� ��� ����

���Path following� ���Pendulum� sti� spring� ���Perturbations

initial data� ��inhomogeneity� ��

Preconditioning� ��Predator�prey model� �Predictor polynomial� ��Predictor�corrector methods� ���Principal error function� �Principal root� ���Projected collocation methods� ���Projected Runge�Kutta methods� ���Projection matrix� orthogonal� ���

Quadrature rulesreview� �

Quasilinearization� �������with midpoint method for BVPs� ���

Radau collocation� ���� ���� ���� ���� ���Reduced solution� �� ���Reduced superposition� ���

Reformulation� boundary value problems����

Regularization �DAE�� ���Review

Basic quadrature rules� �Kronecker product� ��Matrix decompositions� �Matrix eigenvalues� ��Matrix exponential� ��Newton�s method� �Order notation� ��Taylor�s theorem for a function of sev�

eral variables� ��The interpolating polynomial and di�

vided di�erences� ���Riccati method� ���� ���Root condition� ���Rough problems� ��Runge�Kutta codes

dopri�� ���dverk� ���ode�� �Matlab�� ���radau�� ���rkf��� ���rksuite� ���stride� ���

Runge�Kutta methodsabsolute stability� ��� ���Butcher tree theory� ���DAE� �������diagonally implicit �DIRK�� ���Dormand Prince ��� embedded pair�

��embedded methods� ��explicit� ��Fehlberg ��� embedded pair� ��fourth order classical� ��� ��� ��general formulation� ��half�explicit� for DAEs� ���historical development� ���implicit� ���low order� ��mono�implicit� ���

Page 329: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

��� Index

order barriers� ���order of accuracy by Butcher trees� ��order results for DAEs� ���projected� for DAEs� ���singly diagonally implicit �SDIRK�� ���singly implicit �SIRK�� ���

Semi�explicit index�� DAE� ���Sensitivity

analysis� ��� ���boundary value problems� ���parameters� ��

Shooting method� ���algorithm description� ���di culties� ���di culties for nonlinear problems� ���multiple shooting method� ���simple shooting� ���single shooting� ���stability considerations� ���

Similarity transformation� ��� ��Simple pendulum� �� ��� ���� ���Singly diagonally implicit �SDIRK� Runge�

Kutta methods� ���Singly implicitRunge�Kutta methods �SIRK��

���Singular perturbation problems

relation to DAEs� ���Smoothing� ���Sparse linear system� ���� ���Spectral methods �PDEs�� ���Spurious solution� ���Stability

��stability� ��� ��� ��� �������� �������A�stability� �� ���� ���absolute stability� ��algebraic stability� ���AN�stability� ��asymptoticdi�erence equations� ���

asymptotic� of the constraint manifold���

boundary value ODE� �������

di�erence equations� ���initial value DAE� ������initial value ODE� �����asymptotic� ��� �� ��nonlinear� ��

relative stability� ���resonance instability� ���root condition �multistepmethods�� ���scaled stability region� ���strong stability �multistepmethods�� ���weak stability �multistepmethods�� ���

Stability constantboundary value problem� ���� ��� ���initial value problem� ��

StabilizationBaumgarte� ��� ���coordinate projection �DAE�� ���� ���of the constraint �DAE�� ��post�stabilization �DAE�� ���

Stabilized index�� formulation �DAE�� ���Stabilized march method� ���Stage order� ���State space formulation �DAE�� ��Steady state� ��� ���Step size� �Step size selection

multistep methods� ��Runge�Kutta methods� ��

Sti� boundary value problems�nite di�erence methods� ��

Sti� decay� �� �� ���� ���� ���� ���DAE� ���ODE methods for DAEs� ���

Sti)y accurate� ���� ���Sti�ness� ��

boundary value problems� ���de�nition� ��system eigenvalues� ��transient� ��

Strange attractor� ��Superposition method� ���Switching function� ��Symmetric methods� ����

Page 330: Ebooksclub.org Computer Methods for Ordinary Differential Equations and Differential Algebraic Equations

Index ��

Symmetric Runge�Kutta methods� ���Symplectic map� ��Symplectic methods� ���

Taylor series method� ��Taylor�s theorem� several variables

review� ��Test equation� ��� ��Theta method� ��Transformation

decoupling �BVP�� ���decoupling �DAE�� ���

Transformation� stretching� ��Trapezoid method� �� ���

derivation� �dynamic equivalence to midpointmethod�

��explicit� ��explicit� written as Runge�Kutta� ��

Upstream di�erence� ���� ���Upwind di�erence� ���� ���

Variable step size multistep methods�xed leading�coe cient strategy� ��variable�coe cient strategy� ��

Variationalboundary value problem� ��equation� ��� ���

Vibrating spring� �� �

Waveform relaxation� ���Well�posed problem� �

continuous dependence on the data� �existence� �uniqueness� �