Functions of Matrices Nick Higham School of Mathematics The University of Manchester http://www.maths.manchester.ac.uk/~higham/misc/ggsss13.php @nhigham, nickhigham.wordpress.com Gene Golub SIAM Summer School 2013 Fudan University, Shanghai
Research Matters
February 25, 2009
Nick HighamDirector of Research
School of Mathematics
1 / 6
Functions of Matrices
Nick HighamSchool of Mathematics
The University of Manchester
http://www.maths.manchester.ac.uk/~higham/misc/ggsss13.php
@nhigham, nickhigham.wordpress.com
Gene Golub SIAM Summer School 2013Fudan University, Shanghai
An Interview with Gene Golub1
by Nicholas J. Higham2
On July 3, 2005 I interviewed Gene Golub (1932–2007) during a visit he made toThe University of Manchester to attend a workshop. This document provides an editedtranscript of the interview.
The bibliography contains some books and papers mentioned directly or indirectlyin the interview. For more on Golub and his work, I highly recommend Milestones inMatrix Computation: The Selected Works of Gene H. Golub, with Commentaries [3].
I am grateful to Louise Stait for transcribing the interview, and to Gail Corbett andSven Hammarling for help with the editing.
Gene Golub, July 2005. Photograph by N. J. Higham.
Gene Golub by John de Pillis (http://math.ucr.edu/˜jdp), 2007.
1Document dated February 5, 2008.2School of Mathematics, The University of Manchester, Manchester, M13 9PL, UK
([email protected], http://www.ma.man.ac.uk/˜higham).
1
An InterviewwithGene Golub(H, 2008).
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
Free SIAM Student Membership
All graduate students attending the Summer School areeligible for free SIAM Student Membership, which givesyou
Free membership in two SIAM Activity Groups (SIAG).Subscription to SIAM News.Subscription to SIAM Review (electronic).30% discount on all SIAM books.
I will be happy to nominate you.
University of Manchester Nick Higham Matrix Functions 3 / 162
http://www.siam.org/students/memberships.php
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
Outline
1 History2 Definitions3 Properties, Formulas4 Applications5 Fréchet Derivative & Condition No.6 Problem Classification7 Methods for f (A)8 Methods for f (A)b9 Software
10 Case11 Quiz12 Extras
University of Manchester Nick Higham Matrix Functions 5 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
Cayley and Sylvester
Term “matrix” coined in 1850by James Joseph Sylvester,FRS (1814–1897).
Matrix algebra developed byArthur Cayley, FRS (1821–1895).Memoir on the Theory of Ma-trices (1858).
University of Manchester Nick Higham Matrix Functions 6 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
Cayley and Sylvester on Matrix Functions
Cayley considered matrix squareroots in his 1858 memoir.
Tony Crilly, Arthur Cayley: Mathemati-cian Laureate of the Victorian Age,2006.
Sylvester (1883) gave first defini-tion of f (A) for general f .
Karen Hunger Parshall, James JosephSylvester. Jewish Mathematician in aVictorian World, 2006.
University of Manchester Nick Higham Matrix Functions 7 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
Buchheim’s Formula
Buchheim (1886) extended Sylvester’s 1883interpolation formula to arbitrary eigenvalues:
University of Manchester Nick Higham Matrix Functions 8 / 162
Laguerre (1867):
Peano (1888):
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
Matrices in Applied Mathematics
Frazer, Duncan & Collar, Aerodynamics Division ofNPL: aircraft flutter, matrix structural analysis.
Elementary Matrices & Some Applications toDynamics and Differential Equations, 1938.Emphasizes importance of eA.
Arthur Roderick Collar, FRS(1908–1986): “First book to treatmatrices as a branch of appliedmathematics”.
University of Manchester Nick Higham Matrix Functions 10 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
What’s New?
Advances in Matrix Functions and Matrix Equations,University of Manchester, April 10–12, 2013.
Workshop report.
University of Manchester Nick Higham Matrix Functions 12 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
Outline
1 History2 Definitions3 Properties, Formulas4 Applications5 Fréchet Derivative & Condition No.6 Problem Classification7 Methods for f (A)8 Methods for f (A)b9 Software
10 Case11 Quiz12 Extras
University of Manchester Nick Higham Matrix Functions 13 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
Defining by Substitution
Want to define f : Cn×n → Cn×n, but not elementwise.Given f (t), can define f (A) by substituting A for t :
f (t) =1 + t2
1− t⇒ f (A) = (I − A)−1(I + A2).
log(1 + x) = x − x2
2+
x3
3− x4
4+ · · · , |x | < 1
⇒ log(I + A) = A− A2
2+
A3
3− A4
4+ · · · , ρ(A) < 1.
Works for fa polynomial,a rational,or with a convergent power series.
University of Manchester Nick Higham Matrix Functions 14 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
Jordan Canonical Form
Z−1AZ = J = diag(J1, . . . , Jp), Jk︸︷︷︸mk×mk
=
λk 1
λk. . .. . . 1
λk
Definition
f (A) = Zf (J)Z−1 = Zdiag(f (Jk))Z−1,
f (Jk) =
f (λk) f ′(λk) . . .
f (mk−1)(λk)
(mk − 1)!
f (λk). . . .... . . f ′(λk)
f (λk)
.
University of Manchester Nick Higham Matrix Functions 15 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
“Deriving” The Formula for f (Jk)
Write Jk = λk I + Ek ∈ Cmk×mk . For mk = 3 we have
Ek =
0 1 00 0 10 0 0
, E2k =
0 0 10 0 00 0 0
, E3k = 0.
Assume f has Taylor expansion
f (t) = f (λk) + f ′(λk)(t − λk) + · · ·+f (j)(λk)(t − λk)
j
j!+ · · · .
Then
f (Jk) = f (λk)I + f ′(λk)Ek + · · ·+f (mk−1)(λk)E
mk−1k
(mk − 1)!.
University of Manchester Nick Higham Matrix Functions 16 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
Interpolation (1)
Definition (Sylvester, 1883; Buchheim, 1886)Distinct e’vals λ1, . . . , λs, ni = max size of Jordan blocks forλi . Then f (A) = p(A), where p is unique Hermiteinterpolating poly of degree <
∑si=1 ni satisfying
p(j)(λi) = f (j)(λi), j = 0 : ni − 1, i = 1 : s.
Example. Let f (t) = t1/2, A =
[2 21 3
], λ(A) = 1,4.
Taking +ve square roots,
r(t) = f (1)t − 41− 4
+ f (4)t − 14− 1
=13(t + 2).
⇒ A1/2 = r(A) =13(A + 2I) =
13
[4 21 5
].
University of Manchester Nick Higham Matrix Functions 17 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
Interpolation (1)
Definition (Sylvester, 1883; Buchheim, 1886)Distinct e’vals λ1, . . . , λs, ni = max size of Jordan blocks forλi . Then f (A) = p(A), where p is unique Hermiteinterpolating poly of degree <
∑si=1 ni satisfying
p(j)(λi) = f (j)(λi), j = 0 : ni − 1, i = 1 : s.
Example. Let f (t) = t1/2, A =
[2 21 3
], λ(A) = 1,4.
Taking +ve square roots,
r(t) = f (1)t − 41− 4
+ f (4)t − 14− 1
=13(t + 2).
⇒ A1/2 = r(A) =13(A + 2I) =
13
[4 21 5
].
University of Manchester Nick Higham Matrix Functions 17 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
Interpolation (2)
Propertiesf (A) = r(A) is a polynomial in A.Poly r depends on A.f (A) commutes with A.
f (AT ) = f (A)T .
University of Manchester Nick Higham Matrix Functions 18 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
Cayley–Hamilton Theorem
Theorem (Cayley, 1857)If A,B ∈ Cn×n, AB = BA, and f (x , y) = det(xA− yB) thenf (B,A) = 0.
Usual Cayley–Hamilton is:p(t) = det(tI − A) implies p(A) = 0.An =
∑n−1k=0 cnAk .
eA =∑n−1
k=0 dnAk .
University of Manchester Nick Higham Matrix Functions 19 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
Cauchy Integral Theorem
Definition
f (A) =1
2πi
∫Γ
f (z)(zI − A)−1 dz,
where f is analytic on and inside a closed contour Γ thatencloses λ(A).
University of Manchester Nick Higham Matrix Functions 20 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
Schwerdtfeger’s Formula (1938)
DefinitionFor A with distinct e’vals λ1, . . . , λs with indices ni ,
f (A) =s∑
i=1
Ai
ni−1∑j=0
f (j)(λi)
j!(A− λi I)j =
s∑i=1
ni−1∑j=0
f (j)(λi)Zij ,
Ai are Frobenius covariants, Zij depend on A but not f .
University of Manchester Nick Higham Matrix Functions 21 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
Multiplicity of Definitions
There have been proposed in the literature since 1880eight distinct definitions of a matric function,
by Weyr, Sylvester and Buchheim,Giorgi, Cartan, Fantappiè, Cipolla,
Schwerdtfeger and Richter.
— R. F. Rinehart,The Equivalence of Definitions
of a Matric Function (1955)
University of Manchester Nick Higham Matrix Functions 22 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
Equivalence of Definitions
TheoremThe four definitions are equivalent, modulo analyticityassumption for Cauchy.
Interpolation: for basic properties.JCF: for solving matrix equations (e.g., X 2 = A,eX = A). For evaluation (normal A).Cauchy: various uses.
For computation:
Use the definitions (with care).Schur decomposition for general f .Methods specific to particular f and A.
University of Manchester Nick Higham Matrix Functions 23 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
Equivalence of Definitions
TheoremThe four definitions are equivalent, modulo analyticityassumption for Cauchy.
Interpolation: for basic properties.JCF: for solving matrix equations (e.g., X 2 = A,eX = A). For evaluation (normal A).Cauchy: various uses.
For computation:
Use the definitions (with care).Schur decomposition for general f .Methods specific to particular f and A.
University of Manchester Nick Higham Matrix Functions 23 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
Root Oddities (1)
B2n = In, where
B4 =
1 1 1 10 −1 −2 −30 0 1 30 0 0 −1
.Arises in BDF solvers for ODEs.
Turnbull (1927): A3n = In, where
A4 =
−1 1 −1 1−3 2 −1 0−3 1 0 0−1 0 0 0
.
University of Manchester Nick Higham Matrix Functions 24 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
Root Oddities (1)
B2n = In, where
B4 =
1 1 1 10 −1 −2 −30 0 1 30 0 0 −1
.Arises in BDF solvers for ODEs.Turnbull (1927): A3
n = In, where
A4 =
−1 1 −1 1−3 2 −1 0−3 1 0 0−1 0 0 0
.University of Manchester Nick Higham Matrix Functions 24 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
Root Oddities (2)
C2n = I, where
C4 = 2−3/2
1 3 3 11 1 −1 −11 −1 −1 11 −3 3 −1
.
Hill (1932): US patent for involutory matrices incryptography.
Bauer (2002): “since then the value of mathematicalmethods in cryptology has been unchallenged.”
University of Manchester Nick Higham Matrix Functions 25 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
Root Oddities (2)
C2n = I, where
C4 = 2−3/2
1 3 3 11 1 −1 −11 −1 −1 11 −3 3 −1
.
Hill (1932): US patent for involutory matrices incryptography.
Bauer (2002): “since then the value of mathematicalmethods in cryptology has been unchallenged.”
University of Manchester Nick Higham Matrix Functions 25 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
Nonprimary Matrix Functions
If A is derogatory, and a different branch of f is taken in thetwo different Jordan blocks for λ, a nonprimary matrixfunction of A is obtained.
I2 =
[1 00 1
]2
=
[−1 00 −1
]2
primary
=
[1 00 −1
]2
=
[cos θ sin θsin θ − cos θ
]2
nonprimary
Primary: expressible as a polynomial in A.Nonprimary: never so-expressible.
Not all nonprimary functions are obtainable from the JCFdefinition. Consider A = 0.
University of Manchester Nick Higham Matrix Functions 26 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
Square Roots of Rotations
G(θ) =
[cos θ sin θ− sin θ cos θ
].
G(θ/2) is the natural square root of G(θ).
For θ = π,
G(π) =
[−1 00 −1
], G(π/2) =
[0 1−1 0
].
G(π/2) is a nonprimary square root.
Virtually all existing theory and methods are for primaryfunctions.
University of Manchester Nick Higham Matrix Functions 27 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
Matrix Square Root Example
Find a matrix X such that
X 2 = A =
1 1 00 1 00 0 1
.
A solution is
X =
1 1/2 00 1 00 0 1
.All square roots are given by ±X and
Y = ±U
1 1/2 00 1 00 0 −1
U−1, U =
a b d0 a 00 e c
.
University of Manchester Nick Higham Matrix Functions 28 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
Matrix Square Root Example
Find a matrix X such that
X 2 = A =
1 1 00 1 00 0 1
.A solution is
X =
1 1/2 00 1 00 0 1
.
All square roots are given by ±X and
Y = ±U
1 1/2 00 1 00 0 −1
U−1, U =
a b d0 a 00 e c
.
University of Manchester Nick Higham Matrix Functions 28 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
Matrix Square Root Example
Find a matrix X such that
X 2 = A =
1 1 00 1 00 0 1
.A solution is
X =
1 1/2 00 1 00 0 1
.All square roots are given by ±X and
Y = ±U
1 1/2 00 1 00 0 −1
U−1, U =
a b d0 a 00 e c
.University of Manchester Nick Higham Matrix Functions 28 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
Some Open Problems
Find function of minimal norm (possibly nonprimary) orwith particular structure.Does a stochastic A have a stochastic pth root? (H &Lin, 2011).
Very little work. Singer & Spilerman (1976) remains one ofthe best references.
University of Manchester Nick Higham Matrix Functions 29 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
Principal Logarithm and Root
Let A ∈ Cn×n have no eigenvalues on R− .
Principal logX = log A denotes unique X such that
eX = A.−π < Im
(λ(X )
)< π.
Principal pth root
For integer p > 0, X = A1/p is unique X such thatX p = A.−π/p < arg(λ(X )) < π/p.
University of Manchester Nick Higham Matrix Functions 30 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
Principal Logarithm and Root
Let A ∈ Cn×n have no eigenvalues on R− .
Principal logX = log A denotes unique X such that
eX = A.−π < Im
(λ(X )
)< π.
Principal pth root
For integer p > 0, X = A1/p is unique X such thatX p = A.−π/p < arg(λ(X )) < π/p.
University of Manchester Nick Higham Matrix Functions 30 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
Principal Power
Arbitrary Power
For s ∈ R, As = es log A, where log A is the principallogarithm.
As =sin(sπ)
sπA∫ ∞
0(t1/sI + A)−1 dt , s ∈ (0,1).
University of Manchester Nick Higham Matrix Functions 31 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
Outline
1 History2 Definitions3 Properties, Formulas4 Applications5 Fréchet Derivative & Condition No.6 Problem Classification7 Methods for f (A)8 Methods for f (A)b9 Software
10 Case11 Quiz12 Extras
University of Manchester Nick Higham Matrix Functions 32 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
Basic Properties
f (XAX−1) = Xf (A)X−1;
E’vals of f (A) are f (λi), where the λi are theeigenvalues of A (but f may change the Jordanstructure);
if A = (Aij) is block triang then F = f (A) is block triangwith the same block structure as A, and Fii = f (Aii);
University of Manchester Nick Higham Matrix Functions 33 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
More Advanced
f (A) = 0 iff (with JCF as earlier)
f (j)(λi) = 0, j = 0 : ni − 1, i = 1 : s.
Sum, product, composition of functions work “asexpected”:
(sin+ cos)(A) = sin A + cos A,f (t) = cos(sin t) ⇒ f (A) = cos(sin A),
Functional relations preserved: G(f1, . . . , fp) = 0, whereG is a polynomial. E.g.
sin2 A + cos2 A = I,(A1/p)p = A,eiA = cos A + i sin A.
University of Manchester Nick Higham Matrix Functions 34 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
What Can Go Wrong?
f (A∗) 6= f (A)∗ in general.
elog A = A but log eA 6= A in general.
(AB)1/2 6= A1/2B1/2 in general.
eA 6= (eA/α)α in general.
e(A+B)t = eAteBt for all t if and only if AB = BA.
University of Manchester Nick Higham Matrix Functions 35 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
Function of 2× 2 Triangular Matrix
f([
λ1 t12
0 λ2
])=
[f (λ1) t12f [λ1, λ2]
0 f (λ2)
],
where
f [λ1, λ2] =
f (λ2)− f (λ1)
λ2 − λ1, λ1 6= λ2,
f ′(λ1), λ1 = λ2.
Note: (1,2) element is prone to cancellation.
Proof?
University of Manchester Nick Higham Matrix Functions 36 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
Function of n × n Triangular Matrix
Theorem (Davis, 1973; Descloux, 1963; Van Loan, 1975)If T is upper triangular, so is F = f (T ) and fii = f (tii),
fij =∑
(s0,...,sk )∈Sij
ts0,s1ts1,s2 . . . tsk−1,sk f [λs0 , . . . , λsk ],
whereλi = tii ,Sij is set of all strictly increasing sequences of integersstarting at i and ending at j, andf [λs0 , . . . , λsk ] is kth order divided difference.
University of Manchester Nick Higham Matrix Functions 37 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
Diagonalizable Matrices
Theorem
Let D be an open subset of R or C and let f be n − 1 timescontinuously differentiable on D. Then f (A) = 0 for allA ∈ Cn×n with spectrum in D if and only if f (A) = 0 for alldiagonalizable A ∈ Cn×n with spectrum in D.
Theorem (Richter)
For A ∈ Cn×n with no eigenvalues on R−,
log A =
∫ 1
0(A− I)
[t(A− I) + I
]−1 dt .
University of Manchester Nick Higham Matrix Functions 38 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
Outline
1 History2 Definitions3 Properties, Formulas4 Applications5 Fréchet Derivative & Condition No.6 Problem Classification7 Methods for f (A)8 Methods for f (A)b9 Software
10 Case11 Quiz12 Extras
University of Manchester Nick Higham Matrix Functions 39 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
Toolbox of Matrix Functions
d2ydt2 + Ay = 0, y(0) = y0, y ′(0) = y ′0
has solution
y(t) = cos(√
At)y0 +(√
A)−1 sin(
√At)y ′0.
But [y ′
y
]= exp
([0 −tA
t In 0
])[y ′0y0
].
In software want to be able to evaluate interesting f atmatrix args as well as scalar args.MATLAB has expm, logm, sqrtm, funm.
University of Manchester Nick Higham Matrix Functions 40 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
Toolbox of Matrix Functions
d2ydt2 + Ay = 0, y(0) = y0, y ′(0) = y ′0
has solution
y(t) = cos(√
At)y0 +(√
A)−1 sin(
√At)y ′0.
But [y ′
y
]= exp
([0 −tA
t In 0
])[y ′0y0
].
In software want to be able to evaluate interesting f atmatrix args as well as scalar args.MATLAB has expm, logm, sqrtm, funm.
University of Manchester Nick Higham Matrix Functions 40 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
Toolbox of Matrix Functions
d2ydt2 + Ay = 0, y(0) = y0, y ′(0) = y ′0
has solution
y(t) = cos(√
At)y0 +(√
A)−1 sin(
√At)y ′0.
But [y ′
y
]= exp
([0 −tA
t In 0
])[y ′0y0
].
In software want to be able to evaluate interesting f atmatrix args as well as scalar args.MATLAB has expm, logm, sqrtm, funm.
University of Manchester Nick Higham Matrix Functions 40 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
Linear, Constant Coefficient ODE
In nuclear magnetic resonance (NMR) spectroscopy,Solomon equations
dMdt
= −RM, M(0) = I,
where M(t) is matrix of intensities and R a symmetricrelaxation matrix. Thus M(t) = e−Rt .
NMR workers need to solve both forward and inverseproblems.
Burnup calculations in nuclear reactor analysis involve
dXdt
= AX , X (0) = X0,
where A is often upper triangular.University of Manchester Nick Higham Matrix Functions 41 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
Application to Complex NetworksAdjacency matrices of undirected network:
1 2
43 A =
0 0 1 00 0 1 11 1 0 10 1 1 0
.Network measures (Estrada et al., 2005–)
Centrality: (eA)ii , how important node i is.Communicability: (eA)ij , how well information istransferred between nodes i and j .
Can use resolvent (I − αA)−1 in place of eA.trace(cosh(A))/trace(eA) is measure of how closegraph is to bipartite.
University of Manchester Nick Higham Matrix Functions 42 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
The Average Eye
First order character of optical system characterized bytransference matrix
T =
[S δ0 1
]∈ R5×5,
where S ∈ R4×4 is symplectic:
ST JS = J =
[0 I2−I2 0
].
Average m−1∑mi=1 Ti is not a transference matrix.
Harris (2005) proposes the average exp(m−1∑mi=1 log(Ti)).
University of Manchester Nick Higham Matrix Functions 43 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
A1/2b: Application in StatisticsChen, Anitescu & Saad (2011): sample y ∼ N(µ,C),C ∈ Rm×m, m ∈ [1012,1015].
Let x ∼ N(0, I).
If C = LLT is Cholesky factorization,y = µ+ Lx ∼ N(µ,C).
Cholesky factorization may not be computable orstorable.
y = µ+ C1/2x ∼ N(µ,C).
University of Manchester Nick Higham Matrix Functions 44 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
Markov Models (1)
Time-homogeneous continuous-time Markov processwith transition probability matrix P(t) ∈ Rn×n.Transition intensity matrix Q: qij ≥ 0 (i 6= j),∑n
j=1 qij = 0, P(t) = eQt .
For discrete-time Markov processes:
Embeddability problemWhen does a given stochastic P havea real logarithm Q that is an intensitymatrix?
University of Manchester Nick Higham Matrix Functions 45 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
Markov Models (2)—Example
With x = −e−2√
3π ≈ −1.9× 10−5,
P =13
1 + 2x 1− x 1− x1− x 1 + 2x 1− x1− x 1− x 1 + 2x
.P diagonalizable, Λ(P) = 1, x , x.Every primary log complex (can’t have complexconjugate ei’vals).Yet a generator is the non-primary log
Q = 2√
3π
−2/3 1/2 1/61/6 −2/3 1/21/2 1/6 −2/3
.University of Manchester Nick Higham Matrix Functions 46 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
Markov Models (3)–Practicalities
Let P be transition probability matrix for discrete-timeMarkov process.If P is transition matrix for 1 year,P(1/12) = P1/12 = e
112 log P is matrix for 1 month.
Problem: log P, P1/k may have wrong sign patterns⇒“regularize”.In credit risk, P is strictly diagonally dominant.
University of Manchester Nick Higham Matrix Functions 47 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
Chronic Disease Example
Estimated 6-month transition matrix.Four AIDS-free states and 1 AIDS state.2077 observations (Charitos et al., 2008).
P =
0.8149 0.0738 0.0586 0.0407 0.01200.5622 0.1752 0.1314 0.1169 0.01430.3606 0.1860 0.1521 0.2198 0.08150.1676 0.0636 0.1444 0.4652 0.1592
0 0 0 0 1
.Want to estimate the 1-month transition matrix.
Λ(P) = 1,0.9644,0.4980,0.1493,−0.0043.
H & Lin (2011).Lin (2011) for survey of regularization methods.
University of Manchester Nick Higham Matrix Functions 48 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
Phi Functions: Definition
ϕ0(z) = ez , ϕ1(z) =ez − 1
z, ϕ2(z) =
ez − 1− zz2 , . . .
ϕk+1(z) =ϕk(z)− 1/k !
z.
ϕk(z) =∞∑
j=0
z j
(j + k)!.
University of Manchester Nick Higham Matrix Functions 49 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
Phi Functions: Solving DEs
y ∈ Cn, A ∈ Cn×n.
dydt
= Ay , y(0) = y0 ⇒ y(t) = eAty0.
dydt
= Ay + b, y(0) = 0 ⇒ y(t) = t ϕ1(tA)b.
dydt
= Ay + ct , y(0) = 0 ⇒ y(t) = t2ϕ2(tA)c.
...
University of Manchester Nick Higham Matrix Functions 50 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
Phi Functions: Solving DEs
y ∈ Cn, A ∈ Cn×n.
dydt
= Ay , y(0) = y0 ⇒ y(t) = eAty0.
dydt
= Ay + b, y(0) = 0 ⇒ y(t) = t ϕ1(tA)b.
dydt
= Ay + ct , y(0) = 0 ⇒ y(t) = t2ϕ2(tA)c.
...
University of Manchester Nick Higham Matrix Functions 50 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
Phi Functions: Solving DEs
y ∈ Cn, A ∈ Cn×n.
dydt
= Ay , y(0) = y0 ⇒ y(t) = eAty0.
dydt
= Ay + b, y(0) = 0 ⇒ y(t) = t ϕ1(tA)b.
dydt
= Ay + ct , y(0) = 0 ⇒ y(t) = t2ϕ2(tA)c.
...
University of Manchester Nick Higham Matrix Functions 50 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
Exponential Integrators
Considery ′ = Ly + N(y), y(0) = y0.
N(y(t)) ≈ N(y0) implies
y(t) ≈ etLy0 + tϕ1(tL)N(y0).
Exponential Euler method:
yn+1 = ehLyn + hϕ1(hL)N(yn).
Lawson (1967); recent resurgence.
University of Manchester Nick Higham Matrix Functions 51 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
Implementation of Exponential Integrators
u ′(t) = Au(t) + g(t ,u(t)), u(0) = u0, t ≥ 0.
Let uk = g(k−1)(t ,u(t)) |t=0 and ϕ`(z) =∑∞
k=0 zk/(k + `)!.We need to compute
u(t) = etAu0 +∑p
k=1 ϕk(tA)tk uk .
University of Manchester Nick Higham Matrix Functions 52 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
Saad’s Trick (1992)
ϕ1(z) =ez − 1
z.
exp([
A b0 0
])=
[eA ϕ1(A)b0 1
]
University of Manchester Nick Higham Matrix Functions 53 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
Evaluating Sum of Phi Functions
Theorem (Al-Mohy & H, 2011)
Let A ∈ Cn×n, U = [u1,u2, . . . ,up] ∈ Cn×p, τ ∈ C, and define
B =
[A U0 J
]∈ C(n+p)×(n+p), J =
[0 Ip−1
0 0
]∈ Cp×p.
Then for X = eτB we have
X (1 : n,n + j) =∑j
k=1 τk ϕk(τA)uj−k+1, j = 1 : p.
u(t) =[
In 0]
exp(
t[
A U0 J
])[u0
ep
].
University of Manchester Nick Higham Matrix Functions 54 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
Evaluating Sum of Phi Functions
Theorem (Al-Mohy & H, 2011)
Let A ∈ Cn×n, U = [u1,u2, . . . ,up] ∈ Cn×p, τ ∈ C, and define
B =
[A U0 J
]∈ C(n+p)×(n+p), J =
[0 Ip−1
0 0
]∈ Cp×p.
Then for X = eτB we have
X (1 : n,n + j) =∑j
k=1 τk ϕk(τA)uj−k+1, j = 1 : p.
u(t) =[
In 0]
exp(
t[
A U0 J
])[u0
ep
].
University of Manchester Nick Higham Matrix Functions 54 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
Outline
1 History2 Definitions3 Properties, Formulas4 Applications5 Fréchet Derivative & Condition No.6 Problem Classification7 Methods for f (A)8 Methods for f (A)b9 Software
10 Case11 Quiz12 Extras
University of Manchester Nick Higham Matrix Functions 55 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
Fréchet Derivative
Fréchet derivative of f : Cn×n → Cn×n at X ∈ Cn×n
A linear mapping L : Cn×n → Cn×n s.t. for all E ∈ Cn×n
f (X + E)− f (X )− L(X ,E) = o(‖E‖).
Example For f (X ) = X 2 we have
f (X + E)− f (X ) = XE + EX + E2,
so L(X ,E) = XE + EX .
University of Manchester Nick Higham Matrix Functions 56 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
Fréchet Derivative of eA
L(A,E) =
∫ 1
0eA(1−s)EeAs ds.
Simplifies to L(A,E) = EeA = eAE when AE = EA.Another representation:
L(A,E) = E +AE + EA
2!+
A2E + AEA + EA2
3!+ · · · .
University of Manchester Nick Higham Matrix Functions 57 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
Condition Number
cond(f ,A) := limε→0
sup‖E‖≤ε‖A‖
‖f (A + E)− f (A)‖ε‖f (A)‖ .
‖L(A)‖ := maxE 6=0
‖L(A,E)‖‖E‖ .
Lemma
cond(f ,A) =‖L(A)‖‖A‖‖f (A)‖ .
University of Manchester Nick Higham Matrix Functions 58 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
Condition Number of eA
κexp(A) =‖L(A)‖‖A‖‖eA‖ .
‖L(A)‖ ≥ ‖L(A, I)‖ = ‖eA‖ ⇒ κexp(A) ≥ ‖A‖ .
TheoremFor normal A ∈ Cn×n, κexp(A) = ‖A‖2.If A ∈ Rn×n is a nonnegative scalar multiple of astochastic matrix then in the∞-norm, κexp(A) = ‖A‖∞.
University of Manchester Nick Higham Matrix Functions 59 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
Computing Lf : Via 2n × 2n Matrix
TheoremIf f is 2n − 1 times ctsly diffble,
f([
A E0 A
])=
[f (A) Lf (A,E)
0 f (A)
].
Note that Lf (A, αE) = αLf (A,E), but α may effect algused for the evaluation.
University of Manchester Nick Higham Matrix Functions 60 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
Computing Lf : Complex Step
Assume that f : Rn×n → Rn×n and A,E ∈ Rn×n. Then
f (A + ihE)− f (A)− ihLf (A,E) = o(h).
Thus (Al-Mohy & H, 2010)
f (A) ≈ Re f (A + ihE),
Lf (A,E) ≈ Imf (A + ihE)
h.
h not restricted by fl pt arith considerations. Can takeh = 10−100.f alg must not employ complex arith.
University of Manchester Nick Higham Matrix Functions 61 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
Condition Estimation (1)
Since Lf is a linear operator,
vec(Lf (A,E)) = K (A)vec(E)
where K (A) ∈ Cn2×n2 is the Kronecker form of the Fréchetderivative.
Can show ‖Lf (A)‖F = ‖K (A)‖2 .
Let L?f (A) be the adjoint of Lf (A) wrt inner product〈X ,Y 〉 = trace(Y ∗X ). In general, L?f (A) = Lf (A
∗), wheref (z) := f (z).
University of Manchester Nick Higham Matrix Functions 62 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
Power Method
Algorithm
Power method applied to A∗A to produce γ ≤ ‖A‖2.
1 Choose a nonzero starting vector z0 ∈ Cn
2 for k = 0:∞3 wk+1 = Azk
4 zk+1 = A∗wk+1
5 γk+1 = ‖zk+1‖2/‖wk+1‖2
6 if converged, γ = γk+1, quit, end7 end
Normalization needed.Linear convergence.
University of Manchester Nick Higham Matrix Functions 63 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
Condition Estimation (2)
Algorithm (power method on Fréchet derivative)
2-norm power method to produce γ ≤ ‖Lf (A)‖F .
1 Choose a nonzero starting matrix Z0 ∈ Cn×n
2 for k = 0:∞3 Wk+1 = Lf (A,Zk)4 Zk+1 = L?f (A,Wk+1)5 γk+1 = ‖Zk+1‖F/‖Wk+1‖F
6 if converged, γ = γk+1, quit, end7 end
In practice we use instead the block 1-norm estimatorof H & Tisseur (2000).
University of Manchester Nick Higham Matrix Functions 64 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
Outline
1 History2 Definitions3 Properties, Formulas4 Applications5 Fréchet Derivative & Condition No.6 Problem Classification7 Methods for f (A)8 Methods for f (A)b9 Software
10 Case11 Quiz12 Extras
University of Manchester Nick Higham Matrix Functions 65 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
Small/Medium Scale Problems
Decompositions:Normal A: if can compute Schur/spectraldecomposition A = QDQ∗, D = diag(di), thenf (A) = Qdiag(f (di))Q∗.Nonnormal A: if can compute Schur decompositionA = QTQ∗ then use Schur–Parlett method.
Matrix iterations: Xk+1 = g(Xk), X0 = A, for matrix roots,sign function, polar decomposition. Require only matrixmult and multiple RHS linear systems.
Approximation methods: polynomial and rational(Taylor, Padé, . . . ), specific to f .
University of Manchester Nick Higham Matrix Functions 66 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
Large Scale f (A)b Problems (1)
A large and sparse,f (A) cannot be stored,the problem is f (A)b: action of f (A) on b.
Case 1: Can solve Ax = b (sparse direct methods) butnot compute Schur decomp: “backslash matrix”.
Cauchy integral formula can be used (Hale, H &Trefethen, 2008) .Rational Krylov can be used with direct solves.
University of Manchester Nick Higham Matrix Functions 67 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
Large Scale f (A)b Problems (2)
Case 2: Can only compute matrix–vector products Ax(and maybe A∗x).
Perhaps: A symmetric, can estimate [λmin, λmax].
Krylov methods.Polynomial approximations.Others?
Exponential integrators for sufficiently large problems are inthis case.
University of Manchester Nick Higham Matrix Functions 68 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
Accuracy Requirements
Full double precision.
Variable tolerance, e.g., within an ODE integrator.
Given tolerance, where matrix A is subject tomeasurement error, e.g., ≈ 10−4 in engineering,healthcare.
University of Manchester Nick Higham Matrix Functions 69 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
Outline
1 History2 Definitions3 Properties, Formulas4 Applications5 Fréchet Derivative & Condition No.6 Problem Classification7 Methods for f (A)8 Methods for f (A)b9 Software
10 Case11 Quiz12 Extras
University of Manchester Nick Higham Matrix Functions 70 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
Classic MATLAB< M A T L A B >
Version of 01/10/84
HELP is available
<>helpType HELP followed byINTRO (To get started)NEWS (recent revisions)ABS ANS ATAN BASE CHAR CHOL CHOP CLEA COND CONJ COSDET DIAG DIAR DISP EDIT EIG ELSE END EPS EXEC EXITEXP EYE FILE FLOP FLPS FOR FUN HESS HILB IF IMAGINV KRON LINE LOAD LOG LONG LU MACR MAGI NORM ONESORTH PINV PLOT POLY PRIN PROD QR RAND RANK RCON RATREAL RETU RREF ROOT ROUN SAVE SCHU SHOR SEMI SIN SIZESQRT STOP SUM SVD TRIL TRIU USER WHAT WHIL WHO WHY< > ( ) = . , ; / ’ + - * :
University of Manchester Nick Higham Matrix Functions 71 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
Classic MATLAB<>help fun
FUN For matrix arguments X , the functions SIN, COS, ATAN,SQRT, LOG, EXP and X**p are computed using eigenvalues Dand eigenvectors V . If <V,D> = EIG(X) then f(X) =V*f(D)/V . This method may give inaccurate results if Vis badly conditioned. Some idea of the accuracy can beobtained by comparing X**1 with X .For vector arguments, the function is applied to eachcomponent.
The availability of [FUN] in early versions of MATLABquite possibly contributed to
the system’s technical and commercial success.
— Cleve Moler (2003)
University of Manchester Nick Higham Matrix Functions 72 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
Taylor Series
Matrix Taylor series converges if eigenvalues ofincrement matrix lie within radius of convergence ofseries. Thus for all A,
cos(A) = I − A2
2!+
A4
4!− A6
6!+ · · · .
Can bound error in truncated Taylor series in terms ofappropriate derivative at matrix argument.Usual concerns about numerical cancellation in theevaluation.
University of Manchester Nick Higham Matrix Functions 73 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
George Forsythe “Pitfalls” (1970)
University of Manchester Nick Higham Matrix Functions 74 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
Padé Approximation
Rational rkm(x) = pkm(x)/qkm(x) is a[k ,m] Padé approximant tof (x) =
∑∞i=0 αix i if pkm and qkm are polys
of degree at most k and m and
f (x)− rkm(x) = O(xk+m+1).
Generally more efficient thantruncated Taylor series.Possible representations:
Ratio of polys.Continued fraction.Partial fraction.
Henri Padé1863–1953
University of Manchester Nick Higham Matrix Functions 75 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
Similarity Transformations
Can use the formula
A = XBX−1 ⇒ f (A) = Xf (B)X−1,
provided f (B) is easily computable.E.g. B = diag(λi) if A diagonalizable.
Problem : any error ∆B in f (B) magnified by up toκ(X ) = ‖X‖‖X−1‖ ≥ 1.
Prefer to work with unitary X : thus can useeigendecomposition (diagonal B) when A is normal(AA∗ = A∗A),Schur decomposition (triangular B) in general.
University of Manchester Nick Higham Matrix Functions 76 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
Example: Eigendecompositionfunction F = funm_ev(A,fun)[V,D] = eig(A);F = V * diag(feval(fun,diag(D))) / V;
>> A = [3 -1; 1 1]; X = funm_ev(A,@sqrt)X =
1.7678e+000 -3.5355e-0013.5355e-001 1.0607e+000
>> norm(A-X^2) % cond(V) = 9.4e7ans =
9.9519e-009>> Y = sqrtm(A); norm(A-Y^2)ans =
6.4855e-016
University of Manchester Nick Higham Matrix Functions 77 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
Block Diagonalization
Could instead use a block diagonalization A = XDX−1,where
X well conditioned,D = diag(Di) block diagonal.
Can compute starting with Schur:
Need a parameter : max condition of individualtransformations.Di are triangular but have no particular eigenvaluedistribution, so f (Di) nontrivial.
University of Manchester Nick Higham Matrix Functions 78 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
Parlett’s Recurrence
fii = f (tii) is immediate.
Parlett (1976): from FT = TF obtain recurrence
fij = tijfii − fjjtii − tjj
+
j−1∑k=i+1
fik tkj − tik fkj
tii − tjj.
Used in funm in MATLAB 6.5 (2002) and earlier.
Fails when T has repeated eigenvalues.
University of Manchester Nick Higham Matrix Functions 79 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
Parlett’s Recurrence
fii = f (tii) is immediate.
Parlett (1976): from FT = TF obtain recurrence
fij = tijfii − fjjtii − tjj
+
j−1∑k=i+1
fik tkj − tik fkj
tii − tjj.
Used in funm in MATLAB 6.5 (2002) and earlier.
Fails when T has repeated eigenvalues.
University of Manchester Nick Higham Matrix Functions 79 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
Matrix pth Root
Square root: Björck & Hammarling (1983). Compute Schurdecomp. A = QTQ∗, solve U2 = T by
rii =√
tii , rij =tij −
∑j−1k=i+1 uijukj
uii + ujj,
form X = QUQ∗.
Extended to pth roots by Smith (2003)—much morecomplicated recurrence.
These algs
Have essentially optimal numerical stability.Generalize to real Schur decomp.
University of Manchester Nick Higham Matrix Functions 80 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
Parlett vs. Björck & Hammarling
Parlett recurrence is not “optimal”, as clear from sq. rootcase: x12 obtained from
Parlett :a12(√
a11 −√
a22)
a11 − a22=
a12√a11 +
√a22
: B & H.
University of Manchester Nick Higham Matrix Functions 81 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
Block Parlett Recurrence
T = (Tij) block upper triangular with square diagonal blocks.
F = (Fij) has same block structure and Fii = f (Tii).TF = FT leads to Sylvester equations
TiiFij − FijTjj = FiiTij − TijFjj +
j−1∑k=i+1
(FikTkj − TikFkj), i < j .
Compute F a block superdiagonal at a time.
Singular systems possible.So re-order and re-block to produce “well conditioned”systems.
University of Manchester Nick Higham Matrix Functions 82 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
Schur–Parlett Algorithm
H & Davies (2003):
Compute Schur decomposition A = QTQ∗.Re-order T to block triangular form in whicheigenvalues within a block are “close” and those ofseparate blocks are “well separated”.Evaluate Fii = f (Tii).Solve the Sylvester equations
TiiFij − FijTjj = FiiTij − TijFjj +
j−1∑k=i+1
(FikTkj − TikFkj).
Undo the unitary transformations.
University of Manchester Nick Higham Matrix Functions 83 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
Reordering Step
Break the eigenvalues into sets: λi and λj go in sameset if
|λi − λj | ≤ δ = 0.1.
Choose ordering of sets on the diagonal and determinesequence of swaps to produce that ordering.Carry out the swaps by unitary transformations(LAPACK).
University of Manchester Nick Higham Matrix Functions 84 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
Function of Atomic Block
Assume f has Taylor series with∞ radius of cgce andderivatives available.
For diagonal blocks T use
T = σI + M, σ = trace(T )/n : f (T ) =∞∑
k=0
f (k)(σ)k !
Mk .
Truncate series based on strict error bound, not usingsize of terms. NB: for n = 2,
M =
[ε α0 −ε
]⇒ M2k =
[ε2k 00 ε2k
], M2k+1 =
[ε2k+1 αε2k
0 −ε2k+1
].
University of Manchester Nick Higham Matrix Functions 85 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
Features of Algorithm
Costs O(n3) flops, or up to n4/3 flops if large blocksneeded (close or repeated eigenvalues).Needs derivatives if block of size > 1: price to pay fortreating general f and nonnormal A.Parameter δ controls blocking. Default δ = 0.1 goodmost of time.Know example where alg unstable for all δ.Best general f (A) alg. Benchmark for comparing otherf (A) algs—general and specific.The basis of funm in MATLAB.
University of Manchester Nick Higham Matrix Functions 86 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
Example Use of funm
To compute exp(A) + cos(A) with one call to funm, useF = funm(A,@fun_expcos)
where
function f = fun_expcos(x,k)% Return k’th derivative of EXP+COS at X.g = mod(ceil(k/2),2);if mod(k,2)
f = exp(x) + sin(x)*(-1)^g;else
f = exp(x) + cos(x)*(-1)^g;end
University of Manchester Nick Higham Matrix Functions 87 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
Matrix Sign Function
Let A ∈ Cn×n have no pure imaginary eigenvalues and letA = ZJZ−1 be a Jordan canonical form with
J =
[ p q
p J1 0q 0 J2
], Λ(J1) ∈ LHP, Λ(J2) ∈ RHP.
sign(A) = Z[−Ip 00 Iq
]Z−1.
sign(A) = A(A2)−1/2 .
sign(A) =2π
A∫ ∞
0(t2I + A2)−1 dt .
University of Manchester Nick Higham Matrix Functions 88 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
Matrix Sign Function
Let A ∈ Cn×n have no pure imaginary eigenvalues and letA = ZJZ−1 be a Jordan canonical form with
J =
[ p q
p J1 0q 0 J2
], Λ(J1) ∈ LHP, Λ(J2) ∈ RHP.
sign(A) = Z[−Ip 00 Iq
]Z−1.
sign(A) = A(A2)−1/2 .
sign(A) =2π
A∫ ∞
0(t2I + A2)−1 dt .
University of Manchester Nick Higham Matrix Functions 88 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
Matrix Sign Function
Let A ∈ Cn×n have no pure imaginary eigenvalues and letA = ZJZ−1 be a Jordan canonical form with
J =
[ p q
p J1 0q 0 J2
], Λ(J1) ∈ LHP, Λ(J2) ∈ RHP.
sign(A) = Z[−Ip 00 Iq
]Z−1.
sign(A) = A(A2)−1/2 .
sign(A) =2π
A∫ ∞
0(t2I + A2)−1 dt .
University of Manchester Nick Higham Matrix Functions 88 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
Newton’s Method for Sign
Xk+1 = 12(Xk + X−1
k ), X0 = A.
Convergence
Let S := sign(A), G := (A− S)(A + S)−1. Then
Xk = (I −G2k)−1(I + G2k
)S,
Ei’vals of G are (λi − sign(λi))/(λi + sign(λi)).Hence ρ(G) < 1 and Gk → 0 .Easy to show
‖Xk+1 − S‖ ≤ 12‖X−1
k ‖‖Xk − S‖2.
University of Manchester Nick Higham Matrix Functions 89 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
Newton’s Method for Square Root
Newton’s method: X0 given,
Solve XkEk + EkXk = A− X 2k
Xk+1 = Xk + Ek
k = 0,1,2, . . .
Assume AX0 = X0A. Then can show
Xk+1 = 12(Xk + X−1
k A). (∗)
For nonsingular A, local quadratic cgce of full Newtonto a primary square root.To which square root do the iterations converge?(∗) can converge when full Newton breaks down.Lack of symmetry in (∗).
University of Manchester Nick Higham Matrix Functions 90 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
Convergence (Jordan)
Assume X0 = p(A) for some poly p. Let Z−1AZ = J beJordan canonical form and set Z−1XkZ = Yk . Then
Yk+1 = 12(Yk + Y−1
k J), Y0 = J.
Convergence of diagonal of Yk reduces to scalar case:
Heron: yk+1 = 12
(yk +
λ
yk
), y0 = λ.
Can show that off-diagonal converges.Problem: analysis does not generalize to X0A = AX0!X0 not necessarily a polynomial in A.
University of Manchester Nick Higham Matrix Functions 91 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
Convergence (via Sign)
TheoremLet A ∈ Cn×n have no e’vals on R−. The Newton squareroot iterates Xk with X0A = AX0 are related to the Newtonsign iterates
Sk+1 =12(Sk + S−1
k ), S0 = A−1/2X0
by Xk ≡ A1/2Sk . Hence, provided A−1/2X0 has no pure
imag e’vals, Xk are defined and Xk → A1/2sign(S0)
quadratically.
⇒: Xk → A1/2 if Λ(A−1/2X0) ⊆ RHP, e.g., if X0 = A .
University of Manchester Nick Higham Matrix Functions 92 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
History of Newton Sqrt Instability
Xk+1 = 12(Xk + X−1
k A).
Instability of Newton noted by Laasonen (1958):“Newton’s method if carried out indefinitely, isnot stable whenever the ratio of the largest tothe smallest eigenvalue of A exceeds thevalue 9.”
Described informally by Blackwell (1985) inMathematical People: Profiles and Interviews.Analyzed by H (1986) for diagonalizable A by deriving“error amplification factors”.
University of Manchester Nick Higham Matrix Functions 93 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
Stability
DefinitionThe iteration Xk+1 = g(Xk) is stable in a nbhd of a fixedpoint X if Fréchet derivative dgX has bounded powers.
For scalar, superlinearly cgt g, dgX ≡ g′(x) = 0.
Let X0 = X + E0, Ek := Xk − X . Then
Xk+1 = g(Xk) = g(X + Ek) = g(X ) + dgX (Ek) + o(‖Ek‖).So, since g(X ) = X ,
Ek+1 = dgX (Ek) + o(‖Ek‖).If ‖dg i
X (E)‖ ≤ c, then recurring leads to
‖Ek‖ ≤ c‖E0‖+ kc · o(‖E0‖).
University of Manchester Nick Higham Matrix Functions 94 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
Stability
DefinitionThe iteration Xk+1 = g(Xk) is stable in a nbhd of a fixedpoint X if Fréchet derivative dgX has bounded powers.
For scalar, superlinearly cgt g, dgX ≡ g′(x) = 0.
Let X0 = X + E0, Ek := Xk − X . Then
Xk+1 = g(Xk) = g(X + Ek) = g(X ) + dgX (Ek) + o(‖Ek‖).So, since g(X ) = X ,
Ek+1 = dgX (Ek) + o(‖Ek‖).If ‖dg i
X (E)‖ ≤ c, then recurring leads to
‖Ek‖ ≤ c‖E0‖+ kc · o(‖E0‖).
University of Manchester Nick Higham Matrix Functions 94 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
Stability
DefinitionThe iteration Xk+1 = g(Xk) is stable in a nbhd of a fixedpoint X if Fréchet derivative dgX has bounded powers.
For scalar, superlinearly cgt g, dgX ≡ g′(x) = 0.
Let X0 = X + E0, Ek := Xk − X . Then
Xk+1 = g(Xk) = g(X + Ek) = g(X ) + dgX (Ek) + o(‖Ek‖).So, since g(X ) = X ,
Ek+1 = dgX (Ek) + o(‖Ek‖).If ‖dg i
X (E)‖ ≤ c, then recurring leads to
‖Ek‖ ≤ c‖E0‖+ kc · o(‖E0‖).
University of Manchester Nick Higham Matrix Functions 94 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
Stability of Newton Square Root
g(X ) = 12(X + X−1A).
dgX (E) = 12(E − X−1EX−1A).
Relevant fixed point: X = A1/2.dgA1/2(E) = 1
2(E − A−1/2EA1/2).
Ei’vals of dgA1/2 are
12(1− λ−1/2
i λ1/2j ), i , j = 1 : n.
For stability we need
maxi,j12
∣∣∣1− λ−1/2i λ
1/2j
∣∣∣ < 1.
For hpd A, need κ2(A) < 9.
University of Manchester Nick Higham Matrix Functions 95 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
Advantages
Uses only Fréchet derivative of g.No additional assumptions on A.Perturbation analysis is all in the definition.General, unifying approach.Facilitates analysis of families of iterations.
University of Manchester Nick Higham Matrix Functions 96 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
Stability of Sign Iterations (H)
Theorem
Let Xk+1 = g(Xk) be any superlinearly convergentiteration for S = sign(X0).Then dgS(E) = LS(E) = 1
2(E − SES) , where LS is theFréchet derivative of the matrix sign function at S.Hence dgS is idempotent (dgS dgS = dgS) and theiteration is stable.
“All” sign iterations are automatically stable.
University of Manchester Nick Higham Matrix Functions 97 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
More Iterations for Sign Function. . .
Start with Newton for the sign function:
xk+1 =12(xk + x−1
k ).
Invert:xk+1 =
2xk
x2k + 1
.
Halley:
xk+1 =xk(3 + x2
k )
1 + 3x2k.
Newton–Schulz:
xk+1 =xk
2(3− x2
k ).
University of Manchester Nick Higham Matrix Functions 98 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
More Iterations for Sign Function. . .
Start with Newton for the sign function:
xk+1 =12(xk + x−1
k ).
Invert:xk+1 =
2xk
x2k + 1
.
Halley:
xk+1 =xk(3 + x2
k )
1 + 3x2k.
Newton–Schulz:
xk+1 =xk
2(3− x2
k ).
University of Manchester Nick Higham Matrix Functions 98 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
More Iterations for Sign Function. . .
Start with Newton for the sign function:
xk+1 =12(xk + x−1
k ).
Invert:xk+1 =
2xk
x2k + 1
.
Halley:
xk+1 =xk(3 + x2
k )
1 + 3x2k.
Newton–Schulz:
xk+1 =xk
2(3− x2
k ).
University of Manchester Nick Higham Matrix Functions 98 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
More Iterations for Sign Function. . .
Start with Newton for the sign function:
xk+1 =12(xk + x−1
k ).
Invert:xk+1 =
2xk
x2k + 1
.
Halley:
xk+1 =xk(3 + x2
k )
1 + 3x2k.
Newton–Schulz:
xk+1 =xk
2(3− x2
k ).
University of Manchester Nick Higham Matrix Functions 98 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
Outline
1 History2 Definitions3 Properties, Formulas4 Applications5 Fréchet Derivative & Condition No.6 Problem Classification7 Methods for f (A)8 Methods for f (A)b9 Software
10 Case11 Quiz12 Extras
University of Manchester Nick Higham Matrix Functions 99 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
The f (A)b Problem
Givenmatrix function f : Cn×n → Cn×n,A ∈ Cn×n, b ∈ Cn,
compute f (A)b without first computing f (A).
Most important casesf (x) = x−1
f (x) = ex .f (x) = log(x).f (x) = x1/2.f (x) = sign(x).
University of Manchester Nick Higham Matrix Functions 100 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
A1/2 b via Contour Integration
f (A)b =1
2πi
∫Γ
f (z)(zI − A)−1b dz.
A 5× 5 Pascal matrix: λ(A) ∈ [0.01,92.3], f (z) = z1/2.Use repeated trapezium rule to integrate around circlecentre (λmin + λmax)/2, radius λmax/2.Hale, H & Trefethen (2008): conformally map
Ω = C \(−∞,0] ∪ [m,M ]
.
to an annulus: [m,M ]→ inner circle, (−∞,0]→ outercircle.
# points 2 digits 13 digitsCircle 32,000 262,000Conformal map 5 35Refined 29
University of Manchester Nick Higham Matrix Functions 101 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
Aα b via Binomial Expansion
Write A = s(I − C).If λi > 0, s = (λmin + λmax)/2 yieldsρmin = (λmax − λmin)/(λmax + λmin).For any A, s = trace(A∗A)/trace(A∗) minimizes ‖C‖F .
(I − C)α =∞∑
j=0
(α
j
)(−C)j , ρ(C) < 1.
So
Aαb = sα∞∑
j=0
(α
j
)(−C)jb.
For M-matrices, required splitting with C ≥ 0 always exists.
University of Manchester Nick Higham Matrix Functions 102 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
Aα b via Binomial Expansion
Write A = s(I − C).If λi > 0, s = (λmin + λmax)/2 yieldsρmin = (λmax − λmin)/(λmax + λmin).For any A, s = trace(A∗A)/trace(A∗) minimizes ‖C‖F .
(I − C)α =∞∑
j=0
(α
j
)(−C)j , ρ(C) < 1.
So
Aαb = sα∞∑
j=0
(α
j
)(−C)jb.
For M-matrices, required splitting with C ≥ 0 always exists.
University of Manchester Nick Higham Matrix Functions 102 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
Aα b via ODE IVP
dydt
= α(A− I)[t(A− I) + I
]−1y , y(0) = b
has unique solution
y(t) =[t(A− I) + I
]αb
soy(1) = Aαb.
Used by Allen, Baglama & Boyd (2000) for α = 1/2, spd A.
University of Manchester Nick Higham Matrix Functions 103 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
Formulae for eA, A ∈ Cn×n
Taylor series Limit Scaling and squaring
I + A +A2
2!+
A3
3!+ · · · lim
s→∞(I + A/s)s (eA/2s
)2s
Cauchy integral Jordan form Interpolation
12πi
∫Γ
ez(zI − A)−1 dz Zdiag(eJk )Z−1n∑
i=1
f [λ1, . . . , λi ]
i−1∏j=1
(A − λj I)
Differential system Schur form Padé approximation
Y ′(t) = AY (t), Y (0) = I QeT Q∗ pkm(A)qkm(A)−1
Krylov methods: Arnoldi fact. AQk = QkHk + hk+1,kqk+1eTk
with Hessenberg H: eAb ≈ Qk eHk Q∗k b.
University of Manchester Nick Higham Matrix Functions 104 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
The Sixth Dubious WayMoler & Van Loan (1978, 2003)
University of Manchester Nick Higham Matrix Functions 105 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
Computing eAB
A︸︷︷︸n×n
, B︸︷︷︸n×n0
, n0 n. Exploit, for integer s,
eAB = (es−1A)sB = es−1Aes−1A · · · es−1A︸ ︷︷ ︸s times
B.
Choose s so Tm(s−1A) =∑m
j=0(s−1A)j
j!≈ es−1A. Then
Bi+1 = Tm(s−1A)Bi , i = 0 : s − 1, B0 = B
yields Bs ≈ eAB.
How to choose s and m?
University of Manchester Nick Higham Matrix Functions 106 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
Computing eAB
A︸︷︷︸n×n
, B︸︷︷︸n×n0
, n0 n. Exploit, for integer s,
eAB = (es−1A)sB = es−1Aes−1A · · · es−1A︸ ︷︷ ︸s times
B.
Choose s so Tm(s−1A) =∑m
j=0(s−1A)j
j!≈ es−1A. Then
Bi+1 = Tm(s−1A)Bi , i = 0 : s − 1, B0 = B
yields Bs ≈ eAB.
How to choose s and m?
University of Manchester Nick Higham Matrix Functions 106 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
Truncation Analysis
hm+1(A) := log(e−ATm(A)) =∞∑
k=m+1
ck Ak .
Then Tm(A) = eA+hm+1(A). Hence
Tm(2−sA)2s= eA+2shm+1(2−sA) =: eA+∆A.
Aim: select s so that
‖∆A‖‖A‖ =
‖hm+1(2−sA)‖‖2−sA‖ ≤ u ≈ 1.1× 10−16.
Moler & Van Loan (1978): a priori bound for h2m+1 forPadé approximant; m = 6, ‖2−sA‖ ≤ 1/2 in MATLAB.H (2005): used symbolic arithmetic and high precisionto optimize (s,m).
University of Manchester Nick Higham Matrix Functions 107 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
Refined Backward Error Analysis
Lemma (Al-Mohy & H, 2009)
Tm(s−1A)sB = eA+∆AB, where ∆A = shm+1(s−1A) andhm+1(x) = log(e−x Tm(x)) =
∑∞k=m+1 ck xk . Moreover,
‖∆A‖ ≤ s∞∑
k=m+1
|ck | αp(s−1A)k
if m + 1 ≥ p(p − 1), where
αp(A) = max(dp,dp+1), dp = ‖Ap‖1/p.
University of Manchester Nick Higham Matrix Functions 108 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
Bounding the Backward Error
Want to bound norm of h2m+1(X ) =∑∞
k=2m+1 ckX k .
Non-normality implies ρ(A) ‖A‖.
Note that
ρ(A) ≤ ‖Ap‖1/p ≤ ‖A‖, p = 1 : ∞.
and limp→∞ ‖Ap‖1/p = ρ(A).
Use ‖Ap‖1/p instead of ‖A‖ in the truncation bounds.
University of Manchester Nick Higham Matrix Functions 109 / 162
A =
[0.9 5000 −0.5
].
0 5 10 15 2010
0
102
104
106
108
1010
‖A‖k2‖Ak‖2
(‖A5‖1/52 )k
(‖A10‖1/102 )k
‖Ak‖1/k2
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
Key Ideas
Use the ‖Ap‖1/p in the truncation bounds for a few smallp.Choose optimal m, s for given precision u.Preprocess A by shifting A← A− µI, µ = trace(A)/n.Terminate Taylor series evaluation when converged towithin u
University of Manchester Nick Higham Matrix Functions 111 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
Algorithm for F = etAB
1 µ = trace(A)/n2 A = A− µI3 [m∗, s] = parameters(tA) % Includes norm estimation.4 F = B, η = etµ/s
5 for i = 1: s6 c1 = ‖B‖∞7 for j = 1:m∗8 B = t AB/(sj), c2 = ‖B‖∞9 F = F + B
10 if c1 + c2 ≤ tol‖F‖∞, quit, end11 c1 = c2
12 end13 F = ηF , B = F14 end
University of Manchester Nick Higham Matrix Functions 112 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
Conditioning of eAB
κexp(A,B) ≤ ‖eA‖F‖B‖F
‖eAB‖F(1 + κexp(A)).
Relative forward error due to roundoff bounded by
ue‖A‖2‖B‖2/‖eAB‖F .
A normal implies κexp(A) = ‖A‖2. Then instability ife‖A‖2 ‖eA‖2‖A‖2.A Hermitian implies spectrum of A− n−1trace(A)Ihas λmax = −λmin ⇒ (normwise) stability!
University of Manchester Nick Higham Matrix Functions 113 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
Comparison with the Sixth Dubious Way
Advantages of our method over the one-step ODEintegrator:
Fully exploits the linearity of the ODE.Backward error based; ODE integrator controls local(forward) errors.Overscaling avoided.
x ex − (1 + x) ex − (1 + x/2)2
9.9e-9 2.2e-16 6.7e-168.9e-9 0 6.7e-16
University of Manchester Nick Higham Matrix Functions 114 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
Experiment 1
Trefethen, Weideman & Schmelzer (2006):A ∈ R9801×9801, 2D Laplacian,-2500*gallery(’poisson’,99).
Compute eαAb, tol = ud .
α = 0.02 α = 1speed mv diff speed mv diff
Alg AH 1 1010 1 47702expv 2.8 403 7.7e-15 1.3 8835 4.2e-15phipm 1.1 172 3.1e-15 0.2 2504 4.0e-15rational 3.8 7 3.3e-14 0.1 7 1.2e-12
University of Manchester Nick Higham Matrix Functions 115 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
Experiment 2A = -gallery(’triw’,20,4.1), bi = cos i , tol = ud .
0 20 40 60 80 100
10−10
10−5
100
105
t
‖etAb‖2
Alg AH
phipm
rational
expv
University of Manchester Nick Higham Matrix Functions 116 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
Experiment 3Harwell–Boeing matrices:
orani678, n = 2529, t = 100, b = [1,1, . . . ,1]T ;bcspwr10, n = 5300, t = 10, b = [1,0, . . . ,0,1]T .
2D Laplacian matrix, poisson. tol = 6× 10−8.
Alg AH ode15stime error time error
orani678 0.13 4e-8 136 2e-6bcspwr10 0.021 7e-7 2.92 5e-5poisson 3.76 2e-6 2.48 8e-6
4poisson 15 9e-6 3.24 1e-1
University of Manchester Nick Higham Matrix Functions 117 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
Krylov Methods
Run Krylov process on A to get Arnoldi factorization
AQk = QkHk + hk+1,kqk+1eTk
with Hessenberg H. Approximate
eAb ≈ Qk f (Hk)Q∗kb.
University of Manchester Nick Higham Matrix Functions 118 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
Comparison with Krylov Methods
Alg AH Krylov Methods
Most time spent in matrix–vector products.
Krylov recurrence and eH
can be significant.
“Direct method”, cost pre-dictable.
Iterative method; needsstopping test.
No parameters to estimate. Select Krylov subspace size.
Storage: 2 vectors Storage: Krylov basis
Evaluation of eAt at multiplepoints on interval.
Can handle mult col B. Need block Krylov method.
Cost tends to ⇑ with ‖A‖. ‖A‖1/2-dependence for negdef A.
University of Manchester Nick Higham Matrix Functions 119 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
Outline
1 History2 Definitions3 Properties, Formulas4 Applications5 Fréchet Derivative & Condition No.6 Problem Classification7 Methods for f (A)8 Methods for f (A)b9 Software
10 Case11 Quiz12 Extras
University of Manchester Nick Higham Matrix Functions 120 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
MATLAB: “Built-In”
funm: Schur–Parlett alg.expm: scaling and squaring (H, 2005).expmdemo1 (Padé = old expm), expmdemo2 (Taylor),expmdemo3 (eigensystem).logm: inverse scaling and squaring (H, 2008).sqrtm: Schur method (Björck & Hammarling 1983).
Blog post by Clever Moler about expm:
University of Manchester Nick Higham Matrix Functions 121 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
MATLAB: Third-party
The Matrix Function Toolbox:http://www.maths.manchester.ac.uk/~higham/mftoolbox(H, 2008)Expokit: eA (scaling and squaring) and eAb(Krylov)—(Sidje, 1998).Expint: linear combination of φ functions(Berland, Skaflestad & Wright, 2007).
University of Manchester Nick Higham Matrix Functions 122 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
Matrix Functions in the NAG Library
NAG Toolbox for MATLAB (Mark 23, released 2011):
Matrix exponentialGeneral functions of real symmetric or Hermitianmatrices
NAG C Library (Mark 23, released 2012) also contains:Schur-Parlett algorithm for general matrix functionsMatrix logarithm
NAG Fortran Library Mark 24 the NAG Toolbox forMATLAB (released August 2013) also contain:
Action of the matrix exponential on another matrixCondition number estimation for matrix functions
University of Manchester Nick Higham Matrix Functions 123 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
Matrix Functions in the NAG Library (cont.)
Coming in 2014 . . .
Matrix square root.Matrix power As, s ∈ R.Latest algorithms for matrix exponential and logarithm.Latest Fréchet derivative and condition numberalgorithms for matrix exponential, logarithm and power.
Documentation can be found at:http://www.nag.co.uk/support_documentation.asp
University of Manchester Nick Higham Matrix Functions 124 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
NAG Sponsorship of GGSSS
“NAG will provide personal licence (for personal machines)for the NAG Toolbox for MATLAB (and/or NAG Library -flavour of your choice and for any Fortran programmers theNAG Fortran Compiler). We understand the main product ofinterest will be the NAG Toolbox for MATLAB.”
See email from John Holden, NAG.
University of Manchester Nick Higham Matrix Functions 125 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
Octave
expm: Ward’s alg.logm: eigensystem.sqrtm: Schur method (Björck & Hammarling 1983).thfm: trig, hyperbolic and their inverses. Implementstextbook definitions, in terms of expm, logm, sqrtm etc!
University of Manchester Nick Higham Matrix Functions 126 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
Python: SciPy
linalg.expm: scaling and squaring (H, 2005).linalg.logm
linalg.signm
linalg.cosm
linalg.sinm
linalg.tanm
linalg.coshm
linalg.sinhm
linalg.tanhm
linalg.funm: Schur–Parlett (unblocked!).
University of Manchester Nick Higham Matrix Functions 127 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
Python: SciPy in Progress
linalg/_expm_frechet.py
linalg/_onenormest.py
linalg/_expm_multiply.py
linalg/_sqrtm.py (blocked)
University of Manchester Nick Higham Matrix Functions 128 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
R
Goulet, Dutang, Maechler, Firth, Shapira & Stadelmann,R package expm: Matrix exponential, Computation ofthe matrix exponential and related [email protected]://cran.r-project.org/web/packages/expm/index.html
expm (default is expm.Higham08)expmCond
expmFrechet
logm
sqrtm
University of Manchester Nick Higham Matrix Functions 129 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
C++
J. Niesen, Matrix functions module for Eigen C++template library for linear algebra.http://eigen.tuxfamily.org/dox-devel//unsupported/group__MatrixFunctions__Module.html
cos coshexp logpow matrixFunctionsin sinhsqrt
University of Manchester Nick Higham Matrix Functions 130 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
Outline
1 History2 Definitions3 Properties, Formulas4 Applications5 Fréchet Derivative & Condition No.6 Problem Classification7 Methods for f (A)8 Methods for f (A)b9 Software
10 Case11 Quiz12 Extras
University of Manchester Nick Higham Matrix Functions 131 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
Case Study:Schur Method for Matrix Square Root
For more details of the following, including analysis of bestapproach for parallel computation, see Deadman, Higham& Ralha (2013).
University of Manchester Nick Higham Matrix Functions 132 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
Schur Method: Derivation
Compute Schur decomp A = QTQ∗.
Expand U2 = T , where U upper triangular for primarysquare root:
u2ii = tii ,
uiiuij + uijujj = tij −j−1∑
k=i+1
uikukj .
U is found either a column or a superdiagonal at a time.√
A = QUQ∗
University of Manchester Nick Higham Matrix Functions 133 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
The Schur Method: Properties
Cost: 2813n3 flops.
Real arithmetic version uses real Schur decomp (upperquasi-triang T )
Computed square root U satisfies U2 = T +∆T , where
|∆T | ≤ γn|U|2,
or normwise in real arithmetic.
No use of level 3 BLAS: slow!
Now focus on the triangular phase of the algorithm.
University of Manchester Nick Higham Matrix Functions 134 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
The Schur Method: Properties
Cost: 2813n3 flops.
Real arithmetic version uses real Schur decomp (upperquasi-triang T )
Computed square root U satisfies U2 = T +∆T , where
|∆T | ≤ γn|U|2,
or normwise in real arithmetic.
No use of level 3 BLAS: slow!
Now focus on the triangular phase of the algorithm.
University of Manchester Nick Higham Matrix Functions 134 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
Random Complex Triangular
0 1000 2000 3000 4000 5000 6000 7000 80000
50
100
150
200
250
300
350
400
450
n
time
(s)
point
University of Manchester Nick Higham Matrix Functions 135 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
The Blocked Schur Method
The Uij and Tij are now taken to be blocks:
U2ii = Tii , (1)
UiiUij + UijUjj = Tij −j−1∑
k=i+1
UikUkj . (2)
Solve (1) by point method.Solve Sylvester eqn (2) by, e.g., xTRSYL in LAPACK.
Error bounds unchanged.Over 90% of run time spent in GEMM calls and 8% inSylvester equation solution.Insensitive to block size and choice of kernel routines.
University of Manchester Nick Higham Matrix Functions 136 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
The Blocked Schur Method
The Uij and Tij are now taken to be blocks:
U2ii = Tii , (1)
UiiUij + UijUjj = Tij −j−1∑
k=i+1
UikUkj . (2)
Solve (1) by point method.Solve Sylvester eqn (2) by, e.g., xTRSYL in LAPACK.Error bounds unchanged.Over 90% of run time spent in GEMM calls and 8% inSylvester equation solution.Insensitive to block size and choice of kernel routines.
University of Manchester Nick Higham Matrix Functions 136 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
Random Complex Triangular
0 1000 2000 3000 4000 5000 6000 7000 80000
50
100
150
200
250
300
350
400
450
n
time
(s)
pointblock
University of Manchester Nick Higham Matrix Functions 137 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
The Recursive Blocked Schur Method
Recursive solution of the triangular phase:[U11 U12
0 U22
]2
=
[T11 T12
0 T22
].
U211 = T11 and U2
22 = T22 solved recursively.
Solve U11U12 + U12U22 = T12 using the recursivemethod of Jonsson & Kågström (2002)
Point algs are used when the recursion once thresholdreached (e.g. n = 64).
Same error bound as point alg.
University of Manchester Nick Higham Matrix Functions 138 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
Random Complex Triangular Matrices
0 1000 2000 3000 4000 5000 6000 7000 80000
50
100
150
200
250
300
350
400
450
n
time
(s)
pointblockrecursion
University of Manchester Nick Higham Matrix Functions 139 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
Full Complex Matrices
0 200 400 600 800 1000 1200 1400 1600 1800 20000
50
100
150
200
250
n
time
(s)
sqrtmfort_pointfort_recurse
University of Manchester Nick Higham Matrix Functions 140 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
Full Real Matrices
0 200 400 600 800 1000 1200 1400 1600 1800 20000
50
100
150
200
250
n
time
(s)
sqrtm_realsqrtmfort_point_realfort_recurse_real
University of Manchester Nick Higham Matrix Functions 141 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
Outline
1 History2 Definitions3 Properties, Formulas4 Applications5 Fréchet Derivative & Condition No.6 Problem Classification7 Methods for f (A)8 Methods for f (A)b9 Software
10 Case11 Quiz12 Extras
University of Manchester Nick Higham Matrix Functions 144 / 162
Quiz
Liu Hui
Carl Friedrich Gauss Myrick Hascall Doolittle
André-Louis Cholesky Prescott Durand Crout John von Neumann
Liu Hui Carl Friedrich Gauss
Myrick Hascall Doolittle
André-Louis Cholesky Prescott Durand Crout John von Neumann
Liu Hui Carl Friedrich Gauss Myrick Hascall Doolittle
André-Louis Cholesky Prescott Durand Crout John von Neumann
Liu Hui Carl Friedrich Gauss Myrick Hascall Doolittle
André-Louis Cholesky
Prescott Durand Crout John von Neumann
Liu Hui Carl Friedrich Gauss Myrick Hascall Doolittle
André-Louis Cholesky Prescott Durand Crout
John von Neumann
Liu Hui Carl Friedrich Gauss Myrick Hascall Doolittle
André-Louis Cholesky Prescott Durand Crout John von Neumann
Paul Sumner Dwyer
John Todd Alan Turing
Cornelius Lanczos George Forsythe
Paul Sumner Dwyer John Todd
Alan Turing
Cornelius Lanczos George Forsythe
Paul Sumner Dwyer John Todd Alan Turing
Cornelius Lanczos George Forsythe
Paul Sumner Dwyer John Todd Alan Turing
Cornelius Lanczos
George Forsythe
Paul Sumner Dwyer John Todd Alan Turing
Cornelius Lanczos George Forsythe
Peter Lancaster
Olga Taussky-Todd English Electric Lightning
Peter Lancaster
Olga Taussky-Todd
English Electric Lightning
Peter Lancaster
Olga Taussky-Todd English Electric Lightning
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
Outline
1 History2 Definitions3 Properties, Formulas4 Applications5 Fréchet Derivative & Condition No.6 Problem Classification7 Methods for f (A)8 Methods for f (A)b9 Software
10 Case11 Quiz12 Extras
University of Manchester Nick Higham Matrix Functions 152 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
How to Do ResearchDevelop new metric for solved problem then develop anew alg (or tweak an existing one) so it does best onthat metric. E.g. measure communcation costs as wellas arithmetic.
Find hidden assumptions in existing method andremove them. E.g.: Strassen realized use of innerproducts was not necessary for matrix multiplication.Find simpler proof of existing result.Prove (weaker version of) existing result under weakerassumptions.Develop alg that is faster and more accurate thanexisting ones.Develop alg that can solve problems other algs can’t.E.g.: y ∼ N(µ,C) problem in these slides.
University of Manchester Nick Higham Matrix Functions 153 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
How to Do ResearchDevelop new metric for solved problem then develop anew alg (or tweak an existing one) so it does best onthat metric. E.g. measure communcation costs as wellas arithmetic.Find hidden assumptions in existing method andremove them. E.g.: Strassen realized use of innerproducts was not necessary for matrix multiplication.
Find simpler proof of existing result.Prove (weaker version of) existing result under weakerassumptions.Develop alg that is faster and more accurate thanexisting ones.Develop alg that can solve problems other algs can’t.E.g.: y ∼ N(µ,C) problem in these slides.
University of Manchester Nick Higham Matrix Functions 153 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
How to Do ResearchDevelop new metric for solved problem then develop anew alg (or tweak an existing one) so it does best onthat metric. E.g. measure communcation costs as wellas arithmetic.Find hidden assumptions in existing method andremove them. E.g.: Strassen realized use of innerproducts was not necessary for matrix multiplication.Find simpler proof of existing result.
Prove (weaker version of) existing result under weakerassumptions.Develop alg that is faster and more accurate thanexisting ones.Develop alg that can solve problems other algs can’t.E.g.: y ∼ N(µ,C) problem in these slides.
University of Manchester Nick Higham Matrix Functions 153 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
How to Do ResearchDevelop new metric for solved problem then develop anew alg (or tweak an existing one) so it does best onthat metric. E.g. measure communcation costs as wellas arithmetic.Find hidden assumptions in existing method andremove them. E.g.: Strassen realized use of innerproducts was not necessary for matrix multiplication.Find simpler proof of existing result.Prove (weaker version of) existing result under weakerassumptions.
Develop alg that is faster and more accurate thanexisting ones.Develop alg that can solve problems other algs can’t.E.g.: y ∼ N(µ,C) problem in these slides.
University of Manchester Nick Higham Matrix Functions 153 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
How to Do ResearchDevelop new metric for solved problem then develop anew alg (or tweak an existing one) so it does best onthat metric. E.g. measure communcation costs as wellas arithmetic.Find hidden assumptions in existing method andremove them. E.g.: Strassen realized use of innerproducts was not necessary for matrix multiplication.Find simpler proof of existing result.Prove (weaker version of) existing result under weakerassumptions.Develop alg that is faster and more accurate thanexisting ones.
Develop alg that can solve problems other algs can’t.E.g.: y ∼ N(µ,C) problem in these slides.
University of Manchester Nick Higham Matrix Functions 153 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
How to Do ResearchDevelop new metric for solved problem then develop anew alg (or tweak an existing one) so it does best onthat metric. E.g. measure communcation costs as wellas arithmetic.Find hidden assumptions in existing method andremove them. E.g.: Strassen realized use of innerproducts was not necessary for matrix multiplication.Find simpler proof of existing result.Prove (weaker version of) existing result under weakerassumptions.Develop alg that is faster and more accurate thanexisting ones.Develop alg that can solve problems other algs can’t.E.g.: y ∼ N(µ,C) problem in these slides.
University of Manchester Nick Higham Matrix Functions 153 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
Reproducible Research
Wikipediahttp://en.wikipedia.org/wiki/Reproducibility
“Reproducibility . . . refers to the ability of an entireexperiment or study to be reproduced, or by someoneelse working independently.”“The term reproducible research was first proposedby Jon Claerbout at Stanford University and refers tothe idea that the ultimate product of research is thepaper along with the full computational environmentused to produce the results in the paper such as thecode, data, etc. necessary for reproduction of theresults and building upon the research.”
University of Manchester Nick Higham Matrix Functions 154 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
Reproducible Research: Writing
Irreproducible research [NJH 1980s]:paper.tex, paper.tex, paper.tex, . . .
Reproducible but error-prone research [NJH 1990s]:paper1.tex, paper2.tex, paper3.tex, . . . ,
Reproducible research [NJH since 2010]: paper.texin a version controlled repository.
Use of repository also benefits collaboration with co-authors(or just use Dropbox with personal repos).
See the ICERM 2012 workshophttp://icerm.brown.edu/tw12-5-rcemparticularly the introduction by Randy LeVeque.
University of Manchester Nick Higham Matrix Functions 155 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
Reproducible Research: Writing
Irreproducible research [NJH 1980s]:paper.tex, paper.tex, paper.tex, . . .
Reproducible but error-prone research [NJH 1990s]:paper1.tex, paper2.tex, paper3.tex, . . . ,
Reproducible research [NJH since 2010]: paper.texin a version controlled repository.
Use of repository also benefits collaboration with co-authors(or just use Dropbox with personal repos).
See the ICERM 2012 workshophttp://icerm.brown.edu/tw12-5-rcemparticularly the introduction by Randy LeVeque.
University of Manchester Nick Higham Matrix Functions 155 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
Reproducible Research: Writing
Irreproducible research [NJH 1980s]:paper.tex, paper.tex, paper.tex, . . .
Reproducible but error-prone research [NJH 1990s]:paper1.tex, paper2.tex, paper3.tex, . . . ,
Reproducible research [NJH since 2010]: paper.texin a version controlled repository.
Use of repository also benefits collaboration with co-authors(or just use Dropbox with personal repos).
See the ICERM 2012 workshophttp://icerm.brown.edu/tw12-5-rcemparticularly the introduction by Randy LeVeque.
University of Manchester Nick Higham Matrix Functions 155 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
Reproducible Research: Writing
Irreproducible research [NJH 1980s]:paper.tex, paper.tex, paper.tex, . . .
Reproducible but error-prone research [NJH 1990s]:paper1.tex, paper2.tex, paper3.tex, . . . ,
Reproducible research [NJH since 2010]: paper.texin a version controlled repository.
Use of repository also benefits collaboration with co-authors(or just use Dropbox with personal repos).
See the ICERM 2012 workshophttp://icerm.brown.edu/tw12-5-rcemparticularly the introduction by Randy LeVeque.
University of Manchester Nick Higham Matrix Functions 155 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
Git
Git is a widely used distributed version control system.Command line, wysywig interfaces, Emacs interface,. . .gitinfo.sty enables version information to automaticallybe placed into a LATEX document.Github is a popular website to host and browse gitrepositories.
University of Manchester Nick Higham Matrix Functions 156 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
Other Toolslatexdiff is a Perl script:
latexdiff paper_old.tex paper_new.tex ...> paper_diff.tex
“Need to Know” page for theManchester Numerical LinearAlgebra Research group, withinformation about “LaTeX -BibTeX - Beamer - VersionControl - Emacs - MATLAB -Computing - Refereeing - NewsSources - Societies - Dropbox-Personal Web Page - Travelsupport - CV”.
University of Manchester Nick Higham Matrix Functions 157 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
Literate Programming and RR
JSS Journal of Statistical SoftwareJanuary 2012, Volume 46, Issue 3. http://www.jstatsoft.org/
A Multi-Language Computing Environment for
Literate Programming and Reproducible Research
Eric SchulteUniversity of New Mexico
Dan DavisonCounsyl
Thomas DyeUniversity of Hawai‘i
Carsten DominikUniversity of Amsterdam
Abstract
We present a new computing environment for authoring mixed natural and com-puter language documents. In this environment a single hierarchically-organized plaintext source file may contain a variety of elements such as code in arbitrary program-ming languages, raw data, links to external resources, project management data, workingnotes, and text for publication. Code fragments may be executed in situ with graphical,numerical and textual output captured or linked in the file. Export to LATEX, HTML,LATEX beamer, DocBook and other formats permits working reports, presentations andmanuscripts for publication to be generated from the file. In addition, functioning purecode files can be automatically extracted from the file. This environment is implementedas an extension to the Emacs text editor and provides a rich set of features for authoringboth prose and code, as well as sophisticated project management capabilities.
Keywords: literate programming, reproducible research, compendium, WEB, Emacs.
1. Introduction
There are a variety of settings in which it is desirable to mix prose, code, data, and compu-tational results in a single document.
Scientific research increasingly involves the use of computational tools. Successful com-munication and verification of research results requires that this code is distributedtogether with results and explanatory prose.
In software development the exchange of ideas is accomplished through both code and
University of Manchester Nick Higham Matrix Functions 158 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
Emacs + Org-mode + Python in RR
Presentation by John Kitchin at SciPy 2013.
University of Manchester Nick Higham Matrix Functions 159 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
Professional Use of Social Media
SIAM Annual Meeting Minisymposium, San Diego, 2013
Tammy Kolda: how to make your publications easilyavailable online and various ways to maintain such alist.
David Gleich: survey of social media tools.
NJH: how and why to blog and tweet.
Karthika Muthukumaraswamy: importance ofblogging for scientific communication.
University of Manchester Nick Higham Matrix Functions 160 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
Journal Publication Delays
Median time to first review, 2009–present:SISC 3.7 monthsSIMAX 3.4 months
Average months from submission to acceptance:SISC 9.2 monthsSIMAX 11.1 months
Don’t be afraid to contact SIAM to ask for update onpaper’s status. Use “Send ManuscriptCorrespondence” option once logged in to the journal’sweb site.
University of Manchester Nick Higham Matrix Functions 161 / 162
History Defs Properties Applics Fréchet Deriv Prob Class f (A) meth f (A)b meth Software Case Quiz Extras
Householder Symposium
The Householder Symposium XIX on Numerical LinearAlgebra will be held in Spa, Belgium, 8-13 June 2014.
Web page: http://sites.uclouvain.be/HHXIX/Application deadline: 31 October 2013Notification of acceptance: 31 January 2014
Attendance at the meeting is by invitation only.Applications are welcome from researchers innumerical linear algebra, matrix theory, andrelated areas such as optimization, differentialequations, signal processing, and control....The Symposium is very informal, with the interminglingof young and established researchers a priority.
University of Manchester Nick Higham Matrix Functions 162 / 162
References I
A. H. Al-Mohy and N. J. Higham.A new scaling and squaring algorithm for the matrixexponential.SIAM J. Matrix Anal. Appl., 31(3):970–989, 2009.
A. H. Al-Mohy and N. J. Higham.The complex step approximation to the Fréchetderivative of a matrix function.Numer. Algorithms, 53(1):133–148, 2010.
A. H. Al-Mohy and N. J. Higham.Computing the action of the matrix exponential, with anapplication to exponential integrators.SIAM J. Sci. Comput., 33(2):488–511, 2011.
University of Manchester Nick Higham Matrix Functions 1 / 20
References II
E. J. Allen, J. Baglama, and S. K. Boyd.Numerical approximation of the product of the squareroot of a matrix with a vector.Linear Algebra Appl., 310:167–181, 2000.
F. L. Bauer.Decrypted Secrets: Methods and Maxims ofCryptology.Springer-Verlag, Berlin, third edition, 2002.ISBN 3-540-42674-4.xii+474 pp.
University of Manchester Nick Higham Matrix Functions 2 / 20
References III
H. Berland, B. Skaflestad, and W. Wright.EXPINT—A MATLAB package for exponentialintegrators.ACM Trans. Math. Software, 33(1):Article 4, 2007.
Å. Björck and S. Hammarling.A Schur method for the square root of a matrix.Linear Algebra Appl., 52/53:127–140, 1983.
G. Boyd, C. A. Micchelli, G. Strang, and D.-X. Zhou.Binomial matrices.Adv. in Comput. Math., 14:379–391, 2001.
University of Manchester Nick Higham Matrix Functions 3 / 20
References IV
A. Cayley.A memoir on the theory of matrices.Philos. Trans. Roy. Soc. London, 148:17–37, 1858.
T. Charitos, P. R. de Waal, and L. C. van der Gaag.Computing short-interval transition matrices of adiscrete-time Markov chain from partially observeddata.Statistics in Medicine, 27:905–921, 2008.
J. Chen, M. Anitescu, and Y. Saad.Computing f (A)b via least squares polynomialapproximations.SIAM J. Sci. Comput., 33(1):195–222, 2011.
University of Manchester Nick Higham Matrix Functions 4 / 20
References V
T. Crilly.Cayley’s anticipation of a generalised Cayley–Hamiltontheorem.Historia Mathematica, 5:211–219, 1978.
T. Crilly.Arthur Cayley: Mathematician Laureate of the VictorianAge.Johns Hopkins University Press, Baltimore, MD, USA,2006.ISBN 0-8018-8011-4.xxi+610 pp.
University of Manchester Nick Higham Matrix Functions 5 / 20
References VI
P. I. Davies and N. J. Higham.A Schur–Parlett algorithm for computing matrixfunctions.SIAM J. Matrix Anal. Appl., 25(2):464–485, 2003.
C. Davis.Explicit functional calculus.Linear Algebra Appl., 6:193–199, 1973.
University of Manchester Nick Higham Matrix Functions 6 / 20
References VII
E. Deadman, N. J. Higham, and R. Ralha.Blocked Schur algorithms for computing the matrixsquare root.In P. Manninen and P. Öster, editors, Applied Paralleland Scientific Computing: 11th InternationalConference, PARA 2012, Helsinki, Finland, volume7782 of Lecture Notes in Computer Science, pages171–182. Springer-Verlag, Berlin, 2013.
J. Descloux.Bounds for the spectral norm of functions of matrices.Numer. Math., 15:185–190, 1963.
University of Manchester Nick Higham Matrix Functions 7 / 20
References VIII
E. Estrada and D. J. Higham.Network properties revealed through matrix functions.SIAM Rev., 52(4):696–714, 2010.
G. E. Forsythe.Pitfalls in computation, or why a math book isn’tenough.Amer. Math. Monthly, 77(9):931–956, 1970.
University of Manchester Nick Higham Matrix Functions 8 / 20
References IX
R. A. Frazer, W. J. Duncan, and A. R. Collar.Elementary Matrices and Some Applications toDynamics and Differential Equations.Cambridge University Press, Cambridge, UK, 1938.xviii+416 pp.1963 printing.
J. F. Grcar.How ordinary elimination became Gaussian elimination.Historia Mathematica, 38(2):163–218, 2011.
J. F. Grcar.Mathematics of Gaussian elimination.Notices Amer. Math. Soc., 58(8):782–792, 2011.
University of Manchester Nick Higham Matrix Functions 9 / 20
References X
D. A. Grier.When Computers Were Human.Princeton University Press, Princeton, NJ, USA, 2005.ISBN 0-691-09157-9.viii+411 pp.
N. Hale, N. J. Higham, and L. N. Trefethen.Computing Aα, log(A), and related matrix functions bycontour integrals.SIAM J. Numer. Anal., 46(5):2505–2523, 2008.
W. F. Harris.The average eye.Opthal. Physiol. Opt., 24:580–585, 2005.
University of Manchester Nick Higham Matrix Functions 10 / 20
References XI
N. J. Higham.Newton’s method for the matrix square root.Math. Comp., 46(174):537–549, Apr. 1986.
N. J. Higham.The scaling and squaring method for the matrixexponential revisited.SIAM J. Matrix Anal. Appl., 26(4):1179–1193, 2005.
University of Manchester Nick Higham Matrix Functions 11 / 20
References XII
N. J. Higham.Functions of Matrices: Theory and Computation.Society for Industrial and Applied Mathematics,Philadelphia, PA, USA, 2008.ISBN 978-0-898716-46-7.xx+425 pp.
N. J. Higham and A. H. Al-Mohy.Computing matrix functions.Acta Numerica, 19:159–208, 2010.
N. J. Higham and L. Lin.On pth roots of stochastic matrices.Linear Algebra Appl., 435(3):448–463, 2011.
University of Manchester Nick Higham Matrix Functions 12 / 20
References XIII
N. J. Higham and F. Tisseur.A block algorithm for matrix 1-norm estimation, with anapplication to 1-norm pseudospectra.SIAM J. Matrix Anal. Appl., 21(4):1185–1201, 2000.
I. Jonsson and B. Kågström.Recursive blocked algorithms for solving triangularsystems—Part I: One-sided and coupled Sylvester-typematrix equations.ACM Trans. Math. Software, 28(4):392–415, 2002.
University of Manchester Nick Higham Matrix Functions 13 / 20
References XIV
P. Laasonen.On the iterative solution of the matrix equationAX 2 − I = 0.Math. Tables Aids Comput., 12(62):109–116, 1958.
J. D. Lawson.Generalized Runge-Kutta processes for stable systemswith large Lipschitz constants.SIAM J. Numer. Anal., 4(3):372–380, Sept. 1967.
University of Manchester Nick Higham Matrix Functions 14 / 20
References XV
L. Lin.Roots of Stochastic Matrices and Fractional MatrixPowers.PhD thesis, The University of Manchester, Manchester,UK, 2010.117 pp.MIMS EPrint 2011.9, Manchester Institute forMathematical Sciences.
C. B. Moler and C. F. Van Loan.Nineteen dubious ways to compute the exponential of amatrix, twenty-five years later.SIAM Rev., 45(1):3–49, 2003.
University of Manchester Nick Higham Matrix Functions 15 / 20
References XVI
K. H. Parshall.James Joseph Sylvester. Jewish Mathematician in aVictorian World.Johns Hopkins University Press, Baltimore, MD, USA,2006.ISBN 0-8018-8291-5.xiii+461 pp.
M. Pusa and J. Leppänen.Computing the matrix exponential in burnupcalculations.Nuclear Science and Engineering, 164:140–150, 2010.
University of Manchester Nick Higham Matrix Functions 16 / 20
References XVII
Y. Saad.Analysis of some Krylov subspace approximations tothe matrix exponential operator.SIAM J. Numer. Anal., 29(1):209–228, Feb. 1992.
E. Schulte, D. Davison, T. Dye, and C. Dominik.A multi-language computing environment for literateprogramming and reproducible research.Journal of Statistical Software, 46(3):1–24, 2012.
University of Manchester Nick Higham Matrix Functions 17 / 20
References XVIII
D. She, Y. Liu, K. Wang, G. Yu, B. Forget, P. K. Romano,and K. Smith.Development of burnup methods and capabilities inMonte Carlo code RMC.Annals of Nuclear Energy, 51:289–294, 2013.
R. B. Sidje.Expokit: A software package for computing matrixexponentials.ACM Trans. Math. Software, 24(1):130–156, 1998.
University of Manchester Nick Higham Matrix Functions 18 / 20
References XIX
B. Singer and S. Spilerman.The representation of social processes by Markovmodels.Amer. J. Sociology, 82(1):1–54, 1976.
M. I. Smith.A Schur algorithm for computing matrix pth roots.SIAM J. Matrix Anal. Appl., 24(4):971–989, 2003.
L. N. Trefethen, J. A. C. Weideman, and T. Schmelzer.Talbot quadratures and rational approximations.BIT, 46(3):653–670, 2006.
University of Manchester Nick Higham Matrix Functions 19 / 20
References XX
H. W. Turnbull.The matrix square and cube roots of unity.J. London Math. Soc., 2(8):242–244, 1927.
C. F. Van Loan.A study of the matrix exponential.Numerical Analysis Report No. 10, University ofManchester, Manchester, UK, Aug. 1975.Reissued as MIMS EPrint 2006.397, ManchesterInstitute for Mathematical Sciences, The University ofManchester, UK, November 2006.
University of Manchester Nick Higham Matrix Functions 20 / 20