Early Numerical Linear Algebra in the UKhigham/talks/talk05_early-hist.pdf · 2005. 7. 18. · Early Numerical Linear Algebra in the UK Nick Higham School of Mathematics The University

Early Numerical Linear Algebrain the UK

Nick HighamSchool of Mathematics

The University of Manchester

Sven HammarlingNAG Ltd.Oxford

SIAM Annual Meeting, New Orleans, July 2005

Num Lin Alg in UK – p. 1/27

Matrices

Term “matrix” coined in 1850 byJames Joseph Sylvester (1814–1897).

Matrix algebra developed by ArthurCayley (1821–1895).Memoir on the theory of matrices(1858).

Num Lin Alg in UK – p. 2/27

William Thomson (Lord Kelvin, 1824–1907)

On a Machine for the Solution ofSimultaneous Equations,Proc Roy Soc, 1878.

Proposed a system involving tilting plates, cords,pulleys, for 8–10 unknowns.

Suggested iterative refinement: “There is, of course, nolimit to the accuracy thus obtainable by successiveapproximations.”

Actual system for 9 unknowns built by Wilbur (1936) atMIT. Tapes 60ft long. For 3 sig figs, about 3 times fasterthan human with desk calculator.

Num Lin Alg in UK – p. 3/27

Lewis Fry Richardson (1881–1953)

Num soln of PDEs (1910): finite-difference methods,Richardson extrapolation (“deferred approach to thelimit”). Richardson’s method: xk+1 = xk + αk(Axk − b).

Met. office, 1913–1916; Paisley College, 1929–1940.

First to apply mathematics, in particular the method offinite differences, to weather prediction:Weather Prediction by Numerical Process, 1922.

Num Lin Alg in UK – p. 4/27

Richardson on Weather Forecasting

“The detailed example of Ch. IX was worked out inFrance in the intervals of transporting wounded in1916–1918. During the battle of Champagne in April1917 the working copy was sent to the rear, where itbecame lost, to be re-discovered some months laterunder a heap of coal.”

Num Lin Alg in UK – p. 5/27

Richardson on Weather Forecasting

“The detailed example of Ch. IX was worked out inFrance in the intervals of transporting wounded in1916–1918. During the battle of Champagne in April1917 the working copy was sent to the rear, where itbecame lost, to be re-discovered some months laterunder a heap of coal.”

“Perhaps some day in the dim future it will be possibleto advance the computations faster than the weatheradvances . . . But that is a dream.”

“Imagine a large hall like a theatre. . . the walls of thischamber are painted to form a map of the globe.. . . Amyriad computers are at work upon the weather of thepart of the map where each sits, but each computerattends only to one equation or part of an equation.”

Num Lin Alg in UK – p. 5/27

Forecast Factory

Artist’s impression: Francois Schuiten.

Num Lin Alg in UK – p. 6/27

Richard Vynne Southwell (1888–1970)

Relaxation method for Ax = b.Examine patterns & relative magni-tudes of residuals, identify best wayto reduce them.“Like a game of chess” (Fox).

Cgce acceleration: terms overrelax-ation, underrelaxation coined.

Multigrid ideas used.

Relaxation Methods in Engineering Science, 1940.Relaxation Methods in Theoretical Physics, 1946.

“Any attempt to mechanize relaxation methods wouldbe a waste of time” (attrib. by Young, 1990).

Num Lin Alg in UK – p. 7/27

Num Lin Alg in UK – p. 8/27

2D Flow Round Aerofoil

Num Lin Alg in UK – p. 9/27

Mallock’s Machine

R. R. M. Mallock, An Electrical Calculating Machine,1933.

Experimental analogue m/c (variable coil transformers)for solving 6 lin eqns built & tested in 1931.

M/c for 10 equations built by Cambridge Instrument Co.Accurate to ≈ 1% of largest component. Cost ≈ £2000.

Aware of conditioning issue: “if the equations areill-conditioned, these errors may be serious”.

Used equilibration and iterative refinement.

“The machine could not adequately deal with illconditioned equations, letting out a very sharp whistlewhen equilibrium could not be reached” (Croarken).

Num Lin Alg in UK – p. 10/27

Leslie Fox (1918–1992)

PhD (1942) with Southwell. “Outstandingexponent of relaxation method”.

Mathematics Division, NPL, 1945–1956.

Set up Oxford Univ Computing Lab., 1957.

1950s papers on A−1, Ax = b.

An Introduction to Numerical LinearAlgebra, 1964.

Early textbook treatment of computa-tional aspects.

First textbook to describe Wilkinson’sbackward error analysis.

Num Lin Alg in UK – p. 11/27

Gaussian Elimination at NPL

1946 Fox, Goodwin, Turing & Wilkinson solve 18 × 18system on desk calculator in 2 weeks. Obtained smallresidual.

1948 Fox, Huskey & Wilkinson give empirical evidence insupport of GE, even for ill conditioned matrices.

1948 Wilkinson’s confidential NPL report on the AutomaticComputing Engine (ACE) gives program implementingGE with partial pivoting and iterative refinement.

1963 Wilkinson’s backward error analysis: RoundingErrors in Algebraic Processes.

Num Lin Alg in UK – p. 12/27

Num Lin Alg in UK – p. 13/27

Turing’s Paper

Rounding-Off Errors in MatrixProcesses, Quart. J. Mech. andApplied Math., 1948.

Proves ∃ce of A = LU ; shows GE computes it.

Introduces term “condition number”.

Uses term “preconditioning”.

Describes iterative refinement for linear systems.

Exploits backward error ideas.

Analyses GEPP for general A, obtaining bound for

‖x − x‖∞ prop. to ‖A−1‖2∞

.

Paper criticized by Bodewig (1949, Math. Rev.): bounds“impractical”.

Num Lin Alg in UK – p. 14/27

James Hardy Wilkinson

Num Lin Alg in UK – p. 15/27

Num Lin Alg in UK – p. 16/27

Pilot Ace

1950 The Pilot ACE at the National Physical Laboratoryruns for the first time.

Num Lin Alg in UK – p. 17/27

Linear Equation Solvers on the Pilot ACE

An interesting feature of the codes isthat they made a very intensive use of subroutines;

the addition of two vectors,multiplication of a vector by a scalar,

inner products, etc.,were all coded in this way

— J. H. Wilkinson

Num Lin Alg in UK – p. 18/27

Quadratic Eigenvalue Problems in Aerospace

Frazer & Duncan, The Flutter of Airplane Wings,1928. The “flutter bible”.

Worked in Aerodynamics Division of NPL.

Developed matrix structural analysis.

Southwell suggested they combine their various papersinto a book.

Frazer, Duncan & Collar, Elementary Matrices &Some Applications to Dynamics and DifferentialEquations, 1938.

“First book to treat matrices as a branch of appliedmathematics”.“After nearly 40 years, still sells in hundreds of copiesper year”.(Collar, 1978).

Num Lin Alg in UK – p. 19/27

Mathematicians on Flutter

Olga Taussky, in Frazer’s group at NPL, 1940s.6 × 6 QEPs from flutter in supersonic aircraft.Used Gershgorin.

Peter Lancaster, English Electric Co., 1950s.QEPs, 2 ≤ n ≤ 20.

Num Lin Alg in UK – p. 20/27

Modern Computing Methods

Editions 1957 and 1961.

Written by NPL Mathematics Divi-sion staff.

Bible of many practising numericalmathematicians.

Latent Roots chapter: powermethod with deflation, Jacobi forsymmetric.

Backward error analysis in secondedition but not first.

Num Lin Alg in UK – p. 21/27

The QR Algorithm

Origins in Rutishauser’s LR alg (1958), based onearlier quotient-difference alg: T = LR, T ′ = RL,T tridiagonal.

J. G. F. Francis (1961) developed the stable QR alg;exploiting Hessenberg form and using double shifts inreal arithmetic.

Kublanovskaya (1961) independently developed thebasic QR alg.

One of the “Top 10 Algorithms” (Computing in Sci. &Eng., 2000).

Num Lin Alg in UK – p. 22/27

References

[1] Donald J. Albers and G. L. Alexanderson, editors.

Mathematical People: Profiles and Interviews.

Birkhauser, Boston, MA, USA, 1985.

[2] L. Fox. Early numerical analysis in the United

Kingdom. In Stephen G. Nash, editor, A History of

Scientific Computing, pages 280–300.

Addison-Wesley, Reading, MA, USA, 1990.

[3] R. A. Frazer, W. J. Duncan, and A. R. Collar.

Elementary Matrices and Some Applications to

Dynamics and Differential Equations. Cambridge

University Press, tenth edition, 1938. 1963 printing.

[4] D. C. Joyce. Survey of extrapolation processes in

Num Lin Alg in UK – p. 23/27

numerical analysis. SIAM Rev., 13(4):435–490, 1971.

[5] National Physical Laboratory. Modern Computing

Methods. Number 16 in Notes on Applied Science.

Her Majesty’s Stationery Office, London, 1957.

[6] Peter Lancaster. Lambda-Matrices and Vibrating

Systems. Pergamon Press, Oxford, 1966. Reprinted

by Dover, New York, 2002.

[7] R. R. M. Mallock. An electrical calculating machine.

Proc. Royal Society, Series A, 140(841):457–483,

1933.

[8] Beresford N. Parlett. The Symmetric Eigenvalue

Problem. Prentice-Hall, Englewood Cliffs, NJ, USA,

Num Lin Alg in UK – p. 24/27

1980. Reprinted by Society for Industrial and Applied

Mathematics, Philadelphia, PA, USA, 1998.

[9] Beresford N. Parlett. The QR algorithm. Computing

in Science and Engineering, 2(1):38–42, Jan/Feb

2000.

[10] Lewis F. Richardson. Weather Prediction by

Numerical Processes. Cambridge University Press,

Cambridge, UK, 1922.

[11] R. V. Southwell. Relaxation Methods in Engineering

Science: A Treatise on Approximate Computation.

Oxford University Press, 1940.

[12] R. V. Southwell. Relaxation Methods in Theoretical

Num Lin Alg in UK – p. 25/27

Physics: A Continuation of the Treatise ‘Relaxation

Methods in Engineering Science’. Oxford University

Press, 1946.

[13] Olga Taussky. How I became a torchbearer for matrix

theory. Amer. Math. Monthly, 95(9):801–812,

November 1988.

[14] William Thomson. On a machine for the solution of

simultaneous equations. Proc. Royal Society,

28:111–113, 1878-1879.

[15] John B. Wilbur. The mechanical solution of

simultaneous equations. J. Franklin Inst.,

222(6):715–724, 1936.

Num Lin Alg in UK – p. 26/27

[16] J. H. Wilkinson. Modern error analysis. SIAM Rev.,

13(4):548–568, 1971.

[17] David M. Young. A historical review of iterative

methods. In Stephen G. Nash, editor, A History of

Scientific Computing, pages 180–194.

Addison-Wesley, Reading, MA, USA, 1990.

Num Lin Alg in UK – p. 27/27