Page 1
Index
A-optimality, 439
absolute error, 484, 494, 526ACM Transactions on Mathematical
Software, 552, 617
adj(·), 69
adjacency matrix, 331–334, 390Exercise 8.20:, 396
augmented, 390
adjoint (see also conjugate transpose),59
adjoint, classical (see also adjugate), 69
adjugate, 69, 598
adjugate and inverse, 118
affine group, 115, 179affine space, 43
affine transformation, 228
Aitken’s integral, 217
AL(·) (affine group), 115algebraic multiplicity, 144
algorithm, 264, 499–513
batch, 512
definition, 509
direct, 264divide and conquer, 506
greedy, 506
iterative, 264, 508–511, 564
reverse communication, 564online, 512
out-of-core, 512
real-time, 512
algorithmically singular, 121Anaconda, 553, 556, 569
angle(·, ·), 37
angle between matrices, 168angle between vectors, 37, 229, 357ANSI (standards), 475, 562, 563Applied Statistics algorithms, 617approximation and estimation, 431approximation of a matrix, 174, 257,
339, 437, 533approximation of a vector, 41–43arithmetic mean, 35, 37Arnoldi method, 318artificial ill-conditioning, 269ASCII code, 460association matrix, 328–338, 357, 365,
366, 369–371adjacency matrix, 331–332, 390connectivity matrix, 331–332, 390dissimilarity matrix, 369distance matrix, 369incidence matrix, 331–332, 390similarity matrix, 369
ATLAS (Automatically Tuned LinearAlgebra Software), 555
augmented adjacency matrix, 390augmented connectivity matrix, 390Automatically Tuned Linear Algebra
Software (ATLAS), 555autoregressive process, 447–450axpy, 12, 50, 83, 554, 555axpy elementary operator matrix, 83
back substitution, 274, 406backward error analysis, 495, 500Banach space, 33Banachiewicz factorization, 255
Page 2
632 Index
banded matrix, 58inverse, 121
Bartlett decomposition, 346base, 467base point, 466basis, 21–23
Exercise 2.6:, 52orthonormal, 40–41
batch algorithm, 512Bauer-Fike theorem, 307Beowulf (cluster computing), 559bias, in exponent of floating-point
number, 468big endian, 480big integer, 466, 492big O (order), 497, 503, 591big omega (order), 497bilinear form, 91, 134bit, 460bitmap, 461BLACS (software), 557, 558BLAS (software), 553–556
CUDA, 560PBLAS, 558PLASMA, 559PSBLAS, 558
block diagonal matrix, 62determinant of, 70inverse of, 121multiplication, 78
BMvN distribution, 219, 548Bolzano-Weierstrass theorem for
orthogonal matrices, 133Boolean matrix, 390Box M statistic, 368bra·ket notation, 24byte, 460
C (programming language), 474, 489,568–569
C++ (programming language), 475,568–569
CALGO (Collected Algorithms of the
ACM), 617cancellation error, 487, 500canonical form, equivalent, 110canonical form, similar, 149canonical singular value factorization,
162
Cartesian geometry, 35, 73catastrophic cancellation, 486Cauchy matrix, 389Cauchy-Schwarz inequality, 24, 98Cauchy-Schwarz inequality for matrices,
98, 178Cayley multiplication, 75, 94Cayley-Hamilton theorem, 138CDF (Common Data Format), 463centered matrix, 288, 363centered vector, 49chaining of operations, 485character data, 461character string, 461characteristic equation, 138characteristic polynomial, 138characteristic value (see also eigenvalue),
135characteristic vector (see also eigenvec-
tor), 135chasing, 317Chebyshev norm, 28chi-squared distribution, 400
noncentral, 400PDF, equation (9.3), 400
Cholesky decomposition, 253–256, 345,436
computing, 558, 574root-free, 255
circulant matrix, 383classification, 389cluster analysis, 389cluster computing, 559coarray (Fortran construct), 563Cochran’s theorem, 353–356, 401cofactor, 68, 598Collected Algorithms of the ACM
(CALGO), 617collinearity, 265, 406, 430column rank, 99column space, 55, 89, 104, 105column-major, 522, 539, 545column-sum norm, 165Common Data Format (CDF), 463companion matrix, 139, 305compatible linear systems, 106compensated summation, 486complementary projection matrix, 356complementary vector spaces, 19
Page 3
Index 633
complete graph, 329
complete pivoting, 275
complete space, 33
completing the Gramian, 177
complex data type, 491
complex vectors/matrices, 33, 132, 387
Conda, 553
condition (problem or data), 499
condition number, 265, 270, 290, 426,427, 499, 502, 523, 533
computing the number, 533, 574
inverse of matrix, 266, 270
nonfull rank matrices, 290
nonsquare matrices, 290
sample standard deviation, 502
conditional inverse, 128
cone, 43–46, 327
Exercise 2.18:, 53
convex cone, 44, 346
of nonnegative definite matrices, 346
of nonnegative matrices, 371
of positive definite matrices, 349
of positive matrices, 371
conference matrix, 381
configuration matrix, 369
conjugate gradient method, 279–283preconditioning, 282
conjugate norm, 93
conjugate transpose, 59, 132
conjugate vectors, 93, 134
connected vertices, 329, 334
connectivity matrix, 331–334, 390
Exercise 8.20:, 396
augmented, 390
consistency property of matrix norms,164
consistency test, 527, 551
consistent system of equations, 105,272, 277
constrained least squares, equalityconstraints, 413
Exercise 9.4d:, 451
continuous function, 186
contrast, 410
convergence criterion, 508
convergence of a sequence of matrices,134, 152, 171
convergence of a sequence of vectors, 32
convergence of powers of a matrix, 171,375
convergence rate, 509convex combination, 12convex cone, 44–46, 346, 349, 371convex function, 26, 197convex optimization, 346convex set, 12convexity, 26coordinate, 4Cor(·, ·), 51correlation, 51, 90correlation matrix, 365, 422
Exercise 8.8:, 394positive definite approximation, 436pseudo-correlation matrix, 437sample, 422
cost matrix, 369Cov(·, ·), 50covariance, 50covariance matrix, see variance-
covariance matrixCRAN (Comprehensive R Archive
Network), 552, 576cross product of vectors, 47
Exercise 2.19:, 54cross products matrix, 256, 358cross products, computing sum of
Exercise 10.18c:, 519Crout method, 245cuBLAS, 560CUDA, 559curl, 192curse of dimensionality, 511cuSPARSE, 560
D-optimality, 439–440, 533daxpy, 12decomposable matrix, 373decomposition, see also factorization of
a matrixadditive, 353, 354Bartlett decomposition, 346multiplicative, 108nonnegative matrix factorization,
257, 337singular value decomposition,
161–164, 320, 336, 425, 532spectral decomposition, 155
Page 4
634 Index
defective (deficient) matrix, 149, 150deficient (defective) matrix, 149, 150deflation, 308–310degrees of freedom, 360, 362, 407, 430del, 192derivative with respect to a matrix, 195derivative with respect to a vector,
189–195det(·), 66determinant, 66–74
as criterion for optimal design, 439computing, 533derivative of, 195Jacobian, 217of block diagonal matrix, 70of Cayley product, 88of diagonal matrix, 70of elementary operator matrix, 86of inverse, 117of Kronecker product, 96of nonnegative definite matrix, 345of partitioned matrix, 70, 122of permutation matrix, 87of positive definite matrix, 347of transpose, 70of triangular matrix, 70relation to eigenvalues, 141relation to geometric volume, 73, 217
diag(·), 56, 60, 596with matrix arguments, 62
diagonal element, 56diagonal expansion, 73diagonal factorization, 148, 152diagonal matrix, 57
determinant of, 70inverse of, 120multiplication, 78
diagonalizable matrix, 148–152, 306,343
orthogonally, 154unitarily, 147, 343, 387
diagonally dominant matrix, 57, 62,100, 348
differential, 188differentiation of vectors and matrices,
183–220digraph, 332
of a matrix, 333dim(·), 14
dimension of vector space, 14dimension reduction, 20, 356, 425direct method for solving linear systems,
272–277direct product (of matrices), 95direct product (of sets), 5direct product (of vector spaces), 20
basis for, 23direct sum decomposition of a vector
space, 19direct sum of matrices, 62direct sum of vector spaces, 18–20, 64
basis for, 22direct sum decomposition, 19
directed dissimilarity matrix, 369direction cosines, 38, 231discrete Fourier transform, 384discrete Legendre polynomials, 380discretization error, 498, 509dissimilarity matrix, 369distance, 32
between matrices, 175between vectors, 32
distance matrix, 369distributed computing, 463, 508, 544distributed linear algebra machine, 558distribution vector, 377div, 192divergence, 192divide and conquer, 506document-term matrix, 336dominant eigenvalue, 142Doolittle method, 245dot product of matrices, 97dot product of vectors, 23, 91double cone, 43double precision, 472, 480doubly stochastic matrix, 377Drazin inverse, 129–130dual cone, 44
Epq, E(π), Ep(a), Epq(a) (elementaryoperator matrices), 84
E(·) (expectation operator), 216E-optimality, 439echelon form, 111edge of a graph, 329EDP (exact dot product), 493effective degrees of freedom, 362, 430
Page 5
Index 635
efficiency, computational, 503–508eigenpair, 134eigenspace, 144eigenvalue, 134–164, 166, 305–322
computing, 306–318, 558, 574Jacobi method, 313–316Krylov methods, 318power method, 311–313QR method, 316–318
of a graph, 391of a polynomial Exercise 3.26:, 181relation to singular value, 163upper bound on, 142, 145, 306
eigenvector, 134–164, 305–322left eigenvector, 135, 158
eigenvectors, linear independence of,143
EISPACK, 556elementary operation, 79elementary operator matrix, 79–87, 101,
242, 273eigenvalues, 137
elliptic metric, 94elliptic norm, 93endian, 480equivalence of norms, 29, 33, 170equivalence relation, 444equivalent canonical factorization, 112equivalent canonical form, 110, 112equivalent matrices, 110error bound, 496error in computations
cancellation, 487, 500error-free computations, 493measures of, 484, 494–497, 526rounding, 487, 495Exercise 10.10:, 517
error of approximation, 498discretization, 498truncation, 498
error, measures of, 272, 484, 494–497,526
error-free computations, 493errors-in-variables, 405essentially disjoint vector spaces, 15, 64estimable combinations of parameters,
409estimation and approximation, 431Euclidean distance, 32, 369
Euclidean distance matrix, 369Euclidean matrix norm (see also
Frobenius norm), 167Euclidean vector norm, 27Euler’s constant Exercise 10.2:, 516Euler’s integral, 593Euler’s rotation theorem, 231exact computations, 493exact dot product (EDP), 493exception, in computer operations, 483,
487expectation, 212–220exponent, 467exponential order, 503exponential, matrix, 153, 184extended precision, 472extrapolation, 509
factorization of a matrix, 108, 112, 147,148, 161, 225–227, 239–259, 272,274
Banachiewicz factorization, 255Bartlett decomposition, 346canonical singular value factorization,
162Cholesky factorization, 253–256diagonal factorization, 148equivalent canonical factorization,
112full rank factorization, 108, 112Gaussian elimination, 272LQ factorization, 247LU or LDU factorization, 240–246nonnegative matrix factorization,
257, 337orthogonally diagonal factorization,
147QL factorization, 247QR factorization, 246–252root-free Cholesky, 255RQ factorization, 247Schur factorization, 147singular value factorization, 161–164,
320, 336, 425, 532square root factorization, 160unitarily diagonal factorization, 147
fan-in algorithm, 485, 506fast Fourier transform (FFT), 386fast Givens rotation, 239, 525
Page 6
636 Index
fill-in, 259, 526Fisher information, 205fixed-point representation, 465flat, 43floating-point representation, 466FLOP, or flop, 505FLOPS, or flops, 505Fortran, 475–478, 505, 546, 566–568Fourier coefficient, 41, 42, 99, 157, 163,
169Fourier expansion, 36, 41, 98, 157, 163,
169Fourier matrix, 384Frobenius norm, 167–169, 171, 175, 314,
340, 369Frobenius p norm, 169
full precision, 480full rank, 100, 101, 104, 111–113full rank factorization, 108
symmetric matrix, 112full rank partitioning, 104, 122
g1 inverse, 128, 130g2 inverse, 128g4 inverse (see also Moore-Penrose
inverse), 128gamma function, 220, 593GAMS (Guide to Available Mathemati-
cal Software), 539Gauss (software), 570Gauss-Markov theorem, 411Gauss-Newton method, 203Gauss-Seidel method, 277Gaussian elimination, 84, 272, 317Gaussian matrix, 84, 241gemm (general matrix-matrix), 556gemv (general matrix-vector), 556general linear group, 114, 133generalized eigenvalue, 160, 319generalized inverse, 124–125, 127–131,
249, 359relation to QR factorization, 249
generalized least squares, 414generalized least squares with equality
constraints Exercise 9.4d:, 451generalized variance, 366generating set, 14, 21
of a cone, 44generation of random numbers, 440
geometric multiplicity, 144geometry, 35, 73, 227, 231Gershgorin disks, 145GitHub, 539
Exercise 12.3:, 581Givens transformation (rotation),
236–239, 317QR factorization, 251
GL(·) (general linear group), 114GMP (software library), 466, 491, 492
Exercise 10.5:, 516GMRES, 282GNU Scientific Library (GSL), 556GPU (graphical processing unit), 559graceful underflow, 470gradient, 189
projected gradient, 207reduced gradient, 207
gradient descent, 198, 199gradient of a function, 190, 191gradual underflow, 470, 487Gram-Schmidt transformation, 39, 40,
524linear least squares, 289QR factorization, 252
Gramian matrix, 115, 117, 256, 289,358–359
completing the Gramian, 177graph of a matrix, 332graph theory, 8, 329–336, 389graphical processing unit (GPU), 559greedy algorithm, 506group, 114, 133GSL (GNU Scientific Library), 556guard digit, 485
Haar distribution, 219, 549Exercise 4.10:, 221Exercise 8.8:, 394
Haar invariant measure, 220Hadamard matrix, 380Hadamard multiplication, 94Hadamard’s inequality Exercise 5.5:,
260, 606Hadoop, 464, 544Hadoop Distributed File System
(HDFS), 464, 514half precision, 480Hankel matrix, 388
Page 7
Index 637
Hankel norm, 389hat matrix, 360, 408HDF, HDF5 (Hierarchical Data
Format), 463HDFS (Hadoop Distributed File
System), 464, 514Helmert matrix, 378, 410Hemes formula, 286, 415Hermite form, 111Hermitian matrix, 56, 60Hessenberg matrix, 59, 316Hessian matrix, 193
projected Hessian, 207reduced Hessian, 207
Hessian of a function, 193hidden bit, 468Hierarchical Data Format (HDF), 463high-performance computing, 507Hilbert matrix, 548Hilbert space, 33, 168Hilbert-Schmidt norm (see also
Frobenius norm), 167Hoffman-Wielandt theorem, 339Holder norm, 27Holder’s inequality Exercise 2.11a:, 52hollow matrix, 57, 369homogeneous coordinates, 232
in graphics applications, Exercise 5.2:,259
homogeneous system of equations, 43,123
Horner’s method, 512Householder transformation (reflection),
233–236, 250, 317hyperplane, 43hypothesis testing, 408
idempotent matrix, 350–357identity matrix, 60, 76IDL (software), 6, 570IEC standards, 464IEEE standards, 464, 487
Standard 754, 472, 480, 487, 494Standard P1788, 493
IFIP Working Group 2.5, 460, 494ill-conditioned (problem or data), 264,
426, 499, 523artificial, 269stiff data, 502
ill-posed problem, 121image data, 461IMSL Libraries, 556, 560–562incidence matrix, 331–334, 390incomplete data, 435–438incomplete factorization, 258, 526independence, linear, see linear
independenceindependent vertices, 390index-index-value (sparse matrices), 548induced matrix norm, 165infinity, floating-point representation,
473, 487, 488infix operator, 489inner product, 23, 245
inner product of matrices, 96–99inner product space, 24
inner pseudoinverse, 128integer representation, 465integration and expectation, 212–220integration of vectors and matrices, 213Intel Math Kernel Library (MKL), 555,
578intersection graph, 334intersection of vector spaces, 18interval arithmetic, 492, 493invariance property, 227invariant distribution, 445invariant vector (eigenvector), 135inverse of a matrix, 107
determinant of, 117Drazin inverse, 129–130generalized inverse, 124–125, 127–131
Drazin inverse, 129–130Moore-Penrose inverse, 127–129pseudoinverse, 128
Kronecker product, 118left inverse, 108Moore-Penrose inverse, 127–129partitioned matrix, 122products or sums of matrices, 118pseudoinverse, 128right inverse, 108transpose, 107triangular matrix, 121
inverse of a vector, 31IRLS (iteratively reweighted least
squares), 297irreducible Markov chain, 444
Page 8
638 Index
irreducible matrix, 311, 335–336,372–377, 445
is.na, 473isnan, is.nan, 473ISO (standards), 475, 562, 563isometric matrix, 167isometric transformation, 227isotropic transformation, 228iterative method, 277, 284, 305,
508–511, 525for solving linear systems, 277–283iterative refinement, 284
iteratively reweighted least squares, 297
Jacobi method for eigenvalues, 313–316Jacobi transformation (rotation), 236Jacobian, 191, 217Jordan block, 77, 139Jordan decomposition, 151Jordan form, 77, 111
of nilpotent matrix, 77
Kalman filter, 502Kantorovich inequality, 350Karush-Kuhn-Tucker conditions, 210kind (for data types), 476Kronecker multiplication, 94–96
inverse, 118properties, 95symmetric matrices, 96, 156
diagonalization, 156Kronecker structure, 219, 419Krylov method, 281, 318Krylov space, 281Kuhn-Tucker conditions, 210Kulisch accumulator, 493Kullback-Leibler divergence, 175
L1, L2, and L∞ normsof a matrix, 165of a symmetric matrix, 167of a vector, 27relations among, 170
L2 norm of a matrix (see also spectralnorm), 166
Lagrange multiplier, 208, 414Exercise 9.4a:, 450
Lagrangian function, 208Lanczos method, 318
LAPACK, 276, 534, 556, 558LAPACK95, 556
Laplace expansion, 68Laplace operator (∇2), 599Laplace operator (∇2), 192Laplacian matrix, 391lasso regression, 430latent root (see also eigenvalue), 135LAV (least absolute values), 295LDU factorization, 240–246leading principal submatrix, 62, 348least absolute values, 295least squares, 200–204, 256, 286–294
constrained, 206–211nonlinear, 202–204
least squares regression, 201left eigenvector, 135, 158left inverse, 108length of a vector, 4, 27, 31Leslie matrix, 378, 446
Exercise 8.10:, 395Exercise 9.22:, 455
Levenberg-Marquardt method, 204leverage, 408
Exercise 9.6:, 451life table, 447likelihood function, 204line, 43linear convergence, 509linear estimator, 409linear independence, 12, 99linear independence of eigenvectors, 143linear programming, 346linear regression, 401–422, 426–431
variable selection, 427LINPACK, 276, 533, 556Lisp-Stat (software), 570little endian, 480little o (order), 497, 591little omega (order), 497log order, 503log-likelihood function, 205Longley data Exercise 9.10:, 453loop unrolling, 565Lorentz cone, 44lower triangular matrix, 57Lp norm
of a matrix, 165of a vector, 27–28, 186
Page 9
Index 639
LQ factorization, 247LR method, 306LU factorization, 240–246
computing, 558, 574
M-matrix, 393MACHAR, 478
Exercise 10.3(d)i:, 516machine epsilon, 470Mahalanobis distance, 94, 365Manhattan norm, 27manifold of a matrix, 55Maple (software), 491, 570MapReduce, 463, 513, 531, 544Markov chain, 443–445Markov chain Monte Carlo (MCMC),
445Mathematica (software), 491, 570Matlab (software), 546, 578–580matrix, 5matrix derivative, 183–220matrix exponential, 153, 184matrix factorization, 108, 112, 147, 148,
161, 225–227, 239–259, 272, 274matrix function, 152matrix gradient, 191matrix inverse, 107matrix multiplication, 75–99, 528
Cayley, 75, 94CUDA, 560Hadamard, 94inner product, 96–99Kronecker, 94–96MapReduce, 531Strassen algorithm, 529–531
matrix norm, 164–171orthogonally invariant, 164
matrix normal distribution, 218matrix of type 2, 58, 383matrix pencil, 161matrix polynomial, 78
Exercise 3.26:, 181matrix random variable, 218–220matrix storage mode, 546–548Matrix Template Library, 569max norm, 28maximal linearly independent subset,
12maximum likelihood, 204–206
MCMC (Markov chain Monte Carlo),445
mean, 35, 37mean vector, 35message passing, 557Message Passing Library, 557metric, 32, 175metric space, 32Microsoft R Open, 578MIL-STD-1753 standard, 478Minkowski inequality, 27
Exercise 2.11b:, 52Minkowski norm, 27minor, 67, 597missing data, 435–438, 571
representation of, 462, 473MKL (Intel Math Kernel Library), 555,
578mobile Jacobi scheme, 315modified Cholesky decomposition, 436“modified” Gauss-Newton, 203“modified” Gram-Schmidt (see also
Gram-Schmidt transformation),40
Moore-Penrose inverse, 127–129, 248,249, 291
relation to QR factorization, 249MPI (message passing interface), 557,
559MPL (Message Passing Library), 557multicollinearity, 265, 406multigrid method, 283multiple precision, 466, 491multiplicity of an eigenvalue, 144multivariate gamma function, 220multivariate linear regression, 418–422multivariate normal distribution,
217–219, 399, 441singular, 217, 432
multivariate random variable, 215–220
N (·), 126NA (“Not Available”), 462, 473, 571nabla (∇), 190, 191Nag Libraries, 556NaN (“Not-a-Number”), 473, 488NetCDF, 463netlib, xiv, 617network, 329–336, 391
Page 10
640 Index
Newton’s method, 198nilpotent matrix, 77, 174NMF (nonnegative matrix factoriza-
tion), 257, 337noncentral chi-squared distribution, 400
PDF, equation (9.3), 400noncentral Wishart distribution
Exercise 4.12:, 222nonlinear regression, 201nonnegative definite matrix, 91, 159,
253, 344–350summary of properties, 344–345
nonnegative matrix, 258, 369nonnegative matrix factorization, 257,
337nonsingular matrix, 100, 111norm, 25–30
convexity, 26equivalence of norms, 29, 33, 170of a matrix, 164–171
orthogonally invariant, 164of a vector, 27–31weighted, 28, 93
normal distribution, 217–219matrix, 218multivariate, 217–219
normal equations, 256, 289, 404, 420normal matrix, 342
Exercise 8.1:, 394circulant matrix, 384
normal vector, 34normalized floating-point numbers, 468normalized generalized inverse (see also
Moore-Penrose inverse), 128normalized vector, 31normed space, 25not-a-number (“NaN”), 473NP-complete problem, 504nuclear norm, 169null space, 126, 127, 144nullity, 126numpy, 556, 569Nvidia, 559
O(·), 497, 503, 591o(·), 497, 591oblique projection, 356Octave (software), 578OLS (ordinary least squares), 288
one vector, 16, 34online algorithm, 512online processing, 512open-source, 538OpenMP, 557, 559operator matrix, 80, 273operator norm, 165optimal design, 438–440optimization of vector/matrix functions,
196–212constrained, 206–211least squares, 200–204, 206–211
order of a graph, 329order of a vector, 4order of a vector space, 15order of computations, 503order of convergence, 497order of error, 497ordinal relations among matrices, 92,
348ordinal relations among vectors, 16orthogonal array, 380orthogonal basis, 40–41orthogonal complement, 34, 126, 131orthogonal distance regression, 299–302,
405orthogonal group, 133, 220orthogonal matrices, binary relation-
ship, 98orthogonal matrix, 131–134, 228orthogonal residuals, 299–302, 405orthogonal transformation, 228orthogonal vector spaces, 34, 131orthogonal vectors, 33
Exercise 2.6:, 52orthogonalization (Gram-Schmidt
transformations), 38, 252, 524orthogonally diagonalizable, 147, 154,
338, 343, 423orthogonally invariant norm, 164, 167,
168orthogonally similar, 146, 154, 164, 168,
269, 343orthonormal vectors, 33out-of-core algorithm, 512outer product, 90, 245
Exercise 3.14:, 179outer product for matrix multiplication,
529
Page 11
Index 641
outer pseudoinverse, 128, 129outer/inner products matrix, 357overdetermined linear system, 124, 255,
286overfitting, 298, 429overflow, in computer operations, 483,
487Overleaf, 539overloading, 11, 63, 165, 479, 489
p-inverse (see also Moore-Penroseinverse), 128
paging, 565parallel processing, 507, 508, 528, 530,
544, 557parallelogram equality, 27parallelotope, 74Parseval’s identity, 41, 169partial ordering, 16, 92, 348
Exercise 8.2a:, 394partial pivoting, 275partitioned matrix, 61, 79, 131
determinant, 70, 122sum of squares, 361, 413, 420
partitioned matrix, inverse, 122, 131partitioning sum of squares, 361, 413,
420PBLAS (parallel BLAS), 558pencil, 161permutation, 66permutation matrix, 80, 86, 273, 377Perron root, 371, 374Perron theorem, 371Perron vector, 371, 375, 445Perron-Frobenius theorem, 374pivoting, 84, 244, 249, 275PLASMA, 559polar cone, 45polynomial in a matrix, 78
Exercise 3.26:, 181polynomial order, 503polynomial regression, 379polynomial, evaluation of, 512pooled variance-covariance matrix, 368population model, 445portability, 481, 494, 540positive definite matrix, 91, 101,
159–160, 253, 346–350, 422summary of properties, 346–348
positive matrix, 258, 369positive semidefinite matrix, 91positive stable, 159, 393power method for eigenvalues, 311–313precision, 472–480, 491
arbitrary, 466double, 472, 480extended, 472half precision, 480infinite, 466multiple, 466, 491single, 472, 480
preconditioning, 282, 310, 525for eigenvalue computations, 310in the conjugate gradient method,
282primitive Markov chain, 445primitive matrix, 375, 445principal axis, 36principal components, 422–426principal components regression, 428principal diagonal, 56principal minor, 72, 104, 598principal submatrix, 62, 104, 241, 344,
347leading, 62, 348
probabilistic error bound, 497programming model, 463, 544projected gradient, 207projected Hessian, 207projection (of a vector), 20, 36projection matrix, 356–357, 408projective transformation, 228proper value (see also eigenvalue), 135PSBLAS (parallel sparse BLAS), 558pseudo-correlation matrix, 437pseudoinverse (see also Moore-Penrose
inverse), 128PV-Wave (software), 6, 570Pythagorean theorem, 27Python, 478, 546, 569, 570
Q-convergence, 509QL factorization, 247QR factorization, 246–252
and a generalized inverse, 249computing, 558, 574matrix rank, 250skinny, 247
Page 12
642 Index
QR method for eigenvalues, 316–318quadratic convergence, 509quadratic form, 91, 93quasi-Newton method, 199quotient space, 115
R (software), 546, 570–578Microsoft R Open, 578Rcpp, 577RcppArmadillo, 577roxygen, 577RPy, 578RStudio, 578Spotfire S+ (software), 578
radix, 467random graph, 337random matrix, 218–220
BMvN distribution, 219computer generation Exercise 4.10:,
221correlation matrix, 441Haar distribution, 219Exercise 4.10:, 221
normal, 218orthogonal Exercise 4.10:, 221rank Exercise 4.11:, 222Wishart, 122, 346, 421, 432Exercise 4.12:, 222
random number generation, 440–442random matrices Exercise 4.10:, 221
random variable, 215range of a matrix, 55rank deficiency, 100, 144rank determination, 532rank of a matrix, 99–122, 250, 431, 532
of idempotent matrix, 351rank-revealing QR, 250, 431statistical tests, 431–435
rank of an array, 5rank reduction, 533rank(·), 99rank, linear independence, 99, 532rank, number of dimensions, 5rank-one decomposition, 164rank-one update, 234, 285rank-revealing QR, 250, 431, 532rate constant, 509rate of convergence, 509rational fraction, 491
Rayleigh quotient, 90, 157, 209, 392Rcpp, 577RcppBlaze, 559RCR (Replicated Computational
Results), 552real numbers, 466real-time algorithm, 512recursion, 511reduced gradient, 207reduced Hessian, 207reduced rank regression problem, 432reducibility, 311, 334–336, 372
Markov chains, 444reflection, 231–233reflector, 233reflexive generalized inverse, 128register, in computer processor, 485regression, 401–422, 426–431regression variable selection, 427regression, nonlinear, 201regular graph, 329regular matrix (see also diagonalizable
matrix), 149regularization, 298, 405, 429relative error, 484, 494, 526relative spacing, 470Reliable Computing, 493Replicated Computational Results
(RCR), 552Reproducible R Toolkit, 578reproducible research, 551–552, 578residue arithmetic, 493restarting, 525reverse communication, 564ρ(·) (spectral radius), 142Richardson extrapolation, 510ridge regression, 270, 362, 405, 418, 429
Exercise 9.11a:, 453right direct product, 95right inverse, 108robustness (algorithm or software), 500root of a function, 487root-free Cholesky, 255Rosser test matrix, 550rotation, 229–232, 236rounding, 473
rounding error, 487, 495row echelon form, 111row rank, 99
Page 13
Index 643
row space, 56
row-major, 522, 539, 545row-sum norm, 166
roxygen, 577
RPy, 578RQ factorization, 247
RStudio, 578
S (software), 570
Samelson inverse, 31sample variance, computing, 501
saxpy, 12scalability, 506
ScaLAPACK, 558
scalar, 11scalar product, 23
scaled matrix, 365
scaled vector, 49scaling of a vector or matrix, 269
scaling of an algorithm, 503, 506Schatten p norm, 169
Schur complement, 121, 412, 421
Schur factorization, 147–148Schur norm (see also Frobenius norm),
167
SDP (semidefinite programming), 346Seidel adjacency matrix, 332
self-adjoint matrix (see also Hermitianmatrix), 56
semidefinite programming (SDP), 346
seminorm, 25semisimple eigenvalue, 144, 149
sequences of matrices, 171
sequences of vectors, 32shape of matrix, 5
shearing transformation, 228Sherman-Morrison formula, 285, 415
shifting eigenvalues, 310
shrinkage, 405side effect, 554
σ(·) (sign of permutation), 66, 86
σ(·) (spectrum of matrix), 141sign bit, 465
sign(·), 16significand, 467
similar canonical form, 149
similar matrices, 146similarity matrix, 369
similarity transformation, 146–148, 313,317
simple eigenvalue, 144simple graph, 329simple matrix (see also diagonalizable
matrix), 149single precision, 472, 480singular matrix, 100singular multivariate normal distribu-
tion, 217, 432singular value, 162, 425, 532
relation to eigenvalue, 163singular value decomposition, 161–164,
320, 336, 425, 532uniqueness, 163
skew diagonal element, 57skew diagonal matrix, 57skew symmetric matrix, 56, 60skew upper triangular matrix, 58, 388skinny QR factorization, 247smoothing matrix, 361, 418software testing, 548–551SOR (method), 279span(·), 14, 21, 55spanning set, 14, 21
of a cone, 44Spark (software system), 544sparse matrix, 59, 259, 277, 523, 526,
556, 557index-index-value, 548software, 557
CUDA, 560storage mode, 548
spectral circle, 142spectral condition number, 268, 269,
290spectral decomposition, 155, 163spectral norm, 166, 169spectral projector, 155spectral radius, 142, 166, 170, 278spectrum of a graph, 392spectrum of a matrix, 141–145splitting extrapolation, 511Spotfire S+ (software), 578square root matrix, 160, 252, 254, 345stability, 276, 500standard deviation, 49, 364
computing the standard deviation,501
Page 14
644 Index
Standard Template Library, 569standards (see also specific standard),
460stationary point of vector/matrix
functions, 198statistical reference datasets (StRD),
551statlib, xiv, 617steepest descent, 198, 199Stiefel manifold, 133stiff data, 502stochastic matrix, 377stochastic process, 442–450stopping criterion, 508storage mode, for matrices, 546–548storage unit, 464, 467, 480Strassen algorithm, 529–531StRD (statistical reference datasets),
551stride, 522, 539, 554string, character, 461strongly connected graph, 334submatrix, 61, 79subspace of vector space, 17successive overrelaxation, 279summation, 485, 486summing vector, 34Sun ONE Studio Fortran 95, 493superlinear convergence, 509SVD (singular value decomposition),
161–164, 320, 336, 425, 532uniqueness, 163
sweep operator, 413Sylvester’s law of nullity, 117symmetric matrix, 56, 60, 112, 153–160,
338–343eingenvalues/vectors, 153–160equivalent forms, 112inverse of, 120summary of properties, 338
symmetric pair, 319symmetric storage mode, 61, 547
Taylor series, 188, 198Template Numerical Toolkit, 569tensor, 5term-document matrix, 336test problems for algorithms or software,
527, 548–551
Exercise 3.24:, 180consistency test, 527, 551Ericksen matrix, 549Exercise 12.8:, 582
Hilbert matrix, 548Matrix Market, 550randomly generated data, 549Exercise 11.5:, 536
Rosser matrix, 550StRD (statistical reference datasets),
551Wilkinson matrix, 550Exercise 12.9:, 582
Wilkinson’s polynomial, 499testable hypothesis, 410testing software, 548–551thread, 544Tikhonov regularization, 299, 429time series, 447–450
variance-covariance matrix, 383, 449Toeplitz matrix, 382, 449
circulant matrix, 383inverse of, 383
Exercise 8.12:, 395Exercise 12.12:, 583
total least squares, 299, 405tr(·), 65trace, 65
derivative of, 195of Cayley product, 87of idempotent matrix, 351of inner product, 92of Kronecker product, 96of matrix inner product, 97of outer product, 92relation to eigenvalues, 141
trace norm, 169transition matrix, 443translation transformation, 232transpose, 59
determinant of, 70generalized inverse of, 124inverse of, 107norm of, 164of Cayley product of matrices, 76of Kronecker product, 95of partitioned matrices, 63of sum of matrices, 63trace of, 65
Page 15
Index 645
trapezoidal matrix, 58, 241, 246triangle inequality, 25, 164
Exercise 2.11b:, 52triangular matrix, 57, 84, 240
determinant of, 70inverse of, 121multiplication, 78
tridiagonal matrix, 58triple scalar product Exercise 2.19c:, 54triple vector product Exercise 2.19d:, 54truncation error, 42, 99, 498twos-complement representation, 465,
483type 2 matrix, 58, 383
ulp (“unit in the last place”), 471underdetermined linear system, 123underflow, in computer operations, 470,
487Unicode, 461union of vector spaces, 18unit in the last place (ulp), 471unit roundoff, 470unit vector, 16, 22, 36, 76unitarily diagonalizable, 147, 343, 387unitarily similar, 146unitary matrix, 132unrolling do-loop, 565updating a solution, 285, 293, 415–417
regression computations, 415–417upper Hessenberg form, 59, 316upper triangular matrix, 57usual norm (see also Frobenius norm),
167
V(·) (variance operator), 49, 216V(·) (vector space), 21, 55Vandermonde matrix, 379
Fourier matrix, 384variable metric method, 199variable selection, 427variance, computing, 501variance-covariance matrix, 216, 219
Kronecker structure, 219, 419positive definite approximation, 436sample, 365, 422
vec(·), 61vec-permutation matrix, 82vecdiag(·), 56
vech(·), 61
vector, 4centered vector, 48
mean vector, 35
normal vector, 34normalized vector, 31
null vector, 15one vector, 15, 34
“row vector”, 89
scaled vector, 49sign vector, 16
summing vector, 15, 34
unit vector, 16zero vector, 15
vector derivative, 183–220vector processing, 557
vector space, 13–15, 17–23, 55, 64, 126,127
basis, 21–23
definition, 13
dimension, 14direct product, 20
direct sum, 18–20direct sum decomposition, 19
essentially disjoint, 15
intersection, 18null vector space, 13
of matrices, 64
order, 15set operations, 17
subspace, 17union, 18
vector subspace, 17
vectorized processor, 508vertex of a graph, 329
volume as a determinant, 73, 217
weighted graph, 329
weighted least squares, 414with equality constraints Exer-
cise 9.4d:, 451
weighted norm, 28, 93Wilk’s Λ, 421
Wilkinson matrix, 550Wishart distribution, 122, 346
Exercise 4.12:, 222
Woodbury formula, 285, 415word, computer, 464, 467, 480
Page 16
646 Index
XDR (external data representation),481
Yule-Walker equations, 449
Z-matrix, 393zero matrix, 76, 99zero of a function, 487zero vector, 15