Applied GeometricalApplied GeometricalMatrix ComputationsMatrix Computations
Alan EdelmanAlan EdelmanDept of Mathematics: MITDept of Mathematics: MIT
MIT Laboratory for Computer ScienceMIT Laboratory for Computer Science
Householder Symposium XV
June 21, 2002
OutlineOutline
• Geometrical Matrix ComputationsGeometrical Matrix Computations• Illustration with 2x2 matrices:Illustration with 2x2 matrices:• Excursions into eigenland (or why tangency Excursions into eigenland (or why tangency
and curvature matter!!)and curvature matter!!)• Where do matrix factorizations come from?Where do matrix factorizations come from?• Application to Color ScienceApplication to Color Science• Matrix AnimationsMatrix Animations
Working definition:Working definition:•Concerns geometry of matrix space Concerns geometry of matrix space
(n(n2 2 dimensions rather than n)dimensions rather than n)•Involves numerical computation (probably MATLAB)Involves numerical computation (probably MATLAB)•Relates to an NLA problemRelates to an NLA problem
Geometrical Matrix Geometrical Matrix ComputationsComputations
Some Other GMC PeopleSome Other GMC People
Absil, Demmel, Elmroth, Huhtanen, Kagstrom, Absil, Demmel, Elmroth, Huhtanen, Kagstrom, Kahan,Kahan,
Lippert, Ma, Mahony, Malyshev, Sepulchre, Lippert, Ma, Mahony, Malyshev, Sepulchre, Tisseur, Trefethen, Van DoorenTisseur, Trefethen, Van Dooren
Vector Space DiagramsVector Space Diagrams•Points are vectors Points are vectors (not matrices!)(not matrices!)•Geometric relationships for vectors, Geometric relationships for vectors, subspaces, and linear transformationssubspaces, and linear transformations
OutlineOutline• Geometrical Matrix ComputationsGeometrical Matrix Computations• Illustration with 2x2 matrices:Illustration with 2x2 matrices:• Excursions into eigenland (or why tangency Excursions into eigenland (or why tangency
and curvature matter!!)and curvature matter!!)• Where do matrix factorizations come from?Where do matrix factorizations come from?• Application to Color ScienceApplication to Color Science• Matrix AnimationsMatrix Animations
Eigenland (in 2d)Eigenland (in 2d)
2 2x z
x -zM = z -x
xx
zz
Isoeig surfaces are hyperbolasIsoeig surfaces are hyperbolas
The Eigenvalue MapThe Eigenvalue Map
/2/2
0000
zz
xx
Zero MatrixZero Matrix
x -zM = z -x
xx
zz2 2x z
cos z x Eigvec Angle
The Eigenvalue MapThe Eigenvalue Map
/2/2
0000
zz
xx
Zero MatrixZero Matrix
x -zM = z -x
xx
zz2 2x z
cos z x Eigvec Angle
•MM• MM
The Eigenvalue MapThe Eigenvalue Map
/2/2
0000
zz
xx
Zero MatrixZero Matrix
x -zM = z -x
xx
zz2 2x z
cos z x Eigvec Angle
• MM •MM??Uniformly
Pseudospectra (Trefethen)Pseudospectra (Trefethen)
2( ) { : eig( ), with || || }A z z A E E C :““z is an eigenvalue of a matrix near A”z is an eigenvalue of a matrix near A”
min{ : ( - ) }z zI A CPseudoportraits = pictures of contours of zPseudoportraits = pictures of contours of z
zzzz
A = 1 -1 0 0 0 1 1 -1 0 0 1 1 1 -1 0 1 1 1 1 -1 0 1 1 1 1 Random PointsRandom PointsPseudoportraitsPseudoportraits
pseudospectra & geometrypseudospectra & geometry
2nC
1A X X
X
ProjectProject
matrix space eig (w/singularity)spectral portrait2nC
OutlineOutline
• Geometrical Matrix ComputationsGeometrical Matrix Computations• Illustration with 2x2 matrices:Illustration with 2x2 matrices:• Excursions into eigenland (or why tangency Excursions into eigenland (or why tangency
and curvature matter!!)and curvature matter!!)• Where do matrix factorizations come from?Where do matrix factorizations come from?• Application to Color ScienceApplication to Color Science• Matrix AnimationsMatrix Animations
Circle/Hyperbola Tangency = High DensityCircle/Hyperbola Tangency = High Density
have eigenvalue distributions with
2 spikes.
eigenvalueeigenvalue
frequency
frequency
have eigenvalue distributions with
4 spikes.
eigenvalueeigenvalue
frequency
frequency
have eigenvalue distributions with
3 spikes.
frequency
frequency
eigenvalueeigenvalue
•Circles tangent to 2 hyperbolas…
* *
•Circles tangent to 4 hyperbolas…
*
**
*
•Circles tangent to 3 hyperbolas…
*
* *
Radius of Curvature = Highest Radius of Curvature = Highest DensityDensity•Circles are tangent to 3
hyperbolas when two tangency points collide
•The circle also shares a radius of curvature with
the hyperbola at this point
•This is even better than tangency, which means a
higher spike
eigenvalueeigenvalue
frequency
frequency
frequency
frequency
eigenvalueeigenvaluefr
equency
frequency
eigenvalueeigenvalue
*
* *
OutlineOutline
• Geometrical Matrix ComputationsGeometrical Matrix Computations• Illustration with 2x2 matrices:Illustration with 2x2 matrices:• Excursions into eigenland (or why tangency Excursions into eigenland (or why tangency
and curvature matter!!)and curvature matter!!)• Where do matrix factorizations come from?Where do matrix factorizations come from?• Application to Color ScienceApplication to Color Science• Matrix AnimationsMatrix Animations
Where do Matrix Factorizations Where do Matrix Factorizations Come From?Come From?
A=UV’
Classical Classical Answer:Answer:
Representation Representation Theory of Theory of
Semisimple Semisimple GroupsGroups
Semisimple group recipeSemisimple group recipe
•Nicely links factorizationsNicely links factorizations•Three ExamplesThree Examples
NonsingularsNonsingulars UnitaryUnitary OrthogonalOrthogonal
SVDSVD UeUeiiV’V’ CS decompCS decomp
SPDSPD Eigen Eigen UeUeiiU’U’ Essentially Sym Essentially Sym OrthOrth
One more exampleOne more example
Hyperbolic Svd as in last talkHyperbolic Svd as in last talk
Group = SO(p,q) ( XJ=JX)Group = SO(p,q) ( XJ=JX)
Matrix FactorizationsMatrix Factorizations
Where can we look for new factorizations?Where can we look for new factorizations?
• The Mathematics LiteratureThe Mathematics Literature– Lie Algebra: Cartan, Iwasawa, BruhatLie Algebra: Cartan, Iwasawa, Bruhat
– Representation Theory: QuiversRepresentation Theory: Quivers
• Nearness ProblemsNearness Problems
• ApplicationsApplications– Engineering: A factorization is useful if someone can use itEngineering: A factorization is useful if someone can use it
– Mathematics: The useful factorizations are characterized by an Mathematics: The useful factorizations are characterized by an abstract criterionabstract criterion
Ideas to GeneralizeIdeas to Generalize
E = (antisymmetric) + (symmetric)E = (antisymmetric) + (symmetric)
M= Q * S [polar]M= Q * S [polar]
expmexpmexpmexpmexpm
Non-singularNon-singular OrthogonalOrthogonal Pos DefinitePos Definite
1: Cartan Decomposition1: Cartan Decomposition
Ideas to GeneralizeIdeas to Generalize
E = (antisymmetric) + (symmetric)E = (antisymmetric) + (symmetric)
M= Q * S [polar]M= Q * S [polar]
expmexpmexpmexpmexpm
Non-singularNon-singular OrthogonalOrthogonal Pos DefinitePos Definite
1: Cartan Decomposition1: Cartan Decomposition
2:KAK Decomposition2:KAK DecompositionPositive Diagonals = Maximal Group Positive Diagonals = Maximal Group
M=UM=UV’V’
Conjugates give S=QConjugates give S=QQ’Q’
Ideas to GeneralizeIdeas to Generalize
E = (antisymmetric) + (symmetric)E = (antisymmetric) + (symmetric)
M= Q * S [polar]M= Q * S [polar]
expmexpmexpmexpmexpm
Non-singularNon-singular OrthogonalOrthogonal Pos DefinitePos Definite
1: Cartan Decomposition1: Cartan Decomposition
2:KAK Decomposition2:KAK DecompositionPositive Diagonals = Maximal Group Positive Diagonals = Maximal Group
M=UM=UV’V’
Conjugates give S=QConjugates give S=QQ’Q’3:Iwasawa, Bruhat3:Iwasawa, Bruhat
Above not unique at I. GivesAbove not unique at I. Gives
M=M=LU, other permutations, totally LU, other permutations, totally positive, etcpositive, etc
Ideas to GeneralizeIdeas to Generalize
E = (antisymmetric) + (symmetric)E = (antisymmetric) + (symmetric)
M= Q * S [polar]M= Q * S [polar]
expmexpmexpmexpmexpm
Non-singularNon-singular OrthogonalOrthogonal Pos DefinitePos Definite
1: Cartan Decomposition1: Cartan Decomposition
2:KAK Decomposition2:KAK DecompositionPositive Diagonals = Maximal Group Positive Diagonals = Maximal Group
M=UM=UV’V’
Conjugates give S=QConjugates give S=QQ’Q’3:Iwasawa, Bruhat3:Iwasawa, Bruhat
Above not unique at I. GivesAbove not unique at I. Gives
M=M=LU, other permutations, totally LU, other permutations, totally positive, etcpositive, etc4:Eigenvalue, Jordan 4:Eigenvalue, Jordan SchurSchur
Step 1:Cartan DecompositionStep 1:Cartan Decomposition•Group: Group: non-singular matricesnon-singular matrices
•Involution: Involution: (((M))=M(M))=M (M(M11MM22)= )= (M(M11))(M(M22))
(M)=M(M)=M-T-T
•Fixed Points Fixed Points (M)=M are a group (M)=M are a group KK
KK = orthogonal matrices = orthogonal matrices•Near INear I
M = (antisymmetric) + (symmetric)M = (antisymmetric) + (symmetric)•Cartan: expmCartan: expm
M = QS (S>0) (polar)M = QS (S>0) (polar)
Step 1:Cartan Decomposition (U/O)Step 1:Cartan Decomposition (U/O)•Group: Group: unitary matricesunitary matrices•Near INear I
M = (antisymmetric) + (i*symmetric)M = (antisymmetric) + (i*symmetric)•Cartan:Cartan:
M= (real orth)(unitary symmetric) M= (real orth)(unitary symmetric)
Step 2:KAK DecompositionStep 2:KAK Decomposition
P P = sym pos def= sym pos def• A A = biggest group inside= biggest group inside P P (abelian)(abelian)
e.g. e.g. diagonal > 0, or conjugates U diagonal > 0, or conjugates UU’ U’ (fix U)(fix U)•KAKKAK
M=UM=UV’V’•P P = union of conjugates= union of conjugates
S=QS=QQ’Q’
Step 2:KAK Decomposition (U/Q)Step 2:KAK Decomposition (U/Q)
P P = unitary symmetric = unitary symmetric • A A = biggest group inside= biggest group inside P P (abelian)(abelian)
e.g. diagonals (ee.g. diagonals (eii) or conjugates ) or conjugates •KAKKAK
M=UeM=UeiiV’ (U, V real orthogonal)V’ (U, V real orthogonal)•P P = union of conjugates= union of conjugates
S=QeS=QeiiQ’ (Q real orthogonal)Q’ (Q real orthogonal)
Step 2:KAK Decomposition (OStep 2:KAK Decomposition (Onn/O/Op p XX OOq q ))
P P = matrices orthogonally similar to ( = matrices orthogonally similar to ( ) )
• A A = biggest group inside= biggest group inside P P (abelian)(abelian)
e.g. e.g. =( ) or conjugates =( ) or conjugates •KAKKAK
The CS DecompositionThe CS Decomposition
CC SS
-S-S CC
CC SS
-S-S CC
MissingMissing•The constructible decompositionsThe constructible decompositions
Tridiagonalization, BidiagonalizationTridiagonalization, Bidiagonalization•The NNMF (Lee, Seung 1999)The NNMF (Lee, Seung 1999)
•V V WH Input: V WH Input: Vijij>0>0
Output: WOutput: Wijij>0 H>0 Hijij>0 (low >0 (low rank)rank)
Algorithm: H Algorithm: H H .* H .* (W’(W’VV)./(W’)./(W’WHWH))
W W W .* W .* ((VVH’)./(H’)./(WHWHH’)H’)
Original Application: EigenfacesOriginal Application: Eigenfaces
Another Example: Color ScienceAnother Example: Color Science
OutlineOutline
• Geometrical Matrix ComputationsGeometrical Matrix Computations• Illustration with 2x2 matrices:Illustration with 2x2 matrices:• Excursions into eigenland (or why tangency Excursions into eigenland (or why tangency
and curvature matter!!)and curvature matter!!)• Where do matrix factorizations come from?Where do matrix factorizations come from?• Application to Color ScienceApplication to Color Science• Matrix AnimationsMatrix Animations
Color Science: Light Spectra from Color Science: Light Spectra from filmfilm
Reds Greens Blues
Grays
wavelength vs densitywavelength vs density
Film Recording and Film Recording and measurementsmeasurements
Reds
• Solid colors sent to film recorder, e.g. redsSolid colors sent to film recorder, e.g. reds
• Negative is produced: film appears as cyansNegative is produced: film appears as cyans
• Negative sent through projector to spectrometerNegative sent through projector to spectrometer
• Energy data at each Energy data at each wavelengthwavelength• Log ratio with no film (only Log ratio with no film (only bulb)bulb)
film density =film density =
log(no film / with film)log(no film / with film)
The DataThe Data• Inputs (r,g,b) for 1Inputs (r,g,b) for 1r,g,b r,g,b 10 scaled (1000 10 scaled (1000 frames)frames)• Output Space: Densities at 400:3:700 nm’sOutput Space: Densities at 400:3:700 nm’s• Data Structure: 101 x 1000 matrix “A”Data Structure: 101 x 1000 matrix “A”• Compute SVD(A)Compute SVD(A)
indexindex
svd
svd
•Project onto best 3 spaceProject onto best 3 space
Three significantThree significant singular valuessingular values
SVD Basis = no physical SVD Basis = no physical meaningmeaning
The NNMF Basis = primary colorsThe NNMF Basis = primary colors
OutlineOutline• Geometrical Matrix ComputationsGeometrical Matrix Computations• Illustration with 2x2 matrices:Illustration with 2x2 matrices:• Excursions into eigenland (or why tangency Excursions into eigenland (or why tangency
and curvature matter!!)and curvature matter!!)• Where do matrix factorizations come from?Where do matrix factorizations come from?• Application to Color ScienceApplication to Color Science• Matrix AnimationsMatrix Animations
Singular 2x2 Matrices (by svd)Singular 2x2 Matrices (by svd)sin
cos 0 sin co
cos cos
s
sinA
sin
and scales cone
_
A=A=(( ))cos cos cos cos cos cos sin sin -sin -sin cos cos -sin -sin sin sin
All isoeig surfaces are translates of the All isoeig surfaces are translates of the =0 surface!=0 surface!
hyperpolas and hyperboloids are cross sections!hyperpolas and hyperboloids are cross sections!
Torus!Torus!
Torus Cone?Torus Cone?
Bohemian DomeBohemian Dome
Linear Algebra with moviesLinear Algebra with movies
A=QA=QQQTT A=Q A=QQ A=QRQ A=QRHopf FibrationHopf Fibration
Horizontal Vertical Horizontal Vertical VillarceauVillarceau
Incorporate 3d graphics tools directly into Matrix Incorporate 3d graphics tools directly into Matrix computations. Include geometry of matrix space.computations. Include geometry of matrix space.
How should this look?How should this look?
Generalize everything and incorporate into softwareGeneralize everything and incorporate into software
ChallengesChallenges